Centre for Social Anthropology and Computing

Comparative and Mixed Strategy Research on distributed bibliographic datasets

David Zeitlyn
July 1994


Contents:

Aims and Objectives: Background

The introduction of BIDS-ISI and the networking of CD-ROM databases have taken many readers conducting bibliographic research from libraries to computer centres and other points of access to JANET. Librarians and others with interests in both the use of JANET and library resources are posing questions with practical, managerial issues at stake as well as those raised by quality assessment exercises, such as those recommended in the Follett Report and discussed in van House et al. (1990). Typical of such questions are:

  1. Who is using the service?
  2. How satisfied are users with the service?
  3. What are the effects of the widespread use of the service for example on ILL, and the use of other databases?
  4. Can a measure of the quality of the service be established vis a vis other online networked databases?
Such questions are too general to be answered by any one research project. The research proposed here comprises three separate component projects that will contribute to the available pool of knowledge. They address both theoretical concerns about how library research can be undertaken and practical issues as summarized by the above questions.

The general methodological orientation is to conduct detailed qualitative research that will complement research currently being undertaken, for example, by the Centre for Communication and Information Studies, University of Westminster (e.g. as summarized in East and Tilson 1993) and the Department of Education, King's College, London (see Squires 1993).

The three studies to be undertaken are:

  • A) Terminal behaviour
  • B) Mixed strategy research on networked databases
  • C) Pilot survey on the use and non-use of BIDS and other databases
    • The project will employ two RA1A Research Assistants for 18 months. They will work under the direction of Dr Zeitlyn (Chair of BIDS Users Group, and Chair of Oxford University Database Working Party, Research Fellow at the Institute of Social and Cultural Anthropology, Oxford).

    Return to contents list

    Summary of past work

    Most research that is undertaken on library users is based on either questionnaire returns or interviewing techniques (e.g.McClure 1993, ) both of which have well-known problems concerning sampling, the representativeness of those who return the questionnaires or agree to be interviewed) as well as the difficulties of accurate self-reporting in interviews. Other research has used a variety of library statistics such as ILL returns (Metz, & Litchfield 1988; Murfin, & Gugelchuk 1987). Only in the last few years have issues of satisfaction by readers been actively canvassed (Van House, McClure, & Weil 1990). Similarly, the problem of lack of success (however defined) and how it can be studied, is one that has been brought to the fore by users of computer-assisted bibliographic aids (such as OPACs) but has always been an issue in library usage (se e.g Burnett 1967, Urquhart and Schofield 1971 and 1972, and more recently Dwyer, 1991). The research to be undertaken as part of this project is intended to avoid many of the sampling problems of questionnaire-based research. The only comparable research of which I am aware of is that current at King's College, London but there are clear differences in scope between the two projects. Other related research including some precedents has been published as (Barbuto, & Cevallos 1991; Case 1988; Childers, Lopata, & Stafford 1991; Mellon 1986)

    Return to contents list

    Outline of project

    A) Terminal Behaviour

    According to data supplied by BIDS the average length of a BIDS session is 15 minutes. Since most sessions take place out of sight of librarians and other staff there is a very real question of what the users are doing. Quarter of an hour is quite a long time for a bibliographic search. This project will use observational techniques to study users of networked databases in the Radcliffe Science Library, Oxford (henceforth RSL). Since the project will use detailed micro-analysis, it is felt that at this stage, at any rate issues of sampling and the representativeness of the sample are not pressing - it can be left to a further study to assess how far the results from Oxford can be generalized, but prima facie it seems unlikely that those using databases in Oxford behave much differently from those in the rest of the country.

    The core of this research will be the analysis, using conversational analytic techniques of video recording of users conducting searches on some of the machines in the RSL. This will exploit the fact that in the machine room of the RSL the use of terminals is mirrored so they can be easily videotaped without disturbing the users.

    The anticipated protocol will be along the following lines:

    1) Observation of terminal use. A sample of a day's use will be made and the time and manner of use of the terminals will be noted. Of particular concern in this stage will be to record the use of resources other than the computer terminal, notes, books and other reference materials during the session.

    2) Recording of sessions. In order to record sessions and to be able to collect a limited amount of information about the users, the following procedure will be used. A session will be recorded from the RSL machine room mirror. An observer in the RSL terminal area will note the types of other material being consulted. As the user leaves the terminal they will be approached and asked if they wish to participate. If they refuse the recording will be deleted and a fresh one made. If they agree they will be asked about their status and the types of database they have been using and those that they usually use. It is possible that an attempt will be made to conduct longitudinal research on the use made of the results in a similar manner to that envisaged in B2a) below, but this may not prove feasible.

    3) Analysis. The recordings will be transcribed paying particular attention to pauses. The transcription will be analysed using conversational analysis techniques. The basic model will be one of dialogue, the user asking questions of the computer and then interpreting the results. Particular focus will be given to the interpretations made by the user of delays and incomplete results.

    4) Interviews. Those that agree to fully participate will be interviewed about the search and the usefulness of the results obtained. Their comments on the transcript will provide a further information to help the analysis.

    Return to contents list

    B) Mixed Strategy Research on BIDS and other distributed datasets

    Comparative study in several sites will allow assessment of the effect on the use and perceived usefulness of BIDS that results from the differing provision of training and other library facilities. It is not practical, at this stage, to undertake a representative random sample of all UK sites. Rather four sites will be studied. The choice of an "old" university, a "new" university, a university founded in the 1960s and a technology-focused site provides a wide range of different provision of resources and users.

    A mixed strategy of research will allow a range of different but interrelated issues to be addressed while allowing for the methodological limitations of different research techniques.

    Research strategies to be used:

    1) Ethnographic background of each site

    2) Focus group interview data.

    Small groups of both librarians and readers will be asked to discuss the general subject of bibliographic databases. The conversation will be held in the presence of one of the researchers who will use non-directive interviewing techniques to ensure that key topics are discussed. Otherwise the conversation will be allowed to develop among the focus group. The resulting conversation will be tape recorded for subsequent analysis. The groups will be recruited by a combination of stratified sampling and self-selection. The issue of representativeness is secondary to the identification of perceived problems. Analysis of focus group data will reveal the semantic domains which librarians and readers use in their encounters with networked databases. The results of such research can be used in the framing of the questionnaires. We will further analyze the transcribed tape-recordings using content and theme analysis. Focus Groups have had some limited use in library research (see Robbins & Holst 1990, Bertrand et al. 1992, Ward et al. 1991 and Widdows et al. 1991).

    3) Studying the use made of searching networked datasets.

    a) Case study Following the example of Brember (1982) a longitudinal study will be of some searches. Readers will be contacted, and the results of their searches recorded. This may be combined with the research undertaken as part of project A). The readers will be asked to comment on the usefulness of the search and later to say how many of the references they located, how many they read and whether they were useful or not. Comparison will be made with other reference sources, e.g. published bibliographies and reading lists. The question of the provision of abstracts may well be relevant at this point. The case study will provide opportunity to examine whether relevant articles are being missed because of poor search strategy (q.v. Blair and Marron 1985). For example, a trained librarian will be asked to repeat the searches allowing for comparison between the two sets of results obtained.

    b) ILL records - these list source of reference - compare with previous years (allowing for changing subscriptions in the local library).

    c) The use of a journal in the library stack - where records exist for calls in previous years - will be studied to see how many calls are made as a result of BIDS usage.

    d) Quality of service. The different forms of ISI data. ISI data is currently available in the following forms: through BIDS, printed paper, CD Rom, online via DIALOG. A small sample of users of different versions of the ISI data will be contacted on each site and asked to complete a short questionnaire designed to elicit information about the reasons for their choice of form, and whether they are aware of and use other forms of the same data. Users of CD Rom databases and local OPACS will be asked to comment on their experiences of different computer mediated databases.

    Return to contents list

    C) Pilot survey on the use and non-use of BIDS

    The failure of potential users to avail themselves of a service is one of the hardest topics to assess. Yet it is of critical importance for those who train users or publicise the services available. It is also of critical importance to those who must justify the costs of subscriptions to services that apparently have little use.

    We will conduct a pilot survey of two small departments (one science, one humanities) aiming to interview all members of staff and all research students so that a profile can be made of the types of bibliographic resources used and how successful the publicity of the different available datasets has been. The questionnaire will be made as short as possible . It is felt that for a pilot project issues of sampling and non-return of posted questionnaires mean that it will be better to interview everyone face to face at least once. Yet to be manageable this will require a short questionnaire. The results will be contextualized by short accounts of the provision and history of publicity of bibliographic resources in the departments included in the study.

    Expected Benefits

    This research will be of direct relevance to those concerned with training readers in the use of databases, and to those making managerial decisions concerning the provision of datasets and the likely use of library facilities such as ILL. It will also provide general methodological pointers as to how such services are to be studied and assessed.

    Dissemination Plans

    The main report on this research will be made available over the international networks in the form of a searchable WWW document. This will ensure wide dissemination to a large number of interested parties, who will be alerted to the existence of the document by postings to relevant discussion lists and bulletin boards (in UK Lis-link and BUBL). This will be in addition to publication in academic journals the results will be disseminated in the form of papers presented to conferences such as those organised by UKOLN, JUGL and BIDS.

    Return to contents list

    References

    Barbuto, D. M., & Cevallos, E. E. 1991. End-User Searching: Program Review and Future
    Prospects. RQ 31(2), 214-227.
    Bertrand, J. T., Brown, J. E., & Ward, V. M. 1992. Techniques for analyzing focus
    group data. Evaluation Review 16, 198-209.
    Blair, D., & Maron, M. E. 1985. An Evaluation of Retrieval Effectiveness for a Full-Text
    Document-Retrieval System. Communications of the Association for Computer Machinery
    28(3), 289-299.
    Brember, V. L. 1982. Investigations of the library usage and information needs of clinical
    medicine and related disciplines. D.Phil., Oxford.
    Burnett, A. D. 1967. Reader failure: a pilot survey. Research in Librarianship
    1(6), 142-157.
    Case, D. O. 1988. The use of anthropological methods in studying information management
    by American historians Paper presented at: 'Asis '88. Information Technology: planning
    for the next fifty years. Proceedings of the 51st Annual Meeting of the American Society
    for Information Science'. C. L. Borgman, E. Y. H. Pai, & Medford.
    Childers, T., Lopata, C., & Stafford, B. 1991. Measuring the Difficulty of Reference
    Questions. RQ 31(2), 237.
    Dwyer, C. M., Gossen, E. A., & Martin, L. M. 1991. Known-Item Search Failure in
    an OPAC. RQ 31(2), 228-236.
    East, H., & Tilson, Y. (1993). The liberated enduser: developments in practice and
    policy for database provision to the academic community. British Library Research and
    Development Report 6100/ CCIS Policy Paper No 4.
    Lancaster, F. W. 1968. Evaluation of the Medlars Demand Search Service Washington
    D.C.: U.S. Department of Health, Education, and Welfare. Public Health Service.
    Leggate, P., Smith, B., Stow, J., & Williams, M. I. (1973). The BA previews project:
    the development and evaluation of a mechanised S.D.I. service for biologists (OSTI
    Report No. 5140). Oxford: Experimental Information Unit.
    McClure, S. 1993. The use of a survey to establish service priorities in the library
    of the National Museums of Scotland. Library Review 42(2), 23-34.
    Mellon, C. A. 1986. Understanding library use from the standpoint of the user:
    naturalistic inquiry for library research. Journal of Educational Media & Library Sciences
    23(4), 348-364.
    Metz, P., & Litchfield, C. A. 1988. Measuring collection use at Virgina Tech. College
    & Research Libraries 49(6), 501-13.
    Murfin, M. E., & Gugelchuk, G. M. 1987. Development and testing of a reference transaction
    assessment instrument. College & Research Libraries 48(4), 314-338.
    Robbins, K., & Holst, R. 1990. Hospital library evaluation using focus group interviews.
    Bulletin Of The Medical Library Association 78, 311-313.
    Squires, D. 1993. The use of information technology to support information access
    in research. In Informatics and Changes in Learning (A-34). IFIP Technical Committe on Education Working Groups 34 (eds.) D. C. Johnson, & B. Samways. Dordrecht: Elsevier.
    Urquhart, J. A., & Schofield, J. L. 1971. Measuring Readers' Failure at the Shelf.
    Journal of Documentation 27(4), 273-286.
    Urquhart, J. A., & Schofield, J. L. 1972. Measuring Readers' Failure at the Shelf
    in three University Libraries. Journal of Documentation 28(3), 233-241.
    Van House, N., McClure, C. R., & Weil, B. 1990. Measuring academic library performance
    : a practical approach London: American Library Association.
    Ward, V. M., Bertrand, J. T., & Brown, L. F. 1991. The comparability of focus group
    and survey results - 3 case-studies. Evaluation Review 15, 266-283.
    Widdows, R., Hensler, T. A., & Wyncott, M. H. 1991. The focus group interview -
    a method for assessing users evaluation of library-service. College & Research Libraries
    52, 352-359.

    Return to contents list

    Return to BLERBS homepage
    Updated 31st May 1996