Accurate Answers to Reference Queries May Be Provided Less Frequently Than Expected

Abstract

A review of:
Hernon, P., & McClure, C. (1986). Unobtrusive reference testing: The 55 percent rule. Library Journal, 111(7), 37-41.


Objective – To determine the number of government documents reference questions that are answered correctly by professional library staff.

Design – The authors utilized unobtrusive reference testing: reference queries posed to library personnel who were unaware they were being evaluated. As opposed to other designs that require the researcher’s presence in the setting, unobtrusive testing utilizes proxies to administer test questions to the subjects, reducing the possibility of reporter bias.

Setting – Twenty-six public and academic libraries participating in the U.S. Government Printing Office Depository Program located in the Western, Southern, and Midwestern United States. The Federal Depository Program consisted of 1400 libraries at the time of the study. One public and one academic library were chosen for each city.

Subjects – Reference and government documents librarians. These two staff types were selected in order to compare the accuracy of each group’s responses to the queries.

Methods – A set of 15 predetermined factual and bibliographic questions were developed by the authors and administered to library staff respondents by proxies. Government documents were selected as the foundation for the test questions. In selecting federal depository libraries for their sample the authors could ensure all queries may hypothetically be answered using U.S. Government Printing Office documents, as all of the libraries would hold the resources in question.

Graduate students enrolled in the University of Arizona and University of Oklahoma library science programs were trained by the authors to serve as proxies. The proxies posed as library users and administered the set of queries at each selected library. Reference librarians and government documents librarians were tested separately, receiving seven and eight questions respectively at each library visited. Over a four-month period a total of 390 questions were posed and their answers recorded.

Main Results – The respondents correctly answered 241 of 390 queries (62 percent). Government documents librarians accurately answered 65 percent of questions, while reference librarians successfully responded to 59 percent. Hernon and McClure derived the “55 percent rule” for reference accuracy from these results and previous unobtrusive studies conducted by both the authors and other researchers. This body of research estimates the rate of accurate answers of factual and bibliographic questions to be between 50 and 62 percent.
Data regarding the “interview and search process” (I&S), defined as the activities between the time a query was posed and when a resolution was provided, also yielded intriguing findings. Regardless of the question asked, the average I&S duration was three to five minutes. Two-thirds of the accurate answers were supplied within three minutes, and 89 percent within five minutes of the initiation of I&S. The duration of I&S did not vary significantly by library type or librarian type.

Reasons for the provision of inaccurate answers included providing the wrong data (64 percent of instances), responding with “don’t know” and ending the interaction (20 percent), or claiming the library did not own a source that would answer the query (15 percent). Other findings included the fact that respondents infrequently offered referrals, which took place in 17 percent of all interactions, and that the three geographic regions studied had an even distribution of correct answers.

Conclusions - Based on their research results as well as those of similar unobtrusive studies, the authors propose two rules regarding reference assistance that apply to public and academic libraries. First, reference librarians correctly answer approximately 55 percent of factual and bibliographic queries received (the 55 percent reference rule). Second, librarians spend no more than five minutes on most factual and bibliographic questions (the five-minute answer rule).

Ultimately, Hernon and McClure recommend reevaluating the centrality of reference services to library operations. If trained professionals are unable to answer more than 60 percent of factual questions correctly, should reference services continue to receive such considerable staffing and funding? Or, alternatively, should libraries increase efforts to improve the accuracy of answers? The authors close by challenging the profession to address the need for strategic assessment of reference effectiveness.