Users of Virtual Reference Are More Satisfied with the Service They Receive than the Providers of that Service Think They Are

Abstract

Objective – To examine provider pessimism by comparing user and provider perception of the same instant messaging reference transaction.

Design – Instant messaging users and providers completed post-reference transaction surveys which were analyzed using the pair perception comparison method.

Setting – A large research university in the United States.


Subjects – Two hundred undergraduate journalism students (users of the instant messaging service) and 51 Master of Library Science (MLS) students enrolled in a reference services class (providers of the instant messaging service).

Methods – The authors created a research help webpage from which users could access the instant messaging service. Prior to service availability, providers received reference instruction and demonstrated reference aptitude through in-class activities. The authors briefed providers on the project and provided a wiki containing resources they might need during reference transactions. Providers worked in two-hour shifts, and two providers were available during each shift. The service was available for one week while potential users completed a journalism assignment. The authors asked both users and providers of the service to complete an online survey at the conclusion of the reference transaction. Users and providers completed different surveys, but both types included the following four elements: questions to aid in matching a user to a provider; questions about satisfaction with the service based on guidelines put forth by the Reference and User Services Association (RUSA); open-ended questions about the reference transaction; and questions regarding demographics, prior reference service usage, and knowledge of instant messaging. There were 55 valid reference transactions, and from those, the authors matched 26 pairs of user and provider surveys. The authors analyzed paired surveys to (a) compare the user’s perception of the reference transaction with the provider’s guess about the user’s perception and (b) compare the provider’s self-perception of the reference transaction with the provider’s guess about the user’s perception. The authors introduced the pair perception comparison method for the analysis using two-tailed paired t-tests and Wilcoxon signed-rank tests.

Main Results – Analysis of background information showed that users were younger on average than providers and used instant messaging more frequently. Even so, most users and providers felt comfortable with instant messaging.

When providers were asked to guess how satisfied overall they thought the user was with the reference transaction, they reported on average that the user was less satisfied than the user actually was. These results were statistically significant. The authors found no significant difference between the providers’ overall satisfaction with the service they provided and their guesses about the users’ overall satisfaction.

The authors also analyzed the matched pairs on 14 specific aspects of satisfaction gathered from surveys. When comparing the users’ satisfaction with the service they received and the providers’ guesses about the users’ satisfaction, the providers underestimated the users’ satisfaction on average for all 14 dimensions. The authors found statistically significant differences with regard to 7 of the 14 dimensions: tempo, ease of use, friendliness, understanding, accuracy, follow up, and spelling. When comparing the providers’ satisfaction with a given reference transaction and their guesses about the users’ satisfaction with the same reference transaction, the authors found significant differences for 3 of the 14 dimensions: interest, resource type, and accuracy.

Conclusion – This study has shown for instant messaging reference what other studies have shown for face-to-face reference—that provider pessimism exists. Whatever the environment, providers of reference tend to judge themselves more harshly than the people they are helping judge them. Based on a review of the literature, the authors further note that both expert and novice reference providers experience such pessimism. The authors are hopeful that providers will view these results as evidence of their own competence during instant messaging reference transactions.

The results of this study provide valuable information for training instant messaging providers. For example, the fact that providers thought users were less satisfied with the tempo of the reference transaction than they really were suggested to the authors that instant messaging providers need not be so concerned about giving a quick answer. An accurate answer is more desirable. In the same vein, providers thought that users were more concerned with spelling than they really were. Both of these cases, and others gleaned from the results, provide insight into what aspects of the reference transaction providers should spend their time and effort on.

Finally, the authors introduced the pair perception comparison method to compare feedback from matched pairs on individual reference transactions, a methodology not used in any earlier studies. They deemed this method to be an effective way to uncover biases and false assumptions.