Secret Shopping is an Effective Tool for Identifying Local Patterns in Library User Experience

Abstract

A Review of:
Boyce, C. M. (2015). Secret shopping as user experience assessment tool. Public Services Quarterly, 11(4), 237-253. doi:10.1080/15228959.2015.1084903

Objective – To assess library user experience (UX) at two entry-level service desks to determine the need for, and inform the aspects in which to improve, services and staff training.

Design – Observational study using secret shopping.

Setting – A small, private university in Illinois, United States of America.

Subjects – Library employees, comprised primarily of student assistants; and 11 secret shoppers, comprised of 5 faculty members, 4 staff members, and 2 first-year students from the university.

Methods – Recruitment methods for shoppers consisted of the campus electronic mail list, flyers, directed requests, and a $10 gift certificate to the campus bookstore following their participation. Both groups (library employees and secret shoppers) were briefed on the purpose of the study and completed informed consent forms. Shoppers attended face-to-face training sessions in which they selected two questions to ask from a list of options, one for each service desk in the library. Shoppers were not told at which desk to ask their questions. The list of options included informational and research assistance questions; shoppers also had the option of asking a question based on an actual information need. Two service desks were involved: one for circulation and one for research/IT support. Since IT staff and Librarians were excluded from the study, shoppers were directed to tactfully end the transaction if a referral to an expert was made.

Within two weeks of the training session, shoppers made two separate visits to the library at a time convenient to them to ask the question and observe the transaction at each of the two service desks. For each round of secret shopping, shoppers completed an electronic evaluation form afterward on the Qualtrics platform. The evaluation form consisted of yes/no, multiple-choice, and open-ended comment questions with two questions employing skip logic for a total of 29 possible questions. The questions covered the following variables both quantitatively and qualitatively: how well the question was answered, the customer service skills (responsiveness, approachability, and respectfulness) of the library employee, and if applicable, the quality of the referral to other library staff or services.

Main Results – The shoppers evaluated a total of 21 transactions: 11 for the circulation desk and 10 for the research/IT support desk (1 shopper did not evaluate this desk). Eighteen of the transactions were in-person and three3 were by phone. Eight of the questions asked were based on the participants’ actual information need.

On the variable of satisfaction with the answer received, the research/IT support desk scored higher than the circulation desk. The circulation desk received 7 very satisfactory ratings, 3 satisfied ratings, and 1 neutral rating; whereas the research/IT support desk received 10 very satisfactory ratings and 1 satisfied rating. The lower scores of the circulation desk may be related to the variables of responsiveness and approachability, as library employees on the circulation desk were scored lower in these areas and observed as not paying attention in two interactions and this was not observed at all at the research/IT support desk. However, the study did not collect sufficient data to test this relationship. All shoppers gave positive ratings on whether they were treated respectfully and if the library employee waited for the shopper to state the question fully and with the exception of one transaction, clarify the question if necessary.
Responses to the open-ended comment questions were reviewed by investigators, who found that in five cases the transaction would have been improved by consulting library faculty. On the variables related to customer service, responses were generally positive but in several transactions the library employees failed to appear attentive and ready to help the shopper.

Conclusion – The author found secret shopping was an effective tool for evaluating library UX to identify both positive and negative patterns and better inform responses to areas in need of improvement. The author identified two areas for improvement to the library. First, library employees at the circulation desk require additional training that would encourage them to refer transactions to library faculty where necessary. Second, although evaluation of customer service skills was generally positive, library employees will also receive additional training that will emphasize listening and role-playing scenarios. These areas for improvement will also support the library’s plans to combine research and circulation functions into a single service desk on the entry level and move the IT support desk to the third floor.