Library Chat Transcript Evaluation for User Sentiment During the COVID-19 Pandemic
DOI:
https://doi.org/10.18438/eblip30642Abstract
Objective – The purpose of this research was to explore user sentiment on Ask a Librarian, a consortial chat service for university libraries in Ontario, Canada, between 2019 to 2021. We tested how the characteristics of the chat (such as year, semester, user type, operator type, affiliation mismatch, and user complaints) and the onset of the COVID-19 pandemic affected sentiment scores.
Methods – The researchers analyzed 3,339 chat transcripts using VADER, a free, open-source Python natural language processing library for sentiment analysis. We tested the significance of relationships between study variables and sentiment score using either a two-samples t-test or ANOVA.
Results – Between 2019 to 2021, overall sentiment on Ask a Librarian was positive and higher among operators than users. There was a significant relationship between sentiment scores and operator type, affiliation mismatch, and complaints respectively. The year, semester, and pandemic status of the chat were also significantly associated with sentiment score. Chats that took place during the COVID-19 pandemic had a significantly higher overall sentiment score than pre-pandemic chats. Average user sentiment score was also higher during the pandemic, but there were no significant differences in average operator sentiment score.
Conclusion – The COVID-19 pandemic had a significant effect on the emotional tone of the overall chat interaction, as well as the sentiment within the user’s messages. Practitioners can replicate our approach to understand user emotions, opinions, attitudes, or appraisals during times of disruption or emergency, as well as for regular service assessment.
Downloads
References
Barrett, K., Crichton, C., & Logan, J. (2024). An analysis of user complaints on chat reference during the COVID-19 pandemic: Insights into user priorities. Internet Reference Services Quarterly. Advance online publication. https://doi.org/10.1080/10875301.2024.2379816
Barrett, K., & Pagotto, S. (2019). Local users, consortial providers: Seeking points of dissatisfaction with a collaborative virtual reference service. Evidence Based Library and Information Practice, 14(4), 2-20. https://doi.org/10.18438/eblip29624
Barrett, K., & Pagotto, S. (2021). Pay (no) attention to the man behind the curtain: The effects of revealing institutional affiliation in a consortial chat service. Partnership, 16(2), 1-21. https://doi.org/10.21083/partnership.v16i2.6651
Brousseau, C., Johnson, J., & Thacker, C. (2021). Machine learning based chat analysis. Code4Lib, 50.
Catalano, A. J., Glasser, S., Caniano, L., Caniano, W., & Paretta, L. (2018). An analysis of academic libraries’ participation in 21st century library trends. Evidence Based Library and Information Practice, 13(3), 4-16. https://doi.org/10.18438/eblip29450
Chen, X., & Wang, H. (2019). Automated chat transcript analysis using topic modeling for library reference services. Proceedings of the Association for Information Science & Technology, 56(1), 368-371. https://doi.org/10.1002/pra2.31
Chow, A. S., & Croxton, R. A. (2014). Usability evaluation of academic virtual reference services. College & Research Libraries, 75(3), 309-361. https://doi.org/10.5860/crl13-408
Cohn, S., & Hyams, R. (2021). Our year of remote reference: COVID19’s impact on reference services and librarians. Internet Reference Services Quarterly, 25(4), 127-144. https://doi.org/10.1080/10875301.2021.1978031
Connaway, L. S., & Radford, M. L. (2011). Seeking synchronicity: Revelations and recommendations for virtual reference. Online Computer Library Center, Inc. https://www.oclc.org/research/publications/2011/synchronicity.html
De Groote, S., & Scoulas, J. M. (2021). Impact of COVID-19 on the use of the academic library. Reference Services Review, 49(3/4), 281-301. https://doi.org/10.1108/RSR-07-2021-0043
Decker, E. N., & Chapman, K. (2022). Launching chat service during the pandemic: Inaugurating a new public service under emergency conditions. Reference Services Review, 50(2), 163-178. https://doi.org/10.1108/RSR-08-2021-0051
George, L. E., & Birla, L. (2018). A study of topic modeling methods. In 2018 Second International Conference on Intelligent Computing and Control Systems (pp. 109-113). IEEE. https://doi.org/10.1109/ICCONS.2018.8663152
Hervieux, S. (2021). Is the library open? How the pandemic has changed the provision of virtual reference services. Reference Services Review, 49(3/4), 267-280. https://doi.org/10.1108/RSR-04-2021-0014
Hutto, C. J. (2014). VADER-Sentiment-Analysis [Repository]. GitHub. https://github.com/cjhutto/vaderSentiment
Hutto, C. J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216-225. https://doi.org/10.1609/icwsm.v8i1.14550
Kathuria, S. (2021). Library support in times of crisis: An analysis of chat transcripts during COVID-19. Internet Reference Services Quarterly, 25(3), 107-119. https://doi.org/10.1080/10875301.2021.1960669
Koh, H., & Fienup, M. (2021). Topic modeling as a tool for analyzing library transcripts. Information Technology and Libraries, 40(3). https://doi.org/10.6017/ital.v40i3.13333
Kohler, E. (2017). What do your library chats say?: How to analyze webchat transcripts for sentiment and topic extraction. In F. Baudino, K. Hart, & C. Johnson (Eds.), 17th Annual Brick & Click: An Academic Library Conference (pp. 138-148). Northwest Missouri State University. https://eric.ed.gov/?id=ED578189
Kohler, E. (2020). Mining library chats: Sentiment analysis and topic extraction. In 14th International Conference on Performance Measurement in Libraries (pp. 220-229).
Lamba, M., & Madhusudhan, M. (2022). Text mining for information professionals: An uncharted territory. Springer. https://doi.org/10.1007/978-3-030-85085-2
Liu, H., Chatterjee, I., Zhou, M., Lu, X. S., & Abusorrah, A. (2020). Aspect-based sentiment analysis: A survey of deep learning methods. IEEE Transactions on Computational Social Systems, 7(6), 1358-1375. https://doi.org/10.1109/TCSS.2020.3033302
Lund, B. D. (2020). Assessing library topics using sentiment analysis in R: A discussion and code sample. Public Services Quarterly, 16(2), 112-123. https://doi.org/10.1080/15228959.2020.1731402
Mawhinney, T. (2020). User preferences related to virtual reference services in an academic library. The Journal of Academic Librarianship, 46(1), 102094. https://doi.org/10.1016/j.acalib.2019.102094
Munip, L., Tinik, L., Borrelli, S., Randone, G. R., & Paik, E. J. (2022). Lessons learned: A meta-synthesis examining library spaces, services and resources during COVID-19. Library Management, 43(1/2), 80-92. https://doi.org/10.1108/LM-08-2021-0070
Murphy, J. E., Lewis, C. J., McKillop, C. A., & Stoeckle, M. (2022). Expanding digital academic library and archive services at the University of Calgary in response to the COVID-19 pandemic. IFLA Journal, 48(1), 83-98. https://doi.org/10.1177/03400352211023067
Osorio, N., & Droog, A. (2021). Exploring the impact of the pandemic on reference and research services: A literature review. New Review of Academic Librarianship, 27(3), 280-300. https://doi.org/10.1080/13614533.2021.1990092
Ozeran, M., & Martin, P. (2019). “Good night, good day, good luck”: Applying topic modeling to chat reference transcripts. Information Technology and Libraries, 38(2), 49-57. https://doi.org/10.6017/ital.v38i2.10921
Paulus, T. M., Wise, A. F., & Singleton, R. (2019). How will the data be analyzed? Part one: Quantitative approaches including content analysis, statistical modeling, and computational methods. In T. M. Paulus & A. F. Wise (Eds.), Looking for insight, transformation, and learning in online talk (pp. 127-159). Routledge. https://doi.org/10.4324/9781315283258
Radford, M. L., Costello, L., & Montague, K. (2020). Chat reference in the time of COVID-19: Transforming essential user services. Association for Library and Information Science Education (ALISE) 2020 Conference. http://hdl.handle.net/2142/108820
Radford, M. L., Costello, L., & Montague, K. E. (2022). “Death of social encounters”: Investigating COVID-19’s initial impact on virtual reference services in academic libraries. Journal of the Association for Information Science and Technology, 73(11), 1594-1607. https://doi.org/10.1002/asi.24698
Schiller, S. Z. (2016). CHAT for chat: Mediated learning in online chat virtual reference service. Computers in Human Behavior, 65, 651-665. https://doi.org/10.1016/j.chb.2016.06.053
Sharma, A., Barrett, K., & Stapelfeldt, K. (2022). Natural language processing for virtual reference analysis. Evidence Based Library and Information Practice, 17(1), 78-93. https://doi.org/10.18438/eblip30014
Sobol, B., Goncalves, A., Vis-Dunbar, M., Lacey, S., Moist, S., Jantzi, L., Gupta, A., Mussell, J., Foster, P. L., & James, K. (2023). Chat transcripts in the context of the COVID-19 pandemic: Analysis of chats from the AskAway consortia. Evidence Based Library and Information Practice, 18(2), 73-92. https://doi.org/10.18438/eblip30291
Turp, C., & Hervieux, S. (2023). Exploring an automated method for the analysis of virtual reference interactions. Reference Services Review. Advance online publication. https://doi.org/10.1108/RSR-05-2023-0050
Walker, J., & Coleman, J. (2021). Using machine learning to predict chat difficulty. College & Research Libraries, 82(5), 683-707. https://doi.org/10.5860/crl.82.5.683
Wang, Y. (2022). Using machine learning and natural language processing to analyze library chat reference transcripts. Information Technology and Libraries, 41(3). https://doi.org/10.6017/ital.v41i3.14967
Watson, A. P. (2023). Pandemic chat: A comparison of pandemic-era and pre-pandemic online chat questions at the University of Mississippi Libraries. Internet Reference Services Quarterly, 27(1), 25-36. https://doi.org/10.1080/10875301.2022.2117757
Yatcilla, J. K., & Young, S. (2021). Library responses during the early days of the pandemic: A bibliometric study of the 2020 LIS literature. Journal of Library Administration, 61(8), 964-977. https://doi.org/10.1080/0193082
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Kathryn Barrett, Ansh Sharma
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
The Creative Commons-Attribution-Noncommercial-Share Alike License 4.0 International applies to all works published by Evidence Based Library and Information Practice. Authors will retain copyright of the work.
