Research Article
Library Chat Transcript Evaluation for User Sentiment
During the COVID-19 Pandemic
Kathryn
Barrett
Liaison
Librarian
University
of Toronto Scarborough Library
Scarborough,
Ontario, Canada
Email:
kathryn.barrett@utoronto.ca
Ansh Sharma
Computer
Science Student
University
of Toronto Scarborough
Scarborough,
Ontario, Canada
Email:
ansh.sharma@alumni.utoronto.ca
Received: 1 Oct. 2024 Accepted: 17 Mar. 2025
2025 Barrett and Sharma. This is an Open Access article
distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0
International (http://creativecommons.org/licenses/by-nc-sa/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.
DOI: 10.18438/eblip30642
Abstract
Objective
– The purpose of this research was to explore user
sentiment on Ask a Librarian, a consortial chat
service for university libraries in Ontario, Canada, between 2019 to 2021. We
tested how the characteristics of the chat (such as year, semester, user type,
operator type, affiliation mismatch, and user complaints) and the onset of the
COVID-19 pandemic affected sentiment scores.
Methods – The
researchers analyzed 3,339 chat transcripts using VADER, a free, open-source
Python natural language processing library for sentiment analysis. We tested
the significance of relationships between study variables and sentiment score
using either a two-samples t-test or ANOVA.
Results
– Between 2019 to 2021, overall sentiment on Ask a
Librarian was positive and higher among operators than users. There was a
significant relationship between sentiment scores and operator type,
affiliation mismatch, and complaints respectively. The year, semester, and
pandemic status of the chat were also significantly associated with sentiment
score. Chats that took place during the COVID-19 pandemic had a significantly
higher overall sentiment score than pre-pandemic chats. Average user sentiment
score was also higher during the pandemic, but there were no significant differences
in average operator sentiment score.
Conclusion – The COVID-19 pandemic had a significant effect on the emotional tone
of the overall chat interaction, as well as the sentiment within the user’s
messages. Practitioners can replicate our approach to understand user emotions,
opinions, attitudes, or appraisals during times of disruption or emergency, as
well as for regular service assessment.
Introduction
With the onset of the COVID-19 pandemic in March of
2020, academic libraries experienced an immediate and significant disruption to
their operations. As colleges and universities across North America closed
their physical spaces and shifted courses online, academic libraries switched
to online service delivery, including launching or expanding virtual reference
services (Radford et al., 2020; Yatcilla & Young,
2021). Chat reference services were particularly well-positioned to play a role
in pandemic response. Live chat offers synchronous assistance to users in the
online environment, and it meets user preferences for convenience, efficiency,
and personal and informal communication styles (Chow & Croxton,
2014; Connaway & Radford, 2011; Mawhinney, 2020).
Unsurprisingly, many chat services saw surges in demand and rises in chat
volume throughout the pandemic (Radford et al., 2022; Osorio & Droog, 2021).
While some academic libraries launched new online
reference services to respond to the pandemic (Decker & Chapman, 2022),
most already had a virtual reference service in place prior to COVID-19 (Cohn
& Hyams, 2021; Osorio & Droog,
2021). For example, a 2018 survey of ARL libraries found that 91% offered some
form of virtual reference (Catalano et al., 2018). Consequently, the pandemic
transition for most libraries involved improving access to existing virtual
reference services. Common strategies included training new chat operators,
increasing shifts, expanding service hours, drafting best practice documents, creating
new triaging workflows, implementing new features for the chat software, and
making the chat service more prominent (Cohn & Hyams,
2021; Murphy et al., 2022; Osorio & Droog, 2021).
Researchers are beginning to explore how the
COVID-19 pandemic changed the nature of chat reference interactions themselves,
such as volume, temporal distribution, duration, type, complexity,
instructional content, number of complaints, and relational aspects of chat
questions (Barrett et al., 2024; De Groote & Scoulas,
2021; Hervieux, 2021; Munip et al., 2022; Radford et
al., 2022; Watson, 2023). For example, Radford et al. (2022) described shifting
levels of deference, including politeness and expressions of gratitude and
frustration, in user messages during the pandemic. Our study aims to contribute
to this literature by extending our understanding of how the pandemic affected
the emotional tenor of chat interactions. We conducted a sentiment analysis of
chat transcripts from a large, consortial chat
service in Ontario, Canada, and compared chats from the pre-pandemic period in
2019 to pandemic-era chats from 2020 and 2021.
Literature Review
An understanding of user needs enables librarians to
provide efficient and accurate reference services. Reviewing activity on the
library’s various reference services can help staff to identify common patron
needs. Given that chat reference generates and preserves a large volume of data
in the form of chat records and transcripts, librarians can review this data to
identify common user needs and ensure that chat personnel can receive
appropriate training to provide high quality service (Wang, 2022).
Historically, researchers have used qualitative methods to identify chat
trends, such as hand-coding chat transcripts, but these methods are
time-consuming and ill-suited to the large datasets generated by live chat
(Chen & Wang, 2019). Consequently, researchers are beginning to explore
automated, computational approaches to analysis, such as text mining and
machine learning, often using natural language processing techniques (Kohler,
2020; Paulus et al., 2019).
Several researchers have conducted studies to
explore automated methods for the topical analysis of virtual reference
records. For example, Brousseau et al. (2021) used a supervised machine
learning model to code transcripts, and Turp &
Hervieux (2023) used regular expressions to identify themes in virtual
reference. One common approach in the literature is topic modeling, a natural
language processing technique that reveals the hidden structure within
documents by grouping words with similar meanings and separating words with
different meanings (George & Birla, 2018). Several researchers have
conducted studies to explore the viability and application of different topic
modeling techniques to chat reference data. For example, Ozeran
and Martin (2019) tested different algorithms for topic modelling and
determined that Latent Dirichlet Allocation, Phrase-Latent Dirichlet
Allocation, and Non-Negative Matrix Factorization were the most promising for
large datasets. Koh and Fienup (2021) qualitatively
measured the accuracy and interpretability of different topic modelling
techniques and judged that Probabilistic Latent Semantic Analysis performed the
best. Sharma et al. (2022) incorporated a mix of targeted searching for query
terms using regular expressions and natural language processing using the spaCy library and found that it was effective for topical
analysis of chat transcripts.
Other researchers have applied topic modelling
techniques to learn about aspects of their chat services. Schiller (2016)
explored the learning taking place on Wright State University’s chat reference
service using a mix of manual and automated coding using a text mining
software, finding that two teaching styles, “give fish” and “teach fishing,”
are constructed in the process of mediated learning within the chat
interaction, which is facilitated by the chat technology and the social
environment. Kohler (2017) used topic extraction algorithms to identify popular
chat topics, with the results showing that general help, database searching,
interlibrary loan requests, catalogue searching, and login information were
common topics. Walker and Coleman (2021) predicted the difficulty of incoming
chat questions using machine learning and natural language processing techniques, and found that the predictive power of the
modeling processes was statistically significant. Recently, researchers have
also used topic modeling to understand how the COVID-19 pandemic affected the
nature of chat topics, finding that the content of questions remained largely
unchanged (Sobol et al., 2023).
Another popular computational approach employing
natural language processing is sentiment analysis. Also known as opinion
analysis or opinion mining, sentiment analysis extracts patterns of information
from textual data based on the author’s emotions, such as their thoughts,
attitudes, views, opinions, beliefs, or preferences (Lamba & Madhusudhan, 2022). Sentiment analysis extracts feelings in
the form of polarity, measured on a scale of -1 (very negative) to +1 (very
positive), with 0 representing neutrality (Lamba & Madhusudhan,
2022). Sentiment analysis has many applications in business, because it can be
applied to customer reviews to detect changes in client opinion and improve
customer support (Liu et al., 2020). Within libraries, it can be applied to
data from patron feedback, reference transactions, and social media to provide
insights about user satisfaction (Lamba & Madhusudhan,
2022).
There is a small but growing body of literature
about sentiment analysis in libraries. While one study reported on a sentiment
analysis of library tweets (Lund, 2020), the majority of sentiment analysis
research has examined chat transcripts. For example, Kohler (2017) found that
sentiment was overwhelmingly positive on Greenlease
Library’s chat service, while Brousseau et al. (2021) determined that the
number of satisfied chats at Brigham Young University Library decreased over a
three-year period. Several recent studies have looked at the impact of the
COVID-19 pandemic on user sentiment, with mixed results. Kathuria
(2021) conducted sentiment analysis on Georgia State University’s chat
transcripts from 2019 to 2020, finding that overall sentiment was much lower
during the pandemic. There was a spike in positive words early during the
COVID-19 pandemic, but sentiment dropped during summer and fall of 2020. Kohler
(2020) used the VADER sentiment analysis tool to evaluate chat transcripts from
2020 at Virginia Tech and found that sentiment scores were overwhelmingly
positive, with the small group of negative chats mainly being cases of an
inherently negative research topic or lack of access to specific resources. Sobol et al. (2023) used the Linguistic Inquiry and Word
Count tool for sentiment analysis of transcripts from a consortial
chat service covering 2019-2020 and 2020-2021. Overall, the emotional tone of
chats was positive, and higher in the messages of patrons than providers.
During the pandemic, the positive language of chat providers declined, while
sentiment scores for patrons had a small increase.
Aims
The aim of this
research was to explore user sentiment on the Ask a Librarian chat service
between 2019 to 2021, with a particular focus on how the characteristics of the
chat and the onset of the COVID-19 pandemic affected sentiment scores. We
sought to answer the following research questions:
1.
What is the average sentiment score on
Ask a Librarian?
2.
Do average user and operator sentiment scores
differ?
3.
Are there significant differences in
sentiment score based on user or operator type?
4.
Does an affiliation mismatch between the
user and operator affect sentiment scores?
5.
Did sentiment scores vary by year or
semester?
6.
Was there a significance difference
between pre-pandemic and pandemic sentiment scores?
7.
How does the presence of a complaint in
the chat transcript affect sentiment scores?
Methods
Background and Setting
Scholars Portal is the digital services arm of the
Ontario Council of University Libraries (OCUL), a consortium representing the
libraries of the 21 universities in the province of Ontario, Canada. Scholars
Portal manages Ask a Librarian, a collaborative chat service offering real-time
library- and research-related assistance from librarians, paraprofessional
library staff, and graduate student employees. The service is offered at 16
participating universities for 67 hours per week during the academic year,
reaching approximately 445,000 full-time equivalent students, and receiving
over 25,000 chats a year.
The researchers received approval for this study
from the Research Ethics Board of the University of Toronto, the home
institution of the authors, in addition to Scholars Portal’s Ask a Librarian
Research Data Working Group. Users are informed that their chat data can be
used for research purposes through Ask a Librarian’s privacy policy, and
operators are informed during training.
Data Collection, Sampling, and Preparation
This research study employed two approaches to
transcript analysis: manual coding for select variables and natural language
processing for sentiment analysis. Manual coding was performed to enable us to
determine if characteristics of the chat interaction were associated with the
chat’s sentiment score. For hand-coding to be achievable for the research team,
we selected a sample of chats rather than analyzing the entire corpus from the
study period.
All English-language chats that took place between
January 1, 2019, to December 31, 2021, were eligible for sampling. This study
excluded French-language chats (due to the language skills of the research
team) and text message (SMS) interactions. In total, 124,080 eligible chats
occurred over this period.
The researchers downloaded a metadata spreadsheet
for the eligible chats from LibraryH3lp, the chat software. After removing
identifying information about the user and operator, we created new variables
in the spreadsheet to record the year and the semester that the chat took
place. We operationalized the winter semester as the months of January – April,
summer as May – August, and fall as September – December. Through this process,
each chat was assigned to one of 9 possible semesters from the study period.
To create samples for each of the 9 semesters, we
used Excel to randomly select chats according to their unique ID in the
metadata spreadsheet. Sample sizes were calculated for each semester to achieve
a 95% confidence level. Overall, we selected 3,339 chats from the 9 semesters
across the three-year study period.
Variable Creation and Coding
To determine whether each chat took place before or
during the pandemic, we created a variable in the metadata spreadsheet to
record whether the chat took place before or after the World Health Organization
declared COVID-19 a pandemic. Pre-pandemic chats occurred on March 10, 2020, or
earlier. Pandemic-era chats occurred on or after March 11, 2020.
One team member (KB) hand-coded two additional
variables by reviewing the complete transcript of each sampled chat:
1.
User type: This variable
referred to the user’s status at the university. It was coded based on the
user’s response to an auto-generated prompt at the beginning of the chat
requesting that they share information about themself. The options were:
undergraduate student, graduate student, faculty member, staff member, alumni,
member of the public, or other. If the user did not respond to the prompt,
their type was recorded as unknown.
2.
Complaint: This variable
recorded whether there was at least one complaint present within the chat
transcript, which we defined as any expression of grievance, dissatisfaction,
injustice, or wrong suffered on the part of the patron. This could be any
statement from the user that something had gone wrong, was not good enough, was
unsatisfactory, or was unacceptable. Given the subjectivity of identifying
complaints, we chose to be inclusive and coded problems encountered by users as
complaints.
Sentiment Analysis
We used VADER (Hutto & Gilbert, 2014), a Python
natural language processing library, to analyze the chat transcripts. VADER is
a simple rule-based model for general sentiment analysis. It can be used for
text across domains, but it performs especially well in the analysis of social
media text. We selected VADER because it is a free and open
source tool, and because it is especially attuned to sentiments
expressed in social media, which made it a good fit for our corpus of online
chat data.
The VADER library processed a .csv file made up of
rows for each chat, with columns containing the metadata fields and
corresponding transcript. The text within each transcript was analyzed by
parsing every message within the interaction and assigning each message a
score.
The toolchain distinguished whether a particular
message was sent by the user or the operator through the content of the message
in the chat transcript. Messages beginning with the system-generated operator
tag (automatically included in LibraryH3lp chat transcripts) were assumed to be
sent from the operator. Messages beginning with the guest identification string
(automatically assigned by the LibraryH3lp platform) were assumed to be sent
from the user.
The toolchain processed the data and exported a .csv
spreadsheet with its output. Identifying data was automatically removed from
the spreadsheet by the toolchain, including metadata fields related to the user
and operator, as well as the complete text of the transcript.
The output spreadsheet added several new fields for
sentiment score:
1.
Average VADER user sentiment score: mean
sentiment score calculated by VADER for all messages sent by the user within
the chat transcript
2.
Average VADER operator sentiment score: mean
sentiment score calculated by VADER for all messages sent by the operator
within the chat transcript
3.
Average VADER overall sentiment score: calculated
by the researchers, the mean of the combined average user and operator
sentiment scores, reflecting the overall sentiment across all the messages
within the chat transcript
The toolchain also processed and recorded two
additional variables in the output spreadsheet:
1.
Operator type: This
variable referred to the operator’s position within the library and was
determined based on the operator username in the chat metadata. The toolchain
looked up the username in a spreadsheet containing each active operator’s role
at their home library and recorded the response in a new column. The options
were: librarian, library technician, student employee, or unknown. In the Ask a
Librarian context, librarians have graduate degrees in library or information
science, technicians have a college diploma for library and information
technicians (some may also have an advanced degree in LIS), and student
employees are graduate students enrolled in a library or information science
program who have received reference training.
2.
Affiliation mismatch: This
variable recorded whether the user and operator were affiliated with the same
institution. The toolchain compared the queue through which the chat was
submitted (the user’s university) and the operator’s username (which includes a
suffix for their university) in the chat metadata. If they were affiliated with
the same institution, the chat was recorded as an affiliation match. If they
were not, it was recorded as an affiliation mismatch.
Data Compilation and Data Analysis
We merged the spreadsheets containing the chat
metadata, the constructed and hand-coded variables, and the VADER output into a
single spreadsheet based on unique chat ID.
In IBM SPSS Statistics, we generated descriptive
statistics and tested the significance of the relationships between variables
and sentiment score using two-sample t-tests and Analysis of Variance (ANOVA).
A two-samples t-test compares the means of two groups to determine whether the
associated population means are significantly different. ANOVA is a statistical
test used to determine whether there is a significant difference in the means
of more than two groups.
When interpreting results, we used typical threshold
values for VADER to determine if sentiment scores were positive, neutral, or
negative (Hutto, 2014):
1.
Positive sentiment: >= 0.05
2.
Neutral sentiment: < 0.05 and >
-0.05
3.
Negative sentiment: <= -0.05
Results
Average Sentiment Scores
The mean overall VADER sentiment score on Ask a
Librarian between 2019 to 2021 was positive, M = 0.213. Average
sentiment was higher for operators than users (see Table 1).
Table 1
Average Sentiment Scores on Ask a Librarian, 2019 to
2021
|
Sentiment
Score |
M |
SD |
|
Average
Overall |
0.213 |
0.120 |
|
Average
Operator |
0.236 |
0.175 |
|
Average
User |
0.195 |
0.150 |
Association Between User and Operator Type and Sentiment Score
An ANOVA test showed that user type was not significantly
associated with average overall sentiment scores (p = .498). User type
approached but did not meet significance for average user sentiment scores (p
= .059) and for average operator sentiment scores (p = 0.06). For
details, see Table 2.
Table
2
AVOVA
for Sentiment Score and User Type
|
Sentiment
Score |
Undergrad |
Graduate |
Faculty |
Staff |
Member of Public |
Alumni |
Unknown |
Other |
df |
F |
||||||||
|
|
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
|
|
|
Average
Overall |
0.215 |
0.115 |
0.213 |
0.107 |
0.212 |
0.126 |
0.195 |
0.113 |
0.234 |
0.129 |
0.199 |
0.099 |
0.213 |
0.132 |
0.208 |
0.122 |
7, 2832 |
0.909 |
|
Average
Operator |
0.241 |
0.163 |
0.23 |
0.15 |
0.237 |
0.174 |
0.17 |
0.145 |
0.233 |
0.188 |
0.21 |
0.16 |
0.242 |
0.198 |
0.233 |
0.156 |
7, 3086 |
1.934 |
|
Average
User |
0.193 |
0.138 |
0.199 |
0.134 |
0.191 |
0.145 |
0.229 |
0.147 |
0.235 |
0.148 |
0.2 |
0.132 |
0.189 |
0.17 |
0.195 |
0.16 |
7, 3072 |
1.944 |
An ANOVA test determined that operator type was
significantly associated with overall average sentiment scores (p < .001).
Mean sentiment was lowest among the library technician group and highest among
the student employee operator group. Operator type was not significantly
associated with average user sentiment scores (p = .972), but it was
significantly related to average operator sentiment scores (p < .001). Mean
operator sentiment scores were lowest among library technicians and highest
among the student employee group. For details, see Table 3.
Table
3
AVOVA
for Sentiment Score and Operator Type
|
Sentiment
Score |
Librarian |
Library Technician |
Student Employee |
Unknown |
df |
F |
||||
|
|
M |
SD |
M |
SD |
M |
SD |
M |
SD |
|
|
|
Average
Overall |
0.208 |
0.119 |
0.2 |
0.119 |
0.232 |
0.121 |
n/a |
n/a |
2, 2837 |
18.299*** |
|
Average
Operator |
0.217 |
0.166 |
0.208 |
0.157 |
0.268 |
0.158 |
0.284 |
0.279 |
3, 3090 |
29.957*** |
|
Average
User |
0.196 |
0.140 |
0.195 |
0.153 |
0.196 |
0.156 |
n/a |
n/a |
2, 3077 |
0.028 |
*** p < 0.001
Association Between Affiliation Mismatch and Sentiment Score
A
two-samples t-test showed that mean VADER overall sentiment score was
significantly lower in chats in which there was an affiliation mismatch between
the user and the operator compared to chats in which the user and operator were
from the same institution (see Table 4). In chats with affiliation mismatches,
average sentiment scores were lower for both user messages and operator
messages.
Table
4
Two-Samples
T-Test for Sentiment Score and Affiliation Mismatch
|
Sentiment
Score |
Match |
Mismatch |
df |
t |
p |
||
|
|
M |
SD |
M |
SD |
|
|
|
|
Average
Overall |
0.222 |
0.123 |
0.205 |
0.118 |
2826.852 |
3.742 |
< .001*** |
|
Average
Operator |
0.243 |
0.171 |
0.224 |
0.166 |
2958 |
3.019 |
0.003** |
|
Average
User |
0.203 |
0.149 |
0.187 |
0.151 |
3071 |
2.968 |
0.003** |
** p < 0.01
*** p < 0.001
Association Between Year, Semester, and Pandemic Status and Sentiment Score
An ANOVA test showed that the effect of year on
overall average VADER sentiment score was significant (p = 0.03).
Average sentiment score was lowest in 2019 and highest in 2020. The effect on
average patron score and average operator score was not significant (p =
0.122 and p = 0.505 respectively). See Table 5 for details.
Table 5
AVOVA for Sentiment Score and Year
|
Sentiment
Score |
2019 |
2020 |
2021 |
df |
F |
|||
|
|
M |
SD |
M |
SD |
M |
SD |
|
|
|
Average
Overall |
0.205 |
0.114 |
0.218 |
0.126 |
0.217 |
0.121 |
2,
2837 |
3.526* |
|
Average
Operator |
0.232 |
0.175 |
0.234 |
0.176 |
0.241 |
0.174 |
2,
3091 |
0.684 |
|
Average
User |
0.188 |
0.146 |
0.2 |
0.15 |
0.198 |
0.153 |
2, 3077 |
2.102 |
* p < 0.05
An ANOVA test showed the effect of semester on
overall average VADER sentiment score was significant (p < .001). The
semesters with the highest average sentiment scores were summer 2021 and summer
2020. The semesters with the lowest average sentiment scores were summer 2019
and fall 2021. Additional ANOVA tests showed that the effect of semester was
significant on average patron sentiment scores (p = .01), but not on average
operator sentiment scores (p = .103). See Table 6 for details.
Table
6
AVOVA
for Sentiment Score and Semester
|
Sentiment
Score |
Winter 2019 |
Summer 2019 |
Fall 2019 |
Winter 2020 |
Summer 2020 |
Fall 2020 |
Winter 2021 |
Summer 2021 |
Fall 2021 |
df |
F |
|||||||||
|
|
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
M |
SD |
|
|
|
Average
Overall |
0.215 |
0.119 |
0.19 |
0.1 |
0.209 |
0.118 |
0.217 |
0.127 |
0.228 |
0.134 |
0.21 |
0.118 |
0.21 |
0.129 |
0.234 |
0.122 |
0.206 |
0.11 |
8, 2831 |
3.461*** |
|
Average
Operator |
0.244 |
0.19 |
0.215 |
0.168 |
0.237 |
0.164 |
0.233 |
0.173 |
0.244 |
0.194 |
0.225 |
0.159 |
0.23 |
0.181 |
0.258 |
0.176 |
0.235 |
0.165 |
8, 3085 |
1.661 |
|
Average
User |
0.197 |
0.152 |
0.181 |
0.125 |
0.185 |
0.158 |
0.199 |
0.15 |
0.206 |
0.163 |
0.196 |
0.138 |
0.203 |
0.157 |
0.216 |
0.163 |
0.176 |
0.137 |
8, 3071 |
2.527** |
** p < 0.01
*** p < 0.001
A two-samples
t-test found that average overall sentiment scores and average user sentiment
scores were higher during the pandemic compared to pre-pandemic. There was no
significant difference in operators’ average sentiment scores. See Table 7 for
details.
Table
7
Two-Samples
T-Test for Sentiment Score and Pandemic Status
|
Sentiment
Score |
Pre-Pandemic |
Pandemic |
df |
t |
p |
||
|
|
M |
SD |
M |
SD |
|
|
|
|
Average
Overall |
0.206 |
0.115 |
0.218 |
0.124 |
2601.858 |
-2.574 |
0.01** |
|
Average
Operator |
0.233 |
0.173 |
0.238 |
0.176 |
3092 |
-0.806 |
0.421 |
|
Average
User |
0.189 |
0.147 |
0.2 |
0.152 |
2679.729 |
-2.071 |
0.038* |
* p < 0.05
** p < 0.01
Association Between Complaints and Sentiment Score
A two-samples
t-test showed that average overall VADER sentiment scores were significantly
lower when the user had at least one complaint compared to chats without
complaints. In chats with complaints, average sentiment scores were
significantly lower for both user messages and operator messages. See Table 8
for details.
Table
8
Two-Samples
T-Test for Sentiment Score and Complaints
|
Sentiment
Score |
No Complaint |
Complaint |
df |
t |
p |
||
|
|
M |
SD |
M |
SD |
|
|
|
|
Average
Overall |
0.222 |
0.12 |
0.176 |
0.115 |
2838 |
7.992 |
< .001*** |
|
Average
Operator |
0.242 |
0.176 |
0.207 |
0.168 |
3092 |
4.257 |
< .001*** |
|
Average
User |
0.206 |
0.152 |
0.147 |
0.133 |
916.119 |
9.374 |
< .001*** |
*** p < 0.001
Discussion
Chat Characteristics and Sentiment Score
Between 2019 to 2021, overall sentiment on the Ask a
Librarian service was positive, with a mean sentiment score of 0.213. Sentiment
differed between the participants of the chat; the average sentiment score of
operators was higher than that of users. It is difficult to determine if this
observation is valid, because it is possible that the content of the user’s
research question or information need may confound sentiment score. For
example, certain research topics may contain words that VADER assigns a
negative score, which may contribute to user messages having a more negative
sentiment score than the operator messages. It’s worth noting that Sobol et al. (2023) found the opposite pattern. In their
sample, patrons had higher positive language than chat providers.
We found no statistically significant relationships
between user type and sentiment scores, meaning that there were no differences
in sentiment between students, faculty members, staff, alumni, or members of
the public. In contrast, we did identify a statistically significant
association between operator type and average overall sentiment scores, as well
as average operator sentiment scores. For both types of sentiment scores, we
found the scores to be lowest when the operator was a library technician and
highest when the operator was a student employee. Further research is needed to
explore why this relationship exists.
Chats in which there was a mismatch in affiliation
between the user and operator had significantly lower sentiment scores. This
may be because, as previous research has shown, users are more likely to be
dissatisfied when they are made aware that they are being assisted by a library
staff member from outside of their home institution, as patrons may perceive
these operators as lacking knowledge about their local context (Barrett & Pagotto, 2019, 2021). The pandemic also likely exacerbated
the difficulty of serving patrons from other libraries, as the shifting
conditions of pandemic-era services may have made it difficult to share
information between libraries and efficiently and accurately answer users’
questions.
We also determined that chats containing at least
one complaint had a lower overall sentiment score, user sentiment score, and
operator sentiment score. While this association may seem obvious, and the test
redundant, we tested the relationship between these variables to determine if
VADER was being influenced by the tone or attitude of chat participants, given
that the tone of complaints is inherently negative.
Temporal Aspects of Sentiment Score
There was a statistically significant relationship
between the year the chat took place and average overall sentiment score.
Surprisingly, mean scores were lowest in the pre-pandemic year, 2019, and
highest during the year in which the pandemic began, 2020. This result
reinforces those of Kathuria (2021) and Radford et
al. (2022), whose studies noted more positive words and expressions near the
onset of the pandemic in 2020. In addition, there was a statistically
significant relationship between semester and overall sentiment score, with the
lowest sentiment occurring pre-pandemic, in summer 2019, and the highest
sentiment score occurring during the pandemic, in summer 2021. Regardless of
relative differences in sentiment score between years and semesters, we note
that the average sentiment scores always remained above the threshold of 0.05,
reflecting positive sentiment.
Results about the significance of the year and
semester of the chat are consistent with our findings related to pandemic
status: chats that took place after the WHO declared COVID-19 a pandemic had a
significantly higher overall sentiment score than pre-pandemic chats. Our
results are consistent with Kohler’s research (2020), which found that
sentiment was positive during the pandemic, suggesting a “civility of
discourse,” and differ from those of Kathuria (2021),
who identified an increase in negative sentiment during the pandemic. The
differences in our results may be due to the nature of our samples: Kathuria drew on data from 2019 to 2020, Kohler from 2020
alone, and our research covered 2019 to 2021. There were important contextual
differences in the pandemic and library services across these time periods that
may have affected sentiment score. For example, throughout the pandemic, there
were periods of COVID surges and resulting lockdown or stay-at-home orders,
which restricted the availability of library spaces and collections. Our sample
captures Ontario’s second state of emergency (beginning in January 2021), the
third wave of the virus, rising infections from COVID variants of concern, and
Ontario’s third state of emergency (beginning in April 2021). The states of
emergency triggered stay at home orders, which prompted Ontario’s academic
libraries to close and shut down services like curbside pickup and scan and
deliver. Overall, 2021 was a period of significant flux, with users losing and
gaining access to physical spaces and collections, which may have influenced
sentiment scores differently than earlier phases of the pandemic. In addition,
the variation in our results may also be due to the different sentiment
analysis tools we used; Kathuria grouped words into
positive or negative sentiment using a coding system, while Kohler, like us,
used VADER.
The VADER library’s ability to calculate average
patron sentiment score and average operator score for each chat lead us to a
noteworthy finding: average user sentiment score was higher during the
pandemic, while there was no significant difference in average operator
sentiment score. This indicates that it may have been the user’s tone or
attitude that contributed to statistically significant differences in sentiment
score on Ask a Librarian during the pandemic. Sobol
et al. (2023) noted a similar trend on their consortial
chat service: scores for patron chats had a small increase in positive language
during the pandemic, while positive language among chat providers declined.
Additional research is needed to determine why user sentiment score increased
during the pandemic. A study by Radford et al. (2022) may provide an initial
explanation. Many chat operators reported positive changes in user
communication style during the pandemic, such as politeness and expressions of
gratitude. Kohler (2017) also noted the role of politeness in positive
sentiment and added that the user’s sense of being part of the same academic
community as the operator may influence the language used. These elements of
communication style may have been parsed positively by the VADER library.
Limitations and Future Research
Our study has several limitations. Firstly, there is
a great deal of complexity and nuance in textual data, meaning that sentiment
analysis tools may sometimes parse text more inaccurately than a human
researcher. Qualitative studies of pandemic-era chat discourse would be a
helpful complement to the existing computational studies. Secondly, because we
employed a mix of hand-coding and natural language processing for our research,
we selected a sample of chats rather than processing the entire corpus of chats
between 2019 to 2021 using VADER. Although our sample size was large (>3000
chats), failing to utilize the entire population of chat transcripts may have
limited the generalizability of our findings. Future research could employ our
methods using a larger sample size of chats. In addition, given the nature of
our dataset, many of our findings are contextual to the first two years of the
pandemic. While our results provide a rich portrait of users’ sentiments from
2020-2021, further research is needed to explore user sentiment during the
later years of the pandemic (2022-2023). Additional studies are also needed to
determine if the associations we uncovered between chat characteristics and
sentiment score will extend beyond the pandemic. Finally, the VADER library is
somewhat sensitive to the subject of the conversation, meaning that research
topics containing negative terms may result in negative sentiment scores.
Additional research could explore methods to effectively control for the
subject of the chat.
Implications
Our research study outlines a methodology for chat
transcript analysis that combines hand coding and natural language processing.
This approach enables researchers to calculate sentiment scores for chat
transcripts using the VADER library and run inferential statistics to test the
relationships between the hand-coded or toolchain-generated variables and
sentiment scores. This allows for deeper investigations into how the
characteristics of the chat affect sentiment score.
While we used this methodology to examine chat
transcripts from the COVID-19 pandemic on a consortial
chat service, this approach could be used to explore the nature of chat
interactions over any period and on any type of chat service. VADER can also
process content from other forms of virtual reference (such as emails) or
social media. Practitioners can incorporate this methodology into regular
service evaluation or review to understand user sentiment and satisfaction.
Given that using sentiment analysis tools is less time- and labour-intensive
than traditional hand-coding, sentiment analyses could be run more regularly
for real-time assessment.
As libraries increasingly operate in environments of
evidence based or data-informed decision-making, sentiment analysis can be a
helpful approach to identify areas where libraries can make improvements to
customer service, training, policies, or service models. As a free, open-source
tool, VADER is ideal for librarians beginning to explore sentiment analysis.
Conclusion
This study reports on the sentiment analysis of over
3,000 chat transcripts from Ask a Librarian from 2019 to 2021. Overall, we
found that mean sentiment was positive (>0.2), and higher among operators
than patrons. This difference in the sentiment of participants may be due to
the inherent negativity of some users’ research topics or the problems they were
describing. Several characteristics of the chat were significantly associated
with sentiment scores, namely operator type, affiliation mismatch, and
complaints. Sentiment score also varied significantly over time: it was lowest
in 2019 and highest in 2020. The COVID-19 pandemic was also significant: chats
that took place during the pandemic had a higher average overall sentiment
score and higher average user sentiment score. The results of this study
indicate that Ask a Librarian met user needs during the pandemic, as the
polarity of sentiment scores remained positive during pandemic-related
disruptions in library operations. We recommend that sentiment analysis
continue to be conducted as part of regular virtual reference assessment.
Author Contributions
Kathryn Barrett:
Conceptualization, Methodology, Investigation, Data curation, Formal analysis, Supervision,
Writing – original draft, Writing – review &
editing Ansh Sharma: Methodology, Data
curation, Software, Writing – review & editing
Acknowledgements
The authors wish to thank Scholars Portal for
providing access to the chat metadata and transcripts to support this project.
The team also wishes to acknowledge Kirsta Stapelfeldt and Chad Crichton of the University of Toronto
Scarborough Library for their contributions to the research project examining
the Ask a Librarian service during the COVID-19 pandemic.
References
Barrett, K., Crichton, C., & Logan,
J. (2024). An analysis of user complaints on chat reference during the COVID-19
pandemic: Insights into user priorities. Internet Reference Services
Quarterly. Advance online publication. https://doi.org/10.1080/10875301.2024.2379816
Barrett, K., & Pagotto,
S. (2019). Local users, consortial providers: Seeking
points of dissatisfaction with a collaborative virtual reference service. Evidence
Based Library and Information Practice, 14(4), 2-20. https://doi.org/10.18438/eblip29624
Barrett, K., & Pagotto,
S. (2021). Pay (no) attention to the man behind the curtain: The effects of
revealing institutional affiliation in a consortial
chat service. Partnership, 16(2), 1-21. https://doi.org/10.21083/partnership.v16i2.6651
Brousseau, C., Johnson, J., &
Thacker, C. (2021). Machine learning based chat analysis. Code4Lib, 50.
Catalano, A. J., Glasser, S., Caniano, L., Caniano, W., & Paretta, L. (2018). An analysis of academic libraries’
participation in 21st century library trends. Evidence Based
Library and Information Practice, 13(3), 4-16. https://doi.org/10.18438/eblip29450
Chen, X., & Wang, H. (2019).
Automated chat transcript analysis using topic modeling for library reference
services. Proceedings of the Association for Information Science &
Technology, 56(1), 368-371. https://doi.org/10.1002/pra2.31
Chow, A. S., & Croxton,
R. A. (2014). Usability evaluation of academic virtual reference services. College
& Research Libraries, 75(3), 309-361. https://doi.org/10.5860/crl13-408
Cohn, S., & Hyams,
R. (2021). Our year of remote reference: COVID19’s impact on reference services
and librarians. Internet Reference Services Quarterly, 25(4), 127-144. https://doi.org/10.1080/10875301.2021.1978031
Connaway, L.
S., & Radford, M. L. (2011). Seeking synchronicity: Revelations and
recommendations for virtual reference. Online Computer Library Center, Inc.
https://www.oclc.org/research/publications/2011/synchronicity.html
De Groote, S., & Scoulas,
J. M. (2021). Impact of COVID-19 on the use of the academic library. Reference
Services Review, 49(3/4), 281-301. https://doi.org/10.1108/RSR-07-2021-0043
Decker, E. N., & Chapman, K. (2022).
Launching chat service during the pandemic: Inaugurating a new public service
under emergency conditions. Reference Services Review, 50(2), 163-178. https://doi.org/10.1108/RSR-08-2021-0051
George, L. E., & Birla, L. (2018). A
study of topic modeling methods. In 2018 Second International Conference on
Intelligent Computing and Control Systems (pp. 109-113). IEEE. https://doi.org/10.1109/ICCONS.2018.8663152
Hervieux, S. (2021). Is the library
open? How the pandemic has changed the provision of virtual reference services.
Reference Services Review, 49(3/4), 267-280. https://doi.org/10.1108/RSR-04-2021-0014
Hutto, C. J. (2014). VADER-Sentiment-Analysis
[Repository]. GitHub. https://github.com/cjhutto/vaderSentiment
Hutto, C. J., & Gilbert, E. (2014).
VADER: A parsimonious rule-based model for sentiment analysis of social media
text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216-225. https://doi.org/10.1609/icwsm.v8i1.14550
Kathuria, S.
(2021). Library support in times of crisis: An analysis of chat transcripts
during COVID-19. Internet Reference Services Quarterly, 25(3), 107-119. https://doi.org/10.1080/10875301.2021.1960669
Koh, H., & Fienup,
M. (2021). Topic modeling as a tool for analyzing library transcripts. Information
Technology and Libraries, 40(3). https://doi.org/10.6017/ital.v40i3.13333
Kohler, E. (2017). What do your library
chats say?: How to analyze webchat transcripts for
sentiment and topic extraction. In F. Baudino, K.
Hart, & C. Johnson (Eds.), 17th Annual Brick & Click: An
Academic Library Conference (pp. 138-148). Northwest Missouri State
University. https://eric.ed.gov/?id=ED578189
Kohler, E. (2020). Mining library chats:
Sentiment analysis and topic extraction. In 14th International
Conference on Performance Measurement in Libraries (pp. 220-229).
Lamba, M., & Madhusudhan,
M. (2022). Text mining for information professionals: An uncharted territory.
Springer. https://doi.org/10.1007/978-3-030-85085-2
Liu, H., Chatterjee, I., Zhou, M., Lu,
X. S., & Abusorrah, A. (2020). Aspect-based
sentiment analysis: A survey of deep learning methods. IEEE Transactions on
Computational Social Systems, 7(6), 1358-1375. https://doi.org/10.1109/TCSS.2020.3033302
Lund, B. D. (2020). Assessing library
topics using sentiment analysis in R: A discussion and code sample. Public
Services Quarterly, 16(2), 112-123. https://doi.org/10.1080/15228959.2020.1731402
Mawhinney, T. (2020). User preferences
related to virtual reference services in an academic library. The Journal of
Academic Librarianship, 46(1), 102094. https://doi.org/10.1016/j.acalib.2019.102094
Munip, L.,
Tinik, L., Borrelli, S., Randone, G. R., & Paik,
E. J. (2022). Lessons learned: A meta-synthesis examining library spaces,
services and resources during COVID-19. Library Management, 43(1/2),
80-92. https://doi.org/10.1108/LM-08-2021-0070
Murphy, J. E., Lewis, C. J., McKillop,
C. A., & Stoeckle, M. (2022). Expanding digital
academic library and archive services at the University of Calgary in response
to the COVID-19 pandemic. IFLA Journal, 48(1), 83-98. https://doi.org/10.1177/03400352211023067
Osorio, N., & Droog,
A. (2021). Exploring the impact of the pandemic on reference and research
services: A literature review. New Review of Academic Librarianship, 27(3),
280-300. https://doi.org/10.1080/13614533.2021.1990092
Ozeran, M.,
& Martin, P. (2019). “Good night, good day, good luck”: Applying topic
modeling to chat reference transcripts. Information Technology and
Libraries, 38(2), 49-57. https://doi.org/10.6017/ital.v38i2.10921
Paulus, T. M., Wise, A. F., &
Singleton, R. (2019). How will the data be analyzed? Part one: Quantitative
approaches including content analysis, statistical modeling, and computational
methods. In T. M. Paulus & A. F. Wise (Eds.), Looking for insight,
transformation, and learning in online talk (pp. 127-159). Routledge. https://doi.org/10.4324/9781315283258
Radford, M. L., Costello, L., &
Montague, K. (2020). Chat reference in the time of COVID-19: Transforming
essential user services. Association for Library and Information Science
Education (ALISE) 2020 Conference. http://hdl.handle.net/2142/108820
Radford, M. L., Costello, L., &
Montague, K. E. (2022). “Death of social encounters”: Investigating COVID-19’s
initial impact on virtual reference services in academic libraries. Journal
of the Association for Information Science and Technology, 73(11), 1594-1607.
https://doi.org/10.1002/asi.24698
Schiller, S. Z. (2016). CHAT for chat:
Mediated learning in online chat virtual reference service. Computers in
Human Behavior, 65, 651-665. https://doi.org/10.1016/j.chb.2016.06.053
Sharma, A., Barrett, K., & Stapelfeldt, K. (2022). Natural language processing for
virtual reference analysis. Evidence Based Library and Information Practice,
17(1), 78-93. https://doi.org/10.18438/eblip30014
Sobol, B.,
Goncalves, A., Vis-Dunbar, M., Lacey, S., Moist, S., Jantzi,
L., Gupta, A., Mussell, J., Foster, P. L., &
James, K. (2023). Chat transcripts in the context of the COVID-19 pandemic:
Analysis of chats from the AskAway consortia. Evidence
Based Library and Information Practice, 18(2), 73-92. https://doi.org/10.18438/eblip30291
Turp, C.,
& Hervieux, S. (2023). Exploring an automated method for the analysis of
virtual reference interactions. Reference Services Review. Advance
online publication. https://doi.org/10.1108/RSR-05-2023-0050
Walker, J., & Coleman, J. (2021).
Using machine learning to predict chat difficulty. College & Research
Libraries, 82(5), 683-707. https://doi.org/10.5860/crl.82.5.683
Wang, Y. (2022). Using machine learning
and natural language processing to analyze library chat reference transcripts. Information
Technology and Libraries, 41(3). https://doi.org/10.6017/ital.v41i3.14967
Watson, A. P. (2023). Pandemic chat: A
comparison of pandemic-era and pre-pandemic online chat questions at the
University of Mississippi Libraries. Internet Reference Services Quarterly,
27(1), 25-36. https://doi.org/10.1080/10875301.2022.2117757
Yatcilla, J.
K., & Young, S. (2021). Library responses during the early days of the
pandemic: A bibliometric study of the 2020 LIS literature. Journal of
Library Administration, 61(8), 964-977. https://doi.org/10.1080/0193082