RECHERCHE Comparison of Peer Tutor and Librarian Feedback for the Literature Search Component of a Medical School Research Course

: Introduction: The aim of this study is to compare the peer tutor and librarian feedback on second year medical students’ literature search skills as part of a research course at Queen’s University, Kingston, Ontario, Canada. Methods: Student peer tutors and medical librarians each assessed a sample of literature searches for a culminating project. Two separate student cohorts were evaluated, and the marked rubrics were compared. Students also participated in focus groups. An online survey was sent to a third cohort of students who did not work with peer tutors, but instead met with librarians one-on-one to discuss their literature searches. Results: There was a measurable difference in the mark agreement between the peer tutors and the librarians. Unsurprisingly, librarians identified important errors and omissions unseen by the peer tutors. Peer tutors found the process of peer assessment very useful for their own learning and teaching skill development, however, the non-peer tutor students did not appreciate the value of this methodology. After peer tutoring was discontinued, the survey feedback was very positive about the value of the individual librarian consultations. Discussion: Medical students conducting a research project need to perform thorough literature searches. Although librarians found the consultations time-consuming, they found that the consultations improved searches more than having students receive help from peer tutors in the same class. The surveyed students were positive about the librarian consultation .


Introduction
Medical students often have disparate amounts of experience creating research proposals and completing research projects. Supporting a large number of students in the design of individual research projects can be logistically challenging with a small number of librarians. At the Queen's University School of Medicine librarians have worked with faculty to integrate information literacy into the curriculum since 1991 [1]. Over the past 27 years, Queen's librarians have attended curriculum meetings, collaboratively designed course materials (including assessment instruments) with faculty and instructional developers, and jointly participated in educational research projects. In 2010 the School of Medicine implemented major curricular changes, adopting a competency-based framework for undergraduate medical education. Within this framework there are seven key roles. The Medical Information Literacy Program falls within the Scholar Role [2,3]. During the first year of studying undergraduate medicine, students learn to search for medical resources and critically evaluate the findings; they are learning to be effective consumers of medical information [4]. In second year, the Critical Enquiry course acts as a capstone exercise for the development of the Scholar Role. In this course, students become creators of medical information by developing a research project with a faculty tutor over the course of the year.
Prior to the launch of this course, a significant portion of medical student research project activity was concentrated during the last two months of second year (May and June), with many students continuing this research work into the summer months. This activity (known as the Critical Enquiry elective) has been described by Houlden et al [5]. Librarians met with the students individually after they had both chosen a research question and attempted a literature search. Feedback was given in the spirit of improving the literature search to support their project.
With a full curriculum restructure in 2010, the Critical Enquiry elective evolved to become a yearlong course, embedded into the 2 nd medical school year. The course consisted of a series of linked assignments leading to the creation of a research proposal. Early on in the process, one of the assignments required a detailed literature search where the students are expected to demonstrate mastery of the searching skills taught and assessed over first year.
The librarian feedback to students was needed earlier in the year, which was problematic because of other teaching and marking demands on the librarians' time. To facilitate the literature search assignment, the peer tutor process was developed whereby students with research interests or backgrounds volunteered to obtain additional training and then in turn taught and assessed the skills of their peers. This created an enhanced and personalized experience for students with research experience (many of whom already had graduate degrees in medical science fields), with the additional benefit of streamlining the librarians' workload. Students volunteered to be peer tutors knowing that they would need to do extra work but that they would gain the experience of developing their teaching and assessment skills around literature searches.
After three years of using this methodology (2011-2013), and improving it annually based on student feedback and the librarians' experiences, a selection of assignments was reviewed by the librarians in 2014 and 2016. Using the same marking rubric (Appendix A), the librarians' and peer tutors' marks were compared. Both times the librarians arrived at the identical conclusion: students were not getting sufficient advice from their peer tutors to improve their literature searches. In the fall of 2016, peer tutoring was replaced by one-on-one consultations with librarians who assessed the students' searches and gave them personalized feedback. This paper will describe the development, evolution and evaluation of this peer tutor model and the refinement of the assessment process for an assignment-based literature search curriculum embedded within a medical school research course.

Literature Review
While the literature on peer tutoring in undergraduate medicine discusses mainly "near peer tutoring" with more senior students tutoring junior students [6], Eberlein finds that data from peer-led team learning studies "show significant gains in performance, retention, perseverance, and student attitudes and opinions." [7] The intent of our integration of peer tutors was to provide a unique opportunity to develop both scholarship and tutoring skills in medical students with all levels of research experience.
Although there are studies on peer tutoring in medical school [8][9][10][11], most focus on the learning of clinical knowledge and physician roles rather than on teaching evidence-based medicine (EBM) or literature searching skills. Eldredge et al. [12] studied the impact of training students in peer assessment for literature searching and compared their work to that of students who did not receive this training. These authors concluded that there was a large time commitment to using peer tutoring which could be a barrier for integration into librarians' workloads. More recently, Carroll et al. [13] devised a flipped-classroom activity asking medical students to learn and prepare a short class presentation on a specific information literacy skill. Students used a rubric to assess their peers' presentations. This study also reported a significant time commitment, although the value of this project was clearly to increase the collaboration between librarians and faculty. In another study, Rowley et al. [14] reported on the successful peer-to-peer information literacy program designed to teach students studying to become medical and health professionals how to use the Evidence Search portal. The Evidence Search portal was created by the National Institute for Health and Care Excellence, UK and is a resource designed to provide quick access to evidence based information for patient care. Following the peer-to-peer training, the authors reported increased use of the Evidence Search portal by students when they needed to answer medical and health questions.
For the program at Queen's University, the secondyear undergraduate medical students select a research topic with a faculty tutor. The students follow the steps of the EBM process and complete a literature search on their own. We created a new program where peer tutors in the same class volunteered to receive additional training from the librarians and then used a rubric (Appendix A) to assess their peers' literature searches. This novel design has not been previously reported in the medical or library literature and was implemented because faculty saw the importance of assessing the literature search process as an integral part of the critical enquiry.

Methods and Program Description
Three methods were used to evaluate the student literature searching process ( Figure 1): 1) Rubric comparison: librarians marked a selection of student assignments and compared the marks with the peer tutors' marks; 2) Focus groups: facilitators twice asked students (both peer tutors and students assessed by their peers) about the value of the peer tutoring process; 3) An online survey: students were asked to assess the value of librarian consultations.

Fig. 1 Timeline for Critical Enquiry program evaluation
1. Peer tutor and librarian teaching, assessment and rubric comparisons A marking rubric (Appendix A) was designed in collaboration with the School of Medicine's educational designers to provide structure and consistency in the marking process. The rubric was provided to all students as a guide for project expectations. There were three grading levels, with explicit, objective criteria for unacceptable/incomplete work, satisfactory work and exceptional work. In the fall of 2013, the twenty students who had volunteered to be peer tutors attended a separate workshop with the librarians to review the key aspects of literature searching, the assignment expectations, and the marking rubric. In the fall of 2015, the entire class attended the review session with the librarians. The timing of the 2015 session allowed the peer tutors to meet with their groups during the class time. It was hoped that this group meeting with the peer tutors would eliminate the need for individual meetings later. In both cohorts, peer tutors each marked five assignments within a three-week time frame. Peer tutor assignments were marked by another peer tutor. Librarians were available to answer questions from tutors or students at any time. Peer tutors assessed the assignments independently; the librarians were not involved except when approached.
Two cohorts of student assignments were evaluated in duplicate by the librarians, one from 2014 and one from 2016. Librarians randomly selected half (50) of the 100 student assignments from 2014 and 43 from the 99 student papers from 2016. Librarians, blinded to the original peer assessment, used the rubric to perform an independent comparison which was then compared with the original peer tutor grade. Agreement between peer tutor assignment assessments and the health sciences librarians' assessments was assessed using the Cohen Kappa inter-observer agreement using EBMCalc [15]. The values are shown in Table 1. The Cohen Kappa, rather than simple percentages, was calculated to reduce the effect of chance agreements [16]. In addition, the inter-observer agreement between librarians was calculated, with a near-perfect intraclass correlation of 0.816 (P<0.01).

Focus Groups about peer tutor assessments
In 2014, focus groups with peer tutors were led at arm's length by the educational developers at the School of Medicine, Queen's University, to gain insights into the peer tutor processes and give feedback to the librarians and course director. In 2016, educational developers again conducted focus groups, this time including both peer tutors and students who were assessed by their peers. Reported feedback was analysed to determine themes and provide suggestions to improve the peer tutor model.

Online survey of librarian consultations (Spring 2017)
Following an adjustment to the fall workload of the librarians and utilizing the feedback of the students from the three years of the peer tutor program, the literature search assignments were submitted and marked by the six health sciences librarians. Two days after submitting their assignment, each student underwent a 30-minute consultation, so that they could obtain feedback and assistance with their literature search. This flipped design was chosen to maximize student learning. During the consultation, the librarians used the same marking rubric that had been previously used by the peer tutors to assess the students' work. Depending on the project, the bulk of the time was spent either teaching the student how to search an unfamiliar database, or to improve their searches in the chosen databases.
Librarian consultations were evaluated by means of an all-class survey administered in March/April 2017. Ethics approval was granted by the Queen's University Health Sciences Research Ethics Board. The survey was distributed using the subscription-based SurveyMonkey service via an email sent to the students by the School of Medicine (n=92). By completing the survey, the students gave their consent to be a part of this study and were assured of the anonymity of their responses. The survey consisted of 8 questions (Appendix B) to address the outcomes of the librarian consultations. The raw data was exported from SurveyMonkey in CSV format and imported into SPSS for analysis. Microsoft Excel was then used to create all tables and charts.

Peer tutor and librarian rubric comparisons
The results are shown in classes. Agreement between the peer tutors and the librarians was markedly improved in the Meds 2018 group. Students in both classes underused databases for their searches, frequently neglecting to search relevant databases in other disciplines. Peer tutors did not correct this error. For example, the topic of needle exchange programs and public opinion could be searched in sociology and psychology databases. The peer tutors never suggested additional databases. In one instance, the student did not search Medline or Embase when they would have been relevant databases, and the tutor did not recommend that they be included.

4) Search strategy (including advanced features):
In the Meds 2016 class, peer tutors gave all 50 papers the top level (3) mark, but librarians agreed in only four cases that the student deserved a level 3. As a result, the Cohen's Kappa of 0.08 showed only slight agreement. In the Meds 2018 class, more tutors assigned a level two, meaning that the agreement was improved (Cohen's Kappa: 0.32, fair agreement) with 25 of the 43 papers receiving the same mark from the tutors and the librarians.
In Table 2 all the concepts in the first column were covered in first year database searching classes for both cohorts. For Meds 2016, review sessions were led by the peer tutors, and for the Meds 2018 class the review was presented by the librarians. Although these sessions were included in the regular schedule for the Critical Enquiry course, not all students attended. Table 2 presents common searching problems from the Meds 2016 cohort that should have been identified by the peer tutors based on the content covered during the information literacy sessions, and the expectations of the marking rubric.

Focus groups about peer tutor assessments
The School of Medicine Instructional Designer met with 12/25 peer tutors in the spring of 2014 to get their feedback about the peer tutoring process after its inaugural roll-out. The feedback from the tutors was mostly related to requests for scheduling modifications and additional librarian support for their own peer teaching. To address the latter, librarians prepared a presentation for students to use within their group. The librarians also compiled a series of searches with comments indicating how to improve each one, so that the tutors could see what type of feedback they were expected to provide to their classmates.
For the Meds 2018 students, two instructional designers conducted focus groups with 5/20 peer tutors and 8/80 non-peer tutor students. Peer tutors commented that students in their groups felt more comfortable asking questions of their peers than librarians and appreciated improvements to their own learning (e.g. "if you can teach it, you have to know it very well"). The tutors reported feeling comfortable giving feedback to their peers and considered it not "negative" but rather, constructive feedback. Some peer tutors suggested that more time should be spent on designing an answerable research question rather than literature searching.
In the non-peer tutor group, most students said that their confidence in their ability to do a literature search was unchanged after working with their peer tutor and did not feel that the peer tutor assessments offered concrete improvements for their literature search.

Online survey of librarian consultations (Spring 2017)
A brief online survey was sent to the class in the spring of 2017 to seek feedback about the librarian consultations which had occurred in late fall 2016. After three reminders, 31 students (33.7%) completed the survey. According to Nulty [17] a response rate of 31% is considered acceptable for this type of study. Almost 2/3 (64.5%) of the respondents started medical school with a previous undergraduate degree. The remaining students had a master's degree. About 68% of respondents agreed that the librarian consultation had been helpful. 61% received help with the selection of appropriate databases; this number included a few students who needed help searching a different database. When correlated with the students' level of education prior to medical school, 14 of the 20 students (70%) with an undergraduate degree, and 7 of the 11 students with a master's degree (64%) agreed that the librarian had helped them with their research question ( Figure 2). As well, 14 of the 24 students (58%) with an undergraduate degree and 5 of the 12 students (42%) with a master's degree appreciated being shown additional databases (Figure 3). The graph in Figure 4 charts the students' previous knowledge of advanced search features and the changes that occurred after the librarian consultation. During the first year of medicine, librarians present the advanced features of searching Medline (e.g., subject headings, subheadings, explode, focus, etc.) because these powerful techniques can yield higher relevance in the search results when applied during health sciences database searching. Between 26% and 65% of the respondents indicated that they were knowledgeable about these features. However, the librarian consultations helped many students to "get it now" in almost all areas, especially the use of focus, trees, explode and subheadings. Surprisingly, only Boolean logic remained unclear for a large group of students (45%).

Fig. 4 Student self-assessment of advanced search features after the librarian consultation
Librarians helped over 64% of students find additional articles useful for their projects. A large percentage (81%) of respondents who did get help with their research question responded that they had found more articles as a result of the consultation. Only 7 respondents (22.5%) did not get help with their question and did not retrieve more articles.
Overall the majority of students who responded to the survey found the librarian consultation useful both for their current (77%) and future projects (74%) and this was true whether they had a previous undergraduate or master's degree ( Figure 5).

Discussion
This paper describes an iterative process of evaluation and quality improvement in the assessment of a literature search for a capstone research project. The peer tutor initiative was not designed as a replacement for librarians, but instead as a novel structure with two planned benefits: 1) Protecting the availability of librarians to allow students needing advanced support to access them while at the same time assessing all students in basic searching skills. 2) Creating an enhanced experience for students with research background or interest by developing their skills as teachers. After the first round of feedback, it was clear that while the students benefited from becoming peer tutors, few students were accessing librarian services. There were also some logistical challenges. Modifications to the process were made through better supports and teaching techniques to peer tutors' own skills in literature searching and by extension, their tutoring skills for their groups of peers. Additional time was invested in encouraging students, peer tutors and non-peer tutors alike, to come for help at any time during their searching process. Unfortunately, this rarely occurred, either independently or at the recommendation of a peer tutor. Additional concerns about the effectiveness of the peer tutor marking were raised and confirmed through the duplicate marking process [18]. In the meta-analysis by Falchikov and Goldfinch it is reported that "peer assessment activities have been found to promote learning." [19] Although the librarians were pleased with the satisfaction expressed by the peer tutors during the focus group sessions, it was decided that this model was working well only for about 20% of the class (the peer tutors) while the rest of the class was not getting the benefit of a better search to inform their research project. There was additional concern that, rather than freeing up librarian time for complex problems, the process was undermining the importance of librarians as resources for all students, especially when facing searching dilemmas. As a result, the librarian fall workload was restructured to allow all students an individual consultation using a flipped design; students completed a searching assignment and reviewed the rubric assessment during their consultation. This design seemed to offer the best balance between support, skill development and assessment for this important process within the medical school curriculum.
The students who responded to the survey in 2017 clearly benefitted from the librarian consultation. They were able to improve their question and the citation retrieval by improving their search strategies. Librarians were also in a better position than peer tutors to suggest additional databases that would be appropriate to the topic of the research project. During the librarian consultations, when a different database was recommended, the librarian could demonstrate this in real time and retrieve important articles that the student had not found previously. It should be mentioned that peer tutors might have suggested additional databases, but without the advantage of the immediate search to demonstrate the relevance, most students, having found a sufficient number of articles would have no incentive to look for more. Prabha et al. explained the concept of satisficing and in their study they found that "For many students, the amount of time available for doing the assignment and the relative reward (the value being in terms of the final grade in the course) influenced when they stopped looking for more information." [20] Another insight about this concept came from the 2016 peer tutor focus group: all the tutors agreed with the suggestion that assignments should be sent to the faculty tutors for added accountability. The tutor who offered this suggestion seemed to suggest that some work would have been improved as students would have felt "more pressure to do a good job." It is surprising that a large proportion of survey respondents were still unclear about Boolean logic. Could it be because they did not recognise the terminology in the survey? Librarians normally referred to this concept as "Boolean logic" when speaking with students, however, it may have been clearer to label this as "AND/OR" in the survey instrument.
Fortunately, the majority of students said they knew about subject headings and limits. These concepts are explained in detail in the first year of medical school and the students apply these in assignments marked by the librarians during the first year.
Another important point to make is regarding the background sources. As mentioned above, the requirement was removed from the marking rubric after finding that the correlation was very high between peer tutors' and librarians' assessments. The instructions still asked the students to complete this work, but it would not be assessed by the marking rubric. The hope was that this would save some marking time for the peer tutors. In the end, this proved to be a wrong decision: students bypassed this crucial part of the process and then had more difficulties with their question and their literature searches. One student even suggested in a focus group that librarians spend more time on background sources and question formulation. When the librarians started the consultation process in the fall of 2016, the background sources were re-instated in the marking rubric.
Finally, a point should be made about the librarian workload involved in the consultation process. The authors cited earlier in this paper [12,13] mentioned that the workload involved in having peer tutors was considerable and might not be practical in other libraries. The same could be said of having one-on-one librarian consultations. The preparation time and meeting together averaged one hour per student. Sharing this work among a team of experienced librarians (5.6 FTE) makes it feasible at this institution. The mid-November timing does not coincide with other demands on the librarians' time and importantly, the positive outcomes from each consultation reinforce the relevance and basic need for this educational intervention.
One limitation of this study was that librarians had to base the decisions on feedback from a very small sample of students in the focus groups, and from a random selection of the papers marked. The low response rate from the questionnaires may also introduce bias to the feedback. Another limitation of our results is that both the data collection methods and the curricular structures changed over time. The various student cohorts could report only on their own experiences, for example, the last cohort did not have peer tutors. The authors used the cumulative feedback to create a composite picture of the successes and failures of this assessment methodology.

Conclusion
To evaluate the information literacy program in the Critical Enquiry course in second year medicine at Queen's University, the librarians and faculty course director used 3 methods. The results led to a major change in how librarians teach the medical students and assess their literature searching skills. The comparison of the marking in the first instance caused the librarians to enhance the peer tutor interaction and format of the whole-class session. The second time the comparison showed some improvement, however the focus group participants were strongly in favour of discontinuing the peer tutor approach to student assessment. Librarians and faculty agreed to require the students to meet one-on-one with a librarian during the fall term when preparing a comprehensive literature search as part of the Critical Enquiry course. The third evaluation tool was a student survey to assess the librarian consultations. This survey demonstrated perceived value of the consultations by the students who responded. Through informal debriefing about the critical enquiry process, the librarians indicated that they considered the interactions with the students productive. They also felt that the students recognized the value of librarian expertise in searching and question formulation.