PROGRAM DESCRIPTION / DESCRIPTION DU PROGRAMME Explaining the Method Behind Our Madness: 3-part Series on Comprehensive Searches for Knowledge Syntheses

<jats:p>n/a</jats:p>


Introduction
The production of knowledge syntheses (KS), including systematic and scoping reviews, has been steadily increasing over the last twenty years. Recent estimates indicate a three-fold increase in the number of published systematic reviews over the last decade [1], and that nearly half of all published scoping reviews have been published within the last six years alone [2]. This trend is evident at the University of Toronto, where graduate students are being encouraged to include a KS component as part of their comprehensive exams or three-article theses. This has led to an increase in the number of one-on-one consultations between librarians and graduate students. Unfortunately, it is often clear during these consultations that these students are not being formally trained in KS search methods, reporting standards, or citation management solutions. Further evidence indicates this is not just happening at our institution [3][4][5][6].
To address this increasing need at the University of Toronto, librarians at the Gerstein Science Information Centre are offering a three-part workshop series designed to teach graduate students how to search for systematic and scoping reviews. In 2016, Sandra Campbell and colleagues at the University of Alberta's John W. Scott Health Sciences Library described what they believed to be the first published curricula for a three-hour stand-alone KS searching workshop designed for a researcher audience [7]. They observed that while librarians have long been involved in teaching KS search strategies as part of broader systematic review courses, there are few examples in the published literature of distinct KS searching workshops. While there has been recent discussion on how instruction is incorporated into KS service models [8]; we remain unaware of other existing librarian-led KS searching workshops for graduate students that deliver advanced content as a three-part series.

Description
The workshop series, titled Strategies for Systematic, Scoping, or Other Comprehensive Searches of Literature, is composed of three 2.5 hour sessions. It is recommended that participants take each session in order to complete the series, though this is not always the case. Students are required to preregister using the online calendaring platform LibCal, where they can also read the program description, learning objectives, and instructor biographies. We open each session to a maximum of 50 registrants; we can accommodate 40 participants in our electronic classroom. Though each session has always been fully booked, and there has usually been students on the waitlist, we expect a relatively small nonattendance rate of 10-15%. We typically offer the sessions on Tuesday afternoons, three weeks in a row, though sometimes flexibility is required to accommodate our schedules.
Eligible participants of the series can earn two credits towards the Graduate Professional Skills (GPS) program, an initiative of the University of Toronto School of Graduate Studies that is designed to prepare graduate students for their future careers. In order to claim credits, participants must be current graduate students at the University of Toronto, attend and participate in all three sessions of the series, and complete a short reflective questionnaire following the final session.
Two librarians are responsible for delivering the content, and are supported by one student assistant who can help answer questions and keep students on track during the session. Despite the high student to instructor ratio, we encourage an informal atmosphere where students are free to interrupt to ask questions or make comments. Each session utilizes a combination of lecture slides, individual activities, online polls, and group activities. Course materials are made available to participants on a password-protected LibGuides website. We plan content and activities to meet what we call our "hidden agenda": to empower graduate students with the vocabulary and skills necessary to engage in crucial conversations with their supervisors and colleagues and, ultimately, improve the quality of their review research. Our teaching philosophy is rooted in the firm belief that we need to clearly explain and justify review search methods, that our students ought to learn complex database techniques, and that they are capable of thinking critically about systematic and scoping review search strategy development. We believe in authentic and intentional engagement, a focus on processes not tools, and incorporating active learning.

Part I: Structured Approach to Searching the Medical Literature for Knowledge Syntheses
Our introductory session's objectives are to have students be able to:  Identify the key differences between systematic reviews, scoping reviews, and literature reviews as they relate to the search  Incorporate tools and resources for proper reporting and management of their review  Utilize strategies for turning a research question into a searchable question with inclusion/exclusion criteria ( Figure 1)  Identify databases for their review and explain when to use them  Practice using an objective, structured method for developing sensitive search strategies required for knowledge synthesis, utilizing controlled vocabulary, textwords, and advanced techniques  Apply a structured approach to searching in OVID Medline

Fig. 1 Search concepts vs inclusion/exclusion criteria
Guided through a combination of lecture and individual activities, students complete a semicomprehensive search of an example question in Ovid Medline, saved and ready for Part II. After completing a search question activity (Figure 1), we guide students through a process for objective search strategy development. First, we show students how to identify synonyms through various methods beyond brainstorming, including examining MeSH entry terms and interactively scanning known relevant articles [9]. Next, we demonstrate how to discover relevant subject headings by browsing the MeSH hierarchy and using tools such as the Yale MeSH Analyzer [10] and PubReMiner [11]. Finally, we show students how to iteratively test elements of their search strategy (e.g., using the NOT operator to determine optimal proximity operator width), determine whether their search captures previously identified relevant articles, and what to do next if it does not.

Part II: Beyond MEDLINE: Translating Search Strategies for Knowledge Syntheses
This session focuses largely on why and how we translate search strategies; we take an active-learning approach [12] in which students will:  Review Medline strategy from Part 1 and prepare it for translation  Delve deeper into the advanced features of interfaces and databases which allow for editing and refining a search strategy  Identify potential sources for bias in their search and develop strategies to mitigate them  Translate and execute structured search strategies using different databases, including OVID Embase, Ebsco CINAHL, and Cochrane Central (Figure 2)  Prepare database search strategies and compose search methods, such that they can be repeated and to ensure proper reporting Part II is an innovative session for three reasons: 1) we teach students to justify elements of their search strategy as mitigating potential sources of bias; 2) we spend nearly 1.5 hours leading students through a group database translation activity utilizing short demonstrations, Google Docs (Figure 2), and student in-class presentations; and 3) this is an entirely digital day, with no paper materials used for any of the activities.  [13] designed to reinforce the major learning objectives and hidden agenda of the entire series. Through this activity, students are able to see the relationship between search strategies and the overall quality of the review itself.

Outcomes
We have offered the series six times, to 291 students, since its pilot run in March 2017.
A three-part 7.5 hour series is a significant time commitment; however, student feedback and reflections, as well as follow-up consultations, seem to indicate that the advanced techniques and content is appreciated and being absorbed.
We evaluate the series in three ways: observations of student engagement during activities, ticket-out-thedoor evaluation forms, and a short reflection questionnaire. The most valuable assessment method for program development comes from our own in-class observations and conversations with students about their learning. For instance, in the pilot version of our series, we attempted an activity designed to teach students how to translate search strategies. We found that during this activity, students were simply copying and pasting the Medline strategy into Embase. When we asked that they switch to CINAHL, this problem was highlighted as students simply copied the Medline queries (along with the Ovid syntax) into the search bars. These observations made it clear we had spent too much time on database mechanics and not enough time teaching the art and process of translation.
We take the last five minutes of each session to have students pair up and fill out a short ticket-out-thedoor evaluation form (Figure 3). This helps us gauge what learning outcomes are being met, and which require more attention. After each session, we review and summarize the "muddy points", re-visiting them at the beginning of the next session. Typically, students note needing more time to practice at home, difficulty keeping up with database syntax, and how to know when to stop searching. We consistently hear positive feedback regarding Part I's lecture material, our overall instruction style, and the meaningful activities in Part II and Part III.
Finally, we attempt to gauge whether students are learning from a short reflection-based assignment that is required of all GPS participants. One week following Part III, we ask eligible participants to respond to the following three prompts: 1) Can better searches improve the quality of research? If yeshow? No-why? 2) How will you ensure your searches are reproducible and exhaustive? 3) What question(s) has this workshop raised for you? Inspired by the richness of responses, we are now pursuing a qualitative research study on graduate students' attitudes and practices conducting comprehensive searches for systematic and scoping reviews.

Discussion
In our pilot run, we had three sessions as we do now, but we had combined database translation and grey literature in Part II, and Part III was an advanced EndNote session. Instructor observations and student feedback indicated that there was not enough time in Part II for all of the content, and one student astutely pointed out that it was unfair to teach EndNote, a feebased software program at the University of Toronto, as part of the GPS program. The EndNote instructor also found that the students were not prepared to learn advanced functions required for systematic review citation management. We decided to split Part II into the current iterations, and offer two EndNote sessions (one basic, one advanced) shortly after the series, but not as part of the GPS program. We actively promote the EndNote sessions during the series, and these sessions are regularly fully booked.
One of the most significant changes we made following the pilot was to extend the length of each session from 2 hours to 2.5 hours. This has allowed us to spend more time teaching textword syntax in Part I, explaining and coordinating the translation activity in Part II, and leaving plenty of time for discussion during the Part III capstone activity. It also gives us more time to cover muddy points from the previous week's session and to answer any questions as they arise. It is important to note that despite extending the series by 1.5 hours, and replacing the EndNote session with search instruction, the search related learning objectives have stayed exactly the same. Moving forward, we are investigating how best to teach grey literature search strategy in a large group setting to such a diverse group of students, as well as how to discuss emerging review methods.
We believe that students can and should learn advanced database techniques, and that they are capable of thinking critically about KS search strategy development. We hope that other librarians will continue to explore strategies to teach this content to students. To support this goal, we will make our slides and activities available upon request.

Statement of Competing Interests
No competing interests declared.