A Curriculum for an Introductory Systematic Review Searching Workshop for Researchers


JCHLA / JABSC 37: 2–5 (2016) doi: 10.5596/c16-003

A Curriculum for an Introductory Systematic Review Searching Workshop for Researchers

Sandy Campbell, Janice Yu Chen Kung, and Liz Dennett

Sandy Campbell1, Liz Dennett, John W. Scott Health Sciences Library, 2K3.26 Walter C. Mackenzie Health Sciences Centre, University of Alberta, Edmonton, Canada.

Janice Yu Chen Kung, Webster Library, Concordia University, Montreal, Canada.

1Corresponding author (email: sandy.campbell@ualberta.ca)

Abstract: Systematic review searching is becoming an increasing part of the day-to-day work of health librarians. The University of Alberta John W. Scott Health Sciences Library offers a monthly three-hour “Introduction to Systematic Review Searching” workshop to researchers at the University of Alberta. Over the four years that the program has been offered, participants have completed evaluations at the end of each session. Based on these evaluations, the content and the delivery of the workshop have been refined, and the value of the sessions to the participants has been affirmed.


Many libraries offer systematic review searching classes to both library staff and researchers. In 2014, Saleh et al. [1] reviewed the presence of systematic review searching in library school curricula, citing courses at the University of Pittsburgh, the University of Alberta, and Texas Woman’s University. Conte et al. [2] recently described using a “flipped classroom” to teach systematic review searching methods to librarians. Although we are aware of search skills being offered as a part of broader systematic reviewing workshops, for example the University of Alberta’s “Putting Evidence into Practice” [3] workshop, we are unaware of any published curricula for stand-alone systematic review searching workshops designed for the researcher audience.

Following is a description of an instructional program, “Introduction to Systematic Review Searching,” a three-hour hands-on workshop that was offered at the University of Alberta 35 times between 2011 and 2015, and is currently being offered on a monthly basis. This program was developed as one of a number of workshops offered by the The University of Alberta John W. Scott Health Sciences Library (Scott Library) for continuing education credit approved by the Royal College of Physicians and Surgeons of Canada (RCPSC). This session is part of the “Information Skills for Health Professionals” program, a self-approved group-learning activity (a requirement of Section 1) as defined by the Maintenance of Certification program of the RCPSC [4]. To meet the requirement for credit, the workshops have to be taught in a medical facility, by a health librarian. They also have to have clear objectives advertised. Attendance sheets and evaluations are required and must be kept for possible inspection. Questions for the workshop evaluation form were defined by a local representative of the RCPSC. Most of the questions are designed to garner information of interest to the RCPSC; however, two open-ended questions elicited feedback that has been useful to the continued improvement of the workshop: “What changes will you make in your practice as a result of this session?” and “Comments and Suggestions.” Information gleaned from the evaluations was used throughout the four years to continuously improve the workshop, and it also documents the ways in which participants describe the session changing their practice. Although the program was designed as part of the RCPSC program, participation was open to any member of the University of Alberta community. Participants were mainly from the health disciplines (Medicine and Dentistry, Pharmacy and Pharmaceutical Sciences, Nursing, Rehabilitation Medicine, and Public Health). In addition, significant numbers of participants were from Nutrition, Physical Education, and Educational Psychology.


In accordance with the RCPSC requirements, the workshop was advertised through the University of Alberta’s Student Workshops link and includes the following description of outcomes:

By the end of the session participants will be able to:

  1. Identify systematic reviews and distinguish them from other reviews,
  2. Recognize the breadth of resources required to execute a systematic review search,
  3. Develop a well-formulated search question and structure a search using the PICOS format,
  4. Know how to apply appropriate date, language and publication type limits,
  5. Document a search in a standardized form,
  6. Understand the importance of peer-review of systematic review searches,
  7. Recognize the level of expert searching needed for a systematic review.

The workshop is restricted to 18 participants, limited by the size of the computer lab in which it is normally taught. The workshop is ideally conducted by three health sciences librarians, maintaining a 1:6 instructor-to-student ratio. Participants are expected to do advance reading, create a RefWorks account, and prepare a research question in advance of the class. The workshop employs a PowerPoint presentation, a live demonstration of database searches, and hands-on practice with worksheets and database searching. It also includes student-centred teaching practices such as discussion, learning groups, experiential activities, and independent research [5]. An outline of the five parts of Introduction to Systematic Review Searching program follows.

Part 1: Overview of systematic review searching

Part 1 begins with an overview of what a systematic review is, how it compares with other kinds of reviews, and the role of the search and search documentation in a systematic review. Participants review the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram [6] to understand the process of a systematic review.

Part 2: Defining the search question

In Part 2 participants move into two hands-on exercises. First they use a PICOS form (Figure 1) to parse out the concepts in the research that they have brought with them, and then they transfer the concepts into Boolean circles. With the Boolean circles worksheet (Figure 2), participants then build up synonyms. Through lecture, images, and practice, participants are introduced to the identification of searchable concepts, Boolean logic, and several issues related to synonyms including: close synonyms, variant endings, variant spellings, acronyms, homonyms, and broader and narrower terminology. During this time instructors circulate in the lab, speaking with each student to ensure that they receive help in developing their search terms.

Fig. 1. PICOS form for therapy questions.

Fig. 1.

Fig. 2. Boolean circles.

Fig. 2.

Part 3: Systematic searching

Part 3 begins with a demonstration of systematic searching, usually in Ovid MEDLINE, during which the participants may follow along online, replicating the search as it is demonstrated. During this session the need for step-by-step construction of logical and replicable searches is emphasized. This is followed by 10–15 minutes of practice time on Ovid MEDLINE, during which students practice the search that they have described in their Boolean circles exercise.

Part 4: choosing resources and adapting searches to different databases

Part 4 begins with a PowerPoint-supported lecture that describes the kinds of sources searched during a systematic review search and the suite of databases available at the University of Alberta. The need to modify search strategies for different databases is reinforced. Several databases on different platforms (usually EBSCO CINAHL, SCOPUS, ProQuest Dissertations, and Theses Global) are also demonstrated, with opportunities between each demonstration for participants to try their searches on these platforms. Participants review the Peer Review of Electronic Search Strategies (PRESS) checklist [7] to consolidate their learning about search mechanics.

Part 5: Preparing a search methodology and search appendix

Part 5 reviews how to prepare the Search Methodology and Search Appendix sections for publication of a systematic review. A sample of a published search methods section from a systematic review is reviewed as a PowerPoint presentation. There is time available at this point for participants’ questions that have not been answered during the session. The session concludes with distribution of evaluation questionnaires and certificates of attendance.


Over the five years that this workshop has been offered, participants have returned 331 evaluation forms. Of the returned forms, 268 participants responded to the “Suggestions and Comments” question, whereas 263 forms contained responses to the question, “What changes will you make in your practice as a result of this session?” Of the comments received, many were simple notes of appreciation or approval, such as “Great job!,” “Good session,” or “Thanks. I learned a lot.” However, others offered constructive criticism and suggestions for changes. Over successive sessions, we have used the suggestions and comments to improve the delivery and content of the workshop. From the responses to the “changes in practice” question, we get a sense of the impact of the session on the participants.

Improvements made to program delivery

Initially, the workshop was 1.5 hours, offered over a lunch hour. Participant comments and suggestions informed us that it was too short, so the session was lengthened to 3 hours, offered over a morning or an afternoon. We still routinely receive feedback that the session is too short, often requesting a day-long session, but we are also aware that many people cannot commit to a full day of training.

Initially the workshops were taught by two librarians; however, feedback from the evaluation forms revealed that the one-on-one support by the librarians during the hands-on parts of the workshop was one of the things that was most valued. Participants commented: “appreciated having multiple instructors in the class,” “helpful to have more than one librarian present,” and “individual assistance was very helpful.” As a result, we now schedule three librarians for sessions in our usual 18 seat lab or maintain a 1:6 instructor-to-student ratio when we have larger sessions in other venues.

Improvements made to program content

After our early sessions, participant feedback alerted us to the fact that many participants needed to know more about systematic reviews, in general, and also that we were assuming too much knowledge about searching. Due to time constraints and availability of qualified instructors, we could not offer a full introduction to systematic reviews. To ensure the participants have a basic understanding of systematic reviews we began requiring advance reading in this area, specifically: The Institute of Medicine’s Standards for Systematic Reviews [8], Grant and Booth’s A Typology of Reviews [9], and Hemingway and Brereton’s What is a Systematic Review? [10].

We also discerned, through the evaluations and through interacting with participants during the early workshops, that most had a very low level of knowledge about search mechanics. Most had little knowledge of Boolean search operators, structured searching, combining search sets, and using subject headings. As a result, after the first few workshops, we realized that we had to teach not just an introduction to systematic review searching, but an introduction to searching in general.

Because we had to increase the amount of time spent on basic search processes, we had to remove other content. Initially, in the 3-hour workshops, we had dedicated about half an hour to creating RefWorks accounts and demonstrating how to use RefWorks and Write-n-Cite. We now require participants to create a RefWorks account in advance and we point them to RefWorks and Write-n-Cite tutorials and the Library’s RefWorks workshops. Duringthe systematic review searching workshop, the RefWorks demonstration content is restricted to 2–5 minutes of PowerPoint presentation. Participants are encouraged to practice exporting to their accounts as they practice with the databases.

A further change that we made was to focus on basic search skills and reduce the amount of time searching multiple databases and platforms. Participants often comment that they wish there was more time for this in the session. Although we recognize that more practice time on a variety of databases and platforms would be desirable, it is more important that the participants learn the basics well. To compensate, in part, for this our hand-out package now includes search command charts or “cheat sheets,” which list comparative commands across seven platforms for searching and for exporting and saving references. Finally, when we began teaching, we only provided proof of attendance certificates to members of the RCPSC, through which this program was approved for Continuing Medical Education credits. Other participants let us know that they also wanted certificates of attendance. Some countries require that their graduate students studying abroad send documentation of activities to their governments to ensure continued financial support. These students particularly valued the certificates as proof of attendance. We now distribute certificates, which are individually signed, to every participant who completes the workshop.

Impact on participants’ practice

Of the 263 participants who responded to the question, “What changes will you make in your practice as a result of this session?” many reported positive impacts. No one reported negative impacts. To analyze these comments, we first grouped them according to themes. Their frequencies were tallied. Some participants expressed:

  1. having acquired a better understanding of the systematic review method (n = 35);
  2. having learned how to be more organized when conducting a review, for example in saving searches for later replication or in documenting searches for publication (n = 42);
  3. knowing how to select appropriate databases (n = 62);
  4. learning how to execute a search in a systematic manner (n = 72);
  5. general improvements in search techniques or having acquired specific search skills such as using subject headings or truncation (n = 76);
  6. using the RefWorks citation manager was a change of practice (n = 30);
  7. feeling more “motivated” to undertake a systematic review (n = 8);
  8. feeling more “confident” or “brave” and “less stressed” in approaching systematic review projects (n = 11).

Although we do not test whether or not individual learning objectives have been met, the summary of responses shows that participants cited concepts related to many of our stated learning objectives, particularly those related to resources required to execute a systematic review search, formulating a question, and executing a structured search. The emotional changes, feeling more motivated and confident, were not part of our stated outcomes but are nonetheless indicators of the positive value of the workshop.


The purpose of this program description is to give other librarians the opportunity to use and build upon the work that we have done. The evaluations from the workshop indicate that it is highly valued and participants stated that they expected that it would change their practice because they acquired practical skills and knowledge that they could take away and apply immediately. Although the workshop is staff-intensive, based on the participants’ feedback, we believe that the one-on-one, point-of-need assistance is one of the key features that makes the workshop so valuable.

Workshop participants often indicate that they would like longer or additional workshops on other databases, advanced search skills, and other parts of the systematic review process such as data extraction and statistical analysis. Over the years the Library has offered several database specific or advanced searching classes with inconsistent levels of uptake. The teaching of the broader process of systematic reviewing requires a larger teaching team including trained systematic reviewers and biostatisticians as well as librarians. The “Introduction to Systematic Review Searching” workshop has been oversubscribed for much of the time that it has existed. There is almost always a waiting list for the monthly class. It clearly meets a need in our community, so we intend to continue offering it for the foreseeable future, improving and refining it based on participant feedback.

We believe that this workshop can be offered anywhere to any group, limited only by the availability of a computer lab, adequate numbers of staff who are expert searchers, and access to the appropriate databases. To that end, we have made our PowerPoint slides, a detailed Workshop Outline and our Workshop Handouts freely available on the University of Alberta’s institutional repository, ERA [1113].


1.Saleh AA, Ratajeski MA, Bertolet M. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evid Based Libr Inf Pract. 2014;9(3):28–50.
2.Conte ML, MacEachern MP, Mani NS, et al. Flipping the classroom to teach systematic reviews: the development of a continuing education course for librarians. J Med Libr Assoc. 2015;103(2):69–73. doi: 10.3163/1536-5050.103.2.002.
3.Putting evidence into practice [Internet]. 2013. Available from: http://www.pep.ualberta.ca/
4.Maintenance of Certification Program (MOC) [Internet]. 2014. Available from: http://www.royalcollege.ca/portal/page/portal/rc/members/moc/regulations
5.Forsyth DR. College teaching: practical insights from the science of teaching and learning. 2nd ed. 2016.
6.Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):1–34.
7.McGowan J, Sampson M, Lefebvre C. An evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evid Based Libr Inf Pract. 2010;5(1):149–154.
8.Morton S, Levit L, Berg A, Eden J. Finding what works in health care: standards for systematic reviews. National Academies Press; 2011.
9.Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108. doi: 10.1111/j.1471-1842.2009.00848.x.
10.Hemingway P, Brereton N. What is a systematic review? Hayward Medical Communications; 2009. Report No.: NPRO/1111.
11.Campbell S, Dennett E, Kung J. Introduction to systematic review searching: workshop outline. Edmonton: University of Alberta. John W. Scott Health Sciences Library. 2015. Available from: https://era.library.ualberta.ca/files/hm50tv309
12.Campbell S, Dennett E, Kung J. Introduction to systematic review searching: PowerPoint presentation. Edmonton: University of Alberta. John W. Scott Health Sciences Library. 2015. Available from: https://era.library.ualberta.ca/files/jd473008g
13.Campbell S, Dennett E, Kung J. Introduction to systematic review searching: Handouts. Edmonton: University of Alberta. John W. Scott health Sciences Library. 2016. Available from: https://era.library.ualberta.ca/files/2z10ws876


  • There are currently no refbacks.