Review Article


Teaching Knowledge Synthesis Methodologies in a Higher Education Setting: A Scoping Review of Face-to-Face Instructional Programs


Zahra Premji

Research and Learning Librarian

Libraries & Cultural Resources

University of Calgary

Calgary, Alberta, Canada



K. Alix Hayden

Senior Research Librarian

Libraries & Cultural Resources

University of Calgary

Calgary, Alberta, Canada



Shauna Rutherford

Information Literacy Coordinator (Retired)

Libraries & Cultural Resources

University of Calgary

Calgary, Alberta, Canada



Received: 11 Jan. 2021                                                               Accepted: 17 Mar. 2021



cc-ca_logo_xl 2021 Premji, Hayden, and Rutherford. This is an Open Access article distributed under the terms of the Creative CommonsAttributionNoncommercialShare Alike License 4.0 International (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.



DOI: 10.18438/eblip29895




Background - Knowledge synthesis (KS) reviews are increasingly being conducted and published. Librarians are frequently taking a role in training colleagues, faculty, graduate students, and others on aspects of knowledge syntheses methods.

Objective - In order to inform the design of a workshop series, the authors undertook a scoping review to identify what and how knowledge synthesis methods are being taught in higher education settings, and to identify particularly challenging concepts or aspects of KS methods.

Methods - The following databases were searched: MEDLINE, EMBASE & APA PsycInfo (via Ovid); LISA (via ProQuest); ERIC, Education Research Complete, Business Source Complete, Academic Search Complete, CINAHL, Library & Information Science Source, and SocIndex (via EBSCO); and Web of Science core collection. Comprehensive searches in each database were conducted on May 31, 2019 and updated on September 13, 2020. Relevant conferences and journals were hand searched, and forward and backward searching of the included articles was also done. Study selection was conducted by two independent reviews first by title/abstract and then using the full-text articles. Data extraction was completed by one individual and verified independently by a second individual. Discrepancies in study selection and data extraction were resolved by a third individual.

Results - The authors identified 2,597 unique records, of which 48 full-text articles were evaluated for inclusion, leading to 17 included articles. 12 articles reported on credit courses and 5 articles focused on stand-alone workshops or workshop series. The courses/workshops were from a variety of disciplines, at institutions located in North America, Europe, New Zealand, and Africa. They were most often taught by faculty, followed by librarians, and sometimes involved teaching assistants.

Conclusions - The instructional content and methods varied across the courses and workshops, as did the level of detail reported in the articles. Hands-on activities and active learning strategies were heavily encouraged by the authors. More research on the effectiveness of specific teaching strategies is needed in order to determine the optimal ways to teach KS methods.



Knowledge synthesis (KS), also known as evidence synthesis (ES), is defined as “the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic” (Grimshaw, 2010, p. 2; Cochrane, 2020). Furthermore, KS uses methods that are transparent and reproducible (Chandler & Hopewell, 2013).


There are many types of knowledge synthesis reviews (Sutton et al., 2019), and one of the most well-known is the systematic review (SR). A systematic review “seeks to collate evidence that fits pre-specified eligibility criteria in order to answer a specific research question” and attempts “to minimize bias by using explicit, systematic methods documented in advance with a protocol” (Chandler et al., 2020, p. 1). SRs provide an up-to-date synthesis of the state of knowledge on a topic, which can aid in decision-making for practice or policy, identify and indicate gaps in knowledge or lack of evidence, and reveal the limitations of existing studies on a topic (Lasserson et al., 2020). Whereas SRs have been prevalent in the health sciences for some time, they are gaining popularity in a broader range of disciplines.


While systematic reviews are being increasingly published, many have incomplete reporting or were conducted poorly (Bassani et al., 2019; Page et al., 2016; Pussegoda et al., 2017). Experts recommend that both researchers and journal editors should be better educated on SR methodologies (Page & Moher, 2016; Page et al., 2016). They specifically advocate for education focused on strategies to identify bias in a SR, as well as strategies to minimize these biases, which will help to improve the quality of published systematic reviews, and, subsequently, help to “reduce this avoidable waste in research” (Page et al., 2016). Cochrane, an evidence synthesis organization, recommends that first time review authors attend relevant training and work with others who have experience conducting SRs (Lasserson et al., 2020).


Currently, education on KS methods takes many forms such as higher education courses, continuing education courses, workshops, webinars, and eLearning modules. Many evidence synthesis organizations including Cochrane, Joanna Briggs Institute, and the Campbell Collaboration offer fee-based workshops and courses that focus on KS methods (Cochrane, n.d.; Campbell Collaboration, n.d.; Joanna Briggs Institute, n.d.). SR instruction is also offered as credit-bearing courses to undergraduate or graduate students in post-secondary institutions (Himelhoch et al., 2015; Li et al., 2014). Some professional development workshops on KS methods are available at conferences. Additionally, academic libraries offer workshops on some steps of the systematic review methodology (Campbell et al., 2016; Lenton & Fuller, 2019). All of these different programs vary in terms of learner audience, breadth and depth of content covered, and delivery methods, while having the shared goal of educating researchers in the steps and processes necessary to conduct KS reviews. 




We undertook the study as two of the authors were beginning to design of a series of in-person workshops to teach systematic or scoping review methodology. We wanted to learn which teaching methods work well and what challenges we might encounter. We initially considered a systematic review, however, we realized that we were conducting an exploratory study where the literature had not been previously mapped in a structured way. Munn et al. (2018) note that an indication for conducting a scoping review is “to identify key characteristics or factors related to a concept” (p. 2). Further, we wanted to include all forms of evidence, including quantitative or qualitative studies, scholarship of teaching reflections, opinion articles, and program descriptions. Given our openness to all evidence types from all disciplines, we expected that the retrieved literature could be quite heterogenous, which is one reason to choose a scoping review (Peters et al., 2020). Scoping reviews “are more appropriate to assess and understand the extent of the knowledge in an emerging field or to identify, map, report, or discuss the characteristics or concepts in that field” (Peters et al., 2020, p. 2121). Therefore, we decided that a scoping review was the best approach to inform development of both the content and the delivery of our workshop series.


The objective of our scoping review was to identify the extent of the literature and summarize articles that describe the teaching and learning of any knowledge synthesis methodology in a post-secondary setting, with at least a partial in-person (face-to-face) component, to determine:


1)      steps of the knowledge synthesis process taught

2)      teaching methods and learner activities used

3)      learner challenges encountered


A recent environmental scan focusing on online KS instructional courses already exists (Parker et al., 2018). The authors evaluated 20 online training resources against best practices for online instruction using a rubric. To avoid duplication, we decided to exclude online courses and focus solely on face-to-face educational options.




A protocol outlining the objectives, inclusion criteria, and methods for this scoping review was developed in May 2019 to inform our study, and is available from the first author. The protocol is based on the methodological guidelines outlined by the Joanna Briggs Institute (JBI) for the conduct of scoping reviews (Peters et al., 2017). Additionally, our study is reported according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis - Extension for Scoping Reviews) guidelines (Tricco et al., 2018).


Study Eligibility


The population (P) in our scoping review is individuals at post-secondary institutions, which includes students, staff, librarians, and faculty. The concept (C) of interest is instructional interventions for learning knowledge synthesis methodologies. The specific context (C) we are interested in is non-commercial courses that had some in-person component.


Specifically, articles were eligible for inclusion if they describe a course or workshop that:


       taught knowledge synthesis methodology, which we operationalized as the teaching of at least two steps of the knowledge synthesis methodology (protocol development, question formulation, data collection, study selection, data extraction, critical appraisal, narrative synthesis, meta-analysis, or reporting)

       was offered in a higher education setting (for example, credit-bearing, professional development, optional workshop)

       included at least some in-person components (blended courses or entirely in-person course)

       incorporated an evaluative, reflective, or assessment component (this could take the form of assessments of student learning, workshop/course evaluations, or instructor observations or reflection)


Additionally, articles were considered ineligible if they:


       covered a course where teaching was entirely online or via asynchronous methods

       discussed commercially offered courses such as those being offered by organizations involved in knowledge synthesis (e.g. Cochrane, Joanna Briggs Institute, and others)

       focused on evidence based medicine/practice, where methodology of systematic reviews is not significantly covered

       discussed only one step of the knowledge synthesis methodology

       were published in languages other than English


Search Strategy and Information Sources


We utilized a three-step search strategy, as outlined by JBI (Peters et al., 2017). First, we conducted an exploratory search in Google Scholar to discover relevant seed studies that met the inclusion criteria for our review. The articles’ titles and abstracts were analyzed and mined for keywords. As well, we analyzed the seed article records in the MEDLINE (OVID) database to identify relevant subject headings. From this analysis, a search was developed in MEDLINE (OVID), and was piloted against the known seed articles to ensure relevant studies were captured. This MEDLINE search was developed by a librarian (ZP) and peer-reviewed by a second librarian (KAH). The search was then translated for all databases identified in our search protocol. The searches incorporated subject headings when available and free-text terms were combined using appropriate Boolean operators. No language, date, or study design filters were used. The complete search strategies for all databases are included in the Appendix.


The choice of databases was purposefully exhaustive so that as many different disciplines as possible would be represented in our scoping review. The following OVID databases were searched: 


·        MEDLINE(R) and Epub Ahead of Print,In-Process & Other Non-Indexed Citations and Daily (1946 – Sept 13, 2020),

·        EMBASE (1974 – Sept 13, 2020),

·        APA PsycInfo (1806 to Sept 13, 2020).


EBSCO databases were searched simultaneously. One of the major reasons to search databases individually is to ensure effective subject heading searching. However, with the EBSCO databases, none of the existing subject heading were useful in retrieving relevant articles. The subject headings related to KS retrieved knowledge synthesis reviews rather than articles that discuss KS as a topic. As no appropriate subject terms/headings were found, we decided that there was no need to search the databases discretely. EBSCO databases included were:


·        SocINDEX with Full-Text (1908 to Sept 13, 2020)

·        Education Research Complete (1880 to Sept 13, 2020),

·        ERIC (1966 to Sept 13, 2020),

·        CINAHL Plus with Full-Text (1937 to Sept 13, 2020),

·        Library and Information Science Source (1901 to Sept 13, 2020),

·        Academic Search Complete (1887 to Sept 13, 2020),

·        Business Source Complete (1886 to Sept 13, 2020),


Additional databases searched included:


·        LISA: Library and Information Science Abstracts (ProQuest, 1969 to Sept 13, 2020),

·        Web of Science Core Collection. This core collection includes:

·        Science Citation Index-Expanded (1900 to Sept 13, 2020),

·        Social Sciences Citation Index (1900 to Sept 13, 2020),

·        Arts & Humanities Citation Index (1975 to Sept 13, 2020),

·        Conference Proceedings Citation Index - Science (1990 to Sept 13, 2020),

·        Conference Proceedings Citation Index - Social Sciences & Humanities (1990 to Sept 13, 2020),

·        Emerging Sources Citation Index (2005 to Sept 13, 2020).


Searches were conducted on May 31, 2019 and updated on September 13, 2020. Results were downloaded in RIS or text format, and deduplicated in Covidence software ("Covidence," n.d.).


Our third and final step included the hand-searching of relevant journals and conferences, as well as scanning the reference lists of included articles and the associated cited-bys. We hand-searched issues published within the last three years (2017-2019) of the following journals: Journal of the Canadian Health Libraries Association, Journal of the Medical Library Association, Evidence Based Library and Information Practice, and Research Synthesis Methods. We also hand-searched the programs from the following annual conferences: European Association of Health Information and Libraries (2017-2019), Medical Library Association (2017-2019), Canadian Health Libraries Association (2017-2019), Association of European Research Librarian - LIBER (2017-2019), and Evidence based Library and Information Practice (2019). Additionally, we conducted forward and backward citation searching by scanning the reference lists and the cited-bys (via Google Scholar) of all included articles. Where further details were required, authors of the included studies were contacted via email.


Study Selection


Study selection was conducted in two phases, first by title/abstract, and then using the full-text. The process was completed in duplicate, using two independent reviewers (ZP and SR). We first piloted a random set of 50 records to ensure that the eligibility criteria were clear and consistently applied by both screeners. A third independent reviewer resolved discrepancies (KAH). A similar process was followed for the full-text screening, which was also done independently in duplicate (ZP and SR), with a third reviewer resolving discrepancies (KAH). Covidence software was used to facilitate the study selection process.


Data Extraction


Data were extracted in Excel. The following categories were extracted from each included article:



A data extraction template was created in Excel and was piloted by two individuals independently, using 3 studies. Data extraction was then completed by one individual (SR) and was verified independently by a second individual (ZP). Verification was done by checking each data point extracted by the first individual against the original source article. When discrepancies were found, they were first discussed between the data extractor and data verifier. A third individual reviewed any discrepancies in coding that were not easily resolved through the initial consensus process (KAH).




The data collection process identified 4,857 records for title/abstract screening, of which 2,112 were duplicates. After applying inclusion criteria to the 2,597 unique records, 48 articles were left for full-text screening. At the end of the full-text screening process, 17 articles remained that met the inclusion criteria for this scoping review. Inter-rater agreement for the title/abstract screening was 98%, and for the full-text screening was 87%. The inter-rater agreement was calculated automatically by Covidence and is the proportionate agreement level between the two reviewers across the entire set of records or articles. This means that the two reviewers voted the same way on 98% of the total records during title/abstract screening and 87% of the articles during the full-text screening stages. The results of the study selection process are reported in a modified PRISMA flow diagram (Moher et al., 2009) in Figure 1 below.


Figure 1
PRISMA flow diagram.



Description of Included Articles


The population (discipline, learner level) and intervention characteristics (course-integrated or stand-alone, instructors, location) of the 17 included articles in this review are shown in Table 1 below.


The majority of articles describe interventions from North America, with six from the United States, and four from Canada. Three were located in the United Kingdom, with an additional one each from Germany, Italy, New Zealand, and Zimbabwe. Most instruction targeted graduate students as learners. The majority (12) of the articles describe instruction where KS was the focus of an entire credit course or where teaching KS was integrated into such a course, whereas the other five articles describe stand-alone workshops. KS instruction was taught to a broad range of disciplines. Many of the articles describe KS instruction related to the health sciences (i.e., Dentistry, Nursing, Biomedical Sciences, Exercise Science, Public Health and Speech Pathology) which reflects the prevalence of KS in these disciplines. Faculty were involved as instructors in all but three of articles, the remaining of which were taught by librarians. In seven of the articles, teaching was shared to varying degrees among faculty, librarians, teaching assistants and facilitators.


Inclusion criteria for our review dictated that all included workshops or courses covered content related to at least two steps of the KS process, but as Table 2 shows, most covered many more. The stand-alone workshops, which were of shorter duration than the credit courses, included fewer steps of the KS process. The three workshops taught exclusively by librarians (Campbell et al., 2016; Conte et al., 2015; Lenton & Fuller, 2019) taught the fewest steps. This could be due to the fact that these workshops were shortest in length, and also because the steps covered (problem definition, searching, and citation management) are those that align most closely with librarian expertise (Spencer & Eldredge, 2018). All 12 credit-bearing courses taught research question formulation, searching, screening and data extraction. Two of the articles for course-based instruction (Azarpazhooh et al., 2008; Groller et al., 2020) explicitly describe the teaching of five steps, whereas all other courses covered six or more. The most commonly taught step was “Searching the literature,” which all 17 articles describe. This was followed by “Defining the Question” (16 articles), “Reporting” (15 articles) and “Screening” (14 articles). The least common step to be taught was “Narrative Synthesis” (five articles).



Table 1
Population and Intervention Characteristics of Included Studies




















Biomedical Sciences






Educational Psychology






Exercise Science



Health Economics

Professional Librarians







Public Health



Speech Pathology





Learner Level


Graduate Students

















Workshop Design



Course Integrated














Faculty Only




Faculty + Librarian(s)




Librarians Only




Faculty + TAs




Faculty + Librarians + TAs












United States








United Kingdom










New Zealand








Table 2

Steps of the Knowledge Synthesis Process that were Taught in the Content of Each Course or Workshop



Our review captured a very diverse set of courses and workshops teaching knowledge synthesis review methodology. Tables 3 and 4 display the summaries of instruction interventions described in the included literature. The data are presented in two tables, with course-based instruction and stand-alone workshops displayed separately because of some clear differences between the two types of offerings.


Table 3 presents a summary of the stand-alone workshops. All five workshops included limited contact time with learners, ranging from three hours in total (Campbell et al., 2016) to five full days (Flores-Mir et al., 2015; Jack et al., 2020). The course objectives for these workshops are stated in terms of preparing attendees to participate in future reviews, which is appropriate given their short duration. They aim to build capacity rather than to give students extensive experience in conducting reviews. Librarians were the sole instructors in three of the workshops (Campbell et al., 2016; Conte et al., 2015; Lenton & Fuller, 2019). The workshops targeted a more diverse group of learners than the credit courses, usually including a mix of levels (undergraduates, graduate students, post-docs, researchers, librarians, professional staff). Also, without the graded assignments available to instructors in a credit course, there were more limited examples of student assessment. Two of the workshops (Campbell et al., 2016; Flores-Mir et al., 2015) do not mention assessment of student learning at all. Two of the articles mention conducting pretests and posttests (Conte et al., 2015; Jack et al., 2020), and two articles describe assigning participants an assessment activity at the end of the workshop (Jack et al., 2020; Lenton & Fuller, 2019). All of the workshops offered some form of post-course evaluation survey.


The 12 credit courses are summarized in Table 4. Four of the courses were offered to undergraduate students, while eight were at a graduate level. Faculty members were the primary instructors for all the courses, and the sole instructors for seven. Five articles (Briner & Walshe, 2014; Gorczynski et al., 2017; Groller et al., 2020; Li et al., 2014; Proly & Murza, 2009) explicitly mention librarian involvement either within the original course or as a modification for later offerings based on feedback. The course objectives generally focus on students developing an understanding of reviews and the skills to conduct one. Some unique objectives include learning to teach others about systematic reviews (Li et al., 2014) and gaining project management skills and leadership experience (Proly & Murza, 2009). The articles present a variety of graded assignments designed to assess student learning, many of them tied to specific steps of the review process. Oral presentations were assigned in five courses and students created a poster presentation for one course (Bourke & Loveridge, 2013). Ten of the courses required students to hand in either a written summary of findings or a research manuscript based on their review. The most common form of course assessment used was a post-course questionnaire or survey, mentioned in seven articles. Groller et al. (2020) also discusses an online survey specific to the information searching session offered by the librarian. Other forms of course assessment include a focus group (Azarpazhooh et al., 2008), student self-assessments (Briner & Walshe, 2014), faculty observations (Briner & Walshe, 2014), and an analysis of student performance (Land & Booth, 2020).


One of the primary goals of this study was to investigate instruction methods for teaching knowledge synthesis methodology. Table 5 explores the variety of teaching and learning strategies implemented for different steps of the knowledge synthesis process. We did not include traditional lecturing as we were most interested in discovering active teaching and learning strategies. The coding is not discreet - that is, multiple learning strategies may be employed in teaching a single step. For example, database searching may be coded as both hands-on and small group, as the participants worked together to develop search strategies. The majority of articles only briefly mention specific teaching and learning strategies. Li et al. (2014), for instance, states “we developed this course with a philosophy of “learning by doing” (p. 255) but provides little detail on the learning activities and teaching strategies used.  Similarly, Jack et al. (2020) mentions that learners participated in interactive exercises in groups; however, only one example is given. Pieper et al. (2019), who also noted that they used a “learning by doing” philosophy, followed a unique approach implementing a “guiding systematic review” which is a published systematic review used as a “working example throughout the course” (p. 3). A wide range of active learning and teaching strategies were employed across the courses and workshops, with hands-on or small group activities being most commonly mentioned. Hands-on activities were used most for teaching the steps of question development, database searching, screening, data extraction, and critical appraisal. These steps are mirrored in the small group activities, as small group activities often included hands-on experiences.



Table 3
Summary of Workshop Characteristics

Author, Date, Country 



workshop structure


Course Objectives 

Assessment of Student Learning 

Course Evaluation 

Campbell, 2016, Canada 




3hr workshop  


Participants will: identify systematic reviews, recognize the range of resources required to execute a systematic review search, develop a well-formulated search question and structure a search using the PICOS format, learn to apply appropriate search limits, document a search in a standardized form, understand the importance of peer-review of systematic review searches, and recognize the level of expert searching needed for a systematic review 


Evaluation questionnaire  

Conte, 2015,




2-day workshop 


Students will gain knowledge of best practices in conducting systematic reviews and create a personalized action plan to establish their libraries as centers of expertise for systematic reviews  

Online pre and posttests, 

Online post-course survey, MLA evaluation form, Focus group 

Flores-Mir, 2015, Canada 


faculty/graduate students/staff,

5 x 8hr sessions 

Faculty, Librarian (as guest lecturer)  

Students will broaden knowledge of evidence based practice principles in Dentistry and gain hands-on experience in designing, conducting, writing, and critiquing health care systematic reviews.   


Post-workshop evaluation forms 



2020, Zimbabwe 


PhD/Post-doc/Librarians/ Program Managers,

5-day workshop 


To teach trainers from three African countries to conduct systematic review workshops at their home institutions in order to broaden mental health research capacity 

Online pre and posttests, learner presentations at end of workshop 

post-workshop survey assessing learner satisfaction and perception of confidence in conducting a SR 

Lenton, 2019, Canada 


graduate students,

3 x 2.5hr sessions 


Students will learn to identify differences between types of reviews, incorporate tools & resources for proper reporting & management of the review, utilize strategies for creating a searchable question with inclusion/exclusion criteria, identify relevant databases, practice using a structured method for developing advanced search strategies  

Student observation during activities, ticket-out-the-door evaluation forms. Short post-course reflection questionnaire 

Short post-course reflection questionnaire 




Table 4

Summary of Course Characteristics


Date, Country 

Discipline, level,

course structure 


Course Objectives 

Assessment of Student Learning 

Course Evaluation 

Azarpazhooh, 2008, 


Dentistry, Undergrad,

3 x 1hr lectures, 3 x 2-3hr discussion sessions, 1 2-3hr presentation session 

Faculty, Facilitators 

Students will develop and apply skills in evidence based dental practice by finding relevant literature, evaluating and selecting the strongest evidence, summarizing findings, and communicating results  

Students evaluated on quality of participation in group discussions, group presentations and on summary reports of findings 

Online pre and posttests, online post-course survey, MLA evaluation form, focus group 

Baldasarre, 2008,


Electrical Engineering, Masters,

10 sessions 

Faculty, PhD Students 

Students will be introduced to empirical research methods and trained to empirically evaluate software engineering tools, techniques, methods and technologies 

Definition of research protocol assignment, definition of inclusion/exclusion criteria assignment, data extraction assignment  


Post-course questionnaire  




New Zealand  

Educational Psychology, Masters,

not specified 


Not provided in article 

Poster presentations of initial finding of systematic reviews; students then submit full systematic review incorporating faculty & peer feedback on posters. 

Student self-assessments throughout course   




Business, Masters,

7 x 3hr sessions 


Librarian (as guest lecturer) 

Students will gain understanding of evidence based practice and conduct a rapid systematic review 

5-minute presentation on the review question. Research question and outline a few weeks before final deadline. Rapid Evidence Assessment (max 4000 words), evaluated on a clear, answerable review question, sound justification for conducting the review, explicit search strategy, ways of judging the quality of the research, and conclusions that accurately reflect the findings.   

Faculty observations of student experience; student presentations had to answer question "what problems or pleasant surprises have you encountered so far?” 

Gorczynski, 2017,


Exercise Science, Masters,

not specified 


Students will learn to structure evidence based interventions and carry out valid and reliable evaluations 


Students identify an area of mental health and conduct a qualitative systematic review that examines the impact of physical activity on their chosen mental health topic.

Solve weekly case studies using new knowledge and lead discussions presenting their proposed interventions and supporting rationale.  

Quantitative and qualitative mid-year and year-end evaluations 





Nursing, Undergrad, approx. 120 hrs 

Faculty, Librarian 

Students will learn to design, conduct and disseminate results of a collaborative scoping review 

Individual paper reviewing about seven articles, determining suitability for answering research question, and then summarizing implications for clinical practice, policy, education and further research.  

Group oral presentation of research findings, open to campus community. 

Online survey on library session, with three open-ended questions. Online post-course evaluation survey with 15 Likert-scale  & 4 open-ended questions 

Himelhoch, 2015,


Psychology, Residents,

9 lectures 


Students will learn the fundamentals of systematic reviews and meta-analysis, learn to select a good research question, establish eligibility criteria, conduct a reproducible search, assess study quality, organize data and conduct meta-analysis, and present findings 

Eight assignments. 1) create a PICO informed research question 2) Define and describe eligibility criteria 3) Conduct literature search and document results 4) Interrater reliability assignment and PRISMA flow diagram 5) create risk-of-bias table and summary table for included papers 6) Collect, organize, and document data to enable calculation of weighted effect size 7) Present and interpret forest and funnel plots 8) write scientifically formatted manuscript ready for peer review. 

Anonymous course evaluation - 38-questions on Likert scale + 3 open-ended questions 





Biomedical Science, Undergrad, 3 x 2hr classes, ongoing faculty consultation 



Students will develop the skills to conduct an independently researched systematic review and meta-analysis (SRMA) capstone project in their final year 

A systematic review and meta-analysis, done as a proforma report 

Analysis of student performance across program to measure effectiveness of the systematic review exercise




Public Health, Masters & PhD, 6hr/wk x 8 weeks 

Faculty, Librarians, Teaching assistants 

Students will learn the steps of performing  

systematic reviews and meta-analyses and improve their ability to perform, critically appraise, and teach others about systematic reviews 


Graded assignments include three open-book quizzes, individually submitted review protocol, and individually submitted final report on group's systematic review.  Students orally present reviews to class and respond to comment. 

Anonymous evaluation before final paper. Post- course survey offered to students who took course 2004-2012; second survey sent to past participants on long-term effects of course.


2019, Germany

Health Economics, Undergrad,

1.5 hrs x 14 or 15 weeks


Students will learn the fundamentals of systematic review methodology and develop skills to critically appraise other systematic reviews

Students complete a 10-12 page systematic review based on topics selected by instructor and reported according to PRISMA guidelines

Students complete a validated post-course questionnaire to assess instructional quality





Speech Language Pathology, Masters & PhD, not specified 

Faculty, Librarian (as guest lecturer) 

Students will: develop understanding of intervention research design and clinical implications of evidence based practice, develop analytical skills to assess the quality of research evidence, gain project management skills. Doctoral student will gain leadership experience. 

Major course assignment was development of a coding form and code-book specific to each group’s topic, research question and inclusion/exclusion criteria.   Students also had an assignment requiring hand calculation of effect sizes. All students had to register their topic with the Education Coordinating Group of the Campbell Collaboration.  

Not specified 

Upchurch, 2002,


Nursing, Masters,

not specified 


Course 1: Students will gain skills to examine the literature, maintain a bibliographic database, practice statistical analysis, select a problem area and type of data for a research project. Course 2: Students will complete the literature review or simple meta-analysis and prepare a written report.  

Students do a class presentation of their problem area, research question, background and significance. Students design a coding sheet specific to their research question.  Students write a research manuscript emphasizing their methods, findings and implications.   

Not specified 



Assessment Outcomes, Student/Instructor Feedback, and Recommendations


Designing an effective course or workshop series to teach knowledge synthesis methodology requires an understanding of which steps and discrete tasks students find challenging. Instructors must incorporate appropriate time, content, and learning activities to address these challenges and develop assessments to evaluate whether these interventions are effective. All the articles in our study include some form of assessment, although some discuss assessment in far greater detail than others. Challenges and benefits to teaching knowledge synthesis, student feedback, and instructor reflections from the 17 included articles are summarized below, categorized by the methodology step(s) to which they correspond.


Question Formulation and Refinement


Almost all of the included courses and workshops (16) teach question formulation or topic refinement, which often also includes setting inclusion criteria and limits. (see Table 2)


Determining and focusing the research question is an important first step in a knowledge synthesis project. A broad question may be feasible for a research team with many members working over an extended time period, but may be overwhelming for a small group of students completing a course project. Some articles report that students found this step challenging, either due to the ambiguity and iterative nature of the question refinement process, or because of the difficulty in finding a question that is manageable and appropriate for a course assignment (Briner & Walshe, 2014; Upchurch et al., 2002). In one article that describes two sequential research courses, graduate students initially pick a topic of interest, although they do not complete a knowledge synthesis project during the first course (Upchurch et al., 2002). However, in the subsequent course the students take their previously-chosen topic and refine it into a question appropriate for research synthesis. Upchurch et al. (2002) report that developing the final research question and clarifying inclusion criteria is an iterative process that students may find frustrating. The instructors built in extra time at the beginning of the course for students to refine their question. Even when students know they are picking a topic for the purpose of conducting a small systematic review, the process of settling on an appropriate review question can still be challenging. Briner and Walshe (2014) emphasize this through a student’s quote, stating that they “really underestimated the difficulty of asking the right question ahead of formulating a search strategy” (p.426). They also mention that students were often frustrated by the lack of consistency in the way that concepts were defined in the literature, making it difficult to operationalize what seemed like a simple idea or concept. This further adds to the difficulty in settling on an appropriate research question.



Table 5

Teaching and Learning Strategies for Knowledge Synthesis Steps

Knowledge Synthesis Step



Teaching and Learning Strategies

Hands-on Activities/ Experiential Learning

Small Group Work/ Discussion

Student Presentations

Case Studies / Guiding Review

Guided / Facilitated Exercises

Individual Work

Large Group Work/ Discussion

Working in Pairs/

Pair Activities


Peer Feedback/


Role Playing

Analyze Seminal Readings

Protocol Development




3  15


7  9







Defining the Question

1  6  10  11 13

1  2  4  7  10

1  4  6

3  15










1  2  3  4  6  7  8  9  10  11  13  14  15  17

2  3  4  7  10  13  14  15

6  13

3  16

4  11 13

3  6  13  16



6  13




Screening Inclusion / Exclusion

1  2  3  4  6  7  17

1  2  3  4  7 15


3  16  17


15  16


6  17





Citation/Reference Management

1 4   7












Data Extraction

1  2  3  4  6  7  9  17

1  2  3  4  7  15


3  17

3  4

3  15  16







Critical Appraisal / Risk of Bias

2  4  7  10  17

2  4  7  10


16  17























1  7  10  17

1  7  10











Manuscript Draft/ Completed Review


2  7  10

2  5  6  7  10  15



1  7  15  16




1  5



Reporting / Data Management

13  17












Phase not specified







8 12







1 Upchurch, 2002

5 Bourke, 2013

9 Flores-Mir, 2015

13 Lenton, 2019

17  Pieper, 2020


2 Azarpazhooh, 2008

6 Briner, 2014

10 Himelhoch, 2015

14 Jack, 2020



3 Baldassarre, 2008

7 Li, 2014

11 Campbell, 2016

15 Groller, 2020



4 Proly, 2009

8 Conte, 2015

12 Gorczynski, 2017

16 Land, 2020








One strategy to simplify the process of research question development is for the course instructors to pick a list of topics that they know may be feasible for a course assignment. In one course, this was done effectively by using a set of topics or areas of focus that were important to stakeholders as a starting point from which to develop a relevant question (Bourke & Loveridge, 2013). In Pieper et al. (2019) and Land and Booth (2020), students were either given a specific topic or selected from a carefully curated list of topics; topics were vetted by the instructor in order to ensure a manageable volume of results from the search. However, Li et al. (2014) describe another situation where, despite best intentions and a clear set of criteria, some of the topics suggested each year “result in students’ searches that identify tens of thousands of titles and abstracts requiring screening or many more primary research articles meeting the students’ inclusion criteria” (p. 258). Therefore, further intervention and guidance is required from the instructors on how to narrow a topic. However, selecting appropriate topics for students is a challenging task. Pieper et al. (2019) discuss some criteria they felt would be appropriate when identifying suitable topics, such as a small number of search terms and synonyms, reasonable volume of search results, and so on.


Several articles suggest highlighting the difficulty and importance of rigorous question formulation (Briner & Walshe, 2014; Gorczynski et al., 2017; Upchurch et al., 2002). Furthermore, students learned the importance of the research question in determining the body of evidence (Baldassarre et al., 2008) and of choosing the right question (Briner & Walshe, 2014).


Protocol Development


Developing a protocol was either taught or assigned as an assessment of student learning in seven of the courses or workshops. In one class, determining inclusion/exclusion criteria, collecting search terms, and defining the data extraction criteria were assigned as homework , and in the following class students discussed their submissions (Baldassarre et al., 2008). Although not explicitly about protocol development, students in one course requested additional information and assistance with setting inclusion criteria which is one of the components that needs to be defined in a protocol (Gorczynski et al., 2017). In Li et al. (2014), creating the protocol was worth a significant portion of their final course grade, and students suggested that this be a group assignment rather than an individual assignment. Protocol development as a group reflects the real-life experience of researchers when developing their review protocol as a team. In Proly and Murza (2009), the goal of the 15-week course was to submit a review title and protocol to the Campbell Collaboration.


Searching for Studies (Data Collection)


In all of the 17 included articles, searching for evidence was taught as part of the course or workshop.


Searching for KS research must be comprehensive and exhaustive, and attempts must be made to gather all relevant evidence. For students conducting a KS project for the first time, this level of comprehensiveness in searching is likely new. KS course assignments may not require the level of exhaustive searching expected in a full KS review, however the level of comprehensiveness required is still likely greater than what students may be doing for other assignments. In faculty-led courses or workshops, librarians were sometimes invited to teach the search process; this was mentioned in six articles (Briner & Walshe, 2014; Flores-Mir et al., 2015; Gorczynski et al., 2017; Groller et al., 2020; Li et al., 2014; Proly & Murza, 2009).


Student feedback suggests that they recognized the importance, difficulty, or time-consuming nature of searching for evidence (Baldassarre et al., 2008; Briner & Walshe, 2014; Groller et al., 2020). They suggested that more time be allocated for learning how to search, and that additional guidance or handouts to aid with searching be included as part of the content (Campbell et al., 2016; Gorczynski et al., 2017; Lenton & Fuller, 2019). Groller et al. (2020) report that the librarian provided additional, unplanned sessions with each group in order to meet the criteria set out in the pre-established search protocol. These consultations with librarian search experts were found to be beneficial. Despite the challenges, students felt that their experiences in the courses led to improved abilities, skills, or confidence in gathering, searching, or locating evidence (Azarpazhooh et al., 2008; Conte et al., 2015; Proly & Murza, 2009). The course described by Groller et al. (2020) included an evaluation of the library research session. Student feedback highlighted learning about new databases, learning the different way that searches can be executed, and noting that the library skills learned would have been useful throughout their four years at university.


Study Selection and Data Extraction


Study selection and data extraction were taught in 13 courses/workshops (see Table 2). All 12 articles that describe credit-bearing courses covered both study selection and data extraction. Study selection is required to arrive at a set of included studies from which data can be extracted and the evidence synthesized. One of the stand-alone workshops (Jack et al., 2020) discussed the step of study selection, but did not address data extraction.


Even though inclusion criteria are determined in the earlier stages of a KS review, further refinement to the criteria can sometimes occur during the study selection process. Additionally, reading and interpreting academic literature are skills that are required within the study selection and data extraction steps of a knowledge synthesis project. Reading, analyzing, and interpreting academic research were reported as challenging by students (Briner & Walshe, 2014; Upchurch et al., 2002). Students sometimes requested additional information or further assistance with the process of extracting data (Gorczynski et al., 2017). Both students and instructors suggested allocating more time for extracting data (Gorczynski et al., 2017; Li et al., 2014). Briner and Walshe (2014) state that the process of developing and applying criteria for inclusion/exclusion helps learners become active and critical consumers of information.


In Pieper et al. (2019), students practiced data extraction by extracting data for one of the studies included in a previously published systematic review; students then checked their data extraction against the published systematic review, thus allowing students to verify the accuracy of their work. In another course, students participated in a pilot data extraction exercise in class to prepare them for the data extraction process (Baldassarre et al., 2008). Students had to independently extract data from one of two pre-selected papers, and then compared their results with another student who worked on the same paper. Eventually, students received the instructor’s data extraction for final comparison. Feedback on the guided exercise was positive, but “some students found it difficult to understand the meaning of the cells in the table” (p.422). This exercise highlights the value of piloting the data extraction process, but also demonstrates the challenges of the data extraction step. Students also found it difficult to extract data from articles on unfamiliar topics (Baldassarre et al., 2008). This underscores the value of having some familiarity with the topic for data extraction.


Synthesis and Critical Appraisal


5 articles cover narrative synthesis, 9 articles explicitly mention meta-analysis, and 11 articles include the step of critical appraisal/risk of bias (see Table 2).


Analysis or synthesis were either noted as challenging tasks (Upchurch et al., 2002) or mentioned as particularly time-consuming, with a suggestion that additional time be allocated to this step (Li et al., 2014). However, students also felt that learning this step improved their ability to analyze, critically evaluate, or apply information (Azarpazhooh et al., 2008; Groller et al., 2020; Proly & Murza, 2009). They also became more critical of evidence (Bourke & Loveridge, 2013) or skeptical of research findings (Briner & Walshe, 2014). Students were surprised by the limited quantity, quality, and relevance of the research they found. The synthesis and appraisal process thus allowed learners to develop an awareness of the variations in quality and relevance of existing research (Briner & Walshe, 2014). Students improved their critical thinking skills and their ability to critique published systematic reviews (Flores-Mir et al., 2015). However, Land and Booth (2020) note that students “tend to gloss over the detail of forest plots to focus on the bottom-line result” or to focus on the basic interpretation of the funnel plots “without attempting a deeper analysis of the data” (p.283). Suggestions and guidance for addressing these challenges are also provided in their article.


Data Management, Documentation, and Reporting


Due to the volume of references or citations that need to be downloaded and managed, and the explicit requirement to report every aspect of the methods, data management, documentation and reporting are often taught as part of both stand-alone workshops and credit courses. All 17 articles include either data/citation management (10 articles), or documentation and reporting (15 articles), and nearly half included both (see Table 2).


Conte et al. (2015) suggest incorporating additional content on data management, reporting, and documentation. Upchurch et al. (2002) suggest that learners should keep a procedure manual to document the research process. Introducing different reference management software is also recommended (Gorczynski et al., 2017). Campbell et al. (2016) initially included a greater amount of time to cover reference management, but time constraints resulted in less coverage in a later iteration of the workshop. Instead, instructions on reference management were provided via tutorials made available prior to the in-class workshop.


Instructional Design and Teaching Strategies


In addition to discussing challenges, feedback, suggestions, or recommendations related to course content, many articles discuss instructional design or course structure. Azarpazhooh et al. (2008) mention that frequent, shorter sessions were preferred over a longer 3-hour session, however Lenton and Fuller (2019) state that students in their workshops preferred longer sessions in order to more fully cover the content. Allowing more time for learning activities, hands-on practice, or group work is suggested in many articles (Campbell et al., 2016; Flores-Mir et al., 2015; Gorczynski et al., 2017; Li et al., 2014).


There is no consensus on whether group or individual assignments are preferred, however many articles stress the value of students working collaboratively with peers or advisors throughout the review process. Li et al. (2014) mention that assignments should be group rather than individual, whereas