Introduction
Evidence
based medicine (EBM) is healthcare practice based on integrating knowledge
gained from the best available research evidence, clinical expertise, and
patients’ values and circumstances (Dickersin,
Straus, & Bero, 2007). Evidence based practice emphasizes new skills
that learners must acquire and use: question formation, search and retrieval of
the best available evidence, critical appraisal of the evidence, and
application of the evidence to an individual or a group of patients (Finkel, Brown, Gerber, & Supino, 2003; Montori
& Guyatt, 2008). EBM serves as a powerful educational tool or
strategy that allows students and clinicians to become lifelong self-directed
learners and information masters able to fill their knowledge gaps and practice
high quality medicine (Barnett, Smith, &
Swartz, 1999; Bordley, Fagan, & Theige, 1997).
EBM
as an approach to practicing medicine has spawned widespread attention from
medical educators who consider EBM as an important content area to be addressed
in the undergraduate medical education curriculum. The Association of American
Medical Colleges (AAMC) established the
Medical School Graduation Questionnaire (GQ), a national questionnaire, for
medical schools to use in program evaluation and to improve the medical student
experience (Association of American Medical
Colleges, 2012). The questionnaire contains specific items asking
graduating medical students to self-report the level of adequacy in EBM instruction
in their medical education curriculum. The Liaison Committee on Medical
Education (LCME) that accredits medical education programs in the United States
and Canada considers GQ data as part of important evidence demonstrative of the
educational quality of new and established medical educational programs (Melnyk, 2006). The LCME also includes
education standards for EBM related skills being appropriately addressed in a
medical education program (Liaison Committee on
Medical Education, 2012). To develop and implement innovations to
medical schools across the United States, the Undergraduate Medical Education
for the 21st Century project identified the practice of EBM and
population-based medicine as one of nine key content areas to be integrated
into medical school curricula in the clinical years (O'Connell & Pascoe, 2004).
At
the level of graduate medical education, the accreditation body for residency
programs in the United States – the Accreditation Council for Graduate Medical
Education (ACGME) – endorsed six core competencies for residents in residency
programs, one of which is the competency in practice based learning and
improvement (Steward, 2001). The
competency requirement stipulates that residents demonstrate the ability to
locate, appraise, and apply evidence from scientific studies to their patient
health problem. To better prepare medical students for beginning a medical
residency program, a medical education program is expected to incorporate EBM
into its curriculum to teach students basic skills in clinical decision making
and application of evidence based information to medical practice.
As
consumers of health/medical information, physicians are inundated with diverse
and abundant information resources with variable quality and clinical relevance.
They face a formidable challenge of staying current with biomedical knowledge
and applying the growing knowledge to specific patients. “The skills needed to
find potentially relevant studies quickly and reliably, to separate the wheat
from the chaff, and to apply sound research findings to patient care have today
become as essential as skills with a stethoscope” (Glasziou, Burls, & Gilbert, 2008, p. 704). Given the health
care environment that is changing constantly and becoming increasingly sensitive
to optimizing the quality and cost of patient care, medical educators face an
important task of developing curricula that provide a sound scientific
foundation for clinical practice and scholarly investigation and that prepare
medical students to become physicians to practice medicine competently in the
21st century (Mahoney et al., 2004).
Medical
students need to master an enormous volume of medical knowledge in their
medical education from various sources. They may be very comfortable in using
Google or Wikipedia to search for information. However, they lack proficiency
in identifying, searching, and filtering information appropriate to address
different types of clinical questions. Thus, it is essential for them to
develop high level information literacy and acquire fundamental, important
skills for lifelong, self-directed, problem based learning from the early years
of their medical education. Basic EBM skills of searching, appraising, and
applying research evidence to individual patients should be taught early and
applied as an integral part of learning throughout the four years of the
medical education (Glasziou et al., 2008). EBM integrated into the
curriculum will be able to educate students who will become more effective
residents and, ultimately, practicing physicians best able to make informed
clinical decisions about the care of patients and to handle the exponential
growth of biomedical knowledge.
It
is important for medical educators and health sciences librarians to become
cognizant of the available education research evidence and current practice in
teaching EBM in undergraduate medical education. The knowledge will aid them as
they launch their efforts of designing, developing, and implementing an EBM
intervention, whether it is in the format of a curriculum, course, or series of
workshops in preclinical or clinical years. A comprehensive literature search
yielded no systematic or other types of reviews on EBM in the undergraduate
medical education. This qualitative review of studies on EBM was conducted to
determine the year when EBM was introduced and the extent to which medical
students were exposed to EBM in undergraduate medical education. The review
also attempted to examine how EBM interventions were designed, developed,
implemented, and evaluated in the medical curriculum.
Methods
Data
Sources
The
review of the literature covered the period from 1997, the publication year for
David Sackett’s seminal work Evidence-Based Medicine: How to Practice and Teach EBM, to 2011,
when the literature review for this article was performed. Since 1997, Sackett’s work has helped drive the paradigm shift in the
practice of medicine and has had a far reaching impact on the evolution of EBM
teaching across the continuum of medical education.
The
literature search included PubMed, CINAHL, Web of Science, Cochrane Library, ProQuest Dissertations & Thesis, PsycINFO,
and ERIC. PubMed was searched by subject with MeSH
terms (medical subject headings): education, medical, undergraduate; students,
medical; evidence-based practice; and evidence-based medicine. The search
results were limited to the language of English. PubMed and other databases
were searched with the text words of evidence-based medicine, evidence-based
practice, and evidence. These words were combined with medical education,
undergraduate medical education, medical school, clerkship, medical students,
as well as variations of the following text words: teach*, learn*, train*, educat*, instruct*, curricul*,
and program. The search of these databases was supplemented by hand searching
of reference lists of all included articles. Cited references were also
searched in Web of Science to identify relevant articles that cited those
studies selected for the review.
Selection
Criteria
Studies
were included in the review if they evaluated the effect of formal EBM teaching
in undergraduate medical education programs leading to a medical degree
accredited by the LCME. Formal EBM teaching was defined as any intervention in
the form of a curriculum, required or elective course, EBM components
integrated into a preclinical course, curriculum, or clinical clerkship
rotation.
The
types of research design utilized in clinical research may be inadequate in
researching, investigating, and reporting educational phenomena with
complexities of educational programs and wide variations in instructional
methods, learning outcomes, and differences in learner characteristics.
Qualitative research methods or alternative educational research methods are underutilized
and valuable research tools in medical education research. As Hatala and Guyatt pointed out,
well-designed qualitative studies using these methods are equally valuable in
contributing evidence to the research of teaching evidence based medicine (2002). Therefore, studies that utilized the
following types of research design (Table 1) commonly employed in education
research were selected for the review (Campbell
& Stanley, 1963; Gall, Gall, & Borg, 2007). Further information
on each research design type can be found in Appendix A.
Table
1
Types
of Research Design
|
Control-group
designs with random assignment
|
|
1.
Pretest-posttest
control-group design
R O
X O
R O O
|
|
2.
Posttest-only
control-group design
R
X O
R O
|
|
Quasi-experimental
designs
|
|
3.
Static-group comparison design
X O
O
|
|
4.
Nonequivalent
control-group design
O
X O
O O
|
|
Single-group
designs
|
|
5.
One-short case study
X O
|
|
6.
One-group pretest-posttest design
O
X O
|
Note. R =
randomization; O = pretest or posttest; X =
experimental treatment/intervention.
Articles
were excluded from the review if they were studies on:
·
EBM
teaching at postgraduate and continuing education levels
·
EBM
teaching that is not part of, or not integrated into a curriculum, an
existing/required course, program, or clinical clerkship rotation in
undergraduate medical education
·
EBM
in medical schools not accredited by the LCME
·
Effect
of EBM teaching in osteopathic medical education or other allied health
professional education
Data
Extraction
Data
were extracted from each eligible study on study design, year of EBM introduced
in the curriculum, setting, EBM intervention, outcome measured, instructional
method, and evaluation method. Furthermore, information on instructional
strategies was examined using the hierarchy of evidence based medicine teaching
and learning activities (Table 2) as
described by Khan and Coomarasamy (2006).
Kirkpatrick’s four levels of evaluation (Table 3) were
adapted to determine the extent to which EBM outcomes were measured (Kirkpatrick, 1994, 1996). The variation in
research designs, interventions, and outcome measures in the selected studies
precludes quantitatively combining results with a meta-analysis technique.
Table
2
Hierarchy
of Evidence Based Medicine Teaching and Learning Activities
|
Level 1
|
Interactive, and clinically
integrated teaching and learning activities
|
|
Level 2
|
a)
Interactive, classroom based teaching and learning activities
b)
Didactic, but clinically integrated teaching and learning
activities
|
|
Level 3
|
Didactic, and classroom or
standalone teaching and learning activities
|
Table
3
Four
Levels of Evaluation
|
Level 1
|
Learner satisfaction (reaction)
|
Perceptions, opinions, satisfaction
|
|
Level 2
|
Learning outcomes (learning)
|
Change in knowledge, skills, attitudes
|
|
Level 3
|
Performance improvement
(behaviour)
|
Change in behaviours, transfer of learning
to a patient care setting, performance in a practice setting
|
|
Level 4
|
Patient/health outcomes (results)
|
Tangible results--improvement in the health
and wellbeing of patients
|
Results
Literature
Search
The
search strategies identified 13 studies on EBM in undergraduate medical
education which met the inclusion criteria. These articles were derived from a
large pool of 1,279 articles through a process of title scanning, abstract
reading, hand searching, and elimination of duplicate articles from multiple
databases. The review of the studies was conducted within the framework of the
instructional design model ADDIE (the five phases or steps in designing
effective instruction): analysis, design, development, implementation, and
evaluation in instructional design (Gustafson
& Branch, 2007). The phase of analysis is beyond the scope of this
review. Table 4 provides a summary of the studies selected for the review.
Table
4
Summary of Study Design, Year of EBM in the
Curriculum, and Settings
|
Author
(Year)
|
Study
Design
|
Year
of EBM in the Curriculum and Setting
|
|
|
|
Wadland,
Barry, Farquhar, Holzman, and White (1999)
|
Static-group
comparison design
|
Year
3: clinical campuses in 6 communities
|
|
|
Barnett
et al. (2000)
|
Static-group
comparison design
|
Year
1
Year
3 and Year 4: clinical clerkships in medicine, pediatrics, psychiatry, surgery,
obstetrics and gynecology, neurology, community medicine, geriatrics
|
|
|
Ghali
et al. (2000)
|
Nonequivalent
control-group design
|
Year
3: ambulatory care sites during internal medicine clerkship
|
|
|
Thomas
and Cofrancesco (2001)
|
One-shot
case study
|
Year
3 and Year 4: ambulatory medicine clerkship rotation
|
|
|
Srinivasan
et al. (2002)
|
One-shot
case study
|
Year
1
|
|
|
Holloway,
Nesbit, Bordley, and Noyes (2004)
|
One-shot
case study
|
Year
1
Year
2 (same cohorts of first year medical students followed up throughout Year
2)
|
|
|
Dorsch,
Aiyer, and Meyer (2004)
|
One-shot
case study
|
Year
3: internal medicine clerkship
|
|
|
Cayley
(2005)
|
One-group
pretest-posttest design
|
Year
3: family medicine clerkship
|
|
|
Schilling,
Wiecha, Polineni, and
Khalil (2006)
|
Pretest-posttest control-group design
|
Year
3: family medicine clerkship
|
|
|
Nieman,Cheng,
and Foxhall (2009)
|
One-shot
case study
|
Year
1
|
|
|
Aronoff
et al. (2010)
|
One-group
pretest-posttest design
|
Year
3: core clinical clerkships
|
|
|
West,
Jaeger, and McDonald (2011)
|
One
group pretest-posttest design
|
Year
2
Year
3: clinical rotations of internal
medicine, surgery, pediatrics, obstetrics and gynecology, neurology, and
psychiatry
|
|
|
Sastre,
Denny, McCoy, McCoy, and Spickard (2011)
|
One
group pretest-posttest design
|
Year
3: inpatient portion of internal medicine clerkship blocks
|
|
Description
of Studies
The
13 studies demonstrated variability in methodology. One study had the pretest-posttest control group design with random assignment; three
studies had the quasi-experimental design; four studies used one group pretest-posttest design; and five studies utilized one shot case
study design. Participants of the studies ranged from first-year to fourth-year
medical students. Three studies reported the integration of EBM into
preclinical education curricula. EBM was introduced to third-year clinical
clerkships in eight studies.
Two
studies reported a longitudinal EBM intervention that spanned preclinical and
clinical years. Barnett et al. (2000)
evaluated EBM instruction integrated into the first-year medical education
curriculum and clinical education; while the study by West et al. (2011) reported the outcomes of a longitudinal
EBM into the second-year preclinical education and third-year core clinical
rotations. Studies on EBM during clinical rotations were conducted at various
settings of clerkships of internal medicine, family medicine, and pediatrics on
community campuses (Wadland et al., 1999),
outpatient internal medicine clerkship (Ghali et
al., 2000; Thomas & Cofrancesco, 2001), inpatient portion of
internal medicine clerkship blocks (Sastre et
al., 2011), family medicine clerkship (Cayley,
2005; Schilling et al., 2006), or core clinical clerkship rotations such
as internal medicine, surgery, pediatrics, obstetrics and gynecology,
neurology, and psychiatry (Aronoff et al., 2010;
Barnett et al., 2000; West et al., 2011).
Design
and Development of EBM Interventions
The
primary goal of EBM instruction in the 13 studies was to develop medical
students’ essential skills for evidence based practice and evidence based
problem solving; other goals included developing lifelong learners (Barnett et al., 2000; Dorsch et al., 2004),
introducing basic concepts of epidemiology and biostatistics (West et al., 2011), and developing and
presenting a research proposal (Wadland et al.,
1999). EBM skills were the main focus of the EBM interventions in the
studies; however, the EBM interventions reported in three studies included no
content on the second step of the EBM process – acquiring the evidence (Srinivasan et al., 2002; Wadland et al., 1999; West
et al., 2011).
There
were a varied number of contact hours devoted to the EBM interventions in the
13 studies. The time ranged from three student contact hours in a workshop (Sastre et al., 2011) to 30 contact hours in an
EBM curriculum (Holloway et al., 2004).
EBM components were introduced into preclinical years in various ways: offered
as a short EBM course of eight contact hours enhanced with problem based
learning small group discussion sessions facilitated by clinicians (Srinivasan et al., 2002); integrated into courses
such as Library Science and Medical Informatics, Epidemiology, Microbiology (Barnett et al., 2000), and Mastering Medical
Information (Holloway et al., 2004);
taught as part of a series of educational interventions in an elective family
medicine preceptorship program for first and
second-year medical students (Nieman, Cheng,
& Foxhall, 2009); and offered as a course of 22 contact hours at the
end of Year 1 (West et al., 2011). In
clinical years, EBM was integrated with clinical educational experiences through
inpatient and outpatient primary care clerkship rotations of internal medicine,
family medicine, and several other core clinical clerkships (Aronoff et al., 2010; Cayley, 2005; Dorsch et al.,
2004; Ghali et al., 2000; Sastre et al., 2011; Schilling et al., 2006; Thomas
& Cofrancesco, 2001; Wadland et al., 1999). Among the 13 studies,
only two studies, one by Barnett et al. (2000)
and the other by West et al. (2011), had
a longitudinal EBM curriculum that spanned the preclinical curriculum and clinical
core clerkships.
The
content domain of the EBM interventions in the 13 studies was derived from
multiple sources. They included Users’
Guides to the Medical Literature (Guyatt
& Rennie, 2002), a series of articles on “harnessing MEDLINE” (McKibbon, Walker-Dilks, Haynes, & Wilczynski,
1995; McKibbon & Walker-Dilks, 1994a, 1994b), EBM steps as outlined
by Sackett, Rosenberg, Gray, Haynes, and
Richardson (1996), and journal articles on EBM teaching. Four studies
failed to report how EBM content was developed, although the EBM interventions
in these studies reflected some basic EBM principles and processes widely
recognized in the medical literature on EBM teaching and practice. The EBM
curriculum in one study comprised only online content: a literature searching
tutorial, the website of the National Guideline Clearinghouse
(www.guideline.gov/), and the website on the calculation of NNT (number needed
to treat). Three out of the 13 studies included instruction on using
pre-appraised EBM resources such as DynaMed, ACP
Journal Club, Cochrane Library, and the National Guideline Clearinghouse (Nieman et al., 2009; Sastre et al., 2011; Schilling
et al., 2006).
EBM
knowledge and component skills in the EBM interventions were taught with
various methods such as didactic lecture, live demonstration, hands-on
practicum, small group discussion, and case based presentation. In addition,
journal club (Barnett et al., 2000),
problem based small group discussion (Srinivasan
et al., 2002), and worksheets (Dorsch et
al., 2004; Nieman et al., 2009; Srinivasan et al., 2002) were also
reported as interactive teaching and learning activities in the EBM
interventions. To categorize the EBM interventions in the 13 studies with the
hierarchy of effective teaching and learning activities proposed by Khan and Coomarasamy (2006),
five studies incorporated EBM teaching and learning activities of level 1,
which represents the most effective practice of teaching and learning of EBM;
one study used activities of level 2a; six studies integrated activities of
level 2b; and one study fell under level 3 activities.
Out
of the 13 studies, three explored the alternative instruction delivery format
to provide EBM learning experience for students on the Internet or through a
web content management system (WebCT or Blackboard). Given the variability in
faculty’s availability in teaching EBM, online EBM instruction could provide
consistent and equivalent learning experiences for students rotating across
multiple clinical training sites or offer students a tool to enhance their
learning in a traditional classroom setting. Srinivasan
et al. (2002) used a 20 page Web-based
EBM curriculum and an online practice examination developed in WebCT to
supplement a short EBM course made of components of didactic lectures and
interactive, problem based small group discussion sessions. First-year medical
students used the online EBM curriculum as an independent study tool to
reinforce their EBM learning (Srinivasan et al., 2002). Schilling et al. (2006) investigated the use of a Web-based curriculum to teach EBM.
The entire curriculum encompassed an online tutorial and information from two
other websites. In the study, an asynchronous discussion board moderated by
faculty was also used as a learning activity to encourage students’ discussion
of their patient case encounter, clinical questions, search process, and
findings. Although the curriculum was intended as a Web-based EBM instruction
tool, EBM content covered was incomplete since essential EBM skills of critical
appraisal and evidence application were not addressed in the EBM curriculum.
Another
study reported an EBM intervention featuring a two part EBM course for
third-year medical students who went through core clinical clerkship rotations
(Aronoff et al., 2010). The first part was delivered in
six didactic online modules of materials via Blackboard (Blackboard Academic
Suite; Blackboard, Inc., Washington, DC), each of which was followed by a
focused practicum that students completed and submitted to an online faculty
mentor for review and feedback. The second part of the course required that
each student generate a clinical question on a patient seen during each of four
clinical rotations and complete four evidence summaries using the Critically Appraised
Topic format (CAT) developed by Sackett, Richardson,
Rosenberg, and Haynes (1997). The
students sent the evidence summaries to their mentor for review and correction,
if necessary.
Implementation
of EBM Interventions
Nine
out of the 13 studies did not provide any information on any faculty
development opportunities nor delineate the preparation or qualification of any
faculty involved in teaching EBM. Only four studies explained the process of
implementing their EBM intervention, particularly faculty development efforts,
on varying levels of detail. The first of the four studies was conducted at the
College of Human Medicine at Michigan State University, which is a
community-based medical school without a central university hospital. The EBM
intervention of a nine session curriculum was delivered on clinical campuses
across six communities. A course director and course coordinator provided
central management for faculty development and course implementation. Each
community campus had a research director who served as the community course
coordinator and participated in the course content and format selection. A two
day faculty development event was held in which four McMaster University
faculty members led intensive training seminars to train and prepare 50 faculty
members to teach EBM locally (Wadland et al.,
1999).
The
development of a four year longitudinal EBM curriculum reported by Barnett et
al. (2000) started with the establishment
of an Evidence Based Medicine Working Group at the Mount Sinai School of
Medicine. Its multidisciplinary team members were charged with the task of
revamping the EBM component in the traditional medical curriculum in
collaboration with all preclinical course and clinical clerkship directors.
Faculty development opportunities were provided on different levels, including
a full day retreat for course and clerkship directors to receive EBM training
provided by faculty from McMaster University. These trained course and
clerkship directors in turn provided training for other faculty in their own
department.
Srinivasan et al. (2002) investigated the effect of a one month EBM course integrated
into the preclinical medical curriculum at the Indiana University School of
Medicine. The course combined traditional didactic lectures with interactive
small group, problem based learning sessions to teach
138 first-year medical students EBM principles and skills. The small group,
problem based sessions were facilitated by 16 faculty members who represented
four departments of emergency medicine, pediatrics, internal medicine, and
library sciences at the medical school. To prepare faculty for the small group
sessions, a facilitator’s handbook was developed to provide faculty with
consistent small group experiences, detailed objectives, timelines, commonly
asked questions and answers, sample dialog, completed Users’ Guides to the Medical Literature worksheets (Guyatt & Rennie, 2002), critical concept
summaries, background reading material, and small group teaching strategies and
references. Furthermore, dedicated secretarial support was available to faculty
in the early introduction of the EBM course to preclinical medical students.
All faculty facilitators who participated in the small group sessions had EBM
experience through taking formal EBM courses, teaching EBM, holding a MPH
(Master of Public Health) degree, or writing about EBM for peer reviewed
journals.
In
another study on an EBM longitudinal curriculum reported by West et al. (2011), EBM faculty were given supported time
for each graded assignment and administrative time for direct student contact
during the period of teaching and maintaining the short course in the
curriculum. Each instructor involved in EBM teaching had extensive EBM
knowledge and skills through their advanced training in biostatistics and
epidemiology and participation in the workshop offered at McMaster University.
These instructors also had experience in teaching basic and advanced EBM topics
to residents at Mayo Clinic.
The
EBM curriculum in the study by Wadland et al. (1999) was standardized to ensure the
equivalent EBM learning experience in three primary care clerkships of internal
medicine, family medicine, and pediatrics at multiple locations. Srinivasan et al. (2002)
used didactic lectures in conjunction with the problem based learning approach
which demanded a great deal of investment in resources (e.g., faculty and their
time in facilitating small group sessions). Two other studies, one by Barnett
et al. (2000) and the other by West et
al. (2011), were characterized by a
longitudinal curriculum integrated into both preclinical and clinical years of
medical education. These four studies demonstrate that the successful
implementation of an EBM intervention across multiple sites, with a problem
based small group discussion component, or integrated into both preclinical and
clinical education curricula, are contingent on important factors such as
provision of faculty development opportunities, resources coordination, and
instructional and administrative support for EBM faculty.
Evaluation
of EBM Outcomes
Variations
in evaluation methods used to assess the effect of the EBM interventions in the
studies preclude any quantitative analysis of a pooled effect size of the
results from the studies. Therefore, the outcomes reported in the studies were
examined using Kirkpatrick’s four level evaluation model: satisfaction with a
learning experience (level 1), learning in terms of any change in knowledge,
skills, or attitudes (level 2), performance improvement or behavioural changes
in a patient care setting (level 3), and impact on patient health outcomes
(level 4). Five studies incorporated level 1 evaluation; all 13 studies
examined changes in knowledge, skills, or attitudes at level 2; and three
studies had level 3 evaluation of the outcomes of
behavioural changes in a patient care setting. None of the studies went beyond
level 3 evaluation. A summary of EBM interventions, outcomes measured,
evaluation methods, and levels of evaluation is presented in Appendix B.
Level 1
Evaluation: Reaction and Satisfaction
Five
out of the 13 studies evaluated learners’ satisfaction and reaction to the EBM
intervention. Wadland et al. (1999) found that second-year medical students’ rating of an EBM course
was not significantly different
from that of other courses offered in the medical curriculum and that there
were no significant variations in responses across multiple community clinical
sites. Srinivasan et al. (2002) evaluated both students and faculty’s satisfaction with an
EBM course integrated into the first-year medical curriculum. The evaluative
results showed that a majority of students reported enjoying the EBM course and
felt that the course material was appropriate for their learning level and
related to clinical practice. Srinivasan et al. (2002) also sought faculty’s feedback as an
indicator of the effect of EBM instruction. In faculty’s opinions, medical
students could perform well in EBM-related areas and use EBM concepts as well
as or better than residents whom the faculty had supervised in the past year.
Both faculty and student respondents agreed that the course material was
appropriate for first-year medical students. In the study by Sastre et al. (2011),
third-year medical students also reacted to their EBM learning positively. They
felt that EBM learning was useful and that they would incorporate their
acquired skills into clinical care of patients.
Holloway
et al. (2004), however, reported that a
majority of student respondents (58 out of 67 respondents) expressed negative
comments about EBM instruction and evaluation that spanned the preclinical
education. The students felt that there was too much emphasis on EBM (43-page
syllabus) in their preclinical curriculum. They also commented that the test
module with a 15-20 page evaluation packet (a five step EBM evaluation module)
took too much time to complete and was of little value to their learning of
good EBM skills. Thomas and Cofrancesco (2001) found that third- and fourth-year medical
students rated the usefulness of their EBM learning through an EBM report less
favorably than their clinical sessions with their preceptor. The researchers
also found that the students perceived their clinical competence in EBM with a
lower value than diagnostic decision making and self-directed learning.
Level 2 of
Evaluation: Change in Skill and Knowledge
Level
2 evaluation was conducted in all 13 studies to
measure any possible change in learners’ knowledge, skills, or attitudes as a
result of EBM instruction. Learning outcomes in these studies were measured
with a wide array of evaluation methods such as the AAMC Medical School
Graduation Questionnaire (Wadland et al., 1999),
critique of a relevant article (Barnett et al.,
2000; Schilling et al., 2006), self-administered EBM skill survey (Dorsch et al., 2004; Ghali et al., 2000; Schilling et
al., 2006; West et al., 2011), self-administered examination of
knowledge and skills (Aronoff et al., 2010; Dorsch
et al., 2004; Srinivasan et al., 2002; West et al., 2011), self-efficacy
questionnaire (Cayley, 2005; Nieman et al.,
2009), a five step EBM evaluation module (Holloway
et al., 2004), PICO case summary (Nieman
et al., 2009), and CAT (Critically Appraised Topic) summary (Aronoff et al., 2010).
One
study reported an improved response rate for appropriate training in literature
analysis and research skills on the AAMC Medical School Graduation
Questionnaire since one of the objectives for the EBM curriculum in the study
was to develop students’ research skills (Wadland
et al., 1999). Five studies reported improvement in literature searching
and other library skills (Barnett et al., 2000;
Dorsch et al., 2004; Ghali et al., 2000; Holloway et al., 2004; Schilling et
al., 2006). These skills were measured with various instruments such as
a rating scale (Barnett et al., 2000; Schilling
et al., 2006), evaluation rubric (Holloway
et al., 2004), case scenario based skill test (Dorsch et al., 2004), and self-reported survey (Ghali et al., 2000). EBM skill in critical
appraisal was evaluated as an outcome in six studies: four reported significant
improvement on objective measures (Aronoff et
al., 2010; Dorsch et al., 2004; Srinivasan et al., 2002; West et al., 2011);
one reported improvement on self-reported skill test (Ghali et al., 2000); and one failed to report any evaluative
results through a critical appraisal measure – part of two evaluation modules
that tested the five steps of the EBM process (Holloway
et al., 2004). The results of most studies reveal improvement of varying
degrees in terms of learners’ knowledge and skills in EBM.
Level 3 of
Evaluation: Changes in Behaviour
Three
studies incorporated level 3 evaluation measures in evaluating the extent to
which any change was made in students’ behaviour in a clinical setting. Two of
the three studies reported students’ successful use of EBM in actual clinical
work through application of the five steps of the EBM process in an EBM report (Aronoff et al., 2010; Thomas & Cofrancesco, 2001).
Students in the study by Thomas and Cofrancesco
(2001) wrote an EBM report based on a real patient case. Students in the study
by Aronoff et al. (2010) formulated a clinical
question generated by a patient seen during each of four clinical rotations. A
complete CAT form was required of students and reviewed but not scored by
faculty mentors. In the study by Sastre et al. (2011), level 3 evaluation
was conducted through evaluation of citations of EBM resources in students’ inpatient
admission notes and the quality of the EBM component of the discussion portion
in the notes. Another measure to evaluate students’ use of EBM resources in the
same study was computer log recordings of students’ online use of various EBM
resources via an electronic medical resource system. Analysis of students’
patient notes reveals a significant improvement on the overall quality of EBM
incorporation into the discussion of a patient’s problem as a result of EBM
instruction. The computer log data showed a significant increase in students’
usage for all electronic resources. EBM resources such as systematic peer
reviewed resources (e.g., Cochrane Databases, Clinical Guidelines) represented
59% of all resources accessed following the workshop. However, there was no
significant improvement in the number of citations per patient note before and
after the EBM intervention.
Discussion
The
review was conducted to summarize the findings of the studies retrieved from
comprehensive literature searching of relevant resources. The analysis of 13
studies reveals assorted approaches to instructional design, development,
implementation, and evaluation of EBM interventions in undergraduate medical
education. The definition of EBM and component skills required to practice EBM
are not debatable in the current medical literature. However, there is no
standard or model for how EBM should be designed and developed and how it
should be effectively taught and evaluated in undergraduate medical education.
The studies show wide variations in the use of instructional strategies,
delivery formats for EBM instruction, implementation of an EBM intervention,
and outcome measures, all of which remain important issues for further
research.
Effectiveness
of an educational intervention is inescapably linked to the outcomes of
educational activities (Belfield, Thomas,
Bullock, Eynon, & Wall, 2001). Interactive and clinically integrated
activities support active learning and could lead to medical students’ deep
understanding of content material and superior experience in classroom and
clinical settings. The review shows that several studies made efforts to
incorporate clinically integrated EBM learning experiences and clinically
relevant outcome measures into EBM instruction in the medical curriculum.
However, more than half of the studies reviewed fell short of integrating
interactive teaching and learning activities into the EBM interventions. Future
research is required to determine the effect of EBM instruction on learning
outcomes as a result of sound instructional design principles and active
learning strategies employed in the design and development of an EBM
intervention in undergraduate medical education.
Delivery
of EBM instruction with information technology holds promise in affecting EBM
learning outcomes for medical students. A study by Davis, Crabb,
Rogers, Zamora, and Khan (2008)
demonstrated that first-year medical students in a computer-based EBM session
had gains in knowledge and attitude similar to those in the lecture-based EBM
session. Cook et al. (2008) conducted a
meta-analysis of Internet-based learning in the health professions. Their
findings provided further evidence for the effectiveness of Internet-based
instruction similar to traditional instructional methods. Three of the 13
studies utilized online learning as an approach to teaching EBM or
supplementing the traditional method of teaching EBM. The results from these
studies suggest implications for medical educators and health science
librarians in developing and teaching EBM in the medical curriculum.
Online/Internet-based EBM instruction that is properly designed, developed, and
implemented could support students’ self-directed learning, reinforce EBM
learning in a traditional classroom setting, or present great potential for
standardizing the quality of EBM teaching to achieve equivalent learning
experience across multiple clinical teaching sites or clerkship rotations.
To
a great extent, effective delivery of an EBM intervention hinges on a good
implementation plan addressing issues related to faculty recruitment for EBM
teaching, faculty training in EBM, administrative support, time and space
allocated to instruction, and procedures for recording and collecting data
pertaining to course and student evaluation (Gustafson
& Branch, 2007). Nine of the 13 studies failed to provide any
information on the implementation process of an EBM intervention.
Implementation is one of six important steps in curriculum development for
medical education (Kern, Thomas, & Hughes,
2009), as well as in the process of instructional design (Gustafson & Branch, 2007). Successful
integration of EBM into the medical curriculum necessitates EBM training for
busy primary care clinicians or preceptors in the clinical teaching and application
of EBM (Cayley, 2005; Nicholson, Warde, &
Boker, 2007) and for teaching EBM in a longitudinal fashion rather than
as a single innovation (Nieman et al., 2009).
Given that faculty development is likely more important than any other step in
implementation (Bordley et al., 1997),
faculty development efforts are pivotal to any effective EBM instruction for
medical students in either preclinical or clinical years, as evidenced by
higher levels of learner satisfaction and knowledge gains found in the studies
by Srinivasan et al. (2002)
and West et al. (2011). “Teaching EBM may need to focus as much on teachers as on students”
(Del Mar, Glasziou, & Mayer, 2004, p. 990).
Implementation of a successful EBM intervention requires investment of adequate
faculty time, resources (Mi, Moseley, &
Green, 2012), and provision of faculty development opportunities (Green, 2000; Nicholson et al., 2007).
The
analysis of the studies reveals a wide variation in EBM learning outcomes due
to study heterogeneity in research designs, which did not lend to an estimation
of a combined effect from these studies. The adapted 4 level evaluation model
by Kirkpatrick was applied to examine the extent to which learning outcomes
were evaluated. The results showed that the majority of the studies focused on
level 1 and level 2 evaluation to measure learning
that occurs separated from the real-time setting of practice. Few studies went
beyond level 3 evaluation, which is more difficult to
measure because it requires assessment in the practice setting (Straus et al., 2004). However, evaluation of
the effectiveness of EBM as an educational intervention should aim to
incorporate level 3 and level 4 evaluation into curriculum development to
measure the long-term impact of EBM instruction on sustained behavioural
changes in a patient care setting and possibly, patient health outcomes. West
et al. (2011) stressed the need for
research on the sustained increases in perceived and measured EBM knowledge and
benefits beyond medical school into postgraduate medical education and
practice. Clearly, students’ independent application of EBM skills in a
clinical setting and into their residency warrants further research efforts.
The
article only reviewed studies on EBM interventions in the medical curriculum of
U.S. medical schools given the differences that exist between medical schools
in the US and other countries in terms of duration of undergraduate medical
education programs, requirement in competence-based learning outcomes, and
program accreditation standards. Further effort could be made to systematically
review the literature including international studies, related to specific
topics such as the use of different instructional methods in teaching EBM (e.g, team based learning, problem based learning) and
different instruction delivery formats (e.g., online instruction of EBM). Such
review would afford additional insights into EBM teaching in undergraduate
medical education.
The
qualitative review was limited to full research reports published in peer
review journals. The selected reports addressed formal EBM instruction as part
of or integrated into preclinical or clinical curricula, which is the focus of
the review. Future research could be conducted to examine how each EBM step or
component (e.g., clinical question formation, searching the literature for the
evidence, or critical appraisal) is instructed and evaluated in undergraduate
medical education. Systematic reviews could be carried out through exhaustive
searching of literature including conference proceedings and other grey
literature to shed further light on teaching practices and innovations in EBM
instruction in undergraduate medical education.
The
inclusion criteria specifically established in this review circumscribed the
number of studies selected. It should be pointed out that there is a view
frequently held in medical education that “all that is needed in medical
education is commonsense” (Harden, 1986, p. 522). The view may partly account for the level
of evidence based educational research on EBM and other educational phenomena.
Clearly, there is a need to employ different approaches to research and to
break with or question existing teaching practices in medical education in
general and EBM teaching in particular. Relevant research on the use of these
approaches to research and creative and innovative methods of teaching EBM
would add significantly to the literature in this field.
Conclusion
The
review was conducted as an attempt to present research evidence on how EBM was
designed, developed, implemented, and evaluated in undergraduate medical
education and to help inform medical educators and health sciences librarians
in their efforts to practice evidence based teaching of EBM. The review also
pointed out a number of areas that warrant further research on EBM in
undergraduate medical education.
Khan and Coomarasamy
(2006) comment that the substantial heterogeneity in the types of educational
interventions and their effects necessitate an analysis beyond simply focusing
on the overall results of studies on EBM teaching. The article
analyzed and discussed EBM interventions in educational research from the
perspective of instructional design. It examined some relevant issues related
to the treatment of EBM content in the undergraduate medical curriculum. These
issues included EBM instructional design, implementation, faculty development,
and levels of evaluation of learning outcomes. Failure to attend to these issues
could undermine the effectiveness of an EBM intervention in the medical
education curriculum.
References
Aronoff, S. C., Evans, B., Fleece, D., Lyons, P., Kaplan, L.,
& Rojas, R. (2010). Integrating evidence based medicine into undergraduate
medical education: combining online instruction with clinical clerkships. Teaching & Learning in Medicine, 22(3),
219-223. doi:10.1080/10401334.2010.488460
Association of American Medical Colleges. (2012). Medical School
Graduation Questionnaire: 2012 All Schools Summary Report. In Association of American Medical Colleges.
Retrieved 4 Aug. 2012 from https://www.aamc.org/download/300448/data/2012gqallschoolssummaryreport.pdf
Barnett, S. H., Kaiser, S., Morgan, L. K., Sullivant, J.,
Siu, A., Rose, D., Rico, M., Smith, L., Schechter, C., Miller, M.,
Stagnaro-Green, A. (2000). An integrated program for evidence-based medicine in
medical school. Mount Sinai Journal of
Medicine, 67(2), 163-168.
Barnett, S. H., Smith, L. G., & Swartz, M. H. (1999).
Teaching evidence-based medicine skills to medical students and residents. International Journal of Dermatology, 38(12),
893-894. doi:10.1046/j.1365-4362.1999.00844.x
Belfield, C., Thomas, H., Bullock, A., Eynon, R., & Wall,
D. (2001). Measuring effectiveness for best evidence medical education: A
discussion. Medical Teacher, 23(2),
164-170. doi:10.1080/0142150020031084
Bordley, D. R., Fagan, M., & Theige, D. (1997).
Evidence-based medicine: A powerful educational tool for clerkship education. The American Journal of Medicine, 102(5),
427-432.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs
for Research. Boston, MA: Houghton Mifflin Company.
Cayley, W. E., Jr. (2005). Evidence-based medicine for
medical students: Introducing EBM in a primary care rotation. Wisconsin Medical Journal, 104(3),
34-37. Retrieved 18 Jul. 2012 from http://www.wisconsinmedicalsociety.org/_WMS/publications/wmj/issues/wmj_v104n3/Cayley.pdf
Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin,
P. J., & Montori, V. M. (2008). Internet-based learning in the health
professions: A meta-analysis. Journal of
American Medical Association, 300(10), 1181-1196. doi:10.1001/jama.300.10.1181
Davis, J., Crabb, S., Rogers, E., Zamora, J., & Khan, K.
(2008). Computer-based teaching is as good as face to face lecture-based
teaching of evidence based medicine: A randomized controlled trial. Medical Teacher, 30(3), 302-307. doi:10.1080/01421590701784349
Del Mar, C., Glasziou, P., & Mayer, D. (2004). Teaching
evidence based medicine. British Medical
Journal, 329(7473), 990. doi:10.1136/bmj.329.7473.989
Dickersin, K., Straus, S. E., & Bero, L. A. (2007).
Evidence based medicine: Increasing, not dictating, choice. British Medical Journal, 334(Suppl 1),
s10. doi:10.1136/bmj.39062.639444.94
Dorsch, J. L., Aiyer, M. K., & Meyer, L. E. (2004).
Impact of an evidence-based medicine curriculum on medical students' attitudes
and skills. Journal of the Medical
Library Association, 92(4), 397-406. Retrieved 18
Jul. 2012 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC521510/pdf/i0025-7338-092-04-0397.pdf
Finkel, M. L., Brown, H. A., Gerber, L. M., & Supino, P.
G. (2003). Teaching evidence-based medicine to medical students. Medical Teacher, 25(2), 202-204. doi:10.1080/0142159031000092634
Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction
(8th ed.). Boston: Pearson/Allyn & Bacon.
Ghali, W. A., Saitz, R., Eskew, A. H., Gupta, M., Quan, H.,
& Hershman, W. Y. (2000). Successful teaching in evidence-based medicine. Medical Education, 34(1), 18-22. doi:10.1046/j.1365-2923.2000.00402.x
Glasziou, P., Burls, A., & Gilbert, R. (2008). Evidence
based medicine and the medical curriculum. British
Medical Journal, 337, a1253. doi:10.1136/bmj.a1253
Green, M. L. (2000). Evidence-based medicine training in
internal medicine residency programs: A national survey. Journal of General Internal Medicine, 15(2), 129-133. Retrieved 17 Jul. 2012 from
http://www.springerlink.com/content/r351055rj56h1130/fulltext.pdf
Gustafson, K. L., & Branch, R. M. (2007). What Is
Instructional Design? In R. A. Reiser & J. V. Dempsey (Eds.), Trends and Issues in Instructional Design
and Technology (2nd ed., pp. 10-16). Upper Saddle River, NJ:
Pearson/Merrill Prentice Hall.
Guyatt, G., & Rennie, D. (Eds.) (2002). Users' guides to the medical literature:
Essentials of evidence-based clinical practice. Chicago, IL: American
Medical Association.
Harden, R. M. (1986). Approaches to research in medical
education. Medical Education, 20(6),
521-531. doi:10.1111/j.1365-2923.1986.tb01394.x
Hatala, R., & Guyatt, G. (2002). Evaluating the teaching
of evidence-based medicine. JAMA: Journal
of the American Medical Association, 288(9), 1110-1112. doi:10.1001/jama.288.9.1110
Holloway, R., Nesbit, K., Bordley, D., & Noyes, K.
(2004). Teaching and evaluating first and second year medical students'
practice of evidence-based medicine. Medical
Education, 38(8), 868-878. doi:10.1111/j.1365-2929.2004.01817.x
Kern, D. E., Thomas, P. A., & Hughes, M. T. (2009). Curriculum development for medical
education: A six-step approach (2nd ed.). Baltimore, MD: Johns Hopkins
University Press.
Khan, K. S., & Coomarasamy, A. (2006). A hierarchy of
effective teaching and learning to acquire competence in evidenced-based
medicine. BMC Medical Education, 6,
59. doi:10.1186/1472-6920-6-59
Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA:
Berrett-Koehler.
Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, 50(1),
54-59.
Liaison Committee on Medical Education. (2012). Functions and
structure of a medical school: Standards for accreditation of medical education
programs leading to the M.D. Degree. In Liaison Committee on Medicual Education.
Retrieved 4 Aug. 2012 from http://www.lcme.org/functions.pdf
Mahoney, J. F., Cox, M., Gwyther, R. E., O'Dell, D. V.,
Paulman, P. M., & Kowlowitz, V. (2004). Evidence-based and population-based
medicine: National implementation under the UME-21 project. Family Medicine, 36(Jan. Suppl.),
S31-35. Retrieved 17 Jul. 2012 from http://www.stfm.org/fmhub/fm2004/January/JohnS31.pdf
McKibbon, K. A., Walker-Dilks, C., Haynes, R. B., &
Wilczynski, N. (1995). Beyond ACP Journal Club: How to harness MEDLINE for
prognosis problems. ACP Journal Club, 123(1),
A12-A14.
McKibbon, K. A., & Walker-Dilks, C. J. (1994a). Beyond
ACP Journal Club: How to harness MEDLINE for diagnostic problems. ACP Journal Club, 121(Suppl. 2), A10-12.
McKibbon, K. A., & Walker-Dilks, C. J. (1994b). Beyond
ACP Journal Club: How to harness MEDLINE for therapy problems. ACP Journal Club, 121(Suppl. 1), A10-12.
Melnyk, B. M. (2006). Calling all educators to teach and
model evidence-based practice in academic settings. Worldviews on Evidence-Based Nursing, 3(3), 93-94. doi:10.1111/j.1741-6787.2006.00061.x
Mi, M., Moseley, J. L., & Green, M. L. (2012). An
instrument to characterize the environment for residents' evidence-based
medicine learning and practice. Family
Medicine, 44(2), 98-104. Retrieved 17 Jul.
2012 from http://www.stfm.org/fmhub/fm2012/February/Misa98.pdf
Montori, V. M., & Guyatt, G. H. (2008). Progress in
evidence-based medicine. Journal of
American Medical Association, 300(15), 1814-1816. doi:10.1001/jama.300.15.1814
Nicholson, L. J., Warde, C. M., & Boker, J. R. (2007).
Faculty training in evidence-based medicine: Improving evidence acquisition and
critical appraisal. Journal of Continuing
Education in the Health Professions, 27(1), 28-33. doi:10.1002/chp.090
Nieman, L. Z., Cheng, L., & Foxhall, L. E. (2009).
Teaching first-year medical students to apply evidence-based practices to
patient care. Family Medicine, 41(5),
332-336. Retrieved 17 Jul. 2012 from http://www.stfm.org/fmhub/fm2009/May/Linda332.pdf
O'Connell, M. T., & Pascoe, J. M. (2004). Undergraduate medical education for the 21st century: leadership and
teamwork. Family Medicine, 36 (Suppl. January), S51-56. Retrieved 4 Aug. 2012 from http://www.stfm.org/fmhub/fm2004/january/marks51.pdf
Sackett, D. L., Richardson, W. S., Rosenberg, W., &
Haynes, R. B. (1997). Evidenced-based
medicine: How to practice and teach EBM. New York, NY: Churchill
Livingstone.
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes,
R. B., & Richardson, W. S. (1996). Evidence-based medicine: What it is and
what it isn't. British Medical Journal,
312, 71. doi:10.1136/bmj.312.7023.71
Sastre, E. A., Denny, J. C., McCoy, J. A., McCoy, A. B.,
& Spickard, A. (2011). Teaching evidence-based medicine: Impact on
students' literature use and inpatient clinical documentation. Medical Teacher, 33(6), e306-312. doi:10.3109/0142159X.2011.565827
Schilling, K., Wiecha, J., Polineni, D., & Khalil, S.
(2006). An interactive Web-based curriculum on evidence-based medicine: Design
and effectiveness. Family Medicine, 38(2),
126-132. Retrieved 17 Jul. 2012 from https://www.stfm.org/fmhub/fm2006/February/Katherine126.pdf
Slawson, D. C., & Shaughnessy, A. F. (1999).
Teaching information mastery: Creating informed consumers of medical
information. Journal of American Board of Family Practice, 12(6),
444-449.
Srinivasan, M., Weiner, M., Breitfeld, P. P., Brahmi, F.,
Dickerson, K. L., & Weiner, G. (2002). Early introduction of an
evidence-based medicine course to preclinical medical students. Journal of General Internal Medicine, 17(1),
58-65. Retrieved 17 Jul. from http://www.springerlink.com/content/u42tnw482l38441r/fulltext.pdf
Steward, M. G. (2001). Core Competencies. In Accreditation Council for Graduate Medical
Education. Retrieved 4 Aug. 2012 from http://www.acgme.org/acwebsite/RRC_280/280_corecomp.asp
Straus, S. E., Green, M. L., Bell, D. S., Badgett, R., Davis,
D., Gerrity, M., Ortiz, E., Shaneyfelt, T. M., Whalen, C. Mangrulkar, R.
(2004). Evaluating the teaching of evidence based medicine: Conceptual
framework. British Medical Journal, 329(7473),
1029-1032. doi:10.1136/bmj.329.7473.1029
Thomas, P. A., & Cofrancesco, J., Jr. (2001).
Introduction of evidence-based medicine into an ambulatory clinical clerkship. Journal of General Internal Medicine, 16(4),
244-249. Retrieved 17 Jul. 2012 from http://www.springerlink.com/content/b7k81391322u64n1/fulltext.pdf
Wadland, W. C., Barry, H. C., Farquhar, L., Holzman, C.,
& White, A. (1999). Training medical students in evidence-based medicine: A
community campus approach. Family
Medicine, 31(10), 703-708. Retrieved 17 Jul.
2012 from http://stfm.org/fmhub/Fullpdf/NOVDEC99/mse.pdf
West, C. P., Jaeger, T. M., & McDonald, F. S. (2011).
Extended evaluation of a longitudinal medical school evidence-based medicine
curriculum. Journal of General Internal
Medicine, 26(6), 611-615. doi:10.1007/s11606-011-1642-8
2012 Mi. This is an
Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share
Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which
permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.