Evaluation of the Long-Term Impact of a Curriculum-Integrated Medical Information Literacy Program 1

: Introduction: Medical libraries have long provided educational programs to support evidence-based practice. Medical students at Queen’s University, Kingston, Ontario, participate in a curriculum-integrated information literacy program during the first two years of medical school. Do they retain, improve, or forget the skills? Do they continue to use the library resources during clerkship? Did they encounter barriers to prevent them from using the resources? Methods : A short survey was administered to 99 students at the end of medical school. The survey included questions about medical students’ attitudes and behaviours, their use of information resources, and their medical information literacy knowledge. Some of the knowledge questions were compared to pre- and post-tests that the same class completed in first year. Results: Fifty-three students completed the survey. The students rated their abilities very highly but there was only a weak positive relationship with the knowledge scores. Information resources were well used, both for clinical questions and to complete the mini-scholar exercises. Discussion: Medical students feel better prepared to answer clinical questions and their skills improved or remained the same for the content that could be compared between first and fourth year. Different resources were used for day-to day information needs and for the completion of the mini-scholar exercises. The results will inform changes to the Medical Information Literacy program at Queen’s University. The librarians will explore some of the barriers to access to ensure that future students can use information resources with more ease while away from campus.


Introduction
Academic libraries are facing financial challenges. In addition to carefully considering collections decisions, they must examine the value of investing in the delivery of education programs. Medical and health libraries have long provided educational programs to support evidencebased practice. At Queen's University, Kingston, Ontario, a medical information literacy (MIL) program has been integrated in the undergraduate medical curriculum since 1991. Over time, this integration has been linked to the evidence-based medicine (EBM) curriculum. Student assignments and course evaluations consistently show that students learn the MIL skills and can apply them successfully in various endeavours. What has not been assessed is the long-term retention of the MIL knowledge and skills and the long-term use of the library's purchased information resources.
Evidence-based medicine (EBM) training is a regular occurrence in health care professional education. Many research projects have tried to assess EBM skills of practicing physicians, junior doctors, medical residents, and undergraduate students usually soon after an educational intervention. Maggio et al. [1], Shaneyfelt et al. [2], and Just [3] have compiled teaching methodologies and assessment procedures that are as diverse as each article included in their reviews of the literature. One common thread across many of the articles surveyed is that very few studies tested participants after a significant amount of time had elapsed after their EBM training. Gruppen et al. [4] and Just [3] both recommended that more research is needed to examine the long-term retention of skills. Dorsch et al. [5] concluded that medical graduates who were trained in their undergraduate program "retained EBM skills in residency and maintained a positive attitude about the importance of applying EBM principles to patient care." However, Green and Ruff [6] found that health care professionals were having difficulties formulating clinical questions and translating them into effective searches. A few years later, Cullen et al. [7] observed that junior doctors rated their skills higher than assigned by the study raters. These young professionals were relying on synthesized sources of information and did not remember how to search Medline.
This study aimed to determine whether the Queen's University medical students retained their medical information literacy knowledge and skills two years after the last curriculum component offered by the librarians as part of their EBM training. The study also aimed to identify which purchased information resources were being used during the clerkship years and to determine any barriers that might have prevented the students from using them. The results of the study will inform MIL curriculum change, influence collections decisions, and help the library improve access to the academic health information resources while the students are away from campus.

Background and interventions
At Queen's University's Medical School, medical information literacy is integrated in the EBM curriculum throughout the four years of the undergraduate program (Table 1). However, the teaching portion by the librarians is concentrated in the first two years. There are a number of embedded assessment opportunities during these two years.
A pretest, consisting of 5 multiple-choice MIL knowledge questions, is completed online by all incoming students (n0100) in the first few days of medical school (see Supplementary Appendix 1). At the end of the first year, during which the students participated in three online modules and three face-to-face classes, the same test is administered again as a post-test. These two tests are not graded, but used as program evaluation tools, and they document a consistent improvement in student performance in the post-test over the pretest scores each year. The MIL grades are assigned via a quiz and an assignment marked by the librarians, as part of a larger project completed for the course instructor. There is another assignment in second term, linked to a project in the Fundamentals of Therapeutics course. The students are asked to prepare a patient, intervention, comparison, outcome (PICO) question and search the Medline and EMBASE databases for a specific assigned drug. Class averages for this first-year culminating assignment usually hover around 80%. In second year, the students prepare a Critical Enquiry research project working with a faculty tutor. Again, they need to prepare a research question (although not necessarily in PICO format) and a review of the literature. In the past these searches were marked by librarians, but in recent years, a peer-tutor model has been in place and is being evaluated as part of a separate study.
During the two clerkship years, when the students are placed in hospital and other clinical settings, the students complete a mini-scholar exercise (MSE) during each of their eight rotations. The MSE was designed by faculty to ensure continuity in the practice of EBM and the librarians were not involved in the design or assessment of the literature searching portion of this assignment. However, a few questions about this critical incident (defined by Stevenson [8]) were included in this study expecting that the MSE contributed to the students' application of MIL skills during the last two years of their training. It was also important to find out which information resources they searched to answer a question about a "real-life" topic during this assignment, especially considering findings such as that by McKibbon and Fridsma [9] in a 2006 study of information resource usage by health care professionals, which concluded that "physicians did not choose resources wisely." Furthermore, the Library needs to know how the information resources are used to complement usage statistics provided by vendors. During medical education, the university offers various information resources, some essential to all health care programs, while others require careful assessment to be retained during a period of contracting budgets. This study is the first at our university to report on the long-term usage of the resources provided by the Library and the barriers that may prevent students from using them.

How is EBM assessed?
Shaneyfelt et al. [2] compiled various instruments and classified them for their validity and reliability, as well as indicating which instruments would be best for assessing individuals or to assess the impact of complete EBM programs. The Berlin and Fresno tests are often mentioned as valid EBM tests, but were unfortunately not applicable to this study. The Berlin test [3,10] does not include assessment of skills, such as literature searching, and the Fresno test [11] is very demanding in time and resources. The EPIC scale [12] only has one question about the perceived ability to conduct a literature search, whereas the ACE tool [13] asks four questions about the PICO question and search strategy for a given patient scenario. These tools were not detailed enough to capture the students' MIL attitudes, behaviors, and skills that would inform potential curriculum

Methods
Medical students of the 2016 graduating class (n099) were chosen as the target population as they could be surveyed before leaving the university and scattering across the country for their residency training. Ethics approval was granted by the Queen's University Health Sciences Research Ethics Board and the survey was administered at the end of March 2016. At that time, the students had completed all the rotation requirements of the clerkship and were meeting in class for a few weeks before their final qualifying exams. The survey was anonymous and the results would have no bearings on their class standings. During a free 30-minute period, the students who agreed to participate (n 0 53) completed the paper survey. The 10 minutes planned for the survey were sufficient with the exception of one student who did not complete the True/ False section at the very end of the questionnaire. A draw for five small financial incentives ($20) was performed immediately after completion.
Data used in the current study were derived from three sources; a pretest of knowledge, a post-test of knowledge, and the post-program evaluative survey. Where possible, analysis compared results of all three measures to examine evidence of longitudinal effects of the MIL program at Queen's University.

Pre-and post-tests
As noted earlier, students were asked to complete a short, 5-item multiple-choice test both before and after the MIL program in year 1 (pretest, administered in September 2012 (n0100) and post-test, administered in March 2013 (n 059)). Two knowledge questions were consistent across the pretest, post-test, and the survey. The concepts being tested were the same: Boolean logic and choice of resource for a particular clinical scenario. The question format was also the same, only the topic of the scenario varied slightly.

Post-program survey
The School of Medicine recommended a short 10minute survey. It was therefore decided to conduct a quantitative study, mostly using multiple-choice and easy to complete charts or scales. The survey instrument was designed to include two main components. Part 1 focused on attitudes and behaviours and part 2 required the students to answer some knowledge questions based on the Queen's University medical information literacy objectives. Although many authors [7,10] mention that selfrating of EBM skills is not reliable and does not correlate to knowledge, this survey asked the students to rate their confidence levels with the two components of EBM of interest: question formulation and information searching. The goal was to link the confidence levels to the knowledge questions within the survey, attempting to either confirm or disprove the results of earlier studies. A 4-point Likert scale used: strongly agree, agree, disagree, or strongly disagree. The same scale was also used to ask a few questions about the impact of the medical information literacy training and about participants' thoughts regarding the MSE. Stevenson [8] found that the "work environment influences the learner" so the less traditional barriers of "lack of role models," "lack of continued exposure," and "peer pressure" were included in the list.
The survey instrument was tested with four medical students from years 1 and 2. These students are the designated scholar role reps for their classes and were keen to participate. Five librarians at the Health Sciences Library also completed the survey. Based on the feedback from these novice and expert testers, the survey was edited to create the final version. The students also affirmed that a paper survey would be more likely to be completed seriously, referring to the fact that medical students are surveyed so often that they don't want to spend much time completing an online survey. As such, data were collected using hard copy surveys and manually input to electronic software for analysis. All statistical analyses were completed using SPSS.

Pre-and post-tests
Overall scores for the pre-and post-tests were calculated as a sum of correct answers to all questions. Pre-and post-test scores were then compared using a paired t test to assess for change in knowledge through the EMB program.

Post-program survey
Primary analyses of the post-program survey included a review of descriptive statistics for all variables (program of study, educational background, confidence levels, knowledge, frequency of resource usage and barriers encountered). Aggregate scores for test variables were then calculated for use in comparative analyses. Overall knowledge scores were calculated as a sum of responses, where correct answers were scored as a positive 1, incorrect answers as a negative 1 and a response of "I don't know/remember" as 0. Frequency of resource use and confidence were calculated as an average value across all relevant survey items.
Secondary analyses consisted of an ANOVA, assessing for differences in knowledge, confidence, and frequency of resource use by educational background and chosen specialty for residency, as well as a Pearson's correlation analysis assessing relationships between knowledge scores and overall confidence in one's ability.
Comparing pre-and post-tests with the post-program survey As noted earlier, two of the pre-and post-test questions were aligned with content in the post-program survey (Supplementary Appendix 2 Questions 5b and 6). It should be noted that students were not given the correct answers after the pretest (fall 2012), but did immediately receive them after the post-test (spring 2013). Although overall scores have not been compared due to the insufficient detail available in archived data, pre-and post-test scores on these two questions were compared to that of the survey (spring 2016) using descriptive statistics.

Demographics
From the total class of 99 students, 53 agreed to complete the survey (53.5%). They ranged in age from 24 to 33 years (mean026.5; SD02.1). The gender split was almost equal with 51% male and 49% female. This is a representative sample from the class, which in total has the same age range and gender split with 48.5% male and 51.5% female.
Prior to enrolling in their current medical program, 25 of the student participants (66%) held an undergraduate degree, 16 (30.2%) held a Master's degree, and 2 (3.8%) held a PhD or higher. At the time of the survey, students had already received their residency matches. The residency disciplines also follow the total class assignments with about one-third of students going to Family Medicine, less than a third to Internal Medicine, whereas the last portion could be divided into four surgical specialties and six "Other" specialties ( Table 2) [14]. ANOVA found no significant differences in knowledge, confidence, or frequencies of resource use between students of different academic backgrounds or who are currently enrolled in different specialties.

Confidence
Students rated their confidence in their own abilities high, either agreeing or strongly agreeing with each of the statements shown in Table 3.
Students also felt more confident now than at the beginning of medical school in their ability to locate reliable medical information (mean03.53, SD00.67), and that they were more efficient in searching for medical information because they knew how to choose appropriate resources for their information needs (mean03.36, SD00.71).
Knowledge Table 4 shows the compilations of knowledge questions. Only six questions (11 in total) were answered correctly more often than incorrectly or with "I don't remember."

Relationships between knowledge and confidence
A Pearson's correlation analysis assessing the relationship between knowledge scores and confidence found a very weak, nonsignificant positive relationship (R 2 00.17; p 00.25).
Comparing knowledge across pre-and post-tests, and the post-program survey A paired t test comparing scores on pre-and post-tests found a significant increase in overall knowledge scores through the program (t(58)05.95, p B0.001). Although statistical analysis could not be performed to compare frequency of correct responses on the two items across the pre-and post-test, and the post-program survey, comparison of frequencies does show an increase in the proportion of correct responses over time. Scores on a question on Boolean logic increased from 42%, to 59.3%, to 60.4% correct, whereas scores regarding selection of resources increased from 43%, to 47.8%, to 71.7% correct across the pretest, post-test, and post-program survey respectively (Table 5).

Resource usage
Mobile applications were the most frequently used resources. Most students indicate that they use them weekly or daily. Figure 1 displays the frequency of usage for each resource in decreasing order of the mean usage frequency. It should be noted that the survey was specific about the point-of-care (POC) tools being the ones purchased by the university, and we gave the example of Up-To-Date as a resource that would fit in the "Individually purchased" category since it is not offered on campus. Considering the cost of these resources, it was important for the Library to know if the purchased POC tools were continuing to be used during clerkship. Similarly a separate query for drug information resources was prepared, although many are integrated in POC tools. Most of the "Other" responses related to the cost of purchasing resources, no doubt referring to the fact that this university does not offer access to all POC tools. Almost all students agreed (52.8%) or strongly agreed (45.3%) that they would explore available medical information resources at the universities of their forthcoming residencies.

Barriers to access
Students were asked to select any factors that prevented them from accessing and using information resources, other than the ones they purchased individually while they were not on campus. Only 9.4% of participants felt they had never encountered any challenges. Figure 2 shows the proportion of students who encountered barriers, in descending order of prevalence for each barrier.

Confidence and knowledge
Medical students who completed the survey were very confident in their ability to prepare a searchable question for research or clinical needs. However, when asked to identify which portion of a given PICO question would need to be more detailed, only 15% identified that the intervention was not clearly defined. This was a question about drug therapy, something quite common, so the survey authors were sure that students would realize that the dosage would be an essential part of the question and that it was missing. On the other hand, the search strategy question based on this same scenario was answered correctly by 60% of the respondents. The authors were expecting a better response rate considering that the "logic" was clear in the leading information. There were still many students (36%) who were not applying Boolean operators appropriately, echoing Gruppen's findings over 10 years ago [4]. The survey results, when matched to the pre-and post-tests completed by the same students in 2012 and 2013 (Table 5), confirm that the skills improved after the pretest, but remained almost constant with the post-test results (59.3%) and therefore did not deteriorate over time. These findings confirm the need to include search strategy formulation in MIL programs, especially since this skill is applicable to many databases and search interfaces.
Another survey question was compared to the pre-and post-test results. Question 6 asks the students to select the best resource to find recent peer-reviewed articles on a topic. The post-test (47.5%) showed a modest improvement over the pretest (43%), whereas the survey results show a significantly marked improvement with 71.7% of the respondents choosing the correct answer. Prior to the MIL program, students believed that the best source of articles was to go directly to a journal covering the appropriate topic. The advantages of database searching are demonstrated time and again during the MIL program. The timing of the post-test, during the first year, probably does not yet allow the students to internalize that fact. With additional assignments, the Critical Enquiry project in second year and the MSEs in clerkship, the students search databases more often and must then realize that this is more efficient than choosing journals individually.
Medical students indicate that they are heavy users of drug information resources and they do feel confident that they can find independent drug information. Although almost 70% of the respondents were able to identify the resource with a drug interaction tool, only 28% of respondents were correct when they disagreed with the statement that the Canadian drug tool (e-CPS, recently renamed to  RxTx) contained independent information. This resource is a compilation of the drug monographs supplied by the manufacturers. This result will prompt the librarians to stress that information as part of the teaching about drug information resources. For the last six years, class time has been reduced and content about drug handbooks was moved to an online tutorial. From now on, during the class time, librarians will need to include this important fact about the Canadian drug information resource as part of the discussion of the highlights of the online tutorial.
It is important to notice that despite the high confidence recorded at the beginning of the survey, the students did use the "I don't remember" or "I don't know" quite frequently to answer the knowledge questions at the end of the instrument. The True/False questions in particular had a very high percentage for this choice except for the question about the Cochrane Library, which was answered correctly by the highest percentage of respondents among all the knowledge questions. Although overall knowledge scores were not high, and many students chose the "I don't remember" option, there were five questions that were answered correctly by a majority of the students.
Although one may expect to find a relationship between confidence and knowledge, the current study revealed only a very weak relationship between these two variables. This finding reflects what others have found: confidence in abilities is not reflected in knowledge even after two years post intervention. Considering the findings by Salbach et al. [12] that health care professionals "describe a waning in confidence in their ability to access and critically appraise the literature over time," one would think that confidence would have diminished after that amount of time; however, the fact that this did not occur may reflect a desirability bias: medical students rate their confidence high because they feel it is expected or desired.
The discrepancy between confidence and actual knowledge could also be explained by the DunningÁKruger effect [15]. The authors state "that people who lack the knowledge or wisdom to perform well are often unaware of this fact." Although the Queen's University MIL program is quite comprehensive, with multiple opportunities for assessment, the students are still considered novice users of information resources such as Medline and PubMed. Perhaps future MIL programs should focus less on teaching students the complexities of using Medical Subject Headings and concentrate more on the difficulties of navigating the substantial world of medical information. How to select and evaluate an appropriate source of information, for the practitioner or for the patient, remains a challenge that librarians must address with future medical students.

Using information resources
Overall, medical students agreed that their skill in locating reliable medical information had improved during the undergraduate years, and over 80% agreed or strongly agreed that they continued to use the resources presented in the preclinical years during clerkship (question i. in Table 3). They reported using appropriate resources to complete their MSEs (Figure 3), and the survey results found usage in all categories of resources ( Figure 1) with the least used being the citation management software and e-books. Citation management is encouraged, but librarians do not spend a lot of time covering this in the MIL program. The increase in freely available software for this purpose and the declining usage of the university-wide offering led the librarians to believe that students would be choosing their own software to suit their individual needs. Further questions will need to be explored with other classes to determine why the students are not users of e-books. One could surmise that the POC tools and mobile apps are replacing during clerkship the e-books used in the preclinical years. However, based on the recent study by Pickett [16], it is also possible that students simply do not like using e-books. This has far-reaching implications for collection development at this university, as the preference has been to purchase e-books over print for many years.
Mobile apps and individually purchased resources are understandably the highest used of all resources. These would include resources that are easily accessed, potentially without the need for an internet connection, and so would be most convenient in a clinical environment. Point-of-care tools and drug information resources were also used most frequently on a daily or weekly basis. It is important for collection maintenance in a difficult funding environment to confirm that both of these resource types are among the highest used by the clerks.   The high use of web resources is not surprising. Not only do librarians know from observing students that they search Google and Google Scholar to locate articles, but they also learn to use practice guidelines and reliable association or government websites recommended by faculty. Librarians at this institution have observed during recent classes more reluctance to learn and use the Ovid databases (Medline, EMBASE). It seems that structured, but still simple search strategies are not considered to lead to better search results compared to "google-style" searching in PubMed. This finding is influencing the decision to include advanced PubMed searching in term 1, building on the fact that the students all start medical school having searched PubMed before. When and if to introduce Ovid searching is still a debate among the librarian-teachers at this university. Although Ovid searching was ranked low, being used less than monthly by most students responding to that part of the survey, it should be noted that Ovid databases were the most used resources when students were working on their MSEs ( Figure 3). It would be safe to assume that the MSE, requiring them to locate articles, would lead them to use databases, whereas day-to-day questions may well be answered by POC tools or other quick reference resources. Similar to findings by Shanahan [17], almost half the students reported using two or more databases when working on their MSE assignments (Figure 4). Verifying information from more than one source is one of the teaching points of the Queen's University MIL curriculum.
Students also feel confident that they know how to choose appropriate resources and 98% of them agree or strongly agree that they will explore resources available to them during residency. Although Green and Ruff [6] found over 10 years ago that health care professionals were not aware of resources available to them, medical students graduating now face such a wide range of resources that it is perhaps clear to them that it is to their advantage to look into the reliable resources offered at the university.

Barriers to access
Contrary to findings about faculty use of information resources [6,8,18], lack of time was not a highly ranked barrier. The most common barriers referred to access and use of online resources: remote access, access to full text, and mobile interface difficulties. The first two barriers need to be investigated with the affiliated hospitals and clinics to ensure that appropriate internet access and bandwidth are available to the clerks. Mobile interfaces would need to be discussed with future classes to ascertain what the issues are before contacting the content creators.
The lack of role models, which was qualified in the survey with "no guidance on the use of the resources in clinical practice," was also mentioned frequently, confirming findings in the studies by Cullen et al. [7] and Stevenson [8]. Chosen less frequently was the lack of continued exposure (17%) meaning that for some students they would have liked more obvious use of information resources by peers, residents, and faculty working with them in the clinical environment. However, it is reassuring to find that peer pressure (no one else is using them) was not a highly ranked barrier (1.9%).

Limitations of this study
The first key limitation in the current study was the time allotment available to students to complete the survey. A short time allotment (10 minutes) may have impacted students' opportunities to think or reflect through knowledge questions, and could potentially have contributed to lower than expected knowledge scores. Consideration of these short timelines also drove the selection of multiple choice and true/false knowledge question format, which are not optimal to assess MIL skills. In the future, additional time will be requested to allow students the opportunity to think deeper about responses and to allow for open-ended questions.
The second limitation of the current study was the lack of available detailed pre-and post-test survey data. Had detailed responses been available, more rigorous statistical comparisons could have been conducted to assess differences in scores across time. Future studies will remedy this limitation through implementation of consistent measures conducted at multiple points in the program, and intentional retention of data for a longitudinal study.

Conclusion
This study aimed to answer a number of research questions. Did medical students retain their MIL skills two years after the last intervention? Did students change their information seeking practices? Did they continue to use the resources shown to them during the MIL program? Although the results of the knowledge portion of the survey are not encouraging overall, there are some areas of significant change. More encouraging is the positive attitude of students towards the use of information resources presented during MIL sessions and their self-perceived confidence that they can locate more reliable information at this time in their education compared to when they started medical school (Table 3). Their choice of resources for the MSE also demonstrates an acceptance in the use of databases to locate articles that was not evident at the beginning of their program.
With these survey results, the librarians will be able to target improvements to the MIL curriculum specifically related to drug information, consumer health resources, PICO question formulation, and Medline/PubMed searching, including the translation of a question into a search strategy with appropriate Boolean logic. Curriculum revisions will also need to include more content on how best to select a resource for varying information needs.
There are some important barriers to investigate that can hopefully result in improved access for all students. Environmental barriers will be discussed with faculty in charge of the clerkship curriculum.
This survey will be repeated with future fourth year students to gather more evidence about the long-term use of information resources and retention of medical information literacy skills.
A more detailed MSE review is planned for the fall of 2016 which will hopefully provide more information about the use of information resources, the barriers to usage and preceptor feedback for the information research section of the MSE.
Although this study was conducted at a single university, most of the results confirm previous studies. The authors believe that the conclusions may be applicable to other medical school information literacy programs and would be interested in future collaborations. The survey instrument as well as the collected data are made available to encourage others to reproduce this study, using the instrument in its entirety or adapted to local needs.