Research Article

 

Teaching Systematic Searching Methods to Public Health Graduate Students: Repeated Library Instruction Sessions Correlate With Better Assignment Scores

 

John Pell

Assistant Librarian

Social Work & Public Health Library

Hunter College

New York City, New York, United States of America

Email: jpell@hunter.cuny.edu

 

Received: 20 Oct. 2016   Accepted: 11 Mar. 2017 

 

 

cc-ca_logo_xl 2017 Pell. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

 

Abstract

 

Objective – The objective of this study was to devise an assessment plan to determine if repeat attendance at two library instruction sessions is statistically associated with overall assignment scores or specific assignment qualities.

 

Methods – The author used SPSS to calculate correlations between attendance and assignment scores and cross tabulations between attendance and assignment item analysis scores.

 

Results – Repeat attendance at two library instruction sessions was statistically associated with higher overall assignment scores and higher scores on specific assignment sections. The effect is statistically significant.

 

Conclusion – Students who attended two library instruction sessions applied skills and concepts practiced in those sessions on a subsequent research assignment. Not all skills and concepts practiced in the session were applied. Acquisition of more technical skills such as Boolean searching may require a greater number of follow-up sessions.

 

Introduction

 

Systematic reviews are a high quality form of evidence in fields concerned with evidence based practice. Systematic reviews are at the peak of “evidence pyramid” models that rank evidence quality. Numerous reports on research agendas emphasize the importance of consulting and carrying out more systematic reviews (Hawke, Burns, & Landorf, 2009; Howes, Doyle, Jackson, & Waters, 2004; Kite, Indig, Mihrshahi, Milat, & Bauman, 2015; Stewart, 1996; Szajewska, 2013; Whelan, 2014). The attention paid to systematic reviews has proven something of a boon to librarians since completion of a systematic review requires expertise in literature searching. Prominent “gold standard” manuals of systemic review methodologies such as the Cochrane Handbook and the Institute of Medicine Standards have thrown a spotlight on the search expertise of librarians by recommending teaming up with librarians to carry out a systematic search (Higgins & Green, 2011; Research & Medicine, 2011). This emphasis on librarian involvement in systematic review teams has been validated by research into the quality of systematic reviews. This research has shown that having a librarian co-author on a systematic review correlates with higher quality systematic review methodologies (Rethlefsen, Farrell, Osterhaus Trzasko, & Brigham, 2015).

 

Considering this background, there are several good reasons for librarians who support students and researcher in evidence based fields to promote librarian-led training in systematic searching methods to graduate students. It promotes the literature searching expertise of librarians to students and faculty, it can prepare students for a position as a research assistant, and students can apply what they learn from the training to subsequent research assignments. 

 

This paper is specifically concerned with demonstrating, that under the right circumstances, students can learn and apply systematic searching skills to successfully complete research assignments.

 

Aims

 

This paper describes an assessment method to test the following questions:

 

1. Are library research assignment scores correlated with other assignment scores?

2. Is attendance at library instruction sessions associated with better assignment scores?

3. What assignment characteristics are associated with attendance at library instruction sessions?

 

This paper will present the results of an assessment plan developed to answer these questions using data compiled by instructors of a cohort of public health graduate students.

 

Literature review

 

There is a robust body of published assessments of librarian-led training in literature searching for medical students. There are comparatively fewer examples of assessments of learning outcomes from literature search training for graduate students in other fields. There are even fewer examples of assessments of training in systematic search methods for students outside of professional development programs for librarians (Conte et al., 2015). There is at least one example of an effort to teach systematic searching to undergraduate nursing students that shows improved evidence summary outcomes as a result (Whalen & Zentz, 2015). The literature on database training for medical students suggests that librarian-led trainings can be effective at improving the evidence based literature searching skills of medical students. The literature reveals that much of this evidence, pointing to a positive effect, comes from studies with weak designs (Garg & Turtle, 2003; Just, 2012; Maggio & Kung, 2014). A rigorously designed double blind clinical trial did not find any effect on medical student search skills following a single training session (Ilic, Tepper, & Misso, 2012). However, Maggio and Kung propose that this null effect could be consistent with a paradigm in which longitudinal designs and follow-up training sessions are required for effective retention of skills and knowledge (2014). 

 

Effects associated with librarian-led trainings in literature searching for medical students can include increased confidence and use of demonstrated resources (Miller, 2014; Rafferty, 2013). However, some studies offer conflicting findings. For instance, training does not always increase confidence, it can also raise awareness of the complexity of expert literature searching and increase requests for librarian assisted searches (Addison, Glover, & Thornton, 2010). 

 

Much of the assessment literature on librarian involvement with graduate students in fields outside of medicine focuses on needs assessments and student preferences for topics and mode of instruction. The results of these assessments emphasize the need for development of subject-specific content (Baruzzi & Calcagno, 2015; Critz et al., 2012; Fong, Wang, White, & Tipton, 2016; O’Malley & Delwiche, 2012; Roszkowski & Reynolds, 2013; Tomaszewski, 2012). The study I describe in this paper is unique in terms of content area, method of assessment, and student population.

 

Methods

 

Students in a graduate-level public health course (n = 68) had the opportunity to attend two optional two-hour instruction sessions that supported a required library research assignment. Forty-three students attended both sessions. Twenty-five students either attended one session or did not attend any sessions. This study compares the performance of the forty-three students who attended both sessions to the performance of the twenty-five students with incomplete attendance.

 

The instruction sessions were led by faculty librarians with American Library Association-accredited Master of Library Science or Master of Library & Information Science degrees. They provided the students with active learning exercises in stating research questions in a Population/Intervention/Comparison/Outcome (PICO) format, looking up Medical Subject Heading (MeSH) terms related to the concepts in the research question, developing a Boolean-based search strategy that includes keywords and subject heading combinations, identifying literature reviews in PubMed that relate to the research question and hand searching bibliographies for relevant studies, selecting databases other than PubMed/MEDLINE according to the need of the research question, documenting manual search criteria, and using citation management software to format in-text citations and lists of works cited in AMA style. These exercises targeted the same set of skills that the library research assignment required.

 

Students submitted the completed assignments to the librarian instructors, who then evaluated the assignments with a rubric. Librarian instructors received training in the use of the rubric through norming sessions intended to ensure that the librarians applied the rubric consistently.

 

The data set for the assessment consists of collected attendance notes from the library instruction sessions, completed grading rubrics from the library research assignment, and scores on another assignment submitted in the same class. 

 

The author used SPSS to calculate a Pearson correlation between the library assignment scores and scores on another research assignment given in the same course. The purpose of the other assignment was to deliver a public health brief, which is a summary of the current research relating to an assigned topic. The author calculated chi-square cross tabulations in SPSS between attendance at both sessions and assignment score, defining “higher score” as a score above the median score for the cohort.

 

An institutional review board reviewed the protocol for data analysis for this project and determined that it qualified for an exemption from review of human subjects as a research study of existing data, documents, or records.

 

Results

 

The author found that a Pearson correlation showed library research assignment grades were substantially (r = .534) correlated to grades on the other individual research assignment given in the same course. This relationship is statistically significant (p<.01).

 

The author found that a chi-square test showed students who attended both library instruction sessions were likely to score above the median assignment score. This relationship is statistically significant (p<.01).

 

Chi-square tests showed that attendance at both library instruction sessions had a statistically significant association with retrieval of a literature review related to the subject of the student’s research (p = 0.005), PICO structured research questions (p = 0.006), and clear statements of inclusion and exclusion criteria with a logical relationship to the research question (p = 0.01). These three assignment characteristics showed the strongest statistical association with attendance at both library instruction sessions.

 

Attendance at both library instruction sessions was also statistically associated with an accurate summary of primary sources and correct use of AMA citation style (p = 0.03) and, to a lesser extent, use of controlled vocabulary and Boolean logic (p = 0.07). However, these associations, though suggestive, were not statistically significant.

 

Discussion

 

The training sessions described in this study taught students how to systematically search the public health literature. They were not training sessions on how to do systematic reviews. While the sessions introduced the concept of a systematic review and required students to practice some of the skills involved in producing systemic reviews, the training sessions did not cover a comprehensive array of the skills and knowledge required to carry out a full systematic review. Grey literature, clinical trial registries, and publication bias are just a few of the systematic review search skills and concepts left out of the training sessions in this study. The objective of the sessions was to develop skills and knowledge that could serve as a scaffold for further development of more sophisticated search skills. The objective of these sessions was not to produce students capable of conducting systematic reviews. Given that comprehensive systematic review courses can take 24 – 36 hours to complete (Johns Hopkins University, 2017) it would seem inadvisable to attempt to introduce a full array of systematic review concepts and skills in the short sessions assessed in this study. The results of this study suggest some possible limits to what can be accomplished in four hours, especially when training students with little to no prior knowledge about systematic reviews.

 

One of the most statistically significant associations with attendance at the instruction sessions concerns the retrieval of a literature review. A high percentage of students who did not attend both library instructions failed to submit literature reviews related to their research questions. When interpreting this result, it is important to remember that students may be able to identify literature reviews on a topic but may not have the skills required to efficiently search for and access relevant literature reviews for an assignment. One strategy is to teach these students how to use the “publication type” field in PubMed. This could significantly improve retrieval skills, as it did for the cohort that attended both sessions.

 

In contrast to the students’ improved abilities in the areas of literature searching, search criteria, and PICO question formulation, students’ performance on the Boolean searching tasks in the library research assignment did not show a statistically significant association with attendance at the library instruction sessions, despite strong emphasis on Boolean searching. It may be that Boolean coding skills were too far removed from the experience of this cohort to be significantly absorbed after two instruction sessions.

 

This study has some limitations. The data was gathered from a convenience sample of graduate students enrolled in a public health class and students self-selected into the library instruction sessions. It is important to bear these limitations in mind when considering the question of whether the library instruction sessions were a causal factor in the achievement of a higher score on the library assignment or the other individual research assignment in the course.

 

This study was not designed to test the impact of variation in instructor skills and experience on student outcomes. The training sessions employed a student-centered, active-learning pedagogy intended to mitigate for individual differences among instructors and their skills and experiences. Students who attended two sessions often had different instructors for each session. Despite these measures, the fact remains that this study did not collect data on variations in instructor skills and experiences; therefore, it cannot conclusively resolve questions about the impact of individual instructors on student outcomes.

 

Individual variation may have also affected the rubric scores. Although instructors received training intended to standardize their use of the rubric, this training cannot guarantee the elimination of instructor disagreement about rubric scores. Since the instructors each scored non-overlapping segments of the sample, it is not possible to quantify the inter-rater reliability for the instructors who contributed scores to this study.

 

The correlation between repeated attendance at library instruction sessions and higher assignment scores may show the transferability of skills and concepts acquired in the library session. On the other hand, this correlation may merely show that the best students showed up for both library training sessions. Taken alone, the results of the Pearson correlation and the cross tabulations relating voluntary, repeated library instruction attendance to assignment scores could be an artifact of a comparison between students with sufficient time and motivation to attend additional instruction sessions and students without such resources. However, the cross tabulations relating library instruction attendance to performance on different sections of the library research assignment provide evidence to suggest that students who attended two instructional sessions retained and applied specific skills from those sessions (retrieval of literature reviews, documentation of manual search criteria, and PICO formatting of research questions) on the subsequent assignment. When these results are considered together, a stronger case emerges for the causal impact of repeated library instruction sessions on assignment scores. The item analysis of the assignments strongly suggests that students who attended two training sessions retained and applied specific content from those sessions on a subsequent assignment.

 

Further studies with graduate public health students are needed to describe best practices for curriculum plans that ensure appropriately repeated training and exercise in library research methods with this student population.

 

Conclusions

 

The limited success of the two-session plan may be taken as evidence of the importance of repeated instruction sessions addressing systematic search skills for graduate students. More technical skills such as Boolean searching or the use of citation management software in conjunction with AMA formatting may require more follow-up sessions to increase student performance.

 

References

 

Addison, J., Glover, S. W., & Thornton, C. (2010). The impact of information skills training on independent literature searching activity and requests for mediated literature searches. Health Information and Libraries Journal, 27(3), 191–197. http://doi.org/10.1111/j.1471-1842.2009.00871.x

 

Baruzzi, A., & Calcagno, T. (2015). Academic Librarians and Graduate Students: An Exploratory Study. Portal: Libraries and the Academy, 15(3), 393–407. http://doi.org/10.1353/pla.2015.0034

 

Conte, M., MacEachern, M., Mani, N., Townsend, W., Smith, J., Masters, C., & Kelley, C. (2015). Flipping the classroom to teach systematic reviews: the development of a continuing education course for librarians. Journal of the Medical Library Association, 103(2), 69–73. http://doi.org/10.3163/1536-5050.103.2.002

 

Critz, L., Axford, M., Baer, W. M., Doty, C., Lowe, H., & Renfro, C. (2012). Development of the graduate library user education series. Reference Services Review, 40(4), 530–542. http://doi.org/10.1108/00907321211277341

 

Fong, B. L., Wang, M., White, K., & Tipton, R. (2016). Assessing and Serving the Workshop Needs of Graduate Students. The Journal of Academic Librarianship, 42(5), 569–580. http://doi.org/10.1016/j.acalib.2016.06.003

 

Garg, A., & Turtle, K. M. (2003). Effectiveness of training health professionals in literature search skills using electronic health databases--a critical appraisal. Health Information and Libraries Journal, 20(1), 33–41. http://doi.org/10.1046/j.1471-1842.2003.00416.x

 

Hawke, F., Burns, J., & Landorf, K. B. (2009). Evidence-based podiatric medicine: importance of systematic reviews in clinical practice. Journal of the American Podiatric Medical Association, 99(3), 260–266. http://dx.doi.org/10.7547/0980260

 

Higgins, J., & Green, S. (Eds.). (2011). Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org

 

Howes, F., Doyle, J., Jackson, N., & Waters, E. (2004). Evidence-based public health: The importance of finding “difficult to locate” public health and health promotion intervention studies for systematic reviews. Journal of Public Health (Oxford, England), 26(1), 101–104. http://doi.org/10.1093/pubmed/fdh119

 

Ilic, D., Tepper, K., & Misso, M. (2012). Teaching evidence-based medicine literature searching skills to medical students during the clinical years: a randomized controlled trial. Journal of the Medical Library Association: JMLA, 100(3), 190–196. http://doi.org/10.3163/1536-5050.100.3.009

 

Johns Hopkins University. (2017). Introduction to Systematic Review and Meta-Analysis. Retrieved from https://www.coursera.org/learn/systematic-review

 

Just, M. L. (2012). Is literature search training for medical students and residents effective? a literature review. Journal of the Medical Library Association: JMLA, 100(4), 270–276. http://doi.org/10.3163/1536-5050.100.4.008

 

Kite, J., Indig, D., Mihrshahi, S., Milat, A., & Bauman, A. (2015). Assessing the usefulness of systematic reviews for policymakers in public health: A case study of overweight and obesity prevention interventions. Preventive Medicine, 81, 99–107. http://doi.org/10.1016/j.ypmed.2015.08.012

 

Maggio, L. A., & Kung, J. Y. (2014). How are medical students trained to locate biomedical information to practice evidence-based medicine? A review of the 2007-2012 literature. Journal of the Medical Library Association: JMLA, 102(3), 184–191. http://doi.org/10.3163/1536-5050.102.3.008 

 

Miller, L. (2014). First Year Medical Students Use Library Resources Emphasized During Instruction Sessions. Evidence Based Library & Information Practice, 9(1), 48–50. http://dx.doi.org/10.18438/B8F316

 

O’Malley, D. & Delwiche, Francis A. (2012). Aligning library instruction with the needs of basic sciences graduate students: a case study. Journal of the Medical Library Association, 100(4), 284–290. http://dx.doi.org/10.3163/1536-5050.100.4.010

 

Rafferty, R. (2013). The impact of library instruction: do first-year medical students use library resources specifically highlighted during instructional sessions? Journal of the Medical Library Association, 101(3), 213–217. http://dx.doi.org/10.3163%2F1536-5050.101.3.011

 

Research, C. on S. for S. R. of C. E., & Medicine, I. O. (2011). Finding What Works in Health Care: Standards for Systematic Reviews. National Academies Press.

 

Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015). Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology. http://doi.org/10.1016/j.jclinepi.2014.11.025

 

Roszkowski, B., & Reynolds, G. (2013). Assessing, Analyzing, and Adapting: Improving a Graduate Student Instruction Program Through Needs Assessment. Behavioral & Social Sciences Librarian, 32(4), 224–239. http://doi.org/10.1080/01639269.2013.837798

 

Stewart, L. A. (1996). The importance of systematic reviews and meta-analyses in the practice of evidence-based medicine. Annals of the Academy of Medicine, Singapore, 25(4), 483–484.

 

Szajewska, H. (2013). Importance of systematic reviews and meta-analyses in pediatric nutrition. World Review of Nutrition and Dietetics, 108, 1–10. http://doi.org/10.1159/000351479

 

Tomaszewski, R. (2012). Information Needs and Library Services for Doctoral Students and Postdoctoral Scholars at Georgia State University. Science & Technology Libraries, 31(4), 442–462. http://doi.org/10.1080/0194262X.2012.730465

 

Whalen, K. J., & Zentz, S. E. (2015). Teaching Systematic Searching in a Baccalaureate Nursing Research Course. Worldviews on Evidence-Based Nursing / Sigma Theta Tau International, Honor Society of Nursing, 12(4), 246–248. http://doi.org/10.1111/wvn.12090

 

Whelan, K. (2014). Editorial: The importance of systematic reviews and meta-analyses of probiotics and prebiotics. The American Journal of Gastroenterology, 109(10), 1563–1565. http://doi.org/10.1038/ajg.2014.258