Lost in Translation: Supporting Learners to Search Comprehensively Across Databases

PROGRAM DESCRIPTION / DESCRIPTION DU PROGRAMME

JCHLA / JABSC 36: 54–58 (2015) doi: 10.5596/c15-014

Lost in Translation: Supporting Learners to Search Comprehensively Across Databases1

Robin Parker and Maggie Neilson

Robin Parker2, W.K. Kellogg Health Sciences Library, Dalhousie University, 5850 College St, Halifax, NS B3H 4R2.

Maggie Neilson, Vaughan Memorial Library, Acadia University, 50 Acadia Street, Wolfville, NS, B4P 2R6.

1This article has been peer reviewed.
2Corresponding author (e-mail: robin.parker@dal.ca)

Abstract: Introduction: Health sciences librarians play the key role of expert searcher for knowledge synthesis research projects. When students and trainees conduct systematic reviews as academic assignments, academic librarians train learners to search comprehensively for evidence in multiple sources. Description: The authors created an electronic toolkit with handouts and a video tutorial to support instruction on translating search strategies to various databases. Outcomes: The toolkit was well received by users, who provided constructive feedback and reported an increase in comfort with translating searches. Refinements based on the assessment results will improve the tools and supplemental resources will address some gaps in coverage. Most users still expressed the need to consult with a librarian for further training and review of their searches. Discussion: Trainees who need to conduct their own comprehensive searches for academic work will benefit from a variety of training tools to suit different levels of experience and learning styles. Electronic instructional resources such as handouts and videos can effectively supplement hands-on training and feedback from a health sciences librarian.

Introduction

Health sciences librarians have a well-established role in the conduct of knowledge syntheses (KS), which are core to evidence-based clinical practice (EBCP) and knowledge translation [1, 2]. We use the term KS here to refer to the category of research methods including, but not limited to, systematic and scoping reviews, realist reviews, clinical practice guidelines, and meta-narrative reviews. The standards of reputable health research organizations such as Cochrane, the Agency of Health Research Quality (AHRQ), and the Institute of Medicine, recommend or require the involvement of information specialists in search strategy development for systematic reviews [35]. This inclusion in methods guidance acknowledges the importance of the skills and expertise brought by information professionals, and it corresponds with the Medical Library Association statement on the “Role of expert searching in health sciences libraries”: health sciences librarians are crucial for expert searching and for training health sciences practitioners to improve their information retrieval skills and knowledge [6].

For KS research projects undertaken by faculty, clinicians, and researchers, review teams regularly consult health sciences librarians who then develop the search strategies for their systematic reviews [79]. However, when students and trainees are assigned comprehensive reviews as part of their academic programs the role of academic librarians shifts, from directly developing and running the searches to providing instruction so the learner can conduct the comprehensive search independently. As reflected by a recent presentation at the Canadian Health Libraries Association conference, the number of theses and dissertations that rely on KS methods is increasing [10]. Therefore, teaching systematic searching skills has been added to the information literacy roster for many academic librarians.

To search effectively for evidence, trainees should understand the importance of creating a comprehensive search strategy. Comprehensive searching involves developing a search strategy with the appropriate balance between sensitivity and specificity in one citation database, then translating that strategy using the syntax and functionality of other citation databases. Teaching comprehensive search skills therefore includes the EBCP concept of “searching for evidence” (i.e., translating a question into searchable concepts and identifying terms to capture those concepts), as well as bibliographic database instruction for multiple platforms, since standards for systematic reviews recommend searching at least two sources, with the minimum considered to be MEDLINE and the Cochrane Central Register of Controlled Trials [3, 5].

The objective of this program is to provide point-of-need (asynchronous) training tools to support learners working on comprehensive reviews. Supplementing existing instruction on comprehensive searching, these resources help to provide baseline knowledge and skills to trainees so that follow-up consultations with the health sciences librarian can be more in-depth and productive.

Description

To best assist review authors when translating their KS searches from one database to another, we developed a series of tools: a quick reference chart, a more elaborate guide, and a video tutorial. These tools demonstrate functions and search processes for databases commonly consulted for comprehensive reviews and that are available through Dalhousie Libraries, specifically PubMed, CINAHL (EBSCO), and Embase (Elsevier). Each of the three tools delivers a different level of detail to cater to the varying experience levels and learning styles of trainees. This toolkit is intended to complement a series of video tutorials that introduces comprehensive searching and the development of an initial full search in PubMed, which was created previously for a graduate course on systematic reviews. We initiated creation of the toolkit in July 2013 and completed it in January 2014.

We designed the first tool, a quick-reference chart, to provide the least amount of descriptive detail (see Supplementary Table 11). Acting as a “cheat sheet”, the chart provides comparisons of database functions such as subject headings, truncation, and how to combine searches. It was imagined to be most useful for review authors who need brief database reminders. Examples from each database are provided to illustrate how to adapt search techniques during a translation.

The guide, our second tool, is more comprehensive than the chart to provide more support to less experienced learners (Supplementary Figure 11). The guide contains extra translation tips, explanations of database functions and processes, and screenshots to assist users in locating and recognizing different parts of each of the databases. Basic definitions for various elements involved in a search (such as syntax, explosion, or mapping) are provided to assist review authors in gaining a better understanding of the search process. Further, suggestions for an improved search are provided throughout the guide, as well as potential pitfalls and limitations of various databases.

The video tutorial is the most comprehensive and detailed of the three tools, and elaborates significantly on search translation from PubMed into both CINAHL and Embase. Created using Camtasia Studio software (TechSmith Corporation), the video consists of PowerPoint presentation slides and screen capture clips. Audio voice-over guides users through the steps of translation. Users have the ability to jump to different sections of the video as needed using a table of contents function.

In addition to the same tips and explanations found in both the chart and the guide, the video also provides instructions on how to create and use a table to document concepts during searching and translation. Further, the video tutorial takes users through every step of the search translation, from locating the appropriate subject headings to importing search results into the citation management software RefWorks.

We evaluated user satisfaction with and the effectiveness of this suite of electronic tools. After a brief review of similar library tutorial evaluations, we developed survey questions based on common practices and focused on items that would provide the most insight into user needs and preferences. A colleague peer-reviewed our questions, and two librarians and an intern piloted the survey tool to provide further feedback on face and content validity. Our survey focused on the components of the toolkit, and solicited actionable input to inform us of revisions that would improve the quality of the tools. Further, we sought to assess the value of the toolkit to different disciplines, roles, and levels of comprehensive search experience. We were also interested in the ability of the toolkit to replace an initial face-to-face instruction session. The survey focused on ease of use, personal preferences of the users, and impact on search translation outcomes (see Supplementary Figure 21). Owing to limited time, the tool was not tested for reliability.

One author (RP) recruited respondents from among individuals who sought librarian assistance with a synthesis project. Undergraduate, graduate, and doctoral students, as well as faculty of the health sciences and medicine programs were eligible to receive the toolkit and participate in the follow-up survey. Trainees were informed of the evaluation and provided consent, but research ethics approval was not required as the survey consisted of program evaluation.

We collected data using Opinio software and downloaded responses into Microsoft Excel for analysis. Themes and trends were identified via the categorization of text responses, and basic descriptive statistics were used to summarize the responses to closed questions.

Outcomes

Demographics

When the survey closed, 10 of the 18 individuals who had opened the survey had completed and saved their results. A test response was eliminated, as well as the surveys that had not been completed and saved. All responses were from members of the Dalhousie community, and represented diversity in program, role, and type of synthesis project (Figures 1 and 2). Respondents included undergraduate medical students; graduate students from medical, nursing and library degree programs; and librarians without comprehensive searching experience.

Fig. 1. Role and primary faculty/school (n = 10).

Fig. 1.

Fig. 2. Synthesis projects (n = 10).

Fig. 2.

Baseline experience

When asked about their level of experience with searching comprehensively, the respondents reported a range of experience; some reported having searched comprehensively prior to this study, whereas others had very little or limited experience (Figure 3). All respondents reported at least some comfort with PubMed searching; fewer were confident in searching the database comprehensively. When asked about their comfort levels when searching in databases other than PubMed (CINAHL, Embase, and PsycINFO) the spectrum of responses was wider, ranging from no comfort with any of the databases to one person who reported being very comfortable with searching in all of them.

Fig. 3. Experience searching comprehensively (n = 10); respondents could select more than one option.

Fig. 3.

Video evaluation

We asked the respondents about a variety of the video's characteristics, including the arrangement of the steps, language, pacing, clarity, and content. Most or all of the responses were positive regarding the sequence, language, pacing, and ability to understand the concepts. Useful comments that will influence the revision of the video include feedback about the speed of a section that covered exporting of results to RefWorks and the use of more visual highlighting to improve clarity. Another suggestion was to provide more detail about proximity operators and truncation; this has been addressed through the creation of short supplemental videos that are now available on the Dalhousie Libraries website (http://libraries.dal.ca/using_the_library/online_tutorials.html). The final question dedicated to the video tutorial was about the content. Nine of the 10 respondents reported that they would like to see coverage of additional topics. The suggestions in Table 1, extracted from both survey responses and additional email feedback, will inform the creation of additional tutorials and the revision of the search translation video, as indicated.

Table 1. Search translation video feedback.
Topic/concept suggestion Next steps/actions
Explosion of headings Cover briefly in revised video and direct to other existing resources
Searching in other databases (Cochrane, PsycINFO) Direct to other existing resources
Proximity operators Supplemental video completed
Clearing the search field in CINAHL Revise search translation video
Use of index terms and text words Direct to other existing resources (videos in systematic searching series)
Symbols/syntax Direct to other existing resources (search translation chart)
More RefWorks Direct to other existing resources
More complicated search examples Create additional videos/tools
Truncation and phrases Supplemental video created
Appropriate terms to include Direct to other existing resources
Saving search strategies Revise search translation video
Use of text word versus title/abstract in CINAHL Revise search translation video
Use of age filters and limits Create additional tools or direct to other existing resources

Toolkit use and search outcomes

We asked several questions about how the respondents used the components of the toolkit and made inferences about the effectiveness of those components from the respondents’ reported ability and comfort with completing the search translations. Individuals used the combination of tools in various ways and their preferences for each resource varied. The majority of respondents preferred and relied most heavily on the video. The chart was the top choice for ease of use. Overall, respondents found the tools to be useful and effective in developing their skills and confidence when translating initial searches into subsequent databases. Responses indicated that seven out of eight users who had completed their translations were able to do so in three hours or less.

Responses to the virtual toolkit were positive and eight of 10 respondents indicated that they would also want to consult a librarian for additional questions or to confirm their work. This was also reflected both in the written comments and in our experience providing follow-up support for those who had used the toolkit. Trainees who had watched the video and (or) referred to the handouts had subsequent complex and in-depth questions.

Discussion

Historically, librarians have explored various approaches to information literacy, including embracing web-based technology to provide just-in-time instructional resources in addition to the more traditional just-in-case, in-person instruction. In keeping with evidence-based librarianship, librarians have examined the impact of various modes of instruction [1113] and developed best practices for different formats [1417]. We examined our evaluation results in the context of these previously identified practices.

This evaluation provided us with insight into the usefulness of this training resource, as well as improvements that can be made to enhance its usability and content. All of the respondents felt that the toolkit provided adequate information for properly translating a search, which speaks to both the presence of a solid instructional foundation and the ability of the toolkit to provide support to a diverse user group in regards to education level, role, and search experience. Further, the variety of synthesis projects and comprehensive searches the respondents were undertaking demonstrates the toolkit's relevance to different projects. Respondents reported use of the components in a variety of ways to suit individual needs, which demonstrates the toolkit's flexibility. This variation correlates with our expectations that diverse users would prefer different types of resources.

The participants’ suggestions to improve visual elements and presentation to enhance the video corroborated the findings of Foster et al. [17] in their review of video tutorials. The video will be revised in the summer of 2015 to incorporate the survey feedback and best practices as identified by Foster et al. [17], especially regarding learning objectives and interactivity.

Content occasionally lacked in both clarity and coverage, though it was largely deemed suitable by the respondents. Although some of the details requested could be added to the current video, in the interests of length and comprehensiveness, supplementary videos or resources may be more appropriate, as noted in Table 1.

Users still desired the help and support of librarians through their searching process, which is in line with the findings of Boden et al. [18] in their evaluation of screen-capture tutorials. Although all of the respondents felt the toolkit was useful and increased their confidence, open-ended responses emphasized their need for librarians to guide and appraise their searches. Standards for KS increasingly encourage peer-review of search strategies, so further consultation with the search expert is both appropriate and desirable [19]. The continued need for librarian instruction and support highlights the toolkit's usefulness as an aid to, rather than a replacement for, expert consultation and training. Use of online instructional tools has been noted as an effective supplement to face-to-face instruction, particularly when the tasks to be learned are executed in an online environment [13]. We found these online tools facilitated users’ appreciation for librarian support and expertise, and expedited the training and consultation process.

Limitations and next steps

The greatest limitation of this evaluation was the small number of respondents who assessed the toolkit. Nonetheless, the feedback from these 10 respondents will, together with the best practices identified by Foster et al. [17], inform revisions such as incorporation of increased interactivity and more explicit learning outcomes.

A second limitation pertains to evaluation of a tool used by people with different research pursuits. This coincides with both the evaluation of PubMed tutorials by Tuttle et al. [20] and the evaluation of screen capture tutorials by Boden et al. [18]. Since we did not evaluate the resulting search strategies, we assessed the efficacy of the translations based on self-report alone, which is a less reliable measure of impact on outcomes. This limitation regarding the objectivity and quality of studies in education research has been noted frequently [14].

In addition, the authors’ inexperience with survey questions and the lack of a validated evaluation tool to assess the effectiveness and function of the electronic toolkit decrease the generalizability of our results. This assessment will allow us to refine our instructional support resources, and future research plans include evaluation of the search strategies developed by trainees who have used the enhanced electronic toolkit. Learner-oriented and outcomes-based research methods identified by Boden et al. [18] in their pilot study to assess impact of online tutorials will be incorporated. In particular, we will test the intervention against a control group and assess both quantitative and qualitative responses.

Conclusion

This toolkit has been a valuable addition to our instructional resources for comprehensive search training. Our subsequent evaluation informed us of refinements to be made to the toolkit in order to best serve the needs of trainees when learning these skills. Based on feedback on the content and usability of the toolkit, as well as self-reported learning resulting from its use, we will further develop these and other instructional components to achieve improved efficacy and user satisfaction. The resulting tools will support searches for knowledge synthesis projects so trainees can return for more advanced assistance, thereby reducing librarian teaching time and encouraging independent skill development.

1 Supplementary data are available through the journal Web site at http://ejournals.library.ualberta.ca/index.php/jchla/rt/suppFiles/24383/0.

References

1. DiCenso A, Bayley L, Haynes RB. ACP journal club. Editorial: Accessing preappraised evidence: Fine-tuning the 5S model into a 6S model. Ann Intern Med. 2009;151(6): JC3,2, JC3. doi:10.1136/ebn.12.4.99-b.
2. Canadian Institutes of Health Research (CIHR). More About Knowledge Translation at CIHR [Internet]. [cited 23 Mar 2015]. Available from: http://www.cihr-irsc.gc.ca/e/39033.html#Definition.
3. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1. [Internet]. The Cochrane Collaboration. 2006 [updated March 2011; cited 23 March 2015]. Available from www.cochrane-handbook.org.
4. Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews [Internet]. Rockville, MD: Agency for Healthcare Research and Quality. January 2014 [cited 23 Mar 2015]. Available from: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/318/CER-Methods-Guide-140109.pdf.
5. Finding what works in healthcare: Standards for systematic reviews [Internet] Washington, DC: Institute of Medicine of the National Academies. March 2011 [cited 23 Mar 2015]. Available from: http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews.aspx.
6. Medical Library Association. Role of expert searching in health sciences libraries: Policy statement by the Medical Library Association adopted September 2003. J Med Libr Assoc. 2005 Jan;93(1):42–4.
7. Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15. doi:10.1080/02763869.2011.590425.
8. Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. J Med Libr Assoc. 2013;101(4):268–77. doi:10.3163/1536-5050.101.4.008.
9. Heimlich LS. New and emerging roles for medical librarians. J Hosp Librarian. 2014;14(1):24–32. doi:10.1080/15323269.2014.859995.
10. Gore G, Cobus-Kuo L, Kloda LA. Research syntheses produced as graduate theses and doctoral dissertations: a scoping review. Canadian Health Libraries Association (CHLA) Annual Conference; 2014 Jun 16–20; Montreal QC. Available from: http://www.chla-absc.ca/conference/sessions/2013-research-syntheses-produced-graduate-theses-and-doctoral-dissertations-scoping-review.
11. Anderson K. Does the method of instruction matter? An experimental examination of information literacy instruction in the online, blended, and face-to-face classrooms. J Acad Librarian. 2010;36(6):495–500. doi:10.1016/j.acalib.2010.08.005.
12. Schilling K, Applegate R. Best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. JMLA. 2012;100(4):258. doi:10.3163/1536-5050.100.4.007.
13. Silk KJ, Perrault EK, Ladenson S, Nazione SA. The effectiveness of online versus in-person library instruction on finding empirical communication research. J Acad Librarian. 2015 Mar;41(2):149–54. doi:10.1016/j.acalib.2014.12.007.
14. Anderson RP, Wilson SP, Livingston MB, LoCicero AD. Characteristics and content of medical library tutorials: A review. J Med Libr Assoc. 2008 Jan;96(1):61–3. doi:10.3163/1536-5050.96.1.61.
15. Anderson RP, Wilson SP, Yeh F, Phillips B, Livingston MB. Topics and features of academic medical library tutorials. Med Ref Serv Q. 2008;27(4):406–18. doi:10.1080/02763860802368217.
16. Befus R, Byrne K. Redesigned with them in mind: evaluating an online library information literacy tutorial. Urban Library Journal. 2011;17(1):1–26.
17. Foster MJ, Shurtz S, Pepper C. Evaluation of best practices in the design of online evidence-based practice instructional modules. JMLA. 2014;102(1):31. doi:10.3163/1536-5050.102.1.007.
18. Boden C, Neilson CJ, Seaton JX. Efficacy of screen-capture tutorials in literature search training: A pilot study of a research method. Med Ref Serv Q. 2013;32(3):314–27. doi:10.1080/02763869.2013.806863.
19. Niederstadt C, Droste S. Reporting and presenting information retrieval processes: The need for optimizing common practice in health technology assessment. Int J Technol Assess Health Care. 2010 Oct;26(4):450–7. doi:10.1017/S0266462310001066.
20. Tuttle BD, Isenburg Mv, Schardt C, Powers A. PubMed instruction for medical students: Searching for a better way. Med Ref Serv Q. 2009;28(3):199–210. doi:10.1080/02763860903069839.

Refbacks

  • There are currently no refbacks.




 

JCHLA/JABSC | CHLA/ABSC on Twitter