Librarians’ Reported Systematic Review Completion Time Ranges Between 2 and 219 Total Hours with Most Variance due to Information Processing and Instruction
A Review of:
Bullers, K., Howard, A. M., Hanson, A., Kearns, W. D., Orriola, J. J., Polo, R. L., & Sakmar, K. A. (2018). It takes longer than you think: Librarian time spent on systematic review tasks. Journal of the Medical Library Association, 106(2), 198-207.
Peace Ossom Williamson
Director for Research Data Services
University of Texas at Arlington Libraries
Arlington, Texas, United States of America
Received: 16 Nov. 2018 Accepted: 7 Jan. 2019
2019 Williamson. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.
Objective – To investigate how long it takes for medical librarians to complete steps toward completion of a systematic review and to determine if the time differs based on factors including years of experience as a medical librarian and experience completing systematic reviews.
Design – Survey research as a questionnaire disseminated via email distribution lists.
Setting – At institutions that are members of the Association of Academic Health Sciences Libraries (AAHSL) and librarians at Association of American Medical Colleges (AAMC) or American Osteopathic Association (AOA) member institutions.
Subjects – Librarians of member institutions who have worked on systematic reviews.
Methods – On December 11, 2015, AAHSL library directors and librarian members of AAMC and AOA were sent the survey and the recommendation to forward the survey to librarians on staff who have worked on systematic reviews. Reminders were sent on December 17, 2015, and the survey closed for participation on January 7, 2016. Participants who had worked on a systematic review within the past five years were asked to indicate experience by the number of systematic reviews completed, years of experience as a medical librarian, and how much time was spent, in hours, on the following: initial consultations/meetings; developing and testing the initial search strategy; translating the strategy for other databases; documenting the process; delivering the search results; writing their part of the manuscript; other tasks they could identify; and any instruction (i.e., training they provided to team members necessary for completion of the systematic review). Participants also further broke down the amount of their time searching, by percentage of time, in various resources, including literature indexes/databases, included studies’ references, trial registers, grey literature, and hand searching. Participants were also given space to add additional comments. The researchers reported summary statistics for phase one and, for phase two, excluded outliers and performed exploratory factor analysis, beginning with principal components analysis (PCA), followed by a varimax rotation, to determine if there was a relationship between the time on tasks and experience.
Main Results – Of the 185 completed responses, 105 were analyzed for phase one because 80 responses were excluded due to missing data or no recent experience with a systematic review. The average respondent had between 1 and 6 years of experience: 1-3 years in librarianship (49.5%) and 4-6 years (23.8%). The time reported for completion of all tasks ranged from 2 to 219 hours with a mean of 30.7 hours. Most of the variance (61.6%) was caused by “information processing” and “interpersonal instruction/training” components. Search strategy development and testing had the highest average time at 8.4 hours. Within that category, databases accounted for 78.7% of time searching, followed by other searching methods. For remaining systematic review tasks, their averages were as follows: translating research (5.4 hours), delivering results (4.3 hours), conducting preliminary consultations (3.9 hours), instruction (3.8 hours), documentation (3.0 hours), additional tasks that were written-in by respondents (2.2 hours), and writing the manuscript (1.8 hours). The most common written-in tasks were development of inclusion/exclusion criteria, critical appraisal, and deduplication. Other write-ins included retrieving full-text articles, developing protocols, and selecting a journal for publishing the systematic review.
For the second phase of analysis, 12 responses were excluded as extreme outliers, and the remaining 93 responses were analyzed to detect a relationship between experience and time on task. Prior systematic review experience correlated with shorter times performing instruction, consultation, and translation of searches. However, librarian years of experience affected the percentage of time on task, where greater years of experience led to more time spent consulting and instructing than the percentage for librarians with fewer years of experience. Librarians with greater than 7 years of experience skewed trends toward shorter time on task, and, with their data excluded, years of experience showed weak positive correlation with instruction and consultation.
Conclusion – Because the average librarian participating on systematic review teams has had few prior experiences and because the times can vary widely based on assigned roles, duties, years of experience, and complexity of research question, it is not advised to establish expectations for librarians’ time on task. This may be why library administrators have disparate expectations of librarians’ involvement in systematic reviews and find it difficult to allocate and anticipate staff time on systematic review projects. While it may not be possible to set specific overarching guidelines for librarians’ expected time on systematic review tasks, librarian supervisors and library directors planning for their staff to offer systematic review services should work to develop extensive understanding of the steps for conducting and assessing systematic reviews in order to better estimate time commitments.
Librarians typically serve in searching and data management roles on systematic review teams, and many provide expertise in question design and content evaluation (Dudden & Protzko, 2011; Spencer & Eldredge, 2018). Two recent studies investigated total time for systematic reviews and meta-analysis (Gann & Pratt, 2013; Saleh, Ratajeski & Bertolet, 2014); however, this is the first study to investigate the librarian’s time on each task contributing to systematic reviews. The study involved a survey questionnaire emailed to various lists, and the evidence could be stronger if the research were supported by an observational study of librarians, where they tracked their time and effort while conducting one or several systematic reviews. The variation in reporting may also be reduced by stratifying times by systematic review quality. There may be a relationship between time on task and quality of the final product if, for example, fewer databases were searched, search strategies did not include subject headings and were not customized for each database, additional relevant terms were missed, and grey literature was not explored.
The critical appraisal checklist developed by Boynton and Greenhalgh (2004) includes claims of validity and reliability in the reporting of survey research, and the study includes these attributes when explaining the methods of exclusion, quantitative analysis, and interpretation. The authors were transparent in their research by providing the instrument that was used. However, the researchers did not describe what were the statistical criteria for exclusion of outliers and thresholds for factor loadings and whether these were preestablished prior to data analysis, and, in Figure 2 and Table 3, there are numbered components upon which the data relies; however, there is no explanation of what factors (other than factors 1 and 2) with which those numbers align.
The article did not report how the survey was piloted, although the survey was relatively brief with questions that were straightforward. The options for selection and text entry were appropriate. This allowed for clean data and clear structure for analysis. Open-ended commentary elicited further information from participants. One notable strength in the survey was that it allowed for participants to input other tasks that weren’t accounted for in the original list. This allowed the researchers to catch common systematic review steps, like developing a protocol and conducting appraisals, that they would have otherwise omitted.
Ultimately, this study introduces a new way of thinking and compartmentalizing of tasks, and it is clear that the vast majority of time spent by the librarian on systematic reviews is time spent searching. Training on how to search more efficiently may be one of the most impactful ways of reducing time spent on systematic reviews, allowing librarians to better involve themselves in systematic reviews while still performing other job duties.
Boynton, P. M., & Greenhalgh, T. (2004). Hands-on guide to questionnaire research: Selecting, designing, and developing your questionnaire. BMJ: British Medical Journal, 328(7451), 1312-1315.
Dudden, R. F., & Protzko, S. L. (2011). The systematic review team: Contributions of the health sciences librarian. Medical Reference Services Quarterly, 30(3), 301-315.
Gann, L. B., & Pratt, G. F. (2013). Using library search service metrics to demonstrate library value and manage workload. Journal of the Medical Library Association, 101(3), 227-229.
Saleh, A. A., Ratajeski, M. A., & Bertolet, M. (2014). Grey literature searching for health sciences systematic reviews: A prospective study of time spent and resources utilized. Evidence Based Library and Information Practice, 9(3), 28-50.
Spencer, A. J., & Eldredge, J. D. (2018). Roles for librarians in systematic reviews: a scoping review. Journal of the Medical Library Association, 106(1), 46-56.