EBLIParticlelogo   Evidence Based Library and Information Practice




Evidence Summary


PubMed’s Native Interface Remains the Best Tool for Systematic Searching of its Biomedical Citations


A Review of:

Wildgaard, L. E., & Lund, H. (2016). Advancing PubMed? A comparison of third-party PubMed/Medline tools.  Library Hi Tech, 34 (4), 669-684. http://dx.doi.org/doi: 10.1108/LHT-06-2016-0066


Reviewed by:

Ann Glusker

Research & Data Coordinator

National Network of Libraries of Medicine, Pacific Northwest Region

University of Washington Health Sciences Library

Seattle, Washington, United States of America

Email: glusker@uw.edu


Received: 1 Mar. 2017                                                                   Accepted: 17 Apr. 2017



cc-ca_logo_xl 2017 Glusker. This is an Open Access article distributed under the terms of the Creative CommonsAttributionNoncommercialShare Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.




Objective To compare the functionality of third-party PubMed tools for searching biomedical citations in PubMed, in the specific context of systematic searching.


Design Comparative analysis of software functionality.


Setting Online, freely accessible search software.


Subjects – Sixteen third-party tools for searching and managing the full range of PubMed citations (tools which focused on specific disciplines were not included).


Methods – Tools for analysis were identified in two ways; those discussed in two published articles were used, and a supplementary PubMed search was performed. The initial list of 76 possibilities was assessed for study inclusion on 4 criteria: covering the entire range of PubMed content; being freely available; not limiting to a particular bio-medical discipline; and incorporating online PubMed/MEDLINE content. After assessment, 16 tools were chosen for further analysis (the authors provide a list and description of the tools in their Table I). Each was examined in relation to 11 crucial operational aspects. Result sets were tested against a control (a literature search result set on a particular clinical question which was determined by physicians to yield relevant results, details of which are provided by the authors in an online appendix). 


Main Results – The 11 identified aspects related to tool functionality were examined for each tool selected, with results grouped into three sets of factors: 1) supporting the search (field codes, filters, limits and Boolean operators); 2) managing the search (output, related articles, links to articles, number of results, exporting); and 3) documenting the search (saving the search and search history). In some cases, the tests had to be adjusted to accommodate the tool's specifications. In Table II the authors present a grid with the results of the testing, on each of the 11 aspects, for each tool.


The authors found that with many tools it was not straightforward, if even possible, to filter and limit in order to get more specific result sets. Few tools were effective at suggesting related articles within the tool itself, instead linking the user out to PubMed, and only two tools provided the same number of citation results as the comparison PubMed search. In addition, the display of results often made it difficult to assess result sets; and only two tools provided the option to save searches and see search history. Furthermore, due to unexpected tool limitations, it was not possible to assess the relevance of citation result sets delivered by the third-party tools, as compared with the control PubMed search.


Conclusion – Close analysis of the tools studied indicated that they were not created in order to support systematic searches. They lack support for filtering/limiting, saving or exporting searches, which are central functionalities to the work of performing such searches. While some of the tools studied may still be in the early phases of development, and while several of them, in enhancing PubMed searches in particular ways, may suggest additional profitable strategies for performing a systematic search, not one of them can replace the functionalities of the native PubMed interface. It remains the best tool for searching and managing the full range of PubMed citations, for the purposes of performing systematic searches.



This study was an addition to existing literature – specifically articles the authors consulted by Lu (2011) and Keepanasseril (2014), which merely listed and described tools – in that the authors analyzed the functionality of tools using a detailed set of criteria and a validated search as a control. While it turned out that the third-party tools examined are not suited to use for systematic searching, they may be useful for other search enhancements. The authors state that these tools “are beneficial as they give immediate, dynamic visual assessment of relationships between authors, topics and term hierarchies, etc. in the bibliographic data, giving a strong starting point in evaluating and selecting literature to include in a systematic search” (p. 679). For example, in 2010 Kristine Ogden outlined several aspects of the tools HubMed and Quertle which helped her in clinical searches: a citation finder which pulls PubMed records for citations in a bibliography; the ability to run a PubMed search automatically on other sites such as GoogleScholar; the use of natural language to find relevant citations; and separate tabs for keyword search results and citations. She also appreciated the clean user interfaces of these tools.


Furthermore, this paper makes an important contribution toward supporting medical librarians and others who work with systematic reviewers, in showing the crucial importance of the systematic searching that underlies such reviews. It also gives librarians a framework for helping systematic reviewers assess third-party tools to help with those searches.


For this evidence summary, methodologies were systematically assessed using Glynn’s critical appraisal checklist (2006). The checklist was designed to evaluate population-based studies, and so some of its criteria did not apply to this study, but it does focus on freedom from bias and representativeness of the subjects studied. In this instance, one question is whether there are more effective third-party tools for systematic searching which may not be freely available, but rather exist behind pay walls – could including them have changed the results? Also, there may be excellent tools that were designed for specific clinical areas which were excluded from this study. In Table III the authors list excluded third-party tools, which could prove a resource for future analysis.  


Furthermore, in addition to listing closed projects and dead links, Table IV lists potentially relevant third-party tools and sites under construction. The authors are not denigrating the third-party tools they tested (in fact, they mentioned wanting to re-test them, and entries in Table IV may be a starting place). Future methodological advances may contribute to the creation of systematic PubMed search tools. As described in Gonzalez et al. “computational methods contribute…by bringing knowledge from literature, either extracted or curated, together with high-throughput data sets, to identify both known and new relationships” (2016, p. 39). While the context for such computational methods relates to text and data mining, there is every reason to expect that they may eventually contribute to systematic analysis of PubMed citations such as Wildgaard and Lund seek.




Glynn, L. (2006). A critical appraisal tool for library and information research. Library Hi Tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154


Gonzalez, G. H., Tahsin, T., Goodale, B. C., Greene, A. C., & Greene, C. S. (2016). Recent advances and emerging applications in text and data mining for biomedical discovery. Briefings in Bioinformatics, 17(1), 33-42. https://dx.doi.org/10.1093/bib/bbv087


Keepanasseril, A. (2014). PubMed alternatives to search MEDLINE: An environmental scan. Indian Journal of Dental Research, 25(4), 527o. http://dx.doi.org/doi:10.4103/0970-9290.142562


Lu, Z. (2011). PubMed and beyond: A survey of web tools for searching biomedical literature. Database: The Journal of Biological Databases and Curation, baq036. http://dx.doi.org/doi: 10.1093/database/baq036


Ogden, K. (2010, March 16). Why use a third-party tool to interact with PubMed? Retrieved from https://nnlm.gov/pnr/dragonfly/2010/03/16/hubmed_quertle




Evidence Based Library and Information Practice (EBLIP) | EBLIP on Twitter