Research Article

 

Informing Evidence Based Decisions: Usage Statistics for Online Journal Databases

 

Alexei Botchkarev

Senior Information Management Advisor, Ministry of Health and Long-Term Care

Adjunct Professor, Ryerson University

Toronto, Ontario, Canada

Email: alexei.botchkarev@ontario.ca

 

Received: 9 July 2016      Accepted: 20 Feb. 2017  

 

 

cc-ca_logo_xl 2017 Botchkarev. This is an Open Access article distributed under the terms of the Creative CommonsAttributionNoncommercialShare Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

 

Abstract

 

Objective – The primary objective was to examine online journal database usage statistics for a provincial ministry of health in the context of evidence based decision-making. In addition, the study highlights implementation of the Journal Access Centre (JAC) that is housed and powered by the Ontario Ministry of Health and Long-Term Care (MOHLTC) to inform health systems policy-making.

 

Methods – This was a prospective case study using descriptive analysis of the JAC usage statistics of journal articles from January 2009 to September 2013.

 

Results JAC enables ministry employees to access approximately 12,000 journals with full-text articles. JAC usage statistics for the 2011-2012 calendar years demonstrate a steady level of activity in terms of searches, with monthly averages of 5,129. In 2009-2013, a total of 4,759 journal titles were accessed including 1,675 journals with full-text. Usage statistics demonstrate that the actual consumption was over 12,790 full-text downloaded articles or approximately 2,700 articles annually.

 

Conclusion – JAC’s steady level of activities, revealed by the study, reflects continuous demand for JAC services and products. It testifies that access to online journal databases has become part of routine government knowledge management processes. MOHLTC’s broad area of responsibilities with dynamically changing priorities translates into the diverse information needs of its employees and a large set of required journals. Usage statistics indicate that MOHLTC information needs cannot be mapped to a reasonably compact set of “core” journals with a subsequent subscription to those.

 

Introduction

 

Use of research results in medical evidence based decision-making and healthcare evidence-informed policy advice has been recognized as essential to improve health outcomes (Field, Gauld & Lawrence, 2012; Lomas & Brown, 2009; Sosnowy, Weiss, Maylahn, Pirani & Katagiri, 2013). Despite agreement on the importance of the issue and general consensus on the approaches, implementation of evidence based decision-making processes leaves much room for improvement. Several barriers have been identified by researchers, including: lack of access and limited awareness of research results (Wallace, Nwosu and Clarke, 2012); lack of practical use of systematic reviews (Wallace et al., 2012); lack of organizational culture or supports (e.g. behaviour of supervisors, front-line staff and other professionals in the organization) (Rapp et al., 2010); lack of time (Solomons & Spross, 2011); ambiguous and conflicting research (Madhavji, Araujo, Kim & Buschang, 2011; Ubbink et al., 2011) or research having methodological inadequacies (O'Connor & Pettigrew, 2009); lack of skills, training or tools to acquire, assess, synthesize, disseminate and apply research evidence to inform policy related to health systems (Ubbink, Guyatt & Vermeulen, 2013); lack of applicability/relevance of research (Humphries, Stafinski, Mumtaz & Menon, 2014); lack of standard knowledge translation strategies and processes effective in multiple contexts (Humphries et al., 2014); lack of timely research outputs (Oliver, Innvar, Lorenc, Woodman & Thomas, 2014; van der Arend, 2014); and lack of interaction and collaboration between researchers and policymakers (Oliver et al., 2014; Wooding, Hanney, Pollitt, Grant & Buxton, 2014).

 

A significant challenge for health system practitioners (both in a clinical setting and in public service) in implementing research evidence is inadequate access to information, which results in doctors or analysts being unaware of the research (Brownson et al., 2014; Oliver et al., 2014; Ubbink et al., 2013; Ubbink et al., 2011; Wallace et al., 2012). Various types of information are required for producing high-quality evidence based policy advice, including journals, books, research reports, professional/trade magazines, etc. Academic journals and professional magazines are the largest component of the potentially applicable information. Arguably, almost all new research is published in journals. That makes access to journals a key pre-requisite for evidence based policy advice. 

 

In Canada, there are thirteen provinces and territories responsible for implementing evidence based health policies and services for the benefit of their populations. The Ministry of Health and Long-Term Care (MOHLTC) of Ontario, is one of the provincial ministries.

 

MOHLTC “is working to establish a patient-focused, results-driven, integrated and sustainable publicly funded health system” (MOHLTC, 2017a). MOHLTC’s mandate includes (but is not limited to): “establishing overall strategic direction and provincial priorities for the health system; developing legislation, regulations, standards, policies, and directives to support those strategic directions; monitoring and reporting on the performance of the health system and the health of Ontarians; planning for and establishing funding models and levels of funding for the health care system;” etc. (MOHLTC, 2017a). From an organizational point of view, MOHLTC is a large and complex public corporation (MOHLTC, 2017b). The ministry has several locations in Toronto, Ontario and local units in major provincial cities. Among the types of documentation produced by the ministry are: draft legislations and regulations, briefing notes, reports, program reviews and evaluations, recommendations, guidelines, etc. (MOHLTC, 2015).

 

The ministry, within the framework of the government of Ontario, makes decisions or provides advice on a wide spectrum of issues by setting agendas, implementing and evaluating solutions and promoting improvements in the health system. The ministry oversees a health budget of more than C$50 billion per year. The following may serve as examples of types of evidence based decisions made by the ministry on an on-going basis: budget-related planning and control; allocation of public resources to the local health integrated networks; performance management of the health system and its elements; economic evaluation and health technology assessment of new medical interventions, drugs and systems; health system capacity planning, etc. Specific examples of the ministry’s evidence based decision-making can be found in Lomas and Brown (2009) and Khan et al. (2014). In 2015, the government of Ontario established a Centre of Excellence for Evidence-Based Decision Making to enhance the capability of all ministries in making informed decisions.

 

MOHLTC takes specific measures to encourage evidence based policy-making to improve the healthcare system. These measures include, in particular, development and implementation of the policies and procedures of using research evidence, providing financial support to universities in generating new evidence and conducting knowledge transfer, and building and operating information systems to facilitate access to online journals, e.g. the Journal Access Centre (JAC).

 

JAC is made available to all MOHLTC employees – several thousand people. Obviously, actual use varies across the departments. For example, research, analytics, policy development, program evaluation and economic assessment units are known to be regular users of JAC. The departments with primarily operational functions, such as financial management, corporate supply chain, and information technology, use JAC with less regularity. Informal surveys of staff conducted in 2011-2012 show that several hundred employees use JAC weekly or daily and consider their JAC skills as advanced.   

 

Prior to JAC, access to journals was organized through subscriptions by individual ministry departments. Subscriptions were not coordinated. Overall, employees had access to several dozen printed journals. 

 

Objectives

 

The primary objective of the study was to examine online journal databases usage statistics for a provincial ministry of health in the context of evidence based decision-making. In addition, the study highlights implementation of the Journal Access Centre that is housed and powered by the Ontario Ministry of Health and Long-term Care to inform health systems’ policy making.

 

The research questions guiding this study were:

 

  • What are the usage statistics of searches that reflect user demand for online journals?
  • What are the usage statistics of downloaded full-text articles that characterize the desired output of the journal access solution and reflect consumption of information?
  • What are the usage statistics of the journal titles accessed by JAC users?
  • Can information needs of MOHLTC users be satisfied by subscription to a core set of “most important” journals?

 

Intent

 

The study has been undertaken with intent to use the results and recommendations of the research to:

 

  • inform ministry program evaluation and performance management processes;
  • provide input into economic cost-effectiveness models to optimize JAC journal acquisition;
  • inform ministry senior management decisions on JAC funding.

 

Management decisions that were taken based on the recommendations of this study are not discussed.

 

Out of scope

 

This study is the first step in assessing JAC’s outputs. To assess value to the users and organization, the outcomes of higher-level processes of analyzing retrieved information and making evidence based decisions need to be examined. Research questions at these phases would include: What are users doing with the articles they got from JAC?; Are retrieved articles pertinent and of high quality?; How do users analyze information?; How do they integrate evidence?; What is the process of making evidence based decisions in the organization? etc. All these research questions, although related to the topic of this study, are out of scope of this research.

 

The study examines usage statistics at the institutional level. Analyses at lower levels, e.g. departmental or individual, are out of scope. Similarly, usage statistics are examined at the journal title level; article-level considerations are out of scope.

 

Methods

 

This study had a prospective case study design with descriptive analysis of JAC usage statistics for two datasets. Dataset 1 contains JAC user statistics on the numbers of searches and downloaded full-text articles for the period from January 1st 2011 to December 31st 2012 (i.e. two full years of historical data). The following considerations were taken into account while selecting the period of historical data. Firstly, trends in current research practice. Similar or shorter periods of data collection to analyze usage patterns are quite common in academic studies, e.g.: a recent CIBER research report found that two years of journal usage data is sufficient to provide insight into a journal’s usage patterns (CIBER Research Limited, 2011), two-year periods were used in the studies of Wical and Vandenbark (2015), and Chew, Stemper, Lilyard and Schoenborn (2013). Secondly, availability of data. A reasonable effort has been made to collect all pertinent available COUNTER-compliant data (COUNTER, n.d.). Thirdly, alignment of the data collection period with the objective of the study and research questions. This was the first study to examine online journal access for a Canadian ministry of health. The focus was primarily on capturing and interpreting the current state of usage – not on analyzing changes in trends (as there was no prior knowledge), which would have required multiple years of observation periods. Analysis of the usage data allowed the objective of the study to be achieved.

 

It has been acknowledged that information needs and hence usage patterns of the online resources at a ministry of health are different from those in public health organizations (e.g. Barr-Walker, 2017; Ford & Korjonen, 2012), health care providers (e.g., Younger, 2010) or of academic researchers (e.g., Haglund & Olsson, 2008; Niu et al. 2010). Information needs of the policy makers are commonly characterized by: firstly, a broad spectrum of the subject field which spreads beyond health care and includes economics, education, housing, etc. (e.g., Brownson, Royer, Ewing & McBride, 2006); secondly, a wide range of information sources which are not limited to academic journals and include grey literature and professional magazines with inputs on politics, values and opinions (e.g., Ritter 2009); thirdly, a preference towards simple, uncomplicated information, for example evidence summaries (e.g., Petkovic et al., 2016; Ritter 2009; Tricco et al., 2016); and finally, a dynamically changing environment and necessity to make rapid decisions which require availability of tools with timely access to information (e.g., Oliver et al., 2014), making systems like JAC indispensable.   

 

A descriptive analysis of the usage of journal articles was conducted through the JAC access tool from January 2009 to September 2013 (Dataset 2). Journal usage statistics for MOHLTC users were downloaded from the EBSCOhost administrative reporting site (EBSCOhost, n.d.). Journal usage is characterized by the following indicators: i) number of searches, ii) number of full-text articles accessed in PDF or HTML format, iii) number of abstracts accessed, and iv) number of turnaways (access denied). These indicators were selected based on the recommendations of the internationally recognized standard: COUNTER-2008, Counting Online Usage of NeTworked Electronic Resources (COUNTER, n.d.). Definitions of the indicators and related terms are available from COUNTER (n.d.). Numbers of searches characterize overall intensity of use of the JAC and demand for this service. Number of full-text articles characterizes the desired output of the solution and can be linked to the consumption of information provided by the service.

 

It should be noted that at the time of data collection, EBSCO adhered to COUNTER Release 3. Since then, the COUNTER project has published an updated version of its Code of Practice – Release 4. Certain terminology and types of usage reports have been changed. There is no direct correspondence between all types of reports in Release 3 and Release 4. With some approximation, we can indicate that the data for performance measures used in the paper can be found in the following current COUNTER reports: the number of searches – in the Database Report 1 (DB1); number of abstracts – in the Record Views of DB1; number of full text downloads – in the Journal Report 1 (JR1); and number of turnaways (access denied) – in Database Report 2 (DR2).

 

Journal Access Centre Implementation

 

To support evidence based decisions, MOHLTC of Ontario, Canada, built the Journal Access Centre (JAC). JAC - an online access tool supported by journal content selection, acquisition and consultation services – has been in operation with the MOHLTC since 2008, making the ministry one of the Canadian healthcare pioneers of online access to academic journals. It was conceived and developed to facilitate online access to journals and serve as an enabling factor for enhanced evidence-informed policymaking. JAC’s logic model is presented in Figure 1.

 

 

Figure 1

JAC logic model.

 

 

A systems view of JAC is presented in Figure 2. MOHLTC acquire access to journals based on the annual subscriptions procured from various vendors including journal aggregators and individual publishers. The content is discussed and suggested for acquisition by the JAC Content Selection Advisory Network – a permanent working group with representation from each of the ministry’s divisions.

 

Mostly, access to content is acquired by journal databases, which represent collections of journal titles focused on a certain subject area. Examples could be such well-known databases as MEDLINE, CINAHL from EBSCO (EBSCO Information Services, n.d.) or Academic OneFile, Academic OneFile from GALE CENGAGE Learning (Gale Cengage Learning, n.d.). Commonly, each database contains from a few hundred to several thousand journals. Some journals and databases are acquired individually, e.g., The Cochrane Library, Longwoods, etc. The total ministry subscription covers over 17,000 journal titles with over 9.0 million articles. These numbers include archives of prior years. Journals cover such topics as health, medicine, social science, business, policy, economics, finance, management, risk management, etc.

 

Usually, a database contains journals with different levels of access. Some journals are provided with complete full-text article coverage, others are available only at an abstract or indexing level. Some latest full-text journals have embargoes – delays in access from six months to three years. Most databases contain a mix of academic journals and professional non-peer-reviewed magazines. The types of content of several databases are illustrated in Table 1. The prime purpose of JAC is to provide access to the full-text articles because abstracts and bibliographic data for most journals are available on the Internet free of charge. Hence, the most valuable segment of a database constitutes full-text, current, non-embargoed journals. Despite the large number of journals and articles in the JAC repositories, occasionally a need arises for an article that is not available in full-text. In these cases, the Article on Demand Service manned by JAC’s support staff orders materials and sends them to the JAC users. 

 

The technological backbone of the solution is a cloud-based application, search engine, which allows MOHLTC users to access journal databases offered by EBSCO through the ministry’s intranet. The service is provided through the EBSCO Integrated Search function. End users need only a web browser to access online journals. The EBSCO search engine provides integrated coverage of the databases both owned by EBSCO and bought from different providers – so end users can conduct a one-click search through all subscribed content. In addition, the EBSCO search is integrated with Google Scholar search, i.e. when a ministry user is conducting a search in Google Scholar he/she gets reminders if an article, presented in Google Scholar search results, is available in the ministry’s EBSCO subscriptions, and can click on the link to be immediately transferred to a full-text article in EBSCO repositories. The service also provides automatic e-mail notifications of new content at an article level, which may be very specific to meet individual information needs. The service is available 24x7 with short periods of maintenance scheduled during weekends. The service has proved to be highly reliable: just one three-hour incident of service disruption has been observed in more than four years.

 

 

Figure 2

JAC structure.

 

 

Table 1

Types of Database Contenta

Database

MEDLINE
Complete

CINAHL
Complete

Business Source
Complete

Health Business
Elite

Health Policy Reference Center

OmniFile
Full Text

AgeLine

TOTAL

Total Number of Journals in the Database

2,184

5,453

5,023

714

472

3,125

209

17,180

Abstracts Only

0

3,825

1,191

64

37

0

209

5,326

Full-Text

2,184

1,628

3,832

650

435

3,125

0

11,854

Stopped Full-Text

357

537

1,139

414

149

982

0

3,578

Current Full-Text

1,827

1,091

2,693

236

286

2,143

0

8,276

Embargoed Current Full-Text

1,183

277

638

67

81

274

0

2,520

Non-Embargoed Current Full-Text

644

814

2,055

169

205

1,869

0

5,756

Non-Embargoed Current Full-Text

Peer-Reviewed

608

587

943

140

143

1,184

0

3,605

Non-Embargoed Current Full-Text

Magazines, Trade Publications

36

227

1,112

29

62

685

0

2,151

aSome databases have overlapping content (journal titles). The number of unique journal titles in the system is less than shown in the table. 

 

 

Figure 3

Number of searches (monthly average for 2011-2012).

 

 

Figure 4

Number of downloaded full-text articles (quarterly in 2011and 2012).

 

 

Usage Statistics Results

 

This section presents the results for the number of searches from a report covering two full calendar years 2011-2012. Figures 3 and 4 show monthly average number of searches, and quarterly number of downloaded full-text articles respectively. Reported number of turnaways (access denied) is zero.

 

A report on the number of full-text articles and abstracts accessed by journal title for approximately 5000 journals is presented in Additional file (Appendix). This report covers the period from January 1, 2009 through September 30, 2013. A list of the most frequently used journals (top-50) is presented in Table 2.

 

 

Table 2

List of the Most Frequently Used Journalsa

ISSN

Title

Full Text

Abstract

Category

00178012

Harvard Business Review

2208

734

NPR

00900036

American Journal of Public Health

303

129

PR

00284793

New England Journal of Medicine

270

189

PR, E3

08203946

Canadian Medical Association Journal

183

177

PR

00084263

Canadian Journal of Public Health

129

113

PR

15445208

Health Affairs

123

139

PR

00028614

Journal of the American Geriatrics Society

118

74

PR, E12

03092402

Journal of Advanced Nursing

118

40

PR, E12

01628968

Inc.

114

5

NPR

13558196

Journal of Health Services Research and Policy

112

64

PR, E12

1357633X

Journal of Telemedicine and Telecare

94

9

PR

07067437

Canadian Journal of Psychiatry

89

59

PR

01607480

Modern Healthcare

86

47

NPR

09269630

Studies in Health Technology and Informatics

69

49

PR, E12

10688838

HandHN: Hospitals and Health Networks

69

25

NPR

00034819

Annals of Internal Medicine

68

44

PR

11707690

PharmacoEconomics

67

31

PR, E6

01406736

Lancet

66

154

PR

0887378X

Milbank Quarterly

64

40

PR, E12

17561833

BMJ: British Medical Journal

63

115

PR, E3

10966218

Journal of Palliative Medicine

62

29

PR, E12

14726963

BMC Health Services Research

58

50

PR

13652702

Journal of Clinical Nursing

55

23

PR, E12

13869620

Health Care Management Science

54

33

PR, E12

09660410

Health & Social Care in the Community

53

21

PR, E12

13561294

Journal of Evaluation in Clinical Practice

53

19

PR, E12

 

 

01958631

Health Care Financing Review

53

15

PR

07350732

Healthcare Financial Management

51

40

PR

09652140

Addiction

51

13

PR, E12

00197939

Industrial and Labor Relations Review

50

21

PR

14712458

BMC Public Health

48

29

PR

10792082

American Journal of Health-System Pharmacy

48

18

PR

1477030X

Palliative Medicine

48

14

PR

07461739

Nursing Economics

46

20

PR

00333107

Psychology Today

46

8

NPR

03190781

Toronto Star (Toronto, Ontario)

43

32

NPR

00130613

Economist

42

24

NPR

1095158X

Psychiatric Rehabilitation Journal

42

24

PR, E12

87569728

Project Management Journal

41

39

PR, E12

03616878

Journal of Health Politics, Policy and Law

40

40

PR, E12

14756773

Health Services Research

40

34

PR

00413674

Trustee

40

31

NPR

00296570

Nursing Standard

40

25

PR

10903127

Prehospital Emergency Care

38

16

PR, E18

10786767

Journal of Health Care Finance

37

19

PR

08982759

Physician Executive

37

14

PR

00048674

Australian and New Zealand Journal of Psychiatry

36

16

PR

08835381

Healthcare executive

35

24

NPR

15414469

International Journal of Health Services

34

44

PR, E6

08949867

Journal of Traumatic Stress (Wiley)

34

23

PR, E12

aAbbreviations in Table 2: PR – peer-reviewed, NPR – non-peer-reviewed, E – embargoed with indication of the delay in months.

 

 

Figure 5

Distribution of the number of journals falling into journal groups with similar numbers of full-text downloads.

 

 

Based on the usage statistics, a histogram was developed. Figure 5 shows distribution of the number of journals falling into journal groups with similar numbers of full-text downloads. The horizontal axis is non-linear for better visual presentation.

 

Discussion

 

Figures 3 and 4 are interrelated and are built in a COUNTER-compliant way (searches, and downloads).

 

Figure 3 shows the number of searches per month. During 2011-2012, there were over 123,000 searches with monthly averages of 5,129. Smaller increments of time have been used to reveal usage patterns on a relatively short data period of Dataset 1. It should be noted that this was the first effort to collect data and analyze JAC usage. There was no prior information about the usage: whether the system had been used only from time to time (e.g. to support decision-making for an important health care policy or regulation) or whether it had been used at all.

 

It should be acknowledged that the charts are not exposing unexpected trends. Figures 3 and 4 can be interpreted along the following lines. The data provides evidence that December is a slowdown period in JAC use (minimum number of searches). This is quite understandable due to a number of statutory holidays and traditional vacation time. In January and February users increase JAC activity (usage numbers show a steep rise) to make up for a slow year-end and to retrieve information that they need in planning for the next fiscal period (April 1 is the starting date for the government of Ontario fiscal year). Usage during all other months of the year does not display large variations. JAC usage demonstrates a steady level of activity most of the year, i.e. it has become part of routine government knowledge-management processes.

 

The number of full-text articles, either downloaded in PDF format or viewed on the computer screen in HTML, characterizes the desired output of the system. Figure 4 demonstrates the actual consumption of information in JAC databases: in 2011-2012, over 4,600 full-text articles were accessed. During the same period of time more than 5,800 abstracts were accessed. This indicator is secondary, keeping in mind that the main purpose of JAC is to provide access to full-text articles. However, the fact that the user accessed the abstract could be seen as an evidence of interest that the user had in the article but full-text may not have been available. If a journal, not available in full-text, has experienced multiple abstract viewing, it testifies that this journal should be considered for subscription in full-text version.

 

It was noted that some journals were accessed at abstract level extensively, but had zero full-text downloads. That may demonstrate that these journals publish pertinent articles but are not available with full-text. The ten journals that were most frequently accessed in this way (from 117 to 30 times) were: JAMA – Journal of the American Medical Association, Healthcare Quarterly, Medical Care, Healthcare Papers, Health Policy, Vaccine, Journal of Palliative Care, Diabetes Educator. MOHLTC may consider exploring subscription to these journals with full-text taking into account cost-efficiency (Botchkarev, 2013).

 

Usage statistics as numbers of full-text articles and abstracts of individual journals for the period from 2009 to 2013 are presented in Additional file (Appendix). Usage statistics for the top-50 most frequently used journals are shown in Table 2. This data shows that MOHLTC users accessed 12,790 full-text articles and 14,517 abstracts. A total of 4,759 journal titles were accessed including 1,675 journals with full-text. The Harvard Business Review is by far the most frequently used journal – it was used seven times more than the second-ranked journal: the American Journal of Public Health.

 

Table 2 demonstrates that the most frequently used journals include both peer-reviewed (82%) and non-peer-reviewed (18%) periodicals. Approximately 50% of the most frequently used peer-reviewed journals have embargoes from 3 to 18 months, predominantly 12 months.

 

The histogram presented in Figure 5 indicates spread of usage across multiple journal titles. Usage does not demonstrate a “core” set of journals There are only 10 journals with over 100 downloads each. The total number of downloads from these journals is 3,678. Similarly, there are only 30 journals that were accessed in full-text format 50 times or more. These journals contributed only 38% (4,953) of the accessed articles. Of the accessed journals, 54% (908 titles) were accessed with full-text only once or twice. This group contributed 1,192 articles (9%). This pattern of usage can be attributed to the following factors. First, there is an increasing amount of research being conducted that triggers a persistently growing number of publication outlets. Second, MOHLTC has a very broad area of responsibilities with dynamically changing priorities, which translates into the diverse information needs of its employees. A practical conclusion from the usage analysis is that MOHLTC information needs cannot be mapped to a reasonably compact set of “core” journals with a subsequent subscription to those. In this case, subscription economics necessitate the use of journal aggregators, e.g. EBSCO, Gale, etc., as the main source of journal access acquisition (Botchkarev, 2013).

 

Certain JAC user feedback is notable. This information has been collected in non-structured conversations with clients and is not supported by quantitative assessments. Despite availability and actual use of thousands of journals, there is a need to (i) expand access to more peer-reviewed journals, (ii) expand access to more journals with full-text articles, (iii) explore opportunities to reduce embargoes. Some users expect immediate online access to the full-text articles of interest, requesting no abstract-only, no delays/embargoes content. If these expectations are not met, user satisfaction might decline rapidly.

 

The focus of this empirical study is on usage statistics in terms of number of downloads, etc., which represent the outputs of the JAC solution. There is no implication or judgement about the value that JAC collections provide to users. This study is the first step in assessing value to the users and organization. It needs to be emphasized that there is no rationale to judge the value of JAC collections through the rate of downloads per user (high or low – good or bad) because there is nothing to compare to either in MOHLTC or in other organizations (e.g. commonly accepted benchmarks do not exist). To assess value, we would need to consider the outcomes of higher-level processes.

 

The findings reported in this paper can be used in various government organizations to implement journal subscriptions strategies that will better meet ministry employees’ information needs and contribute to cost-efficiency of operations. 

 

Study Assumptions and Limitations

 

To the best of our knowledge, this is the first study to address usage statistics for online journal databases in a Canadian ministry of health. However, this study has certain limitations that should be made explicit.

 

Use of JAC implies that its collections are of interest for the MOHLTC employees. However, numbers of article downloads may not be equal to actual use or satisfaction – users may download an article and find it worthless for their task or they may be unhappy because they did not find specifics they needed.

 

JAC statistics used in this study have been downloaded from the EBSCO reporting site. This data is based on the automatic logs and is believed to be very accurate. However, the following should be noted.

 

Firstly, JAC is not an exclusive channel of information for MOHLTC users. Some users have access to online journals at local universities through their alumni connections. Others have access to journal repositories based on their memberships in professional associations. Certain departments used to have subscriptions to publications in their specific narrow fields. As a result, documented in the study number of downloads represents a lower border of actual downloads.

 

Secondly, JAC is using EBSCO integrated search services that allow access to the databases that are owned by EBSCO, and those of the third parties external to EBSCO. Search results presented to the user include both internal and external documents. When a person clicks on the link to an external database, he/she is transferred to the document in the external database. As soon as a person moves to an external database, EBSCO usually does not have information about what is happening there, and so cannot include activity in the report. That pertains especially to full-text documents. As a result, JAC statistics may be missing data on the use of full-text documents in external databases. An example of this situation could be statistics on the use of the Cochrane database shown in the Additional file (Appendix). JAC has a direct subscription to the Cochrane database with full-text documents, which makes it external to EBSCO. The usage statistics indicate zero downloads of Cochrane full-text documents. However, this is misleading as it has been verified through conversations with JAC users that the Cochrane database actually has been used.

 

Thirdly, in some cases, EBSCO usage reports are not perfect. There were a certain number of duplications of journal titles in the initial version of Additional file (Appendix). For example, (i) some titles were duplicated because of using different online and print ISSNs (e.g., Academic Emergency Medicine: Official Journal Of The Society For Academic Emergency Medicine: Usage was attributed to one title. Another was deleted); (ii) some titles were duplicated because in one case the title had an ISSN and in another the ISSN was blank (e.g., The Academy of Management Executive: Usage was attributed to one title. Another was deleted); (iii) some titles were duplicated because of spelling mistakes (e.g., American Journal of PublicHealth vs American Journal of Public Health); (iv) different title abbreviations were used – most likely in different databases – (e.g., BMC HEALTH SERVICES RESEARCH vs BMC Health Serv Res); (v) Use of & instead of AND; (vi) using titles with or without the definite article; and (vi) just typos.

 

Fourthly, it was shown in annual customer surveys (not reported in the current paper), that JAC users were not satisfied when search results contained a large number of articles with abstracts only. It took additional time to look through several pages of search results to find articles with full text. In March 2012, according to the recommendation of the JAC Content Selection Advisory Network, the JAC default search was configured to present full-text articles only. If a researcher was willing to analyze additional abstracts-only articles, he/she could adjust the search configuration. That may have decreased the number of retrieved abstracts from the second half of 2012.

 

Conclusions

 

This is the first paper, to our knowledge, to examine usage statistics for online journal databases in a ministry of health.

 

The Ministry of Health and Long-Term Care built and has maintained the Journal Access Centre – an online access tool supported by journal content selection, acquisition and consultation services. As a key prerequisite for evidence based policy-making, JAC enables access to approximately 12,000 journals with full-text articles.

 

JAC usage statistics for the 2011 – 2012 calendar years demonstrate a steady level of activities in terms of searches with a monthly average of 5,129. JAC’s steady level of activities, revealed by the study, reflects a continuous demand for the JAC services and products. It testifies that access to online journal databases has become part of routine government knowledge-management processes.

 

The number of downloaded full-text articles characterizes the desired output of the solution. JAC usage statistics demonstrate that the actual consumption in 2009-2013 was over 12,790 full-text articles or approximately 2,700 articles annually.

 

Usage statistics helped to identify a list of journals that were accessed at abstract level extensively (117 to 30 times), but had zero full-text downloads. That led to a practical recommendation to consider subscription to these journals with full-text, taking into account cost-efficiency.

 

JAC usage statistics for the period 2009-2013 provide evidence that a total of 4,759 journal titles were accessed including 1,675 journals with full-text. MOHLTC’s broad area of responsibilities with dynamically changing priorities translates into the diverse information needs of its employees and a large set of required journals. Usage statistics indicate that MOHLTC information needs cannot be mapped to a reasonably compact set of “core” journals with a subsequent subscription to those. In this case, subscription economics necessitate the use of journal aggregators (e.g. EBSCO, Gale, etc.) as the main source of journal access acquisition.

 

Future efforts could be focused on studying: (i) usage statistics complemented with data beyond EBSCO reports and covering all sources of online academic journals available in JAC; (ii) usage statistics by individual EBSCO databases, e.g. MEDLINE, CINAHL, etc., and category of journals, e.g. peer-reviewed, non-peer-reviewed, embargoed, etc.; iii) in-depth usage of information sources and patterns of behaviour at the level of individual article as compared to the journal title level in this paper; (iv) JAC users’ information needs and preferences; (v) JAC’s usability through the customer satisfaction survey; and (vi) development and examination of a model of a value creation chain which would integrate JAC outputs with outcomes of higher-level processes of analyzing retrieved information and making evidence based decisions.    

 

Acknowledgements

 

The views, opinions and conclusions expressed in this document are those of the author alone and do not necessarily represent the views of the Ontario Ministry of Health and Long-Term Care or any of its individual departments.

 

The author is grateful to Dr. Grégory Léon, Université Laval, Dr. Mathieu Ouimet, Université Laval and Dr. John N Lavis, McMaster University, for helpful comments on earlier drafts of the paper that had a distinct scope and objectives. However, mentioned scholars are not responsible for the content and conclusions of the published paper.   

 

References

 

Barr-Walker, J. (2017). Evidence-based information needs of public health workers: A systematized review. Journal of the Medical Library Association: JMLA, 105(1), 69-79. http://dx.doi.org/10.5195/jmla.2017.109

 

Botchkarev, A. (2013). Mining evidence in the online academic journal databases: Access and performance in public sector. E-prints in Library and Information Science (E-LIS). Retrieved from http://eprints.rclis.org/19286 

 

Brownson, R. C., Reis, R. S., Allen, P., Duggan, K., Fields, R., Stamatakis, K. A., & Erwin, P. C. (2014). Understanding administrative evidence-based practices. American Journal of Preventive Medicine, 46(1), 49-57. http://dx.doi.org/10.1016/j.amepre.2013.08.013

 

Brownson, R. C., Royer, C., Ewing, R., & McBride, T. D. (2006). Researchers and policymakers: Travelers in parallel universes. American Journal of Preventive Medicine, 30(2), 164-172. http://dx.doi.org/10.1016/j.amepre.2005.10.004

 

Chew, K., Stemper, J., Lilyard, C. and Schoenborn, M. (2013). User-defined valued metrics for electronic journals. In S. Hiller, M. Kyrillidou, A. Pappalardo, J. Self & A. Yeager (Eds.). Proceedings of the 2012 Library Assessment Conference, Charlottesville, VA, Oct 29-31. Retrieved from http://conservancy.umn.edu/bitstream/handle/11299/144461/1/UserDefinedValueBasedMetricsForElectronicJournalsPDF.pdf

 

CIBER Research Limited. (2011). The journal usage factor: Exploratory data analysis: Stage 2 Final Report, 27 May 2011. Retrieved from http://ciber-research.eu/download/20110527-The_Journal_Usage_Factor.pdf  

 

COUNTER (n.d.). Counting online usage of networked electronic resources. Retrieved from http://www.projectcounter.org 

 

EBSCO Information Services. (n.d.). Retrieved from https://www.ebsco.com 

 

EBSCOhost. (n.d.). Retrieved from http://eadmin.ebscohost.com/EAdmin/Reports/SelectReportsForm.aspx 

 

Field, P., Gauld, R., & Lawrence, M. (2012). Evidence-informed health policy - the crucial role of advocacy. International Journal of Clinical Practice, 66(4), 337-341. http://dx.doi.org/10.1111/j.1742-1241.2012.02883.x

 

Ford, J., & Korjonen, H. (2012). Information needs of public health practitioners: A review of the literature. Health Information & Libraries Journal, 29(4), 260-273. http://dx.doi.org/ 10.1111/hir.12001

 

Gale Cengage Learning. (n.d.). Retrieved from http://www.gale.cengage.com  

 

Haglund, L., & Olsson, P. (2008). The impact on university libraries of changes in information behavior among academic researchers: a multiple case study. The Journal of Academic Librarianship, 34(1), 52-59. http://dx.doi.org/10.1016/j.acalib.2007.11.010

 

Humphries, S., Stafinski, T., Mumtaz, Z., & Menon, D. (2014). Barriers and facilitators to evidence-use in program management: A systematic review of the literature. BMC Health Services Research, 14, 171. http://dx.doi.org/10.1186/1472-6963-14-171

 

Khan, S., Moore, J. E., Gomes, T., Camacho, X., Tran, J., McAuley, G., Juurlink, D.N., Paterson, M., Laupacis, A., & Mamdani, M. M. (2014). The Ontario drug policy research network: Bridging the gap between research and drug policy. Health policy, 117(3), 392-398. http://dx.doi.org/10.1016/j.healthpol.2014.06.007

 

Lomas, J., & Brown, A. D. (2009). Research and advice giving: A functional view of evidence-informed policy advice in a Canadian ministry of health. Milbank Quarterly, 87(4), 903-926. http://dx.doi.org/10.1111/j.1468-0009.2009.00583.x

 

Madhavji, A., Araujo, E. A., Kim, K. B., & Buschang, P. H. (2011). Attitudes, awareness, and barriers toward evidence-based practice in orthodontics. American Journal of Orthodontics and Dentofacial Orthopedics, 140(3), 309-316. http://dx.doi.org/:10.1016/j.ajodo.2010.05.023   

 

MOHLTC. (2015). Publications. Retrieved from http://www.health.gov.on.ca/en/public/publications/default.aspx  

 

MOHLTC. (2017a). About the Ministry. Retrieved from http://www.health.gov.on.ca/en/common/ministry/default.aspx  

 

MOHLTC. (2017b). Organization chart. Retrieved from http://www.health.gov.on.ca/en/common/ministry/orgchart.pdf 

 

Niu, X., Hemminger, B. M., Lown, C., Adams, S., Brown, C., Level, A., McLure, M., Powers, A., Tennant, M.R., & Cataldo, T. (2010). National study of information seeking behavior of academic researchers in the United States. Journal of the American Society for Information Science and Technology, 61(5), 869-890. http://dx.doi.org/ 10.1002/asi.21307

 

O'Connor, S., & Pettigrew, C.M. (2009). The barriers perceived to prevent the successful implementation of evidence-based practice by speech and language therapists. International Journal of Language & Communication Disorders, 44(6), 1018-1035. http://dx.doi.org/10.1080/13682820802585967

 

Oliver, K., Innvar, S., Lorenc, T., Woodman, J., & Thomas, J. (2014). A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research, 14, 2. http://dx.doi.org/10.1186/1472-6963-14-2

 

Petkovic, J., Welch, V., Jacob, M. H., Yoganathan, M., Ayala, A. P., Cunningham, H., & Tugwell, P. (2016). The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: A systematic review. Implementation Science, 11, 162. http://dx.doi.org/10.1186/s13012-016-0530-3

 

Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., Callaghan J., & Holter, M. (2010). Barriers to evidence-based practice implementation: Results of a qualitative study. Community Mental Health Journal, 46(2), 112-118. http://dx.doi.org/10.1007/s10597-009-9238-z

 

Ritter, A. (2009). How do drug policy makers access research evidence? International Journal of Drug Policy, 20(1), 70-75. http://dx.doi.org/10.1016/j.drugpo.2007.11.017

 

Solomons, N. M., & Spross, J. A. (2011). Evidence-based practice barriers and facilitators from a continuous quality improvement perspective: An integrative review. Journal of Nursing Management, 19(1), 109-120. http://dx.doi.org/10.1111/j.1365-2834.2010.01144.x

 

Sosnowy, C. D., Weiss, L. J., Maylahn, C. M., Pirani, S. J., & Katagiri, N. J. (2013). Factors affecting evidence-based decision making in local health departments. American Journal of Preventive Medicine, 45(6), 763-768. http://dx.doi.org/10.1016/j.amepre.2013.08.004

 

Tricco, A. C., Cardoso, R., Thomas, S. M., Motiwala, S., Sullivan, S., Kealey, M. R., Hemmelgarn, B., Ouimet, M., Hillmer, M.P., Perrier, L., Shepperd, S., & Straus, S.E. (2016). Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: A scoping review. Implementation Science, 11, 4. http://dx.doi.org/10.1186/s13012-016-0370-1

 

Ubbink, D. T., Guyatt, G. H., & Vermeulen, H. (2013). Framework of policy recommendations for implementation of evidence-based practice: A systematic scoping review. BMJ Open, 3(1). http://dx.doi.org/10.1136/bmjopen-2012-001881

 

Ubbink, D. T., Vermeulen, H., Knops, A. M., Legemate, D. A., Oude Rengerink, K., Heineman, M. J., Roos Y.B., Fijnvandraat C.J., Heymans H.S., Simons R., & Levi, M. (2011). Implementation of evidence-based practice: outside the box, throughout the hospital. Netherlands Journal of Medicine, 69(2), 87-94.

 

van der Arend, J. (2014). Bridging the research/policy gap: Policy officials' perspectives on the barriers and facilitators to effective links between academic and policy worlds. Policy Studies, 35(6), 611-630. http://dx.doi.org/doi:10.1080/01442872.2014.971731

 

Wallace, J., Nwosu, B., & Clarke, M. (2012). Barriers to the uptake of evidence from systematic reviews and meta-analyses: A systematic review of decision makers’ perceptions. BMJ Open, 2(5). http://dx.doi.org/10.1136/bmjopen-2012-001220

 

Wical, S.H. and Vandenbark, R.T. (2015). Notes on operations: Combining citation studies and usage statistics to build a stronger collection. Library Resources & Technical Services, 59(1), pp.33-42. Retrieved from https://journals.ala.org/lrts/article/view/2755/2733

 

Wooding, S., Hanney, S. R., Pollitt, A., Grant, J., & Buxton, M. J. (2014). Understanding factors associated with the translation of cardiovascular research: A multinational case study approach. Implementation Science, 9, 47. http://dx.doi.org/10.1186/1748-5908-9-47

 

Younger, P. (2010). Internet‐based information‐seeking behaviour amongst doctors and nurses: A short review of the literature. Health Information & Libraries Journal, 27(1), 2-10. http://dx.doi.org/10.1111/j.1471-1842.2010.00883.x


 

 

Appendix

 

Additional file: JAC Usage Statistics by Journal Title 2009-2013

 

 

Comments on this article

View all comments





Evidence Based Library and Information Practice (EBLIP) | EBLIP on Twitter