Introduction

As illustrated by research dating back to as recently as 2010 or as far back as the 1990s (if not earlier), library discovery systems within the networked online environment have evolved, yet continue to struggle to serve users. As a result, the library (or systems supported and maintained by the library) is often not the first stop for research – or worse, not a stop at all. Users have defected, and research continues to illustrate this fact. (Vaughan, 2011, p. 7).

For many years, the University of Manitoba Libraries (UML), like many academic libraries, have offered a wide variety of tools to search for information. Books, audiovisual (AV) materials, and journals were discoverable from the UML’s catalogue, but articles, statistics, and other materials were discoverable only through specific databases, requiring specialized instruction for users in selecting appropriate resources to search. An external website review recommended that the UML implement a single search box as a starting place for all searches for library materials. At the time, the only service on the market was the Summon web-scale discovery service, which offered a single search box searching the majority of the UML’s databases and catalogue records. Summon was presented to the libraries’ staff in 2009, acquired late that year, and implemented in May 2010. Because Summon was a new service to UML and to libraries in general, UML staff were interested in monitoring the effectiveness of Summon, and one way to do so was to compare collection usage pre- and post-Summon implementation. It was expected that clients would find more materials that were relevant to their searches, thereby increasing the use of the UML collections.

Summon was implemented at the UML as an-out-of-the-box service. At the time, Summon allowed very little customization of order and placement of facets, fonts, displayed information, and other aspects of the interface. Summon in different libraries would differ in only in the branding and the content searched, dependent on activation in the Summon knowledgebase of the individual libraries’ available content. Implementation therefore required little decision-making, and the main work for UML’s staff involved making sure that the data from the UML’s catalogue was appearing correctly and linking back to the catalogue, and activating the electronic resources in the Summon knowledgebase.

The University of Manitoba is a doctoral-level university serving over 25,000 students, and the UML’s collections number over 1.8 million titles in 19 libraries, including 8 hospital libraries. Administration and technical services are centralized for the 19 libraries, and the UML uses Sirsi Symphony version 3.3.1, but switched from the Web2 interface to eLibrary in summer 2011. The UML also uses SFX as its OpenURL resolver and has implemented Verde as its electronic resources management system. Access to over 300 separate databases in a variety of subjects is provided by the UML in support of the university’s programs. These databases were made available using a Drupal product, although migration to LibGuides took place in summer 2011. The UML was the second library in Canada to adopt Summon as its resource discovery tool and made it available on the UML’s homepage as a single search option called “One Stop Search.”

Literature Review

Web-scale discovery tools first came onto the market in 2008; articles and books on the topic are now beginning to appear in the literature. There are many articles discussing the role, advantages, and disadvantages of web-scale discovery services, such as Hoy’s “An Introduction to Web Scale Discovery Systems” (2012) and Hoeppner’s “The Ins and Outs of Evaluating Web-Scale Discovery Services” (2012). Jason Vaughan also published a number of articles on the topic and, in 2011, wrote an ALA technology report which discusses web-scale discovery, documents available systems, examines their differences, and suggests questions librarians should ask before acquiring a web-scale discovery system (2011). There have been other reviews of various systems, most notably Ronda Rowe’s (2010) review of Summon, EBSCO Discovery Service, and WorldCat Local in the Charleston Advisor, which identifies some areas to consider when selecting a tool.

Implementation and decisions surrounding system configurations have been discussed in a number of articles and book chapters. Nara L. Newcomer (2011) and Anita Breckbill (2012) look at selecting, configuring, and searching web-scale systems for music-related information resources. More generally, implementation decisions for various systems are discussed in the “Implementation” section of Planning and Implementing Resource Discovery Tools in Academic Libraries, edited by Mary Pagliero Popp and Diane Dallis (2012). In particular, there are discussions on Summon implementation in a consortial environment (Christel, Koehler, & Upfold, 2012), in two British academic libraries (Thoburn, Coates, & Stone, 2012), and at Montana State University (Babbitt, Foster, & Rossmann, 2012).

Search satisfaction and usability testing also receive a good deal of attention in the literature. Julia Gross and Lutie Sheridan (2011) examine the user experience with Summon, finding that students react favourably to the simplified search, using it even when a different search tool might give them better results. This was similar to a finding at the University of Manitoba Libraries where the authors determined that while all participants searching Summon found relevant materials compared to only 60% of those searching pre-Summon, some searches would have retrieved better search results had they been done with more refined tools (O’Hara, Nicholls, & Keiller, 2012).

Change in collection usage as a result of the implementation of web-scale discovery tools is not as well covered as other topics. Doug Way (2010) of Grand Valley State University Libraries examined usage statistics over a three-month period post-Summon implementation, to find the impact of Summon on collection usage at his institution. He determined that there was a dramatic increase in the use of full-text resources and a dramatic decrease in the use of abstracting and indexing databases. Jan Kemp (2012) at the University of Texas at San Antonio also examined changes in collection usage post-Summon implementation, finding that full-text article downloads increased by 23%. Tonia Graves (2012), discussing the effects of a discovery layer on usage a year after implementation, reports that “usage reports show a dramatic increase in patron usage of WorldCat Local. Detailed reports from WorldCat Local indicate that the lowest amount of usage in a single month for the year of 2011 was higher than the highest usage month in 2010” (p. 173).

Methods

To examine changes in collection usage at UML, usage statistics were examined for the year prior to Summon implementation (May 2009 to April 2010 inclusive) and the two years after Summon implementation (May 2010 to April 2011 and May 2011 to April 2012 inclusive). As Way (2010) did, to ensure that measurements for electronic resources were comparable, COUNTER statistics were used in the UML study to find database, e-journal, and e-book statistics. These included database search statistics (DR1) from the COUNTER Database Report 1 to measure database usage; full-text article downloads from the COUNTER Journal Report 1 (JR1) to measure electronic journal usage; and successful section search requests from the COUNTER Book Report 2 (BR2) to measure e-book usage. Because electronic collections are dynamic and change is a constant, only platforms and databases that were licensed by UML over the entire period of the study were included. As well, only publishers, databases, and providers that reported COUNTER statistics for the full period of the study were included. The majority of UML’s publishers and providers supply COUNTER statistics, covering an estimated 95% of the UML’s e-journal collections, less for other electronic collections. For the more static print resources, Sirsi statistics were used to determine changes in usage in the pre- and post-Summon implementation periods, just as Kemp (2012) used OPAC circulation statistics for the same purpose in his study. Other factors such as the increase or decrease in size of collections, changes in collection policies, and recommendations for the use of specific materials or collections by faculty were not considered except peripherally in the discussion.

Results

Print Resources

Print resource circulation increased for monographs fairly steadily until Summon implementation, when they began to decline slightly. Usage for print journals had been declining steadily, but circulations actually increased by 3% over the previous year in the final year examined. One factor affecting this increase might be the addition of a print journal target to the SFX (OpenURL resolver) menu at UML during this period. Adding the target allowed patrons searching for an article in a citation database or in Summon to immediately determine if the UML held the required journal in print. Prior to this, patrons had to perform a separate search in the catalogue to find whether the journal was held in print after they identified required journal articles.

Figure 1

Checkouts for print resources 2007-2012. ^a

^a Note that implementation took place in May 2010; statistics prior to that are pre-Summon implementation and are given to provide some context for general trends in print circulations at UML.

Print Circ Chart.jpg

Database Use

Traditional abstracting and indexing databases saw a fairly uniform decrease in usage post-Summon, according to the usage statistics reported in the table below.

Table 1

Percent Change in Database Searches by Database for the Period May 2009-April 2010 (Y0, Pre-Summon) and May 2010-April 2012 (Y1 & Y2, Post-Summon)

Database Name	%Change Y0-Y1	%Change Y0-Y2	Average Change
Health and Safety Science Abstracts	-84.22%	-95.29%	-89.81%
CSA Social Services Abstracts	-25.84%	-84.09%	-54.96%
Conference Papers Index	-26.37%	-79.42%	-52.90%
Environmental Sciences and Pollution Abstracts	-9.68%	89.09%	49.39%
Biotechnology Research Abstracts	-20.66%	-77.62%	-49.14%
Bacteriology Abstracts (Microbiology B)	-14.97%	-74.57%	-44.77%
CSA Linguistics and Language Behaviour Abstracts	2.4%	-82.37%	-39.99%
COS Scholar Universe: Social Science	5.56%	-80.65%	-37.54%
GeoRef	26.64%	-91.57%	-32.47%
CSA Worldwide Political Science Abstracts	-16.49%	-47.44%	-31.97%
Social Work Abstracts	-9.57%	-49.57%	-29.57%
Child Development & Adolescent Studies	-10.58%	-45.41%	-27.99%
Bibliography of Native North Americans	-8.83%	-44.07%	-26.45%
America: History & Life	-7.38%	-42.71%	-25.04%
Anthropology Plus	-7.44%	-41.09%	-24.26%
Historical Abstracts	-6.99%	-40.79%	-23.89%
ATLA Religion Database with Serials	-5.27%	-41.84%	-23.56%
Peace Research Abstracts	-4.15%	-39.70%	-21.39%
RILM Abstracts of Music Literature	-3.44%	-38.66%	-21.05%
PsycINFO	-33%	-4.74%	-18.87%
CAB Abstracts	-3.36%	-32.89%	-18.13%
SPORTDiscus	1.15%	-35.50%	-17.17%
CINAHL	-9.36%	-13.82%	-11.59%
Food Science and Technology Abstracts	9.85%	-19.75%	-4.95%
GeoBase	-2.45%	-0.39%	-1.42%
Compendex	8.63%	0.11%	4.37%
Avery Index to Architectural Periodicals	-14.42%	170.52%	78.05%
International Pharmaceutical Abstracts	106.70%	91.83%	99.26%
Index Islamicus	-10.52%	284.74%	137.11%
Risk Abstracts	-84.37%	606.72%	261.18%
ARTbibliographies Modern	-19.64%	664.15%	322.26%
METADEX	-31.79%	1186.49%	577.35%
Industrial and Applied Microbiology Abstracts	-20.45%	2474.9%	1227.22%
TOXLINE	-19.08%	2561.01%	1270.96%
Toxicology Abstracts	-30.78%	3050.9%	1510.06%
Ecology Abstracts	-18.82%	3578.33%	1779.75%
Water Resource Abstracts	-18.93%	3581.54%	1781.30%
Environmental Engineering Abstracts	-24.67%	8462.44%	4218.89%
AVERAGE	-11.65%	668.96%	328.66%

Most databases for which usage grew significantly post-Summon implementation were migrated to the new ProQuest platform in 2012, leading staff to suspect that the data were faulty. ProQuest staff confirmed that the issue had also been reported by other libraries. Therefore, the numbers in the table above for “% change Y0-Y2” cannot be considered accurate for the ProQuest databases (shaded in grey) and should be discounted; however, they were included in the study because they constitute a significant portion of UML’s citation databases. When the second-year post-Summon implementation data for the ProQuest databases are removed, the average change from the year before Summon implementation to the second year post-Summon implementation is -11.68%, almost exactly the -11.65% change experienced in the first year post-Summon implementation

Electronic Journal Use

The majority of platforms saw a significant increase in successful journal requests, with an average increase of almost 19% in the first year post-Summon and an increase of 43% from the year before Summon was implemented to the second year after implementation.

Figure 2

Average e-books usage for the period May 2009-April 2010 (Y0, pre-Summon) and May 2010-April 2012 (Y1 & Y2 post-Summon)

Electronic Book Use

The UML provide access to all books and e-books through the library catalogue; a bibliographic record for each e-book is loaded to Summon, with new acquisitions being loaded weekly. This means that every e-book is available through Summon via the catalogue whether the metadata is available in Summon’s knowledgebase or not.

Average usage of e-book platforms increased post-Summon implementation but usage of two platforms dropped overall.

Table 2

Percent Change in Successful Journal Requests by Platform for the Period May 2009-April 2010 (Y0, Pre-Summon) and May 2010-April 2012 (Y1 & Y2, Post-Summon)

Platform	%Change Y0-Y1	%Change Y0-Y2	Average Change
Scitation	-59.85%	-52.46%	-56.16%
Poj	-63.27%	-42.86%	-53.06%
Newsbank	-42.37%	-59.27%	-50.82%
APS	-40.18%	-47.98%	-44.08%
ACS Publications	-36.03%	-34.7%	-35.37%
IIMP	-34.18%	-34.32%	-34.25%
Project Euclid	-25.83%	-21.32%	-23.57%
APS Journals	-10.14%	-32.69%	-21.42%
MIT Press Journals	14.68%	-46.63%	-15.98%
ProQuest	5.51%	-29.5%	-12%
ACM Digital Library	10.62%	-32.71%	-11.04%
JSTOR	-15.74%	1.26%	-7.24%
SwetsWise	-7.21%	4.57%	-1.32%
Highwire Press	0.19%	1.43%	0.81%
BioOne	8.04%	-0.21%	3.92%
Highwire	3.73%	11.22%	7.47%
EBSCOhost	15.55%	11.44%	13.5%
content.karger.com	32.13%	-5.08%	13.52%
BioMed Central	20.61%	31.34%	25.98%
ScienceDirect	19.21%	41.6%	30.4%
nature.com	14.5%	50.02%	32.26%
Wiley	34.48%	50.1%	42.29%
Annual Reviews	44.1%	41.6%	42.85%
IEEE Explore	24.47%	67.61%	46.04%
Project MUSE	22.79%	71.58%	47.19%
MetaPress	11.32%	87.67%	49.49%
rsc.org	31.61%	75.61%	53.61%
JNSPGOnline	10.63%	97.11%	53.87%
Periodicals Archive Online	49.96%	66.08%	58.02%
IngentaConnect	33.7%	83.18%	58.44%
CJO	41.77%	82.46%	62.11%
GOLD	52.05%	75.8%	63.91%
internurse.com	124.15%	123.1%	123.13%
palgrave-journals.com	136.06%	126.6%	131.33%
Thieme Journals	15.02%	295.6%	155.31%
MLA Journals	95%	232.5%	163.75%
CAIRN	170.48%	303.81%	237.14%
AVERAGE	19.1%	43.07%	31.08%

Discussion

The University of Manitoba’s examination of usage statistics did not reveal unexpected results and was typical of the results found in the studies carried out by Way (2010) and Kemp (2012). One area not examined in previous studies was usage of print collections and the decrease in checkouts for print collections at UML was not unexpected. However, there are other factors that might also affect this decrease, including the purchase of large numbers of backfiles, weeding of the print collection, the movement of materials to off-site storage, and a move to e-preferred purchasing. These factors make it impossible to determine if the decrease in usage is due to Summon or to the other factors and might be worth further study. wj

Although Way (2010) described the drop in searches for core subject databases as troubling, it is explained by the fact that usage is no longer tracked in the native interface once citations are included in Summon. The almost uniform decrease in usage of citation databases cannot be accurately measured until it is possible to track usage by database within Summon itself. At present, the source of citations returned from searches cannot be tracked or measured, so it is impossible to determine whether citation databases within Summon are used or useful. The fact that the decrease in usage statistics averaged -11% for both years post-Summon implementation demonstrates that a significant amount of research is still being conducted in these native interfaces and that they are still a necessary acquisition for academic libraries.

For electronic journals, the UML case study shows similar results to those found in the case study Way (2010) conducted at Grand Valley State University Libraries. He found that usage of full-text ejournals increased “regardless of whether the content provider had directly partnered with Serials Solutions to make their content available in Summon” (p. 219).Similarly, UML experienced increases in usage for content not in Summon, as exemplified by the 13.5% average increase in successful journal requests in the EBSCOhost platform. The UML results of an average increase of 19% in successful searches in the first year post-Summon implementation are very close to the results seen at the University of Texas at San Antonio, where full-text downloads increased 23% in the same one-year period (2012).

E-book usage also increased on average for the platforms surveyed. Because the UML are in the process of moving from a mainly print-based monograph collection to an electronic monograph collection, it is more difficult to state confidently that the average increase was due to Summon implementation. Many other factors – including fluctuations in the size of the e-book collections, availability of print books, implementation of demand-driven acquisitions, not to mention the relevancy ranking algorithm used by Summon itself – could affect the e-book usage statistics positively and negatively. It is interesting that the two platforms that saw decreases in usage, Blackwell Online Reference and Royal Society of Chemistry, have somewhat similar content to Credo Reference Online and Springer, which both saw increases in usage. This is certainly an area where further research could be done.

Conclusion

Although the study carried out at the University of Manitoba Libraries examined only those platforms where COUNTER statistics were supplied, a number of conclusions can be drawn. Summon implementation at UML has resulted in an increase in the use of the UML’s electronic full-text collections. In fact, the similarity in results among the UML study, Way’s study (2010), and Kemp’s study (2012) for e-journals suggest that implementation of a web-scale discovery system will increase usage of full-text e-journals in academic libraries. Although neither Way nor Kemp examined e-book usage, the UML study indicates that the same is true for full-text e-books, and so further study is needed in this area.

The fact that citation database usage decreased with the implementation of a discovery layer was not a surprise, since the searches are being conducted within the discovery layer rather than in the native interface. In fact, it can be argued that this decrease points to client satisfaction with Summon searching, assuming that Summon is being used by some segment of the UML user population in place of the native interfaces. However, because there is still significant use of the citation databases recorded, they are still a necessary acquisition.

Although the decrease in print circulations is troubling, it is not possible to determine whether it is the result of the implementation of Summon or other factors, and it will be worthwhile to monitor these circulations in the future. It will also be interesting to observe how resource discovery tools will continue to affect collection usage as vendors and publishers provide better metadata, and advances such as semantic web technology increase search effectiveness.

References

Babbitt, E. P., Foster, A, & Rossmann, D. (2012). Implementation of resource discovery: Lessons learned. In M. P. Popp & D. Dallis (Eds.). Planning and implementing resource discovery tools in academic libraries (pp. 598-607). Hershey, PA: IGI Global.

Breckbill, A. (2012). The trial of searching for musical works using resource discovery tools. In M. P. Popp & D. Dallis (Eds.). Planning and implementing resource discovery tools in academic libraries (pp. 662-676). Hershey, PA: IGI Global.

Christel, M., Koehler, J., & Upfold, M. (2012). Implementing a discovery layer in a consortial environment. In M. P. Popp & D. Dallis (Eds.). Planning and implementing resource discovery tools in academic libraries (pp. 407-418). Hershey, PA: IGI Global.

Graves, T., & Dresselhaus, A. (2012). One academic library – One year of web-scale discovery. The Serials Librarian, 62(1-4), 169-175. doi: 10.1080/0361526X.2012.652915

Gross, J., & Sheridan, L. (2011). Web scale discovery: The user experience. New Library World, 112(5/6), 236-247. Retrieved 22 Oct. 2012 from: http://search.proquest.com/docview/868247945

Hoy, M. B. (2012). An introduction to web scale discovery systems. Medical Reference Services Quarterly, 31(3), 323-329. doi: 10.1080/02763869.2012.698186

Hoeppner, A. (2012). The ins and outs of evaluating web-scale discovery services. Computers in Libraries, 32(3), 6-10. Retrieved 22 Oct. 2012 from http://search.proquest.com/docview/1008145593?accountid=14569

Kemp, J. (2012). Does web-scale discovery make a difference? Changes in collections use after implementing Summon. In M. P. Popp & D. Dallis (Eds.). Planning and implementing resource discovery tools in academic libraries (pp. 456-468). Hershey, PA: IGI Global.

Newcomer, N. L. (2011). The detail behind web-scale: Selecting and configuring web-scale discovery tools to meet music information retrieval needs. Music Reference Services Quarterly, 14(3), 131-145. doi: 10.1080/10588167.2011.596098

O'Hara, L., Nicholls, P., & Keiller, K. (2012). Search success at the University of Manitoba Libraries pre- and post-Summon implementation. In M. P. Popp & D. Dallis (Eds.). Planning and implementing resource discovery tools in academic libraries (pp. 268-287). Hershey, PA: IGI Global.

Popp, M. P., & Dallis, D. (Eds.) (2012). Planning and implementing resource discovery tools in academic libraries. Hershey, PA: IGI Global.

Rowe, R. (2010). Web-scale discovery: A review of Summon, EBSCO Discovery Service, and WorldCat Local. Charleston Advisor, 12(1), 5-10. doi: 10.5260/chara.12.1.5

Thoburn, J., Coates, A., & Stone, G. Simplifying resource discovery and access in academic libraries: Implementing and evaluating Summon at Huddersfield and Northumbria Universities. In M. P. Popp & D. Dallis (Eds.). Planning and implementing resource discovery tools in academic libraries (pp. 580-597). Hershey, PA: IGI Global.

Vaughan, J. (2011). Web scale discovery services. Chicago, IL: ALA Editions. Retrieved 22 Oct. 2012 from http://site.ebrary.com/docDetail.action?docID=10469319

Way, D. (2010). The impact of web-scale discovery on the use of a library collection. Serials Review, 36(4) 214-220. doi: 10.1016/j.serrev.2010.07.002