Consumer health information on the Internet: an evaluation report on the Nova Scotia Health Network

: The Nova Scotia Health Network (NSHN) is a province-wide, Web-based consumer health information service provided through the cooperative efforts of various organizations, including public libraries, health sciences libraries, the Nova Scotia Provincial Library, and Dalhousie University. The primary intent of the NSHN was to build on existing community resources to provide a quality source of local and general health information. Objective – The purpose was to evaluate the NSHN from the perspective of its users in relation to the ease of use of the site and the usability of the information and content. Methods – A descriptive design was chosen to address the study objectives. Results – Study participants were those users of the NSHN site who agreed to complete and submit an online survey between June 2002 and June 2003. The majority of study respondents were white, middle-aged, English-speaking females, who resided in rural Nova Scotia. They were mostly able to find the information they wanted with little effort and in a reasonable time frame. One in four participants indicated that the information had been used to help them change their lifestyle, whereas one in three reported that they had shared the information with their health care pro-vider. The content found on the site was very highly rated, with more than 90% reporting that the information was easy to understand, useful, and of high quality. Conclusion – The data obtained in this study was positive and encour-aging. Sixty-six percent rated the site as a 4 or 5 (with 5 being the best rating) when compared with other health information sites they had visited. Individuals who visited the NSHN site in the past used the acquired information to change health care practices and (or) seek further treatment.


Introduction
The Nova Scotia Health Network (NSHN) is a provincewide, Web-based consumer health information service. It is provided through the cooperative efforts of public libraries, health sciences libraries, the Nova Scotia Provincial Library (Department of Education), several health-related organizations, and a number of departments in Dalhousie University. It is a straightforward initiative that identifies and pulls together quality publicly funded health information resources and services that are available to all Nova Scotians regardless of geographic location, reading level, or Internet access. The services provided by the NSHN include the following: (i) Links to directories of Nova Scotia (N.S.) health-related organizations (ii) Information written by Nova Scotian health care providers (iii) Links to reputable health information sites such as the Canadian Health Network and Medline (iv) Access to the Health and Wellness Resource Center (a database of reference material and full-text journal articles) (v) Links to the databases of N.S. libraries that provide health information (vi) E-mail "Ask a Librarian" service (direct responses to public queries provided by a professional librarian)

Objective
The overall purpose of this research was to evaluate the NSHN Web site from the perspective of its users in relation to the ease of use of the site, usability of the information, and the content itself. More specifically, our purpose was to describe the demographic characteristics of the users of the NSHN, describe the use of the Web site quantitatively and qualitatively, describe the participant's reasons for using the NSHN, and evaluate the "Ask a Librarian" service on the site.

Literature review
Growth in the use of the Internet as a resource for health information has been well documented in the literature. One of Canada's largest full-service marketing, opinion, and so- cial research organizations reported that, as of July 2002, three-quarters of Canadian Internet users were using the Web to seek health information [1]. This figure is a substantial increase compared with the 55% of Canadians reported to be visiting health-related sites in an Ipsos-Reid 2000 study of 1066 Canadian Internet users (Ipsos-Reid is also one of Canada's leading public opinion and market research companies) [2].
Further evidence of such growth over a 4-year period was provided by Statistics Canada. In 1998, only 9.6% of all Canadian households were using the Internet for the purpose of accessing medical and health information [3]. This grew to 15.6% in 1999, 22.9% in 2000, and 30.1% in the last report year of 2001.
Recent studies indicate a growing reliance on online information for health care decision-making. A 2001 RAND Health Report suggested that the increasing consumer interest combined with the vast amount of health-related information available online is beginning to change patientdoctor interactions and perhaps even the health care delivery system. The report stated that "there are both extravagant expectations and serious concerns about the trend" [4]. There are those who propose that the health of the individual will be improved because people will be better informed and able to actively participate in their own care in new ways. Others are fearful that reliance on inaccurate, incomplete, or misleading information will result in actual harm to online consumers. Similarly, Powell and Clarke discussed the role of the Internet in changing the balance of power by empowering the patient with knowledge and contributing to the deprofessionalization of medicine [5].
Alvarez [6] took a uniquely Canadian perspective and argued that e-health can be better used to resolve Canada's inherent health care challenges of geographic isolation, population density, harsh climate, socioeconomic and cultural influences, and rising health care costs. E-health is defined as any exchange of health-related information using information and communication technology, including the Internet, for health care purposes. Alvarez suggested that the health care sector is at least 10 years behind other sectors, such as banking, when it comes to the technology of ehealth. Alvarez also suggested that investment in e-health could significantly reduce or even eliminate some of the challenges facing the Canadian health care system.
On a smaller scale, Sciamanna et al. suggested that great gaps exist between what consumers are currently able to do on the Internet and what they would like to do as it relates to health care activities [7]. Sciamanna et al. contended that opportunity remains for developing more Web sites to address unmet patient needs, such as accessing information on the quality-of-care ratings of specific hospitals, ordering prescription drugs and drug renewals online, scheduling doctor's appointments, and communicating with one's physician by e-mail.
Along with the possibilities and promises are the issues and concerns. One concern addressed in the literature is that of personal privacy. Statistics Canada reported that in 1999, 60% of Canadians were concerned or very concerned that their personal information could be accessible to others while using the Internet [3]. In an article promoting e-health as a solution to Canada's health care woes, Alvarez ex-panded upon the privacy and patient confidentiality issue by stating that "the recent proclamation of Privacy and Confidentiality legislation across the provinces and territories is a considerable challenge to the development of interjurisdictional data sharing arrangements and to storage and manipulation of data holdings (especially patient records)" [6].
In summary, the literature suggests that we are still in the early stages of exploring the use of the Internet in the delivery and facilitation of health care. There are many unexplored opportunities and many challenges to be addressed, making ongoing research a priority.

Methodology
A descriptive design was chosen to address the study objectives and provided a means to describe the characteristics of the users of the NSHN, frequency of site use, perceived usefulness of the site, specific services used when accessing the site, content obtained, and usability of the content. Ethical approval was received from Dalhousie University Health Professional Ethical Review Board in June 2002.
Using the Web for research purposes and as a data collection method is quite new to many researchers: "practical issues related to the use of the Internet [sic] research have rarely been explored and discussed" [8]. Williams et al. recommended criteria for assessing the quality of Internet information [9]. The NSHN study included such criteria. Williams et al. also suggested a multimethod approach combining Web site inspection, user evaluation, and log statistics. All three methodologies were employed by the this study.
A convenience sample (n = 114) included users who chose to complete and submit the online questionnaire. The development of the questionnaire was based on an extensive literature review, and content validity was established by a panel of five educators. The readability level of the questionnaire is below grade eight, which is acceptable as the literature suggests that many people seeking health information read at or below the eighth grade level [10,11].
The setting for this study was the Internet protocol address for the NSHN. The URL is www.nshealthnetwork.ca. When users visited the NSHN, they were asked to complete an online questionnaire as part of a research study that was evaluating the site. Data were collected from June 2002 to April 2003 from users who chose to complete and submit the questionnaire. Twenty people declined to complete the survey indicating that they were worried about their privacy, worried about unexpected risks, did not have enough time, or had other reasons not stated.
The data collected were imported into a database developed for the sole purpose of this study and analyzed using the Statistical Package for the Social Sciences. Qualitative data were compiled, and the research team performed a thematic analysis. Issues of auditability, credibility, and fittedness [12] were addressed as the themes emerged. Data coding was conducted by three people to ensure consistency of findings. The triangulation of data enabled the research team to search for social significance [13].

Results and discussion
Data gathered from study participants indicated that the majority of the users were female (80%), English speaking (97%), white (90%), and indicated that their country of origin was Canada (99%) ( Table 1). Results reported from previous studies of individuals using the Internet to obtain health care information are consistent with the findings of this survey. According to Houston and Allison, the majority of "activist health consumers" (individuals who use the Internet to obtain health care information) are female, white, and English speaking [14].
Pandey et al. conducted a research study that examined women's use of the Internet for health purposes and found that geographic considerations did not have a significant association with Internet health usage [15]. More significant in predicting a woman's use of Internet health-based resources was the number of multiple daily commitments that the woman was "juggling", such as elder care, child care, and occupational responsibilities. The increased number of roles was predictive of the use of the Internet to access health information [15].
In this study, 64% of the sample reported that their age range was 41-60 years, with 23% indicating that they were less than 41 years old and 13% indicating that they were more than 61 years old. This finding is consistent with other research studies on Internet use for health information [14,5]. Previous research demonstrated that education level also affects Internet usage. Ninety-nine percent of the sample indicated that they had a high school diploma as a minimum level of education. Twenty percent reported having a community college education, 34% an undergraduate university education, and 22% a graduate degree. This finding is also consistent with other studies, indicating well-educated users dominate those using the Internet for health information. It is important to note that only 50% of all Nova Scotians understand printed information well enough to make informed decisions about their own health and to put them in a position to improve their family's health [16]. Some authors have questioned whether or not it is education alone or access to the Internet that impacts the lack of Internet use by less-educated people [5,17]. The research suggests that when access to the Internet is given to less-educated individuals, they are just as willing to obtain Internet-based health information.
Ease of use was defined as the degree of ease or difficulty in finding information on the site and whether or not assistance was required to obtain the information. According to the findings, the vast majority (64%) of users were able to find the information they required in less than 15 min and 80% were able to find the information within 30 min. However, 13% reported that they were unable to find the information they were seeking. Consistent with the above findings, 83% of users reported that it was easy or very easy to find the information they wanted (64% and 19%, respectively) ( Table 2). One major contributing factor in the individual's reported interest in obtaining health information from the Internet was their confidence in their ability to use the Internet -"self-efficacy" and "positive outcome expectancy". In other words, the belief that Internet use will help them deal with health con-cerns will encourage further use [18]. The fact that users consistently reported that the NSHN was easy to use increases the probability that they will return to the site for future health care information needs.
The use of the site was conceptually defined as how frequently the site as a whole and specific services were accessed and how the site was used (i.e., in what capacity did users visit the site). Between 2002 and 2003, there were 13 000 hits on the index page of NSHN. More than 4600 of these hits were on the Health and Wellness Directory, and approximately 4500 searches were conducted from the site index page. One of the limitations of using the number of hits as an outcome measure is that this quantitative finding does not indicate the length of time spent on the site and the services used while on the site. A person who inadvertently visits the site is counted the same way as the person who spent hours using the site.
Users came to visit the NSHN site through a number of different pathways. Ten percent of respondents came to the site by using a bookmark, and 8% had seen or heard advertisements for the site.  other Web sites. It was difficult to ascertain how the majority of users had come to the site, with more than 50% of the respondents saying they had used a means other than the choices offered (Table 3). This would be a fruitful area to explore in subsequent site evaluations. The most common reason for getting information for 42% of the respondents was for their own use. The next reason was for a family member or friend (15%), and 10% were seeking information for a client or employer. There is some preliminary research indicating that how a person uses the health information found on the Internet is correlated with the user's health status [14]. For example, it is suggested that people who perceive themselves to be in excellent health were more likely to search for health information for others (friends or family). In future evaluation efforts, it may be beneficial to inquire about the perceived health status of the user. For example, a 2000 study conducted by Cain et al. (see Powell and Clark [5]) characterized users into three groups: (1) the "well", who carry out episodic searches on short-term medical conditions and prevention; (2) the "newly diagnosed", who carry out intensive searches for very specific information; and (3) the "chronically ill", who regularly search for new treatments and alternative therapies.
The research variable of "content" on the NSHN is defined as the perceived characteristics of information materials found on the site. Regarding the completeness of content found on the site, more than 76% of respondents indicated that they found all desired information. In terms of the specific attributes of the health information, 90% of the respondents found the information easy to understand, useful to them, and of high quality. This high percentage is a very positive response. A study by the Health on the Net Foundation found that 33% of the respondents were dissatisfied with the quality of information found on the Internet [19].
Several users did make comments about the cultural sensitivity of materials found on the site, with some requesting more French content. Other respondents had suggestions concerning the need for more languages to be included in the multilingual health site. The literature suggests that English continues to dominate the Internet, with 52% of Web pages in English, compared with 4.6% in French [20]. Alvarez identified one of the challenges of providing ehealth to a country such as Canada, as the issue of the need for multiple languages [6]. It may be timely for consumer health Internet resources to invest energies into providing more content in a language other than English.
Usability of the information is defined as the perceived usefulness of the information found on the NSHN and includes how the individual used or planned to use the information obtained from the site. Eighty-seven out of 90 respondents described the information they found as being "useful to me".
Although not a specific measure of the usefulness of the site, users were asked to indicate where they usually obtained their health care information. Consistent with the literature, a large proportion (71%) of visitors to NSHN were using the Internet as their primary source of health care materials, although printed materials (84%) and speaking with health care providers (79%) were also prevalent sources of information. The findings from this study reflect the documented trend that, increasingly, the general public is turning to the Internet as a means of "self-care" in regard to health matters [5,21].
Data from the survey indicated that the most common reason for looking for information was related to illness, illness treatment, and related care (Table 4). This finding is consistent with several other recent studies. Sciamanna et al. [7] found that regardless of the patient's current access to the Internet, the majority of primary care patients were most interested in using the Internet to obtain information about a specific disease (67%) or medications (53.4%). A Health on the Net study found that 82% of participants were searching for information related to medications [19].
When asked their reasons for seeking information, 23 respondents stated that they were seeking information to stay healthy, and 12 users were looking for information to help with schoolwork or projects. Twenty-two people gave other reasons for seeking health information, such as preparing for a workshop, becoming familiar with what information was available online, and looking for a physician. The most common other reason was related to librarians assisting patrons to find specific health information.
Internet surveys indicate that online information does directly influence users' decision making about health issues [18]. One survey indicated that more than 70% of users said that health information found online influenced a decision about their treatment [22]. There are emerging trends toward health care consumers playing a more active role in their own care. As this occurs, there is a subsequent change in the roles of health professionals as collaborators in illness management and as health promoters [21]: "the Internet is a key  influence in changing the balance of (knowledge) power between health care professionals and the public, empowering patients to become more involved in health care decisionmaking" [5]. However, information concerning how Internet health information is actually used by health care consumers is in its infancy [17]. When asked how they planned to use the information found and how they had used the information found in the past, 11 people (22.91%) indicated that they planned to use the information to help change their lifestyles. Fifteen of the respondents (32.6%) indicated that they would be sharing the information with their health care provider, and 63 (80.76%) people indicated they were going to share the information with others. Twenty people (37.03%) indicated that this information would be used to provide or influence care.
Forty percent of all survey respondents indicated that they had visited the NSHN on at least one previous occasion. Of these, 33.33% indicated that in the past they had used the information to make contact with support groups and services within their communities, while 24% stated they used the information to decide to go to the doctor, and another 8% indicated that the information had helped them to decide not to go to the doctor.
The survey findings indicated that individuals who visited the NSHN site in the past used the information to change health care practices and (or) seek further treatment. One in four users indicated that the information found on the site was used to help them change their lifestyle. A similar percentage (24%) decided to seek medical attention based on information found on the site, and 26% indicated that they provided treatment at home (Table 5).
Almost one in three users reported that they had shared the information with a health care provider. It is interesting that most health information sites, like the NSHN, post a disclaimer stating that the information on the site is not a substitute for information from their health care provider. Consumers are given mixed messages concerning the degree of "suitable self-care". Health care providers recognize that there is a risk to self-care when using information found on the Internet [19]. However, a recent UK study of 800 doctors who use the Web found that twice as many doctors reported that their patients experienced more benefits than harm from Internet use. The most frequently reported benefits were that patients were more self-confident about self-care (34%) and better able to cope with their symptoms or disease [23].
As an overall measure of the usefulness of the NSHN site, users were asked to rank the site as compared with other health information sites they visited (with the ranking of 5 being the best). Sixty-six percent of respondents rated the site either a 4 or a 5. Forty-two respondents included some general comments with their questionnaire. The most common suggestion was for the site to have one search engine that would search the entire site. Other specific comments were related to the lack of French material (n = 6), inaccuracy and (or) incompleteness of information (n = 16), outdated physician links (n = 3), and difficulty navigating the site (n = 11).

Conclusion
The NSHN provides an important service to the citizens of Nova Scotia and beyond. It enables consumers to easily access information about their health, health problems, and resources available in their communities. This study provides a snapshot from the perspective of some of its users. The results discussed here should be useful in guiding future improvements in the availability and quality of health information on the Internet.

Yes
No Unsure In the past, the information I found on the site helped me  Table 5. Usefulness of information found on the site in the past.