Millennial Students’ Online Search Strategies are Associated With Their Mental Models of Search

Abstract

Objective – To examine first-year college students’ information seeking behaviours and determine whether their mental models of the search process influence their ability to effectively search for and find scholarly materials.

Design – Mixed methods including contextual inquiry, concept mapping, observation, and interviews.

Setting – University of Baltimore, a public institution in Maryland, United States of America, offering undergraduate, graduate, and professional degrees.

Subjects – A total of 21 first-year undergraduate students, ages 16 to 19 years, undertaking research assignments for which they chose to use online resources.

Methods – First-year students were recruited in the fall of 2008 and met with the researcher in a university usability lab for about one hour over a three week period. The researcher observed and videotaped the students as they conducted research in their chosen search engines or article databases. The searches were captured using software, and students were encouraged to think aloud about their research process, search strategies, and anticipated search results. Observation sessions concluded with a 10-question interview incorporating a review of the keywords the student used, the student’s reflection on the success of his or her searches, and possible alternate keywords. The interview also offered prompts to help the researcher learn about students’ conceptualizations of search tools’ utilization of keywords to generate results. The researcher then asked the students to provide a visual diagram of the relationship between their search terms and the items retrieved in the search tool.

Data were analyzed by identifying the 21 different search tools used by the students and categorizing all 210 searches and student diagrams for further analysis. A scheme similar to Guinee, Eagleton, and Hall’s (2003) characterized the student searches into four categories: simple single-term searches, topic plus focus searches, phrase searches, and advanced searches employing multiple Boolean operators. Students’ diagrams were put into three different groups: process view, hierarchical view, and network view. The researcher then analyzed the relationships between the students’ search behaviours and their mental models to develop further conclusions.

Main Results – Analysis revealed that this population of students had a limited mental model of the search process and used narrow sets of fairly simple search strategies for retrieving information online.

Search engines were used for the majority (61.9%) of total searches and 72.3% of those conducted in search engines were in Google. The majority of students (76%) began their search process with a search engine while other students began searching in online encyclopedias (10%) or online databases (14%). Academic Search Premiere was used for 73.8% of the database searches. Some students (5%) also performed searches in individual websites (6.3%), for an overall total of 224 searches conducted.

Students performed four varieties of searches: simple searches using short phrases conveying a single concept (34% of total searches); topic plus focus searches using a single Boolean AND (30%); phrase searches consisting of multiple-word descriptive phrases or sentence fragments (17.4%); and advanced Boolean searches combining two or more distinct concepts (13.8%). Generally, students used the same search terms and structure whether they were in a search engine or database, particularly with phrase searches. Nearly 71% of the advanced Boolean searches were inappropriately formed, particularly when used in the databases. Of the few students employing Boolean logic beyond a single AND, only two used it correctly, and only one with successful results.

Students were unable to recognize or explain why a search failed or why they got the results they did. They made frequent incorrect use of punctuation, spelling, and syntax, leading to limited or no search results. Students assumed that obtaining few results indicated a problem of keyword choices rather than search query structure. When faced with no results in the databases, they assumed there were no articles on their topics and did not re-evaluate their search queries. Those with unsuccessful Boolean searches did not recognize that their errors were due to logic, and instead changed their keywords or began a new search altogether.

Several students understood keywords as concepts versus literal strings of letters, yet thought the search tools determined search results based solely on what was typed into the search box. Of those employing phrase searches, some believed that each word was queried, while others thought only the “primary terms” were queried. Most students (61%) offered analogies to print resources to explain how search engines process queries, and all the students’ descriptions included their ideas about what a search tool contained, rather than how the search tools organized information.

Attempts to expand or narrow searches were haphazard. While most students (57%) employed the strategy of adding keywords to narrow searches, only a few (11%) recognized the function of this technique and used it regularly, while others tended to return to their original broad searches in a different tool. Some had a limited understanding that adding terms narrows and reducing terms broadens the search, but their Boolean errors negated the use of synonyms or alternative terms for those purposes. Other strategies included using the search tool’s “advanced search” features or quotes, although all who used the latter did so incorrectly and some mistakenly thought parentheses served the same purpose.

All subjects drew representations of their views of the relationship between keywords used and search results retrieved, though few were able to clearly visualize how a search engine processes a query, or address ideas such as expanding or narrowing searches or synonym use. Three categories of diagrams emerged: the process view, hierarchical view, and network view. The process view displayed a task flow diagram. These students demonstrated the least formed mental models and experienced the search tool as a “black box” that gives results, showing little understanding of how they are generated. They performed the fewest overall searches (11.6%), the majority (79%) of which were simple or phrase searches with no use of Boolean operators. The hierarchical view displayed a broad subject with subtopics, or results highlighting specific aspects of the subject. These students performed nearly 30% of the total searches, 17% of which included the use of Boolean logic. The network view displayed models of interconnected terms. These students performed the majority of the searches (58.7%), and also constructed the most sophisticated queries. Many of their searches employed Boolean logic (83%), and 65% were either Boolean or topic plus focus searches. Students with this mental model tended to focus more on the queries themselves than the results received.

Students indicated feelings of success in their searching and were comfortable relying on simple searches retrieving large results sets. While not central to the research design or driving questions, students’ evaluation of search results was observed and found to be weak. Students displayed rapid searching, scanning, and evaluation processes which may have played a role in many of their mistakes when repeating or attempting to correct faulty searches.

Conclusion – The results show students did not have strong conceptual models of the search process or how search queries impacted results, and were often unable to recognize or troubleshoot problems with searches in order to improve results. Students displaying stronger mental models used more complex search strategies, but still performed unsuccessful searches and demonstrated challenges in remedying defective searches. Students skimmed search results quickly, rarely looking beyond the first two pages, and did not take time to evaluate them for topic relevance. The findings suggest that librarians should rethink how Millennial students are taught search strategies and evaluation, to focus more on problem solving or critical thinking. They also suggest that database developers should continue developing search algorithms and tools, considering this population’s conceptualizations of search. Further research on Millennials’ information processing, critical thinking, and evaluation skills in the context of academic work is needed.