Evidence Summary

Engagement with Search-Based Advertising on Search Engine Results Pages Varies Based on the User’s Prior Knowledge and Screen Size

A Review of:

Schultheiß, S., & Lewandowski, D. (2021). How users’ knowledge of advertisements influences their viewing and selection behavior in search engines. Journal of the Association for Information Science and Technology, 72(3), 285–301. https://doi.org/10.1002/asi.24410

Reviewed by:

Scott Goldstein

Coordinator, Web Services & Library Technology

McGill University Library

Montréal, Québec, Canada

Email: scott.goldstein@mcgill.ca

Received: 1 June 2021 Accepted: 26 July 2021

2021 Goldstein. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

DOI: 10.18438/eblip29989

Abstract

Objective – To examine how users’ understanding of ads on search engine results pages (SERPs) influences their viewing and selection behaviour on computers and smartphones.

Design – Mixed methods approach consisting of pre-study interview, eye-tracking experiment, and post-study questionnaire.

Setting – Usability lab at a university in Germany.

Subjects – 50 students enrolled at the Hamburg University of Applied Sciences and 50 non-students recruited in Hamburg.

Methods – After giving informed consent and receiving payment, participants provided information on demographics as well as how they use search engines as part of a pre-study interview. For the eye-tracking experiment, each participant completed 10 tasks each on a desktop computer and smartphone. Both the device condition order and task order were randomized. Tasks were broken down into five informational tasks (e.g., how do I build a desktop computer?), three transactional tasks (e.g., how would I go about buying a refrigerator?), and two navigational tasks (e.g., I need to go to the Apple website). The software displayed clickable screenshots of SERPs, and all clicks were recorded. iMotions eye-tracking software recorded eye fixations on areas of the page featuring organic search results and paid ads. A post-experiment questionnaire asked participants about Google’s business model and probed them about the extent to which they were able to differentiate between organic results and ads. Answers to the questionnaire were weighted and normalized to form a 0–100 scale.

Main Results – The first set of research hypotheses examining the correlation between participants’ knowledge of ads and viewing and clicking behaviour was partially confirmed. There was no significant correlation between participants’ questionnaire score and visual fixations on ads, but there was a significant negative correlation between questionnaire score and the number of clicks on ads. Users with questionnaire scores in the bottom quartile paid significantly less attention to organic results than those in the top quartile, but users in the top quartile still fixated on ads and did so comparably to users in the bottom quartile. The second set of research hypotheses examining the relationship between viewing and clicking behaviour and device (desktop versus mobile) was also partially confirmed. Users on a smartphone had significantly higher fixation rates on ads than users on a desktop computer, although click rates on ads did not differ significantly between the two conditions.

Conclusion – Knowledge about ads on SERPs influences selection behaviour. Users with a low level of knowledge on search advertising are more likely to click on ads than those with a high level of knowledge. Users on smartphones are also more likely to pay visual attention to ads, probably because the smaller screen size narrows content “above the fold.”

Commentary

This paper is part of the literature on search-based advertising, focusing on the paradigmatic example of Google, which has a Europe-wide market share of over 90%. Advertisements generated in response to user queries and appearing on search engine results pages (SERPs) account for the majority of Google’s revenue. However, according to one of the coauthor’s earlier articles, 40% of German Internet users were unaware of this or listed other incorrect sources of revenue (Lewandowski, 2017). Given that previous studies have shown users trust and rely on Google search, especially organic results high up on the first result page, it is important to know how ads contribute to or interfere with users’ information search behaviour. The authors’ eye-tracking experiment relies on data from 100 individuals, only half of them students, which is a departure from many other eye-tracking studies.

Perryman’s critical appraisal tool was used to appraise this study (Perryman & Rathbun-Grubb, 2014). This research was extremely well planned from start to finish. The authors conducted a thorough literature review and generated two specific research questions. Their recruitment strategy is commendable for its large size and diversity even if it is not a perfectly representative probability sample. Consistent with similar types of studies, the authors minimized bias with condition and question order randomization and withheld certain details of the study to reduce demand characteristics. Considering that 26 of the 100 participants were from the department of information, the authors ought to have looked at whether this large subgroup scored significantly higher on the questionnaire. Nonparametric statistical tests were used. One minor error is the use of “p = .000” (p. 297), which is, strictly speaking, impossible; presumably they meant to write “p < .001.” The authors included a list of limitations at the end of their study. Finally, all of the study’s data and materials are openly available on Zenodo.

There are a couple of implications of this research for libraries. The first is mentioned by the authors themselves, who write that with respect to information literacy, “we deem it imperative to help users understand that search engines do not necessarily act in their best interest, but search engine providers have interests of their own” (p. 299). It is not clear how much of information literacy currently is devoted to Google’s revenue model, but perhaps more should be taught on this to regular users. Secondly, this research has the potential to influence how libraries customize their discovery systems, especially next-generation systems that permit tweaking the display of SERPs. For instance, some libraries might be considering adding widgets and other query-based displays that, while not actually advertisements, might be viewed as such by users. If the benefits of these query-based displays outweigh the cognitive load costs to users, they should be clearly labelled and visually differentiated from the organic results.

References

Lewandowski, D. (2017). Users’ understanding of search engine advertisements. Journal of Information Science Theory and Practice, 5(4), 6–25. https://doi.org/10.1633/JISTaP.2017.5.4.1

Perryman, C., & Rathbun-Grubb, S. (2014). The CAT: A generic critical appraisal tool. In JotForm – Form Builder. Retrieved from https://www.jotform.us/cp1757/TheCat