User Studies Differ Across Some Disciplines and May Not Be Very Effective

Abstract

Objective – As part of a multi-staged project, this study seeks to identify the unanswered questions about users as found in three fields: library and information science (LIS), human computer interaction, and communication and media studies, as well as the convergences and divergences across these fields.

Design – A multi-phased, qualitative study involving individual face-to-face and telephone interviews, as well as self interviewing and focus groups.

Setting – The fields of LIS, human computer interaction, and communication and media studies as examined in interview situations.

Subjects – 83 international experts across the three fields, as well as 31 local experts from central Ohio, USA.

Methods – The majority of the 83 international experts in the fields of LIS, human computer interaction, and communication and media studies were interviewed by telephone (some in person). Thirty-one local experts (7 public and 24 academic library directors) were individually interviewed and also took part in focus groups. The Sense-Making Methodology was used as an interview approach with its emphasis on bridging gaps. Neutral interview questions were used to tease out the gaps in certain situations—in the case of this project, the gaps involve communication and the unanswered questions about users. Brenda Dervin developed this approach, which has been transformed and adapted by Dervin and a host of other LIS researchers over the past 25 years. It is a metatheoretical approach that has “evolved into a generalized communication-based methodology seen as useful for the study of human sense-making (and sense-unmaking) in any context” (Dervin 729). The Sense-Making metatheory is implemented three ways in the method: “in the framing of research questions; in the designing of interviewing; and in the analyzing and concluding processes of research” (Dervin 737). In the research under review for this summary the answers to the gap-identifying questions allow different disciplines to begin to communicate and understand each other. Using Sense-Making in focus groups involves self interviewing (diaries, journals) and group discussions.

Interviews were transcribed using the “smooth verbatim approach” in which non fluencies such as repetition, hesitancies, and partial words are eliminated. Care was taken to ensure anonymity, as this is necessary in the first step of the Sense-Making approach. The transcripts were analyzed for themes to capture a broad picture of what the participants struggle with across disciplinary and research-practice divides. Analysis was carried out by using comparative coding developed in early grounded theory combined with the Sense-Making methodology’s emphasis on gaps and bridging gaps. The “quotable quote” was the unit of analysis, and thematically representative quotes were selected from the transcriptions.

Main Results – In an attempt to analyze communication across and within disciplines, the researchers did a thematic analysis on the interviews conducted with their international and local experts. The thematic analysis found 12 major themes, which included a total of 75 sub themes. The 12 major themes include the following: Participants wanted to make a difference with their work; participants agreed that current user research is not doing the job; there are fundamental disagreements about users and user studies; there are fundamental disagreements about the purposes of using user studies; there are external forces that make carrying out and applying user studies difficult; there was a lengthy list of differing suggestions for improving user studies; interdisciplinary communication across the three fields that do user studies is not effective; it was agreed that interdisciplinary contact is difficult; communication across the research/practice divide is not going well; some participants saw academic researchers as the problem, while some participants viewed practitioners as the problem; and most participants agreed that contact across fields and the research/practice divide would be beneficial.

The researchers contend that this analysis is one of many that could be done on the information retrieved from the interviews. Their goal was not to find the definitive answers, but to describe the difficulties that participants are having across disciplines and across the research/practice divide in terms of communication and relating to user studies. The researchers wanted to tease out implications for communication and to illustrate the multiplicity that they found.

Conclusions – It is difficult for this study to draw conclusions except in the most general sense, as it is part of a larger, multi-staged research project. However, this study did find that although participants across fields wanted a synthesis, they also expressed their inability to understand syntheses from fields other than their own. There were some who wanted more theories, while some claimed there were too many theories already. There was much criticism about communication across disciplines, but few solutions offered. The researchers can offer up no “magic wands” as solutions for these results, but they do suggest that the modes of communication traditionally used in user studies research are not working.