Let’s Agree to Disagree! New Publication by UWSC Researchers Recommends Against Using Agree-Disagree Questions

Although they are ubiquitous, a recent publication by UWSC researchers and collaborators calls to question the use of agree-disagree (AD) questions to measure attitudes and opinions. In “Towards a reconsideration of the use of use of agree-disagree questions in measuring subjective evaluations*,” UWSC researchers provide a review and synthesis of research on the measurement properties and potential limitations of AD questions.

The research leads them to advocate for item-specific (IS) questions, which are written to directly ask about their underlying response dimensions using response categories tailored to match the response dimension. The authors 1) synthesize past research comparing data quality for AD and IS questions; 2) present conceptual models of and review research supporting respondents’ cognitive processing of AD and IS questions; and 3) provide an overview of question characteristics that frequently differ between AD and IS questions and may affect respondents’ cognitive processing and data quality.

Although experimental studies directly comparing AD and IS questions yield some mixed results, more studies find IS questions are associated with desirable data quality outcomes (e.g., validity and reliability) and AD questions are associated with undesirable outcomes (e.g., acquiescence, response effects, etc.). Based on available research, models of cognitive processing, and a review of question characteristics, they recommend IS questions over AD questions for most purposes. For researchers considering the use of previously administered AD questions and instruments, issues surrounding the challenges of translating questions from AD to IS response formats are discussed.

*Dykema, Jennifer, Nora Cate Schaeffer, Dana Garbarski, Nadia Assad, and Steven Blixt. 2021. “Towards a Reconsideration of the Use of Agree-Disagree Questions in Measuring Subjective Evaluations.” Research in Social and Administrative Pharmacy. https://doi.org/10.1016/j.sapharm.2021.06.014