Alvarado-Leiton, Davis & Lee - Impact of Response Styles on Inclusive Measurement. October 27, 2021

Fernanda Alvarado-Leiton completed the Program in Survey and Data Science in August 2021 and earned her Master’s degree in Statistics from the Unversidad de Costa Rica in San Jose, Costa Rica. Fernanda’s research interests focus on the sources of measurement errors in cross-cultural research. In particular, she is interested in response styles and how these impact the survey measurement process across different cultural groups. 

Rachel Davis is an Associate Professor in the Arnold School of Public Health at the University of South Carolina. She earned her Ph.D. from the Department of Health Behavior & Health Education and a certificate from the Program in Survey Methodology from the University of Michigan, as well as a Master’s in Public Health from the University of North Carolina. Dr. Davis uses qualitative and quantitative methods to better understand how culture, race, and ethnicity influence health communication and survey methodology, with a particular focus on working with Latinx and African American populations to mitigate health disparities.

Sunghee Lee is Research Associate Professor at the Survey Research Center, University of Michigan. Her research focuses on sampling and measurement issues for data collection with racial, ethnic and linguistic minorities.

Negated and Polar Opposite Items for Balanced Scale Construction: An Empirical Cross-Cultural Assessment 

Acquiescent Response Style (ARS) is a culturally patterned measurement error in surveys that threatens comparisons across groups with different cultural backgrounds potentially undermining inclusivity estimating attitudes and beliefs in a population. Balanced scales blend items written in different directions and are hypothesized as a method for controlling ARS. This study examined the differences in measurement properties between two types of balanced scales. The first balanced scale type included negated items, which were item reversals formed by inserting a negation, such as, “no” and “not.” The second type included polar opposite items, which used antonyms or opposite terms to reverse the item direction (e.g., “unhappy” as the opposite of “satisfied”). Participants were recruited to a Web survey and randomly assigned to (1) unbalanced, (2) negated balanced or (3) polar opposite balanced scales. Participants came from three groups with different ARS tendencies to contrast the effects of scale wording in mitigating ARS across groups and improving measurement across cultural subgroups. These groups were: Non-Hispanic White respondents, Hispanic respondents in Mexico and Hispanic respondents in the US. Both types of balanced scales outperformed unbalanced scales in convergent validity, with higher correlations between scale scores and validation variables for balanced than unbalanced scales. No statistical differences were observed between negated and polar opposite scales in fit indices of factor models, reliability measures or convergent validity for any group. These findings suggest that negated and polar opposite balanced scales are equivalent for ARS control, and that they yield adequate measurement properties for all groups included in the study.

Response Style and Measurement of Satisfaction with Life

Satisfaction with Life (SWL), a five-item scale, is designed to assess global judgment about one’s satisfaction with life as a whole rather than specific domains of life. Popularly used by many organizations, including the World Health Organization (WHO), the Central Intelligence Agency (CIA) and the United Nations Educational, Scientific and Cultural Organization (UNESCO), it has been translated into over 30 languages. However, with its standard version using a 7-point Likert response scale, it is subject to measurement error due to response style and measurement non-comparability across groups associated with systematically different response styles. More importantly, whether and how this is addressed in research may have implications for its inclusivity. This study examines the utility of balancing the SWL scale experimentally with multiple racial/ethnic/linguistic groups in the US: Latinx dominant in English, Latinx dominant in Spanish, non- Latinx Whites, non-Latinx Blacks, non-Latinx Koreans dominant in English and non-Latinx Koreans dominant in Korean. The results suggest the benefit of balancing measurement scales but not for groups that engage in middle response style.

Reducing Acquiescent Response Style with Conversational Interviewing

Acquiescent response style (ARS), the tendency for survey respondents to select positive answers such as “Strongly Agree,” is of particular concern for increasing measurement error in surveys with populations who are more likely to acquiesce, such as U.S. Latinx respondents. This study enrolled 891 Latinx telephone survey respondents in an experiment to address two questions: (1) Does administering a questionnaire using conversational interviewing (CI) yield less ARS than standardized interviewing (SI)? (2) Do item-specific (IS) response scales reduce ARS when compared to disagree/agree (DA) response formats? No difference was observed in ARS between the DA and IS response scales. However, CI yielded significantly lower ARS than SI, likely due to the CI interviewers' efforts to clarify questions and help with response mapping. Findings from this study suggest that using CI to administer survey questions may decrease use of ARS and improve data quality among survey respondents who are more likely to engage in ARS.