NYU Steinhardt and NYU Wagner

Survey respondents don’t always take surveys as seriously as researchers would like. Sometimes, they provide intentionally untrue, extreme responses. Other times, they skip items or fill in random patterns. We might be tempted to think this just introduces some random error into the estimates, but these responses can have undue effects on estimates of the wellbeing and risk of minoritized populations, such as racially and sexually minoritized youth. Over the past decade, and with a focus on youth who identify as lesbian, gay, bisexual, or questioning (LGBQ), a variety of data-validity screening techniques have been employed in attempts to scrub datasets of “mischievous responders,” youths who systematically provide extreme and untrue responses to outcome items and who tend to falsely report being LGBQ. In this talk, I discuss how mischievous responders—and invalid responses, more generally—can perpetuate narratives of heightened risk, rather than those of greater resilience in the face of obstacles, for LGBQ youth. The talk will review several recent and ongoing studies using pre-registration and replication to test how invalid data affect LGBQ-heterosexual disparities on a wide range of outcomes. Key findings include: (1) potentially invalid responders inflate some (but not all) LGBQ–heterosexual disparities; (2) this is true more among boys than girls; (3) low-incidence outcomes (e.g., heroin use) are particularly susceptible to bias; and (4) the method for detection and mitigation affects the estimates. Yet, these methods do not solve all data validity concerns, and their limitations are discussed. While the empirical focus of this talk is on LGBQ youth, the issues and methods discussed are relevant to research on other minoritized groups and youth generally, and speak to survey development, methodology, and the robustness and transparency of research.

 

Joseph Cimpian, Ph.D., is an Associate Professor of Economics and Education Policy at NYU Steinhardt and associated faculty at NYU Wagner. He earned a Ph.D. in Economics of Education from Stanford University. His research focuses on the use and development of novel and rigorous methods to study equity and policy, particularly concerning language minorities, gender, and sexual minorities. One line of his research examines how “mischievous responders”—youth who provide extreme and untrue responses—can bias estimates of majority-minority group disparities. Some of his other work examines how beliefs about gender and math ability contribute to gender gaps in STEM. Prior to joining the faculty at NYU, Cimpian was an Associate Professor and College of Education Distinguished Scholar at the University of Illinois at Urbana-Champaign. His work has been funded by the Spencer Foundation, the AERA Grants Board, the National Science Foundation, and the Institute of Education Sciences. His research has been published in some of the top journals in education, psychology, health, and policy, and has been featured by the New York Times, the Washington Post, NPR, and Brookings, among other outlets. He presented his work on English learner reclassification policies at a U.S. Congressional briefing and for the Council of Chief State School Officers. At NYU, he teaches intermediate and advanced graduate courses on causal inference. He is currently an Editor of Educational Evaluation and Policy Analysis, and is on the editorial boards of several other education and psychology journals.