Do low survey response rates bias results? Evidence from Japan

Ronald R. Rindfuss, University of North Carolina at Chapel Hill and East-West Center
Minja K. Choe, East-West Center
Noriko Tsuya, Keio University
Larry Bumpass, University of Wisconsin-Madison

Response rates to government and scientific surveys have deteriorated in most developed countries. This has been a growing concern among those in the survey research field, but has received limited attention from demographers who still tend to hold the view that survey data with low response rates should not be used. In this paper we examine the bias that might be associated with two survey efforts in Japan in 2009. The first was a cross-sectional national survey of the Japanese population aged 20-49. The second was a longitudinal follow-up in 2009 of a 2000 national survey of the population aged 20-49. Both the new cross-sectional survey and the follow-up had relatively low response rates. For both data collection efforts, we examine evidence for bias across a wide range of measures: fertility intentions, family and gender attitude items, measures of marital happiness, reports of hours spent on various household tasks, and reports of knowing people who had engaged in such family behaviors as using child care, having a non-marital birth, cohabiting, and planning not to marry. Evidence of bias varies across the two survey efforts, across items, and across methods to assess bias. In general, there is less bias than might be expected given the low response rates and for most items is effectively removed with proper controls in multivariate analyses.