Filtering Survey Responses from Crowdsourcing Platforms: Current Heuristics and Alternative Approaches

2019 | International Conference on Information Systems | Citations: 0

Authors: Schmidt, Lennard; Dost, Florian; Maier, Erik

Abstract: Information Systems research continues to rely on survey participants from crowd ... Expand

Abstract: Information Systems research continues to rely on survey participants from crowdsourcing platforms (e.g., Amazon MTurk). Satisficing behavior of these survey participants may reduce attention and threaten validity. To address this, the current research paradigm mandates excluding participants through filtering heuristics (e.g., time, instructional manipulation checks). Yet, both the selection of the filter and the filtering threshold are not standardized. This flexibility may lead to suboptimal filtering and potentially “p-hacking”, as researchers can pick the most “successful” filter. This research is the first to tests a comprehensive set of established and new filters against key metrics (validity, reliability, effect size, power). Additionally, we introduce a multivariate machine learning approach to identify inattentive participants. We find that while filtering heuristics require high filter levels (33% or 66% of participants), machine learning filters are often superior, especially at lower filter levels. Their “black box” character may also help prevent strategic filtering. Collapse

Topics: logistics management information privacy concern website privacy shapley additive explanation

Methods: survey machine learning artificial neural network regression analysis method computational algorithm