Amazon Turk woes: BOTS the software we love to hate?

  • Context: News 
  • Thread starter Thread starter jedishrfu
  • Start date Start date
  • Tags Tags
    Love Software
Click For Summary
SUMMARY

The recent discussion highlights a significant decline in data quality from Amazon Mechanical Turk (MTurk), as reported by psychology graduate student Max Hui Bai. Researchers noted an alarming increase in nonsensical responses and duplicate GPS locations in survey data, leading to the rejection of nearly half of the collected data. This phenomenon has raised concerns about the presence of bots on the platform, which are undermining the integrity of MTurk's intended human labor model. The widespread nature of these issues has prompted a collective response from the academic community, indicating a pressing need for solutions.

PREREQUISITES
  • Understanding of Amazon Mechanical Turk (MTurk) platform
  • Familiarity with survey design and data collection methods
  • Knowledge of data quality assessment techniques
  • Awareness of automated bots and their impact on online platforms
NEXT STEPS
  • Investigate methods for detecting and mitigating bot activity on MTurk
  • Explore alternative platforms for survey distribution, such as Prolific or Qualtrics
  • Learn about data cleaning techniques to handle low-quality responses
  • Research best practices for designing surveys to minimize the impact of bots
USEFUL FOR

This discussion is beneficial for psychologists, researchers utilizing MTurk for data collection, and data analysts focused on ensuring data integrity in survey research.

jedishrfu
Mentor
Insights Author
Messages
15,627
Reaction score
10,409
https://www.wired.com/story/amazon-mechanical-turk-bot-panic/

FOR THE PAST week, psychologists all over America have been freaking out.

The cause of their agita was an observation by a psychology graduate student from the University of Minnesota named Max Hui Bai. Like many researchers, Bai uses Amazon’s Mechanical Turk platform, where individuals sign up to complete simple tasks, such as taking surveys for academics or marketers, and earn a low fee. On Tuesday, August 7, he posed a simple question in a Facebook group for psychology researchers: "Have anyone used Mturk in the last few weeks and notice any quality drop?"

As he would later elaborate in a blog post, Bai had found that the surveys he conducted with MTurk were full of nonsense answers to open-ended questions and respondents with duplicate GPS locations. He said he had to throw out nearly half of the data in his most recent survey, a sharp increase from what he was used to seeing. His Facebook post garnered 181 comments, with other researchers describing similar signs of low-quality data in their own recent work. A number of them wondered if the culprit was bots—automated programs mimicking human behavior, not the actual human labor MTurk is supposed to supply.
 
Computer science news on Phys.org
Really? [sarcasm]Surprise, surprise. Social scientists.[/sarcasm] :rolleyes::rolleyes:
 
  • Like
Likes   Reactions: jedishrfu

Similar threads

Mathematica This Week's Finds in Mathematical Physics (Week 242)
  • · Replies 1 ·
Replies
1
Views
4K