Back to Search
Start Over
Validity of Online Screening for Autism: Crowdsourcing Study Comparing Paid and Unpaid Diagnostic Tasks
- Source :
- Journal of Medical Internet Research
- Publication Year :
- 2019
- Publisher :
- JMIR Publications Inc., 2019.
-
Abstract
- Background: Obtaining a diagnosis of neuropsychiatric disorders such as autism requires long waiting times that can exceed a year and can be prohibitively expensive. Crowdsourcing approaches may provide a scalable alternative that can accelerate general access to care and permit underserved populations to obtain an accurate diagnosis. Objective: We aimed to perform a series of studies to explore whether paid crowd workers on Amazon Mechanical Turk (AMT) and citizen crowd workers on a public website shared on social media can provide accurate online detection of autism, conducted via crowdsourced ratings of short home video clips. Methods: Three online studies were performed: (1) a paid crowdsourcing task on AMT (N=54) where crowd workers were asked to classify 10 short video clips of children as “Autism” or “Not autism,” (2) a more complex paid crowdsourcing task (N=27) with only those raters who correctly rated ≥8 of the 10 videos during the first study, and (3) a public unpaid study (N=115) identical to the first study. Results: For Study 1, the mean score of the participants who completed all questions was 7.50/10 (SD 1.46). When only analyzing the workers who scored ≥8/10 (n=27/54), there was a weak negative correlation between the time spent rating the videos and the sensitivity (ρ=–0.44, P=.02). For Study 2, the mean score of the participants rating new videos was 6.76/10 (SD 0.59). The average deviation between the crowdsourced answers and gold standard ratings provided by two expert clinical research coordinators was 0.56, with an SD of 0.51 (maximum possible SD is 3). All paid crowd workers who scored 8/10 in Study 1 either expressed enjoyment in performing the task in Study 2 or provided no negative comments. For Study 3, the mean score of the participants who completed all questions was 6.67/10 (SD 1.61). There were weak correlations between age and score (r=0.22, P=.014), age and sensitivity (r=–0.19, P=.04), number of family members with autism and sensitivity (r=–0.195, P=.04), and number of family members with autism and precision (r=–0.203, P=.03). A two-tailed t test between the scores of the paid workers in Study 1 and the unpaid workers in Study 3 showed a significant difference (P
- Subjects :
- Adult
pediatrics
020205 medical informatics
diagnosis
Autism Spectrum Disorder
digital health
autism
Health Informatics
02 engineering and technology
Crowdsourcing
Underserved Population
human-computer interaction
diagnostics
0202 electrical engineering, electronic engineering, information engineering
medicine
Humans
Mass Screening
Intrinsic motivation
Social media
CLIPS
mobile health
computer.programming_language
Original Paper
Internet
Diagnostic Tests, Routine
business.industry
Data Collection
Significant difference
medicine.disease
Corrigenda and Addenda
Digital health
mechanical turk
biomedical data science
citizen healthcare
Child, Preschool
Autism
neuropsychiatric conditions
business
Psychology
Social Media
computer
Clinical psychology
Subjects
Details
- ISSN :
- 14388871
- Volume :
- 21
- Database :
- OpenAIRE
- Journal :
- Journal of Medical Internet Research
- Accession number :
- edsair.doi.dedup.....89cdb368deecadfaa882c30e688678c8