Back to Search
Start Over
Science is not a signal detection problem
- Source :
- Proceedings of the National Academy of Sciences of the United States of America, vol 117, iss 11, Proc Natl Acad Sci U S A
- Publication Year :
- 2020
- Publisher :
- Proceedings of the National Academy of Sciences, 2020.
-
Abstract
- The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, because continuously distributed effects selected using a P < 0.05 filter must be inflated, the fact that they are smaller when replicated (reflecting regression to the mean) is no reason to sound the alarm. Considered from this perspective, recent replication efforts suggest that most published P < 0.05 scientific findings are “true” (i.e., in the correct direction), with observed effect sizes that are inflated to varying degrees. We propose that original science is a screening process, one that adopts NHST logic as a useful fiction for selecting true effects that are potentially large enough to be of interest to other scientists. Unlike original science, replication science seeks to precisely measure the underlying effect size associated with an experimental protocol via large- N direct replication, without regard for statistical significance. Registered reports are well suited to (often resource-intensive) direct replications, which should focus on influential findings and be published regardless of outcome. Conceptual replications play an important but separate role in validating theories. However, because they are part of NHST-based original science, conceptual replications cannot serve as the field’s self-correction mechanism. Only direct replications can do that.
- Subjects :
- Protocol (science)
Replication crisis
Multidisciplinary
Mechanism (biology)
Computer science
Prevention
05 social sciences
Null (mathematics)
replication crisis
050109 social psychology
null hypothesis significance testing
050105 experimental psychology
Outcome (probability)
Field (computer science)
Perspective
Replication (statistics)
Statistics
0501 psychology and cognitive sciences
signal detection theory
Statistical hypothesis testing
Subjects
Details
- ISSN :
- 10916490 and 00278424
- Volume :
- 117
- Database :
- OpenAIRE
- Journal :
- Proceedings of the National Academy of Sciences
- Accession number :
- edsair.doi.dedup.....3e9e854bfc5e4c71e2a859300d683585