Search

For scientific discoveries to be valid—whether in theory or empirically—a phenomenon must be accurately described: The scientist must use appropriate counterfactuals and eliminate competing explanations. Empirical work must also use an appropriate design and method, and empirical claims made about the phenomenon must be correctly characterized. Moreover, valid empirical discoveries must be reliable in the sense that scientists who reexamine the data must be able to reproduce the finding or to replicate the effect from data gathered in a similar context. Only discoveries adhering to the above criteria can be scientifically informative, serve as building blocks for theory, or have policy implications. Unfortunately, as several recent surveys of the literature show, much of the published works in the management and applied psychology fields are uninformative; contributing reasons include several intractable problems in the study design and analysis as well as the failure of the field to adopt open science practices. Against this backdrop, we identify common methodological mistakes made in applied work. We group these mistakes into three major categories: (a) study design and data collection (e.g., fit between hypotheses and methods, design, measurement, open science, literature reviews), (b) data analysis (e.g., data preprocessing, choice of estimators, analysis of data, issues concerning endogeneity, and use of instrumental variables), and (c) diagnostics, inferences, and reporting. We also explain how to avoid these issues, so that published work makes for a useful contribution to the scientific record.

Differences in national culture are commonly invoked to explain cross-country variability in the support for autocratic leaders, that is, leaders who concentrate all decision power in their hands. ...

Violating the distributional assumptions of latent interaction models can lead to biased estimates and invalid inference. However, no statistical procedure is readily available to verify whether th...

Why do we observe either participative or directive leadership in organizations? I test an evolutionary-informed theory suggesting that organizational leadership is currently less participative (i.e., close supervision, rare delegation) among societies that used intensive forms of agriculture in the past. Intensification caused increased social complexity and skewed power distribution, promoting the emergence of directive leaders and eventually shaping followers' preferences for and perceptions of leadership. Combining evidence, secondary data, and methods developed in economics, anthropology, and applied psychology, I document a negative relationship between traditional agricultural intensity and followers' participative leadership prototypes. I then study the link between traditional agriculture and reliance on delegation to subordinates across firms. I discuss competing hypotheses, explore the interplay between traditional agriculture and organizational-level factors, and show that traditional agricultural intensity does not predict most other leadership prototypes and management practices. Implications for leadership theory–with a focus on evolutionary approaches–are finally discussed.

Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state-of-the-science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top-tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best-practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi-experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.