11380: More Tests, Less Progress? The Pitfalls of Empirical Excess

Management scholars have made substantive gains in research methodology sophistication in recent decades. Indeed, newer advanced analytical tools are appropriate when they serve the purpose of addressing violations of the assumptions underlying less complex models and in dealing with various threats to validity. Recently, however, we have seen an explosion in the expected number of robustness tests, alternative specifications, supplemental analyses, and studies expected in a typical study. For OB scholars, a single experiment or field study is now insufficient as many papers are now stretching to 3, 4, and even 5 studies in a single publication. Strategy scholars are often asked to report primary results plus a host of additional tests. While we generally agree with the calls for increasing trust in our research by enhancing rigor, we can’t help but wonder if things have simply gone too far. That is, might there simply be too many tests? And, as a result, might we be encouraging the wrong behaviors, sapping the field of resources, and generally frustrating all involved? This session will attempt to address this by documenting how the empirical tests have changed over the last two decades, discussing how these changes are beneficial and where they cause problems, and offering recommendations that might be used by authors, reviewers, and editors to reign in practices that have little to no benefit (and perhaps harm) which ought to free up resources to more closely focus on the actions that matter for increasing the robustness and trustworthiness of our work.