Star Wars: The empirics strike back

Arbeitspapier

Journals favor rejection of the null hypothesis. This selection upon tests may distort the behavior of researchers. Using 50,000 tests published between 2005 and 2011 in the AER, JPE, and QJE, we identify a residual in the distribution of tests that cannot be explained by selection. The distribution of p-values exhibits a camel shape with abundant p-values above 0.25, a valley between 0.25 and 0.10 and a bump slightly below 0.05. The missing tests (with p-values between 0.25 and 0.10) can be retrieved just after the 0.05 threshold and represent 10% to 20% of marginally rejected tests. Our interpretation is that researchers might be tempted to inflate the value of those almost-rejected tests by choosing a significant specification. We propose a method to measure inflation and decompose it along articles' and authors' characteristics.

Object Details

• Language: Englisch

References and Relationships

• Bibliographic citation: Series: IZA Discussion Papers ; No. 7268

Classification and Topics

• Classification: Wirtschaft ; Role of Economics; Role of Economists; Market for Economists ; Economic Methodology ; Estimation: General ; Operations Research; Statistical Decision Theory
• Subject: hypothesis testing ; distorting incentives ; selection bias ; research in economics

Contributors, Places and Time

• Event: Geistige Schöpfung
• (who): Brodeur, Abel ; Lé, Mathias ; Sangnier, Marc ; Zylberberg, Yanos
• Event: Veröffentlichung
• (who): Institute for the Study of Labor (IZA)
• (where): Bonn
• (when): 2013

Further information

• Handle: http://hdl.handle.net/10419/71700
• Last update: 10.03.2025, 11:42 AM CET