For empirical econ wonks

From Econ Journal Watch: Volume 9, Issue 3, September 2012.

Ziliak and McCloskey’s Criticisms of Significance Tests: An Assessment by Thomas Mayer. The abstract reads:

Stephen Ziliak and D. N. McCloskey have sharply criticized the prevailing use of significance tests. Their work has, in turn, come under vigorous attack. The vehemence of the debate may induce readers to wrongly dismiss it as a “he said-she said” debate, or else to take sides in an unbending way that does not do justice to valid points raised by the other side. This paper aims at a more balanced reading. While Ziliak and McCloskey claim that a substantial majority of economists who use significance tests confuse statistical with substantive significance, or commit the logical error of the transposed conditional, I argue that such errors are much less frequent than they claim, though still much too pervasive. They also argue that since significance tests focus on the existence of an effect rather than on its size, the tests do not answer scientific questions. I respond with counter-examples. Ziliak and McCloskey also complain that significance tests ignore loss functions. I argue that loss functions should be introduced only at a later stage. Ziliak and McCloskey are correct, however, that confidence intervals deserve much more emphasis. The most valuable message of their work is that significance tests should be treated less mechanically.

A reply from McCloskey and Ziliak: Statistical Significance in the New Tom and the Old Tom: A Reply to Thomas Mayer by Deirdre N. McCloskey, Stephen T. Ziliak. The abstract reads:

Econometricians have been claiming proudly since World War II that significance testing is the empirical side of economics. In fact today most young economists think that the word “empirical” simply means “collect enough data to do a significance test”. Tjalling Koopmans’s influential book of 1957, Three Essays on the State of Economic Science, solidified the claim. A century of evidence after Student’s t-test points strongly to the opposite conclusion. Against conventional econometrics we argue that statistical significance is neither necessary nor sufficient for proving commercial, human, or scientific importance. A recent comment by Thomas Mayer, though in parts insightful, does nothing to alter conclusions about the logic and evidence which we and others have assembled against significance testing. Let’s bury it, and get on to empirical work that actually changes minds.