No comment please ..... we're the JPE

Stan Liebowitz it has to be said is no stranger to controversy. But he is now involved in a controversy about a comment he sent to the Journal of Political Economy which was rejected by the then editor Steven Levitt. The JPE is one of the very top economics journals.

Last year the JPE published a paper by Felix Oberholzer-Gee and Koleman Strumpf on file sharing. The paper claims to show that file-sharing websites on the internet have not been responsible for the sharp decline in music sales since the turn of the millennium. Liebowitz claims that the research is fundamentally flawed. He submitted a comment to this effect to the JPE. Liebowitz believes that Oberholzer-Gee and Strumpf are guilty not merely of sloppiness, but of academic dishonesty, and is upset that they refuse to share their data. The Liebowitz comment which was rejected by the JPE editor, Levitt. This story has been commented on before at the Newmark's Door blog, see here and here.

But now this story has been picked up by the German newspaper Handelsblatt. The Handelsblatt article deals, in the main, with Levitt’s decision to ask Strumpf to write a reply and then to use the reply as an anonymous referee report in rejecting Liebowitz’s comment. The paper writes

Ignoring Liebowitz was not possible any more, however, after he submitted his counter-study officially as a comment to the JPE in September 2007. Levitt started by asking one of the authors, Koleman Strumpf, for his opinion. Strumpf handed in his reply in November. He defends the study and retaliates by pointing to alleged mistakes in Liebowitz? comment.

In addition, Levitt asked for a report from an impartial referee. The referee recommends publishing the comment in order to "save subsequent researchers from building on a flawed research foundation." While he advises Liebowitz to rephrase his comment such that it would not contain any overt assertions of data manipulation he sides with him on almost all the critical points and comes to a damming conclusion regarding the file-sharing article: "I would suggest that the authors? conclusions are not warranted given the analysis and evidence that they provide."

However, Levitt is not inclined to publish the comment. He anonymizes the reply by Strumpf and uses it as a second referee-report on which he bases his rejection of Liebowitz? comment. "There is no doubt you raise some reasonable points. Nonetheless, I think the negative referee (negative toward the comment, N.H.) is correct in most of what he says", Levitt writes to Liebowitz. The only point he takes up from the impartial referee is the advice to moderate the tone, should Liebowitz wish to submit the comment to some other, lesser journal.

But what may be more troubling in all of this is the unwillingness of Oberholzer-Gee and Strumpf to share their data (and we could also ask why the JPE hasn't insisted on them doing so). The point is that important empirical results in economics have to be, as a matter of course, carefully and fully scrutinized and shown to be replicable. Sharing data so this can be done must become the standard practice, since we can not have any faith in the results of an empirical study where such a practice has not been followed. The newspaper article notes

The impression that procedural standards of economics journals are not particularly strict is widely shared in the profession. Zurich-based economist Ernst Fehr, an associate editor of the top-five journal "Quarterly Journal of Economics" and of "Science" points to a lack of clear rules as to when an editor should recuse himself because of potential prejudice. Science journals also seem to deal more openly with the competition among scientists. "Authors who submit an article to a science journal can say who they do not want to review their article", praises Fehr, a choice which is typically not given to economists.

One internationally renowned economist, who did not want to be named, expresses the complaint more bluntly: "Little scandals and big scandals are commonplace: editors who publish articles in their own journals, referees or editors who decide about articles submitted by their own doctoral students."

This is not a good look, especially for one of the top economics journals.

(HT: Organizations and Markets and Newmark's Door)