Russ Roberts on empirical economics: part 2

Tyler Cowen takes up the challenge to answer Russ Robert's question from a posting I noted earlier. Roberts asked,

I'd like one example, please. One example, from either micro or macro where people had to give up their prior beliefs about how the world works because of some regression analysis, ideally usually instrumental variables as that is the technique most used to clarify causation.

Cowen writes,

I will cite a few possible examples, although I won't stick with instrumental variables:

1. The interest-elasticity of investment is lower than people once thought.

2. We have a decent sense of the J Curve and why a devaluation or depreciation doesn't improve the trade balance for some while.

3. Dynamic revenue scoring tells us over what time horizon a tax cut is partially (or fully) self-financing.

4. Most resale price maintenance is not for goods and services involving significant ancillary services.

5. More policing can significantly lower the crime rate (that one does use instrumental variables).

6. The term structure of interest rates is whacky.

I see other examples but in general I agree with Russ's point that empirical work fails to settle a great number of important disputes, most disputes in fact. Many of the examples I would cite turn out to involve an elasticity being lower than we had thought. And many more involve macroeconomics (rather than micro) than you might expect.

Peter Boettke also posts on the Robert's question, giving a Austrian view. He says,

Does the fact that statisical tests cannot provide unambiguous refutations of economic theory imply that we learn nothing from statistical analysis? I would argue definitely NO, we can learn a lot from statistical analysis. But what we cannot do is "test" theories with statistical tests. Does it also mean that we cannot refute economic theories? Again, I would argue NO. Refutations of "theoretical" propositions result from demonstrations of logical error, and one can also demonstrate the irrelevance of a logical argument to a contemporary problem due to the inapplicability of the theory to the situation to be examined because one or more of the various subsidiary assumptions that make up the network of statements that constitute a theoretical construction might not hold.

Bryan Caplan's response to the question is to say,

I can't meet Russ's challenge. But his challenge is excessive for two reasons:

1. Ending a controversy and creating a consensus almost always takes dozens, if not hundreds, of empirical studies. As well it should - virtually every specific study is open to reasonable doubt.

2. What exactly counts as "ending a controversy and creating a consensus"? Does every active researcher in the world have veto power? Or is convincing two-thirds of them enough? What if the "former opponents" just quietly give up rather than admitting error?

Having made these points, Caplan modifies the Roberts question to ask:

Name a body of empirical work that is so well done, it won over two-thirds of active researchers and induced half of the unconvinced to quietly give up or recant?

His answer to this question is

By this standard, the obvious response for economists is behavioral economics. I don't know any economist under the age of 40 who denies that there are major exceptions to the standard rational actor model. Many older economists are unconvinced, but they don't publish much about it.

A response to the Caplan example could be that while no one argues that exceptions to the standard rational actor model cannot be found in the lab, there are pressures in the real world that means these exceptions are not of great significance. As Steven D. Levitt and John List have put it,

Perhaps the greatest challenge facing behavioral economics is demonstrating its applicability in the real world. In nearly every instance, the strongest empirical evidence in favor of behavioral anomalies emerges from the lab. Yet, there are many reasons to suspect that these laboratory findings might fail to generalize to real markets. We have recently discussed several factors, ranging from the properties of the situation — such as the nature and extent of scrutiny — to individual expectations and the type of actor involved. For example, the competitive nature of markets encourages individualistic behavior and selects for participants with those tendencies. Compared to lab behavior, therefore, the combination of market forces and experience might lessen the importance of these qualities in everyday markets.

I think the thing to keep in mind is that a single result will not settle a question. But a series of results giving the same answer to the same question involving different data sets, different time periods, different countries, different approaches to answering the question etc will do so. I think what convinces people as the rightness of a result in the accumulation of evidence in its favour.