Macroeconomists do it with models

John Taylor has an interesting post - Measuring the Impact of the Stimulus Package with Economic Models - at his Economics One blog on the problems of evaluating the effects of the stimulus package of 2009 in the US. Taylor writes,

It's been nearly a year since the stimulus package of 2009 was passed. Unfortunately most attempts to answer the question “What was the size of the impact?” are still based on economic models in which the answer is built-in, and was built-in well before the stimulus. Frequently the same economic models that said, a year ago, the impact would be large are now trotted out to show that the impact is large. In other words these assessments are not based on the actual experience with the stimulus. I think this has confused public discourse.

In other words, should we be surprised that we see the rabbit come out of the hat given we saw it carefully put it there in the first place? Taylor continues,

An example is a November 21 news story in the New York Times with the headline “New Consensus Sees Stimulus Package as a Worthy Step.” Authors Jackie Calmes and Michael Cooper write that “the accumulation of hard data and real-life experience has allowed more dispassionate analysts to reach a consensus that the stimulus package, messy as it is, is working. The legislation, a variety of economists say, is helping an economy in free fall a year ago to grow again and shed fewer jobs than it otherwise would.”

As evidence the article includes three graphs, which are reproduced on the left of the chart below. Each of the three graphs on the left corresponds to a Keynesian model maintined by the group shown above the graph. All three graphs show that without the stimulus the recovery would be considerably weaker. The difference between the black line and the gray line is their estimated impact of the stimulus. But this difference was built-in to these models before the stimulus saw the light of day. So there are no new hard data or real life experiences here.

Taylor then makes the point that if the rabbit isn't put in the hat, it doesn't come out.

In fact, a number of other economic models predicted that the stimulus would not be very effective, and, using the same approach, those models now say that it is not very effective. To illustrate this I have added two other graphs on the right-hand side of the chart which did not appear in the New York Times article. The first one is based an a popular and well-regarded new Keynesian model estimated by Frank Smets, Director of Research at the European Central Bank, and his colleague Raf Wouters. Focus again on the difference between the black and the gray lines, which is what is predicted by that model, as shown in research by John Cogan, Volker Wieland, Tobias Cwik, and me. Note that the impact is very small. The second additional graph on the right is based on the research of Professor Robert Barro of Harvard University. As he explained last January, “when I attempted to estimate directly the multiplier associated with peacetime government purchases, I got a number insignificantly different from zero.” So according to that research, the difference between the black and the gray line should be about zero, which is what that graph shows. So there is no consensus.

There seems to be at least two basic messages that follow from this: one, when using models to evaluate policy outcomes it is important to go beyond the use of just a few select models, and check to see if the outcome is robust across a number of different models. Second, it is now time to start looking at the direct impacts of the stimulus by looking at the hard data.