There have been many reasons put forward for market failure. Aspects of these theories can be traced back to the beginnings of economics, but the modern formulations were laid down by Paul Samuelson, James Meade, Francis Bator and others in the 1950s. From then until the 1970s the general consensus would have been that governments should provide at least a few basic public goods, such as national defence, but that markets do the best job of providing most goods and services.
Clearly market don't work perfectly, if they did firms would not exist, for example. But that's not the point, markets don't have to be perfect, they just have to be better than the alternative. So market failure supporters can't just show that markets fail when compared to some hypercritical perfect competition model, they have to show they fail when compared to some real, actually workable alternative. And they have to show there solution is better than any market solution.
Since the 1970s a new set of reasons for market failure based around the idea of asymmetric information has arisen. Arguments like Akerlof's famous example of the "lemons problem" suggest that market failure is widespread. Few if any markets don't suffer from at least some asymmetry in information. According to market failure supporters this will result in mutually beneficial trade not taking place and in the most extreme situations markets collapsing altogether.
But does asymmetric information really cause market failure or does it in fact cause markets to exist? The great Austrian economist Friedrich Hayek would surely argue the latter, markets are the answer to the problem of asymmetric information.
For Hayek markets eliminate the asymmetry by revealing relevant aspects of information in market prices, but not so for the Akerlofs of this world. As Cowen and Crampton (2002: 5) put it "[t]he Canadian plumber's knowledge of substitutes for copper piping influences the French electrician's choice of home wiring through its effect on the market price of copper. For the market failure theorists, however, asymmetry cannot be overcome by exchange precisely because the unequal distribution of information interferes with mutually beneficial exchange." The fundamental point of markets for Hayek is that they utilise dispersed, asymmetric, knowledge in such a manner as to align production plans with consumption plans.
This alignment takes place in three ways: "First, ex ante, prices transmit knowledge about the relative scarcities of goods to various market participants so they may adjust their behavior accordingly. If the price of a good goes up, this informs economic actors that the good has become relatively more scarce and that they should economize on its use. For this reason, participants in the market have an incentive to include the knowledge contained in prices in their actions over time. Second, the price system serves the ex post function of revealing the ultimate profitability or unprofitability of economic actions. Prescient entrepreneurship (in the broad sense of the term) is rewarded with profits; errors are penalized by losses. Market prices, therefore, not only motivate future decisions by conveying information about changing market conditions, but also help market participants evaluate the appropriateness of past market decisions and correct erroneous ones. Seen in this light, the market process is a matter of dynamic adjustment. What is it adjustment to? It is, in effect, adjustment to the gaps between a static equilibrium of universal satisfaction and the many departures from this model that are present in the real world. Each of these gaps between the counterfactual and the factual represent a profit opportunity. Price information is also motivation for profitable real-world adjustment, over time, to the profit opportunities of a particular place." (Boettke 1997: 25-6)
New markets can develop to deal with asymmetric information, if you don't know the condition of a second hand car you take it to the AA and get a report on what's right or wrong with it. If you don't know what maybe wrong with a house you want to buy you can get that checked as well. These services have developed to deal with the asymmetric information in the used car and housing markets. The basic point is that markets constantly innovate, informational asymmetries create demand for product assurance from which profits can be made by alert entrepreneurs.
Klein ("The Demand for and Supply of Assurance" in Cowen and Crampton 2002) is one paper which looks at the institutions that have arisen in the market system to mitigate problems of information, quality and certification. The lesson to be drawn here is that while "[m]arket failure theory predicts massive deadweight losses accruing from the trades that fail to take place because of informational asymmetries. In reality, alert entrepreneurs see deadweight losses as potential profits to be earned by removing the impediment to trade." (Cowen and Cramption 2002: 12-3)
On the empirical front, in a recent paper ("Lemons hypothesis reconsidered: An empirical analysis", Economics Letters 99:3 (June): 541-544) in Economic Letters, Arif Sultan sets out to test this idea. He argues that used cars, if inferior to new cars, would require higher maintenance expenditures. His paper tests the hypothesis that there is no difference in the average maintenance expenditures required for cars acquired used and those acquired new.
The results he gets show that there is no evidence that cars acquired used required more maintenance expenditures than those of a similar age acquired new. The conclusion to the paper reads, in part,
The purpose of this paper was to examine the difference in the quality between cars acquired used and those acquired new. I measured the quality of the car by using maintenance expenditures incurred on a car [... ] I found that cars acquired new required the same maintenance expenditures as those acquired used, all else being equal, implying that cars acquired used are of same quality as cars acquired new of a similar age. If cars acquired used were of lower quality, they would have required more maintenance expenditure.In another test of adverse selection, Eric Bond studies the used car market. His paper is "A Direct Test of the “Lemons” Model: The Market for Used Pickup Trucks." The American Economic Review 72:4 (September): 801-4. Bond concludes that trucks purchased in the used market required no more maintenance than other trucks of similar age and mileage. If the used trucks market suffered from a lemons problem, truck owners would keep high quality older trucks while selling lemons on the used truck market. Yet data collected by the US Department of Transportation reveal no significant differences in maintenance costs between trucks kept by their original owners and trucks sold on the used car market. In the market used as exemplar by Akerlof, Bond finds no empirical support for the lemons hypothesis.
Cawley and Philipson ("An Empirical Examination of Information Barriers to Trade in Insurance." The American Economic Review 89:4 (September) 827-46) test the implications of the asymmetric information model in the term life insurance market and find no evidence of market failure. The asymmetric information theory would predict increasing unit prices for insurance purchases, quantity-constrained low-risk individuals, and prohibitions on the purchasing of multiple small contracts to prevent arbitrage. But Cawley and Philipson find robust evidence for decreasing unit prices, for low-risk individuals holding larger policies than high-risk customers, and for frequent multiple contracting.
Another test for asymmetric information comes from the Chiappori and Salanie ("Testing for Asymmetric Information in Insurance Markets." Journal of Political Economy 108:1 (February): 56-78) study of the French market for automobile insurance. The results are similar to those of Cawley and Philipson. Where asymmetric information models predict that, among observationally identical individuals, those with more coverage should have more accidents, Chiappori and Salanie find no correlation between unobserved riskiness and accident frequency. And there are other similar studies which also cast doubt on the empirical validity of the asymmetric information story for market failure.
I have discussed just one reason for market failure and there are many others. But if you look at the books noted above you will see that these other stories don't hold up any better than the asymmetric information story - Coase shows that lighthouses were privately supplied, Steven N. S. Cheung shows that the moral hazard problems involved sharecropping have market solutions, Liebowitz and Margolis undermined the technology lock-in arguments, Steven N. S. Cheung shows that beekeepers and orchard owners contracted routinely and so on. My basic point is that the standard market failure stories don't hold up well either theoretical or empirically and there are good reasons for thinking that even if there is market failure, market solutions are the best, or at least, the least imperfect solution.
Update: A couple of extra references from Eric Crampton: Propitious selection in insurance by David Hemenway. Abstract:
The theory of propitious selection suggests that there are risk-avoiding personalities who both take physical precautions and buy financial security (insurance). Conversely, there are risk seekers who tend to do neither. Survey evidence is presented that is consistent with the theory. Individuals who obtain motor vehicle liability coverage are less likely than others to drink-and-drive, and are more likely to engage in health-beneficial (risk-avoiding) behaviors. Propitious selection may be a general phenomenon promoting favorable selection in many real world insurance markets. (Emphasis added.)Does Propitious Selection Explain why Riskier People Buy less Insurance by Philippe De Donder and Jean Hindriks. Abstract:
Empirical testing of asymmetric information in the insurance market has uncovered a negative correlation between risk levels and insurance purchases, rather than the positive correlation predicted by the standard insurance theory. Hemenway (1990) proposes an explanation for this negative correlation, called “propitious selection”. He argues that potential insurance buyers have different tastes for risk and that “individuals who are highly risk avoiding are more likely both to try to reduce the hazard and to purchase insurance” (p. 1064). Chiappori and Salanié (2000) also suggest that this line of argument, which they call “cherry picking”, may explain the observed negative correlation.
In this paper, we show that the propitious selection argument does not imply negative correlation between risk levels and insurance purchases, because it fails to take into account the supply side of the insurance market. To illustrate this claim, we provide a model where, although we assume that individuals differ in risk aversion and that the more risk averse individuals exert more precaution and buy more insurance, we end up with a positive correlation between risk and insurance purchases at equilibrium. The reason is that, in any separating equilibrium, the more risk averse individuals face insurance overprovision which, combined with moral hazard, increases their risk relative to the less risk averse individuals. To obtain the negative correlation between risk and insurance purchases, one further needs the extra condition of decreasing marginal willingness to pay for the less risk averse individuals. Finally, we find that propitious selection has profound policy implications for social insurance.