BERL rejects criticism of study (updated x4)

Or so says Adrian Slack. (But then he would, wouldn't he) From the BERL website comes this response to Burgress and Crampton. And I would have to say I'm somewhat underwhelmed by it, but here it is anyway:

BERL’s recent study into the Costs of Harmful Alcohol and Other Drug Use has prompted some criticism from some quarters. In particular, economists Crampton and Burgess have written a critique of our report. BERL soundly rejects the criticism contained in the Crampton and Burgess’ critique and stand by the validity of our work.

[...]

BERL freely accept comment and debate on our publicly released reports. The project brief for this study was focussed on providing detailed information on the costs of alcohol and other drug abuse to New Zealand society. Measurement of benefits was clearly outside the scope of the project. We cannot accept criticism for not covering issues that were outside the project’s terms of reference.

Further, we totally reject suggestions that we adopt assumptions to provide solutions favourable to the client. BERL’s reputation has been built over 50 years of existence and our message is clear: BERL does not write ‘reports to order’. As is the case with this study, where robust evidence is not available, we adopt conservative assumptions or ‘average’ values for parameters.

The Crampton and Burgess critique is fundamentally flawed in that it:

* misinterprets the study’s brief and, on this basis, employs an inappropriate framework for its analysis
* makes some simple, factual errors about our method and the information used
* uses assumptions with a cost-deflating bias, reflecting their own world view.


The project brief

BERL’s report states up front that it is a cost study and uses an international methodology developed by the WHO. The Ministry and Health and ACC’s purpose for the study was, to “[a]ccurately quantif[y] costs… pertaining to alcohol and other drug related harm”.

Despite this information, Crampton and Burgess misinterpret the project’s purpose. They select a framework that is inconsistent with the purpose and proceed with their analysis on the basis of an unsound premise. They argue for the case that all drinking is a rational choice, but take a middle road assuming that for all drinkers the individual benefits offset private costs. They then assume that the majority of costs considered in the BERL study are fully borne by the individual and net out, and assert that these costs are not relevant to policy.

The project brief was not to assess benefits, nor to provide policy analysis. The brief was to quantify the costs of harmful use of alcohol and other drugs using an internationally recognised method. Indeed, the initial request for tender to which BERL responded, stated “You shall complete a cost analysis using an accepted framework and deliver a report on the costs of alcohol and other drug abuse to New Zealand.” Further, the WHO International Guidelines was provided as an example of an “accepted framework”.

The result: a clearly written report soundly based in economic theory.

The Ministry of Health and ACC, as part of good research commissioning practice, required the near-final report be sent for peer review. Australian economists and international experts in this field, Professor David Collins and Professor Helen Lapsley, peer reviewed the report. The reviewers stated “the report is a well-researched report, soundly based in economic theory and quite clearly written…. our overall conclusion that the research has certainly been conducted at a very acceptable level.”

Peer review was also sought from relevant government departments, so they had the opportunity to ensure that the assumptions and use of data relating to their sectors were appropriate.

BERL carefully considered and incorporated the reviewers’ comments into the final report as appropriate. The reviewers noted that “there is often no uniquely acceptable methodology. Frequently the methodology is to some extent determined by issues of data availability. The report provides sufficient information on methodology and data sources for readers to make their own judgments about the quality of particular sub-estimates.”

(A selection of) Their errors

Crampton and Burgess make some simple, but substantial, factual errors. As an illustration, Crampton and Burgess incorrectly cite and misinterpret a New Zealand study on the level of unemployment caused by harmful alcohol use.

They state, “Rayner et al show… that our overall unemployment rate is 10 percent higher than it otherwise would be: in other words, the difference between 4% and 4.4% unemployment”. Rayner et al (1984, p48) clearly state, “For a given employment rate (say 66%) it was assumed that alcohol consumption caused the reduction in paid employment from the national rate (76%) to the given rate.” That is, a 10 percentage point difference, i.e. between 24% and 34% unemployment, which is consistent with the figure we use. This simple, but substantial, error cuts BERL’s social cost estimate by over $830 million (or almost 30 percent of the tangible costs of harmful alcohol use).

A second example of where Crampton and Burgess have got it wrong is that they do not correctly count insurance costs. They assert that BERL double counts insurance costs and that a “tallying of costs should include either the insurance costs or the amounts paid out: not both”. An insurance company’s total costs include both its overheads (fixed costs) plus its claim payouts (which vary with the level of claims). The BERL study correctly, and sensibly counts, the share of insurance companies’ overheads and the actual cost of property damage due to harmful alcohol use.

Their assumptions: a cost-deflating downward bias

Crampton and Burgess use a range of cost-deflating assumptions. For example, they assume that New Zealander drinkers would fare worse in the labour market, even in the absence of harmful drinking. That is, they assert that harmful drinkers would have worse labour outcomes even if they didn’t drink to a harmful level. This ignores the possibility that some extremely successful people may also be heavy drinkers. Indeed, some of these people may be even more productive if they did not suffer alcohol-related problems.

In the absence of robust evidence either way, BERL uses average figures based on New Zealand data. We take a middle line. In the absence of harmful alcohol use these people may have better or worse labour outcomes. We assume, for example, harmful drinkers would be like other New Zealanders – some with depression, some very successful, some who may have depression and be successful. We openly provide sufficient information for others, such as Crampton and Burgess, to draw their own conclusions.

Another example of cost-deflating assumptions, is that Crampton and Burgess assume that all productive resources can be fully and costlessly reallocated, workers bear almost all the cost and count only a “‘value added’ component of 10% as being an external cost”. In the short run, it is not necessarily the case that factors of production can be reallocated. For example, your computer does not keep writing by itself when you have a sick day. Nor may resources be freed up for others to use if a person turns up to work hung-over.

In the longer run, alcohol-related work absences or premature death may reduce the human capital available to the economy. This could be a particularly substantial effect for young people whose drinking impairs their learning, experience and job prospects. There may also be substantial complementarities between workers, so one worker’s alcohol-related absence disrupts other worker’s productivity. Rather than cherry-picking assumptions, we use average figures based as far as possible on New Zealand data and conservative assumptions.

BERL openly admit that we have no expertise in the medical field. Where necessary, BERL was guided by epidemiological experts and evidence as well as medical advice in adopting the necessary thresholds for ‘harmful’ use. The thresholds for harmful consumption, however, were only required for the attributable portion of some health, production and labour costs; all other costs were calculated based on data that show alcohol-related harm has actually occurred.

Their world view and considerations beyond the scope of the BERL study

Setting aside the issue that that the study’s brief did not include considering benefits, Crampton and Burgess assert that all harmful drinking and its consequences, such as becoming addicted, are an acceptable private choice. While a large proportion of the population drink occasionally and moderately, using alcohol in a non-harmful way, many do not. For example, more than half (54.1%) of young New Zealand males admit to drinking large amounts (six or more standard drinks) on a typical drinking occasion, and almost 50% drink three or more times a week.

BERL’s Dr Nana has publicly stated that Crampton and Burgess “had a different world view… that consumers are rational in their decisions about how much alcohol to drink”. But we would suggest that it is nonsense to argue that a drunk driver who wraps themselves around a power pole has made a fully informed, rational choice that is consistent with their long-term welfare an should be of no concern to society.

The following paragraph from Crampton and Burgess is informative. “The economic literature on alcohol and drug use contains a number of results which confound BERL’s assumptions: addiction has been found to have rational foundations, and alcohol and drug abuse tends to be a symptom of other problems; moderate drinkers, many consuming quantities above the lower bound of BERL’s harmful range, earn more and live longer; alcohol saves many more lives than it takes and has health benefits well beyond the point where BERL says harm starts and all benefits stop.”

The ‘result’ that “alcohol saves many more lives than it takes” is an assertion that requires evidence. And the idea that addiction has rational foundations clearly indicates a model view of a consumer that would be at variance others views, including those of some economists. Such a value judgement would not have been appropriate for an independent study such as ours.

The BERL study provides sufficient information for readers to make their own judgment about the costs of harmful alcohol use; it is for the reader to make their own judgments about the benefits of harmful alcohol use (something outside the scope of the project). It is for policy-makers, not BERL, to judge what set of values they use and whether they share Crampton and Burgess’ world view.

Many of these issues have been discussed before, but a couple of quick comments: As to the "[...] simple, but substantial, factual errors", adding in these unemployment and insurance costs increases the Burgess and Crampton costs by a whole $36 million, not much given the amounts being talked about in the report. For more on this point see Eric's post, Errata, at Offsetting Behaviour. That posting makes some other relevant points.

Also Slack writes

The project brief was not to assess benefits, nor to provide policy analysis.

They are not providing policy analysis? Why does he think the Ministry and ACC asked for the study if not for policy purposes? What other purpose could it have? And note the use of the report by the Law Commission. If the BERL report isn't providing policy analysis, why haven't they not pointed out this fact given that the Law Commission seems to very clearly want to use it for policy analysis purposes?

Update: BK Drinkwater notes the Hurly-BERLy: BERL Responds.

Update 2: Kiwblogs comments that BERL responds.

Update 3: Ganesh Nana repeats much of the above material in a piece in the Otago Daily Times.

Update 4: BK Drinkwater comments on the Hurly-BERLy: Nana Responds.

What is missing from the defence is this: an explanation of just why it is that people who drink >40g/day of alcohol are assumed to derive no benefit at all from their drinking; and why this assumption justifies counting these individuals' privately-carried costs as costs to society in general.