Bryan Caplan declines to join the Pigou Club

Over at EconLog Bryan Caplan explains why he is not a member of the Pigou Club. He gives two reasons for his non-membership. The first, public choice arguments. He notes, correctly, that

Once you give the government the power to tax things that it calls "negative externalities," it's got an excuse to tax almost anything, and will probably use it eagerly. Personally, I've noticed a near-perfect correlation between stuff that economists think is bad and stuff that they think has negative externalities.

The second is based on the second reason given by Greg Mankiw for not being a member of the club. Mankiw writes,

You recognize the externalities but you don’t think the government should try to respond to them. You are such a believer in small government that you are willing to live with inferior economic outcomes, such as pollution and congestion.

Caplan's response is based on the idea that Pigovian thinking can be used to justify totalitarianism. Caplan writes,

And contrary to Mankiw, you don't have to be a believer in small government to take this position. Anyone who thinks that speech should not be taxed no matter how offensive it is takes this position. Anyone who thinks that extremely unpopular religions should not be taxed takes this position. Indeed, if you think that you should be able to marry whoever you like, no matter how much your parents hate them, you don't really belong in the Pigou Club.

I'm not going to say that Pigovianism is inherently totalitarian. But I will say that if intolerant preferences are widespread, then Pigovian thinking justifies totalitarianism. There's no denying it: If most people are horrified by the sight of an unveiled woman, then Pigovian logic requires a massive tax on visible female faces.

My view is that people have a right to create all sorts of negative externalities, and other people basically just have to live with them. Unhappy campers have a right to complain about the externalities, refuse to associate with those who create them, or buy a large bloc of land and require visitors to abide by their rules. But they have no right to pass laws to do anything about most externalities. For example, even though I think education has important negative externalities, I don't want to do anything more than eliminate government subsidies for education.

This does not mean, of course, that all negative externalities should be legal. Murder creates externalities, and I'm firmly against its legalization. But the right place to draw the line, in my view, is at physical trespass.

Of course, "physical trespass" itself requires line-drawing, because even the act of breathing shoots atoms at unconsenting strangers. So if someone wants to use a Pigovian approach to e.g. air pollution, I grant that the externality might qualify as a physical trespass. To this much narrower class of externalities, I'm open to Pigovian solutions. But until Pigovians forthrightly acknowledge that the right solution to most negative externalities is tolerance, not taxes, I'm not going to join their club.

So isn't the basic problem here, What is an externality? And of this group, which of them justifies government action?