For education, the positive externality is the benefits that accrue to me from your education. I think that those benefits tend to be pretty small. You get a higher income, and most of those benefits flow to you. I get some of the benefits, because you are more likely to pay taxes and less likely to require government transfers, so that my tax obligations can be correspondingly reduced.Kling has a point. It is not clear that all externalities from education are positive, and even if there are positive externalities its not obvious that there are not also negative ones as well. So its not clear what the net externality effect is. Thus it is not clear that a subsidy, rather than a tax, on education is is the best policy response to deal with education's externalities.
You also get the consumption benefits of your education. I personally don't benefit from your experiments with drugs, sex, rock'n'roll. Nor do I particularly care that you take a class in art appreciation or get tickets to your school's basketball games.
Finally, you are supposed to be a better citizen because of education, and I should be happy about that. But if what you learn is that profits are evil, man-made global warming is beyond doubt, and it is wrong to question that gender differences are socially constructed, then from my perspective your education is not making you a better citizen.
If the higher income that you get from education is due to its signaling effects, then that is a classic negative externality. The investment in the signal is wasteful, and your investment forces others to make a wasteful investment.
On the whole, the case for taxing education rather than subsidizing it is really quite plausible. It is counter-intuitive, perhaps, but the case for free trade is also counter-intuitive to most people.
As to his "tax obligations" argument, it looks like a pecuniary externality. I pay an extra dollar in taxes and so Arnold pays a dollar less, things cancel out. It looks like the effect here is via the budget constraint rather than the utility function, so why do we care?