Using online experiments in business

Economist Susan Athey talks to NPR:

Did you know that every time you do a search on Google or Bing, you are improving the quality of the search engine? The more people click on a search advertisement from a clothing company or on a link on an online news story, the more prominently it is displayed for the next consumer. And the firms constantly experiment to get things right. They watch what consumers do and adapt their products in response to the results of their experiments.

But designing the right experiment is difficult. To see just one example, consider spam. An e-mail provider wants to eliminate spam from your inbox. It is nuisance for all of us. That company might test out a new way to filter spam. The filter may do a great job in short-term experiments, where the spammers don't have a chance to respond. But once the new filter is introduced in practice, spammers may find a way around the filter. So, that means that some legitimate e-mail is filtered out and the end result may be that you haven't solved the spam problem at all. You could very well end up worse than where you started, even though the experimental numbers looked great.

So, if you want to figure out whether a new product will work out the way you hope, you need to be able to anticipate how people will react to your innovation. That is, you don't just have to be a good statistician. It's not just about the numbers that come out of simple experiments; it is about predicting how people will react to the changes you make. You need to understand behavior and how to build models that reflect the choices we all make.

Unfortunately, our universities and business schools haven't figured out how to train students to do this kind of modeling and prediction. That is, we aren't preparing students to manage the new data-driven businesses. And let's face it: This is where our economy is headed, as consumers are spending more and more of their time online. Creating new jobs in this economy is a must, but making sure that the workforce is ready for the jobs where our growth is happening is more important. The good news is that the young people who do develop the talent and skills to capitalize on this opportunity will be in high demand, which puts them in a great position in today's tough economy.