Skip to content Skip to navigation

Susan Athey on how economists can use machine learning to improve policy

March 23, 2016

When Susan Athey answered a question about the impact of machine learning techniques on economics for the website Quora, the page attracted over 445,000 views within the week.

While it may seem surprising that such a technical topic attracts that much attention, it is a sign of the growing interest generated by Athey's recent work bridging the divide between the statistical techniques used by economists and the techniques computer scientists use to extract useful information from massive amounts of data.

The work is a natural extension for Athey. She’s a senior fellow at the Stanford Institute for Economic Policy Research and Economics of Technology professor at the Stanford Graduate School of Business who has worked with big technology companies.

While working at the search engine Bing in 2007, Athey was struck by how the supervised machine learning techniques that tech companies used to decide the placing of online advertisements, among other things, could be used by economists to evaluate policies.

"Supervised machine learning is basically about prediction," Athey says.

In the world of big data, where you have information on many factors that could potentially play a role in a desired outcome, supervised machine learning provides a systematic way of selecting which factors matter and in what way.

"You have some features (x's) and you try to predict (y's)” she says. “The innovations from machine learning have been to find really effective ways to do that, especially in an environment when there are lots of x's and you don't have a theory for exactly how the x's should predict the y's."

In these methods, part of the data is used to build a model and then the remaining data is used to assess how well the model works.

These techniques have immediate applications to policy problems where prediction is important. Economists are starting to apply them to questions such as who should be detained in jail pending trial and who can be released on bail. While some defendants can spend months in jail before being tried, a large share of those released on bail fail to appear in court. Machine learning technology can help judges predict who is likely to return for future hearings, Athey says.

"You don't want to keep someone locked up in jail when they could be out taking care of their family and earning money to feed their kids if they are very likely to show up for court," Athey says.

But these techniques currently do not uncover how changing one factor affects others --which is at the heart of evaluating policy impacts. Questions such as: Does raising the minimum wage reduce employment? Or, do smaller class sizes improve student outcomes?

Athey's work focuses on modifying supervised machine learning methods to be able to ask these types of questions. In one application, her method systematically explores the data in order to determine the population subgroups that are most likely to see policy impacts.

"Using machine learning methods I can estimate for whom the treatment effects are big and for whom they are small," Athey says. "Sometimes a treatment may not have a positive effect on average but it is very effective for certain subpopulations. If you don't have a way to document that, you might just reject the policy altogether."

A key challenge Athey addresses is ensuring that the effects the method finds are valid, not just the product of chance.  The potential payoff in fields like personalized medicine is substantial. For instance, the traditional way in which medical trials ensure that a drug's effect is not a chance result is by having researchers specify in advance which population groups they will examine. If, in the process, researchers discover that their drug works particularly well for a different group, that evidence is not admissible. They are required to design a new clinical trial for that specific group in order to have the drug approved for that population.

The methodologies Athey is developing would allow researchers to find these population groups in advance and avoid the need for a second medical trial.

"In some sense, these are methods that could open up the ability to discover what is in the data without the risk that you are going to end up with a false discovery," Athey says.