Organizations are increasingly using machine learning models to allocate scarce resources or opportunities. For example, these models could help a company review resumes to select interview candidates, or a hospital rank kidney transplant patients based on their chances of survival.
When deploying a model, users typically try to reduce bias to ensure that predictions are fair. This often involves techniques such as adjusting the features the model uses to make decisions or calibrating the scores it generates.
But researchers at MIT and Northeastern University argue that these fairness methods are not enough to address structural injustice and inherent uncertainty. In a new paper, they show how randomizing the model’s decisions in a structured way can improve fairness in certain situations.
For example, if multiple companies use the same machine learning model to rank interview candidates deterministically without randomization, one qualified person may end up getting the lowest ranking for all positions, perhaps because of how the model evaluates the answers provided in an online form. Introducing randomization into the model’s decisions can prevent one qualified person or group from always being denied a scarce resource, such as an interview.
The researchers found that randomization could be particularly beneficial when the model’s decisions involve uncertainty or when the same group consistently makes negative decisions.
They present a framework that can be used to introduce a certain amount of randomness into the model’s decisions by allocating resources through a weighted lottery. This method, which individuals can adjust to their own circumstances, can improve fairness without compromising the model’s efficiency or accuracy.
“Even if we could make fair predictions, should we strictly determine the social distribution of scarce resources or opportunities based on scores or ranks? As things scale and more and more opportunities are determined by these algorithms, the inherent uncertainty in these scores could be amplified. Our results suggest that some form of randomness may be needed for fairness,” said Shomik Jain, a graduate student in the Institute for Data, Systems, and Society (IDSS) and lead author of the paper.
Jain worked with Kathleen Creel, an assistant professor of philosophy and computer science at Northeastern University, and lead author Ashia Wilson, the Lister Brothers Career Development Professor in the Department of Electrical Engineering and Computer Science and a senior scientist in the Laboratory for Information and Decision Systems (LIDS). The research will be presented at the International Conference on Machine Learning.
Consider billing
This study builds on a previous paper that explored the harm that can occur when researchers use deterministic systems at scale. They found that using machine learning models to deterministically allocate resources can amplify inequalities present in the training data, which can reinforce bias and systematic inequalities.
“Randomization is a very useful concept in statistics, and fortunately it satisfies our requirements for fairness from both a systematic and an individual perspective,” says Wilson.
In this paper, they explore the question of when randomization can improve fairness. They structure their analysis around the ideas of philosopher John Broome, who wrote about the value of using lotteries to award scarce resources in a way that respects the claims of all individuals.
A person’s claim to a scarce resource, such as a kidney transplant, may stem from merit, entitlement, or need. For example, Wilson explains, everyone has a right to life, and their claim to a kidney transplant may stem from that right.
“Given that people have different claims to this scarce resource, fairness would require that we respect all of the claims of individuals. If we always gave the resource to the person with the stronger claim, would that be fair?” says Jain.
Deterministic allocations of this kind can lead to systematic exclusion or exacerbate patterned inequality, which occurs when receiving one allocation increases the likelihood that an individual will receive that allocation in the future. Machine learning models can also make mistakes, and deterministic approaches can lead to repeating the same mistakes.
Randomization can help overcome these problems, but that doesn’t mean that every decision the model makes should be equally random.
Structured randomization
Researchers use weighted lotteries to adjust the level of randomization based on the amount of uncertainty involved in the model’s decisions. Less certain decisions should incorporate more randomization.
“In kidney allocation, planning is typically done based on life expectancy, which is very uncertain. If two patients are only five years apart, it’s much harder to measure. We want to use that level of uncertainty to adjust the randomization,” Wilson says.
Researchers used statistical uncertainty quantification methods to determine how much randomization is needed in different situations. They showed that corrected randomization can lead to fairer outcomes for individuals without significantly affecting the utility or effectiveness of the model.
“There is a balance to be struck between overall utility and respecting the rights of individuals to access scarce resources, but often that balance is relatively small,” says Wilson.
However, researchers emphasize that random decision-making does not improve fairness and may even harm individuals in some situations, such as in the criminal justice system.
But there may be other areas where randomization can improve fairness, such as college admissions, and the researchers plan to explore other use cases in future work. They also want to explore how randomization can affect other factors, such as competition or price, and how it can be used to improve the robustness of machine learning models.
“We hope that our paper is a first step toward showing that there may be benefits to randomization. We are providing randomization as a tool. It is up to all stakeholders involved in the allocation to decide how much they want to do. And of course, how they decide that is another research question,” says Wilson.
