For nearly a decade, lawyer Adam Mur phy ’10 advocated for a man who had been incarcerated for more than 45 years to be released by the parole board. The man was 66 and had suffered several bouts of cancer.
“He had completed every program available to him, was in failing health, had expressed sincere remorse, and had long addressed the conditions that led to his crime,” says Murphy, who is Assistant Counsel at the NAACP Legal Defense Fund but was advocating in a volunteer capacity. “He quite clearly did not pose a risk to public safety.”
To apply for parole, those incarcerat ed in New York must answer questions from a risk-assessment tool known as the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions). The tool is one of many examples of the criminal justice system relying on machine-learning tools designed using artificial intelligence to help make predictions about human behavior and assist in crucial deci sion-making by authorities.
Questions asked by the assessment tool include: Were you ever suspended or expelled from school? How many of your friends/acquaintances have ever been arrested? “People in communities of color are often more likely to say ‘yes’ to those questions because they are overpoliced, which leads to the fallacy that a person coming from that com munity is more likely to be ‘high risk.’ Such a result violates the basic tenet articulated by the U.S. Supreme Court that people should be punished for what they do, not who they are or where they are from,” Murphy points out.
Other questions would receive a “yes” response from those who are economi cally disadvantaged, which is correlated to race. For example: How often do you barely have enough money to get by? How often have you moved in the last 12 months? Do you have a working phone?
The tool poses no questions about age or medical history. And the algorithm used to generate an evaluation of each person is not publicly disclosed.
Some of the risk metrics that the COMPAS produced for the man—which is supposed to predict the likelihood of a person committing other crimes—went up from one parole hearing to the next, though he had not incurred any disciplinary infractions in prison during that time, says Murphy, whose work focuses on issues such as parole, bail, and death row. COMPAS also found the man to be at high risk of using drugs again, though he hadn’t used them in decades.
“Algorithms take on a facade of objectivity, but they often reproduce and even amplify the racial disparities that define the criminal legal system,” Murphy says. “And at the same time, studies demonstrate that many of these tools have limited predictive utility.”
After 14 tries, the man was finally released at the age of 75, after having served 52 years.
The criminal justice system has come to rely on AI for many tasks. In addition to its use by parole boards, similar assessment tools have been employed to help make decisions about granting bail, determining sentencing, and analyzing where people convicted of sex crimes can live after release. The technology has also been employed to help police predict where crime will occur. But experts say these tools, in use for decades, are often deeply flawed and reinforce bias already in the justice system rather than eliminating it.
Though experts have pointed out shortcomings in these tools for years, Murphy contends there has been little progress in improving them. “These systems are perpetuating and exacerbating outcomes foundationally correlated to race and class, and yet lawmakers and decision makers either shrug with apathy or actively resist efforts at incorporating empirically validated tools,” Murphy says. “An obvious impediment to change is that people who are incarcerated are disfavored and don’t have a lot of political capital. Rather than making us safer, unreliable and racially biased risk-assessment instruments are antithetical to public safety.”
Simon Hoellerbauer, a post-doctoral fellow in the new multidisciplinary program Data Science and Society at Vassar, cites a widely used risk-assessment tool that was studied in 2016 by ProPublica, a nonprofit news organization specializing in investigative journalism. Using Florida data, ProPublica found that only 20 percent of the people predicted to commit violent crimes by the tool actually went on to do so.
“We know judges have biases, because we all have them, and the program was trying to get around that,” he says. He noted that even when designers omitted information about race, the sentencing recommendations were still found to be biased. “Any machine-learning model learns from the data it receives. If the data are flawed or encode systematic bias, the output you get will be biased. If we use that data uncritically, we will perpetuate the current inequities.”
Technology companies have been working to address these issues in their design process and by hiring ethicists who specialize in machine learning, “but their motivation is profit-based, I believe,” Hoellerbauer says.
To help Vassar students think through these issues, Data Science and Society looks at the societal impact of data science and its application to real-world problems. In 2022, Data Science and Society invited data scientists from academia and industry to present a series of lectures on the ways data science impacts society. The talks explored a variety of issues related to machine learning, including data-based surveillance during COVID, fairness and equity in data science, and the subject “What is scientific about data science?”
“We need to teach people how to think critically about machine learning and AI,” Hoellerbauer says. “We can’t just accept results at face value because a machine said it.”