A few years ago, managers working for Amazon had a brainwave. If they created a computer programme that could automatically pick which job candidates to recruit, it would help the business save money and grow more quickly. That was the plan, anyway. A team of engineers got to work writing software that analysed all the CVs and job applications submitted to the company over the past decade — and then used this data to work out which qualifications were most likely to mean a candidate had been recruited in the past.
Armed with that analysis, the computer system could then look at any new job applicant’s CV, and predict whether or not they’d be a good fit for the firm.
It all worked perfectly, except for one tiny glitch: it turned out that Amazon’s clever new software was totally and utterly sexist.
It marked down any CVs that included the word “women’s” — as in “women’s debating champion”. It downgraded graduates of some all-women colleges. And candidates who used macho verbs like “executed” and “captured” on their CVs were picked more often. In other words, the system overwhelmingly recommended hiring male candidates rather than female ones — and Amazon quickly scrapped the whole endeavour.
So what went wrong? The problem was that by relying on data about past recruits, the algorithm had learned that men tended to be hired far more than women, so it simply repeated (and exacerbated) that historic bias.
This is by no means a one-off case. Advances in technology mean that complex decisions are increasingly being delegated to machines, on the assumption that this will lead to fairer and more objective results. Unfortunately, it doesn’t always work out that way.
Last week, academic journal Science published a paper showing that a software system widely used in American hospitals has been “systematically discriminating” against black people. The software was supposed to work out which patients should receive additional healthcare resources, such as close attention from doctors.
However, the authors of the paper found that only 18 per cent of patients the algorithm had picked to receive extra healthcare resources were black — but if the algorithm had been unbiased, this proportion would have been closer to 50 per cent. Many thousands of patients lost out as a result.
You find the same troubling issue in the US criminal justice system, which is using sophisticated algorithms to help decide who to lock up and who should be granted parole.
Read More
It’s not daft to think that machines could be more dispassionate than human beings when it comes to making decisions like this. After all, if software could analyse historic data about recidivism, it could be harnessed to predict whether or not someone might reoffend in future.
But again, it’s not so easy to escape the taint of human discrimination. Take the software system known as COMPAS, which is used in multiple American states to help judges decide sentences and also determine whether to let someone out of prison on bail. According to one in-depth study, by investigative journalism group ProPublica, COMPAS was “particularly likely to falsely flag black defendants as future criminals” while “white defendants were mislabelled as low-risk more often than black defendants”.
The problem — as in the other examples I’ve mentioned — is that the computer system was basing its decisions on historic datasets that reflect human biases and prejudices.
You find the same pattern with some of the digital services we use in our daily lives. Harvard professor Latanya Sweeney typed more than 2,000 first names into Google, and found that ones most popular among African-Americans (like Darnell and Lakisha) had far greater probability of being served up adverts related to criminal record searches and arrest history databases.
According to Professor Sweeney, the fact that Google’s advertising algorithms appeared to have been skewed in this way exposes an underlying “racial bias in society”— which the search engine was simply reflecting. What links all these stories is the way technology often turns out not to be making the world fairer or more objective. On the contrary, algorithms and ever more intelligent computer systems are frequently revealed to be repeating — and automating — human biases.
As New York University professor Kate Crawford puts it: “What concerns me most is the idea that we’re coming up with systems that are supposed to ameliorate problems, but that might end up exacerbating them.”
The issue is that people tend to trust computers and mathematical models because we assume they’re free from human bias. Unfortunately, it’s much harder to escape our foibles than we like to think.
Now, there may well be ways to mitigate these issues. French President Emmanuel Macron recently announced that his country will publish full details of all algorithms used by his government to allocate resources or determine between different options, in the hope that transparency will make it easier for people to spot biases and fix problems.
Meanwhile an American think tank specialising in technology policy has proposed that the public should always be given the right to appeal against decisions made by computer systems.
However, the main change we need to see is in our own mindset. Technology will never fully save us from some of the darkest aspects of human behaviour, like racism and sexism, and it’s naively utopian to hope that letting machines make decisions will always lead to fairer or less discriminatory outcomes.
Ultimately, technology will always be in some way flawed — because its human creators are flawed. We can never quite escape that imperfection, and it’s daft to think we can. As Leonard Cohen once sang: “There is a crack in everything; that’s how the light gets in.”