Better accountability mechanisms can help protect society’s most vulnerable from AI-based harms

Guest blogger Abeba Birhane explains how AI bias leads to discrimination against marginalised groups – and what to do about it

Last year, Shaun Thompson, a 38-year-old Black man, was wrongfully accused and detained at London Bridge station. Thompson was on his way home from a volunteer shift in South London when the Metropolitan Police’s facial recognition technology misidentified him as a suspect of an unspecified crime

Around the world, AI systems are increasingly integrated into critical social infrastructure mediating decision-making in law enforcement, education, healthcare, child welfare and everything in between. In the US alone, a recent analysis concludes that “92 million low-income people … have some basic aspect of their lives decided by AI”. 

But these systems can suffer from failures, inaccuracies and flaws that often result in worse outcomes for already vulnerable and marginalised groups, encoding and exacerbating historical and societal norms, stereotypes and biases. 

For example, facial recognition technologies used in policing are often less accurate on people of colour. In healthcare, meanwhile, an algorithm used by the NHS to allocate liver transplants was found to prioritise older patients, potentially resulting in increased waiting times for younger people. In the US, the health insurance conglomerate UnitedHealth Group was revealed to use algorithms to limit mental health care to patients who it deemed to be receiving too much therapy, while in Denmark, a decision-support algorithm used by child protection services was found to suffer from issues including information leakage, inconsistent risk scores, and age-based discrimination. Popular generative AI systems such as GPT-4 are also known to encode racism and negative stereotypes about groups such as African Americans, for example based on their dialect

For large institutions, organisations and government offices, cost reduction is usually the main attraction of integrating AI systems in decision-making processes. Oftentimes, however, these AI systems are integrated without rigorous testing and vetting

In short, current capital-driven development and adoption of AI tends to widen inequalities and diminish fundamental rights, while allowing wealth and power to accumulate in the hands of the few. Why is this the case?

Moving targets

Prominent but reductive narratives such as “bias in, bias out” would have us believe that all problems (and thus solutions) with AI systems stem from the data that they are trained on. And whilst it’s true that the quality, size and representativeness of training data to some extent impact the performance of algorithmic systems, that’s far from the whole story. 

Building an AI model, and automation by extension, by its very nature necessitates quantification, datafication and simplification. This might be less of an issue if we are building AI models for, say, wind power prediction or soil health assessment, as these can be quantified and measured relatively accurately. 

But most social, cultural, mental and political phenomena are socially and historically contentious, dynamic and often resist quantification. In other words, they are messy and ambiguous. By virtue of trying to quantify and model these phenomena, we reduce them to simplistic and static proxies. We also often bake in unexamined normative or capitalist assumptions. UnitedHealth Group’s suite of algorithms, for example, used financial (and not clinical) proxies to determine what treatment patients should be given access to. 

Importantly, the overarching objective of an AI model – whether that be optimising efficiency or accuracy – is a key factor in determining whether an algorithm ends up exacerbating injustice or benefitting those that need it. The harms encoded by proxies and assumptions are thus something that AI developers must contend with from the moment a new model is proposed.

AI accountability

Given these challenges and complexities, we need effective ways to unearth issues within opaque AI systems. We need to be able to scrutinise their performance and surface any bias, injustice or discrepancies, so that those who develop and deploy these technologies can be held accountable for any downstream harms their systems might cause. 

Algorithmic audits, which cover a range of processes including analysing an algorithm’s documentation, probing its workings, and testing outputs, represent a crucial mechanism for achieving this. Audits were critical in uncovering many of the AI-based harms mentioned in this piece. 

To become a viable mechanism for accountability, an audit should go beyond merely evaluating the performance of an algorithm. It should be:

  • Approached from a holistic perspective that places the tool being audited in a broader social context, including existing power and resource inequalities between, for example, those creating and implementing the AI system and those affected by it.
  • Designed with a focus on diagnosing (and mitigating, when possible) issues that are concrete and immediately affecting the public, especially people at the margins of society (as opposed to audits designed to protect the liability or reputation of AI developers and vendors).
  • Executed by independent auditors in a process that actively engages and incorporates feedback from civil society and rights groups, and people that tend to be disproportionately negatively impacted.
  • Grounded in a theory of change that outlines an ‘ideal’ outcome and the necessary steps required to progress from the current operations of an algorithm to that ideal. 

Algorithmic audits can only take us so far towards accountability, though. If the objective of a given AI system is to surveil, control or punish, then improvements to its accuracy or performance, or extensive audits, are not sufficient on their own. When deployed in public spaces, even an accurate facial recognition technology still diminishes privacy and rights to freedom of movement and assembly. Similarly, intrusive data harvesting by large social media and video streaming companies erodes privacy and threatens fundamental rights. 

To tackle the discrimination perpetuated by AI systems, then, we need meaningful and enforceable regulations alongside intentional auditing: regulations that ensure companies and governments face appropriate consequences when they fail to adequately protect the public. 

AI offers potential in every walk of life, but it is also failing the most vulnerable. As we continue to develop and discuss these technologies, let’s ensure that accountability is at the heart of the debate. 

Abeba Birhane is the founder and Principal Investigator of Trinity College Dublin’s new AI Accountability Lab (AIAL). Her 2024 Turing Lecture asked ‘Can we trust AI?’

The Turing Lectures bring together speakers from across the AI ecosystem covering a wide range of topics and perspectives. Speakers share their personal views and expertise, which may not necessarily represent the position of the Alan Turing Institute.  

Top image: Turing Commons