Authors
Jonny Freeman, Internal Communications Coordinator
Carolyn Ashurst, Senior Research Associate, Safe and Ethical AI
From November 2021 to January 2022, The Alan Turing Institute and the Information Commissioner’s Office (ICO) hosted a series of workshops about fairness in AI. The aim of the series was to assist the ICO in the development of an update to the fairness component of its existing non-statutory guidance on AI and data protection, by convening key stakeholders from industry, policy, and academia to identify the key issues and challenges around fairness in AI, and to discuss possible solutions. The guidance is part of ICO25’s Action Plan for 2022-2023. The update to the guidance has now been published, and the ICO will ensure any further updates will reflect upcoming changes in relation to AI regulation and data protection.
The series consisted of three workshops divided by three broad themes, though there was significant overlap between each discussion. The first discussion sought to establish the parameters of what we mean by ‘fairness’ and ‘unfairness’ in the context of AI and UK data protection, the second asked how unfairness might be mitigated, and the final workshop focused on the impact of human decision-making on fairness in AI. What follows is a non-exhaustive summary of all three workshops, picking up on key issues and talking points.
Fairness is one of the foundational principles of the UK General Data Protection Regulation (UK GDPR). According to this principle, data controllers and processers “should only handle personal data in ways that people would reasonably expect and not use it in ways that have unjustified adverse effects on them.” The ICO’s guidance for the UK GDPR also calls for measures to mitigate discrimination, although the fairness principle has broader applications than just anti-discrimination. Fairness in data protection is different to ‘algorithmic fairness’ as often discussed in the AI community, as it is not only interested in the distribution of resources or outcomes, but looks at the context a system is deployed in (including the human decision-making that sets the parameters of a system) and the power imbalance between individuals and those who process their data.
Fairness in AI
As automated decision-making governs increasing areas of our day-to-day lives, from our bank accounts to our medical care, it’s more critical than ever to ensure that such systems do not replicate or amplify societal biases that lead to discrimination.
There can be multiple possible sources of such bias across the AI pipeline. It can stem from sampling bias, in which the training data for an AI model is not representative of the population it applies to; or societal bias, where though the data may be accurate, it reflects or reinforces a bias in society. An example of the latter can be found in risk classification when selling insurance policies, which can exacerbate economic inequality by making insurance unaffordable to those living in low-income areas who are at a greater risk of crime. Similarly, societal biases can be reinforced by using machine learning models to calculate the lowest salary a job applicant would be willing to tolerate, or the highest insurance premium they could afford to pay.
The risks are not only to the consumer, however. There are prudential and reputational risks to organisations, and macro-risks to the economy: multiple firms using the same biased datasets increases the likelihood of sudden losses of confidence in financial markets, or an erosion of trust in AI technology across society.
Fairness in human-machine teams
It is not just AI models which can entrench and reproduce bias and discrimination. The AI model may be only one step in a decision-making process where humans have input or ultimate oversight, known as a human-machine team. Focusing only on the machine learning part of the team risks missing biases at the level of human review, or at an organisational level. Risks of bias at the level of data and AI need to be integrated into wider organisational risk management, and responsibility for fairness clearly delineated between relevant stakeholders, from developers to deployers to consumers.
Humans and machines have different strengths and weaknesses when it comes to ensuring fair outcomes. While machines cannot employ emotional intelligence, nuance, or an understanding of the broader context, humans can fall victim to unconscious biases and fallacious reasoning. Humans can also suffer from “automation bias”, becoming overly reliant on automatic processes at the expense of their own good judgement—an infamous example being motorists who blindly follow faulty satnavs into rivers.
Article 22 of UK GDPR currently prohibits “decision(s) based solely on automated processing” that have legal or “similarly significant” consequences for individuals (it is important to note this is an area the government is looking to refine as part of the Data Protection and Digital Information Bill). However, what constitutes “solely” automated is not easily defined, and nor is exactly what constitutes a “decision”. Does automatically prioritising certain candidates in a CV screening constitute a “decision” for instance? And what if, rather than a discreet one-off decision, an organisation is monitoring data on an ongoing basis, with certain behaviours triggering automated actions – for example illegal transactions in a bank account triggering police referral. Does each trigger constitute a decision? And wouldn’t this imply that the inverse—the lack of a trigger—also constitutes a decision? One proposed solution is to define a decision as having not been made if events proceed as a service user expects, which has the advantage of narrowing the scope of what constitutes a “decision” so that human reviewers are not overwhelmed. However, this may not work well for situations when customers receive benefits when certain thresholds are crossed, as not receiving these benefits may be aligned with their expectations, even if they were unfairly excluded from them.
Some participants argued that having “humans in the loop” (HITL) may not be necessary for all applications due to rapid advancements in automation. However, it will still be necessary for humans to set the objectives for what a machine is trying to achieve, and this in itself may be subject to error or bias.
There are times however when human review may not be appropriate. In the case of an AI system designed to identify deepfakes that produces a probability that the content is real or fake, what can a human reviewer meaningfully contribute, when the forgeries are designed to deceive humans? One participant noted that human review also offers the possibility to reintroduce bias as well as to mitigate it. For example, confirmation bias may mean human reviewers support or veto AI decisions depending on whether it reinforces or challenges some existing belief or worldview.
Equally, there are some questions which AI is not well-suited to answer. For all the data an AI model can crunch, it is not possible to definitively know whether someone will be the right candidate for a job or not, for example. One participant cautioned that by delegating too many decisions to AI, we risk creating a “shadow legal system”, where people’s lives end up governed by opaque predictions rather than transparent regulations. Instead of blindly trusting AI, one participant argued we should be communicating exactly what it can and can’t tell us, and how these “known unknowns” define the boundaries of decision-making.
Other issues may arrive during “problem formulation”, the process by which a social or political problem is translated into a quantifiable one, intelligible to a machine learning model. Proxies must be used for unquantifiable constructs: for example, you cannot quantify how sick someone is, but you can see how much money has been spent on their healthcare and then use that as a proxy measure of sickness. However, if you do not control for existing biases (such as the fact that more money is spent on the average white patient than black patient in the United States) you will end up with an algorithm that prioritises white patients over equally unwell black patients.
We’ve seen the varied difficulties inherent in trying to ensure fairness in AI systems. What are some possible solutions?
Tools for mitigating AI bias
Ensuring fairness standards are enshrined in law is crucial, however this can present multiple challenges. Anti-discrimination legislation varies massively across jurisdictions, as do consumer regulation laws, and data protection laws. For multinational companies, this can make implementing a fair AI system particularly challenging. In the absence of clearly defined international standards, many companies have defaulted to using American fairness thresholds, for example the “four-fifths” rule in hiring, which states that the selection rate for any demographic group must be at least 80% of the most hired group (though such measures may fall foul of anti-discrimination laws in other jurisdictions).
However, the limitations of this are obvious. Fairness considerations differ across cultures and societies, and what is appropriate bias mitigation for one part of the supply chain might be near-useless in a different country with a different ethnic make-up, for example. What constitutes fairness also varies widely across sectors, making it necessary to consult with domain-specific experts when designing fair AI systems.
In the UK, many of the regulations governing fairness in AI are to be found in the UK GDPR. For example, Recital 71 of UK GDPR requires data controllers to “implement technical and organisational measures” to ensure lack of discrimination in automated systems. The UK GDPR also requires organisations to undertake a Data Protection Impact Assessment (DPIA) to “analyse, identify and minimise the data protection risks of a project or plan”, specifically identifying “discrimination” as one of the risks to be mitigated. In the case of R (Bridges) v South Wales Police, the Court of Appeal ruled that the force’s use of AI facial recognition technology was unlawful, partly because it had not been subject to an adequate DPIA, and failed to account for the risks of indirect discrimination posed by AI facial recognition technology.
The UK government has also launched an Algorithmic Transparency Reporting Standard for AI systems employed in the public sector, which if adopted could help to inspire confidence that such systems were fair.
Aside from regulation, there are other means of helping to mitigate AI bias. Organisations can make use of “fairness toolkits” i.e. software tools which be used to search for and mitigate bias. However, there are risks associated with these. For example, fairness toolkits can become overfitted to training data, meaning that they may work less well on new data entered later. In addition, fairness toolkits may be overly—or even exclusively—focused on evaluating the model as it is first constructed, at the expense of the remaining model lifecycle. As a means of mitigating this latter risk, the ICO recommends setting a “minimum success criteria” with which to continuously monitor the model’s performance during employment, to ensure that biases don’t become apparent at a later stage. This may occur because fairness considerations are dynamic rather than static, and the parameters may change during the life cycle of a system.
Financial services may employ fairness assessments such those developed by the Veritas Initiative—part of the Singapore National AI Strategy—which ensure that credit scores adhere to the “FEAT” principles, that is, fair, ethical, accountable, and transparent. Other tools include questionnaires, which are a useful means of capturing various biases, including historical, representation, and measurement bias; and wider stakeholder engagement, seeking feedback and input from industry, civil society, and academia. In particular, vulnerable groups and minorities should be consulted at the earliest objective-setting stages to mitigate against biases being introduced. This process of seeking feedback should be ongoing through the lifecycle of the model—and avenues of recourse and complaint for consumers should be made as accessible as possible—so that feedback can be used to continuously update and improve the model.
Updates since the workshop
There have been numerous developments since this workshop: the UK government has published its response following a data reform consultation, introduced a draft Data Protection and Digital Information bill to parliament that included amendments to Article 22 of UK GDPR, and published an AI regulation policy paper outlining the government's thinking ahead of the AI White Paper (which is expected soon). The ICO has published its ICO25 strategic plan for 2022-2025, announcing plans to refresh its AI guidance to help address issues of unfairness and discrimination. This workshop represents a key initial input to this guidance.
Opportunities for further discussion
We’d like to thank everyone who participated in this fascinating series of discussions. If you’re interested in this topic and would like to join future discussions on how data protection’s fairness applies to AI, you can email the ICO at [email protected], or contact the workshop organisers, listed below.
