Introduction
Concerns are growing about how our personal data is used, particular in algorithms making life-altering decisions; from loan and health insurance applications to recruitment and criminal justice systems. The EU’s General Data Protection Regulation (GDPR), which came into effect on 25 May 2018, aims to give individuals more control over the usage of personal data.
However, whilst the GDPR and the UK’s related Data Protection Bill have been produced to make these processes more transparent, researchers at the Turing identified significant gaps in the legislations. They found that a legally binding ‘right to explanation’ for decisions made by algorithms was missing from the proposed GDPR, which could have harmful ramifications for people affected by such algorithmic processes.
Through publishing multiple papers and providing evidence to UK government committees, the researchers provided extensive recommendations for how meaningful, legally binding explanations could be implemented. Such explanations would enable someone to understand a decision, and possibly contest it, without the need for technical, trade-secret information about the process to be disclosed.
These recommendations have been cited by both the House of Lords and the official EU body that is providing expert guidance on how the GDPR should be implemented.
How did it start?
In 2016 Dr Sandra Wachter, now a Turing Fellow, was working as a research assistant at the Turing. “I was helping basically everybody who was working in data ethics”, she says. This included setting up the Data Ethics Group with Turing Fellow Luciano Floridi. The group leads the Institutes’ research into understanding the ethical and societal implications of data.
Dr Wachter, a trained lawyer, became interested in potential loopholes in the GDPR concerning individuals’ rights to explanations of algorithmic decision-making. “It wasn’t supposed to be my actual project”, she confesses, “I worked on it at nights and on weekends because I felt so passionately about it.” Along with Professor Floridi and Turing Research Fellow Brent Mittelstadt, she published the paper ‘Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation’ in January 2017.
The paper showed that the GDPR as it was presented would only require data controllers – everyone who handles personal data – to merely inform individuals about the usage, and basic design, of algorithmic decisions made about them. It wouldn’t require them to give individuals proper explanations of the details of, or steps taken in, the algorithm. The paper garnered international attention and led to Wachter and her colleagues being at the forefront of the subsequent discussion of the issue.
What happened?
Making the case
Despite the Brexit referendum result, the UK announced the Data Protection Bill in August 2017, which aims to transfer the GDPR into UK law to ensure the regulations hold in the UK too. This meant that issues within the GDPR identified by Turing researchers were also present in the Data Protection Bill. Triggered in part by the concerns raised in the Turing researchers’ paper, the House of Commons’ Science and Technology Committee subsequently launched a public inquiry into algorithms in automated decision-making.
Dr Wachter contributed to written evidence provided by the Turing in response to this inquiry, and also provided oral evidence to the committee in November 2017, summarising the problems found in the legal framework. She also provided oral evidence for the related House of Lords Committee on AI. “There is so much resistance to making a right to explanation legally binding”, Wachter explains, “so we felt the need to show decision makers that it is feasible.”
Making explanations explicable
There have been major concerns voiced about providing transparency in algorithms. Reasons such as feasibility (how to go about explaining an algorithm), understandability (algorithms are complex and hard to understand for non-experts), IP infringement (companies are reluctant to give away their data or code), and exploitability (too much transparency could allow people to manipulate or game the system.)
What Dr Wachter and her colleagues, Brent Mittelstadt and Turing Fellow Chris Russell, showed in their paper was that all of these concerns could be mitigated by the use of so-called ‘counterfactual explanations’. These essentially take a decision-making algorithm and show what the outcome would have been for an individual if they had had certain different attributes, without disclosing too much technical, trade-secret information about the process itself. “You only need to give very limited information about what’s happening,” Wachter explains, “to enable someone to understand the decision, to possibly contest the decision, and, if a decision proves to be fair, to know what they need to change in order to alter future decisions in their favour.”
She gives the example of being denied a loan, and this having a significant effect on an individual. The person would go to the financial institution involved and request an explanation. A counterfactual analysis and explanation of the algorithm used might tell the person that the only factor that affected the decision was salary, and if this were higher by a certain amount they would get the loan.
See our Impact Story ‘Fairer algorithm-led decisions’ to see how Turing researchers are applying this thinking to ensure discrimination, based on factors such as race, is eliminated from automated systems.
“Engaging and explaining what is going on with algorithmic decision-making [may help avoid] repercussions for the UK economy”
Sofia Olhede, Director of the UCL Centre for Data Science
The value of this work is being recognised by experts across academia and technology. Hetan Shah, Executive Director of the Royal Statistical Society, says, “The work has been a welcome contribution to the current debate. There is considerable discussion about how far we can peer into ‘black box’ algorithms, but this may well prove to be a cul-de-sac. Approaches like counter-factual explanations are, in my view, likely to yield better results.”
Sofia Olhede, Director of the UCL Centre for Data Science, adds that, “Dr Wachter’s work is very important in tying our everyday understanding to mathematical concepts such as counterfactual fairness. Unless we make the effort in engaging and explaining what is going on with algorithmic decision-making, society may not adopt it, which would have unfortunate repercussions for the UK economy.”
What does the future hold?
The Article 29 Working Party is the official body that gives guidance to EU member states on how to interpret the GDPR and makes recommendations to the public on matters relating to the processing of personal data. In their final guidelines on automated individual decision-making and profiling they cite the work of Dr Wachter and her colleagues when giving their recommendations on explaining algorithms.
In addition, the House of Lords has proposed amendments to the Data Protection Bill which include many of the recommendations proposed by Turing researchers. See parliament’s dedicated page for the current progress of the bill.
These citations show the significant impact that the work of Turing researchers is having on legislation and subsequently the rights of individuals throughout Europe. This work also has the potential to have wider impact on trust and accountability in AI systems in general. Dr Wachter believes that, “because AI is so agile, every area of life that deeply affects people is potentially linked to the need for explanations.”
“We aim to provide a clear path forward for how to achieve meaningful accountability and transparency”
Sandra Wachter, Turing Fellow
Examples of this include in robotics and in healthcare – doctors using AI systems to predict diseases or propose treatment plans need a trusting relationship with AI systems, and to know why a certain course of action has been identified or chosen. There is a need for precise regulation and better methods to certify, explain, and audit inscrutable systems.
Dr Wachter and her Turing colleagues are committed to ensuring this happens. “An open dialogue with industry, government, academia and civil society is needed on how to proceed, and we aim to provide a clear path forward for how to achieve meaningful accountability and transparency in algorithmic and AI systems.”
Recent updates
In September 2018 Google announced that they have included this work's contributions to counterfactual explanations into a 'What-If Tool' for their TensorFlow™ machine learning framework. The tool is designed to let non-programmers understand what their machine learning is doing, with the ability to manually edit examples from datasets and show how the model’s predictions change as any single feature is changed. As detailed above this functionality will help ensure that algorithms produced by software using TensorFlow™ will be easily explainable.
TensorFlow™ is an open source software library for high performance numerical computation and is one of the most widely used deep learning frameworks in the world, powering, for example, all the machine learning in Android phones. Therefore the work of these Turing researchers will be implemented in a huge number of machines around the world as well.
Read more on the Google blog post, and see the code and references to the Turing researcher's work.
PDF Summary
Related people
Professor Luciano Floridi
Professor of Philosophy and Ethics of Information at the University of Oxford