Introduction

Complex control problems arise in the operation of critical infrastructure, including electricity, gas, water and transportation. This project is developing ‘rangl’, an AI competition environment for practitioners (both novice and experienced) to apply classical and machine learning techniques and expert knowledge to UK-centric problems. The project will work with industrial challenge holders to develop insights into the leading classes of solution.

Explaining the science

Infrastructure systems, including electricity, gas, water and transportation, must operate reliably at acceptable cost in the context of ageing infrastructure and new technological possibilities. At the same time a greater amount of measurement and forecast data is becoming available. There is a growing need for appropriate AI controllers to leverage this data. Artificial intelligence, with its speed, scale and accuracy, offers transformative potential in applications to these problems. However, controllers for critical infrastructure should be robustly assessed in an appropriate simulation environment.

This project is developing a competition platform in which challenge environments are formulated in the reinforcement learning framework using the “agent-environment loop”. At each timestep the controller (agent) chooses an action based on the current observation, and the challenge environment returns a new observation and a reward. The aim is to create controllers with intelligent characteristics, capable of handling both quantifiable and unquantifiable uncertainty and encoding expert knowledge. Competition entrants document their controllers, helping to develop insights into the leading classes of solution.

Control system visualisation
Example of a three-zone power system based on Great Britain. Scheduling power plants is challenging due to uncertainty about consumption and renewable generation in the near future. Supply and demand must match in real time, and energy transfers between regions are limited by network capacity. Rangl invites AI to come up with cost effective strategies for scheduling the power system under uncertainty, with the potential to save consumer money, accelerate decarbonisation and improve the reliability of electricity supply.

Project aims

'Rangl' is a competition platform created at The Alan Turing Institute as a new model of collaboration between academia and industry. Through integration with OpenAI Gym, rangl offers a user-friendly environment to develop learning approaches to data-driven control problems. Anybody can propose a rangl challenge, compete in a challenge by designing a controller, or contribute an ‘off-the-shelf’ AI controller for users to customise.

The platforms assess user-submitted algorithms for specific tasks, helping the best classes of solution to emerge; a proven mechanism for realising the potential of AI. 

Project updates

First RangL Challenge

From 18 to 25 January 2021 the RangL team fulfilled a long-held ambition: to run a generation scheduling challenge. The problem involves using continually updated forecasts for energy demand and renewable energy generation to schedule, and so to minimise, the use of fossil fuels. It is challenging partly because the observation space is large — at each step, the agent is given forecasts for all time periods — and also because the forecasts are updated as new information arrives, so are guaranteed to be superseded by better ones.

This ‘look-ahead mode’ generation scheduling was one of the first motivations for RangL, when the project was conceived in early 2019 during the Mathematics of Energy Systems research programme at the Isaac Newton Institute in Cambridge. While not directly connected, it’s interesting to note that the forthcoming special issue of Philosophical Transactions of the Royal Society A based on the MES programme has an article by Peter Glynn and Jacques de Chalendar on theoretical aspects of this kind of problem (titled “On incorporating forecasts into linear state space model Markov decision processes”).

The competition itself was heavily oversubscribed, with applicants from Argentina, Denmark, the Netherlands, Italy, France and the UK, drawn from academia, industry and the third sector. We’d like to thank all participating teams, who generated a fantastic atmosphere on our Slack channel throughout the week. It must have been good, as one competitor even joined the RangL team. The winners were team zeepkist with members from the Intelligent Electrical Power Grids group at TU Delft and Tennet, the Dutch power system operator. The final scores, and zeepkist’s winning code (which used RL), are here in the challenge repository.

We recently argued on the Turing blog that as the world reopens following the pandemic, we will need to make more flexible, responsive and data-driven decisions. Hopefully this first challenge illustrates a small part of the potential role that reinforcement learning can play.

Organisers

Researchers and collaborators

Collaborators