Quartz solar AI nowcasting

Using generative AI to forecast cloud movements, allowing solar energy to be more efficiently integrated into the electricity grid and for the UK to decarbonise faster

Learn more

Project status

Ongoing
Menu

Introduction

Due to cloud cover, solar power generation can be unpredictable. This entry uses AI to forecast cloud formation hours ahead of time, allowing solar energy to be more efficiently integrated into the electricity grid through better prediction of periods of intermittent supply. By leveraging satellite imagery and live solar generation data, it helps better manage solar energy resources and balance a renewables heavy grid. This more efficient approach to renewable energy management can help the UK to decarbonise faster.

Explaining the science

The solution will use generative AI to forecast cloud movements hours ahead, allowing solar energy to be more efficiently integrated into the electricity grid and for the UK to decarbonise faster. 

We will develop an open-source state-of-the-art generative AI model of cloud evolution.  The model of cloud evolution will be combined with our existing AI model for forecasting solar power.  Predicting cloud movement will provide significant improvement in the forecast accuracy for horizons hours into the future. 

Solar power forecasts help our users to optimise and better manage their solar energy assets and balance a renewables-heavy grid, reducing both costs and CO2 emissions.  Our first users are electricity grid operators, solar farms, solar and storage sites, and smart building operators.  For these users, the unpredictability of solar power generation creates challenges in efficiently managing power generation while ensuring end users have an uninterrupted supply of electricity.  

There are a few key trends in the evolution of the electricity grid which are making very short term forecasting ever more important - the rise of weather dependent renewables, the decline of slow to change thermal generation (especially coal) and the rapid rise in battery capacity.  This means much more of the activity on the electricity grid now happens minutes and hours ahead, rather than the hours or days as at the creation of the deregulated GB electricity grid. Our initial focus is the UK market, but being a digital solution, it could be deployed anywhere in the world.  

Most of the forecasts currently available rely solely on numerical weather predictions (NWPs) which do not make use of all the data available and struggle to forecast short-time horizons, which are important when making operational decisions on assets.  Our forecasting solution captures inputs from recent satellite imagery, live solar power generation data and NWPs to forecast across a wide geographical area and from minutes to hours into the future. 

The key innovation is to explicitly create predictions of cloud cover (cloud masks) for hours into the future through the use of generative AI techniques.  We have large histories (hundreds of terabytes) of 5-minutely satellite imagery and weather data which provide the training data for our model.  By applying the latest AI techniques, we are able to effectively generate satellite imagery into the future. From our experience in solar forecasting, having recent satellite imagery improves accuracy by 25% in the short term, so generating future satellite imagery we expect will see large accuracy improvement.

In addition to a central forecast, the model will quantify uncertainty in the cloud and solar power forecasts - important to many end users as the “worst case” situation is often of as much interest as the expected scenario. 

 

Project aims

Most of the forecasts currently available rely solely on numerical weather predictions (NWPs) which do not make use of all the data available and struggle to forecast short-time horizons, which are important when making operational decisions on assets.  Our forecasting solution captures inputs from recent satellite imagery, live solar power generation data and NWPs to forecast across a wide geographical area and from minutes to hours into the future.

The key innovation in this project is to explicitly create predictions of cloud cover three hours into the future using state-of-the-art AI techniques. We have a large dataset of 5-minutely EUMETSAT satellite imagery, which provides the training data for our model. By applying the latest AI models, we are able to effectively generate satellite imagery into the future.

We have conducted an extensive literature review of the existing approaches and integrated the most common methods into our framework as baseline models. We aim to implement transformer models and diffusion models as a part of the solution.

The outputs of the cloud nowcasting model are fused with the PV output prediction model to predict accurate PV generation. Our initial focus is the UK market, but being a digital solution, it could be deployed anywhere in the world.

Applications

The central target industry for a service forecasting cloud movements is the electricity sector, where we have a strong market understanding and value proposition.  Further, the UK Solar Power Market size in terms of installed base is expected to grow from 18.5 GW in 2024 to 53.1 GW by 2029, at a CAGR of 23.53% during the period 2024-2029.  Further, solar power is forecast to become the largest form of generation capacity globally by 2040.  We will focus on capturing the following four markets:

1. Grid Balancing: Accurate solar energy forecasts are essential to manage grid stability and allow integration of increased solar capacity.  The UK has one national and six regional system operators.  Almost every country will have one or more grid operators with similar requirements. 

2. Solar Farm Operators: Improved solar forecasts directly benefit solar farm operators by reducing their costs when selling their power into the grid.  Losses due to solar forecast errors cost UK farms up to 5% of revenues, or nationally £25M p.a.

3. Smart Homes: In this emerging sector, companies help homeowners with solar panels optimise their energy based on accurate forecasts, saving costs and reducing reliance on non-renewable energy sources. This market is projected to grow by 11.66% (2024-2028).

4. Energy Traders: Energy traders need to forecast the balance of supply and demand in order to make the best hedging decisions.  All wholesale energy supply and generation companies have energy traders; in addition to independent trading firms.  There are over 30 such large firms operating in the UK and many more smaller firms. Similar markets exist in the EU, US, Australia, India, Japan. 

Further markets such as leisure and sports can be explored in the longer term. 

Researchers and collaborators