DeepSensor

Introduction

DeepSensor is designed to provide capabilities for:

Multi-scale transitions between global and local;
Infill cloud-covered gaps in satellite-derived data;
Optimise observations and environmental sensing (e.g. trajectories for fleets of autonomous underwater vehicles) to improve environmental prediction.

Explaining the science

Environmental data is challenging for conventional ML architectures because it can be multi-modal, multi-resolution, and have missing data. The various data modalities (e.g. in-situ weather stations, satellites, and simulators) each provide different kinds of information. We need to move beyond vanilla CNNs, MLPs, and GPs if we want to fuse these data streams.

Neural processes (NPs) have emerged as promising ML architectures for environmental data because they can:

Efficiently fuse multi-modal and multi-resolution data,
Handle missing observations,
Capture prediction uncertainty,
Handle both point source and gridded data.

Early research has shown NPs are capable of tackling diverse spatio-temporal modelling tasks, such as sensor placement, forecasting, down-scaling, and satellite gap-filling.

The DeepSensor Python package streamlines the application of NPs to environmental sciences by plugging together the xarray, pandas, and neural processes packages with a user-friendly interface that enables rapid experimentation.

Project aims

DeepSensor aims to:

Reduce the effort required to apply Neural Processes (NPs) to environmental data so users can focus on the science.
Build an open-source software and research community.
Generate a positive feedback loop between research and software.
Accelerate applications of neural processes to environmental research questions.
Stay updated with the latest SOTA models that align with the DeepSensor modelling paradigm.

Applications

The strength of DeepSensor is its ability to be applied to a diverse range of environmental questions from the polar regions to the tropics.

DeepSensor is currently being applied to:

Generate high resolution, high fidelity maps of sea ice conditions across the entire south ocean
Downscale forecasts of precipitation events in New Zealand
Downscale weather temperatures in Germany
Identify optimal placement of environmental monitoring equipment in Antarctica
Produce soil moisture maps of the UK

The key project milestone has been publishing and launching the DeepSensor package to the community.

We are continually looking for new applications of the DeepSensor tool in all scientific discipline, so please get in touch if you have any questions or project suggestions.

We look forward to future milestones as application of this tool increases.

Sensor Placement

ConvGNP	GP Baseline
The convolutional Gaussian neural process (ConvGNP) finds highly informative sensor placement. [Click here for the relevant paper]

Surface Soil Moisture

The example shows the output prediction (mean and standard deviation) of a trained ConvCNP model using only ERA5 Land reanalysis data (a-d) at a given test time using 50% of context stations (shown as black circles). This is compared by another model using the Sim2Real transfer approach (e-h) in which a ConvCNP model is first pre-trained on ERA5 Land reanalysis data, and then a smaller but higher-quality real weather station dataset (COSMOS-UK) is used to fine-tune the model. While the mean prediction looks similar between Sim Only (b) and Sim2Real (f) settings, we observe Sim Only generates unrealistic uncertainty estimates i.e. low variation in the standard deviation (c) if compared with Sim2Real (g). The results impact derived tasks such as Sensor Placement. Further details of the Sim2Real approach are provided in the original paper [Click here for the relevant paper].