Projects

What Distribution Do AI Ensembles Approximate?

One of the most potentially impactful result of the rapid advances in AI weather and climate forecasting is better uncertainty quantification. Real decisions, from evacuation orders in anticipation of severe weather to climate mitigation policy or infrastructure, depend not only on a prediction but on how uncertain that prediction is. AI forecast tools are much cheaper to run allowing very large ensembles, or collections of distinct plausible forecasts, to be generated.

Ensembles are generally assumed to be sampling from some true distribution of possibilities with each forecast representing an equally likely outcome. When evaluating each time and location independently AI forecasts have been shown to perform competitively or even surpass traditional forecasts. But whether they probabilistically capture the relationships over time and space reflecting true uncertainty and physics is a relatively unexplored question.

I try to get at this question about the structure of AI forecasts in my first project working in the Barnes lab. One figure from a recently submitted paper below captures the gist of our results. The Lorenz-96 system is a simple low dimensional analog of the atmosphere for which exact solutions can be generated. We train two different AI models to forecast it at a lead time of 5 days: a diffusion model, which is inherently probabilistic and generates ensemble forecasts conditioned on an initial state, and a deterministic emulator, which forecasts a single predicted state based on the current state. Ensembles are produced by creating a set of slightly perturbed initial conditions and forecasting the future from each one. A “perfect model” dynamical forecast which exactly simulates the true physics is used as a baseline, generating ensembles from perturbed initial conditions.

The blue bars in the figure show how well the forecast captures pointwise behavior, while the orange bars show how well it captures spatial relationships representing the higher dimensional behavior of the system and underlying uncertainty. For all of the AI forecasts, there is a persistent gap which is not replicated for the dynamical model. This is a really interesting result that has implications for how these models get integrated into weather forecasting systems and how we interpret uncertainty estimates from them, particularly uncertainty that implicates multivariate relationships such as compound events. Further results and explanation in a preprint here

ML-Based Radiative Transfer Emulator

Currently, large fractions of satellite observations are not assimilated into weather forecast models. The speed of the forward (or observation) operator is one bottleneck preventing the utilization of more of these data. The Community Radiative Transfer Model (CRTM) is a community model developed and maintained by the Joint Center for Satellite Data Assimilation (JCSDA) and is commonly used as a radiance observation operator for a number of satellite platforms. I have been working with JCSDA to develop a machine learning (ML) emulator of CRTM which will be faster, allowing more observations to be assimilated with the same computational effort.

Critically, our emulator is probabilistic and predicts its own error. In future development, we plan to include a threshold that will allow computations to revert to CRTM for conditions in which the ML model has low confidence in its predictions.

The first paper from this project is published in Artificial Intelligence for the Earth Systems and is available here. The trained ML emulator into the Unified Forward Operator (UFO), a component of JCSDA’s Joint Effort for Data Assimilation Integration (JEDI) framework. This will allow experiments using the ML emulator within a realistic operational setting to determine if forecast quality can be improved using our approach. Active development on this project has been handed over to the JCSDA team and is ongoing.

Skillful Subseasonal to Seasonal Indian Ocean Marine Heat Wave Forecasts Using Deep Learning

A marine heat wave (MHW) is a period of anomalously warm ocean temperatures. These events have significant effects on marine ecosystems, fisheries, and coastal communities. Given sufficient warning, mitigating measures can be taken by resource managers and local governments. Development of forecasts has tended to focus on short-term daily forecasts out to ~2 weeks, or seasonal monthly forecasts out to ~12 months. Subseasonal to seasonal (S2S) forecasts of MHW with weekly resolution has been comparatively less of a focus for the community.

Given the accuracy of AI weather forecast models, and inspired by this work, I was curious if AI/ML forecast methods could be used to predict sea surface temperatures and/or MHWs. The top line, as evidenced by the figure below, is: yes. Our trained U-NET performs competitively with the European Center for Medium Range Weather Forecasting (ECMWF) S2S model across a variety of metrics. This work has been published in Environmental Data Science and is available here.

In addition to training a U-Net to make S2S forecasts, I also experimented with using denoising diffusion models to generate forecast ensemble members. These models, similar to popular image generation AI systems, start with an image of random noise and iteratively remove noise to produce a meaningful result. In this case, the target was a map of predicted Indian Ocean sea surface temperatures (SST). While the approach showed promise, the results were less accurate than using a U-Net directly for forecasts. The primary issue was that the ensembles generated by the diffusion models were underdispersive, meaning they underestimated the true error. There have been recent attempts to address this issue in diffusion models, including by Tim Higgins. I hope to revisit this approach in the future or play a supporting role if someone else wants to pursue the idea.

Emulating the Ensemble Kalman Filter with a Convolutional Neural Network

If we have a measurement of temperature in the atmosphere at a particular location and walk some distance away, that same measurement tells us something about the temperature at our new location. In the context of weather forecast models and data assimilation, the problem of generating new initial conditions can be summarized as: how quickly does the original measurement stop representing the temperature as we walk away from it?

Using jargon, the question is to what degree is the system autocorrelated at the location of the measurement. Different DA methods have different ways of estimating this. From the world of ML, convolutional neural networks (CNN) are really good at processing spatial data and identifying spatial features such as edges. This is at least conceptually analogous to the problem of identifying spatial autocorrelation. My question was: can a CNN successfully capture this structure well enough to assimilate observations?

We performed our experiments using a 1-D test model (Lorenz 96) as a proof of concept. Training the CNN to emulate a traditional DA method (the EnKF), we showed that using the trained CNN on unthinned observations results in better state estimates and forecasts than using EnKF on thinned observations. This was the first project of my PhD, and the results are published in the Journal of Advances in Modeling Earth Systems here.