ICL Characterization of Climate Foundation Models: When Can Transformers Learn Weather and Climate?

Key findings from this study

• The study proves every natural climate task falls into exactly one of two computational complexity categories based on sufficient statistics.
• The authors establish that extreme event detection becomes computationally intractable for transformers when simultaneous events exceed approximately 3 to 5.
• The framework demonstrates weather forecasting remains tractable for lead times under 14 days while individual climate trajectories become fundamentally unpredictable beyond this horizon.

Overview

This theoretical work explains performance disparities in climate foundation models through in-context learning complexity. The study extends ICL characterization frameworks to spatiotemporal climate data. The Climate Prediction Dichotomy Theorem categorizes all natural climate tasks into two computational complexity classes.

Methods and approach

The authors formalize climate tasks using sufficient statistics and computational capacity constraints. They analyze transformer architectures with constant-depth polynomial-size limitations. The framework distinguishes Type A tasks admitting additive sufficient statistics from Type C tasks requiring combinatorial sufficient statistics. Sample complexity relationships between ICL and empirical risk minimization are established. Predictability horizon constraints are derived for weather forecasting lead times.

Results

The Climate Prediction Dichotomy Theorem proves every natural climate task belongs to exactly one of two categories. Type A (ICL-Easy) tasks include temperature, pressure, and wind field prediction, achieving sample complexity nICL = Θ(nERM) through additive sufficient statistics. Type C (ICL-Hard) tasks include extreme event detection, tipping point identification, and compound event localization. These tasks exceed computational capacity of constant-depth polynomial-size transformers when simultaneous events J surpass threshold J* = O(log log n) ≈ 3–5.

Weather forecasting qualifies as ICL-Easy for lead times τ < τL ≈ 14 days. Climate statistics remain ICL-accessible, but individual trajectories prove fundamentally unpredictable beyond this horizon. The analysis generates six testable predictions about climate foundation model behavior. Five deployment guidelines distinguish scenarios where ICL suffices from cases requiring fine-tuning. The dichotomy explains why models like Pangu-Weather and GraphCast excel at field prediction while underperforming at event detection.

Implications

The theoretical framework resolves observed performance disparities between deterministic field prediction and extreme event detection in current climate models. Computational limitations in constant-depth transformers create fundamental barriers for tasks requiring combinatorial reasoning over multiple simultaneous events. The predictability horizon constraint at approximately 14 days provides theoretical grounding for weather versus climate modeling distinctions.

The deployment guidelines inform practitioners when to rely on pretrained models versus when fine-tuning becomes necessary. Architecture choices for next-generation climate AI systems should account for task-specific complexity categories. Models designed for Type C tasks require different computational structures than those optimized for Type A tasks. The framework suggests that improving extreme event detection demands architectural innovations beyond scaling existing transformer designs.

Scope and limitations

This summary is based on the study abstract and available metadata. It does not include a full analysis of the complete paper, supplementary materials, or underlying datasets unless explicitly stated. Findings should be interpreted in the context of the original publication.