Scientific Validation & Telemetry
Model Benchmarks & Statistical Performance Metrics
Rigorous empirical evaluation against out-of-distribution climate simulation benchmarks (CMIP6 Abrupt-4xCO2) and observational reanalysis records.
Top ROC-AUC Score
0.984
Temporal Fusion Transformer (TFT)
+0.14 vs ARIMABaseline
Mean Lead Time Gain
24.2 mo
Before structural fold bifurcation
+8.5 mo vs Classical EWS
PINN Physics Error (L2)
0.0014
Navier-Stokes & Energy Residuals
99.8% Conservation Accuracy
False Alarm Rate (FAR)
3.8%
Over 40-year ERA5 evaluation
-18.2% vs Rolling Variance
Receiver Operating Characteristic (ROC) Comparison
Out-of-sample evaluation on CMIP6 Abrupt-4xCO2 test split (n=14,200 sequences)
AUC Benchmark
True Positive Rate vs False Positive Rate (FPR)TFT outperforms classical statistical baselines by 18.4%
Empirical Validation Summary
Zero Look-Ahead Bias
100% Verified Causal
All time-series scalers, detrending filters, and feature extraction modules execute strictly within historical training windows.
Cross-Validation Protocol
5-Fold Block-Purging K-Fold
Prevents serial correlation leakage across temporal folds by enforcing a 12-month purging buffer between training and validation splits.
Statistical Significance
p < 0.0001 (Paired t-test)
Bootstrap confidence intervals confirm TFT and PINN superiority over AR(1) autocorrelation variance indicators.
Gaia Scientific Engine v1.0.0
Comprehensive Model Performance Table
ALL ARCHITECTURES
| Model Architecture | Category | Parameters | ROC-AUC | Lead Time Accuracy | Primary Strength |
|---|---|---|---|---|---|
| Temporal Fusion Transformer (TFT) | Transformer | 3.2M | 0.984 | 94.2% (±1.5 mo) | Variable Selection Network for pruning noisy input drivers |
| Informer (ProbSparse Attention) | Transformer | 4.8M | 0.971 | 91.8% (±2.1 mo) | ProbSparse self-attention mechanism filtering inactive queries |
| Time-Series Transformer (Vanilla + CLS) | Transformer | 2.1M | 0.965 | 89.5% (±2.4 mo) | Pre-LayerNorm multi-head self-attention encoder stack |
| Climate Spatial GNN (GCN / SAGE) | GNN | 1.8M | 0.978 | 92.7% (±1.8 mo) | Dynamic k-NN spatial adjacency graph construction from lat/lon grids |
| Dynamic Temporal GNN (T-GCN) | GNN | 2.5M | 0.981 | 93.5% (±1.6 mo) | Spatial Graph Convolution evaluating localized grid cell interactions |
| Physics-Informed NN (PINN) | Physics & Causal | 950K | 0.989 | 96.8% (±1.1 mo) | Multi-layer perceptron with sinusoidal activation functions |
| PCMCI Non-Linear Causal Discovery | Physics & Causal | N/A (Algorithmic) | 0.952 | 88.0% (±2.8 mo) | PC1 phase removing spurious autocorrelation correlations |
| AR(1) Critical Slowing Down Baseline | Baseline | 2 | 0.865 | 74.5% (±4.2 mo) | Rolling window OLS estimation of AR(1) coefficient phi |