Forecasting river oxygen, and knowing what not to trust

A short-horizon forecasting and alert-triage system for stream oxygen sensors, built on public USGS data and evaluated more strictly than most projects of this size. What worked, what did not, and the mistakes I caught.

The problem

A field team keeps dozens of dissolved-oxygen sensors running on streams. Low oxygen kills fish; sensors also fail. Each morning they must decide which sites need attention in the next 24 hours: a likely low-oxygen event, or a sensor that cannot be trusted. Alerts must be rare, so the target was at most 0.5 alerts per site per week, with at least 6 hours of warning.

25%less error than the best simple method at 6 h

18%less error at 12 h

78%of outcomes inside the "80%" interval

14sites, one full year of out-of-sample testing

What worked

Forecasts beat strong simple baselines

At 6 hours ahead the simple baseline ("same as now plus yesterday's change") is off by 0.46 mg/L on average; gradient boosting by 0.33. Blending in a pretrained foundation model (Chronos-2) cuts another 0.03 mg/L: statistically real, but below the sensor's own 0.1 mg/L resolution, so the simpler model is the product and the blend is a documented experiment. Weather forecasts matter most at 24 hours (sunlight and temperature, not rain, carry most of the gain).

Uncertainty is calibrated, not decorative

Raw model intervals covered 73% of outcomes when claiming 80%. A conformal calibration step on a year of data brought that to 78% on average (some sites remain lower).

An actual forecast

From the replay dashboard, out-of-sample. The probability warning was right: oxygen fell below 5 mg/L and reached 2.5 mg/L within the day. The 1 to 12 hour forecasts tracked the first dip well, but the forecast for 24 hours ahead missed a second, deeper dip entirely and the actual value falls outside the 80% interval. Across the year, forecasts the model rated 90-100% likely were followed by an event 7 of 7 times in a spot check.

What did not work

Forecast-based alerts barely beat a one-line rule

"Alert when oxygen is below 5.5 mg/L" is hard to beat. The forecast helps when alerts must be very rare (at 0.5 per site per week it warns of 80% of events in advance against 42%), but the difference is not statistically established across 14 sites, and with more alerts allowed the simple rule is as good or better.

Slow sensor drift cannot be detected from one sensor

My first result said the detector caught about 28% of drifts. A "zero-drift" control showed the same rate with no drift at all: longer windows simply contain more random flags. A detector anchored on oxygen from 15-21 days earlier added only a small real signal, and only for 2 mg/L drifts lasting a week or more.

Mistakes I caught

Alert recall of 1.0 for every method. A scoring bug counted warnings raised after the event had begun. Fixed, with a chance baseline added.
"Drift detection at 28%." Chance, as above.
"The baseline's error doubles in storms." I had not computed it; it is 11-22% at 6-24 h. The advantage over persistence is what collapses.
A 97% low-oxygen probability at 7.4 mg/L. It looked wrong; it was right (28 of the previous 31 days had dipped below 5 mg/L).

How it was evaluated

Chronological walk-forward over 8 test folds; preprocessing, thresholds and calibration fit only on earlier data.
Bootstrap confidence intervals clustered by site-week; comparisons on identical forecast origins.
Faults for the detector benchmark are synthetic and labelled as such; no real labelled faults exist in the data.
The final year was reused across experiments, so only a forward holdout (forecasts logged from the freeze date) is truly untouched.

Limits

Sites were chosen for data quality; low-oxygen episodes are rare at some sites (as few as 7 in two years); one site has no usable probability model; weather results use reanalysis as a stand-in for observed rain; the 24 h weather gain is mildly optimistic. Slow drift is out of scope. Details are in the model card.

What is in the repository

Ingestion and validation, baselines, gradient boosting with conformal intervals, a foundation-model comparison, fault-detection benchmark with injected faults, a FastAPI service with monitoring and a retrain rule, a replay dashboard, tests, a data manifest and pinned versions. Start with PROBLEM.md, MODEL_CARD.md and reports/EDA.md.