Confidence intelligence
The story of how ResQNet+ decides what to trust
In an emergency stack, the hardest question isn't "what happened?" — it's should we move people and assets right now? Every escalation spends minutes, fuel, and focus. The model learns from the same evidence your operators see — app, WhatsApp, news, social, grid risk, weather, NASA EONET, and responder readiness — then returns a calibrated probability. LightGBM handles messy interactions, isotonic regression keeps probabilities honest, and deterministic rules keep the API warm when ML pauses.
Listen everywhere
WhatsApp pings, app reports, news, social posts, and NASA EONET become one timeline — no signal fights for attention alone.
Score with receipts
LightGBM ranks patterns; isotonic calibration turns raw scores into probabilities you can defend in a briefing.
Never go blind
If the model hiccups, weighted rules still produce a score — SOS flows stay open, reasons stay visible.
Lab run
Synthetic ResQNet+ signals
Fake-but-plausible traffic so you can see training, metrics, and plots before production labels arrive.
Time-ordered snapshots
~2,200
Reproducible stochastic labels
Chronological split
70/15/15
Train · val · test
Test ROC-AUC
0.5208
Synthetic task; live labels will sharpen
Brier score
0.2511
Lower is better · post-isotonic
Boosting config
600 trees
LR 0.03 · subsample 0.9
Artifacts
LGBM + ISO
lightgbm.pkl · isotonic.pkl · meta.json
Evidence you can show on a slide
Plot gallery
Drop your exports into public/1.png, 2.png, 3.png to light up the gallery.
Operational thresholds
When to act
≥ 0.85
Immediate escalation
0.65 – 0.84
Priority watch + human review
0.45 – 0.64
Monitor + collect more signals
< 0.45
Low confidence — no auto escalation
Engineering playbook
Deep dives for builders
Open only what you need.
01What the model predicts›
Binary probability that a candidate incident is a true high-priority event warranting escalation within a fixed horizon (for example 2h). Production labels freeze to your DB: status transitions, assignments, dispatch, and severity confirmations.
02Stack: speed, honesty, resilience›
- —LightGBM — tabular workhorse for mixed internal + EONET features.
- —Isotonic calibration — maps raw scores to probabilities operators can trust.
- —Rule-based fallback — weighted sources, weather, grid, EONET proximity; never blocks SOS flow.
03Data fusion & NASA EONET›
Each training row is a snapshot at decision time t. We spatially join EONET hazards (50–200 km radius by type), window them (24h / 72h), aggregate counts/distances/recency, then merge with ResQNet+ internal signals.
04Feature blocks (feature_set_v1)›
- —Internal: app_report_count_30m, whatsapp_report_count_30m, news/social counts, unique_source_count_1h, report_velocity, source_entropy
- —Geo-temporal: lat/lng (or geohash), hour_of_day_local, day_of_week, is_night, distance_to_last_confirmed_event_km, local_grid_risk_score
- —Weather: weather_severity, rain_mm_3h, wind_speed, temp_anomaly, forecast_risk_next_3h
- —EONET: eonet_event_count_24h_r50km, eonet_event_count_72h_r100km, nearest_eonet_distance_km, nearest_eonet_age_hours, category match & severity proxy
- —Operational: nearby_ready_responders_5km, eta_best_responder_min, historical_false_alarm_rate_zone, zone_confirmation_rate_30d
05Online scoring & penalties›
- 01Raw positive class probability from LightGBM.
- 02Calibrate through the fitted isotonic regressor.
- 03Apply uncertainty penalties (missing weather/sources, sparse zones, low diversity caps, contradictory signals), clamp to [0, 1].
- 04Return confidence, base_confidence, and auditable reasons[].
06Metrics we watch in production›
ROC-AUC, PR-AUC (rare positives), Brier score, calibration / ECE, recall at the chosen threshold, false-alarm rate by zone and disaster type, plus fairness slices (urban vs rural, high-signal vs low-signal).
07MLOps, audit, and API map›
Persist every inference with event_id, model_version, feature_set_version, base/final confidence, reasons, fallback flag, and timestamp.
| Method | Path | Role |
|---|---|---|
| POST | /ml/confidence/score | Core scorer: LightGBM → isotonic → penalty layer |
| GET | /ai/insights/summary | Dashboard narrative + model_version hook |
| GET | /ai/insights/actions | Ranked playbooks for operators |
| GET | /predictions | Nowcasts consume calibrated confidence |
| GET | /grid | Per-cell risk context for features |
| GET | /grid/nearby | Localized risk without full map sweep |
| GET | /events | Active signals for corroboration counts |
| GET | /reports | Citizen + channel reports for velocity features |
| POST | /external/ingest | NASA EONET / partner feeds enter the same feature fabric |
| GET | /responders/nearby | Responder density & ETA proxies |