Silent-failure detection for a distributed capture fleet

The problem

An uptime check only catches a device that stops. The expensive failures are the ones where volume looks normal but the data is stale, filed under the wrong day, or missing coverage.

This project simulates 314 devices across 110 facilities (28,260 device-days) with 43 failures injected at known times, and compares a week-over-week volume baseline with four detectors.

Headline numbers

• The volume baseline catches 18 of 43 failures; the proposed detectors catch 43 of 43.
• Bad device-days fall from 1,097 to 58, down 95%.
• Stale-repeat, clock-drift and coverage-gap failures: none caught by volume, all caught by content fingerprinting, clock skew and zone coverage.
• On a second fleet with a different seed and nothing retuned: 19 of 42 for the baseline versus 42 of 42.

What it means

Monitoring that watches row counts misses the failures where every dashboard stays green. Content fingerprints, clock skew and coverage checks need no per-device tuning, and alerts work best grouped into incidents and ranked in the customer's units.