Data
Resolution discarded on ingestion cannot be recovered at any price
Averaging telemetry as it arrives makes storage cheap and destroys the signal that identifies the fault. The five-minute question you cannot ask in two years is the one that matters.
Telemetry is expensive to keep, so the standard move is to average it down on the way in. Five-minute readings become hourly. Hourly becomes daily. Storage costs fall by orders of magnitude and the charts look identical.
What the average removes
The charts look identical because averaging preserves exactly the thing charts show — the trend — and removes exactly the thing diagnosis needs: the shape of the deviation.
- A string that drops out for eleven minutes disappears entirely into an hourly mean.
- Shading that moves across an array over the afternoon becomes a single slightly-low number, indistinguishable from mild soiling.
- An inverter clipping at midday and a cloud passing overhead average to the same reduction in output, and only one of them is your problem.
Aggregation is a one-way operation. You can always compute the hourly mean from five-minute data. You can never recover the five minutes from the hour.
Aggregate at query time instead
The alternative is to store signals at the frequency they were produced and aggregate when the question is asked. Someone wanting a monthly summary still gets one. Someone asking why generation was 4% low on a Tuesday in March two years ago still has the interval data to answer it.
This costs more in storage. It is worth stating that plainly rather than pretending the trade-off does not exist. What it buys is that the expensive question — the one asked once a year, in a dispute, with money attached — remains answerable.
There is a related discipline that matters as much: bad data is marked as bad rather than silently interpolated. A gap that is filled invisibly is indistinguishable from a measurement, and an invented value is worse than a missing one.
Start with how it works.
The architecture, the modeling approach, and where each part of the platform actually stands today.