Skip to content

Operations

An alarm is a threshold crossing. A finding is a conclusion.

Monitoring systems generate alarms faster than any team can act on them, so teams stop acting on them. The fix is not better thresholds — it is changing what the system is asked to produce.

· 3 min read

Every operations team has the same relationship with their monitoring system. It generates more alerts than anyone can work through, so the list stops being worked through, and the one alert that mattered arrives into a queue nobody is reading.

The usual response is to tune the thresholds. That helps for a while, then the fleet grows or the weather changes and the queue fills again — because the underlying problem is not where the threshold sits. It is what the system was asked to produce.

The difference

An alarm says a value crossed a line. It carries no opinion about whether that matters, what caused it, or what it costs to ignore. Deciding all three is left to whoever opens the queue, which is why the queue does not get opened.

A finding is a conclusion with the evidence attached and a cost of inaction stated. It is something you can act on or explicitly decline to act on — both of which are progress.

Producing findings rather than alarms requires knowing what the asset should be doing, not just what it is doing. Without a model of expected behaviour, a system can only report numbers back at you and let you decide which are surprising.

What this changes in practice

  • The output is ranked by what it costs to leave alone, not by how many alerts fired.
  • Every item carries the measurement it came from, so an engineer can check the reasoning rather than trust it.
  • Gradual degradation surfaces, because a deviation from an expectation does not have to breach an absolute threshold to be visible.

That last point is where threshold-based monitoring is structurally weakest. Slow degradation stays under any absolute limit until it has already cost a season of generation. Measured against an expectation, it is visible from the first week.

Start with how it works.

The architecture, the modeling approach, and where each part of the platform actually stands today.