Every technology organisation says it values reliability. Far fewer can tell you, at any given moment, what their systems are doing.

The gap between what we assume about our systems and what we observe about them is where most operational pain lives. It is also where the most improvements are available.

Systems fail. Plan for that.

The objective of reliability engineering is not to pretend failures don’t happen. Systems fail: hardware, networks, dependencies, deployments, and the assumptions people made eighteen months ago that no longer hold. The objective is to design organisations and platforms that detect problems quickly, understand them quickly and recover quickly.

Prevention matters, but it is a strategy for a world that doesn’t exist. Detection, understanding and recovery are strategies for the one we operate in.

Reliability is an engineering discipline

I see reliability as an engineering discipline rather than simply an operations responsibility. That has concrete consequences.

Monitoring should be released with the application. If the team that builds a service is not also building its observability, the monitoring will describe what someone guessed the service does, not what it does. Telemetry should tell us what systems are doing rather than what we expect them to be doing.

Operational readiness should be part of engineering. The questions that decide whether a service survives production (what happens when this dependency is slow, when this queue backs up, when this instance disappears) belong in design reviews, not in post-incident retrospectives.

Lessons from production should feed directly back into architecture and development. An incident that ends with a fix but no change to how systems are designed will repeat itself, with better documentation.

What matters

When I look at how production environments degrade, a short list of things keeps appearing. A much longer list of things everyone worries about rarely does.

• Service health from the user’s perspective: internal metrics that look green while customers see errors are worse than no metrics, because they manufacture confidence.
• Deployment safety: most incidents are changes. The faster and more reversible a change is, the less each change costs you. Small, frequent, observable deployments beat large, rare, ceremonial ones.
• Dependencies and their failure modes: every dependency you add is a reliability decision. Fewer dependencies, understood dependencies, and explicit behaviour when they fail.
• Capacity before it becomes an incident: saturation events are incidents you chose to have early.
• Alerting you can act on: an alert that fires when nobody needs to act trains people to ignore alerts. Noise reduction is a reliability function.

What mostly doesn’t: impressive dashboards nobody looks at between incidents. Process that exists to assign blame rather than change outcomes. Tooling debates that substitute for the harder conversation about ownership.

Incidents are information

The best technology organisations I have worked in treated incidents as information about the work, not interruptions to it. A structured incident response (clear roles, a single source of truth, calm communication) followed by root-cause analysis that asks “what about our system made this possible?” rather than “who touched it last?”

Continuous improvement following incidents is the difference between an organisation that has ten years of experience and one that has one year of experience, ten times.

The AI footnote

None of this changes with AI. It gets sharper. AI-assisted monitoring and agents that perform first-pass triage raise the ceiling on how quickly problems can be detected and understood. They also lower the cost of ignoring the fundamentals, because an agent reasoning over rich telemetry is worth far more than an agent reasoning over guesses.

If a system matters, it should be observable. That was true before autonomous operations, and it is the precondition for them now.

Satish Chandran is a senior technology leader specialising in engineering, cloud infrastructure, DevOps, cybersecurity, data and Artificial Intelligence.