Document infra and functional alert scenarios

[KrutikaPhirangi]

Before configuring alerts under #34, document which alerts we need, why each one matters, and what should happen when it fires. This document will be the agreed spec that the alert configuration is built from.

Scope

Define the alert scenarios in two groups:

Infra alerts: the health of the platform the services run on (for example CPU, memory, disk, pod or container restarts, database availability).

Functional alerts: the behaviour of the OAN services themselves (for example API failures, high latency, publish failures, discover errors, registry latency).

For each alert, document

• Name and description: what it detects
• Signal / source: metric, trace or log, and the ClickHouse table or query it comes from
• Condition and threshold: for example error rate > X% over Y minutes
• Severity: critical / warning
• Notification channel: who gets notified and where
• Runbook: first steps to investigate or mitigate, and which dashboard to open

Acceptance criteria

• List of infra alert scenarios, agreed with the team
• List of functional alert scenarios, agreed with the team
• Thresholds, severity and routing defined for each alert
• Document added to the repo and linked from Configure infra and functional alerts #34

[KrutikaPhirangi]

I have attached the documentation outlining the alerting scenarios.
https://docs.google.com/document/d/1ppMeCWyKkn_1UoSKNmBMLB1wrmbP6jkM4VMnfCUuc_A/edit?usp=sharing

[manjudr]

@KrutikaPhirangi, I’ve reviewed the documentation, and it includes granular functional alerts that would be challenging to define. Instead, define the alerts at a high level, and let the description or summary explain the details of each alert.

Please rethink it and come back with updated documentation.