ROX-33626: Add Prometheus metrics to alert manager by dashrews78 · Pull Request #19459 · stackrox/stackrox

@dashrews78 @claude

Add instrumentation to AlertAndNotify and mergeManyAlerts to collect
production data on alert processing performance before optimizing.

Metrics added:
- alert_and_notify_duration_ms: end-to-end duration histogram
- alert_and_notify_incoming_count: incoming alerts per call
- merge_many_alerts_duration_ms: merge duration histogram
- merge_many_alerts_previous_count: previous alerts fetched from DB
- alert_outcome_total: counter vec by outcome (new/updated/resolved)

Partially generated by AI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ajheflin

ajheflin

@dashrews78 @claude

Address PR feedback: extract observeDurationMs helper for the
defer-based duration pattern, and move outcome counting to a plain
recordAlertOutcomes call at the end of mergeManyAlerts instead of
using closures over named return values.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ajheflin

AlexVulaj