Monitoring Dashboard Audit

Structured assessment of monitoring infrastructure for network operations.
Evaluates Grafana dashboards, PromQL query efficiency, alert rule
configuration, SLA/SLO reporting accuracy, and Prometheus data source
health. This skill reads monitoring configuration and metrics — it does
not create, modify, or delete dashboards, alerts, or data sources.

Reference references/cli-reference.md for Grafana and Prometheus API
commands organized by audit step, and references/query-reference.md for
PromQL patterns covering network interface utilization, error rates, BGP
peer state, and SNMP-derived metric evaluation.

When to Use

• - Monitoring gap assessment — verifying that all critical network infrastructure has dashboard coverage and active alerting
• Dashboard quality review — evaluating whether existing Grafana dashboards present accurate, actionable data to operations teams
• Alert fatigue investigation — audit when teams report excessive or irrelevant alert notifications that mask genuine incidents
• SLA/SLO compliance review — validating that error budget calculations and availability metrics reflect actual service delivery
• Pre-migration monitoring readiness — confirming monitoring will survive infrastructure changes (new devices, topology changes, platform migrations)
• Post-incident review — assessing whether monitoring detected the incident, how quickly alerts fired, and what gaps allowed silent failures

Prerequisites

• - Grafana access — API token or service account with Viewer role minimum (grafana_url and Authorization: Bearer <token> header confirmed working)
• Prometheus access — HTTP API reachable at prometheus_url/api/v1/status/config (no authentication required by default, or appropriate auth header configured)
• Network scope defined — device inventory, subnet ranges, and critical service list available for coverage gap analysis
• Baseline documentation — existing SLA/SLO targets, expected alert thresholds, and operations runbooks available for comparison
• Timestamp awareness — confirm NTP synchronization across monitoring stack; Prometheus scrape timestamps and Grafana time range selections depend on consistent clocks

Procedure

Follow these six steps sequentially. Each step produces findings that
feed the monitoring coverage scorecard and optimization recommendations
in Step 6.

Step 1: Dashboard Inventory

Enumerate all Grafana dashboards to establish the monitoring surface
area and identify coverage gaps, stale dashboards, and organizational
issues.

Query the Grafana API to list all dashboards with metadata:

CODEBLOCK0

For each dashboard, record: uid, title, folder, tags, and last-updated
timestamp. Identify dashboards not updated in over 180 days as staleness
candidates — these may reference deprecated metrics or decommissioned
infrastructure.

Examine folder organization. Flat folder structures with 50+ dashboards
at root level indicate organizational debt. Check for naming convention
adherence — dashboards without consistent prefixes or tags reduce
discoverability during incidents.

Retrieve the full JSON model for each dashboard:

CODEBLOCK1

Record panel count, data source references, and template variables.
Flag dashboards with hardcoded time ranges (no relative time selector)
and panels referencing nonexistent data sources — these produce empty
panels that erode operator trust.

Build a coverage matrix: map dashboard panels to infrastructure
inventory. Devices, interfaces, or services present in inventory but
absent from any dashboard represent monitoring blind spots.

Step 2: Panel and Query Analysis

Evaluate PromQL query efficiency, panel threshold configuration,
and visualization appropriateness across all dashboards.

Extract all PromQL expressions from dashboard panel JSON targets.
For each query, assess:

Rate function usage — rate() requires a range vector at least
two scrape intervals wide. Replace irate() in dashboard panels that
display trends over long time ranges — irate() only uses the last
two data points and is appropriate only for volatile, short-window displays.

Recording rule candidates — Complex queries repeating across
multiple dashboards (same expression in 3+ panels) should be recording
rules. Identify these by hashing normalized PromQL expressions.
Common candidates: interface utilization calculations, error rate
ratios, and aggregated availability metrics.

Label cardinality — Queries aggregating across high-cardinality
labels without explicit filtering generate expensive computation.
Flag queries with no label matchers on high-cardinality metrics and
queries using {__name__=~".*"} patterns.

Panel thresholds — Verify gauge and stat panels have threshold
values configured. Panels displaying utilization or error rates
without color-coded thresholds fail to provide at-a-glance severity.
Compare configured thresholds against operational standards (e.g.,
interface utilization warning at 70%, critical at 90%).

Visualization appropriateness — Time series data on gauge panels
loses temporal context. Single-value stats for volatile metrics mislead
operators. Table panels with 100+ rows without sorting are unusable
during incidents.

Step 3: Alert Rule Validation

Assess alert rule configuration for detection coverage, threshold
accuracy, and notification reliability.

Retrieve all alert rules from the Grafana alerting API:

CODEBLOCK2

For Prometheus-native alerting, also query:

CODEBLOCK3

For each alert rule, evaluate:

Threshold appropriateness — Alert thresholds should align with
operational impact, not arbitrary percentages. Interface utilization
alerts at 50% on a 10Gbps link are premature; at 95% on a 100Mbps
WAN link they are late. Cross-reference thresholds against link
capacity and historical peak usage from Prometheus.

Evaluation intervals — Alert rules evaluated every 5 minutes
cannot detect sub-minute outages. For critical infrastructure (WAN
links, core routers), evaluation intervals should match or be less
than the Prometheus scrape interval. Flag alert groups where evaluation
interval exceeds the scrape interval of the underlying metrics.

Pending and for durations — for: 0s alerts fire on transient
spikes and contribute to alert fatigue. for: 30m on critical
infrastructure means 30 minutes of unnotified outage. Recommended
ranges: Critical alerts for: 2m-5m, Warning alerts for: 5m-15m.

Notification channels — Verify all alert rules have at least one
active notification channel. Check channel health — Slack webhooks
return 200, PagerDuty keys are valid, email SMTP is reachable.

Routing and silencing — Review Alertmanager routing tree for
catch-all routes dumping all alerts to a single channel. Verify
silences have expiration times. Active silences without expiration
mask ongoing problems indefinitely.

Escalation completeness — Critical alerts should escalate from
Slack/email to PagerDuty/phone after acknowledgment timeout. Alert
rules with only Slack notification for Critical-severity failures
lack escalation depth.

Step 4: SLA/SLO Reporting

Validate that SLA/SLO dashboards and calculations reflect actual
service delivery accuracy.

Error budget calculation — Verify the formula:
error_budget_remaining = 1 - (actual_errors / allowed_errors).
Common mistakes: using the wrong time window (calendar month vs
rolling 30 days), excluding planned maintenance from downtime
calculations, or computing availability from a single data source
when the service spans multiple components.

Availability SLIs — Check that uptime percentage, MTTR (Mean
Time to Repair), and MTBF (Mean Time Between Failures) use correct
inputs. Uptime should reference probe-based measurement (blackbox
exporter, synthetic checks), not just device-reported status. MTTR
excluding detection time understates actual recovery duration.

Burn rate alerting — Multi-window burn rate alerting provides
early warning when error budget consumption accelerates. Verify burn
rate alerts use at least two windows (e.g., 1-hour and 6-hour). A
single-window alert either fires too late or too often. Check that
severity maps to budget consumption: a 14.4x burn rate over 1 hour
warrants page-level severity.

Multi-window alert patterns — Confirm long-window alerts (6h, 3d)
for trend detection and short-window alerts (5m, 1h) for rapid
response. Verify severity increases with burn rate magnitude.

Step 5: Data Source Health

Assess Prometheus scrape target status, metric cardinality, retention
configuration, and remote write health.

Scrape target status — Query the targets API:

CODEBLOCK4

Check the up metric across all scrape targets. Targets with
up == 0 are failing to scrape — investigate network reachability,
exporter health, or authentication issues. Targets with
scrape_duration_seconds exceeding the scrape interval are timing
out, producing gaps in metrics and potentially triggering false alerts.

Cardinality assessment — Query TSDB status:

CODEBLOCK5

Identify the top 10 metrics by series count. Network environments
commonly see cardinality explosion from per-interface SNMP metrics
on large chassis devices. Metrics exceeding 10,000 active series per
name warrant investigation for label optimization or aggregation.

Retention and storage — Verify retention period covers the longest
SLA reporting window. A 15-day retention with 30-day SLA dashboards
produces incomplete reports. Check WAL size — a WAL larger than 20%
of total TSDB size may indicate write amplification from high churn.

Remote write health — If Prometheus uses remote write (Thanos,
Cortex, Mimir, VictoriaMetrics), check remote storage lag metrics.
Lag exceeding 5 minutes means long-term store is behind real-time.
Flag failed sample counters as write failures creating data gaps.

Metric naming conventions — Verify metrics follow Prometheus
naming conventions: snakecase, unit suffixes (bytes, _seconds,
_total), base units. Inconsistent naming makes dashboard authoring
error-prone and cross-device comparison unreliable.

Step 6: Monitoring Coverage Report

Compile findings into a structured monitoring coverage scorecard
with prioritized optimization recommendations.

Coverage scorecard — Rate each infrastructure category
(core routers, distribution switches, WAN links, firewalls,
load balancers) on a 1–5 scale: 1 = no monitoring, 2 = basic
up/down only, 3 = utilization dashboards, 4 = dashboards with
alerting, 5 = dashboards with alerting and SLO tracking.

Alert quality assessment — Compute alert quality metrics:
alert-to-incident ratio, mean time to acknowledge, silence
frequency per alert rule. High-noise alerts (>80% silenced or
acknowledged-without-action) are candidates for threshold
adjustment or removal.

PromQL optimization recommendations — For each inefficient
query from Step 2, provide current and optimized expressions.
Prioritize recording rule creation for queries appearing in 3+
panels.

Threshold Tables

Domain	Severity	Condition	Example
Coverage	Critical	Production device with zero dashboard panels	Core router absent from all dashboards
Coverage

High | Service with dashboards but no alerting | WAN link monitored but no capacity alert | | Coverage | Medium | Dashboard exists but is stale (>180 days) | Last-modified date precedes device refresh | | Coverage | Low | Dashboard lacks threshold coloring | Utilization panel with no warning/critical bands | | Query | Critical | PromQL uses absent metric name | Panel returns no data due to renamed metric | | Query | High | rate() range vector shorter than 2x scrape interval | rate(metric[15s]) with 30s scrape | | Query | Medium | Repeated query across 3+ panels without recording rule | Same utilization formula in 5 dashboards | | Query | Low | irate() used for trend display over long time range | irate() on 24h overview panel | | Alert | Critical | Alert rule with no notification channel | Silent alarm on Critical infrastructure | | Alert | High | Critical alert with for duration >15m | 15+ minute detection gap | | Alert | Medium | Warning alert with for duration of 0s | Transient spikes cause alert fatigue | | Alert | Low | Single notification channel with no escalation | Slack-only for P1 infrastructure alert | | SLA | Critical | Error budget calculation uses wrong time window | Calendar month vs rolling 30d mismatch | | SLA | High | Availability SLI excludes detection time from MTTR | MTTR underreported by omitting MTTD | | SLA | Medium | Single-window burn rate alerting | Only 1h window, no long-term trend window | | SLA | Low | SLO dashboard missing historical trend comparison | No month-over-month burn rate trend | | DataSource | Critical | Scrape target down (up == 0) for >5m | SNMP exporter unreachable | | DataSource | High | Cardinality >10k series per metric name | Per-interface metrics on 500-port chassis | | DataSource | Medium | Remote write lag >5m | Long-term store behind real-time | | DataSource | Low | Metric naming violates Prometheus conventions | CamelCase or missing unit suffix |

Decision Trees

CODEBLOCK6

Report Template

CODEBLOCK7

Troubleshooting

Grafana API returns 401/403 — Verify the API token has Viewer
permissions. Service accounts require Viewer role on all folders
containing in-scope dashboards. Admin-level tokens are not required
for read-only audit.

Prometheus API unreachable — Check network connectivity and reverse
proxy configuration. Verify the base URL includes the correct path
prefix if Prometheus runs behind a subpath.

Empty dashboard inventory — Grafana API search returns paginated
results. Increase the limit parameter or paginate. Folder-level
permissions can hide dashboards from restricted tokens.

PromQL queries reference absent metrics — SNMP exporter metric
names change between exporter versions. Check exporter version and
generator configuration when dashboards return no data.

Cardinality data not available — The TSDB status endpoint requires
Prometheus 2.14+. If unavailable, estimate cardinality using count
queries, though these are expensive on large installations.

SLA calculations differ from business reports — Time zone handling
is the most common cause. Prometheus stores UTC timestamps. Verify
all SLA dashboards use explicit UTC time ranges or a consistent
time zone variable.

Alert rules exist in both Grafana and Prometheus — Grafana Unified
Alerting coexists with Prometheus-native alerting. Audit both sources
to avoid duplicate or conflicting coverage. Document which system is
authoritative for each alert category.