Interactive Prometheus

Labels, Rules, and Alert Routing

See how metric labels flow through recording rules, alerting rules, and Alertmanager routing.

PromQL label query

avg by (team, environment) (
  probe_success{job="url-monitors"}
)

Core example

probe_success{
  job="url-monitors",
  monitor_id="mon_1001",
  team="checkout",
  environment="prod",
  instance="https://checkout.example.com/health"
} 0

Cardinality diagram

PromQL labels are part of the metric data model

Prometheus stores one time series per unique label set. A single metric name can expand into many stored series as labels multiply.

Metric namehttp_requests_total

servicecheckout, billing, profile3 values

methodGET, POST2 values

status200, 404, 5003 values

environmentprod, staging2 values

Bounded label sets36

3 services x 2 methods x 3 statuses x 2 environments

Bad extra labeluser_id

user_idu_1931, u_1932, u_1933, ...1,000,000+ values

request_idreq_a81, req_a82, req_a83, ...50,000,000+ values

With user_id added36,000,000

Same metric name, dangerous number of unique label sets.

Recording Rules vs Alerting Rules

Recording rules

Precompute PromQL into new time series. Use for repeated dashboard queries, expensive aggregations, SLO base metrics, and shared derived metrics.

Alerting rules

Create alert objects from PromQL conditions. Use when humans or automation need to be notified and Alertmanager should route the event.