Prometheus | DevOps QuickRef

Prometheus scrapes and stores time-series metrics, and PromQL is how you query them. Most real usage is a handful of query patterns repeated against different metric names.

Metric Types

The four types

Counter — cumulative, only increases (request count, errors total). Gauge — a value that goes up or down (memory usage, queue depth). Histogram — samples observations into configurable buckets (request duration). Summary — like a histogram, but computes quantiles client-side instead of letting the server aggregate them.

Basic Queries

up

Returns 1 for every target Prometheus is successfully scraping, 0 for every target it can't reach — the first thing to check when 'metrics are missing.'

http_requests_total

Returns the current value of a counter, labeled by every dimension it was recorded with (method, status, path, …).

http_requests_total{status="500"}

Filters a metric to series matching a specific label value.

http_requests_total{status=~"5.."}

Filters using a regex label matcher — here, any 5xx status code.

Rate & Aggregation

rate(http_requests_total[5m])

Computes the per-second average rate of increase over a 5-minute window. Always wrap a counter in rate() before graphing or alerting on it — a raw counter is a meaningless ever-growing line.

sum(rate(http_requests_total[5m])) by (status)

Sums the per-second rate across all instances, grouped by one label — the standard shape for a request-rate-by-status dashboard panel.

avg(node_memory_available_bytes) by (instance)

Averages a gauge across whatever dimension you group by.

topk(5, rate(http_requests_total[5m]))

Returns only the 5 series with the highest current value — useful for 'which endpoints are hottest right now.'

Histograms & Quantiles

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Computes the 95th percentile latency from a histogram's bucket counts — the standard way to derive p95/p99 from Prometheus histograms.

Histogram quantiles are approximations

histogram_quantile interpolates within whatever buckets you configured — a p99 computed from buckets with a gap around the real p99 value will be inaccurate. Bucket boundaries matter; the default buckets are rarely right for every metric.

Alerting Rules

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Error rate above 5% for 10 minutes"

Always set for:

Without for:, an alert fires on the very first evaluation that crosses the threshold — a single scrape blip pages someone. for: 10m requires the condition to stay true across that whole window first, filtering out noise while still catching anything that's genuinely sustained.

Operations

promtool check config prometheus.yml

Validates a Prometheus config file's syntax before reloading — catches a bad scrape config before it takes the whole server down.

promtool check rules alerts.yml

Validates alerting/recording rule syntax.

curl -X POST http://localhost:9090/-/reload

Hot-reloads config and rules without restarting the process, if Prometheus was started with --web.enable-lifecycle.

curl http://localhost:9090/api/v1/targets

Lists every scrape target and its current health, straight from the API — useful when the UI is slow or unavailable.