Metrics & Alerts

Metrics are derived from traces and pre-aggregated into 1-minute buckets, so they survive trace retention and are cheap to query over long windows.

Available metrics

Metric	Type	Dimensions
run.count	counter	agent, environment, revision, status
run.duration	histogram	agent, environment, revision
run.tokens	counter	agent, environment, model, direction
tool.count	counter	agent, tool, server, status
tool.duration	histogram	agent, tool, server
model.throttled	counter	agent, model
queue.age	gauge	agent, environment

velerion metrics query run.duration --agent support-triage --env prod \
  --since 24h --percentile 95 --group-by revision

What to alert on

These four catch nearly everything that matters, and each has an unambiguous response.

Alert	Condition	Why
Run error rate	status="error" > 2% over 10 min	The agent is failing, not merely degraded.
Tool error rate	any tool > 5% over 10 min	Usually an upstream dependency, not the agent.
Latency regression	p95 up > 50% vs the previous revision	Almost always a prompt or tool change.
Throttling	model.throttled > 0 sustained 5 min	Capacity problem; queues will grow behind it.

velerion alert create high-error-rate \
  --metric run.count --filter 'status = "error"' \
  --condition 'ratio > 0.02' --window 10m \
  --env prod --notify pagerduty:platform-oncall

What not to alert on

• Token spend spikes. Alert on the daily total, not the minute rate. A batch job will page you every time otherwise.
• A single failed run. Individual runs fail for reasons outside your control. Alert on rates.
• stop_reason = "max_tokens". Worth a dashboard and a weekly look; it is a quality signal, not an availability one.
• Absolute p95 latency. Model latency moves on its own. Alert on the change relative to the previous revision, which is the part you caused.

Tip

Alert on queue.age rather than queue length. Length tells you how much work exists; age tells you whether it is being worked through. A long queue draining steadily is fine.

Dashboards

The Console ships a per-agent dashboard covering the four alert conditions plus token spend and tool mix. It is the right default. Build your own only when you need a dimension Velerion does not carry — in which case export to your own stack, since the metrics API is not designed to back a high-refresh dashboard.

Cost reporting

velerion usage --since 2026-07-01 --group-by agent,model --json \
  | jq -r '.[] | [.agent, .model, .input_tokens, .output_tokens] | @tsv'

Usage is reconciled nightly and is the billing source of truth; run.tokens is real-time and may differ slightly for runs in flight at the bucket boundary. Use usage for anything that appears on an invoice or in a chargeback report, and run.tokens for alerting.

Next steps

• Traces — the detail behind these aggregates.
• Access Control — who can read traces and metrics.