Set up alerts

The alert engine runs inside the binary and is on by default. It evaluates persisted rules every PITOT_ALERTS_EVALUATION_INTERVAL (30s) against service rollups, and delivers by webhook, by email, or both.

Create and manage a rule

Open Alerts in the sidebar, then choose New rule. Enter a name and expression, plus an optional description, email recipients and webhook URL. A rule with neither email recipients nor a webhook URL goes to the default email destination when one is configured. Without one, the rule records alerts but delivers nothing. Leave Service empty to evaluate every service, or select one service. Choose Start from a template for an example expression, then set the timers and save.

The new-rule form starts enabled, with Hold for (s) 60, Cooldown (s) 300 and Repeat every (s) 3600. Notify when resolved starts off. The name, description, service, expression, email and webhook fields start empty.

Use the Enabled toggle to pause or resume a rule. The rule’s actions menu has Edit, Test against current data and Delete. Testing evaluates the saved expression without changing alert state, sending a webhook or sending email. Deleting removes the rule and its alert history.

Choose Open in Explore to chart a watched measure from raw telemetry. Its values can differ from the rule’s rollups. Explore shows error rate as a percent: 0.05 in a rule is 5 there.

Read current states and delivery results under History. PITOT_ALERTS_HISTORY_DAYS sets resolved-history retention in days (default 7).

Viewers can read rules and history, without a rule’s delivery settings: webhook URL, headers and template, email recipients, and email failure detail are returned only to operators and administrators. Creating, editing, toggling, testing and deleting require an operator or administrator session; see roles.

What a rule contains

An expression, an optional service to scope it to, timing, and delivery:

Field	Means
name	Required rule name
description	Optional text explaining the rule
enabled	Whether to evaluate the rule
expression	The condition, in CEL
service	Evaluate only for this service; omit for all
for_seconds	How long the condition must hold before firing
cooldown_s	Minimum quiet period after resolving before firing again
repeat_interval_s	How often to re-deliver while still firing
email_recipients	Array of up to 10 addresses to email; omit for none
webhook_url	Where to deliver by webhook
webhook_headers	JSON-encoded object of extra headers, supplied as a string
webhook_template	Go text/template payload body; omit for the default
notify_on_resolve	Also deliver when the alert clears

Nonempty expressions supplied when creating or replacing a rule are type-checked against the declared variables and must return a boolean. An unknown variable or a type mistake is rejected at save time. Supply a complete rule body when replacing it; omitted fields take their zero values.

Variables available to an expression

An expression reads one service’s rollup for the current window. The alert expression reference lists every variable, including error_rate, p95, throughput and the baseline deltas.

For example:

error_rate > 0.02 && throughput > 10

throughput is the total span count in the rollup window. A traffic floor prevents a single failure on a quiet service from triggering this rule.

The lifecycle

An alert moves through three states:

• pending: the condition is true but has not held for for_seconds yet. With for_seconds: 0 a rule skips this and fires immediately.
• firing: it has held. The webhook and email are delivered, and re-delivered every repeat_interval_s while it stays true.
• resolved: the expression went false. If notify_on_resolve is set, that is delivered too.

cooldown_s then holds the rule quiet for that long after resolving, which is what stops a metric hovering at the threshold from delivering repeatedly.

Each alert records its last delivery status and timestamp in GET /api/alerting/alerts. Setting repeat_interval_s to zero disables repeat delivery. Disabling a rule removes pending alerts and resolves firing alerts, sending a resolve notification when notify_on_resolve is set.

The namespace field on a rule is not applied

A rule stores namespace, but evaluation combines service rollups across namespaces. Scope with service; services with the same name in different namespaces are evaluated together.

Email delivery

Email uses the SMTP settings Pitot already uses for sign-in codes. Without SMTP the option is disabled in the rule form, and the API rejects email_recipients with the code email_unavailable. GET /api/alerting/email reports whether email is available.

Set PITOT_ALERTS_EMAIL_DEFAULT_TO to a comma-separated list of addresses to give every rule a destination. It applies only to a rule with no webhook and no recipients of its own, including rules created before email existed. A rule with a webhook and no recipients stays webhook-only.

Email follows the rule’s existing timing: for_seconds, cooldown_s, repeat_interval_s and notify_on_resolve apply as they do to webhooks.

Addresses are checked when saved. Each must be a bare address, without a display name. They are stored in lowercase and a rule can have up to 10.

Retries and restarts

A notification is queued in the control database in the same transaction as the alert’s state change, with one row per recipient. Pitot makes up to five attempts, waiting 1, 5, 15, then 30 minutes between them. Connection failures, including a connection lost partway through a session, and 4xx SMTP replies are retried. A 5xx reply or an unusable address fails at once. Once the server has accepted a message, a failure while closing the session does not count against it. Before a retry, Pitot drops a recipient no longer on the rule, and deleting a rule cancels its queued mail.

Delivery is at least once. A restart resumes queued mail. If Pitot stops or the worker panics while a message is being handed to the mail server, or a connection drops before the server confirms it, Pitot cannot tell whether the server took it, so it sends the message again. The repeat has the same Message-ID. The same happens after a restart if the database could not record an accepted message before Pitot stopped.

The hourly limit

PITOT_ALERTS_EMAIL_HOURLY_LIMIT (default 30) caps the alert emails sent to one address in any rolling hour, across all rules. The count is stored apart from rules, so neither a restart nor deleting a rule resets it. The last email within the limit says that the limit has been reached. Notifications over the limit are suppressed: they are dropped, never sent later, and logged as a warning.

Seeing what happened

A rule whose latest email did not go through shows it in the Rules table, and in email_status and email_status_detail on GET /api/alerting/rules:

email_status	Means
sent	Every recipient accepted the latest notification
retrying	An attempt failed and another is scheduled
failed	At least one recipient could not be reached
suppressed	At least one recipient was over the hourly limit
unavailable	The rule has recipients but SMTP is not configured

Viewers see email_status only. email_status_detail gives the failure class and SMTP code, not the mail server’s response text.

For a rule with no webhook, the alert’s delivery status under History carries success, failed or suppressed for the email. When a rule has a webhook, that column stays the webhook’s result.

Rule names, descriptions and service names are reduced to a single line of printable text in the subject and body, and escaped in the HTML part, so a hostile service name cannot add mail headers or markup.

Why alerts lag the raw data

Evaluation reads analytical rollups. The rollup pass follows durable telemetry publication; the evaluation interval and for_seconds add further delay. How storage works explains the publication horizon and watermark. A shorter evaluation interval does not increase the rollups’ time resolution.

Webhook payload

Leave webhook_template empty to send this body:

{"rule":"{{.Rule.Name}}","service":"{{.Alert.Service}}","state":"{{.Alert.State}}","event":"{{.Event}}","value":{{.Alert.Value}}}

Templates use Go text/template and exported field names, with this context:

Value	Fields
.Rule	ID, Name, Description, Enabled, Service, Namespace, Expression, ForSeconds, CooldownS, RepeatIntervalS, WebhookURL, WebhookHeaders, WebhookTemplate, NotifyOnResolve, CreatedAt, UpdatedAt
.Alert	ID, RuleID, Service, State, Value, FiredAt, ResolvedAt, RepeatedAt, LastEvaluatedAt, LastDeliveryStatus, LastDeliveryAt, CreatedAt
.Event	firing, reminder or resolved
.Env	ErrorRate, P50, P95, Throughput, LogCount, ZScore, HealthScore, ErrorRateDelta, P95Delta, ThroughputDelta, Service
.Time	Delivery context time as a Go time.Time

.Alert.Value is the first variable the expression reads, at the last evaluation. Timestamp fields on .Rule and .Alert are strings.

For a Slack incoming webhook, set its URL and use this template:

{"text":{{printf "%q" (printf "%s: %s on %s (value %g)" .Event .Rule.Name .Alert.Service .Alert.Value)}}}

This uses the built-in printf function; no JSON template helper is registered. text/template does not automatically escape values for JSON. With this example, use names without control characters other than tabs and line breaks.

Pitot sends a POST with Content-Type: application/json unless custom headers override it. Each attempt has a 10-second total timeout and a 5-second response header timeout. Delivery makes up to three attempts, waiting 2 then 4 seconds before retries; 4xx responses are not retried.

Use the HTTP API

The browser uses these routes. Reads require telemetry:read; mutations and tests require alerts:manage.

Endpoint	Does
GET /api/alerting/alerts	Current alert state
GET /api/alerting/summary	Counts by state
GET /api/alerting/rules	List rules
POST /api/alerting/rules	Create and compile a rule
PUT /api/alerting/rules/:id	Replace a rule and check any nonempty expression
POST /api/alerting/rules/:id/test	Evaluate a saved expression against live data without changing alert state or delivering a webhook
DELETE /api/alerting/rules/:id	Delete the rule and its alerts

When creating through HTTP, omitted booleans are false, timers are zero and optional strings are empty. In particular, omitted enabled means disabled. These API defaults differ from the new-rule form above.