--- title: "Set up alerts" summary: "The rule model, the variables available to an expression, the pending/firing/resolved lifecycle, and webhook and email delivery." description: "Write alert rules as expressions over service metrics, and deliver them by webhook or email." status: preview read_when: - "You want to be told when a service degrades." - "A rule is not firing when you expect it to." --- The alert engine runs inside the binary and is on by default. It evaluates persisted rules every `PITOT_ALERTS_EVALUATION_INTERVAL` (30s) against service rollups, and delivers by webhook, by email, or both. ## Create and manage a rule Open **Alerts** in the sidebar, then choose **New rule**. Enter a name and expression, plus an optional description, email recipients and webhook URL. A rule with neither email recipients nor a webhook URL goes to the [default email destination](#email-delivery) when one is configured. Without one, the rule records alerts but delivers nothing. Leave **Service** empty to evaluate every service, or select one service. Choose **Start from a template** for an example expression, then set the timers and save. The new-rule form starts enabled, with **Hold for (s)** 60, **Cooldown (s)** 300 and **Repeat every (s)** 3600. **Notify when resolved** starts off. The name, description, service, expression, email and webhook fields start empty. Use the **Enabled** toggle to pause or resume a rule. The rule's actions menu has **Edit**, **Test against current data** and **Delete**. Testing evaluates the saved expression without changing alert state, sending a webhook or sending email. Deleting removes the rule and its alert history. Choose **Open in Explore** to chart a watched measure from raw telemetry. Its values can differ from the rule's rollups. Explore shows error rate as a percent: `0.05` in a rule is `5` there. Read current states and delivery results under **History**. `PITOT_ALERTS_HISTORY_DAYS` sets resolved-history retention in days (default 7). Viewers can read rules and history, without a rule's delivery settings: webhook URL, headers and template, email recipients, and email failure detail are returned only to operators and administrators. Creating, editing, toggling, testing and deleting require an operator or administrator session; see [roles](/docs/reference/roles). ## What a rule contains An expression, an optional service to scope it to, timing, and delivery: | Field | Means | |---|---| | `name` | Required rule name | | `description` | Optional text explaining the rule | | `enabled` | Whether to evaluate the rule | | `expression` | The condition, in [CEL](https://cel.dev) | | `service` | Evaluate only for this service; omit for all | | `for_seconds` | How long the condition must hold before firing | | `cooldown_s` | Minimum quiet period after resolving before firing again | | `repeat_interval_s` | How often to re-deliver while still firing | | `email_recipients` | Array of up to 10 addresses to email; omit for none | | `webhook_url` | Where to deliver by webhook | | `webhook_headers` | JSON-encoded object of extra headers, supplied as a string | | `webhook_template` | Go text/template payload body; omit for the default | | `notify_on_resolve` | Also deliver when the alert clears | Nonempty expressions supplied when creating or replacing a rule are type-checked against the declared variables and must return a boolean. An unknown variable or a type mistake is rejected at save time. Supply a complete rule body when replacing it; omitted fields take their zero values. ## Variables available to an expression An expression reads one service's rollup for the current window. The [alert expression reference](/docs/reference/alert-expressions) lists every variable, including `error_rate`, `p95`, `throughput` and the baseline deltas. For example: ```text error_rate > 0.02 && throughput > 10 ``` `throughput` is the total span count in the rollup window. A traffic floor prevents a single failure on a quiet service from triggering this rule. ## The lifecycle An alert moves through three states: - **pending**: the condition is true but has not held for `for_seconds` yet. With `for_seconds: 0` a rule skips this and fires immediately. - **firing**: it has held. The webhook and email are delivered, and re-delivered every `repeat_interval_s` while it stays true. - **resolved**: the expression went false. If `notify_on_resolve` is set, that is delivered too. `cooldown_s` then holds the rule quiet for that long after resolving, which is what stops a metric hovering at the threshold from delivering repeatedly. Each alert records its last delivery status and timestamp in `GET /api/alerting/alerts`. Setting `repeat_interval_s` to zero disables repeat delivery. Disabling a rule removes pending alerts and resolves firing alerts, sending a resolve notification when `notify_on_resolve` is set. > **The namespace field on a rule is not applied** > > A rule stores `namespace`, but evaluation combines service rollups across > namespaces. Scope with `service`; services with the same name in different > namespaces are evaluated together. ## Email delivery Email uses the [SMTP settings](/docs/reference/configuration/smtp) Pitot already uses for sign-in codes. Without SMTP the option is disabled in the rule form, and the API rejects `email_recipients` with the code `email_unavailable`. `GET /api/alerting/email` reports whether email is available. Set `PITOT_ALERTS_EMAIL_DEFAULT_TO` to a comma-separated list of addresses to give every rule a destination. It applies only to a rule with no webhook and no recipients of its own, including rules created before email existed. A rule with a webhook and no recipients stays webhook-only. Email follows the rule's existing timing: `for_seconds`, `cooldown_s`, `repeat_interval_s` and `notify_on_resolve` apply as they do to webhooks. Addresses are checked when saved. Each must be a bare address, without a display name. They are stored in lowercase and a rule can have up to 10. ### Retries and restarts A notification is queued in the control database in the same transaction as the alert's state change, with one row per recipient. Pitot makes up to five attempts, waiting 1, 5, 15, then 30 minutes between them. Connection failures, including a connection lost partway through a session, and 4xx SMTP replies are retried. A 5xx reply or an unusable address fails at once. Once the server has accepted a message, a failure while closing the session does not count against it. Before a retry, Pitot drops a recipient no longer on the rule, and deleting a rule cancels its queued mail. Delivery is at least once. A restart resumes queued mail. If Pitot stops or the worker panics while a message is being handed to the mail server, or a connection drops before the server confirms it, Pitot cannot tell whether the server took it, so it sends the message again. The repeat has the same `Message-ID`. The same happens after a restart if the database could not record an accepted message before Pitot stopped. ### The hourly limit `PITOT_ALERTS_EMAIL_HOURLY_LIMIT` (default 30) caps the alert emails sent to one address in any rolling hour, across all rules. The count is stored apart from rules, so neither a restart nor deleting a rule resets it. The last email within the limit says that the limit has been reached. Notifications over the limit are suppressed: they are dropped, never sent later, and logged as a warning. ### Seeing what happened A rule whose latest email did not go through shows it in the **Rules** table, and in `email_status` and `email_status_detail` on `GET /api/alerting/rules`: | `email_status` | Means | |---|---| | `sent` | Every recipient accepted the latest notification | | `retrying` | An attempt failed and another is scheduled | | `failed` | At least one recipient could not be reached | | `suppressed` | At least one recipient was over the hourly limit | | `unavailable` | The rule has recipients but SMTP is not configured | Viewers see `email_status` only. `email_status_detail` gives the failure class and SMTP code, not the mail server's response text. For a rule with no webhook, the alert's delivery status under **History** carries `success`, `failed` or `suppressed` for the email. When a rule has a webhook, that column stays the webhook's result. Rule names, descriptions and service names are reduced to a single line of printable text in the subject and body, and escaped in the HTML part, so a hostile service name cannot add mail headers or markup. ## Why alerts lag the raw data Evaluation reads analytical rollups. The rollup pass follows durable telemetry publication; the evaluation interval and `for_seconds` add further delay. [How storage works](/docs/concepts/storage#rollups-follow-durable-publication) explains the publication horizon and watermark. A shorter evaluation interval does not increase the rollups' time resolution. ## Webhook payload Leave `webhook_template` empty to send this body: ```text {"rule":"{{.Rule.Name}}","service":"{{.Alert.Service}}","state":"{{.Alert.State}}","event":"{{.Event}}","value":{{.Alert.Value}}} ``` Templates use Go `text/template` and exported field names, with this context: | Value | Fields | |---|---| | `.Rule` | `ID`, `Name`, `Description`, `Enabled`, `Service`, `Namespace`, `Expression`, `ForSeconds`, `CooldownS`, `RepeatIntervalS`, `WebhookURL`, `WebhookHeaders`, `WebhookTemplate`, `NotifyOnResolve`, `CreatedAt`, `UpdatedAt` | | `.Alert` | `ID`, `RuleID`, `Service`, `State`, `Value`, `FiredAt`, `ResolvedAt`, `RepeatedAt`, `LastEvaluatedAt`, `LastDeliveryStatus`, `LastDeliveryAt`, `CreatedAt` | | `.Event` | `firing`, `reminder` or `resolved` | | `.Env` | `ErrorRate`, `P50`, `P95`, `Throughput`, `LogCount`, `ZScore`, `HealthScore`, `ErrorRateDelta`, `P95Delta`, `ThroughputDelta`, `Service` | | `.Time` | Delivery context time as a Go `time.Time` | `.Alert.Value` is the first variable the expression reads, at the last evaluation. Timestamp fields on `.Rule` and `.Alert` are strings. For a Slack incoming webhook, set its URL and use this template: ```text {"text":{{printf "%q" (printf "%s: %s on %s (value %g)" .Event .Rule.Name .Alert.Service .Alert.Value)}}} ``` This uses the built-in `printf` function; no JSON template helper is registered. `text/template` does not automatically escape values for JSON. With this example, use names without control characters other than tabs and line breaks. Pitot sends a POST with `Content-Type: application/json` unless custom headers override it. Each attempt has a 10-second total timeout and a 5-second response header timeout. Delivery makes up to three attempts, waiting 2 then 4 seconds before retries; 4xx responses are not retried. ## Use the HTTP API The browser uses these routes. Reads require `telemetry:read`; mutations and tests require `alerts:manage`. | Endpoint | Does | |---|---| | `GET /api/alerting/alerts` | Current alert state | | `GET /api/alerting/summary` | Counts by state | | `GET /api/alerting/rules` | List rules | | `POST /api/alerting/rules` | Create and compile a rule | | `PUT /api/alerting/rules/:id` | Replace a rule and check any nonempty expression | | `POST /api/alerting/rules/:id/test` | Evaluate a saved expression against live data without changing alert state or delivering a webhook | | `DELETE /api/alerting/rules/:id` | Delete the rule and its alerts | When creating through HTTP, omitted booleans are false, timers are zero and optional strings are empty. In particular, omitted `enabled` means disabled. These API defaults differ from the new-rule form above.