Troubleshoot

The process will not start

Pitot validates its whole configuration before opening any listener, so a configuration refusal names the setting. Other startup errors can report storage, listener, or identity-provider failures; read the error before changing settings.

Unknown PITOT_ setting names fail startup, except platform-injected service discovery names. Check spelling against the configuration reference.

An unresolved compaction blocks startup

One startup refusal is not about configuration. If a compaction was interrupted and Pitot can neither finish nor undo it, recovery fails closed and the error names telemetry/COMPACTION.json.

The marker identifies the authoritative batch set during a swap. The replacement is staged under telemetry/compaction/ or published as telemetry/parquet/batches/<output id>.batch. A published replacement lists its inputs in replaces.json, and its presence alone retires them: inputs keep their <id>.batch directories until no reader can still open them, and startup deletes any that remain. Releases before deferred deletion named retired inputs <id>.retired-<output id>. A completed publish can already have deleted some inputs; verify every input before attempting a rollback.

Clear the underlying cause, such as a full disk, and start Pitot again. Recovery runs automatically at startup.

If recovery keeps failing, roll the compaction back by hand. With the process stopped, use telemetry/parquet/batches/ as the batch directory:

1. Read telemetry/COMPACTION.json. It is JSON, with an output object and an inputs list of batch ids.
2. Back up telemetry/, then check every id in inputs. Each must exist in exactly one form: <id>.batch or <id>.retired-<output id>. If an id exists in both forms or neither form, stop here. Both forms make the authoritative copy ambiguous; neither means its rows may survive only inside the replacement. Reconcile the batch from the backup, or complete the compaction instead of rolling it back.
3. Rename each <id>.retired-<output id> back to <id>.batch.
4. Delete telemetry/compaction/<output id> if it exists.
5. Delete telemetry/parquet/batches/<output id>.batch if it exists. While it exists its replaces.json retires the inputs, and startup deletes them; a corrupt replacement continues to block startup.
6. Delete telemetry/COMPACTION.json.
7. Start Pitot.

Verify inputs before deleting the replacement

Restore the retired inputs before removing the compaction marker. The marker protects those directories from startup cleanup. Keep the replacement until every input is restored; it may hold the only surviving copy of some rows.

An unreadable telemetry batch blocks startup

Pitot also refuses to start when an authoritative .batch directory has invalid metadata, a corrupt Parquet file, a missing signal file, or a damaged trace index. First stop Pitot and back up telemetry/, then verify the full authoritative set:

# Use the same environment and data directory as the server.
pitot repair verify

The command is read-only and reports confirmed corrupt batches. It verifies batches concurrently, one two-thread engine with a fixed memory limit per two CPUs; GOMAXPROCS bounds both. Unsupported formats and operational errors stop verification; they do not authorize quarantine. Restore each reported corrupt batch from backup when possible. If one cannot be recovered and discarding only its telemetry is preferable to leaving the instance offline, set it aside explicitly:

pitot repair quarantine --batch <id>

Quarantine is an atomic rename beside the authoritative batch set, not a deletion. It refuses a batch that validates successfully and refuses any input or output protected by a live compaction marker. Preserve the reported quarantine directory for later recovery, run repair verify again, and only then restart Pitot.

Quarantine makes those rows unavailable

The command is intentionally batch-specific and never runs during startup. Every row in the quarantined directory disappears from queries until the batch is restored, so do not use it as a substitute for fixing a transient disk or permission error.

An exporter is rejected

Work down this list in order:

1. Has setup finished? Before the first administrator exists there is no ingest token to check against, and ingest rejects everything. A collector started ahead of setup fails until you complete it.
2. Were all tokens revoked? When no token is active, every exporter gets invalid ingest token over gRPC or 401 over HTTP, the same answer as a wrong token, and the server logs nothing for it. Open /settings/ingest on your Pitot instance. If the list is empty, create a token and update the Collector.
3. Is it the right credential, in the right header? An active ingest token is foi_-prefixed and goes in Authorization: Bearer. Browser sessions and MCP tokens are not ingest credentials.
4. Is the listener reachable? Both OTLP transports use PITOT_ADDR (127.0.0.1:7520 by default; the image sets :7520). Check container port mappings, bind address and firewall rules.
5. Right protocol? Both use port 7520. gRPC requires HTTP/2 through any proxy. Use grpc or http/protobuf consistently with the exporter configuration; HTTP/1.1 proxy forwarding cannot carry gRPC.

For OTLP/HTTP, set the exporter protocol to http/protobuf and give it the base endpoint, http://pitot:7520. Exporters derive /v1/traces, /v1/metrics and /v1/logs from it themselves.

The raw token is shown once when it is created. If it is lost, an administrator creates a new one at /settings/ingest on your Pitot instance, or under Connect telemetry in the account menu, and updates the Collector. A token that was revoked is rejected on the next request.

HTTP 415: unsupported media type

OTLP/HTTP requires Content-Type: application/x-protobuf; JSON returns 415. JavaScript *-otlp-http packages send JSON. Use the *-otlp-proto packages, such as @opentelemetry/exporter-trace-otlp-proto, or a gRPC exporter in Node.js. See Common SDKs for language setup.

HTTP 429 or gRPC RESOURCE_EXHAUSTED

Pitot throttles whole exports under ingest pressure. Enable exporter retries and honor HTTP Retry-After or gRPC RetryInfo. Check disk space, compaction backlog and query pressure. See ingest admission for thresholds and metrics.

Someone cannot sign in and SMTP is not configured

SMTP is optional. Creating a local user without it succeeds and reports invite_delivery: "not_configured" with login_link_required: true. An operator with shell access to the same configuration and data directory then mints a link directly:

pitot login-link [email protected]
# For Docker:
docker exec pitot pitot login-link [email protected]

The link expires after 15 minutes, works once, and is recorded in the authentication audit history. It is available in local auth mode only; OIDC installations use --break-glass.

A configured SMTP relay that fails delivery returns an error. See account recovery for invitation responses and account recovery.

Preserve the account

Deleting a user row cascades to sessions, OAuth grants and owned dashboards. Use a login link for an active local account.

Telemetry arrives but queries look empty

Check the namespace. Pitot reads service.namespace from the OTLP resource and falls back to PITOT_DEFAULT_NAMESPACE, so telemetry exported without one lands in default.

An unscoped query spans every namespace, so this is only a problem when something has scoped the query, through a dashboard filter or an MCP call passing namespace. Clear the scope before concluding the data is missing.

The alert engine and overview read analytical rollups, which follow durable publication. Raw queries can see committed rows before the next rollup pass. See rollup timing.

What to watch in production

The endpoint reference lists health probes, scrape credentials and signals to monitor.