--- title: "Troubleshoot" summary: "Startup refusals, rejected exporters, locked-out users, and where to look when telemetry is not arriving." description: "The failures a Pitot instance actually produces, and what each one means." status: preview read_when: - "The process will not start, or an exporter is being rejected." - "Someone cannot sign in and SMTP is not configured." --- ## The process will not start Pitot validates its whole configuration before opening any listener, so a configuration refusal names the setting. Other startup errors can report storage, listener, or identity-provider failures; read the error before changing settings. Unknown `PITOT_` setting names fail startup, except platform-injected service discovery names. Check spelling against the [configuration reference](/docs/reference/configuration). ## An unresolved compaction blocks startup One startup refusal is not about configuration. If a compaction was interrupted and Pitot can neither finish nor undo it, recovery fails closed and the error names `telemetry/COMPACTION.json`. The marker identifies the authoritative batch set during a swap. The replacement is staged under `telemetry/compaction/` or published as `telemetry/parquet/batches/.batch`. A published replacement lists its inputs in `replaces.json`, and its presence alone retires them: inputs keep their `.batch` directories until no reader can still open them, and startup deletes any that remain. Releases before deferred deletion named retired inputs `.retired-`. A completed publish can already have deleted some inputs; verify every input before attempting a rollback. **Clear the underlying cause, such as a full disk, and start Pitot again.** Recovery runs automatically at startup. If recovery keeps failing, roll the compaction back by hand. With the process stopped, use `telemetry/parquet/batches/` as the batch directory: 1. Read `telemetry/COMPACTION.json`. It is JSON, with an `output` object and an `inputs` list of batch ids. 2. Back up `telemetry/`, then check every id in `inputs`. Each must exist in exactly one form: `.batch` or `.retired-`. **If an id exists in both forms or neither form, stop here.** Both forms make the authoritative copy ambiguous; neither means its rows may survive only inside the replacement. Reconcile the batch from the backup, or complete the compaction instead of rolling it back. 3. Rename each `.retired-` back to `.batch`. 4. Delete `telemetry/compaction/` if it exists. 5. Delete `telemetry/parquet/batches/.batch` if it exists. While it exists its `replaces.json` retires the inputs, and startup deletes them; a corrupt replacement continues to block startup. 6. Delete `telemetry/COMPACTION.json`. 7. Start Pitot. > **Verify inputs before deleting the replacement** > > Restore the retired inputs before removing the compaction marker. The marker > protects those directories from startup cleanup. Keep the replacement until > every input is restored; it may hold the only surviving copy of some rows. ## An unreadable telemetry batch blocks startup Pitot also refuses to start when an authoritative `.batch` directory has invalid metadata, a corrupt Parquet file, a missing signal file, or a damaged trace index. First stop Pitot and back up `telemetry/`, then verify the full authoritative set: ```sh # Use the same environment and data directory as the server. pitot repair verify ``` The command is read-only and reports confirmed corrupt batches. It verifies batches concurrently, one two-thread engine with a fixed memory limit per two CPUs; `GOMAXPROCS` bounds both. Unsupported formats and operational errors stop verification; they do not authorize quarantine. Restore each reported corrupt batch from backup when possible. If one cannot be recovered and discarding only its telemetry is preferable to leaving the instance offline, set it aside explicitly: ```sh pitot repair quarantine --batch ``` Quarantine is an atomic rename beside the authoritative batch set, not a deletion. It refuses a batch that validates successfully and refuses any input or output protected by a live compaction marker. Preserve the reported quarantine directory for later recovery, run `repair verify` again, and only then restart Pitot. > **Quarantine makes those rows unavailable** > > The command is intentionally batch-specific and never runs during startup. > Every row in the quarantined directory disappears from queries until the batch > is restored, so do not use it as a substitute for fixing a transient disk or > permission error. ## An exporter is rejected Work down this list in order: 1. **Has setup finished?** Before the first administrator exists there is no ingest token to check against, and ingest rejects everything. A collector started ahead of setup fails until you complete it. 2. **Were all tokens revoked?** When no token is active, every exporter gets `invalid ingest token` over gRPC or 401 over HTTP, the same answer as a wrong token, and the server logs nothing for it. Open `/settings/ingest` on your Pitot instance. If the list is empty, create a token and update the Collector. 3. **Is it the right credential, in the right header?** An active ingest token is `foi_`-prefixed and goes in `Authorization: Bearer`. Browser sessions and MCP tokens are not ingest credentials. 4. **Is the listener reachable?** Both OTLP transports use `PITOT_ADDR` (`127.0.0.1:7520` by default; the image sets `:7520`). Check container port mappings, bind address and firewall rules. 5. **Right protocol?** Both use port `7520`. gRPC requires HTTP/2 through any proxy. Use `grpc` or `http/protobuf` consistently with the exporter configuration; HTTP/1.1 proxy forwarding cannot carry gRPC. For OTLP/HTTP, set the exporter protocol to `http/protobuf` and give it the base endpoint, `http://pitot:7520`. Exporters derive `/v1/traces`, `/v1/metrics` and `/v1/logs` from it themselves. The raw token is shown once when it is created. If it is lost, an administrator creates a new one at `/settings/ingest` on your Pitot instance, or under **Connect telemetry** in the account menu, and updates the Collector. A token that was revoked is rejected on the next request. ### HTTP 415: unsupported media type OTLP/HTTP requires `Content-Type: application/x-protobuf`; JSON returns 415. JavaScript `*-otlp-http` packages send JSON. Use the `*-otlp-proto` packages, such as `@opentelemetry/exporter-trace-otlp-proto`, or a gRPC exporter in Node.js. See [Common SDKs](/docs/get-started/send-telemetry#common-sdks) for language setup. ### HTTP 429 or gRPC RESOURCE_EXHAUSTED Pitot throttles whole exports under ingest pressure. Enable exporter retries and honor HTTP `Retry-After` or gRPC `RetryInfo`. Check disk space, compaction backlog and query pressure. See [ingest admission](/docs/concepts/storage#rollups-follow-durable-publication) for thresholds and metrics. ## Someone cannot sign in and SMTP is not configured SMTP is optional. Creating a local user without it succeeds and reports `invite_delivery: "not_configured"` with `login_link_required: true`. An operator with shell access to the same configuration and data directory then mints a link directly: ```sh pitot login-link user@example.com # For Docker: docker exec pitot pitot login-link user@example.com ``` The link expires after 15 minutes, works once, and is recorded in the authentication audit history. It is available in `local` auth mode only; OIDC installations use [`--break-glass`](/docs/concepts/authentication#oidc-break-glass-recovery). A configured SMTP relay that fails delivery returns an error. See [account recovery](/docs/concepts/authentication#recover-local-sign-in) for invitation responses and account recovery. > **Preserve the account** > > Deleting a user row cascades to sessions, OAuth grants and owned dashboards. > Use a login link for an active local account. ## Telemetry arrives but queries look empty Check the namespace. Pitot reads `service.namespace` from the OTLP resource and falls back to `PITOT_DEFAULT_NAMESPACE`, so telemetry exported without one lands in `default`. An unscoped query spans every namespace, so this is only a problem when something has scoped the query, through a dashboard filter or an MCP call passing `namespace`. Clear the scope before concluding the data is missing. The alert engine and overview read analytical rollups, which follow durable publication. Raw queries can see committed rows before the next rollup pass. See [rollup timing](/docs/concepts/storage#rollups-follow-durable-publication). ## What to watch in production The [endpoint reference](/docs/reference/endpoints#what-to-alert-on) lists health probes, scrape credentials and signals to monitor.