--- title: "How storage works" summary: "Atomic Parquet ingestion, indexed trace reads, DuckDB analytics, and SQLite control state." description: "What Pitot writes to disk, how it is queried, and why each component exists." status: preview read_when: - "You are sizing disk, or deciding what to back up." - "You want to understand ingest durability or query performance." --- Pitot retains telemetry as ordinary Parquet files in self-contained atomic batch directories. Persistent sidecars index traces, and DuckDB handles SQL, broad scans, and rebuildable analytical rollups in the same process. Application state (users, sessions, dashboards, alert rules, and agent history) lives in SQLite. | Component | Purpose | |---|---| | Atomic Parquet batches | Authoritative retained spans, logs, and metrics | | Trace sidecar | Persistent row-range index for targeted trace reads from Parquet | | DuckDB | SQL, log filtering, broad analytical reads, and rebuildable rollups | | SQLite | Transactional application and identity state |
Telemetry durability and query flow: authenticated OTLP rows pass through group admission, parallel commit workers, and durable publication into immutable Parquet batches; indexed trace reads, DuckDB SQL, retention, and compaction use those batches, while service and edge rollups populate rebuildable query state.
Parquet is authoritative; indexes and DuckDB rollups are derived state.
## Writes are durable immediately Concurrent small OTLP requests may share one group-commit batch. Up to four commit workers encode independent batches in parallel. Each worker writes all present signals and metadata into a staging directory, adds `trace.fidx` when spans are present, fsyncs them, and publishes the complete directory with one atomic rename. Pitot returns success only after every batch belonging to the request is published. A crash before the rename leaves only an unacknowledged staging directory, which startup removes. A crash after the rename leaves a complete batch that startup discovers directly from the filesystem. The atomic directory publish is both the durable commit and the unit startup discovers; acknowledgement occurs only after publication and parent-directory fsync complete. A request whose sender stops waiting (it cancels or its deadline passes) before the commit of its batch starts is dropped unwritten and counted in `pitot_ingest_abandoned_total`, which also counts requests the server cuts off while shutting down. Its memory reservation is released at once, so the sender's retry is admitted and is the only copy. Other requests sharing that group commit are unaffected. Once a commit has started it completes, so a sender that gives up during it and retries stores those rows twice. Delivery is therefore at least once only within that commit window: OTLP has no idempotency key that lets Pitot recognize a resent request. ## One authoritative copy for logs and metrics Every signal is written once, to typed Zstandard-compressed Parquet. DuckDB applies filters, ordering, `LIMIT`, and aggregation inside its vectorized scan. Trace lookup binary-searches a fixed-width sidecar on disk, seeks directly to the matching Parquet row ranges, and verifies the complete trace ID after hashing. The sidecar is an index, not another telemetry representation. Dashboard rollups are rebuildable DuckDB caches. SQLite stores only transactional product state and never holds telemetry rows. Ingest authentication still looks up the token digest there on every export, so an unreadable control database rejects OTLP requests. ## Attributes retain their types Attribute and resource values support up to 128 nested arrays or objects. An export that exceeds this limit is rejected in full with OTLP `INVALID_ARGUMENT` (HTTP 400); accepted values retain their original types. Malformed metric points reject the export the same way, including a gauge or sum point with no value. A point flagged as having no recorded value (`DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK`) is accepted with its `value` stored as NULL, so aggregates skip it; it keeps that bit in `flags`. Span, log, and metric attributes and resource attributes use Parquet VARIANT columns, and the maps keep every key. Standard OpenTelemetry keys also get a typed copy: 18 span attributes, such as `http.route` and `http.response.status_code`, and 29 resource attributes, such as `service.version` and `k8s.namespace.name`, on every signal. A copy holds a value only when its OTLP type matches the semantic conventions; a value of another type stays in the map alone. Messaging attributes are shredded inside the VARIANT encoding instead, and the span read view exposes `messaging_system` and `messaging_destination` directly from those fields, so messaging rollups can avoid decoding the whole attributes object. Values with different types remain in the residual encoding; an integer does not become a string just because another row uses one. Span events and links, metric exemplars, histogram buckets, and summary quantiles are typed parallel lists, one element per item: for example `event_times`, `event_names`, and `event_attributes`, whose attribute maps are VARIANT values. No stored column holds JSON. Integers, booleans, binary values, nulls, arrays, and nested objects survive ingestion, indexed reads, and compaction. Event timestamps retain nanosecond precision as UTC instants (`TIMESTAMPTZ_NS`). Keep timestamp columns unchanged in filters so Parquet row-group statistics can prune the scan. For datetime functions that require a timestamp without a timezone, cast their argument to `TIMESTAMP_NS`; the JSON result boundary handles returned timestamps automatically. Attribute names such as `service.name` are literal keys. In SQL, use `attr(attributes, 'service.name')` and an explicit cast when a scalar type is needed, for example `TRY_CAST(attr(attributes, 'retry.count') AS BIGINT)`. General attribute extracts through views retain their types; extraction across view projections does not reach the Parquet scan. Prefer the scalar service, namespace, and messaging columns for those hot queries. The JSON SQL response converts nested values only when returning results. Integer tokens retain their full precision; clients that parse JSON numbers as floating-point values should cast large integers to strings in their query. Non-finite floats are returned as `"NaN"`, `"Infinity"`, or `"-Infinity"`. Binary values stay binary in storage; cast them explicitly to the desired string encoding when querying them for a JSON client. ## Engine and format are pinned Pitot embeds DuckDB `v2.0.0`, the 2.0 LTS release, with its extensions compiled into the binary. It downloads no extensions at runtime. Native source builds use `scripts/with-duckdb.sh`, which checks the platform archive and uses the matching engine headers; the ordinary `just` build and test recipes invoke it automatically. DuckDB 2's asynchronous I/O uses its default memory-governed read-ahead, which bounds the I/O queue. Rooted service traversal uses keyed recursion with explicit hop, accumulated-node, and execution-time limits. Offline verification decodes every Parquet column with the native engine and separately checks physical schemas, span ordering, row counts, and trace-index ranges. It reuses one bounded in-memory engine and creates no spill files. Verification computes and discards hashes to force decoding, without storing a checksum. Verification failures such as memory exhaustion or cancellation never authorize quarantine. Telemetry batches use format 5. An unsupported batch format makes startup fail before cleanup or schema rewriting, leaving the files intact. ## Small files are bounded by compaction Atomic ingestion creates immutable batch directories. Maintenance drains every eligible compaction group, choosing inputs by count, bytes, and a 25-million-row ceiling. It combines files from the same ingestion hour and generation. Once an hour is more than a day old, its small files may also merge with neighbouring hours of the same UTC day, up to 250,000 rows per file, so a quiet install does not keep one small file per hour for its whole retention. Each merge builds a replacement trace index, and atomically swaps the replacement for its inputs. A durable compaction marker makes an interrupted swap resumable. Active DuckDB readers pin immutable files while retention or compaction removes them, so reads never lose an open input file. The expensive merge runs before readers are excluded. Only the bounded directory swap takes the exclusive reader gate; new ingest batches continue encoding while that work is prepared. Retention is based on ingestion time rather than event time. A service with a bad clock therefore cannot pin a batch on disk by emitting a far-future event. Retention removes whole files, never single rows, as Prometheus, Tempo, and Loki remove whole blocks or chunks. A file expires when its newest row passes the cutoff, so rows in a day-merged file can stay up to one day past `PITOT_RETENTION_DAYS`, plus one maintenance interval. Service and edge rollups expire exactly at the cutoff, so for that last day raw telemetry can outlive its rollups. ## Dashboard reads pin completed batches Dashboard log, metric, and span scans select immutable files from their actual minimum and maximum event timestamps. Bounds come from every Parquet row-group footer, including negative timestamps. Files without complete statistics remain eligible. Late-arriving telemetry is selected by its event time, independently of its ingestion timestamp. Arbitrary SQL retains access to all active files. Operation panels and log counts scan footer-pruned Parquet for the exact requested window. Text searches match the redacted body. Notable-trace selection keeps one indexed candidate per trace, updated from new batches. A candidate records whether its namespace and service are unambiguous; mixed scopes use exact per-batch contributions for the requested scope. Error priority, duration, and trace-ID tie ordering are preserved. Boundary trace parts are recomputed from Parquet. These caches acknowledge complete batch IDs rather than an ingestion watermark. A batch's trace bounds, candidate updates, and acknowledgement commit in one transaction. A read pins active files and reads cache rows and acknowledgements from one DuckDB transaction. Newly published uncached files are read directly, so cache backfill does not delay visibility. Retired batch contributions are excluded by active ID even before cleanup; they cannot invalidate unrelated traces. Compaction derives the output trace contributions from complete cached inputs. If an input is not yet cached, the output remains directly queryable until its own backfill. Trace data in the private cache consists of per-batch trace parts and one incremental candidate per trace. Timestamps are aggregated by batch and trace; scope lookups use those parts and candidates. Complete cached compaction replacements preserve candidate bounds without recomputation. Retention and incomplete replacements recompute only affected trace IDs from active contributions. A semantic version covers the cache schema and aggregation rules; a version change atomically discards all private read tables and rebuilds from authoritative format-5 Parquet. The background worker processes bounded groups; `pitot_read_cache_pending_batches` reports its trace backlog independently of the service and edge rollups' publication horizon. Service-version observations populate their analytical rollup on a separate worker, so an analytical-gate wait cannot delay the completed-batch worker. Completed-batch and analytical writers use separate gates and two connections outside the read pool. They modify disjoint tables; maintenance holds both gates for checkpointing. Trace reads query retained candidates directly. A snapshot-local ambiguity probe includes batch-parts aggregation only when the requested scope needs it. Narrow file selections use explicit lists from footer bounds. Broad selections use a glob with a captured-batch-ID filter, which excludes files published after snapshot capture. Only signals referenced by a query are bound. `PITOT_ROLLUP_SKIP_TO_LATEST` skips historical service and edge aggregation. Completed-batch trace caches still rebuild retained history in bounded passes; uncached files remain immediately queryable. ## Rollups follow durable publication Overview and alert queries read rebuildable DuckDB rollups. Each commit attempt receives a strictly increasing ingestion stamp; a retry receives a new stamp. The repository tracks unfinished attempts and advances the publication horizon only past attempts that have published durably or failed. Service and edge rollups process disjoint ingestion windows up to that horizon, recomputing each affected event-time bucket in full. Raw committed telemetry remains queryable before its next rollup pass. Error-rate, latency, and volume detectors read service-rollup minutes through the last completed minute before the service watermark. The baseline has the same length as the current period. Latency uses the span-weighted mean of the rollup's served-span minute p95 values; log-pattern detection still reads logs. Ingest admission starts adapting a rows-per-second cap at 512 batches of reclaimable generation-0 compaction debt. Rising debt decreases the cap by 15% every 250 ms; falling debt increases it by 10%. The band cap keeps at least one normal batch per pressure window (50,000 rows/s), or its measured entry rate if lower, because further cuts shrink files without reducing their frequency. Rising interactive refusals or queue-wait timeouts also reduce admission, to a floor of half the measured pre-starvation rate. Panel queueing alone does not reduce admission, and the read floor never overrides the compaction cap. Rejected exports receive retryable OTLP responses before row conversion: gRPC `RESOURCE_EXHAUSTED` with `RetryInfo`, or HTTP 429 with `Retry-After`. Senders must enable retries and honor the delay. At 1,024 batches of debt all new exports are refused with a two-second delay; already admitted exports continue draining and retain durable acknowledgement semantics. Under debt pressure, group commits target 200,000 rows over at most one second, including gaps between sparse retry arrivals below 50,000 total rows. Once a 50,000-row batch is available, 20 ms without arrivals closes the group so closed-loop exporters can send again. The 64 MiB group bound and 50,000-row Parquet row groups remain unchanged. Exemplars and trace detail have reserved read capacity and priority over queued panels. Their bounded queue still returns busy responses when interactive demand exceeds capacity. Debt excludes oversized terminal batches and reserves 127 smaller inputs per ingestion window, so it measures reclaimable work rather than total inventory. At 512 batches compaction prioritizes eligible generation-0 groups. Below that threshold it selects the oldest eligible day and then its highest eligible generation. Compaction keeps running under read pressure. `pitot_compaction_backlog_batches` counts all live generation-0 batches, including the exclusions. `pitot_ingest_throttled_total{reason}` counts refused exports (`debt`, `reads`, or `budget`), and `pitot_ingest_admission_rows_per_second` reports the cap (zero means uncapped). `pitot_ingest_backpressure_seconds_total` now measures seconds with a cap active, replacing its earlier blocked-writer semantics; those historical values are not directly comparable. `pitot_compaction_pass_seconds` and `pitot_compaction_retired_batches_total` expose retirement throughput. Service dependency traversal reads the complete edge rollup within the selected window, including edges outside the topology view's displayed page. It follows upstream or downstream edges from a named service, retains the minimum hop count per namespace and service, and bounds depth, accumulated nodes, and execution time. A truncated result states which limit was reached. Rollup lag still applies; dependency edges describe observed calls, not proof of causality.