Tune retention

Retention controls telemetry lifetime. Group commits and compaction control file count and query cost. All configuration changes require a restart.

How long data is kept

export PITOT_RETENTION_DAYS=30

Zero means keep everything. Changing it does not delete anything immediately; expired data goes when maintenance next runs, so disk use falls on the maintenance cycle after restart. Expiry uses ingestion time and removes whole files: a file goes once its newest row is past the cutoff, so data can stay up to one day past the retention period, plus one maintenance interval. Size disk for retention plus a day. See storage semantics.

Observability API, MCP, and dashboard queries can span up to this retention period. With unlimited retention, each query remains capped at 30 days. Pitot automatically uses wider timeline buckets for windows longer than 24 hours.

The maintenance cycle

export PITOT_MAINTENANCE_INTERVAL=1h

The maintenance pass removes expired batches, compacts eligible files and checkpoints query state. Shorter intervals run this work more frequently.

Group-commit target

The target number of telemetry rows that concurrent small requests may share in one atomic Parquet batch:

export PITOT_INGEST_BATCH_SIZE=50000

A larger target can improve sustained write throughput and create fewer files under concurrent ingest. The target groups concurrent small requests; a request above the target remains one atomic batch. Every request is acknowledged only after its Parquet batch is durably published; a request whose sender stops waiting before its commit starts is dropped, so the sender’s retry is the only copy.

What the knobs interact with

Maintenance prepares merges while reads continue, then briefly gates readers while it swaps or removes immutable files. New batches keep encoding and need only the short publication lock. Ingest, queries, and maintenance still share CPU, memory bandwidth and disk. Change one setting at a time and watch /metrics.

The full list, with defaults and types, is in the storage settings reference.