---
title: "Data layout"
summary: "The exact contents of PITOT_DATA_DIR, and why the directory is one unit."
description: "Every path under a Pitot data directory and what it holds."
status: preview
read_when:
  - "You are sizing disk or deciding what to back up."
  - "You are inspecting an instance's files and want to know what you are looking at."
---
Everything Pitot persists lives under `PITOT_DATA_DIR` (`./data` by default,
`/var/lib/pitot/data` in the container).

| Path | Holds |
|---|---|
| `telemetry/parquet/batches/*.batch/` | Authoritative atomic batches containing `metadata.json`, present signal Parquet files, and `trace.fidx` when spans are present. A compacted batch also lists the inputs it replaced in `replaces.json`; those inputs are no longer authoritative and keep their directories only until readers that captured them finish |
| `telemetry/parquet/batches/_schema.batch/` | Empty schema anchors that keep every DuckDB view queryable |
| `telemetry/parquet/batches/*.retired/` | Replaced or expired batches being deleted; startup removes any left by a crash |
| `telemetry/parquet/batches/*.retired-*/` | Compaction inputs from releases before deferred deletion; a live marker protects its own set |
| `telemetry/parquet/batches/*.quarantined-*/` | Unreadable batches explicitly set aside by `pitot repair quarantine`; preserved but not queryable |
| `telemetry/parquet/staging/` | Unacknowledged writes being prepared for atomic publication |
| `telemetry/compaction/` | Prepared replacement batches during compaction |
| `telemetry/COMPACTION.json` | Durable marker for an interrupted compaction swap. Present only while one is unresolved, and blocks startup until it is — see [troubleshoot](/docs/guides/troubleshooting) |
| `query/catalog.duckdb` | Rebuildable DuckDB rollups and query state |
| `query/tmp/` | Spill space for queries that exceed the memory cap |
| `control/fanout.sqlite` | Users, sessions, dashboards, alert rules, agent history |

## Why it is one unit

Each published Parquet batch directory is a self-contained crash-recovery unit.
Its metadata records row counts and time bounds; its fixed-width trace sidecar
maps hashes to sorted span row ranges. The sidecar is verified against the
Parquet rows by offline repair, while DuckDB rollups can be rebuilt. Keeping the
complete data directory is the supported and fastest restore path.

`control/fanout.sqlite` is independent of both in format but not in meaning: it
holds the dashboards and alert rules that refer to the telemetry, and the
accounts that own them.

[Back up and restore](/docs/guides/backup-and-restore) covers cold backups of the
complete directory.

## Why two engines

Telemetry is append-heavy, queried analytically over wide ranges, and never
updated. Application state is small, transactional and updated constantly.
Parquet with DuckDB serves the former; SQLite serves the latter. [How storage
works](/docs/concepts/storage) covers the reasoning and the write path.

## `query/tmp/`

DuckDB spills here when a query needs more memory than `PITOT_DUCKDB_MEMORY`
allows. It is regenerated as needed and holds nothing durable, but it does need
space: a query against a wide time range can spill substantially, and a full
disk causes the query to fail.

## Sizing

`PITOT_RETENTION_DAYS` controls retention, with expiry applied on
the maintenance cycle rather than the moment you change it. Expiry removes whole
files, so data can stay up to one day past the retention period. File count matters
as much as byte count for query latency, which is what the compaction pass exists to
bound. [Tune retention](/docs/guides/retention) covers both.

> **Do not edit these files**
>
> Pitot owns this directory. Editing control SQLite, DuckDB query state, or
> telemetry batches directly can violate application and storage invariants. Use
> the documented commands and APIs instead.