--- title: Platform state snapshots description: Lineage, hourly history, pre-release copies, and restore for the platform database. --- Outer snapshots its platform database (accounts, plans, OAuth grants, app secrets) to the bucket as `outer/state/outer.sqlite.zst` whenever it changes — at most every 15 seconds after a 5-second settle, so worst-case loss is about 22 seconds. This page is the operator's view. ## Lineage: the sidecar Each upload records the snapshot sha256 in the bucket meta object and in a local sidecar (`/data/outer.sqlite.snapshot-sha`, written atomically after the upload lands). Before every upload the host compares the bucket's sha against the sidecar: empty bucket uploads; a sha matching the sidecar (or the bytes just dumped, for the crash window) uploads; a meta written by a pre-lineage binary under this host's id is adopted (below); anything else — including a database with no sidecar facing a lineage-marked snapshot — **refuses loudly** (`refusing state snapshot upload`, naming bucket and key), keeps serving, and retries the check. Never overwrites. ## When an upload refuses - Wrong bucket/prefix in `OUTER_R2_*`: fix the config and restart. If the bad boot already created a fresh local database, stop outer, delete that file (and its sidecar), and boot again — the restore downloads the real snapshot. - A second host claimed the same bucket: the newest claimant wins and the old one stops uploading. Give each install its own bucket. - There is no `--force`: every recovery goes through the restore below. ## Lineage across upgrades Uploads also record `"lineage": 1` in the bucket meta. A meta *without* the marker was written by a pre-lineage binary: if it names this host as `instance`, the upload gate adopts its sha into the sidecar and proceeds (warned in the log), re-checking on every attempt while the bucket meta stays legacy. That covers release handoffs and container swaps: an old binary's last uploads during the overlap can't wedge the newcomer, and once a lineage-aware binary uploads, the strict rule applies. A legacy meta naming another host still refuses. The flip side: any legacy-meta bucket this host id once wrote (a mispointed `OUTER_R2_*` prefix from before the upgrade, say) is adopted and overwritten the same way, so check the bucket and prefix before the first lineage-aware boot. A rollback to a pre-lineage binary still boots — its reader only looks up known meta fields and ignores `lineage` — but uploads ungated while rolled back; the next lineage-aware release re-adopts through the same rule. `outer state adopt-lineage --expect-instance ` remains as a manual operator command: it writes the sidecar from the bucket meta sha when the local DB exists, no sidecar is present, and the meta `instance` equals the given id (exit 0 with `already adopted` when the sidecar exists; every other mismatch exits non-zero naming the reason). Adoption keys off the author `instance` only, not the database bytes: never drop files in place, use the restore below. ## Runbook: is this host backing up? `/ready` reports `snapshot.last_error` (the last failure, truncated) and `snapshot.consecutive_lineage_refusals`. After 30 consecutive lineage refusals `/ready` answers 503 with `snapshot: refusing uploads (N consecutive, backups stopped)`. Compare the three identities: - bucket: the meta object `outer/state/outer.sqlite.meta.json` (`sha256`, `instance`, `generation`); - sidecar: `cat /data/outer.sqlite.snapshot-sha`; - local dump: `sqlite3 /data/outer.sqlite "VACUUM INTO '/tmp/check.sqlite'" && sha256sum /tmp/check.sqlite`. Sidecar == bucket means the host is current (the next change uploads). Bucket == local dump with a stale sidecar is the crash window — the next upload repairs it. Anything else is a wrong bucket/prefix or a foreign database: fix the config, or restore below. ## History and pre-release copies Each successful upload also files `outer/state/history/.sqlite.zst` (first copy per hour wins) and prunes: hourly copies past 48 h collapse to the first copy of each day, days past 30 d go, at most 10 deletions per pass. Release copies (`history/release--.sqlite.zst`, filed by `deploy/release.sh` before the handoff) are kept until removed by hand — the pruner never touches them, so each release leaves a permanent restorable snapshot. Anything the pruner doesn't recognize, and the live key, are never deleted either. ## Restore a snapshot Stop outer first. With any S3-capable client (bucket, endpoint and keys are the `OUTER_R2_*` values in `/srv/outer/outer.env`): 1. Pick a history key and download it; unpack (`zstd -d`) and check it: `sqlite3 file "PRAGMA integrity_check;"` must print `ok`. 2. Replace `/data/outer.sqlite` with the unpacked file and write its sidecar: `sha256sum /data/outer.sqlite | cut -d' ' -f1 > /data/outer.sqlite.snapshot-sha`. 3. Upload the packed bytes back to the live key, and set the meta object's `sha256` to the sidecar's value (leave `generation` alone — the next boot continues it). 4. Start outer: the lineage check passes, the bytes match so nothing re-uploads, and the next change snapshots normally.