Platform state snapshots

Lineage, hourly history, pre-release copies, and restore for the platform database.

Outer snapshots its platform database (accounts, plans, OAuth grants, app secrets) to the bucket as outer/state/outer.sqlite.zst whenever it changes — at most every 15 seconds after a 5-second settle, so worst-case loss is about 22 seconds. This page is the operator’s view.

Lineage: the sidecar

Each upload records the snapshot sha256 in the bucket meta object and in a local sidecar (/data/outer.sqlite.snapshot-sha, written atomically after the upload lands). Before every upload the host compares the bucket’s sha against the sidecar: empty bucket uploads; a sha matching the sidecar (or the bytes just dumped, for the crash window) uploads; a meta written by a pre-lineage binary under this host’s id is adopted (below); anything else — including a database with no sidecar facing a lineage-marked snapshot — refuses loudly (refusing state snapshot upload, naming bucket and key), keeps serving, and retries the check. Never overwrites.

When an upload refuses

Wrong bucket/prefix in OUTER_R2_*: fix the config and restart. If the bad boot already created a fresh local database, stop outer, delete that file (and its sidecar), and boot again — the restore downloads the real snapshot.
A second host claimed the same bucket: the newest claimant wins and the old one stops uploading. Give each install its own bucket.
There is no --force: every recovery goes through the restore below.

Lineage across upgrades

Uploads also record "lineage": 1 in the bucket meta. A meta without the marker was written by a pre-lineage binary: if it names this host as instance, the upload gate adopts its sha into the sidecar and proceeds (warned in the log), re-checking on every attempt while the bucket meta stays legacy. That covers release handoffs and container swaps: an old binary’s last uploads during the overlap can’t wedge the newcomer, and once a lineage-aware binary uploads, the strict rule applies. A legacy meta naming another host still refuses. The flip side: any legacy-meta bucket this host id once wrote (a mispointed OUTER_R2_* prefix from before the upgrade, say) is adopted and overwritten the same way, so check the bucket and prefix before the first lineage-aware boot. A rollback to a pre-lineage binary still boots — its reader only looks up known meta fields and ignores lineage — but uploads ungated while rolled back; the next lineage-aware release re-adopts through the same rule.

outer state adopt-lineage --expect-instance <id> remains as a manual operator command: it writes the sidecar from the bucket meta sha when the local DB exists, no sidecar is present, and the meta instance equals the given id (exit 0 with already adopted when the sidecar exists; every other mismatch exits non-zero naming the reason). Adoption keys off the author instance only, not the database bytes: never drop files in place, use the restore below.

Runbook: is this host backing up?

/ready reports snapshot.last_error (the last failure, truncated) and snapshot.consecutive_lineage_refusals. After 30 consecutive lineage refusals /ready answers 503 with snapshot: refusing uploads (N consecutive, backups stopped). Compare the three identities:

bucket: the meta object outer/state/outer.sqlite.meta.json (sha256, instance, generation);
sidecar: cat /data/outer.sqlite.snapshot-sha;
local dump: sqlite3 /data/outer.sqlite "VACUUM INTO '/tmp/check.sqlite'" && sha256sum /tmp/check.sqlite.

Sidecar == bucket means the host is current (the next change uploads). Bucket == local dump with a stale sidecar is the crash window — the next upload repairs it. Anything else is a wrong bucket/prefix or a foreign database: fix the config, or restore below.

History and pre-release copies

Each successful upload also files outer/state/history/<UTC yyyy-mm-ddTHH>.sqlite.zst (first copy per hour wins) and prunes: hourly copies past 48 h collapse to the first copy of each day, days past 30 d go, at most 10 deletions per pass. Release copies (history/release-<UTC timestamp>-<git sha>.sqlite.zst, filed by deploy/release.sh before the handoff) are kept until removed by hand — the pruner never touches them, so each release leaves a permanent restorable snapshot. Anything the pruner doesn’t recognize, and the live key, are never deleted either.

Restore a snapshot

Stop outer first. With any S3-capable client (bucket, endpoint and keys are the OUTER_R2_* values in /srv/outer/outer.env):

Pick a history key and download it; unpack (zstd -d) and check it: sqlite3 file "PRAGMA integrity_check;" must print ok.
Replace /data/outer.sqlite with the unpacked file and write its sidecar: sha256sum /data/outer.sqlite | cut -d' ' -f1 > /data/outer.sqlite.snapshot-sha.
Upload the packed bytes back to the live key, and set the meta object’s sha256 to the sidecar’s value (leave generation alone — the next boot continues it).
Start outer: the lineage check passes, the bytes match so nothing re-uploads, and the next change snapshots normally.