Skip to main content

Back up and restore

One command creates an encrypted backup of everything TOW needs to come back from a dead host. Two shipped systemd timers run that command and its retention pruning every day. One command restores a backup and keeps TOW offline until the restored system provably honours every account deletion that happened after the backup was taken.

Backups and account deletion are two halves of one design: deletion cannot complete while an old backup still holds the deleted data, so managed backups have a hard maximum age of 30 days and pruning is not optional housekeeping; it is what makes the deletion promise come true. Set backups up before opening the deployment to users.

What a backup contains​

Each backup is a directory of age-encrypted artifacts plus a signed manifest:

  • The PostgreSQL database (custom-format dump).
  • The backend files volume (/app/data for uploads and attachments).
  • When the bundled Authentik profile is in use: the Authentik database and its data, media, and templates volumes. Everything an identity provider needs to come back with its users, OIDC configuration, and MFA intact.

Deliberately not in a backup:

  • .env, config/tow.yaml, and TLS certificates. Recover these from your configuration management or secrets manager, not from personal-data backups.
  • The deletion ledger and its secrets. The ledger is what makes restore safe, so it must survive independently of the data it protects. Keep its directory (TOW_PRIVACY_LEDGER_DIR) on its own durable, separately backed-up path.
  • The Meilisearch index (rebuilt from the database during restore), Docker images, source code, and plaintext dumps of any kind.

Set up backups​

The host needs Docker with Compose, age, Python 3, sha256sum, and systemd. One command does the whole setup (the installer also offers it at the end of installation):

scripts/setup-backups.sh

It performs, in order:

  1. Generates the backup encryption key pair (age-keygen) and records the public key in .env as PRIVACY_BACKUP_AGE_RECIPIENT. The private identity is written once to secrets/; store it in your secrets manager and remove it from the host. It is only needed to restore, and a backup that can be decrypted in place protects nothing.
  2. Configures the deletion-ledger directory (TOW_PRIVACY_LEDGER_DIR, default /var/lib/tow-privacy, an absolute path outside the deployment directory and outside every restore).
  3. Initializes the append-only deletion ledger that restore uses to replay deletions. It is idempotent; later runs just verify the existing chain.
  4. Takes the first encrypted backup (default root /var/backups/tow, override with --output-dir). PostgreSQL and the data volumes stream straight through age encryption; the manifest is signed against the ledger and the backup registered with its destruction deadline.
  5. Installs the daily systemd timers: backup at 03:15 UTC and retention pruning at 04:05 UTC (see the shipped units).

Afterwards, verify and watch the schedule:

systemctl list-timers 'tow-backup*'
journalctl -u tow-backup.service -u tow-backup-prune.service

Alert on failures of either service. Pruning must run daily even if backup creation breaks: it enforces the 30-day maximum and lets pending account deletions complete.

Back up PRIVACY_LEDGER_HMAC_SECRET from .env separately: it authenticates the deletion ledger, and losing it blocks both restore and deletion.

Never replace a used ledger

If a previously used ledger is missing, do not initialize a new one; restore the canonical copy. A fresh ledger cannot re-apply the deletions the old one recorded.

Finally, do a restore drill: run the restore runbook once against a scratch host or an acceptable maintenance window, with the age identity fetched from your secrets manager. An unexercised backup is a hope, not a plan.

Restore a backup​

Restore is one command. It refuses to hand you a half-restored system: every verification gate must pass before application services start, and any failure leaves everything stopped so you can correct and rerun.

  1. Recreate the host, .env, config/, and the deployment directory from configuration management; make sure the ledger directory (TOW_PRIVACY_LEDGER_DIR) is present with its canonical contents; start only the database services (docker compose up -d db meilisearch, plus --profile authentik up -d authentik-db authentik-redis when the backup contains Authentik state).

  2. Make sure the deploy kit's scripts/ directory matches the target release (git pull in the kit; on air-gapped hosts extract the scripts from the backend image).

  3. Fetch the age identity from your secrets manager and run:

    cd /path/to/your/deployment   # the directory with compose.yaml and .env
    scripts/restore-system.sh \
      --backup /var/backups/tow/system-backup-YYYYMMDDTHHMMSSZ-ID \
      --age-identity /path/to/tow-backup-age-identity

  4. Remove the age identity from the host again.

The command authenticates the backup against the signed registry, stops every writer, replaces the data volume and database (and Authentik's, when present), migrates the schema, replays every deletion recorded after the backup was taken, rebuilds the search index, replays deletions again against the rebuilt index, and only then starts TOW. Pass --no-start to keep services stopped after verification.

You know it worked when:

  • The script ends with Restore complete and every gate logged success.
  • docker compose ps shows all services healthy.
  • /api/health responds through the proxy.
  • A user erased after the backup was taken is still absent; attribution shows Former user and sign-in is impossible.

Never extract or restore the archives manually: a hand-restored database silently resurrects deleted accounts and invalidates the deletion evidence.

Bundled Authentik state​

When the authentik Compose profile is active, backup-system.sh detects it and includes the Authentik database and volumes automatically; no extra configuration. restore-system.sh reads the backup's manifest and restores Authentik state when it is present.

  • A backup with Authentik state refuses to restore into a deployment without the profile (pass --skip-authentik to restore core services only).
  • During restore, restored Authentik state is served to the deletion replay with the public proxy stopped, so an identity deleted after the backup never becomes reachable again; the replay deletes it from the restored Authentik before anything is exposed.
  • --skip-authentik on the backup side excludes Authentik state deliberately.
  • The Authentik Redis cache is not backed up; restore flushes it.

Configuration reference​

Configuration is resolved in order: flags, then exported environment variables, then the deployment .env, then defaults. Set TOW_OPS_ENV_FILE to point the scripts at another env file, or to /dev/null to disable .env loading entirely (for example when a secrets manager injects the environment).

SettingFlagDefaultPurpose
PRIVACY_BACKUP_AGE_RECIPIENT--age-recipientUnset (required)Public age key that encrypts every artifact.
TOW_BACKUP_AGE_IDENTITY--age-identityUnset (restore only)Path to the private age identity.
PRIVACY_LEDGER_PATH--journal-pathTOW_PRIVACY_LEDGER_DIR/erasure.jsonlHost path of the deletion ledger.
PRIVACY_LEDGER_HMAC_SECRETN/AUnset (required)Secret authenticating the ledger and manifests.
PRIVACY_LEDGER_HMAC_KEY_FILE--journal-key-fileUnsetOptional file-based alternative to the secret for secrets-manager setups.
PRIVACY_BACKUP_REGISTRY_HEAD_PATH--registry-head-pathBeside the ledgerMonotonic backup-history head.
PRIVACY_BACKUP_RETENTION_DAYS--retention-days30Days until a backup must be destroyed (1–30). Use the same value for backup and prune.
N/A--output-dirbackupsManaged backup root. Use an absolute path such as /var/backups/tow.
N/A--skip-authentikOffExclude (backup) or skip (restore) bundled Authentik state.
N/A--no-pruneOffSkip pruning after a backup.
N/A--no-startOffLeave services stopped after a verified restore.
N/A--dry-runOffPrune only: report expired backups without destroying them.

The shipped systemd units​

install-backup-schedule.sh renders these templates (replacing @PROJECT_DIR@, @BACKUP_ROOT@, and the schedule placeholders) into /etc/systemd/system and enables both timers. Re-running the installer updates them in place.

tow-backup.service
[Unit]
Description=TOW managed encrypted backup
Requires=docker.service
After=docker.service network-online.target

[Service]
Type=oneshot
User=root
WorkingDirectory=@PROJECT_DIR@
ExecStart=@PROJECT_DIR@/scripts/backup-system.sh --output-dir @BACKUP_ROOT@
tow-backup.timer
[Unit]
Description=Daily TOW managed encrypted backup

[Timer]
OnCalendar=@BACKUP_ONCALENDAR@
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target
tow-backup-prune.service
[Unit]
Description=TOW managed backup retention pruning
Requires=docker.service
After=docker.service network-online.target

[Service]
Type=oneshot
User=root
WorkingDirectory=@PROJECT_DIR@
ExecStart=@PROJECT_DIR@/scripts/prune-system-backups.sh --output-dir @BACKUP_ROOT@
tow-backup-prune.timer
[Unit]
Description=Daily TOW managed backup retention pruning

[Timer]
OnCalendar=@PRUNE_ONCALENDAR@
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target

Troubleshooting and recovery​

  • A backup or prune run exits non-zero. Read journalctl -u tow-backup.service (or -u tow-backup-prune.service). The scripts fail closed: valid work still completed, and destruction receipts were synchronized. Fix the cause and rerun; alert on repeated failures.
  • Prune warns about unmanaged entries. Anything in the backup root that is not a managed encrypted backup, including legacy dumps from older tooling and ad-hoc archives, is never touched by the pruner and never expires. Plaintext dumps of personal data defeat the deletion design; review and remove them manually.
  • The ledger or its secret is missing or corrupt. Stop everything that writes (application, backups, restore) and restore the canonical ledger or secret from its protected copy. Never initialize a replacement. See deletion monitoring and recovery.
  • Restore fails at a gate. Services stay stopped by design. Read the last [restore] lines, correct the cause (for example, start the missing authentik profile, or fetch the right age identity), and rerun the same command.
  • Backups taken before this release contain only the core two artifacts; they verify and restore unchanged.
  • Longer than 30 days / offsite requirements. The bundled tooling caps retention at 30 days to keep the deletion promise. Offsite or longer-lived backup systems must implement the full external-system contract described in account deletion.

One-time adoption of a pre-head backup registry​

Only installations whose backup-registry.jsonl was created by an older TOW release that did not yet write an independent registry head need this; it is not a recovery procedure for a head that once existed and was lost. Stop the backup timers, confirm the registry is non-empty and no head has ever existed, then run:

cd /opt/tow
export BACKUP_ROOT=/var/backups/tow
export REGISTRY_HEAD=/var/lib/tow-privacy/backup-registry.head.json

test -s "${BACKUP_ROOT}/backup-registry.jsonl"
test ! -e "${BACKUP_ROOT}/backup-registry.head.json"
test ! -L "${BACKUP_ROOT}/backup-registry.head.json"
test ! -e "${REGISTRY_HEAD}"
test ! -L "${REGISTRY_HEAD}"

docker compose run --rm --no-deps --no-TTY \
  --volume "${BACKUP_ROOT}:/run/tow-adoption/backups" \
  --volume "$(dirname "${REGISTRY_HEAD}"):/run/tow-adoption/ledger" \
  backend python -m app.scripts.privacy_backup adopt-registry-head \
  --backup-root /run/tow-adoption/backups \
  --registry-head-path "/run/tow-adoption/ledger/$(basename "${REGISTRY_HEAD}")" \
  --acknowledge-history-may-already-be-truncated

The command locks and fully authenticates the registry chain before creating the head, and refuses an existing head, an empty or invalid history, and every unsafe path. The acknowledgement flag records the one unavoidable limitation: before a head exists, an already-truncated authenticated suffix cannot be detected. Resume the timers only after a normal backup run verifies the registry.