mix ash_vault.backfill (AshVault v0.2.0)

Copy Markdown View Source

Encrypt an existing plaintext column into its encrypted_<field> column, online.

This is step 3 of the expand / backfill / cut over / contract migration:

  1. Expand — an ordinary Ash/Ecto migration adds encrypted_email. Schema migrations never need keys.
  2. Deploy the transitional resource: encrypt :email, backfill_from: :legacy_email (or pass --from here). The plaintext column is still there.
  3. Backfill — this task. It needs the app runtime and a reachable key provider.
  4. Verify — mix ash_vault.verify, or --verify here.
  5. Cut over — point reads and writes at the encrypted field. This is the irreversible step; do it only after a clean verify.
  6. Contract — drop the plaintext column in a later migration.

There is no decrypt-back mode. Until step 6 the plaintext column is still there, which is the rollback.

Usage

mix ash_vault.backfill MyApp.Accounts.User email --tenant acme
mix ash_vault.backfill MyApp.Accounts.User email --from legacy_email --batch-size 1000
mix ash_vault.backfill MyApp.Accounts.User email --all-tenants MyApp.Accounts.list_tenant_ids/0
mix ash_vault.backfill MyApp.Accounts.User email --tenant acme --dry-run
mix ash_vault.backfill MyApp.Accounts.User email --tenant acme --verify
mix ash_vault.backfill MyApp.Accounts.User email --tenant acme --lookup

--lookup

Populates the deterministic <field>_lookup column of a searchable?: true field for rows that already hold ciphertext — after adding searchable?: true to an existing encrypted field, or after deliberately re-keying the provider's lookup secret.

It reads, decrypts, normalizes, hashes, and writes only the token column. The ciphertext is never rewritten: a fresh nonce would churn a column this mode has no business touching.

It cannot repair a changed normalize:

AshVault encrypts the normalized value, so the ciphertext and the token must agree about what the row holds. Changing normalize: on a populated column makes them disagree, and hashing the newly normalized value would leave those rows findable under one spelling and readable as another. The task detects that and aborts, naming the row, rather than writing the inconsistency. Reconciling the two requires re-encrypting, and by this point there is no plaintext column left to re-encrypt from.

Rotating a lookup key is this task, not mix ash_vault.rotate. rotate mints a new data key version and deliberately leaves every token alone — if it moved them, every existing row would silently stop being findable.

It is NOT part of an ordinary backfill

Back-filling a searchable?: true field writes the ciphertext and its token in the same batch, through AshVault.write_attributes/4 — the same function an ordinary create or update calls. --lookup is for the two cases where the ciphertext is already there and only the token is missing: retro-fitting searchable?: true onto a populated column, and rotating the provider's lookup secret.

Like the ciphertext backfill it is resumable and idempotent with no state file: it selects only rows whose token IS NULL and whose ciphertext IS NOT NULL, so a second run is a no-op. It writes, so it cannot be combined with --verify, which writes nothing — and does not need to be: plain --verify checks the token column for you, by recomputing it from the decrypted value.

Options

--from ATTR          plaintext source attribute (default: the field's `backfill_from`)
--batch-size N       rows per committed transaction (default 500)
--tenant TENANT      required when the key scope is `:tenant`
--all-tenants MFA    zero-arity `Module.function/0` listing tenants
--domain MODULE      Ash domain (inferred from `:ash_domains` when omitted)
--action NAME        update action to write with (default: the primary update
                     action; it must be `require_atomic? false`)
--lookup             populate `<field>_lookup` tokens for rows that already have
                     ciphertext; reads and decrypts, writes only the token column
--verify             decrypt a sample and compare to the plaintext column — and,
                     for a `searchable?` field, recompute and compare its
                     `<field>_lookup` token; write nothing
--dry-run            report what would be done; write nothing
--resume-from ID     start after this primary key (one tenant only; it cannot be
                     combined with `--all-tenants`)
--sample N           rows to check in `--verify` mode (`0` checks every row)
--yes                skip the confirmation prompt

Guarantees

  • Rows are ordered by primary key and paged with a keyset (pk > last_seen), never OFFSET.
  • Only rows whose encrypted column IS NULL are selected, so a re-run is a no-op and an interrupted run resumes with no state file.
  • Each batch is its own transaction. The table is never wrapped in one.
  • The key provider is checked before the first batch; an unreachable provider fails with ProviderUnavailable rather than half a written table. If it goes away mid-run, the task stops at the batch boundary and prints the exact resume command.
  • Only the encrypted column is written — plus the <field>_lookup token when the field is searchable?: true, because a row holding one without the other is a row that decrypts correctly and cannot be found.

Output

One key=value line per batch, then a human summary:

ash_vault.backfill resource=MyApp.Accounts.User field=email source=legacy_email ...
batch=1 rows=500 done=500 remaining=1500 rate=812.3 eta_s=1.8 elapsed_s=0.62
Backfilled 2000 row(s) of MyApp.Accounts.User.email in 4 batch(es) (2.4s).