The online backfill engine behind mix ash_vault.backfill and mix ash_vault.verify.
It turns an existing plaintext column into ciphertext in the resource's
encrypted_<field> column, one committed batch at a time, and never wraps the whole
table in a transaction.
The Mix tasks are thin argument-parsing shells over run/3; everything that matters —
planning, the provider pre-flight, keyset paging, the AAD, verification — lives here so
it can be called from a release without Mix.
AshVault.Backfill.run(MyApp.Accounts.User, :email, tenant: "acme", from: :legacy_email)Why it is idempotent and resumable with no state file
Every batch is selected with encrypted_<field> IS NULL, ordered by primary key, paged
with a keyset (pk > last_seen) rather than OFFSET. A row that has already been
encrypted no longer matches the filter, so a re-run after a crash — or after a provider
outage half way through — simply picks up the rows that are still NULL. --resume-from
is an optimisation that skips re-scanning the committed prefix, never a correctness
requirement.
When the field is configured encrypt_nil?: false, a nil source value legitimately
stores SQL NULL in the encrypted column. Those rows would match the NULL filter forever,
so they are excluded with source IS NOT NULL and reported as skipped_nil_source.
Why backfilled rows decrypt through the ordinary read path
The associated data bound into every AshVault ciphertext is
scope | resource | field, and the scope comes from the Ash tenant. A backfill that
built its own context by hand would be one field name away from writing rows that never
decrypt again, so the context is built by AshVault.Context.Builder.from_changeset/3 —
the same function AshVault.Changes.Encrypt calls on an ordinary write — from a real
changeset over the real record, with the plaintext field name (:email), never the
ciphertext attribute name.
The write
One Ash.bulk_update/4 per batch, strategy: [:stream], transaction: :all (so the
batch, and only the batch, is a transaction), return_records?: false, authorize?: false and skip_unknown_inputs: :*. The per-row attributes are applied through the
:transform_changeset hook — a single bulk input map cannot carry different
ciphertext per row.
Those attributes come from AshVault.write_attributes/4, the same function an ordinary
create or update goes through, so a searchable?: true field gets its <field>_lookup
token written in the same batch as its ciphertext, from a value normalized exactly
once. Nothing else on the row is touched.
Because strategy: [:stream] cannot run an update action declared
require_atomic? true, plan/3 rejects such an action up front with a message naming
--action, rather than letting Ash's NoMatchingBulkStrategy leak out of a batch.
Progress reporting
Pass a :reporter — a one-argument function receiving {event, payload} tuples:
{:preflight, map}— resource, field, source, scope, provider, total rows, batch size{:batch, map}— batch number, rows, done, remaining, rate, ETA, elapsed{:done, map}— final counts{:verify_sample, map}/{:verify_done, map}— verification progress and result
Summary
Functions
Check that the key provider is reachable for this plan's scope before any row is written.
Build the write-path %AshVault.Context{} for a row, exactly as an ordinary create or
update would.
Count the rows still needing a backfill under this plan's filter.
The query selecting rows that still need encrypting, optionally after last_pk.
Plan and validate a backfill without touching the database or the key provider.
Back-fill field on resource, or verify it when verify?: true.
Decrypt a sample of backfilled rows and compare them against the plaintext column.
Types
@type plan() :: %{ resource: module(), field: atom(), encrypted_field: atom(), lookup_field: atom() | false, searchable?: boolean(), lookup?: boolean(), source: atom() | nil, primary_key: atom(), tenant: term(), domain: module() | nil, action: atom(), batch_size: pos_integer(), dry_run?: boolean(), resume_from: term(), sample: non_neg_integer() | nil, encrypt_nil?: boolean(), vault: module(), scope: term(), provider: module(), reporter: (term() -> any()) }
A planned backfill: everything run/3 needs, fully resolved and validated.
Functions
@spec check_provider(plan()) :: {:ok, term()} | {:error, Exception.t()}
Check that the key provider is reachable for this plan's scope before any row is written.
Returns {:ok, :ready | {:current, version}}. A dry run only probes (it must never
mint key material); a real run resolves the current key, minting version 1 on first use
exactly as an ordinary write would.
@spec context(plan(), struct() | nil) :: AshVault.Context.t()
Build the write-path %AshVault.Context{} for a row, exactly as an ordinary create or
update would.
@spec count_pending(plan()) :: {:ok, non_neg_integer()} | {:error, Exception.t()}
Count the rows still needing a backfill under this plan's filter.
@spec pending_query(plan(), term()) :: Ash.Query.t()
The query selecting rows that still need encrypting, optionally after last_pk.
@spec plan(module(), atom(), keyword()) :: {:ok, plan()} | {:error, Exception.t()}
Plan and validate a backfill without touching the database or the key provider.
Returns {:ok, plan} or {:error, %ArgumentError{}} naming the missing piece.
@spec run(module(), atom(), keyword()) :: {:ok, map()} | {:error, Exception.t()} | {:error, Exception.t(), map()}
Back-fill field on resource, or verify it when verify?: true.
Options: :from, :tenant, :domain, :action, :batch_size, :resume_from,
:sample, :dry_run?, :verify?, :reporter.
Returns {:ok, stats}, or {:error, exception, stats} when a batch failed — stats
then carries :resume_from, the last primary key that was successfully committed.
@spec verify(plan()) :: {:ok, map()} | {:error, Exception.t(), map()}
Decrypt a sample of backfilled rows and compare them against the plaintext column.
The sample is 10% of the rows or 1000, whichever is smaller, always
including the first and the last row. sample: 0 checks every row.
Rows are paged with the same keyset as the backfill, and only the sampled rows are
decrypted, so verification costs one table scan and sample provider round-trips.