AshVault.Backfill (AshVault v0.2.0)

Copy Markdown View Source

The online backfill engine behind mix ash_vault.backfill and mix ash_vault.verify.

It turns an existing plaintext column into ciphertext in the resource's encrypted_<field> column, one committed batch at a time, and never wraps the whole table in a transaction.

The Mix tasks are thin argument-parsing shells over run/3; everything that matters — planning, the provider pre-flight, keyset paging, the AAD, verification — lives here so it can be called from a release without Mix.

AshVault.Backfill.run(MyApp.Accounts.User, :email, tenant: "acme", from: :legacy_email)

Why it is idempotent and resumable with no state file

Every batch is selected with encrypted_<field> IS NULL, ordered by primary key, paged with a keyset (pk > last_seen) rather than OFFSET. A row that has already been encrypted no longer matches the filter, so a re-run after a crash — or after a provider outage half way through — simply picks up the rows that are still NULL. --resume-from is an optimisation that skips re-scanning the committed prefix, never a correctness requirement.

When the field is configured encrypt_nil?: false, a nil source value legitimately stores SQL NULL in the encrypted column. Those rows would match the NULL filter forever, so they are excluded with source IS NOT NULL and reported as skipped_nil_source.

Why backfilled rows decrypt through the ordinary read path

The associated data bound into every AshVault ciphertext is scope | resource | field, and the scope comes from the Ash tenant. A backfill that built its own context by hand would be one field name away from writing rows that never decrypt again, so the context is built by AshVault.Context.Builder.from_changeset/3 — the same function AshVault.Changes.Encrypt calls on an ordinary write — from a real changeset over the real record, with the plaintext field name (:email), never the ciphertext attribute name.

The write

One Ash.bulk_update/4 per batch, strategy: [:stream], transaction: :all (so the batch, and only the batch, is a transaction), return_records?: false, authorize?: false and skip_unknown_inputs: :*. The per-row attributes are applied through the :transform_changeset hook — a single bulk input map cannot carry different ciphertext per row.

Those attributes come from AshVault.write_attributes/4, the same function an ordinary create or update goes through, so a searchable?: true field gets its <field>_lookup token written in the same batch as its ciphertext, from a value normalized exactly once. Nothing else on the row is touched.

Because strategy: [:stream] cannot run an update action declared require_atomic? true, plan/3 rejects such an action up front with a message naming --action, rather than letting Ash's NoMatchingBulkStrategy leak out of a batch.

Progress reporting

Pass a :reporter — a one-argument function receiving {event, payload} tuples:

  • {:preflight, map} — resource, field, source, scope, provider, total rows, batch size
  • {:batch, map} — batch number, rows, done, remaining, rate, ETA, elapsed
  • {:done, map} — final counts
  • {:verify_sample, map} / {:verify_done, map} — verification progress and result

Summary

Types

A planned backfill: everything run/3 needs, fully resolved and validated.

Functions

Check that the key provider is reachable for this plan's scope before any row is written.

Build the write-path %AshVault.Context{} for a row, exactly as an ordinary create or update would.

Count the rows still needing a backfill under this plan's filter.

The query selecting rows that still need encrypting, optionally after last_pk.

Plan and validate a backfill without touching the database or the key provider.

Back-fill field on resource, or verify it when verify?: true.

Decrypt a sample of backfilled rows and compare them against the plaintext column.

Types

plan()

@type plan() :: %{
  resource: module(),
  field: atom(),
  encrypted_field: atom(),
  lookup_field: atom() | false,
  searchable?: boolean(),
  lookup?: boolean(),
  source: atom() | nil,
  primary_key: atom(),
  tenant: term(),
  domain: module() | nil,
  action: atom(),
  batch_size: pos_integer(),
  dry_run?: boolean(),
  resume_from: term(),
  sample: non_neg_integer() | nil,
  encrypt_nil?: boolean(),
  vault: module(),
  scope: term(),
  provider: module(),
  reporter: (term() -> any())
}

A planned backfill: everything run/3 needs, fully resolved and validated.

Functions

check_provider(plan)

@spec check_provider(plan()) :: {:ok, term()} | {:error, Exception.t()}

Check that the key provider is reachable for this plan's scope before any row is written.

Returns {:ok, :ready | {:current, version}}. A dry run only probes (it must never mint key material); a real run resolves the current key, minting version 1 on first use exactly as an ordinary write would.

context(plan, record \\ nil)

@spec context(plan(), struct() | nil) :: AshVault.Context.t()

Build the write-path %AshVault.Context{} for a row, exactly as an ordinary create or update would.

count_pending(plan)

@spec count_pending(plan()) :: {:ok, non_neg_integer()} | {:error, Exception.t()}

Count the rows still needing a backfill under this plan's filter.

pending_query(plan, last_pk \\ nil)

@spec pending_query(plan(), term()) :: Ash.Query.t()

The query selecting rows that still need encrypting, optionally after last_pk.

plan(resource, field, opts \\ [])

@spec plan(module(), atom(), keyword()) :: {:ok, plan()} | {:error, Exception.t()}

Plan and validate a backfill without touching the database or the key provider.

Returns {:ok, plan} or {:error, %ArgumentError{}} naming the missing piece.

run(resource, field, opts \\ [])

@spec run(module(), atom(), keyword()) ::
  {:ok, map()} | {:error, Exception.t()} | {:error, Exception.t(), map()}

Back-fill field on resource, or verify it when verify?: true.

Options: :from, :tenant, :domain, :action, :batch_size, :resume_from, :sample, :dry_run?, :verify?, :reporter.

Returns {:ok, stats}, or {:error, exception, stats} when a batch failed — stats then carries :resume_from, the last primary key that was successfully committed.

verify(plan)

@spec verify(plan()) :: {:ok, map()} | {:error, Exception.t(), map()}

Decrypt a sample of backfilled rows and compare them against the plaintext column.

The sample is 10% of the rows or 1000, whichever is smaller, always including the first and the last row. sample: 0 checks every row.

Rows are paged with the same keyset as the backfill, and only the sampled rows are decrypted, so verification costs one table scan and sample provider round-trips.