# `AshVault.Backfill`
[🔗](https://github.com/jakeschurch/ash_vault/blob/main/lib/ash_vault/backfill.ex#L1)

The online backfill engine behind `mix ash_vault.backfill` and `mix ash_vault.verify`.

It turns an existing plaintext column into ciphertext in the resource's
`encrypted_<field>` column, one committed batch at a time, and never wraps the whole
table in a transaction.

The Mix tasks are thin argument-parsing shells over `run/3`; everything that matters —
planning, the provider pre-flight, keyset paging, the AAD, verification — lives here so
it can be called from a release without Mix.

    AshVault.Backfill.run(MyApp.Accounts.User, :email, tenant: "acme", from: :legacy_email)

## Why it is idempotent and resumable with no state file

Every batch is selected with `encrypted_<field> IS NULL`, ordered by primary key, paged
with a keyset (`pk > last_seen`) rather than `OFFSET`. A row that has already been
encrypted no longer matches the filter, so a re-run after a crash — or after a provider
outage half way through — simply picks up the rows that are still NULL. `--resume-from`
is an optimisation that skips re-scanning the committed prefix, never a correctness
requirement.

When the field is configured `encrypt_nil?: false`, a `nil` source value legitimately
stores SQL NULL in the encrypted column. Those rows would match the NULL filter forever,
so they are excluded with `source IS NOT NULL` and reported as `skipped_nil_source`.

## Why backfilled rows decrypt through the ordinary read path

The associated data bound into every AshVault ciphertext is
`scope | resource | field`, and the scope comes from the Ash tenant. A backfill that
built its own context by hand would be one field name away from writing rows that never
decrypt again, so the context is built by `AshVault.Context.Builder.from_changeset/3` —
the same function `AshVault.Changes.Encrypt` calls on an ordinary write — from a real
changeset over the real record, with the plaintext *field* name (`:email`), never the
ciphertext attribute name.

## The write

One `Ash.bulk_update/4` per batch, `strategy: [:stream]`, `transaction: :all` (so the
batch, and only the batch, is a transaction), `return_records?: false`, `authorize?:
false` and `skip_unknown_inputs: :*`. The per-row attributes are applied through the
`:transform_changeset` hook — a single bulk `input` map cannot carry different
ciphertext per row.

Those attributes come from `AshVault.write_attributes/4`, the same function an ordinary
create or update goes through, so a `searchable?: true` field gets its `<field>_lookup`
token written in the same batch as its ciphertext, from a value normalized exactly
once. Nothing else on the row is touched.

Because `strategy: [:stream]` cannot run an update action declared
`require_atomic? true`, `plan/3` rejects such an action up front with a message naming
`--action`, rather than letting Ash's `NoMatchingBulkStrategy` leak out of a batch.

## Progress reporting

Pass a `:reporter` — a one-argument function receiving `{event, payload}` tuples:

  * `{:preflight, map}` — resource, field, source, scope, provider, total rows, batch size
  * `{:batch, map}` — batch number, rows, done, remaining, rate, ETA, elapsed
  * `{:done, map}` — final counts
  * `{:verify_sample, map}` / `{:verify_done, map}` — verification progress and result

# `plan`

```elixir
@type plan() :: %{
  resource: module(),
  field: atom(),
  encrypted_field: atom(),
  lookup_field: atom() | false,
  searchable?: boolean(),
  lookup?: boolean(),
  source: atom() | nil,
  primary_key: atom(),
  tenant: term(),
  domain: module() | nil,
  action: atom(),
  batch_size: pos_integer(),
  dry_run?: boolean(),
  resume_from: term(),
  sample: non_neg_integer() | nil,
  encrypt_nil?: boolean(),
  vault: module(),
  scope: term(),
  provider: module(),
  reporter: (term() -&gt; any())
}
```

A planned backfill: everything `run/3` needs, fully resolved and validated.

# `check_provider`

```elixir
@spec check_provider(plan()) :: {:ok, term()} | {:error, Exception.t()}
```

Check that the key provider is reachable for this plan's scope *before* any row is
written.

Returns `{:ok, :ready | {:current, version}}`. A dry run only probes (it must never
mint key material); a real run resolves the current key, minting version 1 on first use
exactly as an ordinary write would.

# `context`

```elixir
@spec context(plan(), struct() | nil) :: AshVault.Context.t()
```

Build the write-path `%AshVault.Context{}` for a row, exactly as an ordinary create or
update would.

# `count_pending`

```elixir
@spec count_pending(plan()) :: {:ok, non_neg_integer()} | {:error, Exception.t()}
```

Count the rows still needing a backfill under this plan's filter.

# `pending_query`

```elixir
@spec pending_query(plan(), term()) :: Ash.Query.t()
```

The query selecting rows that still need encrypting, optionally after `last_pk`.

# `plan`

```elixir
@spec plan(module(), atom(), keyword()) :: {:ok, plan()} | {:error, Exception.t()}
```

Plan and validate a backfill without touching the database or the key provider.

Returns `{:ok, plan}` or `{:error, %ArgumentError{}}` naming the missing piece.

# `run`

```elixir
@spec run(module(), atom(), keyword()) ::
  {:ok, map()} | {:error, Exception.t()} | {:error, Exception.t(), map()}
```

Back-fill `field` on `resource`, or verify it when `verify?: true`.

Options: `:from`, `:tenant`, `:domain`, `:action`, `:batch_size`, `:resume_from`,
`:sample`, `:dry_run?`, `:verify?`, `:reporter`.

Returns `{:ok, stats}`, or `{:error, exception, stats}` when a batch failed — `stats`
then carries `:resume_from`, the last primary key that was successfully committed.

# `verify`

```elixir
@spec verify(plan()) :: {:ok, map()} | {:error, Exception.t(), map()}
```

Decrypt a sample of backfilled rows and compare them against the plaintext column.

The sample is 10% of the rows or 1000, whichever is smaller, always
including the first and the last row. `sample: 0` checks every row.

Rows are paged with the same keyset as the backfill, and only the sampled rows are
decrypted, so verification costs one table scan and `sample` provider round-trips.

---

*Consult [api-reference.md](api-reference.md) for complete listing*
