Writing a key provider

Copy Markdown View Source

A key provider is the authoritative store of key material. It is the component that makes crypto-erasure real, because it is the one thing that is deliberately not in your database backup. Getting it wrong is how erasure becomes silent data loss.

Read this whole page before writing one, then run the shared contract suite (test/support/key_provider_cases.ex) against your implementation.

The contract

@type scope :: term()          # always a binary in practice — see below
@type version :: non_neg_integer()
@type key_info :: %{version: version(), key: binary(), created_at: DateTime.t()}

@callback current_key(scope()) :: {:ok, key_info()} | {:error, term()}
@callback get_key(scope(), version()) ::
            {:ok, binary()} | {:error, :not_found} | {:error, :destroyed} | {:error, term()}
@callback rotate(scope()) :: {:ok, version()} | {:error, term()}
@callback destroy(scope()) :: :ok | {:error, term()}
@callback key_bytes() :: pos_integer()      # optional, defaults to 32
@callback child_spec(term()) :: Supervisor.child_spec()   # optional, default: no child
@callback setup() :: :ok | {:error, term()}               # optional, default: :ok

Semantics every implementation must honour:

  • current_key/1 mints the scope's key material on first use, at version 1. Callers never create a scope's key explicitly — there is no create_key/1.
  • get_key/2 returns historical versions. Rotation retains them; that is what lets old ciphertext keep decrypting.
  • rotate/1 on a scope that has never been used mints version 1 and returns {:ok, 1}.
  • destroy/1 is a tombstone, not a delete. See below.
  • destroy/1 on a scope that never had a key still records the tombstone and returns :ok.
  • Scopes are binaries. Reject anything else with an ArgumentError.
  • key_bytes/0 is optional; AshVault.KeyProvider.key_bytes/1 falls back to 32.
  • child_spec/1 and setup/0 are optional, and exist so a host application does not have to case over provider modules to know what yours needs. See below.

Error terms are mapped onto AshVault errors in exactly one place, AshVault.Vault.Runtime:

Your returnWhat the caller sees
{:error, :destroyed}AshVault.Errors.KeyDestroyed
{:error, :not_found}AshVault.Errors.KeyNotFound
{:error, anything_else}AshVault.Errors.ProviderUnavailable (retryable)

So the three-way distinction is entirely in your hands.

Starting up, and being set up

Providers differ in what they need before they can serve a key, and a host application should not have to know which. Two optional callbacks carry that:

  • child_spec/1 — define it if your provider owns a process. You get it for free from use GenServer or use Agent; AshVault.KeyProviders.Local and AshVault.KeyProviders.Memory do exactly that. Omit it if you are stateless, as AshVault.KeyProviders.OpenBao is.
  • setup/0 — define it if there is a one-shot, privileged operator step: creating a key root, mounting a secrets engine, provisioning a policy. It must be idempotent, and it must not be something a supervisor does: setup/0 is for a deploy step, a release command, or a mix task.

A host then writes one line each, through the vault, which also knows about the AshVault.KeyProviders.Cached wrapper when one is in play:

children = [MyApp.Repo] ++ MyApp.Vault.child_specs()
:ok = MyApp.Vault.setup()

Both are optional: a provider that defines neither contributes no children and returns :ok from setup, so an existing third-party provider keeps working untouched.

Reading your configuration

Read it through AshVault.KeyProvider.config/1, not Application.get_env/3:

defp config, do: AshVault.KeyProvider.config(__MODULE__)

That resolves config :ash_vault, MyProvider, ... as a base and merges the host application's own config :my_app, MyProvider, ... over it, key by key. Hosts strongly prefer the latter — configuration under someone else's OTP app name surprises anyone running Application.get_env(:my_app, ...) — and going through the helper is what makes it available to your provider too.

Failure modes that matter

These are not hypotheticals. Every one of them was a real finding against AshVault's own providers, and each has a named case in the contract suite.

1. Tombstones must fail closed

Deleting key material leaves the scope absent, and absent looks exactly like never-used. A current_key/1 that cannot positively determine "no tombstone" and mints a fresh version 1 anyway has resurrected an erased tenant. That tenant then writes new rows under the new key while every pre-existing row is unreadable — and nothing anywhere records that an erasure was ever attempted.

So: a tombstone read that cannot complete is never answered with "not destroyed".

  • AshVault.KeyProviders.Local uses File.stat/1, not File.exists?/1. File.exists?/1 returns false for any failure — :eacces on a mode-000 parent, :eio on a failing disk, :estale on NFS, :eloop. Only a positive :enoent counts as absence; anything else is ProviderUnavailable.
  • AshVault.KeyProviders.OpenBao demands a positive identification of the response body in both directions: 200 counts as destroyed only when the body is a map with a map at "data"; 404 counts as absent only when the body is a map with an "errors" key whose list is empty. An HTML error page from an ingress mid-reload is ProviderUnavailable, not "not destroyed" — and not "destroyed" either.

Related: never auto-provision the store that holds your tombstones on a read path. A freshly created, empty tombstone store reads as "nothing was ever destroyed" for every scope you have. That is fail-open by construction. Provisioning belongs in explicit operator setup (AshVault.KeyProviders.OpenBao.setup/0, mix ash_vault.local.init). Creating the store in order to write a tombstone is defensible — it cannot lose an erasure.

And: never create your own key root. If your key store is a directory on its own volume (which is what you should be telling operators to do) and that volume fails to mount, a File.mkdir_p!/1 in init/1 builds an empty store on the underlying filesystem, reports itself healthy, and resurrects every tenant you ever erased. No attacker needed — a reordered systemd unit is enough. Local requires an operator-created .ash_vault_root sentinel and refuses to start without it, with this message:

AshVault.KeyProviders.Local key root /srv/keys exists but holds no .ash_vault_root
sentinel, so it is either uninitialised or not the volume you think it is.

AshVault.KeyProviders.Local never creates its own key root. If it did, a key volume that
failed to mount would be silently replaced by an empty directory on the underlying
filesystem: no tombstones, no key material, and a fresh version 1 minted for every
tenant that was ever crypto-erased.

2. Ordering: tombstone first, then shred

If you shred key material and then write the tombstone, the window between the two — an ENOSPC, an EACCES, a read-only remount, a process crash — leaves the scope with no keys and no tombstone. The next current_key/1 mints a fresh version 1; existing rows say key_version: 1, so get_key/2 hands back the new v1 key, the tag check fails, and the caller is told AshVault.Errors.CiphertextIntegrityFailed — erasure wearing the costume of tampering.

Tombstone-first fails safe: the worst case is a scope marked destroyed whose key files linger, and the provider refuses to serve them anyway. Optionally rewrite the tombstone with a shredded_at once the shred completes, so an interrupted destroy is visible to an operator.

3. :destroyed must never be conflated with :not_found or an outage

Three distinct answers, three distinct causes, three distinct operator responses:

  • :destroyed — deliberate erasure. Close the ticket; the data is gone by design.
  • :not_found — the provider has no such key and no tombstone. Corruption, a wrong scope key, a key file deleted by hand. Investigate.
  • anything else — an outage. Page someone, then retry.

Concretely, this means:

  • corrupt or truncated metadata → ProviderUnavailable, never :destroyed;
  • metadata that references a version whose key material is missing → :not_found, never :destroyed;
  • a wrong or expired credential (OpenBao answers 403 permission denied on every endpoint) → ProviderUnavailable, never :destroyed;
  • never infer "there was nothing to delete" from an error message. AshVault's OpenBao provider once matched the bare substring "not found" — which also appears in policy denials and proxy-surfaced 400s — and so reported destroy/1 as :ok without ever issuing the delete. It now re-reads the key and requires a positive absence before writing the tombstone; if the key is still there, destroy/1 returns {:error, _}.

Validate version too. It is public API: a caller passing "../../../etc/ssl/private/x" must get {:error, :not_found}, not a file read. Guard with is_integer(version) and version > 0.

4. created_at must never be fabricated

Returning DateTime.utc_now() when the real creation time is unavailable produces a key that is never older than any max_age, so age-based rotation silently never fires and nothing logs a reason. Return {:error, ...} instead:

{:error, %AshVault.Errors.ProviderUnavailable{reason: :malformed_key_metadata}}

created_at must be stable across two current_key/1 calls and must not move backwards across a rotate/1.

5. Key size

Hand back exactly the bytes you claim. A truncated key file, a key_bytes: 16 in config, an OpenBao key_type: "aes128-gcm96" under a 256-bit cipher — all of them used to reach the cipher unchecked and be reported as tampering on decrypt and as a retryable outage on encrypt. Reject a key of the wrong size as ProviderUnavailable rather than passing short bytes on; AshVault.Vault.Runtime raises AshVault.Errors.KeySizeMismatch when it sees one anyway, which is at least the truth.

6. Key material must not reach a log

A GenServer whose state holds raw keys emits every one of them in a SASL crash report, straight into your log aggregator and any APM handler attached to it. Implement format_status/1:

@impl GenServer
def format_status(status) do
  Map.update(status, :state, nil, fn state -> %{state | keys: :redacted} end)
end

A worked example

A provider backed by a table in an external database — enough to show every rule. Note that if you put this table in the same database as your encrypted rows, you have defeated the entire point: a restore brings back both.

defmodule MyApp.KeyProviders.Sql do
  @moduledoc """
  Keys in a dedicated PostgreSQL database, separate from the application database.

  The `ash_vault_keys` and `ash_vault_tombstones` tables MUST live in a database that is
  not part of the application database's backup domain. If one restore brings back both,
  crypto-erasure is void.
  """

  @behaviour AshVault.KeyProvider

  alias AshVault.Errors.ProviderUnavailable

  @key_bytes 32

  @impl AshVault.KeyProvider
  def key_bytes, do: @key_bytes

  @impl AshVault.KeyProvider
  def current_key(scope) when is_binary(scope) do
    with :absent <- tombstone(scope) do
      case latest(scope) do
        {:ok, nil} -> mint(scope, 1)
        {:ok, row} -> {:ok, %{version: row.version, key: row.key, created_at: row.created_at}}
        {:error, reason} -> {:error, unavailable(reason)}
      end
    else
      :destroyed -> {:error, :destroyed}
      {:unavailable, reason} -> {:error, unavailable(reason)}
    end
  end

  def current_key(scope), do: raise_non_binary(scope)

  @impl AshVault.KeyProvider
  def get_key(scope, version) when is_binary(scope) and is_integer(version) and version > 0 do
    with :absent <- tombstone(scope) do
      case fetch(scope, version) do
        {:ok, nil} -> {:error, :not_found}
        # A key of the wrong size is a corrupt store, not tampering. Never hand
        # short bytes to the cipher.
        {:ok, %{key: key}} when byte_size(key) != @key_bytes ->
          {:error, unavailable({:bad_key_size, byte_size(key)})}

        {:ok, %{key: key}} -> {:ok, key}
        {:error, reason} -> {:error, unavailable(reason)}
      end
    else
      :destroyed -> {:error, :destroyed}
      {:unavailable, reason} -> {:error, unavailable(reason)}
    end
  end

  def get_key(scope, _version) when is_binary(scope), do: {:error, :not_found}
  def get_key(scope, _version), do: raise_non_binary(scope)

  @impl AshVault.KeyProvider
  def rotate(scope) when is_binary(scope) do
    with :absent <- tombstone(scope) do
      case latest(scope) do
        {:ok, nil} -> with {:ok, %{version: v}} <- mint(scope, 1), do: {:ok, v}
        {:ok, row} -> with {:ok, %{version: v}} <- mint(scope, row.version + 1), do: {:ok, v}
        {:error, reason} -> {:error, unavailable(reason)}
      end
    else
      :destroyed -> {:error, :destroyed}
      {:unavailable, reason} -> {:error, unavailable(reason)}
    end
  end

  def rotate(scope), do: raise_non_binary(scope)

  @impl AshVault.KeyProvider
  def destroy(scope) when is_binary(scope) do
    # Tombstone FIRST. A crash after the delete but before the tombstone would leave
    # the scope with no keys and no record, and the next current_key/1 would mint a
    # fresh v1 — silent, undetectable data loss reported as CiphertextIntegrityFailed.
    with :ok <- write_tombstone(scope),
         :ok <- delete_keys(scope),
         :destroyed <- tombstone(scope) do
      :ok
    else
      {:error, reason} -> {:error, unavailable(reason)}
      # The tombstone did not read back as destroyed: do not claim an erasure.
      other -> {:error, unavailable({:tombstone_unconfirmed, other})}
    end
  end

  def destroy(scope), do: raise_non_binary(scope)

  # Fails closed: only a positive absence counts as "not destroyed".
  defp tombstone(scope) do
    case query_tombstone(scope) do
      {:ok, nil} -> :absent
      {:ok, _row} -> :destroyed
      {:error, reason} -> {:unavailable, reason}
    end
  end

  defp mint(scope, version) do
    key = :crypto.strong_rand_bytes(@key_bytes)
    created_at = DateTime.utc_now() |> DateTime.truncate(:second)

    case insert(scope, version, key, created_at) do
      :ok -> {:ok, %{version: version, key: key, created_at: created_at}}
      {:error, reason} -> {:error, unavailable(reason)}
    end
  end

  defp unavailable(reason),
    do: ProviderUnavailable.exception(provider: __MODULE__, reason: reason)

  defp raise_non_binary(scope) do
    raise ArgumentError,
          "#{inspect(__MODULE__)} scopes must be binaries, got: #{inspect(scope)}"
  end
end

insert/4, latest/1, fetch/2, delete_keys/1, query_tombstone/1 and write_tombstone/1 are yours; each returns {:ok, row_or_nil}, :ok or {:error, reason}. Note what the shape buys you: every database failure becomes ProviderUnavailable, and nothing anywhere can turn one into :destroyed or :not_found.

Then point a vault at it:

defmodule MyApp.Vault do
  use AshVault.Vault, key_provider: MyApp.KeyProviders.Sql
end

Serving the :mac purpose

A scope can hold a second keyring, :mac, used by AshVault.Mac for mac!/2 and verify_mac!/4 (see Key purposes and MACs). It is optional. A provider that only implements the :data callbacks keeps encrypting exactly as before; a vault over it raises AshVault.Errors.PurposeUnsupported if asked to MAC, and refuses to compile if mac: is configured explicitly.

To serve it, implement four optional callbacks:

@impl AshVault.KeyProvider
def purposes, do: [:data, :mac]

@impl AshVault.KeyProvider
def current_key(scope, :data), do: current_key(scope)
def current_key(scope, :mac), do: ...

@impl AshVault.KeyProvider
def get_key(scope, version, :data), do: get_key(scope, version)
def get_key(scope, version, :mac), do: ...

@impl AshVault.KeyProvider
def rotate(scope, :data), do: rotate(scope)
def rotate(scope, :mac), do: ...

Support is read from purposes/0 and never inferred from arity. Memory and Local have long exported a current_key(server, scope); a provider with a same-shaped helper must not be mistaken for one that understands purposes. Callers go through AshVault.KeyProvider.current_key/3, get_key/4 and rotate/3, which route :data to the original callbacks and refuse any purpose purposes/0 does not list.

Every failure mode above applies to the :mac keyring unchanged, plus:

  • Separate key material. Mint :mac keys from fresh randomness at AshVault.KeyProvider.mac_key_bytes/0 (32) bytes, regardless of your data key_bytes/0. Never reuse or derive from the data key or the lookup key.
  • Name keyrings injectively. If you store the :mac keyring under a name derived from the data keyring's (OpenBao uses <data name>.mac), make sure no scope's MAC name can equal another scope's data or lookup name. Use a separator that cannot occur in an encoded scope.
  • One tombstone. destroy/1 destroys the :mac keyring along with everything else, and the existing tombstone answers {:error, :destroyed} for every purpose. Check it first on every :mac path, and fail closed exactly as for :data.
  • Independent rotation. Rotating :mac never moves :data, and the reverse.

Run the contract suite

test/support/key_provider_cases.ex is a __using__ macro holding the cases every provider must pass. Copy it into your own project (or depend on AshVault's test support) and wire it up:

defmodule MyApp.KeyProviders.SqlTest do
  use ExUnit.Case, async: true

  use AshVault.Test.Support.KeyProviderCases, setup: &__MODULE__.start_provider/1

  def start_provider(_context) do
    %{
      provider: MyApp.KeyProviders.Sql,
      scope: fn -> "scope_#{System.unique_integer([:positive])}" end
    }
  end
end

The setup callback returns :provider (a module, or {module, server} when the instance is not default-named), :scope (a zero-arity generator of fresh scopes), and optionally :key_bytes.

The cases, and the rule each one pins down:

CaseRule
first current_key mints version 1minting is implicit
current_key is stable across callsno re-minting
different scopes get different keysscope isolation
rotate mints v2 and v1 stays fetchablehistory is retained
get_key for an unknown version returns :not_foundnot :destroyed
get_key on an unknown scope returns :not_foundnot :destroyed
destroy makes current_key and get_key both :destroyedtombstone covers both
destroy is idempotentre-running is safe
destroy works on a scope that never had a keytombstone anyway
a destroyed scope never re-mintsthe central promise
a non-binary scope is rejected identically by every providerno term_to_binary scope keys
rotate on a scope that has never been used returns {:ok, 1}consistent with minting
a zero, negative or non-integer version is :not_foundversion is validated, not interpolated
created_at is stable across calls and never moves backwards on rotateno fabricated timestamps
destroying one scope leaves others untouchederasure is scoped
the :mac keyring mints, rotates and is fetched like :datasame contract per purpose
the :mac key is never the :data key nor the lookup keykeyrings are separate
rotating either purpose leaves the other aloneindependent rotation
destroy tombstones :mac too, and it never re-mintsone tombstone per scope

The :mac cases run by default; pass mac: false to the use for a provider that serves only :data.

Add your own tests for whatever your backing store can do that these cannot express — Local, for example, additionally asserts that keys and tombstones survive a provider restart, that a corrupt meta.json is ProviderUnavailable rather than :destroyed, and that making the root unwritable mid-destroy does not let the next current_key/1 return {:ok, %{version: 1}}.