AshVault.KeyProviders.Cached (AshVault v0.2.0)

Copy Markdown View Source

An AshVault.KeyProvider that wraps another provider with a bounded, TTL'd key cache.

Caching keys is a deliberate weakening of crypto-erasure, which is why it is off by default. Read this whole moduledoc before turning it on.

Using it

Almost always through the vault's :cache option, which desugars to this module:

use AshVault.Vault, key_provider: MyApp.Keys, cache: true

use AshVault.Vault,
  key_provider: MyApp.Keys,
  cache: [ttl: :timer.seconds(30), max_entries: 4_096,
          backend: AshVaultRustler.KeyCache]

The wrapper is also public and supported, for when a provider needs to be shared between vaults or named explicitly:

defmodule MyApp.CachedKeys do
  use AshVault.KeyProviders.Cached,
    provider: AshVault.KeyProviders.OpenBao,
    ttl: :timer.seconds(60)
end

use AshVault.Vault, key_provider: MyApp.CachedKeys

Either way the generated module is an ordinary AshVault.KeyProvider and must be started, because it owns the cache:

children = [MyApp.Vault.CachedKeyProvider]   # or MyApp.CachedKeys

If it is not started, every read goes straight to the wrapped provider. Slower, and correct. A cache is allowed to fail to cache; it is never allowed to fail to serve.

What this buys, exactly

It removes the long-lived copy of a key from wherever the provider keeps it, and it removes one network round trip per operation. With AshVaultRustler.KeyCache as the backend, the authoritative copy lives outside the BEAM heap and eviction zeroes it.

What it does not buy

get_key/2 still returns a binary to the BEAM, so a transient copy of the key exists on an Elixir process heap for every single encrypt and decrypt. This level bounds the window; it does not eliminate the copy. Eliminating it needs the opaque-handle path (AshVault.Key plus a cipher that can use one), which is a separate, opt-in mechanism.

The pure-Elixir backend (AshVault.KeyCaches.ETS, the default) additionally cannot zero anything at all — an eviction drops a reference and the garbage collector decides the rest. Its guarantee is the observable one: after evict_scope/1 returns, nothing can read that scope from the cache again.

The TTL is your erasure SLA — but only for destroys routed through here

This is the sentence to take away.

destroy/1 evicts the scope synchronously on every connected node before it returns, and fails if any node does not acknowledge. So an erasure performed through this provider — AshVault.destroy_keys!/2, the generic action, mix ash_vault.destroy_keys running in this BEAM — is immediate, not eventual.

An erasure performed out of band is not. A mix ash_vault.destroy_keys in a separate node that is not clustered with the running app, an operator deleting a transit key in OpenBao by hand, another service revoking it — none of those call evict_scope/1, and this cache will keep serving the erased scope's key until the entry expires. For those, the TTL is the maximum time a destroyed tenant stays decryptable, with nothing underneath it. That number is what you can promise in a DPA.

The default TTL is therefore 30 seconds, for both the current key and historical versions. It is short enough to write down without qualification and long enough to collapse the per-operation round trip on any realistic request rate.

Historical key versions are immutable in content, which tempts a long TTL — the spec that asked for this package suggests exactly that. They are not immutable in existence: a destroyed scope's v1 key must stop being served. The erasure SLA is max(ttl, historical_ttl), so raising :historical_ttl raises the SLA. The default keeps them equal on purpose.

The tombstone asymmetry

The single most important rule in this module, stated as code would state it:

  • {:error, :destroyed} is cached, without expiry. Destruction is monotonic — a scope that is destroyed stays destroyed forever — so a cached tombstone can only ever be right, and caching it makes reads fail closed even when the provider is unreachable.
  • The absence of a tombstone is never cached as such. There is no "not destroyed" entry, ever.
  • {:error, :not_found} is never cached. A scope with no key at version N may have one at version N a moment later, and caching absence here would be the same mistake in a smaller costume.
  • Any other error — a transport failure, a provider outage — is never cached, in either direction. An outage is not an answer.

Caching {:ok, key} is, strictly speaking, an implicit cache of "not destroyed", and pretending otherwise would be the fifth fail-open tombstone read in this project's history. What contains it is the two mechanisms above: synchronous cluster-wide eviction on the destroy path, and a short TTL as the floor for everything else.

The generation fence

A read that misses can race a destroy/1 and repopulate the cache after the eviction, which would resurrect the scope for a full TTL. Every fetch returns the scope's generation; evict_scope/1 bumps it; a put carrying a stale generation is dropped. destroy/1 evicts both before and after the wrapped provider's destroy, so the only writes that can survive are ones that were already committed before the erasure began.

:mac keys pass through uncached

Only the :data keyring is cached. A :mac key request (see Purposes in AshVault.KeyProvider) goes straight to the wrapped provider every time, gated only by this cache's own tombstone: a scope already known to be destroyed answers {:error, :destroyed} without a round trip, and a :destroyed answer from the wrapped provider is remembered exactly as it is for :data. Caching MAC keys would need a slot per purpose in every AshVault.KeyCache backend, and a second keyring for destroy/1 to evict before erasure is true; until that exists, passing through is the answer that cannot be stale.

Options

  • :provider — required, the wrapped AshVault.KeyProvider.
  • :backend — an AshVault.KeyCache, default AshVault.KeyCaches.ETS.
  • :ttl — milliseconds for the current key, default 30_000.
  • :historical_ttl — milliseconds for a specific version, default :ttl.
  • :max_entries — default 1_024. With AshVaultRustler.KeyCache this is the bound that matters operationally: that backend spends one locked page per cached key, so size it against RLIMIT_MEMLOCK, not against :max_bytes.
  • :max_bytes — default 1_048_576. Counts key bytes, not pages.
  • :cluster — fan evict_scope/1 out to Node.list(:connected), default true.
  • :evict_timeout — milliseconds to wait for each node, default 5_000.

Summary

Functions

Normalise and validate the wrapper's options into the map the runtime functions take.

Turn :erpc.multicall/5 results into a list of {node, reason} failures.

Fetch the current key for a scope, serving a live cache entry when there is one.

Fetch the current key for a scope and purpose. :data is current_key/2; any other purpose passes through to the wrapped provider uncached (see the moduledoc).

The default TTL in milliseconds, 30_000.

Destroy the scope at the wrapped provider and evict it from every node's cache.

Synchronously drop a scope from this cache on every connected node.

Fetch a specific key version for a scope.

Fetch one key version for a scope and purpose. :data is get_key/3; any other purpose passes through uncached.

The key size the wrapped provider mints.

Fetch the wrapped provider's lookup key, uncached.

The purposes the wrapped provider serves.

Rotate the wrapped provider's key for a scope, then drop the scope from the cache.

Rotate a scope's keyring for a purpose. :data is rotate/2; any other purpose has nothing cached to evict.

Types

opts()

@type opts() :: %{
  provider: module(),
  backend: module(),
  cache_name: atom(),
  ttl: non_neg_integer(),
  historical_ttl: non_neg_integer(),
  max_entries: pos_integer(),
  max_bytes: pos_integer(),
  cluster: boolean(),
  evict_timeout: timeout()
}

Functions

build_opts!(cache_name, opts)

@spec build_opts!(module(), keyword()) :: opts()

Normalise and validate the wrapper's options into the map the runtime functions take.

Raises ArgumentError with a message in the same voice as the vault's own option errors when the shape is wrong.

classify_evictions(results, nodes)

@spec classify_evictions([term()], [node()]) :: [{node(), term()}]

Turn :erpc.multicall/5 results into a list of {node, reason} failures.

Public because it is the branch that decides whether a destroy! is allowed to report success, and a single-node test suite never reaches it through evict_scope/1 — Node.list(:connected) is empty, so the whole fan-out is skipped. Calling it directly with synthetic :erpc shapes is the only way to cover it without standing up a cluster.

Anything that is not a positive {:ok, :ok} is a failure. Not "probably fine", not "the node is probably down anyway": a node that did not say it evicted the key is a node that may still be serving it.

current_key(scope, opts)

@spec current_key(AshVault.KeyProvider.scope(), opts()) ::
  {:ok, AshVault.KeyProvider.key_info()} | {:error, term()}

Fetch the current key for a scope, serving a live cache entry when there is one.

current_key(scope, purpose, opts)

@spec current_key(
  AshVault.KeyProvider.scope(),
  AshVault.KeyProvider.purpose(),
  opts()
) ::
  {:ok, AshVault.KeyProvider.key_info()} | {:error, term()}

Fetch the current key for a scope and purpose. :data is current_key/2; any other purpose passes through to the wrapped provider uncached (see the moduledoc).

default_ttl()

@spec default_ttl() :: pos_integer()

The default TTL in milliseconds, 30_000.

Short on purpose: for an out-of-band erasure this is the whole erasure SLA. See the moduledoc.

destroy(scope, opts)

@spec destroy(AshVault.KeyProvider.scope(), opts()) :: :ok | {:error, term()}

Destroy the scope at the wrapped provider and evict it from every node's cache.

Ordering, and why:

  1. Evict everywhere. Bumps the generation, so any read already in flight cannot write its result back.
  2. provider.destroy/1. If it fails, return the failure and cache nothing — no tombstone is recorded for an erasure that did not happen.
  3. Evict everywhere again, and require every node to acknowledge. A read that started between steps 1 and 2 saw a live key; this is what removes it. Returning :ok while a peer still holds the key is the worst outcome available, so a node that does not answer makes this return {:error, {:cache_evict_failed, _}} and AshVault.Vault.Runtime.destroy!/2 raises.
  4. Only then record the tombstone locally, after the last eviction that would have wiped it. Peers have an empty cache and will read :destroyed from the provider.

evict_scope(scope, opts)

@spec evict_scope(AshVault.KeyProvider.scope(), opts()) :: :ok | {:error, term()}

Synchronously drop a scope from this cache on every connected node.

Returns {:error, {:nodes_failed, [{node, reason}]}} if any node does not acknowledge. Nodes that join after this returns have a cold cache and cannot hold a stale entry, so they need no handling.

get_key(scope, version, opts)

@spec get_key(AshVault.KeyProvider.scope(), AshVault.KeyProvider.version(), opts()) ::
  {:ok, binary()} | {:error, term()}

Fetch a specific key version for a scope.

get_key(scope, version, purpose, opts)

Fetch one key version for a scope and purpose. :data is get_key/3; any other purpose passes through uncached.

key_bytes(opts)

@spec key_bytes(opts()) :: pos_integer()

The key size the wrapped provider mints.

lookup_key(scope, opts)

@spec lookup_key(AshVault.KeyProvider.scope(), opts()) ::
  {:ok, binary()} | {:error, term()}

Fetch the wrapped provider's lookup key, uncached.

A lookup key never rotates, so caching it would save a round trip on a value the wrapped provider can already keep as long as it likes — and would add a second thing destroy/1 has to evict before erasure is true. Passing straight through removes that question entirely.

Returns {:error, :lookup_unsupported} when the wrapped provider has no AshVault.KeyProvider.lookup_key/1, which is also what AshVault.KeyProvider.supports_lookup?/1 reports for this wrapper: it unwraps to the provider underneath rather than believing this delegation.

purposes(opts)

@spec purposes(opts()) :: [AshVault.KeyProvider.purpose()]

The purposes the wrapped provider serves.

rotate(scope, opts)

@spec rotate(AshVault.KeyProvider.scope(), opts()) ::
  {:ok, AshVault.KeyProvider.version()} | {:error, term()}

Rotate the wrapped provider's key for a scope, then drop the scope from the cache.

Eviction failures here are logged, not raised: a stale cached key after a rotation means writes keep using the previous version for up to one TTL, which is a performance and hygiene problem. Erasure is the case where a stale entry is a correctness failure, and destroy/1 treats it as one.

rotate(scope, purpose, opts)

Rotate a scope's keyring for a purpose. :data is rotate/2; any other purpose has nothing cached to evict.