An AshVault.KeyProvider that wraps another provider with a bounded, TTL'd key cache.
Caching keys is a deliberate weakening of crypto-erasure, which is why it is off by default. Read this whole moduledoc before turning it on.
Using it
Almost always through the vault's :cache option, which desugars to this module:
use AshVault.Vault, key_provider: MyApp.Keys, cache: true
use AshVault.Vault,
key_provider: MyApp.Keys,
cache: [ttl: :timer.seconds(30), max_entries: 4_096,
backend: AshVaultRustler.KeyCache]The wrapper is also public and supported, for when a provider needs to be shared between vaults or named explicitly:
defmodule MyApp.CachedKeys do
use AshVault.KeyProviders.Cached,
provider: AshVault.KeyProviders.OpenBao,
ttl: :timer.seconds(60)
end
use AshVault.Vault, key_provider: MyApp.CachedKeysEither way the generated module is an ordinary AshVault.KeyProvider and must be
started, because it owns the cache:
children = [MyApp.Vault.CachedKeyProvider] # or MyApp.CachedKeysIf it is not started, every read goes straight to the wrapped provider. Slower, and correct. A cache is allowed to fail to cache; it is never allowed to fail to serve.
What this buys, exactly
It removes the long-lived copy of a key from wherever the provider keeps it, and it
removes one network round trip per operation. With AshVaultRustler.KeyCache as the
backend, the authoritative copy lives outside the BEAM heap and eviction zeroes it.
What it does not buy
get_key/2 still returns a binary to the BEAM, so a transient copy of the key
exists on an Elixir process heap for every single encrypt and decrypt. This level
bounds the window; it does not eliminate the copy. Eliminating it needs the
opaque-handle path (AshVault.Key plus a cipher that can use one), which is a
separate, opt-in mechanism.
The pure-Elixir backend (AshVault.KeyCaches.ETS, the default) additionally cannot
zero anything at all — an eviction drops a reference and the garbage collector
decides the rest. Its guarantee is the observable one: after evict_scope/1
returns, nothing can read that scope from the cache again.
The TTL is your erasure SLA — but only for destroys routed through here
This is the sentence to take away.
destroy/1 evicts the scope synchronously on every connected node before it
returns, and fails if any node does not acknowledge. So an erasure performed through
this provider — AshVault.destroy_keys!/2, the generic action, mix ash_vault.destroy_keys running in this BEAM — is immediate, not eventual.
An erasure performed out of band is not. A mix ash_vault.destroy_keys in a
separate node that is not clustered with the running app, an operator deleting a
transit key in OpenBao by hand, another service revoking it — none of those call
evict_scope/1, and this cache will keep serving the erased scope's key until the
entry expires. For those, the TTL is the maximum time a destroyed tenant stays
decryptable, with nothing underneath it. That number is what you can promise in a DPA.
The default TTL is therefore 30 seconds, for both the current key and historical versions. It is short enough to write down without qualification and long enough to collapse the per-operation round trip on any realistic request rate.
Historical key versions are immutable in content, which tempts a long TTL — the
spec that asked for this package suggests exactly that. They are not immutable in
existence: a destroyed scope's v1 key must stop being served. The erasure SLA is
max(ttl, historical_ttl), so raising :historical_ttl raises the SLA. The default
keeps them equal on purpose.
The tombstone asymmetry
The single most important rule in this module, stated as code would state it:
{:error, :destroyed}is cached, without expiry. Destruction is monotonic — a scope that is destroyed stays destroyed forever — so a cached tombstone can only ever be right, and caching it makes reads fail closed even when the provider is unreachable.- The absence of a tombstone is never cached as such. There is no "not destroyed" entry, ever.
{:error, :not_found}is never cached. A scope with no key at version N may have one at version N a moment later, and caching absence here would be the same mistake in a smaller costume.- Any other error — a transport failure, a provider outage — is never cached, in either direction. An outage is not an answer.
Caching {:ok, key} is, strictly speaking, an implicit cache of "not destroyed", and
pretending otherwise would be the fifth fail-open tombstone read in this project's
history. What contains it is the two mechanisms above: synchronous cluster-wide
eviction on the destroy path, and a short TTL as the floor for everything else.
The generation fence
A read that misses can race a destroy/1 and repopulate the cache after the
eviction, which would resurrect the scope for a full TTL. Every fetch returns the
scope's generation; evict_scope/1 bumps it; a put carrying a stale generation is
dropped. destroy/1 evicts both before and after the wrapped provider's destroy, so
the only writes that can survive are ones that were already committed before the
erasure began.
:mac keys pass through uncached
Only the :data keyring is cached. A :mac key request (see Purposes in
AshVault.KeyProvider) goes straight to the wrapped provider every time, gated only by
this cache's own tombstone: a scope already known to be destroyed answers
{:error, :destroyed} without a round trip, and a :destroyed answer from the wrapped
provider is remembered exactly as it is for :data. Caching MAC keys would need a slot
per purpose in every AshVault.KeyCache backend, and a second keyring for destroy/1
to evict before erasure is true; until that exists, passing through is the answer that
cannot be stale.
Options
:provider— required, the wrappedAshVault.KeyProvider.:backend— anAshVault.KeyCache, defaultAshVault.KeyCaches.ETS.:ttl— milliseconds for the current key, default30_000.:historical_ttl— milliseconds for a specific version, default:ttl.:max_entries— default1_024. WithAshVaultRustler.KeyCachethis is the bound that matters operationally: that backend spends one locked page per cached key, so size it againstRLIMIT_MEMLOCK, not against:max_bytes.:max_bytes— default1_048_576. Counts key bytes, not pages.:cluster— fanevict_scope/1out toNode.list(:connected), defaulttrue.:evict_timeout— milliseconds to wait for each node, default5_000.
Summary
Functions
Normalise and validate the wrapper's options into the map the runtime functions take.
Turn :erpc.multicall/5 results into a list of {node, reason} failures.
Fetch the current key for a scope, serving a live cache entry when there is one.
Fetch the current key for a scope and purpose. :data is current_key/2; any other
purpose passes through to the wrapped provider uncached (see the moduledoc).
The default TTL in milliseconds, 30_000.
Destroy the scope at the wrapped provider and evict it from every node's cache.
Synchronously drop a scope from this cache on every connected node.
Fetch a specific key version for a scope.
Fetch one key version for a scope and purpose. :data is get_key/3; any other
purpose passes through uncached.
The key size the wrapped provider mints.
Fetch the wrapped provider's lookup key, uncached.
The purposes the wrapped provider serves.
Rotate the wrapped provider's key for a scope, then drop the scope from the cache.
Rotate a scope's keyring for a purpose. :data is rotate/2; any other purpose has
nothing cached to evict.
Types
@type opts() :: %{ provider: module(), backend: module(), cache_name: atom(), ttl: non_neg_integer(), historical_ttl: non_neg_integer(), max_entries: pos_integer(), max_bytes: pos_integer(), cluster: boolean(), evict_timeout: timeout() }
Functions
Normalise and validate the wrapper's options into the map the runtime functions take.
Raises ArgumentError with a message in the same voice as the vault's own option
errors when the shape is wrong.
Turn :erpc.multicall/5 results into a list of {node, reason} failures.
Public because it is the branch that decides whether a destroy! is allowed to report
success, and a single-node test suite never reaches it through evict_scope/1 —
Node.list(:connected) is empty, so the whole fan-out is skipped. Calling it directly
with synthetic :erpc shapes is the only way to cover it without standing up a cluster.
Anything that is not a positive {:ok, :ok} is a failure. Not "probably fine", not
"the node is probably down anyway": a node that did not say it evicted the key is a node
that may still be serving it.
@spec current_key(AshVault.KeyProvider.scope(), opts()) :: {:ok, AshVault.KeyProvider.key_info()} | {:error, term()}
Fetch the current key for a scope, serving a live cache entry when there is one.
@spec current_key( AshVault.KeyProvider.scope(), AshVault.KeyProvider.purpose(), opts() ) :: {:ok, AshVault.KeyProvider.key_info()} | {:error, term()}
Fetch the current key for a scope and purpose. :data is current_key/2; any other
purpose passes through to the wrapped provider uncached (see the moduledoc).
@spec default_ttl() :: pos_integer()
The default TTL in milliseconds, 30_000.
Short on purpose: for an out-of-band erasure this is the whole erasure SLA. See the moduledoc.
@spec destroy(AshVault.KeyProvider.scope(), opts()) :: :ok | {:error, term()}
Destroy the scope at the wrapped provider and evict it from every node's cache.
Ordering, and why:
- Evict everywhere. Bumps the generation, so any read already in flight cannot write its result back.
provider.destroy/1. If it fails, return the failure and cache nothing — no tombstone is recorded for an erasure that did not happen.- Evict everywhere again, and require every node to acknowledge. A read that
started between steps 1 and 2 saw a live key; this is what removes it. Returning
:okwhile a peer still holds the key is the worst outcome available, so a node that does not answer makes this return{:error, {:cache_evict_failed, _}}andAshVault.Vault.Runtime.destroy!/2raises. - Only then record the tombstone locally, after the last eviction that would have
wiped it. Peers have an empty cache and will read
:destroyedfrom the provider.
@spec evict_scope(AshVault.KeyProvider.scope(), opts()) :: :ok | {:error, term()}
Synchronously drop a scope from this cache on every connected node.
Returns {:error, {:nodes_failed, [{node, reason}]}} if any node does not
acknowledge. Nodes that join after this returns have a cold cache and cannot hold a
stale entry, so they need no handling.
@spec get_key(AshVault.KeyProvider.scope(), AshVault.KeyProvider.version(), opts()) :: {:ok, binary()} | {:error, term()}
Fetch a specific key version for a scope.
@spec get_key( AshVault.KeyProvider.scope(), AshVault.KeyProvider.version(), AshVault.KeyProvider.purpose(), opts() ) :: {:ok, AshVault.Key.t()} | {:error, term()}
Fetch one key version for a scope and purpose. :data is get_key/3; any other
purpose passes through uncached.
@spec key_bytes(opts()) :: pos_integer()
The key size the wrapped provider mints.
@spec lookup_key(AshVault.KeyProvider.scope(), opts()) :: {:ok, binary()} | {:error, term()}
Fetch the wrapped provider's lookup key, uncached.
A lookup key never rotates, so caching it would save a round trip on a value the
wrapped provider can already keep as long as it likes — and would add a second thing
destroy/1 has to evict before erasure is true. Passing straight through removes that
question entirely.
Returns {:error, :lookup_unsupported} when the wrapped provider has no
AshVault.KeyProvider.lookup_key/1, which is also what
AshVault.KeyProvider.supports_lookup?/1 reports for this wrapper: it unwraps to the
provider underneath rather than believing this delegation.
@spec purposes(opts()) :: [AshVault.KeyProvider.purpose()]
The purposes the wrapped provider serves.
@spec rotate(AshVault.KeyProvider.scope(), opts()) :: {:ok, AshVault.KeyProvider.version()} | {:error, term()}
Rotate the wrapped provider's key for a scope, then drop the scope from the cache.
Eviction failures here are logged, not raised: a stale cached key after a rotation
means writes keep using the previous version for up to one TTL, which is a
performance and hygiene problem. Erasure is the case where a stale entry is a
correctness failure, and destroy/1 treats it as one.
@spec rotate(AshVault.KeyProvider.scope(), AshVault.KeyProvider.purpose(), opts()) :: {:ok, AshVault.KeyProvider.version()} | {:error, term()}
Rotate a scope's keyring for a purpose. :data is rotate/2; any other purpose has
nothing cached to evict.