Everything an operator needs: which provider to run, how to set it up, what to back up, what never to back up, the mix tasks, and how to read every error AshVault can produce.
Choosing a provider
| Provider | Use it for | Key store | Survives restart | Multi-node |
|---|---|---|---|---|
AshVault.KeyProviders.Memory | tests, a first look | process memory | no | no |
AshVault.KeyProviders.Local | single-node deployments, homelab, dev with persistence | files on disk | yes | no |
AshVault.KeyProviders.OpenBao | production, multi-node | OpenBao/Vault transit | yes | yes |
The requirement that actually decides this is: the key store must be a different system
from the database. Not a different schema, not another table — a different system, with
its own credentials, that a database restore cannot reach. Local satisfies that with a
directory on another volume; Memory cannot (it forgets everything on restart, so in
production it is not a key store, it is a countdown to total data loss).
Memory
# config/dev.exs and config/test.exs
config :my_app, start_memory_key_provider?: true
# lib/my_app/application.ex — a runtime flag, not Mix.env/0, because Mix is not
# available in a release.
children =
[MyApp.Repo] ++
if Application.get_env(:my_app, :start_memory_key_provider?, false) do
[AshVault.KeyProviders.Memory]
else
[]
endAshVault deliberately does not start it for you. In tests, start a uniquely-named instance per test where you can:
start_supervised!({AshVault.KeyProviders.Memory, name: Memory})(the AshVault.KeyProvider callbacks take no server name, so a vault always talks to the
default-named instance — hence name: Memory in tests that go through a vault).
Local
One-time operator setup, with the key volume mounted:
mix ash_vault.local.init /var/lib/my_app/ash_vault_keysInitialised AshVault key root: /var/lib/my_app/ash_vault_keys
mode: 0700
sentinel: .ash_vault_root
Back this directory up separately from the database, and exclude it from the
database backup job. See the AshVault.KeyProviders.Local moduledoc.Then:
# config/runtime.exs
config :my_app, AshVault.KeyProviders.Local,
root: "/var/lib/my_app/ash_vault_keys",
key_bytes: 32
# lib/my_app/application.ex
children = [MyApp.Repo] ++ MyApp.Vault.child_specs()
# a deploy step — `init_root!/1` on the configured root, idempotent
:ok = MyApp.Vault.setup()The provider never creates its own key root, and refuses to start without the
.ash_vault_root sentinel:
AshVault.KeyProviders.Local key root /var/lib/my_app/ash_vault_keys exists but holds no
.ash_vault_root sentinel, so it is either uninitialised or not the volume you think it is.
AshVault.KeyProviders.Local never creates its own key root. If it did, a key volume that
failed to mount would be silently replaced by an empty directory on the underlying
filesystem: no tombstones, no key material, and a fresh version 1 minted for every
tenant that was ever crypto-erased.
Initialise the root once, explicitly, as the operator:
mix ash_vault.local.init /var/lib/my_app/ash_vault_keysDo not run ash_vault.local.init to make a startup error go away
If the provider refuses to start on a root you know was initialised, the sentinel is missing because the volume is missing. Running the task then creates a second, empty key store over the top of the mount point and resurrects every destroyed tenant. Check the mount first. That refusal is the safety mechanism, not an inconvenience.
mix ash_vault.local.init --force overrides the "non-empty directory with no sentinel"
refusal. You want that approximately never.
On-disk layout, so you can find things:
<root>/
.ash_vault_root # sentinel
<scope_dir>/ # Base.url_encode64(scope, padding: false)
meta.json # {"current": 2, "versions": {"1": "...", "2": "..."}}
v1.key # raw key bytes, mode 0600
v2.key
<scope_dir>.tombstone # presence == destroyed; sits BESIDE the directoryAshVault.KeyProviders.Local.scope_dir/1 maps a tenant to its directory:
iex> AshVault.KeyProviders.Local.scope_dir("acme")
"YWNtZQ"The provider logs loudly (and starts anyway) if the root is group- or world-accessible:
AshVault.KeyProviders.Local key root /var/lib/my_app/keys has mode 0755, which is
group- or world-accessible.
Key material is readable by other users on this machine. Run:
chmod 0700 /var/lib/my_app/keys
Starting anyway.Local is single-node. Two nodes sharing one NFS or SMB mount will race on
meta.json and can mint conflicting versions. Use OpenBao.
OpenBao
# config/runtime.exs
config :my_app, AshVault.KeyProviders.OpenBao,
address: System.fetch_env!("BAO_ADDR"),
token: {:system, "BAO_TOKEN"},
transit_mount: "transit",
kv_mount: "ashvault",
key_type: "aes256-gcm96",
receive_timeout: 5_000,
max_retries: 2:token accepts a literal binary, {:system, "VAR"}, or a zero-arity function. The
provider is stateless — there is nothing to add to your supervision tree.
In Kubernetes, use auth: {:kubernetes, role: "..."} in place of :token, and
cacertfile: for a server certificate signed by a private CA. See
Running against OpenBao in Kubernetes.
One-time operator setup, which mounts the KV-v2 engine that holds tombstones:
:ok = MyApp.Vault.setup() # or AshVault.KeyProviders.OpenBao.setup()A mix task does not start your application
OpenBao speaks HTTP through req, whose Finch pool is started by the :req
application. A mix task with @requirements ["app.config"] configures the
application without starting it, and a release command may not have started it either.
Call Application.ensure_all_started(:req) first. If you forget, AshVault says so
precisely — %AshVault.Errors.ProviderUnavailable{reason: {:not_started, :req}}, whose
message states that this is not an outage, that the server was never contacted, and
what to add where. It is deliberately not reported as a transport failure, which is
what an unreachable server looks like.
or equivalently, outside the app:
bao secrets enable -path=ashvault -version=2 kv
bao secrets enable transitThe KV mount is never created on a read path
A missing KV mount answers 404 "no handler for route ..." — status-identical to "no
tombstone here". Mounting it and retrying would find a freshly created, empty store and
report every destroyed scope as intact. The provider therefore reports a missing mount
as ProviderUnavailable and stops. Mounting belongs in setup/0, run once, by you.
The app's token needs transit read/create/rotate/delete and the
transit/export/encryption-key/* capability, plus read/write on ashvault/data/tombstones/*.
Nothing else in your infrastructure should hold the export capability — it is the ability
to read raw key material.
Find a tenant's transit key name:
iex> AshVault.KeyProviders.OpenBao.key_name("acme")
"ashvault_YWNtZQ"Semantics worth knowing:
destroy/1returns:okonly when the transit key is confirmed absent by a fresh read and the tombstone write succeeded. It never infers "there was nothing to delete" from an error message.rotate/1on a scope with no key mints version 1 and returns{:ok, 1}.- A wrong or expired token yields
403 permission deniedon every endpoint, which is alwaysAshVault.Errors.ProviderForbidden— never:destroyed.
The token can reach your APM through Finch telemetry
AshVault never logs the token, never puts it in an error struct, and never puts it in an
exception message. But the token is sent as the x-vault-token header, so it sits in the
Req.Request and Finch.Request structs for the life of the call. Req's Inspect
redacts only authorization, and Finch's [:finch, :request, :start | :stop | :exception]
telemetry metadata carries request headers verbatim. If you attach handlers to those
events, filter x-vault-token out of the metadata yourself. Nothing inside AshVault
can do it for you.
What to back up, and what never to back up
Back up the key store. Separately. Deliberately. With its own retention policy.
Never let the key store and the database land in the same backup, tarball, snapshot or
volume clone. If one restore brings back both, crypto-erasure is defeated and the
library's central promise is void. For Local this means the key root is excluded from the
database backup job and is not on a volume you snapshot with the database. For OpenBao it
is automatic — the keys were never in PostgreSQL.
This is a real, sharp tradeoff with no free answer:
- Keeping key backups makes erasure harder. Every retained copy of the key store can resurrect a destroyed subject, so your key-backup retention window is the real lifetime of a "destroyed" key.
- Keeping no key backups makes data loss easy. Losing the key store destroys every encrypted value in the database, permanently, with no recovery path.
Decide which risk you are underwriting, write it down, and test the restore. Then test the
other restore: database back, keys not — and confirm a destroyed scope stays destroyed.
That is what test/acceptance/backup_restore_test.exs does, and it is the check that
proves the deployment, not just the library.
Also worth knowing: Local fsyncs every key file and every meta.json before renaming it
into place, but OTP offers no way to fsync a directory (:file.open/2 on one returns
{:error, :eisdir}), so directory-entry durability is left to the filesystem's own
ordering. And overwrite-before-unlink guarantees nothing on CoW or log-structured media —
see Crypto-erasure.
The mix tasks
All of them require the app runtime and a reachable key provider, check the provider before
doing anything, exit non-zero on failure, and print one key=value line per unit of
progress with a human summary last.
Shared flags: --tenant TENANT, --all-tenants (a string naming a zero-arity function,
such as MyApp.Accounts.list_tenant_ids plus /0, that returns a list), --domain MODULE (inferred from :ash_domains when omitted), --yes. Pass
--tenant or --all-tenants, never both; pass neither for a scope :global resource.
| Task | Purpose | Destructive |
|---|---|---|
mix ash_vault.local.init | create and initialise a Local key root | no |
mix ash_vault.key_info | show key name, version, timestamps, tombstone state | no |
mix ash_vault.rotate | mint a new key version | no |
mix ash_vault.backfill | encrypt an existing plaintext column, online | no |
mix ash_vault.verify | decrypt a sample and compare to the plaintext column | no |
mix ash_vault.destroy_keys | cryptographic erasure | YES — irreversible |
mix ash_vault.key_info
mix ash_vault.key_info MyApp.Accounts.Organization --tenant acmeash_vault.key_info=scope resource=MyApp.Accounts.Organization tenant=acme scope=acme \
vault=MyApp.Vault provider=AshVault.KeyProviders.OpenBao key_name=ashvault_YWNtZQ \
status=active version=3 created_at=2026-02-01T10:12:00Z versions=1,2,3
MyApp.Vault: scope acme is active at key version 3 (3 version(s) retained).status is active, not_minted or destroyed. It is read-only in the strongest sense:
it never mints key material. current_key/1 mints version 1 on first use, so an unused
scope is probed with get_key/2 first and reported not_minted without creating anything.
A destroyed scope exits 0 — a tombstone is a legitimate answer, not a failure.
mix ash_vault.rotate
See Rotation. Not destructive; old ciphertext keeps decrypting.
mix ash_vault.destroy_keys
See Crypto-erasure. Requires typing the scope back; --yes does not
skip that and there is no --force.
mix ash_vault.backfill and mix ash_vault.verify
The error taxonomy
Twelve errors. Every one of them is a Splode error with class: :invalid, so Ash wraps
them correctly and they arrive from Ash.read/2 and Ash.create/2 as ordinary Ash errors
rather than 500s. The whole point of having twelve rather than one is that the response
differs.
| Error | What it means | First response |
|---|---|---|
KeyDestroyed | the key was crypto-erased; the data is gone by design | close the ticket |
KeyNotFound | no key and no tombstone for this scope/version | investigate the key store |
ProviderUnavailable | the key backend could not be reached | retry, then page |
ProviderForbidden | the key backend answered 403: policy, token or address | fix the configuration; retrying cannot help |
CiphertextIntegrityFailed | ciphertext integrity, not access control. The stored bytes failed their AEAD tag check: modified, or read under a different key/tenant/resource/field. Nothing to do with actors, policies or AshAuthentication — AshVault authorizes nothing | treat as tampering unless a deploy explains it |
KeySizeMismatch | the provider's key size disagrees with the cipher | fix the config; retrying cannot help |
MissingScope | no tenant reached a tenant-scoped field | pass a tenant |
InvalidScope | a custom scope module returned a non-binary | fix the scope module |
InvalidCiphertext | the column is not an AshVault envelope | check the column and the backfill |
UnsupportedEnvelope | the envelope version is newer than this build | deploy a newer AshVault |
UnsupportedCipher | the envelope names an unregistered cipher | register it |
SerializationFailed | the value does not fit the attribute's type | fix the value or the type |
AshVault.Errors.KeyDestroyed — the data is gone, by design
Encryption key for scope "acme" (version 1) has been destroyed.
This data was cryptographically erased and cannot be recovered.Meaning: someone ran a crypto-erasure for this scope. The provider's tombstone is doing its job.
Do: close the ticket. Confirm against your deletion runbook that this scope was meant to be erased. If it was not, you have an incident — but not a recoverable one; the data is unreadable and no restore of the database will change that.
Do not interpret this as corruption or as an attack. Destruction is checked before any decryption is attempted, so this never means "the bytes are wrong".
AshVault.Errors.KeyNotFound — investigate
No encryption key found for scope "acme" (version 2).Meaning: the provider has no such key and no tombstone. This is not erasure and not an outage.
Do: check you are asking with the right scope key — a changed tenant identifier, a
custom to_scope_key/2, a --tenant typo. Then check the store itself: a deleted key
file, a meta.json referencing a version whose key material is gone, a restored key store
older than the rows. mix ash_vault.key_info tells you what the provider thinks exists.
AshVault.Errors.ProviderUnavailable — retry, then page
Key provider AshVault.KeyProviders.OpenBao is unavailable: %Req.TransportError{reason: :econnrefused}.Meaning: transport or backend failure. This is the only retryable error in the list.
Do: retry. If it persists: check the key store is up and reachable, the token is valid
and unexpired, the KV mount exists for
OpenBao, the volume is mounted for Local. It also covers corrupt or unreadable state —
a truncated meta.json, an unreadable tombstone — which is deliberate: a tombstone read
that cannot complete is never answered with "not destroyed".
Do not conclude anything about erasure from this. An outage must never look like erasure, and erasure must never look like an outage.
AshVault.Errors.ProviderForbidden — fix the configuration, do not retry
Key provider AshVault.KeyProviders.OpenBaoTransit was refused by OpenBao (403) on encrypt.Meaning: the key store is up and refused the request. :operation names which one
(:encrypt, :hmac, :create_key, :read_key, :read_tombstone, ...). Retrying the
same request with the same token gets the same answer.
Do: compare the token's policy with the paths the provider documents, and check the
token has not been revoked. A 403 on :encrypt against an HA OpenBao usually
means :address reaches standby nodes: they serve reads and policy checks from their own,
briefly lagging, copy of the data, so an encrypt issued right after AshVault creates a
scope's key can find no key there, and OpenBao treats encrypting with a missing key as
creating it. Point :address at the active node only (the Helm chart's <release>-active
Service), and leave reuse_connections at its default false: a pooled connection
outlives a leader change and keeps talking to the demoted node.
OpenBaoTransit.node_status/0 returning {:ok, :standby} confirms the misrouting.
A 403 on a tombstone read still fails closed: the scope is never reported intact.
AshVault.Errors.CiphertextIntegrityFailed — the bytes do not verify
Ciphertext for MyApp.Accounts.User.email failed its integrity check (key version 1).
The stored bytes were modified, were encrypted for a different tenant, resource or
field, or were encrypted with a different key.
This is not an authorization error. AshVault performs no authorization; if this
operation reached the crypto layer, Ash had already authorized it.Meaning: the AEAD tag did not verify. This is ciphertext integrity, not access
control: the "authentication" in AEAD is the bytes authenticating themselves, and the
error is unrelated to AshAuthentication, Ash.Policy.Authorizer or the actor. Three
realistic causes, in order of likelihood:
- Different associated data. The ciphertext is being read under a different scope, resource or field than it was written under. A relocated blob (which is the attack this catches), a renamed resource module, or a changed scope key.
- The wrong key. Usually a key store restored from a different point in time, or two environments pointed at each other.
- Actual tampering. Someone with database write access modified the column.
Do: check for a recent module rename or scope-key change first — those are the boring answers. Then check which key store the app is pointed at. Only then treat it as tampering — but if the value was not touched by a deploy, treat it as tampering, because that is exactly what this error is for.
AshVault.Errors.KeySizeMismatch — fix the config; retrying cannot help
Key size mismatch: AshVault.Ciphers.AES.GCM requires 32-byte keys, but
AshVault.KeyProviders.Local supplied 16 bytes.
This is a configuration fault, not tampering and not an outage. Retrying will not
help. Check the provider's `:key_bytes` (or OpenBao's `:key_type`) against the
vault's `:cipher`.Meaning: a key_bytes: disagreeing with the cipher, an OpenBao key_type: aes128-gcm96 under a 256-bit cipher, or a truncated key file.
Do: fix the configuration. It is deliberately neither CiphertextIntegrityFailed (which
would tell you your data was tampered with, for a typo) nor ProviderUnavailable (which
would have you retry a permanent misconfiguration forever).
AshVault.Vault.verify_key_sizes!/2 catches the statically-determinable cases at compile
time, with:
MyApp.Vault is misconfigured: its key provider and cipher disagree on key size.
MyApp.KeyProvider.key_bytes() == 16
AshVault.Ciphers.AES.GCM.key_bytes() == 32
Every encryption would fail with AshVault.Errors.KeySizeMismatch.It cannot catch a provider whose size depends on runtime config, which is why the runtime error exists.
AshVault.Errors.MissingScope — pass a tenant
Cannot encrypt MyApp.Accounts.User.email because no Ash tenant was present.
This resource uses tenant-scoped encryption.
Pass a tenant when executing the Ash action or configure another AshVault scope.Meaning: the scope could not be resolved. Almost always a background job, migration or mix task running without a tenant.
Do: pass tenant: to the Ash call, or --tenant / --all-tenants to the task. The
message's verb tracks the operation, so a read says "Cannot decrypt". A different
message, naming :unsupported_tenant_shape, means a tenant was passed in a shape the
scope module cannot reduce to a stable key — see
Tenant-scoped encryption.
AshVault.Errors.InvalidScope — a custom scope module is wrong
MyApp.Scopes.Custom.resolve!/1 returned a 2-tuple, which is not a binary.
AshVault scope keys must be binaries: they name key material in the provider and are
bound into every ciphertext's associated data, so they have to be stable across
processes, releases and OTP upgrades.Do: fix the scope module. Only reachable with a custom AshVault.Scope.
The returned term is described, not printed (AshVault.Scope.describe/1): a scope
term is usually a tenant, and a loaded tenant record is full of customer PII that has no
business in an error this one ends up in a log and an APM trace. The shape is the whole
diagnosis; read the offending value in your own scope module, under a debugger, against
data you are allowed to see.
AshVault.Errors.InvalidCiphertext — the column is not an AshVault envelope
Value is not a valid AshVault envelope: :bad_magic.:reason is :empty, :bad_magic, :not_a_binary, or a decode-specific term.
Meaning: the stored bytes are not an envelope at all. Usually a column that was never
backfilled, a column written by something other than AshVault, or a value mangled in
transit (a bytea that went through a text round trip).
Do: look at the raw bytes. A real envelope starts with "AV" followed by a version
byte.
AshVault.Errors.UnsupportedEnvelope — this build is too old
Unsupported AshVault envelope version: 2.Meaning: the magic was right but the version byte is one this build cannot parse.
Deploying an older release over a newer one that wrote a new envelope version. The same
error, carrying layer: :plaintext, means the plaintext format version ("AVP") was
unrecognised.
Do: deploy the build that can read it.
AshVault.Errors.UnsupportedCipher — register it
Unsupported cipher: "chacha20_poly1305_v1".
Register it under `config :ash_vault, :ciphers, %{...}` if this build should support it.Do: add it back to the registry. Removing a cipher from the registry while rows still name it is how this happens. See Writing a cipher.
AshVault.Errors.SerializationFailed — the value does not fit the type
Could not serialize MyApp.Accounts.User.email (type MyApp.Types.Email) for encryption.
{:dump_to_embedded, {:error, [message: {:redacted, :binary}, value: {:redacted, :binary}]}}
The reason above is redacted: `{:redacted, _}` stands where a type-controlled value
was dropped, because that value is the plaintext this error exists to protect.Meaning: Ash.Type.dump_to_embedded/3 (or cast_from_embedded/3 on the way back)
rejected the value. Not a crypto problem at all.
Why it is redacted: the reason is the type's own return value, and the Ash.Type
contract lets a type return {:error, message: ..., value: value} — where value is the
plaintext AshVault was about to encrypt. A custom type that does so would put plaintext
into an Ash.Error that gets logged, rendered and shipped to APM, so
AshVault.Errors.SerializationFailed rewrites :reason at construction: the shape and
the keys survive, and every leaf that is not an atom or an integer becomes
{:redacted, kind}. Built-in Ash types return :error or {:error, index: 0} and are
therefore unaffected — what you see above only happens with a custom type.
Do: fix the value or the type. The error still names the resource, the field and the
type module, which is where the real message lives — run that type's
dump_to_embedded/2 against a value you are allowed to look at. On the read side this
can also mean the stored term no longer matches the current type; a type change on an
existing encrypted column is a data migration.
Telemetry and the compliance audit log
AshVault emits :telemetry events around every encrypt, every decrypt, and every key
rotation and erasure. There is no on_decrypt callback: an audit log, a metric, a trace
span and a debug log all want the same fact, and a callback can only serve one of them.
AshVault.Telemetry is the reference for the event names, the measurements and every
metadata key. The short version:
| Prefix | Emitted around | Notable metadata |
|---|---|---|
[:ash_vault, :encrypt] | one field of one record, on write | :resource, :field |
[:ash_vault, :decrypt] | one field of one record, on read | :resource, :field, :vault |
[:ash_vault, :key, :rotate] | a scope's key is rotated | :scope, :scope_fingerprint, :key_version |
[:ash_vault, :key, :destroy] | a scope is crypto-erased | :scope, :scope_fingerprint |
Each prefix is a :telemetry.span/3, so it has :start, :stop and :exception under
it. Every event also carries :actor_type and :actor_id, and every :stop carries
:result (:ok or :error) plus, on failure, :error — the error module.
Three things to internalise before you write a handler:
- Watch
:stop, not:exception.AshVault.decrypt_value/5rescues and returns its errors, so a read of a crypto-erased tenant is a:stopwithresult: :error, error: AshVault.Errors.KeyDestroyed. A handler attached only to:exceptionsees none of it. [:ash_vault, :key, :destroy]is the compliance-critical one. It is the record that an erasure request was actually executed, and the only one whose absence is a finding.- Nothing in the metadata is a secret, and it is your job to keep it that way. There is
no plaintext, no ciphertext, no key material and no actor struct in these events — only
an
:actor_id. Do not add any in your handler. (The single exception is the:exceptionevent, where:telemetry.span/3's own contract puts the raised error struct in:reason; seeAshVault.Telemetry.)
A worked handler
Attach once, at application start, after the repo is up:
defmodule MyApp.VaultAudit do
@moduledoc "Writes an immutable audit row for every AshVault key-lifecycle and decrypt event."
require Logger
@events [
[:ash_vault, :decrypt, :stop],
[:ash_vault, :key, :rotate, :stop],
[:ash_vault, :key, :destroy, :stop],
[:ash_vault, :key, :destroy, :exception]
]
def attach do
:telemetry.attach_many("my-app-vault-audit", @events, &__MODULE__.handle/4, nil)
end
# Telemetry handlers run in the caller's process. Keep this cheap and total: a raise
# here detaches the handler for the life of the node, and silently stops your audit log.
def handle(event, measurements, metadata, _config) do
entry = %{
# UTC, always. An audit row in server-local time is not an audit row.
occurred_at: DateTime.utc_now(),
event: Enum.join(event, "."),
# Who.
actor_type: inspect(metadata[:actor_type]),
actor_id: to_string(metadata[:actor_id] || "system"),
# What.
resource: inspect(metadata[:resource]),
field: to_string(metadata[:field] || ""),
scope: metadata[:scope],
key_version: metadata[:key_version],
# Outcome.
result: metadata[:result] || :exception,
error: inspect(metadata[:error]),
duration_us: System.convert_time_unit(measurements[:duration] || 0, :native, :microsecond)
}
MyApp.Repo.insert_all("vault_audit_log", [entry])
rescue
error ->
# Never let the audit log take down the operation it is auditing — but never let it
# fail silently either.
Logger.error("vault audit write failed: #{Exception.message(error)}")
end
endRequest-scoped facts an audit regime often also asks for — the client IP, a request id, a stated purpose for the access — are not in the metadata, because AshVault is not in the request and cannot invent them. Telemetry handlers run in the calling process, so read them from where your web layer already put them:
metadata = Logger.metadata()
entry = Map.merge(entry, %{
request_id: metadata[:request_id],
remote_ip: metadata[:remote_ip]
})(and set them there once, in a plug, rather than threading them through Ash).
The key-lifecycle events carry the raw tenant id
metadata[:scope] is the scope key — under the default AshVault.Scopes.AshTenant that
is the tenant id. It is not secret, but it is frequently a customer identifier, so the
audit table inherits whatever access controls your other customer data has.
A handler that forwards this metadata onward is forwarding tenant identifiers
Logger output from AshVault reports a scope as AshVault.Scope.fingerprint/1, a
truncated SHA-256, precisely because a tenant id is frequently PII. The
[:ash_vault, :key, :rotate] and [:ash_vault, :key, :destroy] telemetry events
deliberately do not: they carry the raw :scope, because the compliance record of
an erasure has to be able to name the tenant it erased, and a fingerprint cannot.
Telemetry metadata is copied verbatim into third-party APMs by most handlers anybody
actually writes. If you forward these events to an external service — an APM, a
log shipper, an error reporter — you are sending it your tenant ids. Both events
also carry :scope_fingerprint so that a handler which must report outward has
something safe to report, but it is a convenience, not a mitigation: the raw :scope
is still in the same map, and a handler that copies the metadata wholesale sends it.
Either drop :scope explicitly in the handler and forward :scope_fingerprint, or
treat the destination as a place that holds customer data. An internal audit table,
which is what the worked handler above writes to, is the case where the raw scope is
the point.
Two notes on what you will not find in these events, both deliberate and both explained
in AshVault.Telemetry: the encrypt and decrypt events carry no :scope (resolving it
would run your AshVault.Scope module a second time per operation, and it is allowed to
have side effects and to raise), and no :key_version (it is inside the envelope). If your
audit requirement needs the tenant on every field read, correlate on Ash's own
[:ash, :*] spans, which carry it, or use a scope module you know is pure and read the
tenant from your request context.
Volume
[:ash_vault, :decrypt] fires once per encrypted field per record. A list endpoint
returning 100 users with 3 encrypted fields emits 300 spans. Auditing every decrypt is what
some regimes ask for, and it is also how you fill a disk. Decide explicitly: many
deployments log the key-lifecycle events in full and sample or aggregate the decrypts.
Retention
Retention is a decision you make with your own counsel; AshVault has no opinion and this is not legal advice. The figures ash_cloak's security considerations cite, reproduced here as the starting point for that conversation rather than as a requirement from us:
| Regime | Figure ash_cloak cites |
|---|---|
| HIPAA | 6 years |
| PCI-DSS | 1 year, with 3 months immediately available |
| GDPR | the duration of processing |
Whatever you choose, note the interaction with crypto-erasure that is specific to this library: the audit log survives the erasure it records. That is the point of it — you must be able to demonstrate that a deletion happened — but it means the audit table is one more place a destroyed subject's identifiers live on. Keep identifiers in it, not data, and make sure its retention window is one you can defend to the same subject.
Monitoring
At minimum, alert differently on these three, because the correct human response differs:
ProviderUnavailable→ page. Encryption and decryption are both down.ProviderForbidden→ page the owner of the key store's policy. Nothing is down; the application is not allowed to do what it is asking.CiphertextIntegrityFailed→ investigate. Either a deploy changed a module name or a scope key, or someone is writing to your database.KeyDestroyed→ do not page. Expected after an erasure; a spike of it for a scope not in your deletion runbook is worth a look.
KeySizeMismatch and InvalidScope should fail your deploy, not your pager — they are
configuration faults that will affect every operation uniformly.
Crash dumps and core dumps
A BEAM crash dump is a complete picture of process memory at the moment of the crash. If a
node dies while any key material is resident — which is every moment between fetching a key
and finishing the operation, and the whole cache lifetime if caching is on — that material can
land in erl_crash.dump as an ordinary file on disk, outside every protection this library
provides. The same is true of an OS core dump.
This is not hypothetical. Developing AshVault produced two crash dumps, 11 MB and 12 MB, from
crashed test runs — one in the project root and one under example/ — and one of them
contained references to the key provider modules. Twice, in a project whose entire subject is
key hygiene.
In production:
# Do not write BEAM crash dumps at all.
export ERL_CRASH_DUMP_SECONDS=0
# Do not write OS core dumps for the beam process.
ulimit -c 0
Under systemd, also set LimitCORE=0 on the unit, and check that no container runtime or
supervisor has re-enabled either.
AshVault warns about this at boot
Because the unsafe state is the default — with ERL_CRASH_DUMP_SECONDS unset, OTP writes
the dump — AshVault logs a single :warning at application start naming the hazard and the
fix.
It is deliberately narrow, because a library that lectures about host configuration earns nothing but muted logs:
- Only when the variable is unset. Setting it to anything,
0or30alike, is a decision, and a decision is respected in silence. The warning is about the absence of one. - Only in
:prod. A development machine legitimately wants crash dumps, and a warning on every test run trains everyone to ignore it. The check is compiled out entirely in other environments — it is not a runtime branch. - Once, at boot, never per operation.
To silence it without changing the environment:
config :ash_vault, warn_on_crash_dumps?: falseNote what the warning does not cover: it cannot read ulimit -c, so an OS core dump is
still your responsibility, and a silenced warning is not a statement that this node is safe.
ERL_CRASH_DUMP_SECONDS=0 disables one of the two hazards on this page.
If you must keep crash dumps for debugging, treat them with exactly the care you give the key
store: restricted directory, short retention, never shipped to a log aggregator or an APM, and
never committed. Add erl_crash.dump to .gitignore in any application that depends on
AshVault.
The same reasoning applies to swap. Key material can be paged to disk and survive there long
after the process ends. The Rust key cache mlocks its buffers to prevent this; the
pure-Elixir cache and the BEAM heap generally cannot. Encrypted swap is the answer at the
operating-system level.
Related
- Crypto-erasure — the operational checklist
- Rotation
- Migrating from plaintext
documentation/adr/0002-distinguishable-crypto-errors.md— why the taxonomy above is granular rather than a single generic "decryption failed", and what a public API owes its users as a resultAshVault.Errors,AshVault.Telemetry,AshVault.KeyProviders.Local,AshVault.KeyProviders.OpenBao