Everything an operator needs: which provider to run, how to set it up, what to back up, what never to back up, the mix tasks, and how to read every error AshVault can produce.

Choosing a provider

ProviderUse it forKey storeSurvives restartMulti-node
AshVault.KeyProviders.Memorytests, a first lookprocess memorynono
AshVault.KeyProviders.Localsingle-node deployments, homelab, dev with persistencefiles on diskyesno
AshVault.KeyProviders.OpenBaoproduction, multi-nodeOpenBao/Vault transityesyes

The requirement that actually decides this is: the key store must be a different system from the database. Not a different schema, not another table — a different system, with its own credentials, that a database restore cannot reach. Local satisfies that with a directory on another volume; Memory cannot (it forgets everything on restart, so in production it is not a key store, it is a countdown to total data loss).

Memory

# config/dev.exs and config/test.exs
config :my_app, start_memory_key_provider?: true

# lib/my_app/application.ex — a runtime flag, not Mix.env/0, because Mix is not
# available in a release.
children =
  [MyApp.Repo] ++
    if Application.get_env(:my_app, :start_memory_key_provider?, false) do
      [AshVault.KeyProviders.Memory]
    else
      []
    end

AshVault deliberately does not start it for you. In tests, start a uniquely-named instance per test where you can:

start_supervised!({AshVault.KeyProviders.Memory, name: Memory})

(the AshVault.KeyProvider callbacks take no server name, so a vault always talks to the default-named instance — hence name: Memory in tests that go through a vault).

Local

One-time operator setup, with the key volume mounted:

mix ash_vault.local.init /var/lib/my_app/ash_vault_keys
Initialised AshVault key root: /var/lib/my_app/ash_vault_keys
  mode:     0700
  sentinel: .ash_vault_root

Back this directory up separately from the database, and exclude it from the
database backup job. See the AshVault.KeyProviders.Local moduledoc.

Then:

# config/runtime.exs
config :my_app, AshVault.KeyProviders.Local,
  root: "/var/lib/my_app/ash_vault_keys",
  key_bytes: 32

# lib/my_app/application.ex
children = [MyApp.Repo] ++ MyApp.Vault.child_specs()

# a deploy step — `init_root!/1` on the configured root, idempotent
:ok = MyApp.Vault.setup()

The provider never creates its own key root, and refuses to start without the .ash_vault_root sentinel:

AshVault.KeyProviders.Local key root /var/lib/my_app/ash_vault_keys exists but holds no
.ash_vault_root sentinel, so it is either uninitialised or not the volume you think it is.

AshVault.KeyProviders.Local never creates its own key root. If it did, a key volume that
failed to mount would be silently replaced by an empty directory on the underlying
filesystem: no tombstones, no key material, and a fresh version 1 minted for every
tenant that was ever crypto-erased.

Initialise the root once, explicitly, as the operator:

    mix ash_vault.local.init /var/lib/my_app/ash_vault_keys

Do not run ash_vault.local.init to make a startup error go away

If the provider refuses to start on a root you know was initialised, the sentinel is missing because the volume is missing. Running the task then creates a second, empty key store over the top of the mount point and resurrects every destroyed tenant. Check the mount first. That refusal is the safety mechanism, not an inconvenience.

mix ash_vault.local.init --force overrides the "non-empty directory with no sentinel" refusal. You want that approximately never.

On-disk layout, so you can find things:

<root>/
  .ash_vault_root         # sentinel
  <scope_dir>/            # Base.url_encode64(scope, padding: false)
    meta.json             # {"current": 2, "versions": {"1": "...", "2": "..."}}
    v1.key                # raw key bytes, mode 0600
    v2.key
  <scope_dir>.tombstone   # presence == destroyed; sits BESIDE the directory

AshVault.KeyProviders.Local.scope_dir/1 maps a tenant to its directory:

iex> AshVault.KeyProviders.Local.scope_dir("acme")
"YWNtZQ"

The provider logs loudly (and starts anyway) if the root is group- or world-accessible:

AshVault.KeyProviders.Local key root /var/lib/my_app/keys has mode 0755, which is
group- or world-accessible.

Key material is readable by other users on this machine. Run:

    chmod 0700 /var/lib/my_app/keys

Starting anyway.

Local is single-node. Two nodes sharing one NFS or SMB mount will race on meta.json and can mint conflicting versions. Use OpenBao.

OpenBao

# config/runtime.exs
config :my_app, AshVault.KeyProviders.OpenBao,
  address: System.fetch_env!("BAO_ADDR"),
  token: {:system, "BAO_TOKEN"},
  transit_mount: "transit",
  kv_mount: "ashvault",
  key_type: "aes256-gcm96",
  receive_timeout: 5_000,
  max_retries: 2

:token accepts a literal binary, {:system, "VAR"}, or a zero-arity function. The provider is stateless — there is nothing to add to your supervision tree.

In Kubernetes, use auth: {:kubernetes, role: "..."} in place of :token, and cacertfile: for a server certificate signed by a private CA. See Running against OpenBao in Kubernetes.

One-time operator setup, which mounts the KV-v2 engine that holds tombstones:

:ok = MyApp.Vault.setup()     # or AshVault.KeyProviders.OpenBao.setup()

A mix task does not start your application

OpenBao speaks HTTP through req, whose Finch pool is started by the :req application. A mix task with @requirements ["app.config"] configures the application without starting it, and a release command may not have started it either.

Call Application.ensure_all_started(:req) first. If you forget, AshVault says so precisely — %AshVault.Errors.ProviderUnavailable{reason: {:not_started, :req}}, whose message states that this is not an outage, that the server was never contacted, and what to add where. It is deliberately not reported as a transport failure, which is what an unreachable server looks like.

or equivalently, outside the app:

bao secrets enable -path=ashvault -version=2 kv
bao secrets enable transit

The KV mount is never created on a read path

A missing KV mount answers 404 "no handler for route ..." — status-identical to "no tombstone here". Mounting it and retrying would find a freshly created, empty store and report every destroyed scope as intact. The provider therefore reports a missing mount as ProviderUnavailable and stops. Mounting belongs in setup/0, run once, by you.

The app's token needs transit read/create/rotate/delete and the transit/export/encryption-key/* capability, plus read/write on ashvault/data/tombstones/*. Nothing else in your infrastructure should hold the export capability — it is the ability to read raw key material.

Find a tenant's transit key name:

iex> AshVault.KeyProviders.OpenBao.key_name("acme")
"ashvault_YWNtZQ"

Semantics worth knowing:

  • destroy/1 returns :ok only when the transit key is confirmed absent by a fresh read and the tombstone write succeeded. It never infers "there was nothing to delete" from an error message.
  • rotate/1 on a scope with no key mints version 1 and returns {:ok, 1}.
  • A wrong or expired token yields 403 permission denied on every endpoint, which is always AshVault.Errors.ProviderForbidden — never :destroyed.

The token can reach your APM through Finch telemetry

AshVault never logs the token, never puts it in an error struct, and never puts it in an exception message. But the token is sent as the x-vault-token header, so it sits in the Req.Request and Finch.Request structs for the life of the call. Req's Inspect redacts only authorization, and Finch's [:finch, :request, :start | :stop | :exception] telemetry metadata carries request headers verbatim. If you attach handlers to those events, filter x-vault-token out of the metadata yourself. Nothing inside AshVault can do it for you.

What to back up, and what never to back up

Back up the key store. Separately. Deliberately. With its own retention policy.

Never let the key store and the database land in the same backup, tarball, snapshot or volume clone. If one restore brings back both, crypto-erasure is defeated and the library's central promise is void. For Local this means the key root is excluded from the database backup job and is not on a volume you snapshot with the database. For OpenBao it is automatic — the keys were never in PostgreSQL.

This is a real, sharp tradeoff with no free answer:

  • Keeping key backups makes erasure harder. Every retained copy of the key store can resurrect a destroyed subject, so your key-backup retention window is the real lifetime of a "destroyed" key.
  • Keeping no key backups makes data loss easy. Losing the key store destroys every encrypted value in the database, permanently, with no recovery path.

Decide which risk you are underwriting, write it down, and test the restore. Then test the other restore: database back, keys not — and confirm a destroyed scope stays destroyed. That is what test/acceptance/backup_restore_test.exs does, and it is the check that proves the deployment, not just the library.

Also worth knowing: Local fsyncs every key file and every meta.json before renaming it into place, but OTP offers no way to fsync a directory (:file.open/2 on one returns {:error, :eisdir}), so directory-entry durability is left to the filesystem's own ordering. And overwrite-before-unlink guarantees nothing on CoW or log-structured media — see Crypto-erasure.

The mix tasks

All of them require the app runtime and a reachable key provider, check the provider before doing anything, exit non-zero on failure, and print one key=value line per unit of progress with a human summary last.

Shared flags: --tenant TENANT, --all-tenants (a string naming a zero-arity function, such as MyApp.Accounts.list_tenant_ids plus /0, that returns a list), --domain MODULE (inferred from :ash_domains when omitted), --yes. Pass --tenant or --all-tenants, never both; pass neither for a scope :global resource.

TaskPurposeDestructive
mix ash_vault.local.initcreate and initialise a Local key rootno
mix ash_vault.key_infoshow key name, version, timestamps, tombstone stateno
mix ash_vault.rotatemint a new key versionno
mix ash_vault.backfillencrypt an existing plaintext column, onlineno
mix ash_vault.verifydecrypt a sample and compare to the plaintext columnno
mix ash_vault.destroy_keyscryptographic erasureYES — irreversible

mix ash_vault.key_info

mix ash_vault.key_info MyApp.Accounts.Organization --tenant acme
ash_vault.key_info=scope resource=MyApp.Accounts.Organization tenant=acme scope=acme \
  vault=MyApp.Vault provider=AshVault.KeyProviders.OpenBao key_name=ashvault_YWNtZQ \
  status=active version=3 created_at=2026-02-01T10:12:00Z versions=1,2,3
MyApp.Vault: scope acme is active at key version 3 (3 version(s) retained).

status is active, not_minted or destroyed. It is read-only in the strongest sense: it never mints key material. current_key/1 mints version 1 on first use, so an unused scope is probed with get_key/2 first and reported not_minted without creating anything. A destroyed scope exits 0 — a tombstone is a legitimate answer, not a failure.

mix ash_vault.rotate

See Rotation. Not destructive; old ciphertext keeps decrypting.

mix ash_vault.destroy_keys

See Crypto-erasure. Requires typing the scope back; --yes does not skip that and there is no --force.

mix ash_vault.backfill and mix ash_vault.verify

See Migrating from plaintext.

The error taxonomy

Twelve errors. Every one of them is a Splode error with class: :invalid, so Ash wraps them correctly and they arrive from Ash.read/2 and Ash.create/2 as ordinary Ash errors rather than 500s. The whole point of having twelve rather than one is that the response differs.

ErrorWhat it meansFirst response
KeyDestroyedthe key was crypto-erased; the data is gone by designclose the ticket
KeyNotFoundno key and no tombstone for this scope/versioninvestigate the key store
ProviderUnavailablethe key backend could not be reachedretry, then page
ProviderForbiddenthe key backend answered 403: policy, token or addressfix the configuration; retrying cannot help
CiphertextIntegrityFailedciphertext integrity, not access control. The stored bytes failed their AEAD tag check: modified, or read under a different key/tenant/resource/field. Nothing to do with actors, policies or AshAuthentication — AshVault authorizes nothingtreat as tampering unless a deploy explains it
KeySizeMismatchthe provider's key size disagrees with the cipherfix the config; retrying cannot help
MissingScopeno tenant reached a tenant-scoped fieldpass a tenant
InvalidScopea custom scope module returned a non-binaryfix the scope module
InvalidCiphertextthe column is not an AshVault envelopecheck the column and the backfill
UnsupportedEnvelopethe envelope version is newer than this builddeploy a newer AshVault
UnsupportedCipherthe envelope names an unregistered cipherregister it
SerializationFailedthe value does not fit the attribute's typefix the value or the type

AshVault.Errors.KeyDestroyed — the data is gone, by design

Encryption key for scope "acme" (version 1) has been destroyed.

This data was cryptographically erased and cannot be recovered.

Meaning: someone ran a crypto-erasure for this scope. The provider's tombstone is doing its job.

Do: close the ticket. Confirm against your deletion runbook that this scope was meant to be erased. If it was not, you have an incident — but not a recoverable one; the data is unreadable and no restore of the database will change that.

Do not interpret this as corruption or as an attack. Destruction is checked before any decryption is attempted, so this never means "the bytes are wrong".

AshVault.Errors.KeyNotFound — investigate

No encryption key found for scope "acme" (version 2).

Meaning: the provider has no such key and no tombstone. This is not erasure and not an outage.

Do: check you are asking with the right scope key — a changed tenant identifier, a custom to_scope_key/2, a --tenant typo. Then check the store itself: a deleted key file, a meta.json referencing a version whose key material is gone, a restored key store older than the rows. mix ash_vault.key_info tells you what the provider thinks exists.

AshVault.Errors.ProviderUnavailable — retry, then page

Key provider AshVault.KeyProviders.OpenBao is unavailable: %Req.TransportError{reason: :econnrefused}.

Meaning: transport or backend failure. This is the only retryable error in the list.

Do: retry. If it persists: check the key store is up and reachable, the token is valid and unexpired, the KV mount exists for OpenBao, the volume is mounted for Local. It also covers corrupt or unreadable state — a truncated meta.json, an unreadable tombstone — which is deliberate: a tombstone read that cannot complete is never answered with "not destroyed".

Do not conclude anything about erasure from this. An outage must never look like erasure, and erasure must never look like an outage.

AshVault.Errors.ProviderForbidden — fix the configuration, do not retry

Key provider AshVault.KeyProviders.OpenBaoTransit was refused by OpenBao (403) on encrypt.

Meaning: the key store is up and refused the request. :operation names which one (:encrypt, :hmac, :create_key, :read_key, :read_tombstone, ...). Retrying the same request with the same token gets the same answer.

Do: compare the token's policy with the paths the provider documents, and check the token has not been revoked. A 403 on :encrypt against an HA OpenBao usually means :address reaches standby nodes: they serve reads and policy checks from their own, briefly lagging, copy of the data, so an encrypt issued right after AshVault creates a scope's key can find no key there, and OpenBao treats encrypting with a missing key as creating it. Point :address at the active node only (the Helm chart's <release>-active Service), and leave reuse_connections at its default false: a pooled connection outlives a leader change and keeps talking to the demoted node. OpenBaoTransit.node_status/0 returning {:ok, :standby} confirms the misrouting.

A 403 on a tombstone read still fails closed: the scope is never reported intact.

AshVault.Errors.CiphertextIntegrityFailed — the bytes do not verify

Ciphertext for MyApp.Accounts.User.email failed its integrity check (key version 1).

The stored bytes were modified, were encrypted for a different tenant, resource or
field, or were encrypted with a different key.

This is not an authorization error. AshVault performs no authorization; if this
operation reached the crypto layer, Ash had already authorized it.

Meaning: the AEAD tag did not verify. This is ciphertext integrity, not access control: the "authentication" in AEAD is the bytes authenticating themselves, and the error is unrelated to AshAuthentication, Ash.Policy.Authorizer or the actor. Three realistic causes, in order of likelihood:

  1. Different associated data. The ciphertext is being read under a different scope, resource or field than it was written under. A relocated blob (which is the attack this catches), a renamed resource module, or a changed scope key.
  2. The wrong key. Usually a key store restored from a different point in time, or two environments pointed at each other.
  3. Actual tampering. Someone with database write access modified the column.

Do: check for a recent module rename or scope-key change first — those are the boring answers. Then check which key store the app is pointed at. Only then treat it as tampering — but if the value was not touched by a deploy, treat it as tampering, because that is exactly what this error is for.

AshVault.Errors.KeySizeMismatch — fix the config; retrying cannot help

Key size mismatch: AshVault.Ciphers.AES.GCM requires 32-byte keys, but
AshVault.KeyProviders.Local supplied 16 bytes.

This is a configuration fault, not tampering and not an outage. Retrying will not
help. Check the provider's `:key_bytes` (or OpenBao's `:key_type`) against the
vault's `:cipher`.

Meaning: a key_bytes: disagreeing with the cipher, an OpenBao key_type: aes128-gcm96 under a 256-bit cipher, or a truncated key file.

Do: fix the configuration. It is deliberately neither CiphertextIntegrityFailed (which would tell you your data was tampered with, for a typo) nor ProviderUnavailable (which would have you retry a permanent misconfiguration forever).

AshVault.Vault.verify_key_sizes!/2 catches the statically-determinable cases at compile time, with:

MyApp.Vault is misconfigured: its key provider and cipher disagree on key size.

  MyApp.KeyProvider.key_bytes() == 16
  AshVault.Ciphers.AES.GCM.key_bytes()   == 32

Every encryption would fail with AshVault.Errors.KeySizeMismatch.

It cannot catch a provider whose size depends on runtime config, which is why the runtime error exists.

AshVault.Errors.MissingScope — pass a tenant

Cannot encrypt MyApp.Accounts.User.email because no Ash tenant was present.

This resource uses tenant-scoped encryption.
Pass a tenant when executing the Ash action or configure another AshVault scope.

Meaning: the scope could not be resolved. Almost always a background job, migration or mix task running without a tenant.

Do: pass tenant: to the Ash call, or --tenant / --all-tenants to the task. The message's verb tracks the operation, so a read says "Cannot decrypt". A different message, naming :unsupported_tenant_shape, means a tenant was passed in a shape the scope module cannot reduce to a stable key — see Tenant-scoped encryption.

AshVault.Errors.InvalidScope — a custom scope module is wrong

MyApp.Scopes.Custom.resolve!/1 returned a 2-tuple, which is not a binary.

AshVault scope keys must be binaries: they name key material in the provider and are
bound into every ciphertext's associated data, so they have to be stable across
processes, releases and OTP upgrades.

Do: fix the scope module. Only reachable with a custom AshVault.Scope.

The returned term is described, not printed (AshVault.Scope.describe/1): a scope term is usually a tenant, and a loaded tenant record is full of customer PII that has no business in an error this one ends up in a log and an APM trace. The shape is the whole diagnosis; read the offending value in your own scope module, under a debugger, against data you are allowed to see.

AshVault.Errors.InvalidCiphertext — the column is not an AshVault envelope

Value is not a valid AshVault envelope: :bad_magic.

:reason is :empty, :bad_magic, :not_a_binary, or a decode-specific term.

Meaning: the stored bytes are not an envelope at all. Usually a column that was never backfilled, a column written by something other than AshVault, or a value mangled in transit (a bytea that went through a text round trip).

Do: look at the raw bytes. A real envelope starts with "AV" followed by a version byte.

AshVault.Errors.UnsupportedEnvelope — this build is too old

Unsupported AshVault envelope version: 2.

Meaning: the magic was right but the version byte is one this build cannot parse. Deploying an older release over a newer one that wrote a new envelope version. The same error, carrying layer: :plaintext, means the plaintext format version ("AVP") was unrecognised.

Do: deploy the build that can read it.

AshVault.Errors.UnsupportedCipher — register it

Unsupported cipher: "chacha20_poly1305_v1".

Register it under `config :ash_vault, :ciphers, %{...}` if this build should support it.

Do: add it back to the registry. Removing a cipher from the registry while rows still name it is how this happens. See Writing a cipher.

AshVault.Errors.SerializationFailed — the value does not fit the type

Could not serialize MyApp.Accounts.User.email (type MyApp.Types.Email) for encryption.

{:dump_to_embedded, {:error, [message: {:redacted, :binary}, value: {:redacted, :binary}]}}

The reason above is redacted: `{:redacted, _}` stands where a type-controlled value
was dropped, because that value is the plaintext this error exists to protect.

Meaning: Ash.Type.dump_to_embedded/3 (or cast_from_embedded/3 on the way back) rejected the value. Not a crypto problem at all.

Why it is redacted: the reason is the type's own return value, and the Ash.Type contract lets a type return {:error, message: ..., value: value} — where value is the plaintext AshVault was about to encrypt. A custom type that does so would put plaintext into an Ash.Error that gets logged, rendered and shipped to APM, so AshVault.Errors.SerializationFailed rewrites :reason at construction: the shape and the keys survive, and every leaf that is not an atom or an integer becomes {:redacted, kind}. Built-in Ash types return :error or {:error, index: 0} and are therefore unaffected — what you see above only happens with a custom type.

Do: fix the value or the type. The error still names the resource, the field and the type module, which is where the real message lives — run that type's dump_to_embedded/2 against a value you are allowed to look at. On the read side this can also mean the stored term no longer matches the current type; a type change on an existing encrypted column is a data migration.

Telemetry and the compliance audit log

AshVault emits :telemetry events around every encrypt, every decrypt, and every key rotation and erasure. There is no on_decrypt callback: an audit log, a metric, a trace span and a debug log all want the same fact, and a callback can only serve one of them.

AshVault.Telemetry is the reference for the event names, the measurements and every metadata key. The short version:

PrefixEmitted aroundNotable metadata
[:ash_vault, :encrypt]one field of one record, on write:resource, :field
[:ash_vault, :decrypt]one field of one record, on read:resource, :field, :vault
[:ash_vault, :key, :rotate]a scope's key is rotated:scope, :scope_fingerprint, :key_version
[:ash_vault, :key, :destroy]a scope is crypto-erased:scope, :scope_fingerprint

Each prefix is a :telemetry.span/3, so it has :start, :stop and :exception under it. Every event also carries :actor_type and :actor_id, and every :stop carries :result (:ok or :error) plus, on failure, :error — the error module.

Three things to internalise before you write a handler:

  • Watch :stop, not :exception. AshVault.decrypt_value/5 rescues and returns its errors, so a read of a crypto-erased tenant is a :stop with result: :error, error: AshVault.Errors.KeyDestroyed. A handler attached only to :exception sees none of it.
  • [:ash_vault, :key, :destroy] is the compliance-critical one. It is the record that an erasure request was actually executed, and the only one whose absence is a finding.
  • Nothing in the metadata is a secret, and it is your job to keep it that way. There is no plaintext, no ciphertext, no key material and no actor struct in these events — only an :actor_id. Do not add any in your handler. (The single exception is the :exception event, where :telemetry.span/3's own contract puts the raised error struct in :reason; see AshVault.Telemetry.)

A worked handler

Attach once, at application start, after the repo is up:

defmodule MyApp.VaultAudit do
  @moduledoc "Writes an immutable audit row for every AshVault key-lifecycle and decrypt event."

  require Logger

  @events [
    [:ash_vault, :decrypt, :stop],
    [:ash_vault, :key, :rotate, :stop],
    [:ash_vault, :key, :destroy, :stop],
    [:ash_vault, :key, :destroy, :exception]
  ]

  def attach do
    :telemetry.attach_many("my-app-vault-audit", @events, &__MODULE__.handle/4, nil)
  end

  # Telemetry handlers run in the caller's process. Keep this cheap and total: a raise
  # here detaches the handler for the life of the node, and silently stops your audit log.
  def handle(event, measurements, metadata, _config) do
    entry = %{
      # UTC, always. An audit row in server-local time is not an audit row.
      occurred_at: DateTime.utc_now(),
      event: Enum.join(event, "."),
      # Who.
      actor_type: inspect(metadata[:actor_type]),
      actor_id: to_string(metadata[:actor_id] || "system"),
      # What.
      resource: inspect(metadata[:resource]),
      field: to_string(metadata[:field] || ""),
      scope: metadata[:scope],
      key_version: metadata[:key_version],
      # Outcome.
      result: metadata[:result] || :exception,
      error: inspect(metadata[:error]),
      duration_us: System.convert_time_unit(measurements[:duration] || 0, :native, :microsecond)
    }

    MyApp.Repo.insert_all("vault_audit_log", [entry])
  rescue
    error ->
      # Never let the audit log take down the operation it is auditing — but never let it
      # fail silently either.
      Logger.error("vault audit write failed: #{Exception.message(error)}")
  end
end

Request-scoped facts an audit regime often also asks for — the client IP, a request id, a stated purpose for the access — are not in the metadata, because AshVault is not in the request and cannot invent them. Telemetry handlers run in the calling process, so read them from where your web layer already put them:

metadata = Logger.metadata()

entry = Map.merge(entry, %{
  request_id: metadata[:request_id],
  remote_ip: metadata[:remote_ip]
})

(and set them there once, in a plug, rather than threading them through Ash).

The key-lifecycle events carry the raw tenant id

metadata[:scope] is the scope key — under the default AshVault.Scopes.AshTenant that is the tenant id. It is not secret, but it is frequently a customer identifier, so the audit table inherits whatever access controls your other customer data has.

A handler that forwards this metadata onward is forwarding tenant identifiers

Logger output from AshVault reports a scope as AshVault.Scope.fingerprint/1, a truncated SHA-256, precisely because a tenant id is frequently PII. The [:ash_vault, :key, :rotate] and [:ash_vault, :key, :destroy] telemetry events deliberately do not: they carry the raw :scope, because the compliance record of an erasure has to be able to name the tenant it erased, and a fingerprint cannot.

Telemetry metadata is copied verbatim into third-party APMs by most handlers anybody actually writes. If you forward these events to an external service — an APM, a log shipper, an error reporter — you are sending it your tenant ids. Both events also carry :scope_fingerprint so that a handler which must report outward has something safe to report, but it is a convenience, not a mitigation: the raw :scope is still in the same map, and a handler that copies the metadata wholesale sends it.

Either drop :scope explicitly in the handler and forward :scope_fingerprint, or treat the destination as a place that holds customer data. An internal audit table, which is what the worked handler above writes to, is the case where the raw scope is the point.

Two notes on what you will not find in these events, both deliberate and both explained in AshVault.Telemetry: the encrypt and decrypt events carry no :scope (resolving it would run your AshVault.Scope module a second time per operation, and it is allowed to have side effects and to raise), and no :key_version (it is inside the envelope). If your audit requirement needs the tenant on every field read, correlate on Ash's own [:ash, :*] spans, which carry it, or use a scope module you know is pure and read the tenant from your request context.

Volume

[:ash_vault, :decrypt] fires once per encrypted field per record. A list endpoint returning 100 users with 3 encrypted fields emits 300 spans. Auditing every decrypt is what some regimes ask for, and it is also how you fill a disk. Decide explicitly: many deployments log the key-lifecycle events in full and sample or aggregate the decrypts.

Retention

Retention is a decision you make with your own counsel; AshVault has no opinion and this is not legal advice. The figures ash_cloak's security considerations cite, reproduced here as the starting point for that conversation rather than as a requirement from us:

RegimeFigure ash_cloak cites
HIPAA6 years
PCI-DSS1 year, with 3 months immediately available
GDPRthe duration of processing

Whatever you choose, note the interaction with crypto-erasure that is specific to this library: the audit log survives the erasure it records. That is the point of it — you must be able to demonstrate that a deletion happened — but it means the audit table is one more place a destroyed subject's identifiers live on. Keep identifiers in it, not data, and make sure its retention window is one you can defend to the same subject.

Monitoring

At minimum, alert differently on these three, because the correct human response differs:

  • ProviderUnavailable → page. Encryption and decryption are both down.
  • ProviderForbidden → page the owner of the key store's policy. Nothing is down; the application is not allowed to do what it is asking.
  • CiphertextIntegrityFailed → investigate. Either a deploy changed a module name or a scope key, or someone is writing to your database.
  • KeyDestroyed → do not page. Expected after an erasure; a spike of it for a scope not in your deletion runbook is worth a look.

KeySizeMismatch and InvalidScope should fail your deploy, not your pager — they are configuration faults that will affect every operation uniformly.

Crash dumps and core dumps

A BEAM crash dump is a complete picture of process memory at the moment of the crash. If a node dies while any key material is resident — which is every moment between fetching a key and finishing the operation, and the whole cache lifetime if caching is on — that material can land in erl_crash.dump as an ordinary file on disk, outside every protection this library provides. The same is true of an OS core dump.

This is not hypothetical. Developing AshVault produced two crash dumps, 11 MB and 12 MB, from crashed test runs — one in the project root and one under example/ — and one of them contained references to the key provider modules. Twice, in a project whose entire subject is key hygiene.

In production:

# Do not write BEAM crash dumps at all.
export ERL_CRASH_DUMP_SECONDS=0

# Do not write OS core dumps for the beam process.
ulimit -c 0

Under systemd, also set LimitCORE=0 on the unit, and check that no container runtime or supervisor has re-enabled either.

AshVault warns about this at boot

Because the unsafe state is the default — with ERL_CRASH_DUMP_SECONDS unset, OTP writes the dump — AshVault logs a single :warning at application start naming the hazard and the fix.

It is deliberately narrow, because a library that lectures about host configuration earns nothing but muted logs:

  • Only when the variable is unset. Setting it to anything, 0 or 30 alike, is a decision, and a decision is respected in silence. The warning is about the absence of one.
  • Only in :prod. A development machine legitimately wants crash dumps, and a warning on every test run trains everyone to ignore it. The check is compiled out entirely in other environments — it is not a runtime branch.
  • Once, at boot, never per operation.

To silence it without changing the environment:

config :ash_vault, warn_on_crash_dumps?: false

Note what the warning does not cover: it cannot read ulimit -c, so an OS core dump is still your responsibility, and a silenced warning is not a statement that this node is safe. ERL_CRASH_DUMP_SECONDS=0 disables one of the two hazards on this page.

If you must keep crash dumps for debugging, treat them with exactly the care you give the key store: restricted directory, short retention, never shipped to a log aggregator or an APM, and never committed. Add erl_crash.dump to .gitignore in any application that depends on AshVault.

The same reasoning applies to swap. Key material can be paged to disk and survive there long after the process ends. The Rust key cache mlocks its buffers to prevent this; the pure-Elixir cache and the BEAM heap generally cannot. Encrypted swap is the answer at the operating-system level.