Deterministic lookup tokens for searchable encrypted fields.
AES-GCM draws a fresh nonce for every write, so the same plaintext encrypts to
different bytes every time: WHERE encrypted_email = $1 matches nothing and a unique
index on the ciphertext column is meaningless. A searchable field therefore carries a
second, deterministic column beside the ciphertext:
plaintext
├── AES-GCM → encrypted_email (randomized, unsearchable)
└── HMAC → email_lookup (deterministic, indexable)The key is not the DEK, and is not derived from it
The token key comes from AshVault.KeyProvider.lookup_key/1 — a separate,
non-rotating, per-scope secret. AshVault.rotate_key!/2 must not change it.
If the token were derived from the rotating data encryption key, rotation would change
every future token while every stored token still reflected the old key. Lookups for
existing rows would silently stop matching, unique? would stop preventing duplicates
and users could not log in — with nothing raised anywhere. That failure passes any test
that writes a row and immediately reads it back, which is why
test/ash_vault/lookup_rotation_test.exs asserts the provider hands back the same
key binary across a rotation rather than inferring it from token equality.
Erasure still erases: AshVault.KeyProvider.destroy/1 destroys the lookup key along
with every data key, and the same tombstone gates it afterwards.
Per-field separation comes from HKDF
lookup_key_for_field =
HKDF-SHA256(ikm: provider_lookup_key(scope),
info: "ash_vault:lookup:v1|" <> inspect(resource) <> "|" <> field)
token = HMAC-SHA256(lookup_key_for_field, normalized_plaintext)Binding the resource and the field means the same plaintext in two columns, or in two resources, produces two unrelated tokens — so a token cannot be replayed across columns any more than a ciphertext can be replayed across its AAD. Binding the scope is what the per-scope provider key already does, and it is what keeps the equality leak inside one tenant.
HKDF is not in OTP, so hkdf/4 implements RFC 5869 extract-and-expand directly on
:crypto.mac/4. It is tested against the RFC's own vectors.
Rotating a lookup key is a backfill, not a rotation
There is deliberately no rotate_lookup_key anywhere. Changing the lookup key — like
changing normalize: — invalidates every stored token, so it is a data migration:
destroy nothing, re-derive, and rewrite the column with mix ash_vault.backfill --lookup.
Summary
Types
A normalization strategy for a searchable field.
Functions
Derive the 32-byte token key for one resource/field from a provider lookup key.
The derived per-field token key for an operation, or an error.
HKDF-SHA256 (RFC 5869): extract-then-expand, in one call.
HKDF-Expand (RFC 5869 §2.3).
HKDF-Extract (RFC 5869 §2.2): HMAC-SHA256(salt, ikm), returning a 32-byte PRK.
The HKDF info string binding a token key to one resource and field.
Apply a field's normalize: strategy to a plaintext value.
Normalize a value with the normalize: strategy configured for a field.
HMAC-SHA256(field_key, normalized_plaintext) — the stored token.
Compute the lookup token for a value, resolving the vault, scope and provider exactly as a write does.
Like token_for/4, but raises the error instead of returning it.
Like token_for/4, for a value that has already been normalized.
Types
Functions
Derive the 32-byte token key for one resource/field from a provider lookup key.
@spec field_key(module(), atom(), AshVault.Context.t()) :: {:ok, binary()} | {:error, Exception.t()}
The derived per-field token key for an operation, or an error.
Exposed so a test can assert the key is stable across a rotation, which is the one property the whole mechanism rests on.
@spec hkdf(binary(), binary(), binary(), pos_integer()) :: binary()
HKDF-SHA256 (RFC 5869): extract-then-expand, in one call.
salt may be "", which RFC 5869 §2.2 defines as a string of HashLen zeros.
RFC 5869 Appendix A.1 (SHA-256, with salt and info) round-trips through here; the
RFC's three SHA-256 vectors are asserted in test/ash_vault/lookup_test.exs.
@spec hkdf_expand(binary(), binary(), pos_integer()) :: binary()
HKDF-Expand (RFC 5869 §2.3).
Raises ArgumentError for a length above 255 * HashLen, which the RFC forbids.
HKDF-Extract (RFC 5869 §2.2): HMAC-SHA256(salt, ikm), returning a 32-byte PRK.
An empty salt is replaced by HashLen zero bytes, as the RFC specifies.
The HKDF info string binding a token key to one resource and field.
Frozen, human-inspectable, and built by stringification rather than
:erlang.term_to_binary/1 — for exactly the reason
AshVault.Vault.Runtime.build_aad/2 is: the external term format is not guaranteed
stable across OTP releases, and stored tokens have to outlive OTP upgrades.
iex> AshVault.Lookup.info(MyApp.User, :email)
"ash_vault:lookup:v1|MyApp.User|email"
Apply a field's normalize: strategy to a plaintext value.
Returns {:ok, normalized}, or {:error, reason} when the strategy produced
something that is not a binary — which is a bug in a custom normalizer, not a user
error, and must not be papered over with to_string/1.
nil passes through as {:ok, nil}: a nil plaintext produces no token.
@spec normalize_for(module() | Spark.Dsl.t(), atom(), term()) :: {:ok, binary() | nil} | {:error, Exception.t()}
Normalize a value with the normalize: strategy configured for a field.
HMAC-SHA256(field_key, normalized_plaintext) — the stored token.
@spec token_for(module(), atom(), term(), AshVault.Context.t()) :: {:ok, binary() | nil} | {:error, Exception.t()}
Compute the lookup token for a value, resolving the vault, scope and provider exactly as a write does.
Returns {:ok, nil} for a nil plaintext. Returns {:error, exception} for a
missing scope, a destroyed scope, an unreachable provider, a provider with no
AshVault.KeyProvider.lookup_key/1, or a normalizer that did not return a binary.
Never raises: like AshVault.decrypt_value/5, crypto failures are values here so the
write path can add them to a changeset and the read path to a query.
@spec token_for!(module(), atom(), term(), AshVault.Context.t()) :: binary() | nil
Like token_for/4, but raises the error instead of returning it.
This is the form AshVault.Query.filter_by/4 uses: a plain function has no changeset
or query to attach an error to, and a lookup that quietly returned no filter would
read as "no such user".
@spec token_for_normalized(module(), atom(), binary() | nil, AshVault.Context.t()) :: {:ok, binary() | nil} | {:error, Exception.t()}
Like token_for/4, for a value that has already been normalized.
The write path normalizes once and encrypts the result, so it must not normalize a
second time here: a custom normalize: MFA is not required to be idempotent, and the
token has to be the hash of exactly the bytes that were encrypted.