This guide configures the OpenBao providers for an application running as a pod. The pod logs in with its service account (no long-lived token in a Secret), and it trusts an OpenBao server certificate signed by a private CA.
Both settings live in the provider's configuration block. They apply to every request the
provider makes: AshVault.KeyProviders.OpenBao, AshVault.KeyProviders.OpenBaoTransit,
and AshVault.Ciphers.OpenBaoTransit and AshVault.Macs.OpenBaoTransit too, because
those two read the OpenBaoTransit block.
Configuration
# config/runtime.exs
config :my_app, AshVault.KeyProviders.OpenBaoTransit,
address: "https://openbao-active.openbao.svc:8200",
cacertfile: "/etc/openbao-ca/ca.crt",
auth: {:kubernetes, role: "ashvault-foundry"}Address the active node
With HA (server.ha.enabled), point :address at the chart's <release>-active Service,
not the <release> Service that spans every pod. OpenBao standbys answer reads and run
policy checks against their own copy of the data, which lags the active node slightly.
AshVault creates a scope's key explicitly and then encrypts with it straight away; a
standby that has not yet seen the new key treats that encrypt as a request to create the
key, and a policy that withholds create on transit/encrypt/* (as recommended) refuses
it with 403. Verified against openbao 2.6.3 in a three-node Raft cluster: through the
all-pods Service, encrypt immediately after create was refused every time on a standby,
and succeeded a few hundred milliseconds later; through the active Service it never failed.
A standby can equally serve a tombstone read from before a destroy/1 landed.
The -active Service only routes new connections. A pooled keep-alive connection opened
while a node was active stays on that node after a leader change demotes it, and Finch has
no maximum connection lifetime, so under steady traffic it never goes idle and never
reconnects. That is why reuse_connections defaults to false: every request carries
connection: close, opens a new connection, and lands on whichever node is active at that
moment. Leave it off against HA. Turning it on is safe only for a single, non-HA node or
servers run with disable_standby_reads = true, which makes standbys forward every
request to the active node. That server setting is the defence for every client, not
just AshVault.
AshVault.KeyProviders.OpenBaoTransit.node_status/0 reports the HA role
(:active, :standby, :sealed, :uninitialized) of the node a request lands on, from
the unauthenticated sys/health. Log it at boot and export it as a gauge: :standby means
:address does not reach the active node, and fresh-scope keys will fail.
The server certificate must name the address you configure (see below).
Use auth: or token:, never both. AshVault raises ArgumentError if both are set
rather than guess which one you meant. The static token: forms (a binary,
{:system, "VAR"} or a zero-arity function) work as before.
Kubernetes auth
auth: {:kubernetes, opts} takes:
| Option | Default | Meaning |
|---|---|---|
:role | required | The role under auth/<mount>/role/ to log in as |
:mount | "kubernetes" | The auth method's mount path |
:jwt_path | /var/run/secrets/kubernetes.io/serviceaccount/token | The projected service-account token |
:refresh_fraction | 2/3 | Log in again after this fraction of the lease |
:backoff_min / :backoff_max | 1_000 / 30_000 | Milliseconds between failed logins, doubling |
:login_timeout | 15_000 | How long a caller waits for a login in flight |
How it behaves, in short. See AshVault.KeyProviders.OpenBao.KubernetesAuth for the
details.
- One token holder per provider. It starts on first use under AshVault's own
supervisor, so there is nothing to add to your supervision tree. The
:ash_vaultapplication must be running. Amixtask or release command that has not started it getsProviderUnavailablewith reason{:not_started, :ash_vault}. - The login. The holder posts the service-account JWT to
/v1/auth/<mount>/loginand caches the client token. - Refreshing. Once two thirds of the lease has passed, it logs in again from
scratch. It re-reads the JWT file each time, because the kubelet rotates projected
tokens. A fresh login is also not limited by the role's
token_max_ttl, whichrenew-selfwould be. - Requests never wait on a refresh. A request that arrives while a refresh is running uses the current token. When there is no valid token, concurrent requests share a single login.
- Failures. A failed refresh keeps the old token in use until it expires, and
retries with backoff. With no valid token, a request inside the backoff window fails
at once instead of starting another login. Every failure is
AshVault.Errors.ProviderUnavailablewith reason{:kubernetes_auth, reason}, which is retryable. Some example reasons are:forbidden,{:http_status, 400},{:jwt_unreadable, path, :enoent}and{:transport, :econnrefused}. A failure is never reported asKeyDestroyed. - Secrets stay out of logs. Neither the JWT nor the client token is logged or
placed in an error. The holder's
:sys.get_status/1and crash reports show the token as:redacted.
A token revoked out of band (for example with bao token revoke) is not noticed
until the next scheduled login. Until then, requests fail with
AshVault.Errors.ProviderForbidden.
OpenBao side
bao auth enable kubernetes
bao write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc
bao write auth/kubernetes/role/ashvault-foundry \
bound_service_account_names=foundry \
bound_service_account_namespaces=foundry \
token_policies=ashvault-foundry \
token_ttl=1h token_max_ttl=4h
Grant the policy only what the provider needs. The OpenBaoTransit moduledoc lists the
paths. Do not grant transit/export/*: AshVault.Macs.OpenBaoTransit computes MACs
with transit/hmac and transit/verify, so the MAC key never leaves OpenBao either.
Server-enforced retirement
Stock Transit is the default (server_retirement: false). Its client-side tombstone
recheck cannot clean up a delayed create when that client dies. For permanent
no-recreation at the server, install the optional plugin from the repository's
openbao_plugin/ directory and set server_retirement: true on either provider.
The plugin uses OpenBao SDK 2.7.1 transactions, tested against OpenBao 2.6.2 with Raft storage. A backend without transactional storage fails with HTTP 503. Every key operation, including export, rotation, AEAD and HMAC, checks the retirement marker in the same transaction as its key access. Retirement deletes policy and archive entries for all three key purposes and permanently marks their names. Concurrent creators cannot commit across that retirement transaction. A lost commit response remains an outage; repeat destruction until confirmed.
This is a restricted Transit backend for AshVault keys, not a general Transit replacement. It supports the AES-GCM and ChaCha20-Poly1305 profiles used by AshVault, versioned exports, rotation, AEAD with associated data and SHA-256 HMAC. Derived, convergent, imported, soft-deleted, auto-rotating or plaintext-backup-enabled keys are refused. Import/restore, implicit creation, retirement deletion and other Transit operations are unavailable. Unsupported request options are rejected. Use a dedicated AshVault mount; inventory existing key profiles before upgrading one.
Build and install
Build from a reviewed repository checkout with Go 1.27.3 or newer. The root Nix shell includes Go and OpenBao; the plugin adds no runtime dependency to an Elixir release.
cd openbao_plugin
go build -trimpath -o ashvault-transit ./cmd/ashvault-transit
sha256sum ashvault-transit
Copy the binary to each OpenBao node's configured plugin_directory, with execute
permission for the OpenBao user. Register its SHA-256 using an operator token:
bao plugin register -sha256=SHA256 -command=ashvault-transit -version=v0.1.0 secret ashvault-transit
bao secrets enable -path=ashvault-transit -plugin-version=v0.1.0 ashvault-transit
bao secrets enable -path=ashvault -version=2 kv
Configure transit_mount: "ashvault-transit" and the existing kv_mount. Do not
point an existing installation at a new empty mount: it would lose access to its keys.
The mount refuses key operations until initialized. Grant the application these
additional paths (replace the Transit mount prefix as configured):
path "ashvault-transit/ashvault/protocol" { capabilities = ["read"] }
path "ashvault-transit/ashvault/retired/*" { capabilities = ["read"] }
path "ashvault-transit/ashvault/retire/*" { capabilities = ["update"] }Retain the provider's ordinary key/crypto and KV permissions. The plugin does not
need delete on key paths; use its retirement endpoint. Keep sudo on initialization,
KV metadata deletion and mount/catalog administration out of application policies.
Upgrade an existing stock Transit mount
Stop writers and drain requests once for this maintenance migration. Keep the same
Transit and KV mounts. Register the binary under the existing plugin name transit
with a custom version, tune the mount and reload it; do not disable the mount:
bao plugin register -sha256=SHA256 -command=ashvault-transit -version=v0.1.0 secret transit
bao secrets tune -plugin-version=v0.1.0 transit/
bao plugin reload -plugin=transit
This preserves stock Transit storage, including keys, versions and archives. Reloading by plugin name affects all mounts using it; use dedicated mounts and schedule accordingly. After installing or upgrading, import all previous KV tombstones before activation:
mix ash_vault.openbao.init_retirement AshVault.KeyProviders.OpenBao
# Or the OpenBaoTransit provider, using its configured mounts.
Run with an operator token configured for that provider: read on sys/mounts to
verify the configured store is KV-v2, read/list on
<kv_mount>/metadata/tombstones, read on <kv_mount>/data/tombstones/*, update
on <transit_mount>/ashvault/retire/*, and update plus sudo on
<transit_mount>/ashvault/initialize, as well as protocol read access. Create the
KV-v2 mount explicitly before a fresh installation's initialization. The task loads
application configuration and starts only AshVault and its HTTP dependencies, not the
consumer application. It imports soft-deleted tombstones too and stops on any unreadable
or ambiguous entry. Interrupted imports are safe to repeat. Never initialize a migrated
mount directly with bao write: skipping the tombstone import would lose old erasures.
Set server_retirement: true, verify provider.setup() succeeds, then resume writers.
No re-encryption or lookup-token rebuild is needed. Subsequent scope retirement needs
no per-scope writer drain. Route requests to the active node as described above.
The library retains KV tombstones as well as server markers. Reverting the application flag keeps recognizing scopes erased through the provider, but does not switch the server backend: stock-mode key deletion is unsupported by the plugin and fails closed. Reverting the mount implementation is a separate operator change: retain its storage and KV tombstones, stop writers, and reconcile every server retirement into KV first if operators used the raw plugin endpoint. Never remove or restore retirement metadata to reactivate a scope. No mode can revoke already-exported keys, recall in-flight responses or erase historical OpenBao snapshots/Raft logs.
Verify the plugin
ASHVAULT_BACKEND is the shared test selector: memory (default) runs without services;
openbao includes both stock Transit and plugin integration checks. From
openbao_plugin/, run ASHVAULT_BACKEND=openbao go test -race -count=1 -timeout 3m ./... with bao, curl,
go and rtk on PATH (Linux). It starts its own disposable Raft server, upgrades stock
keys in place, kills delayed clients and the plugin process, and restarts the server.
For Elixir, ASHVAULT_BACKEND=openbao mix test includes the full PostgreSQL/OpenBao
suite. Prepare stock Transit at transit and an initialized plugin at ashvault-transit.
BAO_ADDR and BAO_TOKEN configure their connection; no separate integration toggles
are needed.
Migrate ambiguous binary scopes
Ordinary text tenant identifiers keep their names. An exporting scope whose canonical
base64url encoding starts with nx_ formerly overlapped the native provider's prefix.
It now uses ashvault.v2.export.<encoded>. Every operation on this reserved binary
range requires explicit namespace readiness in KV; missing or ambiguous readiness is
an error, so existing ciphertext cannot silently acquire a new key version 1.
Stop all writers and drain requests, including older application versions. Install the updated plugin at the existing mount without disabling it, initialize retirement as above, and run with an operator token:
mix ash_vault.openbao.migrate_namespace BASE64URL_SCOPE
Use the same task to provision a fresh scope in this binary range. The token needs
sudo and update on <mount>/ashvault/namespace/*, read on sys/mounts and old
KV tombstones, plus read/write access to <kv_mount>/data/namespaces/* and tombstones.
Applications need read access to the namespaces path. Extend the exporting application's
existing Transit key and export grants to ashvault.v2.export.*; an ashvault_* grant
does not cover the new prefix. Native-provider tokens keep their own key prefix and
must not gain export permissions.
The server atomically moves exportable data/MAC/lookup policies and archives, retaining their key bytes, version numbers and timestamps. Non-exportable keys belonging to the native scope stay untouched. Mixed ownership, unexpected archives and occupied destinations fail closed. The client never receives key bytes. Historical tombstones and server retirements remain erasures; old ambiguous erasure cannot be reinterpreted as a new live scope. KV readiness is written last. Keep writers stopped after a failure and repeat the same command to finish an interrupted migration.
For rollback, keep writers stopped and use the same command with --rollback. It
restores the original keys and disables the new namespace, and refuses an occupied or
retired legacy destination or an erased new scope. It never removes retirement markers.
Resume older code only after rollback completes. After successful forward migration,
stock mode can return to the original stock Transit implementation while retaining the
mount storage and KV readiness; all clients must use the updated library. Keep the
plugin when using server_retirement: true. Neither direction re-encrypts rows or
changes lookup tokens. Never delete readiness or tombstones to force a retry.
A private CA
:cacertfile is a path to a PEM bundle, typically a ConfigMap mounted as a file:
volumes:
- name: openbao-ca
configMap:
name: openbao-ca
containers:
- name: app
volumeMounts:
- name: openbao-ca
mountPath: /etc/openbao-ca
readOnly: trueAshVault passes it to Req as
connect_options: [transport_opts: [cacertfile: path, verify: :verify_peer]]. Mint keeps
its hostname check and SNI on top of that. Things to know:
- The host in
:addressmust match a name in the server certificate.openbao.foundry.svcandopenbao.foundry.svc.cluster.localare different names to the check. Issue the certificate for the name you configure. - A missing or unreadable file fails before any connection is attempted. The error
is
ProviderUnavailablewith reason{:cacertfile_unreadable, path}, so it cannot be mistaken for a network outage. - The CA file is read once per path and cached for the life of the VM. Mint does the caching. After you rotate the CA, restart the pods.
- Other TLS settings go in
:connect_options. Pass it for anything else (timeouts,transport_optssuch asversions). Itstransport_optsare merged with:cacertfile.verify: :verify_noneis refused withArgumentError.
Without :cacertfile, Mint verifies against the OS trust store, or CAStore, the way it
always has. Adding the CA to the image's trust store also works. :cacertfile keeps the
trust scoped to OpenBao.
Other Req options
:req_options is a keyword list merged into every request before AshVault's own
options. AshVault's options win wherever both set the same key. Tests use it to route
requests to a Req.Test stub (req_options: [plug: {Req.Test, MyStub}]). Use it for
anything Req supports that has no dedicated setting here.