Operating the broker

The broker is a stateless HTTP process, and a deployment may run more than one of them behind one address (Running more than one pod). Everything it knows lives in Postgres; it reads who may have which secret, and the secrets’ values, from its secret store, AWS Secrets Manager or Google Cloud Secret Manager, and it starts nothing else.

What to run

The broker ships as envoy-broker in the Envoy image, ghcr.io/cybermonklabs/legion/envoy:<commit>, which .github/workflows/release-envoy-listener.yaml publishes on a push to main that touches the image’s sources (the image’s default entrypoint is the Envoy listener, so run it with --entrypoint envoy-broker). Of the builds waiting at once only the newest runs, and the build that replaces another holds that commit’s change, so a commit may reach main without an image tag of its own. Pin the image by digest and verify it as Running Dispatch shows. To build it yourself:

cd packages/envoy
go build -o envoy-broker ./cmd/broker

It takes no flags (it refuses any flag it is given) and reads all of its configuration from the environment. The configuration reference, generated from the loader, lists every variable with its default and accepted values. A minimal production configuration is:

BROKER_LISTEN_ADDR=0.0.0.0:13380
BROKER_PUBLIC_URL=https://secrets.internal.example
BROKER_MACHINE_LOGIN_URL=https://dispatch.internal.example/credentials/machine
BROKER_DATABASE_URL='postgres://broker:<password>@db.internal.example:5432/broker'
BROKER_K8S_OIDC_ISSUER=https://<cluster issuer>
BROKER_K8S_OIDC_AUDIENCE=legion-broker
BROKER_API_ACCOUNTS=dispatch=system:serviceaccount:example-ns:example-dispatch
BROKER_SIGNIN_ISSUER=https://accounts.google.com
BROKER_SIGNIN_CLIENT_ID=<client id>.apps.googleusercontent.com
BROKER_SIGNIN_CLIENT_SECRET_FILE=/var/run/secrets/broker-signin/client-secret
BROKER_SIGNIN_DOMAIN=example.com
BROKER_SECRETS_PREFIX=production/agent-secrets/
BROKER_SECRETS_KMS_KEY_ARN=arn:aws:kms:<region>:<account>:key/<key id>
AWS_REGION=<region>

BROKER_MACHINE_LOGIN_URL is the page where a person types a machine login’s code, Dispatch’s /credentials/machine. It is optional. With it, the device authorization grant’s verification_uri names that page, so a machine shows its person where to go. Without it there is no device grant at all: POST /v1/oauth/device_authorization is not mounted and answers 404, and the broker’s authorization server metadata leaves the device code out of grant_types_supported, which is how Dispatch knows to serve no machine-code page. Everything else the OAuth provider serves — the token endpoint, the RFC 8693 token exchange, introspection and revocation — is served either way. Leave it unset in a deployment where every machine holds a cloud identity and nobody ever types a code. The boot line says which of the two this broker is: machine login device grant state=on verification_uri=…, or state=off naming the unset variable.

GET /.well-known/oauth-authorization-server is that metadata, RFC 8414. It needs no credential, since it is what a client reads before it holds one, and nothing in it is a fact a caller could not learn by calling the endpoints it names. It names this broker’s issuer (BROKER_PUBLIC_URL), its token, introspection and revocation endpoints, the grants it serves, the DPoP signing algorithms it takes, and the subject token types its exchange is configured with, so a client reads off one document what this deployment turned on rather than being configured with it twice.

BROKER_API_ACCOUNTS lists the callers of the broker’s named-caller routes, here Dispatch’s server as the caller dispatch, each by its own account. An account is written in one of two forms, and the entry’s own text says which: a Kubernetes ServiceAccount subject, as above, or google: and a Google service account’s address (listener=google:example-listener@example-project.iam.gserviceaccount.com). A Kubernetes caller presents its projected token for BROKER_K8S_OIDC_AUDIENCE, verified against BROKER_K8S_OIDC_ISSUER, and is looked up by the token’s sub, so such an entry needs that pair. A Google caller presents its machine’s own Google identity token for BROKER_GOOGLE_OIDC_AUDIENCE — the audience every machine of the deployment already holds a token for, so the Envoy listener on a machine presents what its metadata server already hands it and mints no second token — verified against BROKER_GOOGLE_OIDC_ISSUER, and is looked up by the token’s email, so such an entry needs that audience and no cluster at all. A project’s default Google accounts and Google-managed service agents are refused here as they are in BROKER_VM_SERVICES. No secret is shared between the broker and its callers. A token that lives longer than an hour from its issue is refused, and an account the list does not name is answered 403 ACCOUNT_NOT_LISTED. Through those routes Dispatch lists what waits on a person and relays a person’s machine-login decisions, their machine-login revokes and their sign-in revokes; it decides no secret request (Network and trust).

The four BROKER_SIGNIN_* settings are required, and a broker started without one of them exits naming the missing ones. They are the page where a person approves a secret request, signed in to the broker itself (Approval sign-in).

A caller’s name decides which of those routes its account reaches, so listing an account is not one authority but a choice between them. The broker knows three names: dispatch, which reaches every route Dispatch’s server calls — the approval, denial, revocation and listing routes, POST /v1/oauth/revoke, and POST /v1/oauth/introspect; listener, which reaches POST /v1/oauth/introspect alone, for a resource server such as the Envoy listener that reads one of the broker’s access tokens back and relays nobody’s decision; and access-review, which reaches GET /v1/grant-log alone, for the quarterly review that reads the broker’s record of who asked for what and who said yes, and changes nothing in it. The grant log is no part of what dispatch and listener carry, and neither of them reaches it. A listed caller on a route its name does not carry is answered 403 ACCOUNT_LACKS_POWER, naming the caller and the route, and an entry naming a caller the broker does not know refuses the start, naming that name and the names it knows, so a typo cannot quietly produce a caller that reaches nothing.

Five settings turn the model judge on, which decides one narrow class of the requests that would otherwise wait for a person (Concepts): the endpoint and the id of the model it asks, the audience of the Google identity token the broker presents to that endpoint, and the Dispatch it reads an issue from with the file holding the broker’s own projected ServiceAccount token for it. They are all-or-none: with none of them set there is no judge, and every human-tier request waits for the person who decides it today. A sixth setting names the protocol that endpoint speaks and has a default. The configuration reference names each and gives its accepted values. BROKER_JUDGE_TEST_SKIP_CHECKS turns the judge’s checks before the model off, as the local stack’s negative control of them, and a production start — one with BROKER_FAKE_SECRETS_FILE unset — is refused naming it.

BROKER_SECRETS_STORE picks the secret store: aws-secrets-manager, the default when it is unset, as above, or gcp-secret-manager, which takes the Google Cloud forms of the namespace and the key and reads nothing from AWS:

BROKER_SECRETS_STORE=gcp-secret-manager
BROKER_SECRETS_PREFIX=projects/<project number>/secrets/agent-secrets-
BROKER_SECRETS_KMS_KEY_NAME=projects/<project>/locations/global/keyRings/<ring>/cryptoKeys/<key>

The prefix names the project by its number, not its id, because Secret Manager answers every resource name with the number and the broker matches what the API answers (gcloud projects describe <id> --format='value(projectNumber)' finds it); the id after secrets/ ends in -, so the secret a session asks for as DEEL_API_KEY is projects/<number>/secrets/agent-secrets-deel-api-key. The key is in the global location, the only one a secret with automatic replication can be encrypted with. Each store refuses the other’s key variable, so a deployment moved between stores cannot keep the old key set and silently mean nothing by it.

AWS_REGION is the AWS SDK’s own setting, not the broker’s, read by the AWS store alone: the SDK takes the region of Secrets Manager and KMS only from AWS_REGION, AWS_DEFAULT_REGION or the shared AWS config file, never from the instance it runs on. Without one, the broker’s first read of the namespace fails and it exits at startup with broker: fatal error="list secrets under production/agent-secrets/: operation error Secrets Manager: ListSecrets, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region".

BROKER_SERVICE_ACCOUNT_SECRETS binds each google-service-account secret to the one Google service account the broker may mint a token for, whitespace-separated <SECRET NAME>=<account email> entries:

BROKER_SERVICE_ACCOUNT_SECRETS="GCP_ADMIN=gcp-admin@example-project.iam.gserviceaccount.com DEV_ADA=dev-ada@example-project.iam.gserviceaccount.com"

The name is the uppercase name a session asks for, not the secret’s store id or its slug; the account is a user-managed service account, <account id>@<project id>.iam.gserviceaccount.com in lowercase. The broker refuses to start, naming the entry, on a name no agent secret can carry, an address that is not a user-managed account, a Google-managed service agent (Google’s own services run as those accounts, and the broker must not be able to impersonate one), or a secret named twice. One account may back two secrets, since only the secret’s name is the key.

This setting, and not a secret’s stored value, chooses the account: the broker holds roles/iam.serviceAccountTokenCreator on every account bound here, so a mint happens only where the binding and the stored value name the same account. A secret the setting does not name, and a stored value naming any other account, release nothing: the session is answered 502 MINT_FAILED and the broker logs class=binding, before any call reaches Google. Unset, no secret is bound and every google-service-account release is refused, which is where a deployment stands between this build and the apply that writes its bindings; the broker still boots and still serves every other kind. The policy loader reads the setting as well, for one rule: a secret bound to a dev-<name> account is one person’s own, so it may list no registered service among its requesters.

The broker refuses to start, naming the variable, when one is missing, malformed or out of range, and also while a variable it no longer reads is still set. BROKER_DATABASE_URL is the only database variable: a password goes in the URL itself, URL-escaped, and on Amazon RDS the broker needs none (Signing in to RDS by IAM token).

A secret owned by a service rather than a person needs that service registered. BROKER_SERVICES lists each service with the Kubernetes service accounts its pods run as, whitespace-separated name=<service account> entries, for example BROKER_SERVICES=legion-daemon=system:serviceaccount:legion:legion-worker for the Legion daemon’s worker pods. A service whose pods run as several accounts gets one entry for each, for example legion-daemon=system:serviceaccount:legion:legion-worker legion-daemon=system:serviceaccount:legion:legion-controller.

A pod gets the service’s agent-tier secrets at once when two things hold: a machine login for the listed service enrolled it, and its projected token proved one of that service’s accounts. Every other session is denied them. For the same reason each account is bound to one service: the broker refuses to start, naming both services, if BROKER_SERVICES binds one account to two of them. It also refuses, naming the account, an account given twice for the same service.

The account must be one the service’s pods name. The broker refuses to start, naming the entry, if BROKER_SERVICES binds a namespace’s default account (system:serviceaccount:<namespace>:default). Every pod in a namespace whose spec names no account runs as default, and any of those pods can get a token for the broker’s audience. A default binding would therefore prove nothing about which pods are the service’s. Create a dedicated account, such as legion-worker, set it as the pods’ serviceAccountName, and bind that account.

While a secret’s owner tag names a service the list leaves out, the broker refuses the secret as owner-tag-malformed (Concepts).

A service’s launcher, such as the Legion daemon, logs in with its own projected ServiceAccount token, and no person approves it. BROKER_SERVICE_LOGINS lists which ServiceAccounts may log in as which service, whitespace-separated <service>=<service account> entries, for example BROKER_SERVICE_LOGINS=legion-daemon=system:serviceaccount:legion-system:legion-daemon-legion, one entry per daemon. Each account is named exactly, with no wildcard, and logs in as the one service its entry names. The broker refuses to start on an entry naming a namespace’s default account, which every pod in that namespace whose spec names no account runs as, so any pod there could log in as the daemon. An account BROKER_SERVICES binds for a service’s pods may be listed here as well, which is a pod of that account logging the service in to enrol itself (agent-secrets enroll --pod). Every pod of such an account holds a token for it and so logs the service in, and each login enrols the one pod whose token made it: any other pod UID is refused 403 POD_SELF_ENROLL_ONLY. A pod enrolling itself states no session name either, 403 POD_SELF_ENROLL_NO_SESSION: the pods of one account enrol under one principal, so a name such a pod picked would be a sibling pod’s, and the enrolment would take that sibling’s grants. agent-secrets enroll --pod sends none. The broker verifies the token as it verifies a pod’s, against BROKER_K8S_OIDC_ISSUER and BROKER_K8S_OIDC_AUDIENCE, so the setting needs both, and refuses a token that lives longer than an hour from its issue (exp minus iat, beside a minute of skew) as TOKEN_TOO_LONG_LIVED: give the daemon’s projected token an expirationSeconds of at most 3600, and kubectl create token a --duration of at most 1h. Each account holds one launcher credential: every login moves it to the login’s key and to the token’s expiry, at most an hour away. A running daemon proves the same key at each login, so nothing about it changes but its expiry; a restart’s login brings a new key, so the daemon’s earlier key stops working at once and the pods it enrolled before the restart stay its own. A device-code login that names a service is refused SERVICE_LOGS_IN_WITH_TOKEN. Dispatch’s machine-login page lists a daemon’s token login for everyone signed in, naming its account and no approver, and anyone may revoke it, which ends every pod it enrolled at once; the daemon then logs in again with its token under a new credential, which enrolls the pods it launches next, while a pod already running stays without secrets until it is replaced. To stop a daemon logging in, remove its entry and redeploy the broker: its credential stops authenticating at its next call, and its next login is refused.

An account on that list also registers which pod may write under which session name (POST /v1/oauth/exchange/pod-sessions), which is how a deployment’s daemon tells the broker that a pod it has just started owns an issue’s session. That is a second authority the list carries, so weigh it before listing an account. The request carries the pod’s own projected ServiceAccount token, which the broker verifies against the cluster’s issuer as it verifies an enrolling pod’s, bar the one check it does not make here, that the token names a pod: such a token verifies and the exchange then refuses the registration 400 POD_NOT_BOUND. So a listed account registers a pod whose token it holds and no other. A caller whose own token names a pod may not register that pod, 403 POD_CALLER_IS_THE_POD — a pod that wrote its own row would pick the session name it writes under, including a name a sibling pod holds. The refusal reads the caller’s own token, so it never holds a caller whose token names no pod, such as one kubectl create token mints: whoever mints one for a listed account registers every pod of it whose token they hold, the pod they run in included. So listing a session account, which the paragraph above describes, does not in itself give its pods that authority over one another: the authority follows the token, so a pod reaches another only while it holds that pod’s own token, and a pod holding its own alone reaches nothing here, since its own is the one the refusal takes away. A pod that does hold a sibling’s registers it, which is how one generation of a rollout registers the next. The authority goes no wider than the cluster’s own rights to mint or read a pod’s token — the same kind of access over that pod’s account as the paragraph below names over a listed one — and it stops short of them: the caller needs a BROKER_SERVICE_LOGINS token of its own, and the pod’s account must be one BROKER_EXCHANGE_ACCOUNTS lists, 403 POD_ACCOUNT_NOT_LISTED. A deployment that does not want that lists no such account. A daemon that stops registering costs its pods their writes within the registration’s own retention (24 hours past the last write). Do not list such an account in BROKER_API_ACCOUNTS as well: that is the separate list of the callers that relay a person’s decisions and read tokens back.

Whoever can get a token for a listed account logs in as its service. Kubernetes access control on the daemon’s namespace is that boundary: anyone who may create pods there naming the account, create serviceaccounts/token for it, or exec into the daemon’s pod and read its token file can get one. Deleting the daemon’s ServiceAccount stops new tokens, but a token already issued keeps verifying until it expires, at most an hour after it was minted, because the broker checks tokens against the cluster’s published keys rather than asking the API server. A token login spends its source address’s rate-limit bucket before its token is verified and its account’s own after, so a daemon behind an address other callers share is held to that address’s bucket. A login the broker could not complete (a 5xx, such as while its database is away) spends neither bucket, so the daemon’s retries through a database outage leave it free to log in once the database is back. Set BROKER_TRUSTED_PROXY_HEADER behind a proxy, so each caller is keyed on its own address, and keep the daemon’s and its pods’ traffic to the broker on their own pod addresses rather than translated to a node’s or a gateway’s. The daemon renews halfway through its credential’s life, which carries it through brief contention for its address’s bucket; a caller that keeps that bucket drained for the rest of the credential’s life stops it logging in.

A service login a person approved through a device code before this broker keeps working until it expires, at most BROKER_LAUNCHER_CREDENTIAL_SECONDS (30 days by default) after its approval: nothing refreshes it.

A machine holding a Google identity token — a Compute Engine VM, a pod on Google Kubernetes Engine, or anything a workload identity pool mints one for — trades that token at the broker’s token exchange for a Dispatch access token and the machine login that trade mints. It holds no stored credential and no person approves it. Five settings describe its fleet, and the broker refuses to start with only some of them:

• BROKER_GOOGLE_OIDC_AUDIENCE: the audience the machines ask their metadata server for. The issuer is Google’s, https://accounts.google.com, unless BROKER_GOOGLE_OIDC_ISSUER names a stand-in for a dev stack; the broker reads its keys at startup only when the audience is set, and then logs one google machine identity line naming the issuer, the audience and the machine services. Unset, the broker reads and logs nothing about Google and takes no such token.
• BROKER_VM_SERVICES: whitespace-separated <service>=<service account email> entries. Use a dedicated user-managed account for each fleet: the broker refuses a project’s default Compute Engine or App Engine account, which every workload in the project runs as by default.
• BROKER_VM_PROJECTS and BROKER_VM_INSTANCES: <service account email>=<project id> and <service account email>=<instance-name glob>, in Go’s path.Match syntax. They pin a fleet to one Compute Engine project and the instance names its machines have. The two go together: set both for an account, or neither. An account pinned in neither takes its machines wherever they run, a token carrying no google.compute_engine claims included, which is what a workload identity pool and a pod’s metadata server mint.
• BROKER_VM_EXCHANGE_ACCOUNTS: <service account email>=<project>=<person>[=<session>], the person and the Dispatch project a fleet writes for. An account BROKER_VM_SERVICES lists and this setting does not gets a token that names its service and no person, which Dispatch and the Envoy listener read. Either way it reaches the fleet’s Dispatch token alone and never its secrets: the machine login the exchange mints names no person, so a machine reads what its service owns and the shared agent-tier secrets, and a person reaches their own on that machine only by signing in on it.

One fleet’s entry, example-fleet, whose machines run as one account in example-project, are named example-vm-..., and write for one person:

BROKER_GOOGLE_OIDC_AUDIENCE=example-vm-audience
BROKER_VM_SERVICES=example-fleet=example-fleet@example-project.iam.gserviceaccount.com
BROKER_VM_PROJECTS=example-fleet@example-project.iam.gserviceaccount.com=example-project
BROKER_VM_INSTANCES=example-fleet@example-project.iam.gserviceaccount.com=example-vm-*
BROKER_VM_EXCHANGE_ACCOUNTS=example-fleet@example-project.iam.gserviceaccount.com=EXAMPLE=ada@example.com

A machine of that fleet then reads the secrets example-fleet owns and the shared agent-tier ones, and asks for the rest. ada@example.com is on the Dispatch token it trades for, not on its machine login: her own secrets stay hers until she signs in on that machine.

When she does — agent-secrets login, which needs BROKER_SIGNIN_DEVICE_AUDIENCE set — that sign-in’s trade mints a machine login of its own on that box, carrying her rather than the fleet, and a session enrolled under it reads her secrets and the shared agent-tier ones and nobody else’s. It is bound to the box key the command makes there, so her sign-in on one machine is not her sign-in on another. Revoking that sign-in (POST /v1/person-signins/{id}/revoke, which Dispatch relays from her page) ends the access tokens it minted and the machine login with them, so every session it enrolled ends at once; the fleet’s own machine login, and her sign-ins on other machines, are untouched. That revoke ends the box key, not her grant at Google: agent-secrets login keeps Google’s refresh token on the machine beside the key, and whoever can read that machine’s files can copy both and sign in again from elsewhere, as a sign-in of its own that this revoke does not reach. To cut off a box you believe is compromised, have her revoke the grant at Google — her Google Account’s third-party connections page, the entry for the deployment’s agent-secrets login client — which ends every copy of that refresh token at once; the broker’s own revoke is for a box she has finished with. A sign-in belongs to the machine it was made on: the same box key presented with another machine’s identity token is refused invalid_grant naming agent-secrets login, since the machine login it holds is keyed by the person, the machine and the key together. The broker refuses an exchange whose token is for another audience or does not verify, whose email Google does not hold verified, whose account no entry lists, or whose Compute Engine project or instance name the entry’s pins do not allow, each 400 invalid_request naming what did not match.

Two settings turn a person’s sign-in on. BROKER_SIGNIN_DEVICE_AUDIENCE is the OAuth client id of the device-flow client agent-secrets login runs Google’s device authorization grant as, which is the audience a person’s own Google identity token must carry, since Google puts a client’s id in the aud of the ID tokens it mints for it; set it to the same client id the machines are given as AGENT_SECRETS_SIGNIN_CLIENT_ID. BROKER_SIGNIN_DOMAIN, which the approval sign-in already needs, is the Workspace domain the person’s hd claim must name, so a person outside it signs in nowhere. The issuer of both the person’s token and the machine’s is BROKER_GOOGLE_OIDC_ISSUER, https://accounts.google.com unless a dev stack names a stand-in, and the audience setting needs BROKER_GOOGLE_OIDC_AUDIENCE, whose machines are the boxes a person may sign in on. Unset, the broker accepts no person’s sign-in and refuses either token type by name.

Each machine holds a launcher credential of its own, keyed by the machine and the key its proof presented: <account>@<project>/<zone>/<instance id>#<key thumbprint> for a token minted on a Compute Engine instance, and <account>#<key thumbprint> for one minted anywhere else. The key is part of the name because the identity token is a bearer: a copy of it presented with a key of the copier’s asks for a credential of its own and moves nothing of the machine’s. So one machine’s trade never moves another’s credential, and ending one machine’s enrollment leaves the others. agent-secrets-helper keeps that key in ~/.config/agent-secrets/identity-key.pem, mode 0600, beside the machine login a machine that proves no identity keeps, and makes one only at this machine’s first trade. A restart therefore trades the same subject and moves the one credential the machine already holds, so the sessions enrolled under it keep renewing. A machine that makes a new key — one whose key file was removed, or a machine rebuilt — leaves its old credential behind until that credential’s own expiry, and starts a new one at its next trade; the sessions enrolled under the old one stop renewing when it lapses, and a box enrolled with agent-secrets enroll --helper is enrolled again by running that command again.

Only a trade that asks for a credential holds one, and a trade asks unless it says otherwise, so agent-secrets and the helper hold theirs by doing nothing. A person’s sign-in cannot do otherwise: a sign-in holds its credential, so a trade of a person’s that declined one is refused 400 invalid_request. The clients that want a Dispatch token and read no secret — the dispatch command, the Oh My Pi and Claude Code plugins, envoy-send — decline it, since each of them makes a key per process and per session and a machine-logins list with a row per command run is a list nobody can read. So the machines on that list are the boxes that enrol sessions, not every process that ever traded.

The credential expires with the token it was minted from, an hour after that token was minted, and a machine’s session renews only while that credential is live: not revoked, not past its expiry, and its account still listed for its service. A machine that stops being able to prove its identity therefore stops renewing within the hour, and its sessions lapse one BROKER_LEASE_SECONDS later; a copy of a session’s key and enrollment id taken off the box renews nothing once the box’s login has ended. The broker answers such a renewal 401 ENROLLMENT_NOT_LIVE and logs broker: machine session renewal refused: its machine login has ended. Only a machine’s own session is held this way: a person’s session on that box renews until they revoke the sign-in, which ends it at once, and a pod’s renewal proves the pod’s own projected token.

Dispatch’s machine-login page lists each machine’s login, its host and no approver, and anyone signed in may revoke it, which ends that machine’s enrollment; the machine trades again at its next call. A revoke ends the enrollment, not the Google identity token: someone who copied that token off the machine trades it again within its hour, under a machine login of their own key, and reads what that fleet’s service may read. To end a copied token for certain, take the machine out of its fleet and wait out the hour, as a copied machine token says. To stop a fleet trading, remove its entries and redeploy the broker: each machine’s credential stops authenticating at its next call, its next trade is refused, and its sessions stop renewing.

A revoke alone does not cut off a machine whose key may have been copied. The credential is keyed by the machine and the key it proved, and agent-secrets-helper keeps that key at ~/.config/agent-secrets/identity-key.pem across restarts, so the box’s next trade — within the hour, since the credential lives only as long as the identity token it was minted from — mints a new live credential bound to that same key, and whoever holds a copy of the file signs launcher proofs under the new credential’s id. Three steps, in this order, end that:

1. remove ~/.config/agent-secrets/identity-key.pem on the machine;
2. restart the helper (systemctl --user restart agent-secrets-helper.service, or agent-secrets-helper install --restart-with-sessions, which also ends the grants of the sessions registered with it), so it makes a new key and its next trade holds a credential of a subject of its own;
3. then revoke the old login on the machine-login page, by the credential id the old key held, which ends it and every session it enrolled.

In the other order the revoke is undone: the helper still holds the key, and its next trade inserts a live credential on it again, since a revoked row is no longer the one a trade of that subject moves. None of this reaches the machine’s Google identity token, which is what the trade itself proves: a box you believe is compromised comes out of its fleet, as a copied machine token says.

Whoever can get an identity token for the fleet’s account logs in as its machines. Google Cloud IAM is that boundary: who may run a workload as the account (iam.serviceAccounts.actAs on it), and who may run code on a machine of the fleet, which can fetch that machine’s token.

On the machine itself there is no enrollment command to run and no key to place: agent-secrets, agent-secrets-helper and the Legion controller trade the identity the machine already holds whenever they find no key directory, no sign-in and no helper. What they read is AGENT_SECRETS_URL, the broker’s public URL, and AGENT_SECRETS_MACHINE_IDENTITY, which says which identity this machine proves: metadata with AGENT_SECRETS_MACHINE_AUDIENCE, a Compute Engine machine’s own token from its metadata server, which is the fleet above; kubernetes with AGENT_SECRETS_MACHINE_TOKEN_FILE, a pod’s projected service-account token; or pool, a token minted through a workload identity pool, which takes all four of AGENT_SECRETS_MACHINE_TOKEN_FILE, the projected service-account token the pool federates as the external credential; AGENT_SECRETS_POOL_AUDIENCE, the pool provider that credential is presented to, Google’s full //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider> resource name; AGENT_SECRETS_POOL_ACCOUNT, the service account the pool lets that credential act as, by email address or unique id, which generateIdToken is called on; and AGENT_SECRETS_MACHINE_AUDIENCE, here the aud of the token that mint returns, the client id the broker checks against BROKER_GOOGLE_OIDC_AUDIENCE. A pool machine missing any of the four trades nothing and its helper restart-loops, as Azure dev boxes did for want of the token file (theorem-labs/agent-infra#2200). AGENT_SECRETS_SESSION names the session a trade writes under for a caller that acts as no session of its own — agent-secrets, the helper, a daemon — and is the only name such a caller has. A caller that is a session writes under its own session id and reads that variable for nothing: an agent session’s id is minted when the session starts, and its token must name it, since the Envoy listener refuses a subscribe, an unsubscribe, a role claim or a send from a session whose token names another. So a fleet whose entry fixes a session, or whose pods write under registered names, is a fleet no agent session can trade at: the broker refuses the trade, naming what the entry writes as. For a person to sign in there, give the machine the device-flow client as AGENT_SECRETS_SIGNIN_CLIENT_ID and AGENT_SECRETS_SIGNIN_CLIENT_SECRET_FILE, the client whose id is BROKER_SIGNIN_DEVICE_AUDIENCE. With AGENT_SECRETS_MACHINE_IDENTITY unset nothing trades and every machine works as it did, through a machine login a person approves.

Approval sign-in

A person approves or denies a secret request on the broker’s own page, <BROKER_PUBLIC_URL>/approve/<record>, signed in to the broker with OpenID Connect through an OAuth client of the broker’s own; Dispatch’s Inbox only links there, and no route a service calls decides a secret request (Concepts). Four settings are the sign-in, and every one of them is required:

Setting	What it is
BROKER_SIGNIN_ISSUER	The provider’s issuer, whose discovery document the broker reads at startup and every hour after, in a check of its own (What the sign-in is doing), and for the page at the first sign-in each process serves, which it keeps once a read succeeds; until then a sign-in reads it again at most every 5 seconds. A read that fails stops neither the boot nor the machines the broker serves, so a provider outage never stops it serving them. https://accounts.google.com for Google. An absolute https URL, or http only to a loopback host, for a run on one machine.
BROKER_SIGNIN_CLIENT_ID	The broker’s own OAuth client at that provider. Not Dispatch’s: a token or code made for Dispatch’s client signs no one in to the broker.
BROKER_SIGNIN_CLIENT_SECRET_FILE	A file holding that client’s secret, read at startup and trimmed; mount it from a Kubernetes Secret. The broker refuses to start while BROKER_SIGNIN_CLIENT_SECRET is set, so the secret is never in its environment.
BROKER_SIGNIN_DOMAIN	The Google Workspace domain a person must belong to: the ID token’s hd claim must name it, and its email must be verified. A person’s login is their lowercased email, the form a secret’s owner tag names.
BROKER_SIGNIN_SESSION_SECONDS	Optional. How long a sign-in lasts, counted from the moment it is made and never extended: 86400 (24 hours), the default, at most. Signing out ends it at once. A person whose Workspace account is suspended keeps approving until their session’s expiry, which broker end-sessions ends sooner (Ending a person’s sessions).

A broker missing any of the four refuses to start, naming the ones it is missing: broker: fatal error="the broker's approval page needs BROKER_SIGNIN_ISSUER, BROKER_SIGNIN_CLIENT_ID, BROKER_SIGNIN_CLIENT_SECRET_FILE, BROKER_SIGNIN_DOMAIN; missing BROKER_SIGNIN_DOMAIN". There is no broker that serves machines but approves nothing: a deployment either signs people in or does not run.

On Google, create the client once, in the Google Cloud project whose OAuth consent screen is the Workspace’s Internal one (APIs & Services → Credentials → Create credentials → OAuth client ID, application type Web application), with one authorized redirect URI: <BROKER_PUBLIC_URL>/signin/callback, for example https://secrets.internal.example/signin/callback. Its client id is BROKER_SIGNIN_CLIENT_ID, and its secret goes in the file BROKER_SIGNIN_CLIENT_SECRET_FILE names. The broker asks for openid email profile with PKCE, asks for no offline access and keeps no refresh token, and sends the request’s approver as login_hint, so a person already signed in to Google is asked nothing.

Deploy the broker with these settings before the Dispatch release whose Inbox links the page. In between, an open Dispatch tab’s Approve on a secret request gets the broker’s 403 DECIDE_ON_BROKER_PAGE naming the page, and nothing is approved on Dispatch’s word.

What the sign-in is doing

GET /signin/status answers what this deployment’s sign-in has done, read from Postgres on every call, so every pod answers the same thing. It takes no credential, and every word in it is one of two sets the broker itself chose: no provider’s message, no login and no secret reaches it.

{
  "client_check": "ready",
  "client_checked_at": "2026-03-04T09:00:11Z",
  "last_sign_in_at": "2026-03-04T08:41:02Z",
  "last_failed_sign_in_at": null,
  "last_failed_sign_in_reason": null,
  "last_decision_at": "2026-03-04T08:41:40Z"
}

A fact that has never happened is null, so a deployment nobody has signed in to yet is not a deployment whose sign-in broke. What an operator reads from each field:

• client_check and client_checked_at: what the broker’s own client was found to be at the provider, and when. Each pod checks it at startup and every hour after, by asking the token endpoint to redeem an invalid code with the broker’s client credentials. ready is the provider accepting the client, client_rejected is the provider refusing it (nobody can sign in: the client id, its secret or its registration is wrong), and issuer_unreachable is a provider that could not be asked, which says nothing about the client. A check neither answers a person nor stops the broker: it writes this row, and it logs only a change. When it finds client_rejected or issuer_unreachable and its pod did not log that state last, it logs broker: the sign-in client check. When it finds ready after such a line, it logs broker: the sign-in client check finds the client ready again (Health and logs). So a provider that stays away for a day is one line, not one each hour.
• last_sign_in_at: when a person last finished signing in. Long silence on a deployment people use is the signal, read beside last_decision_at.
• last_failed_sign_in_at and last_failed_sign_in_reason: when a sign-in last failed at the code exchange or at the ID-token check, and why, as one of invalid_client, unauthorized_client, redirect_uri_mismatch, invalid_grant, issuer_unreachable, bad_issuer, bad_audience, bad_signature, bad_nonce, not_in_domain and email_not_verified. The first three are the client’s registration, not_in_domain and email_not_verified are the person rather than the deployment, and a failure none of the eleven names leaves the row alone and is logged instead.
• last_decision_at: when a person last approved or denied a request on the page.

Ending a person’s sessions

A broker session lasts BROKER_SIGNIN_SESSION_SECONDS from the moment it was made and is never extended, so a person whose Workspace account is suspended can still approve until it expires. To end every browser session one login holds, wherever and whenever it was made, run the subcommand in a broker pod:

kubectl exec deploy/broker -- envoy-broker end-sessions ada@example.com
end-sessions: ended 2 sessions of ada@example.com

It reads the same configuration the serving process does, so it reaches the same database and signs in to it the same way, binds no listener and applies no migration: an image older than the servers cannot migrate the database behind them on its way to ending a session. The person’s next visit to a request’s page signs them in again, if the provider still lets them; to stop that, take them out of the Workspace or out of BROKER_SIGNIN_DOMAIN first. It is the only subcommand the broker takes: anything else is refused with a usage line, and envoy-broker with no argument serves.

Signing in to RDS by IAM token

On Amazon RDS or Aurora the broker can sign in to its database with an IAM auth token instead of a password, so no database password exists for it to hold or for RDS to rotate under it. It does so when BROKER_DATABASE_URL names a user and no password and its host is an RDS endpoint, one ending in .rds.amazonaws.com:

BROKER_DATABASE_URL='postgres://broker@<cluster endpoint>:5432/broker?sslmode=verify-full&sslrootcert=/etc/ssl/rds/global-bundle.pem'
AWS_REGION=<region>

Each new connection signs in with a token the broker mints for that user and host, signed with the AWS SDK’s default credentials in the region AWS_REGION, AWS_DEFAULT_REGION or the shared AWS config names; with none, the broker refuses to start. A token is good for 15 minutes from its mint and is checked only when a connection signs in, so a connection the broker holds longer keeps working, and the next connection it opens brings a fresh token. The broker’s AWS identity needs rds-db:connect on the database user, arn:aws:rds-db:<region>:<account>:dbuser:<cluster resource id>/<user>, and the database user must be a member of rds_iam. The cluster needs IAM database authentication turned on. Give the broker a database user of its own, and never grant rds_iam to the RDS master user, directly or through a role it belongs to: once a user holds it, IAM authentication takes precedence over that user’s password (AWS’s limitations of IAM database authentication), so the master could no longer sign in with its password.

A token is a password to the database until it expires, so the broker sends one only to a server it has verified. The URL must name that one host, with sslmode=verify-full and an sslrootcert file (the Envoy image ships the RDS CA bundle at /etc/ssl/rds/global-bundle.pem); otherwise the broker refuses to start, naming the host: sslmode=require encrypts but verifies nothing, and sslrootcert=system (in the URL or PGSSLROOTCERT) names the system trust store, which holds no RDS CA, so every sign-in would fail. The bundle sits outside the system trust store, so nothing else in the image trusts it. It is the global bundle AWS publishes, vendored as packages/envoy/docker/rds-global-bundle.pem with the date it was fetched and its checksum.

A URL with a password, a passwordless URL whose password libpq supplies (PGPASSWORD or a passfile), or a host that is not an RDS endpoint (a local Postgres that trusts its clients, say) connects as given and mints nothing. The broker substitutes no password into the URL: it refuses to start while the removed BROKER_DATABASE_PASSWORD is set or the URL names its ${BROKER_DATABASE_PASSWORD} placeholder.

What it depends on

Dependency	What the broker needs from it
Postgres	A database the broker owns (BROKER_DATABASE_URL). The broker applies its own migrations at startup; they only move forward, so never run an older broker against a database a newer one has migrated. Each process also holds one connection outside its pool, for as long as it runs, to hear what the other processes reread (Running more than one pod). On Amazon RDS it can sign in by IAM token instead of a password (Signing in to RDS by IAM token).
Rolling back	Roll forward past a broker that serves kind secrets, never back. A broker before kinds reads no kind tag and would serve a kind secret’s stored GitHub App key or refresh token as its value. agent-secrets secret writes mint-service-account: kind-secret beside every kind, which every broker from legion-envoy-v13.19.0 to the release before kinds refuses as mint-target-malformed; a broker older than legion-envoy-v13.19.0 reads no such tag and serves the value. A kind secret written without the guard (by hand, or by a deployment that does not set it) is served by every earlier broker. To go back anyway, first disable every kind secret’s versions.
The secret store	The agent secrets: every secret whose name starts with BROKER_SECRETS_PREFIX, carrying owner and tier (AWS tags, or Secret Manager annotations), encrypted with the agent-secrets key, and holding a current value. A kind tag makes the broker mint a release from that value. Concepts describes the tags, the name each secret is asked for by, and how often the broker rereads them. BROKER_SECRETS_STORE names which store, and the two rows below say what each needs.
AWS Secrets Manager (aws-secrets-manager)	The broker calls AWS with the SDK’s default credential chain (environment, shared config, or the workload’s role), in a region (AWS_REGION, AWS_DEFAULT_REGION or the shared config). It needs secretsmanager:ListSecrets (which takes no resource, so on *), secretsmanager:DescribeSecret (to reread one secret) and secretsmanager:GetSecretValue on the namespace’s secrets, kms:Decrypt on the agent-secrets key (BROKER_SECRETS_KMS_KEY_ARN), kms:ListAliases (on *, called only when a secret names its key by an alias), and, when it signs in to RDS by IAM token, rds-db:connect on its database user; nothing else.
Google Cloud Secret Manager (gcp-secret-manager)	The broker calls Secret Manager under Application Default Credentials: on GKE, Workload Identity Federation for the pod’s Kubernetes ServiceAccount, with no key file and no Google service account; the boot log’s secret store … credentials=application-default line says so. Its principal needs secretmanager.secrets.list on the project (the listing takes the project as its parent) and secretmanager.secrets.get, secretmanager.versions.get and secretmanager.versions.access on the namespace’s secrets: roles/secretmanager.viewer on the project and roles/secretmanager.secretAccessor with an IAM condition resource.name.startsWith("projects/<number>/secrets/<id prefix>") cover them. The condition narrows only the accessor binding, so the broker reads values from the namespace alone; the viewer role reaches every secret’s metadata in the project (its annotations, its versions’ state and its IAM policy), never a value. A secret is served while its versions/latest is ENABLED, encrypted with the key BROKER_SECRETS_KMS_KEY_NAME names (replication.automatic.customerManagedEncryption; a secret on Google’s default encryption is refused not-on-agent-secrets-key); one with an expire_time is scheduled for deletion and left out, as a Secrets Manager secret in its recovery window is, and a grant made before it no longer reads its value, though Secret Manager still would until that time. Every Secret Manager call but reading a value counts against the project’s read quota, 600 a minute, shared by everything in the project (reading a value has its own 90,000). The broker’s worst case against it is the sum of: a reload every five minutes, one listing plus one call per secret (a 200-secret namespace, about 40 a minute); the public reread route, two each, bounded across all callers of each process at one reread every N² × 1 second, where N is the pods that process counted at its last beat and never fewer than two — one every 4 seconds at two pods, one every 9 at three, one every 16 at four — which holds the whole fleet at the same 120 metadata reads a minute whatever N is (see Running more than one pod); each session’s rereads for names it was not served, two each, at most one per session every ten seconds; and value releases, a kind secret’s among them, one per distinct secret per 30 seconds (two for a secret whose latest version is disabled), since the broker trusts its last check of a secret, readable or not, for that long (so a delete made outside agent-secrets secret stops releases at most 30 seconds after Secret Manager’s own metadata shows it, which Google documents as eventually consistent; agent-secrets secret delete rereads the name, and the broker then releases nothing more). A busy fleet of sessions requesting unknown names can still spend the quota, and other workloads in the same project draw on it too, so give the agent secrets a project of their own. On the metadata calls the broker retries an UNAVAILABLE answer and never a RESOURCE_EXHAUSTED one, which would only spend more of that quota; reading a value keeps the client library’s own retry of both, against its separate quota. On Secret Manager the agent-secrets key gates no caller: Secret Manager’s service agent makes the KMS calls, so writing a secret onto the key and reading it need no KMS permission, and being on the key is the namespace’s convention, not a second permission check as it is on AWS. Secret Manager makes the KMS calls itself, so the broker needs no KMS permission; the Secret Manager service agent needs roles/cloudkms.cryptoKeyEncrypterDecrypter on the key.
Kind secrets	A github-app secret needs outbound HTTPS access to api.github.com, and a github-owners tag naming the GitHub accounts it mints for, which the deployment sets when it creates the secret. A google-login secret needs outbound HTTPS access to oauth2.googleapis.com. A google-service-account secret needs outbound HTTPS access to iamcredentials.googleapis.com, an entry in BROKER_SERVICE_ACCOUNT_SECRETS binding it to one account, and the broker’s own Google identity holding roles/iam.serviceAccountTokenCreator on each bound account and on no other, since the broker mints the token by impersonating that account; without the role Google refuses the mint and the session is answered 502 MINT_FAILED. The deployment’s binding, not the secret’s stored value, chooses which account the broker impersonates. A secret bound to no account, and a stored value naming an account other than the bound one, release nothing and are logged class=binding, so a secret someone writes with another person’s account in it mints no token, and a secret the deployment never bound mints none either. What the secret’s own tags still decide is who may ask for it (requesters) and what the token can do (scopes), so the write access to a bound secret is an administrator’s (People who manage secrets). The broker reads the stored value for each release, as it reads any secret, and returns only the minted token. It also reads each kind secret’s value once for each version, kind and shape a pod lists, to check it, and refuses a secret whose value its kind’s minter could not mint from (the value read). It needs no IAM permission to mint a GitHub App or a Google login value, and BROKER_SERVICE_ACCOUNT_SECRETS is the one BROKER_* setting any of the three kinds needs. A mint GitHub or Google refuses answers the session 502 MINT_FAILED, naming their reason, and logs broker: mint failed (below). A call to GitHub or Google that answered a 5xx or a rate limit, or got no answer, is attempted up to three times before that (the mint retry). A narrowing the secret cannot serve, an account its tag does not list or one GitHub refuses, answers 403 NARROWING_OWNER_NOT_LISTED or 422 NARROWING_REFUSED instead and logs broker: narrowing refused.
A local stand-in (development only)	BROKER_FAKE_SECRETS_FILE names a JSON file the broker reads in place of the store, in Secrets Manager’s shape whatever BROKER_SECRETS_STORE says, each secret’s kms_key_id naming the agent-secrets key as the selected store configures it (BROKER_SECRETS_KMS_KEY_ARN, or BROKER_SECRETS_KMS_KEY_NAME with the GCP store); the configuration reference gives its format.
Dispatch	Dispatch’s server calls the broker’s account routes as its own ServiceAccount: it lists the requests and machine logins waiting on the signed-in person, each secret request with its page on the broker (approve_url), which its Inbox links to, and relays a person’s machine-login decisions and revokes. Set Dispatch’s DISPATCH_AGENT_SECRETS_URL to the broker’s URL and DISPATCH_AGENT_SECRETS_ACCOUNT_TOKEN_FILE to the path of its projected token for BROKER_K8S_OIDC_AUDIENCE, and list that account in the broker’s BROKER_API_ACCOUNTS. Without the URL, Dispatch hides its credential pages. Deploy a broker with the approval page before a Dispatch whose Inbox links it (Approval sign-in).
The sign-in provider	A person’s browser signs in to the broker at the provider (Google, under the Workspace’s internal consent screen), and the broker reaches its token endpoint and keys over HTTPS at a sign-in, and its discovery document at the first sign-in each process serves (kept once a read succeeds) and at the client check it takes at startup and every hour after. A discovery that could not be read answers the page 503 naming the issuer and logs approval sign-in: reach the provider; machines are served as before, and GET /signin/status reports issuer_unreachable (What the sign-in is doing).
Kubernetes (optional)	To enroll pods, and to let a service log in with its token (BROKER_SERVICE_LOGINS), BROKER_K8S_OIDC_ISSUER and BROKER_K8S_OIDC_AUDIENCE name the cluster’s service-account token issuer and the audience the pods’ and the services’ projected tokens carry. The broker fetches the issuer’s discovery document at startup and refuses to start if it cannot.
Google Cloud (optional)	To let machines trade their Google identity tokens, BROKER_GOOGLE_OIDC_AUDIENCE names the audience of those tokens, and BROKER_VM_SERVICES, BROKER_VM_PROJECTS, BROKER_VM_INSTANCES and BROKER_VM_EXCHANGE_ACCOUNTS describe each fleet. The broker fetches Google’s discovery document at startup and refuses to start if it cannot.
Envoy (optional)	With BROKER_ENVOY_URL, the broker tells a waiting agent session, best effort, when its pending request expires.
The model judge (optional)	Outbound HTTPS access to the model endpoint BROKER_JUDGE_MODEL_URL names, which the broker reaches with its machine’s own Google identity token for BROKER_JUDGE_MODEL_AUDIENCE, and to the Dispatch BROKER_JUDGE_DISPATCH_URL names, which it reads with the projected ServiceAccount token in the file BROKER_JUDGE_DISPATCH_TOKEN_FILE names, read at call time and never at boot, so a rotation needs no restart. It reads one issue and that issue’s spec document there and writes nothing. With those settings unset the broker makes neither call and runs no judge (Concepts).

Network and trust

• Agents, their helpers and launchers, and Dispatch’s server reach the broker at BROKER_PUBLIC_URL. Every signed proof names that URL plus the request’s path, so clients’ AGENT_SECRETS_URL must be exactly it, and a proxy in front of the broker must not change the path. The broker checks each proof against BROKER_PUBLIC_URL, never against the Host, X-Forwarded-Proto or X-Forwarded-Host a request arrives with: a proxy may rewrite Host or terminate TLS without setting X-Forwarded-Proto, and a client that sends those headers itself changes nothing. That holds for the machine login’s token endpoint (POST /v1/oauth/token) too, whose DPoP proofs name BROKER_PUBLIC_URL/v1/oauth/token.
• The model judge adds two outbound calls. The broker reads one Dispatch issue and that issue’s spec document at BROKER_JUDGE_DISPATCH_URL, with its own projected ServiceAccount token, and writes nothing there: no ask, no comment, no status. It calls the model at BROKER_JUDGE_MODEL_URL, presenting its machine’s own Google identity token for BROKER_JUDGE_MODEL_AUDIENCE. Each call carries the broker’s own identity and no session’s, and neither carries a secret value: what reaches the model is the secret’s name and rules, the request’s reason and the issue, as data to judge.
• A person approves a secret request in a browser at BROKER_PUBLIC_URL, which must therefore be an https address people’s browsers reach: the page’s two cookies, __Host-broker-session and __Host-broker-signin, are Secure and host-only. A decision is accepted only from a form on the broker’s own origin (Origin, or Sec-Fetch-Site: same-origin), bound to that browser’s sign-in and to the requests its page printed, so a page on another origin, Dispatch’s included, cannot make one even when both are under one parent domain. The page runs no script and cannot be framed.
• A caller listed as dispatch holds an approval credential: its ServiceAccount token can approve a machine login and revoke grants and machine logins in any person’s name, the broker recording whichever approver that caller names, so whoever holds that account’s token can decide those as anyone. It cannot decide a secret request. List Dispatch’s own account under that name and nothing else, never one an agent’s pod runs as. An account listed as listener reads tokens back and decides nothing, which is the whole of what that name carries.
• Behind a reverse proxy, set BROKER_TRUSTED_PROXY_HEADER so the machine-login, token-endpoint and secret-reread rate limiters see each caller’s address rather than the proxy’s. The token endpoint (POST /v1/oauth/token) has its own per-address bucket in each process, a burst of 60 refilled at two a second, far above a machine’s poll every five seconds and refresh about once a day; a refused request is 429 invalid_request (RATE_LIMITED: …) with Retry-After.
• The settings and reread routes take no credential. GET /v1/settings answers the store, the namespace prefix and, per store, the agent-secrets key’s ARN with its AWS account and region, or the Cloud KMS key’s name with the project number. POST /v1/secrets/{name}/reread makes the broker read one secret from the store at once, and answers whether it now serves it, as which kind (kind: github-app or google-login, absent for a stored secret) and, if not, why. Neither releases a value. What a reread does tell any caller who can reach the broker is whether a name under the namespace exists, the kind it is served as, and, when the broker leaves it out, which of its reasons applies — absent, or a refusal such as owner-tag-missing or not-on-agent-secrets-key. That is more than UNKNOWN_SECRET, which says only that no agent secret has the name, says it only to an enrolled session, and never separates a secret that does not exist from one the broker refuses; it is what anyone allowed to list the namespace reads from the secrets’ own names and tags (or annotations). Rereads are limited twice, by limits each pod derives rather than by settings (RereadOverallFor and RereadLimitFor in packages/envoy/internal/broker/api/limits.go): the broker as a whole takes one reread every N² × 250 ms on the default store, burst 10, however many addresses the rereads come from — each one takes the policy’s writer lock while it reads the secret, so that lock, not the caller, is what this limit bounds — and each source address gets a burst of 30, refilled one every twice that. N is how many broker pods the pod serving the call counted at its last beat, never fewer than two, so a two-pod fleet allows one reread a second and an address one every 2 seconds, a three-pod fleet one every 2.25 seconds and an address one every 4.5, and a four-pod fleet one every 4 seconds and an address one every 8. On Google Secret Manager, where a reread spends two reads of the project’s shared metadata quota, the same arithmetic runs over a budget four times smaller: one reread every N² × 1 second broker-wide, burst 10, and an address’s own one every twice that — 4 and 8 seconds at two pods, 16 and 32 at four. A reread counts against either limit only when both let it run, so one address flooding past its own limit leaves the broker-wide one to everyone else. Past either, the broker answers 429 RATE_LIMITED with a Retry-After header naming the limit that refused, at the rate that limit runs at now: at two pods, 1 second for the broker-wide limit, 4 seconds for Secret Manager’s, and for an address’s own twice whichever of those this process mounted — 2 seconds on the default store, 8 under Secret Manager. Each of these buckets is one process’s, and the broker-wide pair is derived from what the whole fleet spends of the secret store, which is not the same as what it spends in HTTP rereads: one reread the broker accepts costs the store one read per pod, because the pod that served it rereads the secret and then tells every other pod to reread it too. So the fleet reads the store replicas × this rate × replicas times a second, and each bucket is the fleet’s whole budget divided by the replica count squared. At two pods: 2 pods × 1 reread/s × 2 store reads = 4 store reads a second, which is what one pod alone was allowed, and on Secret Manager 2 pods × 0.25 reread/s × 2 store reads × 2 metadata reads = 2 a second, the same 120 metadata reads a minute as before. Read those as the arithmetic rather than as the number two: a third replica divides the fleet’s budget by nine, not by four, and the pod does that division itself. Their burst is not divided at all: it stays at the budget’s ten. A burst is a depth and what it has to admit is one person’s write burst, a hard floor that the replica count does not change, while what it guards — each pod’s own transient, its own burst plus every peer’s — is a soft cost, latency rather than quota, spent once and re-bounded immediately by the rate. Dividing it by the replica count put the depth at 3 for a three-pod fleet, under the floor of four that agent-secrets’ own end-to-end test measures, so ten now carries six tokens of headroom at any fleet size. The per-address bucket is derived from the bucket this process mounts rather than chosen: it refills at half that bucket’s rate, so it always takes at least two addresses to hold the fleet’s budget at zero, on either store and without tightening a deployment for a bucket it does not mount. That ordering matters because the route takes no credential and a refusal costs a caller nothing, so a caller that merely keeps succeeding at the shared refill rate would otherwise starve the fan-out for everyone. Each pod re-derives all three whenever the number of pods it counts changes, and logs the rate it moved to (broker reread rate, Health and logs). A reread one pod’s notification drives on another spends nothing from either bucket, since the pod that served the original call already paid for it — which is what makes the figures above the whole of the fleet’s store traffic rather than a floor under it (Running more than one pod). Every other limiter is per process and unchanged, so a fleet of N pods allows N times each: this per-address reread bucket, the two machine-login buckets, the token endpoint’s, and each enrollment’s budget of rereads for names it was not served. agent-secrets secret sends one reread after each write it makes, from the person’s own address, so writing more than ten secrets at once meets the broker-wide burst. When a limit refuses it, the write stands in the secret store and the CLI exits 1 saying so (manage a secret); only the narrowing of who may read that secret waits for the next token.
• Agents never see the broker’s database or the secret store; whoever can write the database can forge a record, so its access control is part of the broker’s. Whoever can tag (or annotate) a secret under the namespace decides who gets it, so that write access is part of the broker’s too.

People who manage secrets

The broker writes no secret. A person or an agent creates, changes and deletes agent secrets with agent-secrets secret (manage a secret), which calls the store itself under whatever cloud credential its environment gives it - AWS, the sign-in --profile, AWS_PROFILE or the SDK’s default chain names; GCP, an access token in CLOUDSDK_AUTH_ACCESS_TOKEN, else the Application Default Credentials, of any type, else the gcloud login - so that cloud’s own IAM is what decides who may change which secret, and with the tags (AWS) or annotations (GCP), who gets it. The CLI asks the broker only for its settings (GET /v1/settings) and, after each write, for a reread. Before anything else it refuses, on AWS, a sign-in in another account than the agent-secrets key’s (GCP needs no equivalent: every GCP call already names the project explicitly, so the API itself refuses one in another project). It refuses no credential for the kind of identity it is, a person’s or a machine’s: the cloud’s IAM is the boundary, so grant the namespace’s write permissions to no machine identity that should not write the agent secrets. Whoever holds a credential needs each form’s permissions on the namespace’s secrets, and the tables below say what each request carries that a policy’s conditions can match.

One kind is not a person’s to write. A google-service-account secret the deployment binds to an account (BROKER_SERVICE_ACCOUNT_SECRETS) releases a token that acts as that account, and the secret’s own tags say who may ask for it and which scopes its token carries, so whoever may write those tags decides who can get a token as that account. Give every write of a bound secret, its value’s as much as its tags’, to an administrator: name those secrets in the access you grant, since the setting names them, rather than testing the kind tag. On AWS that is a condition on the secret’s own name under the namespace (its ARN, and secretsmanager:Name) for create, PutSecretValue, TagResource and UntagResource; on GCP, where an IAM condition cannot test an annotation’s value, keep secretmanager.secrets.update and secretmanager.versions.add on those secrets for administrators alone.

The binding names a secret by its name alone, so a bound name with no secret behind it is open to whoever may create a secret in the namespace: they create it under that name, own it, list themselves, store the bound account as its value, and the broker mints that account’s token for them. So keep every bound name held by the administrator’s own secret. Create the secret before its binding is written, and remove the binding before the secret is deleted, since a deleted secret’s name is free again once its recovery window ends. On GCP, keep secretmanager.secrets.create on the namespace for administrators alone unless each bound name always holds its secret.

A google-service-account secret under a name the deployment binds to nothing mints nothing, so a person who creates one of their own under such a name, with any account in its value, is answered MINT_FAILED rather than a token. The requesters tag bounds who may ask for a secret; the binding is what bounds which account it mints for.

AWS Secrets Manager

Form	Permissions	What a condition can match
every form	sts:GetCallerIdentity, which needs no permission
list	secretsmanager:ListSecrets (on *: it takes no resource)
show	secretsmanager:DescribeSecret	The secret’s own tags (aws:ResourceTag/owner, aws:ResourceTag/tier)
create	secretsmanager:CreateSecret and secretsmanager:TagResource, since it tags the secret as it creates it; kms:GenerateDataKey and kms:Decrypt on the agent-secrets key	The new secret’s name under the namespace (its ARN, and secretsmanager:Name); the request tags: aws:RequestTag/owner (a person’s email, from --owner me or --owner <email>, or shared) and aws:RequestTag/tier always; aws:RequestTag/kind and aws:RequestTag/mint-service-account (the rollback guard, kind-secret, written beside every kind) with --kind; aws:RequestTag/github-owners (GitHub logins separated by spaces) with --kind github-app, which needs it; aws:RequestTag/requesters (emails and service names separated by spaces) with --requesters; aws:RequestTag/scopes (OAuth scope names separated by spaces) with --scopes; aws:RequestTag/admin (true) with --admin; aws:RequestTag/shape (token or document) with --shape; aws:RequestTag/max-grant with --max-grant; and aws:RequestTag/default-grant with --default-grant (aws:TagKeys is the tags sent, so a policy that limits it must allow owner, tier, kind, mint-service-account, github-owners, requesters, scopes, admin, shape, max-grant and default-grant); the key, named by its ARN (secretsmanager:KmsKeyArn); the TagResource check made on the new secret sees the requested tags as its own (aws:ResourceTag/owner, aws:ResourceTag/tier), so a policy that reserves a shared secret’s tags for administrators still needs a statement that admits a shared create (request and resource owner both shared, the same tier). A secret created with --owner shared and --tier agent is served to every agent session without approval, so only a person whose access allows a shared create can create one.
set	secretsmanager:DescribeSecret, since a set reads the secret’s tags for the kind it checks the value as and the shape it stores the value under before it writes; secretsmanager:PutSecretValue; kms:GenerateDataKey on the agent-secrets key	The secret’s own tags (aws:ResourceTag/owner)
retag	secretsmanager:DescribeSecret, and secretsmanager:TagResource, or secretsmanager:UntagResource for --kind none, --github-owners none, --requesters none, --scopes none, --admin none, --shape none, --max-grant none or --default-grant none. A retag that finds another writer raced it needs both secretsmanager:TagResource and secretsmanager:UntagResource, whichever it sent, since the restore re-sends the tags it read and removes any the race added. A retag that changes the secret’s kind, or the shape of a secret that keeps a kind, also reads the secret’s value to check it as that kind’s minter reads it, before it writes: secretsmanager:GetSecretValue on the secret and kms:Decrypt on the agent-secrets key. The check needs that read and the retag does not: where Secrets Manager refuses the read with AccessDeniedException, the retag writes its tags and prints one line on standard error that says the value was not checked, and the broker then reads the value itself and refuses the secret value-malformed when its minter could not mint from it. Any other failure of the read refuses the retag, which then writes nothing. A retag never prints the value	A retag reads the secret’s tags, sends one request, then reads them back. A tag request carries owner and tier, the unchanged one re-sent as the secret holds it, plus kind and mint-service-account (the rollback guard) when the command sets a kind, github-owners when it names owners, requesters when it names who may ask, scopes when it names scopes, admin when it marks the secret an administrator’s, shape when it sets a shape, max-grant when it sets one, and default-grant when it sets that; a policy that limits aws:TagKeys must allow each of them. An untag request’s aws:TagKeys is the tags none removes: --kind none removes kind and mint-service-account together, and --github-owners none, --requesters none, --scopes none, --admin none, --shape none, --max-grant none and --default-grant none each remove the tag they name; a retag that removes a tag changes nothing else, so a refused call leaves the secret as it was. A retag that would leave a github-app secret without github-owners, or an admin or google-service-account secret without a max-grant of an hour or less, is refused before any write. When the read-back finds another writer’s change left a github-app secret without github-owners, the retag restores the tags it read: a tag request re-sending them, as the secret held them, and an untag request for any tag the race added; then it exits 1. The secret’s own tags let a policy allow a person to tag their own secret (aws:ResourceTag/owner their email) to themselves or shared, and reserve a shared secret’s tags for administrators
delete	secretsmanager:DeleteSecret	The secret’s own tags; secretsmanager:RecoveryWindowInDays is 30, and secretsmanager:ForceDeleteWithoutRecovery is never set
restore	secretsmanager:RestoreSecret	The secret’s own tags

Secrets Manager makes the KMS calls itself, on the caller’s behalf, so the agent-secrets key’s policy can allow them only through Secrets Manager (kms:ViaService of secretsmanager.<region>.amazonaws.com, with kms:CallerAccount). The email in aws:RequestTag/owner is lowercase: --owner me writes the Identity Center session name in lowercase, and --owner <email> lowercases the email it names, so a condition that compares it with a principal tag holding the caller’s email matches only where that tag is lowercase too. No form prints a value. One form reads one: a retag that changes the kind, or the shape of a secret that keeps a kind, calls GetSecretValue to check the value and then drops it (the row above). delete and restore are permissions of their own: an access that grants a person every other form on their own secret but not these refuses those two with AccessDeniedException.

GCP Secret Manager

Form	Permissions
list	secretmanager.secrets.list on the project and secretmanager.versions.get on each secret (to tell whether its versions/latest is ENABLED)
show	secretmanager.secrets.get and secretmanager.versions.list, secretmanager.versions.get (for its versions and whether versions/latest is ENABLED)
create	secretmanager.secrets.create and secretmanager.versions.add. Automatic replication encrypts with the CMEK key through Secret Manager’s own service agent, not the person’s identity: that service agent (service-<project number>@gcp-sa-secretmanager.iam.gserviceaccount.com) needs roles/cloudkms.cryptoKeyEncrypterDecrypter on the key, granted once at setup, not per person
set	secretmanager.secrets.get, since a set reads the secret’s annotations for the kind it checks the value as and the shape it stores the value under before it writes, and secretmanager.versions.add
retag	secretmanager.secrets.get and secretmanager.secrets.update: owner and tier annotations in one call, plus kind and mint-service-account (the rollback guard) when the command sets a kind, github-owners when it names owners, requesters when it names who may ask, scopes when it names scopes, admin when it marks the secret an administrator’s, shape when it sets a shape, max-grant when it sets one, and default-grant when it sets that; or, for none, the annotations it removes (--kind none removes kind and mint-service-account together), and nothing else. A retag that changes the secret’s kind, or the shape of a secret that keeps a kind, also reads the value of versions/latest to check it as that kind’s minter reads it, before it writes: secretmanager.versions.access on the secret. The check needs that read and the retag does not, and people who add versions and change annotations often hold no accessor role: where Secret Manager refuses the read with PermissionDenied, the retag writes its annotations and prints one line on standard error that says the value was not checked, and the broker then reads the value itself and refuses the secret value-malformed when its minter could not mint from it. Any other failure of the read refuses the retag, which then writes nothing. A retag never prints the value
delete	secretmanager.secrets.update (sets expire_time to 30 days out)
restore	secretmanager.secrets.update (clears expire_time)

Unlike AWS’s tags, a GCP IAM condition cannot test an annotation’s value, so GCP has no per-secret counterpart to AWS’s “a person may tag only their own secret” conditions above, and nothing stops a write from reaching another person’s secret: whoever is granted secretmanager.secrets.update on the namespace can retag any secret under it to any owner, tier or kind, shared secrets included, and whoever is granted secretmanager.versions.add can replace any secret’s value under it. On Secret Manager, grant write under the namespace - secrets.create, secrets.update and versions.add alike - to administrators only, and let people ask an administrator to write on their behalf.

Health and logs

GET /healthz answers 200 {"status":"ok"} while the broker can reach Postgres, and 503 DATABASE_UNAVAILABLE when it cannot. From the signal that stops it, it answers 503 SHUTTING_DOWN instead, for the whole drain, while every other route goes on being served exactly as before. Use it as a readiness probe and never as a liveness probe: a liveness probe reads a draining pod as a dead one, so the kubelet would kill the container in the middle of the drain and drop the requests the drain exists to finish (Running more than one pod).

The broker writes its log to stderr as one JSON object per line, in the shape Cloud Logging reads as a structured entry with a severity of its own. Each object carries time (RFC 3339 with nanoseconds, in UTC), severity (INFO, WARNING or ERROR: the broker writes nothing below INFO), message, and each field of the line as a member of its own. A duration is written as Go writes one, such as "every":"1m30s". This is the line a broker writes once it serves, here on the local stack:

{"time":"2026-11-14T02:26:30.105447105Z","severity":"INFO","message":"broker listening","addr":"127.0.0.1:46052"}

A line records a change of state, not a pass of a loop. The five-minute policy reload, each of the request sweeper’s sweeps, the hourly sign-in client check and the read of a kind secret’s value each write one line when they start to fail and one when they succeed again. The policy writes a line when a secret’s state changes, not at each load. A standing state is what the metrics hold (Metrics).

The table names each line by its message, then its fields as field=<value>, and says the severity it is written at (WARN is written WARNING). The lines worth alerting or searching on:

Log line	Meaning
broker stop budget drain=<drain> shutdown=60s total=<their sum>	The first line of the run, before the database or the secret store is reached for: the whole budget a stop spends, BROKER_DRAIN_SECONDS and the 60 seconds in-flight requests get after it. total is what a deployment’s terminationGracePeriodSeconds has to cover, and with the defaults the line reads drain=5s shutdown=60s total=65s (Running more than one pod).
database sign-in method=<method>	At startup, before the broker reaches for AWS or its database: rds-iam when each connection signs in with an RDS IAM token the broker mints, password when it signs in as BROKER_DATABASE_URL and libpq’s defaults say (a password in the URL, PGPASSWORD, a passfile, or none for a server that trusts it).
secret store store=<store> credentials=<source>	At startup, once the store is built: aws-secrets-manager under the AWS SDK’s credential chain (aws-sdk), gcp-secret-manager under Application Default Credentials (application-default), or either name over BROKER_FAKE_SECRETS_FILE (fake secrets file).
approval sign-in on issuer=<issuer> client_id=<client id> domain=<domain>	At startup: the approval page signs people in through that client, holding each person to that Workspace domain (Approval sign-in). The client’s secret is never logged.
broker: the sign-in client check state=<state> issuer=<issuer> error=…	At WARN, from the check each pod takes at startup and every hour after, when it finds a state other than ready that the pod did not log last: the provider refused the broker’s client (state=client_rejected, so nobody can sign in) or could not be asked (state=issuer_unreachable). A check that finds the state the pod logged last, or finds the client ready with no such line before it, logs nothing, so a provider that stays away for a day is one line. Every check writes its answer to GET /signin/status.
broker: the sign-in client check finds the client ready again issuer=<issuer> was=<state>	At INFO: the first check that finds the client ready after the line above. was is the state that line named.
broker: the sign-in client check was not written state=<state> error=…	At WARN: a check could not write its answer to Postgres after the check before it wrote, so GET /signin/status answers the last answer written. The checks after it log nothing more until one writes.
broker: the sign-in client check is written again state=<state>	At INFO: the first check that writes its answer after the line above.
approval sign-in login=<email> / approval sign-out login=<email>	A person signed in to, or out of, the approval page.
approval sign-in refused …	At WARN: a sign-in made no session, the line saying why: the provider refused it, its state or return path was not the sign-in’s, the code exchange or ID token failed, or the person is outside BROKER_SIGNIN_DOMAIN or unverified.
approval page decision login=<email> record=<record id> decision=<approve or deny> outcome=<outcome>	A person decided a secret request on the page; outcome is what became of it (granted, for one).
approval sign-in: reach the provider issuer=<issuer> error=…	At ERROR: the provider’s discovery document could not be read at a sign-in, and the page answered 503 naming the issuer.
broker listening addr=<host:port>	Startup finished: configuration, migrations and the first read of the namespace all succeeded, and every port is bound: the API’s, at this address; the airlock’s proxy port, where the deployment has one (airlock proxy listening addr=<host:port>, just after); and the metrics port (the line below).
broker metrics listening addr=<host:port>	After broker listening, and after airlock proxy listening where the deployment has an airlock: the metrics port is bound at this address (Metrics). A pod that cannot bind it does not start, and logs broker: fatal naming the address.
broker: fatal error=…	Startup refused; the error names the variable or dependency. The process exits 1.
agent secret policy refused name=<secret's name in the store> reason=<reason> owner=<owner tag> first_load=<true or false>	At ERROR, when a secret becomes one the broker refuses to serve, and when it is refused for another reason than before, with was=<the reason before>: one line at the change, not one for each read of the store. A pod’s first load writes one for each secret it refuses, each with first_load=true. A load or a reread that finds a secret refused for the reason it had writes nothing, and broker_unservable_secrets (Metrics) holds every secret refused now. name is the secret’s whole name in the store, such as production/agent-secrets/deel-api-key. owner is its owner tag, and empty for owner-tag-missing and owner-tag-malformed. Reasons include owner-tag-missing, owner-tag-malformed, tier-tag-missing, tier-tag-malformed, name-malformed, service-owner-human-tier, not-on-agent-secrets-key, no-current-value, kind-unknown, github-owners-missing, github-owners-malformed, requesters-malformed (a requesters list holding a comma, naming nobody, carrying an email with a capital letter, or naming a service the deployment does not register), max-grant-required (an admin secret, or a google-service-account one, with no max-grant or one longer than an hour), requesters-not-owner (a google-service-account secret whose requesters are not its one person owner, or that lists a service it may not: any service at agent tier, and any service at all on a secret BROKER_SERVICE_ACCOUNT_SECRETS binds to a dev-<name> account), admin-malformed (an admin tag set to anything but true), admin-tier (an admin secret that is not human tier), admin-requesters (an admin secret with no requesters list), scopes-malformed (a scopes list holding a comma, naming no scope, naming one twice, or holding an entry that is no scope name, a whole https://www.googleapis.com/auth/… URL among them), scopes-kind (a scopes tag on a secret that is not a google-service-account one), shape-malformed (a shape tag set to anything but token or document), max-grant-malformed, default-grant-malformed and value-malformed (a github-app, google-login, google-service-account or azure-login secret whose stored value that kind’s minter could not mint from: the value read).
agent secret policy no longer refused name=<secret's name in the store> reason=<the reason it had> owner=<owner tag> now=<servable or absent>	At INFO, when a secret the broker refused is refused no longer: now=servable when the broker serves it, and now=absent when the store no longer holds it (deleted, or scheduled for deletion).
agent secret policy load failed; previous policy kept error=…	At ERROR, in two cases. The five-minute reload of the namespace failed (the secret store out of reach, or refusing the broker; a Secret Manager failure names its gRPC status code alone, never the status message), and the broker keeps serving the policy from its last good read: the first failed reload of a run writes the line, and the reloads after it write nothing until one succeeds, which writes agent secret policy load succeeded again. Or, with name=<NAME> (the name a request asked for), the request named a secret the broker did not serve and the broker’s reread of that one secret failed, so the request is refused UNKNOWN_SECRET: one line for each such request. A read in flight when the broker shuts down, and a request’s reread its own caller abandoned, are not one of these and log nothing.
agent secret policy load succeeded again	At INFO: the first reload that succeeds after the line above began a run of failed ones.
agent secret value read failed; its state kept name=<secret's name in the store> version=<version> error=…	At WARN: the broker could not read a kind secret’s stored value to check it (the value read), for a permission, a timeout or the store’s own failure, which error names by its call and its class and never by a byte of the value. A stored value that is not text is no failed read: it is refused value-malformed. The secret keeps the state it had, but for a value-malformed refusal of an older version: a failed read of a version the store listed since ends that refusal, which writes agent secret policy no longer refused. Each load after it reads the value again and writes nothing more until a read succeeds.
agent secret value read succeeded again name=<secret's name in the store> version=<version>	At INFO: the first read of that secret’s value that succeeds after the line above.
broker: answer refused answer_id=<id> status=<status> code=<code> method=<method> route=<route> caller_class=<class> caller=<caller> secrets=[<names>] secrets_count=<n>	At WARN, once for each refusal of the API port and of the airlock’s proxy port: every answer from 400 to 499 but a 429 and the two answers a machine gets while it polls a device login, slow_down and authorization_pending, which are counted and not logged. It is written before the answer is sent, and it names the answer id the refused caller was given (Refusals and the answer id). It lists the first secret names the request named, up to 32 KiB of names as the line writes them (JSON-escaped, each with its two quotes), counts them all in secrets_count, and adds secrets_omitted=<n> when it left names out, since Cloud Logging refuses an entry over 256 KiB. An answer Go’s HTTP server writes itself before any of the broker’s code runs, such as a 400 for a malformed request line, has no answer id and writes no line.
broker: <operation> failed error=…	A request failed with a 500 or 503; the line carries the cause the response does not. A reread whose own request ended before it finished - its caller went away, or it reached the 45-second bound - is not one of these: it answers 503 REQUEST_ENDED and logs nothing, since anyone can end a request. One whose request ended only while it read the value of a github-app, google-login, google-service-account or azure-login secret has already put the secret’s new tags in force on that pod, so it still tells the other pods to reread the secret, as a 200 does.
broker: mint failed name=<NAME> kind=<kind> class=<class> attempts=<attempts> error=…	At ERROR, for each release of a kind secret its minter could not make. class says whose fix it is: stored-value, the secret’s value is not a key or login the kind takes, so its owner stores a new one; binding, a google-service-account secret the deployment binds to no account, or to an account its stored value does not name, so the operator fixes BROKER_SERVICE_ACCOUNT_SECRETS — this one is not the secret’s owner to fix by storing a new value, and the mint is refused before any call reaches Google; refused, GitHub or Google answered and refused, such as a Google login that needs a new sign-in or a GitHub App with no installation on a listed account, so its owner signs in again or fixes the App; upstream, GitHub or Google could not be asked, answered no usable reply or rate limited the call, or answered a narrowed GitHub token holding more than the narrowing asked, which the broker revokes at GitHub, so check the broker’s egress and the upstream’s status. attempts is how many times the broker sent the call the mint failed at, 1 to 3 (the mint retry), and 0 for a failure that was no call’s, such as a stored value the minter could not read. A stored-value failure also refuses the secret as value-malformed, so its grants release nothing until a value that decodes is stored. A narrowing the caller got wrong is never this line: see broker: narrowing refused. The error never holds the stored value, a token or a value from GitHub’s answer. The session is answered 502 MINT_FAILED with the same error.
broker: a stored value's trailing line ending was removed name=<NAME>	At INFO, once for each value of a release the shape rule changed, so twice for one release that changed two values: both the value a session is handed and the input a minter reads pass that rule. The line carries the secret’s name and no part of any value. The secret’s shape tag says how the broker reads its value, and a secret carrying no tag is read by the value itself. An owner whose value the rule reads wrongly answers the line with one retag, agent-secrets secret retag <NAME> --shape document.
broker: GitHub did not revoke a narrowed token it minted wider than asked; it lapses at its expiry app=… installation=… expires_at=… error=…	At WARN, beside a mint failed … class=upstream line: GitHub answered a narrowed token holding more than asked, the broker refused it, and GitHub did not confirm the revoke the broker then sent. A 401 or 404 to the revoke’s first attempt is GitHub refusing it and writes this line, since nothing in the revoke can have revoked a token minted moments before. A 401 or 404 to a retried attempt counts as confirmed and writes no line, since an earlier attempt may have revoked the token and lost its answer; an earlier attempt GitHub rate-limited, which revoked nothing, cannot be told apart from that. No caller got the token; GitHub honours it until expires_at, at most an hour. Check the broker’s route to GitHub, as for the line it follows.
broker: narrowing refused grant=… enrollment=… name=<NAME> error=…	At WARN, for each release whose narrowing of its GitHub App tokens names what the secret cannot serve: an account its github-owners tag does not list, refused before any minter runs (403 NARROWING_OWNER_NOT_LISTED), or a narrowing GitHub answered 422, such as a repository the installation does not reach (422 NARROWING_REFUSED); a 422 that says it is a rate limit is an upstream mint failed instead. Nothing was released. The caller fixes its narrowing; the App and the secret are not at fault, so this line needs no action unless one session repeats it.
broker: grant log …	At INFO, once for each request decision, each request.judged row and each grant end. Every line carries the event’s kind, its actor, whichever of the enrollment, request and grant ids the row names, and raw, the audit row’s own detail; it carries no audit id and no timestamp, since the line is written once the row has committed and the log keeps its own time. The rest is what the writer of that line had in hand, which differs by line: a request.created line adds the session and its person, the service the broker verified, the service the session’s credential claims (claimed_service), the secrets and how each was decided, the state, the approver, the lifetime, the reason and the detail; a request.granted or request.denied line adds the session and its person, the claimed service, the state, the approver, who decided, the lifetime, the reason and the detail, and neither the verified service nor the secrets; a request.cancelled or request.expired line adds the state, the detail and, for a cancel, who decided; a request.judged line, written for every judge run that reached a verdict, its referrals and its failed checks included, adds the session and its person, the claimed service, and the grant where the verdict granted one; a grant.revoked or grant.withheld line adds nothing more, except the session’s person when a person’s revoke ended it. A reader joins a line to its request’s own request.created line by request_id for the rest, or reads that row from GET /v1/grant-log, which joins the tables and holds every field. The line is the copy a deployment’s log pipeline keeps of the rows that route answers from the audit table, so a closed period’s decisions can be read without the database (the audit record). A release (grant.used) and an enrollment row write no such line.
broker sweeper: ended an enrollment whose lease lapsed enrollment_id=… kind=… runtime_id=… slot=… lease_expired_at=… grants_revoked=… requests_cancelled=… left_dormant=…	A session stopped renewing (a pod that is gone, a box whose agent-secrets renew stopped) while a broker was serving; the sweep ended it, revoking its grants and cancelling its pending requests. A lease that ran out while no broker was serving is not ended (when no broker is serving).
broker sweeper: end lapsed enrollments sweep=lapsed-enrollments error=…, and the failure lines of the six other sweeps	At WARN, when one of the seven sweeps the sweeper runs every BROKER_SWEEP_SECONDS fails after its run before did not. That sweep logs nothing more until it succeeds, which writes broker sweeper: a sweep that failed succeeded again, so a database that stays away for an hour is two lines for each sweep and not one for each run. Each sweep’s line has its own message and sweep: broker sweeper: end lapsed enrollments (lapsed-enrollments), broker sweeper: end dormant enrollments (dormant-enrollments), broker sweeper: expire pending requests (pending-requests), broker sweeper: expire pending machine logins (pending-machine-logins), broker sweeper: delete spent OAuth state (spent-oauth-state), broker sweeper: delete expired browser sessions (expired-browser-sessions) and broker sweeper: delete the placeholders of grants that have ended (ended-grant-placeholders). A sweep the broker’s shutdown cuts short logs nothing.
broker sweeper: a sweep that failed succeeded again sweep=<sweep>	At INFO: the first run of that sweep that succeeds after its failure line. sweep names it as that line did.
broker: gave back the time no broker was serving to every lease still running when the last broker served gap=… last_serving_at=… enrollments=…	At INFO, on the first beat after more than three sweeps in which no broker beat: every broker process was down, or none could reach the database. Each enrollment not revoked whose lease was still running at last_serving_at had its lease moved on by gap, but no further than BROKER_LEASE_SECONDS from now; enrollments counts them. Their sessions renew as before.
broker: the serving clock could not beat; it retries every interval and logs again once it beats error=… timed_out=…	At WARN: this process could not record its beat, usually because the database is out of reach. A beat may take at most one BROKER_SWEEP_SECONDS; timed_out=true says it ran out of that (its deadline, or a lock or statement past it), as when another transaction holds this process’s row, while the other brokers go on beating. It retries every BROKER_SWEEP_SECONDS and logs broker: the serving clock beats again once it does.
broker reread rate replicas=<n> every=<interval> burst=10	At INFO, at this pod’s first beat of the serving clock and then each time the number of pods it counts changes: the limits it now serves POST /v1/secrets/{name}/reread at. every is how often the broker-wide bucket takes one reread back, and an address’s own bucket takes one back every twice that. A pod that has not beaten yet serves at the two-pod rate (Running more than one pod).
broker reread rate above the replica count this sizing supports replicas=<n> every=<interval> store_reads_at_burst=<reads>	At WARN, beside the line above when a pod counts more than four: the rate is derived the same way, but the burst of 10 is not divided, so a burst at that count reads the secret store store_reads_at_burst times at once, which this sizing was not chosen for. A pod logs it once, when its own count first goes above four, and again only if that count changes while it stays above four. A fleet that stays larger has outgrown the budget limits.go divides, which is where to change it.
broker: pod enrollment renewed without its projected token enrollment=… pod_uid=… slot=…	At WARN: a pod enrollment renewed without its token, and no enrollment naming its pod UID has ever renewed with one, so the broker accepted it; for that pod a token copied out of it still keeps an enrollment alive. It is expected from a pod on a worker image whose agent-secrets renew predates --pod-token-file. A pod on the current image sends its token from its first renewal, so a line naming such a pod means a holder of an enrollment key for it is renewing outside the cluster before the pod’s own first token renewal. List pod images below and replace an older-image pod. If its pod UID is absent or every image is current, end the login that made the line’s enrollment, which ends every enrollment it made, once its subject shows whether it is a Legion daemon’s (the renew token check gives the query and both ways to end it); never pick a machine-login row by its machine name.
broker: pod enrollment renewal refused: projected token required enrollment=…	At WARN: a pod enrollment renewed without its token after an enrollment naming the same pod UID, this one or another, had renewed with one, and was refused. One cause is someone holding its key, possibly enrolled under another machine login with a token copied out of the pod, renewing outside the cluster. The other is an old-image pod refused because someone else marked its UID first: whoever holds an unexpired copy of an old-image pod’s token and a launcher credential that enrolls pods can enroll a key of their own on that UID and renew it with the copy, which marks the UID, and from then on the real old-image pod, whose renewer sends no token, is refused here until its lease lapses. That is weaker than what the copy already allowed, which was to enroll and renew as the pod. The refused enrollment’s pod UID names the pod: list pod images (below). If that pod runs an older image, the copy’s rows are the marked rows of that pod UID: end each one’s login once its subject shows whether it is a Legion daemon’s (the renew token check), never picking a machine-login row by its machine name, and replace the pod with a current image. Otherwise the copy is the line’s own enrollment, as that section says. A current-image renewer that cannot read its token file sends nothing at all, so the broker logs nothing for it; agent-secrets renew prints ERROR: enrollment <id> cannot read its pod token instead, which the worker shim relays into the pod’s log. Its lease and grants end one BROKER_LEASE_SECONDS after its last valid renewal.
broker: machine session renewal refused: its machine login has ended enrollment=…	At WARN: a machine’s own session asked to renew while the launcher credential its Google identity token minted was no longer live — revoked, past the hour’s expiry that token gave it, or its account no longer listed for its service in BROKER_VM_SERVICES. The renewal is answered 401 ENROLLMENT_NOT_LIVE and the session lapses within one BROKER_LEASE_SECONDS. It is expected after a fleet is taken out of the settings and after a machine stops trading. A line for a machine that is still running and still listed means something holding that session’s key is renewing after its box’s login has ended, which is what this refusal is for: see a copied machine token.
broker draining drain=<drain>	SIGTERM or SIGINT: from here on GET /healthz answers 503 SHUTTING_DOWN while every other route is served and the listener stays open, for BROKER_DRAIN_SECONDS. A second signal ends the wait at once, logged as broker: a second signal ended the drain early.
broker shutting down	The drain is over: the broker stops accepting, and the requests still in flight get 60 seconds to finish. That is the server’s own WriteTimeout — the 45-second request deadline plus the 15 seconds a handler needs to write its answer — because it is the longest exchange the broker allows a request past its headers: one admitted in the drain’s last instant still gets its whole deadline. Shutdown closes idle connections at once, and one that has gone five seconds without sending its headers counts as idle, so the budget covers the requests in flight rather than every connection open.
broker: listening for the other pods' secret rereads channel=…	The reread fan-out’s receiving half has its LISTEN connection. The boot waits up to five seconds for this line before it binds, so on a healthy pod it comes just before broker listening; after that it reappears on every redial.
broker: a Postgres listener has no connection; redialing channels=… error=… attempt=… wait_ms=…	At WARN, from pglisten: this pod has lost its LISTEN connection and is redialing with a backoff that reaches thirty seconds. A pod logging this is outside the fan-out for every name — losing the LISTEN does not by itself fail readiness, so the pod can go on passing /healthz and serving the whole time while a retag another pod rereads does not reach it until its own five-minute reload. It often does fail readiness at the same time, since /healthz pings Postgres and a database the pod cannot reach is the commonest reason the LISTEN went: a 200 beside this line is the case to worry about, not the normal one. It is not the only way a pod falls outside: the three rows below leave one behind for at least one name, and of the four only the other pods were not told to reread a secret and a notification's reread of a secret did not run can be matched to a secret. Repeating lines mean the database is not carrying LISTEN sessions for this pod; one or two around a failover are expected.
broker: binding before the reread listener has had an answer waited=5s	At WARN: the boot’s five-second wait for that LISTEN ran out, so this pod began serving before it could hear its peers. It serves normally and the listener keeps redialing; what it loses is the notifications sent in the gap, with its own five-minute reload as the backstop. A pod that logs this takes no catch-up load either, there being no subscription to order a read after.
broker: serving without the boot's catch-up policy load, so a retag another pod served while this one was starting may not be in force here until its five-minute reload error=…	At WARN: once its LISTEN is in place a booting pod reads the namespace a second time, so that a retag a peer served while it was still loading is in force before it binds. That read failed. The pod serves what its first load gave it and catches up at its five-minute reload; the first load succeeded, so this is never a pod with no policy at all.
broker: the other pods were not told to reread a secret secret=… error=…	At WARN, not a failed request: the reread this pod was asked for read the secret’s tags and answered 200, or 503 REQUEST_ENDED when its request ended during the value read alone, and only the Postgres NOTIFY that tells the other pods failed. The retag is in force here and reaches the others at their own next reload.
broker: a notification's reread of a secret did not run secret=… sender=… error=…	At WARN, on no request at all: another pod told this one to reread a secret and the reread failed. This pod goes on serving the tags it last read until its own five-minute reload.
broker: a second signal arrived while the broker was still starting up, so it is stopping without a drain: it had not begun serving when that signal was read	At ERROR: two signals reached a process that had not finished booting, so it exited non-zero without draining. The line says what was true when the second signal was read, which is what the process can observe; a boot that bound its socket in the two statements after that read logs it too. The lines above it say how far the boot got. Expected only when someone or something stopped a pod twice during startup.
broker: signalled while starting up, before it had bound anything stage=…	One signal reached the process before it bound its socket, so it finished the stage it was in and stopped cleanly, serving nothing and running no drain. stage is the first policy load or the catch-up policy load.

Each request is bounded to 45 seconds. Request bodies are JSON of at most 1 MiB.

Refusals and the answer id

Every answer of the API port and of the airlock’s proxy port carries an answer id, 32 hexadecimal characters, in its X-Broker-Answer-Id header. An answer from 400 to 499 carries it in its body too: a JSON one as its answer_id member, a refusal of the airlock’s proxy port at the end of its text as (answer <id>), and a refusal page of the approval page as an id to quote. A 429 is no refusal, so two bodies carry no id: the text of a 429 that the Azure token endpoint answered and the airlock’s proxy port forwards, and the approval page’s 429 page. The header of each still carries it. The exception on both ports is the answers Go’s HTTP server writes itself before any of the broker’s code runs, such as a 400 for a malformed request line, an invalid header name or value, or a missing, repeated or malformed Host header, a 431 for headers over its limit, a 417 for an Expect header it does not know, the 400 Client sent an HTTP request to an HTTPS server on the airlock’s bound connections, and the 200 to OPTIONS *. They carry no answer id and write no refusal line. agent-secrets ends each error the broker answers with the id, a refusal or a failure, so whoever was refused has a thing to quote to whoever runs the broker:

$ agent-secrets NO_SUCH_SECRET -- true
agent-secrets: no agent secret has this name (UNKNOWN_SECRET) (answer ed22ee417e5933b629d7182fc52ce9e6); asked the broker AGENT_SECRETS_URL names (AWS Secrets Manager)

For a refusal, the broker writes its line, broker: answer refused, before it sends the answer, so that id finds the line:

{"time":"2026-11-14T02:28:40.768728789Z","severity":"WARNING","message":"broker: answer refused","answer_id":"ed22ee417e5933b629d7182fc52ce9e6","status":400,"code":"UNKNOWN_SECRET","method":"POST","route":"POST /v1/requests","caller_class":"session","caller":"be3624bc-7b88-4e9c-8657-ebbf4b2651d7","secrets":["NO_SUCH_SECRET"],"secrets_count":1}

Its fields:

• answer_id: the id the answer carries.
• status: the answer’s HTTP status, from 400 to 499.
• code: the code the answer names. The broker reads it from the answer: its JSON body’s code, else the body’s OAuth error, else the code the approval page or the airlock’s proxy port names for that refusal, else HTTP_<status>, which is what a path no route serves (404) and a method a route does not take (405) are written as. A code is a letter and then at most 63 letters, digits and underscores, and anything else is written as HTTP_<status>.
• method and route: the request’s method, at most its first 256 bytes, and the API route that took it, such as POST /v1/requests. The route is empty where no route took the request, and on the airlock’s proxy port.
• caller_class and caller: how the broker learnt who sent the request, and who that is.
  • public: a route that takes no credential, or a path no route serves. caller is empty, but on the OAuth token route, where it is the OAuth client the token request named.
  • unauthenticated: a route that takes a credential, refused before the broker knew who sent the request: a missing or bad credential, a listed caller on a route its name does not carry, a refusal that the two routes that decide a credential request answer before they read the credential, such as DECIDE_ON_BROKER_PAGE, and a refusal of the approval page or its sign-in to a browser with no live sign-in. caller is empty.
  • launcher: a machine’s launcher, by the id of its launcher credential.
  • session: an agent session or a pod, by its enrollment id.
  • account: a caller BROKER_API_ACCOUNTS lists, by the name it gives that caller.
  • service: a service logged in with its own token, as <service>@<namespace>/<name>, the ServiceAccount it presented.
  • sign-in: a person signed in to the approval page, by their login.
  • airlock: a call through the airlock’s proxy port, as <grant>@<enrollment> once its proxy password proved a grant, and empty before.
• secrets: the secret names the request named, as a list, in the request’s order. It is empty when the broker refused the request before it read a name. It lists the first of them up to 32 KiB of names as the line writes them, JSON-escaped and each with its two quotes, each name cut to at most its first 256 bytes, and leaves the rest out, since Cloud Logging refuses an entry over 256 KiB.
• secrets_count: how many secret names the request named, the ones secrets left out included.
• secrets_omitted: how many names secrets left out, and only when it left any out.
• correlation_id: the request’s X-Correlation-Id, at most 256 bytes of its printable ASCII, and only when the request sent one.

A refusal of the airlock’s proxy port has more fields:

• host: the host the call was for, at most its first 256 bytes.
• reason: the proxy’s own words for the refusal, such as not allowed by rule, at most its first 256 bytes, since some of them name the host the caller asked for.
• path: the request’s path, without its query, at most its first 256 bytes.
• issue: the issue key the grant’s reason named, once a grant is proved.
• slot: which of the secret’s values the call used, where the line names one.

A refused call to a bound host writes this line in place of its call line, airlock proxy: a call to a bound host, so each such call still writes one line (Run the broker locally shows both). A 429 that the Azure token endpoint answers and the proxy forwards is no refusal: it keeps its call line at INFO, which names its answer id and its status, and writes no refusal line.

Each field whose text the caller chose is cut to its first 256 bytes, so whatever a caller sends, each line stays within one Cloud Logging entry. On the refusal line those are the method, the host, a reason that names the host, the path and the correlation id; on the call line, the method, the host and the path, which is all of the caller’s text it names.

No refusal line holds a secret’s value or a placeholder, and the API port’s line holds no message text either: it names the code. Each answer, a refusal or not, also counts in broker_answers_total (Metrics), which is the one record of the API port’s 429 and of a device grant’s poll answers.

On the API port, an answer that no refusal line records names its id in no line: a 429, the two answers of a device login’s poll, and every answer of 500 and above. agent-secrets prints the id of a 429 and of a failure as well. For a 502 MINT_FAILED, read the broker: mint failed line instead (Health and logs). It names the secret as name and holds no answer id, so find it by that name and the time of the answer.

The value read

The broker reads the stored value of each github-app, google-login, google-service-account and azure-login secret once for each version, kind and shape of it that a pod lists and has not checked, at the pod’s first load too, and decodes the value as that kind’s minter does. It drops the value at once: no value is written, logged, counted or kept. It reads no value of a secret that has no kind this way. Each read counts as read="value" in broker_policy_store_reads_total (Metrics). So a pod reads each such secret once after it starts, and once more for each new version, kind or shape, and never at each reload.

A value the decoder refuses makes the secret unservable as value-malformed from that load, before any mint: one agent secret policy refused line and one broker_unservable_secrets series. So does a stored value that is not text, which no minter can read either: an empty value or one stored as binary alone on Secrets Manager, and a payload that is not valid UTF-8 on Secret Manager. That is a refusal of the value and no failed read, so it writes no agent secret value read failed line and is not read again at each load. A live grant of the secret releases nothing (GRANT_NOT_LIVE).

The refusal holds while the store lists that version, until a read under a new kind or shape decodes the value. A new version, or a retag of the kind or the shape, is read and decoded again, so when agent-secrets secret set stores a value that decodes, the reread that follows the write serves the secret again and writes one agent secret policy no longer refused line. A mint whose minter cannot read the value refuses the secret in the same way, so a pod whose load could not read the value still refuses the secret at its first mint. That refusal is of the version, kind and shape the release read the value under. When a load or a reread has listed another one since, a retag’s among them, the mint refuses nothing, and the broker decodes the value under the new one.

Any other read that fails, for a permission, a timeout or the store’s own failure, refuses nothing, and each load reads the value again. The secret keeps its state, but for a value-malformed refusal of an older version: a failed read of a version the store listed since ends that refusal, so the broker serves the secret again, and a mint whose minter cannot read the new value refuses it again. The first failure writes agent secret value read failed; its state kept, and the read that succeeds after it writes agent secret value read succeeded again. The read needs no permission the broker lacks, since a release reads the same value (What it depends on). The version these lines name is the store’s own: Secrets Manager’s version id, Secret Manager’s version number, and local-<n> over BROKER_FAKE_SECRETS_FILE.

The mint retry

A minter’s call to GitHub or Google is made again when another attempt can succeed: an answer of 500 or above, a rate limit (a 429, GitHub’s X-RateLimit-Remaining: 0, or a body naming a rate limit), or a call or a read of its answer that failed while the caller’s request was still open. The broker does not retry a 4xx that is no rate limit, a call whose request its caller ended, an answer larger than the minter reads, or a 2xx or 3xx answer other than the one the call asked for.

• Each call gets at most 3 attempts, each under a 5-second timeout.
• Before the second attempt the broker waits a time drawn at random from 0 to 2 seconds, and before the third, from 0 to 4 seconds.
• A wait the upstream names takes the place of that draw: Retry-After, in seconds or as an HTTP date, or GitHub’s X-RateLimit-Reset when X-RateLimit-Remaining is 0. The broker waits it in full when it fits in what is left of the mint’s 10 seconds (the next item) and of the request’s deadline, and otherwise ends the mint with no further attempt. A rate limit that names no wait counts as 60 seconds, so it ends the mint. A wait longer than an hour counts as an hour, the longest GitHub asks for, since GitHub counts its primary rate limits per hour; that ends the mint too. A negative wait counts as none.
• One mint waits at most 10 seconds in all, over every call it makes. A GitHub mint makes several: a page of installations, a token for each owner, perhaps one more without actions, perhaps a revoke.
• Every retry of the process takes one from a budget of 60, which gains one back each second. When the budget is spent, a call that fails ends its mint at once.

Each attempt counts in broker_mint_attempts_total (Metrics), and a mint that fails logs broker: mint failed with the attempts its last call made. A caller that gives up closes its request, and that stops the retry at once.

Metrics

The broker serves its metrics in the Prometheus text format at GET /metrics, on a port of its own: BROKER_METRICS_LISTEN_ADDR, by default 127.0.0.1:13382. The API’s port serves no /metrics, and the metrics port serves nothing else: every other path answers 404, and a method other than GET or HEAD answers 405. The boot binds it after the API’s port and the airlock’s proxy port and logs broker metrics listening addr=<host:port>, and a pod that cannot bind it does not start. A draining pod goes on answering scrapes through its drain, and the port closes with the others.

The default address answers on the pod itself alone, so a scraper in another pod needs an address it can reach, such as 0.0.0.0:13382. Let that scraper alone reach the port, and never route it through the address callers reach the broker at: broker_unservable_secrets names the store id and the owner of each secret the broker refuses, which tells whoever reads it whose secrets the store holds and which of them are broken. No metric holds a value, a token or a name a caller typed: each code is one the broker chose, and a secret’s name reaches a label only from the store’s own listing.

Metric	Type	Labels	What it holds
broker_answers_total	counter	outcome, code, caller_class	Each answer the broker’s code writes on the API port and on the airlock’s proxy port; the answers Go’s HTTP server writes itself before that code runs are not counted (Refusals and the answer id). outcome is served (below 400), refused (a refusal the broker: answer refused line records), rate_limited (a 429, and the device grant’s slow_down), pending (the device grant’s authorization_pending, a machine’s poll while its person has not decided) or failed (500 and above). code is a refusal’s code, as its line names it, or the code the airlock’s proxy port names for a failure of its own, such as UPSTREAM_UNREACHABLE. Every other answer’s code is HTTP_<status>. caller_class is as the refusal line names it. A call the proxy carried to a service counts as served, with the service’s own status as its HTTP_<status>.
broker_mint_attempts_total	counter	kind, outcome, attempt	Each attempt of each call a minter makes to GitHub or Google (the mint retry). kind is the secret’s kind. outcome is ok, upstream (an answer of 500 or above, no answer, or an answer it could not use), rate_limited, refused (a 4xx that is no rate limit) or cancelled (the caller’s request ended). attempt is 1, 2 or 3.
broker_unservable_secrets	gauge	secret, reason, owner	One series of value 1 for each secret the broker refuses to serve now. secret is the secret’s whole name in the store, as the agent secret policy refused line names it, reason is that line’s reason, and owner its owner tag, empty for owner-tag-missing and owner-tag-malformed. A secret that becomes servable, or leaves the store, is gone from the next scrape, and a reread of a name nobody created adds no series.
broker_policy_store_reads_total	counter	read	Each call the policy makes to the secret store, a failed one too: list for a load of the whole namespace, describe for one secret’s reread, and value for one kind secret’s value (the value read). It is how to see that this pod read the store: the broker logs no line for a read whose secrets it serves. On Secret Manager each of them spends the project’s read quota. A list is one ListSecrets for each page and one GetSecretVersion for each secret, and a describe is one GetSecret and one GetSecretVersion. A value is one GetSecret, or none when this pod read that secret in the 30 seconds before, and then the AccessSecretVersion that reads the value, which counts against Secret Manager’s separate access quota.

Each pod counts its own answers, attempts and reads, from zero at its start. Run the broker locally shows a scrape.

Running more than one pod

Nothing in a broker process is private to it but its cache of the policy and its rate-limit buckets, so a deployment may run several identical pods behind one address. A sign-in to the approval page is held in the browser’s cookie until its callback and its session in Postgres, so any pod finishes any sign-in and honours any session. What it has to set, and what it gets:

The stop budget its grace period must cover. On SIGTERM the broker drains for BROKER_DRAIN_SECONDS (5 by default, 0 to 120; a second signal ends the wait at once): it answers GET /healthz with 503 SHUTTING_DOWN while serving every other route exactly as before, and keeps its listener open, because a pod is removed from its Service’s EndpointSlice concurrently with the kubelet beginning termination rather than before it, so for a while after the signal new connections are still routed to it. Only then does it stop accepting and give the requests still in flight 60 seconds. Those 60 are the server’s own WriteTimeout, the 45-second request deadline plus the 15 seconds a handler needs to write its answer, because that is the longest exchange the broker itself allows a request past its headers: one admitted in the drain’s last instant still gets its whole deadline. A connection that has sent no headers for five seconds is closed as idle rather than waited for; its header read has its own ten-second ReadHeaderTimeout. So terminationGracePeriodSeconds must be at least any preStop pause, plus BROKER_DRAIN_SECONDS, plus 60 seconds — 65 seconds with the defaults and no pause. A preStop pause and the drain do the same job and are both paid out of the one grace period, so size the period for their sum, or set the pause to 0 and let the drain do it alone. Past the grace period the kubelet kills the container whatever the broker still believes it is allowed, and the in-flight requests the budget exists for are dropped anyway. The broker logs the whole figure at boot, so nobody adds it up by hand: broker stop budget drain=5s shutdown=60s total=65s.

/healthz as the readiness probe, and as nothing else. A liveness probe on it has the kubelet kill the container in the middle of the drain, which is what the drain exists to prevent (Health and logs).

One more Postgres connection per pod, on a session-mode path. The fan-out below listens on a connection of its own, held for as long as the process runs and outside whatever pool_max_conns sizes, since a LISTEN ends when a pooled connection is recycled. Count one extra connection per pod against the database’s connection limit, and give the broker a session-mode path to Postgres: a transaction pooler (pgbouncer in transaction mode, RDS Proxy) hands each transaction whichever backend is free and carries no long-lived LISTEN. A pod that cannot hold the connection still boots and still serves; its listener redials for as long as it runs.

A reread that reaches every pod, draining ones included. POST /v1/secrets/{name}/reread rereads the secret on the pod the load balancer picked and then, once that pod has put the tags it read in force (its 200 answer, or a 503 REQUEST_ENDED whose request ended during the value read alone), tells every other pod serving the same database to reread the same name, over Postgres LISTEN/NOTIFY on a channel of its own. So one call puts a retag in force across the fleet within seconds rather than at each pod’s own next five-minute reload, and it needs no infrastructure beyond the database the broker already has. A pod that is draining is still one of those pods: it goes on being sent requests for the whole drain, so its listener and its five-minute reload both run until the shutdown has answered the last request or spent its 60 seconds, not until the signal. A pod waits for its first successful LISTEN before it binds, up to five seconds, so an address the load balancer can reach is normally an address a retag can reach; a pod whose LISTEN has not succeeded by then logs a warning and serves anyway, its listener still redialing. The exception, plainly: while a pod’s listener is redialing it hears nothing, and nothing is caught up when it reconnects, so that pod goes on serving what it last read until its own five-minute reload. The promise is therefore “within seconds, except while a pod’s listener is redialing, where the next five-minute reload is the backstop”. A reread a notification drives spends no rate-limit token: the pod that served the original call already paid for it.

The reread limits are three numbers each pod derives from one budget, not one number halved. One reread the broker accepts costs the secret store one read on every pod — the pod that served it, and each peer the fan-out tells — so the fleet’s store traffic is replicas² × the per-pod rate, and the two broker-wide buckets’ rate is the fleet’s whole budget divided by that: at two pods one reread a second, or one every 4 seconds against Google Secret Manager, which leaves the store exactly the 4 reads a second, and Secret Manager the 120 metadata reads a minute, that one pod alone was allowed. At three pods it is one every 2.25 seconds, or one every 9; at four, one every 4 seconds, or one every 16. Each pod does that division itself: its serving clock counts the pods beating in broker_serving at each beat, a count below two counts as two, and a pod whose count changed logs broker reread rate and moves both of its buckets without emptying or refilling either (Health and logs). Their burst is not divided at all and stays at ten, because a burst is a depth and what it admits is one person’s write burst, which the replica count does not change, while what it guards is a transient on each pod’s own writer lock — latency, spent once, and re-bounded by the rate. A burst momentarily lets the fleet accept replicas × burst rereads and each costs a read on every pod, so the store sees replicas² × burst: 40, 90 and 160 reads at two, three and four pods, twice that in metadata reads under Secret Manager (80, 180, 320), which is 8, 18 and 32 seconds of its 600 a minute — transients, re-bounded immediately by the rate. That cost grows with the square of the replica count while the rate is already divided by it, so four pods is the largest count this sizing was chosen for. Above it the rates are derived the same way and every pod logs a warning naming the count and the store reads a burst then costs, because a fleet that stays that large wants its budget re-derived against the quota rather than carrying ten forward unexamined. The per-address bucket is derived from them in turn — it refills at half the rate of the bucket this process mounts, 2 seconds on the default store and 8 under Secret Manager at two pods — so no single address can hold the fleet’s budget at zero on a route that takes no credential. Every other limiter is per process, so a fleet of N pods allows N times each (Network and trust).

A rolling deploy counts one pod more than the Deployment runs. The pod a surge adds beats beside the pod it replaces, and a pod that has stopped goes on being counted until its row is three sweeps old, so a roll of two pods holds the fleet at the three-pod rate for the length of the roll and can reach the four-pod rate while a stopped pod’s row is still live. Nothing has to be set for that: each pod follows the count it sees and goes back to the two-pod rate within a few sweeps of the roll finishing. A roll of a four-pod fleet counts five, which is past what this sizing was chosen for, and each pod logs the warning above once, when its count first goes above four.

A roll serves two generations of the broker on one database, so a schema change that one of them cannot read is staged over two releases. The per-pod machine login is the current one. This release’s migration adds the two indexes it needs and drops none, so the pods still running the previous image go on serving every token login and every machine token exchange through the index they resolve their own statement against; nothing has to be ordered or held, and a roll of this release is an ordinary roll. The release after it drops that index, and the per-pod login — a launcher credential for each pod of a ServiceAccount rather than one for the account — begins there. Each broker reads which of the two schemas it is on at every token login, so an image from either release serves either schema: a roll forward and a roll back both work, and a pod of this release goes on serving while the next release’s pod migrates the database beneath it.

Before you roll a broker, read the lease refusal. A broker whose BROKER_LEASE_SECONDS is too short for its BROKER_SWEEP_SECONDS exits at boot instead of serving, and that rule is stricter than it was: it walks each renewer’s own retry schedule rather than counting two thirds of the lease, so the shortest lease it accepts is 91 s at a sweep of 1 or 5 seconds, 105 s at 10, 225 s at 30 and 360 s at 60. A pair below that boots a broker of an earlier release and does not boot this one — a 60-second lease at the default sweep, or anything from 271 s to 359 s at a sixty-second sweep — so raise BROKER_LEASE_SECONDS to at least the figure for your sweep and roll that first. The refusal prints the figure itself (when no broker is serving).

The renew token check

A pod enrollment is not required to renew with its projected token yet. The broker ratchets each pod UID instead: every enrollment starts unmarked, its first renewal carrying the pod’s token marks it, and once any enrollment naming a pod UID, live or ended, is marked, a renewal without the token from every enrollment naming that pod, whichever machine login made it, is refused POD_TOKEN_REQUIRED. A pod on a worker image whose agent-secrets renew sends the token (--pod-token-file) marks its UID on its first renewal, so the copied-token gap closes for every pod UID a current-image pod has renewed under. Until that first renewal a copied token is used without refusal: seconds, since the Legion daemon logs in with its own token at every start and enrolls a pod as soon as it is up. A pod that ends before it enrolls never renews with its token, so its UID stays open as an old-image pod’s does. A pod on an older image never marks its UID: it renews as before, each renewal logged as the first WARN line above, and the gap stays open for it until the release that requires the token. A pod UID names one pod for ever, so a mark refuses nothing of another pod, and it keeps refusing a copy between the pod’s role generations and after the pod is gone.

Every deploy order keeps every pod renewing: an older broker ignores the token a newer renewer sends, and this broker accepts a renewal without one from a pod no enrollment of which has sent one. That is liveness, not exposure. A grant lasts until its session ends, so a copy renewing an old-image pod’s UID keeps the grant one approval gave it for as long as it renews. So, before or right after the broker deploys, replace every old-image issue pod: a new worker image does not replace a running issue pod, so list the pods’ images to find them.

kubectl get pods -n <namespace> -l legion.dev/issue \
  -o custom-columns=POD:.metadata.name,UID:.metadata.uid,IMAGE:.spec.containers[*].image

Once that list shows no old-image pod, a token-less WARN line no longer names a pod of yours but a copy, and for either line the copy is the line’s own enrollment: a pod on the current image renews with its token, so its own rows are the marked ones, and the line names the row that renewed without it. (Marked rows are a copy’s only when the refused pod runs an older image, as the projected token required row above says.) List every enrollment of the line’s pod UID with the machine login that made it; own is the line’s enrollment, and subject is the ServiceAccount a token login logged in as (none for a login a person approved through a code):

select o.id = e.id as own, o.id as enrollment, o.pod_token_renewed_at is not null as marked,
       o.revoked_at is null and enrollment_lease_live(o.lease_expires_at) as live,
       c.id as credential_id, c.service, c.subject, c.host,
       to_char(c.created_at at time zone 'UTC', 'YYYY-MM-DD HH24:MI:SS') || ' UTC' as issued_at,
       (select ev.login from credential_request_events ev
        where ev.record_id = r.id and ev.event = 'approved') as approved_by
from enrollments e
join enrollments o on o.kind = 'pod' and o.runtime_id = e.runtime_id
join launcher_credentials c on c.id = o.launcher_credential_id
left join credential_requests r on r.id = c.record_id
where e.id = '<the line''s enrollment>'
order by o.created_at;

Whose login the line’s enrollment came from is its subject. A Legion daemon logs in only with its own ServiceAccount’s token, so its login’s subject is an account BROKER_SERVICE_LOGINS names for its service; a login whose subject is none, or an account the setting does not name, is never the daemon’s. A login with no subject is one a person approved through a code: a service login from before token logins (it lasts at most BROKER_LAUNCHER_CREDENTIAL_SECONDS after its approval), since a person’s own machine login enrolls no pod. Revoke it on Dispatch’s machine-login page (/credentials/machine), signed in: find the row whose Credential is the query’s credential_id, check that its Approved by is the query’s approved_by and that its Issued time, in the tooltip, is the query’s issued_at (the tooltip shows that second in the browser’s time zone), then click Revoke and confirm. That ends every enrollment the login made (end a machine’s login), and the broker records the signed-in person as who revoked it. Never pick the row by its machine name: a login’s host is the name its machine reported.

A line whose own enrollment’s subject is a daemon’s account names an enrollment made under that daemon’s one credential: by the daemon itself, renewed by someone holding the enrollment’s key in that pod (the role whose namespace holds it), or by someone holding a token for the daemon’s ServiceAccount. Each login under the account is in the audit table, with the pod the token was bound to; the daemon’s own token is bound to the daemon’s own pod, so a login whose pod_uid is another pod’s, or none (a token kubectl create token minted), did not use the daemon’s own token unless you ran the daemon off the cluster with such a token:

select at, detail->>'pod_uid' as pod_uid, detail->>'key_thumbprint' as key, detail->>'expires_at' as expires_at,
       detail->>'token_expires_at' as token_expires_at
from audit
where kind = 'launcher_credential.token_login' and detail->>'credential_id' = '<the query''s credential_id>'
order by at;

and compare pod_uid with the daemon’s pod (kubectl get pods -n <daemon namespace> -l <the daemon's labels> -o custom-columns=POD:.metadata.name,UID:.metadata.uid). A login under the daemon’s account moves the credential to its own key, so a copy’s login also turns the daemon’s next call into a refusal and a fresh login of its own, logged by the daemon as agent-secrets: the token login failed; retrying or logged in with the daemon's ServiceAccount token; a run of those while nothing restarted the daemon is a copy fighting it. Replacing the line’s pod makes the daemon revoke its enrollment as it lets the process go. If the pod is gone and the enrollment is still live (the daemon logs supervise: agent-secrets: enrollment not revoked after every attempt; the lease ends it, which a copy that keeps renewing never lets happen), or a copy holds the daemon’s token, end the daemon’s login at once: revoke it on Dispatch’s machine-login page by its Credential (the query’s credential_id, Approved by reading No one: and the daemon’s account), which ends every pod it enrolled, and, while a copy holds the daemon’s token, remove its entry from BROKER_SERVICE_LOGINS and redeploy the broker so neither can log in again. Find and remove what copied the token before you restore the entry: the broker refuses a token that lives longer than an hour from its issue, so the copy’s token ends at most an hour after it was minted, and the latest token_expires_at above among the logins that are not the daemon’s says when. Pods the revoke ended stay without secrets until they are replaced.

To end a login you know by its code instead, such as a person’s machine or a service login from before token logins, find its row by that code:

select c.id as credential_id, c.service, c.host,
       to_char(c.created_at at time zone 'UTC', 'YYYY-MM-DD HH24:MI:SS') || ' UTC' as issued_at,
       (select ev.login from credential_request_events ev
        where ev.record_id = r.id and ev.event = 'approved') as approved_by,
       c.revoked_at is not null as revoked
from credential_requests r
join launcher_credentials c on c.record_id = r.id
where r.kind = 'launcher_credential' and r.code = '<the code>';

and revoke the row whose Credential is that credential_id, as above.

The release that requires the token from every pod is safe to deploy only when all three of these hold:

• this query answers 0. A pod’s row is unmarked until its first renewal, seconds after it enrolls, so the query leaves out rows made in the last minute, which a busy fleet always holds:

  select count(*) from enrollments
  where kind = 'pod' and revoked_at is null and pod_token_renewed_at is null
    and created_at < now() - interval '1 minute';

• the broker has logged no broker: pod enrollment renewed without its projected token line for one full BROKER_LEASE_SECONDS.

• the command above lists no issue pod on a worker image older than the one whose renewer sends the token. An old-image issue pod between role generations holds no enrollment, so it counts as 0 and logs nothing, yet its next phase enrolls an unmarked row.

That release refuses every pod renewal without a token, so it also removes what only the ratchet needs: the pod-UID lookup in Renew (podTokenRatcheted) and its partial index enrollments_pod_token_renewed.

A copied machine token

A machine’s Google identity token logs in as that machine for up to an hour after it was minted, wherever it is presented. A copy’s trades show in the audit record as launcher_credential.token_login rows for that machine’s subject from more than one key:

select at, detail->>'credential_id' as credential_id, detail->>'key_thumbprint' as key,
       detail->>'token_expires_at' as token_expires_at
from audit
where kind = 'launcher_credential.token_login'
  and detail->>'subject' like '<account>%'
order by at;

A copy’s access token is bound to the copier’s own key, so it cannot spend the machine’s token, and its credential and enrollment are its own: a credential is keyed by the key its proof presented, so a copy’s trade mints a credential beside the machine’s rather than moving the machine’s onto the copier’s key, and ending one ends nothing of the machine’s. What it does get is whatever the fleet’s service may read. Revoking the machine’s login on Dispatch’s machine-login page ends every enrollment that login made, but not the copied token: within its hour the copy trades again under a login of its own. To end it for certain, take the machine out of the fleet’s settings and redeploy the broker: change BROKER_VM_INSTANCES so the glob no longer matches the machine’s name, or, for the whole fleet, remove the account’s entries from BROKER_VM_SERVICES, BROKER_VM_PROJECTS, BROKER_VM_INSTANCES and BROKER_VM_EXCHANGE_ACCOUNTS. From then every trade and every call of that machine, the copy’s included, is refused. Find and remove what copied the token before you put the machine back: the latest token_expires_at above says when the last copied token ends, at most an hour after it was minted. Rotate any secret the copy may have read; the broker cannot take back a value it released.

The audit record

Requests, records, decisions and the audit trail live in the broker’s Postgres database (Concepts explains them). To read the latest events:

select at, kind, actor, enrollment_id, request_id, grant_id, detail
from audit
order by at desc
limit 50;

and the decisions on credential requests:

select cr.id, cr.kind, cr.approver, r.state as request_state, e.event, e.actor, e.at
from credential_requests cr
left join requests r on r.record_id = cr.id
left join credential_request_events e on e.record_id = cr.id
order by cr.created_at desc
limit 50;

A secret request’s state is its request’s (request_state), which is what the broker goes by: a request an older broker cancelled when its session ended carries no event on its record. A machine login has no request, and its events alone say how it was decided.

A decision the model judge made names judge:<model id> in the event’s actor and in the request’s decided_by, with no login beside it, and writes one request.judged row of its own. That row is there for every judge run that reached a verdict, the referrals and the failed checks included, so it is where a request the judge did not take says why (Concepts):

select at, actor, request_id, detail
from audit
where kind = 'request.judged'
order by at desc
limit 50;

No row any of them answers holds a secret value.

Running it locally

Run the broker locally starts a whole stack (Postgres, the broker on a fake secrets file, and its clients) with one script.