Summary
engram cloud bootstrap admin --issue-token creates the admin but fails to issue the token, leaving an active managed admin with no token and no way to issue one (bootstrap is one-shot; the JSON /admin/* and dashboard token-create routes require an already-valid managed admin token). On a fresh deployment this locks you out of the managed-admin API entirely.
Environment
- engram
v1.19.0 (also present in v1.18.0 — the relevant files are byte-identical between the two tags).
engram cloud serve on Postgres, with ENGRAM_CLOUD_TOKEN_PEPPER set (dedicated, ≥32 bytes, distinct from ENGRAM_JWT_SECRET).
Repro
engram cloud bootstrap admin --username atlas-broker --email atlas-broker@example.com --issue-token atlas-token
Output:
✓ Managed admin created: username=atlas-broker principal_id=4
engram: cloud bootstrap admin: create token with completion audit: cloudstore: auth audit insert failed: cloudstore: sensitive auth audit metadata is not allowed: issued_token
The admin row is created and committed; the token is not persisted.
Root cause
cmd/engram/cloud_bootstrap.go — cloudBootstrapCompletionMetadata(...) sets a metadata key issued_token (metadata["issued_token"] = issuedToken, ~line 261).- That metadata is attached to the completion audit event written by
CreatePrincipalTokenWithAudit.
- The auth-audit metadata validator rejects the event with
ErrSensitiveAuditMetadata ("sensitive auth audit metadata is not allowed") — the key issued_token trips the sensitive-key check (it contains token).
- The token persistence + its completion audit are documented as one atomic unit, so the whole token step fails (
fatal). But the admin creation (CreateFirstAdminHumanUser) is a prior, already-committed step, so it is not rolled back.
Net effect: an active admin with no token, and bootstrap admin now refuses (a managed admin already exists), so the token can never be issued for it via the CLI. The managed-admin JSON API / dashboard token-create both require a managed admin token to call (requireManagedAdmin) — a chicken-and-egg lockout.
Impact
On any v1.18.0/v1.19.0 deployment, the only documented way to obtain the first managed admin token (bootstrap admin --issue-token) is broken. If the bootstrapped admin is the only one, there is no supported recovery path (the docs reference "a documented recovery path" in docs/engram-cloud/troubleshooting.md but none is spelled out). We recovered only by hand-inserting a correctly-hashed token row directly into cloud_principal_tokens.
Suggested fixes
- Don't put an
issued_token key in audit metadata (rename to something the validator accepts, e.g. token_issued is likely also rejected — better: drop it, or encode it as a non-token key like included_credential: true), or allow this specific key in the audit-metadata validator.
- Make admin creation + token issuance atomic (roll back the admin if
--issue-token fails), so a failed --issue-token bootstrap is safely retryable instead of leaving a token-less active admin.
Workaround
Bootstrap without --issue-token (creates the admin cleanly — no completion-audit metadata is written when there are no grants and no token), then issue the token via a supported route once available, or via a direct DB insert as a last resort.
Summary
engram cloud bootstrap admin --issue-tokencreates the admin but fails to issue the token, leaving an active managed admin with no token and no way to issue one (bootstrap is one-shot; the JSON/admin/*and dashboard token-create routes require an already-valid managed admin token). On a fresh deployment this locks you out of the managed-admin API entirely.Environment
v1.19.0(also present inv1.18.0— the relevant files are byte-identical between the two tags).engram cloud serveon Postgres, withENGRAM_CLOUD_TOKEN_PEPPERset (dedicated, ≥32 bytes, distinct fromENGRAM_JWT_SECRET).Repro
Output:
The admin row is created and committed; the token is not persisted.
Root cause
cmd/engram/cloud_bootstrap.go—cloudBootstrapCompletionMetadata(...)sets a metadata keyissued_token(metadata["issued_token"] = issuedToken, ~line 261).CreatePrincipalTokenWithAudit.ErrSensitiveAuditMetadata("sensitive auth audit metadata is not allowed") — the keyissued_tokentrips the sensitive-key check (it containstoken).fatal). But the admin creation (CreateFirstAdminHumanUser) is a prior, already-committed step, so it is not rolled back.Net effect: an active admin with no token, and
bootstrap adminnow refuses (a managed admin already exists), so the token can never be issued for it via the CLI. The managed-admin JSON API / dashboard token-create both require a managed admin token to call (requireManagedAdmin) — a chicken-and-egg lockout.Impact
On any v1.18.0/v1.19.0 deployment, the only documented way to obtain the first managed admin token (
bootstrap admin --issue-token) is broken. If the bootstrapped admin is the only one, there is no supported recovery path (the docs reference "a documented recovery path" indocs/engram-cloud/troubleshooting.mdbut none is spelled out). We recovered only by hand-inserting a correctly-hashed token row directly intocloud_principal_tokens.Suggested fixes
issued_tokenkey in audit metadata (rename to something the validator accepts, e.g.token_issuedis likely also rejected — better: drop it, or encode it as a non-tokenkey likeincluded_credential: true), or allow this specific key in the audit-metadata validator.--issue-tokenfails), so a failed--issue-tokenbootstrap is safely retryable instead of leaving a token-less active admin.Workaround
Bootstrap without
--issue-token(creates the admin cleanly — no completion-audit metadata is written when there are no grants and no token), then issue the token via a supported route once available, or via a direct DB insert as a last resort.