ACME Governed AI Workforce — Administrator Guide

The complete administrator reference for deploying, configuring, and operating the governed, zero-trust AI agent workforce — identity and directory, agent lifecycle, policy, model routing, the runner, governance operations, audit, access and entitlements, observability, durability and DR, security hardening, troubleshooting, and a full environment / port / API reference.

Host placeholders: these guides use placeholder host names — acme-demo-host, acme-directory-host, acme-directory-internal, acme-soul-db-private, your-ops-ip — for the deployment's real addresses. The consoles are also reachable at their stable domains (console.asphodel.ai, pdp.tiresias.watch). Substitute the actual values from your operations record.

1. Introduction, audience, product overview, how to use this guide

This guide is the administrator's reference for the ACME governed AI workforce: a zero-trust platform that runs autonomous AI agents under continuous, provable governance. It covers deployment, day-to-day operation, policy authoring, identity linking, behavioural monitoring, audit, and troubleshooting. Read it end to end before your first production rollout; thereafter use it as a task reference.

Who this guide is for

This guide is written for the people accountable for standing up and running the platform:

• Platform administrators who deploy and operate the control plane, enforcement point, data tier, and runner; manage durability and backup; and provision the agent fleet.
• Security administrators and governance officers who author deny-by-default policy, review access, work the exception queue, release quarantines, and verify the audit chain.
• Identity administrators who sync the corporate directory into the platform and link agents to their accountable human owners.

Readers are expected to be comfortable with Linux service administration, PostgreSQL, container orchestration, LDAP/SCIM/OIDC identity concepts, and reading JSON configuration. No prior knowledge of the ACME products is assumed — this section establishes the vocabulary the rest of the guide relies on.

End users of the workforce (people who chat with their own agent, inspect its memory, or run playground tasks) are served by the companion User Guide. Where an admin task exposes a user-facing surface, this guide cross-references the relevant User Guide section rather than duplicating it.

What the governed AI workforce is

The ACME deployment is a governed, zero-trust AI agent workforce. In the reference fleet, 367 governed agents operate: 360 fleet agents mapped like-for-like onto a 325,154-principal corporate directory spanning 28 departments, 10 regions, and 626 roles, plus 7 showcase agents wired on a 14-edge hand-off DAG.

Every agent operates under the same non-negotiable control loop. On each model call:

1. The call is two-factor authenticated — a tenant API key plus a per-agent HS256 capability token.
2. It is evaluated deny-by-default at the Policy Enforcement Point (PEP); nothing runs on an absent or ambiguous rule.
3. On allow, the request is forwarded verbatim to the agent's row-level-security-isolated data tenant — the platform never rewrites the payload.
4. The decision is signed into a post-quantum-hybrid audit chain (Ed25519 + ML-DSA-44), producing a tamper-evident record.
5. Agent behaviour is monitored by SoulWatch; a composite drift score of ≥ 75 triggers automatic quarantine.
6. Token usage and spend are metered for cost and ROI reporting.

A governing invariant runs through the whole design: an agent can never exceed the entitlements of its accountable human principal. Every agent has a sponsor ("no sponsor → no run") and an operational manager, and every capability an agent holds is a provably narrowed subset of what its owner holds in the directory.

The product split: Tiresias-ZT, Asphodel, and soul-svc

The platform is three products with clean seams. They are sold and deployed independently or together.

The division of labour is deliberate: Tiresias enforces, Asphodel manages, soul-svc stores. Tiresias-ZT is the zero-trust spine — it holds the durable stores (registry, enrollment, usage, audit, quarantine, identity), derives capability envelopes, and makes the allow/deny decision at the PEP. Asphodel is the human control surface that manages agents, owners, policy, and reviews, but enforces nothing itself; it always acts through the control plane's own RBAC. soul-svc is the isolated memory tier where each agent's data lives under forced RLS, reachable only on an allowed, forwarded call.

Underneath these products sit six planes — Identity/Directory, Control Plane, Enforcement (PEP), Data, Agents & runner, and Observability/ROI — introduced in the architecture section and expanded across the guide.

Conventions used in this guide

• Imperative voice is used for administrator procedures ("Provision the agent…"). Second person is used when describing what an end user sees ("You select an agent…").
• Numbered steps carry procedures; where the outcome is observable, a step or procedure ends with a Result: line stating what you should see.
• Fixed-width marks routes, environment variables, filenames, roles, and literal values — for example X-Admin-Key, ACME_SITE=acme-site, runner/config.json.
• Fenced blocks carry configuration, request bodies, and log lines verbatim.
• Reference tables summarise fields, environment variables, and options.
• Screenshots are referenced as ![caption](images/NAME.png); the guide lives in docs/, so image paths are repo-relative (images/console-observe.png).
• Cross-references point to other sections by name and number.

Critical convention — the site slug. Throughout, the enforcement slice is ACME_SITE=acme-site, not acme. A mismatched site slug leaves agents ungoverned ("Unknown API key"). This recurs in provisioning and troubleshooting.

A map of this guide

To deploy from scratch, read sections 02–05 in order. To operate an existing deployment, jump to the task at hand and follow its cross-references. To govern the fleet day to day, live in sections 06–09.

2. Architecture & deployment topology (administrator view)

This section describes the governed AI workforce as an operator will encounter it: the six planes and what each one is responsible for at runtime, the lifecycle of a single governed model call, where each plane physically runs, and the trust model that makes the whole fleet provably governed. It is a reference for administrators who operate the deployment; for the step-by-step build-out of each plane (env vars, compose profiles, verification), see IMPLEMENTATION-GUIDE.md.

2.1 The six planes at a glance

The system is composed of six cooperating planes. Each is independently deployable and communicates with its neighbours over a small, typed contract, so a fault or a restart in one plane does not corrupt another.

The management console (Asphodel) and the analyst/enforcement console are operator surfaces onto these planes, not planes in their own right; they are covered in their own sections.

2.2 The governed-call lifecycle

Every single model call an agent makes traverses the same path. Understanding it is the key to operating the fleet, because every control the platform offers hangs off one of these steps.

1. The agent presents two secrets. It sends its tenant API key (X-Tiresias-Api-Key) and a short-lived per-agent capability token (X-Capability-Token) that it mints itself, HS256, from its own 64-hex capability secret. The PEP validates both; it does not issue capability tokens. Result: a call missing either factor is rejected before any policy is evaluated.
2. The PEP evaluates deny-by-default. It looks up the tenant's policy slice — pulled from the control plane and ETag-cached — and evaluates the route deny-by-default. Absent an explicit allow, the request is denied. Result: denials surface as 401/403 in the logs; that is the PEP working, not failing.
3. On allow, the request is forwarded verbatim. The PEP passes the request unchanged to the agent's isolated soul-svc tenant. It never rewrites or enriches the payload and adds no privilege of its own. On deny, nothing reaches the data plane. Result: the data tier only ever sees traffic that already passed policy.
4. The decision is signed into the audit chain. Every decision is written into a PQC-hybrid (Ed25519 + ML-DSA-44) audit chain with a durable write-ahead log, then shipped to the control plane. Result: the chain is independently verifiable from the analyst Verify tab — you do not take the platform's word for what happened.
5. SoulWatch scores behaviour. A content-blind behavioural monitor scores the agent; composite drift ≥ 75 trips it into quarantine automatically. Result: a drifting agent is held without operator intervention and appears in the analyst Quarantine tab and the admin Observe exception queue.
6. Usage is metered. Tokens and cost are reported to /cp/usage/report, landing in the tzt_usage table. Result: the row rolls up on the Grafana ROI board within the next panel refresh.

The essential invariant: a model call can reach data only through the PEP, so authentication, deny-by-default policy, verbatim forwarding, signed audit, and drift detection apply uniformly to all 367 agents. There is no side channel.

2.3 Deployment topology — where each plane runs

The reference deployment spans three substrates. Administrators should know which plane lives where, because exposure posture, backup scope, and failure blast radius all follow the topology.

Host-published surfaces on the acme-demo VM are the PEP (:8343), the Asphodel console (:3001 → container 3000), change-svc (:8085), Grafana (:3000), Loki (:3100), and Promtail syslog intake (:1514). The Control Plane listens on 8343 inside the demostack network — a distinct listener from the host :8343 PEP, despite the shared port number. The internal PEP→CP hop runs cleartext (TZT_ALLOW_CLEARTEXT_CP=true) solely because it never leaves that network.

For the full env, port, and compose-profile detail behind this topology — and the enterprise swap table for substituting your own directory, SIEM, Postgres, vault, and OIDC IdP — see IMPLEMENTATION-GUIDE.md §2 and §13. The reference runs single-node on Docker Compose; production splits the planes across nodes or Kubernetes (deploy/k8s/) with the stateless PEP scaled horizontally and the CP kept store-backed.

Durability note. The design is deliberately stateless-in-containers, durable-in-stores. All six CP stores must be set to postgres or governance state is lost on restart; the PEP audit WAL at /data/audit-wal is crash-durable; and the runner's fleet-state.json must persist or agents over-run their ~1–2 jobs/day pacing after a reboot. See §2.5 and IMPLEMENTATION-GUIDE.md §12.

2.4 The trust model

The platform's stance is zero trust: position is policy, not privilege. Nothing is trusted by virtue of where it sits on the network. The trust model rests on four properties an administrator can rely on:

• Two-factor call authentication. No call is honoured without both the tenant API key and a valid, freshly minted per-agent capability token. Compromise of one agent's secret is scoped to that one tenant.
• Deny-by-default enforcement. The PEP grants nothing that policy has not explicitly allowed. Access is added by authoring policy, never assumed.
• Provable non-escalation. An agent's effective entitlements are derived as a monotonically narrowing chain — E0 (the owning human principal's real directory entitlements) ⊇ E1 (Asphodel's authored scoped subset) ⊇ Tiresias call rules → effective (E0 ∩ E1 ∩ rules). Every layer can only restrict, so "an agent can never exceed its owning principal" is provable, not asserted. This is why every agent must answer to a human owner: no sponsor → no run.
• Tamper-evident accountability. Each decision is signed into the PQC-hybrid audit chain and independently verifiable; forced per-tenant RLS guarantees no agent can read another's data even at the database owner level.

Because these properties are enforced inline on the single mandatory path (§2.2) rather than bolted on around it, governance is not something the fleet can opt out of — it is a structural property of how a call reaches data.

2.5 Operator implications

• Back up the CP Postgres. It is the system of record for registry, enrollment, usage, audit, quarantine, and identity. The data-plane Cloud SQL (acme_soul) separately holds agent memory under forced RLS. Both belong in your DR plan.
• Confirm the site slug. The PEP is scoped to acme-site (not acme). An agent provisioned under the wrong site is ungoverned and answers 401 "Unknown API key"; verify enrollment with GET /cp/admin/peps.
• Keep RLS forced. Never grant standing BYPASSRLS; scope service_role per-connection for legitimate cross-tenant maintenance.
• Publish only what must be public. Keep the control plane, soul-svc, and Postgres in-network; publish the PEP, console, and observability UIs, and front them with TLS and (in production) OIDC.

For the acceptance checklist that confirms all six planes are healthy end-to-end, see IMPLEMENTATION-GUIDE.md §11.

3. Initial configuration & first-run checklist

This section takes a freshly installed ACME governed-workforce stack — containers created but not yet configured — and brings it to a known-good, governed first-run state: the Control Plane admin key set, the enforcement site slug correct, all durable stores pointed at Postgres, liveness confirmed on /healthz, and the management console reachable and rendering both planes. Complete it in order; every step ends with a verifiable result, and the section closes with a first-run acceptance checklist you can sign off against.

Perform these procedures on the acme-demo VM from the demo-stack project root:

/opt/acme/tiresias-zt/deploy/demo-stack/

All values in angle brackets come from your operations record or vault. Substitute the deployment's real host names for the placeholders (acme-demo-host, acme-soul-db-private, your-ops-ip).

3.1 Set the Control Plane admin key

The admin key (TZT_ADMIN_KEY) gates every /cp/admin/* route, is presented as the X-Admin-Key header, and is reused by the management console BFF (TIRESIAS_ADMIN_KEY), the change-control service (CHANGE_ADMIN_KEY), and the analyst console login. Set it once, from your vault — never from git or a printed environment dump.

1. Generate a long random value in your vault and record it as TZT_ADMIN_KEY.
2. Set it in the Control Plane container environment:

ini # acme-tiresias-cp TZT_ADMIN_KEY=<long-random-admin-key> # from your vault 3. Mirror the same value into the dependent services that authenticate against the CP:

4. Recreate the affected containers so the new environment takes effect:

bash docker compose up -d tiresias-cp asphodel-ui change-svc

Result: TZT_ADMIN_KEY is set on the CP and mirrored to the console BFF and change-svc. A later call to any /cp/admin/* route without this header returns api key required; with it, the route responds.

Production note. The BFF admin key is a demo simplification. In production, replace it with the caller's own OIDC session and set ASPHODEL_DEMO_FALLBACK=0 (see §3.5). The key is BFF-only and must never reach the browser.

3.2 Set the site slug — the acme-site gotcha

The enforcement plane is site-scoped. The PEP is enrolled to the Control Plane under a site slug, and the CP serves it exactly the tenant slice keyed to that site (the 367-agent slice for acme-site). The slug must be acme-site — not acme.

1. Set the site slug on the PEP:

ini # acme-tiresias-zt (PEP) ACME_SITE=acme-site # MUST match the enrolled slice — not "acme" 2. Provision every agent under the same site (the site field in the provision body must read "site": "acme-site"; see the provisioning section of this guide). 3. Recreate the PEP and confirm the enrollment:

bash docker compose up -d tiresias-zt curl -sS http://acme-demo-host:8343/cp/admin/peps -H "X-Admin-Key: $TZT_ADMIN_KEY"

Result: GET /cp/admin/peps lists the PEP enrolled under site acme-site. Any agent whose provisioned site does not match this slug falls outside the PEP's policy slice and its calls are rejected with 401 "Unknown API key" — the single most common first-run misconfiguration.

Do not confuse the two slugs. ACME_SITE=acme-site is the enforcement site and is load-bearing. ASPHODEL_CUSTOMER=acme (§3.5) is the console customer label and is cosmetic. They are different values for different purposes; the site slug is the one that must never be acme.

3.3 Point the Control Plane at Postgres

The CP owns all durable state — the agent registry, PEP enrollment, per-tenant policy, usage, the audit sink, the quarantine store, and the identity index. Each of these has an in-memory default that is lost on restart. Every store must be set to postgres, or agents silently "disappear" after a bounce.

1. Set the DSN and all six store selectors on the CP:

ini # acme-tiresias-cp TZT_POSTGRES_DSN=postgresql://tzt:tzt@postgres:5432/tzt TZT_REGISTRY_STORE=postgres TZT_ENROLLMENT_STORE=postgres TZT_USAGE_STORE=postgres TZT_IDENTITY_STORE=postgres TZT_AUDIT_SINK=postgres TZT_QUARANTINE_STORE=postgres

2. Recreate the CP:

bash docker compose up -d tiresias-cp 3. Confirm durability by bouncing the CP and re-reading the registry — provisioned agents must still be present.

Result: All six stores read postgres, backed by the shared tzt database on the in-network postgres:5432. Registry, usage, audit, and quarantine state survive a restart. The shared CP Postgres is the system of record — back it up (see the durability section of this guide).

3.4 Verify liveness on /healthz

/healthz is an unauthenticated, plain-liveness endpoint on the Control Plane (and, when configured, on the runner control API). It carries no admin key and returns a simple ok status.

1. Probe the CP from inside the network:

bash docker exec acme-tiresias-cp wget -qO- http://localhost:8343/healthz # → ok 2. (Optional) If the runner control API is enabled, probe it too:

bash curl -sS http://runner:8090/healthz # → {"status":"ok"}

Result: The CP answers ok on /healthz. This confirms the process is live and the listener is bound before you rely on any authenticated route. A non-ok or connection-refused response means the container is not up — resolve that before proceeding to the console.

/healthz is liveness only. It does not assert that the stores are Postgres-backed or that the site slug is correct — those are separately verified in §3.3 and §3.2.

3.5 First login to the management console

The management console (Asphodel) is one role-adaptive Next.js app with two RBAC-gated planes. Its browser layer talks only to a back-end-for-front-end (BFF) that proxies the Control Plane server-side.

1. Set the console's server-side environment:

ini # acme-asphodel-ui (BFF — server-side only) TIRESIAS_API_URL=http://tiresias-cp:8343 TIRESIAS_ADMIN_KEY=<= TZT_ADMIN_KEY> # BFF-only, never exposed to the browser ASPHODEL_CUSTOMER=acme # customer label (cosmetic) ASPHODEL_DEMO_FALLBACK=1 # 1 = guest read-only demo; 0 = REQUIRE login 2. Bring up the console:

bash docker compose --profile console up -d asphodel-ui # host :3001 → container :3000 3. Open http://acme-demo-host:3001 (or console.asphodel.ai). With ASPHODEL_DEMO_FALLBACK=1, a guest lands in read-only DEMO mode with no login (the BFF performs the admin-key read); with ASPHODEL_DEMO_FALLBACK=0, sign in via OIDC / demo-auth first. 4. Confirm both planes render. The Personal plane (My agent /, Chat /chat, Memory /memory) should render an agent's governed memory; the Admin plane should render the exception-first Dashboard at /observe, plus Principals (/principals) and Access (/access).

Result: The console loads, the BFF reaches the CP, and the exception-first admin Dashboard renders. If the browser shows data but the network tab never exposes the admin key, the BFF boundary is intact — the key must stay server-side.

3.6 First-run acceptance checklist

Sign off against this checklist before declaring the instance ready. Each item maps to a procedure above or to a downstream section of this guide.

• [ ] Admin key set — TZT_ADMIN_KEY set on the CP and mirrored to the BFF (TIRESIAS_ADMIN_KEY) and change-svc (CHANGE_ADMIN_KEY); a /cp/admin/* call without the header returns api key required (§3.1).
• [ ] Site slug correct — ACME_SITE=acme-site (not acme); GET /cp/admin/peps lists the PEP enrolled under acme-site (§3.2).
• [ ] Stores durable — all six TZT_*_STORE/TZT_AUDIT_SINK read postgres against TZT_POSTGRES_DSN; state survives a CP restart (§3.3).
• [ ] Liveness ok — /healthz returns ok on the CP (§3.4).
• [ ] Console both planes — the console renders Personal memory and the Admin exception Dashboard, principals, and access; the admin key never reaches the browser (§3.5).
• [ ] PEP enforcing — a call without a valid capability token is denied 401; deny-by-default is observable in the analyst console Status tab.
• [ ] Governed work flowing — once agents are provisioned and the runner is started, fleet logs show -> governed write and N/N fleet covered, and usage rows accrue.
• [ ] Audit verifiable — the analyst console Verify tab reports the chain intact / signatures valid.

Result: With every item checked, the stack is in a governed first-run state. Proceed to provisioning agents, running jobs, and directory synchronization in the following sections. For deeper deploy-time detail see IMPLEMENTATION-GUIDE.md (§4 Control Plane, §5 Enforcement, §9 the console, §11 the full verification checklist); for the plane-by-plane design and port map see ARCHITECTURE.md.

4. Identity & directory administration

Every governed agent in the ACME fleet ultimately answers to a human principal in your corporate directory. The control plane (tiresias-cp) keeps a queryable identity index — the tzt_principal table — projecting that directory: each principal's groups, department (OU), region, role, and policy segments. This index is the substrate for the like-for-like fleet mapping, for the accountable-sponsor invariant ("no sponsor → no run"), and for the Access explorer's provable envelope derivation (E0 ⊇ E1 ⊇ call rules → effective). Identity flows in one direction only — directory → CP store → policy — with the directory as the system of record and Postgres as the durable projection. This section covers configuring the identity provider and LDAP bind, running the sync, administering ownership from the Principals surface, and verifying the result.

4.1 Configure the identity provider and store

The control plane selects its identity source with two environment variables on the tiresias-cp service. Set both, then restart the control plane.

Use TZT_IDENTITY_STORE=postgres for any real deployment — memory loses the index on restart. The reference ACME instance runs TZT_IDENTITY_PROVIDER=ldap against NethServer / Samba-AD; native is reserved for the seeded-lab path in §4.3.

4.2 Configure the LDAP bind

For an AD/LDAP source, add the bind configuration to the tiresias-cp environment. Always bind with a read-only service account and source the password from your vault — never commit it.

TZT_IDENTITY_PROVIDER=ldap
TZT_IDENTITY_STORE=postgres
TZT_POSTGRES_DSN=postgresql://<user>:<pw>@<host>:5432/<db>
TZT_IDENTITY_LDAP_URL=ldaps://<your-dc>:636
TZT_IDENTITY_LDAP_BASE_DN=DC=corp,DC=example
TZT_IDENTITY_LDAP_BIND_DN=CN=svc-tiresias,CN=Users,DC=corp,DC=example
TZT_IDENTITY_LDAP_PASSWORD=<bind-account-password>   # from your vault, never in git
TZT_IDENTITY_LDAP_USE_SSL=true

To bind against the ACME reference directory:

1. Provision the svc-tiresias read-only bind account in Samba-AD.
2. Expose LDAPS on port 636 to the control-plane host only — over the VPC via the internal acme-directory-internal address, never published to the internet.
3. Populate the LDAP variables above and restart: docker compose up -d tiresias-cp.
4. Confirm the container came up healthy: docker exec <cp> wget -qO- http://localhost:8343/healthz. Result: the endpoint returns ok, and the CP begins syncing on start.

Segments (departments, divisions) are derived from OU / group membership. If your OU layout does not map cleanly to departments, tune the derivation with TZT_IDENTITY_SEGMENT_RULES.

4.3 Seeded-directory lab (no external directory)

For a self-contained lab with no AD/LDAP to point at, seed a compact synthetic directory instead of binding LDAP.

1. Run the CP with TZT_IDENTITY_STORE=postgres and TZT_IDENTITY_PROVIDER=native.
2. Seed the directory and link the showcase agents in one pass: bash docker cp scripts/identity/seed_directory.py <cp>:/tmp/ docker exec <cp> python /tmp/seed_directory.py
3. Restart the control plane so it reads the seeded index. Result: seed_directory.py reports directory now: {'human': N, 'agent': M} and the humans/agents are linked.

A synthetic directory is a demo convenience, not a system of record. In production, prefer LDAP sync (§4.2) plus link_agents.py (§4.6) against your real directory.

4.4 Sync principals to tzt_principal

With TZT_IDENTITY_PROVIDER=ldap, the control plane runs sync_ldap.py: it connects to Samba-AD over LDAPS, walks the directory under the base DN, and upserts each principal into tzt_principal on start and on schedule. Each row carries:

The reference directory holds 325,154 principals — 120,120 humans plus 205,034 NHI / service identities — across 28 departments, 10 regions, and 626 roles. Note that directory scale is not agent scale: only 360 fleet agents are mapped like-for-like (by department × role-family headcount), plus 7 showcase agents.

Because policy binds to segment:<dept>, the segments field is load-bearing. A principal synced without correct department segmentation will not resolve to the intended policy — verify segmentation whenever you add an OU or reorganize departments.

4.5 The Principals & Ownership surface

Open the admin console at Principals (/principals). Every agent answers to two humans:

The principal browser is a server-side identity-index search over GET /cp/admin/identities. The full 100k+ principal graph is intentionally not browsable — you search it. A suggested-owner queue lists owner-unattested agents and offers an Attest owner action, authored through change control (Contract B).

Attesting an owner. Attestation is a governed, auditable act — an administrator affirming that a named human is accountable for an agent.

1. On the Principals surface, review the suggested-owner queue of owner-unattested agents.
2. Select an agent and confirm the suggested human is correct (matching segment, seniority, and reporting line).
3. Click Attest owner. The action is authored through the change-control service and enters the approval inbox (/approvals).
4. Once approved, the change writes owner_principal_id (and sponsor_id where applicable) to the agent's tzt_principal row. Result: the agent leaves the suggested queue, and its Access-explorer confidence advances from stub toward live.

Succession on departure. When a principal leaves the organization, their sponsored/owned agents must not be orphaned (an orphaned agent loses its sponsor and can no longer run).

1. On the Principals surface, locate the departing principal by search.
2. Choose Succession-on-departure and select the successor human — a peer or manager in the same segment.
3. Submit; the reassignment is authored through change control and approved in /approvals. Result: every agent whose sponsor_id / owner_principal_id pointed at the departing principal is reassigned to the successor, preserving the no-sponsor-no-run invariant and E0 continuity.

4.6 Link agents to owners (E0/E1)

Sync populates human principals; linking gives each agent an owner so its envelope resolves live instead of showing an honest stub. The idempotent link_agents.py script picks a distinct senior human owner per agent's segment and upserts the agent's agent-type row with owner_principal_id and scope_bindings.

docker cp scripts/identity/link_agents.py <cp>:/tmp/
docker exec <cp> python /tmp/link_agents.py
# LINKED security-analyst            -> Cyrus Duval [Director, InfoSec] group=infosec
# DONE: N linked, M skipped

E0 then resolves live from the owner's real directory groups, dept/OU, region, and role; E1 is the authored scoped subset. To move an agent from stub to live, it must resolve to a directory principal (by matching id, LDAP external_id, or the registry leaf's principal_dn) carrying real groups/dept/region/role — against your AD/LDAP those come straight from the sync. See section 07 (Access & entitlements) for the full derivation.

4.7 Verify the sync

Confirm the index is populated before running any jobs.

1. Count principals by type directly against the store: bash docker exec <cp> python -c "import os,psycopg;c=psycopg.connect(os.environ['TZT_POSTGRES_DSN']);\ cur=c.cursor();cur.execute(\"SELECT node_type,count(*) FROM tzt_principal GROUP BY node_type\");print(dict(cur.fetchall()))" # → {'human': 120120, 'agent': 205034} (ACME's live numbers) Result: both counts are non-zero and match your directory population.
2. Cross-check in the console: the Principals surface returns the same principals via server-side search (GET /cp/admin/identities).
3. Spot-check segmentation: search a known department and confirm principals carry the expected segment:<dept> label, since policy resolution depends on it.
4. Confirm ownership: search a linked agent and verify it shows an accountable sponsor and operational manager, not the suggested-owner queue.

If the counts are zero, re-check the bind DN, password, TZT_IDENTITY_LDAP_USE_SSL, and that the CP was restarted after configuration. If principals appear but policy does not resolve, inspect the segments field and TZT_IDENTITY_SEGMENT_RULES.

Cross-references: deployment topology and the port map, section 11; Access & entitlements (E0/E1/effective), section 07; governance operations and change control, section 08.

5. Agent lifecycle administration

Every governed agent in the ACME workforce is a tenant in the Tiresias Control Plane (CP) with a matching data tenant in soul-svc. Administering an agent's lifecycle — provisioning it, editing it, enabling or disabling it, and decommissioning it — is therefore a control-plane operation, performed either through the Asphodel management console or directly against the CP admin API (X-Admin-Key: $TIRESIAS_ADMIN_KEY). This section covers the full lifecycle for a single agent and for the fleet.

A defining property of the platform is that an agent's character is immutable after creation while its operational wiring stays editable. Understand that split before you edit anything.

5.1 The genesis lock: character vs. operational wiring

At creation an agent is assigned a persona. Four fields form its genesis-locked character and can never be changed by any subsequent edit; the remaining fields are operational wiring and may be changed at any time. The runner enforces this on every persona write and rejects a locked-field change with a locked_field error.

The seven showcase agents defined in personas.json are flagged genesis in the roster; the 360 fleet agents are provisioned dynamically. The genesis flag marks provenance — the locked-field guard applies uniformly to every agent regardless of origin.

5.2 Provisioning a single agent

Provisioning both registers the tenant in the CP registry and creates its soul-svc data tenant (a _soul_tenants row plus a _soul_api_keys key where key_hash = sha512(bearer)).

To provision one agent:

1. Choose a slug matching ^[a-z0-9][a-z0-9-]*$. Confirm it is free: GET /cp/admin/agents/{slug} must return 404 (a 200 means the slug is taken).
2. Issue the provision call:

``` POST http://tiresias-cp:8343/cp/admin/tenants/{slug}/provision X-Admin-Key: $TIRESIAS_ADMIN_KEY Content-Type: application/json

{ "tier": "enterprise", "customer": "acme", "site": "acme-site", "backend": { "kind": "soul-svc", "url": "", "auth": { "type": "bearer", "token": "" } } } ```

3. Capture the response.

Result: the CP returns the tenant's api_key and capability_secret. Both are shown exactly once and are never recoverable — store them in your secret manager immediately. The 64-hex capability_secret is the seed from which the caller mints per-call HS256 capability tokens; the CP validates capability tokens but never re-issues the secret.

At runtime the agent presents two credentials at the PEP: X-Tiresias-Api-Key (the tenant api key) and X-Capability-Token (the minted HS256 token). Both must validate before the PEP evaluates policy.

The site must match the PEP's enrolled slice. The value "acme-site" (settable via ACME_SITE) must equal the slice the enforcement PEP is enrolled to. If it does not, the new agent is outside the PEP's policy slice and every governed call fails with "Unknown API key" — the agent is effectively ungoverned. Confirm the enrolled slice with GET /cp/admin/peps.

5.3 Batch provisioning the fleet

For fleet-scale provisioning, derive the roster from the directory rather than authoring agents by hand. runner/agent_manifest.py reads the tzt_principal identity index and emits an agent per department × role-family, weighted by headcount, capped at 360 agents. Each manifest entry carries slug, dept, role, region, headcount, an intel/research domain, and a route block (home / ceiling / floor tiers derived from seniority).

1. Ensure the directory has synced into tzt_principal (the manifest is empty without it).
2. Run agent_manifest.py to produce agent-manifest.json and review the printed summary (manifest agents, departments covered, home-tier spread).
3. Drive the batch provisioning path, which reuses one database connection and one HTTP session across the whole roster instead of reconnecting per agent.

Result: 360 fleet agents are registered like-for-like against the real ACME org across all 28 departments. Together with the 7 showcase agents this yields the 367 governed agents the platform operates.

5.4 Editing an agent

Edit an agent's operational wiring through the console (Agents → agent detail → Edit agent) or by PUTting the editable fields. A persona edit validates in three passes: the locked-field guard, the merge, and a full persona re-validation; only then is the override persisted and the agent hot-reloaded.

• Permitted: change title, seniority, expertise, responsibilities, job, system_prompt, memory_seed, assigned knowledge_bases, mindsets, and the routing_order table.
• Rejected: any attempt to change slug, identity, voice, or values returns a locked-field error and is not applied. To change an agent's fundamental character you must decommission it and provision a new one.

Result: the edit is written to the persona-override store and takes effect on the next cycle; the roster and agent-detail reads reflect it immediately.

5.5 Enabling and disabling an agent

Enable/disable is a runner-scheduling control, distinct from quarantine (a governance action, see Governance operations). A disabled agent is skipped by the scheduler and the fleet runner but remains fully provisioned and governed.

1. From the agent-detail page click Disable (or POST /control/agents/{slug}/enabled with {"enabled": false}).

Result: the agent's enabled flag is set to false, persisted to runtime-state.json, and honored across restarts. The agent runs no further scheduled cycles until re-enabled.

2. To resume, click Enable (or POST with {"enabled": true}).

The runner control surface is fail-closed: every /control/* call requires the X-Runner-Token header, and the surface is disabled entirely (503) when RUNNER_CONTROL_TOKEN is unset.

5.6 The agent-detail management actions

The Agents → agent detail page (/agents/{slug}) is the single pane for one agent's full persona and lifecycle actions.

The page presents the full persona (identity / voice / values / expertise / responsibilities / job) alongside these management actions:

Four governed tabs sit below the actions: Memory, Reasoning (CoT) (chain-of-thought timeline, newest first), Jobs (deterministic cycles grouped by session), and Access & Policy (the capability-issuance scheme and Ed25519 public key, the API-key hash ledger with no secret material, and the editable deny-by-default route policy).

5.7 Decommissioning an agent

Decommission an agent that is being retired. The safe order:

1. Disable the agent (§5.5) so no further cycles run.
2. Optionally place a memory hold if the agent's governed memory must be retained for review (see Memory governance).
3. Deprovision the tenant: POST /cp/admin/tenants/{slug}/deprovision with X-Admin-Key. This is best-effort — a missing endpoint (404/405) is non-fatal to teardown.
4. Remove the soul-svc data tenant: delete the _soul_api_keys row(s) then the _soul_tenants row for the agent's tenant id.
5. Remove the agent's stored credentials (agent-creds.json) and its persona override so it no longer appears in the roster.

Result: the agent no longer exists in the CP registry or the data plane, its api key stops validating at the PEP, and it drops out of the fleet. Its historical audit-chain entries remain intact — decommissioning removes the agent, not the tamper-evident record of what it did.

A provisioning failure triggers the same teardown automatically: create_agent rolls back the CP tenant and both soul-pg rows best-effort and raises a ProvisionError naming the failing step, so a half-created agent never lingers.

6. Policy administration

Policy in the ACME governed AI workforce is enforced at the Policy Enforcement Point (PEP), never inside the agent. Two distinct surfaces govern what a governed agent may do:

• The deny-by-default route policy — the explicit table of {methods, path_prefix, scope} grants that the PEP evaluates on every forwarded call. Any route with no matching rule is denied at the PEP.
• The per-agent policy — the runtime posture applied to the agent's model traffic: capability tier, request-rate ceiling (rpm / rpm_mode), and spend budget.

Both are held in the Control Plane registry, propagated to the PEP as a site-scoped slice, and edited through the Asphodel admin console or directly against the Control Plane admin API (/cp/admin/*, gated by X-Admin-Key: $TZT_ADMIN_KEY). This section covers editing both surfaces, how position is policy constrains what any edit can grant, and how to confirm the resulting effective permissions in the Access explorer.

All admin-API examples target the Control Plane at tiresias-cp:8343. Substitute the agent's tenant slug for {slug}. The admin key is BFF-only in the console and must never be exposed to a browser.

6.1 The two policy surfaces at a glance

6.2 Deny-by-default route policy

The route policy is an ordered list of grant rules. Each rule is a JSON object:

{
  "methods":     ["GET"],            // HTTP methods this rule grants
  "path_prefix": "/v1/memory",       // data-tier path prefix it matches
  "scope":       "memory:read"       // capability scope required/asserted
}

On each forwarded call the PEP matches the request method and path against the rule table. If a rule matches, the call is allowed and forwarded verbatim to the agent's RLS-isolated data tenant; if no rule matches, the call is denied. There is no implicit allow — an agent granted only reads can never write, and an agent with no rule for a route never reaches it. Denials are deterministic (a fixed policy decision, not a heuristic) and are surfaced in the analyst console's Tiresias-denials panel.

A worked example — a read-only reporting identity versus a write-only ingest identity:

To edit the route policy in the console:

1. Open the admin console and navigate to Agents → the agent's detail page (/agents/{slug}).
2. Select the Access & Policy tab. The editable deny-by-default route-policy table is shown alongside the capability-issuance scheme (Ed25519 pubkey) and the API-key hash ledger.
3. Add, edit, or remove rules, then save. The change is authored into the Control Plane registry and re-served to the PEP's cached slice.

Result: The new rule table takes effect on the next forwarded call; removing a rule immediately denies the route.

To edit the route policy from the admin API, PUT the full replacement rule list (the write replaces the table, so include every rule you want kept):

curl -X PUT https://tiresias-cp:8343/cp/admin/tenants/{slug}/policy \
  -H "X-Admin-Key: $TZT_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "policies": [
          {"methods": ["GET"],  "path_prefix": "/v1/memory",        "scope": "memory:read"},
          {"methods": ["POST"], "path_prefix": "/v1/memory/search", "scope": "memory:read"}
        ]
      }'

Result: The Control Plane persists the new table and the PEP re-fetches the changed slice; the agent's effective route grants update on the next call.

6.3 Per-agent policy — tier, rpm, rpm_mode, and budget

The per-agent policy editor on the Policy cockpit (/governance) sets the runtime posture applied to the agent's model traffic:

To edit tier / rpm / rpm_mode in the console:

1. Navigate to Policy (/governance). The cockpit lists each agent with its tier / rpm / policy summary, signed-audit count, and chain-intact status.
2. Open the per-agent policy editor, adjust tier, rpm, and rpm_mode, and save.

Result: The updated policy is written to the Control Plane (PUT /cp/admin/tenants/{slug}/policy) and applied on the agent's next governed call.

To set the daily budget from the admin API:

curl -X PUT https://tiresias-cp:8343/cp/admin/tenants/{slug}/agents/{slug}/budget \
  -H "X-Admin-Key: $TZT_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"daily_usd": 25.0}'

Result: The agent's daily spend cap updates; the runner's tier step-down honours the cap (it steps a call down toward the floor tier when the cap would be exceeded), and spend is metered against it.

6.4 Position is policy — the narrowing envelope

No policy edit can grant an agent more than its owning human principal already holds. Entitlements resolve as a strictly monotonically narrowing envelope in which each layer can only restrict the one above it:

E0 (the owning principal's real directory entitlements — LDAP groups, department/OU, region, role) ⊇ E1 (Asphodel's authored scoped subset, from the agent's scope_bindings) ⊇ Tiresias call rules (the deny-by-default route policy of §6.2) → effective (E0 ∩ E1 ∩ rules).

The invariants E1 ⊆ E0 and effective ⊆ E1 always hold. Practically, this means:

• Editing the route policy (§6.2) or the scoped subset can only narrow an agent — it can never lift the agent above E0. Granting a rule for a route the principal does not hold in E0 has no effect: the intersection still excludes it.
• To widen an agent's reach, you must change its position — the directory entitlements of its owner principal, or the agent-to-principal link (owner_principal_id) that supplies E0. This is why the system is described as position is policy: the human's directory standing is the ceiling, and every policy layer below it only subtracts.

An agent not yet linked to a directory principal derives E0 from no owner and shows an honest stub state — link it (see Directory / linking) before relying on the effective set.

6.5 Verifying effective permissions with the Access explorer

After any policy edit, confirm the resulting permissions rather than reasoning about layers in isolation. The Access view (/access) resolves, for a selected agent, the deterministic Resolved-Set-of-Permissions (RSoP) in plain verbs, with full provenance and honest per-layer confidence.

1. Open Access (/access) and select the agent.
2. Read the derivation across the tiles: E0 ENTITLEMENTS, E1 SCOPED, CALL RULES, EFFECTIVE, NOT-GRANTABLE, and JIT ACTIVE.
3. Check the confidence badge on each layer — live (derived from a linked directory principal), partial, or stub (agent not yet linked; an honest state, not a failure).

Result: The EFFECTIVE tile is the ground truth for what the agent can actually do after E0 ∩ E1 ∩ rules; NOT-GRANTABLE shows what a policy edit cannot grant because the principal does not hold it.

Because every layer only narrows, "an agent can't exceed its principal" is provable from this view — use it as the final check on any policy change before considering the edit complete.

7. Model routing administration

Model routing decides which LLM answers each governed task, within limits you set per agent, at a cost you cap per tier. It is the one control surface that flexes an agent's reasoning capacity without ever touching its genesis-locked persona (identity, voice, values). Routing is defined entirely in a single JSON file — runner/config.json — which is validated on load, hot-reloadable at runtime, and readable without exposing a single secret.

This section describes the file's structure, the four provider lanes, per-agent home/ceiling/floor bands, keyword escalation, budget step-down, fallback-to-floor degradation, provider-key wiring, how to point a tier at any OpenAI-compatible endpoint, and how per-call pricing surfaces on the ROI board (§04, Grafana acme-roi-economics).

7.1 The configuration file and its resolution order

The runner resolves its config path in this order:

1. An explicit path argument passed by a caller.
2. The environment variable ACME_CONFIG (point this at a mounted file to override without rebuilding the image).
3. The bundled runner/config.json shipped next to config.py.

The file has five top-level blocks: providers, tiers, routing, default_tier, and escalation. Every load runs a strict validator (config.validate_config); an invalid config raises ConfigError and is refused — the runner never boots on a malformed routing table.

API keys are never stored in this file. Each provider names the environment variable that holds its key via api_key_env. The config holds the variable name, never the secret. This lets the console and control API render the full config read-only with no risk of key leakage.

7.2 Providers and their model catalogs

A provider is one OpenAI-compatible chat endpoint (POST {base_url}/chat/completions) plus a catalog of models, each carrying price_per_million as an [input, output] USD pair. Four lanes ship by default across three providers:

OpenRouter is deliberately dual-lane: it carries the paid frontier models (anthropic/claude-opus-4.8 at [15.0, 75.0], anthropic/claude-sonnet-4.5 at [3.0, 15.0]) alongside five :free open-weight models priced [0.0, 0.0]. Free models cost $0 but still consume OpenRouter rate limits — a 429 on a free lane is the primary trigger for the fallback in §7.6.

7.3 Capability tiers

A tier binds a provider + model to a rank (higher = more capable) and an optional daily budget. Tiers are what escalation selects between. The nine shipped tiers, rank-ordered:

A daily_usd of null (or an omitted budget) means uncapped. default_tier is free_large — the tier applied to any agent slug absent from routing.

7.4 Per-agent routing bands (home / ceiling / floor)

Each agent's routing entry defines a band:

• home — the tier the agent sits at by default.
• ceiling — the most-capable tier it may escalate to, on cause.
• floor — the lowest tier it degrades to (optional; omitting it defaults to the lowest-rank tier overall).

The validator enforces rank(floor) ≤ rank(home) ≤ rank(ceiling). Any tier whose rank falls inside [floor, ceiling] is eligible, so escalation can land on an intermediate tier, not only the three named ones. The seven showcase agents:

High-stakes work (compliance) homes on paid with a free floor; routine work homes on free with a paid ceiling reserved for genuine escalation.

7.5 Keyword escalation

Escalation is deterministic, pure, and content-driven — no LLM, no clock. With escalation.enabled: true, each task is scored 0..100:

score = min(40, len(text)//200)              # task size
      + 10 * min(3, num_upstreams)           # DAG fan-in
      + 15 * (# distinct keywords_high in text)
      - 15 * (# distinct keywords_low  in text)

keywords_high (escalate): incident, breach, critical, escalate, zero-day, exploit, compromise, urgent, legal, compliance, risk. keywords_low (de-escalate): routine, summary, digest, triage, noop, fyi, status.

The score maps onto the agent's [floor, ceiling] band centered on home: 50 → home's rank, >50 interpolates up toward ceiling, <50 drifts down toward floor. The eligible tier whose rank is closest to that target wins (ties break to the cheaper tier). Set escalation.enabled: false to pin every agent to its home tier.

Budget step-down. After escalation picks a tier, the runner estimates the call cost (tier price × ~4000 in / ~1500 out tokens) against that tier's daily spend. While the chosen tier is over its daily_usd and a lower eligible tier exists, it steps down one rank. Each step is recorded in the decision trail, so a capped frontier cleanly becomes standard, then a free tier, with the reason auditable.

7.6 Fallback-to-floor on provider errors

Escalation and budget handle policy. Provider failures are handled separately: any LLM 4xx/5xx — a free-tier 429, a retired-model 410, or a 5xx outage — must never fail a governed agent.

Degradation procedure (automatic):

1. The call at the selected tier raises LLMError after its retry/backoff budget is spent (401/403 and 400/404 are surfaced immediately as auth/bad-model errors; 429/5xx are retried first).
2. The runner builds a fallback chain: the agent's floor tier, then ACME_FALLBACK_TIER (default ollama_oss), keeping only distinct tiers that differ from the failed one and are eligible for the agent.
3. It re-issues the call down the chain until one succeeds.
4. The downgrade is stamped into the decision record (fell back <from>-><to> after LLM failure).

Result: the governed work completes on a cheaper model, and both the failure and the substitution appear on the ROI board and in the CoT trail. If the entire chain fails, the original error is raised and the agent run is marked failed.

7.7 Provider keys and endpoint overrides

Supply keys through the process environment (e.g. runner/.env); the config only references them by name.

A missing key fails only the tiers that use that provider (LLMError: Missing API key … set env <NAME>), not the whole runner — so a partial deployment still routes on the lanes it can reach.

Pointing a tier at any OpenAI-compatible endpoint. Because every provider is just a base_url + bearer key + /chat/completions, you can retarget or add lanes:

• Override an existing provider's endpoint without editing the file via the well-known env overrides (env > config): OPENROUTER_BASE_URL, OLLAMA_BASE_URL, HF_BASE_URL. The live deployment uses this to pin OLLAMA_BASE_URL.
• Add a new provider by inserting a block under providers with its base_url, a new api_key_env name, and a models catalog with pricing, then defining tiers that reference it. To route a self-hosted vLLM/Ollama/LM Studio gateway, set its base_url to the internal endpoint — the model always comes from the tier catalog, never from env.

To apply an edited config at runtime, PUT /control/config on the runner control API (authenticated with X-Runner-Token: $RUNNER_CONTROL_TOKEN), or use the console Integrations page (§02) — both validate, hot-reload, and persist the override to the state file so it survives restart. GET /control/config returns the sanitized view (env-var names only). See §14, Troubleshooting, for RUNNER_CONTROL_TOKEN unset → 503.

7.8 Cost implications and the ROI board

Every price_per_million pair is the economic ground truth for a model. On each governed call the runner meters tokens_in/tokens_out, prices them against the tier's model, and writes a cost_usd row to the usage table — the datasource behind the Grafana ROI board (:3000/d/acme-roi-economics).

This makes the whole routing policy measurable. The board's per-1M-token panel shows the spread the tiers are chosen to exploit — e.g. opus-4.8 ≈ $37, sonnet-4.5 ≈ $10, deepseek-v4-pro ≈ $0.62, gpt-oss:120b ≈ $0.44, and the :free tiers at $0. Homing routine agents on free lanes with paid ceilings reserved for escalation is precisely what keeps blended spend low while retaining frontier capability on cause; fallback-to-floor further biases spend downward under provider stress. Log in as admin for the board to render data (an anonymous session renders only the shell).

Cross-references: §02 (Integrations / routing editor), §04 (ROI board), §05 (fleet run cadence), §14 (Troubleshooting: free-tier 429 saturation, control-API 503).

8. The runner — scheduler, fleet, and the Integrations surface

The runner is the execution engine of the governed workforce: it drives agents through their deterministic jobs, routes every memory and chain-of-thought (CoT) operation through the Tiresias PEP, and reports token spend so work lands in the audit chain and the ROI board. This section covers the three ways the runner executes work — the scheduled runner, the fleet runner, and the token-protected Control API — and how administrators steer all of it from the Asphodel console's Integrations (/runtime) surface.

The runner never enforces policy itself. It authenticates as each agent (tenant API key + per-agent capability token) and writes through the PEP, which evaluates every call deny-by-default. A runner misconfiguration therefore fails closed: unauthorised writes are denied, not silently accepted.

Execution modes

The runner ships one CLI (runner.py) plus a separate large-fleet driver (fleet_runner.py). Use the mode that matches the workload.

Cycle id determinism. Cycle id resolves as --cycle argument → ACME_CYCLE env → a UTC-date fallback (logged as a warning). Pass --cycle or set ACME_CYCLE for reproducible runs; the scheduler appends an incrementing -NNNN suffix to the base id each loop.

The scheduled runner (runner.py schedule)

The scheduler loops run-cycle at a fixed interval, running the 7-agent hand-off DAG each pass. Every DAG edge is one audited governed read (as the upstream agent) plus one audited governed write (as the downstream agent). A crash in one cycle is caught and logged; the loop continues.

To run the standalone scheduler:

1. Set the per-agent credentials and provider keys in runner/.env (see section 05).
2. Start the scheduler with your cadence: sh python runner.py schedule --interval 3600 --cycle prod
3. Confirm cycles are completing in the logs (see Reading runner logs, below).

Result: each pass logs ========== CYCLE prod-0000 START ========== … DONE (7 ok, 0 failed, 0 skipped), then Sleeping 3600s until next cycle....

The default container command is schedule --interval 3600. Standalone schedule has no remote controls — to pause, retime, or run agents on demand from the console, run serve mode instead.

The fleet runner (fleet_runner.py)

Where the showcase DAG is a hand-built 7-agent graph, the fleet runner scales governed jobs across the large org-mapped fleet (~360 agents, one per real Acme org function) as independent rolling batches. Each cycle selects the N agents most overdue for a job, runs one governed research/synthesis job each at the agent's home tier (with floor-tier fallback on any LLM error), writes the CoT + deliverable through the PEP as that agent, and reports usage. A persisted last-run state (fleet-state.json) rolls coverage across the whole fleet so every agent receives roughly 1–2 jobs per day without a DAG.

Tune the fleet runner entirely through environment variables:

The defaults (4 agents every 600s, 10h minimum gap) yield ~1–2 governed jobs/day/agent across the fleet. Raise FLEET_BATCH or lower FLEET_INTERVAL to increase throughput; widen FLEET_MIN_GAP_HOURS to reduce it.

To start the fleet runner:

1. Ensure agent-manifest.json (from runner/agent_manifest.py) and agent-creds.json are present under ACME_OUTPUT_DIR; only agents with credentials are eligible.
2. Launch it: sh FLEET_BATCH=4 FLEET_INTERVAL=600 FLEET_MIN_GAP_HOURS=10 python fleet_runner.py

Result: startup logs Fleet runner up: batch=4 interval=600s min_gap=10h max_tokens=420, and each cycle ends with batch done: N ran; 360/360 fleet covered.

The runner Control API

Run serve mode to make the runner remotely manageable. It starts a FastAPI service on 0.0.0.0:<RUNNER_CONTROL_PORT> (default 8090) and a managed scheduler in a background thread. A single lock guarantees at most one cycle runs at a time: the scheduler acquires it blocking; every manual trigger acquires it non-blocking and returns 409 busy if a cycle is already in flight.

Authentication is fail-closed. Every /control/* route requires header X-Runner-Token equal to env RUNNER_CONTROL_TOKEN:

• RUNNER_CONTROL_TOKEN unset → the entire control surface is disabled (503); only /healthz stays up.
• Header absent or mismatched → 401 (constant-time compare).

Runtime state — scheduler {enabled, interval_seconds}, per-agent {enabled}, and the active config override — is persisted to the state file and survives restarts. Interval changes are clamped to a minimum of 60 seconds.

The principal endpoints:

The config view is rebuilt field-by-field and never returns key values — only api_key_env names. A PUT /control/config is validated, hot-reloaded (the LLM provider table and agent routing are rebuilt for subsequent cycles), and persisted; agent personas are genesis-locked and unaffected.

Managing the runner from the console Integrations surface

The Asphodel admin console exposes all of the above at Integrations (/runtime). The BFF calls the Control API using RUNNER_CONTROL_URL + RUNNER_CONTROL_TOKEN; the token is never exposed to the browser. The page polls GET /control/status every 5 seconds so scheduler state and per-agent last-run times stay live. If the runner token is not configured, agent detail pages read "RUNTIME NOT CONFIGURED" instead of "WIRED · RUNNER CONTROL".

To pause or resume the scheduler:

1. Open Integrations (/runtime).
2. Toggle the scheduler control (pause clears enabled; resume sets it).

Result: the change persists to the state file and the scheduler wakes immediately to recompute its next-run estimate.

To change the cycle interval:

1. Edit the interval field and save (minimum 60 seconds).

Result: the running scheduler is woken early and the next cycle uses the new cadence.

To run a cycle on demand, click Run cycle now. Result: POST /control/cycle returns a new cycle id and 202; if a cycle is already running the console reports busy (409) rather than starting a second.

To manage a single agent, use the per-agent table:

1. Toggle enable to include or exclude an agent from scheduled cycles. A disabled agent is skipped; downstream agents fall back to their last governed read (then the in-process cache).
2. Click Run on a row to execute just that agent now.

Result: the row's live last-run cycle and timestamp update on the next 5-second poll.

To adjust providers, models, or routing, use the config editor, which reads GET /control/config and writes PUT /control/config. Result: on save the config is validated and hot-reloaded; an invalid config is rejected with the validation reason and the running config is unchanged.

Reading runner logs

Runner logs use %(asctime)s %(levelname)s %(name)s: %(message)s. Set verbosity with ACME_LOG_LEVEL. Key lines to watch for:

• Governed write (showcase): governed write as <slug> cot=<id> memory=<id> — the CoT and deliverable were written through the PEP as that agent.
• Governed read: governed read as <slug> session=<sid> -> N row(s); a PDP deny logs DENIED by PDP … FAIL-CLOSED (not serving cache).
• Cycle boundaries: ========== CYCLE <id> START ========== and DONE (N ok, M failed, K skipped).
• Fleet job: [<slug>] <model> tok=<n> $<cost> -> governed write, and per batch batch done: N ran; 360/360 fleet covered (or no agents due (min gap 10h) when nothing is overdue).
• Fallback: FALLING BACK to '<tier>' after a provider 4xx/5xx (free-tier 429 / retired-model 410 / outage) — the job still completes on a cheaper model, and the downgrade is recorded so it stays auditable and visible in the ROI board.

Cross-references. For provisioning agents and the roster/manifest, see section 05; for model routing (home/ceiling/floor, escalation, fallback) see section 05's routing coverage; for reading spend produced by these jobs see the ROI board in the observability section; for the deny decisions these writes are evaluated against see the policy and access-governance section.

9. Governance operations

Governance operations are the day-to-day disciplines that keep the agent workforce inside its declared behavioural and policy envelope: continuous behavioural monitoring, exception triage, quarantine handling, drift analysis, and the periodic compliance sign-off. Two consoles divide the work. The Asphodel admin console manages governance state — the Observe exception queue (/observe), the Policy/governance cockpit (/governance), and Drift & behavior analytics (/analytics). The Tiresias analyst console (the PEP) enforces it — SoulWatch detections, held agents, and audit-chain verification. This section covers both, in the order an operator uses them.

09.1 SoulWatch behavioural monitoring and auto-quarantine

SoulWatch is the behavioural-monitoring subsystem embedded in the enforcement plane (PEP). It observes every governed call content-blind — it scores behavioural markers (call cadence, scope reach, ladder-rung trajectory, deviation from the agent's population baseline) without inspecting prompt or response payloads. Markers roll up into a single composite behavioural risk score per agent.

The auto-quarantine rule is deterministic:

When SoulWatch quarantines an agent it writes the verdict to the shared Postgres quarantine store (TZT_QUARANTINE_STORE=postgres), so the control plane and every PEP replica agree on which agents are held. A quarantined agent's subsequent calls are denied deny-by-default at the PEP until an analyst releases it. SoulWatch posture is reported as ENFORCING on the analyst Status tab.

09.2 The Observe exception queue

The admin console Dashboard / Observe view (/observe) is the operator's first stop each shift. It presents an exception-first queue: SoulWatch detections, active quarantines, correlated campaigns, and baseline drift are folded into one ranked incident per entity, impact-ranked with a visible cut line so low-signal noise stays below the fold.

Four tiles head the page, each a clickable filter over the queue:

Two panels sit below the tiles: a Segregation-of-duties (SoD) conflict scan and a rollup of deterministic-PEP Tiresias denials over the last 24 hours. The legend colour-codes severity: critical = act now / warn = review / healthy = benign.

To triage: click the CRITICAL tile to filter the queue, open the top incident, and follow its entity through to the agent detail page (/agents/{slug}) or the analyst case view. Confirm whether the incident is an active quarantine before acting.

09.3 The Policy / governance cockpit

The Asphodel Policy view (/governance) is the fleet governance cockpit. For each agent it summarises:

• Sovereignty — the agent's governance ownership and data-residency posture.
• Tier / rpm / policy summary — the model tier, rate-per-minute ceiling, and the current deny-by-default route policy in brief.
• Signed-audit count and chain intact — the number of governed calls signed into the audit chain and whether the chain's cryptographic integrity is unbroken.
• Compliance sign-off — the compliance officer's latest attestation (see §09.6).

The cockpit exposes two editable surfaces:

1. A per-agent policy editor for tier, rpm, rpm_mode, and budget.
2. An audit-trail browser listing each governed decision with its decision, scope, status, and ts, filterable by chain.

Policy and budget changes write through the control-plane admin API:

PUT /cp/admin/tenants/{slug}/policy          # tier / rpm / rpm_mode / route policy
PUT /cp/admin/tenants/{slug}/agents/{slug}/budget   # per-agent budget

Both endpoints require the admin key header X-Admin-Key: $TZT_ADMIN_KEY and are proxied server-side by the console BFF — the key is never exposed to the browser.

09.4 Handling a quarantine — release from the analyst console

Releasing a held agent is an enforcement action performed from the Tiresias analyst console, not from Asphodel.

1. Open the analyst console at pdp.tiresias.watch (or the VM at :8343/admin/console — not :8343/admin, which is the governed API and returns "api key required").
2. Paste the admin key (X-Admin-Key = $TZT_ADMIN_KEY) into the #key field and click connect. Subsequent XHRs carry the key. Result: the console authenticates and the tab strip becomes live.
3. Open the Quarantine tab to list held agents.
4. Confirm the incident on the Detections and Agent case view tabs — review the SoulWatch markers that drove the composite risk over 75 before you release.
5. Click release on the target agent. The console POSTs:

POST /admin/quarantine/{id}/release

Result: the release verdict is written to the shared quarantine store; the CP and every PEP replica clear the hold, and the agent resumes governed calls under its existing policy. The QUARANTINED tile on Observe decrements.

Release an agent only after the underlying behaviour is understood and, where warranted, its policy tightened via the governance cockpit (§09.3). Releasing without remediation invites re-quarantine at the next risk threshold crossing.

09.5 Drift and behaviour analytics

The admin Drift & behavior view (/analytics) is a content-blind behavioural cockpit for population-scale analysis, organised into tabs:

On a fresh fleet with little history, each tab degrades to an honest empty state rather than fabricating a signal — a maturing baseline is expected, not a fault. Use this view to distinguish an isolated agent excursion (surfaced on Observe) from a coordinated pattern across many agents (a campaign cluster) before deciding whether to quarantine broadly or tune a single policy.

09.6 The compliance sign-off

The audit chain underpins every attestation. Each governed call is signed into a tamper-evident chain using PQC-hybrid SoulKey signatures — Ed25519 + ML-DSA-44 — buffered to a durable WAL and shipped to the control plane. Verify chain integrity from the analyst Verify tab; the governance cockpit surfaces the same result as the chain intact indicator.

The compliance sign-off is the compliance officer's periodic attestation, recorded per agent and shown in the governance cockpit (§09.3) as the latest sign-off. A sign-off is meaningful only against an intact chain and a current signed-audit count: confirm both read healthy in the cockpit before recording the attestation, so the sign-off certifies a verified, unbroken record rather than an unchecked one.

Cross-references: exception triage begins at Observe (§09.2); policy and budget writes are detailed under provisioning and policy operations; audit-chain durability and the WAL are covered under durability and restart.

10. Access & entitlement administration

Access administration in the ACME governed AI workforce answers one question with proof rather than assertion: what can this agent actually do, and why? The Asphodel Access explorer (/access) resolves that question for any selected agent as a deterministic, monotonically-narrowing entitlement envelope derived from your real directory. This section explains the envelope model, how to link agents to directory principals so the envelope becomes live, how to read the explorer as an administrator, how per-layer confidence is reported, and how to run access reviews and recertification.

10.1 The entitlement envelope model

Every agent's effective permissions are derived, never authored directly. The control plane computes four layers in a fixed order, each of which can only restrict the one above it:

E0  (owner's REAL directory entitlements: LDAP groups, dept/OU, region, role)
  ⊇ E1  (Asphodel's authored scoped subset — from the agent's scope_bindings)
    ⊇ Tiresias call rules (the PEP's deny-by-default route policy)
      → effective   (E0 ∩ E1 ∩ rules)

The system holds a monotonic-narrowing invariant: E1 ⊆ E0 and effective ⊆ E1. Because every layer can only subtract, the statement "an agent can never exceed its principal" is provable from the derivation itself, not asserted by policy. This is the core zero-trust guarantee of access administration: an agent's authority is a strict subset of a named, accountable human's authority.

10.2 Linking agents to directory principals

The envelope is only as real as its E0 root. E0 derives from the agent's owner — the human principal referenced by owner_principal_id on the agent's tzt_principal row. Until that link exists, the explorer honestly reports a stub (see §10.4). Linking is therefore the single most important access-administration task on a new fleet.

An agent principal is an agent-type row in tzt_principal whose owner_principal_id points at a human-type principal in a matching segment. Two idempotent scripts under scripts/identity/ establish this. Both read the DSN from TZT_POSTGRES_DSN and upsert on tzt_principal.id, so they are safe to re-run.

Linking against an existing directory

Use scripts/identity/link_agents.py when tzt_principal is already populated from your AD/LDAP (via TZT_IDENTITY_PROVIDER=ldap). For each agent slug in its MAPPING table, the script selects a distinct senior human owner in the matching segment — a principal whose role matches lead|head|director|manager|chief|vp|senior — and upserts the agent principal with owner_principal_id, sponsor_id, and scope_bindings set. E0 then resolves live from that owner's directory groups, department, region, and role.

1. Copy the script into the running control-plane container: bash docker cp scripts/identity/link_agents.py <cp-container>:/tmp/
2. Execute it inside the container (it has psycopg and the DSN): bash docker exec <cp-container> python /tmp/link_agents.py
3. Read the per-agent link log and the summary line: LINKED security-analyst -> Cyrus Duval [cn=cyrus.1,ou=infosec,...] group=infosec ... DONE: 7 linked, 0 skipped

Result: each linked agent now resolves to a named owner; the Access explorer derives E0 live for that agent. Any agent for which no senior owner was found in its segment is reported under skip and remains a stub until a suitable owner exists.

Edit the MAPPING dict (slug → display name, segment) to match your roster before running. The default mapping covers the seven showcase agents plus common service agents.

Seeding a directory for a lab with no LDAP

Use scripts/identity/seed_directory.py for a self-contained demo with no external directory. It creates the tzt_principal table, seeds a compact set of human owners across segments (infosec, digital, legal, finance, medialabs, publishing, merch), links each agent to an owner, and prints a directory census. The control plane must be running in native identity mode:

1. Set the control plane to serve identity from Postgres natively: TZT_IDENTITY_STORE=postgres TZT_IDENTITY_PROVIDER=native
2. Run the seeder inside the CP container: bash docker cp scripts/identity/seed_directory.py <cp-container>:/tmp/ docker exec <cp-container> python /tmp/seed_directory.py
3. Restart the control plane so it picks up the seeded directory.

Result: the explorer shows live E0/E1 for the seeded agents. A synthetic directory is a demo convenience, not a system of record — for production always prefer link_agents.py against your real AD/LDAP.

A stub agent is not a failure. It is the honest state of an agent that has not yet been linked to a directory principal. To move an agent from stub to live, ensure it resolves to a human principal (by id, LDAP external_id, or the registry leaf's DN) and that the principal carries real groups/dept/region/role.

10.3 Reading the Access explorer

Open the Access explorer at /access in the Admin console and select an agent. The explorer renders the full derivation with provenance for every layer.

The header tiles summarise the envelope:

To read a derivation:

1. Select the agent from the roster or identity search.
2. Confirm the confidence badge on E0 (see §10.4). A live badge means E0 was read from a real directory principal.
3. Walk the layers top to bottom. Each entry shows its provenance — where the entitlement came from (a specific LDAP group, the OU, the region, the authored binding, or a route rule).
4. Cross-check the monotonic invariant visually: E1 must be a subset of E0, and effective a subset of E1. The explorer renders narrowing at each step; a widening would indicate a linking or policy defect.

Result: you have a plain-verb, provenance-backed statement of the agent's resolved permissions, suitable for evidence in a review.

For the complementary per-agent view — capability issuance scheme, Ed25519 public key, the API-key hash ledger, and the editable route policy — use the Access & Policy tab on the agent detail page (/agents/{slug}). Fleet-wide policy, budgets, and the signed-audit chain live in Policy (/governance).

10.4 Per-layer confidence: live, partial, stub

Confidence is reported per layer, never as a single blended score, so you always know how much of an envelope rests on real data.

A stub is displayed honestly — it is never dressed up as live. This is deliberate: an access-administration console that fabricated confidence would defeat the purpose of provable governance.

10.5 Reviews and recertification

Access reviews close the loop between what the envelope is and what the organisation has approved. The Reviews workflow (/reviews) drives periodic access-review and recertification campaigns, and — like every mutating governance action — is authored through change control so each decision is captured as an approvable, auditable record.

A typical recertification cycle:

1. Open Reviews (/reviews) and start (or resume) a recertification campaign scoped to a department, region, or the whole fleet.
2. For each agent, review the resolved effective envelope and its E0 owner in the Access explorer. Confirm the owner is still the correct accountable sponsor and operational manager (see the Principals & Ownership workflow, /principals).
3. Recertify or revoke. Revocation narrows E1 (the authored scope) or the route rules; because layers only restrict, a revocation is always safe.
4. Route the campaign's decisions through the approval inbox (/approvals, Change & approvals) so they enter the signed audit chain.

Result: a complete, signed recertification record — each agent's access affirmed or withdrawn by a named human, with full provenance.

Two ownership events intersect access administration directly. Owner attestation — the suggested-owner queue in Principals & Ownership lists owner-unattested agents with an Attest owner action; attesting establishes or confirms the E0 root. Succession on departure — when a principal leaves, reassign their owned agents to a new owner; every reassignment re-derives E0 from the new owner, and the change flows through change control. The §2 invariant holds throughout: no sponsor → no run.

Cross-references: for the derivation internals and linking recipes see the Implementation Guide §10; for ownership and succession see the Principals & Ownership workflow (/principals); for the route-policy editor and signed-audit chain see Policy (/governance); for the capability-issuance scheme and API-key hash ledger see the agent detail Access & Policy tab.

11. Audit administration & compliance

Audit is the evidentiary backbone of the ACME governed workforce: every governed model call — allow or deny — is signed into a tamper-evident chain the moment the Policy Enforcement Point (PEP) decides it. This section is the administrator's reference for that chain — its post-quantum-hybrid signing scheme, its durable write-ahead log, its shipment to the Control Plane (CP), how to prove its integrity, where to browse and report on it, how to export security events to your SIEM, and how to meet retention and write-once (WORM) obligations. The goal is a chain of custody you can check, not merely trust: when an auditor asks "how do you know this record is real?", verification produces a cryptographic answer rather than an assurance.

11.1 The signed audit chain

Every governed call is signed into a tamper-evident audit chain using PQC-hybrid SoulKey signatures — Ed25519 + ML-DSA-44. Pairing the classical Ed25519 signature with the NIST-standardised ML-DSA-44 lattice signature makes each record verifiable today and resistant to a future quantum adversary, without waiting for a migration. Records are chained — each links to its predecessor — so any alteration, insertion, or removal breaks continuity and is detectable on verification.

The chain is buffered to a durable write-ahead log (WAL) at /data/audit-wal on the PEP host. The WAL is crash-durable: a signed call survives a PEP restart or unclean shutdown, and no record is lost between signing and shipment. The PEP continuously ships the signed chain to the CP at /cp/audit, where it lands in the CP's Postgres audit home (the store selected by TZT_AUDIT_SINK=postgres). Signing and shipping are independent of the data path — the PEP signs the call whether or not the forwarded backend request succeeds — so the audit trail is complete even for denied or failed calls.

Load-bearing durability. The WAL at /data/audit-wal must be on durable storage, and TZT_AUDIT_SINK must be postgres on the CP. An in-memory audit sink would lose the shipped chain on restart. See section 8 (Durability and restart) and section 12 (Backup/DR) — the CP Postgres is the system of record for the audit home and must be backed up.

11.2 Verifying chain integrity (Verify tab)

Verification is how you prove, cryptographically, that the audit trail has not been tampered with. Run it from the analyst/enforcement console's Verify tab.

1. Open the analyst console at pdp.tiresias.watch or the VM at :8343/admin/console (note: /admin/console, not /admin, which is the governed API and returns "api key required").
2. Paste the admin key (X-Admin-Key = TZT_ADMIN_KEY) into the #key field and click connect. Subsequent XHRs carry the key.
3. Select the Verify tab and run the check (control #v-run).

Result: The console reports the audit chain intact — signatures verified and the chain of signed calls continuous, with no call altered, inserted, or removed.

Run Verify on a schedule and after any incident, restart, or restore. If verification fails, treat it as a chain-of-custody event: preserve the WAL at /data/audit-wal and the CP audit home before any further writes, and escalate. A cross-check of posture is available on the Status tab, which reports the audit signer is present and signing alongside event and deny counts (for example, 11,765 events / 11,337 denies), SoulWatch posture, SIEM, and license.

11.3 The Audit & reports surface

Day-to-day audit review lives in the Asphodel admin console.

• Audit & reports (/reports) — the audit-chain browser and reporting surface. Browse signed records and produce compliance reports from the shipped chain.
• Policy (/governance) — the fleet governance cockpit surfaces the signed-audit count and the chain status (intact) per the last verification, alongside per-agent tier/rpm/policy summary and the compliance officer's latest sign-off. Its audit-trail browser lists records by decision / scope / status / timestamp, with a chain filter to scope the view.

Both surfaces are read projections of the same signed chain; neither can alter a record. Policy and budget writes (PUT /cp/admin/tenants/{slug}/policy, PUT .../agents/{slug}/budget) are themselves governed operations that appear in the trail. For the recertification workflow that consumes these reports, see the Reviews surface (/reviews) and section 7 (Policy & access governance).

11.4 SIEM export (RFC5424 syslog)

Enforcement and SoulWatch emit security events as RFC5424 syslog down a single path to your SIEM (or, in the reference stack, to the bundled Loki/Grafana):

PEP / SoulWatch  --RFC5424 syslog-->  Promtail :1514  -->  Loki  -->  Grafana

To export to your own SIEM, point the :1514/TCP receiver — or a syslog collector of your own — at your platform and consume the RFC5424 stream. Governed-call decisions, SoulWatch drift detections, and quarantine events all travel this path, making them queryable alongside cost data.

Security caveat. The :1514 syslog receiver carries no authentication — any host that can reach it can inject syslog lines. Firewall the port to trusted collectors only, and treat the bundled acme-promtail receiver as demo-scoped. The authoritative, tamper-evident chain of custody is the signed audit chain (WAL + CP audit home), not the syslog copy in Loki, which is a convenience log. SIEM export is license-gated (siem); confirm on the Status tab.

11.5 Retention and WORM guidance

The signed audit chain is the compliance system of record; retain it accordingly.

Because records are chained and PQC-hybrid signed, retention plus periodic verification yields a defensible, quantum-resistant evidence trail: the record's authenticity is provable from the signatures, and its completeness is provable from the unbroken chain.

12. Observability administration

The observability plane turns governed-call telemetry into an operator-facing signal in two forms: a security-event stream (behavioural drift, quarantine, and audit signals shipped as RFC5424 syslog) and an economics story (per-model cost rendered on the Grafana ROI board). The plane is deliberately read-mostly and downstream — it never sits on the governed request path. It consumes two derived streams: usage rows the control plane writes into tzt_usage, and syslog events the PEP and SoulWatch push into a log pipeline. Nothing an operator does here can weaken enforcement; telemetry lands, operators read.

This section is written against the reference stack ACME runs — Grafana + Loki + Promtail — but every component is a drop-in point for your existing equivalent. The enforcement plane emits standard RFC5424 syslog, so the first decision is where that stream goes.

12.1 Choose a destination: your SIEM or the example stack

The PEP emits RFC5424 syslog for every governed event — audit records, deny-by-default denials, SoulWatch drift verdicts, and quarantine actions all flow down the same path. You have two supported options, and they are not mutually exclusive:

You may forward to your SIEM instead of, or alongside, the example stack. For long-term compliance, forward the signed audit and syslog to immutable/WORM storage for retention in addition to whichever live destination you choose.

12.2 The example pipeline — syslog → Promtail → Loki → Grafana

In the reference deployment the observability plane runs as part of the acme-demo-stack docker-compose project on the demostack network. The components are:

Bring the pipeline up with the siem compose profile:

docker compose --profile siem up -d loki promtail grafana

Result: Promtail begins listening on :1514 for the PEP's RFC5424 stream, Loki aggregates it, and Grafana is reachable on :3000. SoulWatch drift-to-quarantine events ship down the same path, so quarantine and drift are queryable in Grafana alongside cost data.

To confirm the SIEM leg is flowing end-to-end, open the analyst Status tab (:8343/admin/console) and read the SIEM indicator — it reports the export pipeline (RFC5424 syslog → Promtail:1514 → Loki) as healthy when events are landing.

12.3 The ROI board and its datasource

The ROI board answers ACME's central business question — is self-hosting cheaper than paying per API call? — by rendering real per-model usage as a $/1M-token economics story.

Reach it at:

http://acme-demo-host:3000/d/acme-roi-economics

The dashboard uid is acme-roi-economics. Anonymous access renders the dashboard shell, but the data panels stay empty — the datasource query runs under an authenticated session. Log in as admin to see the numbers.

The board reads the tzt_usage table through a Grafana Postgres datasource named acme-tzt-usage, pointing at the shared tzt database. That table is the single source of truth for economics: every governed job ends with a usage report to /cp/usage/report, which the CP writes into tzt_usage (postgres-backed via TZT_USAGE_STORE=postgres, so the data survives restart).

The tzt_usage schema:

The board's panels:

Observed $/1M-token figures (live, will grow as the fleet runs): opus-4.8 $37, sonnet-4.5 $10, deepseek-v4-pro $0.62, gpt-oss:120b $0.44, gemma:free $0. The paid frontier tiers cost one to two orders of magnitude more per million tokens than the self-hosted tiers — which is the self-host-vs-API argument made concrete.

12.4 Building and adjusting dashboards

The ROI board ships pre-built, but you will extend it as the fleet grows. To add or edit a panel:

1. Log in to Grafana as admin at :3000 and open the ROI board (/d/acme-roi-economics).
2. Enter edit mode, then Add panel (or open an existing panel's menu → Edit).
3. Set the panel's datasource to acme-tzt-usage.
4. Write the SQL against tzt_usage, selecting the columns you need and grouping by model, agent, or provider lane as appropriate.
5. Apply the mandatory time filter on the bucket column — $__unixEpochFilter(bucket) (see §12.5).
6. Save the dashboard.

Result: the new panel queries live usage under your authenticated session and renders alongside the existing economics panels.

A minimal panel query, for a per-model spend rollup, looks like:

SELECT bucket AS "time", model, SUM(cost_usd) AS spend
FROM tzt_usage
WHERE $__unixEpochFilter(bucket)
GROUP BY bucket, model
ORDER BY bucket

If you render panels headlessly (for image export or scheduled reports), drop the color-background gradient cell option on table panels — the gradient does not render cleanly in the headless image path.

12.5 The $__unixEpochFilter gotcha

This is the single most common dashboard error, so it is worth stating plainly. The tzt_usage.bucket column stores epoch seconds, not a SQL timestamp. Grafana panels must therefore filter time with:

WHERE $__unixEpochFilter(bucket)

Do not use:

WHERE $__timeFilter(to_timestamp(bucket))   -- WRONG: mis-expands and errors

$__timeFilter expects a timestamp column and wraps the expression in a way that mis-expands against an epoch-seconds value, producing a query error and an empty panel. Any panel showing "no data" against a datasource you know is populated should be checked for this first.

12.6 Alerting guidance

The observability plane is where cost and behavioural signals become alertable. Two families of alert are worth standing up:

• Economics alerts (Grafana over tzt_usage) — alert on anomalous spend: a sudden rise in cost_usd per hour, a spike in paid-lane call volume, or an agent's spend crossing its expected envelope. These catch runaway escalation and misrouting before they show up on the invoice.
• Reliability alerts (Grafana over tzt_usage / Loki) — the free tier is nominally $0 but saturates under fleet load. hermes-405b:free in particular suffers chronic HTTP 429 rate-limit errors, which is why free agents home their fallback floor at the self-hosted ollama_oss tier. Alert on a rising 429 rate or on fallback-to-floor frequency so you can see reliability degradation as it happens rather than reading it after the fact. Treat $0 cost lines as "free but not dependable at scale."

Behavioural alerting — SoulWatch drift and auto-quarantine — is governed at the enforcement layer (composite behavioural risk ≥ 75 → auto-quarantine) and surfaced on the admin Observe exception queue and the analyst Detections and Quarantine tabs, rather than configured as Grafana alerts. Because SoulWatch's verdicts ship down the same syslog path into Loki, you can additionally build Grafana or SIEM alerts over the drift/quarantine log stream for teams that want a single alerting surface. See §9, Governance operations for the SoulWatch quarantine workflow, and the analyst Verify tab for audit-chain integrity.

Cross-references: SIEM/audit shipping and WORM retention are covered under production hardening; the usage-report path (/cp/usage/report → tzt_usage) originates in the runner and CP usage plane; SoulWatch drift and quarantine handling are detailed in §9 (Governance operations).

13. Durability, backup & disaster recovery

The governed workforce is designed so that a container restart, a host reboot, or a full VM loss never destroys governed state. Every plane that holds authoritative data is backed by Postgres; the runner's operating state is persisted to disk and baked into a pinned image; and every change is reversible through an image-tag + source-backup rollback pattern. This section documents what is durable, where it lives, how to back it up and restore it, and how to roll back to a known-good state.

13.1 Durability model — what survives a restart

Three stores hold authoritative state, on two Postgres instances plus one on-disk working set.

The PEP (acme-tiresias-zt) is deliberately stateless (TZT_STORE=memory); its durable state lives in the control plane, and its only local durable artifact is the crash-recoverable audit write-ahead log at /data/audit-wal, which is signed and shipped to the CP. Losing the PEP loses nothing — re-enroll it and it re-pulls its policy slice.

⚠️ Store-backed or it vanishes. If any CP *_STORE is left at its in-memory default, that state is lost on restart and agents "disappear." Confirm all six are postgres before relying on any backup (see §13.2).

13.2 The control-plane Postgres — system of record

The control plane owns durable governance state across six stores, all bound to a single Postgres DSN (TZT_POSTGRES_DSN). Confirm every store is postgres:

# tiresias-cp
TZT_REGISTRY_STORE=postgres     # the 367-agent registry (personas, tiers, keys' hashes)
TZT_ENROLLMENT_STORE=postgres   # PEP enrollment / pull tokens
TZT_USAGE_STORE=postgres        # metered spend/tokens (tzt_usage → ROI board)
TZT_AUDIT_SINK=postgres         # PQC-hybrid signed audit chain
TZT_QUARANTINE_STORE=postgres   # SoulWatch quarantine decisions
TZT_IDENTITY_STORE=postgres     # identity index (tzt_principal, 325,154 principals)
TZT_POSTGRES_DSN=postgresql://<user>:<pw>@<host>:5432/<db>

Because these six stores are the system of record, backing up acme-postgres backs up the fleet. The registry, enrollment tokens, usage history, audit chain, quarantine state, and the identity projection are all recoverable from one dump. Verify durability empirically:

1. Note the current agent count: GET /cp/admin/peps (site acme-site should report the 367-agent slice).
2. Restart the CP: docker compose up -d acme-tiresias-cp from /opt/acme/tiresias-zt/deploy/demo-stack/.
3. Re-query GET /cp/admin/peps.

Result: the agent count, usage totals, and audit chain length are unchanged — no re-provisioning is required.

13.3 The soul-svc Postgres — forced-RLS agent memory

Each agent's memory, chain-of-thought, and soul state live in Cloud SQL DB acme_soul behind soul-svc → PostgREST, with FORCE ROW LEVEL SECURITY on the content tables so even the table owner is subject to tenant policy. This is a separate system of record from the CP: it holds what agents remember, tenant-isolated all the way to the row.

• Managed backups: Cloud SQL automated backups + point-in-time recovery cover acme_soul. Do not disable them.
• Restore/recovery writes and cross-tenant inserts require SET ROLE service_role via the DSN option options='-c role=service_role'; on managed Postgres you cannot create a BYPASSRLS role, so grant service_role and set it per-connection. Never disable RLS to restore data — a bulk load done as the plain owner is still filtered, silently dropping rows.
• Keep the firewall authorized-networks at your-ops-ip/32; add an ops IP only for the duration of DB work, then restore it.

13.4 The runner — reboot-durable working set

The scheduler (acme-runner-sched) and fleet runner (acme-fleet-runner) share the baked acme-runner:latest image and run restart=unless-stopped, so both come back after a host reboot. Their dependencies persist on the host at /opt/acme/acme-runner-data/:

⚠️ fleet-state.json must persist. It tracks the batch cursor and FLEET_MIN_GAP_HOURS timing. If it is lost, agents over-run their ~1–2 governed-jobs/day pacing after a restart. Include /opt/acme/acme-runner-data/ in the host backup.

Bundled config (providers, models, routing, pricing) is baked into the image at runner/config.json; a mounted ACME_CONFIG overrides it without a rebuild. agent-creds.json is the only copy of each agent's capability_secret (the CP stores hashes, not secret material) — treat it as a secret-tier backup artifact.

13.5 Backup procedure

Run on a schedule and before every change window.

1. Dump the CP system of record:

bash docker exec acme-postgres pg_dump -Fc -U <user> <db> > cp-$(date +%Y%m%d).dump 2. Confirm Cloud SQL automated backup + PITR is enabled for soulsvc-db (managed; no manual dump needed for routine DR). 3. Snapshot the runner working set:

bash tar czf runner-data-$(date +%Y%m%d).tgz /opt/acme/acme-runner-data/ 4. Confirm the deployment sources are pushed to GitHub (compose files, runner/, config.json, envelope config).

Result: three restorable artifacts (CP dump, Cloud SQL PITR window, runner-data archive) plus a source backup — the full recovery set.

13.6 Restore procedure

1. Bring up Postgres and restore the CP dump:

bash docker exec -i acme-postgres pg_restore -c -U <user> -d <db> < cp-YYYYMMDD.dump 2. Restore agent memory from Cloud SQL PITR to the target timestamp (console or gcloud sql), keeping RLS forced throughout. 3. Restore /opt/acme/acme-runner-data/ from the archive, then start the stack: docker compose up -d. 4. Re-enroll the PEP if it was rebuilt (it re-pulls its policy slice from /cp/policy).

Result: the fleet returns to its backed-up state — registry, usage, audit chain, quarantine, memory, and runner pacing all consistent. Verify per §13.2 step 1 and the Verify tab (audit chain intact).

13.7 Rollback pattern (image tag + source backup)

This deployment rolls back through two coupled handles rather than in-place edits, so any change is reversible:

• Image tag — services run pinned, dated image tags (e.g. soul-svc:rls-complete-20260717, acme-runner:latest, saluca/tiresias-zt:demo). The prior tag is retained, so a bad rollout is reverted by re-pinning the previous tag and docker compose up -d <svc> — no rebuild.
• Source backup — the deployment sources (compose, envelope/config) are backed up in GitHub; a config or envelope change is reverted by checking out the prior commit and re-applying.

Envelope and CP changes are made rollback-safe: stage behind a canary at 0% traffic, ramp 10 → 50 → 100 with a go/no-go at each step, and keep the previous image tag pinned as the instant fallback. If a stage fails its check, re-pin the prior tag (or revert the source commit) and re-ramp.

Cross-references. Store/DSN configuration — §4 (Control Plane) and the environment reference. Forced-RLS provisioning and the service_role option — §5 (Data plane) and the troubleshooting gotchas. Audit-chain verification after restore — the analyst Verify tab (§3).

14. Security hardening & RBAC

This section defines the security posture required to run the ACME governed AI workforce beyond a demo: how secrets are sourced and rotated, how row-level isolation is forced, how the management console is authenticated with OIDC SSO, how the two-plane RBAC boundary is enforced, and how the deployment is exposed on the network. Apply every control in this section before the fleet handles production data.

The controls compose. No single control is the boundary: secrets keep credentials out of reach, forced RLS keeps tenants isolated even from a compromised owner role, OIDC removes standing privilege from the console, control-plane RBAC re-evaluates every forwarded call deny-by-default, and network scoping keeps the sensitive planes off the public surface. Assume any one layer can fail and the next must still hold.

14.1 Secrets management

Every secret the platform consumes — TZT_ADMIN_KEY, the LDAP bind password, provider API keys (OPENROUTER_API_KEY, OLLAMA_API_KEY, HF_TOKEN), the OpenAI embeddings key, all database DSNs, the soul-svc bearer tokens, and each agent's capability_secret — MUST be sourced from your secrets vault (HashiCorp Vault, cloud KMS, or a managed Secret Manager). Never commit a secret to git and never emit one to a printed environment dump, a log line, or a shell history.

The provisioning call (POST /cp/admin/tenants/{slug}/provision, Section 5) returns api_key and capability_secret exactly once. Capture them programmatically into the runtime creds store; do not echo them. On the runner host the store is agent-creds.json under /opt/acme/acme-runner-data/, written owner-only (mode 0600) because it holds every tenant's cap_secret and soul-svc bearer in plaintext — a group- or world-readable creds file is a finding.

To confirm the creds store is not leaking material:

1. On the runner host, inspect the file mode: bash stat -c '%a %n' /opt/acme/acme-runner-data/agent-creds.json
2. Confirm the result is 600 (owner read/write only).
3. Grep your CI and container logs for known secret prefixes to confirm none were printed.

Result: the creds file is 0600 and no secret material appears in logs, git history, or environment dumps.

Rotation

Rotate every secret on a fixed schedule and immediately on any suspected exposure.

1. Write the new secret value into your vault.
2. Roll the consuming service so it reads the new value at start (CP, PEP, console BFF for TZT_ADMIN_KEY; runner for provider keys).
3. For an agent capability secret, re-provision or re-issue through the control plane, then update the runner creds store.
4. Revoke the old value in the vault once every consumer is confirmed on the new one.

Result: the secret is rotated with no window in which both the old and new values are simultaneously valid and unmanaged.

14.2 Forced row-level security

The soul-svc data tier (acme_soul) runs with FORCE ROW LEVEL SECURITY on the content tables _memories and _soul_api_keys. Forced RLS applies the tenant predicate to every role including the table owner, so per-tenant isolation holds even if an owner role is compromised. On Cloud SQL the postgres role is also subject to FORCE RLS — it does not bypass row-level security the way a self-hosted superuser would, and Cloud SQL will not let you create a BYPASSRLS role at all.

Rules:

• Keep FORCE ROW LEVEL SECURITY on the content tables. Do not disable it for convenience.
• Never grant a standing BYPASSRLS role. Any legitimate cross-tenant read, write, or maintenance must assume service_role explicitly and per-connection.
• Scope service_role to the single connection that needs it — do not leave it set on a shared pool.

For a cross-tenant operation, set the role per-connection rather than provisioning a privileged standing role:

# psql: assume service_role for this connection only
PGOPTIONS="-c role=service_role" psql "$ACME_SOUL_DSN" -c "…"

-- inside a transaction, scoped and reset
SET LOCAL ROLE service_role;
-- cross-tenant statement here

Forgetting the service_role assumption is not a security failure but a visibility one: rows are silently invisible even to postgres. This forced-RLS posture pairs with Cloud SQL authorized-networks discipline (Section 10) — the database is both network-restricted and row-level-isolated. See also Section 4.3 and the RLS gotcha in Section 12.

14.3 Console SSO and the BFF-only admin key

The Asphodel management console authenticates its session identity via OIDC (or demo-auth in the lab). Read-only DEMO mode renders without login by having the guest read through the BFF with the admin key; this is a demo simplification and MUST be disabled in production.

Two rules define the production console posture:

1. Require login. Set ASPHODEL_DEMO_FALLBACK=0 so a guest is never served data without an authenticated OIDC session. Wire the BFF to your OIDC IdP (Okta, Entra ID, Ping, or any compliant provider).
2. Keep the admin key server-side. TIRESIAS_ADMIN_KEY is consumed by the BFF only and MUST never reach the browser. When a user is signed in, the BFF carries the caller's own scoped bearer and holds no standing privileged credential.

To harden the console:

1. Configure the console BFF environment (server-side / BFF only — none of these are exposed to the browser): ini TIRESIAS_API_URL=http://tiresias-cp:8343 TIRESIAS_ADMIN_KEY=<read-only admin key> # BFF-only, never browser-exposed ASPHODEL_CUSTOMER=acme ASPHODEL_DEMO_FALLBACK=0 # REQUIRE login — no guest fallback
2. Point the BFF's OIDC client at your IdP and map IdP groups/claims to console roles.
3. Restart the console container and load console.asphodel.ai (or the VM :3001) in a clean browser session.
4. Confirm you are redirected to the IdP login and that no page renders data before authentication.
5. In browser dev tools, confirm no request carries X-Admin-Key and that TIRESIAS_ADMIN_KEY never appears in any client bundle or network payload.

Result: the console requires an OIDC login, the admin key stays server-side, and a signed-in user's calls carry only that user's scoped bearer.

14.4 RBAC — the plane gate and control-plane deny-by-default

The console is one role-adaptive Next.js app with two planes (Personal and Admin); the plane is a property of the URL. Two independent mechanisms enforce the boundary:

• Console plane gate. ADMIN_ROUTE_PREFIXES single-sources plane derivation and drives a server-side redirect gate. A non-admin session never receives a control-plane route (/observe, /fleet, /principals, /access, /governance, /reports, /runtime, …) — the redirect happens server-side, so an admin route is never delivered to a non-admin browser.
• Control-plane deny-by-default RBAC. The real boundary is the control plane's own RBAC on every forwarded call. The BFF carries the caller's own scoped bearer; there is no standing privileged credential when a user is signed in. Even if a route were reached, the control plane re-evaluates the caller's scope deny-by-default per call and forwards nothing the caller is not entitled to.

Treat the plane gate as a UX and defence-in-depth layer and the control-plane RBAC as the authoritative boundary. Do not rely on hiding routes alone.

To verify the boundary:

1. Sign in as a non-admin (Personal-plane) user.
2. Request an admin route directly, e.g. GET /observe.
3. Confirm the server redirects away from the control-plane route rather than rendering it.
4. Confirm any direct control-plane API call made with that user's bearer is denied by the control plane, not merely hidden by the UI.

Result: the non-admin is both redirected off the admin route and denied at the control plane — the boundary holds at two independent layers.

14.5 Network exposure

Publish only the surfaces that must be reachable, and keep the sensitive planes in-network. Publish the PEP (:8343), the console (:3001), and the observability UIs; keep the Control Plane, soul-svc, and Postgres off the host.

The published observability ports carry no gateway auth in the demo and are demo-scoped — front them with authentication before production. The internal PEP→CP hop runs cleartext (TZT_ALLOW_CLEARTEXT_CP=true) solely because it never leaves the demostack network; this is not a general cleartext allowance. Do not add addresses to the Cloud SQL authorized-networks list beyond your-ops-ip/32; if you add an ops IP for maintenance, restore the list afterward (Section 10).

14.6 TLS and LDAPS

• TLS everywhere on the public surface. Terminate TLS at the gateway for the PEP, the console, and any published observability UI. DNS + TLS for the public hostnames (pdp.tiresias.watch, console.asphodel.ai) is a prerequisite for production.
• LDAPS only. Directory sync connects to Samba-AD over ldaps://acme-directory-internal:636, base DC=acme,DC=example. Access is TLS-encrypted and confined to the VPC via the internal address — the directory is never published to the internet. Expose LDAPS on 636 to the control-plane host only (Section 9).
• Cloud SQL stays restricted to authorized networks and row-level-isolated (Sections 10 and 14.2).

To confirm the transport posture:

1. Confirm the PEP and console are reachable only over TLS at their public hostnames.
2. Confirm the directory sync uses ldaps://…:636 and rejects a plaintext LDAP bind.
3. Confirm the Control Plane admin API is not reachable from the host — only in-network from the PEP and the console BFF.

Result: public surfaces are TLS-terminated, the directory is LDAPS-only over the VPC, and the control-plane admin API is unreachable from outside demostack.

Cross-references. Directory sync and LDAPS — Section 9. Cloud SQL firewall discipline — Section 10. Exposure posture and full port map — Section 11. Forced-RLS and site-slug gotchas — Section 12. Provisioning and secret issuance — Section 5.

15. Troubleshooting

This section is the diagnostic reference for the ACME governed AI workforce. Use the symptom index to locate a failure mode, then follow the matching subsection for cause, fix, and a verification command that confirms the repair. Every procedure ends with a Result: line stating what a healthy system reports. Most fixes are configuration corrections, not code changes — the platform fails closed by design, so a misconfiguration surfaces as a denied call or an honest empty state rather than a silent ungoverned path.

15.1 Symptom index

15.2 "Unknown API key" — wrong site slug

Cause: The agent was provisioned with "site": "acme" (or another value) rather than "site": "acme-site". The tenant is then outside the PEP's enrolled policy slice, so the two-factor call auth cannot resolve the API key and the call is denied deny-by-default. A stale cached slice on the PEP produces the same symptom.

1. Confirm the PEP's enrolled site and the tenant roster.

bash curl -s -H "X-Admin-Key: $TZT_ADMIN_KEY" \ "$TZT_CP_URL/cp/admin/peps"

Verify the enrolled slice is acme-site and the affected tenant slug appears under it. 2. If the agent was provisioned to the wrong site, re-provision it with the correct body (Section 5):

bash curl -s -X POST -H "X-Admin-Key: $TZT_ADMIN_KEY" \ -H "Content-Type: application/json" \ -d '{"tier":"standard","customer":"acme","site":"acme-site","backend":{"kind":"soul-svc","url":"...","auth":{"type":"bearer","token":"..."}}}' \ "$TZT_CP_URL/cp/admin/tenants/{slug}/provision" 3. If provisioning was correct but the PEP is serving a stale slice, the PEP re-fetches /cp/policy on ETag change — confirm the CP change has propagated.

Result: GET /cp/admin/peps lists the tenant under acme-site, and the agent's next governed call is authorised rather than returning 401 "Unknown API key".

15.3 RLS insert or cross-tenant read fails

Cause: The data tier enforces forced row-level security (FORCE ROW LEVEL SECURITY) on the memory and key tables. On managed Postgres even the postgres superuser is subject to FORCE RLS and cannot be granted BYPASSRLS, so any cross-tenant INSERT, read, or maintenance silently returns nothing or fails the row policy.

1. Assume service_role per connection by adding the DSN option:

options='-c role=service_role' 2. Never grant standing BYPASSRLS; scope service_role to the individual connection that needs it. 3. Verify the active role on the connection:

sql SELECT current_user, current_setting('role');

Result: current_setting('role') returns service_role and the cross-tenant write commits. Ordinary per-tenant calls continue to see only their own rows.

15.4 State lost on restart

Cause: One or more control-plane stores is set to memory. All CP stores must be postgres for durability; a memory store discards registry, enrollment, usage, audit, quarantine, or identity data on restart.

1. Confirm every CP store variable is postgres: TZT_REGISTRY_STORE, TZT_ENROLLMENT_STORE, TZT_USAGE_STORE, TZT_IDENTITY_STORE, TZT_AUDIT_STORE, TZT_QUARANTINE_STORE.
2. Confirm TZT_POSTGRES_DSN points at the durable CP database and the CP reports healthy:

bash curl -s "$TZT_CP_URL/healthz" 3. Restart the CP and confirm the agent roster survives.

Result: /healthz is ok, all six stores read postgres, and provisioned agents, usage rows, and audit entries persist across a CP restart.

15.5 Grafana ROI board shows no data

Cause: The tzt_usage bucket column is an epoch-seconds time bucket. Panels using $__timeFilter(to_timestamp(bucket)) filter incorrectly and return nothing. Separately, an anonymous session renders the dashboard shell but cannot run the datasource query.

1. Edit each panel query to use the epoch macro:

sql WHERE $__unixEpochFilter(bucket) 2. Log in as admin in Grafana — anonymous access renders the shell only; the Postgres datasource query needs an authed session. 3. For headless/scheduled renders, drop gradient cell styling so panels render reliably.

Result: The ROI board at :3000/d/acme-roi-economics shows total spend, per-model $/1M-token, and per-agent spend.

15.6 Free-tier 429 saturation

Cause: Free provider tiers are nominally $0 but unreliable under fleet load; hermes-405b:free in particular throws chronic HTTP 429 (rate-limit), and retired models return 410. This is the concrete driver of the self-host argument.

1. This is handled by design: on any LLM 4xx/5xx (including 429/410/outage), the runner's fallback-to-floor mechanism (Section 5) routes the job to the agent's reliable floor model rather than failing.
2. Confirm the floor is a reliable self-hosted or paid lane (e.g. ollama_oss), not another free tier, in runner/config.json.

Result: Free-homed agents degrade to their floor and complete governed work; runner logs show -> governed write and batch done: N ran despite upstream 429s.

15.7 Console admin-key exposure

Cause: The console admin key (TIRESIAS_ADMIN_KEY) must be BFF-only and never reach the browser. If it appears in a client bundle or a browser network request, the deployment is misconfigured.

1. Confirm TIRESIAS_API_URL and TIRESIAS_ADMIN_KEY are set as server-side environment only; the browser calls the console's own /api/cp/* BFF, which injects the key server-side.
2. In production, replace the demo admin-key read path with OIDC so no standing privileged credential exists when a user is signed in; set ASPHODEL_DEMO_FALLBACK=0 to require login.
3. Rotate TZT_ADMIN_KEY/TIRESIAS_ADMIN_KEY immediately on any suspected exposure.

Result: Browser dev tools show only calls to /api/cp/* with no X-Admin-Key header; the key never leaves the server.

15.8 Access explorer shows "stub"

Cause: A stub confidence badge in the Access explorer means the agent is not yet linked to a directory principal — an honest state, not a failure. Without an owner principal, E0 (the owner's real directory entitlements) cannot be derived.

1. Link the agent to its owner principal (Section 4):

bash python scripts/identity/link_agents.py

This sets owner_principal_id; E0 then derives from the owner's LDAP groups, dept/OU, region, and role. 2. Re-open the Access explorer for the agent and confirm the badge.

Result: The Access explorer reports live (or partial) confidence with real E0 entitlements, and the monotonic-narrowing chain E0 ⊇ E1 ⊇ rules → effective resolves.

15.9 Analyst console path

Cause: :8343/admin is the governed API, which correctly answers "api key required" — it is not the console. The analyst/enforcement console lives at /admin/console.

1. Open pdp.tiresias.watch or the VM :8343/admin/console.
2. Paste the admin key (X-Admin-Key = TZT_ADMIN_KEY) into the #key field and click connect; subsequent XHRs carry the key.

Result: The Status tab shows SoulWatch posture ENFORCING with event/deny counts, audit signer, SIEM target, and license.

15.10 Runtime not configured

Cause: An agent detail page shows "RUNTIME NOT CONFIGURED" (rather than "WIRED · RUNNER CONTROL") because the console has no runner control token. Live run/enable actions and the Integrations page require it.

1. Set both RUNNER_CONTROL_URL and RUNNER_CONTROL_TOKEN on the console (server-side).
2. Reload the agent detail page or the Integrations page (/runtime), which polls status every 5 s.

Result: The agent detail page reads "WIRED · RUNNER CONTROL", and the Integrations page shows live scheduler status and a per-agent run table.

15.11 Change-svc not connected

Cause: The Change & approvals inbox (/approvals) degrades to "not connected" when the change-control service (Contract B) is absent or unreachable. Attestation, reviews, and succession actions authored through change control are unavailable.

1. Set CHANGE_SVC_URL on the console and confirm the change-svc is reachable from the console network.
2. Reload /approvals.

Result: The approval inbox renders pending change items; owner attestation, access reviews, and succession-on-departure actions route through change control.

15.12 Verification command reference

See also Section 8 (behavioural monitoring and quarantine release), Section 10 (observability and ROI), and Section 11 (operations, backup, and DR) for the operational context behind these fixes.

16. Reference — environment, ports, API, CLI

This section is the consolidated reference for the ACME governed-AI-workforce deployment: every environment variable that shapes runtime behaviour, the port map for the acme-demo-stack compose project, the Control-Plane (CP) admin API, the runner control API, and the operator command-line surface. Values are drawn from the running instance and the repository (runner/.env.example, runner/config.json, runner/control.py, runner/provisioning.py, runner/runner.py, scripts/identity/). Do not substitute values that do not appear here or in the repo.

Two secrets recur throughout and must never be exposed to a browser or a log: TZT_ADMIN_KEY (the CP X-Admin-Key) and RUNNER_CONTROL_TOKEN (the runner X-Runner-Token). Both fail closed — a missing token disables the surface it guards rather than opening it.

16.1 Environment variable reference

Control Plane and enforcement (TZT_*)

Every CP store above is set to postgres in production. The in-memory default does not survive a restart, so these settings are load-bearing. DSN: postgresql://tzt:tzt@postgres:5432/tzt.

Per-agent and app-fleet credentials

<SLUG> is the persona slug upper-snake-cased (security-analyst → SECURITY_ANALYST). App slugs: BILLING_SVC, INVENTORY_SVC, REPORTING_SVC, WEBHOOK_RELAY, PAYMENTS_GATEWAY, NOTIFICATION_SVC.

Runner behaviour and control API (ACME_*, RUNNER_*, FLEET_*)

New-agent provisioning (runner POST /control/agents)

Required only for the create endpoint; unset values make it return 503 ("provisioning not configured"). Playground and persona edits work without them.

Console / BFF (ASPHODEL_*, server-side only)

Never expose these to the browser; the admin key is BFF-only.

Provider keys and overrides

16.2 Port reference

All host-published ports below are on the acme-demo VM (acme-demo-host).

The two runner containers (acme-runner-sched, acme-fleet-runner) expose no inbound socket; they are pure outbound clients of the PEP and the LLM providers.

16.3 Control-Plane admin API (/cp/admin/*)

All routes require header X-Admin-Key: $TZT_ADMIN_KEY. Base host tiresias-cp:8343 (in-network) or via the console BFF proxy /api/cp/*.

Quarantine release is a PEP admin endpoint, not /cp/admin. The analyst console (:8343/admin/console → Quarantine tab) issues:

POST /admin/quarantine/{id}/release
X-Admin-Key: $TZT_ADMIN_KEY

Access derivation. The Access explorer (/access) is composed by the console from CP reads — GET /cp/admin/identities (E0, the owner's real directory entitlements), the agent's authored scope_bindings (E1), and the PEP route policy (call rules) — yielding effective = E0 ∩ E1 ∩ rules. There is no single "access" write endpoint; each layer is governed through its own surface above.

The CP also exposes a PEP-facing slice consumed by the enforcement plane: POST /cp/enroll (redeem a single-use enroll token for a durable pull token), GET /cp/policy (the ETag-cached, site-scoped tenant slice), /cp/usage/report (metering), and /cp/audit (audit-chain shipping). These carry PEP credentials, not the admin key.

16.4 Runner control API (/control/*)

Served by runner.py serve on :8090. Every route requires header X-Runner-Token: $RUNNER_CONTROL_TOKEN; a mismatch is 401, and an unset token disables the surface with 503. GET /healthz is unauthenticated.

16.5 Command-line surface

There is no separate bin/tiresias binary in this deployment; operators drive the runner and identity tooling directly through Python entry points.

runner.py — the governed multi-agent runner:

python runner.py run-cycle [--cycle <id>]        # run all 7 showcase agents once
python runner.py run-agent <slug> [--cycle <id>] # run one agent once
python runner.py schedule --interval 3600        # loop run-cycle on a cadence (7-agent DAG)
python runner.py serve --port 8090               # token-gated control API + managed scheduler
python runner.py mindset list [--agent <slug>]   # list catalog + registered mindsets
python runner.py mindset register <slug>
python runner.py mindset query <slug> "<query>" [-k N] [--agent <slug>]
python runner.py mindset cleanup-tenant <slug>

fleet_runner.py — the 360-agent fleet runner (rolling batches governed by FLEET_BATCH / FLEET_INTERVAL / FLEET_MIN_GAP_HOURS):

python fleet_runner.py

Provisioning and identity scripts:

python runner/agent_manifest.py            # derive the fleet roster from tzt_principal
python -m appfleet.provision               # provision the 6 deterministic app identities
python scripts/identity/link_agents.py     # set owner_principal_id (links agents → owners; E0 live)
python scripts/identity/seed_directory.py  # seed a lab tzt_principal (native provider)

Batch agent provisioning reuses one DB connection and one HTTP session across the roster (see §5, Provisioning agents). After a directory seed, restart the CP so TZT_IDENTITY_PROVIDER=native takes effect.

Cross-references. Provisioning contract and the site=acme-site invariant: §4–5. Routing tiers and provider lanes: §6. Governance, quarantine, and audit: §7. Directory sync and agent→principal linking: §9. Troubleshooting the ungoverned-agent and forced-RLS gotchas: §12.