ADR-0014: Human-on-the-Loop Delivery Model — Agent Orchestration, Merge Authority and Escalation
Field | Value |
|---|---|
Status | Accepted (2026-11-15, rev 5) — §5 item 1 signed off by the owner |
Date | 2026-11-15 |
Author | Alex Henshaw (drafted by Claude Code; revised in claude.ai review) |
Repo path |
|
Sync status | Synced from YouTrack CSPROD-A-48 (rev 4); this file is now canonical |
Relates to | ADR-0002 (CI/CD Pipeline and Supply Chain Security), QA-Gated Release Promotion Strategy (YouTrack draft CSPROD-A-35, numbered ADR-0006 there but not yet migrated; it takes a new number on migration because this repo's ADR-0006 is the Keycloak deployment topology ADR), ADR-0008 (Secrets Management), ADR-0009 (Build Concurrency and Object Pruning), ADR-0011 (Autonomous Dependency Update Pipeline) |
Supersedes (in part) | ADR-0011 (Autonomous Dependency Update Pipeline): the policy-file, merge-path, kill-switch and model-credential mechanisms, for the implemented flows — see §2.7 |
Tracked in | CSPROD-340 (drafting); Epic CSPROD-334 |
1. Context
CEREBRAL STRATUM has a single maintainer. Until November 2026 every step of delivery needed that maintainer: they read each Dependabot PR, merged it to qa, merged the gitops digest bump the Tekton pipeline opened, and watched the canary. CI was not the bottleneck. Approvals were.
ADR-0011 proposed removing the human from dependency review. CSPROD-334 broadens the aim. Delivery moves from human-in-the-loop, where Alex approves each step, to human-on-the-loop, where Alex sets policy, watches a digest, and is pulled in only when a change crosses a defined threshold.
By 2026-11-15 the first slice had been built and verified end to end:
CSPROD-333.
dependabot-automerge.yamlin frontend and backend triages Dependabot PRs deterministically:patch and Docker digest bumps auto-merge;
minor and non-semver bumps get a Claude review, which emits an
approveorhumanverdict;majors and workflow changes escalate.
CSPROD-331 and CSPROD-332. Only
qapushes open gitops bump PRs. Those PRs are authored by theblueguardian-co-tektonGitHub App, with signed (Verified) commits.CSPROD-342.
qa-bump-automerge.yamlin gitops checks each bump PR againstvalidate-qa-bump.sh, read frommain, and merges it, or labels itneeds-human-review.Verification. Dependabot PR (backend #141, frontend #146) → PaC build → App-signed bump PR (gitops #228) → automated merge (49d089e, Verified) → Argo sync → qa canary Healthy. No human acted at any step.
The implementation diverges from ADR-0011 as drafted (see §5). The broader model also covers work that ADR-0011 does not: agent-authored bug fixes via the AIDLC agent, and the orchestration role of Claude itself. This ADR records the model as a whole: who acts, who can merge where, when a human is pulled in, how decisions are audited, and how autonomy is switched off.
2. Decision
Autonomy extends to qa and stops there. Every autonomous merge is performed by a deterministic workflow under a GitHub identity, after machine-checkable validation. Deterministic rules set the highest tier a change can reach; a model's output can only hold a change at or below that tier, and is never sufficient on its own to merge. Production promotion, security boundaries, and decisions stay with the human.
2.1 Roles
Role | Actor | Does | Does not |
|---|---|---|---|
Owner | Alex | Sets policy (this ADR, guard policy, rulesets); merges prod promotions, workflow and security-boundary changes; answers escalations; reads the daily digest | Approve routine |
Orchestrator | Claude Code in the | Pulls work from YouTrack, hands implementation to the AIDLC agent, reviews its plans and PRs within guard policy, updates tickets with evidence, escalates | Merge to any branch; change guard policy, rulesets or its own permissions; handle secrets |
Implementer | AIDLC agent (Claude Code framework in | Plans and implements tickets in the sibling repos; opens PRs | Merge; touch protected paths outside its approved scope |
Reviewer (in CI) |
| Reads the diff and changelog; writes a verdict file | Push, comment, label or merge. Runs in a job with |
Automation | GitHub Actions workflows; Tekton PaC; Argo CD / Argo Rollouts | Triage, wait for checks, validate, merge to | Merge to |
2.2 Work types and human involvement
Work type | Path | Human involvement |
|---|---|---|
Dependabot patch / Docker digest | Triage → wait for PaC green → workflow merges to | None (digest only) |
Dependabot minor / non-semver | Triage → Claude verdict → PaC green → merge | None unless the verdict is not |
Dependabot bump of a package on the high-risk deny-list (§2.3.2) | Labelled | Always |
Dependabot major, | Labelled | Always |
gitops |
| None unless validation rejects |
Clear-cut bug (AIDLC bugfix / infra scope) | AIDLC plans → orchestrator approves the plan within guard policy → PR to | Pending CSPROD-336: merge authority for agent-authored code is not yet granted |
Production promotion ( | PR, whose body lists every autonomous merge it includes | Always: Alex merges |
ADRs, policy, guard policy, rulesets, secrets, credentials, Chains signing key and its public key on | — | Always |
2.3 Merge authority
Only workflows merge autonomously, and only into
qaor the gitopsqaoverlay. Each autonomous merge usesgh pr merge --merge --match-head-commit <sha>, so a head the checks did not cover can never be merged. Immediately before the merge call, the job re-reads the PR's labels and aborts ifneeds-human-reviewis present (--match-head-commitpins the commit, not the labels). The org ruleset allows only merge commits and requires signatures; GitHub signs the merge commit.Validation is deterministic and lives outside the PR under review.
Dependabot triage uses
dependabot/fetch-metadataand file paths, plus a high-risk package deny-list maintained onmain. It covers auth/identity (e.g. Keycloak and OIDC clients), crypto/TLS, persistence drivers and ORM, messaging clients, and serialization libraries. Any bump of a listed package is forced tohumanbefore the reviewer runs. This puts §2.4's identity, crypto and secrets trigger into effect for dependencies.The gitops validator is fetched from
mainthrough the contents API, so a bump PR cannot change the rules it is judged by. One identifier throughout: a bump is named by the first 12 hex characters of the new image'ssha256digest (<digest12>), in the branch, the PR title and the kustomization change. The source commit SHA is recorded in the PR body for humans, but is trusted only via the provenance attestation (check 5), never from the branch name. Checks run in this order, and the first failure rejects:a PR authored by the Tekton App, with every commit by the bot and verified;
a branch matching
bump/<component>-<digest12>, with a title containing the same<digest12>;exactly one digest line changed, under the right image name, with the new full digest starting with
sha256:<digest12>;at most a +1 Rollout
versionchange;provenance (key-based): Tekton Chains signs with a key pair. The Chains public key is committed to gitops
main(e.g..github/trust/chains-cosign.pub) and fetched through the contents API alongside the validator, so a PR can't swap it. Using thecosignCLI against the public Quay image (no registry credentials needed), the validator runscosign verify-attestation --key <pinned pub> --type <slsa predicate type>on the full digest. It then extracts the source repo URI and commit from the attestation's predicate (jq) and checks that:(a) the repo is the component's expected source repo;
(b) the commit is reachable from
qa, via the GitHub compare API (gh api repos/<org>/<repo>/compare/<commit>...qa, whereidenticaloraheadmeans reachable).
Check 5 is the only place the source commit is trusted;
a successful
kubectl kustomizeof the qa overlay. This runs only after the diff-shape checks pass, so kustomize never runs on an arbitrary PR.
Check 5 means a leaked Tekton App key or registry credential is not enough on its own to get an image into
qa. It does not protect against compromise of the Chains signing key itself (see §4 and §5 item 10). Thecosignbinary is installed throughsigstore/cosign-installer, pinned by commit SHA. Do not pipe it throughcurl | sh.Model output is advisory and fail-closed. A missing, malformed or non-
approveverdict is treated ashuman. The model cannot assign or raise a tier. The reviewer and the merge step run in separate jobs. The reviewer job holds onlycontents: readand the Claude token. The merge job holds the write token, never runs a model, and accepts only the enumapprove|humanfrom the reviewer's output; anything else counts ashuman.A human label always wins.
needs-human-reviewon a PR stops every merge path, including on re-runs and up to the moment of merge (rule 1).Identities. Bump PRs come only from the Tekton GitHub App, whose key reaches the cluster via 1Password Operator (ADR-0008). Orchestrator and implementer commit through separate GitHub Apps (
agentic-orchestratorandagentic-implementer, names TBC) viacreateCommitOnBranch. Each gives Verified commits attributable to an App rather than a person, and the audit trail can tell who planned from who built. This replaces the singleagentic-actionsApp.The orchestrator never holds merge authority, even where it could technically obtain it. Granting merge authority for agent-authored code is an owner decision (CSPROD-336).
2.4 Escalation triggers
Automation or the orchestrator stops and pulls in Alex when any of the following applies:
semver-major bumps, workflow or CI-tooling changes, or a
github_actionsecosystem update;a bump of a package on the high-risk deny-list (§2.3.2);
a validation reject (including a missing or invalid provenance attestation), a failed or timed-out check, or a refused merge;
conflict with an ADR, or a change that reveals an ADR needs amending;
anything touching tenant isolation, secrets or credentials, identity, or crypto;
anything bound for production;
a second failed attempt at the same fix;
cost above threshold: a single merge job running more than 110 minutes, or daily CI-review or model usage above a set budget (thresholds set in workflow config; reported in the digest).
Escalation has three parts:
a
needs-human-reviewlabel;one comment stating the reason;
a Slack notification in
#team-cerebral-stratum.
The Slack notification depends on urgency:
Immediate review goes out as a real-time post with an @-mention of the on-call reviewer (Alex today; a Slack user group later, so more maintainers can be added without workflow changes). Immediate review covers: validation or provenance rejects, refused merges, deny-list hits, and anything touching tenant isolation, secrets, identity or crypto.
Everything else is listed in the daily digest without a mention.
2.5 Audit trail
Record | Where |
|---|---|
Why a change was merged or held | The workflow run (triage decision, deny-list hit, verdict, validator |
Who merged |
|
Who authored agent work |
|
What work was asked for and its outcome | The YouTrack ticket: state transitions plus an evidence comment with PRs, SHAs and runs |
How the agent reasoned | AIDLC intent artefacts (requirements, reviews) and plan-approval records under |
What shipped | gitops history, Tekton Chains provenance (ADR-0002), Argo Rollouts history |
Daily summary | Scheduled Claude routine posting to Slack |
Digest liveness | Dead-man's switch: a separate scheduled GitHub Actions workflow, not the Claude routine, checks via the Slack API that today's digest has posted in |
2.6 Rollback and kill switch
Per PR. Add
needs-human-review.Per workflow.
gh workflow disable <workflow>in the relevant repo. The underlying flow reverts to human merges, with no code change needed.Org-wide (decided here, not yet built). An org-level Actions variable,
ON_THE_LOOP_AUTONOMY=auto|off, read by every merge job:offlabels instead of merging. Fail-closed: any value other than exactlyauto, including missing, empty or unreadable, is treated asoff. This supersedes ADR-0011's per-repoFACTORY_AUTONOMYfor the implemented flows. Implementation is a follow-up task. Verify first that thevarscontext resolves in Dependabot-triggered runs (§5 item 5).Signing-key revocation. If the Chains key is suspected compromised, remove or replace the pinned public key on gitops
main(an owner PR). From then on every bump PR fails check 5 and is held, which stops autonomous qa deploys until the key is rotated.Deployment rollback. Unchanged. Argo Rollouts aborts a failing canary; reverting the gitops bump commit rolls
qaback.Agent. AIDLC runs only when started by a human or the orchestrator. Stopping the session stops it, and its guard hooks remain in force.
2.7 Relationship to ADR-0011 (partial supersession)
ADR-0011 (Autonomous Dependency Update Pipeline) stays in force as the target state for the parts of the Dependabot pipeline that have not been built. For the flows that have been built (CSPROD-333, CSPROD-342), this ADR supersedes ADR-0011's mechanism sections. Where the two disagree on a superseded item, this ADR wins.
ADR-0011 section | Status | Replaced by |
|---|---|---|
§1 stage 5, Merge: GitHub native auto-merge ( | Superseded | §2.3.1: the workflow waits for the PaC checks itself, then merges with |
§2, risk tiers in | Superseded (location only) | Tiers live in the workflow YAML ( |
§5, per-repo | Superseded | The single org-level, fail-closed |
§6, | Superseded | The subscription OAuth token |
§1 stages 3–4, rework and independent re-review (T1) | Remains target | Not built. Today a failed check or a non- |
§4, two-App author/reviewer split with most-recent-push approval | Remains target | Not built for Dependabot. It is a prerequisite for agent-authored merge authority (§3, CSPROD-336). |
§5, shadow mode and the shadow → auto promotion thresholds | Remains target | Not built. The override-rate metric is tracked as §5 item 6. |
Everything else (context, trust boundary, high-risk list contents, firmware cap, cooldown, open items) | Unchanged | — |
ADR-0011 is still a YouTrack draft (CSPROD-A-43) and has not been committed to this repo. Whenever it is synced or next revised, it must carry a note saying it is superseded in part by ADR-0014 §2.7 and that it uses ON_THE_LOOP_AUTONOMY as its kill switch.
3. Alternatives Considered
Alternative | Reason rejected |
|---|---|
Keep human approval for every step (status quo) | The maintainer is the bottleneck. Routine bumps queued and went stale, which widened CVE exposure without adding proportionate safety. |
Repo-native auto-merge ( | The org ruleset has no required checks today (CSPROD-335 is open). Native auto-merge would then merge on approval alone. Merge jobs instead wait for the PaC checks themselves and merge by explicit head SHA. Revisit once the aggregate required check exists. |
Let Claude (orchestrator or CI reviewer) merge directly | Places merge authority in a model that reads untrusted text: release notes, PR bodies, issue text. A prompt injection could induce a merge. Workflows make the enforcement point deterministic. |
ADR-0011's two-App author/reviewer split for Dependabot PRs, with most-recent-push approval | Not needed for the Dependabot scope: no agent pushes rework to Dependabot PRs, so there is no self-grading risk there. Agent-authored work is different. The orchestrator already approves the implementer's plans, so this ADR separates their identities now (§2.3.5). The full author/reviewer split, including review approval rules, is required before agent-authored code gets merge authority (CSPROD-336). |
Validate gitops bump PRs with the validator from the PR head | A PR could weaken its own validator. Reading it from |
Gitops validation on diff shape and authorship only (no provenance check) | Trusts the Tekton App key and registry credentials completely. If either leaks, an attacker can open a well-formed bump PR to an arbitrary image. Checking provenance ties every merged digest to a signed build of a |
Keyless (Fulcio/Rekor) verification against RHTAS from CI | Chains already uses a key pair. Pinning the public key on |
Name bump branches by source commit SHA | Puts two different hashes (git SHA and OCI digest) side by side in one PR, which invites mismatch and confusion. It also tempts the validator to trust a commit SHA it read from a branch name. Naming by digest keeps one identifier end to end, and the commit is bound through provenance instead. |
Dead-man's switch inside the digest routine | A routine that has failed cannot report its own failure. The liveness check must run on an independent execution path. |
API-key billing for CI Claude | The subscription OAuth token ( |
4. Consequences
Positive
Routine dependency and qa deploy traffic needs no maintainer time. Alex's attention goes to the work types in §2.2 that genuinely need it.
Every autonomous merge is reproducible from the workflow log, attributable to an App identity, signed, provenance-verified (gitops), and bounded to
qa.Production risk is unchanged: the QA-gated release promotion ADR's (CSPROD-A-35)
qa→maingate and the production overlays stay human.Gives a template for extending autonomy (AIDLC bug fixes) one work type at a time, each with its own validator and escalation rule.
Public Quay images plus a pinned public key mean provenance verification needs no registry credentials and no external trust services in CI.
The same pinned public key can drive a Kyverno
verifyImagesrule, so CI and admission enforce one signer policy.
Negative / Trade-offs
The tests are the oracle. A repo with thin coverage ships its gaps to
qafaster. The qa canary and the human prod gate are the backstops.Merge jobs poll. They wait up to 110 minutes for PaC checks, which costs Actions minutes and couples to check-run names (
REQUIRED_CHECKS). An aggregate required check (CSPROD-335) would remove this.More surface to maintain: three GitHub Apps (Tekton, orchestrator, implementer), a validator with a provenance step, a pinned signing public key, a deny-list, three workflow copies, a Dependabot secret, a scheduled routine and its liveness workflow, and a Slack bot token for the liveness check.
A single-maintainer blind spot remains. A bad change that passes CI and the canary reaches
qawithout anyone noticing until the digest. This is acceptable becauseqais not customer-facing. Note, though, thatqastill holds credentials (e.g. via 1Password Operator), so "not customer-facing" does not mean "no blast radius". The deny-list and provenance check exist partly for this reason.The signing key is now load-bearing, and co-located. If the Chains private key sits on the same cluster as the Tekton App key (e.g. the
signing-secretsSecret inopenshift-pipelines), compromising that one cluster yields both. An attacker could then sign an arbitrary image and open a valid bump PR for it, which defeats check 5. A key pair also has no transparency log by default, so misuse of the key would leave no public record. Mitigation: a non-exportable KMS-backed Chains key (§5 item 10).Larger production promotions. Autonomous merges pile up on
qa, so the humanqa→mainPR grows and gets harder to review. To counter this, the promotion PR body lists every autonomous merge since the last promotion.Subscription-token coupling. CI review stops if the OAuth token expires or is revoked. It fails closed to
human, which is safe but noisy.Shared quota. CI review, the orchestrator, AIDLC and the digest routine all draw on one subscription's usage limits. Heavy interactive use can exhaust the quota and push CI review into fail-closed
humanverdicts across the board. This is another reason to move CI to Bedrock + OIDC.
5. Open Items
ADR-0011 status. Resolved (rev 5, owner sign-off 2026-11-15): ADR-0014 supersedes ADR-0011's mechanism sections (policy file, merge path, kill switch, token) for the implemented flows. ADR-0011 stays the target state for the rework stage, the author/reviewer split and shadow mode. A single kill-switch name,
ON_THE_LOOP_AUTONOMY, applies. The section-by-section mapping is in §2.7. Still to do: add the matching supersession note to ADR-0011 itself (YouTrack draft CSPROD-A-43, not yet in this repo).High-risk deny-list contents. Decided in principle (§2.3.2); the exact package list per repo still needs writing and committing to
main.AIDLC guard policy and merge authority (CSPROD-336). What the orchestrator can approve without Alex, and whether agent-authored bug fixes may be merged to
qaby automation. Prerequisite: the full author/reviewer split (§3).Aggregate required check (CSPROD-335). It replaces polling and enables native auto-merge.
Org-wide kill switch (
ON_THE_LOOP_AUTONOMY, §2.6). It needs implementing across the three workflows, fail-closed. First confirm thevarscontext is available to Dependabot-triggered runs; if it isn't, pick another source that Dependabot-context jobs can read.Override-rate metric. Track how often Alex overturns an automated decision, as ADR-0011's shadow-mode metric, before widening autonomy.
Firmware. Out of scope until on-device signature verification exists (ADR-0011).
Digest dead-man's switch mechanism. Resolved (rev 3): a Slack-based, independent Actions liveness workflow with an @-mention for immediate review (§2.5). Remaining implementation details:
the digest deadline time;
the Slack user group for the on-call reviewer;
the Slack bot token scope (read channel history, post) and where it is stored (ADR-0008).
Branch
<sha12>semantics. Resolved (rev 3):<digest12>, the image digest prefix (§2.3.2). Confirm the Tekton bump task already names branches this way; if it doesn't, update it alongside the validator.Provenance check: signing mode resolved; key custody open (rev 4). Chains signs with a key pair, so verification is
cosign verify-attestation --keyagainst a public key pinned on gitopsmain(§2.3.2 check 5). Still open:Key custody. Where the Chains private key lives today, and whether to move it to a non-exportable KMS key that Chains supports via a KMS URI (e.g. AWS KMS through STS/IRSA from ROSA, or Vault transit for the on-prem QA SNOs). Do this so a cluster compromise cannot extract it, and so it is never stored alongside the Tekton App key.
Transparency log. Whether Chains uploads to the RHTAS Rekor. If it doesn't, the validator verifies with
--insecure-ignore-tlog=trueagainst the pinned key. This is acceptable with a pinned key, but means there is no audit record of key use. If it does, decide whether CI should reach RHTAS Rekor (via Cloudflare) to check inclusion.Predicate format. Check the Chains config (
artifacts.taskrun.format/artifacts.pipelinerun.format). The SLSA predicate version it emits (v0.2materialsvs v1resolvedDependencies) sets the--typeflag and thejqpath used to extract the source repo and commit.Rotation procedure. Keep a short runbook: generate the new key (or a new KMS key version), update the Chains config, commit the new public key to
main(allowing old and new keys during overlap), then remove the old key.
6. Forward Pointers
CSPROD-339: AIDLC pilot on real tickets under the CSPROD-336 guard policy.
CSPROD-338: the daily digest routine (Slack
#team-cerebral-stratum), plus its independent liveness workflow.Bedrock + GitHub OIDC for CI Claude, replacing the subscription token. This also removes the shared-quota coupling.
Native auto-merge plus required checks once CSPROD-335 lands.
Kyverno / RHACS admission control as the runtime backstop (ADR-0002). This complements, not replaces, the validator's provenance check. Reuse the same pinned Chains public key in a Kyverno
verifyImagesrule.KMS-backed Chains signing key (§5 item 10).
Revision history
Rev | Date | Change |
|---|---|---|
1 | 2026-11-15 | Initial draft (Claude Code) |
2 | 2026-11-15 | claude.ai review changes: provenance check in gitops validator; high-risk deny-list moved from open item to decision; precise model-authority wording plus reviewer/merge job separation; label re-check before merge; separate orchestrator/implementer Apps; fail-closed kill switch; digest dead-man's switch; shared-quota and promotion-size trade-offs; cost-trigger thresholds; validator check ordering; ADR-0011 resolution proposed; open items 8–10 added |
3 | 2026-11-15 | Owner decisions: Slack escalation with @-mention for immediate review, plus an independent liveness workflow (item 8 resolved); bump branches named by image digest prefix, with the commit trusted only via provenance (item 9 resolved); provenance tooling set as cosign CLI against public Quay with the GitHub compare API (item 10 narrowed to trust root and signing mode) |
4 | 2026-11-15 | Chains confirmed key-pair signing: check 5 now verifies against a public key pinned on gitops |
5 | 2026-11-15 | Owner signed off the partial supersession of ADR-0011 (§5 item 1 resolved); added §2.7 mapping each ADR-0011 section to superseded or remains target; status Proposed → Accepted; the QA-gated release promotion reference now points at CSPROD-A-35, so it can't be confused with this repo's ADR-0006 |