ADR-0002: CI/CD Pipeline and Supply Chain Security
Field | Value |
|---|---|
Status | Accepted |
Migrated from | Project-wide ADR-008 (CI/CD Pipeline) |
Context
CEREBRAL STRATUM requires a CI/CD pipeline capable of building and deploying containerised microservices across a multi-region OpenShift estate (ROSA HCP in ap-southeast-2, us-east-2, and eu-west-2, plus three on-premises SNO clusters for QA). The pipeline must satisfy supply chain security requirements appropriate for a platform handling sensitive customer location data, including cryptographic proof of build provenance for all artefacts delivered to production.
The platform runs natively on OpenShift, and the build tooling selection should align with that operational context to minimise integration overhead and operational complexity.
Decision
Tekton (OpenShift Pipelines) is selected as the CI/CD pipeline engine, with Tekton Chains providing automated supply chain security for container image artefacts. Red Hat Trusted Artifact Signer (RHTAS) will be used for signing non-container artefacts (firmware binaries, Helm charts, OTA update packages, etc.).
Tekton + Chains (container images)
Tekton Chains integrates natively with Tekton Pipelines and observes TaskRun completions to automatically:
Sign container images using Sigstore/cosign with keys managed in the cluster keystore
Generate SLSA provenance attestations (build inputs, pipeline definition, git commit, timestamp) and attach them to the image in the OCI registry
Produce in-toto attestations describing the build environment
This requires no additional pipeline steps — Chains operates as a controller watching the Tekton API, making provenance generation a side-effect of the normal build process rather than an opt-in step.
Attestation storage: Attestations are stored alongside the container image in the OCI registry (Quay.io or AWS ECR depending on region) using OCI referrers. This keeps provenance co-located with the image, avoids a separate attestation store, and is natively supported by both Quay and ECR. The signing backend is configured as oci in the Chains TektonConfig. SLSA predicate type slsaprovenance02 is used to align with current tooling defaults and downstream verifier compatibility.
Red Hat Trusted Artifact Signer (non-container artefacts)
RHTAS provides a managed Sigstore stack (Fulcio CA, Rekor transparency log, and TSA) deployed on OpenShift. It will be used to sign:
ESP32 firmware binaries (Rust/ESP-IDF build outputs)
Helm chart packages
OTA update manifests and delta packages
RHTAS allows signing with short-lived OIDC-bound certificates (via Fulcio) rather than long-lived key material, and records all signing events to the Rekor append-only transparency log — providing auditability without key rotation overhead. The Rekor instance is operator-controlled, satisfying data sovereignty requirements.
Admission enforcement
Kyverno will be used to enforce supply chain policy at admission time. A ClusterPolicy will require that all workload images:
Have a valid Cosign signature verifiable against the Chains signing key
Have an attached SLSA attestation in the OCI registry
Pods that do not satisfy both conditions will be blocked at scheduling time. This makes signing and attestation a hard deployment gate rather than a post-hoc audit check.
Consequences
Positive:
Provenance and signing are automatic for container builds; no per-pipeline boilerplate required
Attestations stored in the OCI registry (Quay/ECR) keep provenance co-located with the image with no additional storage infrastructure
SLSA attestations are machine-verifiable at deploy time via Kyverno
ClusterPolicy, making "signed and attested" a hard deployment gateRHTAS centralises non-container signing with audit trail via Rekor (operator-controlled), aligning with data sovereignty requirements
Keyless signing (OIDC-bound short-lived certs) eliminates long-lived signing key management overhead
Consistent with the broader Red Hat/OpenShift ecosystem already in use (Keycloak, AMQ Streams, Crunchy Postgres Operator, OpenShift Pipelines)
Negative / Trade-offs:
Tekton's YAML-heavy task/pipeline model has a higher initial authoring overhead compared to GitHub Actions or similar
RHTAS adds operational surface area — the Sigstore stack (Fulcio, Rekor, TSA) must be maintained on-cluster
Chains configuration (signing backend, OCI annotation format, SLSA predicate type) requires careful initial setup; changing predicate type post-deployment requires re-signing historical images
Kyverno
ClusterPolicyscope must be carefully defined to avoid blocking system/platform workloads that are not subject to the same signing requirements
Alternatives Considered
GitHub Actions with third-party signing: Would introduce an external CI dependency outside the OpenShift estate, complicating network policy and data sovereignty posture. Supply chain security would require explicit pipeline steps rather than automatic attestation.
Jenkins: Mature but not cloud-native; no native Chains equivalent. Signing would require custom scripting with no standardised attestation format.
Storing attestations in a separate store (e.g. Rekor only): Rejected in favour of OCI referrers — co-location with the image simplifies verification tooling and avoids a dependency on a separate lookup service at admission time.
Future Considerations
Kyverno policy scope: Initial
ClusterPolicywill target application namespaces only. Platform and operator-managed namespaces (e.g.openshift-*) require exclusion rules.RHACS evaluation: If adopted for runtime security, the interaction between Kyverno admission webhooks and RHACS admission control must be explicitly designed. A dedicated ADR will cover this decision.
On-device firmware signature verification: RHTAS provides the signing infrastructure for OTA firmware packages, but the ESP32-side verification (public key pinning or certificate bundle stored in NVS) is a separate design decision, addressed in the firmware security ADR.
Policy-as-code in GitOps: Kyverno
ClusterPolicyresources should be managed via the same GitOps pipeline as application manifests (OpenShift GitOps / ArgoCD), ensuring policy changes are versioned, reviewed, and auditable.ECR vs Quay registry strategy: The per-region registry strategy should be formalised as part of the multi-region deployment topology documentation.