ADR-0010: Cross-Region Observability Egress Boundary

2.1 Topology: one Alloy Deployment per region

Deploy Grafana Alloy as a namespaced Deployment per region — hub (au-canberra-core, on-premises) plus spokes us-east-2, eu-west-2, ap-southeast-2 today — not a single global instance and not a bare per-node DaemonSet for application/mesh telemetry. Each region's Alloy is reachable only via an in-cluster Service DNS name local to that region/namespace group (e.g. grafana-alloy.<region>-cerebral-stratum.svc.cluster.local:4317).

Applications send OTLP to their own region's Alloy — never across a region boundary. This is set via quarkus.otel.exporter.otlp.endpoint, supplied as an environment variable per-environment in the gitops Rollout manifests (not hardcoded in application.yml, so the same container image is region-portable).

2.2 Credential centralization

Each regional Alloy instance holds the sole export credential for its region — Grafana Cloud remote_write /OTLP credential for the three AWS spokes, or the local Grafana OSS stack's credential for au-canberra-core (see §2.5) — injected via the existing 1Password Operator/Vault pattern already used elsewhere in gitops (prod-core-secrets). Application pods in QA/prod never receive a Grafana Cloud credential — that pattern (QUARKUS_OTEL_EXPORTER_OTLP_HEADERS via hack_modules.sh) stays strictly %dev /developer-laptop-only, as it is today.

2.3 Scrub-then-forward boundary (the actual sovereignty decision)

Before any signal leaves the region, the regional Alloy pipeline classifies it against ADR-0005's two-tier model:

• Aggregate/non-attributable metrics (JVM stats, HTTP latency histograms, error rates, resource usage — no device_id/tenant_id /request-path-with-identifier content) ship centrally to Grafana Cloud without restriction. This is the ADR-0005 "aggregate/anonymised operational metrics" carve-out.

• Traces and logs are higher risk — span attributes and log fields can carry device_id, tenant_id, or identifier-bearing request paths (ADR-0005's named risk). An OTel Collector transform/attributes processor stage in each regional Alloy pipeline strips or hashes these fields before central export. Signals that cannot be safely scrubbed (or haven't been reviewed yet — see Open Items) are not exported centrally; they are retained in the region's own Grafana OSS stack (see §2.5) instead of Grafana Cloud.

• Mesh telemetry (Envoy access logs/metrics via OSSM, per platform ADR-0001) feeds the same regional Alloy instance and is subject to the identical scrub-then-forward rule — access logs carry request paths and must not bypass the boundary just because they originate from the mesh rather than the app.

2.4 PostgreSQL view scraping (backend ADR-0002) is unaffected

The existing cerebralstratum-backend ADR-0002 design (Alloy scraping per-tenant-schema PostgreSQL views, tenant_id promoted to a label) already produces tenant-labelled fleet metrics by design, for customer-facing dashboards accessed via scoped, per-tenant Grafana embed URLs (customer dashboards are inherently single-tenant-scoped at the presentation layer). This ADR does not change that flow. It governs the separate, lower-level application/infra/mesh telemetry path (JVM, HTTP, Envoy, logs) that platform ADR-0001 describes, which has no equivalent per-tenant scoping and is the path actually implicated in the ADR-0005 open item.

2.5 Regional storage backend, and the au-canberra-core exception

The three AWS spoke regions (us-east-2, eu-west-2, ap-southeast-2) run on ROSA HCP, so any scrub-failed traces/logs that need a region-local sink land on Loki/Tempo backed by native S3 — no additional storage backend is required, since S3 is already there. Loki and Tempo themselves are still net-new services to deploy and operate per spoke region, though: confirmed via a check of both infra and gitops that neither repo references Loki, Tempo, or MinIO anywhere today. That deployment work is called out explicitly in Forward Pointers below rather than folded into the Core-specific bullet, so it doesn't get silently dropped when this becomes implementation tickets.

au-canberra-core is different in three ways that this ADR treats as a single, related decision:

• It's on-premises SNO with no native S3. Red Hat OpenShift Data Foundation (Ceph) was considered and rejected for this role: on a single-node cluster already running the full app stack, Ceph's resource footprint competes directly with the workloads it's meant to support, and it's a paid OCP add-on rather than bundled entitlement. Object storage is instead provided by MinIO (Apache-licensed, no additional licensing cost, consistent with the platform's existing AGPL/Apache/MIT licensing posture) backing a self-hosted Grafana OSS + Loki + Tempo stack. Given Core's node count, MinIO runs as a single instance, PVC-backed, without erasure coding — this is an accepted non-HA component, consistent with the single-point-of-failure profile Core's SNO topology already carries for the app stack; this ADR does not introduce that trade-off, it inherits it. MinIO's object storage sits on the SSD tier only (Loki/Tempo indexing and recent-data queries are latency-sensitive; the HDD tier is not used unless a future ILM/tiering rule is added for genuinely cold, archival data). Retention on this store is tied to the same purge cadence as the platform's four-state data lifecycle grace period, not an independent retention clock — a concrete TTL is still an open item (see §5).

• It hosts a full QA and a full production cerebral-stratum stack simultaneously, on top of its hub duties (spoke lifecycle management, image builds via Tekton, ArgoCD hub management). Each app stack gets its own Alloy Deployment in its own namespace (qa-cerebral-stratum, production-cerebral-stratum) — QA and prod telemetry must not share a pipeline or credential on Core, exactly as they don't share one across the AWS regions.

• Core's production stack is treated as carrying real tenant-classified data, even though its actual traffic is BlueGuardian-employee-only. This is deliberate: it means the scrub-then-forward mechanism gets exercised end-to-end against genuine tenant-schema data before any customer-facing spoke goes live, rather than being validated for the first time against real customer traffic.

Reachability. Core's self-hosted Grafana OSS is exposed using the platform's existing pattern for Core-hosted services — Cloudflare Tunnel + a published application, routed through the OSSM ingress gateway into the mesh — rather than a new exposure mechanism. No bespoke ingress path is introduced for observability.

Positive

• Closes the ADR-0005 open item with a concrete, reviewable boundary instead of an implicit assumption.

• Directly explains and fixes the empty-Grafana-metrics symptom that triggered this investigation.

• Centralizes the export credential to N regional Alloy instances instead of N×pods.

• Gives backend/notification-dispatcher/device-registrar/device-simulator a single, consistent pattern for wiring %prod/%qa OTLP export (currently missing in all four).

• The au-canberra-core self-hosted Grafana OSS stack, built to satisfy the sovereignty boundary, doubles as the entire observability answer for ENTITLEMENT_MODE=community self-hosted operators — no separate design needed for that case.

Negative / Trade-offs

• Alloy becomes a new per-region component that itself needs monitoring — an Alloy outage in a region is now a telemetry blind spot for that region, with no automatic failover to another region's Alloy (crossing regions to fail over would itself violate the boundary this ADR sets).

• The scrub pipeline (OTTL/River transform config) becomes sovereignty-relevant code, not just infra config — it needs the same review rigor as application code handling tenant data, not a one-time setup.

• One additional network hop (app → regional Alloy → Grafana Cloud/regional Grafana OSS) versus direct export, though this is the same shape ADR-0001 already committed to for mesh telemetry.

• au-canberra-core now runs a materially heavier observability footprint (Alloy ×2, Grafana OSS, Loki, Tempo, MinIO) alongside dual app stacks, image builds, and hub-management duties, all on a single node currently provisioned at 80 vCPU / ~64GB (128GB planned) — see Open Items: this lands on an already-tight memory budget, not spare headroom.