Joshua Belke
b6443a823b
Some checks failed
control plane / chart (push) Has been cancelled
control plane / test (push) Has been cancelled
control plane / browser-e2e (push) Has been cancelled
control plane / Build control plane image (linux/amd64) (push) Has been cancelled
control plane / Build control plane image (linux/arm64) (push) Has been cancelled
control plane / Publish signed control plane image (push) Has been cancelled
helm chart / lint + unittest + render matrix (push) Has been cancelled
helm chart / install on kind (gated) (push) Has been cancelled
helm chart / publish chart to GHCR (push) Has been cancelled
Meridian Harness / Build (aarch64-unknown-linux-musl) (push) Has been cancelled
Meridian Harness / Build (x86_64-unknown-linux-musl) (push) Has been cancelled
Meridian Harness / Publish rolling release (push) Has been cancelled
Meridian Harness / Publish tagged release (push) Has been cancelled
CI / Detect Changed Paths (push) Has been cancelled
CI / Dead Token Reference Guard (push) Has been cancelled
Docker image / Build (linux/amd64) (push) Has been cancelled
Docker image / Build (linux/arm64) (push) Has been cancelled
Docker image / Build public push gateway (linux/amd64) (push) Has been cancelled
Docker image / Build public push gateway (linux/arm64) (push) Has been cancelled
CI / Rust Lint (push) Has been cancelled
CI / Unit Tests (push) Has been cancelled
CI / Isolated DB Gate (push) Has been cancelled
CI / Desktop Core (push) Has been cancelled
CI / Desktop Smoke E2E (1) (push) Has been cancelled
CI / Desktop Smoke E2E (2) (push) Has been cancelled
CI / Desktop Smoke E2E (3) (push) Has been cancelled
CI / Desktop Smoke E2E (4) (push) Has been cancelled
CI / Desktop (push) Has been cancelled
CI / Desktop E2E Relay (push) Has been cancelled
CI / Desktop E2E Integration (1/2) (push) Has been cancelled
CI / Desktop E2E Integration (2/2) (push) Has been cancelled
CI / Desktop E2E Integration (push) Has been cancelled
CI / Backend Integration (relay e2e) (push) Has been cancelled
CI / Relay E2E (push) Has been cancelled
CI / Web (push) Has been cancelled
CI / Admin Web (push) Has been cancelled
CI / Mobile (push) Has been cancelled
CI / Security (push) Has been cancelled
CI / Server Cross-Compile (push) Has been cancelled
CI / Server Cross-Compile-1 (push) Has been cancelled
CI / Windows Rust (x86_64-pc-windows-msvc) (push) Has been cancelled
CI / Desktop Build (macOS) (push) Has been cancelled
Docker image / Merge release multi-arch manifest (push) Has been cancelled
Docker image / Merge debug multi-arch manifest (push) Has been cancelled
Docker image / Publish public push gateway image (push) Has been cancelled
`meridian_zenoh_ready` was 1 and `/_readiness` `bus.ready` was true on both pods for the entire period during which cross-pod delivery was completely broken (meridian-ty6l). The cause is structural: `probe()` publishes on the session's own health key and is satisfied by that same session's subscriber, which Zenoh delivers without involving the router at all (`zenoh-1.8.0/src/net/runtime/mod.rs:325`). The probe proves the session object is alive and proves nothing about reachability. The obvious fix -- require a witness for readiness -- is wrong, because a single-relay deployment has no peer to witness it and would be held closed for ever. So the signal is split rather than tightened: - `bus_ready()` / `meridian_zenoh_ready` keep exactly their conditions and are now named as liveness in code, in `/_readiness` and in the runbook. Redis and Shadow paths are untouched: Shadow still answers from Redis alone. - A separate `ZenohReachability` is reported and gates nothing. It carries two router-external witnesses that fail independently. The declaration plane is `Publisher::matching_status()` on a publisher restricted to `Locality::Remote`, which `net/runtime/mod.rs:326` answers by `matches.values().any(|dir| dir.id != face_id)` -- every face but our own, and for a client session those faces are populated only by declarations the router forwarded. The data plane is the health subscriber widened from `_health/<own-zid>` to `_health/*`: a sample keyed to an id this process does not own cannot have been delivered locally. The peer map is bounded in both directions, since its key now arrives over the bus. - `meridian_zenoh_received_total` is registered at 0 for all eleven lanes at startup, and every new series likewise. An absent series and a zero series are the same picture on a dashboard and opposite facts, and the live re-probe hit exactly that ambiguity. `MeridianBusPeersUnreachable` pages when a pod reports ready, sees no peer on either plane, and more than one pod in the namespace has an open session. That last clause is the whole design: no single session can tell "alone" from "cut off", so the discriminator is the fleet's own pod count. A one-pod deployment never trips it. Listed in `MUST_BE_ALERTED` under a second, written-down criterion -- a metric whose failure to *move* is the fault -- with a runbook section and an inventory row, in this commit rather than a later one. The reproduction test keeps asserting the absence of delivery and now also asserts that reachability reads zero on every field while it happens. Its counterpart, `an_isolated_pod_is_indistinguishable_from_a_lone_one`, stands up one relay against a real zenohd and proves it reaches readiness with no witness -- the evidence that this must not gate. Verified: `just zenoh-check` 0, `just zenoh-router-check` 0 (3/3 against a live eclipse/zenoh:1.8.0), clippy `-D warnings` with `--features zenoh` 0, `just fmt-check` 0, `just check-alert-runbooks` 0 (35 alerts / 204 metrics), `just check-unwrap-budget` 0, `cargo test -p meridian-relay --lib` 889 passed. Refs: meridian-ty6l Signed-off-by: Joshua Belke <joshua@innovationhub-act.org> |
||
|---|---|---|
| .. | ||
| admin | ||
| assets/screenshots | ||
| benchmarks | ||
| formal | ||
| mips | ||
| nips | ||
| plans | ||
| runbooks | ||
| spec | ||
| AGENTS.md | ||
| bridge-channel-window.md | ||
| community-registration-launch-policy.md | ||
| control-plane-deployment.md | ||
| git-on-object-storage.md | ||
| linux-rendering-troubleshooting.md | ||
| MCP_DRIVEN_HOOKS.md | ||
| meridian-shared-compute-dev.md | ||
| multi-tenant-conformance.md | ||
| multi-tenant-relay.md | ||
| nostr-nip-support-matrix.md | ||
| projects-qa-runbook.md | ||
| push-gateway-deployment.md | ||
| ux-end-to-end-workflow.md | ||
| welcome-kickoff-silent-failures.md | ||
| workflow-variables.md | ||