RAID/R2D2-Holocron
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(deploy): reconfigure deploy skill for holocron r2d2 target

Browse source
This commit is contained in:
Joshua Belke 2026-06-02 00:57:31 -04:00
commit fab7c7c020
1 changed files with 149 additions and 388 deletions
Download patch file Download diff file Expand all files Collapse all files

533
.settings/deploy/skill.md
View file

@ -1,265 +1,134 @@
---
name: deploy-r2d2-app
description: Deploy the STANAG 4817 stack (Next.js docs site in web/ + the FastAPI validation API in interop/validation-api/, one Helm chart at the repo root) to the r2d2 cluster end-to-end — push code to Gitea (RAID/4817), build both images, push to Gitea's OCI registry, apply the Helm overlay, install into k8s, and verify the docs site AND the live API via the public ingress at 4817.r2d2.office.ilab.zone. Use when the user wants to ship the docs site and/or the validation API to git.office.ilab.zone + the cluster. Triggers on "deploy 4817 / the docs site to r2d2", "ship the docs site to the cluster", "redeploy the validation API", or invoking /deploy-r2d2-app. Do NOT invoke for non-r2d2 deploys or other clusters.
name: deploy-holocron
description: Deploy the Holocron tactical display (the holocron-frontend React/Cesium SPA + its Express/WebSocket MQTT bridge) to the RAID-K8-ILAB (r2d2) cluster end-to-end — build the frontend image, push it to Gitea's OCI registry, install the Helm chart at deploy/holocron, and verify the site + WebSocket + health via the public host holocron.r2d2.office.ilab.zone. Use when the user wants to ship Holocron to git.office.ilab.zone + the cluster. Triggers on "deploy holocron to r2d2", "ship the tactical display", "redeploy holocron", or invoking /deploy-holocron. Do NOT invoke for non-r2d2 deploys, the holocron-sim chart, or other clusters.
---
# Deploy the STANAG 4817 docs site to the r2d2 cluster
# Deploy the Holocron tactical display to the r2d2 cluster
End-to-end runbook for landing the stack at `4817.r2d2.office.ilab.zone`. Two
workloads sit behind **one ingress on one host**:
End-to-end runbook for landing Holocron at `holocron.r2d2.office.ilab.zone`.
1. **Docs site** — a **Next.js (fumadocs) standalone server behind an nginx sidecar**;
nginx on **:8080** proxies the app on **:3000**, the same shape in Docker Compose
and in k8s (nginx Ingress → Service :8080 → container :3000). Serves `/`.
2. **Validation API** — a **FastAPI service (uvicorn :8817)** deployed as a second
Deployment+Service (`stanag-4817-api`). The ingress routes the API path prefixes
(`/validate`, `/route`, `/fuse`, `/fusion`, `/correlate`, `/messages`, `/examples`,
`/versions`, `/publish`, `/harness`, `/monitor`, `/health`) to it; everything else
(`/`, `/docs`, `/archive`, `/api-reference`, `/testing`) stays on the docs app.
This is what makes the Scalar **Try it** panel work against the public host.
**What deploys:** the **holocron-frontend** Node app — an Express server (`server.cjs`)
that serves the built React/Cesium SPA + REST API on **:3001** and runs a separate
**WebSocket bridge on :9002** that relays MQTT asset updates to the browser. One Helm
chart (`deploy/holocron`) packages it as a Deployment+Service behind one **nginx
Ingress**, plus an in-cluster **mosquitto** broker (Deployment+Service) the bridge
subscribes to.
Both are in the **same Helm chart**; the API is gated by `validationApi.enabled`
(off in `values.yaml`, on in `values.override.yaml`). The docs and API images
version independently (separate OCI repos), but the canonical flow below builds
**both at the same `$TAG`** from one clean commit.
**Topology at the host:** the office edge (`174.77.95.250`) terminates TLS for the
wildcard `*.r2d2.office.ilab.zone` and proxies to the cluster's **nginx ingress**
(HTTP, port 80 — same pattern as `farsight.r2d2`, `flora.r2d2`). The ingress routes:
- `/ws` → Service port **9002** (WebSocket upgrade)
- `/` (everything else, incl. `/api`, `/health`, the SPA) → Service port **3001**
**All concrete values for this deploy live in `.settings/SKILL/deploy/.devploy.env`**
(gitignored). Source it once per shell before running anything:
**⚠️ IFRAME EMBEDDING:** Holocron is embedded as an `<iframe>` on
`r2d2.office.ilab.zone`. `server.cjs` emits **no** `X-Frame-Options` /
`frame-ancestors` CSP, and the chart adds **no** header annotations that would
block framing. **Never** add `X-Frame-Options` or a restrictive `frame-ancestors`
CSP (to the ingress, the server, or by routing through `holocron-proxy`, whose
nginx sets `X-Frame-Options SAMEORIGIN`). Doing so breaks the embed.
**All concrete values live in `.settings/deploy/.devploy.env`** (gitignored).
Source it once per shell:
```bash
set -a; . .settings/SKILL/deploy/.devploy.env; set +a
export KUBECONFIG=$KUBECONFIG
set -a; . .settings/deploy/.devploy.env; set +a
export KUBECONFIG
```
Everything hardened below was learned deploying real apps to this cluster; the
gotchas are inline.
## Scope guardrails
- **Cluster scope is strict.** Only ever create/modify resources in the `$RELEASE`
namespace (`stanag-4817`). Don't touch other namespaces, releases, or cluster-wide
objects beyond what this skill creates.
- **Outbound actions confirm-or-block.** Never `docker push`, `git push`, or
`helm upgrade` without the inputs in `.devploy.env`. If a credential is missing,
stop and ask — don't invent or scrape secrets.
- **No Claude attribution.** Commits made under this skill use the project's existing
`git config user.{name,email}` and carry no `Co-Authored-By: Claude*` / "Generated
with Claude Code" footer. The user's deploys must look like the user's.
- **Cluster scope is strict.** Only create/modify resources in the `$NAMESPACE`
namespace (`holocron`). Don't touch other namespaces, releases, or cluster-wide
objects. `kubectl get … -A` is a smell.
- **Outbound actions confirm-or-block.** Never `docker push` or `helm upgrade`
without the inputs in `.devploy.env`. If a credential is missing, stop and ask.
- **No Claude attribution.** Commits use the project's `git config user.{name,email}`
and carry no `Co-Authored-By: Claude*` / "Generated with Claude Code" footer.
## Key facts (this app)
| Thing | Value |
| ---------------- | ------------------------------------------------------------- |
| `RELEASE` / ns | `stanag-4817` (Service names are DNS-1035 — can't start with a digit, so not `4817`) |
| `HOSTNAME` | `4817.r2d2.office.ilab.zone` |
| `GIT_REPO_URL` | `https://git.office.ilab.zone/RAID/4817.git` |
| `IMAGE_FULL` | docs: `git.office.ilab.zone/raid/r2d2-4817` |
| `API_IMAGE_FULL` | API: `git.office.ilab.zone/raid/r2d2-4817-validation-api` |
| `MCP_IMAGE_FULL` | MCP: `git.office.ilab.zone/raid/r2d2-4817-mcp` |
| App source | `web/` — Next.js standalone, runs as user **node (UID 1000)** |
| API source | `interop/validation-api/` — FastAPI/uvicorn `:8817`, runs as **UID 1000** |
| MCP source | `interop/mcp-server/` — FastMCP Streamable HTTP `:8819`, **sidecar in the docs pod**, runs as **UID 1000** |
| Build context | **repo root** for ALL THREE (docs generator needs `STANAG/Draft-*/`; API + MCP read the spec pack / baked `message-docs.json`) |
| `DOCKERFILE` | docs: `web/Dockerfile` (relative to the repo-root build context) |
| `API_DOCKERFILE` | API: `interop/validation-api/Dockerfile` (relative to repo root) |
| `MCP_DOCKERFILE` | MCP: `interop/mcp-server/Dockerfile` (relative to repo root) |
| API service / k8s| Deployment+Service `stanag-4817-api` (`$RELEASE-api`), 1 replica, stateless (`MQTT_AUTOCONNECT=false`, `REDIS_URL=memory://`) |
| `CHART_DIR` | **repo root** (`.`) — Chart.yaml + values* live at the top (r2d2-fleet convention) |
| Helm values | `values.yaml` (defaults) **+** `values.override.yaml` (r2d2), both at repo root — apply both |
| `.helmignore` | excludes the non-chart sibling trees (`web/ STANAG/ robot/ …`) so helm doesn't load the whole monorepo |
| Persistence | **none** — stateless (read-only rootfs; `/tmp` + `.next/cache` are emptyDir) |
| Probe / verify | `/` (no `/healthz` on this app) |
| `RELEASE` / ns | `holocron` / `holocron` |
| `HOSTNAME` | `holocron.r2d2.office.ilab.zone` |
| `IMAGE_FULL` | `git.office.ilab.zone/raid/r2d2-holocron` |
| App source | `holocron-frontend/` — Express + React/Cesium, runs as **uid 1001** |
| Build context | `holocron-frontend/` (the Dockerfile is self-contained there) |
| `DOCKERFILE` | `holocron-frontend/Dockerfile.local` (multi-stage prod image) |
| Ports | HTTP/API/SPA **:3001** (`/health`), WebSocket **:9002** (`/ws`) |
| MQTT broker | in-cluster `holocron-mqtt:1883` (mosquitto, chart-deployed) |
| `CHART_DIR` | `deploy/holocron` (Chart.yaml + values* + templates/) |
| Helm values | `values.yaml` (defaults) **+** `values.override.yaml` (r2d2) — apply both |
| Ingress | nginx, host-based, HTTP only (edge does TLS). `/ws`→9002, `/`→3001 |
| Persistence | none (SPA + bridge are stateless; mosquitto data is emptyDir) |
| Probe / verify | `/health` (returns 200 even when MQTT is disconnected) |
## Step 0 — Source config & sanity-check inputs
```bash
set -a; . .settings/SKILL/deploy/.devploy.env; set +a
set -a; . .settings/deploy/.devploy.env; set +a
export KUBECONFIG
: "${GIT_USER:?set GIT_USER in .devploy.env}"
: "${GIT_TOKEN:?set GIT_TOKEN in .devploy.env}"
echo "release=$RELEASE host=$HOSTNAME chart=$CHART_DIR"
echo "docs image=$IMAGE_FULL api image=$API_IMAGE_FULL mcp image=${MCP_IMAGE_FULL:-git.office.ilab.zone/raid/r2d2-4817-mcp}"
echo "release=$RELEASE ns=$NAMESPACE host=$HOSTNAME image=$IMAGE_FULL"
echo "chart=$CHART_DIR context=$BUILD_CONTEXT"
```
If `GIT_USER`/`GIT_TOKEN` are empty, **stop and ask** the user to fill them in
(`.settings/` is gitignored, so they're safe there). Don't guess `HOSTNAME` or
`GIT_REPO_URL` — they're public and hard to reverse.
If `GIT_USER`/`GIT_TOKEN` are empty, **stop and ask**.
## Step 1 — Pre-flight checks
Run **all** of these. Report each pass/fail. Don't proceed past Step 2 until green.
```bash
ROOT=$LOCAL_SRC
# 1.1 — Local tooling
for t in docker helm kubectl jq pnpm node; do command -v $t >/dev/null || echo "MISSING $t"; done
# 1.2 — Cluster reachable, Longhorn default SC (informational), nginx ingress class
kubectl get ingressclass nginx -o jsonpath='{.spec.controller}{"\n"}' # expect k8s.io/ingress-nginx
# 1.3 — Target namespace: first-install vs upgrade
# • NotFound → first install (created in 7.1)
# • Active → upgrade; confirm a helm release secret for THIS release exists,
# otherwise STOP (something else owns the namespace).
kubectl get ns $RELEASE 2>&1
kubectl -n $RELEASE get secret -l owner=helm,name=$RELEASE 2>&1 | head -3
# 1.4 — Chart renders & lints with BOTH value files (defaults + r2d2 override)
for t in docker helm kubectl jq node npm; do command -v $t >/dev/null || echo "MISSING $t"; done
# 1.2 — Cluster reachable + nginx ingress class present
kubectl get ingressclass nginx -o jsonpath='{.spec.controller}{"\n"}' # expect k8s.io/ingress-nginx
# 1.3 — Namespace: NotFound → first install (created in 6.1); Active → upgrade
kubectl get ns $NAMESPACE 2>&1
kubectl -n $NAMESPACE get secret -l owner=helm,name=$RELEASE 2>&1 | head -3
# 1.4 — Chart renders & lints with BOTH value files
helm lint $CHART_DIR -f $VALUES -f $OVERLAY
helm template $RELEASE $CHART_DIR -n $RELEASE -f $VALUES -f $OVERLAY >/dev/null
# 1.4b — Ingress covers every public API route (catches missing /encode-style gaps)
cd $ROOT/interop/validation-api && pytest test_ingress_routes.py -q
helm template $RELEASE $CHART_DIR -n $NAMESPACE -f $VALUES -f $OVERLAY --set image.tag=preflight >/dev/null
# 1.5 — Gitea registry reachable (GET, not HEAD: /v2/ returns 401 + Bearer challenge)
curl -s --max-time 10 -o /dev/null -w '%{http_code}\n' https://git.office.ilab.zone/v2/ # expect 401
# 1.6 — Docker daemon up
docker buildx version >/dev/null # if it errors: `orb start` (OrbStack) and retry
```
## Step 2 — Local smoke test, docs (compose + nginx sidecar)
Build and run the **production image behind the nginx sidecar** locally before
pushing anything. This is the same image the cluster runs.
```bash
cd $ROOT/web
docker compose up -d --build # prod profile: app (:3000) + nginx (:8080)
docker compose ps # both services Up; app healthcheck healthy
/usr/bin/curl -s -o /dev/null -w 'home=%{http_code}\n' http://localhost:8080/
/usr/bin/curl -s -o /dev/null -w 'archive=%{http_code}\n' http://localhost:8080/archive
docker compose down
```
If the build fails on `pnpm install`/generate, fix it here — never push a broken
image. The generator runs as `prebuild`; a missing `STANAG/Draft-*/draft.json`
makes it throw "cannot resolve the current draft".
> The host `:8080` is sometimes already held by an unrelated OrbStack container.
> If `docker compose up` can't bind it, run the smoke test on another port:
> `INGRESS_PORT=8085 docker compose up -d --build` (compose honours `${INGRESS_PORT:-8080}`),
> and probe `http://localhost:8085/…`.
## Step 2b — Local smoke test, validation API (as UID 1000)
Build the API image and run it **as the same non-root UID the pod uses (1000)** so
a read/permission problem surfaces here, not in the cluster. It's stateless — no
broker, no Redis:
```bash
cd $ROOT
API_IMG_LOCAL=stanag-4817-validation-api:smoke
docker buildx build --platform linux/amd64 --load \
-t $API_IMG_LOCAL -f $API_DOCKERFILE $BUILD_CONTEXT
docker rm -f va-smoke >/dev/null 2>&1
docker run -d --name va-smoke --user 1000 \
-e MQTT_AUTOCONNECT=false -e REDIS_URL=memory:// -p 18817:8817 $API_IMG_LOCAL
# Wait for /health, then POST a real example to /validate (expect valid:true).
for i in $(seq 1 20); do
[ "$(/usr/bin/curl -s -o /dev/null -w '%{http_code}' http://localhost:18817/health)" = "200" ] && break
sleep 1
done
/usr/bin/curl -s -o /dev/null -w 'health=%{http_code}\n' http://localhost:18817/health
EX=$(/usr/bin/curl -s http://localhost:18817/examples/catl_2_node_status.json)
/usr/bin/curl -s -X POST http://localhost:18817/validate -H 'content-type: application/json' \
-d "{\"message\": $(echo "$EX" | jq -c '.message // .')}" | jq '{valid, schema_ref}'
docker rm -f va-smoke
```
Expect `health=200` and `valid:true`. If the container exits immediately as UID
1000, it's a file-permission issue in the image (the spec pack must be world-readable).
## Step 3 — Sensitive cleanup before any `git push`
```bash
cd $ROOT
grep -iE 'env|secret|kubeconfig|\.settings' .gitignore # .settings/ must be ignored
git ls-files | grep -iE '\.env$|secret|kubeconfig|id_rsa' # expect only web/.env (committed, non-secret compose defaults)
git config user.name; git config user.email # must be the user, not "Claude"
git log --all --grep='Claude\|Anthropic' --pretty='%h %s' | head # surface, don't rewrite
```
`web/.env` IS tracked on purpose — it holds non-secret Compose defaults. Real
secrets live in `web/.env.local` (gitignored) and `.settings/` (gitignored).
**Do not push** until clean.
## Step 4 — Git: push the project to Gitea (`RAID/4817`)
```bash
cd $ROOT
git remote -v
git remote add origin $GIT_REPO_URL # only if 'origin' isn't set
git push -u origin main
```
**Auth note:** Gitea HTTPS push uses a personal access token, not the web password.
The token needs `read/write:repository` for git and `read/write:package` for the
docker push in Step 6. A `write:package`-less token fails Step 6 with
`unauthorized: reqPackageAccess`.
## Step 5 — Build the container image
## Step 2 — Build the frontend image
```bash
cd $ROOT
SHA=$(git rev-parse --short HEAD)
# Clean tree → sha-<short> (reproducible). Dirty (modified OR untracked) → preflight-<short>.
# Clean tree → sha-<short> (reproducible). Dirty → preflight-<short>.
if [ -z "$(git status --short)" ]; then TAG=sha-$SHA; else TAG=preflight-$SHA; fi
IMG=$IMAGE_FULL:$TAG
# Build from the REPO ROOT (-f web/Dockerfile), linux/amd64 for the cluster.
# Build linux/amd64 for the cluster (nodes are amd64), context = holocron-frontend/.
docker buildx build --platform linux/amd64 --load -t $IMG -f $DOCKERFILE $BUILD_CONTEXT
```
The image is large-ish (Next standalone + bundled archive artifacts). To trim it,
the Dockerfile defaults `BUNDLE_MAX_BYTES=16MiB` so the big SD-2 XMI/QEA exports
link out instead of being baked in. If the Docker daemon is down
(`Cannot connect to the Docker daemon`), `orb start` and retry.
## Step 3 — Local smoke test (frontend serves /health)
## Step 5b — Build the validation-api image
The FastAPI validation/verification API is a **second container** in the same Helm
release. Build it with the **same `$TAG`** as the docs image:
Run the production image and confirm `/health` is 200 **and that no
frame-blocking headers are present** (iframe-embedding guard):
```bash
API_IMG=$API_IMAGE_FULL:$TAG
# Same repo-root build context as the docs image; the API reads the spec pack +
# interop/mqtt/topic_map.py from their in-repo locations.
docker buildx build --platform linux/amd64 --load \
-t $API_IMG -f $API_DOCKERFILE $BUILD_CONTEXT
docker rm -f holo-smoke >/dev/null 2>&1
docker run -d --name holo-smoke -e NODE_ENV=production -p 13001:3001 $IMG >/dev/null
for i in $(seq 1 20); do
[ "$(/usr/bin/curl -s -o /dev/null -w '%{http_code}' http://localhost:13001/health)" = "200" ] && break; sleep 1
done
/usr/bin/curl -s http://localhost:13001/health; echo
/usr/bin/curl -sI http://localhost:13001/ | grep -iE 'x-frame|content-security' \
&& echo "FAIL: frame-blocking header present (breaks iframe embed)" || echo "OK: no frame-blocking headers"
docker rm -f holo-smoke >/dev/null 2>&1
```
The ingress routes `/validate`, `/route`, `/fuse`, `/health`, etc. to this Service
while `/` stays on the docs app — Scalar **Try it** at the public host depends on
this pod being up. It is stateless (`MQTT_AUTOCONNECT=false`, `REDIS_URL=memory://`).
## Step 4 — Push to Gitea's OCI registry
## Step 5c — Build the MCP server image
The open-source MCP server (`interop/mcp-server/`) is a **sidecar container in the
docs pod** — it serves Streamable HTTP at `/mcp` so external chatbots can explain and
exercise the 4817 docs. Build it with the **same `$TAG`** as the others:
```bash
MCP_IMAGE_FULL=${MCP_IMAGE_FULL:-git.office.ilab.zone/raid/r2d2-4817-mcp}
MCP_DOCKERFILE=${MCP_DOCKERFILE:-interop/mcp-server/Dockerfile}
MCP_IMG=$MCP_IMAGE_FULL:$TAG
# Same repo-root build context; the Dockerfile bakes interop/validation-api/message-docs.json.
docker buildx build --platform linux/amd64 --load \
-t $MCP_IMG -f $MCP_DOCKERFILE $BUILD_CONTEXT
```
Its documentation tools run offline from the baked guide; its live tools proxy to the
validation-api Service on `:8817` (allowed by the docs pod's egress NetworkPolicy).
The ingress routes `/mcp` to the `stanag-4817-mcp` Service. Because the MCP server is
a sidecar in the docs pod, **enabling `mcp` does not add a Deployment** — it adds a
container to `deploy/stanag-4817` plus the `stanag-4817-mcp` Service.
## Step 6 — Push to Gitea's OCI registry
`docker login` is fragile on macOS (osxkeychain `-25299` even on success). Use a
**session-local docker config dir** with inline base64 auth:
`docker login` is fragile on macOS (osxkeychain `-25299`). Use a session-local
docker config dir with inline base64 auth:
```bash
DC=/tmp/${RELEASE}-dc-dir; mkdir -p $DC
@ -267,243 +136,135 @@ AUTH=$(printf '%s' "$GIT_USER:$GIT_TOKEN" | base64)
cat > $DC/config.json <<EOF
{ "auths": { "git.office.ilab.zone": { "auth": "$AUTH" } } }
EOF
# Verify the token has push scope BEFORE pushing (avoids orphan layers on 401):
# Verify push scope BEFORE pushing (avoids orphan layers on 401):
curl -s --max-time 6 -H "Authorization: Basic $AUTH" \
"https://git.office.ilab.zone/v2/token?service=container_registry&scope=repository:raid/$IMAGE_NAME:push,pull" \
| jq '{has_token: (.token!=null)}'
docker --config $DC push $IMG
docker --config $DC push $API_IMG
docker --config $DC push $MCP_IMG
# Confirm all three tags landed:
# Confirm the tag landed:
curl -s --max-time 8 -H "Authorization: Basic $AUTH" \
"https://git.office.ilab.zone/v2/raid/$IMAGE_NAME/tags/list"
curl -s --max-time 8 -H "Authorization: Basic $AUTH" \
"https://git.office.ilab.zone/v2/raid/$API_IMAGE_NAME/tags/list"
curl -s --max-time 8 -H "Authorization: Basic $AUTH" \
"https://git.office.ilab.zone/v2/raid/${MCP_IMAGE_FULL##*/}/tags/list"
```
`unauthorized: reqPackageAccess` ⇒ token missing `write:package`. Stop, ask the
`unauthorized: reqPackageAccess` ⇒ the PAT lacks `write:package`. Stop, ask the
user to add it, update `GIT_TOKEN` in `.devploy.env`, re-source, retry.
## Step 7 — Deploy
The r2d2 overlay (`values.override.yaml`) already exists and is correct (image
repo, host, `gitea-registry-cred`, `fullnameOverride: stanag-4817`, node UID 1000,
probe `/`, no PVC). The only field that changes per build is `image.tag`.
## Step 5 — Deploy
```bash
# 7.1 — Namespace (idempotent)
kubectl get ns $RELEASE >/dev/null 2>&1 || kubectl create namespace $RELEASE
# 5.1 — Namespace (idempotent)
kubectl get ns $NAMESPACE >/dev/null 2>&1 || kubectl create namespace $NAMESPACE
# 7.2 — Image-pull secret in THIS namespace only (idempotent: delete-if-exists then create)
kubectl -n $RELEASE delete secret gitea-registry-cred --ignore-not-found
kubectl -n $RELEASE create secret docker-registry gitea-registry-cred \
# 5.2 — Image-pull secret in THIS namespace only (idempotent)
kubectl -n $NAMESPACE delete secret gitea-registry-cred --ignore-not-found
kubectl -n $NAMESPACE create secret docker-registry gitea-registry-cred \
--docker-server=git.office.ilab.zone \
--docker-username="$GIT_USER" --docker-password="$GIT_TOKEN" \
--docker-email="$(git config user.email)"
# 7.2b — Ask-AI gateway secret (idempotent). The docs pod reads this via
# secretKeyRef (values.yaml env AI_API_KEY, key=token; optional:true so a
# missing secret won't crash the pod). The key NEVER lives in the repo — it
# comes from .devploy.env (gitignored). Skip cleanly if AI_API_KEY is empty.
if [ -n "${AI_API_KEY:-}" ]; then
kubectl -n $RELEASE delete secret ask-ai --ignore-not-found
kubectl -n $RELEASE create secret generic ask-ai \
--from-literal=token="$AI_API_KEY" \
${AI_USERNAME:+--from-literal=username="$AI_USERNAME"}
echo "ask-ai secret created (token$([ -n "${AI_USERNAME:-}" ] && echo " + username"))"
else
echo "AI_API_KEY empty in .devploy.env — skipping ask-ai secret (assistant won't auth)"
fi
# 7.3 — Point the overlay at the tag we just pushed, then install/upgrade.
# (image.tag is the only volatile field; set it explicitly each rollout.)
# validationApi.image.tag + mcp.image.tag use the same tag — built from
# interop/validation-api/Dockerfile (Step 5b) and interop/mcp-server/Dockerfile
# (Step 5c), pushed as r2d2-4817-validation-api and r2d2-4817-mcp.
helm upgrade --install $RELEASE $CHART_DIR -n $RELEASE \
# 5.3 — Install/upgrade. image.tag is the only volatile field; set it explicitly.
# Pass MAPTILER_KEY through as a chart secret only when set (gitignored env).
helm upgrade --install $RELEASE $CHART_DIR -n $NAMESPACE \
-f $VALUES -f $OVERLAY \
--set image.tag=$TAG \
--set validationApi.image.tag=$TAG \
--set mcp.image.tag=$TAG
${MAPTILER_KEY:+--set secrets.MAPTILER_KEY=$MAPTILER_KEY}
# 7.4 — Wait for rollout (first pull can take a minute). Confirm the image tags:
kubectl -n $RELEASE rollout status deploy/$RELEASE --timeout=180s
kubectl -n $RELEASE rollout status deploy/$RELEASE-api --timeout=180s
kubectl -n $RELEASE get pods,svc,ingress
kubectl -n $RELEASE get deploy/$RELEASE -o jsonpath='docs={.spec.template.spec.containers[0].image}{"\n"}'
# The MCP server is a sidecar in the docs pod — it's the 2nd container, not a Deployment.
kubectl -n $RELEASE get deploy/$RELEASE -o jsonpath='mcp={.spec.template.spec.containers[1].image}{"\n"}'
kubectl -n $RELEASE get deploy/$RELEASE-api -o jsonpath='api={.spec.template.spec.containers[0].image}{"\n"}'
# 5.4 — Wait for rollout (first pull can take a minute).
kubectl -n $NAMESPACE rollout status deploy/$RELEASE --timeout=180s
kubectl -n $NAMESPACE rollout status deploy/$RELEASE-mqtt --timeout=120s
kubectl -n $NAMESPACE get pods,svc,ingress
kubectl -n $NAMESPACE get deploy/$RELEASE -o jsonpath='image={.spec.template.spec.containers[0].image}{"\n"}'
```
## Step 8 — End-to-end verify (docs **and** API)
## Step 6 — End-to-end verify
Use **`/usr/bin/curl` explicitly** — the `rtk` token-killer hook can strip `-v`/`-I`
and make probes look like failures. NB: this harness runs under zsh, which does NOT
word-split an unquoted `$var` — use `${=VAR}` or you'll probe one giant URL and get
`000` on every line.
Use **`/usr/bin/curl` explicitly** — the `rtk` token-killer hook can strip
`-v`/`-I`. This harness runs zsh, which does NOT word-split unquoted `$var` — use
`${=VAR}`.
```bash
host $HOSTNAME # resolves via office DNS → cluster ingress IP
host $HOSTNAME # resolves via office DNS (wildcard) → edge → ingress
# 8.1 — Docs routes (expect 200).
# 6.1 — HTTP routes (expect 200).
for p in ${=VERIFY_PATHS}; do
/usr/bin/curl -sk --max-time 6 -o /dev/null -w "$p = %{http_code}\n" "https://$HOSTNAME$p"
/usr/bin/curl -sk --max-time 8 -o /dev/null -w "$p = %{http_code}\n" "https://$HOSTNAME$p"
done
# 8.2 — API GET routes via the public host (require the stanag-4817-api pod; expect 200).
for p in ${=API_VERIFY_PATHS}; do
/usr/bin/curl -sk --max-time 6 -o /dev/null -w "$p = %{http_code}\n" "https://$HOSTNAME$p"
done
# 6.2 — /api/config returns JSON with the public wss URL.
/usr/bin/curl -sk --max-time 8 "https://$HOSTNAME/api/config" | jq '{wsUrl, subscribeTopic}'
# 8.3 — API POST /validate with a REAL example body (expect valid:true, NOT a Next 404).
EX=$(/usr/bin/curl -sk --max-time 6 "https://$HOSTNAME/examples/catl_2_node_status.json")
/usr/bin/curl -sk --max-time 8 -X POST "https://$HOSTNAME/validate" \
-H 'content-type: application/json' \
-d "{\"message\": $(echo "$EX" | jq -c '.message // .')}" \
| jq '{valid, schema_ref, message_type}'
# 6.3 — IFRAME GUARD: the response must NOT carry frame-blocking headers.
/usr/bin/curl -skI --max-time 8 "https://$HOSTNAME/" | grep -iE 'x-frame|content-security' \
&& echo "FAIL: frame-blocking header present (breaks the r2d2 iframe embed)" \
|| echo "OK: embeddable (no X-Frame-Options / CSP frame block)"
# 8.4 — API POST /route resolves the MQTT topic (expect a topic/qos/retain object).
/usr/bin/curl -sk --max-time 8 -X POST "https://$HOSTNAME/route" \
-H 'content-type: application/json' \
-d "{\"message\": $(echo "$EX" | jq -c '.message // .')}" | jq '{topic, qos, retain}'
# 8.4b — MCP server (sidecar) reachable through the ingress. Only /mcp is routed to
# the sidecar (the container's /health is probed directly by the kubelet, not via
# ingress). The MCP endpoint is POST /mcp and answers `initialize` with an SSE event.
/usr/bin/curl -sk --max-time 8 -X POST "https://$HOSTNAME/mcp" \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"deploy-smoke","version":"0"}}}' \
| grep -qi '"serverInfo"' && echo "OK: /mcp initialize" || echo "FAIL: /mcp did not initialize"
# 8.5 — CRITICAL routing check: /docs must be the DOCS site, NOT FastAPI's Swagger.
# (The API's own /docs, /openapi.json, /reference are deliberately NOT routed to
# the API — the docs site owns those, and Scalar reads the static web/public/openapi.json.)
/usr/bin/curl -sk --max-time 6 "https://$HOSTNAME/docs" | grep -qi 'swagger' \
&& echo "FAIL: /docs is FastAPI swagger (ingress mis-routed)" || echo "OK: /docs is the docs site"
# 8.6 — TLS cert is the Let's Encrypt one for this host:
echo | openssl s_client -connect $HOSTNAME:443 -servername $HOSTNAME 2>/dev/null \
| openssl x509 -noout -subject -issuer
# 6.4 — WebSocket upgrade through /ws (expect HTTP 101 Switching Protocols).
/usr/bin/curl -sk --max-time 8 -o /dev/null -w 'ws=%{http_code}\n' \
-H 'Connection: Upgrade' -H 'Upgrade: websocket' \
-H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
"https://$HOSTNAME/ws"
```
Expect: docs routes `200`; API GET routes `200`; `POST /validate` ⇒ `valid:true`
with a real `schema_ref` (never a Next.js 404 page); `POST /route` ⇒ a topic object;
`/docs` ⇒ the docs site. A `404`/HTML body on an API route means the ingress path
rule is missing or the `stanag-4817-api` pod is down — check
`kubectl -n $RELEASE get pods -l app.kubernetes.io/component=validation-api` and its logs.
Expect: VERIFY_PATHS `200`; `/api/config` JSON with `wsUrl=wss://…/ws`; **no**
frame-blocking headers; `/ws` ⇒ `101`. A `404`/HTML on `/health` means the pod
is down or the ingress host is wrong — check
`kubectl -n $NAMESPACE get pods -l app.kubernetes.io/component=frontend` + logs.
## Step 9 — Iterate (rev2, rev3…) when code or values change
## Step 7 — Iterate (rev2, rev3…)
```bash
# Both changed → rebuild + push both (Step 5/5b–6) at a fresh $TAG, then:
helm upgrade $RELEASE $CHART_DIR -n $RELEASE -f $VALUES -f $OVERLAY \
--set image.tag=$TAG --set validationApi.image.tag=$TAG
kubectl -n $RELEASE rollout status deploy/$RELEASE --timeout=180s
kubectl -n $RELEASE rollout status deploy/$RELEASE-api --timeout=180s
# Code changed → rebuild + push at a fresh $TAG (Steps 2–4), then:
helm upgrade $RELEASE $CHART_DIR -n $NAMESPACE -f $VALUES -f $OVERLAY --set image.tag=$TAG
kubectl -n $NAMESPACE rollout status deploy/$RELEASE --timeout=180s
```
**Only one image changed** — the docs and API images version independently, so
rebuild/push just the one that changed and reuse the other's existing tag:
```bash
# e.g. only the validation API changed (web/ untouched): keep the docs tag, bump the API.
helm upgrade $RELEASE $CHART_DIR -n $RELEASE -f $VALUES -f $OVERLAY \
--set image.tag=$DOCS_TAG_ALREADY_DEPLOYED --set validationApi.image.tag=$NEW_API_TAG
kubectl -n $RELEASE rollout status deploy/$RELEASE-api --timeout=180s
```
Get the currently-deployed tags with
`kubectl -n $RELEASE get deploy/$RELEASE -o jsonpath='{..image}'` (and `$RELEASE-api`).
**Config-only change** (no rebuild): edit `values.override.yaml`, `helm upgrade`,
then `kubectl -n $RELEASE rollout restart deploy/$RELEASE deploy/$RELEASE-api` so
pods pick up the change.
**Rotating the Ask-AI key** (no rebuild): update `AI_API_KEY` in `.devploy.env`,
re-source, re-run Step 7.2b (delete+create `ask-ai`), then
`kubectl -n $RELEASE rollout restart deploy/$RELEASE` so the docs pod re-reads it.
**Adding a new spec draft:** drop `STANAG/Draft-N-CODE/` with a `draft.json`,
rebuild the image (the generator picks it up automatically), push, upgrade. No
code change needed — that's the point of the manifest contract.
then `kubectl -n $NAMESPACE rollout restart deploy/$RELEASE`.
## Rules / anti-rules
### DO
- **Build `--platform linux/amd64`** even on Apple Silicon — nodes are amd64.
- **Build from the repo root** (`-f web/Dockerfile $LOCAL_SRC`); the generator needs
every `STANAG/Draft-*/`. The root `.dockerignore` (keeps only `web/` + `STANAG/`)
keeps the context lean.
- **Smoke-test with compose first** (Step 2) — the nginx sidecar + image are the
same shape as prod; catch build/generate breaks locally.
- **Smoke-test the API as UID 1000** (Step 2b) — running the container with
`--user 1000` locally catches read/permission breaks before the cluster does.
- **Verify the API end-to-end, not just the docs** (Step 8.2–8.4) — `POST /validate`
returning `valid:true` through the public host is the real "Try it works" signal.
- **Smoke-test the image locally first** (Step 3) and assert no frame headers.
- **Use a session-local docker config dir**, not `docker login` (osxkeychain `-25299`).
- **Tag dirty trees `preflight-<sha>`**, never `sha-<short>`.
- **Make every step idempotent** (`get…||create…`, `delete --ignore-not-found && create…`).
- **Use GET (not HEAD) on OCI `/v2/` endpoints** — HEAD returns 405.
- **Confirm the rendered/rolled `image:` tag** before and after `helm upgrade` —
wrong tag is the #1 "I deployed but nothing changed".
- **Use GET (not HEAD) on OCI `/v2/`** — HEAD returns 405.
- **Confirm the rolled `image:` tag** before/after `helm upgrade`.
### DON'T
- **Don't touch namespaces other than `$RELEASE`.** `kubectl get pods -A` is a smell.
- **Don't `docker push` without the `/v2/token` scope check** (Step 6) — a 401
mid-push leaves orphan layers.
- **Don't name the release `4817`.** k8s Service names are DNS-1035 labels (must start
with a letter) — use `stanag-4817`. The public host keeps `4817.r2d2…`.
- **Don't add a PVC.** This app is stateless; the chart's emptyDir mounts cover the
read-only-rootfs writable paths (`/tmp`, `.next/cache`).
- **Don't route `/docs`, `/openapi.json`, or `/reference` to the API.** The docs site
owns those paths; Scalar reads the static `web/public/openapi.json`. Routing them to
the API would shadow the docs with FastAPI's own Swagger (Step 8.5 guards this).
- **Don't give the API pod the docs selector labels.** The chart uses a DISTINCT
`app.kubernetes.io/name: <name>-api` so the docs Service selector never matches API
pods (and vice-versa). Reusing the docs labels makes traffic leak between them.
- **Don't rebuild for env-only changes** — `helm upgrade` + `rollout restart`.
- **Don't add `Co-Authored-By: Claude*` / "Generated with Claude Code"** to commits.
- **Don't trust browser DNS as a deploy signal** — if `/usr/bin/curl` from the build
host is 200 but the browser is `ERR_NAME_NOT_RESOLVED`, it's the browser's resolver
(split-DNS/VPN), not the deploy.
- **Don't add `X-Frame-Options` / `frame-ancestors` CSP** anywhere — it breaks the
`r2d2.office.ilab.zone` iframe embed. Don't front the app with `holocron-proxy`
(its nginx sets `X-Frame-Options SAMEORIGIN`).
- **Don't touch namespaces other than `holocron`.**
- **Don't repurpose `holocron-sim/helm/sim`** — that's a different app.
- **Don't add a PVC** — the app is stateless.
- **Don't `docker push` without the `/v2/token` scope check** (Step 4).
- **Don't add `Co-Authored-By: Claude*`** to commits.
## Common failures & quick fixes
| Symptom | Cause | Fix |
| --- | --- | --- |
| `Cannot connect to the Docker daemon` | OrbStack stopped | `orb start`, wait for `docker info` |
| `error storing credentials … keychain (-25299)` on `docker login` | osxkeychain helper | Use the session-local config dir (Step 6) |
| `unauthorized: reqPackageAccess` on push | Token lacks `write:package` | Reissue/edit the PAT, add `read/write:package`, update `.devploy.env` |
| `Service "4817" is invalid: metadata.name … DNS-1035` | Release name starts with a digit | Use `RELEASE=stanag-4817` (already set) |
| Pod `CrashLoopBackOff`, logs `EACCES`/read-only FS on `.next/cache` or `/tmp` | Missing emptyDir mount | The chart mounts both as emptyDir; confirm you applied `values.yaml` (not only the override) |
| `generate: cannot resolve the current draft` at build | No `STANAG/Draft-*/draft.json` with `status: current` | Ensure the current draft's `draft.json` has `"status": "current"` |
| `/archive/sd2` shows XMI/QEA as external links that 404 | `BUNDLE_MAX_BYTES` excluded them but the repo isn't pushed yet | Push to Gitea (Step 4) so `ARTIFACT_BASE_URL` resolves, or build with `BUNDLE_MAX_BYTES` unset to bundle them |
| `Get "https://git.office.ilab.zone/v2/": EOF` mid-push | Office edge (Caddy) hiccup | Wait + retry |
| API route (`/validate` etc.) returns a Next.js 404 / HTML page | ingress path rule missing, or `stanag-4817-api` pod down | Check `kubectl -n $RELEASE get pods -l app.kubernetes.io/component=validation-api`; confirm the API paths are in `values.override.yaml` `ingress.hosts[].paths` with `service: stanag-4817-api` |
| `/docs` shows FastAPI Swagger instead of the docs site | `/docs` (or a broad prefix) routed to the API | Remove that path from the API routes — the docs site owns `/docs`, `/openapi.json`, `/reference` |
| API pod `CrashLoopBackOff` / exits as UID 1000 | spec-pack files not world-readable, or a missing COPY in the API Dockerfile | Reproduce with Step 2b (`docker run --user 1000`); fix file perms / the Dockerfile COPYs |
| `error storing credentials … keychain (-25299)` | osxkeychain helper | Use the session-local config dir (Step 4) |
| `unauthorized: reqPackageAccess` on push | PAT lacks `write:package` | Reissue the PAT with `read/write:package`, update `.devploy.env` |
| Pod `ImagePullBackOff` | `gitea-registry-cred` missing/wrong in ns | Re-run Step 5.2 |
| iframe shows blank / "refused to connect" | a frame-blocking header crept in | Step 6.3 finds it; remove the X-Frame-Options/CSP source |
| `/ws` not `101` | nginx didn't upgrade, or ws pod down | confirm ingress `/ws`→port 9002 + the proxy-read-timeout annotation |
| `/health` returns Next/HTML 404 | wrong host or pod down | check the ingress host + `kubectl -n holocron logs` |
## What this skill produces (final artifacts)
- **Gitea**: `https://git.office.ilab.zone/RAID/4817` with `main` pushed.
- **Gitea OCI registry**: `git.office.ilab.zone/raid/r2d2-4817:<tag>` (docs) **and**
`git.office.ilab.zone/raid/r2d2-4817-validation-api:<tag>` (API).
- **Cluster**: namespace `stanag-4817` containing: docs Deployment+Service, the
validation-api Deployment+Service `stanag-4817-api` (routed at `/validate`, `/route`,
… on the same host), one Ingress fronting both, `gitea-registry-cred`, the `ask-ai`
Secret (Open WebUI key the docs pod reads for the "Ask AI" assistant), Helm release
secret. No PVC.
- **Public URL**: `https://4817.r2d2.office.ilab.zone/` serves the docs; `/archive`
serves the version history; `/api-reference` + the API path prefixes are the live
Validation API (Scalar **Try it** works).
- **Gitea OCI registry**: `git.office.ilab.zone/raid/r2d2-holocron:<tag>`.
- **Cluster**: namespace `holocron` with the holocron Deployment+Service, the
`holocron-mqtt` mosquitto Deployment+Service, one nginx Ingress, the
`gitea-registry-cred` pull secret, the Helm release secret. No PVC.
- **Public URL**: `https://holocron.r2d2.office.ilab.zone/` serves the tactical
display (embeddable as an iframe under `r2d2.office.ilab.zone`); `/ws` is the
live WebSocket bridge.
## Memory hook
When this completes, record (via `bd remember`) the last-deployed image tag and any
project-specific gotcha hit during *this* rollout (not the generic ones above).
When this completes, record (via `bd remember`) the last-deployed image tag and
any project-specific gotcha hit during *this* rollout.