# cluster-iac — AI Agent Context

GitOps infrastructure repo for Kubernetes deployments via Flux CD. This file is the entry point for AI agents working with any app that deploys to this cluster.

> Published at **https://cluster.nbrx.com/AGENTS.md** and referenced from every chart schema
> (`https://cluster.nbrx.com/schemas/charts/webapp.json`). If you are an agent working in an
> *app* repo (not this one), fetch that URL to get the full cluster context.

## How It Works

App repos contain a `.cluster/` directory with deployment config. On push, a GitHub Actions workflow runs the `deploy` CLI (from `packages/deploy` in this repo) which synthesizes the chart into raw Kubernetes manifests (via cdk8s), pushes them to this repo, and waits for Flux to reconcile.

```
App Repo (.cluster/app.yaml) → deploy CLI → cluster-iac (raw manifests) → Flux CD → Kubernetes
```

## 1. App Configuration (`.cluster/app.yaml`)

### Schema & IDE Validation

Every `app.yaml` must start with this comment for IDE autocompletion and validation:

```yaml
# yaml-language-server: $schema=https://cluster.nbrx.com/schemas/charts/webapp.json
```

The schema is published to GitHub Pages at `cluster.nbrx.com` by `.github/workflows/schema-pages.yaml`. Its root `description` links to https://cluster.nbrx.com/AGENTS.md — this document — so an agent that only sees an app repo can fetch the full cluster context from the schema reference alone.

### Top-Level Fields

```yaml
# .cluster/app.yaml
chart: webapp                        # chart (default: webapp, only option currently)
requiredSecretKeys:                  # GitHub Environment Secrets to seal and inject
  - DATABASE_URL
  - API_KEY
expandDatabaseUrl: DATABASE_URL      # Parse DATABASE_URL into DB_HOST, DB_PORT, DB_DATABASE, DB_USER, DB_PASSWORD
expandDatabaseUrlPrefix: ""          # Prefix for expanded DB_* keys (default: none)
secretPrefix: ""                     # Prefix added to all secret key names in the SealedSecret
secretMapping:                       # Rename specific secret keys (takes precedence over secretPrefix)
  API_KEY: EXTERNAL_API_KEY
values:                              # 1:1 chart values (see below)
  ...
```

Only `values` is required. All other fields are optional.

### Environment Overlays

Per-environment overrides in `app.<env>.yaml` are deep-merged over `app.yaml`:

```
.cluster/
├── app.yaml            # Base (all environments)
├── app.dev.yaml        # Dev overrides
├── app.staging.yaml    # Staging overrides
└── app.prod.yaml       # Prod overrides
```

Deep-merge: objects are recursively merged, arrays are replaced entirely. Only the `values` key from overlay files is used.

### Values Reference (most common)

All keys under `values` map 1:1 to `charts/webapp/values.yaml`. The full schema is at `charts/webapp/values.schema.json`. Here are the commonly used keys:

| Key | Default | Description |
|-----|---------|-------------|
| `replicaCount` | `1` | Pod replicas (or use `autoscaling`) |
| `containerPort` | `3000` | Port the app listens on |
| `image.repository` | `""` | GHCR image (auto-set by deploy CLI if empty) |
| `image.tag` | `latest` | Image tag (set by deploy CLI via `DEPLOY_TAG`) |
| `service.port` | `3000` | ClusterIP service port |
| `ingress.hosts[].host` | — | Ingress hostname (typically per-env) |
| `ingress.annotations` | `{}` | Nginx ingress annotations |
| `ingress.clusterIssuer` | `letsencrypt-dns01` | cert-manager issuer for TLS |
| `resources.requests.cpu` | `100m` | CPU request |
| `resources.requests.memory` | `128Mi` | Memory request |
| `resources.limits.cpu` | `300m` | CPU limit |
| `resources.limits.memory` | `256Mi` | Memory limit |
| `probes.startup.path` | `/healthz` | Startup probe path |
| `probes.readiness.path` | `/readyz` | Readiness probe path |
| `probes.liveness.path` | `/livez` | Liveness probe path |
| `config` | `{LOG_FORMAT: json}` | Env vars via ConfigMap (key=name, value=value) |
| `extraEnv` | `[]` | Extra env vars in k8s EnvVar format (for `valueFrom`) |
| `database.enabled` | `false` | In-cluster CNPG PostgreSQL |
| `database.instances` | `1` | DB replicas (2+ = HA with streaming replication) |
| `database.size` | `5Gi` | PVC size per instance |
| `database.expose` | `false` | Expose DB externally via Envoy Gateway TLSRoute on port 5432 |
| `database.hostname` | `""` | External hostname for DB access. If empty and expose=true, defaults to `db.<ingress.hosts[0].host>` |
| `database.postgresql.parameters` | `{}` | PostgreSQL GUCs (e.g. `log_min_duration_statement: "1000"`) |
| `database.postgresql.sharedPreloadLibraries` | `[]` | e.g. `[pg_stat_statements]` (rolling restart) |
| `cronTasks` | `[]` | Scheduled HTTP tasks via hurl CronJobs (ConfigMap + CronJob per task; hits the in-cluster Service) |
| `autoscaling.enabled` | `false` | HPA (mutually exclusive with `replicaCount`) |
| `podDefaults.nodeSelector` | `{cfke.io/region: europe, cfke.io/subregion: central}` | Node placement |
| `podDefaults.spreadAcrossNodes` | `false` | Anti-affinity across nodes |
| `monitoring.probe.enabled` | `false` | Blackbox HTTP availability monitoring |
| `secretRefs` | `[]` | **Auto-managed by deploy CLI** — do not set manually |

### Extra Resources

Place standalone Kubernetes manifests in `.cluster/resources/*.yaml`. They are committed alongside the rendered chart manifests with variable substitution:

| Placeholder | Value | Example |
|------------|-------|---------|
| `%{APP_NAME}` | `fullnameOverride` or repo name | `my-app` |
| `%{NAMESPACE}` | Target namespace | `my-app-prod` |
| `%{RELEASE_NAME}` | Release name (`app.kubernetes.io/instance`) | `app` |

Use `%{}` syntax (not `${}` or `{{}}`).

### Database

Two patterns for database access:

#### In-Cluster CNPG Database (`database.enabled: true`)

Setting `database.enabled: true` creates a CNPG PostgreSQL cluster in the namespace. The chart automatically injects these env vars into the app container (no `extraEnv` or `requiredSecretKeys` needed):

| Env Var | Source | Example |
|---------|--------|---------|
| `DATABASE_URL` | CNPG Secret `uri` key | `postgresql://app:secret@my-app-db-rw:5432/app` |
| `DB_HOST` | CNPG Secret `host` key | `my-app-db-rw` |
| `DB_PORT` | CNPG Secret `port` key | `5432` |
| `DB_NAME` | CNPG Secret `dbname` key | `app` |
| `DB_USER` | CNPG Secret `username` key | `app` |
| `DB_PASSWORD` | CNPG Secret `password` key | (generated) |

The CNPG operator creates a Secret named `<fullnameOverride>-db-<owner>` (default owner: `app`).

**External access (`database.expose`):**

| Setting | Default | Effect |
|---------|---------|--------|
| `database.expose` | `false` | DB only reachable inside the cluster |
| `database.expose: true` | — | Exposes DB externally on port 5432 + generates TLS certificates |
| `database.hostname` | `""` | If empty and expose=true, defaults to `db.<ingress.hosts[0].host>` |
| `database.hostname: "db.example.com"` | — | Explicit hostname for external DB access |

When `expose: true`, the chart generates:
- A TLSRoute (Gateway API) for SNI-based routing configuration
- A self-signed CA + server TLS certificate (via cert-manager) for encrypted connections
- Traffic path: nginx TCP:5432 → pg-sni-proxy (PostgreSQL STARTTLS handling + SNI routing) → CNPG

**Connection string (external):** `postgresql://<user>:<pass>@<hostname>:5432/<dbname>?sslmode=require`

Works with **any PostgreSQL client** (psql, TablePlus, DBeaver, node-postgres, libpq, etc.). No special parameters required. The pg-sni-proxy handles the PostgreSQL STARTTLS protocol and routes based on SNI extracted from the TLS ClientHello.

**Server tuning (`database.postgresql`):**

Pass-through to the CNPG `Cluster.spec.postgresql` block, so PostgreSQL itself can be tuned from the app repo without touching cluster-iac:

```yaml
values:
  database:
    enabled: true
    postgresql:
      parameters:
        log_min_duration_statement: "1000"   # log every statement slower than 1s
        log_lock_waits: "on"
      sharedPreloadLibraries:
        - pg_stat_statements
```

| Setting | Default | Description |
|---------|---------|-------------|
| `database.postgresql.parameters` | `{}` | PostgreSQL GUCs merged into `spec.postgresql.parameters`. **Values must be strings** (quote numbers). Applied via config reload. |
| `database.postgresql.sharedPreloadLibraries` | `[]` | Rendered as `shared_preload_libraries`. Triggers a **rolling restart**. |

Notes:
- `pg_stat_statements` only needs to be listed in `sharedPreloadLibraries` — CNPG's *managed extensions* feature runs `CREATE EXTENSION` in the databases automatically, also on existing clusters. No manual SQL required.
- CNPG owns a set of parameters (WAL, replication, `log_destination`, `logging_collector`, …) and silently enforces its own values; they cannot be overridden here.
- The block is only rendered when non-empty, so existing manifests stay unchanged for apps that don't use it.

**Backups (`database.backup`):**

Enables continuous WAL archiving + scheduled base backups to S3-compatible object storage. This provides **Point-in-Time Recovery (PITR)**.

Implemented via the **Barman Cloud Plugin** (CNPG-I), not the deprecated in-tree `barmanObjectStore` (removed in CNPG 1.30.0). When `database.backup.enabled: true`, the chart renders three resources in the app namespace: a `barmancloud.cnpg.io/v1` `ObjectStore` (`<app>-db-backup`) holding the storage config + retention policy, a `plugins` entry on the `Cluster` (`barman-cloud.cloudnative-pg.io`, `isWALArchiver: true`) for continuous WAL archiving, and a `ScheduledBackup` (`method: plugin`) for scheduled base backups. The plugin itself is installed cluster-wide via the `plugin-barman-cloud` HelmRelease in `infrastructure/controllers/barman-cloud-plugin/` (namespace `cnpg-system`).

Enabling backups on an **existing** database is non-destructive: adding the `plugins` entry triggers only a rolling restart of the CNPG instances — no data loss, no PVC/cluster recreation.


| Setting | Default | Description |
|---------|---------|-------------|
| `database.backup.enabled` | `false` | Enable CNPG backups |
| `database.backup.schedule` | `0 0 2 * * *` | Base-backup cron (6-field: sec min hour dom mon dow) |
| `database.backup.destinationPath` | `""` | S3 path, e.g. `s3://my-bucket/my-app-prod` (required when enabled) |
| `database.backup.endpointURL` | `""` | S3-compatible endpoint (Hetzner: `https://fsn1.your-objectstorage.com`); empty = AWS S3 |
| `database.backup.retentionPolicy` | `14d` | Retention for base backups + WALs (e.g. `14d`, `4w`) |
| `database.backup.immediate` | `true` | Take a base backup immediately when the ScheduledBackup is created |
| `database.backup.compression` | `gzip` | Compression for data + WAL (`gzip`/`bzip2`/`snappy`) |
| `database.backup.s3Credentials.secretName` | `app-secrets` | Secret with the S3 credentials |
| `database.backup.s3Credentials.accessKeyIdKey` | `BACKUP_S3_ACCESS_KEY_ID` | Key in the Secret holding the access key id |
| `database.backup.s3Credentials.secretAccessKeyKey` | `BACKUP_S3_SECRET_ACCESS_KEY` | Key in the Secret holding the secret access key |

Credentials default to the deploy-managed `app-secrets` Secret. Add `BACKUP_S3_ACCESS_KEY_ID` and `BACKUP_S3_SECRET_ACCESS_KEY` to the relevant GitHub Environment Secrets so the deploy CLI seals them into `app-secrets` (works automatically with `includeSecretKeys: '*'`, otherwise list them in `requiredSecretKeys`). Backup status is visible in the CNPG Grafana dashboard.

Example (`app.prod.yaml`):

```yaml
values:
  database:
    backup:
      enabled: true
      schedule: "0 0 */6 * * *"   # every 6 hours
      destinationPath: "s3://my-backups/my-app-prod"
      endpointURL: "https://fsn1.your-objectstorage.com"
      retentionPolicy: "14d"
```

#### External Database (`expandDatabaseUrl`)

For managed databases (Neon, Supabase, RDS) where the connection URL is a GitHub Environment Secret:

```yaml
requiredSecretKeys:
  - DATABASE_URL

expandDatabaseUrl: DATABASE_URL       # Parses URL into individual components
expandDatabaseUrlPrefix: ""           # Optional prefix for expanded keys
```

`expandDatabaseUrl` parses the URL and adds these keys to the SealedSecret:

| Key | Parsed from | Example |
|-----|-------------|---------|
| `DB_HOST` | hostname:port | `ep-cool-rain.us-east-2.aws.neon.tech:5432` |
| `DB_PORT` | port (default: 5432) | `5432` |
| `DB_DATABASE` | path | `mydb` |
| `DB_USER` | username | `app` |
| `DB_PASSWORD` | password (URL-decoded) | `s3cr3t` |

With `expandDatabaseUrlPrefix: "PG_"`, keys become `PG_DB_HOST`, `PG_DB_PORT`, etc. The original URL key is unaffected by the prefix.

## 2. CI/CD Workflow

### Standard Structure

Workflow is always `.github/workflows/ci.yaml` with `name: ci`.

```yaml
name: ci
on:
  push:
    branches: [main]       # + optional: tags: ["v*"]

env:
  IMAGE: ghcr.io/${{ github.repository }}

jobs:
  validate:                 # Validates .cluster/app.yaml, detects if build is needed
    runs-on: cluster-runners
    outputs:
      needs_build: ${{ steps.changes.outputs.needs_build }}
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 0
      - name: Detect build-relevant changes
        id: changes
        run: |
          # Compare changed files: if only .cluster/ → skip build
          # Initial pushes and tags → always build
      - run: deploy validate

  build:                    # Builds and pushes Docker image (skipped if only .cluster/ changed)
    needs: validate
    if: needs.validate.outputs.needs_build == 'true'
    runs-on: ubuntu-latest
    outputs:
      tag: ${{ steps.tag.outputs.value }}

  deploy-<env>:             # Deploys via the deploy CLI on the cluster-runners scale set
    needs: [validate, build]
    if: always() && needs.validate.result == 'success' && (needs.build.result == 'success' || needs.build.result == 'skipped')
    runs-on: cluster-runners
    environment:
      name: <env>
      url: ${{ steps.deploy.outputs.url }}
    env:
      DEPLOY_TAG: ${{ needs.build.outputs.tag }}   # Empty when build skipped → reuses existing tag
      DEPLOY_ENV: <env>
      DEPLOY_SECRETS: ${{ toJSON(secrets) }}
    steps:
      - uses: actions/checkout@v6
      - run: deploy
```

### Build-Skip Optimization

When a push only changes files in `.cluster/`, the `build` job is skipped entirely. The `deploy` CLI detects the missing `DEPLOY_TAG` and reuses the existing image tag from the last deployment in cluster-iac. This saves ~40s on config-only changes.

### Deployment Models

| Model | Trigger | Environments | Example |
|-------|---------|-------------|---------|
| **Dev-Only** | Push to `main` | dev | `examples/workflow-dev-only.yaml` |
| **Release Train** | `main` → dev, `v*` tag → staging → prod | dev, staging, prod | `examples/workflow-release-train.yaml` |
| **Semver** | `v*` tag only | staging → prod | `examples/workflow-semver.yaml` |
| **Review Apps** | Pull Request | `pr-<n>` (temporary) | `examples/workflow-review-apps.yaml` |

### Environment Variables

| Variable | Source | Description |
|----------|--------|-------------|
| `DEPLOY_TAG` | `build` job output | Image tag (empty = reuse existing) |
| `DEPLOY_ENV` | Hardcoded per job | Target environment (`dev`, `staging`, `prod`) |
| `DEPLOY_SECRETS` | `toJSON(secrets)` | All GitHub Environment Secrets |
| `CLUSTER_APP` | Workflow | Override which `app.<env>.yaml` overlay to load (e.g., `review` → `app.review.yaml`) |
| `DEPLOY_REVIEW` | Workflow | Force review mode (`true`/`false`) |
| `DEPLOY_REVIEW_DOMAIN` | Runner config | Base domain for review app ingress (default: `review.nbrx.com`) |
| `DEPLOY_REVIEW_NAMESPACE` | Workflow | Override the shared review namespace name |

### Conventions

- Workflow file: always `.github/workflows/ci.yaml`
- Workflow name: always `name: ci`
- File extension: `.yaml` (not `.yml`)
- Deploy runner: always `runs-on: cluster-runners`
- Build runner: `ubuntu-latest`
- Actions versions: `checkout@v6`, `docker/login-action@v4`, `docker/setup-buildx-action@v4`, `docker/build-push-action@v7`
- Docker cache: `cache-from: type=gha`, `cache-to: type=gha,mode=max`

### Review Apps

Review apps create ephemeral environments per pull request. They share a namespace and (optionally) a CNPG database cluster.

**How it works:**

1. Set `DEPLOY_ENV: pr-<number>` (or any `pr-*`/`review-*` pattern, or `DEPLOY_REVIEW=true`)
2. The deploy CLI detects review mode and:
   - Uses a shared namespace: `<app>-review`
   - Generates a unique resource-name prefix from the slugified env name
   - Auto-generates ingress: `<slug>.<app>.<DEPLOY_REVIEW_DOMAIN>`
   - If `database.enabled: true`: creates a shared CNPG Cluster (once) and a per-PR Database CRD
3. On PR close, `deploy cleanup` removes the PR's manifests and waits for Flux to prune resources

**Review app with database:**

```yaml
# .cluster/app.yaml
values:
  database:
    enabled: true
    # These settings are used for the shared review CNPG Cluster

# Optional: .cluster/app.review.yaml — overlay for review apps
# Typically: smaller resources, review-specific config
```

The deploy CLI:
- Disables `database.enabled` on the per-PR app (no per-PR CNPG Cluster)
- Creates a shared CNPG Cluster in `<app>-review` namespace (written to `review-base/`)
- Creates a CNPG `Database` CRD per PR (`databaseReclaimPolicy: delete` → auto-dropped on cleanup)
- Injects `DATABASE_URL`, `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME` via `extraEnv`

**Cleanup:**

```yaml
# In workflow:
- run: deploy cleanup
  env:
    DEPLOY_ENV: pr-${{ github.event.pull_request.number }}
```

The cleanup:
- Removes per-PR directory and Flux Kustomization from cluster-iac
- Waits for Flux to prune all K8s resources (including the Database CRD → CNPG drops the database)
- If it was the last review app, also removes the shared base (CNPG Cluster + namespace)

## 3. Deploy Process

### What `deploy` Does

1. Parses `.cluster/app.yaml` + `app.<env>.yaml` → deep-merged values
2. Validates values against `charts/webapp/values.schema.json` (via ajv)
3. Clones `nbrx-ag/cluster-iac`
4. Synthesizes the chart into raw manifests (`deployment.yaml`, `service.yaml`, …) + Kustomization YAML
5. Seals secrets with kubeseal (skips if hash unchanged)
6. In review mode with `database.enabled`: writes shared CNPG base + per-PR Database CRD
7. Commits and pushes to cluster-iac
8. Annotates Flux GitRepository + Kustomization for immediate reconciliation
9. Watches Kustomization for errors (fail-fast)
10. Waits for the Flux Kustomization to become Ready
11. Validates deployed revision matches expected revision
12. Checks runtime workload health (Deployments, Pods, Jobs, CNPG)
13. Writes Job Summary + sets `url` output for GitHub Deployments page

### What `deploy cleanup` Does

1. Parses `.cluster/app.yaml` to derive app name and identifiers
2. Clones `nbrx-ag/cluster-iac`
3. Removes environment directory + Flux Kustomization file
4. Removes entry from `apps/kustomization.yaml`
5. If last review app: also removes shared review base (CNPG Cluster + namespace)
6. Commits and pushes to cluster-iac
7. Watches Flux Kustomization until deleted (waits for prune)

Usage: `deploy cleanup` or `deploy cleanup --no-wait`

### Secrets Lifecycle

GitHub Environment Secrets → `DEPLOY_SECRETS` env var → deploy CLI extracts keys listed in `requiredSecretKeys` → kubeseal encrypts → SealedSecret committed to cluster-iac → Sealed Secrets controller decrypts in-cluster → Secret available to pods via `secretRefs`.

Secrets are only re-sealed when their content hash changes.

### Revision Pinning

After pushing to cluster-iac, `deploy` records the git commit SHA. After the Kustomization is Ready, it verifies the Kustomization's `lastAppliedRevision` matches this SHA. This prevents overlapping deploys from masking failures — each deploy validates its own revision.

### No-Change Fast Path

When the rendered manifests are identical to what's already in cluster-iac (e.g., re-running a deploy), the CLI skips commit/push and instead verifies the existing Kustomization and workloads are healthy. Fast path completes in ~2s.

### Failure Diagnostics

On failure, the CLI outputs:
- Flux Kustomization status with messages
- Deployment rollout status (ready/total replicas)
- Unhealthy pod details with recent events
- Pod logs (last 100 lines) in collapsible groups
- Failed Job diagnostics from events

Common failure causes visible in diagnostics: `ImagePullBackOff`, `CrashLoopBackOff`, startup/readiness probe failures, scheduling issues, PVC mount errors.

### Runner Infrastructure

The `deploy` CLI runs on self-hosted ARC (Actions Runner Controller) runners in the cluster. The runner image (`ghcr.io/nbrx-ag/cluster-iac/runner`, built by this repo's `.github/workflows/runner-image.yaml` from `packages/deploy/Dockerfile`) includes Bun, kubeseal, the deploy CLI, the PostgreSQL 17 client tools (`psql`, `pg_dump`, `pg_restore`) and `rclone`. After a runner image update, idle runner pods must be recycled for changes to take effect.

Runner label: `runs-on: cluster-runners`. Because the runners live *inside* the cluster, they can reach in-cluster Services directly (e.g. `<app>-db-rw.<namespace>.svc:5432`) — useful for migration, dump/restore and backup jobs in CI.

**Eviction protection.** Runner pods annotate themselves with `karpenter.sh/do-not-disrupt` for the duration of a job, via the ARC hooks `ACTIONS_RUNNER_HOOK_JOB_STARTED` / `_COMPLETED` (`/opt/cluster-sync/hooks/job-*.sh` → `deploy protect on|off`). Without this, node consolidation can evict a runner mid-job; the job then fails with **no steps and no log**, and every dependent job is skipped. Idle runners stay disruptible on purpose, so `minRunners: 1` does not permanently pin a node.

## 4. Cluster Platform (Cloudfleet CFKE)

The cluster is a **Cloudfleet Kubernetes Engine (CFKE)** cluster (`cluster-1`, region `europe-central-1a`, tier `pro`). Compute nodes are Hetzner Cloud VMs in `fsn1`/`nbg1`, provisioned **just-in-time** by the CFKE node auto-provisioner (Karpenter-style) — there are no static node pools. Networking is Cilium over WireGuard.

### Node reality — READ THIS BEFORE DEBUGGING SCHEDULING OR OOM

**The nodes run permanently close to their capacity limit.** Typical steady state:

| Metric | Typical range across nodes |
|--------|---------------------------|
| CPU **requests** vs. allocatable | 40–80 % |
| Memory **requests** vs. allocatable | **67–97 %** |
| CPU **limits** vs. allocatable | 240–475 % (heavily oversubscribed) |
| Memory **limits** vs. allocatable | 190–300 % (heavily oversubscribed) |

Consequences you must assume when diagnosing a problem:

- **Memory requests are the binding constraint.** A pod stuck in `Pending` is almost always a memory-request fit problem, not a bug. Either lower `resources.requests.memory` or wait for the auto-provisioner to add a node (it reacts to `Pending` pods, so this resolves itself within minutes).
- **Limits are massively oversubscribed**, so a container can be OOM-killed or CPU-throttled even though its own limit looks generous — the node is contended. `CrashLoopBackOff` with exit code 137 = OOMKilled: raise `resources.limits.memory` *and* the request, don't just retry.
- **Never raise resource requests "just to be safe".** Every unnecessary MiB of request either blocks scheduling or provisions another paid Hetzner node. Right-size against actual `kubectl top pod` usage.
- **Always set requests on every container** (including CNPG, init and sidecar containers). Without them the auto-provisioner cannot size nodes and the cluster mis-scales.
- Nodes come and go. A pod disappearing or restarting on a different node is normal, not an incident.

### Consolidation — pods get evicted, by design

The fleet runs the **`aggressive`** auto-provisioning profile: CFKE evicts running pods off under-utilized nodes to repack them onto a cheaper layout. Expect periodic eviction waves that hit many unrelated workloads at once.

- The NodePool/NodeClass are **managed by Cloudfleet and not writable** — `kubectl patch nodepool fleet-1` is rejected by admission. Disruption budgets, `consolidateAfter` and `consolidationPolicy` cannot be tuned from the cluster.
- The only Fleet-level lever is `scalingProfile` (`aggressive` ⇄ `conservative`), changed in the Cloudfleet console. The CLI's `fleets update` is a full replace and would clobber the Hetzner credentials/constraints — don't use it for this.
- What we *can* control per workload: `karpenter.sh/do-not-disrupt` on pods, `PodDisruptionBudget`, and ≥ 2 replicas with `topologySpreadConstraints`. Never annotate a long-lived pod permanently — it pins its node and disables consolidation cluster-wide.
- Consolidation never blocks **scale-up**; `Pending` pods still trigger new nodes.

Details and the diagnostic commands: `.github/instructions/cfke-node-provisioner.instructions.md`.

### Node placement

The webapp chart defaults to `podDefaults.nodeSelector: {cfke.io/region: europe, cfke.io/subregion: central}`. Do not pin to a specific `node.kubernetes.io/instance-type` — that removes the provisioner's freedom to pick a cost-optimal node.

| Label | Example values |
|-------|----------------|
| `cfke.io/provider` | `hetzner` |
| `cfke.io/region` / `cfke.io/subregion` | `europe` / `central` |
| `cfke.io/instance-family` | `cx`, `cpx`, `ccx`, `cax` (ARM) |
| `node.kubernetes.io/instance-type` | `cx23`, `cx33`, … |
| `topology.kubernetes.io/region` | `fsn1`, `nbg1` |
| `karpenter.sh/capacity-type` | `on-demand`, `spot` |
| `kubernetes.io/arch` | `amd64`, `arm64` |

For LoadBalancer Services always set `externalTrafficPolicy: Local` — the default `Cluster` provisions a load balancer in *every* region where nodes exist. Details: `.github/instructions/cfke-node-provisioner.instructions.md`.

### Cloudfleet MCP server — use it for inspection

A **read-only** Cloudfleet MCP server is available (`clusters_list`, `clusters_get`, `clusters_query`, `who_am_i`, plus support tickets). `clusters_query` is a passthrough to the Kubernetes API restricted to HTTP GET.

**Prefer the Cloudfleet MCP over `kubectl` for read-only inspection** — it works without a local kubeconfig and cannot mutate the cluster. Use `kubectl` only for things the MCP cannot do: `logs`, `exec`, `describe`-style event correlation, and any write.

```
clusters_list                                  → cluster id (25d616fa-…), status, ready
clusters_query /api/v1/nodes                   → node inventory, capacity/allocatable, labels
clusters_query /api/v1/namespaces/<ns>/pods    → pod status, restart counts, container statuses
clusters_query /api/v1/namespaces/<ns>/events?fieldSelector=involvedObject.name=<pod>
clusters_query /apis/apps/v1/namespaces/<ns>/deployments
clusters_query /apis/postgresql.cnpg.io/v1/namespaces/<ns>/clusters
clusters_query /apis/kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations
```

Always narrow with `labelSelector` / `fieldSelector` / `limit`. Never query Secrets through it.

## 5. Databases (CNPG) — operational notes

In-cluster PostgreSQL is managed by the **CloudNativePG** operator (`database.enabled: true`, see section 1 for the configuration surface). Server image: `ghcr.io/cloudnative-pg/postgresql:17.x` — **PostgreSQL 17**. Match any client tooling to major 17 (`pg_dump` refuses to dump a newer server).

| Fact | Value |
|------|-------|
| Cluster CR | `postgresql.cnpg.io/v1 Cluster`, named `<app>-db` in the app namespace |
| Services | `<app>-db-rw` (primary), `<app>-db-ro`, `<app>-db-r` — port 5432 |
| Credentials Secret | `<app>-db-app` (keys `uri`, `host`, `port`, `dbname`, `username`, `password`) |
| Injected env | `DATABASE_URL`, `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD` (automatic) |
| Backups | Barman Cloud **plugin** (CNPG-I), *not* the removed in-tree `barmanObjectStore` |
| Operator namespace | `cnpg-system` |

Operational rules:

- **`instances: 1` is not highly available.** A single-instance cluster has downtime during node replacement — expected on a just-in-time node cluster. Use `instances: 2` for anything that must survive a node going away.
- CNPG owns the pods. Never `kubectl delete pod` a database pod to "fix" it; use the CNPG CRs (`Cluster`, `Backup`, `ScheduledBackup`) and let the operator reconcile. Switchover/restart is `kubectl cnpg` plugin territory, not manual pod surgery.
- Adding `database.backup.enabled: true` to an existing cluster is non-destructive — it triggers only a rolling restart.
- The CNPG Grafana dashboard shows replication lag, WAL archiving and backup status.
- Set database resource requests deliberately: the CNPG pod's memory request competes with app pods on already-saturated nodes.
- For external access set `database.expose: true`; the path is nginx TCP:5432 → `pg-sni-proxy` (STARTTLS + SNI routing) → CNPG. Connect with `?sslmode=require`.

Dump/restore from a `cluster-runners` job (client tools are in the runner image):

```bash
pg_dump  --format=custom --no-owner --no-acl "$DATABASE_URL" -f dump.pgc
pg_restore --clean --if-exists --no-owner --no-acl -d "$DATABASE_URL" dump.pgc
rclone copy dump.pgc "s3remote:my-bucket/$(date +%F)/"   # S3-compatible object storage
```

## 6. Debugging Playbook

Work top-down; stop as soon as the cause is identified.

1. **Is it a deploy problem or a runtime problem?** Check the Flux Kustomization for the app first:
   `clusters_query /apis/kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations` →
   `status.conditions` + `lastAppliedRevision`. A stale revision means the manifests never landed.
2. **Pod state** via `clusters_query /api/v1/namespaces/<ns>/pods`. Map the symptom:

   | Symptom | Most likely cause on this cluster |
   |---------|-----------------------------------|
   | `Pending` | Memory request does not fit — nodes are near capacity. Wait for auto-provisioning or lower the request. |
   | `CrashLoopBackOff`, exit 137 | OOMKilled under node memory contention → raise request *and* limit. |
   | `CrashLoopBackOff`, app error | Missing env/secret — check `requiredSecretKeys` vs. the GitHub Environment Secrets. |
   | `ImagePullBackOff` | Wrong tag, or the build job was skipped and no previous tag exists. |
   | Readiness failing | Probe path/port mismatch (`probes.*.path`, `containerPort`), or slow start under CPU throttling — extend the startup probe rather than the liveness probe. |
   | Pod vanished / rescheduled | Normal node churn. Not an incident unless it repeats. |

3. **Events** before logs: `clusters_query /api/v1/namespaces/<ns>/events?fieldSelector=involvedObject.name=<pod>`.
4. **Logs** need `kubectl logs -n <ns> <pod> --previous` (the MCP cannot fetch logs). Loki/Grafana at `grafana.nbrx.com` holds history for pods that no longer exist.
5. **Node pressure check** when several unrelated apps misbehave at once: `kubectl top nodes` and the `Allocated resources` block of `kubectl describe node`. Memory requests above ~95 % on every node means the cluster is genuinely full — fix by right-sizing requests, not by restarting workloads.
6. **Never "fix" by scaling up blindly.** Extra replicas or bigger requests on a saturated cluster provision additional paid nodes.

Alert triage for Grafana alerts follows `.github/instructions/alert.instructions.md`.

## Key Files in This Repo

| Path | Purpose |
|------|---------|
| `charts/webapp/` | cdk8s chart for web/API workloads |
| `charts/webapp/values.yaml` | All available chart values with defaults |
| `charts/webapp/values.schema.json` | JSON Schema for validation |
| `apps/tenants/<org>/<app>/<env>/` | Rendered manifests per app/env (managed by deploy CLI) |
| `examples/` | Example `app.yaml` and workflow files |
| `examples/README.md` | Detailed guide with all deployment patterns |
| `infrastructure/` | Cluster infrastructure (controllers, CRDs, etc.) |
| `.github/instructions/` | VS Code Copilot instructions for this repo |

## Quick Reference: New App Setup

1. Create `.cluster/app.yaml` with schema comment and values
2. Create `.cluster/app.<env>.yaml` for each environment (at minimum: `ingress.hosts`, `resources`)
3. Create `.github/workflows/ci.yaml` (copy from `examples/workflow-dev-only.yaml` or `workflow-release-train.yaml`)
4. Create `Dockerfile`
5. Add GitHub Environment Secrets matching `requiredSecretKeys`
6. Push to `main` — the deploy CLI handles everything else

## Pre-Commit Validation

Before committing changes to Kubernetes resources (YAML manifests, CRDs, Grafana resources, etc.), validate them against the cluster API server if cluster access is available:

```bash
kubectl --dry-run=server apply -f <file.yaml>
```

This catches schema errors, invalid field names, and CRD-specific validation issues before they reach Flux. Use `--context` if multiple clusters are configured.