go-bananas/docs/deployment.md

# Deployment

Running go-bananas, from a laptop to a production cluster.

---

## Contents

- [The deployment checklist](#the-deployment-checklist)
- [Configuration](#configuration)
- [Authentication](#authentication)
- [Helm](#helm)
- [The production checklist](#the-production-checklist)
- [Scaling](#scaling)
- [Backup and restore](#backup-and-restore)
- [Behind an ingress](#behind-an-ingress)
- [Monitoring](#monitoring)
- [Upgrading](#upgrading)
- [Troubleshooting](#troubleshooting)

---

## The deployment checklist

Four things must be right. Everything else has a working default.

1. **`config.publicBaseUrl`** — the externally reachable origin, no trailing slash.
   It builds the OIDC redirect URI, the absolute URLs in the runtime manifest and the
   HSTS decision. Wrong means a login loop.
2. **`identity.mode`** — `oidc` in production. `dev` and `disabled` are refused by the
   application outside a development environment.
3. **The OIDC client is registered** with that exact redirect URI:
   `{publicBaseUrl}/auth/callback`.
4. **`identity.oidc.existingSecret`** — a Secret you manage, holding a session secret
   of at least 32 bytes and the provider's client secret.

The Helm chart refuses to render a manifest it knows is wrong, so items 1–2 fail at
`helm template` time rather than at run time.

---

## Configuration

Everything is an environment variable. Both `BANANAS_` and `GO_BANANAS_` prefixes are
accepted, so a ConfigMap can be mounted under a distinct prefix without colliding
with the pod's own environment. That is what makes one image usable across
environments.

### Server

| Variable | Default | Notes |
| --- | --- | --- |
| `BANANAS_ENVIRONMENT` | `development` | `production` disables every development affordance |
| `BANANAS_HTTP_ADDR` | `:8080` | |
| `BANANAS_PUBLIC_BASE_URL` | `http://localhost:8080` | No trailing slash. Required in practice |
| `BANANAS_BASE_PATH` | — | For a path-based ingress |
| `BANANAS_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
| `BANANAS_LOG_FORMAT` | `text` | `json` in production |
| `BANANAS_HTTP_READ_TIMEOUT` | `15s` | |
| `BANANAS_HTTP_WRITE_TIMEOUT` | `30s` | Generous: this is the hot asset path |
| `BANANAS_SHUTDOWN_TIMEOUT` | `25s` | Must be below the pod's grace period |

### Storage

| Variable | Default | Notes |
| --- | --- | --- |
| `BANANAS_DATABASE_PATH` | `./data/go-bananas.db` | |
| `BANANAS_DB_MAX_CONNS` | `8` | Small on purpose. SQLite serialises writers |
| `BANANAS_DB_BUSY_TIMEOUT` | `5s` | How long a blocked writer waits |
| `BANANAS_SEED_ON_START` | `true` | Reconciles the built-in themes |

### Runtime

| Variable | Default | Notes |
| --- | --- | --- |
| `BANANAS_RUNTIME_PUBLIC` | `true` | The runtime API requires no credential by design |
| `BANANAS_RUNTIME_CACHE_CONTROL` | `public, max-age=60, s-maxage=300, stale-while-revalidate=86400` | |
| `BANANAS_RUNTIME_CDN_BASE_URL` | — | Absolute base for the manifest, when assets come from a CDN |
| `BANANAS_RATE_LIMIT_PER_MINUTE` | `3000` | High: the pattern is one request per page view |

### Network

| Variable | Default | Notes |
| --- | --- | --- |
| `BANANAS_TRUSTED_PROXIES` | — | CIDRs whose `X-Forwarded-For` is believed. Empty means forwarding headers are ignored entirely |
| `BANANAS_CORS_ALLOWED_ORIGINS` | — | For the admin API only. The runtime API is public by design |
| `BANANAS_ENABLE_METRICS` | `true` | |
| `BANANAS_ENABLE_PROFILER` | `false` | A network-reachable profiler is a DoS surface. Pair with a NetworkPolicy |

### Authentication

| Variable | Default | Notes |
| --- | --- | --- |
| `BANANAS_AUTH_MODE` | `dev` | `oidc`, `dev`, `none`. The latter two are refused outside development |
| `BANANAS_SESSION_SECRET` | ephemeral in dev | At least 32 bytes. Rotating it logs out everyone |
| `BANANAS_SESSION_TTL` | `8h` | |
| `BANANAS_SESSION_SECURE` | `true` outside dev | |
| `BANANAS_SESSION_SAMESITE` | `Lax` | `None` requires `Secure` |
| `BANANAS_OIDC_ISSUER` | — | Must match the `iss` claim exactly |
| `BANANAS_OIDC_CLIENT_ID` | — | |
| `BANANAS_OIDC_CLIENT_SECRET` | — | From a Secret, never a ConfigMap |
| `BANANAS_OIDC_SCOPES` | `profile,email,groups` | `groups` is what the role mapping reads |
| `BANANAS_OIDC_ADMIN_GROUPS` | — | Comma separated |
| `BANANAS_OIDC_EDITOR_GROUPS` | — | |
| `BANANAS_OIDC_VIEWER_GROUPS` | — | |
| `BANANAS_OIDC_REDIRECT_URL` | derived | Override only if the provider has a different URI registered |
| `BANANAS_OIDC_SKIP_TLS_VERIFY` | `false` | Development only. Refused in production |

Full reference in `internal/config/config.go`, which is the single source of truth and
carries the reasoning for each default.

---

## Authentication

### The three modes

| Mode | Behaviour | Allowed in production |
| --- | --- | --- |
| `oidc` | OpenID Connect against a real provider | **Yes. This is the mode.** |
| `dev` | Every request is a local administrator | No. Refused by the application |
| `none` | Every request is granted administrator access | No. Refused by the application |

The refusal is in the code, not in documentation, and it is a hard `fail` in the
constructor — an accidentally deployed bypass is a total authentication bypass, so it
is made impossible to construct.

### The shared development realm

The chart defaults to `https://auth-dev.urpi.be/realms/urpi`, so a development cluster
works with no identity configuration.

The realm path is part of the issuer. The value must match the `iss` claim in the ID
token **exactly**, including a trailing slash if the provider emits one. A mismatch
here is the single most common cause of `invalid issuer` at the callback, and it is
worth checking first when a login loop appears.

### Registering the client

Redirect URIs, exact:

```
https://themes.example.com/auth/callback
https://bananas-dev.example.com/auth/callback
```

Post-logout redirect, if the provider requires one:

```
https://themes.example.com/
```

Web origins, for a cross-origin admin UI (same-origin needs none):

```
https://app.example.com
```

### Group-to-role mapping

```yaml
identity:
  oidc:
    adminGroups: [platform-admins]
    editorGroups: [design-team]
    viewerGroups: [everyone]
```

The most privileged matching group wins. **A user in no configured group gets
viewer.** That is deliberate: a mapping mistake must fail closed.

Group names are realm-specific and are deliberately not guessed by the chart. Read
them from the provider:

```bash
curl -s "$ISSUER/protocol/openid-connect/certs" | jq -r '.keys[].kid'
# or, with a token:
curl -s -H "Authorization: Bearer $TOKEN" "$ISSUER/admin/realms/urpi/groups" | jq '.[].name'
```

### A note on sessions

The session is a signed cookie with no server-side record
([ADR 0003](adr/0003-stateful-sessions.md)). Concretely:

- any replica authenticates any request, so a rollout logs nobody out;
- a stolen session cannot be revoked before it expires — bounded by
  `BANANAS_SESSION_TTL`, eight hours by default;
- full invalidation means rotating the session secret, which logs out everyone.

For a design-token service this is the right trade. If you need per-session
revocation, or your organisation's policy requires it, this is the one component to
replace.

---

## Helm

```bash
helm upgrade --install go-bananas oci://ghcr.io/urpi/charts/go-bananas \
  --namespace go-bananas --create-namespace \
  --set config.publicBaseUrl=https://themes.example.com \
  --set identity.oidc.issuer=https://auth.example.com/realms/urpi \
  --set identity.oidc.clientId=go-bananas \
  --set identity.oidc.existingSecret=go-bananas-auth \
  --set identity.oidc.adminGroups[0]=platform-admins \
  --set ingress.enabled=true \
  --set ingress.hosts[0].host=themes.example.com \
  --set ingress.tls[0].secretName=go-bananas-tls \
  --set ingress.tls[0].hosts[0]=themes.example.com \
  --wait --timeout 5m
```

### The Secret

```bash
kubectl create secret generic go-bananas-auth \
  --namespace go-bananas \
  --from-literal=session-secret="$(openssl rand -base64 48)" \
  --from-literal=oidc-client-secret="$CLIENT_SECRET_FROM_YOUR_SECRET_MANAGER"
```

Key names are configurable through `identity.sessionSecretKey`,
`identity.oidc.clientSecretKey` and `identity.oidc.existingSecretKeys`, for a
Secret managed by an external secrets operator.

A chart-managed Secret (`identity.oidc.generateSecret=true`) generates the session key
and preserves it across upgrades by reading the live Secret back — so an upgrade does
not sign everyone out. It **cannot** generate the OIDC client secret, because that has
to come from the provider. A generated Secret therefore has an empty client secret,
which means the deployment fails to authenticate until one is supplied. For anything
real, use `existingSecret`.

### Validation at render time

The chart fails rather than producing a manifest it knows is wrong:

| Misconfiguration | Message |
| --- | --- |
| `publicBaseUrl` empty | Required; names what it is used for |
| `publicBaseUrl` has a trailing slash | The exact acceptable form |
| `shutdownTimeoutMs` ≥ grace period | Kubernetes would SIGKILL mid-shutdown |
| `identity.mode` unknown | The three valid modes |
| `identity.mode=oidc`, no issuer | Required, with the development-realm form |
| `identity.mode=oidc`, no secret | Names both ways to supply one |
| Both `existingSecret` and `generateSecret` | Mutually exclusive |
| `skipTLSVerify` with `environment: production` | Refused, with the reason |
| Any value outside the schema | The JSON-schema path |

Failing at render time means a bad release never reaches the API server: there is no
half-created Deployment and no `CrashLoopBackOff` to diagnose at 3am.

### Render it before you apply it

```bash
make helm-template          # defaults
make helm-template-auth     # configured for OIDC
helm lint deploy/helm/go-bananas --strict
```

---

## The production checklist

- [ ] `config.environment: production`
- [ ] `config.publicBaseUrl` is the real HTTPS origin, no trailing slash
- [ ] `identity.mode: oidc`
- [ ] The OIDC client registered with `{publicBaseUrl}/auth/callback`
- [ ] `identity.oidc.issuer` verified against the provider's discovery document
- [ ] Group-to-role mapping populated and **tested with a real account** in each role
- [ ] `identity.oidc.existingSecret` set; the session secret is at least 32 random bytes
- [ ] Ingress with TLS; `service.sessionSecure: true`
- [ ] `config.trustedProxies` set to the ingress controller, so the audit log records real client addresses
- [ ] Persistence enabled; a snapshot scheduled
- [ ] `networkPolicy.enabled: true`
- [ ] Resources set, with a memory limit
- [ ] Prometheus scraping `/metrics`
- [ ] A backup job that copies the SQLite file
- [ ] `make helm-template` reviewed

### Verify a deployment

```bash
kubectl --namespace go-bananas get pods -l app.kubernetes.io/instance=go-bananas
kubectl --namespace go-bananas port-forward svc/go-bananas 8080:80

curl -s localhost:8080/readyz | jq            # "status": "ready"
curl -s localhost:8080/rt/v1/manifest.json | jq '.themes[].slug'
curl -sI 'localhost:8080/rt/v1/theme.css?slug=midnight' | grep -i x-theme-version
```

Then open the admin interface and confirm a real account can sign in and see a theme.

---

## Scaling

### Read scaling: put a CDN in front

The runtime is the hot path and it is entirely cacheable. Immutable-by-URL responses
mean a CDN is a complete answer for read scaling.

```yaml
config:
  runtimeCdnBaseUrl: https://cdn.example.com/themes
```

The manifest then hands consumers the CDN URL, so they never reach the service at all.

### The one real constraint: one writer

SQLite on a local volume is single-node. Three shapes:

**One replica, one volume.** The recommended production shape. One replica serves a
very large number of consumers, because after the render cache a request is a map
lookup.

**Many replicas, a shared filesystem.** Works with NFS, CephFS or a ReadWriteMany
volume — anything that permits concurrent mounts and shared mmap, which WAL mode
requires. Expect lower throughput than local disk; measure before choosing it.

```yaml
persistence:
  type: persistentVolumeClaim
  accessMode: ReadWriteMany
  size: 2Gi
  storageClassName: nfs-client
deployment:
  replicaCount: 3
```

The chart annotates the Deployment when a ReadWriteOnce volume is combined with more
than one replica, because that combination cannot work: only one pod mounts the
volume and the others crash-loop.

### If you outgrow SQLite

Move to PostgreSQL. The change is contained to `internal/store` — the only package
with SQL — and its interface is the set of methods it exposes rather than SQLite's
features. [ADR 0001](adr/0001-embedded-sqlite.md) covers when that point arrives.

---

## Backup and restore

A backup is a copy of one file, and the supported way to take it is the `backup`
subcommand:

```bash
kubectl exec -n go-bananas deploy/go-bananas -- \
  /usr/local/bin/go-bananas backup --to /srv/data/snapshots/$(date -u +%Y%m%dT%H%M%SZ).db
```

It uses SQLite's `VACUUM INTO`, which reads through the database's own backup
machinery. Two consequences matter:

- **It is safe against a running service.** The snapshot is consistent even while
  the application is serving writes, and it does not block them for the length of
  the copy. There is no need to scale to zero.
- **Do not substitute `cp` for it.** Copying a SQLite file while a write is in
  flight produces a torn copy: the header can describe one set of pages while the
  file holds another. The result frequently opens without complaint and is corrupt.
  Copying the `-wal` file alongside it is worse, because the two files are then
  captured at different instants and the pair may not correspond to any point in
  time.

The command refuses to overwrite an existing file, so a stale snapshot can never be
mistaken for a fresh one, and it will not snapshot an in-memory database, which has
nothing to preserve.

Run it from cron, a Kubernetes CronJob, or by hand. When you do it by hand, the
output path is printed in full so you can confirm where the file went.

### Restoring

```bash
kubectl scale deploy/go-bananas -n go-bananas --replicas=0
kubectl cp ./backup.db go-bananas-0:/srv/data/go-bananas.db
kubectl delete pod -n go-bananas -l app.kubernetes.io/instance=go-bananas
kubectl scale deploy/go-bananas -n go-bananas --replicas=1
```

A restored database serves immediately. The built-in themes are reconciled at
start-up, and a built-in theme an operator had edited is recognised by its version and
left alone.

Scale to zero before replacing the file. Copying over a database that another
process has open produces exactly the torn copy described above, and the running
service will keep writing to the file you just replaced.

### Retention

The audit log is the only unbounded table. Everything else is bounded by the number of
themes and their versions. Prune it on a schedule:

```sql
DELETE FROM audit_log WHERE at < datetime('now', '-180 days');
```

The chart's snapshot CronJob (`persistence.snapshot.enabled`) writes to
`<mountPath>/snapshots` on the same volume, so a snapshot survives the pod that took
it. Restore from there with the same `backup` machinery in reverse.

---

## Behind an ingress

### Path-based

```yaml
config:
  publicBaseUrl: https://example.com/themes
  basePath: /themes
```

Every route, including the OIDC redirect, is prefixed. The provider must be registered
with `https://example.com/themes/auth/callback`.

### Correct client addresses

`X-Forwarded-For` is believed only from a configured proxy. Without this, any client
can set it and spoof its address in the audit log.

```yaml
config:
  trustedProxies:
    - 10.42.0.0/16      # the ingress controller's node or pod CIDR
```

### TLS

Terminate at the ingress. `service.sessionSecure: true` marks the session cookie
`Secure`, and HSTS is advertised only when `publicBaseUrl` is `https://`.

Behind a proxy that terminates TLS and forwards `X-Forwarded-Proto`, the service
derives scheme from `publicBaseUrl`, not from the header, which is the correct trust
boundary: the operator states the truth in configuration rather than a client
asserting it.

---

## Monitoring

### Metrics

| Metric | Type | Alert when |
| --- | --- | --- |
| `bananas_http_requests_total` | counter | Rate collapses, or 5xx ratio climbs |
| `bananas_http_request_duration_seconds` | histogram | p99 above ~250 ms |
| `bananas_cache_hits_total` / `_misses_total` | counter | Hit rate below ~80% after warm-up |
| `bananas_go_memory_bytes` | gauge | Approaches the limit |
| `bananas_uptime_seconds` | gauge | Resets: the process restarted |

A sustained cache miss rate means a publish loop, a version-keyed cache storm, or
more themes than expected. It is the first thing to look at when latency rises.

### Logs

One JSON line per request, carrying `requestId`, `route`, `status`, `duration`,
`clientIp`, `subject` and `query`. The `subject` field is the authenticated identity,
so "who did this" needs no second query.

```bash
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.status >= 500)'
```

### Probes

| Path | Use | Touches the database |
| --- | --- | --- |
| `/livez` | Liveness | No |
| `/healthz` | Liveness, with build information | No |
| `/readyz` | Readiness | **Yes** |

The split is deliberate. A liveness probe that touched the database would have the
kubelet restart a process that is otherwise healthy, turning a dependency blip into
an outage. Readiness does check the database, because a pod that cannot reach it
cannot serve.

---

## Upgrading

```bash
helm upgrade go-bananas oci://ghcr.io/urpi/charts/go-bananas \
  --namespace go-bananas \
  --set image.tag=1.4.2 \
  --wait --timeout 5m
```

Migrations apply at start-up and are idempotent. The built-in themes are reconciled:
missing ones are created, and ones an operator has edited are recognised by version
and left alone.

Enable `persistence.snapshot.enabled` for a scheduled snapshot. It is a CronJob
that runs before each upgrade, and it costs a few megabytes.

### Rolling back

```bash
helm rollback go-bananas 3 --namespace go-bananas --wait
```

Safe, because the schema is additive and migrations are idempotent. If a future change
is destructive, it gets a snapshot step in the chart and an entry in
[the runbook](operations.md).

---

## Troubleshooting

**A login loop.** The redirect URI does not match what the provider has registered.
Check in this order:

1. `{publicBaseUrl}/auth/callback` in the provider's client configuration.
2. `publicBaseUrl` has no trailing slash and matches the scheme exactly.
3. `basePath` is reflected in the registered URI.
4. `identity.oidc.issuer` matches the discovery document's `issuer` field **exactly**.

```bash
kubectl exec -n go-bananas deploy/go-bananas -- \
  wget -qO- "$ISSUER/.well-known/openid-configuration" | jq -r .issuer
```

**`invalid issuer`.** The `iss` claim in the ID token differs from the configured
issuer. Usually a trailing slash, or a realm path that is not the one configured.

**`no such user in the roles mapping`.** Not an error this service produces: a user in
no configured group gets viewer, silently. Check the `groups` claim is present —
`groups` must be in `OIDC_SCOPES`.

**A pod stuck in `CrashLoopBackOff`.** Almost always configuration. The first log line
is the reason; every configuration problem is reported in full at start-up.

**`SQLITE_BUSY`.** A writer is waiting for the lock. Raise
`config.database.busyTimeoutMs`, or check for a publish loop.

**Replicas crash-looping with a ReadWriteOnce volume.** Exactly one pod can mount it.
Set `deployment.replicaCount: 1`, or use a filesystem that permits concurrent mounts.

**A theme looks wrong in one application.** Compare `X-Theme-Version` across
applications. A CDN or browser holding an old version explains it; the runtime
manifest's `baseUrl` will show whether a CDN is in the path.

**The stylesheet is stale after a publish.** Expected within one `s-maxage` window
(300 s by default). To force it, lower `config.runtimeCacheControl` or purge the CDN.