go-bananas/docs/operations.md

# Operations

Running go-bananas in production: what to watch, what to do when something breaks,
and how to change things safely.

## Contents

- [Health](#health)
- [The runbook](#the-runbook)
- [Routine operations](#routine-operations)
- [Alerts](#alerts)
- [Diagnostics](#diagnostics)
- [Capacity](#capacity)

## Health

Three endpoints, deliberately different in what they check.

| Endpoint | Checks | Touches the database | Used by |
| --- | --- | --- | --- |
| `/livez` | Nothing. The process can serve HTTP | No | Liveness probe |
| `/healthz` | The process can serve HTTP, plus build information | No | Human, uptime checks |
| `/readyz` | The database answers | Yes | Readiness probe |

The liveness probe does not touch the database on purpose. A probe that failed on a
slow database would have the kubelet restart a process that is otherwise healthy,
turning a dependency blip into an outage. Readiness does check, because a pod that
cannot reach its database cannot serve anything and should leave the load balancer.

`/readyz` warns when the database is in-memory. That is legitimate in a test and
never legitimate in production, so it is surfaced rather than quietly accepted.

## The runbook

### The service will not start

The first log line is the reason. Every configuration problem is reported in full at
start-up, listing each problem rather than only the first, so one deploy tells you
everything that is wrong.

```
go-bananas: invalid configuration:
  - SESSION_SECRET must be at least 32 bytes when auth is enabled, got 8
  - OIDC_ISSUER must use https outside local development, got "http"
```

Configuration never reaches the log renderer, because the logger's own configuration
may be part of what is wrong.

### Users cannot sign in

Work down this list; the first three cover almost every case.

1. **The redirect URI.** It must be exactly `{publicBaseUrl}/auth/callback`, including
   the base path if there is one.

   ```bash
   kubectl exec -n go-bananas deploy/go-bananas -- \
     wget -qO- "$ISSUER/.well-known/openid-configuration" | jq -r .authorization_endpoint
   ```

   Register that callback with the provider, then compare it to `publicBaseUrl`.

2. **The issuer string.** It must match the `iss` claim in the ID token exactly,
   including a trailing slash. Check both:

   ```bash
   kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data.BANANAS_OIDC_ISSUER}{"\n"}'
   curl -s "$ISSUER/.well-known/openid-configuration" | jq -r .issuer
   ```

3. **The client secret.** A stale one produces an invalid-grant error at the token
   endpoint, which surfaces as a redirect back with an error parameter.

4. **The groups claim.** Absent when `groups` is not in `OIDC_SCOPES`. Users then land
   with viewer rights and no way to tell why.

5. **The cookie.** A session rejected because `Secure` is set but the site is served
   over plain HTTP, or `SameSite=None` without `Secure`. The browser refuses to store
   it and the loop looks like an authentication failure.

### Everything can sign in but nobody can edit

The role mapping. A user in no configured group gets viewer, by design. The mapping
fails closed, so a typo in a group name silently produces read-only access rather
than an error.

```bash
kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data.BANANAS_OIDC_ADMIN_GROUPS}{"\n"}'
```

### The runtime is slow

1. **Cache hit rate.** Below 80% after warm-up is the first thing to look at.

   ```bash
   curl -s :8080/metrics | grep bananas_cache_
   ```

   A sustained low rate means a publish loop, a version-keyed cache storm, or more
   themes than expected.

2. **p99 latency.** Over about 250 ms for the runtime path means a cache miss rate
   problem, not a CPU problem.

3. **Is a CDN in front?** If not, that is the fix. Runtime responses are
   immutable by URL and designed to be cached hard.

4. **Database contention.** `SQLITE_BUSY` in the logs means writers are queueing.
   At the expected write rate that is a symptom, not a cause.

### The database is growing

Expected: a few megabytes for thousands of themes. `bananas_database_size_bytes` is
exposed on `/api/v1/stats`.

The audit log is the only table that grows without bound. Everything else is bounded
by the number of themes and their versions.

```sql
DELETE FROM audit_log WHERE at < datetime('now', '-180 days');
```

### A theme is not appearing for users

1. Is it published? A draft returns 404 from the runtime API by design, so a probe
   cannot confirm it exists.

   ```bash
   curl -s "$THEMES/rt/v1/manifest.json" | jq '.themes[].slug'
   ```

2. Which version did they get? Compare the response header across applications.

   ```bash
   curl -sI "$THEMES/rt/v1/theme.css?slug=acme" | grep -i 'x-theme-version'
   ```

3. **Cached.** Expected within one `s-maxage` window, 300 s by default. To force it,
   purge the CDN or lower `config.runtimeCacheControl`.

4. **Pinned.** A URL with `?v=7` is immutable by construction. If a consumer pinned a
   version, it will keep getting that version, which is the point.

### A replica is crash-looping

`ReadWriteOnce` volume with more than one replica. Exactly one pod can mount it; the
others fail. The chart annotates the Deployment when it detects this combination.

```bash
kubectl get deploy go-bananas -n go-bananas -o jsonpath='{.spec.template.metadata.annotations}'
```

Fix: `deployment.replicaCount: 1`, or a filesystem that permits concurrent mounts.

## Routine operations

### Publishing a theme

An operator action in the admin interface, not a deployment. No rollout, no restart, no
cache flush. Consumers pick it up within one `s-maxage` window.

For a controlled release, publish a draft, pin consumers to the version, then
re-point. Two-step, and the old version remains available throughout.

### Rotating the session secret

```bash
kubectl create secret generic go-bananas-auth \
  --namespace go-bananas \
  --from-literal=session-secret="$(openssl rand -base64 48)" \
  --from-literal=oidc-client-secret="$CURRENT_CLIENT_SECRET" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deploy/go-bananas -n go-bananas
```

Every session is invalidated. Do it during a quiet period, and expect one login from
each user.

### Rotating the OIDC client secret

Update the Secret with both keys, then restart. The chart reads the value at start-up,
so a rolling restart is needed to pick it up.

### Adding a theme

Through the admin interface. To seed a set of themes into a fresh environment, export
them from the source instance and import them on the target.

```bash
curl -s -H "Cookie: $SESSION" "$SOURCE/api/v1/themes/acme/export" > acme.theme.json
curl -X POST -H "Cookie: $SESSION" -H 'Content-Type: application/json' \
  --data @<(jq '{slug, name, tokens, meta, asDraft: true}' acme.theme.json) \
  "$TARGET/api/v1/themes/import"
```

`asDraft: true` means an import can never publish something by accident.

### Backups

See [deployment.md](deployment.md#backup-and-restore). A backup is a copy of one
file; restore by copying it back and restarting.

Test a restore. An untested backup is a hypothesis.

## Alerts

| Alert | Condition | First response |
| --- | --- | --- |
| `GoBananasDown` | No `/readyz` success for 5 min | Check the pod. Configuration is the usual cause |
| `GoBananasHighErrorRate` | 5xx above 2% for 10 min | Check the logs. A database failure appears here first |
| `GoBananasLatencyHigh` | p99 runtime latency above 250 ms | Cache hit rate first |
| `GoBananasCacheHitRateLow` | Below 80% for 15 min | Publish loop, or more themes than expected |
| `GoBananasMemoryHigh` | Above 85% of the limit | The render cache, or too many themes |
| `GoBananasDatabaseBusy` | `SQLITE_BUSY` in the logs | Writer contention. Symptom of something else |
| `GoBananasDeploymentRestarted` | `kube_deployment_status_replicas` resets | Read the previous container's logs first |

## Diagnostics

```bash
# What is running?
kubectl get deploy go-bananas -n go-bananas -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

# What configuration is it actually using?
kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data}' | tr ',' '\n'

# Why did the last pod die?
kubectl logs -n go-bananas deploy/go-bananas --previous --tail=100

# What has it been doing?
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.status >= 400)' | tail -50

# What did the provider say?
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq -r 'select(.msg == "login rejected") | .reason'

# Is it healthy?
kubectl port-forward -n go-bananas svc/go-bananas 8080:80
curl -s localhost:8080/readyz | jq
curl -s localhost:8080/api/v1/stats -b cookies.txt | jq '.store'
```

The correlation id ties a user-visible failure to its log lines. The response body
carries `requestId` on every error, and `X-Request-ID` is on every response:

```bash
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.requestId == "0f90d3cb9ab8e449")'
```

## Capacity

The service is small and the render cache does the work. Realistic figures for one
replica on a modest node:

| | |
| --- | --- |
| Steady-state runtime request | sub-millisecond, no database access |
| Cold render, 200 tokens | ~100 µs |
| Memory | ~40 MiB base, plus roughly 60 kB per cached theme per variant |
| Database | a few MB for a thousand themes |
| Capacity | a very large number of consumers, limited by network, not by the service |

Memory is the resource to watch: the render cache grows with the number of published
themes, multiplied by appearances and variants. A thousand themes across six
combinations is on the order of hundreds of megabytes. Either raise the limit or
publish fewer themes.

When memory becomes the constraint, `/api/v1/admin/cache/rebuild` re-warms it and
`/api/v1/stats` reports the entry count. A restart clears the cache entirely and
re-warms from the database, so a rolling restart is a valid way to reclaim memory.