go-bananas/docs/operations.md

Operations

Running go-bananas in production: what to watch, what to do when something breaks, and how to change things safely.

Contents

• Health
• The runbook
• Routine operations
• Alerts
• Diagnostics
• Capacity

Health

Three endpoints, deliberately different in what they check.

Endpoint	Checks	Touches the database	Used by
/livez	Nothing. The process can serve HTTP	No	Liveness probe
/healthz	The process can serve HTTP, plus build information	No	Human, uptime checks
/readyz	The database answers	Yes	Readiness probe

The liveness probe does not touch the database on purpose. A probe that failed on a slow database would have the kubelet restart a process that is otherwise healthy, turning a dependency blip into an outage. Readiness does check, because a pod that cannot reach its database cannot serve anything and should leave the load balancer.

/readyz warns when the database is in-memory. That is legitimate in a test and never legitimate in production, so it is surfaced rather than quietly accepted.

The runbook

The service will not start

The first log line is the reason. Every configuration problem is reported in full at start-up, listing each problem rather than only the first, so one deploy tells you everything that is wrong.

go-bananas: invalid configuration:
  - SESSION_SECRET must be at least 32 bytes when auth is enabled, got 8
  - OIDC_ISSUER must use https outside local development, got "http"

Configuration never reaches the log renderer, because the logger's own configuration may be part of what is wrong.

Users cannot sign in

Work down this list; the first three cover almost every case.

1. The redirect URI. It must be exactly {publicBaseUrl}/auth/callback, including the base path if there is one.

   kubectl exec -n go-bananas deploy/go-bananas -- \
     wget -qO- "$ISSUER/.well-known/openid-configuration" | jq -r .authorization_endpoint
   

   Register that callback with the provider, then compare it to publicBaseUrl.

2. The issuer string. It must match the iss claim in the ID token exactly, including a trailing slash. Check both:

   kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data.BANANAS_OIDC_ISSUER}{"\n"}'
   curl -s "$ISSUER/.well-known/openid-configuration" | jq -r .issuer
   

3. The client secret. A stale one produces an invalid-grant error at the token endpoint, which surfaces as a redirect back with an error parameter.

4. The groups claim. Absent when groups is not in OIDC_SCOPES. Users then land with viewer rights and no way to tell why.

5. The cookie. A session rejected because Secure is set but the site is served over plain HTTP, or SameSite=None without Secure. The browser refuses to store it and the loop looks like an authentication failure.

Everything can sign in but nobody can edit

The role mapping. A user in no configured group gets viewer, by design. The mapping fails closed, so a typo in a group name silently produces read-only access rather than an error.

kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data.BANANAS_OIDC_ADMIN_GROUPS}{"\n"}'

The runtime is slow

1. Cache hit rate. Below 80% after warm-up is the first thing to look at.

   curl -s :8080/metrics | grep bananas_cache_
   

   A sustained low rate means a publish loop, a version-keyed cache storm, or more themes than expected.

2. p99 latency. Over about 250 ms for the runtime path means a cache miss rate problem, not a CPU problem.

3. Is a CDN in front? If not, that is the fix. Runtime responses are immutable by URL and designed to be cached hard.

4. Database contention. SQLITE_BUSY in the logs means writers are queueing. At the expected write rate that is a symptom, not a cause.

The database is growing

Expected: a few megabytes for thousands of themes. bananas_database_size_bytes is exposed on /api/v1/stats.

The audit log is the only table that grows without bound. Everything else is bounded by the number of themes and their versions.

DELETE FROM audit_log WHERE at < datetime('now', '-180 days');

A theme is not appearing for users

1. Is it published? A draft returns 404 from the runtime API by design, so a probe cannot confirm it exists.

   curl -s "$THEMES/rt/v1/manifest.json" | jq '.themes[].slug'
   

2. Which version did they get? Compare the response header across applications.

   curl -sI "$THEMES/rt/v1/theme.css?slug=acme" | grep -i 'x-theme-version'
   

3. Cached. Expected within one s-maxage window, 300 s by default. To force it, purge the CDN or lower config.runtimeCacheControl.

4. Pinned. A URL with ?v=7 is immutable by construction. If a consumer pinned a version, it will keep getting that version, which is the point.

A replica is crash-looping

ReadWriteOnce volume with more than one replica. Exactly one pod can mount it; the others fail. The chart annotates the Deployment when it detects this combination.

kubectl get deploy go-bananas -n go-bananas -o jsonpath='{.spec.template.metadata.annotations}'

Fix: deployment.replicaCount: 1, or a filesystem that permits concurrent mounts.

Routine operations

Publishing a theme

An operator action in the admin interface, not a deployment. No rollout, no restart, no cache flush. Consumers pick it up within one s-maxage window.

For a controlled release, publish a draft, pin consumers to the version, then re-point. Two-step, and the old version remains available throughout.

Rotating the session secret

kubectl create secret generic go-bananas-auth \
  --namespace go-bananas \
  --from-literal=session-secret="$(openssl rand -base64 48)" \
  --from-literal=oidc-client-secret="$CURRENT_CLIENT_SECRET" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deploy/go-bananas -n go-bananas

Every session is invalidated. Do it during a quiet period, and expect one login from each user.

Rotating the OIDC client secret

Update the Secret with both keys, then restart. The chart reads the value at start-up, so a rolling restart is needed to pick it up.

Adding a theme

Through the admin interface. To seed a set of themes into a fresh environment, export them from the source instance and import them on the target.

curl -s -H "Cookie: $SESSION" "$SOURCE/api/v1/themes/acme/export" > acme.theme.json
curl -X POST -H "Cookie: $SESSION" -H 'Content-Type: application/json' \
  --data @<(jq '{slug, name, tokens, meta, asDraft: true}' acme.theme.json) \
  "$TARGET/api/v1/themes/import"

asDraft: true means an import can never publish something by accident.

Backups

See deployment.md. A backup is a copy of one file; restore by copying it back and restarting.

Test a restore. An untested backup is a hypothesis.

Alerts

Alert	Condition	First response
GoBananasDown	No /readyz success for 5 min	Check the pod. Configuration is the usual cause
GoBananasHighErrorRate	5xx above 2% for 10 min	Check the logs. A database failure appears here first
GoBananasLatencyHigh	p99 runtime latency above 250 ms	Cache hit rate first
GoBananasCacheHitRateLow	Below 80% for 15 min	Publish loop, or more themes than expected
GoBananasMemoryHigh	Above 85% of the limit	The render cache, or too many themes
GoBananasDatabaseBusy	SQLITE_BUSY in the logs	Writer contention. Symptom of something else
GoBananasDeploymentRestarted	kube_deployment_status_replicas resets	Read the previous container's logs first

Diagnostics

# What is running?
kubectl get deploy go-bananas -n go-bananas -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

# What configuration is it actually using?
kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data}' | tr ',' '\n'

# Why did the last pod die?
kubectl logs -n go-bananas deploy/go-bananas --previous --tail=100

# What has it been doing?
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.status >= 400)' | tail -50

# What did the provider say?
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq -r 'select(.msg == "login rejected") | .reason'

# Is it healthy?
kubectl port-forward -n go-bananas svc/go-bananas 8080:80
curl -s localhost:8080/readyz | jq
curl -s localhost:8080/api/v1/stats -b cookies.txt | jq '.store'

The correlation id ties a user-visible failure to its log lines. The response body carries requestId on every error, and X-Request-ID is on every response:

kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.requestId == "0f90d3cb9ab8e449")'

Capacity

The service is small and the render cache does the work. Realistic figures for one replica on a modest node:

Steady-state runtime request	sub-millisecond, no database access
Cold render, 200 tokens	~100 µs
Memory	~40 MiB base, plus roughly 60 kB per cached theme per variant
Database	a few MB for a thousand themes
Capacity	a very large number of consumers, limited by network, not by the service

Memory is the resource to watch: the render cache grows with the number of published themes, multiplied by appearances and variants. A thousand themes across six combinations is on the order of hundreds of megabytes. Either raise the limit or publish fewer themes.

When memory becomes the constraint, /api/v1/admin/cache/rebuild re-warms it and /api/v1/stats reports the entry count. A restart clears the cache entirely and re-warms from the database, so a rolling restart is a valid way to reclaim memory.