279 lines
10 KiB
Markdown
279 lines
10 KiB
Markdown
<!-- go-doc:generated:begin source=go-bananas:docs/operations.md signature=7d8b696ba971c1cbd2e19854d26de304 generated=2026-10-05 18:14 UTC -->
|
|
Imported from docs/operations.md in the go-bananas repository.
|
|
|
|
---
|
|
|
|
# Operations
|
|
|
|
Running go-bananas in production: what to watch, what to do when something breaks,
|
|
and how to change things safely.
|
|
|
|
## Contents
|
|
|
|
- [Health](#health)
|
|
- [The runbook](#the-runbook)
|
|
- [Routine operations](#routine-operations)
|
|
- [Alerts](#alerts)
|
|
- [Diagnostics](#diagnostics)
|
|
- [Capacity](#capacity)
|
|
|
|
## Health
|
|
|
|
Three endpoints, deliberately different in what they check.
|
|
|
|
| Endpoint | Checks | Touches the database | Used by |
|
|
| --- | --- | --- | --- |
|
|
| `/livez` | Nothing. The process can serve HTTP | No | Liveness probe |
|
|
| `/healthz` | The process can serve HTTP, plus build information | No | Human, uptime checks |
|
|
| `/readyz` | The database answers | Yes | Readiness probe |
|
|
|
|
The liveness probe does not touch the database on purpose. A probe that failed on a
|
|
slow database would have the kubelet restart a process that is otherwise healthy,
|
|
turning a dependency blip into an outage. Readiness does check, because a pod that
|
|
cannot reach its database cannot serve anything and should leave the load balancer.
|
|
|
|
`/readyz` warns when the database is in-memory. That is legitimate in a test and
|
|
never legitimate in production, so it is surfaced rather than quietly accepted.
|
|
|
|
## The runbook
|
|
|
|
### The service will not start
|
|
|
|
The first log line is the reason. Every configuration problem is reported in full at
|
|
start-up, listing each problem rather than only the first, so one deploy tells you
|
|
everything that is wrong.
|
|
|
|
```
|
|
go-bananas: invalid configuration:
|
|
- SESSION_SECRET must be at least 32 bytes when auth is enabled, got 8
|
|
- OIDC_ISSUER must use https outside local development, got "http"
|
|
```
|
|
|
|
Configuration never reaches the log renderer, because the logger's own configuration
|
|
may be part of what is wrong.
|
|
|
|
### Users cannot sign in
|
|
|
|
Work down this list; the first three cover almost every case.
|
|
|
|
1. **The redirect URI.** It must be exactly `{publicBaseUrl}/auth/callback`, including
|
|
the base path if there is one.
|
|
|
|
```bash
|
|
kubectl exec -n go-bananas deploy/go-bananas -- \
|
|
wget -qO- "$ISSUER/.well-known/openid-configuration" | jq -r .authorization_endpoint
|
|
```
|
|
|
|
Register that callback with the provider, then compare it to `publicBaseUrl`.
|
|
|
|
2. **The issuer string.** It must match the `iss` claim in the ID token exactly,
|
|
including a trailing slash. Check both:
|
|
|
|
```bash
|
|
kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data.BANANAS_OIDC_ISSUER}{"\n"}'
|
|
curl -s "$ISSUER/.well-known/openid-configuration" | jq -r .issuer
|
|
```
|
|
|
|
3. **The client secret.** A stale one produces an invalid-grant error at the token
|
|
endpoint, which surfaces as a redirect back with an error parameter.
|
|
|
|
4. **The groups claim.** Absent when `groups` is not in `OIDC_SCOPES`. Users then land
|
|
with viewer rights and no way to tell why.
|
|
|
|
5. **The cookie.** A session rejected because `Secure` is set but the site is served
|
|
over plain HTTP, or `SameSite=None` without `Secure`. The browser refuses to store
|
|
it and the loop looks like an authentication failure.
|
|
|
|
### Everything can sign in but nobody can edit
|
|
|
|
The role mapping. A user in no configured group gets viewer, by design. The mapping
|
|
fails closed, so a typo in a group name silently produces read-only access rather
|
|
than an error.
|
|
|
|
```bash
|
|
kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data.BANANAS_OIDC_ADMIN_GROUPS}{"\n"}'
|
|
```
|
|
|
|
### The runtime is slow
|
|
|
|
1. **Cache hit rate.** Below 80% after warm-up is the first thing to look at.
|
|
|
|
```bash
|
|
curl -s :8080/metrics | grep bananas_cache_
|
|
```
|
|
|
|
A sustained low rate means a publish loop, a version-keyed cache storm, or more
|
|
themes than expected.
|
|
|
|
2. **p99 latency.** Over about 250 ms for the runtime path means a cache miss rate
|
|
problem, not a CPU problem.
|
|
|
|
3. **Is a CDN in front?** If not, that is the fix. Runtime responses are
|
|
immutable by URL and designed to be cached hard.
|
|
|
|
4. **Database contention.** `SQLITE_BUSY` in the logs means writers are queueing.
|
|
At the expected write rate that is a symptom, not a cause.
|
|
|
|
### The database is growing
|
|
|
|
Expected: a few megabytes for thousands of themes. `bananas_database_size_bytes` is
|
|
exposed on `/api/v1/stats`.
|
|
|
|
The audit log is the only table that grows without bound. Everything else is bounded
|
|
by the number of themes and their versions.
|
|
|
|
```sql
|
|
DELETE FROM audit_log WHERE at < datetime('now', '-180 days');
|
|
```
|
|
|
|
### A theme is not appearing for users
|
|
|
|
1. Is it published? A draft returns 404 from the runtime API by design, so a probe
|
|
cannot confirm it exists.
|
|
|
|
```bash
|
|
curl -s "$THEMES/rt/v1/manifest.json" | jq '.themes[].slug'
|
|
```
|
|
|
|
2. Which version did they get? Compare the response header across applications.
|
|
|
|
```bash
|
|
curl -sI "$THEMES/rt/v1/theme.css?slug=acme" | grep -i 'x-theme-version'
|
|
```
|
|
|
|
3. **Cached.** Expected within one `s-maxage` window, 300 s by default. To force it,
|
|
purge the CDN or lower `config.runtimeCacheControl`.
|
|
|
|
4. **Pinned.** A URL with `?v=7` is immutable by construction. If a consumer pinned a
|
|
version, it will keep getting that version, which is the point.
|
|
|
|
### A replica is crash-looping
|
|
|
|
`ReadWriteOnce` volume with more than one replica. Exactly one pod can mount it; the
|
|
others fail. The chart annotates the Deployment when it detects this combination.
|
|
|
|
```bash
|
|
kubectl get deploy go-bananas -n go-bananas -o jsonpath='{.spec.template.metadata.annotations}'
|
|
```
|
|
|
|
Fix: `deployment.replicaCount: 1`, or a filesystem that permits concurrent mounts.
|
|
|
|
## Routine operations
|
|
|
|
### Publishing a theme
|
|
|
|
An operator action in the admin interface, not a deployment. No rollout, no restart, no
|
|
cache flush. Consumers pick it up within one `s-maxage` window.
|
|
|
|
For a controlled release, publish a draft, pin consumers to the version, then
|
|
re-point. Two-step, and the old version remains available throughout.
|
|
|
|
### Rotating the session secret
|
|
|
|
```bash
|
|
kubectl create secret generic go-bananas-auth \
|
|
--namespace go-bananas \
|
|
--from-literal=session-secret="$(openssl rand -base64 48)" \
|
|
--from-literal=oidc-client-secret="$CURRENT_CLIENT_SECRET" \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
kubectl rollout restart deploy/go-bananas -n go-bananas
|
|
```
|
|
|
|
Every session is invalidated. Do it during a quiet period, and expect one login from
|
|
each user.
|
|
|
|
### Rotating the OIDC client secret
|
|
|
|
Update the Secret with both keys, then restart. The chart reads the value at start-up,
|
|
so a rolling restart is needed to pick it up.
|
|
|
|
### Adding a theme
|
|
|
|
Through the admin interface. To seed a set of themes into a fresh environment, export
|
|
them from the source instance and import them on the target.
|
|
|
|
```bash
|
|
curl -s -H "Cookie: $SESSION" "$SOURCE/api/v1/themes/acme/export" > acme.theme.json
|
|
curl -X POST -H "Cookie: $SESSION" -H 'Content-Type: application/json' \
|
|
--data @<(jq '{slug, name, tokens, meta, asDraft: true}' acme.theme.json) \
|
|
"$TARGET/api/v1/themes/import"
|
|
```
|
|
|
|
`asDraft: true` means an import can never publish something by accident.
|
|
|
|
### Backups
|
|
|
|
See [deployment.md](deployment.md#backup-and-restore). A backup is a copy of one
|
|
file; restore by copying it back and restarting.
|
|
|
|
Test a restore. An untested backup is a hypothesis.
|
|
|
|
## Alerts
|
|
|
|
| Alert | Condition | First response |
|
|
| --- | --- | --- |
|
|
| `GoBananasDown` | No `/readyz` success for 5 min | Check the pod. Configuration is the usual cause |
|
|
| `GoBananasHighErrorRate` | 5xx above 2% for 10 min | Check the logs. A database failure appears here first |
|
|
| `GoBananasLatencyHigh` | p99 runtime latency above 250 ms | Cache hit rate first |
|
|
| `GoBananasCacheHitRateLow` | Below 80% for 15 min | Publish loop, or more themes than expected |
|
|
| `GoBananasMemoryHigh` | Above 85% of the limit | The render cache, or too many themes |
|
|
| `GoBananasDatabaseBusy` | `SQLITE_BUSY` in the logs | Writer contention. Symptom of something else |
|
|
| `GoBananasDeploymentRestarted` | `kube_deployment_status_replicas` resets | Read the previous container's logs first |
|
|
|
|
## Diagnostics
|
|
|
|
```bash
|
|
# What is running?
|
|
kubectl get deploy go-bananas -n go-bananas -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
|
|
|
|
# What configuration is it actually using?
|
|
kubectl get cm go-bananas -n go-bananas -o jsonpath='{.data}' | tr ',' '\n'
|
|
|
|
# Why did the last pod die?
|
|
kubectl logs -n go-bananas deploy/go-bananas --previous --tail=100
|
|
|
|
# What has it been doing?
|
|
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
|
|
| jq 'select(.status >= 400)' | tail -50
|
|
|
|
# What did the provider say?
|
|
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
|
|
| jq -r 'select(.msg == "login rejected") | .reason'
|
|
|
|
# Is it healthy?
|
|
kubectl port-forward -n go-bananas svc/go-bananas 8080:80
|
|
curl -s localhost:8080/readyz | jq
|
|
curl -s localhost:8080/api/v1/stats -b cookies.txt | jq '.store'
|
|
```
|
|
|
|
The correlation id ties a user-visible failure to its log lines. The response body
|
|
carries `requestId` on every error, and `X-Request-ID` is on every response:
|
|
|
|
```bash
|
|
kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
|
|
| jq 'select(.requestId == "0f90d3cb9ab8e449")'
|
|
```
|
|
|
|
## Capacity
|
|
|
|
The service is small and the render cache does the work. Realistic figures for one
|
|
replica on a modest node:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| Steady-state runtime request | sub-millisecond, no database access |
|
|
| Cold render, 200 tokens | ~100 µs |
|
|
| Memory | ~40 MiB base, plus roughly 60 kB per cached theme per variant |
|
|
| Database | a few MB for a thousand themes |
|
|
| Capacity | a very large number of consumers, limited by network, not by the service |
|
|
|
|
Memory is the resource to watch: the render cache grows with the number of published
|
|
themes, multiplied by appearances and variants. A thousand themes across six
|
|
combinations is on the order of hundreds of megabytes. Either raise the limit or
|
|
publish fewer themes.
|
|
|
|
When memory becomes the constraint, `/api/v1/admin/cache/rebuild` re-warms it and
|
|
`/api/v1/stats` reports the entry count. A restart clears the cache entirely and
|
|
re-warms from the database, so a rolling restart is a valid way to reclaim memory.
|
|
<!-- go-doc:generated:end -->
|