go-bananas/docs/deployment.md

Deployment

Running go-bananas, from a laptop to a production cluster.

---

Contents

• The deployment checklist
• Configuration
• Authentication
• Helm
• Local deployment (lcl)
• The production checklist
• Scaling
• Backup and restore
• Behind an ingress
• Monitoring
• Upgrading
• Troubleshooting

---

The deployment checklist

Four things must be right. Everything else has a working default.

1. config.publicBaseUrl — the externally reachable origin, no trailing slash. It builds the OIDC redirect URI, the absolute URLs in the runtime manifest and the HSTS decision. Wrong means a login loop.
2. identity.mode — oidc in production. dev and disabled are refused by the application outside a development environment.
3. The OIDC client is registered with that exact redirect URI: {publicBaseUrl}/auth/callback.
4. identity.oidc.existingSecret — a Secret you manage, holding a session secret of at least 32 bytes and the provider's client secret.

The Helm chart refuses to render a manifest it knows is wrong, so items 1–2 fail at helm template time rather than at run time.

---

Configuration

Everything is an environment variable. Both BANANAS_ and GO_BANANAS_ prefixes are accepted, so a ConfigMap can be mounted under a distinct prefix without colliding with the pod's own environment. That is what makes one image usable across environments.

Server

Variable	Default	Notes
BANANAS_ENVIRONMENT	development	production disables every development affordance
BANANAS_HTTP_ADDR	:8080
BANANAS_PUBLIC_BASE_URL	http://localhost:8080	No trailing slash. Required in practice
BANANAS_BASE_PATH	—	For a path-based ingress
BANANAS_LOG_LEVEL	info	debug, info, warn, error
BANANAS_LOG_FORMAT	text	json in production
BANANAS_HTTP_READ_TIMEOUT	15s
BANANAS_HTTP_WRITE_TIMEOUT	30s	Generous: this is the hot asset path
BANANAS_SHUTDOWN_TIMEOUT	25s	Must be below the pod's grace period

Storage

Variable	Default	Notes
BANANAS_DATABASE_PATH	./data/go-bananas.db
BANANAS_DB_MAX_CONNS	8	Small on purpose. SQLite serialises writers
BANANAS_DB_BUSY_TIMEOUT	5s	How long a blocked writer waits
BANANAS_SEED_ON_START	true	Reconciles the built-in themes

Runtime

Variable	Default	Notes
BANANAS_RUNTIME_PUBLIC	true	The runtime API requires no credential by design
BANANAS_RUNTIME_CACHE_CONTROL	public, max-age=60, s-maxage=300, stale-while-revalidate=86400
BANANAS_RUNTIME_CDN_BASE_URL	—	Absolute base for the manifest, when assets come from a CDN
BANANAS_RATE_LIMIT_PER_MINUTE	3000	High: the pattern is one request per page view

Network

Variable	Default	Notes
BANANAS_TRUSTED_PROXIES	—	CIDRs whose X-Forwarded-For is believed. Empty means forwarding headers are ignored entirely
BANANAS_CORS_ALLOWED_ORIGINS	—	For the admin API only. The runtime API is public by design
BANANAS_ENABLE_METRICS	true
BANANAS_ENABLE_PROFILER	false	A network-reachable profiler is a DoS surface. Pair with a NetworkPolicy

Authentication

Variable	Default	Notes
BANANAS_AUTH_MODE	dev	oidc, dev, none. The latter two are refused outside development
BANANAS_SESSION_SECRET	ephemeral in dev	At least 32 bytes. Rotating it logs out everyone
BANANAS_SESSION_TTL	8h
BANANAS_SESSION_SECURE	true outside dev
BANANAS_SESSION_SAMESITE	Lax	None requires Secure
BANANAS_OIDC_ISSUER	—	Must match the iss claim exactly
BANANAS_OIDC_CLIENT_ID	—
BANANAS_OIDC_CLIENT_SECRET	—	From a Secret, never a ConfigMap
BANANAS_OIDC_SCOPES	profile,email,groups	groups is what the role mapping reads
BANANAS_OIDC_ADMIN_GROUPS	—	Comma separated
BANANAS_OIDC_EDITOR_GROUPS	—
BANANAS_OIDC_VIEWER_GROUPS	—
BANANAS_OIDC_REDIRECT_URL	derived	Override only if the provider has a different URI registered
BANANAS_OIDC_SKIP_TLS_VERIFY	false	Development only. Refused in production

Full reference in internal/config/config.go, which is the single source of truth and carries the reasoning for each default.

---

Authentication

The three modes

Mode	Behaviour	Allowed in production
oidc	OpenID Connect against a real provider	Yes. This is the mode.
dev	Every request is a local administrator	No. Refused by the application
none	Every request is granted administrator access	No. Refused by the application

The refusal is in the code, not in documentation, and it is a hard fail in the constructor — an accidentally deployed bypass is a total authentication bypass, so it is made impossible to construct.

The shared development realm

The chart defaults to https://auth.lcl-urpi.be/realms/urpi, so a development cluster works with no identity configuration.

The realm path is part of the issuer. The value must match the iss claim in the ID token exactly, including a trailing slash if the provider emits one. A mismatch here is the single most common cause of invalid issuer at the callback, and it is worth checking first when a login loop appears.

Registering the client

Redirect URIs, exact:

https://themes.example.com/auth/callback
https://bananas-dev.example.com/auth/callback

Post-logout redirect, if the provider requires one:

https://themes.example.com/

Web origins, for a cross-origin admin UI (same-origin needs none):

https://app.example.com

Group-to-role mapping

identity:
  oidc:
    adminGroups: [platform-admins]
    editorGroups: [design-team]
    viewerGroups: [everyone]

The most privileged matching group wins. A user in no configured group gets viewer. That is deliberate: a mapping mistake must fail closed.

Group names are realm-specific and are deliberately not guessed by the chart. Read them from the provider:

curl -s "$ISSUER/protocol/openid-connect/certs" | jq -r '.keys[].kid'
# or, with a token:
curl -s -H "Authorization: Bearer $TOKEN" "$ISSUER/admin/realms/urpi/groups" | jq '.[].name'

A note on sessions

The session is a signed cookie with no server-side record (ADR 0003). Concretely:

• any replica authenticates any request, so a rollout logs nobody out;
• a stolen session cannot be revoked before it expires — bounded by BANANAS_SESSION_TTL, eight hours by default;
• full invalidation means rotating the session secret, which logs out everyone.

For a design-token service this is the right trade. If you need per-session revocation, or your organisation's policy requires it, this is the one component to replace.

---

Helm

helm upgrade --install go-bananas oci://ghcr.io/urpi/charts/go-bananas \
  --namespace go-bananas --create-namespace \
  --set config.publicBaseUrl=https://themes.example.com \
  --set identity.oidc.issuer=https://auth.example.com/realms/urpi \
  --set identity.oidc.clientId=go-bananas \
  --set identity.oidc.existingSecret=go-bananas-auth \
  --set identity.oidc.adminGroups[0]=platform-admins \
  --set ingress.enabled=true \
  --set ingress.hosts[0].host=themes.example.com \
  --set ingress.tls[0].secretName=go-bananas-tls \
  --set ingress.tls[0].hosts[0]=themes.example.com \
  --wait --timeout 5m

The Secret

kubectl create secret generic go-bananas-auth \
  --namespace go-bananas \
  --from-literal=session-secret="$(openssl rand -base64 48)" \
  --from-literal=oidc-client-secret="$CLIENT_SECRET_FROM_YOUR_SECRET_MANAGER"

Key names are configurable through identity.sessionSecretKey, identity.oidc.clientSecretKey and identity.oidc.existingSecretKeys, for a Secret managed by an external secrets operator.

A chart-managed Secret (identity.oidc.generateSecret=true) generates the session key and preserves it across upgrades by reading the live Secret back — so an upgrade does not sign everyone out. It cannot generate the OIDC client secret, because that has to come from the provider. A generated Secret therefore has an empty client secret, which means the deployment fails to authenticate until one is supplied. For anything real, use existingSecret.

Validation at render time

The chart fails rather than producing a manifest it knows is wrong:

Misconfiguration	Message
publicBaseUrl empty	Required; names what it is used for
publicBaseUrl has a trailing slash	The exact acceptable form
shutdownTimeoutMs ≥ grace period	Kubernetes would SIGKILL mid-shutdown
identity.mode unknown	The three valid modes
identity.mode=oidc, no issuer	Required, with the development-realm form
identity.mode=oidc, no secret	Names both ways to supply one
Both existingSecret and generateSecret	Mutually exclusive
skipTLSVerify with environment: production	Refused, with the reason
Any value outside the schema	The JSON-schema path

Failing at render time means a bad release never reaches the API server: there is no half-created Deployment and no CrashLoopBackOff to diagnose at 3am.

Render it before you apply it

make helm-template          # defaults
make helm-template-auth     # configured for OIDC
helm lint deploy/helm/go-bananas --strict

---

Local deployment (lcl)

The lcl environment is the developer's own single-node k3s cluster used for local integration testing. It is defined by deploy/helm/go-bananas/values-lcl.yaml:

• config.environment: development so the local auth bypass is allowed.
• identity.mode: dev so you can sign in without an OIDC provider.
• image.registry: localhost:5000 so the cluster pulls from a local registry.
• Ingress host bananas-lcl.intern-urpi.be, terminated by the local TLS edge.

All the go-* applications share the go-insane-lcl namespace; go-bananas is a Helm release named go-bananas inside it.

Prerequisites

1. A running local k3s cluster and a registry at localhost:5000.
2. bananas-lcl.intern-urpi.be resolving to the node (/etc/hosts) behind the TLS edge.
3. docker buildx, helm and kubectl.

Kubeconfig caveat. On this machine the default local context goes through a Rancher proxy (https://rancher.urpi.be/k8s/clusters/local) which can 502. When it does, talk to k3s directly instead: export KUBECONFIG=/etc/rancher/k3s/k3s.yaml. Everything below respects that variable.

One-shot build + deploy + verify

KUBECONFIG=/etc/rancher/k3s/k3s.yaml ./hack/deploy-lcl.sh
# or, equivalently:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml make deploy-lcl

This builds a fresh image (with the web admin UI and runtime SDK compiled in), pushes it to the local registry, runs helm upgrade --install, waits for the rollout, and curls the runtime manifest through the TLS edge. The image tag defaults to a build timestamp so a rebuild is always re-pulled (IfNotPresent would otherwise serve a stale cached tag). Override it with LCL_TAG=....

Step by step

make install             # Go and Node dependencies
make image-lcl           # build + push to localhost:5000/go-bananas:<LCL_TAG>
make helm-template-lcl   # sanity-check the rendered manifest
make helm-install-lcl    # helm upgrade --install into go-insane-lcl

LCL_TAG, LCL_REGISTRY and LCL_NAMESPACE are overridable:

make deploy-lcl LCL_TAG=v1 LCL_REGISTRY=registry:5000 LCL_NAMESPACE=go-insane-lcl

Sign in and verify

The release uses dev authentication, so any user is a local administrator. Open https://bananas-lcl.intern-urpi.be/?dev-user=you@example.test, then:

curl -sk https://bananas-lcl.intern-urpi.be/readyz
curl -sk https://bananas-lcl.intern-urpi.be/rt/v1/manifest.json | jq '.themes[].slug'

Remove

KUBECONFIG=/etc/rancher/k3s/k3s.yaml make helm-uninstall-lcl

---

The production checklist

• config.environment: production
• config.publicBaseUrl is the real HTTPS origin, no trailing slash
• identity.mode: oidc
• The OIDC client registered with {publicBaseUrl}/auth/callback
• identity.oidc.issuer verified against the provider's discovery document
• Group-to-role mapping populated and tested with a real account in each role
• identity.oidc.existingSecret set; the session secret is at least 32 random bytes
• Ingress with TLS; service.sessionSecure: true
• config.trustedProxies set to the ingress controller, so the audit log records real client addresses
• Persistence enabled; a snapshot scheduled
• networkPolicy.enabled: true
• Resources set, with a memory limit
• Prometheus scraping /metrics
• A backup job that copies the SQLite file
• make helm-template reviewed

Verify a deployment

kubectl --namespace go-bananas get pods -l app.kubernetes.io/instance=go-bananas
kubectl --namespace go-bananas port-forward svc/go-bananas 8080:80

curl -s localhost:8080/readyz | jq            # "status": "ready"
curl -s localhost:8080/rt/v1/manifest.json | jq '.themes[].slug'
curl -sI 'localhost:8080/rt/v1/theme.css?slug=midnight' | grep -i x-theme-version

Then open the admin interface and confirm a real account can sign in and see a theme.

---

Scaling

Read scaling: put a CDN in front

The runtime is the hot path and it is entirely cacheable. Immutable-by-URL responses mean a CDN is a complete answer for read scaling.

config:
  runtimeCdnBaseUrl: https://cdn.example.com/themes

The manifest then hands consumers the CDN URL, so they never reach the service at all.

The one real constraint: one writer

SQLite on a local volume is single-node. Three shapes:

One replica, one volume. The recommended production shape. One replica serves a very large number of consumers, because after the render cache a request is a map lookup.

Many replicas, a shared filesystem. Works with NFS, CephFS or a ReadWriteMany volume — anything that permits concurrent mounts and shared mmap, which WAL mode requires. Expect lower throughput than local disk; measure before choosing it.

persistence:
  type: persistentVolumeClaim
  accessMode: ReadWriteMany
  size: 2Gi
  storageClassName: nfs-client
deployment:
  replicaCount: 3

The chart annotates the Deployment when a ReadWriteOnce volume is combined with more than one replica, because that combination cannot work: only one pod mounts the volume and the others crash-loop.

If you outgrow SQLite

Move to PostgreSQL. The change is contained to internal/store — the only package with SQL — and its interface is the set of methods it exposes rather than SQLite's features. ADR 0001 covers when that point arrives.

---

Backup and restore

A backup is a copy of one file, and the supported way to take it is the backup subcommand:

kubectl exec -n go-bananas deploy/go-bananas -- \
  /usr/local/bin/go-bananas backup --to /srv/data/snapshots/$(date -u +%Y%m%dT%H%M%SZ).db

It uses SQLite's VACUUM INTO, which reads through the database's own backup machinery. Two consequences matter:

• It is safe against a running service. The snapshot is consistent even while the application is serving writes, and it does not block them for the length of the copy. There is no need to scale to zero.
• Do not substitute cp for it. Copying a SQLite file while a write is in flight produces a torn copy: the header can describe one set of pages while the file holds another. The result frequently opens without complaint and is corrupt. Copying the -wal file alongside it is worse, because the two files are then captured at different instants and the pair may not correspond to any point in time.

The command refuses to overwrite an existing file, so a stale snapshot can never be mistaken for a fresh one, and it will not snapshot an in-memory database, which has nothing to preserve.

Run it from cron, a Kubernetes CronJob, or by hand. When you do it by hand, the output path is printed in full so you can confirm where the file went.

Restoring

kubectl scale deploy/go-bananas -n go-bananas --replicas=0
kubectl cp ./backup.db go-bananas-0:/srv/data/go-bananas.db
kubectl delete pod -n go-bananas -l app.kubernetes.io/instance=go-bananas
kubectl scale deploy/go-bananas -n go-bananas --replicas=1

A restored database serves immediately. The built-in themes are reconciled at start-up, and a built-in theme an operator had edited is recognised by its version and left alone.

Scale to zero before replacing the file. Copying over a database that another process has open produces exactly the torn copy described above, and the running service will keep writing to the file you just replaced.

Retention

The audit log is the only unbounded table. Everything else is bounded by the number of themes and their versions. Prune it on a schedule:

DELETE FROM audit_log WHERE at < datetime('now', '-180 days');

The chart's snapshot CronJob (persistence.snapshot.enabled) writes to <mountPath>/snapshots on the same volume, so a snapshot survives the pod that took it. Restore from there with the same backup machinery in reverse.

---

Behind an ingress

Path-based

config:
  publicBaseUrl: https://example.com/themes
  basePath: /themes

Every route, including the OIDC redirect, is prefixed. The provider must be registered with https://example.com/themes/auth/callback.

Correct client addresses

X-Forwarded-For is believed only from a configured proxy. Without this, any client can set it and spoof its address in the audit log.

config:
  trustedProxies:
    - 10.42.0.0/16      # the ingress controller's node or pod CIDR

TLS

Terminate at the ingress. service.sessionSecure: true marks the session cookie Secure, and HSTS is advertised only when publicBaseUrl is https://.

Behind a proxy that terminates TLS and forwards X-Forwarded-Proto, the service derives scheme from publicBaseUrl, not from the header, which is the correct trust boundary: the operator states the truth in configuration rather than a client asserting it.

---

Monitoring

Metrics

Metric	Type	Alert when
bananas_http_requests_total	counter	Rate collapses, or 5xx ratio climbs
bananas_http_request_duration_seconds	histogram	p99 above ~250 ms
bananas_cache_hits_total / _misses_total	counter	Hit rate below ~80% after warm-up
bananas_go_memory_bytes	gauge	Approaches the limit
bananas_uptime_seconds	gauge	Resets: the process restarted

A sustained cache miss rate means a publish loop, a version-keyed cache storm, or more themes than expected. It is the first thing to look at when latency rises.

Logs

One JSON line per request, carrying requestId, route, status, duration, clientIp, subject and query. The subject field is the authenticated identity, so "who did this" needs no second query.

kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.status >= 500)'

Probes

Path	Use	Touches the database
/livez	Liveness	No
/healthz	Liveness, with build information	No
/readyz	Readiness	Yes

The split is deliberate. A liveness probe that touched the database would have the kubelet restart a process that is otherwise healthy, turning a dependency blip into an outage. Readiness does check the database, because a pod that cannot reach it cannot serve.

---

Upgrading

helm upgrade go-bananas oci://ghcr.io/urpi/charts/go-bananas \
  --namespace go-bananas \
  --set image.tag=1.4.2 \
  --wait --timeout 5m

Migrations apply at start-up and are idempotent. The built-in themes are reconciled: missing ones are created, and ones an operator has edited are recognised by version and left alone.

Enable persistence.snapshot.enabled for a scheduled snapshot. It is a CronJob that runs before each upgrade, and it costs a few megabytes.

Rolling back

helm rollback go-bananas 3 --namespace go-bananas --wait

Safe, because the schema is additive and migrations are idempotent. If a future change is destructive, it gets a snapshot step in the chart and an entry in the runbook.

---

Troubleshooting

A login loop. The redirect URI does not match what the provider has registered. Check in this order:

1. {publicBaseUrl}/auth/callback in the provider's client configuration.
2. publicBaseUrl has no trailing slash and matches the scheme exactly.
3. basePath is reflected in the registered URI.
4. identity.oidc.issuer matches the discovery document's issuer field exactly.

kubectl exec -n go-bananas deploy/go-bananas -- \
  wget -qO- "$ISSUER/.well-known/openid-configuration" | jq -r .issuer

invalid issuer. The iss claim in the ID token differs from the configured issuer. Usually a trailing slash, or a realm path that is not the one configured.

no such user in the roles mapping. Not an error this service produces: a user in no configured group gets viewer, silently. Check the groups claim is present — groups must be in OIDC_SCOPES.

A pod stuck in CrashLoopBackOff. Almost always configuration. The first log line is the reason; every configuration problem is reported in full at start-up.

SQLITE_BUSY. A writer is waiting for the lock. Raise config.database.busyTimeoutMs, or check for a publish loop.

Replicas crash-looping with a ReadWriteOnce volume. Exactly one pod can mount it. Set deployment.replicaCount: 1, or use a filesystem that permits concurrent mounts.

A theme looks wrong in one application. Compare X-Theme-Version across applications. A CDN or browser holding an old version explains it; the runtime manifest's baseUrl will show whether a CDN is in the path.

The stylesheet is stale after a publish. Expected within one s-maxage window (300 s by default). To force it, lower config.runtimeCacheControl or purge the CDN.