dnoulet/go-doc

Fork 0

Daniel Noulet 81c5112409 chore: adding docs

2026-10-07 15:05:48 +02:00

20 KiB

Raw Blame History

Imported from docs/deployment.md in the go-bananas repository.

Deployment

Running go-bananas, from a laptop to a production cluster.

The deployment checklist
Configuration
Authentication
Helm
The production checklist
Scaling
Backup and restore
Behind an ingress
Monitoring
Upgrading
Troubleshooting

The deployment checklist

Four things must be right. Everything else has a working default.

config.publicBaseUrl — the externally reachable origin, no trailing slash. It builds the OIDC redirect URI, the absolute URLs in the runtime manifest and the HSTS decision. Wrong means a login loop.
identity.mode — oidc in production. dev and disabled are refused by the application outside a development environment.
The OIDC client is registered with that exact redirect URI: {publicBaseUrl}/auth/callback.
identity.oidc.existingSecret — a Secret you manage, holding a session secret of at least 32 bytes and the provider's client secret.

The Helm chart refuses to render a manifest it knows is wrong, so items 1–2 fail at helm template time rather than at run time.

Configuration

Everything is an environment variable. Both BANANAS_ and GO_BANANAS_ prefixes are accepted, so a ConfigMap can be mounted under a distinct prefix without colliding with the pod's own environment. That is what makes one image usable across environments.

Server

Variable	Default	Notes
`BANANAS_ENVIRONMENT`	`development`	`production` disables every development affordance
`BANANAS_HTTP_ADDR`	`:8080`
`BANANAS_PUBLIC_BASE_URL`	`http://localhost:8080`	No trailing slash. Required in practice
`BANANAS_BASE_PATH`	—	For a path-based ingress
`BANANAS_LOG_LEVEL`	`info`	`debug`, `info`, `warn`, `error`
`BANANAS_LOG_FORMAT`	`text`	`json` in production
`BANANAS_HTTP_READ_TIMEOUT`	`15s`
`BANANAS_HTTP_WRITE_TIMEOUT`	`30s`	Generous: this is the hot asset path
`BANANAS_SHUTDOWN_TIMEOUT`	`25s`	Must be below the pod's grace period

Storage

Variable	Default	Notes
`BANANAS_DATABASE_PATH`	`./data/go-bananas.db`
`BANANAS_DB_MAX_CONNS`	`8`	Small on purpose. SQLite serialises writers
`BANANAS_DB_BUSY_TIMEOUT`	`5s`	How long a blocked writer waits
`BANANAS_SEED_ON_START`	`true`	Reconciles the built-in themes

Runtime

Variable	Default	Notes
`BANANAS_RUNTIME_PUBLIC`	`true`	The runtime API requires no credential by design
`BANANAS_RUNTIME_CACHE_CONTROL`	`public, max-age=60, s-maxage=300, stale-while-revalidate=86400`
`BANANAS_RUNTIME_CDN_BASE_URL`	—	Absolute base for the manifest, when assets come from a CDN
`BANANAS_RATE_LIMIT_PER_MINUTE`	`3000`	High: the pattern is one request per page view

Network

Variable	Default	Notes
`BANANAS_TRUSTED_PROXIES`	—	CIDRs whose `X-Forwarded-For` is believed. Empty means forwarding headers are ignored entirely
`BANANAS_CORS_ALLOWED_ORIGINS`	—	For the admin API only. The runtime API is public by design
`BANANAS_ENABLE_METRICS`	`true`
`BANANAS_ENABLE_PROFILER`	`false`	A network-reachable profiler is a DoS surface. Pair with a NetworkPolicy

Authentication

Variable	Default	Notes
`BANANAS_AUTH_MODE`	`dev`	`oidc`, `dev`, `none`. The latter two are refused outside development
`BANANAS_SESSION_SECRET`	ephemeral in dev	At least 32 bytes. Rotating it logs out everyone
`BANANAS_SESSION_TTL`	`8h`
`BANANAS_SESSION_SECURE`	`true` outside dev
`BANANAS_SESSION_SAMESITE`	`Lax`	`None` requires `Secure`
`BANANAS_OIDC_ISSUER`	—	Must match the `iss` claim exactly
`BANANAS_OIDC_CLIENT_ID`	—
`BANANAS_OIDC_CLIENT_SECRET`	—	From a Secret, never a ConfigMap
`BANANAS_OIDC_SCOPES`	`profile,email,groups`	`groups` is what the role mapping reads
`BANANAS_OIDC_ADMIN_GROUPS`	—	Comma separated
`BANANAS_OIDC_EDITOR_GROUPS`	—
`BANANAS_OIDC_VIEWER_GROUPS`	—
`BANANAS_OIDC_REDIRECT_URL`	derived	Override only if the provider has a different URI registered
`BANANAS_OIDC_SKIP_TLS_VERIFY`	`false`	Development only. Refused in production

Full reference in internal/config/config.go, which is the single source of truth and carries the reasoning for each default.

Authentication

The three modes

Mode	Behaviour	Allowed in production
`oidc`	OpenID Connect against a real provider	Yes. This is the mode.
`dev`	Every request is a local administrator	No. Refused by the application
`none`	Every request is granted administrator access	No. Refused by the application

The refusal is in the code, not in documentation, and it is a hard fail in the constructor — an accidentally deployed bypass is a total authentication bypass, so it is made impossible to construct.

The shared development realm

The chart defaults to https://auth-dev.urpi.be/realms/urpi, so a development cluster works with no identity configuration.

The realm path is part of the issuer. The value must match the iss claim in the ID token exactly, including a trailing slash if the provider emits one. A mismatch here is the single most common cause of invalid issuer at the callback, and it is worth checking first when a login loop appears.

Registering the client

Redirect URIs, exact:

https://themes.example.com/auth/callback
https://bananas-dev.example.com/auth/callback

Post-logout redirect, if the provider requires one:

https://themes.example.com/

Web origins, for a cross-origin admin UI (same-origin needs none):

https://app.example.com

Group-to-role mapping

identity:
  oidc:
    adminGroups: [platform-admins]
    editorGroups: [design-team]
    viewerGroups: [everyone]

The most privileged matching group wins. A user in no configured group gets viewer. That is deliberate: a mapping mistake must fail closed.

Group names are realm-specific and are deliberately not guessed by the chart. Read them from the provider:

curl -s "$ISSUER/protocol/openid-connect/certs" | jq -r '.keys[].kid'
# or, with a token:
curl -s -H "Authorization: Bearer $TOKEN" "$ISSUER/admin/realms/urpi/groups" | jq '.[].name'

A note on sessions

The session is a signed cookie with no server-side record (ADR 0003). Concretely:

any replica authenticates any request, so a rollout logs nobody out;
a stolen session cannot be revoked before it expires — bounded by BANANAS_SESSION_TTL, eight hours by default;
full invalidation means rotating the session secret, which logs out everyone.

For a design-token service this is the right trade. If you need per-session revocation, or your organisation's policy requires it, this is the one component to replace.

Helm

helm upgrade --install go-bananas oci://ghcr.io/urpi/charts/go-bananas \
  --namespace go-bananas --create-namespace \
  --set config.publicBaseUrl=https://themes.example.com \
  --set identity.oidc.issuer=https://auth.example.com/realms/urpi \
  --set identity.oidc.clientId=go-bananas \
  --set identity.oidc.existingSecret=go-bananas-auth \
  --set identity.oidc.adminGroups[0]=platform-admins \
  --set ingress.enabled=true \
  --set ingress.hosts[0].host=themes.example.com \
  --set ingress.tls[0].secretName=go-bananas-tls \
  --set ingress.tls[0].hosts[0]=themes.example.com \
  --wait --timeout 5m

The Secret

kubectl create secret generic go-bananas-auth \
  --namespace go-bananas \
  --from-literal=session-secret="$(openssl rand -base64 48)" \
  --from-literal=oidc-client-secret="$CLIENT_SECRET_FROM_YOUR_SECRET_MANAGER"

Key names are configurable through identity.sessionSecretKey, identity.oidc.clientSecretKey and identity.oidc.existingSecretKeys, for a Secret managed by an external secrets operator.

A chart-managed Secret (identity.oidc.generateSecret=true) generates the session key and preserves it across upgrades by reading the live Secret back — so an upgrade does not sign everyone out. It cannot generate the OIDC client secret, because that has to come from the provider. A generated Secret therefore has an empty client secret, which means the deployment fails to authenticate until one is supplied. For anything real, use existingSecret.

Validation at render time

The chart fails rather than producing a manifest it knows is wrong:

Misconfiguration	Message
`publicBaseUrl` empty	Required; names what it is used for
`publicBaseUrl` has a trailing slash	The exact acceptable form
`shutdownTimeoutMs` ≥ grace period	Kubernetes would SIGKILL mid-shutdown
`identity.mode` unknown	The three valid modes
`identity.mode=oidc`, no issuer	Required, with the development-realm form
`identity.mode=oidc`, no secret	Names both ways to supply one
Both `existingSecret` and `generateSecret`	Mutually exclusive
`skipTLSVerify` with `environment: production`	Refused, with the reason
Any value outside the schema	The JSON-schema path

Failing at render time means a bad release never reaches the API server: there is no half-created Deployment and no CrashLoopBackOff to diagnose at 3am.

Render it before you apply it

make helm-template          # defaults
make helm-template-auth     # configured for OIDC
helm lint deploy/helm/go-bananas --strict

The production checklist

config.environment: production
config.publicBaseUrl is the real HTTPS origin, no trailing slash
identity.mode: oidc
The OIDC client registered with {publicBaseUrl}/auth/callback
identity.oidc.issuer verified against the provider's discovery document
Group-to-role mapping populated and tested with a real account in each role
identity.oidc.existingSecret set; the session secret is at least 32 random bytes
Ingress with TLS; service.sessionSecure: true
config.trustedProxies set to the ingress controller, so the audit log records real client addresses
Persistence enabled; a snapshot scheduled
networkPolicy.enabled: true
Resources set, with a memory limit
Prometheus scraping /metrics
A backup job that copies the SQLite file
make helm-template reviewed

Verify a deployment

kubectl --namespace go-bananas get pods -l app.kubernetes.io/instance=go-bananas
kubectl --namespace go-bananas port-forward svc/go-bananas 8080:80

curl -s localhost:8080/readyz | jq            # "status": "ready"
curl -s localhost:8080/rt/v1/manifest.json | jq '.themes[].slug'
curl -sI 'localhost:8080/rt/v1/theme.css?slug=midnight' | grep -i x-theme-version

Then open the admin interface and confirm a real account can sign in and see a theme.

Scaling

Read scaling: put a CDN in front

The runtime is the hot path and it is entirely cacheable. Immutable-by-URL responses mean a CDN is a complete answer for read scaling.

config:
  runtimeCdnBaseUrl: https://cdn.example.com/themes

The manifest then hands consumers the CDN URL, so they never reach the service at all.

The one real constraint: one writer

SQLite on a local volume is single-node. Three shapes:

One replica, one volume. The recommended production shape. One replica serves a very large number of consumers, because after the render cache a request is a map lookup.

Many replicas, a shared filesystem. Works with NFS, CephFS or a ReadWriteMany volume — anything that permits concurrent mounts and shared mmap, which WAL mode requires. Expect lower throughput than local disk; measure before choosing it.

persistence:
  type: persistentVolumeClaim
  accessMode: ReadWriteMany
  size: 2Gi
  storageClassName: nfs-client
deployment:
  replicaCount: 3

The chart annotates the Deployment when a ReadWriteOnce volume is combined with more than one replica, because that combination cannot work: only one pod mounts the volume and the others crash-loop.

If you outgrow SQLite

Move to PostgreSQL. The change is contained to internal/store — the only package with SQL — and its interface is the set of methods it exposes rather than SQLite's features. ADR 0001 covers when that point arrives.

Backup and restore

A backup is a copy of one file, and the supported way to take it is the backup subcommand:

kubectl exec -n go-bananas deploy/go-bananas -- \
  /usr/local/bin/go-bananas backup --to /srv/data/snapshots/$(date -u +%Y%m%dT%H%M%SZ).db

It uses SQLite's VACUUM INTO, which reads through the database's own backup machinery. Two consequences matter:

It is safe against a running service. The snapshot is consistent even while the application is serving writes, and it does not block them for the length of the copy. There is no need to scale to zero.
Do not substitute cp for it. Copying a SQLite file while a write is in flight produces a torn copy: the header can describe one set of pages while the file holds another. The result frequently opens without complaint and is corrupt. Copying the -wal file alongside it is worse, because the two files are then captured at different instants and the pair may not correspond to any point in time.

The command refuses to overwrite an existing file, so a stale snapshot can never be mistaken for a fresh one, and it will not snapshot an in-memory database, which has nothing to preserve.

Run it from cron, a Kubernetes CronJob, or by hand. When you do it by hand, the output path is printed in full so you can confirm where the file went.

Restoring

kubectl scale deploy/go-bananas -n go-bananas --replicas=0
kubectl cp ./backup.db go-bananas-0:/srv/data/go-bananas.db
kubectl delete pod -n go-bananas -l app.kubernetes.io/instance=go-bananas
kubectl scale deploy/go-bananas -n go-bananas --replicas=1

A restored database serves immediately. The built-in themes are reconciled at start-up, and a built-in theme an operator had edited is recognised by its version and left alone.

Scale to zero before replacing the file. Copying over a database that another process has open produces exactly the torn copy described above, and the running service will keep writing to the file you just replaced.

Retention

The audit log is the only unbounded table. Everything else is bounded by the number of themes and their versions. Prune it on a schedule:

DELETE FROM audit_log WHERE at < datetime('now', '-180 days');

The chart's snapshot CronJob (persistence.snapshot.enabled) writes to <mountPath>/snapshots on the same volume, so a snapshot survives the pod that took it. Restore from there with the same backup machinery in reverse.

Behind an ingress

Path-based

config:
  publicBaseUrl: https://example.com/themes
  basePath: /themes

Every route, including the OIDC redirect, is prefixed. The provider must be registered with https://example.com/themes/auth/callback.

Correct client addresses

X-Forwarded-For is believed only from a configured proxy. Without this, any client can set it and spoof its address in the audit log.

config:
  trustedProxies:
    - 10.42.0.0/16      # the ingress controller's node or pod CIDR

TLS

Terminate at the ingress. service.sessionSecure: true marks the session cookie Secure, and HSTS is advertised only when publicBaseUrl is https://.

Behind a proxy that terminates TLS and forwards X-Forwarded-Proto, the service derives scheme from publicBaseUrl, not from the header, which is the correct trust boundary: the operator states the truth in configuration rather than a client asserting it.

Monitoring

Metrics

Metric	Type	Alert when
`bananas_http_requests_total`	counter	Rate collapses, or 5xx ratio climbs
`bananas_http_request_duration_seconds`	histogram	p99 above ~250 ms
`bananas_cache_hits_total` / `_misses_total`	counter	Hit rate below ~80% after warm-up
`bananas_go_memory_bytes`	gauge	Approaches the limit
`bananas_uptime_seconds`	gauge	Resets: the process restarted

A sustained cache miss rate means a publish loop, a version-keyed cache storm, or more themes than expected. It is the first thing to look at when latency rises.

Logs

One JSON line per request, carrying requestId, route, status, duration, clientIp, subject and query. The subject field is the authenticated identity, so "who did this" needs no second query.

kubectl logs -n go-bananas -l app.kubernetes.io/instance=go-bananas \
  | jq 'select(.status >= 500)'

Probes

Path	Use	Touches the database
`/livez`	Liveness	No
`/healthz`	Liveness, with build information	No
`/readyz`	Readiness	Yes

The split is deliberate. A liveness probe that touched the database would have the kubelet restart a process that is otherwise healthy, turning a dependency blip into an outage. Readiness does check the database, because a pod that cannot reach it cannot serve.

Upgrading

helm upgrade go-bananas oci://ghcr.io/urpi/charts/go-bananas \
  --namespace go-bananas \
  --set image.tag=1.4.2 \
  --wait --timeout 5m

Migrations apply at start-up and are idempotent. The built-in themes are reconciled: missing ones are created, and ones an operator has edited are recognised by version and left alone.

Enable persistence.snapshot.enabled for a scheduled snapshot. It is a CronJob that runs before each upgrade, and it costs a few megabytes.

Rolling back

helm rollback go-bananas 3 --namespace go-bananas --wait

Safe, because the schema is additive and migrations are idempotent. If a future change is destructive, it gets a snapshot step in the chart and an entry in the runbook.

Troubleshooting

A login loop. The redirect URI does not match what the provider has registered. Check in this order:

{publicBaseUrl}/auth/callback in the provider's client configuration.
publicBaseUrl has no trailing slash and matches the scheme exactly.
basePath is reflected in the registered URI.
identity.oidc.issuer matches the discovery document's issuer field exactly.

kubectl exec -n go-bananas deploy/go-bananas -- \
  wget -qO- "$ISSUER/.well-known/openid-configuration" | jq -r .issuer

invalid issuer. The iss claim in the ID token differs from the configured issuer. Usually a trailing slash, or a realm path that is not the one configured.

no such user in the roles mapping. Not an error this service produces: a user in no configured group gets viewer, silently. Check the groups claim is present — groups must be in OIDC_SCOPES.

A pod stuck in CrashLoopBackOff. Almost always configuration. The first log line is the reason; every configuration problem is reported in full at start-up.

SQLITE_BUSY. A writer is waiting for the lock. Raise config.database.busyTimeoutMs, or check for a publish loop.

Replicas crash-looping with a ReadWriteOnce volume. Exactly one pod can mount it. Set deployment.replicaCount: 1, or use a filesystem that permits concurrent mounts.

A theme looks wrong in one application. Compare X-Theme-Version across applications. A CDN or browser holding an old version explains it; the runtime manifest's baseUrl will show whether a CDN is in the path.

The stylesheet is stale after a publish. Expected within one s-maxage window (300 s by default). To force it, lower config.runtimeCacheControl or purge the CDN.

20 KiB Raw Blame History Unescape Escape

Deployment

Contents

The deployment checklist

Configuration

Server

Storage

Runtime

Network

Authentication

Authentication

The three modes

The shared development realm

Registering the client

Group-to-role mapping

A note on sessions

Helm

The Secret

Validation at render time

Render it before you apply it

The production checklist

Verify a deployment

Scaling

Read scaling: put a CDN in front

The one real constraint: one writer

If you outgrow SQLite

Backup and restore

Restoring

Retention

Behind an ingress

Path-based

Correct client addresses

TLS

Monitoring

Metrics

Logs

Probes

Upgrading

Rolling back

Troubleshooting

20 KiB

Raw Blame History