chore: adding docs
This commit is contained in:
parent
23816d00f7
commit
81c5112409
39 changed files with 7257 additions and 2 deletions
182
data/docs/go-bananas-0001-embedded-sqlite.md
Normal file
182
data/docs/go-bananas-0001-embedded-sqlite.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
<!-- go-doc:generated:begin source=go-bananas:docs/adr/0001-embedded-sqlite.md signature=7d8b696ba971c1cbd2e19854d26de304 generated=2026-10-05 18:14 UTC -->
|
||||
Imported from docs/adr/0001-embedded-sqlite.md in the go-bananas repository.
|
||||
|
||||
---
|
||||
|
||||
# ADR 0001: Embedded SQLite rather than a client/server database
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-10-04
|
||||
- **Deciders:** platform team
|
||||
- **Applies to:** all deployments of the current major version
|
||||
|
||||
## Context
|
||||
|
||||
go-bananas stores themes. A theme is a slug, some metadata and a JSON document of
|
||||
design tokens, typically 10–60 kB. A useful instance has tens to a few thousand of
|
||||
them.
|
||||
|
||||
The workload shape is:
|
||||
|
||||
- **Reads:** constant and bursty. Every page view in every consuming application
|
||||
requests CSS. A service with ten consumer applications and a few hundred users
|
||||
can easily see thousands of stylesheet requests a minute, and 95%+ of them are
|
||||
the same handful of bytes served from a render cache.
|
||||
- **Writes:** rare. A theme is edited by a human. A busy week might be fifty writes.
|
||||
- **Total size:** a few megabytes. Even ten thousand large themes would fit in
|
||||
fifty.
|
||||
|
||||
The conventional answer for a service with this shape is PostgreSQL. That answer is
|
||||
right for almost every service and wrong for this one, for reasons worth writing
|
||||
down rather than rediscovering in a year.
|
||||
|
||||
## Decision
|
||||
|
||||
Use SQLite through `modernc.org/sqlite`, a pure-Go translation, on a local volume.
|
||||
|
||||
## Rationale
|
||||
|
||||
### What a client/server database would add
|
||||
|
||||
For this dataset, Postgres buys nothing and costs:
|
||||
|
||||
- **A network hop per query.** A local SQLite read is microseconds; a query over a
|
||||
pod network is a millisecond and a tail-latency distribution with a long right
|
||||
tail. For a stylesheet endpoint on the critical path of every page view, that
|
||||
tail is the difference between a fast site and a janky one.
|
||||
- **A connection pool to get wrong.** Every deployment needs its own tuning, and the
|
||||
failure mode of under-tuning is a saturated pool under load.
|
||||
- **Credentials to create, distribute, rotate and revoke.** Another secret in
|
||||
Kubernetes, another thing to get wrong.
|
||||
- **A schema migration process.** For a dataset that fits in a few megabytes, this
|
||||
is a disproportionate amount of ceremony.
|
||||
- **A second thing to operate.** Backups, upgrades, monitoring, incident response,
|
||||
for a component whose whole job is to hold some JSON.
|
||||
- **A second failure mode.** The database becoming unavailable is now the most
|
||||
likely way the theme service goes down.
|
||||
|
||||
### What SQLite gives up
|
||||
|
||||
Honesty requires stating the costs:
|
||||
|
||||
- **One writer at a time.** Correct at this write rate. A theme edit is a
|
||||
millisecond-scale transaction; even a hundred concurrent editors would not
|
||||
saturate it.
|
||||
- **Single-node.** This is the real constraint. See Scaling below.
|
||||
- **No network access.** A consuming application cannot query the database directly.
|
||||
It does not need to; the runtime API is the interface.
|
||||
|
||||
### Why this particular configuration is fast
|
||||
|
||||
The default SQLite configuration is not fast. These settings are what make it fast
|
||||
enough that the question does not arise:
|
||||
|
||||
| Setting | Effect |
|
||||
| --- | --- |
|
||||
| `journal_mode=WAL` | Readers never block the writer and the writer never blocks readers. This is the single most important setting. |
|
||||
| `synchronous=NORMAL` | In WAL mode this is durable across an application crash and risks only the last transactions on an operating-system crash. A lost publish is re-doable; an outage is not. |
|
||||
| `cache_size=-16000` | A 16 MiB page cache, which holds the entire dataset. |
|
||||
| `mmap_size=268435456` | The file is read through the page cache directly, removing a copy from the read path. |
|
||||
| `temp_store=MEMORY` | Sort and group-by spill stays in memory. |
|
||||
| Max 8 connections | SQLite serialises writers. A large pool adds contention, not throughput. This is counter-intuitive and correct. |
|
||||
| In-memory render cache | The steady-state runtime request performs **no database access at all**. |
|
||||
|
||||
The render cache is the decisive optimisation. Rendering a theme traverses its whole
|
||||
token document; serving the cached string is one map lookup. Because the cache key
|
||||
includes the theme version, publishing invalidates exactly one theme's entries and
|
||||
disturbs nothing else. The database is consulted only on a cache miss, which is a
|
||||
publish rather than a read.
|
||||
|
||||
### Why `modernc.org/sqlite` rather than `mattn/go-sqlite3`
|
||||
|
||||
`mattn/go-sqlite3` is a cgo binding to the C library. It is faster, and it means:
|
||||
|
||||
- `CGO_ENABLED=1`, so the release image needs a C toolchain at build time and libc
|
||||
at run time, and cross-compilation needs a cross-C toolchain;
|
||||
- a statically linked binary is impossible, so the image carries a C library whose
|
||||
CVEs need tracking;
|
||||
- `go install` from source fails on a machine without a compiler.
|
||||
|
||||
`modernc.org/sqlite` is a transpiled Go implementation: no cgo, a fully static
|
||||
binary, and cross-compilation is `GOOS=linux GOARCH=arm64` and nothing else. The
|
||||
performance difference is irrelevant here because the hot path is the in-memory
|
||||
render cache, not SQLite.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**PostgreSQL.** The conventional choice. Rejected on the grounds above. It remains
|
||||
the right answer if the product changes shape — see Scaling.
|
||||
|
||||
**Redis.** Excellent for this read-heavy workload, and a good fit for a render
|
||||
cache. But it is another service to operate for a cache that is already
|
||||
process-local and loses nothing on restart, because the source of truth is the
|
||||
database. Rejected as unnecessary operational surface. Worth revisiting if a
|
||||
many-node deployment needs a shared cache; the render cache is an interface, so
|
||||
that change is contained.
|
||||
|
||||
**bbolt.** Faster than SQLite for pure key/value workloads and simpler. Rejected
|
||||
because the schema needs indexes for listing, filtering and sorting, and because
|
||||
SQLite's query planner handles the "a few hundred rows filtered by status and
|
||||
ordered by date" shape without the ceremony of managing bbolt buckets by hand.
|
||||
|
||||
**A JSON file per theme on disk.** No dependency at all. Rejected because concurrent
|
||||
writes need locking, and because every listing requires reading and parsing every
|
||||
file. SQLite is a better version of this idea that is already written.
|
||||
|
||||
**An in-memory store with write-through to object storage.** Attractive for scale.
|
||||
Rejected for v1: it removes the transactional guarantee that makes version history
|
||||
and rollback correct, and it adds an object-storage dependency. The write path
|
||||
here is not hot enough to justify losing the invariant.
|
||||
|
||||
## Scaling
|
||||
|
||||
SQLite on a local volume is single-node. Two shapes are supported:
|
||||
|
||||
1. **One replica, one volume.** The recommended production shape. Horizontally
|
||||
scalable only in the sense that the service is so cheap that one replica serves
|
||||
a very large number of consumers.
|
||||
|
||||
2. **Many replicas, a shared filesystem.** Works when the storage class supports
|
||||
concurrent mounts — NFS, CephFS, or a ReadWriteMany block device. WAL mode
|
||||
requires the filesystem to support shared memory mapping, which these do. The
|
||||
Helm chart warns when a ReadWriteOnce volume is combined with more than one
|
||||
replica, because that combination cannot work.
|
||||
|
||||
If the workload ever outgrows both — genuinely concurrent multi-writer access across
|
||||
nodes — the migration is to Postgres. It is contained: `internal/store` is the only
|
||||
package that writes SQL, and its interface is defined by the methods it exposes
|
||||
rather than by SQLite's features.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good**
|
||||
|
||||
- One less thing to operate, back up, monitor and secure.
|
||||
- No credential to leak.
|
||||
- A static binary and a small image with no libc.
|
||||
- A backup is `cp` of one file. There is no consistent-snapshot dance.
|
||||
- The render cache can be process-local because there is only one process.
|
||||
|
||||
**Bad**
|
||||
|
||||
- Cannot scale horizontally past one node without a shared filesystem.
|
||||
- A shared-filesystem deployment is slower than local disk, and WAL over NFS is
|
||||
sensitive to latency. Measure before choosing it.
|
||||
- Single-writer means a pathological write loop would queue. Nothing in the
|
||||
application can produce one.
|
||||
|
||||
**Neutral**
|
||||
|
||||
- The `store` package has a wider interface than a repository might otherwise need,
|
||||
because it must express listing, filtering and sorting. Moving to another engine
|
||||
means reimplementing that surface.
|
||||
|
||||
## Revisit when
|
||||
|
||||
- A single instance cannot serve the load, and profiling shows the database rather
|
||||
than the render cache is the bottleneck.
|
||||
- Concurrent writers become a real requirement, for example if themes can be edited
|
||||
by an automated pipeline from several deployments.
|
||||
- The dataset outgrows a few hundred megabytes, at which point the time to back up
|
||||
by copying a file becomes its own problem.
|
||||
<!-- go-doc:generated:end -->
|
||||
Loading…
Add table
Add a link
Reference in a new issue