go-store/docs/history.md

Project History — go-store

Origin

Extracted from forge.lthn.ai/core/go (pkg/store/) on 19 February 2026 by Virgil. The extraction gave the package its own module path (forge.lthn.ai/core/go-store), its own repository, and independent versioning.

At extraction the package comprised a single source file and a single test file. It provided basic CRUD with group namespacing and template rendering but had no TTL, no namespace isolation, and no event system. Test coverage was 73.1%.

---

Phase 0 — Hardening and Test Coverage

Agent: Charon Completed: 2026-02-20

Concurrency fix: SQLITE_BUSY under contention

Problem. The database/sql connection pool hands out different physical connections for each Exec or Query call. SQLite pragmas (PRAGMA journal_mode=WAL, PRAGMA busy_timeout) are per-connection. Under concurrent write load (10 goroutines, 100 ops each), connections from the pool that had not received the WAL pragma would block and return SQLITE_BUSY immediately rather than waiting.

Fix. db.SetMaxOpenConns(1) serialises all database access through a single connection. Because SQLite is a single-writer database by design (it serialises writes at the file-lock level regardless of pool size), this does not reduce write throughput. It eliminates the BUSY errors by ensuring the pragma settings always apply.

Defence in depth. PRAGMA busy_timeout=5000 was added to make the single connection wait up to 5 seconds before reporting a timeout error, providing additional resilience.

Extended test coverage

Added tests for:

• Concurrent read/write with 10 goroutines and the race detector
• Render() with invalid template syntax (parse error)
• Render() with template execution error (calling a string as a function)
• Get() on a non-existent group vs. a non-existent key
• DeleteGroup() followed by GetAll() returning empty
• Count() after 500 bulk inserts
• In-memory vs. file-backed store (persistence across open/close)
• WAL mode verification via PRAGMA journal_mode query
• Edge cases: empty key, empty value, empty group, 10K-character key, binary-like values with null bytes, Unicode (accented, CJK, Arabic), SQL injection attempts, special characters

Coverage: 73.1% to 90.9%.

The remaining 9.1% comprised defensive error paths in New(), GetAll(), and Render() that are unreachable through integration tests against a healthy SQLite database (driver initialisation failures, scan errors on NULL columns, rows iteration errors on corrupted pages).

Benchmarks

Established baseline benchmark results:

BenchmarkSet-32               119280    10290 ns/op
BenchmarkGet-32               335707     3589 ns/op
BenchmarkGetAll-32 (10K keys)    258  4741451 ns/op
BenchmarkSet_FileBacked-32      4525   265868 ns/op

go vet ./... made clean. No warnings.

---

Phase 1 — TTL Support

Agent: Charon Completed: 2026-02-20

Added optional time-to-live for keys.

Changes

• expires_at INTEGER nullable column added to the kv schema.
• SetWithTTL(group, key, value string, ttl time.Duration) stores the current time plus TTL as a Unix millisecond timestamp in expires_at.
• Get() performs lazy deletion: if a key is found with an expires_at in the past, it is deleted and ErrNotFound is returned.
• Count(), GetAll(), and Render() include (expires_at IS NULL OR expires_at > ?) in all queries, excluding expired keys from results.
• PurgeExpired() public method deletes all physically stored expired rows and returns the count removed.
• Background goroutine calls PurgeExpired() every 60 seconds, controlled by a context.WithCancel that is cancelled on Close().
• Set() clears any existing TTL when overwriting a key (sets expires_at = NULL).
• Schema migration: ALTER TABLE kv ADD COLUMN expires_at INTEGER runs on New(). The "duplicate column" error on already-upgraded databases is silently ignored.

Tests added

TTL functionality tests covering: normal expiry on Get, exclusion from Count/GetAll/Render, SetWithTTL upsert, plain Set clearing TTL, future TTL remaining accessible, background purge goroutine, concurrent TTL access with 10 goroutines, schema migration from a pre-TTL database (manually constructed without expires_at).

Coverage: 90.9% to 94.7%.

---

Phase 2 — Namespace Isolation

Agent: Charon Completed: 2026-02-20

Added ScopedStore for multi-tenant namespace isolation.

Changes

• scope.go introduced with ScopedStore wrapping *Store.
• Namespace strings validated against ^[a-zA-Z0-9-]+$.
• NewScoped(store, namespace) constructor.
• All Store methods delegated with group automatically prefixed as namespace + ":" + group.
• QuotaConfig{MaxKeys, MaxGroups int} struct; zero means unlimited.
• NewScopedWithQuota(store, namespace, quota) constructor.
• ErrQuotaExceeded sentinel error.
• checkQuota(group, key) internal method: skips upserts (existing key), checks CountAll(namespace+":") against MaxKeys, checks Groups(namespace+":") against MaxGroups only when the group is new.
• CountAll(prefix string) added to Store: counts non-expired keys across all groups matching a prefix. Empty prefix counts across all groups.
• Groups(prefix string) added to Store: returns distinct non-expired group names matching a prefix. Empty prefix returns all groups.

Tests added

ScopedStore basic CRUD, cross-namespace isolation, GetAll scoping. Quota tests: MaxKeys limit, upsert does not count, delete and re-insert stays within quota, MaxGroups limit, zero quota is unlimited, TTL-expired keys do not count towards quota. CountAll and Groups tests with mixed namespaces and expired key exclusion.

Coverage: 94.7% to 95.5% (approximate; coverage_test.go added to cover defensive paths).

---

Phase 3 — Event Hooks

Agent: Charon Completed: 2026-02-20

Added a reactive notification system for store mutations.

Changes

• events.go introduced with EventType (EventSet, EventDelete, EventDeleteGroup), Event struct, Watcher struct, callbackEntry struct.
• watcherBufSize = 16 constant.
• Watch(group, key string) *Watcher: creates a buffered channel watcher. Wildcard "*" supported for both group and key. Uses atomic.AddUint64 for monotonic watcher IDs.
• Unwatch(w *Watcher): removes watcher from the registry and closes its channel. Idempotent.
• OnChange(fn func(Event)) func(): registers a synchronous callback. Returns an idempotent unregister function using sync.Once.
• notify(e Event): internal dispatch. Acquires read-lock on s.mu; non-blocking send to each matching watcher channel (drop-on-full); calls each callback synchronously. Separate watcherMatches helper handles wildcard logic.
• Set(), SetWithTTL(), Delete(), DeleteGroup() each call notify() after the successful database write.
• Store struct extended with watchers []*Watcher, callbacks []callbackEntry, mu sync.RWMutex, nextID uint64.
• ScopedStore mutations automatically emit events with the full prefixed group name — no extra implementation required.

Tests added

Specific-key watcher receives matching events and ignores non-matching keys. Wildcard-key watcher receives all keys in a group. Global wildcard ("*", "*") receives all mutations across all groups. Unwatch stops delivery and closes the channel. Unwatch is idempotent. Delete and DeleteGroup emit correct event types with correct populated fields. OnChange callback fires on Set and Delete. OnChange unregister stops future invocations (idempotent). Buffer-full (32 writes against cap-16 channel) does not block the writer. Multiple watchers on the same key receive events independently. Concurrent Watch/Unwatch during concurrent writes (race test, 10 goroutines). ScopedStore events carry the prefixed group name. SetWithTTL emits EventSet. EventType.String() returns correct labels including "unknown" for undefined values.

Coverage: 94.7% to 95.5%.

---

Coverage Test Suite

coverage_test.go exercises defensive error paths that integration tests cannot reach through normal usage:

• Schema conflict: pre-existing SQLite index named kv causes New() to return store.New: schema: ....
• GetAll scan error: NULL key in a row (requires manually altering the schema to remove the NOT NULL constraint).
• GetAll rows iteration error: physically corrupting database pages mid-file to trigger rows.Err() during multi-page scans.
• Render scan error: same NULL-key technique.
• Render rows iteration error: same corruption technique.

These tests exercise correct defensive code. They must continue to pass but are not indicative of real failure modes in production.

---

Known Limitations

Single writer. SetMaxOpenConns(1) serialises all access through one connection. Write throughput is bounded by SQLite's single-writer architecture. This is appropriate for the intended use cases (configuration storage, session state, per-tenant key-value data) but is not suitable for high-throughput append-only workloads.

File-backed write throughput. File-backed Set operations (~3,800 ops/sec on Apple M-series) are dominated by fsync. Applications writing at higher rates should use in-memory stores or consider WAL checkpoint tuning.

GetAll memory usage. Fetching a group with 10,000 keys allocates approximately 2.3 MB per call. There is no pagination API. Applications with very large groups should restructure data into smaller groups or query selectively.

No cross-group transactions. There is no API for atomic multi-group operations. Each method is individually atomic at the SQLite level, but there is no Begin/Commit exposed to callers.

No wildcard deletes. There is no DeletePrefix or pattern-based delete. To delete all groups under a namespace, callers must retrieve the group list via Groups() and delete each individually.

Callback deadlock risk. OnChange callbacks run synchronously in the writer's goroutine while holding s.mu (read). Calling any Store method that calls notify() from within a callback will attempt to re-acquire s.mu (read), which is permitted with a read-lock but calling Watch/Unwatch/OnChange within a callback will deadlock (they require a write-lock). Document this constraint prominently in callback usage.

No persistence of watcher registrations. Watchers and callbacks are in-memory only. They are not persisted across Close/New cycles.

---

Future Considerations

These are design notes, not committed work:

• Pagination for GetAll. A GetPage(group string, offset, limit int) method would support large groups without full in-memory materialisation.
• Indexed prefix keys. An additional index on (grp, key) prefix would accelerate prefix scans without a full-table scan.
• TTL background purge interval as constructor option. Currently only settable by mutating s.purgeInterval directly in tests. A WithPurgeInterval(d time.Duration) functional option would make this part of the public API.
• Cross-group atomic operations. Exposing a Transaction(func(tx *StoreTx) error) API would allow callers to compose atomic multi-group operations.
• DeletePrefix(prefix string) method. Would enable efficient cleanup of an entire namespace without first listing groups.