go-store/docs/history.md

Project History — go-store

Origin

Extracted from forge.lthn.ai/core/go (pkg/store/) on 19 February 2026 by Virgil. The extraction gave the package its own module path (forge.lthn.ai/core/go-store), its own repository, and independent versioning.

At extraction the package comprised a single source file and a single test file. It provided basic CRUD with group namespacing and template rendering but had no TTL, no namespace isolation, and no event system. Test coverage was 73.1%.

---

Phase 0 — Hardening and Test Coverage

Agent: Charon Completed: 2026-02-20

Concurrency fix: SQLITE_BUSY under contention

Problem. The database/sql connection pool hands out different physical connections for each Exec or Query call. SQLite pragmas (PRAGMA journal_mode=WAL, PRAGMA busy_timeout) are per-connection. Under concurrent write load (10 goroutines, 100 ops each), connections from the pool that had not received the WAL pragma would block and return SQLITE_BUSY immediately rather than waiting.

Fix. database.SetMaxOpenConns(1) serialises all database access through a single connection. Because SQLite is a single-writer database by design (it serialises writes at the file-lock level regardless of pool size), this does not reduce write throughput. It eliminates the BUSY errors by ensuring the pragma settings always apply.

Defence in depth. PRAGMA busy_timeout=5000 was added to make the single connection wait up to 5 seconds before reporting a timeout error, providing additional resilience.

Extended test coverage

Added tests for:

• Concurrent read/write with 10 goroutines and the race detector
• Render() with invalid template syntax (parse error)
• Render() with template execution error (calling a string as a function)
• Get() on a non-existent group vs. a non-existent key
• DeleteGroup() followed by GetAll() returning empty
• Count() after 500 bulk inserts
• In-memory vs. file-backed store (persistence across open/close)
• WAL mode verification via PRAGMA journal_mode query
• Edge cases: empty key, empty value, empty group, 10K-character key, binary-like values with null bytes, Unicode (accented, CJK, Arabic), SQL injection attempts, special characters

Coverage: 73.1% to 90.9%.

The remaining 9.1% comprised defensive error paths in New(), GetAll(), and Render() that are unreachable through integration tests against a healthy SQLite database (driver initialisation failures, scan errors on NULL columns, rows iteration errors on corrupted pages).

Benchmarks

Established baseline benchmark results:

BenchmarkSet-32               119280    10290 ns/op
BenchmarkGet-32               335707     3589 ns/op
BenchmarkGetAll-32 (10K keys)    258  4741451 ns/op
BenchmarkSet_FileBacked-32      4525   265868 ns/op

go vet ./... made clean. No warnings.

---

Phase 1 — TTL Support

Agent: Charon Completed: 2026-02-20

Added optional time-to-live for keys.

Changes

• expires_at INTEGER nullable column added to the key-value schema.
• SetWithTTL(group, key, value string, timeToLive time.Duration) stores the current time plus TTL as a Unix millisecond timestamp in expires_at.
• Get() performs lazy deletion: if a key is found with an expires_at in the past, it is deleted and NotFoundError is returned.
• Count(), GetAll(), and Render() include (expires_at IS NULL OR expires_at > ?) in all queries, excluding expired keys from results.
• PurgeExpired() public method deletes all physically stored expired rows and returns the count removed.
• Background goroutine calls PurgeExpired() every 60 seconds, controlled by a context.WithCancel that is cancelled on Close().
• Set() clears any existing TTL when overwriting a key (sets expires_at = NULL).
• Schema migration: ALTER TABLE entries ADD COLUMN expires_at INTEGER runs on New(). The "duplicate column" error on already-upgraded databases is silently ignored.

Tests added

TTL functionality tests covering: normal expiry on Get, exclusion from Count/GetAll/Render, SetWithTTL upsert, plain Set clearing TTL, future TTL remaining accessible, background purge goroutine, concurrent TTL access with 10 goroutines, schema migration from a pre-TTL database (manually constructed without expires_at).

Coverage: 90.9% to 94.7%.

---

Phase 2 — Namespace Isolation

Agent: Charon Completed: 2026-02-20

Added ScopedStore for multi-tenant namespace isolation.

Changes

• scope.go introduced with ScopedStore wrapping *Store.
• Namespace strings validated against ^[a-zA-Z0-9-]+$.
• NewScoped(store, namespace) constructor.
• All Store methods delegated with group automatically prefixed as namespace + ":" + group.
• QuotaConfig{MaxKeys, MaxGroups int} struct; zero means unlimited.
• NewScopedWithQuota(store, namespace, quota) constructor.
• QuotaExceededError sentinel error.
• checkQuota(group, key) internal method: skips upserts (existing key), checks CountAll(namespace+":") against MaxKeys, checks Groups(namespace+":") against MaxGroups only when the group is new.
• CountAll(prefix string) added to Store: counts non-expired keys across all groups matching a prefix. Empty prefix counts across all groups.
• Groups(prefix string) added to Store: returns distinct non-expired group names matching a prefix. Empty prefix returns all groups.

Tests added

ScopedStore basic CRUD, cross-namespace isolation, GetAll scoping. Quota tests: MaxKeys limit, upsert does not count, delete and re-insert stays within quota, MaxGroups limit, zero quota is unlimited, TTL-expired keys do not count towards quota. CountAll and Groups tests with mixed namespaces and expired key exclusion.

Coverage: 94.7% to 95.5% (approximate; coverage_test.go added to cover defensive paths).

---

Phase 3 — Event Hooks

Agent: Charon Completed: 2026-02-20

Added a reactive notification system for store mutations.

Changes

• events.go introduced with EventType (EventSet, EventDelete, EventDeleteGroup), Event struct, Watcher struct, changeCallbackRegistration struct.
• watcherEventBufferCapacity = 16 constant.
• Watch(group, key string) *Watcher: creates a buffered channel watcher. Wildcard "*" supported for both group and key. Uses atomic.AddUint64 for monotonic watcher IDs.
• Unwatch(watcher *Watcher): removes watcher from the registry and closes its channel. Idempotent.
• OnChange(callback func(Event)) func(): registers a synchronous callback. Returns an idempotent unregister function using sync.Once.
• notify(event Event): internal dispatch. Acquires read-lock on watchersLock; non-blocking send to each matching watcher channel (drop-on-full); calls each callback synchronously. Separate watcherMatches helper handles wildcard logic.
• Set(), SetWithTTL(), Delete(), DeleteGroup() each call notify() after the successful database write.
• Store struct extended with watchers []*Watcher, callbacks []changeCallbackRegistration, watchersLock sync.RWMutex, callbacksLock sync.RWMutex, nextWatcherID uint64, nextCallbackID uint64.
• ScopedStore mutations automatically emit events with the full prefixed group name — no extra implementation required.

Tests added

Specific-key watcher receives matching events and ignores non-matching keys. Wildcard-key watcher receives all keys in a group. Global wildcard ("*", "*") receives all mutations across all groups. Unwatch stops delivery and closes the channel. Unwatch is idempotent. Delete and DeleteGroup emit correct event types with correct populated fields. OnChange callback fires on Set and Delete. OnChange unregister stops future invocations (idempotent). Buffer-full (32 writes against cap-16 channel) does not block the writer. Multiple watchers on the same key receive events independently. Concurrent Watch/Unwatch during concurrent writes (race test, 10 goroutines). ScopedStore events carry the prefixed group name. SetWithTTL emits EventSet. EventType.String() returns correct labels including "unknown" for undefined values.

Coverage: 94.7% to 95.5%.

---

Phase 4 — AX API Cleanup

Agent: Codex Completed: 2026-03-30

Aligned the public API with the AX naming rules by removing compatibility aliases that were no longer used inside the repository.

Changes

• Removed the legacy compatibility aliases for the not-found error, quota error, key-value pair, and watcher channel.
• Kept the primary names NotFoundError, QuotaExceededError, KeyValue, and Watcher.Events.
• Updated docs and examples to describe the primary names only.

---

Phase 5 — Re-entrant Event Dispatch

Agent: Codex Completed: 2026-03-30

Changes

• Split watcher and callback registry locks so callbacks can register or unregister subscriptions without deadlocking.
• Updated notify() to dispatch watcher events under the watcher lock, snapshot callbacks under the callback lock, and invoke callbacks after both locks are released.

Tests added

• Re-entrant callback coverage for Watch, Unwatch, and OnChange from inside the same callback while a write is in flight.

---

Phase 6 — AX Schema Naming Cleanup

Agent: Codex Completed: 2026-03-30

Renamed the internal SQLite schema to use descriptive names that are easier for agents to read and reason about.

Changes

• Replaced the abbreviated key-value table with the descriptive entries table.
• Renamed the grp, key, and value schema columns to group_name, entry_key, and entry_value.
• Added a startup migration that copies legacy key-value databases into the new schema and preserves TTL data when present.
• Kept the public Go API unchanged; the migration only affects the internal storage layout.

---

Coverage Test Suite

coverage_test.go exercises defensive error paths that integration tests cannot reach through normal usage:

• Schema conflict: pre-existing SQLite index named entries causes New() to return store.New: ensure schema: ....
• GetAll scan error: NULL key in a row (requires manually altering the schema to remove the NOT NULL constraint) to trigger store.All: scan row: ....
• GetAll rows iteration error: physically corrupting database pages mid-file to trigger store.All: rows iteration: ....
• Render scan error: same NULL-key technique, surfaced as store.All: scan row: ....
• Render rows iteration error: same corruption technique, surfaced as store.All: rows iteration: ....

These tests exercise correct defensive code. They must continue to pass but are not indicative of real failure modes in production.

---

Known Limitations

Single writer. SetMaxOpenConns(1) serialises all access through one connection. Write throughput is bounded by SQLite's single-writer architecture. This is appropriate for the intended use cases (configuration storage, session state, per-tenant key-value data) but is not suitable for high-throughput append-only workloads.

File-backed write throughput. File-backed Set operations (~3,800 ops/sec on Apple M-series) are dominated by fsync. Applications writing at higher rates should use in-memory stores or consider WAL checkpoint tuning.

GetAll memory usage. Fetching a group with 10,000 keys allocates approximately 2.3 MB per call. Use GetPage() when you need offset/limit pagination over a large group. Applications with very large groups should still prefer smaller groups or selective queries.

No persistence of watcher registrations. Watchers and callbacks are in-memory only. They are not persisted across Close/New cycles.

---

Future Considerations

These are design notes, not committed work:

• Indexed prefix keys. An additional index on (group_name, entry_key) prefix would accelerate prefix scans without a full-table scan.