docs: graduate TODO/FINDINGS into production documentation
Replace internal task tracking (TODO.md, FINDINGS.md) with structured documentation in docs/. Trim CLAUDE.md to agent instructions only. Co-Authored-By: Virgil <virgil@lethean.io>
This commit is contained in:
Snider
parent
4628b2b094
commit
6e8ae2a1ec
6 changed files with 814 additions and 250 deletions
63
CLAUDE.md
63
CLAUDE.md
|
|
@ -1,66 +1,29 @@
|
|||
# CLAUDE.md
|
||||
|
||||
## What This Is
|
||||
## Project Overview
|
||||
|
||||
AI service lifecycle and task management client. Module: `forge.lthn.ai/core/go-agentic`
|
||||
AI service lifecycle and task management library. Module: `forge.lthn.ai/core/go-agentic`
|
||||
|
||||
Implements the Core framework service pattern for AI-assisted CLI operations: allowance/quota management, context building, completion requests, and claude CLI subprocess spawning.
|
||||
Implements the Core framework service pattern for AI-assisted CLI operations: allowance/quota management, agent registry, task routing and dispatch, context building, completion requests, and `claude` CLI subprocess spawning.
|
||||
|
||||
## Commands
|
||||
See `docs/architecture.md` for component details.
|
||||
|
||||
## Build Commands
|
||||
|
||||
```bash
|
||||
go test ./... # Run all tests
|
||||
go test -v -run TestAllowance # Single test
|
||||
go test -v -run TestName ./... # Single test
|
||||
go test -race ./... # With race detector
|
||||
go test -bench=. ./... # Benchmarks
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
| File | LOC | Purpose |
|
||||
|------|-----|---------|
|
||||
| `service.go` | Service lifecycle (OnStartup, ServiceRuntime) |
|
||||
| `allowance.go` | Quota/allowance tracking per workspace |
|
||||
| `allowance_service.go` | Allowance persistence and enforcement |
|
||||
| `client.go` | HTTP client for AI provider APIs |
|
||||
| `completion.go` | Completion request/response types and logic |
|
||||
| `config.go` | YAML/env configuration loading |
|
||||
| `context.go` | Context building for AI prompts |
|
||||
| `types.go` | Shared types (Task, Plan, etc.) |
|
||||
| `embed.go` | Embedded prompt templates |
|
||||
| `prompts/` | Markdown prompt templates (commit.md) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `forge.lthn.ai/core/go` — Core DI framework (ServiceRuntime, framework, io, log)
|
||||
- `gopkg.in/yaml.v3` — Configuration parsing
|
||||
- `github.com/stretchr/testify` — Tests
|
||||
|
||||
## Key Interfaces
|
||||
|
||||
```go
|
||||
// Implements Core service pattern
|
||||
type Service struct {
|
||||
*framework.ServiceRuntime[Config]
|
||||
}
|
||||
|
||||
// Allowance management
|
||||
type Allowance struct {
|
||||
WorkspaceID string
|
||||
Model string
|
||||
Used int
|
||||
Limit int
|
||||
}
|
||||
```
|
||||
|
||||
## Coding Standards
|
||||
|
||||
- UK English
|
||||
- Tests: testify assert/require
|
||||
- UK English (colour, organisation, behaviour, licence, serialise)
|
||||
- Exported types and functions require Go doc comments
|
||||
- Error strings use `log.E(op, "message", err)` from the Core framework
|
||||
- Defensive copies on all in-memory store Get/Set operations
|
||||
- Tests: `github.com/stretchr/testify/assert` and `require`
|
||||
- Conventional commits: `type(scope): description`
|
||||
- Co-Author: `Co-Authored-By: Virgil <virgil@lethean.io>`
|
||||
- Licence: EUPL-1.2
|
||||
|
||||
## Task Queue
|
||||
|
||||
See `TODO.md` for prioritised work.
|
||||
See `FINDINGS.md` for research notes.
|
||||
|
|
|
|||
58
FINDINGS.md
58
FINDINGS.md
|
|
@ -1,58 +0,0 @@
|
|||
# FINDINGS.md — go-agentic Research & Discovery
|
||||
|
||||
## 2026-02-19: Split from go-ai (Virgil)
|
||||
|
||||
### Origin
|
||||
|
||||
Extracted from `forge.lthn.ai/core/go-ai/agentic/`. Zero internal go-ai dependencies — cleanest split candidate.
|
||||
|
||||
### What Was Extracted
|
||||
|
||||
- 14 Go files (~1,968 LOC excluding tests)
|
||||
- 5 test files (allowance, client, completion, config, context)
|
||||
- 1 embedded prompt template (commit.md)
|
||||
|
||||
### Key Finding: Completely Independent
|
||||
|
||||
agentic/ imports only:
|
||||
- `forge.lthn.ai/core/go` (framework, io, log)
|
||||
- `gopkg.in/yaml.v3`
|
||||
- Standard library
|
||||
|
||||
No coupling to ml/, rag/, mcp/, or any other go-ai package. This is a pure CLI service that spawns claude subprocess and manages task/allowance state.
|
||||
|
||||
## 2026-02-20: Phase 1 Test Coverage (Charon)
|
||||
|
||||
### Coverage: 70.1% -> 85.6%
|
||||
|
||||
Added 7 new test files (130+ tests total, all passing):
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `lifecycle_test.go` | Full claim -> process -> complete integration; fail/cancel flows; concurrent agents |
|
||||
| `allowance_edge_test.go` | Boundary tests: exact limit, one-over, zero allowance, warning threshold table |
|
||||
| `allowance_error_test.go` | Mock `errorStore` to cover all error paths in RecordUsage/Check/ResetAgent |
|
||||
| `embed_test.go` | Prompt() hit/miss + trimming |
|
||||
| `service_test.go` | DefaultServiceOptions, TaskPrompt Set/GetTaskID, TaskCommit fields |
|
||||
| `completion_git_test.go` | Real git repos: AutoCommit, CreateBranch, CommitAndSync, GetDiff, HasUncommittedChanges |
|
||||
| `context_git_test.go` | findRelatedCode in git repos: keyword search, 10-file limit, truncation |
|
||||
|
||||
### Remaining uncovered (intentionally skipped)
|
||||
|
||||
- `service.go` (NewService, OnStartup, handleTask, doCommit, doPrompt) — 0%
|
||||
These require `framework.Core` DI container and spawn a `claude` subprocess.
|
||||
Testing would need either a full Core bootstrap or an interface extraction.
|
||||
Recommended for Phase 4 when CLI integration lands.
|
||||
|
||||
- `completion.go` CreatePR — 14.3%
|
||||
Requires `gh` CLI installed and authenticated. Not suitable for unit tests.
|
||||
|
||||
### Key Discovery: MemoryStore Copy Semantics
|
||||
|
||||
MemoryStore correctly copies values on Set/Get (defensive copies prevent aliasing).
|
||||
Tests confirm mutations to the original struct after SetAllowance do not affect stored data.
|
||||
|
||||
### Key Discovery: AllowanceService Check Priority
|
||||
|
||||
The check order is: model allowlist -> daily tokens -> daily jobs -> concurrent jobs -> global model budget.
|
||||
When multiple limits are exceeded simultaneously, the first in this order is reported as the reason.
|
||||
142
TODO.md
142
TODO.md
|
|
@ -1,142 +0,0 @@
|
|||
# TODO.md -- go-agentic
|
||||
|
||||
## Phase 1: Test Coverage
|
||||
|
||||
- [x] Verify all 5 test files pass standalone after split (`go test ./...`)
|
||||
- [x] Add integration test for full task lifecycle: claim -> process -> complete
|
||||
- [x] Add edge-case tests for allowance exhaustion mid-task
|
||||
- [x] Fill coverage gaps: embed, service types, git operations, config YAML/env paths, error store mock
|
||||
- [x] Target 85%+ coverage achieved: 85.6% (from 70.1%) — `23aa635`
|
||||
|
||||
## Phase 2: Allowance Persistence
|
||||
|
||||
- [x] MemoryStore is in-memory only -- state lost on restart
|
||||
- [x] Add Redis backend for `AllowanceStore` interface (multi-process safe) — `0be744e`
|
||||
- [x] Add SQLite backend for `AllowanceStore` interface (single-node fallback)
|
||||
- [x] Config already supports YAML -- wire backend selection into config loader
|
||||
|
||||
## Phase 3: Multi-Agent Coordination — `646cc02`
|
||||
|
||||
### 3.1 Agent Registry
|
||||
|
||||
- [x] **Create `registry.go`** — `AgentInfo` struct (ID, Name, Capabilities []string, Status enum, LastHeartbeat, CurrentLoad int, MaxLoad int), `AgentStatus` enum (Available/Busy/Offline)
|
||||
- [x] **`AgentRegistry` interface** — `Register(AgentInfo) error`, `Deregister(id string) error`, `Get(id string) (AgentInfo, error)`, `List() []AgentInfo`, `Heartbeat(id string) error`, `Reap(ttl time.Duration) []string` (returns IDs of reaped agents)
|
||||
- [x] **`MemoryRegistry` implementation** — `sync.RWMutex` guarded map, `Reap()` marks agents offline if heartbeat older than TTL
|
||||
- [x] **Tests** — registration, deregistration, heartbeat updates, reap stale agents, concurrent access
|
||||
|
||||
### 3.2 Task Router
|
||||
|
||||
- [x] **Create `router.go`** — `TaskRouter` interface with `Route(task *Task, agents []AgentInfo) (string, error)` returning agent ID
|
||||
- [x] **`DefaultRouter` implementation** — capability matching (task.Labels ⊆ agent.Capabilities), then least-loaded agent (CurrentLoad / MaxLoad ratio), priority weighting (critical tasks skip load balancing)
|
||||
- [x] **`ErrNoEligibleAgent`** sentinel error when no agents match capabilities or all are at capacity
|
||||
- [x] **Tests** — capability matching, load distribution, critical priority bypass, no eligible agent error, tie-breaking by agent ID (deterministic)
|
||||
|
||||
### 3.3 Dispatcher
|
||||
|
||||
- [x] **Create `dispatcher.go`** — `Dispatcher` struct wrapping `AgentRegistry`, `TaskRouter`, `AllowanceService`, and `Client`
|
||||
- [x] **`Dispatch(ctx, task) (string, error)`** — Route → allowance check → claim via client → record usage. Returns assigned agent ID
|
||||
- [x] **`DispatchLoop(ctx, interval)`** — polls client.ListTasks(pending) → dispatches each. Respects context cancellation
|
||||
- [x] **Tests** — full dispatch flow with mock client, allowance rejection path, no-agent-available path, loop cancellation
|
||||
|
||||
## Phase 4: CLI Backing Functions — `ef81db7`
|
||||
|
||||
Phase 4 provides the data-fetching and formatting functions that `core agent` CLI commands will call. The CLI commands themselves live in `core/cli`.
|
||||
|
||||
### 4.1 Status Summary
|
||||
|
||||
- [x] **Create `status.go`** — `StatusSummary` struct (Agents []AgentInfo, PendingTasks int, InProgressTasks int, AllowanceRemaining map[string]int64)
|
||||
- [x] **`GetStatus(ctx, registry, client, allowanceSvc) (*StatusSummary, error)`** — aggregates registry.List(), client.ListTasks counts, allowance remaining per agent
|
||||
- [x] **`FormatStatus(summary) string`** — tabular text output for CLI rendering
|
||||
- [x] **Tests** — with mock registry + nil client, full summary with mock client
|
||||
|
||||
### 4.2 Task Submission
|
||||
|
||||
- [x] **`SubmitTask(ctx, client, title, description, labels, priority) (*Task, error)`** — creates a new task via client (requires new Client.CreateTask method)
|
||||
- [x] **Add `Client.CreateTask(ctx, task) (*Task, error)`** — POST /api/tasks
|
||||
- [x] **Tests** — creation with all fields, validation (empty title), httptest mock
|
||||
|
||||
### 4.3 Log Streaming
|
||||
|
||||
- [x] **Create `logs.go`** — `StreamLogs(ctx, client, taskID, writer) error` — polls task updates and writes progress to io.Writer
|
||||
- [x] **Tests** — mock client with progress updates, context cancellation
|
||||
|
||||
## Phase 5: Persistent Agent Registry — `04a30df`
|
||||
|
||||
The `AgentRegistry` interface only has `MemoryRegistry` — a restart drops all agent registrations. This mirrors the AllowanceStore pattern: memory → SQLite → Redis.
|
||||
|
||||
### 5.1 SQLite Registry
|
||||
|
||||
- [x] **Create `registry_sqlite.go`** — `SQLiteRegistry` implementing `AgentRegistry` interface
|
||||
- [x] Schema: `agents` table (id TEXT PK, name TEXT, capabilities TEXT JSON, status TEXT, last_heartbeat DATETIME, current_load INT, max_load INT, registered_at DATETIME)
|
||||
- [x] Use `modernc.org/sqlite` (already a transitive dep via go-store) with WAL mode
|
||||
- [x] `Register` → UPSERT, `Deregister` → DELETE, `Get` → SELECT, `List` → SELECT all, `Heartbeat` → UPDATE last_heartbeat, `Reap(ttl)` → UPDATE status=Offline WHERE last_heartbeat < now-ttl
|
||||
- [x] **Tests** — full parity with `registry_test.go` using `:memory:` SQLite, concurrent access under `-race`
|
||||
|
||||
### 5.2 Redis Registry
|
||||
|
||||
- [x] **Create `registry_redis.go`** — `RedisRegistry` implementing `AgentRegistry` with TTL-based reaping
|
||||
- [x] Key pattern: `{prefix}:agent:{id}` → JSON AgentInfo, with TTL = heartbeat interval * 3
|
||||
- [x] `Heartbeat` → re-SET with TTL refresh (natural expiry = auto-reap)
|
||||
- [x] `List` → SCAN `{prefix}:agent:*`, `Reap` → explicit scan for expired (backup to natural TTL)
|
||||
- [x] **Tests** — skip-if-no-Redis pattern, unique prefix per test
|
||||
|
||||
### 5.3 Config Factory
|
||||
|
||||
- [x] **Add `RegistryConfig`** to `config.go` — `RegistryBackend string` (memory/sqlite/redis), `RegistryPath string`, `RegistryRedisAddr string`
|
||||
- [x] **`NewAgentRegistryFromConfig(cfg) (AgentRegistry, error)`** — factory mirroring `NewAllowanceStoreFromConfig`
|
||||
- [x] **Tests** — all backends, unknown backend error
|
||||
|
||||
## Phase 6: Dead Code Cleanup + Rate Enforcement
|
||||
|
||||
`HourlyRateLimit` and `CostCeiling` on `ModelQuota` are stored but never enforced in `AllowanceService.Check`. Either implement or remove.
|
||||
|
||||
### 6.1 Enforce or Remove Dead Fields
|
||||
|
||||
- [x] **Audit `HourlyRateLimit` and `CostCeiling`** — fields kept as reserved with doc annotations. Enforcement deferred: requires `AllowanceStore.GetHourlyUsage` (sliding window), which would break all 3 implementations. Fields stored/round-tripped correctly in all backends.
|
||||
|
||||
### 6.2 Fix DefaultBaseURL
|
||||
|
||||
- [x] **`DefaultBaseURL`** changed from `api.core-agentic.dev` (nonexistent) to `http://localhost:8080` for local dev. Set `AGENTIC_BASE_URL` env var for production.
|
||||
|
||||
## Phase 7: Priority-Ordered Dispatch + Retry — `ba8c19d`
|
||||
|
||||
`DispatchLoop` dispatches tasks in arbitrary order with no retry backoff.
|
||||
|
||||
### 7.1 Priority Sorting
|
||||
|
||||
- [x] **Sort pending tasks** in `DispatchLoop` by `Priority` (Critical > High > Normal > Low) before dispatching
|
||||
- [x] **Tie-break** by `CreatedAt` (oldest first within same priority)
|
||||
- [x] **Tests** — 5 tasks with mixed priorities dispatched in correct order
|
||||
|
||||
### 7.2 Retry Backoff
|
||||
|
||||
- [x] **Add `MaxRetries` and `RetryCount` fields** to `Task` type
|
||||
- [x] **Exponential backoff** in `DispatchLoop` — skip tasks where `RetryCount > 0` and `LastAttempt + backoff(RetryCount) > now`
|
||||
- [x] **Dead-letter** — tasks exceeding `MaxRetries` (default 3) get status `TaskFailed` with reason "max retries exceeded"
|
||||
- [x] **Tests** — retry delay respected, dead-letter after max retries, backoff calculation
|
||||
|
||||
## Phase 8: Event Hooks
|
||||
|
||||
Production orchestration needs event notifications for task lifecycle transitions.
|
||||
|
||||
### 8.1 EventEmitter Interface
|
||||
|
||||
- [x] **Create `events.go`** — `Event` struct (Type EventType, TaskID string, AgentID string, Timestamp time.Time, Payload any), 8 event types
|
||||
- [x] **`EventEmitter` interface** — `Emit(ctx context.Context, event Event) error`
|
||||
- [x] **`ChannelEmitter`** — in-process `chan Event` for local subscribers (buffered, non-blocking, drops on overflow)
|
||||
- [x] **`MultiEmitter`** — fans out to multiple emitters, continues on failure
|
||||
- [x] **Tests** — 8 tests: emit/receive, buffer overflow drops, default buffer size, multi-emitter fan-out, add emitter, failure tolerance, concurrent emit, event type coverage
|
||||
|
||||
### 8.2 Dispatcher Integration
|
||||
|
||||
- [x] **Wire `EventEmitter`** into `Dispatcher` — `SetEventEmitter()` + `emit()` helper. Events: task_dispatched, task_claimed, dispatch_failed_no_agent, dispatch_failed_quota, task_dead_lettered
|
||||
- [x] **Wire into `AllowanceService`** — `SetEventEmitter()` + `emitEvent()` helper. Events: quota_warning (80%), quota_exceeded (5 check paths), usage_recorded (job started + completed)
|
||||
- [x] **Tests** — 12 integration tests verifying all emission points, including nil-emitter safety
|
||||
|
||||
---
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Virgil in core/go writes tasks here after research
|
||||
2. This repo's dedicated session picks up tasks in phase order
|
||||
3. Mark `[x]` when done, note commit hash
|
||||
382
docs/architecture.md
Normal file
382
docs/architecture.md
Normal file
|
|
@ -0,0 +1,382 @@
|
|||
# go-agentic Architecture
|
||||
|
||||
Module: `forge.lthn.ai/core/go-agentic`
|
||||
|
||||
This document describes the internal structure of go-agentic: how components relate, what interfaces they implement, and how data flows through the system.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
go-agentic is an AI service lifecycle and task management library. It provides:
|
||||
|
||||
- A Core framework service that spawns `claude` subprocess invocations
|
||||
- Quota enforcement (allowance management) with three storage backends
|
||||
- An agent registry with two persistence backends
|
||||
- A task router and dispatcher for multi-agent coordination
|
||||
- Event hooks for lifecycle notification
|
||||
- CLI-oriented helper functions for status display, task submission, and log streaming
|
||||
|
||||
The package has no dependency on go-ai, go-ml, go-rag, or go-mcp. Its only non-standard imports are `forge.lthn.ai/core/go` (the Core DI framework), `forge.lthn.ai/core/go-store` (SQLite KV), and `github.com/redis/go-redis/v9`.
|
||||
|
||||
---
|
||||
|
||||
## Service Lifecycle
|
||||
|
||||
`Service` wraps `framework.ServiceRuntime[ServiceOptions]` and integrates with the Core dependency injection container.
|
||||
|
||||
```go
|
||||
type Service struct {
|
||||
*framework.ServiceRuntime[ServiceOptions]
|
||||
}
|
||||
```
|
||||
|
||||
On startup, `OnStartup` registers a task handler with the Core runtime:
|
||||
|
||||
```go
|
||||
func (s *Service) OnStartup(ctx context.Context) error {
|
||||
s.Core().RegisterTask(s.handleTask)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
The handler dispatches on task type:
|
||||
|
||||
- `TaskCommit` — runs the embedded `prompts/commit.md` prompt via `claude -p <prompt> --allowedTools Bash,Read,Glob,Grep` in the specified directory. If `CanEdit` is true, `Write` and `Edit` are added to the tool list.
|
||||
- `TaskPrompt` — runs an arbitrary prompt. Allowed tools default to `ServiceOptions.DefaultTools` (Bash, Read, Glob, Grep) but can be overridden per task. Reports progress via `Core().Progress()` using the task ID.
|
||||
|
||||
Both task types spawn `claude` as a subprocess with `os.Stdout`/`os.Stderr`/`os.Stdin` connected directly, so output streams to the terminal.
|
||||
|
||||
### ServiceOptions
|
||||
|
||||
```go
|
||||
type ServiceOptions struct {
|
||||
DefaultTools []string // default tool list for TaskPrompt
|
||||
AllowEdit bool // global permission for Write/Edit tools
|
||||
}
|
||||
```
|
||||
|
||||
Default tools are `["Bash", "Read", "Glob", "Grep"]`. Write/Edit require explicit opt-in either globally or per task.
|
||||
|
||||
---
|
||||
|
||||
## Allowance Management
|
||||
|
||||
Allowance management enforces per-agent and per-model token quotas. It is composed of three layers:
|
||||
|
||||
1. `AllowanceStore` — persistence interface
|
||||
2. `AllowanceService` — enforcement logic
|
||||
3. Three `AllowanceStore` implementations — Memory, SQLite, Redis
|
||||
|
||||
### AllowanceStore Interface
|
||||
|
||||
```go
|
||||
type AllowanceStore interface {
|
||||
GetAllowance(agentID string) (*AgentAllowance, error)
|
||||
SetAllowance(a *AgentAllowance) error
|
||||
GetUsage(agentID string) (*UsageRecord, error)
|
||||
IncrementUsage(agentID string, tokens int64, jobs int) error
|
||||
DecrementActiveJobs(agentID string) error
|
||||
ReturnTokens(agentID string, tokens int64) error
|
||||
ResetUsage(agentID string) error
|
||||
GetModelQuota(model string) (*ModelQuota, error)
|
||||
GetModelUsage(model string) (int64, error)
|
||||
IncrementModelUsage(model string, tokens int64) error
|
||||
}
|
||||
```
|
||||
|
||||
All three implementations satisfy this interface with identical semantics. The `AgentAllowance` struct holds per-agent limits; `UsageRecord` holds current period usage.
|
||||
|
||||
### MemoryStore
|
||||
|
||||
In-memory implementation using `sync.RWMutex`. State is lost on process restart. Suitable for single-process use, testing, and development. Uses defensive copy semantics: `Get` and `Set` return copies of stored structs to prevent aliasing.
|
||||
|
||||
### SQLiteStore
|
||||
|
||||
Persistent single-node implementation using `forge.lthn.ai/core/go-store` (a SQLite KV abstraction over `modernc.org/sqlite`). Data is namespaced into four groups: `allowances`, `usage`, `model_quotas`, `model_usage`. Read-modify-write operations are serialised with a `sync.Mutex`. WAL mode is enabled for concurrent reads. `time.Duration` values are serialised as int64 nanoseconds to avoid locale-dependent string parsing.
|
||||
|
||||
Default database path: `~/.config/agentic/allowance.db`.
|
||||
|
||||
### RedisStore
|
||||
|
||||
Multi-process persistent implementation using `github.com/redis/go-redis/v9`. Suitable for deployments where multiple agent processes share quota state. Atomic read-modify-write is implemented via Lua scripts registered on the Redis server (`EVAL`). Key pattern: `{prefix}:{type}:{id}`. Default prefix: `agentic`. Pings the server at construction time.
|
||||
|
||||
### Backend Selection
|
||||
|
||||
`NewAllowanceStoreFromConfig(cfg AllowanceConfig)` is the factory:
|
||||
|
||||
```go
|
||||
type AllowanceConfig struct {
|
||||
StoreBackend string // "memory" | "sqlite" | "redis"
|
||||
StorePath string // SQLite path (optional)
|
||||
RedisAddr string // host:port (optional)
|
||||
}
|
||||
```
|
||||
|
||||
An empty or unrecognised `StoreBackend` defaults to `"memory"`.
|
||||
|
||||
### AllowanceService
|
||||
|
||||
`AllowanceService` wraps an `AllowanceStore` and enforces quota policy. The check order in `Check(agentID, model string)` is:
|
||||
|
||||
1. Model allowlist — rejects if agent's `ModelAllowlist` is non-empty and the model is not in it
|
||||
2. Daily token limit — rejects if `TokensUsed >= DailyTokenLimit`; emits `EventQuotaWarning` at 80% usage
|
||||
3. Daily job limit — rejects if `JobsStarted >= DailyJobLimit`
|
||||
4. Concurrent jobs — rejects if `ActiveJobs >= ConcurrentJobs`
|
||||
5. Global model budget — rejects if model-level `TokensUsed >= DailyTokenBudget`
|
||||
|
||||
When multiple limits are simultaneously exceeded, the first in this order is reported as the reason.
|
||||
|
||||
`RecordUsage(report UsageReport)` handles four quota events:
|
||||
|
||||
| Event | Effect |
|
||||
|---|---|
|
||||
| `QuotaEventJobStarted` | Increments `JobsStarted` and `ActiveJobs` by 1 |
|
||||
| `QuotaEventJobCompleted` | Adds `TokensIn + TokensOut` to `TokensUsed`; decrements `ActiveJobs`; records model usage |
|
||||
| `QuotaEventJobFailed` | Records full token usage; returns 50% of tokens; decrements `ActiveJobs` |
|
||||
| `QuotaEventJobCancelled` | Returns 100% of tokens; decrements `ActiveJobs`; no model usage recorded |
|
||||
|
||||
Two fields on `ModelQuota` — `HourlyRateLimit` and `CostCeiling` — are stored and round-tripped by all backends but are not currently enforced by `AllowanceService.Check`. Enforcement requires a sliding-window `GetHourlyUsage` method on `AllowanceStore`, which would be a breaking interface change. The fields are annotated as reserved in the source.
|
||||
|
||||
---
|
||||
|
||||
## Agent Registry
|
||||
|
||||
`AgentRegistry` tracks the set of known agents and their health state.
|
||||
|
||||
### Interface
|
||||
|
||||
```go
|
||||
type AgentRegistry interface {
|
||||
Register(agent AgentInfo) error
|
||||
Deregister(id string) error
|
||||
Get(id string) (AgentInfo, error)
|
||||
List() []AgentInfo
|
||||
Heartbeat(id string) error
|
||||
Reap(ttl time.Duration) []string
|
||||
}
|
||||
```
|
||||
|
||||
`Reap` transitions agents whose `LastHeartbeat` is older than `ttl` to `AgentOffline` and returns their IDs. `Heartbeat` transitions an `AgentOffline` agent back to `AgentAvailable`.
|
||||
|
||||
### MemoryRegistry
|
||||
|
||||
In-memory implementation with `sync.RWMutex`. Uses copy-on-read semantics consistent with `MemoryStore`.
|
||||
|
||||
### SQLiteRegistry
|
||||
|
||||
Persistent implementation backed by a `database/sql` SQLite connection (using `modernc.org/sqlite` directly, not via go-store). Schema:
|
||||
|
||||
```sql
|
||||
CREATE TABLE agents (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL DEFAULT '',
|
||||
capabilities TEXT NOT NULL DEFAULT '[]',
|
||||
status TEXT NOT NULL DEFAULT 'available',
|
||||
last_heartbeat DATETIME NOT NULL DEFAULT (datetime('now')),
|
||||
current_load INTEGER NOT NULL DEFAULT 0,
|
||||
max_load INTEGER NOT NULL DEFAULT 0,
|
||||
registered_at DATETIME NOT NULL DEFAULT (datetime('now'))
|
||||
)
|
||||
```
|
||||
|
||||
`Register` uses `INSERT ... ON CONFLICT(id) DO UPDATE` (UPSERT). WAL mode and `busy_timeout=5000ms` are set at open time. Capabilities are stored as a JSON array string.
|
||||
|
||||
### RedisRegistry
|
||||
|
||||
TTL-based implementation where each agent is stored as a JSON value at key `{prefix}:agent:{id}`. Natural key expiry serves as the reap mechanism. `Heartbeat` refreshes the key's TTL. `Reap` performs an explicit SCAN for expired keys as a backup to natural expiry. Uses `WithRedisPassword`, `WithRedisDB`, and `WithRedisPrefix` functional options.
|
||||
|
||||
### Backend Selection
|
||||
|
||||
`NewAgentRegistryFromConfig(cfg RegistryConfig)` mirrors the allowance factory:
|
||||
|
||||
```go
|
||||
type RegistryConfig struct {
|
||||
RegistryBackend string // "memory" | "sqlite" | "redis"
|
||||
RegistryPath string // SQLite path (optional)
|
||||
RegistryRedisAddr string // host:port (optional)
|
||||
}
|
||||
```
|
||||
|
||||
Default registry path: `~/.config/agentic/registry.db`.
|
||||
|
||||
---
|
||||
|
||||
## Task Queue and Dispatcher
|
||||
|
||||
### Task Model
|
||||
|
||||
`Task` is the core data type:
|
||||
|
||||
```go
|
||||
type Task struct {
|
||||
ID string
|
||||
Title string
|
||||
Description string
|
||||
Priority TaskPriority // critical | high | medium | low
|
||||
Status TaskStatus // pending | in_progress | completed | blocked | failed
|
||||
Labels []string // capability requirements for routing
|
||||
Files []string // relevant files for context building
|
||||
Dependencies []string // task IDs that must complete first
|
||||
MaxRetries int // 0 uses DefaultMaxRetries (3)
|
||||
RetryCount int // failed dispatch attempts so far
|
||||
LastAttempt *time.Time // when the last dispatch attempt occurred
|
||||
FailReason string // set when StatusFailed
|
||||
// ... timestamps, ClaimedBy, Project, etc.
|
||||
}
|
||||
```
|
||||
|
||||
### TaskRouter
|
||||
|
||||
`DefaultRouter` selects an agent using two strategies:
|
||||
|
||||
- For `PriorityCritical` tasks: picks the agent with the lowest `CurrentLoad`, tie-broken alphabetically by agent ID.
|
||||
- For all other tasks: scores agents as `1.0 - (CurrentLoad / MaxLoad)` and picks the highest scorer. Agents with `MaxLoad == 0` score 1.0 (unlimited capacity).
|
||||
|
||||
Eligibility filtering runs first: only agents with status `AgentAvailable` or `AgentBusy` with remaining capacity are considered. The task's `Labels` field maps to capability requirements — all labels must be present in the agent's `Capabilities` slice.
|
||||
|
||||
Returns `ErrNoEligibleAgent` when no agent qualifies.
|
||||
|
||||
### Dispatcher
|
||||
|
||||
`Dispatcher` combines `AgentRegistry`, `TaskRouter`, `AllowanceService`, and `Client`:
|
||||
|
||||
```
|
||||
Dispatch(ctx, task):
|
||||
1. registry.List() — get candidate agents
|
||||
2. router.Route(task, agents) — select best agent
|
||||
3. allowance.Check(agentID) — verify quota
|
||||
4. client.ClaimTask(task.ID) — mark in-progress via API
|
||||
5. allowance.RecordUsage(JobStarted) — record job start
|
||||
→ emit EventTaskDispatched
|
||||
```
|
||||
|
||||
Failure paths emit `EventDispatchFailedNoAgent` or `EventDispatchFailedQuota`.
|
||||
|
||||
### DispatchLoop
|
||||
|
||||
`DispatchLoop(ctx, interval)` polls `client.ListTasks(pending)` at the given interval and dispatches each task. Key behaviours:
|
||||
|
||||
- Tasks are sorted by priority (Critical first) then by `CreatedAt` (oldest first) before dispatch.
|
||||
- Tasks with `RetryCount > 0` are skipped if `LastAttempt + backoff(RetryCount) > now`. Backoff is exponential starting at 5 seconds (5s, 10s, 20s, ...).
|
||||
- Tasks that fail dispatch have `RetryCount` incremented. Tasks reaching `MaxRetries` (default 3) are updated to `StatusFailed` via the API (dead-lettered) and emit `EventTaskDeadLettered`.
|
||||
- Transient API errors listing tasks are logged and the loop continues.
|
||||
- The loop exits when the context is cancelled and returns `ctx.Err()`.
|
||||
|
||||
---
|
||||
|
||||
## Context Builder
|
||||
|
||||
`BuildTaskContext(task, dir)` gathers context for AI collaboration:
|
||||
|
||||
1. Reads files explicitly listed in `task.Files` from the working directory.
|
||||
2. Runs `git status --porcelain` and `git log --oneline -10`.
|
||||
3. Extracts keywords from the task title and description (stop-word filtered, minimum 3 chars, up to 5 keywords) and runs `git grep -l -i <keyword> -- *.go *.ts *.js *.py` for each.
|
||||
4. Reads up to 10 matching source files, truncating files over 5,000 bytes.
|
||||
|
||||
`TaskContext.FormatContext()` renders the collected data as a Markdown string with sections for task metadata, task files, git status, recent commits, and related code — suitable for inclusion in a prompt sent to an AI model.
|
||||
|
||||
`GatherRelatedFiles(task, dir)` is the public sub-function that reads only the explicitly listed files.
|
||||
|
||||
---
|
||||
|
||||
## CLI Client
|
||||
|
||||
`Client` is an HTTP client for the core-agentic REST API. It sends `Bearer` token authentication on every request.
|
||||
|
||||
Endpoints used:
|
||||
|
||||
| Method | Path | Client function |
|
||||
|---|---|---|
|
||||
| GET | /api/tasks | ListTasks |
|
||||
| GET | /api/tasks/{id} | GetTask |
|
||||
| POST | /api/tasks | CreateTask |
|
||||
| POST | /api/tasks/{id}/claim | ClaimTask |
|
||||
| PATCH | /api/tasks/{id} | UpdateTask |
|
||||
| POST | /api/tasks/{id}/complete | CompleteTask |
|
||||
| GET | /api/health | Ping |
|
||||
|
||||
`ClaimTask` accepts both `ClaimResponse{"task": ...}` and bare `Task` JSON to handle varying API implementations. `checkResponse` parses error bodies as `APIError{code, message, details}` or falls back to a generic HTTP status message.
|
||||
|
||||
Default timeout: 30 seconds. Default base URL: `http://localhost:8080` (override with `AGENTIC_BASE_URL`).
|
||||
|
||||
### Configuration
|
||||
|
||||
`LoadConfig(dir)` resolves configuration in this order:
|
||||
|
||||
1. `.env` file in the specified directory (or current directory if `dir` is empty), reading `AGENTIC_BASE_URL`, `AGENTIC_TOKEN`, `AGENTIC_PROJECT`, `AGENTIC_AGENT_ID`.
|
||||
2. `~/.core/agentic.yaml` with the same fields in YAML form.
|
||||
3. Environment variable overrides applied on top of whichever file was loaded.
|
||||
|
||||
A missing token always returns an error. `SaveConfig` writes to `~/.core/agentic.yaml`, creating the directory if needed.
|
||||
|
||||
---
|
||||
|
||||
## Event Hooks
|
||||
|
||||
All lifecycle events are published through the `EventEmitter` interface:
|
||||
|
||||
```go
|
||||
type EventEmitter interface {
|
||||
Emit(ctx context.Context, event Event) error
|
||||
}
|
||||
```
|
||||
|
||||
`Event` carries a typed string `EventType`, optional `TaskID` and `AgentID`, a `Timestamp`, and an untyped `Payload`.
|
||||
|
||||
### Event Types
|
||||
|
||||
| Type | Emitter | Trigger |
|
||||
|---|---|---|
|
||||
| `task_dispatched` | Dispatcher | Successful routing, quota check, and claim |
|
||||
| `task_claimed` | Dispatcher | API claim call succeeded |
|
||||
| `dispatch_failed_no_agent` | Dispatcher | `ErrNoEligibleAgent` from router |
|
||||
| `dispatch_failed_quota` | Dispatcher | Allowance check denied |
|
||||
| `task_dead_lettered` | Dispatcher | Task exceeded `MaxRetries` |
|
||||
| `quota_warning` | AllowanceService | Agent at 80%+ of daily token limit |
|
||||
| `quota_exceeded` | AllowanceService | Any limit check fails |
|
||||
| `usage_recorded` | AllowanceService | `JobStarted` or `JobCompleted` events |
|
||||
|
||||
### Implementations
|
||||
|
||||
`ChannelEmitter` — buffered `chan Event`. Drops events silently if the buffer is full to avoid blocking the dispatch path. Default buffer size: 64. Use `Events()` to receive.
|
||||
|
||||
`MultiEmitter` — fans out to a slice of emitters. Failures in one emitter do not stop the others. Thread-safe: emitters can be added with `Add()` at any time.
|
||||
|
||||
Both `Dispatcher` and `AllowanceService` accept an emitter via `SetEventEmitter()`. A nil emitter is safe — all emit calls are no-ops.
|
||||
|
||||
---
|
||||
|
||||
## CLI Backing Functions
|
||||
|
||||
These functions are intended to be called by `core agent` CLI commands implemented in the `core/cli` repository.
|
||||
|
||||
`GetStatus(ctx, registry, client, allowanceSvc)` — aggregates `registry.List()`, counts of pending and in-progress tasks from the client, and remaining tokens per agent from the allowance service. Any argument can be nil; those sections are skipped. Returns `*StatusSummary`.
|
||||
|
||||
`FormatStatus(summary)` — renders a `StatusSummary` as a human-readable table with agent rows sorted alphabetically by ID.
|
||||
|
||||
`SubmitTask(ctx, client, title, description, labels, priority)` — validates that `title` is non-empty, constructs a `Task` with `StatusPending` and current timestamp, and calls `client.CreateTask`.
|
||||
|
||||
`StreamLogs(ctx, client, taskID, interval, writer)` — polls `client.GetTask` at the given interval, writing timestamped status lines to `writer`. Stops on `StatusCompleted` or `StatusBlocked`. Transient errors are written to the stream but do not stop polling. Returns `ctx.Err()` on cancellation.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Templates
|
||||
|
||||
`embed.go` embeds all files under `prompts/*.md` at compile time. `Prompt(name)` reads a template by name (without `.md`), trims whitespace, and returns an empty string if the file does not exist. Currently one template is provided: `prompts/commit.md`, used by `TaskCommit` to guide Claude through generating a conventional commit.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
go-agentic
|
||||
forge.lthn.ai/core/go (ServiceRuntime, framework, io, log)
|
||||
forge.lthn.ai/core/go-store (SQLite KV — SQLiteStore)
|
||||
github.com/redis/go-redis/v9 (RedisStore, RedisRegistry)
|
||||
gopkg.in/yaml.v3 (config file parsing)
|
||||
modernc.org/sqlite (SQLiteRegistry — direct database/sql)
|
||||
github.com/stretchr/testify (tests only)
|
||||
```
|
||||
|
||||
The module uses Go workspace `replace` directives to reference `../go` and `../go-store` during development.
|
||||
246
docs/development.md
Normal file
246
docs/development.md
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
# go-agentic Development Guide
|
||||
|
||||
Module: `forge.lthn.ai/core/go-agentic`
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Go 1.25 or later (the module declares `go 1.25.5`)
|
||||
- Go workspace containing `../go` and `../go-store` (the `go.mod` replace directives assume sibling directories)
|
||||
- `git` on `$PATH` (required at runtime by `context.go` and `completion.go`)
|
||||
- `claude` on `$PATH` (required at runtime by `service.go` — not needed for tests)
|
||||
- `gh` on `$PATH` and authenticated (required at runtime by `completion.go` CreatePR — not needed for unit tests)
|
||||
- Redis (optional — required only for `RedisStore` and `RedisRegistry` integration tests)
|
||||
|
||||
### Go Workspace Setup
|
||||
|
||||
The repository is part of a Go workspace rooted one level up. If running outside the workspace, the replace directives in `go.mod` must resolve:
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go → ../go
|
||||
forge.lthn.ai/core/go-store → ../go-store
|
||||
```
|
||||
|
||||
Either work within the workspace or run `go work sync` after cloning adjacent repositories.
|
||||
|
||||
---
|
||||
|
||||
## Build and Test Commands
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
go test ./...
|
||||
|
||||
# Run a single named test
|
||||
go test -v -run TestAllowance ./...
|
||||
|
||||
# Run with race detector (recommended before committing)
|
||||
go test -race ./...
|
||||
|
||||
# Run benchmarks
|
||||
go test -bench=. ./...
|
||||
|
||||
# Generate coverage report
|
||||
go test -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
There is no Taskfile or Makefile in this module. All automation uses plain `go` commands.
|
||||
|
||||
---
|
||||
|
||||
## Test Patterns
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
Tests use a `_Good`, `_Bad`, `_Ugly` suffix pattern inherited from the Core Go framework:
|
||||
|
||||
- `_Good` — happy path, expected success
|
||||
- `_Bad` — expected error conditions (invalid input, limit exceeded, not found)
|
||||
- `_Ugly` — panics, race conditions, or extreme edge cases
|
||||
|
||||
### File Organisation
|
||||
|
||||
Test files are co-located with their source files. The following test files exist:
|
||||
|
||||
| File | Covers |
|
||||
|---|---|
|
||||
| `allowance_test.go` | MemoryStore CRUD, UsageRecord, MemoryRegistry |
|
||||
| `allowance_edge_test.go` | Boundary conditions: exact limit, one-over, warning threshold |
|
||||
| `allowance_error_test.go` | Error paths in RecordUsage, Check, and ResetAgent via mock store |
|
||||
| `allowance_sqlite_test.go` | SQLiteStore with `:memory:` database |
|
||||
| `allowance_redis_test.go` | RedisStore integration (skip if Redis unavailable) |
|
||||
| `client_test.go` | HTTP client with `httptest.Server` mocks |
|
||||
| `completion_git_test.go` | AutoCommit, CreateBranch, CommitAndSync, GetDiff using real git repos |
|
||||
| `completion_test.go` | Completion type parsing |
|
||||
| `config_test.go` | YAML and .env loading, env var overrides |
|
||||
| `context_git_test.go` | findRelatedCode with real git repos, keyword search, 10-file limit, truncation |
|
||||
| `context_test.go` | BuildTaskContext, GatherRelatedFiles |
|
||||
| `coverage_boost_test.go` | Miscellaneous coverage for edge cases |
|
||||
| `dispatcher_test.go` | Full dispatch flow, allowance rejection, no-agent path, loop cancellation |
|
||||
| `embed_test.go` | Prompt() hit/miss and whitespace trimming |
|
||||
| `events_integration_test.go` | Event emission from Dispatcher and AllowanceService |
|
||||
| `events_test.go` | ChannelEmitter, MultiEmitter |
|
||||
| `lifecycle_test.go` | Full claim -> process -> complete integration; fail/cancel flows; concurrent agents |
|
||||
| `logs_test.go` | StreamLogs polling, context cancellation |
|
||||
| `registry_redis_test.go` | RedisRegistry integration (skip if Redis unavailable) |
|
||||
| `registry_sqlite_test.go` | SQLiteRegistry with `:memory:` database |
|
||||
| `registry_test.go` | MemoryRegistry |
|
||||
| `router_test.go` | Capability matching, load distribution, critical priority, tie-breaking |
|
||||
| `service_test.go` | DefaultServiceOptions, TaskPrompt Set/GetTaskID, TaskCommit fields |
|
||||
| `status_test.go` | GetStatus with nil and mock components, FormatStatus |
|
||||
| `submit_test.go` | SubmitTask validation and creation |
|
||||
|
||||
### Writing Tests
|
||||
|
||||
Use `github.com/stretchr/testify/assert` for non-fatal assertions and `require` for fatal ones:
|
||||
|
||||
```go
|
||||
func TestAllowance_Check_Good(t *testing.T) {
|
||||
store := NewMemoryStore()
|
||||
svc := NewAllowanceService(store)
|
||||
|
||||
err := store.SetAllowance(&AgentAllowance{
|
||||
AgentID: "agent-1",
|
||||
DailyTokenLimit: 1000,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
result, err := svc.Check("agent-1", "claude-sonnet-4-5")
|
||||
require.NoError(t, err)
|
||||
assert.True(t, result.Allowed)
|
||||
assert.Equal(t, AllowanceOK, result.Status)
|
||||
}
|
||||
```
|
||||
|
||||
### Redis Tests
|
||||
|
||||
Redis tests use a skip-if-no-Redis pattern:
|
||||
|
||||
```go
|
||||
func TestRedisStore_GetAllowance_Good(t *testing.T) {
|
||||
rs, err := NewRedisStore("localhost:6379")
|
||||
if err != nil {
|
||||
t.Skip("Redis not available:", err)
|
||||
}
|
||||
defer rs.Close()
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Each Redis test uses a unique key prefix to avoid cross-test interference:
|
||||
|
||||
```go
|
||||
rs, _ := NewRedisStore("localhost:6379",
|
||||
WithRedisPrefix("test-"+t.Name()),
|
||||
)
|
||||
defer rs.FlushPrefix(context.Background())
|
||||
```
|
||||
|
||||
### Mock Store Pattern
|
||||
|
||||
Use the `errorStore` mock (defined in `allowance_error_test.go`) to test error paths in `AllowanceService` without a real store:
|
||||
|
||||
```go
|
||||
type errorStore struct{ AllowanceStore }
|
||||
|
||||
func (e *errorStore) GetAllowance(string) (*AgentAllowance, error) {
|
||||
return nil, errors.New("store failure")
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### Language
|
||||
|
||||
UK English throughout. Use: colour, organisation, behaviour, licence (noun), centre, serialise, recognise. Never use American spellings.
|
||||
|
||||
### Go Style
|
||||
|
||||
- All files must have `package agentic` as the first declaration.
|
||||
- The module does not use `declare(strict_types)` (this is a Go project, not PHP), but all function signatures must have explicit parameter and return types.
|
||||
- Exported types and functions must have Go doc comments.
|
||||
- Error strings use the `"op: what failed"` convention via `log.E(op, "message", err)` from the Core framework.
|
||||
- Sentinel errors (such as `ErrNoEligibleAgent`) are package-level `var` declarations using `errors.New`.
|
||||
- Defensive copies on all `Get`/`Set` operations in in-memory stores to prevent aliasing.
|
||||
|
||||
### Conventional Commits
|
||||
|
||||
Commit message format:
|
||||
|
||||
```
|
||||
type(scope): short description
|
||||
|
||||
Optional longer body explaining why, not what.
|
||||
|
||||
Co-Authored-By: Virgil <virgil@lethean.io>
|
||||
```
|
||||
|
||||
Types: `feat`, `fix`, `test`, `docs`, `refactor`, `chore`
|
||||
|
||||
Scopes used in this project: `allowance`, `registry`, `dispatch`, `cli`, `events`, `config`, `test`
|
||||
|
||||
Examples from the commit log:
|
||||
|
||||
```
|
||||
feat(events): Phase 8 — event hooks for task lifecycle and quota notifications
|
||||
feat(dispatch): Phase 7 — priority-ordered dispatch with retry backoff and dead-letter
|
||||
feat(registry): Phase 5 — persistent agent registry (SQLite + Redis + config factory)
|
||||
feat(allowance): add Redis backend for AllowanceStore
|
||||
test: achieve 85.6% coverage with 7 new test files
|
||||
fix(config): change DefaultBaseURL to localhost, annotate reserved fields
|
||||
```
|
||||
|
||||
### Licence
|
||||
|
||||
All source files are licensed under EUPL-1.2. Do not add MIT, Apache, or other licence headers.
|
||||
|
||||
### Co-Author
|
||||
|
||||
Every commit must include:
|
||||
|
||||
```
|
||||
Co-Authored-By: Virgil <virgil@lethean.io>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Coverage Target
|
||||
|
||||
The project targets 85%+ line coverage. Current coverage after Phase 8 is above 96%. The following are intentionally excluded from coverage:
|
||||
|
||||
- `service.go` — `NewService`, `OnStartup`, `handleTask`, `doCommit`, `doPrompt` require the full Core DI container and a `claude` subprocess. Testing these functions needs either a complete Core bootstrap or interface extraction of the subprocess call.
|
||||
- `completion.go` CreatePR — requires `gh` CLI installed and authenticated. Not suitable for unit tests.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Backend
|
||||
|
||||
To add a new `AllowanceStore` or `AgentRegistry` backend:
|
||||
|
||||
1. Create a new file (e.g., `allowance_memcache.go`) implementing all interface methods.
|
||||
2. Add a `Close() error` method if the backend holds resources.
|
||||
3. Add a test file using the same pattern as `allowance_sqlite_test.go` (use an in-memory or mock connection; add a skip guard if the backend requires an external service).
|
||||
4. Add a case to `NewAllowanceStoreFromConfig` or `NewAgentRegistryFromConfig` in `config.go`.
|
||||
5. Add a corresponding config field to `AllowanceConfig` or `RegistryConfig`.
|
||||
|
||||
---
|
||||
|
||||
## Running Integration Tests Locally
|
||||
|
||||
```bash
|
||||
# With Redis running on default port
|
||||
go test -v -run TestRedis ./...
|
||||
go test -v -run TestRedisRegistry ./...
|
||||
|
||||
# With race detector for concurrent-access tests
|
||||
go test -race -v -run TestConcurrent ./...
|
||||
|
||||
# Run only fast unit tests (skips Redis)
|
||||
go test -short ./...
|
||||
```
|
||||
|
||||
The `-short` flag is respected: Redis tests check `testing.Short()` and skip accordingly if you add that guard, though the current pattern skips on connection failure rather than the short flag.
|
||||
173
docs/history.md
Normal file
173
docs/history.md
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
# go-agentic Project History
|
||||
|
||||
Module: `forge.lthn.ai/core/go-agentic`
|
||||
|
||||
---
|
||||
|
||||
## Origin: Extraction from go-ai
|
||||
|
||||
**Date**: 19 February 2026
|
||||
**Commit**: `68c108f feat: extract go-agentic from go-ai as standalone service package`
|
||||
|
||||
The package was extracted from `forge.lthn.ai/core/go-ai/agentic/`. The agentic subdirectory in go-ai imported only `forge.lthn.ai/core/go`, `gopkg.in/yaml.v3`, and the standard library — no coupling to `go-ai/ml`, `go-ai/rag`, `go-ai/mcp`, or any other subpackage. This made it the cleanest extraction candidate in the go-ai monolith.
|
||||
|
||||
What was extracted at the split point:
|
||||
|
||||
- 14 Go source files (~1,968 lines, excluding tests)
|
||||
- 5 test files covering allowance, client, completion, config, and context
|
||||
- 1 embedded prompt template (`prompts/commit.md`)
|
||||
|
||||
After extraction, a `go.mod` replace directive was corrected from `../core` to `../go` to match the actual sibling directory name (`af110be`).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Test Coverage
|
||||
|
||||
**Commit**: `23aa635 test: achieve 85.6% coverage with 7 new test files`
|
||||
**Operator**: Charon
|
||||
|
||||
Coverage improved from 70.1% to 85.6% with 7 new test files and over 130 tests:
|
||||
|
||||
| New file | Purpose |
|
||||
|---|---|
|
||||
| `lifecycle_test.go` | Full claim -> process -> complete integration; fail/cancel flows; concurrent agents |
|
||||
| `allowance_edge_test.go` | Boundary: exact limit, one-over, zero allowance, warning threshold |
|
||||
| `allowance_error_test.go` | Mock `errorStore` to exercise all error paths in RecordUsage/Check/ResetAgent |
|
||||
| `embed_test.go` | Prompt() hit/miss and whitespace trimming |
|
||||
| `service_test.go` | DefaultServiceOptions, TaskPrompt Set/GetTaskID, TaskCommit fields |
|
||||
| `completion_git_test.go` | AutoCommit, CreateBranch, CommitAndSync, GetDiff using real git repositories |
|
||||
| `context_git_test.go` | findRelatedCode in git repos: keyword search, 10-file cap, truncation |
|
||||
|
||||
**Discovery**: `MemoryStore` correctly uses defensive copies on `Set`/`Get`. Mutations to a struct after `SetAllowance` do not affect the stored data.
|
||||
|
||||
**Discovery**: `AllowanceService.Check` enforces limits in a fixed priority order: model allowlist -> daily tokens -> daily jobs -> concurrent jobs -> global model budget. When multiple limits are exceeded simultaneously, the first in this order is reported.
|
||||
|
||||
A follow-up commit (`5d02695`) pushed coverage further to 96.5%.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Allowance Persistence
|
||||
|
||||
**Commit**: `3e43233 feat: Phase 2 — SQLite AllowanceStore backend + config wiring`
|
||||
**Commit**: `0be744e feat(allowance): add Redis backend for AllowanceStore`
|
||||
|
||||
`MemoryStore` lost all state on process restart. Two persistent backends were added:
|
||||
|
||||
- `SQLiteStore` — single-node persistence via `forge.lthn.ai/core/go-store` (SQLite KV). Read-modify-write operations are serialised with `sync.Mutex`. `time.Duration` is stored as int64 nanoseconds to avoid locale-dependent string parsing.
|
||||
- `RedisStore` — multi-process persistence via `github.com/redis/go-redis/v9`. Atomic increment/decrement operations use Lua scripts (`EVAL`) to avoid TOCTOU races.
|
||||
|
||||
`AllowanceConfig` and `NewAllowanceStoreFromConfig` were added to `config.go` as the backend selection factory.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Multi-Agent Coordination
|
||||
|
||||
**Commit**: `646cc02 feat(coordination): add agent registry, task router, and dispatcher`
|
||||
|
||||
Three new files introduced the multi-agent layer:
|
||||
|
||||
- `registry.go` — `AgentInfo` struct, `AgentRegistry` interface, `MemoryRegistry` implementation. `Reap(ttl)` marks stale agents offline and returns their IDs.
|
||||
- `router.go` — `TaskRouter` interface, `DefaultRouter` implementation. Capability matching (task labels must be a subset of agent capabilities), load-based scoring (1 - load/max), least-loaded selection for critical tasks. `ErrNoEligibleAgent` sentinel.
|
||||
- `dispatcher.go` — `Dispatcher` combining registry, router, allowance service, and API client. `Dispatch` executes the five-step pipeline. `DispatchLoop` polls for pending tasks on a ticker.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: CLI Backing Functions
|
||||
|
||||
**Commit**: `ef81db7 feat(cli): add status summary, task submission, and log streaming`
|
||||
|
||||
Three files added to serve the `core agent` CLI commands (implemented separately in `core/cli`):
|
||||
|
||||
- `status.go` — `StatusSummary`, `GetStatus`, `FormatStatus`. Aggregates registry, task counts, and allowance remaining. All components are optional (nil-safe).
|
||||
- `submit.go` — `SubmitTask`. Validates title, sets `StatusPending` and `CreatedAt`, delegates to `client.CreateTask`.
|
||||
- `logs.go` — `StreamLogs`. Polls `GetTask` at an interval, writes timestamped status lines to an `io.Writer`, stops on terminal states.
|
||||
- `client.go` gained `CreateTask` (POST /api/tasks).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Persistent Agent Registry
|
||||
|
||||
**Commit**: `ce502c0 feat(registry): Phase 5 — persistent agent registry (SQLite + Redis + config factory)`
|
||||
|
||||
`MemoryRegistry` lost all agent registrations on restart. The same persistence pattern from Phase 2 was applied to the registry:
|
||||
|
||||
- `registry_sqlite.go` — `SQLiteRegistry` using `database/sql` with `modernc.org/sqlite` directly. Schema: `agents` table with UPSERT on Register, WAL mode, `busy_timeout=5000ms`.
|
||||
- `registry_redis.go` — `RedisRegistry` with TTL-based natural expiry serving as the reap mechanism. SCAN-based `Reap` as a backup.
|
||||
- `RegistryConfig` and `NewAgentRegistryFromConfig` added to `config.go`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Dead Code Cleanup
|
||||
|
||||
**Commit**: `779132a fix(config): change DefaultBaseURL to localhost, annotate reserved fields`
|
||||
|
||||
Two issues addressed:
|
||||
|
||||
- `DefaultBaseURL` was `api.core-agentic.dev` (a non-existent host). Changed to `http://localhost:8080`. Production deployments must set `AGENTIC_BASE_URL`.
|
||||
- `HourlyRateLimit` and `CostCeiling` on `ModelQuota` were stored but never enforced in `AllowanceService.Check`. Enforcement would require `AllowanceStore.GetHourlyUsage` (a sliding window query), which would be a breaking interface change. The fields are retained and annotated as reserved. All three backends correctly store and round-trip both values.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Priority-Ordered Dispatch and Retry
|
||||
|
||||
**Commit**: `ba8c19d feat(dispatch): Phase 7 — priority-ordered dispatch with retry backoff and dead-letter`
|
||||
|
||||
`DispatchLoop` previously dispatched tasks in arbitrary API order with no retry handling. Two improvements:
|
||||
|
||||
- **Priority sorting**: tasks are sorted by `priorityRank` (Critical=0, High=1, Medium=2, Low=3) then by `CreatedAt` ascending (oldest first for tasks of equal priority). `sort.SliceStable` is used for determinism.
|
||||
- **Exponential backoff and dead-letter**: tasks with `RetryCount > 0` are skipped until `LastAttempt + backoffDuration(RetryCount) > now`. Backoff starts at 5 seconds and doubles per retry (5s, 10s, 20s, ...). Tasks reaching `MaxRetries` (default 3) are updated to `StatusFailed` via `client.UpdateTask` and the failure reason is set to `"max retries exceeded"`. `MaxRetries` and `RetryCount` fields were added to the `Task` type.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Event Hooks
|
||||
|
||||
**Commit**: `a29ded5 feat(events): Phase 8 — event hooks for task lifecycle and quota notifications`
|
||||
|
||||
Production orchestration required external notification of lifecycle transitions. Three new constructs:
|
||||
|
||||
- `events.go` — `Event` struct, `EventType` (8 constants), `EventEmitter` interface, `ChannelEmitter` (buffered channel, drops on overflow), `MultiEmitter` (fan-out, failure-tolerant).
|
||||
- `Dispatcher.SetEventEmitter` and `emit` helper — emits `task_dispatched`, `task_claimed`, `dispatch_failed_no_agent`, `dispatch_failed_quota`, `task_dead_lettered`.
|
||||
- `AllowanceService.SetEventEmitter` and `emitEvent` helper — emits `quota_warning` (at 80% of daily token limit), `quota_exceeded` (five distinct check paths), `usage_recorded` (on job started and completed).
|
||||
|
||||
Both `SetEventEmitter` callers are nil-safe: emission is always a no-op when no emitter is set.
|
||||
|
||||
12 integration tests verify all emission points in `events_integration_test.go`.
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
The following limitations were documented during development and have not yet been addressed.
|
||||
|
||||
### service.go Coverage Gap
|
||||
|
||||
`NewService`, `OnStartup`, `handleTask`, `doCommit`, and `doPrompt` are at 0% test coverage. These functions require a full `framework.Core` DI container and spawn a `claude` subprocess. A mock subprocess approach (`exec.Command` with a test binary) is possible but was deferred to a later phase. A minimal mock binary approach was explored in `service_test.go` (commit `9636cdb`) for `HandleTask` tests.
|
||||
|
||||
### completion.go CreatePR
|
||||
|
||||
`CreatePR` calls `gh pr create` as a subprocess. It is at 14.3% coverage. Testing requires `gh` installed and authenticated against a real or stub GitHub API. Not suitable for unit tests.
|
||||
|
||||
### HourlyRateLimit and CostCeiling Not Enforced
|
||||
|
||||
Both fields on `ModelQuota` are stored correctly by all three backends but are never checked in `AllowanceService.Check`. Enforcement would require a new `AllowanceStore.GetHourlyUsage(agentID string, since time.Time) (int64, error)` method — a breaking interface change that would require updates to all three implementations plus their tests. This was deferred indefinitely.
|
||||
|
||||
### DispatchLoop Task State Not Re-fetched
|
||||
|
||||
After a failed dispatch that increments `RetryCount` and sets `LastAttempt`, the loop modifies the local copy of the task in the `tasks` slice. It does not call `client.UpdateTask` to persist `RetryCount`/`LastAttempt`. If the process restarts, those values are lost and the task will be dispatched again from a zero retry count. Fixing this requires either persisting retry state via the API or a separate task state store.
|
||||
|
||||
### No Heartbeat Loop
|
||||
|
||||
`AgentRegistry.Heartbeat` and `Reap` exist but there is no built-in background goroutine to call them. Callers must schedule heartbeats and reaping externally.
|
||||
|
||||
---
|
||||
|
||||
## Future Considerations
|
||||
|
||||
Items identified during development but not yet scoped:
|
||||
|
||||
- **Interface extraction for `service.go`**: Extract the subprocess invocation into a `ClaudeRunner` interface to enable unit testing without a real `claude` binary.
|
||||
- **`AllowanceStore.GetHourlyUsage`**: Required to enforce `HourlyRateLimit`. Would enable per-hour rate limiting across all three backends.
|
||||
- **Persist retry state via API**: `DispatchLoop` should call `client.UpdateTask` to persist `RetryCount` and `LastAttempt` so that retries survive process restarts.
|
||||
- **Built-in heartbeat loop**: A `StartHeartbeat(ctx, registry, agentID, interval)` helper to call `Heartbeat` on a ticker and `Reap` on a longer interval.
|
||||
- **WebSocket event emitter**: A `WebSocketEmitter` backed by `forge.lthn.ai/core/go/pkg/ws` for real-time event streaming to external consumers.
|
||||
- **Allowance daily reset scheduler**: A background goroutine that calls `AllowanceService.ResetAgent` at midnight UTC for each registered agent.
|
||||
Loading…
Add table
Reference in a new issue