Merge pull request 'feat(agentci): Clotho orchestrator, rate limiting, and security hardening' (#49) from feat/agentci-packaging into new

This commit is contained in:
Snider 2026-02-10 03:08:36 +00:00
commit 3473d5729e
27 changed files with 4867 additions and 203 deletions

213
docs/pkg-batch1-analysis.md Normal file
View file

@ -0,0 +1,213 @@
Here is the technical documentation for the Core framework packages.
# Core Framework Documentation
## Package: pkg/log
### 1. Overview
`pkg/log` acts as the central observability and error handling primitive for the framework. It combines structured logging with a rich error type system (`Err`), allowing operational context (Operations, Codes) to travel with errors up the stack. It is designed to be used both standalone and as an injectable service within the Core framework.
### 2. Public API
**Error Types & Functions**
* `type Err`: Struct implementing `error` with fields for `Op` (operation), `Msg` (message), `Err` (wrapped error), and `Code` (machine-readable code).
* `func E(op, msg string, err error) error`: Creates a new error with operational context.
* `func Wrap(err error, op, msg string) error`: Wraps an existing error, preserving existing codes if present.
* `func WrapCode(err error, code, op, msg string) error`: Wraps an error and assigns a specific error code.
* `func NewCode(code, msg string) error`: Creates a sentinel error with a code.
* `func Is(err, target error) bool`: Wrapper for `errors.Is`.
* `func As(err error, target any) bool`: Wrapper for `errors.As`.
* `func Join(errs ...error) error`: Wrapper for `errors.Join`.
* `func Op(err error) string`: Extracts the operation name from an error chain.
* `func ErrCode(err error) string`: Extracts the error code from an error chain.
* `func StackTrace(err error) []string`: Returns a slice of operations leading to the error.
* `func LogError(err error, op, msg string) error`: Logs an error and returns it wrapped (reduces boilerplate).
* `func LogWarn(err error, op, msg string) error`: Logs a warning and returns it wrapped.
* `func Must(err error, op, msg string)`: Panics if error is not nil, logging it first.
**Logging Types & Functions**
* `type Logger`: The main logging struct.
* `type Level`: Integer type for log verbosity (`LevelQuiet` to `LevelDebug`).
* `type Options`: Configuration struct for Logger (Level, Output, Rotation).
* `type RotationOptions`: Config for log file rotation (Size, Age, Backups, Compression).
* `func New(opts Options) *Logger`: Constructor.
* `func Default() *Logger`: Returns the global default logger.
* `func SetDefault(l *Logger)`: Sets the global default logger.
* `func (l *Logger) Debug/Info/Warn/Error/Security(msg string, keyvals ...any)`: Leveled logging methods.
**Service Integration**
* `type Service`: Wraps `Logger` for framework integration.
* `func NewService(opts Options) func(*framework.Core) (any, error)`: Factory for dependency injection.
* `type QueryLevel`, `type TaskSetLevel`: Message types for runtime management.
### 3. Internal Design
* **Contextual Errors**: The `Err` struct forms a linked list via the `Err` field (inner error), allowing the reconstruction of a logical stack trace (`op` sequence) distinct from the runtime stack trace.
* **Concurrency**: The `Logger` uses a `sync.RWMutex` to guard configuration and writes, ensuring thread safety.
* **Rotation Strategy**: The `RotatingWriter` implements `io.WriteCloser`. It lazily opens files and checks size thresholds on every write, leveraging `pkg/io` to abstract the filesystem.
* **Framework Integration**: The `Service` struct embeds `framework.ServiceRuntime`, utilizing the Actor pattern (Queries and Tasks) to allow dynamic log level adjustment at runtime without restarting the application.
### 4. Dependencies
* `github.com/host-uk/core/pkg/io`: Used by `rotation.go` to handle file operations (renaming, deleting, writing) abstractly.
* `github.com/host-uk/core/pkg/framework`: Used by `service.go` to hook into the application lifecycle and message bus.
* Standard Lib: `errors`, `fmt`, `os`, `sync`, `time`.
### 5. Test Coverage Notes
* **Error Unwrapping**: Verify `errors.Is` and `errors.As` work correctly through deep chains of `log.Err`.
* **Logical Stack Traces**: Ensure `StackTrace()` returns the correct order of operations `["app.Run", "db.Query", "net.Dial"]`.
* **Log Rotation**: Critical to test the boundary conditions of `MaxSize` and `MaxBackups` using a Mock Medium to avoid actual disk I/O.
* **Concurrency**: Race detection on `Logger` when changing levels while logging is active.
### 6. Integration Points
* **Application-wide**: This is the most imported package. All other packages should use `log.E` or `log.Wrap` instead of `fmt.Errorf` or `errors.New`.
* **Core Framework**: The `Service` is designed to be passed to `core.New()`.
---
## Package: pkg/config
### 1. Overview
`pkg/config` provides 12-factor app configuration management. It layers configuration sources in a specific precedence (Environment > Config File > Defaults) and exposes them via a typed API or a dot-notation getter. It abstracts the underlying storage, allowing configs to be loaded from disk or memory.
### 2. Public API
* `type Config`: The main configuration manager.
* `type Option`: Functional option pattern for configuration.
* `func New(opts ...Option) (*Config, error)`: Constructor.
* `func LoadEnv(prefix string) map[string]any`: Helper to parse environment variables into a map.
* `func (c *Config) Get(key string, out any) error`: Unmarshals a key (or root) into a struct.
* `func (c *Config) Set(key string, v any) error`: Sets a value and persists it to storage.
* `func (c *Config) LoadFile(m coreio.Medium, path string) error`: Merges a file into the current config.
* `type Service`: Framework service wrapper for `Config`.
* `func NewConfigService(c *core.Core) (any, error)`: Factory for dependency injection.
### 3. Internal Design
* **Engine**: Uses `spf13/viper` as the underlying configuration engine for its merging and unmarshalling logic.
* **Abstraction**: Unlike standard Viper usage, this package decouples the filesystem using `pkg/io.Medium`. This allows the config system to work in sandboxed environments or with mock filesystems.
* **Persistence**: The `Set` method triggers an immediate write-back to the storage medium, making the config file the source of truth for runtime changes.
* **Environment Mapping**: Automatically maps `CORE_CONFIG_FOO_BAR` to `foo.bar` using a `strings.Replacer`.
### 4. Dependencies
* `github.com/spf13/viper`: Core logic for map merging and unmarshalling.
* `gopkg.in/yaml.v3`: For marshalling data when saving.
* `github.com/host-uk/core/pkg/io`: For reading/writing config files.
* `github.com/host-uk/core/pkg/framework/core`: For service integration and error handling.
### 5. Test Coverage Notes
* **Precedence**: Verify that Environment variables override File values.
* **Persistence**: Test that `Set()` writes valid YAML back to the `Medium`.
* **Type Safety**: Ensure `Get()` correctly unmarshals into complex structs and returns errors on type mismatches.
### 6. Integration Points
* **Bootstrap**: Usually the first service initialized in `core.New()`.
* **Service Configuration**: Other services (like `auth` or `log`) should inject `config.Service` to retrieve their startup settings.
---
## Package: pkg/io
### 1. Overview
`pkg/io` provides a filesystem abstraction layer (`Medium`). Its philosophy is to decouple business logic from the `os` package, facilitating easier testing (via mocks) and security (via sandboxing).
### 2. Public API
* `type Medium`: Interface defining filesystem operations (`Read`, `Write`, `List`, `Stat`, `Open`, `Create`, `Delete`, `Rename`, etc.).
* `var Local`: A pre-initialized `Medium` for the host root filesystem.
* `func NewSandboxed(root string) (Medium, error)`: Returns a `Medium` restricted to a specific directory.
* `type MockMedium`: In-memory implementation of `Medium` for testing.
* `func NewMockMedium() *MockMedium`: Constructor for the mock.
* **Helpers**: `Read`, `Write`, `Copy`, `EnsureDir`, `IsFile`, `ReadStream`, `WriteStream` (accept `Medium` as first arg).
### 3. Internal Design
* **Interface Segregation**: The `Medium` interface mimics the capabilities of `os` and `io/fs` but bundles them into a single dependency.
* **Mocking**: `MockMedium` uses `map[string]string` for files and `map[string]bool` for directories. It implements manual path logic to simulate filesystem behavior (e.g., verifying a directory is empty before deletion) without touching the disk.
* **Sandboxing**: The `local` implementation (imported internally) enforces path scoping to prevent traversal attacks when using `NewSandboxed`.
### 4. Dependencies
* Standard Lib: `io`, `io/fs`, `os`, `path/filepath`, `strings`, `time`.
* `github.com/host-uk/core/pkg/io/local`: (Implied) The concrete implementation for OS disk access.
### 5. Test Coverage Notes
* **Mock fidelity**: The `MockMedium` must behave exactly like the OS. E.g., `Rename` should fail if the source doesn't exist; `Delete` should fail if a directory is not empty.
* **Sandboxing**: Verify that `..` traversal attempts in `NewSandboxed` cannot access files outside the root.
### 6. Integration Points
* **Universal Dependency**: Used by `log` (rotation), `config` (loading), and `auth` (user DB).
* **Testing**: Application code should accept `io.Medium` in constructors rather than using `os.Open` directly, enabling unit tests to use `NewMockMedium()`.
---
## Package: pkg/crypt
### 1. Overview
`pkg/crypt` provides "batteries-included," opinionated cryptographic primitives. It abstracts away the complexity of parameter selection (salt length, iteration counts, nonce generation) to prevent misuse of crypto algorithms.
### 2. Public API
* **Hashing**: `HashPassword` (Argon2id), `VerifyPassword`, `HashBcrypt`, `VerifyBcrypt`.
* **Symmetric**: `Encrypt`/`Decrypt` (ChaCha20-Poly1305), `EncryptAES`/`DecryptAES` (AES-GCM).
* **KDF**: `DeriveKey` (Argon2), `DeriveKeyScrypt`, `HKDF`.
* **Checksums**: `SHA256File`, `SHA512File`, `SHA256Sum`, `SHA512Sum`.
* **HMAC**: `HMACSHA256`, `HMACSHA512`, `VerifyHMAC`.
### 3. Internal Design
* **Safe Defaults**: Uses Argon2id for password hashing with tuned parameters (64MB memory, 3 iterations).
* **Container Format**: Symmetric encryption functions return a concatenated byte slice: `[Salt (16b) | Nonce (Variable) | Ciphertext]`. This ensures the decryption function has everything it needs without separate state management.
* **Randomness**: Automatically handles salt and nonce generation using `crypto/rand`.
### 4. Dependencies
* `golang.org/x/crypto`: For Argon2, ChaCha20, HKDF, Scrypt.
* Standard Lib: `crypto/aes`, `crypto/cipher`, `crypto/rand`, `crypto/sha256`.
### 5. Test Coverage Notes
* **Interoperability**: Ensure `Encrypt` output can be read by `Decrypt`.
* **Tamper Resistance**: manually modifying a byte in the ciphertext or nonce must result in a decryption failure (AuthTag check).
* **Vectors**: Validate hashing against known test vectors where possible.
### 6. Integration Points
* **Auth**: Heavily used by `pkg/auth` for password storage and potentially for encrypted user data.
* **Data Protection**: Any service requiring data at rest encryption should use `crypt.Encrypt`.
---
## Package: pkg/auth
### 1. Overview
`pkg/auth` implements a persistent user identity system based on OpenPGP challenge-response authentication. It supports a unique "Air-Gapped" workflow where challenges and responses are exchanged via files, alongside standard online methods. It manages user lifecycles, sessions, and key storage.
### 2. Public API
* `type Authenticator`: Main logic controller.
* `type User`: User metadata struct.
* `type Session`: Active session token struct.
* `func New(m io.Medium, opts ...Option) *Authenticator`: Constructor.
* `func (a *Authenticator) Register(username, password string) (*User, error)`: Creates new user and PGP keys.
* `func (a *Authenticator) Login(userID, password string) (*Session, error)`: Password-based fallback login.
* `func (a *Authenticator) CreateChallenge(userID string) (*Challenge, error)`: Starts PGP auth flow.
* `func (a *Authenticator) ValidateResponse(userID string, signedNonce []byte) (*Session, error)`: Completes PGP auth flow.
* `func (a *Authenticator) ValidateSession(token string) (*Session, error)`: Checks token validity.
* `func (a *Authenticator) WriteChallengeFile(userID, path string) error`: For air-gapped flow.
* `func (a *Authenticator) ReadResponseFile(userID, path string) (*Session, error)`: For air-gapped flow.
### 3. Internal Design
* **Storage Layout**: Uses a flat-file database approach on `io.Medium`:
* `users/{id}.pub`: Public Key.
* `users/{id}.key`: Encrypted Private Key.
* `users/{id}.lthn`: Password Hash.
* `users/{id}.json`: Encrypted metadata.
* **Identity**: User IDs are hashes of usernames to anonymize storage structure.
* **Flow**:
1. Server generates random nonce.
2. Server encrypts nonce with User Public Key.
3. User decrypts nonce (client-side) and signs it.
4. Server validates signature against User Public Key.
### 4. Dependencies
* `github.com/host-uk/core/pkg/io`: For user database storage.
* `github.com/host-uk/core/pkg/crypt/lthn`: (Implied) Specific password hashing.
* `github.com/host-uk/core/pkg/crypt/pgp`: (Implied) OpenPGP operations.
* `github.com/host-uk/core/pkg/framework/core`: Error handling.
### 5. Test Coverage Notes
* **Flow Verification**: Full integration test simulating a client: Register -> Get Challenge -> Decrypt/Sign (Mock Client) -> Validate -> Get Token.
* **Security**: Ensure `server` user cannot be deleted. Ensure expired sessions are rejected.
* **Persistence**: Ensure user data survives an `Authenticator` restart (i.e., data is actually written to medium).
### 6. Integration Points
* **API Gateways**: HTTP handlers would call `ValidateSession` on every request.
* **CLI Tools**: Would use `WriteChallengeFile`/`ReadResponseFile` for offline authentication.

255
docs/pkg-batch2-analysis.md Normal file
View file

@ -0,0 +1,255 @@
# Package Analysis — Batch 2
Generated by: gemini-batch-runner.sh
Models: gemini-2.5-flash-lite → gemini-3-flash-preview → gemini-3-pro-preview
Date: 2026-02-09
Packages: cli help session workspace
Total tokens: 125308
---
Here is the documentation for the analyzed framework packages.
# Core Framework Documentation
## Package: `pkg/cli`
### 1. Overview
The `cli` package is a comprehensive application runtime and UI framework designed to build uniform, aesthetic, and robust command-line interfaces. It acts as a high-level wrapper around `cobra`, handling application lifecycle (signals, daemonization), output styling (ANSI colors, glyphs, layouts), interactive prompts, and internationalization (i18n). Its design philosophy prioritizes developer ergonomics ("fluent" APIs) and consistent user experience across different execution modes (interactive vs. headless).
### 2. Public API
#### Application Lifecycle
- `func Init(opts Options) error`: Initialises the global CLI runtime, sets up the root command, and registers services.
- `func Main()`: The main entry point. Handles panic recovery, service initialization, and command execution. Exits process on completion.
- `func Execute() error`: Executes the root command structure.
- `func Shutdown()`: Triggers graceful shutdown of the runtime and all services.
- `func Run(ctx context.Context) error`: Blocking helper for daemon/simple modes.
- `func RunWithTimeout(timeout time.Duration) func()`: Returns a shutdown function that enforces a timeout.
#### Command Building
- `func NewCommand(use, short, long string, run func(*Command, []string) error) *Command`: Factory for standard commands.
- `func NewGroup(use, short, long string) *Command`: Factory for parent commands (no run logic).
- `func RegisterCommands(fn CommandRegistration)`: Registers a callback to add commands to the root at runtime.
#### Output & Styling
- `type AnsiStyle`: Fluent builder for text styling (Bold, Dim, Foreground, Background).
- `func Success(msg string)`, `func Error(msg string)`, `func Warn(msg string)`, `func Info(msg string)`: Semantic logging to stdout/stderr with glyphs.
- `func Table`: Struct and methods for rendering ASCII/Unicode tables.
- `func Check(name string) *CheckBuilder`: Fluent builder for test/verification status lines (Pass/Fail/Skip).
- `func Task(label, message string)`: Prints a task header.
- `func Progress(verb string, current, total int, item ...string)`: Prints a transient progress line.
- `func Layout(variant string) *Composite`: Creates an HLCRF (Header, Left, Content, Right, Footer) terminal layout.
#### Input & Interaction
- `func Confirm(prompt string, opts ...ConfirmOption) bool`: Interactive yes/no prompt.
- `func Prompt(label, defaultVal string) (string, error)`: Standard text input.
- `func Select(label string, options []string) (string, error)`: Interactive list selection.
- `func Choose[T](prompt string, items []T, opts ...ChooseOption[T]) T`: Generic selection helper.
#### Utilities
- `func GhAuthenticated() bool`: Checks GitHub CLI authentication status.
- `func GitClone(ctx, org, repo, path string) error`: Smart clone (uses `gh` if auth, else `git`).
### 3. Internal Design
- **Singleton Runtime**: The package relies on a package-level singleton `instance` (`runtime` struct) initialized via `Init`. This holds the `cobra.Command` tree and the Service Container.
- **Service Layering**: It integrates heavily with `pkg/framework`. Services like `log`, `i18n`, and `crypt` are injected into the runtime during initialization.
- **Mode Detection**: The `daemon.go` logic automatically detects if the app is running interactively (TTY), via pipe, or as a background daemon, adjusting output styling accordingly.
- **Global Error Handling**: Custom error types (`ExitError`) and wrappers (`WrapVerb`) utilize semantic grammar for consistent error messaging.
- **Glyph Abstraction**: The `Glyph` system abstracts symbols, allowing runtime switching between Unicode, Emoji, and ASCII themes based on terminal capabilities.
### 4. Dependencies
- `github.com/spf13/cobra`: The underlying command routing engine.
- `github.com/host-uk/core/pkg/framework`: The dependency injection and service lifecycle container.
- `github.com/host-uk/core/pkg/i18n`: For translation and semantic grammar generation.
- `github.com/host-uk/core/pkg/log`: For structured logging.
- `golang.org/x/term`: For TTY detection.
### 5. Test Coverage Notes
- **Interactive Prompts**: Tests must mock `stdin` to verify `Confirm`, `Prompt`, and `Select` behavior without hanging.
- **Command Registration**: Verify `RegisterCommands` works both before and after `Init` is called.
- **Daemon Lifecycle**: Tests needed for `PIDFile` locking and `HealthServer` endpoints (/health, /ready).
- **Layout Rendering**: Snapshot testing is recommended for `Layout` and `Table` rendering to ensure ANSI codes and alignment are correct.
### 6. Integration Points
- **Entry Point**: This package is the entry point for the entire application (`main.go` should call `cli.Main()`).
- **Service Registry**: Other packages (like `workspace` or custom logic) are registered as services via `cli.Options.Services`.
- **UI Standard**: All other packages should use `cli.Success`, `cli.Error`, etc., instead of `fmt.Println` to maintain visual consistency.
---
## Package: `pkg/help`
### 1. Overview
The `help` package provides an embedded documentation system. It treats documentation as data, parsing Markdown files into structured topics, and provides an in-memory full-text search engine to allow users to query help topics directly from the CLI.
### 2. Public API
- `type Catalog`: The central registry of help topics.
- `func DefaultCatalog() *Catalog`: Creates a catalog with built-in topics.
- `func (c *Catalog) Add(t *Topic)`: Registers a topic.
- `func (c *Catalog) Search(query string) []*SearchResult`: Performs full-text search.
- `func (c *Catalog) Get(id string) (*Topic, error)`: Retrieves a specific topic.
- `func ParseTopic(path string, content []byte) (*Topic, error)`: Parses raw Markdown content into a Topic struct.
- `type Topic`: Struct representing a documentation page (includes Title, Content, Sections, Tags).
### 3. Internal Design
- **In-Memory Indexing**: The `searchIndex` struct builds a reverse index (word -> topic IDs) on initialization. It does not use an external database.
- **Scoring Algorithm**: Search results are ranked based on a scoring system where matches in Titles > Section Headers > Content.
- **Markdown Parsing**: It uses Regex (`frontmatterRegex`, `headingRegex`) rather than a full AST parser to extract structure, prioritizing speed and simplicity for this specific use case.
- **Snippet Extraction**: The search logic includes a highlighter that extracts relevant text context around search terms.
### 4. Dependencies
- `gopkg.in/yaml.v3`: Used to parse the YAML frontmatter at the top of Markdown files.
### 5. Test Coverage Notes
- **Search Ranking**: Tests should verify that a keyword in a Title ranks higher than the same keyword in the body text.
- **Frontmatter Parsing**: Test with valid, invalid, and missing YAML frontmatter.
- **Tokenization**: Ensure `tokenize` handles punctuation and case insensitivity correctly to ensure search accuracy.
### 6. Integration Points
- **CLI Help Command**: The `pkg/cli` package would likely have a `help` command that instantiates the `Catalog` and calls `Search` or `Get` based on user input.
---
## Package: `pkg/session`
### 1. Overview
The `session` package is a specialized toolkit for parsing, analyzing, and visualizing "Claude Code" session transcripts (`.jsonl` files). It allows developers to replay AI interactions, search through past sessions, and generate visual artifacts (HTML reports or MP4 videos).
### 2. Public API
- `func ListSessions(projectsDir string) ([]Session, error)`: Scans a directory for session files.
- `func ParseTranscript(path string) (*Session, error)`: Reads a JSONL file and structures it into a `Session` object with a timeline of events.
- `func Search(projectsDir, query string) ([]SearchResult, error)`: specific search across all session files.
- `func RenderHTML(sess *Session, outputPath string) error`: Generates a self-contained HTML file visualizing the session.
- `func RenderMP4(sess *Session, outputPath string) error`: Uses `vhs` to render a video replay of the terminal session.
### 3. Internal Design
- **Streaming Parser**: `ParseTranscript` uses `bufio.Scanner` to handle potentially large JSONL files line-by-line, reconstructing the state of tool use (e.g., matching a `tool_use` event with its corresponding `tool_result`).
- **External Dependency Wrapper**: `RenderMP4` generates a `.tape` file dynamically and executes the external `vhs` binary to produce video.
- **HTML embedding**: `RenderHTML` embeds CSS and JS directly into the Go source strings to produce a single-file portable output without static asset dependencies.
### 4. Dependencies
- `github.com/charmbracelet/vhs` (Runtime dependency): The `vhs` binary must be installed for `RenderMP4` to work.
- Standard Library (`encoding/json`, `html/template` equivalents).
### 5. Test Coverage Notes
- **JSON Parsing**: Critical to test against the exact schema of Claude Code logs, including edge cases like partial streams or error states.
- **VHS Generation**: Test that the generated `.tape` content follows the VHS syntax correctly.
- **Tool Mapping**: Verify that specific tools (Bash, Edit, Write) are correctly categorized and parsed from the raw JSON arguments.
### 6. Integration Points
- **CLI Commands**: Likely used by commands like `core session list`, `core session play`, or `core session export`.
- **Filesystem**: Reads directly from the user's Claude Code project directory (usually `~/.claude/`).
---
## Package: `pkg/workspace`
### 1. Overview
The `workspace` package implements the `core.Workspace` interface, providing isolated, secure working environments. It manages the directory structure, file I/O, and cryptographic identity (PGP keys) associated with specific projects or contexts.
### 2. Public API
- `func New(c *core.Core) (any, error)`: Service factory function compatible with the framework registry.
- `func (s *Service) CreateWorkspace(identifier, password string) (string, error)`: Initialises a new workspace directory with keys.
- `func (s *Service) SwitchWorkspace(name string) error`: Sets the active context.
- `func (s *Service) WorkspaceFileGet(filename string) (string, error)`: Reads a file from the active workspace.
- `func (s *Service) WorkspaceFileSet(filename, content string) error`: Writes a file to the active workspace.
### 3. Internal Design
- **Service Implementation**: Implements `core.Workspace`.
- **IPC Handling**: Contains `HandleIPCEvents` to respond to generic framework messages (`workspace.create`, `workspace.switch`), allowing loose coupling with other components.
- **Path Hashing**: Uses SHA-256 to hash workspace identifiers into directory names (referred to as "LTHN proxy" in comments), likely to sanitize paths and obscure names.
- **Key Management**: Delegates actual key generation to the core's `Crypt()` service but manages the storage of the resulting keys within the workspace layout.
### 4. Dependencies
- `github.com/host-uk/core/pkg/framework/core`: Interfaces.
- `github.com/host-uk/core/pkg/io`: File system abstraction (`io.Medium`).
- `crypt` service (Runtime dependency): Required for `CreateWorkspace`.
### 5. Test Coverage Notes
- **Mocking IO**: Use an in-memory `io.Medium` implementation to test directory creation and file writing without touching the real disk.
- **State Management**: Test that `WorkspaceFileGet` fails correctly if `SwitchWorkspace` hasn't been called yet.
- **Concurrency**: `sync.RWMutex` is used; tests should verify race conditions aren't possible during rapid switching/reading.
### 6. Integration Points
- **Core Framework**: Registered in `pkg/cli/app.go` via `framework.WithName("workspace", workspace.New)`.
- **IPC**: Can be controlled by other plugins or UI components via the framework's message bus.
---
## Quick Reference (Flash Summary)
### Package: `pkg/cli`
**Description:** A comprehensive CLI framework providing terminal styling, command management, interactive prompts, and daemon lifecycles.
**Key Exported Types and Functions:**
- `AnsiStyle`: Struct for chaining terminal text styles (bold, colors, etc.).
- `Main()`: The primary entry point that initializes services and executes the root command.
- `Command`: Re-exported `cobra.Command` for simplified dependency management.
- `NewDaemon(opts)`: Manages background process lifecycles, PID files, and health checks.
- `Check(name)`: Fluent API for rendering status check lines (e.g., "✓ audit passed").
- `Confirm/Question/Choose`: Interactive prompt utilities for user input and selection.
- `Composite`: Implements a region-based layout system (Header, Left, Content, Right, Footer).
- `Table`: Helper for rendering aligned tabular data in the terminal.
**Dependencies:**
- `pkg/crypt/openpgp`
- `pkg/framework`
- `pkg/log`
- `pkg/workspace`
- `pkg/i18n`
- `pkg/io`
**Complexity:** Complex
---
### Package: `pkg/help`
**Description:** Manages display-agnostic help content with markdown parsing and full-text search capabilities.
**Key Exported Types and Functions:**
- `Catalog`: Registry for managing and searching help topics.
- `Topic`: Represents a help page including content, sections, and metadata.
- `ParseTopic(path, content)`: Parses markdown files with YAML frontmatter into structured topics.
- `searchIndex`: Internal engine providing scored full-text search and snippet extraction.
- `GenerateID(title)`: Utility to create URL-safe identifiers from strings.
**Dependencies:**
- *None* (Internal `pkg/*` imports)
**Complexity:** Moderate
---
### Package: `pkg/session`
**Description:** Parses, searches, and renders Claude Code session transcripts (JSONL) into HTML or video formats.
**Key Exported Types and Functions:**
- `Session`: Holds metadata and a timeline of `Event` objects from a transcript.
- `ParseTranscript(path)`: Reads JSONL files and reconstructs tool usage, user, and assistant interactions.
- `RenderHTML(sess, path)`: Generates a self-contained, interactive HTML timeline of a session.
- `RenderMP4(sess, path)`: Uses VHS to generate a terminal-style video recording of a session.
- `Search(dir, query)`: Scans a directory of session files for specific text or tool usage.
**Dependencies:**
- *None*
**Complexity:** Moderate
---
### Package: `pkg/workspace`
**Description:** Manages isolated, encrypted filesystem environments for different CLI projects.
**Key Exported Types and Functions:**
- `Service`: Core service managing active workspaces and their storage roots.
- `CreateWorkspace(id, pass)`: Initializes a hashed directory structure and generates PGP keypairs.
- `SwitchWorkspace(name)`: Sets the active workspace for subsequent file operations.
- `WorkspaceFileSet/Get`: Encrypted file I/O within the active workspace context.
- `HandleIPCEvents`: Processes workspace-related commands via the internal message bus.
**Dependencies:**
- `pkg/framework/core`
- `pkg/io`
**Complexity:** Moderate

384
docs/pkg-batch3-analysis.md Normal file
View file

@ -0,0 +1,384 @@
# Package Analysis — Batch 3
Generated by: gemini-batch-runner.sh
Models: gemini-2.5-flash-lite → gemini-3-flash-preview → gemini-3-pro-preview
Date: 2026-02-09
Packages: build container process jobrunner
Total tokens: 96300
---
Here is the technical documentation for the Core framework packages, analyzing the provided source code.
# Core Framework Package Documentation
## Table of Contents
1. [pkg/build](#package-pkgbuild)
2. [pkg/container](#package-pkgcontainer)
3. [pkg/process](#package-pkgprocess)
4. [pkg/jobrunner](#package-pkgjobrunner)
---
## Package: `pkg/build`
### 1. Overview
The `build` package provides a standardized system for detecting project types, loading build configurations, and packaging artifacts. It is designed around an abstraction of the filesystem (`io.Medium`) to facilitate testing and cross-platform compatibility, handling compression formats (gzip, xz, zip) and SHA256 checksum generation.
### 2. Public API
#### Project Detection & Configuration
```go
// Represents a detected project type (e.g., "go", "wails", "node")
type ProjectType string
// Detects project types in a directory based on marker files
func Discover(fs io.Medium, dir string) ([]ProjectType, error)
func PrimaryType(fs io.Medium, dir string) (ProjectType, error)
// Helper predicates for detection
func IsGoProject(fs io.Medium, dir string) bool
func IsWailsProject(fs io.Medium, dir string) bool
func IsNodeProject(fs io.Medium, dir string) bool
func IsPHPProject(fs io.Medium, dir string) bool
func IsCPPProject(fs io.Medium, dir string) bool
// Loads configuration from .core/build.yaml
func LoadConfig(fs io.Medium, dir string) (*BuildConfig, error)
func ConfigExists(fs io.Medium, dir string) bool
```
#### Artifact Management
```go
type Artifact struct {
Path, OS, Arch, Checksum string
}
type ArchiveFormat string // "gz", "xz", "zip"
// Archiving functions
func Archive(fs io.Medium, artifact Artifact) (Artifact, error) // Default gzip
func ArchiveXZ(fs io.Medium, artifact Artifact) (Artifact, error)
func ArchiveWithFormat(fs io.Medium, artifact Artifact, format ArchiveFormat) (Artifact, error)
func ArchiveAll(fs io.Medium, artifacts []Artifact) ([]Artifact, error)
// Checksum functions
func Checksum(fs io.Medium, artifact Artifact) (Artifact, error)
func ChecksumAll(fs io.Medium, artifacts []Artifact) ([]Artifact, error)
func WriteChecksumFile(fs io.Medium, artifacts []Artifact, path string) error
```
#### Interfaces
```go
// Interface for project-specific build logic
type Builder interface {
Name() string
Detect(fs io.Medium, dir string) (bool, error)
Build(ctx context.Context, cfg *Config, targets []Target) ([]Artifact, error)
}
```
### 3. Internal Design
* **Filesystem Abstraction**: Heavily relies on dependency injection via `io.Medium` rather than direct `os` calls, enabling safe unit testing of file operations.
* **Strategy Pattern**: The `Builder` interface allows different build logic (Go, Docker, Node) to be swapped dynamically based on detection.
* **Priority Detection**: `Discovery` uses an ordered slice of markers (`markers` var) to handle hybrid projects (e.g., Wails is detected before Go).
* **Configuration Overlay**: Uses `mapstructure` to parse YAML config, applying sensible defaults via `applyDefaults` if fields are missing.
### 4. Dependencies
* `archive/tar`, `archive/zip`, `compress/gzip`: Standard library for archiving.
* `github.com/Snider/Borg/pkg/compress`: External dependency for XZ compression support.
* `github.com/host-uk/core/pkg/io`: Internal interface for filesystem abstraction.
* `github.com/host-uk/core/pkg/config`: Internal centralized configuration loading.
### 5. Test Coverage Notes
* **Mocking IO**: Tests must implement a mock `io.Medium` to simulate file existence (`Detect`) and write operations (`Archive`) without touching the disk.
* **Format Specifics**: Verify that Windows builds automatically default to `.zip` regardless of the requested format in `ArchiveWithFormat`.
* **Config Parsing**: Test `LoadConfig` with malformed YAML and missing fields to ensure defaults are applied correctly.
### 6. Integration Points
* **CLI Build Commands**: This package is the backend for any `core build` CLI command.
* **CI Pipelines**: Used to generate release artifacts and `CHECKSUMS.txt` files for releases.
---
## Package: `pkg/container`
### 1. Overview
This package manages the lifecycle of local LinuxKit virtual machines. It abstracts underlying hypervisors (QEMU on Linux, Hyperkit on macOS) to provide a container-like experience (start, stop, logs, exec) for running VM images.
### 2. Public API
#### Manager & Lifecycle
```go
type Manager interface {
Run(ctx context.Context, image string, opts RunOptions) (*Container, error)
Stop(ctx context.Context, id string) error
List(ctx context.Context) ([]*Container, error)
Logs(ctx context.Context, id string, follow bool) (io.ReadCloser, error)
Exec(ctx context.Context, id string, cmd []string) error
}
// Factory
func NewLinuxKitManager(m io.Medium) (*LinuxKitManager, error)
```
#### Templates
```go
type TemplateManager struct { ... }
func NewTemplateManager(m io.Medium) *TemplateManager
func (tm *TemplateManager) ListTemplates() []Template
func (tm *TemplateManager) GetTemplate(name string) (string, error)
func (tm *TemplateManager) ApplyTemplate(name string, vars map[string]string) (string, error)
```
#### Types
```go
type Container struct {
ID, Name, Image string
Status Status // "running", "stopped", "error"
PID int
// ... ports, memory stats
}
type RunOptions struct {
Name string
Detach bool
Memory, CPUs, SSHPort int
Ports, Volumes map[string]string
}
```
### 3. Internal Design
* **Hypervisor Abstraction**: The `Hypervisor` interface hides the complexity of building CLI arguments for `qemu-system-x86_64` vs `hyperkit`.
* **State Persistence**: Uses a JSON file (`.core/containers.json`) protected by a `sync.RWMutex` to track VM state across process restarts.
* **Embedded Assets**: Uses Go `embed` to package default LinuxKit YAML templates (`templates/*.yml`) inside the binary.
* **Log Following**: Implements a custom `followReader` to emulate `tail -f` behavior for VM logs.
### 4. Dependencies
* `os/exec`: Essential for spawning the hypervisor processes.
* `embed`: For built-in templates.
* `github.com/host-uk/core/pkg/io`: Filesystem access for state and logs.
### 5. Test Coverage Notes
* **Process Management**: Difficult to test `Run` in standard CI. Mocking `exec.Command` or the `Hypervisor` interface is required.
* **State Integrity**: Test `LoadState` and `SaveState` handles corruption or concurrent writes.
* **Template Interpolation**: Verify `ApplyVariables` correctly handles required vs optional `${VAR:-default}` syntax.
### 6. Integration Points
* **Dev Environments**: Used to spin up isolated development environments defined by LinuxKit YAMLs.
* **Testing**: Can be used to launch disposable VMs for integration testing.
---
## Package: `pkg/process`
### 1. Overview
A sophisticated wrapper around `os/exec` that integrates with the Core framework's event bus. It features output streaming, ring-buffer capturing, dependency-based task execution (DAG), and a global singleton service for ease of use.
### 2. Public API
#### Service & Global Access
```go
// Global singletons (require Init)
func Init(c *framework.Core) error
func Start(ctx, cmd string, args ...string) (*Process, error)
func Run(ctx, cmd string, args ...string) (string, error)
func Kill(id string) error
// Service Factory
func NewService(opts Options) func(*framework.Core) (any, error)
```
#### Process Control
```go
type Process struct { ... }
func (p *Process) Wait() error
func (p *Process) Kill() error
func (p *Process) Output() string
func (p *Process) IsRunning() bool
func (p *Process) SendInput(input string) error
func (p *Process) Done() <-chan struct{}
```
#### Task Runner
```go
type Runner struct { ... }
type RunSpec struct {
Name, Command string
After []string // Dependencies
// ... args, env
}
func NewRunner(svc *Service) *Runner
func (r *Runner) RunAll(ctx context.Context, specs []RunSpec) (*RunAllResult, error)
func (r *Runner) RunParallel(ctx context.Context, specs []RunSpec) (*RunAllResult, error)
```
### 3. Internal Design
* **Event Sourcing**: Instead of just logging, the service broadcasts events (`ActionProcessStarted`, `ActionProcessOutput`) via `framework.Core`. This allows UI frontends to subscribe to real-time output.
* **Ring Buffer**: Uses a fixed-size circular buffer (`RingBuffer`) to store logs, preventing memory exhaustion from long-running processes.
* **DAG Execution**: The `Runner.RunAll` method implements a dependency graph resolver to run tasks in parallel waves based on the `After` field.
* **Global Singleton**: Uses `atomic.Pointer` for a thread-safe global `Default()` service instance.
### 4. Dependencies
* `os/exec`: The underlying execution engine.
* `github.com/host-uk/core/pkg/framework`: Creates the `ServiceRuntime` and provides the IPC/Action bus.
### 5. Test Coverage Notes
* **Concurrency**: The `Runner` needs tests for race conditions during parallel execution.
* **Dependency Resolution**: Test circular dependencies (deadlock detection) and skip logic when a dependency fails.
* **Buffer Overflow**: Verify `RingBuffer` overwrites old data correctly when full.
### 6. Integration Points
* **Task Runners**: The `Runner` struct is the engine for tools like `make` or `Taskfile`.
* **UI/TUI**: The Action-based output streaming is designed to feed data into a TUI or Web frontend in real-time.
---
## Package: `pkg/jobrunner`
### 1. Overview
A polling-based workflow engine designed to ingest "signals" (e.g., GitHub Issues/PRs), match them to specific handlers, and record execution results in a structured journal. It implements a "dry-run" capability and detailed audit logging.
### 2. Public API
#### Poller
```go
type Poller struct { ... }
type PollerConfig struct {
Sources []JobSource
Handlers []JobHandler
Journal *Journal
// ... interval, dryRun
}
func NewPoller(cfg PollerConfig) *Poller
func (p *Poller) Run(ctx context.Context) error
func (p *Poller) AddSource(s JobSource)
func (p *Poller) AddHandler(h JobHandler)
```
#### Journaling
```go
type Journal struct { ... }
func NewJournal(baseDir string) (*Journal, error)
func (j *Journal) Append(signal *PipelineSignal, result *ActionResult) error
```
#### Interfaces
```go
type JobSource interface {
Poll(ctx context.Context) ([]*PipelineSignal, error)
Report(ctx context.Context, result *ActionResult) error
}
type JobHandler interface {
Match(signal *PipelineSignal) bool
Execute(ctx context.Context, signal *PipelineSignal) (*ActionResult, error)
}
```
### 3. Internal Design
* **Poller Loop**: Runs a continuous loop (ticker-based) that snapshots sources and handlers at the start of every cycle to allow dynamic registration.
* **Data Models**: Defines rigid structures (`PipelineSignal`, `ActionResult`) to decouple data sources (GitHub) from logic handlers.
* **Journaling**: Writes `jsonl` (JSON Lines) files partitioned by repository and date (`baseDir/owner/repo/YYYY-MM-DD.jsonl`), ensuring an append-only audit trail.
### 4. Dependencies
* `github.com/host-uk/core/pkg/log`: Internal logging.
* `encoding/json`: For journal serialization.
### 5. Test Coverage Notes
* **Matching Logic**: Test that `findHandler` picks the correct handler for a given signal.
* **Dry Run**: Ensure `Execute` is *not* called when `dryRun` is true, but logs are generated.
* **Journal Locking**: Verify concurrent writes to the journal do not corrupt the JSONL file.
### 6. Integration Points
* **CI Bots**: The primary framework for building bots that automate Pull Request management or Issue triage.
* **Dashboarding**: The generated JSONL journal files are structured to be ingested by analytics tools.
---
## Quick Reference (Flash Summary)
### Package: `pkg/build`
Provides project type detection, build configuration management, and cross-compilation utilities.
**Key Exported Types and Functions**
* `Builder` (interface): Defines the interface for project-specific build implementations (Go, Node, PHP, etc.).
* `Config` / `BuildConfig` (structs): Hold runtime and file-based build parameters.
* `Artifact` (struct): Represents a build output file with path, OS, architecture, and checksum metadata.
* `ProjectType` (type): Constants identifying project types (e.g., `ProjectTypeGo`, `ProjectTypeWails`).
* `Archive`, `ArchiveXZ`, `ArchiveWithFormat`: Functions to create compressed archives (tar.gz, tar.xz, zip) of build artifacts.
* `Checksum`, `ChecksumAll`: Compute SHA256 hashes for build artifacts.
* `Discover`, `PrimaryType`: Detect project types based on marker files (e.g., `go.mod`, `package.json`).
* `LoadConfig`: Loads build settings from `.core/build.yaml`.
**Dependencies**
* `pkg/io`
* `pkg/config`
* `pkg/build/signing`
**Complexity Rating**
Moderate
---
### Package: `pkg/container`
Manages the lifecycle of LinuxKit virtual machines using platform-native hypervisors.
**Key Exported Types and Functions**
* `Manager` (interface): Defines container lifecycle operations (Run, Stop, List, Logs, Exec).
* `LinuxKitManager` (struct): Core implementation for managing LinuxKit VM instances.
* `Container` (struct): Represents a running or stopped VM instance with metadata like PID and status.
* `Hypervisor` (interface): Abstract interface for VM backends (QEMU, Hyperkit).
* `TemplateManager` (struct): Handles LinuxKit YAML templates and variable substitution.
* `State` (struct): Manages persistent storage of container metadata in JSON format.
* `DetectHypervisor`: Automatically selects the appropriate hypervisor for the current OS.
* `ApplyVariables`: Performs `${VAR:-default}` string interpolation in configuration files.
**Dependencies**
* `pkg/io`
**Complexity Rating**
Complex
---
### Package: `pkg/process`
Advanced process management system featuring output streaming, circular buffering, and dependency-aware task execution.
**Key Exported Types and Functions**
* `Service` (struct): Manages multiple processes with Core framework IPC integration.
* `Process` (struct): Represents a managed external process with non-blocking output capture.
* `Runner` (struct): Orchestrates complex task execution with dependency graph support (DAG).
* `RingBuffer` (struct): A thread-safe circular buffer for efficient process output storage.
* `RunOptions` (struct): Detailed configuration for spawning processes (env, dir, capture settings).
* `ActionProcessOutput`, `ActionProcessExited`: IPC message types for broadcasting process events via the Core framework.
* `Start`, `Run`, `Kill`: Global convenience functions for rapid process control.
**Dependencies**
* `pkg/framework`
**Complexity Rating**
Moderate/Complex
---
### Package: `pkg/jobrunner`
A poll-dispatch automation system designed to process structural signals from issues or pull requests.
**Key Exported Types and Functions**
* `Poller` (struct): Implements the main loop that discovers work from sources and dispatches to handlers.
* `PipelineSignal` (struct): A metadata snapshot of a work item (e.g., PR state, thread counts, mergeability).
* `JobSource` (interface): Interface for external systems (like GitHub) that provide actionable items.
* `JobHandler` (interface): Interface for logic that matches and executes actions on signals.
* `Journal` (struct): Provides persistent, date-partitioned JSONL audit logging for all actions.
* `ActionResult` (struct): Captures the success, failure, and duration of a completed job.
**Dependencies**
* `pkg/log`
**Complexity Rating**
Moderate

366
docs/pkg-batch4-analysis.md Normal file
View file

@ -0,0 +1,366 @@
# Package Analysis — Batch 4
Generated by: gemini-batch-runner.sh
Models: gemini-2.5-flash-lite → gemini-3-flash-preview → gemini-3-pro-preview
Date: 2026-02-09
Packages: git repos gitea forge release
Total tokens: 92202
---
Here is the technical documentation for the analyzed packages, written from the perspective of a Senior Go Engineer.
# Framework Package Documentation
## 1. Package: `pkg/git`
### Overview
The `git` package provides a high-level abstraction over local Git operations, specifically designed for multi-repo workspace management. It combines direct shell execution for complex operations (push/pull with interactive auth) with concurrent status checking. It is designed to run both as a standalone utility library and as a registered `framework.Service` within the Core application.
### Public API
**Types**
```go
type RepoStatus struct {
Name, Path string
Modified, Untracked, Staged, Ahead, Behind int
Branch string
Error error
}
func (s *RepoStatus) IsDirty() bool
func (s *RepoStatus) HasUnpushed() bool
func (s *RepoStatus) HasUnpulled() bool
type StatusOptions struct {
Paths []string
Names map[string]string
}
type PushResult struct {
Name, Path string
Success bool
Error error
}
// Service integration
type Service struct { ... }
type ServiceOptions struct { WorkDir string }
```
**Functions**
```go
// Concurrent status checking
func Status(ctx context.Context, opts StatusOptions) []RepoStatus
// Interactive operations (hooks into os.Stdin/Stdout)
func Push(ctx context.Context, path string) error
func Pull(ctx context.Context, path string) error
func PushMultiple(ctx context.Context, paths []string, names map[string]string) []PushResult
// Error handling
func IsNonFastForward(err error) bool
// Service Factory
func NewService(opts ServiceOptions) func(*framework.Core) (any, error)
```
### Internal Design
* **Shell Wrapper**: Uses `os/exec` to invoke the system `git` binary rather than using a native Go implementation (like go-git). This ensures 100% compatibility with the user's local git configuration (SSH keys, hooks, GPG signing).
* **Concurrency**: `Status()` uses a `sync.WaitGroup` pattern to check multiple repository statuses in parallel, significantly speeding up workspace checks.
* **Interactive Mode**: `Push` and `Pull` explicitly wire `os.Stdin` and `os.Stdout` to the subprocess to allow SSH passphrase prompts or GPG pin entry to function correctly in a terminal environment.
* **Service Pattern**: Implements the `framework.ServiceRuntime` interface, registering distinct Queries (`QueryStatus`) and Tasks (`TaskPush`) to decouple the UI/CLI from the git logic.
### Dependencies
* `os/exec`: For invoking git commands.
* `github.com/host-uk/core/pkg/framework`: For service registration and message passing types.
### Test Coverage Notes
* **Mocking**: Testing requires abstracting `exec.Command`. Since this package calls `exec.CommandContext` directly, tests likely require overriding a package-level variable or using a "fake exec" pattern during test initialization.
* **Parsing**: Unit tests should cover the parsing logic of `git status --porcelain` in `getStatus` to ensure modified/staged/untracked counts are accurate.
* **Concurrency**: Race detection should be enabled to ensure `Status()` result slice assignment is thread-safe (it uses index-based assignment, which is safe).
### Integration Points
* **CLI**: The CLI command `core git status` consumes the `Service` via the framework's message bus.
* **Workspace Managers**: Packages managing multi-repo setups (like `pkg/repos`) use this to report health.
---
## 2. Package: `pkg/repos`
### Overview
This package manages the "Registry" of a multi-repository ecosystem. It acts as the source of truth for repository locations, types (foundation, module, product), and dependencies. It supports loading from a static `repos.yaml` or scanning the filesystem as a fallback.
### Public API
**Types**
```go
type Registry struct {
Repos map[string]*Repo
Defaults RegistryDefaults
...
}
type Repo struct {
Name, Type, Description, CI, Domain string
DependsOn []string
Docs bool
Path string // Computed
}
type RepoType string // "foundation", "module", "product", "template"
```
**Functions**
```go
// Loading
func LoadRegistry(m io.Medium, path string) (*Registry, error)
func FindRegistry(m io.Medium) (string, error)
func ScanDirectory(m io.Medium, dir string) (*Registry, error)
// Registry Methods
func (r *Registry) List() []*Repo
func (r *Registry) Get(name string) (*Repo, bool)
func (r *Registry) ByType(t string) []*Repo
func (r *Registry) TopologicalOrder() ([]*Repo, error)
// Repo Methods
func (repo *Repo) Exists() bool
func (repo *Repo) IsGitRepo() bool
```
### Internal Design
* **Abstraction**: Uses `io.Medium` to abstract filesystem access, making the registry testable without disk I/O.
* **Computed Fields**: The YAML struct is separate from the logic; `LoadRegistry` enriches the raw data with computed absolute paths and back-references.
* **Graph Theory**: `TopologicalOrder` implements a Depth-First Search (DFS) with cycle detection (`visiting` vs `visited` maps) to resolve build orders based on the `depends_on` field.
### Dependencies
* `gopkg.in/yaml.v3`: For parsing `repos.yaml`.
* `github.com/host-uk/core/pkg/io`: For filesystem abstraction (`io.Medium`).
### Test Coverage Notes
* **Circular Dependencies**: Critical test cases must define a registry with `A->B->A` dependencies to ensure `TopologicalOrder` returns a clear error and doesn't stack overflow.
* **Path Expansion**: Verify `~` expansion logic works across different OS mocks in `LoadRegistry`.
### Integration Points
* **Build System**: The build package uses `TopologicalOrder()` to determine the sequence in which to build libraries before products.
* **CI/CD**: Uses `Repo.Type` to apply different linting/testing rules (e.g., Foundation repos might require stricter coverage).
---
## 3. Packages: `pkg/gitea` & `pkg/forge`
*(Note: These packages share a very similar design pattern. `pkg/forge` is essentially a port of `pkg/gitea` for Forgejo.)*
### Overview
These packages provide typed clients for Gitea and Forgejo instances. They abstract the underlying SDKs to provide "Configuration-Aware" clients that automatically resolve authentication (Config vs Env vs Flags) and provide specialized helper methods for AI-driven metadata extraction (`PRMeta`).
### Public API (Common to both)
**Types**
```go
type Client struct { ... }
// Structural signals for AI analysis
type PRMeta struct {
Number int64
Title, State, Author, Branch, BaseBranch string
Labels, Assignees []string
IsMerged bool
CommentCount int
...
}
```
**Functions**
```go
// Construction
func New(url, token string) (*Client, error)
func NewFromConfig(flagURL, flagToken string) (*Client, error)
// Meta-data Extraction
func (c *Client) GetPRMeta(owner, repo string, pr int64) (*PRMeta, error)
func (c *Client) GetCommentBodies(...)
func (c *Client) GetIssueBody(...)
// Repo Management
func (c *Client) CreateMirror(...) // Gitea specific migration
func (c *Client) MigrateRepo(...) // Forgejo specific migration
func (c *Client) ListOrgRepos(...)
func (c *Client) ListUserRepos(...)
```
### Internal Design
* **Config Precedence Layer**: `ResolveConfig` implements a strict hierarchy: CLI Flags > Environment Variables > Config File (`~/.core/config.yaml`). This allows seamless switching between local dev and CI environments.
* **Dual-End Reader**: The `GetPRMeta` method aggregates data from multiple API endpoints (PR details + Issue Comments + Labels) into a flattened struct designed specifically to be fed into an LLM or policy engine.
* **Workarounds**: `pkg/forge/prs.go` implements a raw `net/http` PATCH request for `SetPRDraft` because the specific feature was missing or broken in the imported version of the Forgejo SDK.
### Dependencies
* `code.gitea.io/sdk/gitea` (for `pkg/gitea`)
* `codeberg.org/mvdkleijn/forgejo-sdk` (for `pkg/forge`)
* `github.com/host-uk/core/pkg/config`: For persistent auth storage.
### Test Coverage Notes
* **Draft Status**: The raw HTTP patch in `pkg/forge` needs integration testing against a real instance or a high-fidelity HTTP mock to ensure payload format matches Forgejo's API expectation.
* **Pagination**: `List*` methods implement manual pagination loops. Tests should simulate API responses with multiple pages to verify all items are collected.
### Integration Points
* **CI Pipelines**: Used to fetch PR context for "Smart CI" decisions.
* **Migration Tools**: The `CreateMirror`/`MigrateRepo` functions are used to synchronize repositories between GitHub and local Gitea/Forgejo instances.
---
## 4. Package: `pkg/release`
### Overview
The `release` package allows fully automated releases. It handles Semantic Versioning detection, Conventional Commit parsing for changelogs, build orchestration, and publishing to multiple downstream targets (GitHub, Docker, LinuxKit, etc.).
### Public API
**Types**
```go
type Config struct { ... } // Maps to release.yaml
type Release struct {
Version string
Artifacts []build.Artifact
Changelog string
...
}
```
**Functions**
```go
// Main Entry Points
func Run(ctx context.Context, cfg *Config, dryRun bool) (*Release, error)
func Publish(ctx context.Context, cfg *Config, dryRun bool) (*Release, error)
func RunSDK(ctx context.Context, cfg *Config, dryRun bool) (*SDKRelease, error)
// Utilities
func DetermineVersion(dir string) (string, error)
func Generate(dir, fromRef, toRef string) (string, error) // Changelog
func IncrementVersion(current string) string
```
### Internal Design
* **Pipeline Architecture**: `Run()` executes a linear pipeline:
1. **Versioner**: Checks Git tags -> parses SemVer -> increments patch (default).
2. **Changelog**: Parses `git log` via `ParseCommitType` (Conventional Commits regex) -> Buckets by type (Feat, Fix) -> Renders Markdown.
3. **Builder**: Delegates to `pkg/build` to compile binaries and generate checksums.
4. **Publisher**: Iterates over `Config.Publishers`, instantiates specific strategies via a Factory pattern (`getPublisher`), and executes them.
* **Separation of Concerns**: `Publish()` exists separately from `Run()` to support CI workflows where the *Build* step is separate from the *Release* step. It locates pre-existing artifacts in `dist/`.
* **SDK Generation**: Includes a specialized sub-pipeline (`RunSDK`) that handles OpenAPI diffing and client generation.
### Dependencies
* `github.com/host-uk/core/pkg/build`: For compiling artifacts.
* `github.com/host-uk/core/pkg/release/publishers`: Interface definitions for publishing targets.
* `golang.org/x/text`: For title casing in changelogs.
### Test Coverage Notes
* **SemVer Logic**: Extensive unit tests needed for `DetermineVersion` and `IncrementVersion` covering edge cases (v-prefix, no tags, pre-releases).
* **Regex**: Validate `conventionalCommitRegex` against a corpus of valid and invalid commit messages to ensure changelogs are generated correctly.
* **Config Unmarshaling**: `LoadConfig` uses complex nesting; tests should verify that `release.yaml` maps correctly to the internal structs, especially the `map[string]any` used for publisher-specific config.
### Integration Points
* **CI Runner**: This is the engine behind `core ci release`.
* **Build System**: Tightly coupled with `pkg/build`—it assumes artifacts are placed in `dist/` and accompanied by a `CHECKSUMS.txt`.
---
## Quick Reference (Flash Summary)
### Package: `pkg/git`
Provides utilities for git operations across multiple repositories and a service runtime for managing repository states.
**Key Exported Types and Functions**
- `RepoStatus`: Struct representing the state of a repository (ahead/behind counts, dirty status, branch).
- `Status()`: Checks git status for multiple repositories in parallel using goroutines.
- `Push()` / `Pull()`: Performs git operations in interactive mode to support SSH passphrase prompts.
- `PushMultiple()`: Executes pushes for multiple repositories sequentially.
- `Service`: A framework-compatible service that handles git-related tasks and queries.
- `IsNonFastForward()`: Utility to detect specific git push rejection errors.
**Dependencies**
- `pkg/framework`
**Complexity Rating**
Moderate
---
### Package: `pkg/repos`
Manages multi-repo workspaces by parsing a registry configuration and handling repository discovery.
**Key Exported Types and Functions**
- `Registry`: Represents a collection of repositories defined in `repos.yaml`.
- `Repo`: Represents a single repository with metadata and dependency information.
- `LoadRegistry()`: Loads and parses the repository registry from a given storage medium.
- `FindRegistry()`: Searches for a `repos.yaml` file in local and config directories.
- `ScanDirectory()`: Fallback mechanism to generate a registry by scanning a filesystem for git folders.
- `TopologicalOrder()`: Sorts repositories based on their dependency graph for build ordering.
**Dependencies**
- `pkg/io`
**Complexity Rating**
Moderate
---
### Package: `pkg/gitea`
A wrapper around the Gitea Go SDK for managing repositories, issues, and pull requests.
**Key Exported Types and Functions**
- `Client`: Primary wrapper for the Gitea API client.
- `NewFromConfig()`: Resolves authentication (token/URL) from flags, environment, or config files.
- `GetPRMeta()`: Extracts structural metadata from a pull request for pipeline analysis.
- `ListOrgRepos()` / `ListUserRepos()`: Lists repositories for organizations or the authenticated user.
- `CreateMirror()`: Uses the migration API to set up a pull mirror from a remote source.
- `GetCommentBodies()`: Retrieves all text content for PR comments.
**Dependencies**
- `pkg/log`
- `pkg/config`
**Complexity Rating**
Moderate
---
### Package: `pkg/forge`
A wrapper around the Forgejo Go SDK for repository management, issue tracking, and PR orchestration.
**Key Exported Types and Functions**
- `Client`: Primary wrapper for the Forgejo API client.
- `NewFromConfig()`: Tiered configuration loader for Forgejo instance connectivity.
- `GetPRMeta()`: Collects PR metadata, including state, labels, and comment counts.
- `MergePullRequest()`: Merges a PR using squash, rebase, or merge styles.
- `SetPRDraft()`: Manages draft status via raw HTTP PATCH (working around SDK limitations).
- `MigrateRepo()`: Imports repositories and metadata from external services.
**Dependencies**
- `pkg/log`
- `pkg/config`
**Complexity Rating**
Moderate
---
### Package: `pkg/release`
Orchestrates release automation, including changelog generation, versioning, and publishing to various targets.
**Key Exported Types and Functions**
- `Generate()`: Parses conventional commits to create markdown changelogs.
- `DetermineVersion()`: Calculates the next semantic version based on git tags and commit history.
- `Run()` / `Publish()`: Orchestrates the full process of building, archiving, and distributing artifacts.
- `RunSDK()`: Handles OpenAPI-based SDK generation and breaking change detection.
- `LoadConfig()`: Parses `.core/release.yaml` to configure build targets and publishers.
- `Config`: Struct defining project metadata, build targets, and distribution channels (Docker, Homebrew, etc.).
**Dependencies**
- `pkg/build`
- `pkg/io`
- `pkg/config`
- `pkg/log`
**Complexity Rating**
Complex

303
docs/pkg-batch5-analysis.md Normal file
View file

@ -0,0 +1,303 @@
# Package Analysis — Batch 5
Generated by: gemini-batch-runner.sh
Models: gemini-2.5-flash-lite → gemini-3-flash-preview → gemini-3-pro-preview
Date: 2026-02-09
Packages: agentci agentic ai rag
Total tokens: 78402
---
Here is the detailed documentation for the framework's AI and Agent capabilities.
# Host-UK Core Framework: AI & Agent Packages
This document outlines the architecture, API, and design patterns for the AI automation subsystem within the Core framework. These packages provide the foundation for LLM-assisted development, task management, and RAG (Retrieval Augmented Generation) context.
---
## Package: `pkg/agentci`
### 1. Overview
`pkg/agentci` serves as the configuration bridge between the Core config system and the Agent CI dispatch logic. Its primary purpose is to manage the definitions of "Agent Targets"—machines or environments capable of running AI workloads (e.g., specific GPU nodes or cloud runners)—allowing the job runner to dynamically load and dispatch tasks to active agents.
### 2. Public API
```go
type AgentConfig struct {
Host string `yaml:"host" mapstructure:"host"`
QueueDir string `yaml:"queue_dir" mapstructure:"queue_dir"`
ForgejoUser string `yaml:"forgejo_user" mapstructure:"forgejo_user"`
Model string `yaml:"model" mapstructure:"model"`
Runner string `yaml:"runner" mapstructure:"runner"`
Active bool `yaml:"active" mapstructure:"active"`
}
// LoadAgents reads agent targets from config and returns a map suitable for the dispatch handler.
func LoadAgents(cfg *config.Config) (map[string]handlers.AgentTarget, error)
// SaveAgent writes an agent config entry to the config file.
func SaveAgent(cfg *config.Config, name string, ac AgentConfig) error
// RemoveAgent removes an agent from the config file.
func RemoveAgent(cfg *config.Config, name string) error
// ListAgents returns all configured agents (active and inactive).
func ListAgents(cfg *config.Config) (map[string]AgentConfig, error)
```
### 3. Internal Design
* **Configuration Mapping**: The package acts as a Data Transfer Object (DTO) layer. It maps raw YAML/MapStructure data into the strictly typed `handlers.AgentTarget` struct required by the job runner.
* **Defaults Handling**: `LoadAgents` applies specific logic defaults (e.g., default queue directories, default models like "sonnet") to ensure the system works with minimal configuration.
### 4. Dependencies
* `github.com/host-uk/core/pkg/config`: For reading/writing the persistent configuration state.
* `github.com/host-uk/core/pkg/jobrunner/handlers`: To map local config structs to the runtime types used by the job dispatch system.
### 5. Test Coverage Notes
* **Configuration Persistence**: Tests should verify that `SaveAgent` correctly updates the underlying config file and that `LoadAgents` retrieves it accurately.
* **Validation**: Edge cases where `Host` is empty or defaults are applied need unit testing.
### 6. Integration Points
* **Job Runner**: The main dispatch loop calls `LoadAgents` to determine where AI jobs can be sent.
* **CLI Tools**: CLI commands for managing build agents (e.g., `core agent add`) would use `SaveAgent` and `ListAgents`.
---
## Package: `pkg/agentic`
### 1. Overview
`pkg/agentic` is the heavy-lifting package for AI-assisted task management. It provides both a REST client for the `core-agentic` backend service and a Core Framework Service implementation to execute local AI operations (Git automation, context gathering, and Claude invocations).
### 2. Public API
**Client & API Types**
```go
type Client struct { /* ... */ }
type Task struct { /* ID, Title, Priority, Status, etc. */ }
type TaskContext struct { /* Task, Files, GitStatus, RAGContext, etc. */ }
// Client Factory
func NewClient(baseURL, token string) *Client
func NewClientFromConfig(cfg *Config) *Client
// API Operations
func (c *Client) ListTasks(ctx context.Context, opts ListOptions) ([]Task, error)
func (c *Client) GetTask(ctx context.Context, id string) (*Task, error)
func (c *Client) ClaimTask(ctx context.Context, id string) (*Task, error)
func (c *Client) UpdateTask(ctx context.Context, id string, update TaskUpdate) error
func (c *Client) CompleteTask(ctx context.Context, id string, result TaskResult) error
func (c *Client) Ping(ctx context.Context) error
```
**Git & Automation Ops**
```go
func AutoCommit(ctx context.Context, task *Task, dir string, message string) error
func CreatePR(ctx context.Context, task *Task, dir string, opts PROptions) (string, error)
func CreateBranch(ctx context.Context, task *Task, dir string) (string, error)
func CommitAndSync(ctx context.Context, client *Client, task *Task, dir string, message string, progress int) error
func BuildTaskContext(task *Task, dir string) (*TaskContext, error)
```
**Framework Service**
```go
type Service struct { /* ... */ }
type TaskCommit struct { Path, Name string; CanEdit bool }
type TaskPrompt struct { Prompt, WorkDir string; AllowedTools []string }
func NewService(opts ServiceOptions) func(*framework.Core) (any, error)
```
### 3. Internal Design
* **Service Runtime**: Implements the `framework.ServiceRuntime` pattern, registering task handlers (`TaskCommit`, `TaskPrompt`) allowing other parts of the Core framework to request AI actions via the event bus.
* **Heuristic Context Gathering**: `BuildTaskContext` uses a mix of git commands (`git grep`, `git status`) and file reading to assemble a prompt context for the LLM automatically.
* **Tooling Integration**: Wraps the `claude` CLI binary directly via `os/exec` to perform actual inference, exposing tool capabilities (Bash, Read, Write) based on permissions.
* **Embeds**: Uses Go embed (`//go:embed`) to store system prompts (e.g., `prompts/commit.md`) within the binary.
### 4. Dependencies
* `pkg/framework`: To integrate as a background service.
* `pkg/ai`: Uses `ai.QueryRAGForTask` to inject documentation context into task execution.
* `pkg/config` & `pkg/io`: For loading credentials and file operations.
* `pkg/log`: Structured logging.
### 5. Test Coverage Notes
* **HTTP Client**: Requires mocking `http.Client` to verify request payload serialization and error handling for 4xx/5xx responses.
* **Git Operations**: Needs integration tests with a temporary git repository to verify `AutoCommit` and branch creation logic.
* **Context Building**: Unit tests should verify `extractKeywords` and `GatherRelatedFiles` logic on a known file structure.
### 6. Integration Points
* **Developer CLI**: The `core` CLI uses this package to fetch tasks (`core task list`) and start work (`core task start`).
* **Agents**: Autonomous agents use the `Client` to claim work and the `Service` to execute the necessary code changes.
---
## Package: `pkg/ai`
### 1. Overview
`pkg/ai` is the canonical entry point and facade for the framework's AI capabilities. It unifies RAG (from `pkg/rag`) and metrics collection, providing a simplified interface for other packages to consume AI features without managing low-level clients.
### 2. Public API
```go
// Metrics
type Event struct { /* Type, Timestamp, AgentID, etc. */ }
func Record(event Event) (err error)
func ReadEvents(since time.Time) ([]Event, error)
func Summary(events []Event) map[string]any
// RAG Facade
type TaskInfo struct { Title, Description string }
func QueryRAGForTask(task TaskInfo) string
```
### 3. Internal Design
* **Facade Pattern**: Hides the initialization complexity of `rag.QdrantClient` and `rag.OllamaClient`. `QueryRAGForTask` instantiates these on demand with sensible defaults, ensuring graceful degradation (returns empty string) if services aren't running.
* **Dependency Inversion**: It defines `TaskInfo` locally to accept task data from `pkg/agentic` without importing `pkg/agentic` directly, breaking potential circular dependencies.
* **Local Metrics Store**: Implements a lightweight, file-based (JSONL) telemetry system stored in `~/.core/ai/metrics`.
### 4. Dependencies
* `pkg/rag`: For vector database and embedding operations.
* `pkg/agentic`: (Conceptually composed, though `ai` is the higher-level import).
### 5. Test Coverage Notes
* **Metrics I/O**: Tests for `Record` and `ReadEvents` to ensure concurrent writes to the JSONL file do not corrupt data and dates are filtered correctly.
* **Graceful Failure**: `QueryRAGForTask` must be tested to ensure it does not panic if Qdrant/Ollama are offline.
### 6. Integration Points
* **Agentic Context**: `pkg/agentic` calls `QueryRAGForTask` to enhance prompts.
* **Dashboard**: A UI or CLI dashboard would consume `Summary` to show AI usage stats.
---
## Package: `pkg/rag`
### 1. Overview
`pkg/rag` implements the Retrieval Augmented Generation pipeline. It handles the full lifecycle of document processing: reading Markdown files, chunking them intelligently, generating embeddings via Ollama, storing them in Qdrant, and performing semantic search.
### 2. Public API
**Ingestion & Chunking**
```go
type IngestConfig struct { /* Directory, Collection, ChunkConfig */ }
type Chunk struct { Text, Section string; Index int }
func DefaultIngestConfig() IngestConfig
func Ingest(ctx context.Context, qdrant *QdrantClient, ollama *OllamaClient, cfg IngestConfig, progress IngestProgress) (*IngestStats, error)
func ChunkMarkdown(text string, cfg ChunkConfig) []Chunk
```
**Querying**
```go
type QueryConfig struct { /* Collection, Limit, Threshold */ }
type QueryResult struct { Text, Source, Score float32 /* ... */ }
func Query(ctx context.Context, qdrant *QdrantClient, ollama *OllamaClient, query string, cfg QueryConfig) ([]QueryResult, error)
func FormatResultsContext(results []QueryResult) string
```
**Clients**
```go
type QdrantClient struct { /* ... */ }
func NewQdrantClient(cfg QdrantConfig) (*QdrantClient, error)
type OllamaClient struct { /* ... */ }
func NewOllamaClient(cfg OllamaConfig) (*OllamaClient, error)
func (o *OllamaClient) Embed(ctx context.Context, text string) ([]float32, error)
```
### 3. Internal Design
* **Pipeline Architecture**: Separation of concerns between `Ingest` (Crawler/Loader), `Chunk` (Processor), `Embed` (Transformation), and `Qdrant` (Storage).
* **Semantic Chunking**: `ChunkMarkdown` is designed specifically for documentation, splitting by H2 (`##`) headers first, then by paragraphs, maintaining overlap to preserve context.
* **Adapter Pattern**: Wraps external libraries (`qdrant-go-client`, `ollama/api`) to strictly define the interface required by the Core framework.
### 4. Dependencies
* `github.com/qdrant/go-client/qdrant`: Vector database driver.
* `github.com/ollama/ollama/api`: Embedding model API.
* `pkg/log`: Error reporting.
### 5. Test Coverage Notes
* **Chunking Logic**: Critical to test `ChunkMarkdown` with various markdown structures (headers, lists, code blocks) to ensure chunks don't break mid-sentence or lose header context.
* **Embed Dimensions**: Tests should verify that the vector size created by the Ollama client matches the collection configuration in Qdrant.
* **Integration**: Requires running Qdrant and Ollama containers for full integration testing.
### 6. Integration Points
* **CLI Admin**: An ingestion command (e.g., `core docs ingest`) would trigger the `Ingest` function.
* **AI Package**: `pkg/ai` consumes `Query` to augment prompts.
---
## Quick Reference (Flash Summary)
### Package: `pkg/agentci`
**Description**: Manages configuration and lifecycle for AgentCI dispatch targets and remote runner machines.
**Key Exported Types and Functions**:
- `AgentConfig`: Struct representing an agent's host, queue directory, model, and runner type.
- `LoadAgents`: Reads agent configurations from the global config and maps them to dispatch targets.
- `SaveAgent`: Adds or updates a specific agent entry in the configuration file.
- `RemoveAgent`: Deletes an agent configuration entry by name.
- `ListAgents`: Retrieves all configured agents, including inactive ones.
**Dependencies**:
- `pkg/config`
- `pkg/jobrunner/handlers`
**Complexity**: Simple
---
### Package: `pkg/agentic`
**Description**: Provides an API client and automation tools for AI-assisted task management, git operations, and context gathering.
**Key Exported Types and Functions**:
- `Client`: API client for interacting with the core-agentic task service.
- `Task` / `TaskUpdate`: Data structures representing development tasks and their status updates.
- `BuildTaskContext`: Aggregates task details, relevant file contents, git status, and RAG data for AI consumption.
- `AutoCommit`: Automatically stages changes and creates a git commit with a task reference.
- `CreatePR`: Uses the `gh` CLI to create a pull request based on task metadata.
- `Service`: A framework-compatible service for handling asynchronous AI tasks like automated commits and prompts.
- `LoadConfig`: Multi-source configuration loader (Env, `.env` files, YAML) for API credentials.
**Dependencies**:
- `pkg/log`
- `pkg/config`
- `pkg/io`
- `pkg/ai`
- `pkg/framework`
**Complexity**: Complex
---
### Package: `pkg/ai`
**Description**: Unified entry point for AI features, orchestrating vector search, task context, and usage metrics.
**Key Exported Types and Functions**:
- `Event`: Represents a recorded AI or security metric event for telemetry.
- `Record`: Persists metric events to daily JSONL files in the user's home directory.
- `ReadEvents` / `Summary`: Retrieves and aggregates stored metrics for reporting.
- `QueryRAGForTask`: High-level helper that queries the vector database for documentation relevant to a specific task.
- `TaskInfo`: A minimal structure used to pass task data to the RAG system without circular dependencies.
**Dependencies**:
- `pkg/rag`
**Complexity**: Moderate
---
### Package: `pkg/rag`
**Description**: Implements Retrieval-Augmented Generation (RAG) using Qdrant for vector storage and Ollama for embeddings.
**Key Exported Types and Functions**:
- `QdrantClient`: Wrapper for the Qdrant database providing collection management and vector search.
- `OllamaClient`: Client for generating text embeddings using local models (e.g., `nomic-embed-text`).
- `ChunkMarkdown`: Semantically splits markdown text into smaller chunks based on headers and paragraphs.
- `Ingest`: Processes a directory of markdown files, generates embeddings, and stores them in Qdrant.
- `Query`: Performs vector similarity searches and filters results by score thresholds.
- `FormatResultsContext`: Formats retrieved document chunks into XML-style tags for LLM prompt injection.
**Dependencies**:
- `pkg/log`
**Complexity**: Moderate

520
docs/pkg-batch6-analysis.md Normal file
View file

@ -0,0 +1,520 @@
# Package Analysis — Batch 6
Generated by: gemini-batch-runner.sh
Models: gemini-2.5-flash-lite → gemini-3-flash-preview → gemini-3-pro-preview
Date: 2026-02-09
Packages: ansible deploy devops framework mcp plugin unifi webview ws collect i18n cache
Total tokens: 458153
---
# Framework Documentation
This document provides a detailed technical analysis of the core packages within the framework.
---
## === Package: pkg/ansible ===
### 1. Overview
A native Go implementation of an Ansible playbook runner. Unlike wrappers that call the `ansible` CLI, this package parses YAML playbooks and inventories, handles variable interpolation (Jinja2-style), and executes modules directly over SSH or local connections. It is designed for embedding configuration management directly into the application without external dependencies.
### 2. Public API
#### Executor
The primary entry point for running playbooks.
```go
type Executor struct { ... }
func NewExecutor(basePath string) *Executor
func (e *Executor) SetInventory(path string) error
func (e *Executor) SetInventoryDirect(inv *Inventory)
func (e *Executor) SetVar(key string, value any)
func (e *Executor) Run(ctx context.Context, playbookPath string) error
func (e *Executor) Close()
```
#### Callback Hooks
Callbacks to monitor execution progress.
```go
e.OnPlayStart = func(play *Play)
e.OnTaskStart = func(host string, task *Task)
e.OnTaskEnd = func(host string, task *Task, result *TaskResult)
e.OnPlayEnd = func(play *Play)
```
#### Types
```go
type Playbook struct { Plays []Play }
type Play struct { ... }
type Task struct { ... }
type Inventory struct { ... }
type TaskResult struct { Changed, Failed bool; Msg, Stdout string; ... }
```
### 3. Internal Design
* **Parser**: `parser.go` handles recursive YAML parsing, resolving role paths, and `include_tasks`.
* **Module Dispatch**: `modules.go` contains a switch statement dispatching tasks to native Go functions (e.g., `moduleShell`, `moduleCopy`, `moduleApt`) based on the task name.
* **Templating**: Implements a custom Jinja2-subset parser in `executor.go` (`templateString`, `resolveExpr`) to handle variables like `{{ var }}` and filters.
* **SSH Abstraction**: `ssh.go` wraps `golang.org/x/crypto/ssh` to handle connection pooling, key management, and `sudo` escalation (become).
### 4. Dependencies
* `github.com/host-uk/core/pkg/log`: structured logging.
* `golang.org/x/crypto/ssh`: Underlying SSH transport.
* `gopkg.in/yaml.v3`: YAML parsing.
### 5. Test Coverage Notes
* **Templating Logic**: Critical to test variable resolution, filters (`default`, `bool`), and nested lookups.
* **Module Idempotency**: Verify that file/apt modules return `Changed: false` when state matches.
* **SSH/Sudo**: Test `become` functionality with password handling.
### 6. Integration Points
Used by higher-level orchestration tools or the `pkg/devops` package to provision environments.
---
## === Package: pkg/devops ===
### 1. Overview
Manages a portable, sandboxed development environment using LinuxKit images. It handles the lifecycle (download, install, boot, stop) of a QEMU-based VM and provides utilities to bridge the host and VM (SSH forwarding, file mounting).
### 2. Public API
#### Lifecycle Management
```go
type DevOps struct { ... }
func New(m io.Medium) (*DevOps, error)
func (d *DevOps) Install(ctx, progress func(int64, int64)) error
func (d *DevOps) Boot(ctx, opts BootOptions) error
func (d *DevOps) Stop(ctx) error
func (d *DevOps) Status(ctx) (*DevStatus, error)
```
#### Interaction
```go
func (d *DevOps) Shell(ctx, opts ShellOptions) error
func (d *DevOps) Serve(ctx, projectDir, opts ServeOptions) error
func (d *DevOps) Test(ctx, projectDir, opts TestOptions) error
func (d *DevOps) Claude(ctx, projectDir, opts ClaudeOptions) error
```
### 3. Internal Design
* **Image Management**: `images.go` handles versioning and downloading QCOW2 images from GitHub/CDN.
* **Container Abstraction**: Delegates low-level VM execution to `pkg/container` (likely a wrapper around QEMU/LinuxKit).
* **SSH Bridging**: Heavily relies on `exec.Command("ssh", ...)` to tunnel ports, mount filesystems via SSHFS, and forward agents.
* **Auto-Detection**: `DetectServeCommand` and `DetectTestCommand` inspect project files (`package.json`, `go.mod`) to determine how to run projects.
### 4. Dependencies
* `pkg/container`: VM runtime management.
* `pkg/io`: Filesystem abstraction.
* `pkg/config`: Configuration loading.
### 5. Test Coverage Notes
* **Command Detection**: Unit tests for `Detect*Command` with various mock file structures.
* **SSH Config**: Verify `ensureHostKey` correctly parses and updates `known_hosts`.
### 6. Integration Points
The primary interface for CLI commands (`core dev ...`). Bridges `pkg/mcp` agents into a sandboxed environment.
---
## === Package: pkg/framework ===
### 1. Overview
A facade package that re-exports types and functions from `pkg/framework/core`. It serves as the primary entry point for the Dependency Injection (DI) framework, providing a cleaner import path for consumers.
### 2. Public API
Re-exports `Core`, `Option`, `Message`, `Startable`, `Stoppable`, and constructors like `New`, `WithService`, `ServiceFor`.
### 3. Internal Design
Purely structural; contains type aliases and variable assignments to expose the internal `core` package.
### 4. Dependencies
* `github.com/host-uk/core/pkg/framework/core`
### 5. Test Coverage Notes
No logic to test directly; coverage belongs in `pkg/framework/core`.
### 6. Integration Points
Imported by `main.go` and all service packages to register themselves with the DI container.
---
## === Package: pkg/mcp ===
### 1. Overview
Implements a Model Context Protocol (MCP) server. It acts as a bridge between AI models (like Claude) and the system tools, exposing file operations, process management, web browsing, and RAG capabilities as callable tools.
### 2. Public API
```go
type Service struct { ... }
func New(opts ...Option) (*Service, error)
func (s *Service) Run(ctx context.Context) error
func (s *Service) ServeTCP(ctx, addr string) error
func (s *Service) ServeStdio(ctx) error
```
#### Configuration Options
```go
func WithWorkspaceRoot(root string) Option
func WithProcessService(svc *process.Service) Option
func WithWSHub(hub *ws.Hub) Option
```
### 3. Internal Design
* **Tool Registry**: Registers functions (e.g., `s.readFile`, `s.processStart`) with the MCP SDK.
* **Sandboxing**: `WithWorkspaceRoot` creates a restricted `io.Medium` to prevent AI from accessing files outside the workspace.
* **Subsystems**: Segregates tools into files (`tools_rag.go`, `tools_webview.go`, etc.).
* **Transports**: Supports Stdio (for CLI pipes), TCP, and Unix sockets.
### 4. Dependencies
* `github.com/modelcontextprotocol/go-sdk`: MCP protocol implementation.
* `pkg/process`, `pkg/ws`, `pkg/rag`, `pkg/webview`: Capability providers.
* `pkg/io`: Filesystem access.
### 5. Test Coverage Notes
* **Security**: Verify `WithWorkspaceRoot` actually prevents accessing `/etc/passwd`.
* **Tool I/O**: Ensure JSON inputs/outputs for tools map correctly to internal service calls.
### 6. Integration Points
Runs as a standalone server or subprocess for AI agents. Consumes `pkg/process` and `pkg/webview`.
---
## === Package: pkg/plugin ===
### 1. Overview
Provides a plugin system for the CLI, allowing extensions to be installed from GitHub. It manages a local registry of installed plugins and handles their lifecycle (install, update, remove).
### 2. Public API
```go
type Plugin interface { Name(); Version(); Init(); Start(); Stop() }
type Registry struct { ... }
func NewRegistry(m io.Medium, basePath string) *Registry
func (r *Registry) List() []*PluginConfig
type Installer struct { ... }
func (i *Installer) Install(ctx, source string) error // source: "org/repo"
func (i *Installer) Update(ctx, name string) error
```
### 3. Internal Design
* **Manifest**: Relies on `plugin.json` in the root of the plugin repo.
* **Git Integration**: Uses the `gh` CLI via `exec` to clone/pull repositories.
* **Persistence**: Stores plugin metadata in a `registry.json` file.
### 4. Dependencies
* `pkg/io`: Filesystem access.
* `pkg/framework/core`: Error handling.
* External `gh` and `git` binaries.
### 5. Test Coverage Notes
* **Manifest Validation**: Test valid/invalid `plugin.json` parsing.
* **Source Parsing**: Test parsing of `org/repo`, `org/repo@v1`, etc.
### 6. Integration Points
Used by the main CLI application to load dynamic commands at startup.
---
## === Package: pkg/unifi ===
### 1. Overview
A strongly-typed client for Ubiquiti UniFi controllers. It wraps the `unpoller` SDK but adds configuration resolution (config file -> env var -> flags) and specific helper methods for data extraction not easily accessible in the raw SDK.
### 2. Public API
```go
func NewFromConfig(...) (*Client, error)
func (c *Client) GetClients(filter ClientFilter) ([]*uf.Client, error)
func (c *Client) GetDeviceList(site, type string) ([]DeviceInfo, error)
func (c *Client) GetRoutes(site string) ([]Route, error)
func (c *Client) GetNetworks(site string) ([]NetworkConf, error)
```
### 3. Internal Design
* **Config Cascade**: `ResolveConfig` logic ensures hierarchical configuration overrides.
* **Raw API Fallback**: Methods like `GetRoutes` and `GetNetworks` bypass the SDK's high-level structs to hit specific API endpoints (`/api/s/%s/stat/routing`) and decode into custom structs.
### 4. Dependencies
* `github.com/unpoller/unifi/v5`: Base SDK.
* `pkg/config`: Config file management.
### 5. Test Coverage Notes
* **Config Resolution**: Verify priority order (Flag > Env > Config).
* **JSON Unmarshalling**: Test `GetRoutes`/`GetNetworks` against sample JSON responses from a controller.
### 6. Integration Points
Used by network management plugins or diagnostic tools.
---
## === Package: pkg/webview ===
### 1. Overview
A browser automation package using the Chrome DevTools Protocol (CDP). It is designed for headless testing, scraping, and AI-driven interaction. It supports advanced features like Angular-specific waiting strategies.
### 2. Public API
```go
type Webview struct { ... }
func New(opts ...Option) (*Webview, error)
func (wv *Webview) Navigate(url string) error
func (wv *Webview) Click(selector string) error
func (wv *Webview) Type(selector, text string) error
func (wv *Webview) Screenshot() ([]byte, error)
func (wv *Webview) Evaluate(script string) (any, error)
```
#### Angular Helpers
```go
func NewAngularHelper(wv *Webview) *AngularHelper
func (ah *AngularHelper) WaitForAngular() error
func (ah *AngularHelper) GetNgModel(selector string) (any, error)
```
### 3. Internal Design
* **CDP Client**: `cdp.go` implements a raw WebSocket client for the DevTools protocol, managing message IDs and event dispatching.
* **Action Sequence**: `actions.go` implements the Command pattern (`Action` interface) to chain browser interactions.
* **Angular Awareness**: `angular.go` injects JS to probe `window.ng` or `getAllAngularRootElements` to interact with Angular's Zone.js and component state.
### 4. Dependencies
* `github.com/gorilla/websocket`: WebSocket transport.
### 5. Test Coverage Notes
* **CDP Protocol**: Mock the WebSocket server to ensure correct message serialization/response handling.
* **Angular Helpers**: Requires an actual Angular app (or mock environment) to verify Zone.js stabilization logic.
### 6. Integration Points
Used by `pkg/mcp` to expose browser tools to AI agents.
---
## === Package: pkg/ws ===
### 1. Overview
A concurrent WebSocket hub implementation. It handles client registration, broadcasting, and channel-based subscriptions (e.g., subscribing only to logs for a specific process).
### 2. Public API
```go
type Hub struct { ... }
func NewHub() *Hub
func (h *Hub) Run(ctx)
func (h *Hub) Handler() http.HandlerFunc
func (h *Hub) SendProcessOutput(id, output string) error
func (h *Hub) SendEvent(type string, data any) error
```
### 3. Internal Design
* **Hub Pattern**: Central `Hub` struct manages a map of clients and channels. Uses unbuffered channels for registration to avoid race conditions.
* **Channel Routing**: Maintains a `map[string]map[*Client]bool` to route messages efficiently to subscribers.
* **Goroutines**: Each client spawns a `readPump` and `writePump` to handle I/O concurrently.
### 4. Dependencies
* `github.com/gorilla/websocket`
### 5. Test Coverage Notes
* **Concurrency**: Test registering/unregistering clients while broadcasting heavily.
* **Subscription**: Verify messages only go to subscribed clients.
### 6. Integration Points
Used by `pkg/mcp` to stream process output to a web UI.
---
## === Package: pkg/collect ===
### 1. Overview
A data collection pipeline for gathering data from various sources (GitHub, Forums, Market Data, Papers). It standardizes the collection process into a `Collector` interface and handles common concerns like rate limiting, state tracking (resume support), and formatting.
### 2. Public API
```go
type Collector interface {
Name() string
Collect(ctx, cfg *Config) (*Result, error)
}
type Excavator struct { Collectors []Collector ... }
func (e *Excavator) Run(ctx, cfg) (*Result, error)
```
### 3. Internal Design
* **Excavator**: The orchestrator that runs collectors sequentially.
* **RateLimiter**: Implements token bucket-like delays per source type (e.g., GitHub, CoinGecko).
* **State Persistence**: Saves a JSON cursor file to resume interrupted collections.
* **Formatters**: `process.go` converts raw HTML/JSON into Markdown for easier consumption by LLMs.
### 4. Dependencies
* `pkg/io`: File storage.
* `golang.org/x/net/html`: HTML parsing for forums/papers.
* `gh` CLI: Used for GitHub data fetching.
### 5. Test Coverage Notes
* **HTML Parsing**: Test `ParsePostsFromHTML` with sample forum HTML.
* **Rate Limit**: Verify `Wait` respects context cancellation and time delays.
### 6. Integration Points
Used as a standalone CLI command or by AI agents to gather context.
---
## === Package: pkg/i18n ===
### 1. Overview
A sophisticated internationalization library that goes beyond simple key-value lookups. It includes a grammar engine to handle pluralization, verb conjugation, and semantic sentence generation ("Subject verbed object").
### 2. Public API
```go
func T(key string, args ...any) string // Main translation function
func S(noun string, value any) *Subject // Create a semantic subject
func N(format string, value any) string // Number formatting
func SetLanguage(lang string) error
```
### 3. Internal Design
* **Grammar Engine**: `grammar.go` applies rules for past tense, gerunds, and pluralization based on language-specific JSON rules or algorithmic fallbacks.
* **Namespace Handlers**: `handler.go` intercepts keys like `i18n.count.*` or `i18n.done.*` to auto-generate phrases based on the grammar engine.
* **Loader**: `loader.go` flattens nested JSON translation files and extracts grammar rules (`gram.verb.*`).
### 4. Dependencies
* `golang.org/x/text/language`: Standard language tag parsing.
### 5. Test Coverage Notes
* **Pluralization**: Test complex rules (e.g., Slavic/Arabic plural categories).
* **Grammar generation**: Test `PastTense` and `Gerund` for regular and irregular English verbs.
### 6. Integration Points
Used pervasively across the CLI for all user-facing output.
---
## === Package: pkg/cache ===
### 1. Overview
A simple, file-based JSON cache with Time-To-Live (TTL) support.
### 2. Public API
```go
func New(m io.Medium, baseDir string, ttl time.Duration) (*Cache, error)
func (c *Cache) Get(key string, dest interface{}) (bool, error)
func (c *Cache) Set(key string, data interface{}) error
```
### 3. Internal Design
* Stores data as JSON files: `{ "data": ..., "expires_at": ... }`.
* Uses `pkg/io` abstraction for storage independence.
### 4. Dependencies
* `pkg/io`
### 5. Test Coverage Notes
* **Expiry**: Verify `Get` returns false after TTL expires.
* **Serialization**: Ensure struct round-tripping works correctly.
### 6. Integration Points
Used by `pkg/collect` or `pkg/plugin` to cache API responses (e.g., GitHub releases).
---
## Quick Reference (Flash Summary)
### pkg/ansible
**Description:** Implements a Go-based Ansible-lite engine for executing playbooks and roles over SSH with YAML parsing and fact gathering.
- **Executor (Type):** Main runner that manages inventory, variables, and execution state.
- **NewExecutor (Func):** Initialises the executor with a base path for roles and playbooks.
- **Task (Type):** Represents a single Ansible task with module parameters and conditional logic.
- **Run (Func):** Parses and executes a playbook from a file path.
- **Inventory (Type):** Holds the host and group structure for targeting remote machines.
**Dependencies:** `pkg/log`
**Complexity:** Complex
### pkg/devops
**Description:** Manages a portable development environment using LinuxKit VM images and QEMU/SSH integration.
- **DevOps (Type):** Core service for environment lifecycle, mounting, and tool execution.
- **Boot (Func):** Configures and starts the dev environment container.
- **Claude (Func):** Launches a sandboxed AI session with project mounting and auth forwarding.
- **Serve (Func):** Auto-detects project types and runs local development servers inside the VM.
- **ImageManager (Type):** Handles downloading and updating dev environment system images.
**Dependencies:** `pkg/config`, `pkg/container`, `pkg/io`, `pkg/devops/sources`
**Complexity:** Moderate
### pkg/framework
**Description:** Provides a facade for the Core dependency injection and service runtime framework.
- **Core (Type):** The central DI container and service registry.
- **New (Func):** Creates and initialises a new Core instance.
- **ServiceFor (Func):** Retrieves a type-safe service from the container by name.
- **Runtime (Type):** Manages the lifecycle and configuration of application services.
**Dependencies:** `pkg/framework/core`
**Complexity:** Simple
### pkg/mcp
**Description:** Implements a Model Context Protocol (MCP) server providing filesystem, process, and RAG tools to AI agents.
- **Service (Type):** The MCP server instance managing tools and transport.
- **New (Func):** Initialises the server with workspace sandboxing and optional service integration.
- **WithWorkspaceRoot (Option):** Restricts file operations to a specific directory for security.
- **ServeTCP / ServeStdio (Func):** Transport-specific server implementations.
- **Subsystem (Interface):** Allows external packages to register custom toolsets.
**Dependencies:** `pkg/io`, `pkg/io/local`, `pkg/log`, `pkg/process`, `pkg/ws`, `pkg/ai`, `pkg/rag`, `pkg/webview`
**Complexity:** Complex
### pkg/plugin
**Description:** A dynamic plugin system that manages gits-based extensions for the core CLI.
- **Plugin (Interface):** Defines the lifecycle (Init/Start/Stop) for extensions.
- **Registry (Type):** Manages metadata and persistence for installed plugins.
- **Installer (Type):** Handles git-based installation and updates from GitHub.
- **Loader (Type):** Discovers and loads plugin manifests from the filesystem.
**Dependencies:** `pkg/framework/core`, `pkg/io`
**Complexity:** Moderate
### pkg/unifi
**Description:** A wrapper for the UniFi SDK providing simplified access to network controller devices, clients, and routing.
- **Client (Type):** Main API client for interacting with UniFi controllers.
- **NewFromConfig (Func):** Resolves credentials from config/env and initialises a client.
- **GetClients (Func):** Returns a filtered list of connected wired and wireless clients.
- **GetDeviceList (Func):** Returns flat metadata for infrastructure hardware (APs, Switches, Gateways).
**Dependencies:** `pkg/config`, `pkg/log`
**Complexity:** Simple
### pkg/webview
**Description:** Provides browser automation and framework-specific testing (Angular) via the Chrome DevTools Protocol (CDP).
- **Webview (Type):** High-level controller for browser navigation and interaction.
- **CDPClient (Type):** Manages raw WebSocket communication with Chrome.
- **AngularHelper (Type):** Specialized tools for waiting on Zone.js stability and interacting with Angular components.
- **ConsoleWatcher (Type):** Captures and filters browser console logs and exceptions.
- **ActionSequence (Type):** Chains multiple browser interactions (click, type, navigate) into a single execution.
**Dependencies:** None
**Complexity:** Complex
### pkg/ws
**Description:** Implements a WebSocket hub for real-time message broadcasting and channel-based subscriptions.
- **Hub (Type):** Manages client connections, message loops, and channel routing.
- **Run (Func):** Starts the central event loop for broadcasting and registration.
- **Broadcast (Func):** Sends a message to every connected client.
- **SendToChannel (Func):** Targets messages to clients subscribed to specific topics (e.g., process logs).
**Dependencies:** None
**Complexity:** Moderate
### pkg/collect
**Description:** An orchestration subsystem for scraping and processing data from GitHub, forums, market APIs, and academic sources.
- **Collector (Interface):** Standard interface for data sources (e.g., `GitHubCollector`, `BitcoinTalkCollector`).
- **Excavator (Type):** Orchestrates multiple collectors with rate limiting and state resume support.
- **Processor (Type):** Converts raw HTML/JSON data into cleaned Markdown files.
- **RateLimiter (Type):** Manages per-source API delays to prevent IP bans.
- **State (Type):** Persists progress to allow incremental collection runs.
**Dependencies:** `pkg/framework/core`, `pkg/io`
**Complexity:** Complex
### pkg/i18n
**Description:** A localization engine supporting nested translations, grammatical rules (plurals, gender, verbs), and semantic composition.
- **Service (Type):** Manages loaded locales and message resolution logic.
- **T / Raw (Func):** Translates keys with or without automatic grammatical composition.
- **Subject (Type):** Provides context (count, gender, formality) for semantic intent templates.
- **RegisterLocales (Func):** Allows packages to register embedded translation files.
- **GrammarData (Type):** Defines language-specific rules for past tense, gerunds, and articles.
**Dependencies:** None
**Complexity:** Complex
### pkg/cache
**Description:** Provides a persistent, file-based JSON cache with TTL-based expiration.
- **Cache (Type):** Main handler for storing and retrieving cached entries.
- **Entry (Type):** Internal wrapper for data, including cached and expiry timestamps.
- **Get / Set (Func):** Thread-safe operations for managing cached data.
- **Age (Func):** Calculates how long an item has been stored.
**Dependencies:** `pkg/io`
**Complexity:** Simple

View file

@ -36,7 +36,7 @@ func loadConfig() (*config.Config, error) {
func agentAddCmd() *cli.Command { func agentAddCmd() *cli.Command {
cmd := &cli.Command{ cmd := &cli.Command{
Use: "add <name> <user@host>", Use: "add <name> <user@host>",
Short: "Add an agent to the config", Short: "Add an agent to the config and verify SSH",
Args: cli.ExactArgs(2), Args: cli.ExactArgs(2),
RunE: func(cmd *cli.Command, args []string) error { RunE: func(cmd *cli.Command, args []string) error {
name := args[0] name := args[0]
@ -50,14 +50,38 @@ func agentAddCmd() *cli.Command {
if queueDir == "" { if queueDir == "" {
queueDir = "/home/claude/ai-work/queue" queueDir = "/home/claude/ai-work/queue"
} }
model, _ := cmd.Flags().GetString("model")
dualRun, _ := cmd.Flags().GetBool("dual-run")
// Test SSH connectivity. // Scan and add host key to known_hosts.
// TODO: Replace exec ssh with charmbracelet/ssh native Go client + keygen. parts := strings.Split(host, "@")
hostname := parts[len(parts)-1]
fmt.Printf("Scanning host key for %s... ", hostname)
scanCmd := exec.Command("ssh-keyscan", "-H", hostname)
keys, err := scanCmd.Output()
if err != nil {
fmt.Println(errorStyle.Render("FAILED"))
return fmt.Errorf("failed to scan host keys: %w", err)
}
home, _ := os.UserHomeDir()
knownHostsPath := filepath.Join(home, ".ssh", "known_hosts")
f, err := os.OpenFile(knownHostsPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)
if err != nil {
return fmt.Errorf("failed to open known_hosts: %w", err)
}
if _, err := f.Write(keys); err != nil {
f.Close()
return fmt.Errorf("failed to write known_hosts: %w", err)
}
f.Close()
fmt.Println(successStyle.Render("OK"))
// Test SSH with strict host key checking.
fmt.Printf("Testing SSH to %s... ", host) fmt.Printf("Testing SSH to %s... ", host)
out, err := exec.Command("ssh", testCmd := agentci.SecureSSHCommand(host, "echo ok")
"-o", "StrictHostKeyChecking=accept-new", out, err := testCmd.CombinedOutput()
"-o", "ConnectTimeout=10",
host, "echo ok").CombinedOutput()
if err != nil { if err != nil {
fmt.Println(errorStyle.Render("FAILED")) fmt.Println(errorStyle.Render("FAILED"))
return fmt.Errorf("SSH failed: %s", strings.TrimSpace(string(out))) return fmt.Errorf("SSH failed: %s", strings.TrimSpace(string(out)))
@ -73,6 +97,8 @@ func agentAddCmd() *cli.Command {
Host: host, Host: host,
QueueDir: queueDir, QueueDir: queueDir,
ForgejoUser: forgejoUser, ForgejoUser: forgejoUser,
Model: model,
DualRun: dualRun,
Active: true, Active: true,
} }
if err := agentci.SaveAgent(cfg, name, ac); err != nil { if err := agentci.SaveAgent(cfg, name, ac); err != nil {
@ -85,6 +111,8 @@ func agentAddCmd() *cli.Command {
} }
cmd.Flags().String("forgejo-user", "", "Forgejo username (defaults to agent name)") cmd.Flags().String("forgejo-user", "", "Forgejo username (defaults to agent name)")
cmd.Flags().String("queue-dir", "", "Remote queue directory (default: /home/claude/ai-work/queue)") cmd.Flags().String("queue-dir", "", "Remote queue directory (default: /home/claude/ai-work/queue)")
cmd.Flags().String("model", "sonnet", "Primary AI model")
cmd.Flags().Bool("dual-run", false, "Enable Clotho dual-run verification")
return cmd return cmd
} }
@ -108,22 +136,21 @@ func agentListCmd() *cli.Command {
return nil return nil
} }
table := cli.NewTable("NAME", "HOST", "FORGEJO USER", "ACTIVE", "QUEUE") table := cli.NewTable("NAME", "HOST", "MODEL", "DUAL", "ACTIVE", "QUEUE")
for name, ac := range agents { for name, ac := range agents {
active := dimStyle.Render("no") active := dimStyle.Render("no")
if ac.Active { if ac.Active {
active = successStyle.Render("yes") active = successStyle.Render("yes")
} }
dual := dimStyle.Render("no")
if ac.DualRun {
dual = successStyle.Render("yes")
}
// Quick SSH check for queue depth. // Quick SSH check for queue depth.
// TODO: Replace exec ssh with charmbracelet/ssh native Go client.
queue := dimStyle.Render("-") queue := dimStyle.Render("-")
out, err := exec.Command("ssh", checkCmd := agentci.SecureSSHCommand(ac.Host, fmt.Sprintf("ls %s/ticket-*.json 2>/dev/null | wc -l", ac.QueueDir))
"-o", "StrictHostKeyChecking=accept-new", out, err := checkCmd.Output()
"-o", "ConnectTimeout=5",
ac.Host,
fmt.Sprintf("ls %s/ticket-*.json 2>/dev/null | wc -l", ac.QueueDir),
).Output()
if err == nil { if err == nil {
n := strings.TrimSpace(string(out)) n := strings.TrimSpace(string(out))
if n != "0" { if n != "0" {
@ -133,7 +160,7 @@ func agentListCmd() *cli.Command {
} }
} }
table.AddRow(name, ac.Host, ac.ForgejoUser, active, queue) table.AddRow(name, ac.Host, ac.Model, dual, active, queue)
} }
table.Render() table.Render()
return nil return nil
@ -182,11 +209,7 @@ func agentStatusCmd() *cli.Command {
fi fi
` `
// TODO: Replace exec ssh with charmbracelet/ssh native Go client. sshCmd := agentci.SecureSSHCommand(ac.Host, script)
sshCmd := exec.Command("ssh",
"-o", "StrictHostKeyChecking=accept-new",
"-o", "ConnectTimeout=10",
ac.Host, script)
sshCmd.Stdout = os.Stdout sshCmd.Stdout = os.Stdout
sshCmd.Stderr = os.Stderr sshCmd.Stderr = os.Stderr
return sshCmd.Run() return sshCmd.Run()
@ -218,19 +241,12 @@ func agentLogsCmd() *cli.Command {
return fmt.Errorf("agent %q not found", name) return fmt.Errorf("agent %q not found", name)
} }
// TODO: Replace exec ssh with charmbracelet/ssh native Go client. remoteCmd := fmt.Sprintf("tail -n %d ~/ai-work/logs/runner.log", lines)
tailArgs := []string{
"-o", "StrictHostKeyChecking=accept-new",
"-o", "ConnectTimeout=10",
ac.Host,
}
if follow { if follow {
tailArgs = append(tailArgs, fmt.Sprintf("tail -f -n %d ~/ai-work/logs/runner.log", lines)) remoteCmd = fmt.Sprintf("tail -f -n %d ~/ai-work/logs/runner.log", lines)
} else {
tailArgs = append(tailArgs, fmt.Sprintf("tail -n %d ~/ai-work/logs/runner.log", lines))
} }
sshCmd := exec.Command("ssh", tailArgs...) sshCmd := agentci.SecureSSHCommand(ac.Host, remoteCmd)
sshCmd.Stdout = os.Stdout sshCmd.Stdout = os.Stdout
sshCmd.Stderr = os.Stderr sshCmd.Stderr = os.Stderr
sshCmd.Stdin = os.Stdin sshCmd.Stdin = os.Stdin
@ -307,7 +323,6 @@ func agentRemoveCmd() *cli.Command {
// findSetupScript looks for agent-setup.sh in common locations. // findSetupScript looks for agent-setup.sh in common locations.
func findSetupScript() string { func findSetupScript() string {
// Relative to executable.
exe, _ := os.Executable() exe, _ := os.Executable()
if exe != "" { if exe != "" {
dir := filepath.Dir(exe) dir := filepath.Dir(exe)
@ -322,7 +337,6 @@ func findSetupScript() string {
} }
} }
// Working directory.
cwd, _ := os.Getwd() cwd, _ := os.Getwd()
if cwd != "" { if cwd != "" {
p := filepath.Join(cwd, "scripts", "agent-setup.sh") p := filepath.Join(cwd, "scripts", "agent-setup.sh")
@ -333,4 +347,3 @@ func findSetupScript() string {
return "" return ""
} }

View file

@ -69,6 +69,12 @@ func initCommands() {
// Add agent management commands (core ai agent ...) // Add agent management commands (core ai agent ...)
AddAgentCommands(aiCmd) AddAgentCommands(aiCmd)
// Add rate limit management commands (core ai ratelimits ...)
AddRateLimitCommands(aiCmd)
// Add dispatch commands (core ai dispatch run/watch/status)
AddDispatchCommands(aiCmd)
} }
// AddAICommands registers the 'ai' command and all subcommands. // AddAICommands registers the 'ai' command and all subcommands.

View file

@ -0,0 +1,498 @@
package ai
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"os/exec"
"os/signal"
"path/filepath"
"sort"
"strconv"
"strings"
"syscall"
"time"
"github.com/host-uk/core/pkg/cli"
"github.com/host-uk/core/pkg/log"
)
// AddDispatchCommands registers the 'dispatch' subcommand group under 'ai'.
// These commands run ON the agent machine to process the work queue.
func AddDispatchCommands(parent *cli.Command) {
dispatchCmd := &cli.Command{
Use: "dispatch",
Short: "Agent work queue processor (runs on agent machine)",
}
dispatchCmd.AddCommand(dispatchRunCmd())
dispatchCmd.AddCommand(dispatchWatchCmd())
dispatchCmd.AddCommand(dispatchStatusCmd())
parent.AddCommand(dispatchCmd)
}
// dispatchTicket represents the work item JSON structure.
type dispatchTicket struct {
ID string `json:"id"`
RepoOwner string `json:"repo_owner"`
RepoName string `json:"repo_name"`
IssueNumber int `json:"issue_number"`
IssueTitle string `json:"issue_title"`
IssueBody string `json:"issue_body"`
TargetBranch string `json:"target_branch"`
EpicNumber int `json:"epic_number"`
ForgeURL string `json:"forge_url"`
ForgeToken string `json:"forge_token"`
ForgeUser string `json:"forgejo_user"`
Model string `json:"model"`
Runner string `json:"runner"`
Timeout string `json:"timeout"`
CreatedAt string `json:"created_at"`
}
const (
defaultWorkDir = "ai-work"
lockFileName = ".runner.lock"
)
type runnerPaths struct {
root string
queue string
active string
done string
logs string
jobs string
lock string
}
func getPaths(baseDir string) runnerPaths {
if baseDir == "" {
home, _ := os.UserHomeDir()
baseDir = filepath.Join(home, defaultWorkDir)
}
return runnerPaths{
root: baseDir,
queue: filepath.Join(baseDir, "queue"),
active: filepath.Join(baseDir, "active"),
done: filepath.Join(baseDir, "done"),
logs: filepath.Join(baseDir, "logs"),
jobs: filepath.Join(baseDir, "jobs"),
lock: filepath.Join(baseDir, lockFileName),
}
}
func dispatchRunCmd() *cli.Command {
cmd := &cli.Command{
Use: "run",
Short: "Process a single ticket from the queue",
RunE: func(cmd *cli.Command, args []string) error {
workDir, _ := cmd.Flags().GetString("work-dir")
paths := getPaths(workDir)
if err := ensureDispatchDirs(paths); err != nil {
return err
}
if err := acquireLock(paths.lock); err != nil {
log.Info("Runner locked, skipping run", "lock", paths.lock)
return nil
}
defer releaseLock(paths.lock)
ticketFile, err := pickOldestTicket(paths.queue)
if err != nil {
return err
}
if ticketFile == "" {
return nil
}
return processTicket(paths, ticketFile)
},
}
cmd.Flags().String("work-dir", "", "Working directory (default: ~/ai-work)")
return cmd
}
func dispatchWatchCmd() *cli.Command {
cmd := &cli.Command{
Use: "watch",
Short: "Run as a daemon, polling the queue",
RunE: func(cmd *cli.Command, args []string) error {
workDir, _ := cmd.Flags().GetString("work-dir")
interval, _ := cmd.Flags().GetDuration("interval")
paths := getPaths(workDir)
if err := ensureDispatchDirs(paths); err != nil {
return err
}
log.Info("Starting dispatch watcher", "dir", paths.root, "interval", interval)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
ticker := time.NewTicker(interval)
defer ticker.Stop()
runCycle(paths)
for {
select {
case <-ticker.C:
runCycle(paths)
case <-sigChan:
log.Info("Shutting down watcher...")
return nil
case <-ctx.Done():
return nil
}
}
},
}
cmd.Flags().String("work-dir", "", "Working directory (default: ~/ai-work)")
cmd.Flags().Duration("interval", 5*time.Minute, "Polling interval")
return cmd
}
func dispatchStatusCmd() *cli.Command {
cmd := &cli.Command{
Use: "status",
Short: "Show runner status",
RunE: func(cmd *cli.Command, args []string) error {
workDir, _ := cmd.Flags().GetString("work-dir")
paths := getPaths(workDir)
lockStatus := "IDLE"
if data, err := os.ReadFile(paths.lock); err == nil {
pidStr := strings.TrimSpace(string(data))
pid, _ := strconv.Atoi(pidStr)
if isProcessAlive(pid) {
lockStatus = fmt.Sprintf("RUNNING (PID %d)", pid)
} else {
lockStatus = fmt.Sprintf("STALE (PID %d)", pid)
}
}
countFiles := func(dir string) int {
entries, _ := os.ReadDir(dir)
count := 0
for _, e := range entries {
if !e.IsDir() && strings.HasPrefix(e.Name(), "ticket-") {
count++
}
}
return count
}
fmt.Println("=== Agent Dispatch Status ===")
fmt.Printf("Work Dir: %s\n", paths.root)
fmt.Printf("Status: %s\n", lockStatus)
fmt.Printf("Queue: %d\n", countFiles(paths.queue))
fmt.Printf("Active: %d\n", countFiles(paths.active))
fmt.Printf("Done: %d\n", countFiles(paths.done))
return nil
},
}
cmd.Flags().String("work-dir", "", "Working directory (default: ~/ai-work)")
return cmd
}
func runCycle(paths runnerPaths) {
if err := acquireLock(paths.lock); err != nil {
log.Debug("Runner locked, skipping cycle")
return
}
defer releaseLock(paths.lock)
ticketFile, err := pickOldestTicket(paths.queue)
if err != nil {
log.Error("Failed to pick ticket", "error", err)
return
}
if ticketFile == "" {
return
}
if err := processTicket(paths, ticketFile); err != nil {
log.Error("Failed to process ticket", "file", ticketFile, "error", err)
}
}
func processTicket(paths runnerPaths, ticketPath string) error {
fileName := filepath.Base(ticketPath)
log.Info("Processing ticket", "file", fileName)
activePath := filepath.Join(paths.active, fileName)
if err := os.Rename(ticketPath, activePath); err != nil {
return fmt.Errorf("failed to move ticket to active: %w", err)
}
data, err := os.ReadFile(activePath)
if err != nil {
return fmt.Errorf("failed to read ticket: %w", err)
}
var t dispatchTicket
if err := json.Unmarshal(data, &t); err != nil {
return fmt.Errorf("failed to unmarshal ticket: %w", err)
}
jobDir := filepath.Join(paths.jobs, fmt.Sprintf("%s-%s-%d", t.RepoOwner, t.RepoName, t.IssueNumber))
repoDir := filepath.Join(jobDir, t.RepoName)
if err := os.MkdirAll(jobDir, 0755); err != nil {
return err
}
if err := prepareRepo(t, repoDir); err != nil {
reportToForge(t, false, fmt.Sprintf("Git setup failed: %v", err))
moveToDone(paths, activePath, fileName)
return err
}
prompt := buildPrompt(t)
logFile := filepath.Join(paths.logs, fmt.Sprintf("%s-%s-%d.log", t.RepoOwner, t.RepoName, t.IssueNumber))
success, exitCode, runErr := runAgent(t, prompt, repoDir, logFile)
msg := fmt.Sprintf("Agent completed work on #%d. Exit code: %d.", t.IssueNumber, exitCode)
if !success {
msg = fmt.Sprintf("Agent failed on #%d (exit code: %d). Check logs on agent machine.", t.IssueNumber, exitCode)
if runErr != nil {
msg += fmt.Sprintf(" Error: %v", runErr)
}
}
reportToForge(t, success, msg)
moveToDone(paths, activePath, fileName)
log.Info("Ticket complete", "id", t.ID, "success", success)
return nil
}
func prepareRepo(t dispatchTicket, repoDir string) error {
user := t.ForgeUser
if user == "" {
host, _ := os.Hostname()
user = fmt.Sprintf("%s-%s", host, os.Getenv("USER"))
}
cleanURL := strings.TrimPrefix(t.ForgeURL, "https://")
cleanURL = strings.TrimPrefix(cleanURL, "http://")
cloneURL := fmt.Sprintf("https://%s:%s@%s/%s/%s.git", user, t.ForgeToken, cleanURL, t.RepoOwner, t.RepoName)
if _, err := os.Stat(filepath.Join(repoDir, ".git")); err == nil {
log.Info("Updating existing repo", "dir", repoDir)
cmds := [][]string{
{"git", "fetch", "origin"},
{"git", "checkout", t.TargetBranch},
{"git", "pull", "origin", t.TargetBranch},
}
for _, args := range cmds {
cmd := exec.Command(args[0], args[1:]...)
cmd.Dir = repoDir
if out, err := cmd.CombinedOutput(); err != nil {
if args[1] == "checkout" {
createCmd := exec.Command("git", "checkout", "-b", t.TargetBranch, "origin/"+t.TargetBranch)
createCmd.Dir = repoDir
if _, err2 := createCmd.CombinedOutput(); err2 == nil {
continue
}
}
return fmt.Errorf("git command %v failed: %s", args, string(out))
}
}
} else {
log.Info("Cloning repo", "url", t.RepoOwner+"/"+t.RepoName)
cmd := exec.Command("git", "clone", "-b", t.TargetBranch, cloneURL, repoDir)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("git clone failed: %s", string(out))
}
}
return nil
}
func buildPrompt(t dispatchTicket) string {
return fmt.Sprintf(`You are working on issue #%d in %s/%s.
Title: %s
Description:
%s
The repo is cloned at the current directory on branch '%s'.
Create a feature branch from '%s', make minimal targeted changes, commit referencing #%d, and push.
Then create a PR targeting '%s' using the forgejo MCP tools or git push.`,
t.IssueNumber, t.RepoOwner, t.RepoName,
t.IssueTitle,
t.IssueBody,
t.TargetBranch,
t.TargetBranch, t.IssueNumber,
t.TargetBranch,
)
}
func runAgent(t dispatchTicket, prompt, dir, logPath string) (bool, int, error) {
timeout := 30 * time.Minute
if t.Timeout != "" {
if d, err := time.ParseDuration(t.Timeout); err == nil {
timeout = d
}
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
model := t.Model
if model == "" {
model = "sonnet"
}
log.Info("Running agent", "runner", t.Runner, "model", model)
// For Gemini runner, wrap with rate limiting.
if t.Runner == "gemini" {
return executeWithRateLimit(ctx, model, prompt, func() (bool, int, error) {
return execAgent(ctx, t.Runner, model, prompt, dir, logPath)
})
}
return execAgent(ctx, t.Runner, model, prompt, dir, logPath)
}
func execAgent(ctx context.Context, runner, model, prompt, dir, logPath string) (bool, int, error) {
var cmd *exec.Cmd
switch runner {
case "codex":
cmd = exec.CommandContext(ctx, "codex", "exec", "--full-auto", prompt)
case "gemini":
args := []string{"-p", "-", "-y", "-m", model}
cmd = exec.CommandContext(ctx, "gemini", args...)
cmd.Stdin = strings.NewReader(prompt)
default: // claude
cmd = exec.CommandContext(ctx, "claude", "-p", "--model", model, "--dangerously-skip-permissions", "--output-format", "text")
cmd.Stdin = strings.NewReader(prompt)
}
cmd.Dir = dir
f, err := os.Create(logPath)
if err != nil {
return false, -1, err
}
defer f.Close()
cmd.Stdout = f
cmd.Stderr = f
if err := cmd.Run(); err != nil {
exitCode := -1
if exitErr, ok := err.(*exec.ExitError); ok {
exitCode = exitErr.ExitCode()
}
return false, exitCode, err
}
return true, 0, nil
}
func reportToForge(t dispatchTicket, success bool, body string) {
url := fmt.Sprintf("%s/api/v1/repos/%s/%s/issues/%d/comments",
strings.TrimSuffix(t.ForgeURL, "/"), t.RepoOwner, t.RepoName, t.IssueNumber)
payload := map[string]string{"body": body}
jsonBody, _ := json.Marshal(payload)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonBody))
if err != nil {
log.Error("Failed to create request", "err", err)
return
}
req.Header.Set("Authorization", "token "+t.ForgeToken)
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
log.Error("Failed to report to Forge", "err", err)
return
}
defer resp.Body.Close()
if resp.StatusCode >= 300 {
log.Warn("Forge reported error", "status", resp.Status)
}
}
func moveToDone(paths runnerPaths, activePath, fileName string) {
donePath := filepath.Join(paths.done, fileName)
if err := os.Rename(activePath, donePath); err != nil {
log.Error("Failed to move ticket to done", "err", err)
}
}
func ensureDispatchDirs(p runnerPaths) error {
dirs := []string{p.queue, p.active, p.done, p.logs, p.jobs}
for _, d := range dirs {
if err := os.MkdirAll(d, 0755); err != nil {
return fmt.Errorf("mkdir %s failed: %w", d, err)
}
}
return nil
}
func acquireLock(lockPath string) error {
if data, err := os.ReadFile(lockPath); err == nil {
pidStr := strings.TrimSpace(string(data))
pid, _ := strconv.Atoi(pidStr)
if isProcessAlive(pid) {
return fmt.Errorf("locked by PID %d", pid)
}
log.Info("Removing stale lock", "pid", pid)
_ = os.Remove(lockPath)
}
return os.WriteFile(lockPath, []byte(fmt.Sprintf("%d", os.Getpid())), 0644)
}
func releaseLock(lockPath string) {
_ = os.Remove(lockPath)
}
func isProcessAlive(pid int) bool {
if pid <= 0 {
return false
}
process, err := os.FindProcess(pid)
if err != nil {
return false
}
return process.Signal(syscall.Signal(0)) == nil
}
func pickOldestTicket(queueDir string) (string, error) {
entries, err := os.ReadDir(queueDir)
if err != nil {
return "", err
}
var tickets []string
for _, e := range entries {
if !e.IsDir() && strings.HasPrefix(e.Name(), "ticket-") && strings.HasSuffix(e.Name(), ".json") {
tickets = append(tickets, filepath.Join(queueDir, e.Name()))
}
}
if len(tickets) == 0 {
return "", nil
}
sort.Strings(tickets)
return tickets[0], nil
}

View file

@ -0,0 +1,213 @@
package ai
import (
"fmt"
"os"
"strconv"
"text/tabwriter"
"time"
"github.com/host-uk/core/pkg/cli"
"github.com/host-uk/core/pkg/config"
"github.com/host-uk/core/pkg/ratelimit"
)
// AddRateLimitCommands registers the 'ratelimits' subcommand group under 'ai'.
func AddRateLimitCommands(parent *cli.Command) {
rlCmd := &cli.Command{
Use: "ratelimits",
Short: "Manage Gemini API rate limits",
}
rlCmd.AddCommand(rlShowCmd())
rlCmd.AddCommand(rlResetCmd())
rlCmd.AddCommand(rlCountCmd())
rlCmd.AddCommand(rlConfigCmd())
rlCmd.AddCommand(rlCheckCmd())
parent.AddCommand(rlCmd)
}
func rlShowCmd() *cli.Command {
return &cli.Command{
Use: "show",
Short: "Show current rate limit usage",
RunE: func(cmd *cli.Command, args []string) error {
rl, err := ratelimit.New()
if err != nil {
return err
}
if err := rl.Load(); err != nil {
return err
}
stats := rl.AllStats()
w := tabwriter.NewWriter(os.Stdout, 0, 0, 3, ' ', 0)
fmt.Fprintln(w, "MODEL\tRPM\tTPM\tRPD\tSTATUS")
for model, s := range stats {
rpmStr := fmt.Sprintf("%d/%s", s.RPM, formatLimit(s.MaxRPM))
tpmStr := fmt.Sprintf("%d/%s", s.TPM, formatLimit(s.MaxTPM))
rpdStr := fmt.Sprintf("%d/%s", s.RPD, formatLimit(s.MaxRPD))
status := "OK"
if (s.MaxRPM > 0 && s.RPM >= s.MaxRPM) ||
(s.MaxTPM > 0 && s.TPM >= s.MaxTPM) ||
(s.MaxRPD > 0 && s.RPD >= s.MaxRPD) {
status = "LIMITED"
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", model, rpmStr, tpmStr, rpdStr, status)
}
w.Flush()
return nil
},
}
}
func rlResetCmd() *cli.Command {
return &cli.Command{
Use: "reset [model]",
Short: "Reset usage counters for a model (or all)",
RunE: func(cmd *cli.Command, args []string) error {
rl, err := ratelimit.New()
if err != nil {
return err
}
if err := rl.Load(); err != nil {
return err
}
model := ""
if len(args) > 0 {
model = args[0]
}
rl.Reset(model)
if err := rl.Persist(); err != nil {
return err
}
if model == "" {
fmt.Println("Reset stats for all models.")
} else {
fmt.Printf("Reset stats for model %q.\n", model)
}
return nil
},
}
}
func rlCountCmd() *cli.Command {
return &cli.Command{
Use: "count <model> <text>",
Short: "Count tokens for text using Gemini API",
Args: cli.ExactArgs(2),
RunE: func(cmd *cli.Command, args []string) error {
model := args[0]
text := args[1]
cfg, err := config.New()
if err != nil {
return err
}
var apiKey string
if err := cfg.Get("agentci.gemini_api_key", &apiKey); err != nil || apiKey == "" {
apiKey = os.Getenv("GEMINI_API_KEY")
}
if apiKey == "" {
return fmt.Errorf("GEMINI_API_KEY not found in config or env")
}
count, err := ratelimit.CountTokens(apiKey, model, text)
if err != nil {
return err
}
fmt.Printf("Model: %s\nTokens: %d\n", model, count)
return nil
},
}
}
func rlConfigCmd() *cli.Command {
return &cli.Command{
Use: "config",
Short: "Show configured quotas",
RunE: func(cmd *cli.Command, args []string) error {
rl, err := ratelimit.New()
if err != nil {
return err
}
w := tabwriter.NewWriter(os.Stdout, 0, 0, 3, ' ', 0)
fmt.Fprintln(w, "MODEL\tMAX RPM\tMAX TPM\tMAX RPD")
for model, q := range rl.Quotas {
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n",
model,
formatLimit(q.MaxRPM),
formatLimit(q.MaxTPM),
formatLimit(q.MaxRPD))
}
w.Flush()
return nil
},
}
}
func rlCheckCmd() *cli.Command {
return &cli.Command{
Use: "check <model> <estimated-tokens>",
Short: "Check rate limit capacity for a model",
Args: cli.ExactArgs(2),
RunE: func(cmd *cli.Command, args []string) error {
model := args[0]
tokens, err := strconv.Atoi(args[1])
if err != nil {
return fmt.Errorf("invalid token count: %w", err)
}
rl, err := ratelimit.New()
if err != nil {
return err
}
if err := rl.Load(); err != nil {
fmt.Printf("Warning: could not load existing state: %v\n", err)
}
stats := rl.Stats(model)
canSend := rl.CanSend(model, tokens)
status := "RATE LIMITED"
if canSend {
status = "OK"
}
fmt.Printf("Model: %s\n", model)
fmt.Printf("Request Cost: %d tokens\n", tokens)
fmt.Printf("Status: %s\n", status)
fmt.Printf("\nCurrent Usage (1m window):\n")
fmt.Printf(" RPM: %d / %s\n", stats.RPM, formatLimit(stats.MaxRPM))
fmt.Printf(" TPM: %d / %s\n", stats.TPM, formatLimit(stats.MaxTPM))
fmt.Printf(" RPD: %d / %s (reset: %s)\n", stats.RPD, formatLimit(stats.MaxRPD), stats.DayStart.Format(time.RFC3339))
return nil
},
}
}
func formatLimit(limit int) string {
if limit == 0 {
return "∞"
}
if limit >= 1000000 {
return fmt.Sprintf("%dM", limit/1000000)
}
if limit >= 1000 {
return fmt.Sprintf("%dK", limit/1000)
}
return fmt.Sprintf("%d", limit)
}

View file

@ -0,0 +1,49 @@
package ai
import (
"context"
"github.com/host-uk/core/pkg/log"
"github.com/host-uk/core/pkg/ratelimit"
)
// executeWithRateLimit wraps an agent execution with rate limiting logic.
// It estimates token usage, waits for capacity, executes the runner, and records usage.
func executeWithRateLimit(ctx context.Context, model, prompt string, runner func() (bool, int, error)) (bool, int, error) {
rl, err := ratelimit.New()
if err != nil {
log.Warn("Failed to initialize rate limiter, proceeding without limits", "error", err)
return runner()
}
if err := rl.Load(); err != nil {
log.Warn("Failed to load rate limit state", "error", err)
}
// Estimate tokens from prompt length (1 token ≈ 4 chars)
estTokens := len(prompt) / 4
if estTokens == 0 {
estTokens = 1
}
log.Info("Checking rate limits", "model", model, "est_tokens", estTokens)
if err := rl.WaitForCapacity(ctx, model, estTokens); err != nil {
return false, -1, err
}
success, exitCode, runErr := runner()
// Record usage with conservative output estimate (actual tokens unknown from shell runner).
outputEst := estTokens / 10
if outputEst < 50 {
outputEst = 50
}
rl.RecordUsage(model, estTokens, outputEst)
if err := rl.Persist(); err != nil {
log.Warn("Failed to persist rate limit state", "error", err)
}
return success, exitCode, runErr
}

View file

@ -67,17 +67,23 @@ func startHeadless() {
enableAutoMerge := handlers.NewEnableAutoMergeHandler(forgeClient) enableAutoMerge := handlers.NewEnableAutoMergeHandler(forgeClient)
tickParent := handlers.NewTickParentHandler(forgeClient) tickParent := handlers.NewTickParentHandler(forgeClient)
// Agent dispatch — load targets from ~/.core/config.yaml // Agent dispatch — Clotho integration
cfg, cfgErr := config.New() cfg, cfgErr := config.New()
var agentTargets map[string]handlers.AgentTarget var agentTargets map[string]agentci.AgentConfig
var clothoCfg agentci.ClothoConfig
if cfgErr == nil { if cfgErr == nil {
agentTargets, _ = agentci.LoadAgents(cfg) agentTargets, _ = agentci.LoadActiveAgents(cfg)
clothoCfg, _ = agentci.LoadClothoConfig(cfg)
} }
if agentTargets == nil { if agentTargets == nil {
agentTargets = map[string]handlers.AgentTarget{} agentTargets = map[string]agentci.AgentConfig{}
} }
log.Printf("Loaded %d agent targets", len(agentTargets))
dispatch := handlers.NewDispatchHandler(forgeClient, forgeURL, forgeToken, agentTargets) spinner := agentci.NewSpinner(clothoCfg, agentTargets)
log.Printf("Loaded %d agent targets. Strategy: %s", len(agentTargets), clothoCfg.Strategy)
dispatch := handlers.NewDispatchHandler(forgeClient, forgeURL, forgeToken, spinner)
// Build poller // Build poller
poller := jobrunner.NewPoller(jobrunner.PollerConfig{ poller := jobrunner.NewPoller(jobrunner.PollerConfig{

87
pkg/agentci/clotho.go Normal file
View file

@ -0,0 +1,87 @@
package agentci
import (
"context"
"strings"
"github.com/host-uk/core/pkg/jobrunner"
)
// RunMode determines the execution strategy for a dispatched task.
type RunMode string
const (
ModeStandard RunMode = "standard"
ModeDual RunMode = "dual" // The Clotho Protocol — dual-run verification
)
// Spinner is the Clotho orchestrator that determines the fate of each task.
type Spinner struct {
Config ClothoConfig
Agents map[string]AgentConfig
}
// NewSpinner creates a new Clotho orchestrator.
func NewSpinner(cfg ClothoConfig, agents map[string]AgentConfig) *Spinner {
return &Spinner{
Config: cfg,
Agents: agents,
}
}
// DeterminePlan decides if a signal requires dual-run verification based on
// the global strategy, agent configuration, and repository criticality.
func (s *Spinner) DeterminePlan(signal *jobrunner.PipelineSignal, agentName string) RunMode {
if s.Config.Strategy != "clotho-verified" {
return ModeStandard
}
agent, ok := s.Agents[agentName]
if !ok {
return ModeStandard
}
if agent.DualRun {
return ModeDual
}
// Protect critical repos with dual-run (Axiom 1).
if signal.RepoName == "core" || strings.Contains(signal.RepoName, "security") {
return ModeDual
}
return ModeStandard
}
// GetVerifierModel returns the model for the secondary "signed" verification run.
func (s *Spinner) GetVerifierModel(agentName string) string {
agent, ok := s.Agents[agentName]
if !ok || agent.VerifyModel == "" {
return "gemini-1.5-pro"
}
return agent.VerifyModel
}
// FindByForgejoUser resolves a Forgejo username to the agent config key and config.
// This decouples agent naming (mythological roles) from Forgejo identity.
func (s *Spinner) FindByForgejoUser(forgejoUser string) (string, AgentConfig, bool) {
if forgejoUser == "" {
return "", AgentConfig{}, false
}
// Direct match on config key first.
if agent, ok := s.Agents[forgejoUser]; ok {
return forgejoUser, agent, true
}
// Search by ForgejoUser field.
for name, agent := range s.Agents {
if agent.ForgejoUser != "" && agent.ForgejoUser == forgejoUser {
return name, agent, true
}
}
return "", AgentConfig{}, false
}
// Weave compares primary and verifier outputs. Returns true if they converge.
// This is a placeholder for future semantic diff logic.
func (s *Spinner) Weave(ctx context.Context, primaryOutput, signedOutput []byte) (bool, error) {
return string(primaryOutput) == string(signedOutput), nil
}

View file

@ -1,12 +1,10 @@
// Package agentci provides configuration and management for AgentCI dispatch targets. // Package agentci provides configuration, security, and orchestration for AgentCI dispatch targets.
package agentci package agentci
import ( import (
"fmt" "fmt"
"github.com/host-uk/core/pkg/config" "github.com/host-uk/core/pkg/config"
"github.com/host-uk/core/pkg/jobrunner/handlers"
"github.com/host-uk/core/pkg/log"
) )
// AgentConfig represents a single agent machine in the config file. // AgentConfig represents a single agent machine in the config file.
@ -14,49 +12,85 @@ type AgentConfig struct {
Host string `yaml:"host" mapstructure:"host"` Host string `yaml:"host" mapstructure:"host"`
QueueDir string `yaml:"queue_dir" mapstructure:"queue_dir"` QueueDir string `yaml:"queue_dir" mapstructure:"queue_dir"`
ForgejoUser string `yaml:"forgejo_user" mapstructure:"forgejo_user"` ForgejoUser string `yaml:"forgejo_user" mapstructure:"forgejo_user"`
Model string `yaml:"model" mapstructure:"model"` // claude model: sonnet, haiku, opus (default: sonnet) Model string `yaml:"model" mapstructure:"model"` // primary AI model
Runner string `yaml:"runner" mapstructure:"runner"` // runner binary: claude, codex (default: claude) Runner string `yaml:"runner" mapstructure:"runner"` // runner binary: claude, codex, gemini
VerifyModel string `yaml:"verify_model" mapstructure:"verify_model"` // secondary model for dual-run
SecurityLevel string `yaml:"security_level" mapstructure:"security_level"` // low, high
Roles []string `yaml:"roles" mapstructure:"roles"`
DualRun bool `yaml:"dual_run" mapstructure:"dual_run"`
Active bool `yaml:"active" mapstructure:"active"` Active bool `yaml:"active" mapstructure:"active"`
} }
// LoadAgents reads agent targets from config and returns a map suitable for the dispatch handler. // ClothoConfig controls the orchestration strategy.
type ClothoConfig struct {
Strategy string `yaml:"strategy" mapstructure:"strategy"` // direct, clotho-verified
ValidationThreshold float64 `yaml:"validation_threshold" mapstructure:"validation_threshold"` // divergence limit (0.0-1.0)
SigningKeyPath string `yaml:"signing_key_path" mapstructure:"signing_key_path"`
}
// LoadAgents reads agent targets from config and returns a map of AgentConfig.
// Returns an empty map (not an error) if no agents are configured. // Returns an empty map (not an error) if no agents are configured.
func LoadAgents(cfg *config.Config) (map[string]handlers.AgentTarget, error) { func LoadAgents(cfg *config.Config) (map[string]AgentConfig, error) {
var agents map[string]AgentConfig var agents map[string]AgentConfig
if err := cfg.Get("agentci.agents", &agents); err != nil { if err := cfg.Get("agentci.agents", &agents); err != nil {
// No config is fine — just no agents. return map[string]AgentConfig{}, nil
return map[string]handlers.AgentTarget{}, nil
} }
targets := make(map[string]handlers.AgentTarget) // Validate and apply defaults.
for name, ac := range agents { for name, ac := range agents {
if !ac.Active { if !ac.Active {
continue continue
} }
if ac.Host == "" { if ac.Host == "" {
return nil, log.E("agentci.LoadAgents", fmt.Sprintf("agent %q: host is required", name), nil) return nil, fmt.Errorf("agent %q: host is required", name)
} }
queueDir := ac.QueueDir if ac.QueueDir == "" {
if queueDir == "" { ac.QueueDir = "/home/claude/ai-work/queue"
queueDir = "/home/claude/ai-work/queue"
} }
model := ac.Model if ac.Model == "" {
if model == "" { ac.Model = "sonnet"
model = "sonnet"
} }
runner := ac.Runner if ac.Runner == "" {
if runner == "" { ac.Runner = "claude"
runner = "claude"
}
targets[name] = handlers.AgentTarget{
Host: ac.Host,
QueueDir: queueDir,
Model: model,
Runner: runner,
} }
agents[name] = ac
} }
return targets, nil return agents, nil
}
// LoadActiveAgents returns only active agents.
func LoadActiveAgents(cfg *config.Config) (map[string]AgentConfig, error) {
all, err := LoadAgents(cfg)
if err != nil {
return nil, err
}
active := make(map[string]AgentConfig)
for name, ac := range all {
if ac.Active {
active[name] = ac
}
}
return active, nil
}
// LoadClothoConfig loads the Clotho orchestrator settings.
// Returns sensible defaults if no config is present.
func LoadClothoConfig(cfg *config.Config) (ClothoConfig, error) {
var cc ClothoConfig
if err := cfg.Get("agentci.clotho", &cc); err != nil {
return ClothoConfig{
Strategy: "direct",
ValidationThreshold: 0.85,
}, nil
}
if cc.Strategy == "" {
cc.Strategy = "direct"
}
if cc.ValidationThreshold == 0 {
cc.ValidationThreshold = 0.85
}
return cc, nil
} }
// SaveAgent writes an agent config entry to the config file. // SaveAgent writes an agent config entry to the config file.
@ -67,6 +101,7 @@ func SaveAgent(cfg *config.Config, name string, ac AgentConfig) error {
"queue_dir": ac.QueueDir, "queue_dir": ac.QueueDir,
"forgejo_user": ac.ForgejoUser, "forgejo_user": ac.ForgejoUser,
"active": ac.Active, "active": ac.Active,
"dual_run": ac.DualRun,
} }
if ac.Model != "" { if ac.Model != "" {
data["model"] = ac.Model data["model"] = ac.Model
@ -74,6 +109,15 @@ func SaveAgent(cfg *config.Config, name string, ac AgentConfig) error {
if ac.Runner != "" { if ac.Runner != "" {
data["runner"] = ac.Runner data["runner"] = ac.Runner
} }
if ac.VerifyModel != "" {
data["verify_model"] = ac.VerifyModel
}
if ac.SecurityLevel != "" {
data["security_level"] = ac.SecurityLevel
}
if len(ac.Roles) > 0 {
data["roles"] = ac.Roles
}
return cfg.Set(key, data) return cfg.Set(key, data)
} }
@ -81,10 +125,10 @@ func SaveAgent(cfg *config.Config, name string, ac AgentConfig) error {
func RemoveAgent(cfg *config.Config, name string) error { func RemoveAgent(cfg *config.Config, name string) error {
var agents map[string]AgentConfig var agents map[string]AgentConfig
if err := cfg.Get("agentci.agents", &agents); err != nil { if err := cfg.Get("agentci.agents", &agents); err != nil {
return log.E("agentci.RemoveAgent", "no agents configured", err) return fmt.Errorf("no agents configured")
} }
if _, ok := agents[name]; !ok { if _, ok := agents[name]; !ok {
return log.E("agentci.RemoveAgent", fmt.Sprintf("agent %q not found", name), nil) return fmt.Errorf("agent %q not found", name)
} }
delete(agents, name) delete(agents, name)
return cfg.Set("agentci.agents", agents) return cfg.Set("agentci.agents", agents)

329
pkg/agentci/config_test.go Normal file
View file

@ -0,0 +1,329 @@
package agentci
import (
"testing"
"github.com/host-uk/core/pkg/config"
"github.com/host-uk/core/pkg/io"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func newTestConfig(t *testing.T, yaml string) *config.Config {
t.Helper()
m := io.NewMockMedium()
if yaml != "" {
m.Files["/tmp/test/config.yaml"] = yaml
}
cfg, err := config.New(config.WithMedium(m), config.WithPath("/tmp/test/config.yaml"))
require.NoError(t, err)
return cfg
}
func TestLoadAgents_Good(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
darbs-claude:
host: claude@192.168.0.201
queue_dir: /home/claude/ai-work/queue
forgejo_user: darbs-claude
model: sonnet
runner: claude
active: true
`)
agents, err := LoadAgents(cfg)
require.NoError(t, err)
require.Len(t, agents, 1)
agent := agents["darbs-claude"]
assert.Equal(t, "claude@192.168.0.201", agent.Host)
assert.Equal(t, "/home/claude/ai-work/queue", agent.QueueDir)
assert.Equal(t, "sonnet", agent.Model)
assert.Equal(t, "claude", agent.Runner)
}
func TestLoadAgents_Good_MultipleAgents(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
darbs-claude:
host: claude@192.168.0.201
queue_dir: /home/claude/ai-work/queue
active: true
local-codex:
host: localhost
queue_dir: /home/claude/ai-work/queue
runner: codex
active: true
`)
agents, err := LoadAgents(cfg)
require.NoError(t, err)
assert.Len(t, agents, 2)
assert.Contains(t, agents, "darbs-claude")
assert.Contains(t, agents, "local-codex")
}
func TestLoadAgents_Good_SkipsInactive(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
active-agent:
host: claude@10.0.0.1
active: true
offline-agent:
host: claude@10.0.0.2
active: false
`)
agents, err := LoadAgents(cfg)
require.NoError(t, err)
// Both are returned, but only active-agent has defaults applied.
assert.Len(t, agents, 2)
assert.Contains(t, agents, "active-agent")
}
func TestLoadActiveAgents_Good(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
active-agent:
host: claude@10.0.0.1
active: true
offline-agent:
host: claude@10.0.0.2
active: false
`)
active, err := LoadActiveAgents(cfg)
require.NoError(t, err)
assert.Len(t, active, 1)
assert.Contains(t, active, "active-agent")
}
func TestLoadAgents_Good_Defaults(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
minimal:
host: claude@10.0.0.1
active: true
`)
agents, err := LoadAgents(cfg)
require.NoError(t, err)
require.Len(t, agents, 1)
agent := agents["minimal"]
assert.Equal(t, "/home/claude/ai-work/queue", agent.QueueDir)
assert.Equal(t, "sonnet", agent.Model)
assert.Equal(t, "claude", agent.Runner)
}
func TestLoadAgents_Good_NoConfig(t *testing.T) {
cfg := newTestConfig(t, "")
agents, err := LoadAgents(cfg)
require.NoError(t, err)
assert.Empty(t, agents)
}
func TestLoadAgents_Bad_MissingHost(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
broken:
queue_dir: /tmp
active: true
`)
_, err := LoadAgents(cfg)
assert.Error(t, err)
assert.Contains(t, err.Error(), "host is required")
}
func TestLoadAgents_Good_WithDualRun(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
gemini-agent:
host: localhost
runner: gemini
model: gemini-2.0-flash
verify_model: gemini-1.5-pro
dual_run: true
active: true
`)
agents, err := LoadAgents(cfg)
require.NoError(t, err)
agent := agents["gemini-agent"]
assert.Equal(t, "gemini", agent.Runner)
assert.Equal(t, "gemini-2.0-flash", agent.Model)
assert.Equal(t, "gemini-1.5-pro", agent.VerifyModel)
assert.True(t, agent.DualRun)
}
func TestLoadClothoConfig_Good(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
clotho:
strategy: clotho-verified
validation_threshold: 0.9
signing_key_path: /etc/core/keys/clotho.pub
`)
cc, err := LoadClothoConfig(cfg)
require.NoError(t, err)
assert.Equal(t, "clotho-verified", cc.Strategy)
assert.Equal(t, 0.9, cc.ValidationThreshold)
assert.Equal(t, "/etc/core/keys/clotho.pub", cc.SigningKeyPath)
}
func TestLoadClothoConfig_Good_Defaults(t *testing.T) {
cfg := newTestConfig(t, "")
cc, err := LoadClothoConfig(cfg)
require.NoError(t, err)
assert.Equal(t, "direct", cc.Strategy)
assert.Equal(t, 0.85, cc.ValidationThreshold)
}
func TestSaveAgent_Good(t *testing.T) {
cfg := newTestConfig(t, "")
err := SaveAgent(cfg, "new-agent", AgentConfig{
Host: "claude@10.0.0.5",
QueueDir: "/home/claude/ai-work/queue",
ForgejoUser: "new-agent",
Model: "haiku",
Runner: "claude",
Active: true,
})
require.NoError(t, err)
agents, err := ListAgents(cfg)
require.NoError(t, err)
require.Contains(t, agents, "new-agent")
assert.Equal(t, "claude@10.0.0.5", agents["new-agent"].Host)
assert.Equal(t, "haiku", agents["new-agent"].Model)
}
func TestSaveAgent_Good_WithDualRun(t *testing.T) {
cfg := newTestConfig(t, "")
err := SaveAgent(cfg, "verified-agent", AgentConfig{
Host: "claude@10.0.0.5",
Model: "gemini-2.0-flash",
VerifyModel: "gemini-1.5-pro",
DualRun: true,
Active: true,
})
require.NoError(t, err)
agents, err := ListAgents(cfg)
require.NoError(t, err)
require.Contains(t, agents, "verified-agent")
assert.True(t, agents["verified-agent"].DualRun)
}
func TestSaveAgent_Good_OmitsEmptyOptionals(t *testing.T) {
cfg := newTestConfig(t, "")
err := SaveAgent(cfg, "minimal", AgentConfig{
Host: "claude@10.0.0.1",
Active: true,
})
require.NoError(t, err)
agents, err := ListAgents(cfg)
require.NoError(t, err)
assert.Contains(t, agents, "minimal")
}
func TestRemoveAgent_Good(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
to-remove:
host: claude@10.0.0.1
active: true
to-keep:
host: claude@10.0.0.2
active: true
`)
err := RemoveAgent(cfg, "to-remove")
require.NoError(t, err)
agents, err := ListAgents(cfg)
require.NoError(t, err)
assert.NotContains(t, agents, "to-remove")
assert.Contains(t, agents, "to-keep")
}
func TestRemoveAgent_Bad_NotFound(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
existing:
host: claude@10.0.0.1
active: true
`)
err := RemoveAgent(cfg, "nonexistent")
assert.Error(t, err)
assert.Contains(t, err.Error(), "not found")
}
func TestRemoveAgent_Bad_NoAgents(t *testing.T) {
cfg := newTestConfig(t, "")
err := RemoveAgent(cfg, "anything")
assert.Error(t, err)
assert.Contains(t, err.Error(), "no agents configured")
}
func TestListAgents_Good(t *testing.T) {
cfg := newTestConfig(t, `
agentci:
agents:
agent-a:
host: claude@10.0.0.1
active: true
agent-b:
host: claude@10.0.0.2
active: false
`)
agents, err := ListAgents(cfg)
require.NoError(t, err)
assert.Len(t, agents, 2)
assert.True(t, agents["agent-a"].Active)
assert.False(t, agents["agent-b"].Active)
}
func TestListAgents_Good_Empty(t *testing.T) {
cfg := newTestConfig(t, "")
agents, err := ListAgents(cfg)
require.NoError(t, err)
assert.Empty(t, agents)
}
func TestRoundTrip_SaveThenLoad(t *testing.T) {
cfg := newTestConfig(t, "")
err := SaveAgent(cfg, "alpha", AgentConfig{
Host: "claude@alpha",
QueueDir: "/home/claude/work/queue",
ForgejoUser: "alpha-bot",
Model: "opus",
Runner: "claude",
Active: true,
})
require.NoError(t, err)
err = SaveAgent(cfg, "beta", AgentConfig{
Host: "claude@beta",
ForgejoUser: "beta-bot",
Runner: "codex",
Active: true,
})
require.NoError(t, err)
agents, err := LoadActiveAgents(cfg)
require.NoError(t, err)
assert.Len(t, agents, 2)
assert.Equal(t, "claude@alpha", agents["alpha"].Host)
assert.Equal(t, "opus", agents["alpha"].Model)
assert.Equal(t, "codex", agents["beta"].Runner)
}

49
pkg/agentci/security.go Normal file
View file

@ -0,0 +1,49 @@
package agentci
import (
"fmt"
"os/exec"
"path/filepath"
"regexp"
"strings"
)
var safeNameRegex = regexp.MustCompile(`^[a-zA-Z0-9\-\_\.]+$`)
// SanitizePath ensures a filename or directory name is safe and prevents path traversal.
// Returns filepath.Base of the input after validation.
func SanitizePath(input string) (string, error) {
base := filepath.Base(input)
if !safeNameRegex.MatchString(base) {
return "", fmt.Errorf("invalid characters in path element: %s", input)
}
if base == "." || base == ".." || base == "/" {
return "", fmt.Errorf("invalid path element: %s", base)
}
return base, nil
}
// EscapeShellArg wraps a string in single quotes for safe remote shell insertion.
// Prefer exec.Command arguments over constructing shell strings where possible.
func EscapeShellArg(arg string) string {
return "'" + strings.ReplaceAll(arg, "'", "'\\''") + "'"
}
// SecureSSHCommand creates an SSH exec.Cmd with strict host key checking and batch mode.
func SecureSSHCommand(host string, remoteCmd string) *exec.Cmd {
return exec.Command("ssh",
"-o", "StrictHostKeyChecking=yes",
"-o", "BatchMode=yes",
"-o", "ConnectTimeout=10",
host,
remoteCmd,
)
}
// MaskToken returns a masked version of a token for safe logging.
func MaskToken(token string) string {
if len(token) < 8 {
return "*****"
}
return token[:4] + "****" + token[len(token)-4:]
}

View file

@ -75,6 +75,17 @@ func (c *Client) EditIssue(owner, repo string, number int64, opts forgejo.EditIs
return issue, nil return issue, nil
} }
// AssignIssue assigns an issue to the specified users.
func (c *Client) AssignIssue(owner, repo string, number int64, assignees []string) error {
_, _, err := c.api.EditIssue(owner, repo, number, forgejo.EditIssueOption{
Assignees: assignees,
})
if err != nil {
return log.E("forge.AssignIssue", "failed to assign issue", err)
}
return nil
}
// ListPullRequests returns pull requests for the given repository. // ListPullRequests returns pull requests for the given repository.
func (c *Client) ListPullRequests(owner, repo string, state string) ([]*forgejo.PullRequest, error) { func (c *Client) ListPullRequests(owner, repo string, state string) ([]*forgejo.PullRequest, error) {
st := forgejo.StateOpen st := forgejo.StateOpen

View file

@ -1,6 +1,9 @@
package forge package forge
import ( import (
"fmt"
"strings"
forgejo "codeberg.org/mvdkleijn/forgejo-sdk/forgejo/v2" forgejo "codeberg.org/mvdkleijn/forgejo-sdk/forgejo/v2"
"github.com/host-uk/core/pkg/log" "github.com/host-uk/core/pkg/log"
@ -58,3 +61,52 @@ func (c *Client) CreateRepoLabel(owner, repo string, opts forgejo.CreateLabelOpt
return label, nil return label, nil
} }
// GetLabelByName retrieves a specific label by name from a repository.
func (c *Client) GetLabelByName(owner, repo, name string) (*forgejo.Label, error) {
labels, err := c.ListRepoLabels(owner, repo)
if err != nil {
return nil, err
}
for _, l := range labels {
if strings.EqualFold(l.Name, name) {
return l, nil
}
}
return nil, fmt.Errorf("label %s not found in %s/%s", name, owner, repo)
}
// EnsureLabel checks if a label exists, and creates it if it doesn't.
func (c *Client) EnsureLabel(owner, repo, name, color string) (*forgejo.Label, error) {
label, err := c.GetLabelByName(owner, repo, name)
if err == nil {
return label, nil
}
return c.CreateRepoLabel(owner, repo, forgejo.CreateLabelOption{
Name: name,
Color: color,
})
}
// AddIssueLabels adds labels to an issue.
func (c *Client) AddIssueLabels(owner, repo string, number int64, labelIDs []int64) error {
_, _, err := c.api.AddIssueLabels(owner, repo, number, forgejo.IssueLabelsOption{
Labels: labelIDs,
})
if err != nil {
return log.E("forge.AddIssueLabels", "failed to add labels to issue", err)
}
return nil
}
// RemoveIssueLabel removes a label from an issue.
func (c *Client) RemoveIssueLabel(owner, repo string, number int64, labelID int64) error {
_, err := c.api.DeleteIssueLabel(owner, repo, number, labelID)
if err != nil {
return log.E("forge.RemoveIssueLabel", "failed to remove label from issue", err)
}
return nil
}

View file

@ -0,0 +1,87 @@
package handlers
import (
"context"
"fmt"
"time"
"github.com/host-uk/core/pkg/forge"
"github.com/host-uk/core/pkg/jobrunner"
)
const (
ColorAgentComplete = "#0e8a16" // Green
)
// CompletionHandler manages issue state when an agent finishes work.
type CompletionHandler struct {
forge *forge.Client
}
// NewCompletionHandler creates a handler for agent completion events.
func NewCompletionHandler(client *forge.Client) *CompletionHandler {
return &CompletionHandler{
forge: client,
}
}
// Name returns the handler identifier.
func (h *CompletionHandler) Name() string {
return "completion"
}
// Match returns true if the signal indicates an agent has finished a task.
func (h *CompletionHandler) Match(signal *jobrunner.PipelineSignal) bool {
return signal.Type == "agent_completion"
}
// Execute updates the issue labels based on the completion status.
func (h *CompletionHandler) Execute(ctx context.Context, signal *jobrunner.PipelineSignal) (*jobrunner.ActionResult, error) {
start := time.Now()
// Remove in-progress label.
if inProgressLabel, err := h.forge.GetLabelByName(signal.RepoOwner, signal.RepoName, LabelInProgress); err == nil {
_ = h.forge.RemoveIssueLabel(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), inProgressLabel.ID)
}
if signal.Success {
completeLabel, err := h.forge.EnsureLabel(signal.RepoOwner, signal.RepoName, LabelAgentComplete, ColorAgentComplete)
if err != nil {
return nil, fmt.Errorf("ensure label %s: %w", LabelAgentComplete, err)
}
if err := h.forge.AddIssueLabels(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), []int64{completeLabel.ID}); err != nil {
return nil, fmt.Errorf("add completed label: %w", err)
}
if signal.Message != "" {
_ = h.forge.CreateIssueComment(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), signal.Message)
}
} else {
failedLabel, err := h.forge.EnsureLabel(signal.RepoOwner, signal.RepoName, LabelAgentFailed, ColorAgentFailed)
if err != nil {
return nil, fmt.Errorf("ensure label %s: %w", LabelAgentFailed, err)
}
if err := h.forge.AddIssueLabels(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), []int64{failedLabel.ID}); err != nil {
return nil, fmt.Errorf("add failed label: %w", err)
}
msg := "Agent reported failure."
if signal.Error != "" {
msg += fmt.Sprintf("\n\nError: %s", signal.Error)
}
_ = h.forge.CreateIssueComment(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), msg)
}
return &jobrunner.ActionResult{
Action: "completion",
RepoOwner: signal.RepoOwner,
RepoName: signal.RepoName,
EpicNumber: signal.EpicNumber,
ChildNumber: signal.ChildNumber,
Success: true,
Timestamp: time.Now(),
Duration: time.Since(start),
}, nil
}

View file

@ -1,28 +1,31 @@
package handlers package handlers
import ( import (
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"os/exec"
"path/filepath" "path/filepath"
"strings"
"time" "time"
"github.com/host-uk/core/pkg/agentci"
"github.com/host-uk/core/pkg/forge" "github.com/host-uk/core/pkg/forge"
"github.com/host-uk/core/pkg/jobrunner" "github.com/host-uk/core/pkg/jobrunner"
"github.com/host-uk/core/pkg/log" "github.com/host-uk/core/pkg/log"
) )
// AgentTarget maps a Forgejo username to an SSH-reachable agent machine. const (
type AgentTarget struct { LabelAgentReady = "agent-ready"
Host string // SSH destination (e.g., "claude@192.168.0.201") LabelInProgress = "in-progress"
QueueDir string // Remote queue directory (e.g., "~/ai-work/queue") LabelAgentFailed = "agent-failed"
Model string // AI model: sonnet, haiku, opus (default: sonnet) LabelAgentComplete = "agent-completed"
Runner string // Runner binary: claude, codex (default: claude)
} ColorInProgress = "#1d76db" // Blue
ColorAgentFailed = "#c0392b" // Red
)
// DispatchTicket is the JSON payload written to the agent's queue. // DispatchTicket is the JSON payload written to the agent's queue.
// The ForgeToken is transferred separately via a .env file with 0600 permissions.
type DispatchTicket struct { type DispatchTicket struct {
ID string `json:"id"` ID string `json:"id"`
RepoOwner string `json:"repo_owner"` RepoOwner string `json:"repo_owner"`
@ -33,28 +36,29 @@ type DispatchTicket struct {
TargetBranch string `json:"target_branch"` TargetBranch string `json:"target_branch"`
EpicNumber int `json:"epic_number"` EpicNumber int `json:"epic_number"`
ForgeURL string `json:"forge_url"` ForgeURL string `json:"forge_url"`
ForgeToken string `json:"forge_token"`
ForgeUser string `json:"forgejo_user"` ForgeUser string `json:"forgejo_user"`
Model string `json:"model,omitempty"` Model string `json:"model,omitempty"`
Runner string `json:"runner,omitempty"` Runner string `json:"runner,omitempty"`
VerifyModel string `json:"verify_model,omitempty"`
DualRun bool `json:"dual_run"`
CreatedAt string `json:"created_at"` CreatedAt string `json:"created_at"`
} }
// DispatchHandler dispatches coding work to remote agent machines via SSH/SCP. // DispatchHandler dispatches coding work to remote agent machines via SSH.
type DispatchHandler struct { type DispatchHandler struct {
forge *forge.Client forge *forge.Client
forgeURL string forgeURL string
token string token string
agents map[string]AgentTarget spinner *agentci.Spinner
} }
// NewDispatchHandler creates a handler that dispatches tickets to agent machines. // NewDispatchHandler creates a handler that dispatches tickets to agent machines.
func NewDispatchHandler(client *forge.Client, forgeURL, token string, agents map[string]AgentTarget) *DispatchHandler { func NewDispatchHandler(client *forge.Client, forgeURL, token string, spinner *agentci.Spinner) *DispatchHandler {
return &DispatchHandler{ return &DispatchHandler{
forge: client, forge: client,
forgeURL: forgeURL, forgeURL: forgeURL,
token: token, token: token,
agents: agents, spinner: spinner,
} }
} }
@ -64,67 +68,113 @@ func (h *DispatchHandler) Name() string {
} }
// Match returns true for signals where a child issue needs coding (no PR yet) // Match returns true for signals where a child issue needs coding (no PR yet)
// and the assignee is a known agent. // and the assignee is a known agent (by config key or Forgejo username).
func (h *DispatchHandler) Match(signal *jobrunner.PipelineSignal) bool { func (h *DispatchHandler) Match(signal *jobrunner.PipelineSignal) bool {
if !signal.NeedsCoding { if !signal.NeedsCoding {
return false return false
} }
_, ok := h.agents[signal.Assignee] _, _, ok := h.spinner.FindByForgejoUser(signal.Assignee)
return ok return ok
} }
// Execute creates a ticket JSON and SCPs it to the agent's queue directory. // Execute creates a ticket JSON and transfers it securely to the agent's queue directory.
func (h *DispatchHandler) Execute(ctx context.Context, signal *jobrunner.PipelineSignal) (*jobrunner.ActionResult, error) { func (h *DispatchHandler) Execute(ctx context.Context, signal *jobrunner.PipelineSignal) (*jobrunner.ActionResult, error) {
start := time.Now() start := time.Now()
agent, ok := h.agents[signal.Assignee] agentName, agent, ok := h.spinner.FindByForgejoUser(signal.Assignee)
if !ok { if !ok {
return nil, log.E("dispatch.Execute", fmt.Sprintf("unknown agent: %s", signal.Assignee), nil) return nil, fmt.Errorf("unknown agent: %s", signal.Assignee)
} }
// Determine target branch (default to repo default). // Sanitize inputs to prevent path traversal.
safeOwner, err := agentci.SanitizePath(signal.RepoOwner)
if err != nil {
return nil, fmt.Errorf("invalid repo owner: %w", err)
}
safeRepo, err := agentci.SanitizePath(signal.RepoName)
if err != nil {
return nil, fmt.Errorf("invalid repo name: %w", err)
}
// Ensure in-progress label exists on repo.
inProgressLabel, err := h.forge.EnsureLabel(safeOwner, safeRepo, LabelInProgress, ColorInProgress)
if err != nil {
return nil, fmt.Errorf("ensure label %s: %w", LabelInProgress, err)
}
// Check if already in progress to prevent double-dispatch.
issue, err := h.forge.GetIssue(safeOwner, safeRepo, int64(signal.ChildNumber))
if err == nil {
for _, l := range issue.Labels {
if l.Name == LabelInProgress || l.Name == LabelAgentComplete {
log.Info("issue already processed, skipping", "issue", signal.ChildNumber, "label", l.Name)
return &jobrunner.ActionResult{
Action: "dispatch",
Success: true,
Timestamp: time.Now(),
Duration: time.Since(start),
}, nil
}
}
}
// Assign agent and add in-progress label.
if err := h.forge.AssignIssue(safeOwner, safeRepo, int64(signal.ChildNumber), []string{signal.Assignee}); err != nil {
log.Warn("failed to assign agent, continuing", "err", err)
}
if err := h.forge.AddIssueLabels(safeOwner, safeRepo, int64(signal.ChildNumber), []int64{inProgressLabel.ID}); err != nil {
return nil, fmt.Errorf("add in-progress label: %w", err)
}
// Remove agent-ready label if present.
if readyLabel, err := h.forge.GetLabelByName(safeOwner, safeRepo, LabelAgentReady); err == nil {
_ = h.forge.RemoveIssueLabel(safeOwner, safeRepo, int64(signal.ChildNumber), readyLabel.ID)
}
// Clotho planning — determine execution mode.
runMode := h.spinner.DeterminePlan(signal, agentName)
verifyModel := ""
if runMode == agentci.ModeDual {
verifyModel = h.spinner.GetVerifierModel(agentName)
}
// Build ticket.
targetBranch := "new" // TODO: resolve from epic or repo default targetBranch := "new" // TODO: resolve from epic or repo default
ticketID := fmt.Sprintf("%s-%s-%d-%d", safeOwner, safeRepo, signal.ChildNumber, time.Now().Unix())
ticket := DispatchTicket{ ticket := DispatchTicket{
ID: fmt.Sprintf("%s-%s-%d-%d", signal.RepoOwner, signal.RepoName, signal.ChildNumber, time.Now().Unix()), ID: ticketID,
RepoOwner: signal.RepoOwner, RepoOwner: safeOwner,
RepoName: signal.RepoName, RepoName: safeRepo,
IssueNumber: signal.ChildNumber, IssueNumber: signal.ChildNumber,
IssueTitle: signal.IssueTitle, IssueTitle: signal.IssueTitle,
IssueBody: signal.IssueBody, IssueBody: signal.IssueBody,
TargetBranch: targetBranch, TargetBranch: targetBranch,
EpicNumber: signal.EpicNumber, EpicNumber: signal.EpicNumber,
ForgeURL: h.forgeURL, ForgeURL: h.forgeURL,
ForgeToken: h.token,
ForgeUser: signal.Assignee, ForgeUser: signal.Assignee,
Model: agent.Model, Model: agent.Model,
Runner: agent.Runner, Runner: agent.Runner,
VerifyModel: verifyModel,
DualRun: runMode == agentci.ModeDual,
CreatedAt: time.Now().UTC().Format(time.RFC3339), CreatedAt: time.Now().UTC().Format(time.RFC3339),
} }
ticketJSON, err := json.MarshalIndent(ticket, "", " ") ticketJSON, err := json.MarshalIndent(ticket, "", " ")
if err != nil { if err != nil {
return &jobrunner.ActionResult{ h.failDispatch(signal, "Failed to marshal ticket JSON")
Action: "dispatch", return nil, fmt.Errorf("marshal ticket: %w", err)
RepoOwner: signal.RepoOwner,
RepoName: signal.RepoName,
EpicNumber: signal.EpicNumber,
ChildNumber: signal.ChildNumber,
Success: false,
Error: fmt.Sprintf("marshal ticket: %v", err),
Timestamp: time.Now(),
Duration: time.Since(start),
}, nil
} }
// Check if ticket already exists on agent (dedup). // Check if ticket already exists on agent (dedup).
ticketName := fmt.Sprintf("ticket-%s-%s-%d.json", signal.RepoOwner, signal.RepoName, signal.ChildNumber) ticketName := fmt.Sprintf("ticket-%s-%s-%d.json", safeOwner, safeRepo, signal.ChildNumber)
if h.ticketExists(agent, ticketName) { if h.ticketExists(ctx, agent, ticketName) {
log.Info("ticket already queued, skipping", "ticket", ticketName, "agent", signal.Assignee) log.Info("ticket already queued, skipping", "ticket", ticketName, "agent", signal.Assignee)
return &jobrunner.ActionResult{ return &jobrunner.ActionResult{
Action: "dispatch", Action: "dispatch",
RepoOwner: signal.RepoOwner, RepoOwner: safeOwner,
RepoName: signal.RepoName, RepoName: safeRepo,
EpicNumber: signal.EpicNumber, EpicNumber: signal.EpicNumber,
ChildNumber: signal.ChildNumber, ChildNumber: signal.ChildNumber,
Success: true, Success: true,
@ -133,30 +183,55 @@ func (h *DispatchHandler) Execute(ctx context.Context, signal *jobrunner.Pipelin
}, nil }, nil
} }
// SCP ticket to agent queue. // Transfer ticket JSON.
remotePath := filepath.Join(agent.QueueDir, ticketName) remoteTicketPath := filepath.Join(agent.QueueDir, ticketName)
if err := h.scpTicket(ctx, agent.Host, remotePath, ticketJSON); err != nil { if err := h.secureTransfer(ctx, agent, remoteTicketPath, ticketJSON, 0644); err != nil {
h.failDispatch(signal, fmt.Sprintf("Ticket transfer failed: %v", err))
return &jobrunner.ActionResult{ return &jobrunner.ActionResult{
Action: "dispatch", Action: "dispatch",
RepoOwner: signal.RepoOwner, RepoOwner: safeOwner,
RepoName: signal.RepoName, RepoName: safeRepo,
EpicNumber: signal.EpicNumber, EpicNumber: signal.EpicNumber,
ChildNumber: signal.ChildNumber, ChildNumber: signal.ChildNumber,
Success: false, Success: false,
Error: fmt.Sprintf("scp ticket: %v", err), Error: fmt.Sprintf("transfer ticket: %v", err),
Timestamp: time.Now(),
Duration: time.Since(start),
}, nil
}
// Transfer token via separate .env file with 0600 permissions.
envContent := fmt.Sprintf("FORGE_TOKEN=%s\n", h.token)
remoteEnvPath := filepath.Join(agent.QueueDir, fmt.Sprintf(".env.%s", ticketID))
if err := h.secureTransfer(ctx, agent, remoteEnvPath, []byte(envContent), 0600); err != nil {
// Clean up the ticket if env transfer fails.
_ = h.runRemote(ctx, agent, fmt.Sprintf("rm -f %s", agentci.EscapeShellArg(remoteTicketPath)))
h.failDispatch(signal, fmt.Sprintf("Token transfer failed: %v", err))
return &jobrunner.ActionResult{
Action: "dispatch",
RepoOwner: safeOwner,
RepoName: safeRepo,
EpicNumber: signal.EpicNumber,
ChildNumber: signal.ChildNumber,
Success: false,
Error: fmt.Sprintf("transfer token: %v", err),
Timestamp: time.Now(), Timestamp: time.Now(),
Duration: time.Since(start), Duration: time.Since(start),
}, nil }, nil
} }
// Comment on issue. // Comment on issue.
comment := fmt.Sprintf("Dispatched to **%s** agent queue.", signal.Assignee) modeStr := "Standard"
_ = h.forge.CreateIssueComment(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), comment) if runMode == agentci.ModeDual {
modeStr = "Clotho Verified (Dual Run)"
}
comment := fmt.Sprintf("Dispatched to **%s** agent queue.\nMode: **%s**", signal.Assignee, modeStr)
_ = h.forge.CreateIssueComment(safeOwner, safeRepo, int64(signal.ChildNumber), comment)
return &jobrunner.ActionResult{ return &jobrunner.ActionResult{
Action: "dispatch", Action: "dispatch",
RepoOwner: signal.RepoOwner, RepoOwner: safeOwner,
RepoName: signal.RepoName, RepoName: safeRepo,
EpicNumber: signal.EpicNumber, EpicNumber: signal.EpicNumber,
ChildNumber: signal.ChildNumber, ChildNumber: signal.ChildNumber,
Success: true, Success: true,
@ -165,37 +240,51 @@ func (h *DispatchHandler) Execute(ctx context.Context, signal *jobrunner.Pipelin
}, nil }, nil
} }
// scpTicket writes ticket data to a remote path via SSH. // failDispatch handles cleanup when dispatch fails (adds failed label, removes in-progress).
// TODO: Replace exec ssh+cat with charmbracelet/ssh for native Go SSH. func (h *DispatchHandler) failDispatch(signal *jobrunner.PipelineSignal, reason string) {
func (h *DispatchHandler) scpTicket(ctx context.Context, host, remotePath string, data []byte) error { if failedLabel, err := h.forge.EnsureLabel(signal.RepoOwner, signal.RepoName, LabelAgentFailed, ColorAgentFailed); err == nil {
// Use ssh + cat instead of scp for piping stdin. _ = h.forge.AddIssueLabels(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), []int64{failedLabel.ID})
// TODO: Use charmbracelet/keygen for key management, native Go SSH client for transport. }
cmd := exec.CommandContext(ctx, "ssh",
"-o", "StrictHostKeyChecking=accept-new", if inProgressLabel, err := h.forge.GetLabelByName(signal.RepoOwner, signal.RepoName, LabelInProgress); err == nil {
"-o", "ConnectTimeout=10", _ = h.forge.RemoveIssueLabel(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), inProgressLabel.ID)
host, }
fmt.Sprintf("cat > %s", remotePath),
) _ = h.forge.CreateIssueComment(signal.RepoOwner, signal.RepoName, int64(signal.ChildNumber), fmt.Sprintf("Agent dispatch failed: %s", reason))
cmd.Stdin = strings.NewReader(string(data)) }
// secureTransfer writes data to a remote path via SSH stdin, preventing command injection.
func (h *DispatchHandler) secureTransfer(ctx context.Context, agent agentci.AgentConfig, remotePath string, data []byte, mode int) error {
safeRemotePath := agentci.EscapeShellArg(remotePath)
remoteCmd := fmt.Sprintf("cat > %s && chmod %o %s", safeRemotePath, mode, safeRemotePath)
cmd := agentci.SecureSSHCommand(agent.Host, remoteCmd)
cmd.Stdin = bytes.NewReader(data)
output, err := cmd.CombinedOutput() output, err := cmd.CombinedOutput()
if err != nil { if err != nil {
return log.E("dispatch.scp", fmt.Sprintf("ssh to %s failed: %s", host, string(output)), err) return log.E("dispatch.transfer", fmt.Sprintf("ssh to %s failed: %s", agent.Host, string(output)), err)
} }
return nil return nil
} }
// runRemote executes a command on the agent via SSH.
func (h *DispatchHandler) runRemote(ctx context.Context, agent agentci.AgentConfig, cmdStr string) error {
cmd := agentci.SecureSSHCommand(agent.Host, cmdStr)
return cmd.Run()
}
// ticketExists checks if a ticket file already exists in queue, active, or done. // ticketExists checks if a ticket file already exists in queue, active, or done.
// TODO: Replace exec ssh with native Go SSH client (charmbracelet/ssh). func (h *DispatchHandler) ticketExists(ctx context.Context, agent agentci.AgentConfig, ticketName string) bool {
func (h *DispatchHandler) ticketExists(agent AgentTarget, ticketName string) bool { safeTicket, err := agentci.SanitizePath(ticketName)
cmd := exec.Command("ssh", if err != nil {
"-o", "StrictHostKeyChecking=accept-new", return false
"-o", "ConnectTimeout=10", }
agent.Host, qDir := agent.QueueDir
fmt.Sprintf("test -f %s/%s || test -f %s/../active/%s || test -f %s/../done/%s", checkCmd := fmt.Sprintf(
agent.QueueDir, ticketName, "test -f %s/%s || test -f %s/../active/%s || test -f %s/../done/%s",
agent.QueueDir, ticketName, qDir, safeTicket, qDir, safeTicket, qDir, safeTicket,
agent.QueueDir, ticketName),
) )
cmd := agentci.SecureSSHCommand(agent.Host, checkCmd)
return cmd.Run() == nil return cmd.Run() == nil
} }

View file

@ -1,16 +1,30 @@
package handlers package handlers
import ( import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing" "testing"
"github.com/host-uk/core/pkg/agentci"
"github.com/host-uk/core/pkg/jobrunner" "github.com/host-uk/core/pkg/jobrunner"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
) )
// newTestSpinner creates a Spinner with the given agents for testing.
func newTestSpinner(agents map[string]agentci.AgentConfig) *agentci.Spinner {
return agentci.NewSpinner(agentci.ClothoConfig{Strategy: "direct"}, agents)
}
// --- Match tests ---
func TestDispatch_Match_Good_NeedsCoding(t *testing.T) { func TestDispatch_Match_Good_NeedsCoding(t *testing.T) {
h := NewDispatchHandler(nil, "", "", map[string]AgentTarget{ spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue"}, "darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue", Active: true},
}) })
h := NewDispatchHandler(nil, "", "", spinner)
sig := &jobrunner.PipelineSignal{ sig := &jobrunner.PipelineSignal{
NeedsCoding: true, NeedsCoding: true,
Assignee: "darbs-claude", Assignee: "darbs-claude",
@ -18,10 +32,24 @@ func TestDispatch_Match_Good_NeedsCoding(t *testing.T) {
assert.True(t, h.Match(sig)) assert.True(t, h.Match(sig))
} }
func TestDispatch_Match_Bad_HasPR(t *testing.T) { func TestDispatch_Match_Good_MultipleAgents(t *testing.T) {
h := NewDispatchHandler(nil, "", "", map[string]AgentTarget{ spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue"}, "darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue", Active: true},
"local-codex": {Host: "localhost", QueueDir: "~/ai-work/queue", Active: true},
}) })
h := NewDispatchHandler(nil, "", "", spinner)
sig := &jobrunner.PipelineSignal{
NeedsCoding: true,
Assignee: "local-codex",
}
assert.True(t, h.Match(sig))
}
func TestDispatch_Match_Bad_HasPR(t *testing.T) {
spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue", Active: true},
})
h := NewDispatchHandler(nil, "", "", spinner)
sig := &jobrunner.PipelineSignal{ sig := &jobrunner.PipelineSignal{
NeedsCoding: false, NeedsCoding: false,
PRNumber: 7, PRNumber: 7,
@ -31,9 +59,10 @@ func TestDispatch_Match_Bad_HasPR(t *testing.T) {
} }
func TestDispatch_Match_Bad_UnknownAgent(t *testing.T) { func TestDispatch_Match_Bad_UnknownAgent(t *testing.T) {
h := NewDispatchHandler(nil, "", "", map[string]AgentTarget{ spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue"}, "darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue", Active: true},
}) })
h := NewDispatchHandler(nil, "", "", spinner)
sig := &jobrunner.PipelineSignal{ sig := &jobrunner.PipelineSignal{
NeedsCoding: true, NeedsCoding: true,
Assignee: "unknown-user", Assignee: "unknown-user",
@ -42,12 +71,257 @@ func TestDispatch_Match_Bad_UnknownAgent(t *testing.T) {
} }
func TestDispatch_Match_Bad_NotAssigned(t *testing.T) { func TestDispatch_Match_Bad_NotAssigned(t *testing.T) {
h := NewDispatchHandler(nil, "", "", map[string]AgentTarget{ spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue"}, "darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue", Active: true},
}) })
h := NewDispatchHandler(nil, "", "", spinner)
sig := &jobrunner.PipelineSignal{ sig := &jobrunner.PipelineSignal{
NeedsCoding: true, NeedsCoding: true,
Assignee: "", Assignee: "",
} }
assert.False(t, h.Match(sig)) assert.False(t, h.Match(sig))
} }
func TestDispatch_Match_Bad_EmptyAgentMap(t *testing.T) {
spinner := newTestSpinner(map[string]agentci.AgentConfig{})
h := NewDispatchHandler(nil, "", "", spinner)
sig := &jobrunner.PipelineSignal{
NeedsCoding: true,
Assignee: "darbs-claude",
}
assert.False(t, h.Match(sig))
}
// --- Name test ---
func TestDispatch_Name_Good(t *testing.T) {
spinner := newTestSpinner(nil)
h := NewDispatchHandler(nil, "", "", spinner)
assert.Equal(t, "dispatch", h.Name())
}
// --- Execute tests ---
func TestDispatch_Execute_Bad_UnknownAgent(t *testing.T) {
srv := httptest.NewServer(withVersion(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})))
defer srv.Close()
client := newTestForgeClient(t, srv.URL)
spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "claude@192.168.0.201", QueueDir: "~/ai-work/queue", Active: true},
})
h := NewDispatchHandler(client, srv.URL, "test-token", spinner)
sig := &jobrunner.PipelineSignal{
NeedsCoding: true,
Assignee: "nonexistent-agent",
RepoOwner: "host-uk",
RepoName: "core",
ChildNumber: 1,
}
_, err := h.Execute(context.Background(), sig)
require.Error(t, err)
assert.Contains(t, err.Error(), "unknown agent")
}
func TestDispatch_TicketJSON_Good(t *testing.T) {
ticket := DispatchTicket{
ID: "host-uk-core-5-1234567890",
RepoOwner: "host-uk",
RepoName: "core",
IssueNumber: 5,
IssueTitle: "Fix the thing",
IssueBody: "Please fix this bug",
TargetBranch: "new",
EpicNumber: 3,
ForgeURL: "https://forge.lthn.ai",
ForgeUser: "darbs-claude",
Model: "sonnet",
Runner: "claude",
DualRun: false,
CreatedAt: "2026-02-09T12:00:00Z",
}
data, err := json.MarshalIndent(ticket, "", " ")
require.NoError(t, err)
var decoded map[string]any
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)
assert.Equal(t, "host-uk-core-5-1234567890", decoded["id"])
assert.Equal(t, "host-uk", decoded["repo_owner"])
assert.Equal(t, "core", decoded["repo_name"])
assert.Equal(t, float64(5), decoded["issue_number"])
assert.Equal(t, "Fix the thing", decoded["issue_title"])
assert.Equal(t, "Please fix this bug", decoded["issue_body"])
assert.Equal(t, "new", decoded["target_branch"])
assert.Equal(t, float64(3), decoded["epic_number"])
assert.Equal(t, "https://forge.lthn.ai", decoded["forge_url"])
assert.Equal(t, "darbs-claude", decoded["forgejo_user"])
assert.Equal(t, "sonnet", decoded["model"])
assert.Equal(t, "claude", decoded["runner"])
// Token should NOT be present in the ticket.
_, hasToken := decoded["forge_token"]
assert.False(t, hasToken, "forge_token must not be in ticket JSON")
}
func TestDispatch_TicketJSON_Good_DualRun(t *testing.T) {
ticket := DispatchTicket{
ID: "test-dual",
RepoOwner: "host-uk",
RepoName: "core",
IssueNumber: 1,
ForgeURL: "https://forge.lthn.ai",
Model: "gemini-2.0-flash",
VerifyModel: "gemini-1.5-pro",
DualRun: true,
}
data, err := json.Marshal(ticket)
require.NoError(t, err)
var roundtrip DispatchTicket
err = json.Unmarshal(data, &roundtrip)
require.NoError(t, err)
assert.True(t, roundtrip.DualRun)
assert.Equal(t, "gemini-1.5-pro", roundtrip.VerifyModel)
}
func TestDispatch_TicketJSON_Good_OmitsEmptyModelRunner(t *testing.T) {
ticket := DispatchTicket{
ID: "test-1",
RepoOwner: "host-uk",
RepoName: "core",
IssueNumber: 1,
TargetBranch: "new",
ForgeURL: "https://forge.lthn.ai",
}
data, err := json.MarshalIndent(ticket, "", " ")
require.NoError(t, err)
var decoded map[string]any
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)
_, hasModel := decoded["model"]
_, hasRunner := decoded["runner"]
assert.False(t, hasModel, "model should be omitted when empty")
assert.False(t, hasRunner, "runner should be omitted when empty")
}
func TestDispatch_TicketJSON_Good_ModelRunnerVariants(t *testing.T) {
tests := []struct {
name string
model string
runner string
}{
{"claude-sonnet", "sonnet", "claude"},
{"claude-opus", "opus", "claude"},
{"codex-default", "", "codex"},
{"gemini-default", "", "gemini"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ticket := DispatchTicket{
ID: "test-" + tt.name,
RepoOwner: "host-uk",
RepoName: "core",
IssueNumber: 1,
TargetBranch: "new",
ForgeURL: "https://forge.lthn.ai",
Model: tt.model,
Runner: tt.runner,
}
data, err := json.Marshal(ticket)
require.NoError(t, err)
var roundtrip DispatchTicket
err = json.Unmarshal(data, &roundtrip)
require.NoError(t, err)
assert.Equal(t, tt.model, roundtrip.Model)
assert.Equal(t, tt.runner, roundtrip.Runner)
})
}
}
func TestDispatch_Execute_Good_PostsComment(t *testing.T) {
var commentPosted bool
var commentBody string
srv := httptest.NewServer(withVersion(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/repos/host-uk/core/labels":
json.NewEncoder(w).Encode([]any{})
return
case r.Method == http.MethodPost && r.URL.Path == "/api/v1/repos/host-uk/core/labels":
json.NewEncoder(w).Encode(map[string]any{"id": 1, "name": "in-progress", "color": "#1d76db"})
return
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/repos/host-uk/core/issues/5":
json.NewEncoder(w).Encode(map[string]any{"id": 5, "number": 5, "labels": []any{}, "title": "Test"})
return
case r.Method == http.MethodPatch && r.URL.Path == "/api/v1/repos/host-uk/core/issues/5":
json.NewEncoder(w).Encode(map[string]any{"id": 5, "number": 5})
return
case r.Method == http.MethodPost && r.URL.Path == "/api/v1/repos/host-uk/core/issues/5/labels":
json.NewEncoder(w).Encode([]any{map[string]any{"id": 1, "name": "in-progress"}})
return
case r.Method == http.MethodPost && r.URL.Path == "/api/v1/repos/host-uk/core/issues/5/comments":
commentPosted = true
var body map[string]string
_ = json.NewDecoder(r.Body).Decode(&body)
commentBody = body["body"]
json.NewEncoder(w).Encode(map[string]any{"id": 1, "body": body["body"]})
return
}
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]any{})
})))
defer srv.Close()
client := newTestForgeClient(t, srv.URL)
spinner := newTestSpinner(map[string]agentci.AgentConfig{
"darbs-claude": {Host: "localhost", QueueDir: "/tmp/nonexistent-queue", Active: true},
})
h := NewDispatchHandler(client, srv.URL, "test-token", spinner)
sig := &jobrunner.PipelineSignal{
NeedsCoding: true,
Assignee: "darbs-claude",
RepoOwner: "host-uk",
RepoName: "core",
ChildNumber: 5,
EpicNumber: 3,
IssueTitle: "Test issue",
IssueBody: "Test body",
}
result, err := h.Execute(context.Background(), sig)
require.NoError(t, err)
assert.Equal(t, "dispatch", result.Action)
assert.Equal(t, "host-uk", result.RepoOwner)
assert.Equal(t, "core", result.RepoName)
assert.Equal(t, 3, result.EpicNumber)
assert.Equal(t, 5, result.ChildNumber)
if result.Success {
assert.True(t, commentPosted)
assert.Contains(t, commentBody, "darbs-claude")
}
}

View file

@ -26,6 +26,10 @@ type PipelineSignal struct {
Assignee string // issue assignee username (for dispatch) Assignee string // issue assignee username (for dispatch)
IssueTitle string // child issue title (for dispatch prompt) IssueTitle string // child issue title (for dispatch prompt)
IssueBody string // child issue body (for dispatch prompt) IssueBody string // child issue body (for dispatch prompt)
Type string // signal type (e.g., "agent_completion")
Success bool // agent completion success flag
Error string // agent error message
Message string // agent completion message
} }
// RepoFullName returns "owner/repo". // RepoFullName returns "owner/repo".

382
pkg/ratelimit/ratelimit.go Normal file
View file

@ -0,0 +1,382 @@
package ratelimit
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
"time"
"gopkg.in/yaml.v3"
)
// ModelQuota defines the rate limits for a specific model.
type ModelQuota struct {
MaxRPM int `yaml:"max_rpm"` // Requests per minute
MaxTPM int `yaml:"max_tpm"` // Tokens per minute
MaxRPD int `yaml:"max_rpd"` // Requests per day (0 = unlimited)
}
// TokenEntry records a token usage event.
type TokenEntry struct {
Time time.Time `yaml:"time"`
Count int `yaml:"count"`
}
// UsageStats tracks usage history for a model.
type UsageStats struct {
Requests []time.Time `yaml:"requests"` // Sliding window (1m)
Tokens []TokenEntry `yaml:"tokens"` // Sliding window (1m)
DayStart time.Time `yaml:"day_start"`
DayCount int `yaml:"day_count"`
}
// RateLimiter manages rate limits across multiple models.
type RateLimiter struct {
mu sync.RWMutex
Quotas map[string]ModelQuota `yaml:"quotas"`
State map[string]*UsageStats `yaml:"state"`
filePath string
}
// New creates a new RateLimiter with default quotas.
func New() (*RateLimiter, error) {
home, err := os.UserHomeDir()
if err != nil {
return nil, err
}
rl := &RateLimiter{
Quotas: make(map[string]ModelQuota),
State: make(map[string]*UsageStats),
filePath: filepath.Join(home, ".core", "ratelimits.yaml"),
}
// Default quotas based on Tier 1 observations (Feb 2026)
rl.Quotas["gemini-3-pro-preview"] = ModelQuota{MaxRPM: 150, MaxTPM: 1000000, MaxRPD: 1000}
rl.Quotas["gemini-3-flash-preview"] = ModelQuota{MaxRPM: 150, MaxTPM: 1000000, MaxRPD: 1000}
rl.Quotas["gemini-2.5-pro"] = ModelQuota{MaxRPM: 150, MaxTPM: 1000000, MaxRPD: 1000}
rl.Quotas["gemini-2.0-flash"] = ModelQuota{MaxRPM: 150, MaxTPM: 1000000, MaxRPD: 0} // Unlimited RPD
rl.Quotas["gemini-2.0-flash-lite"] = ModelQuota{MaxRPM: 0, MaxTPM: 0, MaxRPD: 0} // Unlimited
return rl, nil
}
// Load reads the state from disk.
func (rl *RateLimiter) Load() error {
rl.mu.Lock()
defer rl.mu.Unlock()
data, err := os.ReadFile(rl.filePath)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
return yaml.Unmarshal(data, rl)
}
// Persist writes the state to disk.
func (rl *RateLimiter) Persist() error {
rl.mu.RLock()
defer rl.mu.RUnlock()
data, err := yaml.Marshal(rl)
if err != nil {
return err
}
dir := filepath.Dir(rl.filePath)
if err := os.MkdirAll(dir, 0755); err != nil {
return err
}
return os.WriteFile(rl.filePath, data, 0644)
}
// prune removes entries older than the sliding window (1 minute).
// Caller must hold lock.
func (rl *RateLimiter) prune(model string) {
stats, ok := rl.State[model]
if !ok {
return
}
now := time.Now()
window := now.Add(-1 * time.Minute)
// Prune requests
validReqs := 0
for _, t := range stats.Requests {
if t.After(window) {
stats.Requests[validReqs] = t
validReqs++
}
}
stats.Requests = stats.Requests[:validReqs]
// Prune tokens
validTokens := 0
for _, t := range stats.Tokens {
if t.Time.After(window) {
stats.Tokens[validTokens] = t
validTokens++
}
}
stats.Tokens = stats.Tokens[:validTokens]
// Reset daily counter if day has passed
if now.Sub(stats.DayStart) >= 24*time.Hour {
stats.DayStart = now
stats.DayCount = 0
}
}
// CanSend checks if a request can be sent without violating limits.
func (rl *RateLimiter) CanSend(model string, estimatedTokens int) bool {
rl.mu.Lock()
defer rl.mu.Unlock()
quota, ok := rl.Quotas[model]
if !ok {
return true // Unknown models are allowed
}
// Unlimited check
if quota.MaxRPM == 0 && quota.MaxTPM == 0 && quota.MaxRPD == 0 {
return true
}
// Ensure state exists
if _, ok := rl.State[model]; !ok {
rl.State[model] = &UsageStats{
DayStart: time.Now(),
}
}
rl.prune(model)
stats := rl.State[model]
// Check RPD
if quota.MaxRPD > 0 && stats.DayCount >= quota.MaxRPD {
return false
}
// Check RPM
if quota.MaxRPM > 0 && len(stats.Requests) >= quota.MaxRPM {
return false
}
// Check TPM
if quota.MaxTPM > 0 {
currentTokens := 0
for _, t := range stats.Tokens {
currentTokens += t.Count
}
if currentTokens+estimatedTokens > quota.MaxTPM {
return false
}
}
return true
}
// RecordUsage records a successful API call.
func (rl *RateLimiter) RecordUsage(model string, promptTokens, outputTokens int) {
rl.mu.Lock()
defer rl.mu.Unlock()
if _, ok := rl.State[model]; !ok {
rl.State[model] = &UsageStats{
DayStart: time.Now(),
}
}
stats := rl.State[model]
now := time.Now()
stats.Requests = append(stats.Requests, now)
stats.Tokens = append(stats.Tokens, TokenEntry{Time: now, Count: promptTokens + outputTokens})
stats.DayCount++
}
// WaitForCapacity blocks until capacity is available or context is cancelled.
func (rl *RateLimiter) WaitForCapacity(ctx context.Context, model string, tokens int) error {
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for {
if rl.CanSend(model, tokens) {
return nil
}
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
// check again
}
}
}
// Reset clears stats for a model (or all if model is empty).
func (rl *RateLimiter) Reset(model string) {
rl.mu.Lock()
defer rl.mu.Unlock()
if model == "" {
rl.State = make(map[string]*UsageStats)
} else {
delete(rl.State, model)
}
}
// ModelStats represents a snapshot of usage.
type ModelStats struct {
RPM int
MaxRPM int
TPM int
MaxTPM int
RPD int
MaxRPD int
DayStart time.Time
}
// Stats returns current stats for a model.
func (rl *RateLimiter) Stats(model string) ModelStats {
rl.mu.Lock()
defer rl.mu.Unlock()
rl.prune(model)
stats := ModelStats{}
quota, ok := rl.Quotas[model]
if ok {
stats.MaxRPM = quota.MaxRPM
stats.MaxTPM = quota.MaxTPM
stats.MaxRPD = quota.MaxRPD
}
if s, ok := rl.State[model]; ok {
stats.RPM = len(s.Requests)
stats.RPD = s.DayCount
stats.DayStart = s.DayStart
for _, t := range s.Tokens {
stats.TPM += t.Count
}
}
return stats
}
// AllStats returns stats for all tracked models.
func (rl *RateLimiter) AllStats() map[string]ModelStats {
rl.mu.Lock()
defer rl.mu.Unlock()
result := make(map[string]ModelStats)
// Collect all model names
for m := range rl.Quotas {
result[m] = ModelStats{}
}
for m := range rl.State {
result[m] = ModelStats{}
}
now := time.Now()
window := now.Add(-1 * time.Minute)
for m := range result {
// Prune inline
if s, ok := rl.State[m]; ok {
validReqs := 0
for _, t := range s.Requests {
if t.After(window) {
s.Requests[validReqs] = t
validReqs++
}
}
s.Requests = s.Requests[:validReqs]
validTokens := 0
for _, t := range s.Tokens {
if t.Time.After(window) {
s.Tokens[validTokens] = t
validTokens++
}
}
s.Tokens = s.Tokens[:validTokens]
if now.Sub(s.DayStart) >= 24*time.Hour {
s.DayStart = now
s.DayCount = 0
}
}
ms := ModelStats{}
if q, ok := rl.Quotas[m]; ok {
ms.MaxRPM = q.MaxRPM
ms.MaxTPM = q.MaxTPM
ms.MaxRPD = q.MaxRPD
}
if s, ok := rl.State[m]; ok {
ms.RPM = len(s.Requests)
ms.RPD = s.DayCount
ms.DayStart = s.DayStart
for _, t := range s.Tokens {
ms.TPM += t.Count
}
}
result[m] = ms
}
return result
}
// CountTokens calls the Google API to count tokens for a prompt.
func CountTokens(apiKey, model, text string) (int, error) {
url := fmt.Sprintf("https://generativelanguage.googleapis.com/v1beta/models/%s:countTokens?key=%s", model, apiKey)
reqBody := map[string]any{
"contents": []any{
map[string]any{
"parts": []any{
map[string]string{"text": text},
},
},
},
}
jsonBody, err := json.Marshal(reqBody)
if err != nil {
return 0, err
}
resp, err := http.Post(url, "application/json", bytes.NewBuffer(jsonBody))
if err != nil {
return 0, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return 0, fmt.Errorf("API error %d: %s", resp.StatusCode, string(body))
}
var result struct {
TotalTokens int `json:"totalTokens"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return 0, err
}
return result.TotalTokens, nil
}

View file

@ -0,0 +1,176 @@
package ratelimit
import (
"context"
"path/filepath"
"testing"
"time"
)
func TestCanSend_Good(t *testing.T) {
rl, _ := New()
rl.filePath = filepath.Join(t.TempDir(), "ratelimits.yaml")
model := "test-model"
rl.Quotas[model] = ModelQuota{MaxRPM: 10, MaxTPM: 1000, MaxRPD: 100}
if !rl.CanSend(model, 100) {
t.Errorf("Expected CanSend to return true for fresh state")
}
}
func TestCanSend_RPMExceeded_Bad(t *testing.T) {
rl, _ := New()
model := "test-rpm"
rl.Quotas[model] = ModelQuota{MaxRPM: 2, MaxTPM: 1000000, MaxRPD: 100}
rl.RecordUsage(model, 10, 10)
rl.RecordUsage(model, 10, 10)
if rl.CanSend(model, 10) {
t.Errorf("Expected CanSend to return false after exceeding RPM")
}
}
func TestCanSend_TPMExceeded_Bad(t *testing.T) {
rl, _ := New()
model := "test-tpm"
rl.Quotas[model] = ModelQuota{MaxRPM: 10, MaxTPM: 100, MaxRPD: 100}
rl.RecordUsage(model, 50, 40) // 90 tokens used
if rl.CanSend(model, 20) { // 90 + 20 = 110 > 100
t.Errorf("Expected CanSend to return false when estimated tokens exceed TPM")
}
}
func TestCanSend_RPDExceeded_Bad(t *testing.T) {
rl, _ := New()
model := "test-rpd"
rl.Quotas[model] = ModelQuota{MaxRPM: 10, MaxTPM: 1000000, MaxRPD: 2}
rl.RecordUsage(model, 10, 10)
rl.RecordUsage(model, 10, 10)
if rl.CanSend(model, 10) {
t.Errorf("Expected CanSend to return false after exceeding RPD")
}
}
func TestCanSend_UnlimitedModel_Good(t *testing.T) {
rl, _ := New()
model := "test-unlimited"
rl.Quotas[model] = ModelQuota{MaxRPM: 0, MaxTPM: 0, MaxRPD: 0}
// Should always be allowed
for i := 0; i < 1000; i++ {
rl.RecordUsage(model, 100, 100)
}
if !rl.CanSend(model, 999999) {
t.Errorf("Expected unlimited model to always allow sends")
}
}
func TestRecordUsage_PrunesOldEntries_Good(t *testing.T) {
rl, _ := New()
model := "test-prune"
rl.Quotas[model] = ModelQuota{MaxRPM: 5, MaxTPM: 1000000, MaxRPD: 100}
// Manually inject old data
oldTime := time.Now().Add(-2 * time.Minute)
rl.State[model] = &UsageStats{
Requests: []time.Time{oldTime, oldTime, oldTime},
Tokens: []TokenEntry{
{Time: oldTime, Count: 100},
{Time: oldTime, Count: 100},
},
DayStart: time.Now(),
}
// CanSend triggers prune
if !rl.CanSend(model, 10) {
t.Errorf("Expected CanSend to return true after pruning old entries")
}
stats := rl.State[model]
if len(stats.Requests) != 0 {
t.Errorf("Expected 0 requests after pruning old entries, got %d", len(stats.Requests))
}
}
func TestPersistAndLoad_Good(t *testing.T) {
tmpDir := t.TempDir()
path := filepath.Join(tmpDir, "ratelimits.yaml")
rl1, _ := New()
rl1.filePath = path
model := "persist-test"
rl1.Quotas[model] = ModelQuota{MaxRPM: 50, MaxTPM: 5000, MaxRPD: 500}
rl1.RecordUsage(model, 100, 100)
if err := rl1.Persist(); err != nil {
t.Fatalf("Persist failed: %v", err)
}
rl2, _ := New()
rl2.filePath = path
if err := rl2.Load(); err != nil {
t.Fatalf("Load failed: %v", err)
}
stats := rl2.Stats(model)
if stats.RPM != 1 {
t.Errorf("Expected RPM 1 after load, got %d", stats.RPM)
}
if stats.TPM != 200 {
t.Errorf("Expected TPM 200 after load, got %d", stats.TPM)
}
}
func TestWaitForCapacity_Ugly(t *testing.T) {
rl, _ := New()
model := "wait-test"
rl.Quotas[model] = ModelQuota{MaxRPM: 1, MaxTPM: 1000000, MaxRPD: 100}
rl.RecordUsage(model, 10, 10) // Use up the 1 RPM
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
err := rl.WaitForCapacity(ctx, model, 10)
if err != context.DeadlineExceeded {
t.Errorf("Expected DeadlineExceeded, got %v", err)
}
}
func TestDefaultQuotas_Good(t *testing.T) {
rl, _ := New()
expected := []string{
"gemini-3-pro-preview",
"gemini-3-flash-preview",
"gemini-2.0-flash",
}
for _, m := range expected {
if _, ok := rl.Quotas[m]; !ok {
t.Errorf("Expected default quota for %s", m)
}
}
}
func TestAllStats_Good(t *testing.T) {
rl, _ := New()
rl.RecordUsage("gemini-3-pro-preview", 1000, 500)
all := rl.AllStats()
if len(all) < 5 {
t.Errorf("Expected at least 5 models in AllStats, got %d", len(all))
}
pro := all["gemini-3-pro-preview"]
if pro.RPM != 1 {
t.Errorf("Expected RPM 1 for pro, got %d", pro.RPM)
}
if pro.TPM != 1500 {
t.Errorf("Expected TPM 1500 for pro, got %d", pro.TPM)
}
}

View file

@ -1,5 +1,5 @@
#!/bin/bash #!/bin/bash
# agent-runner.sh — One-at-a-time queue runner for Claude Code agents. # agent-runner.sh — Clotho-Verified Queue Runner for AgentCI.
# Deployed to agent machines, triggered by cron every 5 minutes. # Deployed to agent machines, triggered by cron every 5 minutes.
# #
# Usage: */5 * * * * ~/ai-work/agent-runner.sh >> ~/ai-work/logs/runner.log 2>&1 # Usage: */5 * * * * ~/ai-work/agent-runner.sh >> ~/ai-work/logs/runner.log 2>&1
@ -26,14 +26,7 @@ if [ -f "$LOCK_FILE" ]; then
rm -f "$LOCK_FILE" rm -f "$LOCK_FILE"
fi fi
# --- 2. Check credits --- # --- 2. Pick oldest ticket ---
# Parse remaining usage from claude. If under 5% remaining, skip.
if command -v claude &>/dev/null; then
USAGE_OUTPUT=$(claude --output-format json -p "Reply with just the word OK" 2>/dev/null | head -1 || echo "")
# Fallback: if we can't check, proceed anyway.
fi
# --- 3. Pick oldest ticket ---
TICKET=$(find "$QUEUE_DIR" -name 'ticket-*.json' -type f 2>/dev/null | sort | head -1) TICKET=$(find "$QUEUE_DIR" -name 'ticket-*.json' -type f 2>/dev/null | sort | head -1)
if [ -z "$TICKET" ]; then if [ -z "$TICKET" ]; then
exit 0 # No work exit 0 # No work
@ -42,19 +35,24 @@ fi
TICKET_BASENAME=$(basename "$TICKET") TICKET_BASENAME=$(basename "$TICKET")
echo "$(date -Iseconds) Processing ticket: $TICKET_BASENAME" echo "$(date -Iseconds) Processing ticket: $TICKET_BASENAME"
# --- 4. Lock --- # --- 3. Lock ---
echo $$ > "$LOCK_FILE" echo $$ > "$LOCK_FILE"
cleanup() { cleanup() {
rm -f "$LOCK_FILE" rm -f "$LOCK_FILE"
# Secure cleanup of env file if it still exists.
if [ -n "${ENV_FILE:-}" ] && [ -f "$ENV_FILE" ]; then
rm -f "$ENV_FILE"
fi
echo "$(date -Iseconds) Lock released." echo "$(date -Iseconds) Lock released."
} }
trap cleanup EXIT trap cleanup EXIT
# --- 5. Move to active --- # --- 4. Move to active ---
mv "$TICKET" "$ACTIVE_DIR/" mv "$TICKET" "$ACTIVE_DIR/"
TICKET_FILE="$ACTIVE_DIR/$TICKET_BASENAME" TICKET_FILE="$ACTIVE_DIR/$TICKET_BASENAME"
# --- 6. Extract ticket data --- # --- 5. Extract ticket data ---
ID=$(jq -r .id "$TICKET_FILE")
REPO_OWNER=$(jq -r .repo_owner "$TICKET_FILE") REPO_OWNER=$(jq -r .repo_owner "$TICKET_FILE")
REPO_NAME=$(jq -r .repo_name "$TICKET_FILE") REPO_NAME=$(jq -r .repo_name "$TICKET_FILE")
ISSUE_NUM=$(jq -r .issue_number "$TICKET_FILE") ISSUE_NUM=$(jq -r .issue_number "$TICKET_FILE")
@ -62,10 +60,30 @@ ISSUE_TITLE=$(jq -r .issue_title "$TICKET_FILE")
ISSUE_BODY=$(jq -r .issue_body "$TICKET_FILE") ISSUE_BODY=$(jq -r .issue_body "$TICKET_FILE")
TARGET_BRANCH=$(jq -r .target_branch "$TICKET_FILE") TARGET_BRANCH=$(jq -r .target_branch "$TICKET_FILE")
FORGE_URL=$(jq -r .forge_url "$TICKET_FILE") FORGE_URL=$(jq -r .forge_url "$TICKET_FILE")
FORGE_TOKEN=$(jq -r .forge_token "$TICKET_FILE") DUAL_RUN=$(jq -r '.dual_run // false' "$TICKET_FILE")
MODEL=$(jq -r '.model // "sonnet"' "$TICKET_FILE")
RUNNER=$(jq -r '.runner // "claude"' "$TICKET_FILE")
VERIFY_MODEL=$(jq -r '.verify_model // ""' "$TICKET_FILE")
echo "$(date -Iseconds) Issue: ${REPO_OWNER}/${REPO_NAME}#${ISSUE_NUM} - ${ISSUE_TITLE}" echo "$(date -Iseconds) Issue: ${REPO_OWNER}/${REPO_NAME}#${ISSUE_NUM} - ${ISSUE_TITLE}"
# --- 6. Load secure token from .env file ---
ENV_FILE="$QUEUE_DIR/.env.$ID"
if [ -f "$ENV_FILE" ]; then
source "$ENV_FILE"
rm -f "$ENV_FILE" # Delete immediately after sourcing
else
echo "$(date -Iseconds) ERROR: Token file not found for ticket $ID"
mv "$TICKET_FILE" "$DONE_DIR/"
exit 1
fi
if [ -z "${FORGE_TOKEN:-}" ]; then
echo "$(date -Iseconds) ERROR: FORGE_TOKEN missing from env file."
mv "$TICKET_FILE" "$DONE_DIR/"
exit 1
fi
# --- 7. Clone or update repo --- # --- 7. Clone or update repo ---
JOB_DIR="$WORK_DIR/jobs/${REPO_OWNER}-${REPO_NAME}-${ISSUE_NUM}" JOB_DIR="$WORK_DIR/jobs/${REPO_OWNER}-${REPO_NAME}-${ISSUE_NUM}"
REPO_DIR="$JOB_DIR/$REPO_NAME" REPO_DIR="$JOB_DIR/$REPO_NAME"
@ -90,8 +108,11 @@ else
cd "$REPO_DIR" cd "$REPO_DIR"
fi fi
# --- 8. Build prompt --- # --- 8. Agent execution function ---
PROMPT="You are working on issue #${ISSUE_NUM} in ${REPO_OWNER}/${REPO_NAME}. run_agent() {
local model="$1"
local log_suffix="$2"
local prompt="You are working on issue #${ISSUE_NUM} in ${REPO_OWNER}/${REPO_NAME}.
Title: ${ISSUE_TITLE} Title: ${ISSUE_TITLE}
@ -102,46 +123,76 @@ The repo is cloned at the current directory on branch '${TARGET_BRANCH}'.
Create a feature branch from '${TARGET_BRANCH}', make minimal targeted changes, commit referencing #${ISSUE_NUM}, and push. Create a feature branch from '${TARGET_BRANCH}', make minimal targeted changes, commit referencing #${ISSUE_NUM}, and push.
Then create a PR targeting '${TARGET_BRANCH}' using the forgejo MCP tools or git push." Then create a PR targeting '${TARGET_BRANCH}' using the forgejo MCP tools or git push."
# --- 9. Run AI agent --- local log_file="$LOG_DIR/${ID}-${log_suffix}.log"
MODEL=$(jq -r '.model // "sonnet"' "$TICKET_FILE") echo "$(date -Iseconds) Running ${RUNNER} (model: ${model}, suffix: ${log_suffix})..."
RUNNER=$(jq -r '.runner // "claude"' "$TICKET_FILE")
LOG_FILE="$LOG_DIR/${REPO_OWNER}-${REPO_NAME}-${ISSUE_NUM}.log"
echo "$(date -Iseconds) Running ${RUNNER} (model: ${MODEL})..." case "$RUNNER" in
case "$RUNNER" in
codex) codex)
codex exec --full-auto \ codex exec --full-auto "$prompt" > "$log_file" 2>&1
"$PROMPT" \
> "$LOG_FILE" 2>&1
;; ;;
gemini) gemini)
MODEL_FLAG="" local model_flag=""
if [ -n "$MODEL" ] && [ "$MODEL" != "sonnet" ]; then if [ -n "$model" ] && [ "$model" != "sonnet" ]; then
MODEL_FLAG="-m $MODEL" model_flag="-m $model"
fi fi
echo "$PROMPT" | gemini -p - -y $MODEL_FLAG \ echo "$prompt" | gemini -p - -y $model_flag > "$log_file" 2>&1
> "$LOG_FILE" 2>&1
;; ;;
*) *)
echo "$PROMPT" | claude -p \ echo "$prompt" | claude -p \
--model "$MODEL" \ --model "$model" \
--dangerously-skip-permissions \ --dangerously-skip-permissions \
--output-format text \ --output-format text \
> "$LOG_FILE" 2>&1 > "$log_file" 2>&1
;; ;;
esac esac
EXIT_CODE=$? return $?
echo "$(date -Iseconds) ${RUNNER} exited with code: $EXIT_CODE" }
# --- 9. Execute ---
run_agent "$MODEL" "primary"
EXIT_CODE_A=$?
FINAL_EXIT=$EXIT_CODE_A
COMMENT=""
if [ "$DUAL_RUN" = "true" ] && [ -n "$VERIFY_MODEL" ]; then
echo "$(date -Iseconds) Clotho Dual Run: resetting for verifier..."
HASH_A=$(git rev-parse HEAD)
git checkout "$TARGET_BRANCH" 2>/dev/null || true
run_agent "$VERIFY_MODEL" "verifier"
EXIT_CODE_B=$?
HASH_B=$(git rev-parse HEAD)
# Compare the two runs.
echo "$(date -Iseconds) Comparing threads..."
DIFF_COUNT=$(git diff --shortstat "$HASH_A" "$HASH_B" 2>/dev/null | wc -l || echo "1")
if [ "$DIFF_COUNT" -eq 0 ] && [ "$EXIT_CODE_A" -eq 0 ] && [ "$EXIT_CODE_B" -eq 0 ]; then
echo "$(date -Iseconds) Clotho Verification: Threads converged."
FINAL_EXIT=0
git checkout "$HASH_A" 2>/dev/null
git push origin "HEAD:refs/heads/feat/issue-${ISSUE_NUM}"
else
echo "$(date -Iseconds) Clotho Verification: Divergence detected."
FINAL_EXIT=1
COMMENT="**Clotho Verification Failed**\n\nPrimary ($MODEL) and Verifier ($VERIFY_MODEL) produced divergent results.\nPrimary Exit: $EXIT_CODE_A | Verifier Exit: $EXIT_CODE_B"
fi
else
# Standard single run — push if successful.
if [ $FINAL_EXIT -eq 0 ]; then
git push origin "HEAD:refs/heads/feat/issue-${ISSUE_NUM}" 2>/dev/null || true
fi
fi
# --- 10. Move to done --- # --- 10. Move to done ---
mv "$TICKET_FILE" "$DONE_DIR/" mv "$TICKET_FILE" "$DONE_DIR/"
# --- 11. Report result back to Forgejo --- # --- 11. Report result back to Forgejo ---
if [ $EXIT_CODE -eq 0 ]; then if [ $FINAL_EXIT -eq 0 ] && [ -z "$COMMENT" ]; then
COMMENT="Agent completed work on #${ISSUE_NUM}. Exit code: 0." COMMENT="Agent completed work on #${ISSUE_NUM}. Exit code: 0."
else elif [ -z "$COMMENT" ]; then
COMMENT="Agent failed on #${ISSUE_NUM} (exit code: ${EXIT_CODE}). Check logs on agent machine." COMMENT="Agent failed on #${ISSUE_NUM} (exit code: ${FINAL_EXIT}). Check logs on agent machine."
fi fi
curl -s -X POST "${FORGE_URL}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/issues/${ISSUE_NUM}/comments" \ curl -s -X POST "${FORGE_URL}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/issues/${ISSUE_NUM}/comments" \
@ -150,4 +201,4 @@ curl -s -X POST "${FORGE_URL}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/issues/${I
-d "$(jq -n --arg body "$COMMENT" '{body: $body}')" \ -d "$(jq -n --arg body "$COMMENT" '{body: $body}')" \
> /dev/null 2>&1 || true > /dev/null 2>&1 || true
echo "$(date -Iseconds) Done: $TICKET_BASENAME (exit: $EXIT_CODE)" echo "$(date -Iseconds) Done: $TICKET_BASENAME (exit: $FINAL_EXIT)"

View file

@ -8,7 +8,7 @@
set -euo pipefail set -euo pipefail
HOST="${1:?Usage: agent-setup.sh <user@host>}" HOST="${1:?Usage: agent-setup.sh <user@host>}"
SSH_OPTS="-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10" SSH_OPTS="-o StrictHostKeyChecking=yes -o BatchMode=yes -o ConnectTimeout=10"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
RUNNER_SCRIPT="${SCRIPT_DIR}/agent-runner.sh" RUNNER_SCRIPT="${SCRIPT_DIR}/agent-runner.sh"

203
scripts/gemini-batch-runner.sh Executable file
View file

@ -0,0 +1,203 @@
#!/bin/bash
# gemini-batch-runner.sh — Rate-limit-aware tiered Gemini analysis pipeline.
#
# Uses cheap models to prep work for expensive models, respecting TPM limits.
# Designed for Tier 1 (1M TPM) with 80% safety margin (800K effective).
#
# Usage: ./scripts/gemini-batch-runner.sh <batch-number> <pkg1> <pkg2> ...
# Example: ./scripts/gemini-batch-runner.sh 1 log config io crypt auth
set -euo pipefail
BATCH_NUM="${1:?Usage: gemini-batch-runner.sh <batch-num> <pkg1> [pkg2] ...}"
shift
PACKAGES=("$@")
if [ ${#PACKAGES[@]} -eq 0 ]; then
echo "Error: No packages specified" >&2
exit 1
fi
# --- Config ---
API_KEY="${GEMINI_API_KEY:?Set GEMINI_API_KEY}"
API_BASE="https://generativelanguage.googleapis.com/v1beta/models"
TPM_LIMIT=800000 # 80% of 1M Tier 1 limit
OUTPUT_DIR="${OUTPUT_DIR:-docs}"
REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
# Models (cheapest → most expensive)
MODEL_LITE="gemini-2.5-flash-lite"
MODEL_FLASH="gemini-3-flash-preview"
MODEL_PRO="gemini-3-pro-preview"
# --- Helpers ---
log() { echo "$(date -Iseconds) $*"; }
api_call() {
local model="$1" prompt_file="$2" max_tokens="${3:-4096}"
local tmpfile
tmpfile=$(mktemp /tmp/gemini-payload-XXXXXX.json)
trap "rm -f '$tmpfile'" RETURN
# Read prompt from file to avoid argument length limits.
jq -n --rawfile text "$prompt_file" --argjson max "$max_tokens" \
'{contents: [{parts: [{text: $text}]}], generationConfig: {maxOutputTokens: $max}}' \
> "$tmpfile"
local response
response=$(curl -s "${API_BASE}/${model}:generateContent?key=${API_KEY}" \
-H 'Content-Type: application/json' \
-d "@${tmpfile}")
# Check for errors
local error
error=$(echo "$response" | jq -r '.error.message // empty')
if [ -n "$error" ]; then
log "ERROR from $model: $error"
# Rate limited — wait and retry once
if echo "$error" | grep -qi "rate\|quota\|resource_exhausted"; then
log "Rate limited. Waiting 60s..."
sleep 60
response=$(curl -s "${API_BASE}/${model}:generateContent?key=${API_KEY}" \
-H 'Content-Type: application/json' \
-d "@${tmpfile}")
else
echo "$response"
return 1
fi
fi
echo "$response"
}
extract_text() {
jq -r '.candidates[0].content.parts[0].text // "ERROR: no output"'
}
extract_tokens() {
jq -r '.usageMetadata.totalTokenCount // 0'
}
# --- 1. Build context bundle ---
log "Building context for batch ${BATCH_NUM}: ${PACKAGES[*]}"
CONTEXT_FILE=$(mktemp /tmp/gemini-context-XXXXXX.txt)
trap "rm -f '$CONTEXT_FILE' /tmp/gemini-prompt-*.txt" EXIT
TOTAL_LINES=0
for pkg in "${PACKAGES[@]}"; do
PKG_DIR="${REPO_ROOT}/pkg/${pkg}"
if [ ! -d "$PKG_DIR" ]; then
log "WARN: pkg/${pkg} not found, skipping"
continue
fi
echo "=== Package: pkg/${pkg} ===" >> "$CONTEXT_FILE"
while IFS= read -r -d '' f; do
echo "--- $(basename "$f") ---" >> "$CONTEXT_FILE"
cat "$f" >> "$CONTEXT_FILE"
echo "" >> "$CONTEXT_FILE"
TOTAL_LINES=$((TOTAL_LINES + $(wc -l < "$f")))
done < <(find "$PKG_DIR" -maxdepth 1 -name '*.go' ! -name '*_test.go' -type f -print0 | sort -z)
done
EST_TOKENS=$((TOTAL_LINES * 4))
log "Context: ${TOTAL_LINES} lines (~${EST_TOKENS} tokens estimated)"
if [ "$EST_TOKENS" -gt "$TPM_LIMIT" ]; then
log "WARNING: Estimated tokens (${EST_TOKENS}) exceeds TPM budget (${TPM_LIMIT})"
log "Consider splitting this batch further."
exit 1
fi
# Helper: write prompt to temp file (prefix + context)
write_prompt() {
local outfile="$1" prefix="$2"
echo "$prefix" > "$outfile"
echo "" >> "$outfile"
cat "$CONTEXT_FILE" >> "$outfile"
}
# --- 2. Flash Lite: quick scan (verify batch is reasonable) ---
log "Step 1/3: Flash Lite scan..."
LITE_FILE=$(mktemp /tmp/gemini-prompt-XXXXXX.txt)
write_prompt "$LITE_FILE" "For each Go package below, give a one-line description and list the exported types. Be very concise."
LITE_RESP=$(api_call "$MODEL_LITE" "$LITE_FILE" 2048)
LITE_TOKENS=$(echo "$LITE_RESP" | extract_tokens)
log "Flash Lite used ${LITE_TOKENS} tokens"
# --- 3. Flash: structured prep ---
log "Step 2/3: Gemini 3 Flash prep..."
FLASH_FILE=$(mktemp /tmp/gemini-prompt-XXXXXX.txt)
write_prompt "$FLASH_FILE" "You are analyzing Go packages for documentation. For each package below, produce:
1. A one-line description
2. Key exported types and functions (names + one-line purpose)
3. Dependencies on other packages in this codebase (pkg/* imports only)
4. Complexity rating (simple/moderate/complex)
Output as structured markdown. Be concise."
FLASH_RESP=$(api_call "$MODEL_FLASH" "$FLASH_FILE" 4096)
FLASH_TEXT=$(echo "$FLASH_RESP" | extract_text)
FLASH_TOKENS=$(echo "$FLASH_RESP" | extract_tokens)
log "Gemini 3 Flash used ${FLASH_TOKENS} tokens"
# Check cumulative TPM before hitting Pro
CUMULATIVE=$((LITE_TOKENS + FLASH_TOKENS))
if [ "$CUMULATIVE" -gt "$((TPM_LIMIT / 2))" ]; then
log "Cumulative tokens high (${CUMULATIVE}). Pausing 60s before Pro call..."
sleep 60
fi
# --- 4. Pro: deep analysis ---
log "Step 3/3: Gemini 3 Pro deep analysis..."
PRO_FILE=$(mktemp /tmp/gemini-prompt-XXXXXX.txt)
write_prompt "$PRO_FILE" "You are a senior Go engineer documenting a framework. Analyze each package below and produce a detailed markdown document with:
For each package:
1. **Overview**: 2-3 sentence description of purpose and design philosophy
2. **Public API**: All exported types, functions, methods with type signatures and brief purpose
3. **Internal Design**: Key patterns used (interfaces, generics, dependency injection, etc.)
4. **Dependencies**: What pkg/* packages it imports and why
5. **Test Coverage Notes**: What would need testing based on the API surface
6. **Integration Points**: How other packages would use this package
Output as a single structured markdown document."
PRO_RESP=$(api_call "$MODEL_PRO" "$PRO_FILE" 8192)
PRO_TEXT=$(echo "$PRO_RESP" | extract_text)
PRO_TOKENS=$(echo "$PRO_RESP" | extract_tokens)
log "Gemini 3 Pro used ${PRO_TOKENS} tokens"
TOTAL_TOKENS=$((LITE_TOKENS + FLASH_TOKENS + PRO_TOKENS))
log "Total tokens for batch ${BATCH_NUM}: ${TOTAL_TOKENS}"
# --- 5. Save output ---
mkdir -p "${REPO_ROOT}/${OUTPUT_DIR}"
OUTPUT_FILE="${REPO_ROOT}/${OUTPUT_DIR}/pkg-batch${BATCH_NUM}-analysis.md"
cat > "$OUTPUT_FILE" << HEADER
# Package Analysis — Batch ${BATCH_NUM}
Generated by: gemini-batch-runner.sh
Models: ${MODEL_LITE}${MODEL_FLASH}${MODEL_PRO}
Date: $(date -I)
Packages: ${PACKAGES[*]}
Total tokens: ${TOTAL_TOKENS}
---
HEADER
echo "$PRO_TEXT" >> "$OUTPUT_FILE"
cat >> "$OUTPUT_FILE" << FOOTER
---
## Quick Reference (Flash Summary)
${FLASH_TEXT}
FOOTER
log "Output saved to ${OUTPUT_FILE}"
log "Done: Batch ${BATCH_NUM} (${#PACKAGES[@]} packages, ${TOTAL_TOKENS} tokens)"