go/docs/pkg-batch6-analysis.md

Package Analysis — Batch 6

Generated by: gemini-batch-runner.sh Models: gemini-2.5-flash-lite → gemini-3-flash-preview → gemini-3-pro-preview Date: 2026-02-09 Packages: ansible deploy devops framework mcp plugin unifi webview ws collect i18n cache Total tokens: 458153

---

Framework Documentation

This document provides a detailed technical analysis of the core packages within the framework.

---

=== Package: pkg/ansible ===

1. Overview

A native Go implementation of an Ansible playbook runner. Unlike wrappers that call the ansible CLI, this package parses YAML playbooks and inventories, handles variable interpolation (Jinja2-style), and executes modules directly over SSH or local connections. It is designed for embedding configuration management directly into the application without external dependencies.

2. Public API

Executor

The primary entry point for running playbooks.

type Executor struct { ... }
func NewExecutor(basePath string) *Executor
func (e *Executor) SetInventory(path string) error
func (e *Executor) SetInventoryDirect(inv *Inventory)
func (e *Executor) SetVar(key string, value any)
func (e *Executor) Run(ctx context.Context, playbookPath string) error
func (e *Executor) Close()

Callback Hooks

Callbacks to monitor execution progress.

e.OnPlayStart = func(play *Play)
e.OnTaskStart = func(host string, task *Task)
e.OnTaskEnd   = func(host string, task *Task, result *TaskResult)
e.OnPlayEnd   = func(play *Play)

Types

type Playbook struct { Plays []Play }
type Play struct { ... }
type Task struct { ... }
type Inventory struct { ... }
type TaskResult struct { Changed, Failed bool; Msg, Stdout string; ... }

3. Internal Design

• Parser: parser.go handles recursive YAML parsing, resolving role paths, and include_tasks.
• Module Dispatch: modules.go contains a switch statement dispatching tasks to native Go functions (e.g., moduleShell, moduleCopy, moduleApt) based on the task name.
• Templating: Implements a custom Jinja2-subset parser in executor.go (templateString, resolveExpr) to handle variables like {{ var }} and filters.
• SSH Abstraction: ssh.go wraps golang.org/x/crypto/ssh to handle connection pooling, key management, and sudo escalation (become).

4. Dependencies

• forge.lthn.ai/core/cli/pkg/log: structured logging.
• golang.org/x/crypto/ssh: Underlying SSH transport.
• gopkg.in/yaml.v3: YAML parsing.

5. Test Coverage Notes

• Templating Logic: Critical to test variable resolution, filters (default, bool), and nested lookups.
• Module Idempotency: Verify that file/apt modules return Changed: false when state matches.
• SSH/Sudo: Test become functionality with password handling.

6. Integration Points

Used by higher-level orchestration tools or the pkg/devops package to provision environments.

---

=== Package: pkg/devops ===

1. Overview

Manages a portable, sandboxed development environment using LinuxKit images. It handles the lifecycle (download, install, boot, stop) of a QEMU-based VM and provides utilities to bridge the host and VM (SSH forwarding, file mounting).

2. Public API

Lifecycle Management

type DevOps struct { ... }
func New(m io.Medium) (*DevOps, error)
func (d *DevOps) Install(ctx, progress func(int64, int64)) error
func (d *DevOps) Boot(ctx, opts BootOptions) error
func (d *DevOps) Stop(ctx) error
func (d *DevOps) Status(ctx) (*DevStatus, error)

Interaction

func (d *DevOps) Shell(ctx, opts ShellOptions) error
func (d *DevOps) Serve(ctx, projectDir, opts ServeOptions) error
func (d *DevOps) Test(ctx, projectDir, opts TestOptions) error
func (d *DevOps) Claude(ctx, projectDir, opts ClaudeOptions) error

3. Internal Design

• Image Management: images.go handles versioning and downloading QCOW2 images from GitHub/CDN.
• Container Abstraction: Delegates low-level VM execution to pkg/container (likely a wrapper around QEMU/LinuxKit).
• SSH Bridging: Heavily relies on exec.Command("ssh", ...) to tunnel ports, mount filesystems via SSHFS, and forward agents.
• Auto-Detection: DetectServeCommand and DetectTestCommand inspect project files (package.json, go.mod) to determine how to run projects.

4. Dependencies

• pkg/container: VM runtime management.
• pkg/io: Filesystem abstraction.
• pkg/config: Configuration loading.

5. Test Coverage Notes

• Command Detection: Unit tests for Detect*Command with various mock file structures.
• SSH Config: Verify ensureHostKey correctly parses and updates known_hosts.

6. Integration Points

The primary interface for CLI commands (core dev ...). Bridges pkg/mcp agents into a sandboxed environment.

---

=== Package: pkg/framework ===

1. Overview

A facade package that re-exports types and functions from pkg/framework/core. It serves as the primary entry point for the Dependency Injection (DI) framework, providing a cleaner import path for consumers.

2. Public API

Re-exports Core, Option, Message, Startable, Stoppable, and constructors like New, WithService, ServiceFor.

3. Internal Design

Purely structural; contains type aliases and variable assignments to expose the internal core package.

4. Dependencies

• forge.lthn.ai/core/cli/pkg/framework/core

5. Test Coverage Notes

No logic to test directly; coverage belongs in pkg/framework/core.

6. Integration Points

Imported by main.go and all service packages to register themselves with the DI container.

---

=== Package: pkg/mcp ===

1. Overview

Implements a Model Context Protocol (MCP) server. It acts as a bridge between AI models (like Claude) and the system tools, exposing file operations, process management, web browsing, and RAG capabilities as callable tools.

2. Public API

type Service struct { ... }
func New(opts ...Option) (*Service, error)
func (s *Service) Run(ctx context.Context) error
func (s *Service) ServeTCP(ctx, addr string) error
func (s *Service) ServeStdio(ctx) error

Configuration Options

func WithWorkspaceRoot(root string) Option
func WithProcessService(svc *process.Service) Option
func WithWSHub(hub *ws.Hub) Option

3. Internal Design

• Tool Registry: Registers functions (e.g., s.readFile, s.processStart) with the MCP SDK.
• Sandboxing: WithWorkspaceRoot creates a restricted io.Medium to prevent AI from accessing files outside the workspace.
• Subsystems: Segregates tools into files (tools_rag.go, tools_webview.go, etc.).
• Transports: Supports Stdio (for CLI pipes), TCP, and Unix sockets.

4. Dependencies

• github.com/modelcontextprotocol/go-sdk: MCP protocol implementation.
• pkg/process, pkg/ws, pkg/rag, pkg/webview: Capability providers.
• pkg/io: Filesystem access.

5. Test Coverage Notes

• Security: Verify WithWorkspaceRoot actually prevents accessing /etc/passwd.
• Tool I/O: Ensure JSON inputs/outputs for tools map correctly to internal service calls.

6. Integration Points

Runs as a standalone server or subprocess for AI agents. Consumes pkg/process and pkg/webview.

---

=== Package: pkg/plugin ===

1. Overview

Provides a plugin system for the CLI, allowing extensions to be installed from GitHub. It manages a local registry of installed plugins and handles their lifecycle (install, update, remove).

2. Public API

type Plugin interface { Name(); Version(); Init(); Start(); Stop() }
type Registry struct { ... }
func NewRegistry(m io.Medium, basePath string) *Registry
func (r *Registry) List() []*PluginConfig

type Installer struct { ... }
func (i *Installer) Install(ctx, source string) error // source: "org/repo"
func (i *Installer) Update(ctx, name string) error

3. Internal Design

• Manifest: Relies on plugin.json in the root of the plugin repo.
• Git Integration: Uses the gh CLI via exec to clone/pull repositories.
• Persistence: Stores plugin metadata in a registry.json file.

4. Dependencies

• pkg/io: Filesystem access.
• pkg/framework/core: Error handling.
• External gh and git binaries.

5. Test Coverage Notes

• Manifest Validation: Test valid/invalid plugin.json parsing.
• Source Parsing: Test parsing of org/repo, org/repo@v1, etc.

6. Integration Points

Used by the main CLI application to load dynamic commands at startup.

---

=== Package: pkg/unifi ===

1. Overview

A strongly-typed client for Ubiquiti UniFi controllers. It wraps the unpoller SDK but adds configuration resolution (config file -> env var -> flags) and specific helper methods for data extraction not easily accessible in the raw SDK.

2. Public API

func NewFromConfig(...) (*Client, error)
func (c *Client) GetClients(filter ClientFilter) ([]*uf.Client, error)
func (c *Client) GetDeviceList(site, type string) ([]DeviceInfo, error)
func (c *Client) GetRoutes(site string) ([]Route, error)
func (c *Client) GetNetworks(site string) ([]NetworkConf, error)

3. Internal Design

• Config Cascade: ResolveConfig logic ensures hierarchical configuration overrides.
• Raw API Fallback: Methods like GetRoutes and GetNetworks bypass the SDK's high-level structs to hit specific API endpoints (/api/s/%s/stat/routing) and decode into custom structs.

4. Dependencies

• github.com/unpoller/unifi/v5: Base SDK.
• pkg/config: Config file management.

5. Test Coverage Notes

• Config Resolution: Verify priority order (Flag > Env > Config).
• JSON Unmarshalling: Test GetRoutes/GetNetworks against sample JSON responses from a controller.

6. Integration Points

Used by network management plugins or diagnostic tools.

---

=== Package: pkg/webview ===

1. Overview

A browser automation package using the Chrome DevTools Protocol (CDP). It is designed for headless testing, scraping, and AI-driven interaction. It supports advanced features like Angular-specific waiting strategies.

2. Public API

type Webview struct { ... }
func New(opts ...Option) (*Webview, error)
func (wv *Webview) Navigate(url string) error
func (wv *Webview) Click(selector string) error
func (wv *Webview) Type(selector, text string) error
func (wv *Webview) Screenshot() ([]byte, error)
func (wv *Webview) Evaluate(script string) (any, error)

Angular Helpers

func NewAngularHelper(wv *Webview) *AngularHelper
func (ah *AngularHelper) WaitForAngular() error
func (ah *AngularHelper) GetNgModel(selector string) (any, error)

3. Internal Design

• CDP Client: cdp.go implements a raw WebSocket client for the DevTools protocol, managing message IDs and event dispatching.
• Action Sequence: actions.go implements the Command pattern (Action interface) to chain browser interactions.
• Angular Awareness: angular.go injects JS to probe window.ng or getAllAngularRootElements to interact with Angular's Zone.js and component state.

4. Dependencies

• github.com/gorilla/websocket: WebSocket transport.

5. Test Coverage Notes

• CDP Protocol: Mock the WebSocket server to ensure correct message serialization/response handling.
• Angular Helpers: Requires an actual Angular app (or mock environment) to verify Zone.js stabilization logic.

6. Integration Points

Used by pkg/mcp to expose browser tools to AI agents.

---

=== Package: pkg/ws ===

1. Overview

A concurrent WebSocket hub implementation. It handles client registration, broadcasting, and channel-based subscriptions (e.g., subscribing only to logs for a specific process).

2. Public API

type Hub struct { ... }
func NewHub() *Hub
func (h *Hub) Run(ctx)
func (h *Hub) Handler() http.HandlerFunc
func (h *Hub) SendProcessOutput(id, output string) error
func (h *Hub) SendEvent(type string, data any) error

3. Internal Design

• Hub Pattern: Central Hub struct manages a map of clients and channels. Uses unbuffered channels for registration to avoid race conditions.
• Channel Routing: Maintains a map[string]map[*Client]bool to route messages efficiently to subscribers.
• Goroutines: Each client spawns a readPump and writePump to handle I/O concurrently.

4. Dependencies

• github.com/gorilla/websocket

5. Test Coverage Notes

• Concurrency: Test registering/unregistering clients while broadcasting heavily.
• Subscription: Verify messages only go to subscribed clients.

6. Integration Points

Used by pkg/mcp to stream process output to a web UI.

---

=== Package: pkg/collect ===

1. Overview

A data collection pipeline for gathering data from various sources (GitHub, Forums, Market Data, Papers). It standardizes the collection process into a Collector interface and handles common concerns like rate limiting, state tracking (resume support), and formatting.

2. Public API

type Collector interface {
    Name() string
    Collect(ctx, cfg *Config) (*Result, error)
}
type Excavator struct { Collectors []Collector ... }
func (e *Excavator) Run(ctx, cfg) (*Result, error)

3. Internal Design

• Excavator: The orchestrator that runs collectors sequentially.
• RateLimiter: Implements token bucket-like delays per source type (e.g., GitHub, CoinGecko).
• State Persistence: Saves a JSON cursor file to resume interrupted collections.
• Formatters: process.go converts raw HTML/JSON into Markdown for easier consumption by LLMs.

4. Dependencies

• pkg/io: File storage.
• golang.org/x/net/html: HTML parsing for forums/papers.
• gh CLI: Used for GitHub data fetching.

5. Test Coverage Notes

• HTML Parsing: Test ParsePostsFromHTML with sample forum HTML.
• Rate Limit: Verify Wait respects context cancellation and time delays.

6. Integration Points

Used as a standalone CLI command or by AI agents to gather context.

---

=== Package: pkg/i18n ===

1. Overview

A sophisticated internationalization library that goes beyond simple key-value lookups. It includes a grammar engine to handle pluralization, verb conjugation, and semantic sentence generation ("Subject verbed object").

2. Public API

func T(key string, args ...any) string // Main translation function
func S(noun string, value any) *Subject // Create a semantic subject
func N(format string, value any) string // Number formatting
func SetLanguage(lang string) error

3. Internal Design

• Grammar Engine: grammar.go applies rules for past tense, gerunds, and pluralization based on language-specific JSON rules or algorithmic fallbacks.
• Namespace Handlers: handler.go intercepts keys like i18n.count.* or i18n.done.* to auto-generate phrases based on the grammar engine.
• Loader: loader.go flattens nested JSON translation files and extracts grammar rules (gram.verb.*).

4. Dependencies

• golang.org/x/text/language: Standard language tag parsing.

5. Test Coverage Notes

• Pluralization: Test complex rules (e.g., Slavic/Arabic plural categories).
• Grammar generation: Test PastTense and Gerund for regular and irregular English verbs.

6. Integration Points

Used pervasively across the CLI for all user-facing output.

---

=== Package: pkg/cache ===

1. Overview

A simple, file-based JSON cache with Time-To-Live (TTL) support.

2. Public API

func New(m io.Medium, baseDir string, ttl time.Duration) (*Cache, error)
func (c *Cache) Get(key string, dest interface{}) (bool, error)
func (c *Cache) Set(key string, data interface{}) error

3. Internal Design

• Stores data as JSON files: { "data": ..., "expires_at": ... }.
• Uses pkg/io abstraction for storage independence.

4. Dependencies

• pkg/io

5. Test Coverage Notes

• Expiry: Verify Get returns false after TTL expires.
• Serialization: Ensure struct round-tripping works correctly.

6. Integration Points

Used by pkg/collect or pkg/plugin to cache API responses (e.g., GitHub releases).

---

Quick Reference (Flash Summary)

pkg/ansible

Description: Implements a Go-based Ansible-lite engine for executing playbooks and roles over SSH with YAML parsing and fact gathering.

• Executor (Type): Main runner that manages inventory, variables, and execution state.
• NewExecutor (Func): Initialises the executor with a base path for roles and playbooks.
• Task (Type): Represents a single Ansible task with module parameters and conditional logic.
• Run (Func): Parses and executes a playbook from a file path.
• Inventory (Type): Holds the host and group structure for targeting remote machines. Dependencies: pkg/log Complexity: Complex

pkg/devops

Description: Manages a portable development environment using LinuxKit VM images and QEMU/SSH integration.

• DevOps (Type): Core service for environment lifecycle, mounting, and tool execution.
• Boot (Func): Configures and starts the dev environment container.
• Claude (Func): Launches a sandboxed AI session with project mounting and auth forwarding.
• Serve (Func): Auto-detects project types and runs local development servers inside the VM.
• ImageManager (Type): Handles downloading and updating dev environment system images. Dependencies: pkg/config, pkg/container, pkg/io, pkg/devops/sources Complexity: Moderate

pkg/framework

Description: Provides a facade for the Core dependency injection and service runtime framework.

• Core (Type): The central DI container and service registry.
• New (Func): Creates and initialises a new Core instance.
• ServiceFor (Func): Retrieves a type-safe service from the container by name.
• Runtime (Type): Manages the lifecycle and configuration of application services. Dependencies: pkg/framework/core Complexity: Simple

pkg/mcp

Description: Implements a Model Context Protocol (MCP) server providing filesystem, process, and RAG tools to AI agents.

• Service (Type): The MCP server instance managing tools and transport.
• New (Func): Initialises the server with workspace sandboxing and optional service integration.
• WithWorkspaceRoot (Option): Restricts file operations to a specific directory for security.
• ServeTCP / ServeStdio (Func): Transport-specific server implementations.
• Subsystem (Interface): Allows external packages to register custom toolsets. Dependencies: pkg/io, pkg/io/local, pkg/log, pkg/process, pkg/ws, pkg/ai, pkg/rag, pkg/webview Complexity: Complex

pkg/plugin

Description: A dynamic plugin system that manages gits-based extensions for the core CLI.

• Plugin (Interface): Defines the lifecycle (Init/Start/Stop) for extensions.
• Registry (Type): Manages metadata and persistence for installed plugins.
• Installer (Type): Handles git-based installation and updates from GitHub.
• Loader (Type): Discovers and loads plugin manifests from the filesystem. Dependencies: pkg/framework/core, pkg/io Complexity: Moderate

pkg/unifi

Description: A wrapper for the UniFi SDK providing simplified access to network controller devices, clients, and routing.

• Client (Type): Main API client for interacting with UniFi controllers.
• NewFromConfig (Func): Resolves credentials from config/env and initialises a client.
• GetClients (Func): Returns a filtered list of connected wired and wireless clients.
• GetDeviceList (Func): Returns flat metadata for infrastructure hardware (APs, Switches, Gateways). Dependencies: pkg/config, pkg/log Complexity: Simple

pkg/webview

Description: Provides browser automation and framework-specific testing (Angular) via the Chrome DevTools Protocol (CDP).

• Webview (Type): High-level controller for browser navigation and interaction.
• CDPClient (Type): Manages raw WebSocket communication with Chrome.
• AngularHelper (Type): Specialized tools for waiting on Zone.js stability and interacting with Angular components.
• ConsoleWatcher (Type): Captures and filters browser console logs and exceptions.
• ActionSequence (Type): Chains multiple browser interactions (click, type, navigate) into a single execution. Dependencies: None Complexity: Complex

pkg/ws

Description: Implements a WebSocket hub for real-time message broadcasting and channel-based subscriptions.

• Hub (Type): Manages client connections, message loops, and channel routing.
• Run (Func): Starts the central event loop for broadcasting and registration.
• Broadcast (Func): Sends a message to every connected client.
• SendToChannel (Func): Targets messages to clients subscribed to specific topics (e.g., process logs). Dependencies: None Complexity: Moderate

pkg/collect

Description: An orchestration subsystem for scraping and processing data from GitHub, forums, market APIs, and academic sources.

• Collector (Interface): Standard interface for data sources (e.g., GitHubCollector, BitcoinTalkCollector).
• Excavator (Type): Orchestrates multiple collectors with rate limiting and state resume support.
• Processor (Type): Converts raw HTML/JSON data into cleaned Markdown files.
• RateLimiter (Type): Manages per-source API delays to prevent IP bans.
• State (Type): Persists progress to allow incremental collection runs. Dependencies: pkg/framework/core, pkg/io Complexity: Complex

pkg/i18n

Description: A localization engine supporting nested translations, grammatical rules (plurals, gender, verbs), and semantic composition.

• Service (Type): Manages loaded locales and message resolution logic.
• T / Raw (Func): Translates keys with or without automatic grammatical composition.
• Subject (Type): Provides context (count, gender, formality) for semantic intent templates.
• RegisterLocales (Func): Allows packages to register embedded translation files.
• GrammarData (Type): Defines language-specific rules for past tense, gerunds, and articles. Dependencies: None Complexity: Complex

pkg/cache

Description: Provides a persistent, file-based JSON cache with TTL-based expiration.

• Cache (Type): Main handler for storing and retrieving cached entries.
• Entry (Type): Internal wrapper for data, including cached and expiry timestamps.
• Get / Set (Func): Thread-safe operations for managing cached data.
• Age (Func): Calculates how long an item has been stored. Dependencies: pkg/io Complexity: Simple