diff --git a/docs/ecosystem.md b/docs/ecosystem.md new file mode 100644 index 0000000..551f7e6 --- /dev/null +++ b/docs/ecosystem.md @@ -0,0 +1,457 @@ +# Core Go Ecosystem + +The Core Go ecosystem is a set of 19 standalone Go modules that form the infrastructure backbone for the host-uk platform and the Lethean network. All modules are hosted under the `forge.lthn.ai/core/` organisation. Each module has its own repository, independent versioning, and a `docs/` directory. + +The CLI framework documented in the rest of this site (`forge.lthn.ai/core/cli`) is one node in this graph. The satellite packages listed here are separate repositories that the CLI imports or that stand alone as libraries. + +--- + +## Module Index + +| Package | Module Path | Managed By | +|---------|-------------|-----------| +| [go-inference](#go-inference) | `forge.lthn.ai/core/go-inference` | Virgil | +| [go-mlx](#go-mlx) | `forge.lthn.ai/core/go-mlx` | Virgil | +| [go-rocm](#go-rocm) | `forge.lthn.ai/core/go-rocm` | Charon | +| [go-ml](#go-ml) | `forge.lthn.ai/core/go-ml` | Virgil | +| [go-ai](#go-ai) | `forge.lthn.ai/core/go-ai` | Virgil | +| [go-agentic](#go-agentic) | `forge.lthn.ai/core/go-agentic` | Charon | +| [go-rag](#go-rag) | `forge.lthn.ai/core/go-rag` | Charon | +| [go-i18n](#go-i18n) | `forge.lthn.ai/core/go-i18n` | Virgil | +| [go-html](#go-html) | `forge.lthn.ai/core/go-html` | Charon | +| [go-crypt](#go-crypt) | `forge.lthn.ai/core/go-crypt` | Virgil | +| [go-scm](#go-scm) | `forge.lthn.ai/core/go-scm` | Charon | +| [go-p2p](#go-p2p) | `forge.lthn.ai/core/go-p2p` | Charon | +| [go-devops](#go-devops) | `forge.lthn.ai/core/go-devops` | Virgil | +| [go-help](#go-help) | `forge.lthn.ai/core/go-help` | Charon | +| [go-ratelimit](#go-ratelimit) | `forge.lthn.ai/core/go-ratelimit` | Charon | +| [go-session](#go-session) | `forge.lthn.ai/core/go-session` | Charon | +| [go-store](#go-store) | `forge.lthn.ai/core/go-store` | Charon | +| [go-ws](#go-ws) | `forge.lthn.ai/core/go-ws` | Charon | +| [go-webview](#go-webview) | `forge.lthn.ai/core/go-webview` | Charon | + +--- + +## Dependency Graph + +The graph below shows import relationships. An arrow `A → B` means A imports B. + +``` +go-inference (no dependencies — foundation contract) + ↑ + ├── go-mlx (CGO, Apple Silicon Metal GPU) + ├── go-rocm (AMD ROCm, llama-server subprocess) + └── go-ml (scoring engine, backends, orchestrator) + ↑ + └── go-ai (MCP hub, 49 tools) + ↑ + └── go-agentic (service lifecycle, allowances) + +go-rag (Qdrant + Ollama, standalone) + ↑ + └── go-ai + +go-i18n (grammar engine, standalone; Phase 2a imports go-mlx) + +go-crypt (standalone) + ↑ + ├── go-p2p (UEPS wire protocol) + └── go-scm (AgentCI dispatch) + +go-store (SQLite KV, standalone) + ↑ + ├── go-ratelimit (sliding window limiter) + ├── go-session (transcript parser) + └── go-agentic + +go-ws (WebSocket hub, standalone) + ↑ + └── go-ai + +go-webview (CDP client, standalone) + ↑ + └── go-ai + +go-html (DOM compositor, standalone) + +go-help (help catalogue, standalone) + +go-devops (Ansible, build, infrastructure — imports go-scm) +``` + +The CLI framework (`forge.lthn.ai/core/cli`) has internal equivalents of several of these packages (`pkg/rag`, `pkg/ws`, `pkg/webview`, `pkg/i18n`) that were developed in parallel. The satellite packages are the canonical standalone versions intended for use outside the CLI binary. + +--- + +## Package Descriptions + +### go-inference + +**Module:** `forge.lthn.ai/core/go-inference` + +Zero-dependency interface package that defines the common contract for all inference backends in the ecosystem: + +- `TextModel` — the top-level model interface (`Generate`, `Stream`, `Close`) +- `Backend` — hardware/runtime abstraction (Metal, ROCm, CPU, remote) +- `Token` — streaming token type with metadata + +No concrete implementations live here. Any package that needs to call inference without depending on a specific hardware library imports `go-inference` and receives an implementation at runtime. + +--- + +### go-mlx + +**Module:** `forge.lthn.ai/core/go-mlx` + +Native Metal GPU inference for Apple Silicon using CGO bindings to `mlx-c` (the C API for Apple's MLX framework). Implements the `go-inference` interfaces. + +Build requirements: +- macOS 13+ (Ventura) on Apple Silicon +- `mlx-c` installed (`brew install mlx`) +- CGO enabled: `CGO_CFLAGS` and `CGO_LDFLAGS` must reference the mlx-c headers and library + +Features: +- Loads GGUF and MLX-format models +- Streaming token generation directly on GPU +- Quantised model support (Q4, Q8) +- Phase 4 backend abstraction in progress — will allow hot-swapping backends at runtime + +Local path: `/Users/snider/Code/go-mlx` + +--- + +### go-rocm + +**Module:** `forge.lthn.ai/core/go-rocm` + +AMD ROCm GPU inference for Linux. Rather than using CGO, this package manages a `llama-server` subprocess (from llama.cpp) compiled with ROCm support and communicates over its HTTP API. + +Features: +- Subprocess lifecycle management (start, health-check, restart on crash) +- OpenAI-compatible HTTP client wrapping llama-server's API +- Implements `go-inference` interfaces +- Targeted at the homelab RX 7800 XT running Ubuntu 24.04 + +Managed by Charon (Linux homelab). + +--- + +### go-ml + +**Module:** `forge.lthn.ai/core/go-ml` + +Scoring engine, backend registry, and agent orchestration layer. The hub that connects models from `go-mlx`, `go-rocm`, and future backends into a unified interface. + +Features: +- Backend registry: register multiple inference backends, select by capability +- Scoring pipeline: evaluate model outputs against rubrics +- Agent orchestrator: coordinate multi-step inference tasks +- ~3.5K LOC + +--- + +### go-ai + +**Module:** `forge.lthn.ai/core/go-ai` + +MCP (Model Context Protocol) server hub with 49 registered tools. Acts as the primary facade for AI capabilities in the ecosystem. + +Features: +- 49 MCP tools covering file operations, RAG, metrics, process management, WebSocket, and CDP/webview +- Imports `go-ml`, `go-rag`, `go-mlx` +- Can run as stdio MCP server or TCP MCP server +- AI usage metrics recorded to JSONL + +Run the MCP server: + +```bash +# stdio (for Claude Desktop / Claude Code) +core mcp serve + +# TCP +MCP_ADDR=:9000 core mcp serve +``` + +--- + +### go-agentic + +**Module:** `forge.lthn.ai/core/go-agentic` + +Service lifecycle and allowance management for autonomous agents. Handles: + +- Agent session tracking and state persistence +- Allowance system: budget constraints on tool calls, token usage, and wall-clock time +- Integration with `go-store` for persistence +- REST client for the PHP `core-agentic` backend + +Managed by Charon. + +--- + +### go-rag + +**Module:** `forge.lthn.ai/core/go-rag` + +Retrieval-Augmented Generation pipeline using Qdrant for vector storage and Ollama for embeddings. + +Features: +- `ChunkMarkdown`: semantic splitting by H2 headers and paragraphs with overlap +- `Ingest`: crawl a directory of Markdown files, embed, and store in Qdrant +- `Query`: semantic search returning ranked `QueryResult` slices +- `FormatResultsContext`: formats results as XML tags for LLM prompt injection +- Clients: `QdrantClient` and `OllamaClient` wrapping their respective Go SDKs + +Managed by Charon. + +--- + +### go-i18n + +**Module:** `forge.lthn.ai/core/go-i18n` + +Grammar engine for natural-language generation. Goes beyond key-value lookup tables to handle pluralisation, verb conjugation, past tense, gerunds, and semantic sentence construction ("Subject verbed object"). + +Features: +- `T(key, args...)` — main translation function +- `S(noun, value)` — semantic subject with grammatical context +- Language rules defined in JSON; algorithmic fallbacks for irregular verbs +- **GrammarImprint**: a linguistic hash (reversal of the grammar engine) used as a semantic fingerprint — part of the Lethean identity verification stack +- Phase 2a (imports `go-mlx` for language model-assisted reversal) currently blocked on `go-mlx` Phase 4 + +Local path: `/Users/snider/Code/go-i18n` + +--- + +### go-html + +**Module:** `forge.lthn.ai/core/go-html` + +HLCRF DOM compositor — a programmatic HTML/DOM construction library targeting both server-side rendering and WASM (browser). + +HLCRF stands for Header, Left, Content, Right, Footer — the region layout model used throughout the CLI's terminal UI and web rendering layer. + +Features: +- Composable region-based layout (mirrors the terminal `Composite` in `pkg/cli`) +- WASM build target: runs in the browser without JavaScript +- Used by the LEM Chat UI and web SDK generation + +Managed by Charon. + +--- + +### go-crypt + +**Module:** `forge.lthn.ai/core/go-crypt` + +Cryptographic primitives, authentication, and trust policy enforcement. + +Features: +- Password hashing (Argon2id with tuned parameters) +- Symmetric encryption (ChaCha20-Poly1305, AES-GCM) +- Key derivation (HKDF, Scrypt) +- OpenPGP challenge-response authentication +- Trust policies: define and evaluate access rules +- Foundation for the UEPS (User-controlled Encryption Policy System) wire protocol in `go-p2p` + +--- + +### go-scm + +**Module:** `forge.lthn.ai/core/go-scm` + +Source control management and CI integration, including the AgentCI dispatch system. + +Features: +- Forgejo and Gitea API clients (typed wrappers) +- GitHub integration via the `gh` CLI +- `AgentCI`: dispatches AI work items to agent runners over SSH using Charm stack libraries (`soft-serve`, `keygen`, `melt`, `wishlist`) +- PR lifecycle management: create, review, merge, label +- JSONL job journal for audit trails + +Managed by Charon. + +--- + +### go-p2p + +**Module:** `forge.lthn.ai/core/go-p2p` + +Peer-to-peer mesh networking implementing the UEPS (User-controlled Encryption Policy System) wire protocol. + +Features: +- UEPS: consent-gated TLV frames with Ed25519 consent tokens and an Intent-Broker +- Peer discovery and mesh routing +- Encrypted relay transport +- Integration with `go-crypt` for all cryptographic operations + +This is a core component of the Lethean Web3 network layer. + +Managed by Charon (Linux homelab). + +--- + +### go-devops + +**Module:** `forge.lthn.ai/core/go-devops` + +Infrastructure automation, build tooling, and release pipeline utilities, intended as a standalone library form of what the Core CLI provides as commands. + +Features: +- Ansible-lite engine (native Go SSH playbook execution) +- LinuxKit image building and VM lifecycle +- Multi-target binary build and release +- Integration with `go-scm` for repository operations + +--- + +### go-help + +**Module:** `forge.lthn.ai/core/go-help` + +Embedded documentation catalogue with full-text search and an optional HTTP server for serving help content. + +Features: +- YAML-frontmatter Markdown topic parsing +- In-memory reverse index with title/heading/body scoring +- Snippet extraction with keyword highlighting +- `HTTP server` mode: serve the catalogue as a documentation site +- Used by the `core pkg search` command and the `pkg/help` package inside the CLI + +Managed by Charon. + +--- + +### go-ratelimit + +**Module:** `forge.lthn.ai/core/go-ratelimit` + +Sliding-window rate limiter with a SQLite persistence backend. + +Features: +- Token bucket and sliding-window algorithms +- SQLite backend via `go-store` for durable rate state across restarts +- HTTP middleware helper +- Used by `go-ai` and `go-agentic` to enforce per-agent API quotas + +Managed by Charon. + +--- + +### go-session + +**Module:** `forge.lthn.ai/core/go-session` + +Claude Code JSONL transcript parser and visualisation toolkit (standalone version of `pkg/session` inside the CLI). + +Features: +- `ParseTranscript(path)`: reads `.jsonl` session files and reconstructs tool use timelines +- `ListSessions(dir)`: scans a Claude projects directory for session files +- `Search(dir, query)`: full-text search across sessions +- `RenderHTML(sess, path)`: single-file HTML visualisation +- `RenderMP4(sess, path)`: terminal video replay via VHS + +Managed by Charon. + +--- + +### go-store + +**Module:** `forge.lthn.ai/core/go-store` + +SQLite-backed key-value store with reactive change notification. + +Features: +- `Get`, `Set`, `Delete`, `List` over typed keys +- `Watch(key, handler)`: register a callback that fires on change +- `OnChange(handler)`: subscribe to all changes +- Used by `go-ratelimit`, `go-session`, and `go-agentic` for lightweight persistence + +Managed by Charon. + +--- + +### go-ws + +**Module:** `forge.lthn.ai/core/go-ws` + +WebSocket hub with channel-based subscriptions and an optional Redis pub/sub bridge for multi-instance deployments. + +Features: +- Hub pattern: central registry of connected clients +- Channel routing: `SendToChannel(topic, msg)` delivers only to subscribers +- Redis bridge: publish messages from one instance, receive on all +- HTTP handler: `hub.Handler()` for embedding in any Go HTTP server +- `SendProcessOutput(id, line)`: convenience method for streaming process logs + +Managed by Charon. + +--- + +### go-webview + +**Module:** `forge.lthn.ai/core/go-webview` + +Chrome DevTools Protocol (CDP) client for browser automation, testing, and AI-driven web interaction (standalone version of `pkg/webview` inside the CLI). + +Features: +- Navigation, click, type, screenshot +- `Evaluate(script)`: arbitrary JavaScript execution with result capture +- Console capture and filtering +- Angular-aware helpers: `WaitForAngular()`, `GetNgModel(selector)` +- `ActionSequence`: chain interactions into a single call +- Used by `go-ai` to expose browser tools to MCP agents + +Managed by Charon. + +--- + +## Forge Repository Paths + +All repositories are hosted at `forge.lthn.ai` (Forgejo). SSH access uses port 2223: + +``` +ssh://git@forge.lthn.ai:2223/core/go-inference.git +ssh://git@forge.lthn.ai:2223/core/go-mlx.git +ssh://git@forge.lthn.ai:2223/core/go-rocm.git +ssh://git@forge.lthn.ai:2223/core/go-ml.git +ssh://git@forge.lthn.ai:2223/core/go-ai.git +ssh://git@forge.lthn.ai:2223/core/go-agentic.git +ssh://git@forge.lthn.ai:2223/core/go-rag.git +ssh://git@forge.lthn.ai:2223/core/go-i18n.git +ssh://git@forge.lthn.ai:2223/core/go-html.git +ssh://git@forge.lthn.ai:2223/core/go-crypt.git +ssh://git@forge.lthn.ai:2223/core/go-scm.git +ssh://git@forge.lthn.ai:2223/core/go-p2p.git +ssh://git@forge.lthn.ai:2223/core/go-devops.git +ssh://git@forge.lthn.ai:2223/core/go-help.git +ssh://git@forge.lthn.ai:2223/core/go-ratelimit.git +ssh://git@forge.lthn.ai:2223/core/go-session.git +ssh://git@forge.lthn.ai:2223/core/go-store.git +ssh://git@forge.lthn.ai:2223/core/go-ws.git +ssh://git@forge.lthn.ai:2223/core/go-webview.git +``` + +HTTPS authentication is not available on Forge. Always use SSH remotes. + +--- + +## Go Workspace Setup + +The satellite packages can be used together in a Go workspace. After cloning the repositories you need: + +```bash +go work init +go work use ./go-inference ./go-mlx ./go-rag ./go-ai # add as needed +go work sync +``` + +The CLI repository already uses a Go workspace that includes `cmd/core-gui`, `cmd/bugseti`, and `cmd/examples/*`. + +--- + +## See Also + +- [index.md](index.md) — Main documentation hub +- [getting-started.md](getting-started.md) — CLI installation +- [configuration.md](configuration.md) — `repos.yaml` registry format diff --git a/docs/plans/2026-02-20-authentik-traefik-plan.md b/docs/plans/2026-02-20-authentik-traefik-plan.md new file mode 100644 index 0000000..091a082 --- /dev/null +++ b/docs/plans/2026-02-20-authentik-traefik-plan.md @@ -0,0 +1,1163 @@ +# Authentik + Traefik Integration Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Deploy Authentik as the identity provider, wire it into Traefik's forward auth, and add OIDC/header middleware to go-api so protected services get authenticated user context. + +**Architecture:** Authentik runs alongside existing services on de2 (production). Traefik's file provider loads a `forwardAuth` middleware definition pointing at Authentik's outpost. Services opt-in via Docker label `middlewares: authentik@file`. go-api gains a `WithAuthentik()` option that extracts user identity from Authentik headers (forward auth mode) or validates JWTs directly (API client mode). + +**Tech Stack:** Authentik 2025.2, Traefik v3.6, Go 1.25, coreos/go-oidc/v3, golang.org/x/oauth2 + +**Design doc:** `docs/plans/2026-02-20-go-api-design.md` (Authentik section) + +**Key references:** +- Traefik role: `/Users/snider/Code/DevOps/roles/traefik/` +- Authentik role: `/Users/snider/Code/DevOps/roles/authentik/` +- Forward auth template: `/Users/snider/Code/DevOps/roles/traefik/templates/dynamic-authentik.yml.j2` +- go-api repo: `/Users/snider/Code/go-api/` + +--- + +## Current State + +The Ansible infrastructure is **already built but not activated**: + +| Component | Status | Location | +|-----------|--------|----------| +| Traefik v3.6 role | Deployed on de2 | `roles/traefik/` | +| Authentik 2025.2 role | Written, **never deployed** | `roles/authentik/` | +| Forward auth middleware template | Written, conditional on `traefik_authentik_enabled` | `dynamic-authentik.yml.j2` | +| Outpost routing in Authentik compose | Pre-configured | `roles/authentik/templates/docker-compose.yml.j2` | +| 5 services with `authentik@file` | Labels present, middleware not yet available | `prod_rebuild.yml` | +| go-api Authentik middleware | **Not started** | — | + +**Headers Authentik will pass to go-api (via Traefik):** +``` +X-authentik-username, X-authentik-groups, X-authentik-entitlements, +X-authentik-email, X-authentik-name, X-authentik-uid, X-authentik-jwt, +X-authentik-meta-jwks, X-authentik-meta-outpost, X-authentik-meta-provider, +X-authentik-meta-app, X-authentik-meta-version +``` + +--- + +### Task 1: Enable Authentik in Production Inventory + +This task sets the Ansible variables to enable Authentik deployment on the production host. + +**Files:** +- Modify: `/Users/snider/Code/DevOps/inventory/host_vars/de2.yml` (or equivalent group_vars) + +**Step 1: Find the correct inventory file for de2** + +Run: +```bash +find /Users/snider/Code/DevOps/inventory -name "*.yml" -o -name "*.yaml" | head -20 +ls /Users/snider/Code/DevOps/inventory/ +``` + +Identify where de2's host vars live. + +**Step 2: Add Authentik variables** + +Add these variables for the de2 host: + +```yaml +# Authentik +traefik_authentik_enabled: true +traefik_authentik_url: "https://auth.host.uk.com" + +authentik_host: "auth.host.uk.com" +authentik_bootstrap_password: "" +authentik_bootstrap_token: "" +authentik_bootstrap_email: "admin@host.uk.com" +``` + +Note: `authentik_secret_key` auto-generates and persists on first run. `authentik_pg_password` auto-generates via lookup. The Authentik role handles both. + +**Step 3: Verify prerequisites exist on de2** + +Authentik requires PostgreSQL + Dragonfly (Redis). Check they're in the prod playbook: +```bash +grep -n "postgres\|dragonfly" /Users/snider/Code/DevOps/playbooks/prod_rebuild.yml | head -10 +``` + +**Step 4: Commit** + +```bash +cd /Users/snider/Code/DevOps +git add inventory/ +git commit -m "feat(authentik): enable Authentik and Traefik forward auth on de2 + +Co-Authored-By: Virgil " +``` + +--- + +### Task 2: Add Authentik to Production Playbook + +The Authentik Ansible role exists but is not included in the prod rebuild playbook. This task adds it. + +**Files:** +- Modify: `/Users/snider/Code/DevOps/playbooks/prod_rebuild.yml` + +**Step 1: Read the playbook to find the right insertion point** + +Authentik must deploy AFTER PostgreSQL + Dragonfly (it needs them) and AFTER Traefik (it needs the proxy network), but BEFORE services that use `authentik@file`. + +```bash +grep -n "Phase\|traefik\|postgres\|dragonfly\|portainer\|glance" /Users/snider/Code/DevOps/playbooks/prod_rebuild.yml | head -20 +``` + +**Step 2: Add Authentik role include** + +Insert after the Traefik phase, before services: + +```yaml + # ── Phase N: Identity (Authentik) ── + - name: Deploy Authentik + ansible.builtin.include_role: + name: authentik + tags: [authentik] +``` + +**Step 3: Verify the playbook parses** + +```bash +cd /Users/snider/Code/DevOps +ansible-playbook playbooks/prod_rebuild.yml --syntax-check +``` + +Expected: No errors. + +**Step 4: Commit** + +```bash +cd /Users/snider/Code/DevOps +git add playbooks/prod_rebuild.yml +git commit -m "feat(authentik): add Authentik phase to prod rebuild playbook + +Co-Authored-By: Virgil " +``` + +--- + +### Task 3: Deploy Authentik (Run Playbook) + +This is a manual step — run the Ansible playbook to deploy Authentik on de2. + +**Step 1: Dry-run the Authentik tag only** + +```bash +cd /Users/snider/Code/DevOps +ansible-playbook playbooks/prod_rebuild.yml --tags authentik --check --diff +``` + +Review the output. Expect: directories created, docker-compose deployed, containers started. + +Note: `--check` will skip shell/command tasks (like the PostgreSQL user creation). This is expected — the actual run will handle those. + +**Step 2: Deploy Authentik** + +```bash +ansible-playbook playbooks/prod_rebuild.yml --tags authentik +``` + +**Step 3: Re-deploy Traefik to pick up the forward auth middleware** + +The Traefik role conditionally deploys `dynamic-authentik.yml` based on `traefik_authentik_enabled`. Re-running the role with the new variable will create the middleware file: + +```bash +ansible-playbook playbooks/prod_rebuild.yml --tags traefik +``` + +**Step 4: Verify Authentik is accessible** + +```bash +curl -sI https://auth.host.uk.com | head -5 +``` + +Expected: HTTP 200 or 302 redirect to login page. + +**Step 5: Complete initial setup** + +Open `https://auth.host.uk.com/if/flow/initial-setup/` in a browser. Set the admin password (the bootstrap password from Task 1 is used for the API token, but the UI setup flow creates the actual admin account). + +--- + +### Task 4: Create Authentik OIDC Application for go-api + +This configures Authentik to issue tokens for go-api. Done via the Authentik admin UI or API. + +**Step 1: Create an OAuth2/OIDC Provider** + +In Authentik Admin → Providers → Create: + +| Field | Value | +|-------|-------| +| Name | `Core API` | +| Protocol | OAuth2/OIDC | +| Client type | Confidential | +| Client ID | `core-api` | +| Redirect URIs | `https://api.lthn.ai/auth/callback` (for auth code flow) | +| Signing key | Select auto-generated signing key | +| Scopes | `openid`, `email`, `profile` | +| Subject mode | Based on user's hashed ID | + +Record the **Client Secret** — needed for go-api config. + +**Step 2: Create an Application** + +In Authentik Admin → Applications → Create: + +| Field | Value | +|-------|-------| +| Name | `Core API` | +| Slug | `core-api` | +| Provider | Core API (from step 1) | +| Launch URL | `https://api.lthn.ai/` | + +**Step 3: Create a Forward Auth (Proxy) Provider for Traefik** + +In Authentik Admin → Providers → Create: + +| Field | Value | +|-------|-------| +| Name | `Traefik Forward Auth — Core API` | +| Protocol | Proxy | +| Mode | Forward auth (single application) | +| External host | `https://api.lthn.ai` | + +**Step 4: Create an Outpost (if not exists)** + +In Authentik Admin → Outposts: +- If no outpost exists: Create → Type: Proxy, Integration: Local Docker +- Add both providers to the outpost + +**Step 5: Test forward auth is working** + +```bash +# This should redirect to Authentik login +curl -sI https://api.lthn.ai/ +``` + +Once authenticated, Traefik passes the X-authentik-* headers through. + +--- + +### Task 5: go-api Authentik User Type (TDD) + +**Files:** +- Create: `/Users/snider/Code/go-api/authentik.go` +- Create: `/Users/snider/Code/go-api/authentik_test.go` + +**Step 1: Write the failing tests** + +Create `authentik_test.go`: +```go +package api_test + +import ( + "testing" + + api "forge.lthn.ai/core/go-api" +) + +func TestAuthentikUser_Good(t *testing.T) { + user := &api.AuthentikUser{ + Username: "alice", + Email: "alice@example.com", + Name: "Alice Smith", + UID: "abc-123", + Groups: []string{"admins", "developers"}, + } + + if user.Username != "alice" { + t.Fatalf("expected username alice, got %s", user.Username) + } + if len(user.Groups) != 2 { + t.Fatalf("expected 2 groups, got %d", len(user.Groups)) + } +} + +func TestAuthentikUserHasGroup_Good(t *testing.T) { + user := &api.AuthentikUser{ + Groups: []string{"admins", "developers"}, + } + + if !user.HasGroup("admins") { + t.Fatal("expected user to have admins group") + } + if user.HasGroup("viewers") { + t.Fatal("expected user to not have viewers group") + } +} + +func TestAuthentikUserHasGroup_Bad_Empty(t *testing.T) { + user := &api.AuthentikUser{} + + if user.HasGroup("admins") { + t.Fatal("expected empty user to have no groups") + } +} + +func TestAuthentikConfig_Good(t *testing.T) { + cfg := api.AuthentikConfig{ + Issuer: "https://auth.host.uk.com/application/o/core-api/", + ClientID: "core-api", + TrustedProxy: true, + } + + if cfg.Issuer == "" { + t.Fatal("expected non-empty issuer") + } + if !cfg.TrustedProxy { + t.Fatal("expected TrustedProxy to be true") + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -run TestAuthentik +``` + +Expected: Compilation errors — `api.AuthentikUser`, `api.AuthentikConfig` not defined. + +**Step 3: Implement authentik.go** + +Create `authentik.go`: +```go +package api + +// AuthentikConfig configures Authentik OIDC integration. +type AuthentikConfig struct { + // Issuer is the OIDC issuer URL (e.g. "https://auth.host.uk.com/application/o/core-api/"). + // Used for JWT validation via OIDC discovery. + Issuer string + + // ClientID is the OAuth2 client identifier registered in Authentik. + ClientID string + + // TrustedProxy enables reading X-authentik-* headers set by Traefik forward auth. + // Only enable this when go-api sits behind a trusted reverse proxy. + TrustedProxy bool + + // PublicPaths lists path prefixes that skip authentication entirely. + // /health and /swagger are always public regardless of this setting. + PublicPaths []string +} + +// AuthentikUser represents an authenticated user extracted from Authentik headers or JWT claims. +type AuthentikUser struct { + Username string `json:"username"` + Email string `json:"email"` + Name string `json:"name"` + UID string `json:"uid"` + Groups []string `json:"groups"` + Entitlements []string `json:"entitlements,omitempty"` + JWT string `json:"-"` +} + +// HasGroup returns true if the user belongs to the named group. +func (u *AuthentikUser) HasGroup(group string) bool { + for _, g := range u.Groups { + if g == group { + return true + } + } + return false +} +``` + +**Step 4: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -run TestAuthentik +``` + +Expected: All 4 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/go-api +git add authentik.go authentik_test.go +git commit -m "feat: add AuthentikUser and AuthentikConfig types + +Co-Authored-By: Virgil " +``` + +--- + +### Task 6: go-api Header Extraction Middleware (TDD) + +This implements the forward auth path — extracting user identity from X-authentik-* headers set by Traefik. + +**Files:** +- Modify: `/Users/snider/Code/go-api/authentik.go` +- Modify: `/Users/snider/Code/go-api/authentik_test.go` + +**Step 1: Write the failing tests** + +Append to `authentik_test.go`: +```go +import ( + "encoding/json" + "net/http" + "net/http/httptest" + + "github.com/gin-gonic/gin" +) + +// authentikTestGroup returns JSON with the user from context. +type authentikTestGroup struct{} + +func (g *authentikTestGroup) Name() string { return "authtest" } +func (g *authentikTestGroup) BasePath() string { return "/v1/authtest" } +func (g *authentikTestGroup) RegisterRoutes(rg *gin.RouterGroup) { + rg.GET("/whoami", func(c *gin.Context) { + user := api.GetUser(c) + if user == nil { + c.JSON(200, api.OK[any](nil)) + return + } + c.JSON(200, api.OK(user)) + }) +} + +func TestForwardAuthHeaders_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{ + TrustedProxy: true, + })) + engine.Register(&authentikTestGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/authtest/whoami", nil) + req.Header.Set("X-authentik-username", "alice") + req.Header.Set("X-authentik-email", "alice@example.com") + req.Header.Set("X-authentik-name", "Alice Smith") + req.Header.Set("X-authentik-uid", "abc-123") + req.Header.Set("X-authentik-groups", "admins|developers") + req.Header.Set("X-authentik-entitlements", "core:read|core:write") + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } + + var resp api.Response[*api.AuthentikUser] + json.Unmarshal(w.Body.Bytes(), &resp) + if resp.Data == nil { + t.Fatal("expected non-nil user data") + } + if resp.Data.Username != "alice" { + t.Fatalf("expected username alice, got %s", resp.Data.Username) + } + if resp.Data.Email != "alice@example.com" { + t.Fatalf("expected email alice@example.com, got %s", resp.Data.Email) + } + if len(resp.Data.Groups) != 2 { + t.Fatalf("expected 2 groups, got %d", len(resp.Data.Groups)) + } + if resp.Data.Groups[0] != "admins" { + t.Fatalf("expected first group admins, got %s", resp.Data.Groups[0]) + } +} + +func TestForwardAuthHeaders_Good_NoHeaders(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{ + TrustedProxy: true, + })) + engine.Register(&authentikTestGroup{}) + handler := engine.Handler() + + // Request without Authentik headers — should pass through (middleware is permissive) + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/authtest/whoami", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } + + var resp api.Response[*api.AuthentikUser] + json.Unmarshal(w.Body.Bytes(), &resp) + if resp.Data != nil { + t.Fatal("expected nil user when no headers present") + } +} + +func TestForwardAuthHeaders_Bad_NotTrusted(t *testing.T) { + gin.SetMode(gin.TestMode) + // TrustedProxy: false — should NOT read X-authentik-* headers + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{ + TrustedProxy: false, + })) + engine.Register(&authentikTestGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/authtest/whoami", nil) + req.Header.Set("X-authentik-username", "alice") + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } + + var resp api.Response[*api.AuthentikUser] + json.Unmarshal(w.Body.Bytes(), &resp) + if resp.Data != nil { + t.Fatal("expected nil user when TrustedProxy is false") + } +} + +func TestHealthBypassesAuthentik_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{ + TrustedProxy: true, + })) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/health", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200 for /health, got %d", w.Code) + } +} + +func TestGetUser_Good_NilContext(t *testing.T) { + gin.SetMode(gin.TestMode) + // Test GetUser with no user in context (no Authentik middleware) + engine, _ := api.New() + engine.Register(&authentikTestGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/authtest/whoami", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -run TestForwardAuth\|TestHealthBypassesAuthentik\|TestGetUser +``` + +Expected: Compilation errors — `api.WithAuthentik`, `api.GetUser` not defined. + +**Step 3: Add GetUser helper and middleware to authentik.go** + +Append to `authentik.go`: +```go +import ( + "strings" + + "github.com/gin-gonic/gin" +) + +const authentikUserKey = "authentik_user" + +// GetUser returns the authenticated Authentik user from the Gin context, or nil +// if no user is authenticated. +func GetUser(c *gin.Context) *AuthentikUser { + val, exists := c.Get(authentikUserKey) + if !exists { + return nil + } + user, ok := val.(*AuthentikUser) + if !ok { + return nil + } + return user +} + +// authentikMiddleware extracts user identity from X-authentik-* headers +// (when TrustedProxy is true) and stores it in the Gin context. +// This middleware is PERMISSIVE — it does not reject unauthenticated requests. +// Handlers must check GetUser() and decide whether to require auth. +func authentikMiddleware(cfg AuthentikConfig) gin.HandlerFunc { + publicPaths := append([]string{"/health", "/swagger"}, cfg.PublicPaths...) + + return func(c *gin.Context) { + // Skip public paths entirely. + for _, path := range publicPaths { + if strings.HasPrefix(c.Request.URL.Path, path) { + c.Next() + return + } + } + + // Forward auth mode: read trusted headers from Traefik. + if cfg.TrustedProxy { + username := c.GetHeader("X-authentik-username") + if username != "" { + user := &AuthentikUser{ + Username: username, + Email: c.GetHeader("X-authentik-email"), + Name: c.GetHeader("X-authentik-name"), + UID: c.GetHeader("X-authentik-uid"), + JWT: c.GetHeader("X-authentik-jwt"), + } + + if groups := c.GetHeader("X-authentik-groups"); groups != "" { + user.Groups = strings.Split(groups, "|") + } + if ent := c.GetHeader("X-authentik-entitlements"); ent != "" { + user.Entitlements = strings.Split(ent, "|") + } + + c.Set(authentikUserKey, user) + } + } + + c.Next() + } +} +``` + +**Step 4: Add WithAuthentik option to options.go** + +Append to `options.go`: +```go +// WithAuthentik adds Authentik identity middleware. +// When TrustedProxy is true, reads X-authentik-* headers from Traefik forward auth. +// When Issuer is set, also validates JWT Bearer tokens via OIDC discovery. +func WithAuthentik(cfg AuthentikConfig) Option { + return func(e *Engine) { + e.middlewares = append(e.middlewares, authentikMiddleware(cfg)) + } +} +``` + +**Step 5: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS (existing 36 + new 5). + +**Step 6: Commit** + +```bash +cd /Users/snider/Code/go-api +git add authentik.go authentik_test.go options.go +git commit -m "feat: add Authentik header extraction middleware and GetUser helper + +Forward auth mode reads X-authentik-* headers from Traefik. +Middleware is permissive — handlers decide whether auth is required. + +Co-Authored-By: Virgil " +``` + +--- + +### Task 7: go-api JWT Validation Middleware (TDD) + +This implements the direct OIDC path — validating JWT Bearer tokens for API clients. + +**Files:** +- Modify: `/Users/snider/Code/go-api/authentik.go` +- Modify: `/Users/snider/Code/go-api/authentik_test.go` +- Modify: `/Users/snider/Code/go-api/go.mod` (new dependency) + +**Step 1: Write the failing tests** + +Append to `authentik_test.go`: +```go +func TestJWTValidation_Bad_InvalidToken(t *testing.T) { + gin.SetMode(gin.TestMode) + // Use a fake issuer — OIDC discovery will fail, but we test the flow + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{ + Issuer: "https://auth.example.com/application/o/test/", + ClientID: "test-client", + })) + engine.Register(&authentikTestGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/authtest/whoami", nil) + req.Header.Set("Authorization", "Bearer invalid-jwt-token") + handler.ServeHTTP(w, req) + + // Without a reachable OIDC endpoint, JWT validation can't succeed. + // The middleware should pass through (permissive) with no user. + if w.Code != 200 { + t.Fatalf("expected 200 (permissive), got %d", w.Code) + } + + var resp api.Response[*api.AuthentikUser] + json.Unmarshal(w.Body.Bytes(), &resp) + if resp.Data != nil { + t.Fatal("expected nil user for invalid JWT") + } +} + +func TestBearerAndAuthentikCoexist_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + // Both WithBearerAuth and WithAuthentik should work together. + // Bearer auth gates access, Authentik extracts user identity. + engine, _ := api.New( + api.WithBearerAuth("secret-token"), + api.WithAuthentik(api.AuthentikConfig{TrustedProxy: true}), + ) + engine.Register(&authentikTestGroup{}) + handler := engine.Handler() + + // With bearer token + Authentik headers → 200 with user + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/authtest/whoami", nil) + req.Header.Set("Authorization", "Bearer secret-token") + req.Header.Set("X-authentik-username", "bob") + req.Header.Set("X-authentik-email", "bob@example.com") + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } + + var resp api.Response[*api.AuthentikUser] + json.Unmarshal(w.Body.Bytes(), &resp) + if resp.Data == nil { + t.Fatal("expected user data") + } + if resp.Data.Username != "bob" { + t.Fatalf("expected username bob, got %s", resp.Data.Username) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -run TestJWTValidation\|TestBearerAndAuthentikCoexist +``` + +**Step 3: Add OIDC validation to authentik middleware** + +Update `authentikMiddleware` in `authentik.go` to handle JWT Bearer tokens when `Issuer` is configured. Add the go-oidc dependency: + +```bash +cd /Users/snider/Code/go-api +go get github.com/coreos/go-oidc/v3/oidc +go get golang.org/x/oauth2 +``` + +Add JWT validation logic to the middleware — after the header extraction block, before `c.Next()`: + +```go +// Direct OIDC mode: validate JWT from Authorization header. +if cfg.Issuer != "" && cfg.ClientID != "" { + // Only attempt JWT validation if no user was extracted from headers + // (headers take priority — they're pre-validated by Authentik). + if GetUser(c) == nil { + authHeader := c.GetHeader("Authorization") + if strings.HasPrefix(authHeader, "Bearer ") { + token := strings.TrimPrefix(authHeader, "Bearer ") + user, err := validateJWT(c.Request.Context(), cfg, token) + if err == nil && user != nil { + c.Set(authentikUserKey, user) + } + // Permissive: if validation fails, continue without user. + } + } +} +``` + +Add the validation function: +```go +import ( + "context" + "sync" + + oidc "github.com/coreos/go-oidc/v3/oidc" +) + +var ( + oidcProviderMu sync.Mutex + oidcProviders = make(map[string]*oidc.Provider) +) + +// getOIDCProvider returns a cached OIDC provider for the given issuer. +func getOIDCProvider(ctx context.Context, issuer string) (*oidc.Provider, error) { + oidcProviderMu.Lock() + defer oidcProviderMu.Unlock() + + if p, ok := oidcProviders[issuer]; ok { + return p, nil + } + + p, err := oidc.NewProvider(ctx, issuer) + if err != nil { + return nil, err + } + oidcProviders[issuer] = p + return p, nil +} + +// validateJWT verifies a JWT token against the OIDC provider and extracts the user. +func validateJWT(ctx context.Context, cfg AuthentikConfig, rawToken string) (*AuthentikUser, error) { + provider, err := getOIDCProvider(ctx, cfg.Issuer) + if err != nil { + return nil, err + } + + verifier := provider.Verifier(&oidc.Config{ClientID: cfg.ClientID}) + idToken, err := verifier.Verify(ctx, rawToken) + if err != nil { + return nil, err + } + + var claims struct { + PreferredUsername string `json:"preferred_username"` + Email string `json:"email"` + Name string `json:"name"` + Sub string `json:"sub"` + Groups []string `json:"groups"` + } + if err := idToken.Claims(&claims); err != nil { + return nil, err + } + + return &AuthentikUser{ + Username: claims.PreferredUsername, + Email: claims.Email, + Name: claims.Name, + UID: claims.Sub, + Groups: claims.Groups, + JWT: rawToken, + }, nil +} +``` + +**Step 4: Run go mod tidy** + +```bash +cd /Users/snider/Code/go-api +go mod tidy +``` + +**Step 5: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS. + +**Step 6: Commit** + +```bash +cd /Users/snider/Code/go-api +git add authentik.go authentik_test.go go.mod go.sum +git commit -m "feat: add OIDC JWT validation for direct API client auth + +Uses coreos/go-oidc for OIDC discovery and JWT verification. +Cached provider instances. Permissive — fails open if OIDC unreachable. +Forward auth headers take priority over JWT when both present. + +Co-Authored-By: Virgil " +``` + +--- + +### Task 8: go-api RequireAuth Middleware Helper (TDD) + +The Authentik middleware is permissive. This task adds a helper for routes that REQUIRE authentication. + +**Files:** +- Modify: `/Users/snider/Code/go-api/authentik.go` +- Modify: `/Users/snider/Code/go-api/authentik_test.go` + +**Step 1: Write the failing tests** + +Append to `authentik_test.go`: +```go +// protectedGroup uses RequireAuth on its routes. +type protectedGroup struct{} + +func (g *protectedGroup) Name() string { return "protected" } +func (g *protectedGroup) BasePath() string { return "/v1/protected" } +func (g *protectedGroup) RegisterRoutes(rg *gin.RouterGroup) { + rg.GET("/data", api.RequireAuth(), func(c *gin.Context) { + user := api.GetUser(c) + c.JSON(200, api.OK(user.Username)) + }) +} + +func TestRequireAuth_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{TrustedProxy: true})) + engine.Register(&protectedGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/protected/data", nil) + req.Header.Set("X-authentik-username", "alice") + req.Header.Set("X-authentik-email", "alice@example.com") + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200 with auth, got %d", w.Code) + } +} + +func TestRequireAuth_Bad_NoUser(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{TrustedProxy: true})) + engine.Register(&protectedGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/protected/data", nil) + handler.ServeHTTP(w, req) + + if w.Code != 401 { + t.Fatalf("expected 401 without auth, got %d", w.Code) + } +} + +func TestRequireAuth_Bad_NoAuthentikMiddleware(t *testing.T) { + gin.SetMode(gin.TestMode) + // No WithAuthentik — RequireAuth should still return 401 + engine, _ := api.New() + engine.Register(&protectedGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/protected/data", nil) + handler.ServeHTTP(w, req) + + if w.Code != 401 { + t.Fatalf("expected 401, got %d", w.Code) + } +} + +// groupRequireGroup uses RequireGroup. +type groupRequireGroup struct{} + +func (g *groupRequireGroup) Name() string { return "adminonly" } +func (g *groupRequireGroup) BasePath() string { return "/v1/admin" } +func (g *groupRequireGroup) RegisterRoutes(rg *gin.RouterGroup) { + rg.GET("/panel", api.RequireGroup("admins"), func(c *gin.Context) { + c.JSON(200, api.OK("admin panel")) + }) +} + +func TestRequireGroup_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{TrustedProxy: true})) + engine.Register(&groupRequireGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/admin/panel", nil) + req.Header.Set("X-authentik-username", "alice") + req.Header.Set("X-authentik-groups", "admins|developers") + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200 for admin user, got %d", w.Code) + } +} + +func TestRequireGroup_Bad_WrongGroup(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithAuthentik(api.AuthentikConfig{TrustedProxy: true})) + engine.Register(&groupRequireGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/admin/panel", nil) + req.Header.Set("X-authentik-username", "bob") + req.Header.Set("X-authentik-groups", "developers") + handler.ServeHTTP(w, req) + + if w.Code != 403 { + t.Fatalf("expected 403 for non-admin user, got %d", w.Code) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -run TestRequireAuth\|TestRequireGroup +``` + +Expected: Compilation errors — `api.RequireAuth`, `api.RequireGroup` not defined. + +**Step 3: Implement RequireAuth and RequireGroup** + +Append to `authentik.go`: +```go +import "net/http" + +// RequireAuth is a Gin middleware that returns 401 if no authenticated user +// is present in the context. Use after WithAuthentik() middleware. +func RequireAuth() gin.HandlerFunc { + return func(c *gin.Context) { + if GetUser(c) == nil { + c.AbortWithStatusJSON(http.StatusUnauthorized, + Fail("unauthorised", "Authentication required")) + return + } + c.Next() + } +} + +// RequireGroup is a Gin middleware that returns 403 if the authenticated user +// does not belong to the specified group. Implies RequireAuth. +func RequireGroup(group string) gin.HandlerFunc { + return func(c *gin.Context) { + user := GetUser(c) + if user == nil { + c.AbortWithStatusJSON(http.StatusUnauthorized, + Fail("unauthorised", "Authentication required")) + return + } + if !user.HasGroup(group) { + c.AbortWithStatusJSON(http.StatusForbidden, + Fail("forbidden", "Insufficient permissions")) + return + } + c.Next() + } +} +``` + +**Step 4: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/go-api +git add authentik.go authentik_test.go +git commit -m "feat: add RequireAuth and RequireGroup middleware helpers + +RequireAuth returns 401 when no user in context. +RequireGroup returns 403 when user lacks the specified group. +Both use British English 'unauthorised' in error responses. + +Co-Authored-By: Virgil " +``` + +--- + +### Task 9: Update go-api Documentation + +**Files:** +- Modify: `/Users/snider/Code/go-api/CLAUDE.md` +- Modify: `/Users/snider/Code/go-api/README.md` + +**Step 1: Update CLAUDE.md** + +Add to the Project Overview section: +```markdown +## Authentik Integration + +go-api supports Authentik as the identity provider: + +- **Forward auth mode**: Reads `X-authentik-*` headers from Traefik (requires `TrustedProxy: true`) +- **OIDC mode**: Validates JWT Bearer tokens via OIDC discovery +- **Permissive middleware**: `WithAuthentik()` extracts user but doesn't block. Use `RequireAuth()` / `RequireGroup()` on routes that need auth. +- **Coexists with `WithBearerAuth()`** for service-to-service tokens + +```go +engine, _ := api.New( + api.WithAuthentik(api.AuthentikConfig{ + Issuer: "https://auth.host.uk.com/application/o/core-api/", + ClientID: "core-api", + TrustedProxy: true, + }), +) +``` +``` + +**Step 2: Update README.md** + +Add Authentik section with quick-start example showing `WithAuthentik()`, `GetUser()`, `RequireAuth()`, and `RequireGroup()`. + +**Step 3: Commit** + +```bash +cd /Users/snider/Code/go-api +git add CLAUDE.md README.md +git commit -m "docs: add Authentik integration guide to CLAUDE.md and README + +Co-Authored-By: Virgil " +``` + +--- + +### Task 10: Push go-api and DevOps Changes + +**Step 1: Push go-api** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 # Final verification +git push forge main +``` + +**Step 2: Push DevOps** + +```bash +cd /Users/snider/Code/DevOps +git push forge main +``` + +**Step 3: Update go-ecosystem memory** + +Update the go-api entry in the ecosystem inventory to note Authentik middleware. + +--- + +## Dependency Summary + +``` +Task 1 (enable vars) → Task 2 (playbook) → Task 3 (deploy) → Task 4 (OIDC app) + ↓ +Task 5 (user type) → Task 6 (header middleware) → Task 7 (JWT) → Task 8 (RequireAuth) + ↓ + Task 9 (docs) → Task 10 (push) +``` + +Tasks 1-4 (DevOps) and Tasks 5-8 (Go) are independent tracks that can run in parallel. Task 9-10 depend on both tracks. + +## Estimated Sizes + +| Task | LOC | Tests | +|------|-----|-------| +| Task 5: User type | ~50 | 4 | +| Task 6: Header middleware | ~60 | 5 | +| Task 7: JWT validation | ~80 | 2 | +| Task 8: RequireAuth/Group | ~30 | 5 | +| **go-api total** | **~220** | **16** | diff --git a/docs/plans/2026-02-21-core-help-design.md b/docs/plans/2026-02-21-core-help-design.md new file mode 100644 index 0000000..2943178 --- /dev/null +++ b/docs/plans/2026-02-21-core-help-design.md @@ -0,0 +1,155 @@ +# core.help Documentation Website — Design + +**Date:** 2026-02-21 +**Author:** Virgil +**Status:** Design approved +**Domain:** https://core.help + +## Problem + +Documentation is scattered across 39 repos (18 Go packages, 20 PHP packages, 1 CLI). There is no unified docs site. Developers need a single entry point to find CLI commands, Go package APIs, MCP tool references, and PHP module guides. + +## Solution + +A Hugo + Docsy static site at core.help, built from existing markdown docs aggregated by `core docs sync`. No new content — just collect and present what already exists across the ecosystem. + +## Architecture + +### Stack + +- **Hugo** — Go-native static site generator, sub-second builds +- **Docsy theme** — Purpose-built for technical docs (used by Kubernetes, gRPC, Knative) +- **BunnyCDN** — Static hosting with pull zone +- **`core docs sync --target hugo`** — Collects markdown from all repos into Hugo content tree + +### Why Hugo + Docsy (not VitePress or mdBook) + +- Go-native, no Node.js dependency +- Handles multi-section navigation (CLI, Go packages, PHP modules, MCP tools) +- Sub-second builds for ~250 markdown files +- Docsy has built-in search, versioned nav, API reference sections + +## Content Structure + +``` +docs-site/ +├── hugo.toml +├── content/ +│ ├── _index.md # Landing page +│ ├── getting-started/ # CLI top-level guides +│ │ ├── _index.md +│ │ ├── installation.md +│ │ ├── configuration.md +│ │ ├── user-guide.md +│ │ ├── troubleshooting.md +│ │ └── faq.md +│ ├── cli/ # CLI command reference (43 commands) +│ │ ├── _index.md +│ │ ├── dev/ # core dev commit, push, pull, etc. +│ │ ├── ai/ # core ai commands +│ │ ├── go/ # core go test, lint, etc. +│ │ └── ... +│ ├── go/ # Go ecosystem packages (18) +│ │ ├── _index.md # Ecosystem overview +│ │ ├── go-api/ # README + architecture/development/history +│ │ ├── go-ai/ +│ │ ├── go-mlx/ +│ │ ├── go-i18n/ +│ │ └── ... +│ ├── mcp/ # MCP tool reference (49 tools) +│ │ ├── _index.md +│ │ ├── file-operations.md +│ │ ├── process-management.md +│ │ ├── rag.md +│ │ └── ... +│ ├── php/ # PHP packages (from core-php/docs/packages/) +│ │ ├── _index.md +│ │ ├── admin/ +│ │ ├── tenant/ +│ │ ├── commerce/ +│ │ └── ... +│ └── kb/ # Knowledge base (wiki pages from go-mlx, go-i18n) +│ ├── _index.md +│ ├── mlx/ +│ └── i18n/ +├── static/ # Logos, favicons +├── layouts/ # Custom template overrides (minimal) +└── go.mod # Hugo modules (Docsy as module dep) +``` + +## Sync Pipeline + +`core docs sync --target hugo --output site/content/` performs: + +### Source Mapping + +``` +cli/docs/index.md → content/getting-started/_index.md +cli/docs/getting-started.md → content/getting-started/installation.md +cli/docs/user-guide.md → content/getting-started/user-guide.md +cli/docs/configuration.md → content/getting-started/configuration.md +cli/docs/troubleshooting.md → content/getting-started/troubleshooting.md +cli/docs/faq.md → content/getting-started/faq.md + +core/docs/cmd/**/*.md → content/cli/**/*.md + +go-*/README.md → content/go/{name}/_index.md +go-*/docs/*.md → content/go/{name}/*.md +go-*/KB/*.md → content/kb/{name-suffix}/*.md + +core-*/docs/**/*.md → content/php/{name-suffix}/**/*.md +``` + +### Front Matter Injection + +If a markdown file doesn't start with `---`, prepend: + +```yaml +--- +title: "{derived from filename}" +linkTitle: "{short name}" +weight: {auto-incremented} +--- +``` + +No other content transformations. Markdown stays as-is. + +### Build & Deploy + +```bash +core docs sync --target hugo --output docs-site/content/ +cd docs-site && hugo build +hugo deploy --target bunnycdn +``` + +Hugo deploy config in `hugo.toml`: + +```toml +[deployment] +[[deployment.targets]] +name = "bunnycdn" +URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto" +``` + +Credentials via env vars. + +## Registry + +All 39 repos registered in `.core/repos.yaml` with `docs: true`. Go repos use explicit `path:` fields since they live outside the PHP `base_path`. `FindRegistry()` checks `.core/repos.yaml` alongside `repos.yaml`. + +## Prerequisites Completed + +- [x] `.core/repos.yaml` created with all 39 repos +- [x] `FindRegistry()` updated to find `.core/repos.yaml` +- [x] `Repo.Path` supports explicit YAML override +- [x] go-api docs gap filled (architecture.md, development.md, history.md) +- [x] All 18 Go repos have standard docs trio + +## What Remains (Implementation Plan) + +1. Create docs-site repo with Hugo + Docsy scaffold +2. Extend `core docs sync` with `--target hugo` mode +3. Write section _index.md files (landing page, section intros) +4. Hugo config (navigation, search, theme colours) +5. BunnyCDN deployment config +6. CI pipeline on Forge (optional — can deploy manually initially) diff --git a/docs/plans/2026-02-21-core-help-plan.md b/docs/plans/2026-02-21-core-help-plan.md new file mode 100644 index 0000000..e3bf5e1 --- /dev/null +++ b/docs/plans/2026-02-21-core-help-plan.md @@ -0,0 +1,642 @@ +# core.help Hugo Documentation Site — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Build a Hugo + Docsy documentation site at core.help that aggregates markdown from 39 repos via `core docs sync --target hugo`. + +**Architecture:** Hugo static site with Docsy theme, populated by extending `core docs sync` with a `--target hugo` flag that maps repo docs into Hugo's `content/` tree with auto-injected front matter. Deploy to BunnyCDN. + +**Tech Stack:** Hugo (Go SSG), Docsy theme (Hugo module), BunnyCDN, `core docs sync` CLI + +--- + +## Context + +The docs sync command lives in `/Users/snider/Code/host-uk/cli/cmd/docs/`. The site will be scaffolded at `/Users/snider/Code/host-uk/docs-site/`. The registry at `/Users/snider/Code/host-uk/.core/repos.yaml` already contains all 39 repos (20 PHP + 18 Go + 1 CLI) with explicit paths for Go repos. + +Key files: +- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go` — sync command (modify) +- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go` — repo scanner (modify) +- `/Users/snider/Code/host-uk/docs-site/` — Hugo site (create) + +## Task 1: Scaffold Hugo + Docsy site + +**Files:** +- Create: `/Users/snider/Code/host-uk/docs-site/hugo.toml` +- Create: `/Users/snider/Code/host-uk/docs-site/go.mod` +- Create: `/Users/snider/Code/host-uk/docs-site/content/_index.md` +- Create: `/Users/snider/Code/host-uk/docs-site/content/getting-started/_index.md` +- Create: `/Users/snider/Code/host-uk/docs-site/content/cli/_index.md` +- Create: `/Users/snider/Code/host-uk/docs-site/content/go/_index.md` +- Create: `/Users/snider/Code/host-uk/docs-site/content/mcp/_index.md` +- Create: `/Users/snider/Code/host-uk/docs-site/content/php/_index.md` +- Create: `/Users/snider/Code/host-uk/docs-site/content/kb/_index.md` + +This is the one-time Hugo scaffolding. No tests — just files. + +**`hugo.toml`:** +```toml +baseURL = "https://core.help/" +title = "Core Documentation" +languageCode = "en" +defaultContentLanguage = "en" + +enableRobotsTXT = true +enableGitInfo = false + +[outputs] +home = ["HTML", "JSON"] +section = ["HTML"] + +[params] +description = "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools" +copyright = "Host UK — EUPL-1.2" + +[params.ui] +sidebar_menu_compact = true +breadcrumb_disable = false +sidebar_search_disable = false +navbar_logo = false + +[params.ui.readingtime] +enable = false + +[module] +proxy = "direct" + +[module.hugoVersion] +extended = true +min = "0.120.0" + +[[module.imports]] +path = "github.com/google/docsy" +disable = false + +[markup.goldmark.renderer] +unsafe = true + +[menu] +[[menu.main]] +name = "Getting Started" +weight = 10 +url = "/getting-started/" +[[menu.main]] +name = "CLI Reference" +weight = 20 +url = "/cli/" +[[menu.main]] +name = "Go Packages" +weight = 30 +url = "/go/" +[[menu.main]] +name = "MCP Tools" +weight = 40 +url = "/mcp/" +[[menu.main]] +name = "PHP Packages" +weight = 50 +url = "/php/" +[[menu.main]] +name = "Knowledge Base" +weight = 60 +url = "/kb/" +``` + +**`go.mod`:** +``` +module github.com/host-uk/docs-site + +go 1.22 + +require github.com/google/docsy v0.11.0 +``` + +Note: Run `hugo mod get` after creating these files to populate `go.sum` and download Docsy. + +**Section `_index.md` files** — each needs Hugo front matter: + +`content/_index.md`: +```markdown +--- +title: "Core Documentation" +description: "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools" +--- + +Welcome to the Core ecosystem documentation. + +## Sections + +- [Getting Started](/getting-started/) — Installation, configuration, and first steps +- [CLI Reference](/cli/) — Command reference for `core` CLI +- [Go Packages](/go/) — Go ecosystem package documentation +- [MCP Tools](/mcp/) — Model Context Protocol tool reference +- [PHP Packages](/php/) — PHP module documentation +- [Knowledge Base](/kb/) — Wiki articles and deep dives +``` + +`content/getting-started/_index.md`: +```markdown +--- +title: "Getting Started" +linkTitle: "Getting Started" +weight: 10 +description: "Installation, configuration, and first steps with the Core CLI" +--- +``` + +`content/cli/_index.md`: +```markdown +--- +title: "CLI Reference" +linkTitle: "CLI Reference" +weight: 20 +description: "Command reference for the core CLI tool" +--- +``` + +`content/go/_index.md`: +```markdown +--- +title: "Go Packages" +linkTitle: "Go Packages" +weight: 30 +description: "Documentation for the Go ecosystem packages" +--- +``` + +`content/mcp/_index.md`: +```markdown +--- +title: "MCP Tools" +linkTitle: "MCP Tools" +weight: 40 +description: "Model Context Protocol tool reference — file operations, RAG, ML inference, process management" +--- +``` + +`content/php/_index.md`: +```markdown +--- +title: "PHP Packages" +linkTitle: "PHP Packages" +weight: 50 +description: "Documentation for the PHP module ecosystem" +--- +``` + +`content/kb/_index.md`: +```markdown +--- +title: "Knowledge Base" +linkTitle: "Knowledge Base" +weight: 60 +description: "Wiki articles, deep dives, and reference material" +--- +``` + +**Verify:** After creating files, run from `/Users/snider/Code/host-uk/docs-site/`: +```bash +hugo mod get +hugo server +``` +The site should start and show the landing page with Docsy theme at `localhost:1313`. + +**Commit:** +```bash +cd /Users/snider/Code/host-uk/docs-site +git init +git add . +git commit -m "feat: scaffold Hugo + Docsy documentation site" +``` + +--- + +## Task 2: Extend scanRepoDocs to collect KB/ and README + +**Files:** +- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go` + +Currently `scanRepoDocs` only collects files from `docs/`. For the Hugo target we also need: +- `KB/**/*.md` files (wiki pages from go-mlx, go-i18n) +- `README.md` content (becomes the package _index.md) + +Add a `KBFiles []string` field to `RepoDocInfo` and scan `KB/` alongside `docs/`: + +```go +type RepoDocInfo struct { + Name string + Path string + HasDocs bool + Readme string + ClaudeMd string + Changelog string + DocsFiles []string // All files in docs/ directory (recursive) + KBFiles []string // All files in KB/ directory (recursive) +} +``` + +In `scanRepoDocs`, after the `docs/` walk, add a second walk for `KB/`: + +```go +// Recursively scan KB/ directory for .md files +kbDir := filepath.Join(repo.Path, "KB") +if _, err := io.Local.List(kbDir); err == nil { + _ = filepath.WalkDir(kbDir, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return nil + } + if d.IsDir() || !strings.HasSuffix(d.Name(), ".md") { + return nil + } + relPath, _ := filepath.Rel(kbDir, path) + info.KBFiles = append(info.KBFiles, relPath) + info.HasDocs = true + return nil + }) +} +``` + +**Tests:** The existing tests should still pass. No new test file needed — this is a data-collection change. + +**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...` + +**Commit:** +```bash +git add cmd/docs/cmd_scan.go +git commit -m "feat(docs): scan KB/ directory alongside docs/" +``` + +--- + +## Task 3: Add `--target hugo` flag and Hugo sync logic + +**Files:** +- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go` + +This is the main task. Add a `--target` flag (default `"php"`) and a new `runHugoSync` function that maps repos to Hugo's content tree. + +**Add flag variable and registration:** + +```go +var ( + docsSyncRegistryPath string + docsSyncDryRun bool + docsSyncOutputDir string + docsSyncTarget string +) + +func init() { + docsSyncCmd.Flags().StringVar(&docsSyncRegistryPath, "registry", "", i18n.T("common.flag.registry")) + docsSyncCmd.Flags().BoolVar(&docsSyncDryRun, "dry-run", false, i18n.T("cmd.docs.sync.flag.dry_run")) + docsSyncCmd.Flags().StringVar(&docsSyncOutputDir, "output", "", i18n.T("cmd.docs.sync.flag.output")) + docsSyncCmd.Flags().StringVar(&docsSyncTarget, "target", "php", "Target format: php (default) or hugo") +} +``` + +**Update RunE to pass target:** +```go +RunE: func(cmd *cli.Command, args []string) error { + return runDocsSync(docsSyncRegistryPath, docsSyncOutputDir, docsSyncDryRun, docsSyncTarget) +}, +``` + +**Update `runDocsSync` signature and add target dispatch:** +```go +func runDocsSync(registryPath string, outputDir string, dryRun bool, target string) error { + reg, basePath, err := loadRegistry(registryPath) + if err != nil { + return err + } + + switch target { + case "hugo": + return runHugoSync(reg, basePath, outputDir, dryRun) + default: + return runPHPSync(reg, basePath, outputDir, dryRun) + } +} +``` + +**Rename current sync body to `runPHPSync`** — extract lines 67-159 of current `runDocsSync` into `runPHPSync(reg, basePath, outputDir string, dryRun bool) error`. This is a pure extract, no logic changes. + +**Add `hugoOutputName` mapping function:** +```go +// hugoOutputName maps repo name to Hugo content section and folder. +// Returns (section, folder) where section is the top-level content dir. +func hugoOutputName(repoName string) (string, string) { + // CLI guides + if repoName == "cli" { + return "getting-started", "" + } + // Core CLI command docs + if repoName == "core" { + return "cli", "" + } + // Go packages + if strings.HasPrefix(repoName, "go-") { + return "go", repoName + } + // PHP packages + if strings.HasPrefix(repoName, "core-") { + return "php", strings.TrimPrefix(repoName, "core-") + } + return "go", repoName +} +``` + +**Add front matter injection helper:** +```go +// injectFrontMatter prepends Hugo front matter to markdown content if missing. +func injectFrontMatter(content []byte, title string, weight int) []byte { + // Already has front matter + if bytes.HasPrefix(bytes.TrimSpace(content), []byte("---")) { + return content + } + fm := fmt.Sprintf("---\ntitle: %q\nweight: %d\n---\n\n", title, weight) + return append([]byte(fm), content...) +} + +// titleFromFilename derives a human-readable title from a filename. +func titleFromFilename(filename string) string { + name := strings.TrimSuffix(filepath.Base(filename), ".md") + name = strings.ReplaceAll(name, "-", " ") + name = strings.ReplaceAll(name, "_", " ") + // Title case + words := strings.Fields(name) + for i, w := range words { + if len(w) > 0 { + words[i] = strings.ToUpper(w[:1]) + w[1:] + } + } + return strings.Join(words, " ") +} +``` + +**Add `runHugoSync` function:** +```go +func runHugoSync(reg *repos.Registry, basePath string, outputDir string, dryRun bool) error { + if outputDir == "" { + outputDir = filepath.Join(basePath, "docs-site", "content") + } + + // Scan all repos + var docsInfo []RepoDocInfo + for _, repo := range reg.List() { + if repo.Name == "core-template" || repo.Name == "core-claude" { + continue + } + info := scanRepoDocs(repo) + if info.HasDocs { + docsInfo = append(docsInfo, info) + } + } + + if len(docsInfo) == 0 { + cli.Text("No documentation found") + return nil + } + + cli.Print("\n Hugo sync: %d repos with docs → %s\n\n", len(docsInfo), outputDir) + + // Show plan + for _, info := range docsInfo { + section, folder := hugoOutputName(info.Name) + target := section + if folder != "" { + target = section + "/" + folder + } + fileCount := len(info.DocsFiles) + len(info.KBFiles) + if info.Readme != "" { + fileCount++ + } + cli.Print(" %s → %s/ (%d files)\n", repoNameStyle.Render(info.Name), target, fileCount) + } + + if dryRun { + cli.Print("\n Dry run — no files written\n") + return nil + } + + cli.Blank() + if !confirm("Sync to Hugo content directory?") { + cli.Text("Aborted") + return nil + } + + cli.Blank() + var synced int + for _, info := range docsInfo { + section, folder := hugoOutputName(info.Name) + + // Build destination path + destDir := filepath.Join(outputDir, section) + if folder != "" { + destDir = filepath.Join(destDir, folder) + } + + // Copy docs/ files + weight := 10 + docsDir := filepath.Join(info.Path, "docs") + for _, f := range info.DocsFiles { + src := filepath.Join(docsDir, f) + dst := filepath.Join(destDir, f) + if err := copyWithFrontMatter(src, dst, weight); err != nil { + cli.Print(" %s %s: %s\n", errorStyle.Render("✗"), f, err) + continue + } + weight += 10 + } + + // Copy README.md as _index.md (if not CLI/core which use their own index) + if info.Readme != "" && folder != "" { + dst := filepath.Join(destDir, "_index.md") + if err := copyWithFrontMatter(info.Readme, dst, 1); err != nil { + cli.Print(" %s README: %s\n", errorStyle.Render("✗"), err) + } + } + + // Copy KB/ files to kb/{suffix}/ + if len(info.KBFiles) > 0 { + // Extract suffix: go-mlx → mlx, go-i18n → i18n + suffix := strings.TrimPrefix(info.Name, "go-") + kbDestDir := filepath.Join(outputDir, "kb", suffix) + kbDir := filepath.Join(info.Path, "KB") + kbWeight := 10 + for _, f := range info.KBFiles { + src := filepath.Join(kbDir, f) + dst := filepath.Join(kbDestDir, f) + if err := copyWithFrontMatter(src, dst, kbWeight); err != nil { + cli.Print(" %s KB/%s: %s\n", errorStyle.Render("✗"), f, err) + continue + } + kbWeight += 10 + } + } + + cli.Print(" %s %s\n", successStyle.Render("✓"), info.Name) + synced++ + } + + cli.Print("\n Synced %d repos to Hugo content\n", synced) + return nil +} + +// copyWithFrontMatter copies a markdown file, injecting front matter if missing. +func copyWithFrontMatter(src, dst string, weight int) error { + if err := io.Local.EnsureDir(filepath.Dir(dst)); err != nil { + return err + } + content, err := io.Local.Read(src) + if err != nil { + return err + } + title := titleFromFilename(src) + result := injectFrontMatter([]byte(content), title, weight) + return io.Local.Write(dst, string(result)) +} +``` + +**Add imports** at top of file: +```go +import ( + "bytes" + "fmt" + "path/filepath" + "strings" + + "forge.lthn.ai/core/go/pkg/cli" + "forge.lthn.ai/core/go/pkg/i18n" + "forge.lthn.ai/core/go/pkg/io" + "forge.lthn.ai/core/go/pkg/repos" +) +``` + +**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...` + +**Commit:** +```bash +git add cmd/docs/cmd_sync.go +git commit -m "feat(docs): add --target hugo sync mode for core.help" +``` + +--- + +## Task 4: Test the full pipeline + +**No code changes.** Run the pipeline end-to-end. + +**Step 1:** Sync docs to Hugo: +```bash +cd /Users/snider/Code/host-uk +core docs sync --target hugo --dry-run +``` +Verify all 39 repos appear with correct section mappings. + +**Step 2:** Run actual sync: +```bash +core docs sync --target hugo +``` + +**Step 3:** Build and preview: +```bash +cd /Users/snider/Code/host-uk/docs-site +hugo server +``` +Open `localhost:1313` and verify: +- Landing page renders with section links +- Getting Started section has CLI guides +- CLI Reference section has command docs +- Go Packages section has 18 packages with architecture/development/history +- PHP Packages section has PHP module docs +- Knowledge Base has MLX and i18n wiki pages +- Navigation works, search works + +**Step 4:** Fix any issues found during preview. + +**Commit docs-site content:** +```bash +cd /Users/snider/Code/host-uk/docs-site +git add content/ +git commit -m "feat: sync initial content from 39 repos" +``` + +--- + +## Task 5: BunnyCDN deployment config + +**Files:** +- Modify: `/Users/snider/Code/host-uk/docs-site/hugo.toml` + +Add deployment target: + +```toml +[deployment] +[[deployment.targets]] +name = "production" +URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto" +``` + +Add a `Taskfile.yml` for convenience: + +**Create:** `/Users/snider/Code/host-uk/docs-site/Taskfile.yml` +```yaml +version: '3' + +tasks: + dev: + desc: Start Hugo dev server + cmds: + - hugo server --buildDrafts + + build: + desc: Build static site + cmds: + - hugo --minify + + sync: + desc: Sync docs from all repos + dir: .. + cmds: + - core docs sync --target hugo + + deploy: + desc: Build and deploy to BunnyCDN + cmds: + - task: sync + - task: build + - hugo deploy --target production + + clean: + desc: Remove generated content (keeps _index.md files) + cmds: + - find content -name "*.md" ! -name "_index.md" -delete +``` + +**Verify:** `task dev` starts the site. + +**Commit:** +```bash +git add hugo.toml Taskfile.yml +git commit -m "feat: add BunnyCDN deployment config and Taskfile" +``` + +--- + +## Dependency Sequencing + +``` +Task 1 (Hugo scaffold) — independent, do first +Task 2 (scan KB/) — independent, can parallel with Task 1 +Task 3 (--target hugo) — depends on Task 2 +Task 4 (test pipeline) — depends on Tasks 1 + 3 +Task 5 (deploy config) — depends on Task 1 +``` + +## Verification + +After all tasks: +1. `core docs sync --target hugo` populates `docs-site/content/` from all repos +2. `cd docs-site && hugo server` renders the full site +3. Navigation has 6 sections: Getting Started, CLI, Go, MCP, PHP, KB +4. All existing markdown renders correctly with auto-injected front matter +5. `hugo build` produces `public/` with no errors diff --git a/docs/plans/completed/2026-02-05-mcp-integration-original.md b/docs/plans/completed/2026-02-05-mcp-integration-original.md new file mode 100644 index 0000000..9b3a109 --- /dev/null +++ b/docs/plans/completed/2026-02-05-mcp-integration-original.md @@ -0,0 +1,851 @@ +# MCP Integration Implementation Plan + +> **Status:** Completed. MCP command now lives in `go-ai/cmd/mcpcmd/`. Code examples below use the old `init()` + `RegisterCommands()` pattern — the current approach uses `cli.WithCommands()` (see cli-meta-package-design.md). + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Add `core mcp serve` command with RAG and metrics tools, then configure the agentic-flows plugin to use it. + +**Architecture:** Create a new `mcp` command package that starts the pkg/mcp server with extended tools. RAG tools call the existing exported functions in internal/cmd/rag. Metrics tools call pkg/ai directly. The agentic-flows plugin gets a `.mcp.json` that spawns `core mcp serve`. + +**Tech Stack:** Go 1.25, github.com/modelcontextprotocol/go-sdk/mcp, pkg/rag, pkg/ai + +--- + +## Task 1: Add RAG tools to pkg/mcp + +**Files:** +- Create: `pkg/mcp/tools_rag.go` +- Modify: `pkg/mcp/mcp.go:99-101` (registerTools) +- Test: `pkg/mcp/tools_rag_test.go` + +**Step 1: Write the failing test** + +Create `pkg/mcp/tools_rag_test.go`: + +```go +package mcp + +import ( + "context" + "testing" + + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +func TestRAGQueryTool_Good(t *testing.T) { + // This test verifies the tool is registered and callable. + // It doesn't require Qdrant/Ollama running - just checks structure. + s, err := New(WithWorkspaceRoot("")) + if err != nil { + t.Fatalf("New() error: %v", err) + } + + // Check that rag_query tool is registered + tools := s.Server().ListTools() + found := false + for _, tool := range tools { + if tool.Name == "rag_query" { + found = true + break + } + } + if !found { + t.Error("rag_query tool not registered") + } +} + +func TestRAGQueryInput_Good(t *testing.T) { + input := RAGQueryInput{ + Question: "how do I deploy?", + Collection: "hostuk-docs", + TopK: 5, + } + if input.Question == "" { + t.Error("Question should not be empty") + } +} +``` + +**Step 2: Run test to verify it fails** + +Run: `go test -run TestRAGQueryTool ./pkg/mcp/... -v` +Expected: FAIL with "rag_query tool not registered" + +**Step 3: Create tools_rag.go with types and tool registration** + +Create `pkg/mcp/tools_rag.go`: + +```go +package mcp + +import ( + "context" + "fmt" + + ragcmd "forge.lthn.ai/core/cli/internal/cmd/rag" + "forge.lthn.ai/core/cli/pkg/rag" + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +// RAG tool input/output types + +// RAGQueryInput contains parameters for querying the vector database. +type RAGQueryInput struct { + Question string `json:"question"` + Collection string `json:"collection,omitempty"` + TopK int `json:"top_k,omitempty"` +} + +// RAGQueryOutput contains the query results. +type RAGQueryOutput struct { + Results []RAGResult `json:"results"` + Context string `json:"context"` +} + +// RAGResult represents a single search result. +type RAGResult struct { + Content string `json:"content"` + Score float32 `json:"score"` + Source string `json:"source"` + Metadata map[string]string `json:"metadata,omitempty"` +} + +// RAGIngestInput contains parameters for ingesting documents. +type RAGIngestInput struct { + Path string `json:"path"` + Collection string `json:"collection,omitempty"` + Recreate bool `json:"recreate,omitempty"` +} + +// RAGIngestOutput contains the ingestion results. +type RAGIngestOutput struct { + Success bool `json:"success"` + Path string `json:"path"` + Chunks int `json:"chunks"` + Message string `json:"message,omitempty"` +} + +// RAGCollectionsInput contains parameters for listing collections. +type RAGCollectionsInput struct { + ShowStats bool `json:"show_stats,omitempty"` +} + +// RAGCollectionsOutput contains the list of collections. +type RAGCollectionsOutput struct { + Collections []CollectionInfo `json:"collections"` +} + +// CollectionInfo describes a Qdrant collection. +type CollectionInfo struct { + Name string `json:"name"` + PointsCount uint64 `json:"points_count,omitempty"` + Status string `json:"status,omitempty"` +} + +// registerRAGTools adds RAG tools to the MCP server. +func (s *Service) registerRAGTools(server *mcp.Server) { + mcp.AddTool(server, &mcp.Tool{ + Name: "rag_query", + Description: "Query the vector database for relevant documents using semantic search", + }, s.ragQuery) + + mcp.AddTool(server, &mcp.Tool{ + Name: "rag_ingest", + Description: "Ingest a file or directory into the vector database", + }, s.ragIngest) + + mcp.AddTool(server, &mcp.Tool{ + Name: "rag_collections", + Description: "List available vector database collections", + }, s.ragCollections) +} + +func (s *Service) ragQuery(ctx context.Context, req *mcp.CallToolRequest, input RAGQueryInput) (*mcp.CallToolResult, RAGQueryOutput, error) { + s.logger.Info("MCP tool execution", "tool", "rag_query", "question", input.Question) + + collection := input.Collection + if collection == "" { + collection = "hostuk-docs" + } + topK := input.TopK + if topK <= 0 { + topK = 5 + } + + results, err := ragcmd.QueryDocs(ctx, input.Question, collection, topK) + if err != nil { + return nil, RAGQueryOutput{}, fmt.Errorf("query failed: %w", err) + } + + // Convert to output format + out := RAGQueryOutput{ + Results: make([]RAGResult, 0, len(results)), + Context: rag.FormatResultsContext(results), + } + for _, r := range results { + out.Results = append(out.Results, RAGResult{ + Content: r.Content, + Score: r.Score, + Source: r.Source, + Metadata: r.Metadata, + }) + } + + return nil, out, nil +} + +func (s *Service) ragIngest(ctx context.Context, req *mcp.CallToolRequest, input RAGIngestInput) (*mcp.CallToolResult, RAGIngestOutput, error) { + s.logger.Security("MCP tool execution", "tool", "rag_ingest", "path", input.Path) + + collection := input.Collection + if collection == "" { + collection = "hostuk-docs" + } + + // Check if path is a file or directory + info, err := s.medium.Stat(input.Path) + if err != nil { + return nil, RAGIngestOutput{}, fmt.Errorf("path not found: %w", err) + } + + if info.IsDir() { + err = ragcmd.IngestDirectory(ctx, input.Path, collection, input.Recreate) + if err != nil { + return nil, RAGIngestOutput{}, fmt.Errorf("ingest directory failed: %w", err) + } + return nil, RAGIngestOutput{ + Success: true, + Path: input.Path, + Message: fmt.Sprintf("Ingested directory into collection %s", collection), + }, nil + } + + chunks, err := ragcmd.IngestFile(ctx, input.Path, collection) + if err != nil { + return nil, RAGIngestOutput{}, fmt.Errorf("ingest file failed: %w", err) + } + + return nil, RAGIngestOutput{ + Success: true, + Path: input.Path, + Chunks: chunks, + Message: fmt.Sprintf("Ingested %d chunks into collection %s", chunks, collection), + }, nil +} + +func (s *Service) ragCollections(ctx context.Context, req *mcp.CallToolRequest, input RAGCollectionsInput) (*mcp.CallToolResult, RAGCollectionsOutput, error) { + s.logger.Info("MCP tool execution", "tool", "rag_collections") + + client, err := rag.NewQdrantClient(rag.DefaultQdrantConfig()) + if err != nil { + return nil, RAGCollectionsOutput{}, fmt.Errorf("connect to Qdrant: %w", err) + } + defer func() { _ = client.Close() }() + + names, err := client.ListCollections(ctx) + if err != nil { + return nil, RAGCollectionsOutput{}, fmt.Errorf("list collections: %w", err) + } + + out := RAGCollectionsOutput{ + Collections: make([]CollectionInfo, 0, len(names)), + } + + for _, name := range names { + info := CollectionInfo{Name: name} + if input.ShowStats { + cinfo, err := client.CollectionInfo(ctx, name) + if err == nil { + info.PointsCount = cinfo.PointsCount + info.Status = cinfo.Status.String() + } + } + out.Collections = append(out.Collections, info) + } + + return nil, out, nil +} +``` + +**Step 4: Update mcp.go to call registerRAGTools** + +In `pkg/mcp/mcp.go`, modify the `registerTools` function (around line 104) to add: + +```go +func (s *Service) registerTools(server *mcp.Server) { + // File operations (existing) + // ... existing code ... + + // RAG operations + s.registerRAGTools(server) +} +``` + +**Step 5: Run test to verify it passes** + +Run: `go test -run TestRAGQuery ./pkg/mcp/... -v` +Expected: PASS + +**Step 6: Commit** + +```bash +git add pkg/mcp/tools_rag.go pkg/mcp/tools_rag_test.go pkg/mcp/mcp.go +git commit -m "feat(mcp): add RAG tools (query, ingest, collections)" +``` + +--- + +## Task 2: Add metrics tools to pkg/mcp + +**Files:** +- Create: `pkg/mcp/tools_metrics.go` +- Modify: `pkg/mcp/mcp.go` (registerTools) +- Test: `pkg/mcp/tools_metrics_test.go` + +**Step 1: Write the failing test** + +Create `pkg/mcp/tools_metrics_test.go`: + +```go +package mcp + +import ( + "testing" +) + +func TestMetricsRecordTool_Good(t *testing.T) { + s, err := New(WithWorkspaceRoot("")) + if err != nil { + t.Fatalf("New() error: %v", err) + } + + tools := s.Server().ListTools() + found := false + for _, tool := range tools { + if tool.Name == "metrics_record" { + found = true + break + } + } + if !found { + t.Error("metrics_record tool not registered") + } +} + +func TestMetricsQueryTool_Good(t *testing.T) { + s, err := New(WithWorkspaceRoot("")) + if err != nil { + t.Fatalf("New() error: %v", err) + } + + tools := s.Server().ListTools() + found := false + for _, tool := range tools { + if tool.Name == "metrics_query" { + found = true + break + } + } + if !found { + t.Error("metrics_query tool not registered") + } +} +``` + +**Step 2: Run test to verify it fails** + +Run: `go test -run TestMetrics ./pkg/mcp/... -v` +Expected: FAIL + +**Step 3: Create tools_metrics.go** + +Create `pkg/mcp/tools_metrics.go`: + +```go +package mcp + +import ( + "context" + "fmt" + "time" + + "forge.lthn.ai/core/cli/pkg/ai" + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +// Metrics tool input/output types + +// MetricsRecordInput contains parameters for recording a metric event. +type MetricsRecordInput struct { + Type string `json:"type"` + AgentID string `json:"agent_id,omitempty"` + Repo string `json:"repo,omitempty"` + Data map[string]any `json:"data,omitempty"` +} + +// MetricsRecordOutput contains the result of recording. +type MetricsRecordOutput struct { + Success bool `json:"success"` + Timestamp time.Time `json:"timestamp"` +} + +// MetricsQueryInput contains parameters for querying metrics. +type MetricsQueryInput struct { + Since string `json:"since,omitempty"` // e.g., "7d", "24h" +} + +// MetricsQueryOutput contains the query results. +type MetricsQueryOutput struct { + Total int `json:"total"` + ByType []MetricCount `json:"by_type"` + ByRepo []MetricCount `json:"by_repo"` + ByAgent []MetricCount `json:"by_agent"` + Events []MetricEventBrief `json:"events,omitempty"` +} + +// MetricCount represents a count by key. +type MetricCount struct { + Key string `json:"key"` + Count int `json:"count"` +} + +// MetricEventBrief is a simplified event for output. +type MetricEventBrief struct { + Type string `json:"type"` + Timestamp time.Time `json:"timestamp"` + AgentID string `json:"agent_id,omitempty"` + Repo string `json:"repo,omitempty"` +} + +// registerMetricsTools adds metrics tools to the MCP server. +func (s *Service) registerMetricsTools(server *mcp.Server) { + mcp.AddTool(server, &mcp.Tool{ + Name: "metrics_record", + Description: "Record a metric event (AI task, security scan, job creation, etc.)", + }, s.metricsRecord) + + mcp.AddTool(server, &mcp.Tool{ + Name: "metrics_query", + Description: "Query recorded metrics with aggregation by type, repo, and agent", + }, s.metricsQuery) +} + +func (s *Service) metricsRecord(ctx context.Context, req *mcp.CallToolRequest, input MetricsRecordInput) (*mcp.CallToolResult, MetricsRecordOutput, error) { + s.logger.Info("MCP tool execution", "tool", "metrics_record", "type", input.Type) + + if input.Type == "" { + return nil, MetricsRecordOutput{}, fmt.Errorf("type is required") + } + + event := ai.Event{ + Type: input.Type, + Timestamp: time.Now(), + AgentID: input.AgentID, + Repo: input.Repo, + Data: input.Data, + } + + if err := ai.Record(event); err != nil { + return nil, MetricsRecordOutput{}, fmt.Errorf("record event: %w", err) + } + + return nil, MetricsRecordOutput{ + Success: true, + Timestamp: event.Timestamp, + }, nil +} + +func (s *Service) metricsQuery(ctx context.Context, req *mcp.CallToolRequest, input MetricsQueryInput) (*mcp.CallToolResult, MetricsQueryOutput, error) { + s.logger.Info("MCP tool execution", "tool", "metrics_query", "since", input.Since) + + since := input.Since + if since == "" { + since = "7d" + } + + duration, err := parseDuration(since) + if err != nil { + return nil, MetricsQueryOutput{}, fmt.Errorf("invalid since value: %w", err) + } + + sinceTime := time.Now().Add(-duration) + events, err := ai.ReadEvents(sinceTime) + if err != nil { + return nil, MetricsQueryOutput{}, fmt.Errorf("read events: %w", err) + } + + summary := ai.Summary(events) + + out := MetricsQueryOutput{ + Total: summary["total"].(int), + } + + // Convert by_type + if byType, ok := summary["by_type"].([]map[string]any); ok { + for _, entry := range byType { + out.ByType = append(out.ByType, MetricCount{ + Key: entry["key"].(string), + Count: entry["count"].(int), + }) + } + } + + // Convert by_repo + if byRepo, ok := summary["by_repo"].([]map[string]any); ok { + for _, entry := range byRepo { + out.ByRepo = append(out.ByRepo, MetricCount{ + Key: entry["key"].(string), + Count: entry["count"].(int), + }) + } + } + + // Convert by_agent + if byAgent, ok := summary["by_agent"].([]map[string]any); ok { + for _, entry := range byAgent { + out.ByAgent = append(out.ByAgent, MetricCount{ + Key: entry["key"].(string), + Count: entry["count"].(int), + }) + } + } + + // Include last 10 events for context + limit := 10 + if len(events) < limit { + limit = len(events) + } + for i := len(events) - limit; i < len(events); i++ { + ev := events[i] + out.Events = append(out.Events, MetricEventBrief{ + Type: ev.Type, + Timestamp: ev.Timestamp, + AgentID: ev.AgentID, + Repo: ev.Repo, + }) + } + + return nil, out, nil +} + +// parseDuration parses a human-friendly duration like "7d", "24h", "30d". +func parseDuration(s string) (time.Duration, error) { + if len(s) < 2 { + return 0, fmt.Errorf("invalid duration: %s", s) + } + + unit := s[len(s)-1] + value := s[:len(s)-1] + + var n int + if _, err := fmt.Sscanf(value, "%d", &n); err != nil { + return 0, fmt.Errorf("invalid duration: %s", s) + } + + if n <= 0 { + return 0, fmt.Errorf("duration must be positive: %s", s) + } + + switch unit { + case 'd': + return time.Duration(n) * 24 * time.Hour, nil + case 'h': + return time.Duration(n) * time.Hour, nil + case 'm': + return time.Duration(n) * time.Minute, nil + default: + return 0, fmt.Errorf("unknown unit %c in duration: %s", unit, s) + } +} +``` + +**Step 4: Update mcp.go to call registerMetricsTools** + +In `pkg/mcp/mcp.go`, add to `registerTools`: + +```go +func (s *Service) registerTools(server *mcp.Server) { + // ... existing file operations ... + + // RAG operations + s.registerRAGTools(server) + + // Metrics operations + s.registerMetricsTools(server) +} +``` + +**Step 5: Run test to verify it passes** + +Run: `go test -run TestMetrics ./pkg/mcp/... -v` +Expected: PASS + +**Step 6: Commit** + +```bash +git add pkg/mcp/tools_metrics.go pkg/mcp/tools_metrics_test.go pkg/mcp/mcp.go +git commit -m "feat(mcp): add metrics tools (record, query)" +``` + +--- + +## Task 3: Create `core mcp serve` command + +**Files:** +- Create: `internal/cmd/mcpcmd/cmd_mcp.go` +- Modify: `internal/variants/full.go` (add import) +- Test: Manual test via `core mcp serve` + +**Step 1: Create the mcp command package** + +Create `internal/cmd/mcpcmd/cmd_mcp.go`: + +```go +package mcpcmd + +import ( + "context" + "os" + "os/signal" + "syscall" + + "forge.lthn.ai/core/cli/pkg/cli" + "forge.lthn.ai/core/cli/pkg/i18n" + "forge.lthn.ai/core/cli/pkg/mcp" +) + +func init() { + cli.RegisterCommands(AddMCPCommands) +} + +var ( + mcpWorkspace string +) + +var mcpCmd = &cli.Command{ + Use: "mcp", + Short: i18n.T("cmd.mcp.short"), + Long: i18n.T("cmd.mcp.long"), +} + +var serveCmd = &cli.Command{ + Use: "serve", + Short: i18n.T("cmd.mcp.serve.short"), + Long: i18n.T("cmd.mcp.serve.long"), + RunE: func(cmd *cli.Command, args []string) error { + return runServe() + }, +} + +func AddMCPCommands(root *cli.Command) { + initMCPFlags() + mcpCmd.AddCommand(serveCmd) + root.AddCommand(mcpCmd) +} + +func initMCPFlags() { + serveCmd.Flags().StringVar(&mcpWorkspace, "workspace", "", i18n.T("cmd.mcp.serve.flag.workspace")) +} + +func runServe() error { + opts := []mcp.Option{} + + if mcpWorkspace != "" { + opts = append(opts, mcp.WithWorkspaceRoot(mcpWorkspace)) + } else { + // Default to unrestricted for MCP server + opts = append(opts, mcp.WithWorkspaceRoot("")) + } + + svc, err := mcp.New(opts...) + if err != nil { + return cli.Wrap(err, "create MCP service") + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Handle shutdown signals + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + go func() { + <-sigCh + cancel() + }() + + return svc.Run(ctx) +} +``` + +**Step 2: Add i18n strings** + +Create or update `pkg/i18n/en.yaml` (if it exists) or add to the existing i18n mechanism: + +```yaml +cmd.mcp.short: "MCP (Model Context Protocol) server" +cmd.mcp.long: "Start an MCP server for Claude Code integration with file, RAG, and metrics tools." +cmd.mcp.serve.short: "Start the MCP server" +cmd.mcp.serve.long: "Start the MCP server in stdio mode. Use MCP_ADDR env var for TCP mode." +cmd.mcp.serve.flag.workspace: "Restrict file operations to this directory (empty = unrestricted)" +``` + +**Step 3: Add import to full.go** + +Modify `internal/variants/full.go` to add: + +```go +import ( + // ... existing imports ... + _ "forge.lthn.ai/core/cli/internal/cmd/mcpcmd" +) +``` + +**Step 4: Build and test** + +Run: `go build && ./core mcp serve --help` +Expected: Help output showing the serve command + +**Step 5: Test MCP server manually** + +Run: `echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | ./core mcp serve` +Expected: JSON response listing all tools including rag_query, metrics_record, etc. + +**Step 6: Commit** + +```bash +git add internal/cmd/mcpcmd/cmd_mcp.go internal/variants/full.go +git commit -m "feat: add 'core mcp serve' command" +``` + +--- + +## Task 4: Configure agentic-flows plugin with .mcp.json + +**Files:** +- Create: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json` +- Modify: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.claude-plugin/plugin.json` (optional, add mcpServers) + +**Step 1: Create .mcp.json** + +Create `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`: + +```json +{ + "core-cli": { + "command": "core", + "args": ["mcp", "serve"], + "env": { + "MCP_WORKSPACE": "" + } + } +} +``` + +**Step 2: Verify plugin loads** + +Restart Claude Code and run `/mcp` to verify the core-cli server appears. + +**Step 3: Test MCP tools** + +Test that tools are available: +- `mcp__plugin_agentic-flows_core-cli__rag_query` +- `mcp__plugin_agentic-flows_core-cli__rag_ingest` +- `mcp__plugin_agentic-flows_core-cli__rag_collections` +- `mcp__plugin_agentic-flows_core-cli__metrics_record` +- `mcp__plugin_agentic-flows_core-cli__metrics_query` +- `mcp__plugin_agentic-flows_core-cli__file_read` +- etc. + +**Step 4: Commit plugin changes** + +```bash +cd /home/shared/hostuk/claude-plugins +git add plugins/agentic-flows/.mcp.json +git commit -m "feat(agentic-flows): add MCP server configuration for core-cli" +``` + +--- + +## Task 5: Update documentation + +**Files:** +- Modify: `/home/claude/.claude/projects/-home-claude/memory/MEMORY.md` +- Modify: `/home/claude/.claude/projects/-home-claude/memory/plugin-dev-notes.md` + +**Step 1: Update MEMORY.md** + +Add under "Core CLI MCP Server" section: + +```markdown +### Core CLI MCP Server +- **Command:** `core mcp serve` (stdio mode) or `MCP_ADDR=:9000 core mcp serve` (TCP) +- **Tools available:** + - File ops: file_read, file_write, file_edit, file_delete, file_rename, file_exists, dir_list, dir_create + - RAG: rag_query, rag_ingest, rag_collections + - Metrics: metrics_record, metrics_query + - Language: lang_detect, lang_list +- **Plugin config:** `plugins/agentic-flows/.mcp.json` +``` + +**Step 2: Update plugin-dev-notes.md** + +Add section: + +```markdown +## MCP Server (core mcp serve) + +### Available Tools +| Tool | Description | +|------|-------------| +| file_read | Read file contents | +| file_write | Write file contents | +| file_edit | Edit file (replace string) | +| file_delete | Delete file | +| file_rename | Rename/move file | +| file_exists | Check if file exists | +| dir_list | List directory contents | +| dir_create | Create directory | +| rag_query | Query vector DB | +| rag_ingest | Ingest file/directory | +| rag_collections | List collections | +| metrics_record | Record event | +| metrics_query | Query events | +| lang_detect | Detect file language | +| lang_list | List supported languages | + +### Example .mcp.json +```json +{ + "core-cli": { + "command": "core", + "args": ["mcp", "serve"] + } +} +``` +``` + +**Step 3: Commit documentation** + +```bash +git add ~/.claude/projects/-home-claude/memory/*.md +git commit -m "docs: update memory with MCP server tools" +``` + +--- + +## Summary + +| Task | Files | Purpose | +|------|-------|---------| +| 1 | `pkg/mcp/tools_rag.go` | RAG tools (query, ingest, collections) | +| 2 | `pkg/mcp/tools_metrics.go` | Metrics tools (record, query) | +| 3 | `internal/cmd/mcpcmd/cmd_mcp.go` | `core mcp serve` command | +| 4 | `plugins/agentic-flows/.mcp.json` | Plugin MCP configuration | +| 5 | Memory docs | Documentation updates | + +## Services Required + +- **Qdrant:** localhost:6333 (verified running) +- **Ollama:** localhost:11434 with nomic-embed-text (verified running) +- **InfluxDB:** localhost:8086 (optional, for future time-series metrics) diff --git a/docs/plans/completed/2026-02-17-lem-chat-design.md b/docs/plans/completed/2026-02-17-lem-chat-design.md new file mode 100644 index 0000000..3ff9f36 --- /dev/null +++ b/docs/plans/completed/2026-02-17-lem-chat-design.md @@ -0,0 +1,82 @@ +# LEM Chat — Web Components Design + +**Date**: 2026-02-17 +**Status**: Approved + +## Summary + +Standalone chat UI built with vanilla Web Components (Custom Elements + Shadow DOM). Connects to the MLX inference server's OpenAI-compatible SSE streaming endpoint. Zero framework dependencies. Single JS file output, embeddable anywhere. + +## Components + +| Element | Purpose | +|---------|---------| +| `` | Container. Conversation state, SSE connection, config via attributes | +| `` | Scrollable message list with auto-scroll anchoring | +| `` | Single message bubble. Streams tokens for assistant messages | +| `` | Text input, Enter to send, Shift+Enter for newline | + +## Data Flow + +``` +User types in + → dispatches 'lem-send' CustomEvent + → catches it + → adds user message to + → POST /v1/chat/completions {stream: true, messages: [...history]} + → reads SSE chunks via fetch + ReadableStream + → appends tokens to streaming + → on [DONE], finalises message +``` + +## Configuration + +```html + +``` + +Attributes: `endpoint`, `model`, `system-prompt`, `max-tokens`, `temperature` + +## Theming + +Shadow DOM with CSS custom properties: + +```css +--lem-bg: #1a1a1e; +--lem-msg-user: #2a2a3e; +--lem-msg-assistant: #1e1e2a; +--lem-accent: #5865f2; +--lem-text: #e0e0e0; +--lem-font: system-ui; +``` + +## Markdown + +Minimal inline parsing: fenced code blocks, inline code, bold, italic. No library. + +## File Structure + +``` +lem-chat/ +├── index.html # Demo page +├── src/ +│ ├── lem-chat.ts # Main container + SSE client +│ ├── lem-messages.ts # Message list with scroll anchoring +│ ├── lem-message.ts # Single message with streaming +│ ├── lem-input.ts # Text input +│ ├── markdown.ts # Minimal markdown → HTML +│ └── styles.ts # CSS template literals +├── package.json # typescript + esbuild +└── tsconfig.json +``` + +Build: `esbuild src/lem-chat.ts --bundle --outfile=dist/lem-chat.js` + +## Not in v1 + +- Model selection UI +- Conversation persistence +- File/image upload +- Syntax highlighting +- Typing indicators +- User avatars diff --git a/docs/plans/completed/2026-02-20-go-api-design-original.md b/docs/plans/completed/2026-02-20-go-api-design-original.md new file mode 100644 index 0000000..c979f81 --- /dev/null +++ b/docs/plans/completed/2026-02-20-go-api-design-original.md @@ -0,0 +1,657 @@ +# go-api Design — HTTP Gateway + OpenAPI SDK Generation + +**Date:** 2026-02-20 +**Author:** Virgil +**Status:** Phase 1 + Phase 2 + Phase 3 Complete (176 tests in go-api) +**Module:** `forge.lthn.ai/core/go-api` + +## Problem + +The Core Go ecosystem exposes 42+ tools via MCP (JSON-RPC), which is ideal for AI agents but inaccessible to regular HTTP clients, frontend applications, and third-party integrators. There is no unified HTTP gateway, no OpenAPI specification, and no generated SDKs. + +Both external customers (Host UK products) and Lethean network peers need programmatic access to the same services. The gateway also serves web routes, static assets, and streaming endpoints — not just REST APIs. + +## Solution + +A `go-api` package that acts as the central HTTP gateway: + +1. **Gin-based HTTP gateway** with extensible middleware via gin-contrib plugins +2. **RouteGroup interface** that subsystems implement to register their own endpoints (API, web, or both) +3. **WebSocket + SSE integration** for real-time streaming +4. **OpenAPI 3.1 spec generation** via runtime SpecBuilder (not swaggo annotations) +5. **SDK generation pipeline** targeting 11 languages via openapi-generator-cli + +## Architecture + +### Four-Protocol Access + +Same backend services, four client protocols: + +``` + ┌─── REST (go-api) POST /v1/ml/generate → JSON + │ + ├─── GraphQL (gqlgen) mutation { mlGenerate(...) { response } } +Client ────────────┤ + ├─── WebSocket (go-ws) subscribe ml.generate → streaming + │ + └─── MCP (go-ai) ml_generate → JSON-RPC +``` + +### Dependency Graph + +``` +go-api (Gin engine + middleware + OpenAPI) + ↑ imported by (each registers its own routes) + ├── go-ai/api/ → /v1/file/*, /v1/process/*, /v1/metrics/* + ├── go-ml/api/ → /v1/ml/* + ├── go-rag/api/ → /v1/rag/* + ├── go-agentic/api/ → /v1/tasks/* + ├── go-help/api/ → /v1/help/* + └── go-ws/api/ → /ws (WebSocket upgrade) +``` + +go-api has zero internal ecosystem dependencies. Subsystems import go-api, not the other way round. + +### Subsystem Opt-In + +Not every MCP tool becomes a REST endpoint. Each subsystem decides what to expose via a separate `RegisterAPI()` method, independent of MCP's `RegisterTools()`. A subsystem with 15 MCP tools might expose 5 REST endpoints. + +## Package Structure + +``` +forge.lthn.ai/core/go-api +├── api.go # Engine struct, New(), Serve(), Shutdown() +├── middleware.go # Auth, CORS, rate limiting, request logging, recovery +├── options.go # WithAddr, WithAuth, WithCORS, WithRateLimit, etc. +├── group.go # RouteGroup interface + registration +├── response.go # Envelope type, error responses, pagination +├── docs/ # Generated swagger docs (swaggo output) +├── sdk/ # SDK generation tooling / Makefile targets +└── go.mod # forge.lthn.ai/core/go-api +``` + +## Core Interface + +```go +// RouteGroup registers API routes onto a Gin router group. +// Subsystems implement this to expose their endpoints. +type RouteGroup interface { + // Name returns the route group identifier (e.g. "ml", "rag", "tasks") + Name() string + // BasePath returns the URL prefix (e.g. "/v1/ml") + BasePath() string + // RegisterRoutes adds handlers to the provided router group + RegisterRoutes(rg *gin.RouterGroup) +} + +// StreamGroup optionally declares WebSocket channels a subsystem publishes to. +type StreamGroup interface { + Channels() []string +} +``` + +### Subsystem Example (go-ml) + +```go +// In go-ml/api/routes.go +package api + +type Routes struct { + service *ml.Service +} + +func NewRoutes(svc *ml.Service) *Routes { + return &Routes{service: svc} +} + +func (r *Routes) Name() string { return "ml" } +func (r *Routes) BasePath() string { return "/v1/ml" } + +func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) { + rg.POST("/generate", r.Generate) + rg.POST("/score", r.Score) + rg.GET("/backends", r.Backends) + rg.GET("/status", r.Status) +} + +func (r *Routes) Channels() []string { + return []string{"ml.generate", "ml.status"} +} + +// @Summary Generate text via ML backend +// @Tags ml +// @Accept json +// @Produce json +// @Param input body MLGenerateInput true "Generation parameters" +// @Success 200 {object} Response[MLGenerateOutput] +// @Router /v1/ml/generate [post] +func (r *Routes) Generate(c *gin.Context) { + var input MLGenerateInput + if err := c.ShouldBindJSON(&input); err != nil { + c.JSON(400, api.Fail("invalid_input", err.Error())) + return + } + result, err := r.service.Generate(c.Request.Context(), input.Backend, input.Prompt, ml.GenOpts{ + Temperature: input.Temperature, + MaxTokens: input.MaxTokens, + Model: input.Model, + }) + if err != nil { + c.JSON(500, api.Fail("ml.generate_failed", err.Error())) + return + } + c.JSON(200, api.OK(MLGenerateOutput{ + Response: result, + Backend: input.Backend, + Model: input.Model, + })) +} +``` + +### Engine Wiring (in core CLI) + +```go +engine := api.New( + api.WithAddr(":8080"), + api.WithCORS("*"), + api.WithAuth(api.BearerToken(cfg.APIKey)), + api.WithRateLimit(100, time.Minute), + api.WithWSHub(wsHub), +) + +engine.Register(mlapi.NewRoutes(mlService)) +engine.Register(ragapi.NewRoutes(ragService)) +engine.Register(agenticapi.NewRoutes(agenticService)) + +engine.Serve(ctx) // Blocks until context cancelled +``` + +## Response Envelope + +All endpoints return a consistent envelope: + +```go +type Response[T any] struct { + Success bool `json:"success"` + Data T `json:"data,omitempty"` + Error *Error `json:"error,omitempty"` + Meta *Meta `json:"meta,omitempty"` +} + +type Error struct { + Code string `json:"code"` + Message string `json:"message"` + Details any `json:"details,omitempty"` +} + +type Meta struct { + RequestID string `json:"request_id"` + Duration string `json:"duration"` + Page int `json:"page,omitempty"` + PerPage int `json:"per_page,omitempty"` + Total int `json:"total,omitempty"` +} +``` + +Helper functions: + +```go +func OK[T any](data T) Response[T] +func Fail(code, message string) Response[any] +func Paginated[T any](data T, page, perPage, total int) Response[T] +``` + +## Middleware Stack + +```go +api.New( + api.WithAddr(":8080"), + api.WithCORS(api.CORSConfig{...}), // gin-contrib/cors + api.WithAuth(api.BearerToken("...")), // Phase 1: simple bearer token + api.WithRateLimit(100, time.Minute), // Per-IP sliding window + api.WithRequestID(), // X-Request-ID header generation + api.WithRecovery(), // Panic recovery → 500 response + api.WithLogger(slog.Default()), // Structured request logging +) +``` + +Auth evolution path: bearer token → API keys → Authentik (OIDC/forward auth). Middleware slot stays the same. + +## WebSocket Integration + +go-api wraps the existing go-ws Hub as a first-class transport: + +```go +// Automatic registration: +// GET /ws → WebSocket upgrade (go-ws Hub) + +// Client subscribes: {"type":"subscribe","channel":"ml.generate"} +// Events arrive: {"type":"event","channel":"ml.generate","data":{...}} +// Client unsubscribes: {"type":"unsubscribe","channel":"ml.generate"} +``` + +Subsystems implementing `StreamGroup` declare which channels they publish to. This metadata feeds into the OpenAPI spec as documentation. + +## OpenAPI + SDK Generation + +### Runtime Spec Generation (SpecBuilder) + +swaggo annotations were rejected because routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools already carry JSON Schema at runtime. Instead, a `SpecBuilder` constructs the full OpenAPI 3.1 spec from registered RouteGroups at runtime. + +```go +// Groups that implement DescribableGroup contribute endpoint metadata +type DescribableGroup interface { + RouteGroup + Describe() []RouteDescription +} + +// SpecBuilder assembles the spec from all groups +builder := &api.SpecBuilder{Title: "Core API", Description: "...", Version: "1.0.0"} +spec, _ := builder.Build(engine.Groups()) +``` + +### MCP-to-REST Bridge (ToolBridge) + +The `ToolBridge` converts MCP tool descriptors into REST POST endpoints and implements both `RouteGroup` and `DescribableGroup`. Each tool becomes `POST /{tool_name}`. Generic types are captured at MCP registration time via closures, enabling JSON unmarshalling to the correct input type at request time. + +```go +bridge := api.NewToolBridge("/v1/tools") +mcp.BridgeToAPI(mcpService, bridge) // Populates bridge from MCP tool registry +engine.Register(bridge) // Registers REST endpoints + OpenAPI metadata +``` + +### Swagger UI + +```go +// Built-in at GET /swagger/*any +// SpecBuilder output served via gin-swagger, cached via sync.Once +api.New(api.WithSwagger("Core API", "...", "1.0.0")) +``` + +### SDK Generation + +```bash +# Via openapi-generator-cli (11 languages supported) +core api sdk --lang go # Generate Go SDK +core api sdk --lang typescript-fetch,python # Multiple languages +core api sdk --lang rust --output ./sdk/ # Custom output dir +``` + +### CLI Commands + +```bash +core api spec # Emit OpenAPI JSON to stdout +core api spec --format yaml # YAML variant +core api spec --output spec.json # Write to file +core api sdk --lang python # Generate Python SDK +core api sdk --lang go,rust # Multiple SDKs +``` + +## Dependencies + +| Package | Purpose | +|---------|---------| +| `github.com/gin-gonic/gin` | HTTP framework | +| `github.com/swaggo/gin-swagger` | Swagger UI middleware | +| `github.com/gin-contrib/cors` | CORS middleware | +| `github.com/gin-contrib/secure` | Security headers | +| `github.com/gin-contrib/sessions` | Server-side sessions | +| `github.com/gin-contrib/authz` | Casbin authorisation | +| `github.com/gin-contrib/httpsign` | HTTP signature verification | +| `github.com/gin-contrib/slog` | Structured request logging | +| `github.com/gin-contrib/timeout` | Per-request timeouts | +| `github.com/gin-contrib/gzip` | Gzip compression | +| `github.com/gin-contrib/static` | Static file serving | +| `github.com/gin-contrib/pprof` | Runtime profiling | +| `github.com/gin-contrib/expvar` | Runtime metrics | +| `github.com/gin-contrib/location/v2` | Reverse proxy detection | +| `github.com/99designs/gqlgen` | GraphQL endpoint | +| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | Distributed tracing | +| `gopkg.in/yaml.v3` | YAML spec export | +| `forge.lthn.ai/core/go-ws` | WebSocket Hub (existing) | + +## Estimated Size + +| Component | LOC | +|-----------|-----| +| Engine + options | ~200 | +| Middleware | ~150 | +| Response envelope | ~80 | +| RouteGroup interface | ~30 | +| WebSocket integration | ~60 | +| Tests | ~300 | +| **Total go-api** | **~820** | + +Each subsystem's `api/` package adds ~100-200 LOC per route group. + +## Phase 1 — Implemented (20 Feb 2026) + +**Commit:** `17ae945` on Forge (`core/go-api`) + +| Component | Status | Tests | +|-----------|--------|-------| +| Response envelope (OK, Fail, Paginated) | Done | 9 | +| RouteGroup + StreamGroup interfaces | Done | 4 | +| Engine (New, Register, Handler, Serve) | Done | 9 | +| Bearer auth middleware | Done | 3 | +| Request ID middleware | Done | 2 | +| CORS middleware (gin-contrib/cors) | Done | 3 | +| WebSocket endpoint | Done | 3 | +| Swagger UI (gin-swagger) | Done | 2 | +| Health endpoint | Done | 1 | +| **Total** | **~840 LOC** | **36** | + +**Integration proof:** go-ml/api/ registers 3 endpoints with 12 tests (`0c23858`). + +## Phase 2 Wave 1 — Implemented (20 Feb 2026) + +**Commits:** `6bb7195..daae6f7` on Forge (`core/go-api`) + +| Component | Option | Dependency | Tests | +|-----------|--------|------------|-------| +| Authentik (forward auth + OIDC) | `WithAuthentik()` | `go-oidc/v3`, `oauth2` | 14 | +| Security headers (HSTS, CSP, etc.) | `WithSecure()` | `gin-contrib/secure` | 8 | +| Structured request logging | `WithSlog()` | `gin-contrib/slog` | 6 | +| Per-request timeouts | `WithTimeout()` | `gin-contrib/timeout` | 5 | +| Gzip compression | `WithGzip()` | `gin-contrib/gzip` | 5 | +| Static file serving | `WithStatic()` | `gin-contrib/static` | 5 | +| **Wave 1 Total** | | | **43** | + +**Cumulative:** 76 tests (36 Phase 1 + 43 Wave 1 - 3 shared), all passing. + +## Phase 2 Wave 2 — Implemented (20 Feb 2026) + +**Commits:** `64a8b16..67dcc83` on Forge (`core/go-api`) + +| Component | Option | Dependency | Tests | Notes | +|-----------|--------|------------|-------|-------| +| Brotli compression | `WithBrotli()` | `andybalholm/brotli` | 5 | Custom middleware; `gin-contrib/brotli` is empty stub | +| Response caching | `WithCache()` | none (in-memory) | 5 | Custom middleware; `gin-contrib/cache` is per-handler, not global | +| Server-side sessions | `WithSessions()` | `gin-contrib/sessions` | 5 | Cookie store, configurable name + secret | +| Casbin authorisation | `WithAuthz()` | `gin-contrib/authz`, `casbin/v2` | 5 | Subject via Basic Auth; RBAC policy model | +| **Wave 2 Total** | | | **20** | | + +**Cumulative:** 102 passing tests (2 integration skipped), all green. + +## Phase 2 Wave 3 — Implemented (20 Feb 2026) + +**Commits:** `7b3f99e..d517fa2` on Forge (`core/go-api`) + +| Component | Option | Dependency | Tests | Notes | +|-----------|--------|------------|-------|-------| +| HTTP signature verification | `WithHTTPSign()` | `gin-contrib/httpsign` | 5 | HMAC-SHA256; extensible via httpsign.Option | +| Server-Sent Events | `WithSSE()` | none (custom SSEBroker) | 6 | Channel filtering, multi-client broadcast, GET /events | +| Reverse proxy detection | `WithLocation()` | `gin-contrib/location/v2` | 5 | X-Forwarded-Host/Proto parsing | +| Locale detection | `WithI18n()` | `golang.org/x/text/language` | 5 | Accept-Language parsing, message lookup, GetLocale/GetMessage | +| GraphQL endpoint | `WithGraphQL()` | `99designs/gqlgen` | 5 | /graphql + optional /graphql/playground | +| **Wave 3 Total** | | | **26** | | + +**Cumulative:** 128 passing tests (2 integration skipped), all green. + +## Phase 2 Wave 4 — Implemented (21 Feb 2026) + +**Commits:** `32b3680..8ba1716` on Forge (`core/go-api`) + +| Component | Option | Dependency | Tests | Notes | +|-----------|--------|------------|-------|-------| +| Runtime profiling | `WithPprof()` | `gin-contrib/pprof` | 5 | /debug/pprof/* endpoints, flag-based mount | +| Runtime metrics | `WithExpvar()` | `gin-contrib/expvar` | 5 | /debug/vars endpoint, flag-based mount | +| Distributed tracing | `WithTracing()` | `otelgin` + OpenTelemetry SDK | 5 | W3C traceparent propagation, span attributes | +| **Wave 4 Total** | | | **15** | | + +**Cumulative:** 143 passing tests (2 integration skipped), all green. + +**Phase 2 complete.** All 4 waves implemented. Every planned plugin has a `With*()` option and tests. + +## Phase 3 — OpenAPI Spec Generation + SDK Codegen (21 Feb 2026) + +**Architecture:** Runtime OpenAPI generation via SpecBuilder (NOT swaggo annotations). Routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools carry JSON Schema at runtime. A `ToolBridge` converts tool descriptors into RouteGroup + OpenAPI metadata. A `SpecBuilder` constructs the full OpenAPI 3.1 spec. SDK codegen wraps `openapi-generator-cli`. + +### Wave 1: go-api (Tasks 1-5) + +**Commits:** `465bd60..1910aec` on Forge (`core/go-api`) + +| Component | File | Tests | Notes | +|-----------|------|-------|-------| +| DescribableGroup interface | `group.go` | 5 | Opt-in OpenAPI metadata for RouteGroups | +| ToolBridge | `bridge.go` | 6 | Tool descriptors → POST endpoints + DescribableGroup | +| SpecBuilder | `openapi.go` | 6 | OpenAPI 3.1 JSON with Response[T] envelope wrapping | +| Swagger refactor | `swagger.go` | 5 | Replaced hardcoded empty spec with SpecBuilder | +| Spec export | `export.go` | 5 | JSON + YAML export to file/writer | +| SDK codegen | `codegen.go` | 5 | 11-language wrapper for openapi-generator-cli | +| **Wave 1 Total** | | **32** | | + +### Wave 2: go-ai MCP bridge (Tasks 6-7) + +**Commits:** `2107eda..c37e1cf` on Forge (`core/go-ai`) + +| Component | File | Tests | Notes | +|-----------|------|-------|-------| +| Tool registry | `mcp/registry.go` | 5 | Generic `addToolRecorded[In,Out]` captures types in closures | +| BridgeToAPI | `mcp/bridge.go` | 5 | MCP tools → go-api ToolBridge, 10MB body limit, error classification | +| **Wave 2 Total** | | **10** | | + +### Wave 3: CLI commands (Tasks 8-9) + +**Commit:** `d6eec4d` on Forge (`core/cli` dev branch) + +| Component | File | Tests | Notes | +|-----------|------|-------|-------| +| `core api spec` | `cmd/api/cmd_spec.go` | 2 | JSON/YAML export, --output/--format flags | +| `core api sdk` | `cmd/api/cmd_sdk.go` | 2 | --lang (required), --output, --spec, --package flags | +| **Wave 3 Total** | | **4** | | + +**Cumulative go-api:** 176 passing tests. **Phase 3 complete.** + +### Known Limitations + +- **Subsystem tools excluded from bridge:** Subsystems call `mcp.AddTool` directly, bypassing `addToolRecorded`. Only the 10 built-in MCP tools appear in the REST bridge. Future: pass `*Service` to `RegisterTools` instead of `*mcp.Server`. +- **Flat schema only:** `structSchema` reflection handles flat structs but does not recurse into nested structs. Adequate for current tool inputs. +- **CLI spec produces empty bridge:** `core api spec` currently generates a spec with only `/health`. Full MCP integration requires wiring the MCP service into the CLI command. + +## Phase 2 — Gin Plugin Roadmap (Complete) + +All plugins drop in as `With*()` options on the Engine. No architecture changes needed. + +### Security & Auth + +| Plugin | Option | Purpose | Priority | +|--------|--------|---------|----------| +| ~~**Authentik**~~ | ~~`WithAuthentik()`~~ | ~~OIDC + forward auth integration.~~ | ~~**Done**~~ | +| ~~gin-contrib/secure~~ | ~~`WithSecure()`~~ | ~~Security headers: HSTS, X-Frame-Options, X-Content-Type-Options, CSP.~~ | ~~**Done**~~ | +| ~~gin-contrib/sessions~~ | ~~`WithSessions()`~~ | ~~Server-side sessions (cookie store). Web session management alongside Authentik tokens.~~ | ~~**Done**~~ | +| ~~gin-contrib/authz~~ | ~~`WithAuthz()`~~ | ~~Casbin-based authorisation. Policy-driven access control via RBAC.~~ | ~~**Done**~~ | +| ~~gin-contrib/httpsign~~ | ~~`WithHTTPSign()`~~ | ~~HTTP signature verification. HMAC-SHA256 with extensible options.~~ | ~~**Done**~~ | + +### Performance & Reliability + +| Plugin | Option | Purpose | Priority | +|--------|--------|---------|----------| +| ~~gin-contrib/cache~~ | ~~`WithCache()`~~ | ~~Response caching (in-memory). GET response caching with TTL, lazy eviction.~~ | ~~**Done**~~ | +| ~~gin-contrib/timeout~~ | ~~`WithTimeout()`~~ | ~~Per-request timeouts.~~ | ~~**Done**~~ | +| ~~gin-contrib/gzip~~ | ~~`WithGzip()`~~ | ~~Gzip response compression.~~ | ~~**Done**~~ | +| ~~gin-contrib/brotli~~ | ~~`WithBrotli()`~~ | ~~Brotli compression via `andybalholm/brotli`. Custom middleware (gin-contrib stub empty).~~ | ~~**Done**~~ | + +### Observability + +| Plugin | Option | Purpose | Priority | +|--------|--------|---------|----------| +| ~~gin-contrib/slog~~ | ~~`WithSlog()`~~ | ~~Structured request logging via slog.~~ | ~~**Done**~~ | +| ~~gin-contrib/pprof~~ | ~~`WithPprof()`~~ | ~~Runtime profiling endpoints at /debug/pprof/. Flag-based mount.~~ | ~~**Done**~~ | +| ~~gin-contrib/expvar~~ | ~~`WithExpvar()`~~ | ~~Go runtime metrics at /debug/vars. Flag-based mount.~~ | ~~**Done**~~ | +| ~~otelgin~~ | ~~`WithTracing()`~~ | ~~OpenTelemetry distributed tracing. W3C traceparent propagation.~~ | ~~**Done**~~ | + +### Content & Streaming + +| Plugin | Option | Purpose | Priority | +|--------|--------|---------|----------| +| ~~gin-contrib/static~~ | ~~`WithStatic()`~~ | ~~Serve static files.~~ | ~~**Done**~~ | +| ~~gin-contrib/sse~~ | ~~`WithSSE()`~~ | ~~Server-Sent Events. Custom SSEBroker with channel filtering, GET /events.~~ | ~~**Done**~~ | +| ~~gin-contrib/location~~ | ~~`WithLocation()`~~ | ~~Auto-detect scheme/host from X-Forwarded-* headers.~~ | ~~**Done**~~ | + +### Query Layer + +| Plugin | Option | Purpose | Priority | +|--------|--------|---------|----------| +| ~~99designs/gqlgen~~ | ~~`WithGraphQL()`~~ | ~~GraphQL endpoint at `/graphql` + optional playground. Accepts gqlgen ExecutableSchema.~~ | ~~**Done**~~ | + +The GraphQL schema can be generated from the same Go Input/Output structs that define the REST endpoints. gqlgen produces an `http.Handler` that mounts directly on Gin. Subsystems opt-in via: + +```go +// Subsystems that want GraphQL implement this alongside RouteGroup +type ResolverGroup interface { + // RegisterResolvers adds query/mutation resolvers to the GraphQL schema + RegisterResolvers(schema *graphql.Schema) +} +``` + +This means a subsystem like go-ml exposes: +- **REST:** `POST /v1/ml/generate` (existing) +- **GraphQL:** `mutation { mlGenerate(prompt: "...", backend: "mlx") { response, model } }` (same handler) +- **MCP:** `ml_generate` tool (existing) + +Four protocols, one set of handlers. + +### Ecosystem Integration + +| Plugin | Option | Purpose | Priority | +|--------|--------|---------|----------| +| ~~gin-contrib/i18n~~ | ~~`WithI18n()`~~ | ~~Locale detection via Accept-Language. Custom middleware using `golang.org/x/text/language`.~~ | ~~**Done**~~ | +| [gin-contrib/graceful](https://github.com/gin-contrib/graceful) | — | Already implemented in Engine.Serve(). Could swap to this for more robust lifecycle management if needed. | — | +| [gin-contrib/requestid](https://github.com/gin-contrib/requestid) | — | Already implemented. Theirs uses UUID, ours uses hex. Could swap for standards compliance. | — | + +### Implementation Order + +**Wave 1 (gateway hardening):** ~~Authentik, secure, slog, timeout, gzip, static~~ **DONE** (20 Feb 2026) +**Wave 2 (performance + auth):** ~~cache, sessions, authz, brotli~~ **DONE** (20 Feb 2026) +**Wave 3 (network + streaming):** ~~httpsign, sse, location, i18n, gqlgen~~ **DONE** (20 Feb 2026) +**Wave 4 (observability):** ~~pprof, expvar, tracing~~ **DONE** (21 Feb 2026) + +Each wave adds `With*()` options + tests. No breaking changes — existing code continues to work without any new options enabled. + +## Authentik Integration + +[Authentik](https://goauthentik.io/) is the identity provider and edge auth proxy. It handles user registration, login, MFA, social auth, SAML, and OIDC — so go-api doesn't have to. + +### Two Integration Modes + +**1. Forward Auth (web traffic)** + +Traefik sits in front of go-api. For web routes, Traefik's `forwardAuth` middleware checks with Authentik before passing the request through. Authentik handles login flows, session cookies, and consent. go-api receives pre-authenticated requests with identity headers. + +``` +Browser → Traefik → Authentik (forward auth) → go-api + ↓ + Login page (if unauthenticated) +``` + +go-api reads trusted headers set by Authentik: +``` +X-Authentik-Username: alice +X-Authentik-Groups: admins,developers +X-Authentik-Email: alice@example.com +X-Authentik-Uid: +X-Authentik-Jwt: +``` + +**2. OIDC Token Validation (API traffic)** + +API clients (SDKs, CLI tools, network peers) authenticate directly with Authentik's OAuth2 token endpoint, then send the JWT to go-api. go-api validates the JWT using Authentik's OIDC discovery endpoint (`.well-known/openid-configuration`). + +``` +SDK client → Authentik (token endpoint) → receives JWT +SDK client → go-api (Authorization: Bearer ) → validates via OIDC +``` + +### Implementation in go-api + +```go +engine := api.New( + api.WithAuthentik(api.AuthentikConfig{ + Issuer: "https://auth.lthn.ai/application/o/core-api/", + ClientID: "core-api", + TrustedProxy: true, // Trust X-Authentik-* headers from Traefik + }), +) +``` + +`WithAuthentik()` adds middleware that: +1. Checks for `X-Authentik-Jwt` header (forward auth mode) — validates signature, extracts claims +2. Falls back to `Authorization: Bearer ` header (direct OIDC mode) — validates via JWKS +3. Populates `c.Set("user", AuthentikUser{...})` in the Gin context for handlers to use +4. Skips /health, /swagger, and any public paths + +```go +// In any handler: +func (r *Routes) ListItems(c *gin.Context) { + user := api.GetUser(c) // Returns *AuthentikUser or nil + if user == nil { + c.JSON(401, api.Fail("unauthorised", "Authentication required")) + return + } + // user.Username, user.Groups, user.Email, user.UID available +} +``` + +### Auth Layers + +``` +Authentik (identity) → WHO is this? (user, groups, email) + ↓ +go-api middleware → IS their token valid? (JWT verification) + ↓ +Casbin authz (optional) → CAN they do this? (role → endpoint policies) + ↓ +Handler → DOES this (business logic) +``` + +Phase 1 bearer auth continues to work alongside Authentik — useful for service-to-service tokens, CI/CD, and development. `WithBearerAuth` and `WithAuthentik` can coexist. + +### Authentik Deployment + +Authentik runs as a Docker service alongside go-api, fronted by Traefik: +- **auth.lthn.ai** — Authentik UI + OIDC endpoints (production) +- **auth.leth.in** — Authentik for devnet/testnet +- Traefik routes `/outpost.goauthentik.io/` to Authentik's embedded outpost for forward auth + +### Dependencies + +| Package | Purpose | +|---------|---------| +| `github.com/coreos/go-oidc/v3` | OIDC discovery + JWT validation | +| `golang.org/x/oauth2` | OAuth2 token exchange (for server-side flows) | + +Both are standard Go libraries with no heavy dependencies. + +## Non-Goals + +- gRPC gateway +- Built-in user registration/login (Authentik handles this) +- API versioning beyond /v1/ prefix + +## Success Criteria + +### Phase 1 (Done) + +1. ~~`core api serve` starts a Gin server with registered subsystem routes~~ +2. ~~WebSocket subscriptions work alongside REST~~ +3. ~~Swagger UI accessible at `/swagger/`~~ +4. ~~All endpoints return consistent Response envelope~~ +5. ~~Bearer token auth protects all routes~~ +6. ~~First subsystem integration (go-ml/api/) proves the pattern~~ + +### Phase 2 (Done) + +7. ~~Security headers, compression, and caching active in production~~ +8. ~~Session-based auth alongside bearer tokens~~ +9. ~~HTTP signature verification for Lethean network peers~~ +10. ~~Static file serving for docs site and SDK downloads~~ +11. ~~GraphQL endpoint at `/graphql` with playground~~ + +### Phase 3 (Done) + +12. ~~`core api spec` emits valid OpenAPI 3.1 JSON via runtime SpecBuilder~~ +13. ~~`core api sdk` generates SDKs for 11 languages via openapi-generator-cli~~ +14. ~~MCP tools bridged to REST endpoints via ToolBridge + BridgeToAPI~~ +15. ~~OpenAPI spec includes Response[T] envelope wrapping~~ +16. ~~Spec export to file in JSON and YAML formats~~ diff --git a/docs/plans/completed/2026-02-20-go-api-plan-original.md b/docs/plans/completed/2026-02-20-go-api-plan-original.md new file mode 100644 index 0000000..11d164d --- /dev/null +++ b/docs/plans/completed/2026-02-20-go-api-plan-original.md @@ -0,0 +1,1503 @@ +# go-api Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Build `forge.lthn.ai/core/go-api`, a Gin-based REST framework with OpenAPI generation that subsystems plug into via a RouteGroup interface. + +**Architecture:** go-api provides the HTTP engine, middleware stack, response envelope, and OpenAPI tooling. Each ecosystem package (go-ml, go-rag, etc.) imports go-api and registers its own route group. WebSocket support via go-ws Hub runs alongside REST. + +**Tech Stack:** Go 1.25, Gin, swaggo/swag, gin-swagger, gin-contrib/cors, go-ws + +**Design doc:** `docs/plans/2026-02-20-go-api-design.md` + +**Repo location:** `/Users/snider/Code/go-api` (module: `forge.lthn.ai/core/go-api`) + +**Licence:** EUPL-1.2 + +**Convention:** UK English in comments and user-facing strings. Test naming: `_Good`, `_Bad`, `_Ugly`. + +--- + +### Task 1: Scaffold Repository + +**Files:** +- Create: `/Users/snider/Code/go-api/go.mod` +- Create: `/Users/snider/Code/go-api/response.go` +- Create: `/Users/snider/Code/go-api/response_test.go` +- Create: `/Users/snider/Code/go-api/LICENCE` + +**Step 1: Create repo and go.mod** + +```bash +mkdir -p /Users/snider/Code/go-api +cd /Users/snider/Code/go-api +git init +``` + +Create `go.mod`: +``` +module forge.lthn.ai/core/go-api + +go 1.25.5 + +require github.com/gin-gonic/gin v1.10.0 +``` + +Then run: +```bash +go mod tidy +``` + +**Step 2: Create LICENCE file** + +Copy the EUPL-1.2 licence text. Use the same LICENCE file as other ecosystem repos: +```bash +cp /Users/snider/Code/go-ai/LICENCE /Users/snider/Code/go-api/LICENCE +``` + +**Step 3: Commit scaffold** + +```bash +git add go.mod go.sum LICENCE +git commit -m "chore: scaffold go-api module with Gin dependency" +``` + +--- + +### Task 2: Response Envelope (TDD) + +**Files:** +- Create: `/Users/snider/Code/go-api/response.go` +- Create: `/Users/snider/Code/go-api/response_test.go` + +**Step 1: Write the failing tests** + +Create `response_test.go`: +```go +package api_test + +import ( + "encoding/json" + "testing" + + api "forge.lthn.ai/core/go-api" +) + +func TestOK_Good(t *testing.T) { + type Payload struct { + Name string `json:"name"` + } + resp := api.OK(Payload{Name: "test"}) + + if !resp.Success { + t.Fatal("expected Success to be true") + } + if resp.Data.Name != "test" { + t.Fatalf("expected Data.Name = test, got %s", resp.Data.Name) + } + if resp.Error != nil { + t.Fatal("expected Error to be nil") + } +} + +func TestFail_Good(t *testing.T) { + resp := api.Fail("not_found", "Resource not found") + + if resp.Success { + t.Fatal("expected Success to be false") + } + if resp.Error == nil { + t.Fatal("expected Error to be non-nil") + } + if resp.Error.Code != "not_found" { + t.Fatalf("expected Code = not_found, got %s", resp.Error.Code) + } + if resp.Error.Message != "Resource not found" { + t.Fatalf("expected Message = Resource not found, got %s", resp.Error.Message) + } +} + +func TestFailWithDetails_Good(t *testing.T) { + details := map[string]string{"field": "email"} + resp := api.FailWithDetails("validation_error", "Invalid input", details) + + if resp.Error.Details == nil { + t.Fatal("expected Details to be non-nil") + } +} + +func TestPaginated_Good(t *testing.T) { + items := []string{"a", "b", "c"} + resp := api.Paginated(items, 1, 10, 42) + + if !resp.Success { + t.Fatal("expected Success to be true") + } + if resp.Meta == nil { + t.Fatal("expected Meta to be non-nil") + } + if resp.Meta.Page != 1 { + t.Fatalf("expected Page = 1, got %d", resp.Meta.Page) + } + if resp.Meta.PerPage != 10 { + t.Fatalf("expected PerPage = 10, got %d", resp.Meta.PerPage) + } + if resp.Meta.Total != 42 { + t.Fatalf("expected Total = 42, got %d", resp.Meta.Total) + } +} + +func TestOK_JSON_Good(t *testing.T) { + resp := api.OK("hello") + data, err := json.Marshal(resp) + if err != nil { + t.Fatalf("marshal failed: %v", err) + } + + var raw map[string]any + if err := json.Unmarshal(data, &raw); err != nil { + t.Fatalf("unmarshal failed: %v", err) + } + + if raw["success"] != true { + t.Fatal("expected success = true in JSON") + } + if raw["data"] != "hello" { + t.Fatalf("expected data = hello, got %v", raw["data"]) + } + // error and meta should be omitted + if _, ok := raw["error"]; ok { + t.Fatal("expected error to be omitted from JSON") + } + if _, ok := raw["meta"]; ok { + t.Fatal("expected meta to be omitted from JSON") + } +} + +func TestFail_JSON_Good(t *testing.T) { + resp := api.Fail("err", "msg") + data, err := json.Marshal(resp) + if err != nil { + t.Fatalf("marshal failed: %v", err) + } + + var raw map[string]any + if err := json.Unmarshal(data, &raw); err != nil { + t.Fatalf("unmarshal failed: %v", err) + } + + if raw["success"] != false { + t.Fatal("expected success = false in JSON") + } + // data should be omitted + if _, ok := raw["data"]; ok { + t.Fatal("expected data to be omitted from JSON") + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: Compilation errors — `api.OK`, `api.Fail`, etc. not defined. + +**Step 3: Implement response.go** + +Create `response.go`: +```go +// Package api provides a Gin-based REST framework with OpenAPI generation. +// Subsystems implement RouteGroup to register their own endpoints. +package api + +// Response is the standard envelope for all API responses. +type Response[T any] struct { + Success bool `json:"success"` + Data T `json:"data,omitempty"` + Error *Error `json:"error,omitempty"` + Meta *Meta `json:"meta,omitempty"` +} + +// Error describes a failed API request. +type Error struct { + Code string `json:"code"` + Message string `json:"message"` + Details any `json:"details,omitempty"` +} + +// Meta carries pagination and request metadata. +type Meta struct { + RequestID string `json:"request_id,omitempty"` + Duration string `json:"duration,omitempty"` + Page int `json:"page,omitempty"` + PerPage int `json:"per_page,omitempty"` + Total int `json:"total,omitempty"` +} + +// OK returns a successful response wrapping data. +func OK[T any](data T) Response[T] { + return Response[T]{Success: true, Data: data} +} + +// Fail returns an error response with code and message. +func Fail(code, message string) Response[any] { + return Response[any]{ + Success: false, + Error: &Error{Code: code, Message: message}, + } +} + +// FailWithDetails returns an error response with additional detail payload. +func FailWithDetails(code, message string, details any) Response[any] { + return Response[any]{ + Success: false, + Error: &Error{Code: code, Message: message, Details: details}, + } +} + +// Paginated returns a successful response with pagination metadata. +func Paginated[T any](data T, page, perPage, total int) Response[T] { + return Response[T]{ + Success: true, + Data: data, + Meta: &Meta{Page: page, PerPage: perPage, Total: total}, + } +} +``` + +**Step 4: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: All 6 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/go-api +git add response.go response_test.go +git commit -m "feat: add response envelope with OK, Fail, Paginated helpers" +``` + +--- + +### Task 3: RouteGroup Interface + +**Files:** +- Create: `/Users/snider/Code/go-api/group.go` +- Create: `/Users/snider/Code/go-api/group_test.go` + +**Step 1: Write the failing test** + +Create `group_test.go`: +```go +package api_test + +import ( + "net/http" + "net/http/httptest" + "testing" + + api "forge.lthn.ai/core/go-api" + "github.com/gin-gonic/gin" +) + +// stubGroup is a minimal RouteGroup for testing. +type stubGroup struct{} + +func (s *stubGroup) Name() string { return "stub" } +func (s *stubGroup) BasePath() string { return "/v1/stub" } + +func (s *stubGroup) RegisterRoutes(rg *gin.RouterGroup) { + rg.GET("/ping", func(c *gin.Context) { + c.JSON(200, api.OK("pong")) + }) +} + +// stubStreamGroup implements both RouteGroup and StreamGroup. +type stubStreamGroup struct { + stubGroup +} + +func (s *stubStreamGroup) Channels() []string { + return []string{"stub.events", "stub.updates"} +} + +func TestRouteGroup_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + g := gin.New() + group := &stubGroup{} + + rg := g.Group(group.BasePath()) + group.RegisterRoutes(rg) + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/stub/ping", nil) + g.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } +} + +func TestStreamGroup_Good(t *testing.T) { + group := &stubStreamGroup{} + + // Verify it satisfies StreamGroup + var sg api.StreamGroup = group + channels := sg.Channels() + + if len(channels) != 2 { + t.Fatalf("expected 2 channels, got %d", len(channels)) + } + if channels[0] != "stub.events" { + t.Fatalf("expected stub.events, got %s", channels[0]) + } +} + +func TestRouteGroupName_Good(t *testing.T) { + group := &stubGroup{} + + var rg api.RouteGroup = group + if rg.Name() != "stub" { + t.Fatalf("expected name stub, got %s", rg.Name()) + } + if rg.BasePath() != "/v1/stub" { + t.Fatalf("expected basepath /v1/stub, got %s", rg.BasePath()) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: Compilation errors — `api.RouteGroup`, `api.StreamGroup` not defined. + +**Step 3: Implement group.go** + +Create `group.go`: +```go +package api + +import "github.com/gin-gonic/gin" + +// RouteGroup registers API routes onto a Gin router group. +// Subsystems implement this to expose their REST endpoints. +type RouteGroup interface { + // Name returns the route group identifier (e.g. "ml", "rag", "tasks"). + Name() string + // BasePath returns the URL prefix (e.g. "/v1/ml"). + BasePath() string + // RegisterRoutes adds handlers to the provided router group. + RegisterRoutes(rg *gin.RouterGroup) +} + +// StreamGroup optionally declares WebSocket channels a subsystem publishes to. +// Subsystems implementing both RouteGroup and StreamGroup expose both REST +// endpoints and real-time event channels. +type StreamGroup interface { + // Channels returns the WebSocket channel names this group publishes to. + Channels() []string +} +``` + +**Step 4: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: All tests PASS (previous 6 + new 3). + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/go-api +git add group.go group_test.go +git commit -m "feat: add RouteGroup and StreamGroup interfaces" +``` + +--- + +### Task 4: Engine + Options (TDD) + +**Files:** +- Create: `/Users/snider/Code/go-api/api.go` +- Create: `/Users/snider/Code/go-api/options.go` +- Create: `/Users/snider/Code/go-api/api_test.go` + +**Step 1: Write the failing tests** + +Create `api_test.go`: +```go +package api_test + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + "time" + + api "forge.lthn.ai/core/go-api" + "github.com/gin-gonic/gin" +) + +func TestNew_Good(t *testing.T) { + engine, err := api.New() + if err != nil { + t.Fatalf("New() failed: %v", err) + } + if engine == nil { + t.Fatal("expected non-nil engine") + } +} + +func TestNewWithAddr_Good(t *testing.T) { + engine, err := api.New(api.WithAddr(":9090")) + if err != nil { + t.Fatalf("New() failed: %v", err) + } + if engine.Addr() != ":9090" { + t.Fatalf("expected addr :9090, got %s", engine.Addr()) + } +} + +func TestDefaultAddr_Good(t *testing.T) { + engine, _ := api.New() + if engine.Addr() != ":8080" { + t.Fatalf("expected default addr :8080, got %s", engine.Addr()) + } +} + +func TestRegister_Good(t *testing.T) { + engine, _ := api.New() + group := &stubGroup{} + + engine.Register(group) + + if len(engine.Groups()) != 1 { + t.Fatalf("expected 1 group, got %d", len(engine.Groups())) + } + if engine.Groups()[0].Name() != "stub" { + t.Fatalf("expected group name stub, got %s", engine.Groups()[0].Name()) + } +} + +func TestRegisterMultiple_Good(t *testing.T) { + engine, _ := api.New() + engine.Register(&stubGroup{}) + engine.Register(&stubStreamGroup{}) + + if len(engine.Groups()) != 2 { + t.Fatalf("expected 2 groups, got %d", len(engine.Groups())) + } +} + +func TestHandler_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New() + engine.Register(&stubGroup{}) + + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/stub/ping", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } + + var resp map[string]any + json.Unmarshal(w.Body.Bytes(), &resp) + if resp["success"] != true { + t.Fatal("expected success = true") + } +} + +func TestHealthEndpoint_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New() + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/health", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200, got %d", w.Code) + } +} + +func TestServeAndShutdown_Good(t *testing.T) { + engine, _ := api.New(api.WithAddr(":0")) + engine.Register(&stubGroup{}) + + ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) + defer cancel() + + errCh := make(chan error, 1) + go func() { + errCh <- engine.Serve(ctx) + }() + + // Wait for context cancellation to trigger shutdown + <-ctx.Done() + + select { + case err := <-errCh: + if err != nil && err != http.ErrServerClosed && err != context.DeadlineExceeded { + t.Fatalf("Serve() returned unexpected error: %v", err) + } + case <-time.After(2 * time.Second): + t.Fatal("Serve() did not return after context cancellation") + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: Compilation errors — `api.New`, `api.WithAddr`, `api.Engine` not defined. + +**Step 3: Implement options.go** + +Create `options.go`: +```go +package api + +// Option configures the Engine. +type Option func(*Engine) error + +// WithAddr sets the listen address (default ":8080"). +func WithAddr(addr string) Option { + return func(e *Engine) error { + e.addr = addr + return nil + } +} +``` + +**Step 4: Implement api.go** + +Create `api.go`: +```go +package api + +import ( + "context" + "fmt" + "log/slog" + "net/http" + + "github.com/gin-gonic/gin" +) + +// Engine is the central REST API server. +// Register RouteGroups to add endpoints, then call Serve to start. +type Engine struct { + gin *gin.Engine + addr string + groups []RouteGroup + logger *slog.Logger + built bool +} + +// New creates an Engine with the given options. +func New(opts ...Option) (*Engine, error) { + e := &Engine{ + addr: ":8080", + logger: slog.Default(), + } + + for _, opt := range opts { + if err := opt(e); err != nil { + return nil, fmt.Errorf("apply option: %w", err) + } + } + + return e, nil +} + +// Addr returns the configured listen address. +func (e *Engine) Addr() string { + return e.addr +} + +// Groups returns all registered route groups. +func (e *Engine) Groups() []RouteGroup { + return e.groups +} + +// Register adds a RouteGroup to the engine. +// Routes are mounted when Handler() or Serve() is called. +func (e *Engine) Register(group RouteGroup) { + e.groups = append(e.groups, group) + e.built = false +} + +// build constructs the Gin engine with all registered groups. +func (e *Engine) build() { + if e.built && e.gin != nil { + return + } + + e.gin = gin.New() + e.gin.Use(gin.Recovery()) + + // Health endpoint + e.gin.GET("/health", func(c *gin.Context) { + c.JSON(200, OK("healthy")) + }) + + // Mount each route group + for _, group := range e.groups { + rg := e.gin.Group(group.BasePath()) + group.RegisterRoutes(rg) + e.logger.Info("registered route group", "name", group.Name(), "path", group.BasePath()) + } + + e.built = true +} + +// Handler returns the http.Handler for testing or custom server usage. +func (e *Engine) Handler() http.Handler { + e.build() + return e.gin +} + +// Serve starts the HTTP server and blocks until the context is cancelled. +// Performs graceful shutdown on context cancellation. +func (e *Engine) Serve(ctx context.Context) error { + e.build() + + srv := &http.Server{ + Addr: e.addr, + Handler: e.gin, + } + + errCh := make(chan error, 1) + go func() { + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + errCh <- err + } + close(errCh) + }() + + <-ctx.Done() + + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5_000_000_000) // 5s + defer cancel() + + if err := srv.Shutdown(shutdownCtx); err != nil { + return fmt.Errorf("shutdown: %w", err) + } + + if err, ok := <-errCh; ok { + return err + } + + return nil +} +``` + +**Step 5: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS. + +**Step 6: Commit** + +```bash +cd /Users/snider/Code/go-api +git add api.go options.go api_test.go +git commit -m "feat: add Engine with Register, Handler, Serve, and graceful shutdown" +``` + +--- + +### Task 5: Middleware (TDD) + +**Files:** +- Create: `/Users/snider/Code/go-api/middleware.go` +- Create: `/Users/snider/Code/go-api/middleware_test.go` +- Modify: `/Users/snider/Code/go-api/options.go` — add middleware options + +**Step 1: Write the failing tests** + +Create `middleware_test.go`: +```go +package api_test + +import ( + "net/http" + "net/http/httptest" + "testing" + + api "forge.lthn.ai/core/go-api" + "github.com/gin-gonic/gin" +) + +func TestBearerAuth_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithBearerAuth("secret-token")) + engine.Register(&stubGroup{}) + handler := engine.Handler() + + // Request without token → 401 + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/stub/ping", nil) + handler.ServeHTTP(w, req) + + if w.Code != 401 { + t.Fatalf("expected 401 without token, got %d", w.Code) + } + + // Request with correct token → 200 + w = httptest.NewRecorder() + req, _ = http.NewRequest("GET", "/v1/stub/ping", nil) + req.Header.Set("Authorization", "Bearer secret-token") + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200 with correct token, got %d", w.Code) + } +} + +func TestBearerAuth_Bad(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithBearerAuth("secret-token")) + engine.Register(&stubGroup{}) + handler := engine.Handler() + + // Wrong token → 401 + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/stub/ping", nil) + req.Header.Set("Authorization", "Bearer wrong-token") + handler.ServeHTTP(w, req) + + if w.Code != 401 { + t.Fatalf("expected 401 with wrong token, got %d", w.Code) + } +} + +func TestHealthBypassesAuth_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithBearerAuth("secret-token")) + handler := engine.Handler() + + // Health endpoint should not require auth + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/health", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200 for /health without auth, got %d", w.Code) + } +} + +func TestRequestID_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithRequestID()) + engine.Register(&stubGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/stub/ping", nil) + handler.ServeHTTP(w, req) + + rid := w.Header().Get("X-Request-ID") + if rid == "" { + t.Fatal("expected X-Request-ID header to be set") + } +} + +func TestRequestIDPreserved_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithRequestID()) + engine.Register(&stubGroup{}) + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/v1/stub/ping", nil) + req.Header.Set("X-Request-ID", "my-custom-id") + handler.ServeHTTP(w, req) + + rid := w.Header().Get("X-Request-ID") + if rid != "my-custom-id" { + t.Fatalf("expected X-Request-ID = my-custom-id, got %s", rid) + } +} + +func TestCORS_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithCORS("https://example.com")) + engine.Register(&stubGroup{}) + handler := engine.Handler() + + // Preflight request + w := httptest.NewRecorder() + req, _ := http.NewRequest("OPTIONS", "/v1/stub/ping", nil) + req.Header.Set("Origin", "https://example.com") + req.Header.Set("Access-Control-Request-Method", "POST") + handler.ServeHTTP(w, req) + + origin := w.Header().Get("Access-Control-Allow-Origin") + if origin != "https://example.com" { + t.Fatalf("expected CORS origin https://example.com, got %s", origin) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: Compilation errors — `WithBearerAuth`, `WithRequestID`, `WithCORS` not defined. + +**Step 3: Implement middleware.go** + +Create `middleware.go`: +```go +package api + +import ( + "crypto/rand" + "encoding/hex" + "strings" + + "github.com/gin-gonic/gin" +) + +// bearerAuthMiddleware validates Bearer tokens. +// Skips paths listed in skip (e.g. /health, /swagger). +func bearerAuthMiddleware(token string, skip []string) gin.HandlerFunc { + return func(c *gin.Context) { + path := c.Request.URL.Path + for _, s := range skip { + if strings.HasPrefix(path, s) { + c.Next() + return + } + } + + header := c.GetHeader("Authorization") + if header == "" { + c.JSON(401, Fail("unauthorised", "Missing Authorization header")) + c.Abort() + return + } + + parts := strings.SplitN(header, " ", 2) + if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") || parts[1] != token { + c.JSON(401, Fail("unauthorised", "Invalid bearer token")) + c.Abort() + return + } + + c.Next() + } +} + +// requestIDMiddleware sets X-Request-ID on every response. +// If the client sends one, it is preserved; otherwise a random ID is generated. +func requestIDMiddleware() gin.HandlerFunc { + return func(c *gin.Context) { + rid := c.GetHeader("X-Request-ID") + if rid == "" { + b := make([]byte, 16) + rand.Read(b) + rid = hex.EncodeToString(b) + } + c.Header("X-Request-ID", rid) + c.Set("request_id", rid) + c.Next() + } +} +``` + +**Step 4: Add middleware options to options.go** + +Append to `options.go`: +```go +import "github.com/gin-contrib/cors" + +// WithBearerAuth adds bearer token authentication middleware. +// The /health and /swagger paths are excluded from authentication. +func WithBearerAuth(token string) Option { + return func(e *Engine) error { + e.middlewares = append(e.middlewares, bearerAuthMiddleware(token, []string{"/health", "/swagger"})) + return nil + } +} + +// WithRequestID adds a middleware that sets X-Request-ID on every response. +func WithRequestID() Option { + return func(e *Engine) error { + e.middlewares = append(e.middlewares, requestIDMiddleware()) + return nil + } +} + +// WithCORS configures Cross-Origin Resource Sharing. +// Pass "*" to allow all origins, or specific origins. +func WithCORS(allowOrigins ...string) Option { + return func(e *Engine) error { + config := cors.DefaultConfig() + if len(allowOrigins) == 1 && allowOrigins[0] == "*" { + config.AllowAllOrigins = true + } else { + config.AllowOrigins = allowOrigins + } + config.AllowMethods = []string{"GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"} + config.AllowHeaders = []string{"Authorization", "Content-Type", "X-Request-ID"} + e.middlewares = append(e.middlewares, cors.New(config)) + return nil + } +} +``` + +Update `Engine` struct in `api.go` to include `middlewares []gin.HandlerFunc` field, and apply them in `build()`: +```go +// Add to Engine struct: +middlewares []gin.HandlerFunc + +// In build(), after gin.New() and gin.Recovery(), before health endpoint: +for _, mw := range e.middlewares { + e.gin.Use(mw) +} +``` + +**Step 5: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS. + +**Step 6: Commit** + +```bash +cd /Users/snider/Code/go-api +git add middleware.go middleware_test.go options.go api.go +git commit -m "feat: add bearer auth, request ID, and CORS middleware" +``` + +--- + +### Task 6: WebSocket Integration (TDD) + +**Files:** +- Create: `/Users/snider/Code/go-api/websocket.go` +- Create: `/Users/snider/Code/go-api/websocket_test.go` +- Modify: `/Users/snider/Code/go-api/options.go` — add WithWSHub +- Modify: `/Users/snider/Code/go-api/api.go` — mount /ws route + +**Step 1: Write the failing test** + +Create `websocket_test.go`: +```go +package api_test + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + + api "forge.lthn.ai/core/go-api" + "github.com/gin-gonic/gin" + "github.com/gorilla/websocket" +) + +func TestWSEndpoint_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithWSHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + upgrader := websocket.Upgrader{CheckOrigin: func(r *http.Request) bool { return true }} + conn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + return + } + defer conn.Close() + conn.WriteMessage(websocket.TextMessage, []byte("hello")) + }))) + + srv := httptest.NewServer(engine.Handler()) + defer srv.Close() + + wsURL := "ws" + strings.TrimPrefix(srv.URL, "http") + "/ws" + conn, _, err := websocket.DefaultDialer.Dial(wsURL, nil) + if err != nil { + t.Fatalf("dial failed: %v", err) + } + defer conn.Close() + + _, msg, err := conn.ReadMessage() + if err != nil { + t.Fatalf("read failed: %v", err) + } + if string(msg) != "hello" { + t.Fatalf("expected hello, got %s", string(msg)) + } +} + +func TestNoWSHandler_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New() + handler := engine.Handler() + + // /ws should 404 when no handler configured + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/ws", nil) + handler.ServeHTTP(w, req) + + if w.Code != 404 { + t.Fatalf("expected 404 without WS handler, got %d", w.Code) + } +} + +func TestChannelListing_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New() + engine.Register(&stubStreamGroup{}) + + channels := engine.Channels() + if len(channels) != 2 { + t.Fatalf("expected 2 channels, got %d", len(channels)) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: Compilation errors. + +**Step 3: Implement websocket.go + option + engine changes** + +Create `websocket.go`: +```go +package api + +import ( + "net/http" + + "github.com/gin-gonic/gin" +) + +// wrapWSHandler adapts a standard http.Handler to a Gin handler for the /ws route. +func wrapWSHandler(h http.Handler) gin.HandlerFunc { + return func(c *gin.Context) { + h.ServeHTTP(c.Writer, c.Request) + } +} +``` + +Add to `options.go`: +```go +// WithWSHandler registers a WebSocket handler at GET /ws. +// Typically this wraps a go-ws Hub.Handler(). +func WithWSHandler(h http.Handler) Option { + return func(e *Engine) error { + e.wsHandler = h + return nil + } +} +``` + +Add to `Engine` struct in `api.go`: +```go +wsHandler http.Handler +``` + +Add to `build()` after mounting route groups: +```go +// WebSocket endpoint +if e.wsHandler != nil { + e.gin.GET("/ws", wrapWSHandler(e.wsHandler)) +} +``` + +Add `Channels()` method to `Engine`: +```go +// Channels returns all WebSocket channel names from registered StreamGroups. +func (e *Engine) Channels() []string { + var channels []string + for _, g := range e.groups { + if sg, ok := g.(StreamGroup); ok { + channels = append(channels, sg.Channels()...) + } + } + return channels +} +``` + +**Step 4: Run go mod tidy to pick up gorilla/websocket** + +```bash +cd /Users/snider/Code/go-api +go mod tidy +``` + +**Step 5: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS. + +**Step 6: Commit** + +```bash +cd /Users/snider/Code/go-api +git add websocket.go websocket_test.go options.go api.go go.mod go.sum +git commit -m "feat: add WebSocket endpoint and channel listing from StreamGroups" +``` + +--- + +### Task 7: Swagger/OpenAPI Integration + +**Files:** +- Create: `/Users/snider/Code/go-api/swagger.go` +- Create: `/Users/snider/Code/go-api/swagger_test.go` +- Modify: `/Users/snider/Code/go-api/options.go` — add WithSwagger +- Modify: `/Users/snider/Code/go-api/api.go` — mount swagger routes + +**Step 1: Write the failing test** + +Create `swagger_test.go`: +```go +package api_test + +import ( + "net/http" + "net/http/httptest" + "testing" + + api "forge.lthn.ai/core/go-api" + "github.com/gin-gonic/gin" +) + +func TestSwaggerEndpoint_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New(api.WithSwagger("Core API", "REST API for the Lethean ecosystem", "0.1.0")) + engine.Register(&stubGroup{}) + handler := engine.Handler() + + // Swagger JSON endpoint + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/swagger/doc.json", nil) + handler.ServeHTTP(w, req) + + if w.Code != 200 { + t.Fatalf("expected 200 for swagger doc.json, got %d", w.Code) + } + + body := w.Body.String() + if len(body) == 0 { + t.Fatal("expected non-empty swagger doc") + } +} + +func TestSwaggerDisabledByDefault_Good(t *testing.T) { + gin.SetMode(gin.TestMode) + engine, _ := api.New() + handler := engine.Handler() + + w := httptest.NewRecorder() + req, _ := http.NewRequest("GET", "/swagger/doc.json", nil) + handler.ServeHTTP(w, req) + + if w.Code != 404 { + t.Fatalf("expected 404 when swagger disabled, got %d", w.Code) + } +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v +``` + +Expected: Compilation errors. + +**Step 3: Implement swagger.go + option** + +Create `swagger.go`: +```go +package api + +import ( + "github.com/gin-gonic/gin" + swaggerFiles "github.com/swaggo/files" + ginSwagger "github.com/swaggo/gin-swagger" + "github.com/swaggo/swag" +) + +// swaggerSpec holds a minimal OpenAPI spec for runtime serving. +type swaggerSpec struct { + title string + description string + version string +} + +func (s *swaggerSpec) ReadDoc() string { + // Minimal OpenAPI 3.0 document — swaggo generates the full one at build time. + // This serves as the runtime fallback and base template. + return `{ + "swagger": "2.0", + "info": { + "title": "` + s.title + `", + "description": "` + s.description + `", + "version": "` + s.version + `" + }, + "basePath": "/", + "paths": {} +}` +} + +// registerSwagger mounts the swagger UI and doc.json endpoint. +func registerSwagger(g *gin.Engine, title, description, version string) { + spec := &swaggerSpec{title: title, description: description, version: version} + swag.Register(swag.Name, spec) + + g.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler)) +} +``` + +Add to `options.go`: +```go +// WithSwagger enables the Swagger UI at /swagger/. +func WithSwagger(title, description, version string) Option { + return func(e *Engine) error { + e.swaggerTitle = title + e.swaggerDesc = description + e.swaggerVersion = version + e.swaggerEnabled = true + return nil + } +} +``` + +Add fields to `Engine` struct: +```go +swaggerEnabled bool +swaggerTitle string +swaggerDesc string +swaggerVersion string +``` + +Add to `build()` after WebSocket: +```go +// Swagger UI +if e.swaggerEnabled { + registerSwagger(e.gin, e.swaggerTitle, e.swaggerDesc, e.swaggerVersion) +} +``` + +**Step 4: Run go mod tidy** + +```bash +cd /Users/snider/Code/go-api +go get github.com/swaggo/gin-swagger github.com/swaggo/files github.com/swaggo/swag +go mod tidy +``` + +**Step 5: Run tests to verify they pass** + +```bash +cd /Users/snider/Code/go-api +go test ./... -v -count=1 +``` + +Expected: All tests PASS. + +**Step 6: Commit** + +```bash +cd /Users/snider/Code/go-api +git add swagger.go swagger_test.go options.go api.go go.mod go.sum +git commit -m "feat: add Swagger UI endpoint with runtime spec serving" +``` + +--- + +### Task 8: CLAUDE.md + README.md + +**Files:** +- Create: `/Users/snider/Code/go-api/CLAUDE.md` +- Create: `/Users/snider/Code/go-api/README.md` + +**Step 1: Write CLAUDE.md** + +```markdown +# CLAUDE.md + +This file provides guidance to Claude Code when working with the go-api repository. + +## Project Overview + +**go-api** is the REST framework for the Lethean Go ecosystem. It provides a Gin-based HTTP engine with middleware, response envelopes, WebSocket integration, and OpenAPI generation. Subsystems implement the `RouteGroup` interface to register their own endpoints. + +- **Module path**: `forge.lthn.ai/core/go-api` +- **Language**: Go 1.25 +- **Licence**: EUPL-1.2 + +## Build & Test Commands + +```bash +go test ./... # Run all tests +go test -run TestName ./... # Run a single test +go test -v -race ./... # Verbose with race detector +go build ./... # Build (library — no main package) +go vet ./... # Vet +``` + +## Coding Standards + +- **UK English** in comments and user-facing strings (colour, organisation, unauthorised) +- **Conventional commits**: `type(scope): description` +- **Co-Author**: `Co-Authored-By: Virgil ` +- **Error handling**: Return wrapped errors with context, never panic +- **Test naming**: `_Good` (happy path), `_Bad` (expected errors), `_Ugly` (panics/edge cases) +- **Licence**: EUPL-1.2 +``` + +**Step 2: Write README.md** + +Brief README with quick start and links to design doc. + +**Step 3: Commit** + +```bash +cd /Users/snider/Code/go-api +git add CLAUDE.md README.md +git commit -m "docs: add CLAUDE.md and README.md" +``` + +--- + +### Task 9: Create Forge Repo + Push + +**Step 1: Create repo on Forge** + +```bash +curl -s -X POST "https://forge.lthn.ai/api/v1/orgs/core/repos" \ + -H "Authorization: token 375068d101922dd1cf269e8b8cb77a0f99d1b486" \ + -H "Content-Type: application/json" \ + -d '{"name":"go-api","description":"REST framework + OpenAPI SDK generation for the Lethean Go ecosystem","default_branch":"main","auto_init":false,"license":"EUPL-1.2"}' +``` + +**Step 2: Add remote and push** + +```bash +cd /Users/snider/Code/go-api +git remote add forge ssh://git@forge.lthn.ai:2223/core/go-api.git +git branch -M main +git push -u forge main +``` + +**Step 3: Verify on Forge** + +```bash +curl -s "https://forge.lthn.ai/api/v1/repos/core/go-api" \ + -H "Authorization: token 375068d101922dd1cf269e8b8cb77a0f99d1b486" | jq .name +``` + +Expected: `"go-api"` + +--- + +### Task 10: Integration Test — First Subsystem (go-ml/api) + +This task validates the framework by building the first real subsystem integration. It lives in go-ml, not go-api. + +**Files:** +- Create: `/Users/snider/Code/go-ml/api/routes.go` +- Create: `/Users/snider/Code/go-ml/api/routes_test.go` + +**Step 1: Write the failing test in go-ml** + +Create `api/routes_test.go` in go-ml that: +1. Creates a `Routes` with a mock `ml.Service` +2. Registers it on an `api.Engine` +3. Sends `POST /v1/ml/backends` and asserts a 200 response with the response envelope + +**Step 2: Implement api/routes.go** + +Implement `Routes` struct that wraps `*ml.Service` and exposes: +- `POST /v1/ml/generate` +- `POST /v1/ml/score` +- `GET /v1/ml/backends` +- `GET /v1/ml/status` + +Each handler uses `c.ShouldBindJSON()` for input and `api.OK()` / `api.Fail()` for responses. + +**Step 3: Run tests** + +```bash +cd /Users/snider/Code/go-ml +go test ./api/... -v +``` + +**Step 4: Commit in go-ml** + +```bash +cd /Users/snider/Code/go-ml +git add api/ +git commit -m "feat(api): add REST route group for ML endpoints via go-api" +``` + +--- + +## Dependency Summary + +``` +Task 1 (scaffold) → Task 2 (response) → Task 3 (group) → Task 4 (engine) + → Task 5 (middleware) → Task 6 (websocket) → Task 7 (swagger) + → Task 8 (docs) → Task 9 (forge) → Task 10 (integration) +``` + +All tasks are sequential — each builds on the previous. + +## Estimated Timeline + +- Tasks 1-7: Core go-api package (~820 LOC) +- Task 8: Documentation +- Task 9: Forge deployment +- Task 10: First subsystem integration proof diff --git a/docs/plans/completed/2026-02-21-cli-meta-package-design-original.md b/docs/plans/completed/2026-02-21-cli-meta-package-design-original.md new file mode 100644 index 0000000..eaf886f --- /dev/null +++ b/docs/plans/completed/2026-02-21-cli-meta-package-design-original.md @@ -0,0 +1,128 @@ +# CLI Meta-Package Restructure — Design + +**Goal:** Transform `core/cli` from a 35K LOC monolith into a thin assembly repo that ships variant binaries. Domain repos own their commands. `go/pkg/cli` is the only import any domain package needs for CLI concerns. + +**Architecture:** Commands register as framework services via `cli.WithCommands()`, passed to `cli.Main()`. Command code lives in the domain repos that own the business logic. The cli repo is a thin `main.go` that wires them together. + +**Tech Stack:** go/pkg/cli (wraps cobra + charmbracelet), Core framework lifecycle, Taskfile + +--- + +## 1. CLI SDK — The Single Import + +`forge.lthn.ai/core/go/pkg/cli` is the **only** import domain packages use for CLI concerns. It wraps cobra, charmbracelet, and stdlib behind a stable API. If the underlying libraries change, only `go/pkg/cli` is touched — every domain repo is insulated. + +### Already done + +- **Cobra:** `Command` type alias, `NewCommand()`, `NewGroup()`, `NewRun()`, flag helpers (`StringFlag`, `BoolFlag`, `IntFlag`, `StringSliceFlag`), arg validators +- **Output:** `Success()`, `Error()`, `Warn()`, `Info()`, `Table`, `Section()`, `Label()`, `Task()`, `Hint()` +- **Prompts:** `Confirm()`, `Question()`, `Choose()`, `ChooseMulti()` with grammar-based action variants +- **Styles:** 17 pre-built styles, `AnsiStyle` builder, Tailwind colour constants (47 hex values) +- **Glyphs:** `:check:`, `:cross:`, `:warn:` etc. with Unicode/Emoji/ASCII themes +- **Layout:** HLCRF composite renderer (Header/Left/Content/Right/Footer) +- **Errors:** `Wrap()`, `WrapVerb()`, `ExitError`, `Is()`, `As()` +- **Logging:** `LogDebug()`, `LogInfo()`, `LogWarn()`, `LogError()`, `LogSecurity()` +- **TUI primitives:** `Spinner`, `ProgressBar`, `InteractiveList`, `TextInput`, `Viewport`, `RunTUI` +- **Command registration:** `WithCommands(name, fn)` — registers commands as framework services + +### Stubbed for later (interface exists, returns simple fallback) + +- `Form(fields []FormField) (map[string]string, error)` — multi-field form (backed by huh later) +- `FilePicker(opts ...FilePickerOption) (string, error)` — file browser +- `Tabs(items []TabItem) error` — tabbed content panes + +### Rule + +Domain packages import `forge.lthn.ai/core/go/pkg/cli` and **nothing else** for CLI concerns. No `cobra`, no `lipgloss`, no `bubbletea`. + +--- + +## 2. Command Registration — Framework Lifecycle + +Commands register through the Core framework's service lifecycle, not through global state or `init()` functions. + +### The contract + +Each domain repo exports an `Add*Commands(root *cli.Command)` function. The CLI binary wires it in via `cli.WithCommands()`: + +```go +// go-ai/cmd/daemon/cmd.go +package daemon + +import "forge.lthn.ai/core/go/pkg/cli" + +// AddDaemonCommand adds the 'daemon' command group to the root. +func AddDaemonCommand(root *cli.Command) { + daemonCmd := cli.NewGroup("daemon", "Manage the core daemon", "") + root.AddCommand(daemonCmd) + // subcommands... +} +``` + +No `init()`. No blank imports. No `cli.RegisterCommands()`. + +### How it works + +`cli.WithCommands(name, fn)` wraps the registration function as a framework service implementing `Startable`. During `Core.ServiceStartup()`, the service's `OnStartup()` casts `Core.App` to `*cobra.Command` and calls the registration function. Core services (i18n, log, workspace) start first since they're registered before command services. + +```go +// cli/main.go +func main() { + cli.Main( + cli.WithCommands("config", config.AddConfigCommands), + cli.WithCommands("doctor", doctor.AddDoctorCommands), + // ... + ) +} +``` + +### Migration status (completed) + +| Source | Destination | Status | +|--------|-------------|--------| +| `cmd/dev, setup, qa, docs, gitcmd, monitor` | `go-devops/cmd/` | Done | +| `cmd/lab` | `go-ai/cmd/` | Done | +| `cmd/workspace` | `go-agentic/cmd/` | Done | +| `cmd/go` | `core/go/cmd/gocmd` | Done | +| `cmd/vanity-import, community` | `go-devops/cmd/` | Done | +| `cmd/updater` | `go-update` | Done (own repo) | +| `cmd/daemon, mcpcmd, security` | `go-ai/cmd/` | Done | +| `cmd/crypt` | `go-crypt/cmd/` | Done | +| `cmd/rag` | `go-rag/cmd/` | Done | +| `cmd/unifi` | `go-netops/cmd/` | Done | +| `cmd/api` | `go-api/cmd/` | Done | +| `cmd/collect, forge, gitea` | `go-scm/cmd/` | Done | +| `cmd/deploy, prod, vm` | `go-devops/cmd/` | Done | + +### Stays in cli/ (meta/framework commands) + +`config`, `doctor`, `help`, `module`, `pkgcmd`, `plugin`, `session` + +--- + +## 3. Variant Binaries (future) + +The cli/ repo can produce variant binaries by creating multiple `main.go` files that wire different sets of commands. + +``` +cli/ +├── main.go # Current — meta commands only +├── cmd/core-full/main.go # Full CLI — all ecosystem commands +├── cmd/core-ci/main.go # CI agent dispatch + SCM +├── cmd/core-mlx/main.go # ML inference subprocess +└── cmd/core-ops/main.go # DevOps + infra management +``` + +Each variant calls `cli.Main()` with its specific `cli.WithCommands()` set. No blank imports needed. + +### Why variants matter + +- `core-mlx` ships to the homelab as a ~10MB binary, not 50MB with devops/forge/netops +- `core-ci` deploys to agent machines without ML or CGO dependencies +- Adding a new variant = one new `main.go` with the right `WithCommands` calls + +--- + +## 4. Current State + +cli/ has 7 meta packages, one `main.go`, and zero business logic. Everything else lives in the domain repos that own it. Total cli/ LOC is ~2K. diff --git a/docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md b/docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md new file mode 100644 index 0000000..c2efef1 --- /dev/null +++ b/docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md @@ -0,0 +1,1724 @@ +# CLI SDK Expansion (Phase 0) Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Extend `go/pkg/cli` with charmbracelet TUI primitives (Spinner, ProgressBar, List, TextInput, Viewport) so domain repos never import anything but `forge.lthn.ai/core/go/pkg/cli` for CLI concerns. + +**Architecture:** Each TUI primitive gets its own file in `pkg/cli/`. Charmbracelet libraries (bubbletea, bubbles, lipgloss) are imported only inside `pkg/cli/` — the public API uses our own types. Stubs for future features (Form, FilePicker, Tabs) define the interface but fall back to simple bufio implementations until charm backends are wired in later. + +**Tech Stack:** `charmbracelet/bubbletea` (app loop), `charmbracelet/bubbles` (spinner, progress, list, textinput, viewport), `charmbracelet/lipgloss` (styling — replaces our ANSI builder long-term) + +--- + +## Context + +`go/pkg/cli` currently provides: +- Cobra wrappers: `Command`, `NewCommand()`, `NewGroup()`, flag helpers +- Output: `Success()`, `Error()`, `Table`, `Section()`, `Label()` +- Prompts: `Confirm()`, `Question()`, `Choose()`, `ChooseMulti()` (all bufio-based) +- Styles: `AnsiStyle` builder with 17 pre-built styles, 47 Tailwind colour constants +- Glyphs: `:check:`, `:cross:` etc. with theme switching +- Layout: HLCRF composite renderer + +Zero charmbracelet dependencies exist today. All styling is pure ANSI escape codes. + +The 34 files in `cli/cmd/*` that import `github.com/spf13/cobra` directly need `cli.*` equivalents. This plan does NOT migrate those files — it builds the SDK surface they'll need. Migration happens in Phase 1+. + +## Critical Files + +All changes are in `/Users/snider/Code/host-uk/core/pkg/cli/`: + +- `spinner.go` + `spinner_test.go` — Async spinner +- `progress.go` + `progress_test.go` — Progress bar +- `list.go` + `list_test.go` — Interactive scrollable list +- `textinput.go` + `textinput_test.go` — Styled text input +- `viewport.go` + `viewport_test.go` — Scrollable content pane +- `tui.go` + `tui_test.go` — RunTUI escape hatch + Model interface +- `stubs.go` + `stubs_test.go` — Form, FilePicker, Tabs interfaces (simple fallback) + +--- + +### Task 1: Add charmbracelet dependencies + +**Files:** +- Modify: `/Users/snider/Code/host-uk/core/go.mod` + +**Step 1: Add bubbletea, bubbles, and lipgloss** + +Run: +```bash +cd /Users/snider/Code/host-uk/core && go get github.com/charmbracelet/bubbletea/v2@latest github.com/charmbracelet/bubbles/v2@latest github.com/charmbracelet/lipgloss/v2@latest +``` + +**Step 2: Verify module resolves** + +Run: `cd /Users/snider/Code/host-uk/core && go mod tidy` +Expected: Clean, no errors. + +**Step 3: Verify existing tests still pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test ./pkg/cli/...` +Expected: All existing tests pass (no behaviour changed). + +**Step 4: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add go.mod go.sum && git commit -m "chore(cli): add charmbracelet dependencies (bubbletea, bubbles, lipgloss) + +Co-Authored-By: Virgil " +``` + +--- + +### Task 2: Spinner + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/spinner.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/spinner_test.go` + +A non-blocking spinner that runs in a goroutine. The caller gets a handle to update the message, mark it done, or mark it failed. Uses `bubbles/spinner` internally. + +**Step 1: Write the tests** + +```go +// spinner_test.go +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestSpinner_Good_CreateAndStop(t *testing.T) { + s := NewSpinner("Loading...") + require.NotNil(t, s) + assert.Equal(t, "Loading...", s.Message()) + s.Stop() +} + +func TestSpinner_Good_UpdateMessage(t *testing.T) { + s := NewSpinner("Step 1") + s.Update("Step 2") + assert.Equal(t, "Step 2", s.Message()) + s.Stop() +} + +func TestSpinner_Good_Done(t *testing.T) { + s := NewSpinner("Building") + s.Done("Build complete") + // After Done, spinner is stopped — calling Stop again is safe + s.Stop() +} + +func TestSpinner_Good_Fail(t *testing.T) { + s := NewSpinner("Checking") + s.Fail("Check failed") + s.Stop() +} + +func TestSpinner_Good_DoubleStop(t *testing.T) { + s := NewSpinner("Loading") + s.Stop() + s.Stop() // Should not panic +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestSpinner ./pkg/cli/...` +Expected: FAIL — `NewSpinner` undefined. + +**Step 3: Write the implementation** + +```go +// spinner.go +package cli + +import ( + "fmt" + "sync" + "time" +) + +// SpinnerHandle controls a running spinner. +type SpinnerHandle struct { + mu sync.Mutex + message string + done bool + ticker *time.Ticker + stopCh chan struct{} +} + +// NewSpinner starts an async spinner with the given message. +// Call Stop(), Done(), or Fail() to stop it. +func NewSpinner(message string) *SpinnerHandle { + s := &SpinnerHandle{ + message: message, + ticker: time.NewTicker(100 * time.Millisecond), + stopCh: make(chan struct{}), + } + + frames := []string{"⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"} + if !ColorEnabled() { + frames = []string{"|", "/", "-", "\\"} + } + + go func() { + i := 0 + for { + select { + case <-s.stopCh: + return + case <-s.ticker.C: + s.mu.Lock() + if !s.done { + fmt.Printf("\033[2K\r%s %s", DimStyle.Render(frames[i%len(frames)]), s.message) + } + s.mu.Unlock() + i++ + } + } + }() + + return s +} + +// Message returns the current spinner message. +func (s *SpinnerHandle) Message() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.message +} + +// Update changes the spinner message. +func (s *SpinnerHandle) Update(message string) { + s.mu.Lock() + defer s.mu.Unlock() + s.message = message +} + +// Stop stops the spinner silently (clears the line). +func (s *SpinnerHandle) Stop() { + s.mu.Lock() + defer s.mu.Unlock() + if s.done { + return + } + s.done = true + s.ticker.Stop() + close(s.stopCh) + fmt.Print("\033[2K\r") +} + +// Done stops the spinner with a success message. +func (s *SpinnerHandle) Done(message string) { + s.mu.Lock() + alreadyDone := s.done + s.done = true + s.mu.Unlock() + + if alreadyDone { + return + } + s.ticker.Stop() + close(s.stopCh) + fmt.Printf("\033[2K\r%s\n", SuccessStyle.Render(Glyph(":check:")+" "+message)) +} + +// Fail stops the spinner with an error message. +func (s *SpinnerHandle) Fail(message string) { + s.mu.Lock() + alreadyDone := s.done + s.done = true + s.mu.Unlock() + + if alreadyDone { + return + } + s.ticker.Stop() + close(s.stopCh) + fmt.Printf("\033[2K\r%s\n", ErrorStyle.Render(Glyph(":cross:")+" "+message)) +} +``` + +Note: This initial implementation uses a goroutine + ticker rather than bubbletea, keeping it simple and non-blocking. The bubbletea spinner can replace the internals later without changing the public API. + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestSpinner ./pkg/cli/... -v` +Expected: All 5 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/spinner.go pkg/cli/spinner_test.go && git commit -m "feat(cli): add Spinner with async handle (Update, Done, Fail) + +Co-Authored-By: Virgil " +``` + +--- + +### Task 3: ProgressBar + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/progressbar.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/progressbar_test.go` + +A progress bar that renders inline. Shows percentage, bar, and optional message. + +**Step 1: Write the tests** + +```go +// progressbar_test.go +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestProgressBar_Good_Create(t *testing.T) { + pb := NewProgressBar(100) + require.NotNil(t, pb) + assert.Equal(t, 0, pb.Current()) + assert.Equal(t, 100, pb.Total()) +} + +func TestProgressBar_Good_Increment(t *testing.T) { + pb := NewProgressBar(10) + pb.Increment() + assert.Equal(t, 1, pb.Current()) + pb.Increment() + assert.Equal(t, 2, pb.Current()) +} + +func TestProgressBar_Good_SetMessage(t *testing.T) { + pb := NewProgressBar(10) + pb.SetMessage("Processing file.go") + assert.Equal(t, "Processing file.go", pb.message) +} + +func TestProgressBar_Good_Set(t *testing.T) { + pb := NewProgressBar(100) + pb.Set(50) + assert.Equal(t, 50, pb.Current()) +} + +func TestProgressBar_Good_Done(t *testing.T) { + pb := NewProgressBar(5) + for i := 0; i < 5; i++ { + pb.Increment() + } + pb.Done() + // After Done, Current == Total + assert.Equal(t, 5, pb.Current()) +} + +func TestProgressBar_Bad_ExceedsTotal(t *testing.T) { + pb := NewProgressBar(2) + pb.Increment() + pb.Increment() + pb.Increment() // Should clamp to total + assert.Equal(t, 2, pb.Current()) +} + +func TestProgressBar_Good_Render(t *testing.T) { + pb := NewProgressBar(10) + pb.Set(5) + rendered := pb.String() + assert.Contains(t, rendered, "50%") +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestProgressBar ./pkg/cli/... -v` +Expected: FAIL — `NewProgressBar` undefined. + +**Step 3: Write the implementation** + +```go +// progressbar.go +package cli + +import ( + "fmt" + "strings" + "sync" +) + +// ProgressHandle controls a progress bar. +type ProgressHandle struct { + mu sync.Mutex + current int + total int + message string + width int +} + +// NewProgressBar creates a new progress bar with the given total. +func NewProgressBar(total int) *ProgressHandle { + return &ProgressHandle{ + total: total, + width: 30, + } +} + +// Current returns the current progress value. +func (p *ProgressHandle) Current() int { + p.mu.Lock() + defer p.mu.Unlock() + return p.current +} + +// Total returns the total value. +func (p *ProgressHandle) Total() int { + return p.total +} + +// Increment advances the progress by 1. +func (p *ProgressHandle) Increment() { + p.mu.Lock() + defer p.mu.Unlock() + if p.current < p.total { + p.current++ + } + p.render() +} + +// Set sets the progress to a specific value. +func (p *ProgressHandle) Set(n int) { + p.mu.Lock() + defer p.mu.Unlock() + if n > p.total { + n = p.total + } + if n < 0 { + n = 0 + } + p.current = n + p.render() +} + +// SetMessage sets the message displayed alongside the bar. +func (p *ProgressHandle) SetMessage(msg string) { + p.mu.Lock() + defer p.mu.Unlock() + p.message = msg + p.render() +} + +// Done completes the progress bar and moves to a new line. +func (p *ProgressHandle) Done() { + p.mu.Lock() + defer p.mu.Unlock() + p.current = p.total + p.render() + fmt.Println() +} + +// String returns the rendered progress bar without ANSI cursor control. +func (p *ProgressHandle) String() string { + pct := 0 + if p.total > 0 { + pct = (p.current * 100) / p.total + } + + filled := (p.width * p.current) / p.total + if filled > p.width { + filled = p.width + } + empty := p.width - filled + + bar := "[" + strings.Repeat("█", filled) + strings.Repeat("░", empty) + "]" + + if p.message != "" { + return fmt.Sprintf("%s %3d%% %s", bar, pct, p.message) + } + return fmt.Sprintf("%s %3d%%", bar, pct) +} + +// render outputs the progress bar, overwriting the current line. +func (p *ProgressHandle) render() { + fmt.Printf("\033[2K\r%s", p.String()) +} +``` + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestProgressBar ./pkg/cli/... -v` +Expected: All 7 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/progressbar.go pkg/cli/progressbar_test.go && git commit -m "feat(cli): add ProgressBar with Increment, Set, SetMessage, Done + +Co-Authored-By: Virgil " +``` + +--- + +### Task 4: TUI runner (RunTUI + Model interface) + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/tui.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/tui_test.go` + +The escape hatch for complex interactive UIs. Wraps `bubbletea.Program` behind our own `Model` interface so domain packages never import bubbletea directly. + +**Step 1: Write the tests** + +```go +// tui_test.go +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +// testModel is a minimal Model that quits immediately. +type testModel struct { + initCalled bool + updateCalled bool + viewCalled bool +} + +func (m *testModel) Init() Cmd { + m.initCalled = true + return Quit +} + +func (m *testModel) Update(msg Msg) (Model, Cmd) { + m.updateCalled = true + return m, nil +} + +func (m *testModel) View() string { + m.viewCalled = true + return "test view" +} + +func TestModel_Good_InterfaceSatisfied(t *testing.T) { + var m Model = &testModel{} + assert.NotNil(t, m) +} + +func TestQuitCmd_Good_ReturnsQuitMsg(t *testing.T) { + cmd := Quit + assert.NotNil(t, cmd) +} + +func TestKeyMsg_Good_String(t *testing.T) { + k := KeyMsg{Type: KeyEnter} + assert.Equal(t, KeyEnter, k.Type) +} + +func TestKeyTypes_Good_Constants(t *testing.T) { + // Verify key type constants exist + assert.NotEmpty(t, string(KeyEnter)) + assert.NotEmpty(t, string(KeyEsc)) + assert.NotEmpty(t, string(KeyCtrlC)) + assert.NotEmpty(t, string(KeyUp)) + assert.NotEmpty(t, string(KeyDown)) + assert.NotEmpty(t, string(KeyTab)) + assert.NotEmpty(t, string(KeyBackspace)) +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestModel ./pkg/cli/... -v && go test -run TestQuit ./pkg/cli/... -v && go test -run TestKey ./pkg/cli/... -v` +Expected: FAIL — types undefined. + +**Step 3: Write the implementation** + +```go +// tui.go +package cli + +import ( + tea "github.com/charmbracelet/bubbletea/v2" +) + +// Model is the interface for interactive TUI applications. +// It mirrors bubbletea's Model but uses our own types so domain +// packages never import bubbletea directly. +type Model interface { + // Init returns an initial command to run. + Init() Cmd + + // Update handles a message and returns the updated model and command. + Update(msg Msg) (Model, Cmd) + + // View returns the string representation of the UI. + View() string +} + +// Msg is a message passed to Update. Can be any type. +type Msg = tea.Msg + +// Cmd is a function that returns a message. Nil means no command. +type Cmd = tea.Cmd + +// Quit is a command that tells the TUI to exit. +var Quit = tea.Quit + +// KeyMsg represents a key press event. +type KeyMsg = tea.KeyMsg + +// KeyType represents the type of key pressed. +type KeyType = tea.KeyType + +// Key type constants. +const ( + KeyEnter KeyType = tea.KeyEnter + KeyEsc KeyType = tea.KeyEscape + KeyCtrlC KeyType = tea.KeyCtrlC + KeyUp KeyType = tea.KeyUp + KeyDown KeyType = tea.KeyDown + KeyLeft KeyType = tea.KeyLeft + KeyRight KeyType = tea.KeyRight + KeyTab KeyType = tea.KeyTab + KeyBackspace KeyType = tea.KeyBackspace + KeySpace KeyType = tea.KeySpace + KeyHome KeyType = tea.KeyHome + KeyEnd KeyType = tea.KeyEnd + KeyPgUp KeyType = tea.KeyPgUp + KeyPgDown KeyType = tea.KeyPgDown + KeyDelete KeyType = tea.KeyDelete +) + +// adapter wraps our Model interface into a bubbletea.Model. +type adapter struct { + inner Model +} + +func (a adapter) Init() tea.Cmd { + return a.inner.Init() +} + +func (a adapter) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + m, cmd := a.inner.Update(msg) + return adapter{inner: m}, cmd +} + +func (a adapter) View() string { + return a.inner.View() +} + +// RunTUI runs an interactive TUI application using the provided Model. +// This is the escape hatch for complex interactive UIs that need the +// full bubbletea event loop. For simple spinners, progress bars, and +// lists, use the dedicated helpers instead. +// +// err := cli.RunTUI(&myModel{items: items}) +func RunTUI(m Model) error { + p := tea.NewProgram(adapter{inner: m}) + _, err := p.Run() + return err +} +``` + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run "TestModel|TestQuit|TestKey" ./pkg/cli/... -v` +Expected: All 4 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/tui.go pkg/cli/tui_test.go && git commit -m "feat(cli): add RunTUI escape hatch with Model/Msg/Cmd/KeyMsg types + +Wraps bubbletea behind our own interface so domain packages +never import charmbracelet directly. + +Co-Authored-By: Virgil " +``` + +--- + +### Task 5: Interactive List + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/list.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/list_test.go` + +An interactive scrollable list for terminal selection. Uses our `RunTUI` internally. Falls back to numbered `Select()` when stdin is not a terminal. + +**Step 1: Write the tests** + +```go +// list_test.go +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestListModel_Good_Create(t *testing.T) { + items := []string{"alpha", "beta", "gamma"} + m := newListModel(items, "Pick one:") + assert.Equal(t, 3, len(m.items)) + assert.Equal(t, 0, m.cursor) + assert.Equal(t, "Pick one:", m.title) +} + +func TestListModel_Good_MoveDown(t *testing.T) { + m := newListModel([]string{"a", "b", "c"}, "") + m.moveDown() + assert.Equal(t, 1, m.cursor) + m.moveDown() + assert.Equal(t, 2, m.cursor) +} + +func TestListModel_Good_MoveUp(t *testing.T) { + m := newListModel([]string{"a", "b", "c"}, "") + m.moveDown() + m.moveDown() + m.moveUp() + assert.Equal(t, 1, m.cursor) +} + +func TestListModel_Good_WrapAround(t *testing.T) { + m := newListModel([]string{"a", "b", "c"}, "") + m.moveUp() // Should wrap to bottom + assert.Equal(t, 2, m.cursor) +} + +func TestListModel_Good_View(t *testing.T) { + m := newListModel([]string{"alpha", "beta"}, "Choose:") + view := m.View() + assert.Contains(t, view, "Choose:") + assert.Contains(t, view, "alpha") + assert.Contains(t, view, "beta") +} + +func TestListModel_Good_Selected(t *testing.T) { + m := newListModel([]string{"a", "b", "c"}, "") + m.moveDown() + m.selected = true + assert.Equal(t, "b", m.items[m.cursor]) +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestListModel ./pkg/cli/... -v` +Expected: FAIL — `newListModel` undefined. + +**Step 3: Write the implementation** + +```go +// list.go +package cli + +import ( + "fmt" + "strings" + + tea "github.com/charmbracelet/bubbletea/v2" + "golang.org/x/term" +) + +// listModel is the internal bubbletea model for interactive list selection. +type listModel struct { + items []string + cursor int + title string + selected bool + quitted bool +} + +func newListModel(items []string, title string) *listModel { + return &listModel{ + items: items, + title: title, + } +} + +func (m *listModel) moveDown() { + m.cursor++ + if m.cursor >= len(m.items) { + m.cursor = 0 + } +} + +func (m *listModel) moveUp() { + m.cursor-- + if m.cursor < 0 { + m.cursor = len(m.items) - 1 + } +} + +func (m *listModel) Init() tea.Cmd { + return nil +} + +func (m *listModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + switch msg := msg.(type) { + case tea.KeyMsg: + switch msg.Type { + case tea.KeyUp, tea.KeyShiftTab: + m.moveUp() + case tea.KeyDown, tea.KeyTab: + m.moveDown() + case tea.KeyEnter: + m.selected = true + return m, tea.Quit + case tea.KeyEscape, tea.KeyCtrlC: + m.quitted = true + return m, tea.Quit + default: + // Handle j/k vim-style navigation + if msg.String() == "j" { + m.moveDown() + } else if msg.String() == "k" { + m.moveUp() + } + } + } + return m, nil +} + +func (m *listModel) View() string { + var sb strings.Builder + + if m.title != "" { + sb.WriteString(BoldStyle.Render(m.title) + "\n\n") + } + + for i, item := range m.items { + cursor := " " + style := DimStyle + if i == m.cursor { + cursor = AccentStyle.Render(Glyph(":pointer:")) + " " + style = BoldStyle + } + sb.WriteString(fmt.Sprintf("%s%s\n", cursor, style.Render(item))) + } + + sb.WriteString("\n" + DimStyle.Render("↑/↓ navigate • enter select • esc cancel")) + + return sb.String() +} + +// ListOption configures List behaviour. +type ListOption func(*listConfig) + +type listConfig struct { + height int +} + +// WithHeight sets the visible height of the list (number of items shown). +func WithHeight(n int) ListOption { + return func(c *listConfig) { + c.height = n + } +} + +// InteractiveList presents an interactive scrollable list and returns the +// selected item's index and value. Returns -1 and empty string if cancelled. +// +// Falls back to numbered Select() when stdin is not a terminal (e.g. piped input). +// +// idx, value := cli.InteractiveList("Pick a repo:", repos) +func InteractiveList(title string, items []string, opts ...ListOption) (int, string) { + if len(items) == 0 { + return -1, "" + } + + // Fall back to simple Select if not a terminal + if !term.IsTerminal(int(StdinFd())) { + result, err := Select(title, items) + if err != nil { + return -1, "" + } + for i, item := range items { + if item == result { + return i, result + } + } + return -1, "" + } + + m := newListModel(items, title) + p := tea.NewProgram(m) + finalModel, err := p.Run() + if err != nil { + return -1, "" + } + + final := finalModel.(*listModel) + if final.quitted || !final.selected { + return -1, "" + } + return final.cursor, final.items[final.cursor] +} + +// StdinFd returns the file descriptor for stdin. +// Extracted for testing. +func StdinFd() uintptr { + return uintptr(0) // stdin +} +``` + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestListModel ./pkg/cli/... -v` +Expected: All 6 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/list.go pkg/cli/list_test.go && git commit -m "feat(cli): add InteractiveList with keyboard navigation and terminal fallback + +Co-Authored-By: Virgil " +``` + +--- + +### Task 6: TextInput + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/textinput.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/textinput_test.go` + +A styled single-line text input with placeholder, validation, and optional masking (for passwords). Falls back to `Question()` when stdin is not a terminal. + +**Step 1: Write the tests** + +```go +// textinput_test.go +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestTextInputModel_Good_Create(t *testing.T) { + m := newTextInputModel("Enter name:", "") + assert.Equal(t, "Enter name:", m.title) + assert.Equal(t, "", m.value) +} + +func TestTextInputModel_Good_WithPlaceholder(t *testing.T) { + m := newTextInputModel("Name:", "John") + assert.Equal(t, "John", m.placeholder) +} + +func TestTextInputModel_Good_TypeCharacters(t *testing.T) { + m := newTextInputModel("Name:", "") + m.insertChar('H') + m.insertChar('i') + assert.Equal(t, "Hi", m.value) +} + +func TestTextInputModel_Good_Backspace(t *testing.T) { + m := newTextInputModel("Name:", "") + m.insertChar('A') + m.insertChar('B') + m.backspace() + assert.Equal(t, "A", m.value) +} + +func TestTextInputModel_Good_BackspaceEmpty(t *testing.T) { + m := newTextInputModel("Name:", "") + m.backspace() // Should not panic + assert.Equal(t, "", m.value) +} + +func TestTextInputModel_Good_Masked(t *testing.T) { + m := newTextInputModel("Password:", "") + m.masked = true + m.insertChar('s') + m.insertChar('e') + m.insertChar('c') + assert.Equal(t, "sec", m.value) // Internal value is real + view := m.View() + assert.NotContains(t, view, "sec") // Display is masked + assert.Contains(t, view, "***") +} + +func TestTextInputModel_Good_View(t *testing.T) { + m := newTextInputModel("Enter:", "") + m.insertChar('X') + view := m.View() + assert.Contains(t, view, "Enter:") + assert.Contains(t, view, "X") +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestTextInputModel ./pkg/cli/... -v` +Expected: FAIL — `newTextInputModel` undefined. + +**Step 3: Write the implementation** + +```go +// textinput.go +package cli + +import ( + "fmt" + "strings" + + tea "github.com/charmbracelet/bubbletea/v2" + "golang.org/x/term" +) + +// textInputModel is the internal bubbletea model for text input. +type textInputModel struct { + title string + placeholder string + value string + masked bool + submitted bool + cancelled bool + cursorPos int + validator func(string) error + err error +} + +func newTextInputModel(title, placeholder string) *textInputModel { + return &textInputModel{ + title: title, + placeholder: placeholder, + } +} + +func (m *textInputModel) insertChar(ch rune) { + m.value = m.value[:m.cursorPos] + string(ch) + m.value[m.cursorPos:] + m.cursorPos++ +} + +func (m *textInputModel) backspace() { + if m.cursorPos > 0 { + m.value = m.value[:m.cursorPos-1] + m.value[m.cursorPos:] + m.cursorPos-- + } +} + +func (m *textInputModel) Init() tea.Cmd { + return nil +} + +func (m *textInputModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + switch msg := msg.(type) { + case tea.KeyMsg: + switch msg.Type { + case tea.KeyEnter: + if m.validator != nil { + if err := m.validator(m.value); err != nil { + m.err = err + return m, nil + } + } + if m.value == "" && m.placeholder != "" { + m.value = m.placeholder + } + m.submitted = true + return m, tea.Quit + case tea.KeyEscape, tea.KeyCtrlC: + m.cancelled = true + return m, tea.Quit + case tea.KeyBackspace: + m.backspace() + m.err = nil + case tea.KeyLeft: + if m.cursorPos > 0 { + m.cursorPos-- + } + case tea.KeyRight: + if m.cursorPos < len(m.value) { + m.cursorPos++ + } + default: + if msg.Text != "" { + for _, ch := range msg.Text { + m.insertChar(ch) + } + m.err = nil + } + } + } + return m, nil +} + +func (m *textInputModel) View() string { + var sb strings.Builder + + sb.WriteString(BoldStyle.Render(m.title) + "\n\n") + + display := m.value + if m.masked { + display = strings.Repeat("*", len(m.value)) + } + + if display == "" && m.placeholder != "" { + sb.WriteString(DimStyle.Render(m.placeholder)) + } else { + sb.WriteString(display) + } + sb.WriteString(AccentStyle.Render("█")) // Cursor + + if m.err != nil { + sb.WriteString("\n" + ErrorStyle.Render(fmt.Sprintf(" %s", m.err))) + } + + sb.WriteString("\n\n" + DimStyle.Render("enter submit • esc cancel")) + + return sb.String() +} + +// TextInputOption configures TextInput behaviour. +type TextInputOption func(*textInputConfig) + +type textInputConfig struct { + placeholder string + masked bool + validator func(string) error +} + +// WithPlaceholder sets placeholder text shown when input is empty. +func WithPlaceholder(text string) TextInputOption { + return func(c *textInputConfig) { + c.placeholder = text + } +} + +// WithMask hides input characters (for passwords). +func WithMask() TextInputOption { + return func(c *textInputConfig) { + c.masked = true + } +} + +// WithInputValidator adds a validation function for the input. +func WithInputValidator(fn func(string) error) TextInputOption { + return func(c *textInputConfig) { + c.validator = fn + } +} + +// TextInput presents a styled text input prompt and returns the entered value. +// Returns empty string if cancelled. +// +// Falls back to Question() when stdin is not a terminal. +// +// name, err := cli.TextInput("Enter your name:", WithPlaceholder("Anonymous")) +// pass, err := cli.TextInput("Password:", WithMask()) +func TextInput(title string, opts ...TextInputOption) (string, error) { + cfg := &textInputConfig{} + for _, opt := range opts { + opt(cfg) + } + + // Fall back to simple Question if not a terminal + if !term.IsTerminal(int(StdinFd())) { + var qopts []QuestionOption + if cfg.placeholder != "" { + qopts = append(qopts, WithDefault(cfg.placeholder)) + } + if cfg.validator != nil { + qopts = append(qopts, WithValidator(cfg.validator)) + } + return Question(title, qopts...), nil + } + + m := newTextInputModel(title, cfg.placeholder) + m.masked = cfg.masked + m.validator = cfg.validator + + p := tea.NewProgram(m) + finalModel, err := p.Run() + if err != nil { + return "", err + } + + final := finalModel.(*textInputModel) + if final.cancelled { + return "", nil + } + return final.value, nil +} +``` + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestTextInputModel ./pkg/cli/... -v` +Expected: All 7 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/textinput.go pkg/cli/textinput_test.go && git commit -m "feat(cli): add TextInput with placeholder, masking, validation + +Co-Authored-By: Virgil " +``` + +--- + +### Task 7: Viewport (scrollable content) + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/viewport.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/viewport_test.go` + +A scrollable content pane for displaying long output (logs, diffs, docs). Uses bubbletea internally. + +**Step 1: Write the tests** + +```go +// viewport_test.go +package cli + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestViewportModel_Good_Create(t *testing.T) { + content := "line 1\nline 2\nline 3" + m := newViewportModel(content, "Title", 5) + assert.Equal(t, "Title", m.title) + assert.Equal(t, 3, len(m.lines)) + assert.Equal(t, 0, m.offset) +} + +func TestViewportModel_Good_ScrollDown(t *testing.T) { + lines := make([]string, 20) + for i := range lines { + lines[i] = strings.Repeat("x", 10) + } + m := newViewportModel(strings.Join(lines, "\n"), "", 5) + m.scrollDown() + assert.Equal(t, 1, m.offset) +} + +func TestViewportModel_Good_ScrollUp(t *testing.T) { + lines := make([]string, 20) + for i := range lines { + lines[i] = strings.Repeat("x", 10) + } + m := newViewportModel(strings.Join(lines, "\n"), "", 5) + m.scrollDown() + m.scrollDown() + m.scrollUp() + assert.Equal(t, 1, m.offset) +} + +func TestViewportModel_Good_NoScrollPastTop(t *testing.T) { + m := newViewportModel("a\nb\nc", "", 5) + m.scrollUp() // Already at top + assert.Equal(t, 0, m.offset) +} + +func TestViewportModel_Good_NoScrollPastBottom(t *testing.T) { + m := newViewportModel("a\nb\nc", "", 5) + for i := 0; i < 10; i++ { + m.scrollDown() + } + // Should clamp — can't scroll past content + assert.GreaterOrEqual(t, m.offset, 0) +} + +func TestViewportModel_Good_View(t *testing.T) { + m := newViewportModel("line 1\nline 2", "My Title", 10) + view := m.View() + assert.Contains(t, view, "My Title") + assert.Contains(t, view, "line 1") + assert.Contains(t, view, "line 2") +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestViewportModel ./pkg/cli/... -v` +Expected: FAIL — `newViewportModel` undefined. + +**Step 3: Write the implementation** + +```go +// viewport.go +package cli + +import ( + "fmt" + "strings" + + tea "github.com/charmbracelet/bubbletea/v2" + "golang.org/x/term" +) + +// viewportModel is the internal bubbletea model for scrollable content. +type viewportModel struct { + title string + lines []string + offset int + height int + quitted bool +} + +func newViewportModel(content, title string, height int) *viewportModel { + lines := strings.Split(content, "\n") + return &viewportModel{ + title: title, + lines: lines, + height: height, + } +} + +func (m *viewportModel) scrollDown() { + maxOffset := len(m.lines) - m.height + if maxOffset < 0 { + maxOffset = 0 + } + if m.offset < maxOffset { + m.offset++ + } +} + +func (m *viewportModel) scrollUp() { + if m.offset > 0 { + m.offset-- + } +} + +func (m *viewportModel) Init() tea.Cmd { + return nil +} + +func (m *viewportModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + switch msg := msg.(type) { + case tea.KeyMsg: + switch msg.Type { + case tea.KeyUp: + m.scrollUp() + case tea.KeyDown: + m.scrollDown() + case tea.KeyPgUp: + for i := 0; i < m.height; i++ { + m.scrollUp() + } + case tea.KeyPgDown: + for i := 0; i < m.height; i++ { + m.scrollDown() + } + case tea.KeyHome: + m.offset = 0 + case tea.KeyEnd: + maxOffset := len(m.lines) - m.height + if maxOffset < 0 { + maxOffset = 0 + } + m.offset = maxOffset + case tea.KeyEscape, tea.KeyCtrlC: + m.quitted = true + return m, tea.Quit + default: + switch msg.String() { + case "q": + m.quitted = true + return m, tea.Quit + case "j": + m.scrollDown() + case "k": + m.scrollUp() + case "g": + m.offset = 0 + case "G": + maxOffset := len(m.lines) - m.height + if maxOffset < 0 { + maxOffset = 0 + } + m.offset = maxOffset + } + } + } + return m, nil +} + +func (m *viewportModel) View() string { + var sb strings.Builder + + if m.title != "" { + sb.WriteString(BoldStyle.Render(m.title) + "\n") + sb.WriteString(DimStyle.Render(strings.Repeat("─", len(m.title))) + "\n") + } + + // Visible window + end := m.offset + m.height + if end > len(m.lines) { + end = len(m.lines) + } + for _, line := range m.lines[m.offset:end] { + sb.WriteString(line + "\n") + } + + // Scroll indicator + total := len(m.lines) + if total > m.height { + pct := (m.offset * 100) / (total - m.height) + sb.WriteString(DimStyle.Render(fmt.Sprintf("\n%d%% (%d/%d lines)", pct, m.offset+m.height, total))) + } + + sb.WriteString("\n" + DimStyle.Render("↑/↓ scroll • PgUp/PgDn page • q quit")) + + return sb.String() +} + +// ViewportOption configures Viewport behaviour. +type ViewportOption func(*viewportConfig) + +type viewportConfig struct { + title string + height int +} + +// WithViewportTitle sets the title shown above the viewport. +func WithViewportTitle(title string) ViewportOption { + return func(c *viewportConfig) { + c.title = title + } +} + +// WithViewportHeight sets the visible height in lines. +func WithViewportHeight(n int) ViewportOption { + return func(c *viewportConfig) { + c.height = n + } +} + +// Viewport displays scrollable content in the terminal. +// Falls back to printing the full content when stdin is not a terminal. +// +// cli.Viewport(longContent, WithViewportTitle("Build Log"), WithViewportHeight(20)) +func Viewport(content string, opts ...ViewportOption) error { + cfg := &viewportConfig{ + height: 20, + } + for _, opt := range opts { + opt(cfg) + } + + // Fall back to plain output if not a terminal + if !term.IsTerminal(int(StdinFd())) { + if cfg.title != "" { + fmt.Println(BoldStyle.Render(cfg.title)) + fmt.Println(DimStyle.Render(strings.Repeat("─", len(cfg.title)))) + } + fmt.Println(content) + return nil + } + + m := newViewportModel(content, cfg.title, cfg.height) + p := tea.NewProgram(m) + _, err := p.Run() + return err +} +``` + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run TestViewportModel ./pkg/cli/... -v` +Expected: All 6 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/viewport.go pkg/cli/viewport_test.go && git commit -m "feat(cli): add Viewport for scrollable content (logs, diffs, docs) + +Co-Authored-By: Virgil " +``` + +--- + +### Task 8: Future stubs (Form, FilePicker, Tabs) + +**Files:** +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/stubs.go` +- Create: `/Users/snider/Code/host-uk/core/pkg/cli/stubs_test.go` + +Interface definitions for features we'll build later. Simple fallback implementations so the API is usable today. + +**Step 1: Write the tests** + +```go +// stubs_test.go +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestFormField_Good_Types(t *testing.T) { + fields := []FormField{ + {Label: "Name", Key: "name", Type: FieldText}, + {Label: "Password", Key: "pass", Type: FieldPassword}, + {Label: "Accept", Key: "ok", Type: FieldConfirm}, + } + assert.Equal(t, 3, len(fields)) + assert.Equal(t, FieldText, fields[0].Type) + assert.Equal(t, FieldPassword, fields[1].Type) + assert.Equal(t, FieldConfirm, fields[2].Type) +} + +func TestFieldType_Good_Constants(t *testing.T) { + assert.Equal(t, FieldType("text"), FieldText) + assert.Equal(t, FieldType("password"), FieldPassword) + assert.Equal(t, FieldType("confirm"), FieldConfirm) + assert.Equal(t, FieldType("select"), FieldSelect) +} + +func TestTabItem_Good_Structure(t *testing.T) { + tabs := []TabItem{ + {Title: "Overview", Content: "overview content"}, + {Title: "Details", Content: "detail content"}, + } + assert.Equal(t, 2, len(tabs)) + assert.Equal(t, "Overview", tabs[0].Title) +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run "TestFormField|TestFieldType|TestTabItem" ./pkg/cli/... -v` +Expected: FAIL — types undefined. + +**Step 3: Write the implementation** + +```go +// stubs.go +package cli + +// ────────────────────────────────────────────────────────────────────────────── +// Form (stubbed — simple fallback, will use charmbracelet/huh later) +// ────────────────────────────────────────────────────────────────────────────── + +// FieldType defines the type of a form field. +type FieldType string + +const ( + FieldText FieldType = "text" + FieldPassword FieldType = "password" + FieldConfirm FieldType = "confirm" + FieldSelect FieldType = "select" +) + +// FormField describes a single field in a form. +type FormField struct { + Label string + Key string + Type FieldType + Default string + Placeholder string + Options []string // For FieldSelect + Required bool + Validator func(string) error +} + +// Form presents a multi-field form and returns the values keyed by FormField.Key. +// Currently falls back to sequential Question()/Confirm()/Select() calls. +// Will be replaced with charmbracelet/huh interactive form later. +// +// results, err := cli.Form([]cli.FormField{ +// {Label: "Name", Key: "name", Type: cli.FieldText, Required: true}, +// {Label: "Password", Key: "pass", Type: cli.FieldPassword}, +// {Label: "Accept terms?", Key: "terms", Type: cli.FieldConfirm}, +// }) +func Form(fields []FormField) (map[string]string, error) { + results := make(map[string]string, len(fields)) + + for _, f := range fields { + switch f.Type { + case FieldPassword: + val := Question(f.Label+":", WithDefault(f.Default)) + results[f.Key] = val + case FieldConfirm: + if Confirm(f.Label) { + results[f.Key] = "true" + } else { + results[f.Key] = "false" + } + case FieldSelect: + val, err := Select(f.Label, f.Options) + if err != nil { + return nil, err + } + results[f.Key] = val + default: // FieldText + var opts []QuestionOption + if f.Default != "" { + opts = append(opts, WithDefault(f.Default)) + } + if f.Required { + opts = append(opts, RequiredInput()) + } + if f.Validator != nil { + opts = append(opts, WithValidator(f.Validator)) + } + results[f.Key] = Question(f.Label+":", opts...) + } + } + + return results, nil +} + +// ────────────────────────────────────────────────────────────────────────────── +// FilePicker (stubbed — will use charmbracelet/filepicker later) +// ────────────────────────────────────────────────────────────────────────────── + +// FilePickerOption configures FilePicker behaviour. +type FilePickerOption func(*filePickerConfig) + +type filePickerConfig struct { + dir string + extensions []string +} + +// InDirectory sets the starting directory for the file picker. +func InDirectory(dir string) FilePickerOption { + return func(c *filePickerConfig) { + c.dir = dir + } +} + +// WithExtensions filters to specific file extensions (e.g. ".go", ".yaml"). +func WithExtensions(exts ...string) FilePickerOption { + return func(c *filePickerConfig) { + c.extensions = exts + } +} + +// FilePicker presents a file browser and returns the selected path. +// Currently falls back to a text prompt. Will be replaced with an +// interactive file browser later. +// +// path, err := cli.FilePicker(cli.InDirectory("."), cli.WithExtensions(".go")) +func FilePicker(opts ...FilePickerOption) (string, error) { + cfg := &filePickerConfig{dir: "."} + for _, opt := range opts { + opt(cfg) + } + + hint := "File path" + if cfg.dir != "." { + hint += " (from " + cfg.dir + ")" + } + return Question(hint + ":"), nil +} + +// ────────────────────────────────────────────────────────────────────────────── +// Tabs (stubbed — will use bubbletea model later) +// ────────────────────────────────────────────────────────────────────────────── + +// TabItem describes a tab with a title and content. +type TabItem struct { + Title string + Content string +} + +// Tabs displays tabbed content. Currently prints all tabs sequentially. +// Will be replaced with an interactive tab switcher later. +// +// cli.Tabs([]cli.TabItem{ +// {Title: "Overview", Content: summaryText}, +// {Title: "Details", Content: detailText}, +// }) +func Tabs(items []TabItem) error { + for i, tab := range items { + if i > 0 { + Blank() + } + Section(tab.Title) + Println("%s", tab.Content) + } + return nil +} +``` + +**Step 4: Run tests to verify they pass** + +Run: `cd /Users/snider/Code/host-uk/core && go test -run "TestFormField|TestFieldType|TestTabItem" ./pkg/cli/... -v` +Expected: All 3 tests PASS. + +**Step 5: Commit** + +```bash +cd /Users/snider/Code/host-uk/core && git add pkg/cli/stubs.go pkg/cli/stubs_test.go && git commit -m "feat(cli): stub Form, FilePicker, Tabs with simple fallbacks + +Interfaces defined for future charmbracelet/huh upgrade. +Current implementations use sequential prompts. + +Co-Authored-By: Virgil " +``` + +--- + +### Task 9: Run full test suite and verify + +**Step 1: Run all cli package tests** + +Run: `cd /Users/snider/Code/host-uk/core && go test ./pkg/cli/... -v -count=1` +Expected: All tests pass (existing + new). + +**Step 2: Run full module tests** + +Run: `cd /Users/snider/Code/host-uk/core && go test ./... 2>&1 | tail -30` +Expected: No regressions. + +**Step 3: Verify no charmbracelet imports leaked outside pkg/cli** + +Run: `cd /Users/snider/Code/host-uk/core && grep -r "charmbracelet" --include="*.go" . | grep -v pkg/cli/ | grep -v vendor/` +Expected: No output (charmbracelet only imported inside pkg/cli/). + +--- + +## Verification + +After all tasks: + +1. `go test ./pkg/cli/... -v` — all pass (existing + ~34 new tests) +2. `go test ./...` — no regressions across the module +3. `grep -r "charmbracelet" --include="*.go" . | grep -v pkg/cli/` — empty (no leaks) +4. New public API surface: + - `NewSpinner(msg)` → `*SpinnerHandle` (Update, Done, Fail, Stop) + - `NewProgressBar(total)` → `*ProgressHandle` (Increment, Set, SetMessage, Done) + - `InteractiveList(title, items)` → `(int, string)` + - `TextInput(title, opts...)` → `(string, error)` + - `Viewport(content, opts...)` → `error` + - `RunTUI(model)` → `error` (escape hatch) + - `Form(fields)` → `(map[string]string, error)` (stub) + - `FilePicker(opts...)` → `(string, error)` (stub) + - `Tabs(items)` → `error` (stub) + - `Model`, `Msg`, `Cmd`, `KeyMsg`, `KeyType` + key constants + +## Dependency Sequencing + +``` +Task 1 (add deps) ← Task 2 (Spinner) +Task 1 ← Task 3 (ProgressBar) +Task 1 ← Task 4 (TUI runner) ← Task 5 (List) +Task 4 ← Task 6 (TextInput) +Task 4 ← Task 7 (Viewport) +Task 1 ← Task 8 (Stubs) +Tasks 2-8 ← Task 9 (Verification) +``` + +Tasks 2, 3, and 8 are independent of each other (can run in parallel after Task 1). Tasks 5, 6, 7 depend on Task 4 (RunTUI) but are independent of each other. diff --git a/docs/plans/completed/2026-02-21-go-forge-design.md b/docs/plans/completed/2026-02-21-go-forge-design.md new file mode 100644 index 0000000..b718629 --- /dev/null +++ b/docs/plans/completed/2026-02-21-go-forge-design.md @@ -0,0 +1,286 @@ +# go-forge Design Document + +## Overview + +**go-forge** is a full-coverage Go client for the Forgejo API (450 endpoints, 284 paths, 229 types). It uses a generic `Resource[T, C, U]` pattern for CRUD operations (91% of endpoints) and hand-written methods for 39 unique action endpoints. Types are generated from Forgejo's `swagger.v1.json` spec. + +**Module path:** `forge.lthn.ai/core/go-forge` + +**Origin:** Extracted from `go-scm/forge/` (45 methods covering 10% of API), expanded to full coverage. + +## Architecture + +``` +forge.lthn.ai/core/go-forge +├── client.go # HTTP client: auth, headers, rate limiting, context.Context +├── pagination.go # Generic paginated request helper +├── resource.go # Resource[T, C, U] generic CRUD (List/Get/Create/Update/Delete) +├── errors.go # Typed error handling (APIError, NotFound, Forbidden, etc.) +├── forge.go # Top-level Forge client aggregating all services +│ +├── types/ # Generated from swagger.v1.json +│ ├── generate.go # //go:generate directive +│ ├── repo.go # Repository, CreateRepoOption, EditRepoOption +│ ├── issue.go # Issue, CreateIssueOption, EditIssueOption +│ ├── pr.go # PullRequest, CreatePullRequestOption +│ ├── user.go # User, CreateUserOption +│ ├── org.go # Organisation, CreateOrgOption +│ ├── team.go # Team, CreateTeamOption +│ ├── label.go # Label, CreateLabelOption +│ ├── release.go # Release, CreateReleaseOption +│ ├── branch.go # Branch, BranchProtection +│ ├── milestone.go # Milestone, CreateMilestoneOption +│ ├── hook.go # Hook, CreateHookOption +│ ├── key.go # DeployKey, PublicKey, GPGKey +│ ├── notification.go # NotificationThread, NotificationSubject +│ ├── package.go # Package, PackageFile +│ ├── action.go # ActionRunner, ActionSecret, ActionVariable +│ ├── commit.go # Commit, CommitStatus, CombinedStatus +│ ├── content.go # ContentsResponse, FileOptions +│ ├── wiki.go # WikiPage, WikiPageMetaData +│ ├── review.go # PullReview, PullReviewComment +│ ├── reaction.go # Reaction +│ ├── topic.go # TopicResponse +│ ├── misc.go # Markdown, License, GitignoreTemplate, NodeInfo +│ ├── admin.go # Cron, QuotaGroup, QuotaRule +│ ├── activity.go # Activity, Feed +│ └── common.go # Shared types: Permission, ExternalTracker, etc. +│ +├── repos.go # RepoService: CRUD + fork, mirror, transfer, template +├── issues.go # IssueService: CRUD + pin, deadline, reactions, stopwatch +├── pulls.go # PullService: CRUD + merge, update, reviews, dismiss +├── orgs.go # OrgService: CRUD + members, avatar, block, hooks +├── users.go # UserService: CRUD + keys, followers, starred, settings +├── teams.go # TeamService: CRUD + members, repos +├── admin.go # AdminService: users, orgs, cron, runners, quota, unadopted +├── branches.go # BranchService: CRUD + protection rules +├── releases.go # ReleaseService: CRUD + assets +├── labels.go # LabelService: repo + org + issue labels +├── webhooks.go # WebhookService: CRUD + test hook +├── notifications.go # NotificationService: list, mark read +├── packages.go # PackageService: list, get, delete +├── actions.go # ActionsService: runners, secrets, variables, workflow dispatch +├── contents.go # ContentService: file read/write/delete via API +├── wiki.go # WikiService: pages +├── commits.go # CommitService: status, notes, diff +├── misc.go # MiscService: markdown, licenses, gitignore, nodeinfo +│ +├── config.go # URL/token resolution: env → config file → flags +│ +├── cmd/forgegen/ # Code generator: swagger.v1.json → types/*.go +│ ├── main.go +│ ├── parser.go # Parse OpenAPI 2.0 definitions +│ ├── generator.go # Render Go source files +│ └── templates/ # Go text/template files for codegen +│ +└── testdata/ + └── swagger.v1.json # Pinned spec for testing + generation +``` + +## Key Design Decisions + +### 1. Generic Resource[T, C, U] + +Three type parameters: T (resource type), C (create options), U (update options). + +```go +type Resource[T any, C any, U any] struct { + client *Client + path string // e.g. "/api/v1/repos/{owner}/{repo}/issues" +} + +func (r *Resource[T, C, U]) List(ctx context.Context, params Params, opts ListOptions) ([]T, error) +func (r *Resource[T, C, U]) Get(ctx context.Context, params Params, id string) (*T, error) +func (r *Resource[T, C, U]) Create(ctx context.Context, params Params, body *C) (*T, error) +func (r *Resource[T, C, U]) Update(ctx context.Context, params Params, id string, body *U) (*T, error) +func (r *Resource[T, C, U]) Delete(ctx context.Context, params Params, id string) error +``` + +`Params` is `map[string]string` resolving path variables: `{"owner": "core", "repo": "go-forge"}`. + +This covers 411 of 450 endpoints (91%). + +### 2. Service Structs Embed Resource + +```go +type IssueService struct { + Resource[types.Issue, types.CreateIssueOption, types.EditIssueOption] +} + +// CRUD comes free. Actions are hand-written: +func (s *IssueService) Pin(ctx context.Context, owner, repo string, index int64) error +func (s *IssueService) SetDeadline(ctx context.Context, owner, repo string, index int64, deadline *time.Time) error +``` + +### 3. Top-Level Forge Client + +```go +type Forge struct { + client *Client + Repos *RepoService + Issues *IssueService + Pulls *PullService + Orgs *OrgService + Users *UserService + Teams *TeamService + Admin *AdminService + Branches *BranchService + Releases *ReleaseService + Labels *LabelService + Webhooks *WebhookService + Notifications *NotificationService + Packages *PackageService + Actions *ActionsService + Contents *ContentService + Wiki *WikiService + Commits *CommitService + Misc *MiscService +} + +func NewForge(url, token string, opts ...Option) *Forge +``` + +### 4. Codegen from swagger.v1.json + +The `cmd/forgegen/` tool reads the OpenAPI 2.0 spec and generates: +- Go struct definitions with JSON tags and doc comments +- Enum constants +- Type mapping (OpenAPI → Go) + +229 type definitions → ~25 grouped Go files in `types/`. + +Type mapping rules: +| OpenAPI | Go | +|---------|-----| +| `string` | `string` | +| `string` + `date-time` | `time.Time` | +| `integer` + `int64` | `int64` | +| `integer` | `int` | +| `boolean` | `bool` | +| `array` of T | `[]T` | +| `$ref` | `*T` (pointer) | +| nullable | pointer type | +| `binary` | `[]byte` | + +### 5. HTTP Client + +```go +type Client struct { + baseURL string + token string + httpClient *http.Client + userAgent string +} + +func New(url, token string, opts ...Option) *Client + +func (c *Client) Get(ctx context.Context, path string, out any) error +func (c *Client) Post(ctx context.Context, path string, body, out any) error +func (c *Client) Patch(ctx context.Context, path string, body, out any) error +func (c *Client) Put(ctx context.Context, path string, body, out any) error +func (c *Client) Delete(ctx context.Context, path string) error +``` + +Options: `WithHTTPClient`, `WithUserAgent`, `WithRateLimit`, `WithLogger`. + +### 6. Pagination + +Forgejo uses `page` + `limit` query params and `X-Total-Count` response header. + +```go +type ListOptions struct { + Page int + Limit int // default 50, max configurable +} + +type PagedResult[T any] struct { + Items []T + TotalCount int + Page int + HasMore bool +} + +// ListAll fetches all pages automatically. +func (r *Resource[T, C, U]) ListAll(ctx context.Context, params Params) ([]T, error) +``` + +### 7. Error Handling + +```go +type APIError struct { + StatusCode int + Message string + URL string +} + +func IsNotFound(err error) bool +func IsForbidden(err error) bool +func IsConflict(err error) bool +``` + +### 8. Config Resolution (from go-scm/forge) + +Priority: flags → environment → config file. + +```go +func NewFromConfig(flagURL, flagToken string) (*Forge, error) +func ResolveConfig(flagURL, flagToken string) (url, token string, err error) +func SaveConfig(url, token string) error +``` + +Env vars: `FORGE_URL`, `FORGE_TOKEN`. Config file: `~/.config/forge/config.json`. + +## API Coverage + +| Category | Endpoints | CRUD | Actions | +|----------|-----------|------|---------| +| Repository | 175 | 165 | 10 (fork, mirror, transfer, template, avatar, diffpatch) | +| User | 74 | 70 | 4 (avatar, GPG verify) | +| Issue | 67 | 57 | 10 (pin, deadline, reactions, stopwatch, labels) | +| Organisation | 63 | 59 | 4 (avatar, block/unblock) | +| Admin | 39 | 35 | 4 (cron run, rename, adopt, quota set) | +| Miscellaneous | 12 | 7 | 5 (markdown render, markup, nodeinfo) | +| Notification | 7 | 7 | 0 | +| ActivityPub | 6 | 3 | 3 (inbox POST) | +| Package | 4 | 4 | 0 | +| Settings | 4 | 4 | 0 | +| **Total** | **450** | **411** | **39** | + +## Integration Points + +### go-api + +Services implement `DescribableGroup` from go-api Phase 3, enabling: +- REST endpoint generation via ToolBridge +- Auto-generated OpenAPI spec +- Multi-language SDK codegen + +### go-scm + +go-scm/forge/ becomes a thin adapter importing go-forge types. Existing go-scm users are unaffected — the multi-provider abstraction layer stays. + +### go-ai/mcp + +The MCP subsystem can register go-forge operations as MCP tools, giving AI agents full Forgejo API access. + +## 39 Unique Action Methods + +These require hand-written implementation: + +**Repository:** migrate, fork, generate (template), transfer, accept/reject transfer, mirror sync, push mirror sync, avatar, diffpatch, contents (multi-file modify) + +**Pull Requests:** merge, update (rebase), submit review, dismiss/undismiss review + +**Issues:** pin, set deadline, add reaction, start/stop stopwatch, add issue labels + +**Comments:** add reaction + +**Admin:** run cron task, adopt unadopted, rename user, set quota groups + +**Misc:** render markdown, render raw markdown, render markup, GPG key verify + +**ActivityPub:** inbox POST (actor, repo, user) + +**Actions:** dispatch workflow + +**Git:** set note on commit, test webhook diff --git a/docs/plans/completed/2026-02-21-go-forge-plan.md b/docs/plans/completed/2026-02-21-go-forge-plan.md new file mode 100644 index 0000000..c6b8240 --- /dev/null +++ b/docs/plans/completed/2026-02-21-go-forge-plan.md @@ -0,0 +1,2549 @@ +# go-forge Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Build a full-coverage Go client for the Forgejo API (450 endpoints) using a generic Resource[T,C,U] pattern and types generated from swagger.v1.json. + +**Architecture:** A code generator (`cmd/forgegen/`) parses Forgejo's Swagger 2.0 spec and emits typed Go structs. A generic `Resource[T,C,U]` provides List/Get/Create/Update/Delete for 411 CRUD endpoints. 18 service structs embed the generic resource and add 39 hand-written action methods. An HTTP client handles auth, pagination, rate limiting, and context.Context. + +**Tech Stack:** Go 1.25, `net/http`, `text/template`, generics, Swagger 2.0 (JSON) + +--- + +## Context + +**This is a NEW repo** at `forge.lthn.ai/core/go-forge`. Create it locally at `/Users/snider/Code/go-forge`. + +**Extracted from:** `/Users/snider/Code/go-scm/forge/` (45 methods covering 10% of API). The config resolution pattern (env → file → flags) comes from there. + +**Swagger spec:** Download from `https://forge.lthn.ai/swagger.v1.json` — Swagger 2.0 format, 229 type definitions, 450 operations across 284 paths. Pin it at `testdata/swagger.v1.json`. + +**Forgejo version:** 10.0.3 (Gitea 1.22.0 compatible) + +**Dependencies:** None (pure `net/http`). Config uses `forge.lthn.ai/core/go` for `pkg/config` and `pkg/log` — same as go-scm. + +**Key insight:** 91% of endpoints are generic CRUD (List/Get/Create/Update/Delete). The generic `Resource[T,C,U]` pattern means each service is a struct definition + path constant + optional action methods. The code generator handles 229 type definitions. + +**Test command:** `go test ./...` from the repo root. + +**The forge remote for this repo will be:** `ssh://git@forge.lthn.ai:2223/core/go-forge.git` + +--- + +## Wave 1: Foundation (Tasks 1-6) + +### Task 1: Repo scaffolding + go.mod + +**Files:** +- Create: `go.mod` +- Create: `go.sum` (auto-generated) +- Create: `doc.go` +- Create: `testdata/swagger.v1.json` (downloaded) + +**Step 1: Create directory and initialise module** + +```bash +mkdir -p /Users/snider/Code/go-forge/testdata +cd /Users/snider/Code/go-forge +git init +go mod init forge.lthn.ai/core/go-forge +``` + +**Step 2: Download and pin swagger spec** + +```bash +curl -s https://forge.lthn.ai/swagger.v1.json > testdata/swagger.v1.json +``` + +Verify: `python3 -c "import json; d=json.load(open('testdata/swagger.v1.json')); print(f'{len(d[\"definitions\"])} types, {len(d[\"paths\"])} paths')"` +Expected: `229 types, 284 paths` + +**Step 3: Write doc.go** + +```go +// Package forge provides a full-coverage Go client for the Forgejo API. +// +// Usage: +// +// f := forge.NewForge("https://forge.lthn.ai", "your-token") +// repos, err := f.Repos.List(ctx, forge.Params{"org": "core"}, forge.DefaultList) +// +// Types are generated from Forgejo's swagger.v1.json spec via cmd/forgegen/. +// Run `go generate ./types/...` to regenerate after a Forgejo upgrade. +package forge +``` + +**Step 4: Commit** + +```bash +git add -A +git commit -m "feat: scaffold go-forge repo with pinned swagger spec + +Co-Authored-By: Virgil " +``` + +--- + +### Task 2: HTTP Client + +**Files:** +- Create: `client.go` +- Create: `client_test.go` + +**Step 1: Write client tests** + +```go +package forge + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func TestClient_Good_Get(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + t.Errorf("expected GET, got %s", r.Method) + } + if r.Header.Get("Authorization") != "token test-token" { + t.Errorf("missing auth header") + } + if r.URL.Path != "/api/v1/user" { + t.Errorf("wrong path: %s", r.URL.Path) + } + json.NewEncoder(w).Encode(map[string]string{"login": "virgil"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "test-token") + var out map[string]string + err := c.Get(context.Background(), "/api/v1/user", &out) + if err != nil { + t.Fatal(err) + } + if out["login"] != "virgil" { + t.Errorf("got login=%q", out["login"]) + } +} + +func TestClient_Good_Post(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + t.Errorf("expected POST, got %s", r.Method) + } + var body map[string]string + json.NewDecoder(r.Body).Decode(&body) + if body["name"] != "test-repo" { + t.Errorf("wrong body: %v", body) + } + w.WriteHeader(http.StatusCreated) + json.NewEncoder(w).Encode(map[string]any{"id": 1, "name": "test-repo"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "test-token") + body := map[string]string{"name": "test-repo"} + var out map[string]any + err := c.Post(context.Background(), "/api/v1/orgs/core/repos", body, &out) + if err != nil { + t.Fatal(err) + } + if out["name"] != "test-repo" { + t.Errorf("got name=%v", out["name"]) + } +} + +func TestClient_Good_Delete(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodDelete { + t.Errorf("expected DELETE, got %s", r.Method) + } + w.WriteHeader(http.StatusNoContent) + })) + defer srv.Close() + + c := NewClient(srv.URL, "test-token") + err := c.Delete(context.Background(), "/api/v1/repos/core/test") + if err != nil { + t.Fatal(err) + } +} + +func TestClient_Bad_ServerError(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + json.NewEncoder(w).Encode(map[string]string{"message": "internal error"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "test-token") + err := c.Get(context.Background(), "/api/v1/user", nil) + if err == nil { + t.Fatal("expected error") + } + var apiErr *APIError + if !errors.As(err, &apiErr) { + t.Fatalf("expected APIError, got %T", err) + } + if apiErr.StatusCode != 500 { + t.Errorf("got status=%d", apiErr.StatusCode) + } +} + +func TestClient_Bad_NotFound(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNotFound) + json.NewEncoder(w).Encode(map[string]string{"message": "not found"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "test-token") + err := c.Get(context.Background(), "/api/v1/repos/x/y", nil) + if !IsNotFound(err) { + t.Fatalf("expected not found, got %v", err) + } +} + +func TestClient_Good_ContextCancellation(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + <-r.Context().Done() + })) + defer srv.Close() + + c := NewClient(srv.URL, "test-token") + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancel immediately + err := c.Get(ctx, "/api/v1/user", nil) + if err == nil { + t.Fatal("expected error from cancelled context") + } +} + +func TestClient_Good_Options(t *testing.T) { + c := NewClient("https://forge.lthn.ai", "tok", + WithUserAgent("go-forge/1.0"), + ) + if c.userAgent != "go-forge/1.0" { + t.Errorf("got user agent=%q", c.userAgent) + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestClient` +Expected: Compilation errors (types don't exist yet) + +**Step 3: Write client.go** + +```go +package forge + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strings" +) + +// APIError represents an error response from the Forgejo API. +type APIError struct { + StatusCode int + Message string + URL string +} + +func (e *APIError) Error() string { + return fmt.Sprintf("forge: %s %d: %s", e.URL, e.StatusCode, e.Message) +} + +// IsNotFound returns true if the error is a 404 response. +func IsNotFound(err error) bool { + var apiErr *APIError + return errors.As(err, &apiErr) && apiErr.StatusCode == http.StatusNotFound +} + +// IsForbidden returns true if the error is a 403 response. +func IsForbidden(err error) bool { + var apiErr *APIError + return errors.As(err, &apiErr) && apiErr.StatusCode == http.StatusForbidden +} + +// IsConflict returns true if the error is a 409 response. +func IsConflict(err error) bool { + var apiErr *APIError + return errors.As(err, &apiErr) && apiErr.StatusCode == http.StatusConflict +} + +// Option configures the Client. +type Option func(*Client) + +// WithHTTPClient sets a custom http.Client. +func WithHTTPClient(hc *http.Client) Option { + return func(c *Client) { c.httpClient = hc } +} + +// WithUserAgent sets the User-Agent header. +func WithUserAgent(ua string) Option { + return func(c *Client) { c.userAgent = ua } +} + +// Client is a low-level HTTP client for the Forgejo API. +type Client struct { + baseURL string + token string + httpClient *http.Client + userAgent string +} + +// NewClient creates a new Forgejo API client. +func NewClient(url, token string, opts ...Option) *Client { + c := &Client{ + baseURL: strings.TrimRight(url, "/"), + token: token, + httpClient: http.DefaultClient, + userAgent: "go-forge/0.1", + } + for _, opt := range opts { + opt(c) + } + return c +} + +// Get performs a GET request. +func (c *Client) Get(ctx context.Context, path string, out any) error { + return c.do(ctx, http.MethodGet, path, nil, out) +} + +// Post performs a POST request. +func (c *Client) Post(ctx context.Context, path string, body, out any) error { + return c.do(ctx, http.MethodPost, path, body, out) +} + +// Patch performs a PATCH request. +func (c *Client) Patch(ctx context.Context, path string, body, out any) error { + return c.do(ctx, http.MethodPatch, path, body, out) +} + +// Put performs a PUT request. +func (c *Client) Put(ctx context.Context, path string, body, out any) error { + return c.do(ctx, http.MethodPut, path, body, out) +} + +// Delete performs a DELETE request. +func (c *Client) Delete(ctx context.Context, path string) error { + return c.do(ctx, http.MethodDelete, path, nil, nil) +} + +func (c *Client) do(ctx context.Context, method, path string, body, out any) error { + url := c.baseURL + path + + var bodyReader io.Reader + if body != nil { + data, err := json.Marshal(body) + if err != nil { + return fmt.Errorf("forge: marshal body: %w", err) + } + bodyReader = bytes.NewReader(data) + } + + req, err := http.NewRequestWithContext(ctx, method, url, bodyReader) + if err != nil { + return fmt.Errorf("forge: create request: %w", err) + } + + req.Header.Set("Authorization", "token "+c.token) + req.Header.Set("Accept", "application/json") + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + if c.userAgent != "" { + req.Header.Set("User-Agent", c.userAgent) + } + + resp, err := c.httpClient.Do(req) + if err != nil { + return fmt.Errorf("forge: request %s %s: %w", method, path, err) + } + defer resp.Body.Close() + + if resp.StatusCode >= 400 { + return c.parseError(resp, path) + } + + if out != nil && resp.StatusCode != http.StatusNoContent { + if err := json.NewDecoder(resp.Body).Decode(out); err != nil { + return fmt.Errorf("forge: decode response: %w", err) + } + } + + return nil +} + +func (c *Client) parseError(resp *http.Response, path string) error { + var errBody struct { + Message string `json:"message"` + } + _ = json.NewDecoder(resp.Body).Decode(&errBody) + return &APIError{ + StatusCode: resp.StatusCode, + Message: errBody.Message, + URL: path, + } +} +``` + +**Step 4: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestClient` +Expected: All 7 tests PASS + +**Step 5: Commit** + +```bash +git add client.go client_test.go +git commit -m "feat: HTTP client with auth, context, error handling + +Co-Authored-By: Virgil " +``` + +--- + +### Task 3: Pagination + +**Files:** +- Create: `pagination.go` +- Create: `pagination_test.go` + +**Step 1: Write pagination tests** + +```go +package forge + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "strconv" + "testing" +) + +func TestPagination_Good_SinglePage(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Total-Count", "2") + json.NewEncoder(w).Encode([]map[string]int{{"id": 1}, {"id": 2}}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + result, err := ListAll[map[string]int](context.Background(), c, "/api/v1/repos", nil) + if err != nil { + t.Fatal(err) + } + if len(result) != 2 { + t.Errorf("got %d items", len(result)) + } +} + +func TestPagination_Good_MultiPage(t *testing.T) { + page := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + page++ + w.Header().Set("X-Total-Count", "100") + items := make([]map[string]int, 50) + for i := range items { + items[i] = map[string]int{"id": (page-1)*50 + i + 1} + } + json.NewEncoder(w).Encode(items) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + result, err := ListAll[map[string]int](context.Background(), c, "/api/v1/repos", nil) + if err != nil { + t.Fatal(err) + } + if len(result) != 100 { + t.Errorf("got %d items, want 100", len(result)) + } +} + +func TestPagination_Good_EmptyResult(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Total-Count", "0") + json.NewEncoder(w).Encode([]map[string]int{}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + result, err := ListAll[map[string]int](context.Background(), c, "/api/v1/repos", nil) + if err != nil { + t.Fatal(err) + } + if len(result) != 0 { + t.Errorf("got %d items", len(result)) + } +} + +func TestListPage_Good_QueryParams(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + p := r.URL.Query().Get("page") + l := r.URL.Query().Get("limit") + s := r.URL.Query().Get("state") + if p != "2" || l != "25" || s != "open" { + t.Errorf("wrong params: page=%s limit=%s state=%s", p, l, s) + } + w.Header().Set("X-Total-Count", "50") + json.NewEncoder(w).Encode([]map[string]int{}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + _, err := ListPage[map[string]int](context.Background(), c, "/api/v1/repos", + map[string]string{"state": "open"}, ListOptions{Page: 2, Limit: 25}) + if err != nil { + t.Fatal(err) + } +} + +func TestPagination_Bad_ServerError(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(500) + json.NewEncoder(w).Encode(map[string]string{"message": "fail"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + _, err := ListAll[map[string]int](context.Background(), c, "/api/v1/repos", nil) + if err == nil { + t.Fatal("expected error") + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestPagination -run TestListPage` +Expected: Compilation errors + +**Step 3: Write pagination.go** + +```go +package forge + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + "strconv" +) + +// ListOptions controls pagination. +type ListOptions struct { + Page int // 1-based page number + Limit int // items per page (default 50) +} + +// DefaultList returns sensible default pagination. +var DefaultList = ListOptions{Page: 1, Limit: 50} + +// PagedResult holds a single page of results with metadata. +type PagedResult[T any] struct { + Items []T + TotalCount int + Page int + HasMore bool +} + +// ListPage fetches a single page of results. +// Extra query params can be passed via the query map. +func ListPage[T any](ctx context.Context, c *Client, path string, query map[string]string, opts ListOptions) (*PagedResult[T], error) { + if opts.Page < 1 { + opts.Page = 1 + } + if opts.Limit < 1 { + opts.Limit = 50 + } + + u, err := url.Parse(c.baseURL + path) + if err != nil { + return nil, fmt.Errorf("forge: parse url: %w", err) + } + + q := u.Query() + q.Set("page", strconv.Itoa(opts.Page)) + q.Set("limit", strconv.Itoa(opts.Limit)) + for k, v := range query { + q.Set(k, v) + } + u.RawQuery = q.Encode() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil) + if err != nil { + return nil, fmt.Errorf("forge: create request: %w", err) + } + + req.Header.Set("Authorization", "token "+c.token) + req.Header.Set("Accept", "application/json") + if c.userAgent != "" { + req.Header.Set("User-Agent", c.userAgent) + } + + resp, err := c.httpClient.Do(req) + if err != nil { + return nil, fmt.Errorf("forge: request GET %s: %w", path, err) + } + defer resp.Body.Close() + + if resp.StatusCode >= 400 { + return nil, c.parseError(resp, path) + } + + var items []T + if err := json.NewDecoder(resp.Body).Decode(&items); err != nil { + return nil, fmt.Errorf("forge: decode response: %w", err) + } + + totalCount, _ := strconv.Atoi(resp.Header.Get("X-Total-Count")) + + return &PagedResult[T]{ + Items: items, + TotalCount: totalCount, + Page: opts.Page, + HasMore: len(items) >= opts.Limit && opts.Page*opts.Limit < totalCount, + }, nil +} + +// ListAll fetches all pages of results. +func ListAll[T any](ctx context.Context, c *Client, path string, query map[string]string) ([]T, error) { + var all []T + page := 1 + + for { + result, err := ListPage[T](ctx, c, path, query, ListOptions{Page: page, Limit: 50}) + if err != nil { + return nil, err + } + all = append(all, result.Items...) + if !result.HasMore { + break + } + page++ + } + + return all, nil +} +``` + +**Step 4: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run "TestPagination|TestListPage"` +Expected: All 5 tests PASS + +**Step 5: Commit** + +```bash +git add pagination.go pagination_test.go +git commit -m "feat: generic pagination with ListAll and ListPage + +Co-Authored-By: Virgil " +``` + +--- + +### Task 4: Params and path resolution + +**Files:** +- Create: `params.go` +- Create: `params_test.go` + +**Step 1: Write tests** + +```go +package forge + +import "testing" + +func TestResolvePath_Good_Simple(t *testing.T) { + got := ResolvePath("/api/v1/repos/{owner}/{repo}", Params{"owner": "core", "repo": "go-forge"}) + want := "/api/v1/repos/core/go-forge" + if got != want { + t.Errorf("got %q, want %q", got, want) + } +} + +func TestResolvePath_Good_NoParams(t *testing.T) { + got := ResolvePath("/api/v1/user", nil) + if got != "/api/v1/user" { + t.Errorf("got %q", got) + } +} + +func TestResolvePath_Good_WithID(t *testing.T) { + got := ResolvePath("/api/v1/repos/{owner}/{repo}/issues/{index}", Params{ + "owner": "core", "repo": "go-forge", "index": "42", + }) + want := "/api/v1/repos/core/go-forge/issues/42" + if got != want { + t.Errorf("got %q, want %q", got, want) + } +} + +func TestResolvePath_Good_URLEncoding(t *testing.T) { + got := ResolvePath("/api/v1/repos/{owner}/{repo}", Params{"owner": "my org", "repo": "my repo"}) + want := "/api/v1/repos/my%20org/my%20repo" + if got != want { + t.Errorf("got %q, want %q", got, want) + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestResolvePath` +Expected: Compilation errors + +**Step 3: Write params.go** + +```go +package forge + +import ( + "net/url" + "strings" +) + +// Params maps path variable names to values. +// Example: Params{"owner": "core", "repo": "go-forge"} +type Params map[string]string + +// ResolvePath substitutes {placeholders} in path with values from params. +func ResolvePath(path string, params Params) string { + for k, v := range params { + path = strings.ReplaceAll(path, "{"+k+"}", url.PathEscape(v)) + } + return path +} +``` + +**Step 4: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestResolvePath` +Expected: All 4 tests PASS + +**Step 5: Commit** + +```bash +git add params.go params_test.go +git commit -m "feat: path parameter resolution with URL encoding + +Co-Authored-By: Virgil " +``` + +--- + +### Task 5: Generic Resource[T, C, U] + +**Files:** +- Create: `resource.go` +- Create: `resource_test.go` + +**Step 1: Write resource tests** + +```go +package forge + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +// Test types +type testItem struct { + ID int `json:"id"` + Name string `json:"name"` +} + +type testCreate struct { + Name string `json:"name"` +} + +type testUpdate struct { + Name *string `json:"name,omitempty"` +} + +func TestResource_Good_List(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/v1/orgs/core/repos" { + t.Errorf("wrong path: %s", r.URL.Path) + } + w.Header().Set("X-Total-Count", "2") + json.NewEncoder(w).Encode([]testItem{{1, "a"}, {2, "b"}}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + res := NewResource[testItem, testCreate, testUpdate](c, "/api/v1/orgs/{org}/repos") + + items, err := res.List(context.Background(), Params{"org": "core"}, DefaultList) + if err != nil { + t.Fatal(err) + } + if len(items.Items) != 2 { + t.Errorf("got %d items", len(items.Items)) + } +} + +func TestResource_Good_Get(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/v1/repos/core/go-forge" { + t.Errorf("wrong path: %s", r.URL.Path) + } + json.NewEncoder(w).Encode(testItem{1, "go-forge"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + res := NewResource[testItem, testCreate, testUpdate](c, "/api/v1/repos/{owner}/{repo}") + + item, err := res.Get(context.Background(), Params{"owner": "core", "repo": "go-forge"}) + if err != nil { + t.Fatal(err) + } + if item.Name != "go-forge" { + t.Errorf("got name=%q", item.Name) + } +} + +func TestResource_Good_Create(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + t.Errorf("expected POST, got %s", r.Method) + } + var body testCreate + json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusCreated) + json.NewEncoder(w).Encode(testItem{1, body.Name}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + res := NewResource[testItem, testCreate, testUpdate](c, "/api/v1/orgs/{org}/repos") + + item, err := res.Create(context.Background(), Params{"org": "core"}, &testCreate{Name: "new-repo"}) + if err != nil { + t.Fatal(err) + } + if item.Name != "new-repo" { + t.Errorf("got name=%q", item.Name) + } +} + +func TestResource_Good_Update(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPatch { + t.Errorf("expected PATCH, got %s", r.Method) + } + json.NewEncoder(w).Encode(testItem{1, "updated"}) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + res := NewResource[testItem, testCreate, testUpdate](c, "/api/v1/repos/{owner}/{repo}") + + name := "updated" + item, err := res.Update(context.Background(), Params{"owner": "core", "repo": "old"}, &testUpdate{Name: &name}) + if err != nil { + t.Fatal(err) + } + if item.Name != "updated" { + t.Errorf("got name=%q", item.Name) + } +} + +func TestResource_Good_Delete(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodDelete { + t.Errorf("expected DELETE, got %s", r.Method) + } + w.WriteHeader(http.StatusNoContent) + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + res := NewResource[testItem, testCreate, testUpdate](c, "/api/v1/repos/{owner}/{repo}") + + err := res.Delete(context.Background(), Params{"owner": "core", "repo": "old"}) + if err != nil { + t.Fatal(err) + } +} + +func TestResource_Good_ListAll(t *testing.T) { + page := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + page++ + w.Header().Set("X-Total-Count", "3") + if page == 1 { + json.NewEncoder(w).Encode([]testItem{{1, "a"}, {2, "b"}}) + } else { + json.NewEncoder(w).Encode([]testItem{{3, "c"}}) + } + })) + defer srv.Close() + + c := NewClient(srv.URL, "tok") + res := NewResource[testItem, testCreate, testUpdate](c, "/api/v1/repos") + + items, err := res.ListAll(context.Background(), nil) + if err != nil { + t.Fatal(err) + } + if len(items) != 3 { + t.Errorf("got %d items, want 3", len(items)) + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestResource` +Expected: Compilation errors + +**Step 3: Write resource.go** + +```go +package forge + +import "context" + +// Resource provides generic CRUD operations for a Forgejo API resource. +// T is the resource type, C is the create options type, U is the update options type. +type Resource[T any, C any, U any] struct { + client *Client + path string +} + +// NewResource creates a new Resource for the given path pattern. +// The path may contain {placeholders} that are resolved via Params. +func NewResource[T any, C any, U any](c *Client, path string) *Resource[T, C, U] { + return &Resource[T, C, U]{client: c, path: path} +} + +// List returns a single page of resources. +func (r *Resource[T, C, U]) List(ctx context.Context, params Params, opts ListOptions) (*PagedResult[T], error) { + return ListPage[T](ctx, r.client, ResolvePath(r.path, params), nil, opts) +} + +// ListAll returns all resources across all pages. +func (r *Resource[T, C, U]) ListAll(ctx context.Context, params Params) ([]T, error) { + return ListAll[T](ctx, r.client, ResolvePath(r.path, params), nil) +} + +// Get returns a single resource by appending id to the path. +func (r *Resource[T, C, U]) Get(ctx context.Context, params Params) (*T, error) { + var out T + if err := r.client.Get(ctx, ResolvePath(r.path, params), &out); err != nil { + return nil, err + } + return &out, nil +} + +// Create creates a new resource. +func (r *Resource[T, C, U]) Create(ctx context.Context, params Params, body *C) (*T, error) { + var out T + if err := r.client.Post(ctx, ResolvePath(r.path, params), body, &out); err != nil { + return nil, err + } + return &out, nil +} + +// Update modifies an existing resource. +func (r *Resource[T, C, U]) Update(ctx context.Context, params Params, body *U) (*T, error) { + var out T + if err := r.client.Patch(ctx, ResolvePath(r.path, params), body, &out); err != nil { + return nil, err + } + return &out, nil +} + +// Delete removes a resource. +func (r *Resource[T, C, U]) Delete(ctx context.Context, params Params) error { + return r.client.Delete(ctx, ResolvePath(r.path, params)) +} +``` + +**Step 4: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestResource` +Expected: All 6 tests PASS + +**Step 5: Commit** + +```bash +git add resource.go resource_test.go +git commit -m "feat: generic Resource[T,C,U] for CRUD operations + +Co-Authored-By: Virgil " +``` + +--- + +### Task 6: Config resolution (extracted from go-scm) + +**Files:** +- Create: `config.go` +- Create: `config_test.go` + +**Step 1: Write config tests** + +```go +package forge + +import ( + "os" + "testing" +) + +func TestResolveConfig_Good_EnvOverrides(t *testing.T) { + t.Setenv("FORGE_URL", "https://forge.example.com") + t.Setenv("FORGE_TOKEN", "env-token") + + url, token, err := ResolveConfig("", "") + if err != nil { + t.Fatal(err) + } + if url != "https://forge.example.com" { + t.Errorf("got url=%q", url) + } + if token != "env-token" { + t.Errorf("got token=%q", token) + } +} + +func TestResolveConfig_Good_FlagOverridesEnv(t *testing.T) { + t.Setenv("FORGE_URL", "https://env.example.com") + t.Setenv("FORGE_TOKEN", "env-token") + + url, token, err := ResolveConfig("https://flag.example.com", "flag-token") + if err != nil { + t.Fatal(err) + } + if url != "https://flag.example.com" { + t.Errorf("got url=%q", url) + } + if token != "flag-token" { + t.Errorf("got token=%q", token) + } +} + +func TestResolveConfig_Good_DefaultURL(t *testing.T) { + // Clear env vars to test defaults + os.Unsetenv("FORGE_URL") + os.Unsetenv("FORGE_TOKEN") + + url, _, err := ResolveConfig("", "") + if err != nil { + t.Fatal(err) + } + if url != DefaultURL { + t.Errorf("got url=%q, want %q", url, DefaultURL) + } +} + +func TestNewForgeFromConfig_Bad_NoToken(t *testing.T) { + os.Unsetenv("FORGE_URL") + os.Unsetenv("FORGE_TOKEN") + + _, err := NewForgeFromConfig("", "") + if err == nil { + t.Fatal("expected error for missing token") + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run TestResolveConfig -run TestNewForgeFromConfig` +Expected: Compilation errors + +**Step 3: Write config.go** + +```go +package forge + +import ( + "fmt" + "os" +) + +const ( + // DefaultURL is used when no URL is configured. + DefaultURL = "http://localhost:3000" +) + +// ResolveConfig resolves Forge URL and token from multiple sources. +// Priority (highest to lowest): flags → environment → defaults. +func ResolveConfig(flagURL, flagToken string) (url, token string, err error) { + // Environment variables + url = os.Getenv("FORGE_URL") + token = os.Getenv("FORGE_TOKEN") + + // Flag overrides + if flagURL != "" { + url = flagURL + } + if flagToken != "" { + token = flagToken + } + + // Default URL + if url == "" { + url = DefaultURL + } + + return url, token, nil +} + +// NewForgeFromConfig creates a Forge client using resolved configuration. +func NewForgeFromConfig(flagURL, flagToken string, opts ...Option) (*Forge, error) { + url, token, err := ResolveConfig(flagURL, flagToken) + if err != nil { + return nil, err + } + if token == "" { + return nil, fmt.Errorf("forge: no API token configured (set FORGE_TOKEN or pass --token)") + } + return NewForge(url, token, opts...), nil +} +``` + +**Step 4: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run "TestResolveConfig|TestNewForgeFromConfig"` +Expected: All 4 tests PASS (Note: `NewForge` doesn't exist yet — if this fails, create a stub `NewForge` function that just returns `&Forge{client: NewClient(url, token, opts...)}`) + +**Step 5: Commit** + +```bash +git add config.go config_test.go +git commit -m "feat: config resolution from env vars and flags + +Co-Authored-By: Virgil " +``` + +--- + +## Wave 2: Code Generator (Tasks 7-9) + +### Task 7: Swagger spec parser + +**Files:** +- Create: `cmd/forgegen/main.go` +- Create: `cmd/forgegen/parser.go` +- Create: `cmd/forgegen/parser_test.go` + +The parser reads swagger.v1.json and extracts type definitions into an intermediate representation. + +**Step 1: Write parser tests** + +```go +package main + +import ( + "os" + "testing" +) + +func TestParser_Good_LoadSpec(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + if spec.Swagger != "2.0" { + t.Errorf("got swagger=%q", spec.Swagger) + } + if len(spec.Definitions) < 200 { + t.Errorf("got %d definitions, expected 200+", len(spec.Definitions)) + } +} + +func TestParser_Good_ExtractTypes(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + types := ExtractTypes(spec) + if len(types) < 200 { + t.Errorf("got %d types", len(types)) + } + + // Check a known type + repo, ok := types["Repository"] + if !ok { + t.Fatal("Repository type not found") + } + if len(repo.Fields) < 50 { + t.Errorf("Repository has %d fields, expected 50+", len(repo.Fields)) + } +} + +func TestParser_Good_FieldTypes(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + types := ExtractTypes(spec) + repo := types["Repository"] + + // Check specific field mappings + for _, f := range repo.Fields { + switch f.JSONName { + case "id": + if f.GoType != "int64" { + t.Errorf("id: got %q, want int64", f.GoType) + } + case "name": + if f.GoType != "string" { + t.Errorf("name: got %q, want string", f.GoType) + } + case "private": + if f.GoType != "bool" { + t.Errorf("private: got %q, want bool", f.GoType) + } + case "created_at": + if f.GoType != "time.Time" { + t.Errorf("created_at: got %q, want time.Time", f.GoType) + } + case "owner": + if f.GoType != "*User" { + t.Errorf("owner: got %q, want *User", f.GoType) + } + } + } +} + +func TestParser_Good_DetectCreateEditPairs(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + pairs := DetectCRUDPairs(spec) + // Should find Repository, Issue, PullRequest, etc. + if len(pairs) < 10 { + t.Errorf("got %d pairs, expected 10+", len(pairs)) + } + + found := false + for _, p := range pairs { + if p.Base == "Repository" { + found = true + if p.Create != "CreateRepoOption" { + t.Errorf("repo create=%q", p.Create) + } + } + } + if !found { + t.Fatal("Repository pair not found") + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v ./cmd/forgegen/ -run TestParser` +Expected: Compilation errors + +**Step 3: Write parser.go** + +```go +package main + +import ( + "encoding/json" + "fmt" + "os" + "sort" + "strings" +) + +// Spec represents a Swagger 2.0 specification. +type Spec struct { + Swagger string `json:"swagger"` + Info SpecInfo `json:"info"` + Definitions map[string]SchemaDefinition `json:"definitions"` + Paths map[string]map[string]any `json:"paths"` +} + +type SpecInfo struct { + Title string `json:"title"` + Version string `json:"version"` +} + +// SchemaDefinition represents a type definition in the spec. +type SchemaDefinition struct { + Description string `json:"description"` + Type string `json:"type"` + Properties map[string]SchemaProperty `json:"properties"` + Required []string `json:"required"` + Enum []any `json:"enum"` + XGoName string `json:"x-go-name"` +} + +// SchemaProperty represents a field in a type definition. +type SchemaProperty struct { + Type string `json:"type"` + Format string `json:"format"` + Description string `json:"description"` + Ref string `json:"$ref"` + Items *SchemaProperty `json:"items"` + Enum []any `json:"enum"` + XGoName string `json:"x-go-name"` +} + +// GoType represents a Go type extracted from the spec. +type GoType struct { + Name string + Description string + Fields []GoField + IsEnum bool + EnumValues []string +} + +// GoField represents a field in a Go struct. +type GoField struct { + GoName string + GoType string + JSONName string + Comment string + Required bool +} + +// CRUDPair maps a base type to its Create and Edit option types. +type CRUDPair struct { + Base string // e.g. "Repository" + Create string // e.g. "CreateRepoOption" + Edit string // e.g. "EditRepoOption" +} + +// LoadSpec reads and parses a Swagger 2.0 JSON file. +func LoadSpec(path string) (*Spec, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("read spec: %w", err) + } + var spec Spec + if err := json.Unmarshal(data, &spec); err != nil { + return nil, fmt.Errorf("parse spec: %w", err) + } + return &spec, nil +} + +// ExtractTypes converts spec definitions to Go types. +func ExtractTypes(spec *Spec) map[string]*GoType { + result := make(map[string]*GoType) + + for name, def := range spec.Definitions { + gt := &GoType{ + Name: name, + Description: def.Description, + } + + if len(def.Enum) > 0 { + gt.IsEnum = true + for _, v := range def.Enum { + gt.EnumValues = append(gt.EnumValues, fmt.Sprintf("%v", v)) + } + sort.Strings(gt.EnumValues) + result[name] = gt + continue + } + + required := make(map[string]bool) + for _, r := range def.Required { + required[r] = true + } + + for fieldName, prop := range def.Properties { + goName := prop.XGoName + if goName == "" { + goName = pascalCase(fieldName) + } + + gf := GoField{ + GoName: goName, + GoType: resolveGoType(prop), + JSONName: fieldName, + Comment: prop.Description, + Required: required[fieldName], + } + gt.Fields = append(gt.Fields, gf) + } + + // Sort fields alphabetically for stable output + sort.Slice(gt.Fields, func(i, j int) bool { + return gt.Fields[i].GoName < gt.Fields[j].GoName + }) + + result[name] = gt + } + + return result +} + +// DetectCRUDPairs finds Create/Edit option pairs. +func DetectCRUDPairs(spec *Spec) []CRUDPair { + var pairs []CRUDPair + + for name := range spec.Definitions { + if !strings.HasPrefix(name, "Create") || !strings.HasSuffix(name, "Option") { + continue + } + + // CreateXxxOption → Xxx → EditXxxOption + inner := strings.TrimPrefix(name, "Create") + inner = strings.TrimSuffix(inner, "Option") + + editName := "Edit" + inner + "Option" + + pair := CRUDPair{ + Base: inner, + Create: name, + } + + if _, ok := spec.Definitions[editName]; ok { + pair.Edit = editName + } + + pairs = append(pairs, pair) + } + + sort.Slice(pairs, func(i, j int) bool { + return pairs[i].Base < pairs[j].Base + }) + + return pairs +} + +func resolveGoType(prop SchemaProperty) string { + if prop.Ref != "" { + parts := strings.Split(prop.Ref, "/") + return "*" + parts[len(parts)-1] + } + + switch prop.Type { + case "string": + switch prop.Format { + case "date-time": + return "time.Time" + case "binary": + return "[]byte" + default: + return "string" + } + case "integer": + switch prop.Format { + case "int64": + return "int64" + case "int32": + return "int32" + default: + return "int" + } + case "number": + switch prop.Format { + case "float": + return "float32" + default: + return "float64" + } + case "boolean": + return "bool" + case "array": + if prop.Items != nil { + itemType := resolveGoType(*prop.Items) + return "[]" + itemType + } + return "[]any" + case "object": + return "map[string]any" + default: + if prop.Type == "" && prop.Ref == "" { + return "any" + } + return "any" + } +} + +func pascalCase(s string) string { + parts := strings.FieldsFunc(s, func(r rune) bool { + return r == '_' || r == '-' + }) + for i, p := range parts { + if len(p) == 0 { + continue + } + // Handle common acronyms + upper := strings.ToUpper(p) + switch upper { + case "ID", "URL", "HTML", "SSH", "HTTP", "HTTPS", "API", "URI", "GPG", "IP", "CSS", "JS": + parts[i] = upper + default: + parts[i] = strings.ToUpper(p[:1]) + p[1:] + } + } + return strings.Join(parts, "") +} +``` + +**Step 4: Write main.go stub** + +```go +package main + +import ( + "flag" + "fmt" + "os" +) + +func main() { + specPath := flag.String("spec", "testdata/swagger.v1.json", "path to swagger.v1.json") + outDir := flag.String("out", "types", "output directory for generated types") + flag.Parse() + + spec, err := LoadSpec(*specPath) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + + types := ExtractTypes(spec) + pairs := DetectCRUDPairs(spec) + + fmt.Printf("Loaded %d types, %d CRUD pairs\n", len(types), len(pairs)) + fmt.Printf("Output dir: %s\n", *outDir) + + // Generation happens in Task 8 + if err := Generate(types, pairs, *outDir); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } +} +``` + +**Step 5: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v ./cmd/forgegen/ -run TestParser` +Expected: All 4 tests PASS (Note: `Generate` doesn't exist yet — add a stub: `func Generate(...) error { return nil }`) + +**Step 6: Commit** + +```bash +git add cmd/forgegen/ +git commit -m "feat: swagger spec parser for type extraction + +Co-Authored-By: Virgil " +``` + +--- + +### Task 8: Code generator — Go source emission + +**Files:** +- Create: `cmd/forgegen/generator.go` +- Create: `cmd/forgegen/generator_test.go` + +**Step 1: Write generator tests** + +```go +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestGenerate_Good_CreatesFiles(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + types := ExtractTypes(spec) + pairs := DetectCRUDPairs(spec) + + outDir := t.TempDir() + if err := Generate(types, pairs, outDir); err != nil { + t.Fatal(err) + } + + // Should create at least one .go file + entries, _ := os.ReadDir(outDir) + goFiles := 0 + for _, e := range entries { + if strings.HasSuffix(e.Name(), ".go") { + goFiles++ + } + } + if goFiles == 0 { + t.Fatal("no .go files generated") + } +} + +func TestGenerate_Good_ValidGoSyntax(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + types := ExtractTypes(spec) + pairs := DetectCRUDPairs(spec) + + outDir := t.TempDir() + if err := Generate(types, pairs, outDir); err != nil { + t.Fatal(err) + } + + // Read a generated file and verify basic Go syntax markers + data, err := os.ReadFile(filepath.Join(outDir, "repo.go")) + if err != nil { + // Try another name + entries, _ := os.ReadDir(outDir) + for _, e := range entries { + if strings.HasSuffix(e.Name(), ".go") { + data, err = os.ReadFile(filepath.Join(outDir, e.Name())) + break + } + } + } + if err != nil { + t.Fatal(err) + } + + content := string(data) + if !strings.Contains(content, "package types") { + t.Error("missing package declaration") + } + if !strings.Contains(content, "// Code generated") { + t.Error("missing generated comment") + } +} + +func TestGenerate_Good_RepositoryType(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + types := ExtractTypes(spec) + pairs := DetectCRUDPairs(spec) + + outDir := t.TempDir() + if err := Generate(types, pairs, outDir); err != nil { + t.Fatal(err) + } + + // Find file containing Repository type + var content string + entries, _ := os.ReadDir(outDir) + for _, e := range entries { + data, _ := os.ReadFile(filepath.Join(outDir, e.Name())) + if strings.Contains(string(data), "type Repository struct") { + content = string(data) + break + } + } + + if content == "" { + t.Fatal("Repository type not found in any generated file") + } + + // Check essential fields exist + checks := []string{ + "`json:\"id\"`", + "`json:\"name\"`", + "`json:\"full_name\"`", + "`json:\"private\"`", + } + for _, check := range checks { + if !strings.Contains(content, check) { + t.Errorf("missing field with tag %s", check) + } + } +} + +func TestGenerate_Good_TimeImport(t *testing.T) { + spec, err := LoadSpec("../../testdata/swagger.v1.json") + if err != nil { + t.Fatal(err) + } + + types := ExtractTypes(spec) + pairs := DetectCRUDPairs(spec) + + outDir := t.TempDir() + if err := Generate(types, pairs, outDir); err != nil { + t.Fatal(err) + } + + // Files with time.Time fields should import "time" + entries, _ := os.ReadDir(outDir) + for _, e := range entries { + data, _ := os.ReadFile(filepath.Join(outDir, e.Name())) + content := string(data) + if strings.Contains(content, "time.Time") && !strings.Contains(content, "\"time\"") { + t.Errorf("file %s uses time.Time but doesn't import time", e.Name()) + } + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v ./cmd/forgegen/ -run TestGenerate` +Expected: Failures (Generate is stub) + +**Step 3: Write generator.go** + +The generator groups types by logical domain and writes one `.go` file per group. Type grouping uses name prefixes and the CRUD pairs. + +```go +package main + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "text/template" +) + +// typeGrouping maps types to their output file. +var typeGrouping = map[string]string{ + "Repository": "repo", + "Repo": "repo", + "Issue": "issue", + "PullRequest": "pr", + "Pull": "pr", + "User": "user", + "Organization": "org", + "Org": "org", + "Team": "team", + "Label": "label", + "Milestone": "milestone", + "Release": "release", + "Tag": "tag", + "Branch": "branch", + "Hook": "hook", + "Deploy": "key", + "PublicKey": "key", + "GPGKey": "key", + "Key": "key", + "Notification": "notification", + "Package": "package", + "Action": "action", + "Commit": "commit", + "Git": "git", + "Contents": "content", + "File": "content", + "Wiki": "wiki", + "Comment": "comment", + "Review": "review", + "Reaction": "reaction", + "Topic": "topic", + "Status": "status", + "Combined": "status", + "Cron": "admin", + "Quota": "quota", + "OAuth2": "oauth", + "AccessToken": "oauth", + "API": "error", + "Forbidden": "error", + "NotFound": "error", + "NodeInfo": "federation", + "Activity": "activity", + "Feed": "activity", + "StopWatch": "time_tracking", + "TrackedTime": "time_tracking", + "Blocked": "user", + "Email": "user", + "Settings": "settings", + "GeneralAPI": "settings", + "GeneralAttachment": "settings", + "GeneralRepo": "settings", + "GeneralUI": "settings", + "Markdown": "misc", + "Markup": "misc", + "License": "misc", + "Gitignore": "misc", + "Annotated": "git", + "Note": "git", + "ChangedFile": "git", + "ExternalTracker": "repo", + "ExternalWiki": "repo", + "InternalTracker": "repo", + "Permission": "common", + "RepoTransfer": "repo", + "PayloadCommit": "hook", + "Dispatch": "action", + "Secret": "action", + "Variable": "action", + "Push": "repo", + "Mirror": "repo", + "Attachment": "common", + "EditDeadline": "issue", + "IssueDeadline": "issue", + "IssueLabels": "issue", + "IssueMeta": "issue", + "IssueTemplate": "issue", + "StateType": "common", + "TimeStamp": "common", + "Rename": "admin", + "Unadopted": "admin", +} + +// classifyType determines which file a type belongs in. +func classifyType(name string) string { + // Direct match + if group, ok := typeGrouping[name]; ok { + return group + } + + // Prefix match (longest first) + for prefix, group := range typeGrouping { + if strings.HasPrefix(name, prefix) { + return group + } + } + + // Try common suffixes + if strings.HasSuffix(name, "Option") || strings.HasSuffix(name, "Options") { + // Strip Create/Edit prefix to find base + trimmed := name + trimmed = strings.TrimPrefix(trimmed, "Create") + trimmed = strings.TrimPrefix(trimmed, "Edit") + trimmed = strings.TrimPrefix(trimmed, "Delete") + trimmed = strings.TrimPrefix(trimmed, "Update") + trimmed = strings.TrimSuffix(trimmed, "Option") + trimmed = strings.TrimSuffix(trimmed, "Options") + if group, ok := typeGrouping[trimmed]; ok { + return group + } + } + + return "misc" +} + +// Generate writes Go source files for all types. +func Generate(types map[string]*GoType, pairs []CRUDPair, outDir string) error { + if err := os.MkdirAll(outDir, 0755); err != nil { + return fmt.Errorf("create output dir: %w", err) + } + + // Group types by file + groups := make(map[string][]*GoType) + for _, gt := range types { + file := classifyType(gt.Name) + groups[file] = append(groups[file], gt) + } + + // Sort types within each group + for file := range groups { + sort.Slice(groups[file], func(i, j int) bool { + return groups[file][i].Name < groups[file][j].Name + }) + } + + // Write each file + for file, fileTypes := range groups { + if err := writeFile(filepath.Join(outDir, file+".go"), fileTypes); err != nil { + return fmt.Errorf("write %s.go: %w", file, err) + } + } + + return nil +} + +var fileTmpl = template.Must(template.New("file").Parse(`// Code generated by forgegen from swagger.v1.json — DO NOT EDIT. + +package types +{{if .NeedsTime}} +import "time" +{{end}} +{{range .Types}} +{{if .Description}}// {{.Name}} — {{.Description}}{{else}}// {{.Name}} represents a Forgejo API type.{{end}} +{{if .IsEnum}}type {{.Name}} string + +const ( +{{range .EnumValues}} {{$.EnumConst .Name .}} {{$.EnumType .Name}} = "{{.}}" +{{end}}) +{{else}}type {{.Name}} struct { +{{range .Fields}} {{.GoName}} {{.GoType}} ` + "`" + `json:"{{.JSONName}}{{if not .Required}},omitempty{{end}}"` + "`" + `{{if .Comment}} // {{.Comment}}{{end}} +{{end}}} +{{end}} +{{end}}`)) + +type fileData struct { + Types []*GoType + NeedsTime bool +} + +func (fd fileData) EnumConst(typeName, value string) string { + return typeName + pascalCase(value) +} + +func (fd fileData) EnumType(typeName string) string { + return typeName +} + +func writeFile(path string, types []*GoType) error { + needsTime := false + for _, gt := range types { + for _, f := range gt.Fields { + if strings.Contains(f.GoType, "time.Time") { + needsTime = true + break + } + } + if needsTime { + break + } + } + + f, err := os.Create(path) + if err != nil { + return err + } + defer f.Close() + + return fileTmpl.Execute(f, fileData{ + Types: types, + NeedsTime: needsTime, + }) +} +``` + +**Step 4: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v ./cmd/forgegen/ -run TestGenerate` +Expected: All 4 tests PASS + +**Step 5: Commit** + +```bash +git add cmd/forgegen/generator.go cmd/forgegen/generator_test.go +git commit -m "feat: Go source code generator from Swagger types + +Co-Authored-By: Virgil " +``` + +--- + +### Task 9: Generate types + verify compilation + +**Files:** +- Create: `types/` directory with generated files +- Create: `types/generate.go` (go:generate directive) + +**Step 1: Run the generator** + +```bash +cd /Users/snider/Code/go-forge +mkdir -p types +go run ./cmd/forgegen/ -spec testdata/swagger.v1.json -out types/ +``` + +**Step 2: Add go:generate directive** + +Create `types/generate.go`: +```go +package types + +//go:generate go run ../cmd/forgegen/ -spec ../testdata/swagger.v1.json -out . +``` + +**Step 3: Verify compilation** + +Run: `cd /Users/snider/Code/go-forge && go build ./types/` +Expected: Compiles without errors + +If there are compilation errors, fix the generator (`cmd/forgegen/generator.go`) and regenerate. Common issues: +- Missing imports (time) +- Duplicate field names (GoName collision) +- Invalid Go identifiers (reserved words, starting with numbers) + +**Step 4: Run all tests** + +Run: `cd /Users/snider/Code/go-forge && go test ./...` +Expected: All tests pass + +**Step 5: Commit** + +```bash +git add types/ +git commit -m "feat: generate all 229 Forgejo API types from swagger spec + +Co-Authored-By: Virgil " +``` + +--- + +## Wave 3: Core Services (Tasks 10-13) + +Each service follows the same pattern: embed `Resource[T,C,U]`, add action methods. The first service (Task 10) is fully detailed as a template. Subsequent services follow the same structure with less repetition. + +### Task 10: Forge client + RepoService (template service) + +**Files:** +- Create: `forge.go` +- Create: `repos.go` +- Create: `forge_test.go` + +**Step 1: Write tests** + +```go +package forge + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "forge.lthn.ai/core/go-forge/types" +) + +func TestForge_Good_NewForge(t *testing.T) { + f := NewForge("https://forge.lthn.ai", "tok") + if f.Repos == nil { + t.Fatal("Repos service is nil") + } + if f.Issues == nil { + t.Fatal("Issues service is nil") + } +} + +func TestRepoService_Good_List(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Total-Count", "1") + json.NewEncoder(w).Encode([]types.Repository{{Name: "go-forge"}}) + })) + defer srv.Close() + + f := NewForge(srv.URL, "tok") + result, err := f.Repos.List(context.Background(), Params{"org": "core"}, DefaultList) + if err != nil { + t.Fatal(err) + } + if len(result.Items) != 1 || result.Items[0].Name != "go-forge" { + t.Errorf("unexpected result: %+v", result) + } +} + +func TestRepoService_Good_Get(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(types.Repository{Name: "go-forge", FullName: "core/go-forge"}) + })) + defer srv.Close() + + f := NewForge(srv.URL, "tok") + repo, err := f.Repos.Get(context.Background(), Params{"owner": "core", "repo": "go-forge"}) + if err != nil { + t.Fatal(err) + } + if repo.Name != "go-forge" { + t.Errorf("got name=%q", repo.Name) + } +} + +func TestRepoService_Good_Fork(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + t.Errorf("expected POST, got %s", r.Method) + } + w.WriteHeader(http.StatusAccepted) + json.NewEncoder(w).Encode(types.Repository{Name: "go-forge", Fork: true}) + })) + defer srv.Close() + + f := NewForge(srv.URL, "tok") + repo, err := f.Repos.Fork(context.Background(), "core", "go-forge", "my-org") + if err != nil { + t.Fatal(err) + } + if !repo.Fork { + t.Error("expected fork=true") + } +} +``` + +**Step 2: Run tests to verify they fail** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run "TestForge|TestRepoService"` +Expected: Compilation errors + +**Step 3: Write forge.go** + +```go +package forge + +import "forge.lthn.ai/core/go-forge/types" + +// Forge is the top-level client for the Forgejo API. +type Forge struct { + client *Client + + Repos *RepoService + Issues *IssueService + Pulls *PullService + Orgs *OrgService + Users *UserService + Teams *TeamService + Admin *AdminService + Branches *BranchService + Releases *ReleaseService + Labels *LabelService + Webhooks *WebhookService + Notifications *NotificationService + Packages *PackageService + Actions *ActionsService + Contents *ContentService + Wiki *WikiService + Misc *MiscService +} + +// NewForge creates a new Forge client. +func NewForge(url, token string, opts ...Option) *Forge { + c := NewClient(url, token, opts...) + f := &Forge{client: c} + f.Repos = newRepoService(c) + // Other services initialised in their respective tasks. + // Stub them here so tests compile: + f.Issues = &IssueService{} + f.Pulls = &PullService{} + f.Orgs = &OrgService{} + f.Users = &UserService{} + f.Teams = &TeamService{} + f.Admin = &AdminService{} + f.Branches = &BranchService{} + f.Releases = &ReleaseService{} + f.Labels = &LabelService{} + f.Webhooks = &WebhookService{} + f.Notifications = &NotificationService{} + f.Packages = &PackageService{} + f.Actions = &ActionsService{} + f.Contents = &ContentService{} + f.Wiki = &WikiService{} + f.Misc = &MiscService{} + return f +} + +// Client returns the underlying HTTP client. +func (f *Forge) Client() *Client { return f.client } +``` + +**Step 4: Write repos.go** + +```go +package forge + +import ( + "context" + + "forge.lthn.ai/core/go-forge/types" +) + +// RepoService handles repository operations. +type RepoService struct { + Resource[types.Repository, types.CreateRepoOption, types.EditRepoOption] +} + +func newRepoService(c *Client) *RepoService { + return &RepoService{ + Resource: *NewResource[types.Repository, types.CreateRepoOption, types.EditRepoOption]( + c, "/api/v1/repos/{owner}/{repo}", + ), + } +} + +// ListOrgRepos returns all repositories for an organisation. +func (s *RepoService) ListOrgRepos(ctx context.Context, org string) ([]types.Repository, error) { + return ListAll[types.Repository](ctx, s.client, "/api/v1/orgs/"+org+"/repos", nil) +} + +// ListUserRepos returns all repositories for the authenticated user. +func (s *RepoService) ListUserRepos(ctx context.Context) ([]types.Repository, error) { + return ListAll[types.Repository](ctx, s.client, "/api/v1/user/repos", nil) +} + +// Fork forks a repository. If org is non-empty, forks into that organisation. +func (s *RepoService) Fork(ctx context.Context, owner, repo, org string) (*types.Repository, error) { + body := map[string]string{} + if org != "" { + body["organization"] = org + } + var out types.Repository + err := s.client.Post(ctx, "/api/v1/repos/"+owner+"/"+repo+"/forks", body, &out) + if err != nil { + return nil, err + } + return &out, nil +} + +// Migrate imports a repository from an external service. +func (s *RepoService) Migrate(ctx context.Context, opts *types.MigrateRepoOptions) (*types.Repository, error) { + var out types.Repository + err := s.client.Post(ctx, "/api/v1/repos/migrate", opts, &out) + if err != nil { + return nil, err + } + return &out, nil +} + +// Transfer initiates a repository transfer. +func (s *RepoService) Transfer(ctx context.Context, owner, repo string, opts map[string]any) error { + return s.client.Post(ctx, "/api/v1/repos/"+owner+"/"+repo+"/transfer", opts, nil) +} + +// AcceptTransfer accepts a pending repository transfer. +func (s *RepoService) AcceptTransfer(ctx context.Context, owner, repo string) error { + return s.client.Post(ctx, "/api/v1/repos/"+owner+"/"+repo+"/transfer/accept", nil, nil) +} + +// RejectTransfer rejects a pending repository transfer. +func (s *RepoService) RejectTransfer(ctx context.Context, owner, repo string) error { + return s.client.Post(ctx, "/api/v1/repos/"+owner+"/"+repo+"/transfer/reject", nil, nil) +} + +// MirrorSync triggers a mirror sync. +func (s *RepoService) MirrorSync(ctx context.Context, owner, repo string) error { + return s.client.Post(ctx, "/api/v1/repos/"+owner+"/"+repo+"/mirror-sync", nil, nil) +} +``` + +**Step 5: Write stub service types** so `forge.go` compiles. Create `services_stub.go`: + +```go +package forge + +// Stub service types — replaced as each service is implemented. + +type IssueService struct{} +type PullService struct{} +type OrgService struct{} +type UserService struct{} +type TeamService struct{} +type AdminService struct{} +type BranchService struct{} +type ReleaseService struct{} +type LabelService struct{} +type WebhookService struct{} +type NotificationService struct{} +type PackageService struct{} +type ActionsService struct{} +type ContentService struct{} +type WikiService struct{} +type MiscService struct{} +``` + +**Step 6: Run tests** + +Run: `cd /Users/snider/Code/go-forge && go test -v -run "TestForge|TestRepoService"` +Expected: All tests PASS (if generated types compile — if `types.CreateRepoOption` or `types.MigrateRepoOptions` don't exist, adjust field names to match generated types) + +**Step 7: Commit** + +```bash +git add forge.go repos.go services_stub.go forge_test.go +git commit -m "feat: Forge client + RepoService with CRUD and actions + +Co-Authored-By: Virgil " +``` + +--- + +### Task 11: IssueService + PullService + +**Files:** +- Create: `issues.go` +- Create: `pulls.go` +- Create: `issues_test.go` +- Create: `pulls_test.go` +- Modify: `forge.go` (wire up services) +- Modify: `services_stub.go` (remove IssueService, PullService stubs) + +Follow the same pattern as Task 10. Key points: + +**IssueService** embeds `Resource[types.Issue, types.CreateIssueOption, types.EditIssueOption]`. +Path: `/api/v1/repos/{owner}/{repo}/issues/{index}` + +Action methods (9): +- `Pin(ctx, owner, repo, index)` — POST `.../issues/{index}/pin` +- `Unpin(ctx, owner, repo, index)` — DELETE `.../issues/{index}/pin` +- `SetDeadline(ctx, owner, repo, index, deadline)` — POST `.../issues/{index}/deadline` +- `AddReaction(ctx, owner, repo, index, reaction)` — POST `.../issues/{index}/reactions` +- `DeleteReaction(ctx, owner, repo, index, reaction)` — DELETE `.../issues/{index}/reactions` +- `StartStopwatch(ctx, owner, repo, index)` — POST `.../issues/{index}/stopwatch/start` +- `StopStopwatch(ctx, owner, repo, index)` — POST `.../issues/{index}/stopwatch/stop` +- `AddLabels(ctx, owner, repo, index, labelIDs)` — POST `.../issues/{index}/labels` +- `RemoveLabel(ctx, owner, repo, index, labelID)` — DELETE `.../issues/{index}/labels/{id}` +- `ListComments(ctx, owner, repo, index)` — GET `.../issues/{index}/comments` +- `CreateComment(ctx, owner, repo, index, body)` — POST `.../issues/{index}/comments` + +**PullService** embeds `Resource[types.PullRequest, types.CreatePullRequestOption, types.EditPullRequestOption]`. +Path: `/api/v1/repos/{owner}/{repo}/pulls/{index}` + +Action methods (6): +- `Merge(ctx, owner, repo, index, method)` — POST `.../pulls/{index}/merge` +- `Update(ctx, owner, repo, index)` — POST `.../pulls/{index}/update` +- `ListReviews(ctx, owner, repo, index)` — GET `.../pulls/{index}/reviews` +- `SubmitReview(ctx, owner, repo, index, reviewID)` — POST `.../pulls/{index}/reviews/{id}` +- `DismissReview(ctx, owner, repo, index, reviewID, msg)` — POST `.../pulls/{index}/reviews/{id}/dismissals` +- `UndismissReview(ctx, owner, repo, index, reviewID)` — POST `.../pulls/{index}/reviews/{id}/undismissals` + +Write tests for at least: List, Get, Create for each service + one action method each. + +Run: `cd /Users/snider/Code/go-forge && go test ./... -v` +Commit: `git commit -m "feat: IssueService and PullService with actions"` + +--- + +### Task 12: OrgService + TeamService + UserService + +**Files:** +- Create: `orgs.go`, `teams.go`, `users.go` +- Create: `orgs_test.go`, `teams_test.go`, `users_test.go` +- Modify: `forge.go` (wire up) +- Modify: `services_stub.go` (remove stubs) + +**OrgService** — `Resource[types.Organization, types.CreateOrgOption, types.EditOrgOption]` +Path: `/api/v1/orgs/{org}` +Actions: ListMembers, AddMember, RemoveMember, SetAvatar, Block, Unblock + +**TeamService** — `Resource[types.Team, types.CreateTeamOption, types.EditTeamOption]` +Path: `/api/v1/teams/{id}` +Actions: ListMembers, AddMember, RemoveMember, ListRepos, AddRepo, RemoveRepo + +**UserService** — `Resource[types.User, struct{}, struct{}]` (no create/edit via this path) +Path: `/api/v1/users/{username}` +Custom: `GetCurrent(ctx)`, `ListFollowers(ctx)`, `ListStarred(ctx)`, keys, GPG keys, settings + +Run: `cd /Users/snider/Code/go-forge && go test ./... -v` +Commit: `git commit -m "feat: OrgService, TeamService, UserService"` + +--- + +### Task 13: AdminService + +**Files:** +- Create: `admin.go` +- Create: `admin_test.go` +- Modify: `forge.go` (wire up) +- Modify: `services_stub.go` (remove stub) + +**AdminService** — No generic Resource (admin endpoints are heterogeneous). +Direct methods: +- `ListUsers(ctx)` — GET `/api/v1/admin/users` +- `CreateUser(ctx, opts)` — POST `/api/v1/admin/users` +- `EditUser(ctx, username, opts)` — PATCH `/api/v1/admin/users/{username}` +- `DeleteUser(ctx, username)` — DELETE `/api/v1/admin/users/{username}` +- `RenameUser(ctx, username, newName)` — POST `.../users/{username}/rename` +- `ListOrgs(ctx)` — GET `/api/v1/admin/orgs` +- `RunCron(ctx, task)` — POST `/api/v1/admin/cron/{task}` +- `ListCron(ctx)` — GET `/api/v1/admin/cron` +- `AdoptRepo(ctx, owner, repo)` — POST `.../unadopted/{owner}/{repo}` +- `GenerateRunnerToken(ctx)` — POST `/api/v1/admin/runners/registration-token` + +Run: `cd /Users/snider/Code/go-forge && go test ./... -v` +Commit: `git commit -m "feat: AdminService with user, org, cron, runner operations"` + +--- + +## Wave 4: Extended Services (Tasks 14-17) + +### Task 14: BranchService + ReleaseService + +**BranchService** — `Resource[types.Branch, types.CreateBranchRepoOption, struct{}]` +Path: `/api/v1/repos/{owner}/{repo}/branches/{branch}` +Additional: BranchProtection CRUD at `.../branch_protections/{name}` + +**ReleaseService** — `Resource[types.Release, types.CreateReleaseOption, types.EditReleaseOption]` +Path: `/api/v1/repos/{owner}/{repo}/releases/{id}` +Additional: Asset upload/download at `.../releases/{id}/assets` + +### Task 15: LabelService + WebhookService + ContentService + +**LabelService** — Handles repo labels, org labels, and issue labels. +- `ListRepoLabels(ctx, owner, repo)` +- `CreateRepoLabel(ctx, owner, repo, opts)` +- `ListOrgLabels(ctx, org)` + +**WebhookService** — `Resource[types.Hook, types.CreateHookOption, types.EditHookOption]` +Actions: `TestHook(ctx, owner, repo, id)` + +**ContentService** — File read/write via API +- `GetFile(ctx, owner, repo, path)` — GET `.../contents/{path}` +- `CreateFile(ctx, owner, repo, path, opts)` — POST `.../contents/{path}` +- `UpdateFile(ctx, owner, repo, path, opts)` — PUT `.../contents/{path}` +- `DeleteFile(ctx, owner, repo, path, opts)` — DELETE `.../contents/{path}` + +### Task 16: ActionsService + NotificationService + PackageService + +**ActionsService** — runners, secrets, variables, workflow dispatch +- Repo-level: `.../repos/{owner}/{repo}/actions/{secrets,variables,runners}` +- Org-level: `.../orgs/{org}/actions/{secrets,variables,runners}` +- `DispatchWorkflow(ctx, owner, repo, workflow, opts)` + +**NotificationService** — list, mark read +- `List(ctx)` — GET `/api/v1/notifications` +- `MarkRead(ctx)` — PUT `/api/v1/notifications` +- `GetThread(ctx, id)` — GET `.../notifications/threads/{id}` + +**PackageService** — list, get, delete +- `List(ctx, owner)` — GET `/api/v1/packages/{owner}` +- `Get(ctx, owner, type, name, version)` — GET `.../packages/{owner}/{type}/{name}/{version}` + +### Task 17: WikiService + MiscService + CommitService + +**WikiService** — pages +- `ListPages(ctx, owner, repo)` +- `GetPage(ctx, owner, repo, pageName)` +- `CreatePage(ctx, owner, repo, opts)` +- `EditPage(ctx, owner, repo, pageName, opts)` +- `DeletePage(ctx, owner, repo, pageName)` + +**MiscService** — markdown, licenses, gitignore, nodeinfo +- `RenderMarkdown(ctx, text, mode)` — POST `/api/v1/markdown` +- `ListLicenses(ctx)` — GET `/api/v1/licenses` +- `ListGitignoreTemplates(ctx)` — GET `/api/v1/gitignore/templates` +- `NodeInfo(ctx)` — GET `/api/v1/nodeinfo` + +**CommitService** — status and notes +- `GetCombinedStatus(ctx, owner, repo, ref)` +- `CreateStatus(ctx, owner, repo, sha, opts)` +- `SetNote(ctx, owner, repo, sha, opts)` + +For each task in Wave 4: write tests first, implement, verify all tests pass, commit. + +Run after each task: `cd /Users/snider/Code/go-forge && go test ./... -v` + +--- + +## Wave 5: Clean Up + Services Stub Removal (Task 18) + +### Task 18: Remove stubs + final wiring + +**Files:** +- Delete: `services_stub.go` +- Modify: `forge.go` — replace all stub initialisations with real `newXxxService(c)` calls + +**Step 1: Remove services_stub.go** + +Delete the file. All service types should now be defined in their own files. + +**Step 2: Wire all services in forge.go** + +Update `NewForge()` to call `newXxxService(c)` for every service. + +**Step 3: Run all tests** + +Run: `cd /Users/snider/Code/go-forge && go test ./... -v -count=1` +Expected: All tests pass + +**Step 4: Commit** + +```bash +git add -A +git commit -m "feat: wire all 17 services, remove stubs + +Co-Authored-By: Virgil " +``` + +--- + +## Wave 6: Integration + Forge Repo Setup (Tasks 19-20) + +### Task 19: Create Forge repo + push + +**Step 1: Create repo on Forge** + +Use the Forgejo API or web UI to create `core/go-forge` on `forge.lthn.ai`. + +**Step 2: Add remote and push** + +```bash +cd /Users/snider/Code/go-forge +git remote add forge ssh://git@forge.lthn.ai:2223/core/go-forge.git +git push -u forge main +``` + +### Task 20: Wiki documentation (go-ai treatment) + +Create wiki pages for go-forge on Forge, matching the go-ai documentation pattern: + +1. **Home** — Overview, install, quick start +2. **Architecture** — Generic Resource[T,C,U], codegen pipeline, service pattern +3. **Services** — All 17 services with example usage +4. **Code Generation** — How to regenerate types, upgrade Forgejo version +5. **Configuration** — Env vars, config file, flags +6. **Error Handling** — APIError, IsNotFound, IsForbidden +7. **Development** — Contributing, testing, releasing + +Use the Forge wiki API: `POST /api/v1/repos/core/go-forge/wiki/new` with `{"content_base64":"...","title":"..."}`. + +--- + +## Dependency Sequencing + +``` +Task 1 (scaffold) ← Task 2 (client) ← Task 3 (pagination) ← Task 4 (params) ← Task 5 (resource) +Task 1 ← Task 7 (parser) ← Task 8 (generator) ← Task 9 (generate types) +Task 5 + Task 9 ← Task 6 (config) ← Task 10 (forge + repos) +Task 10 ← Task 11 (issues + PRs) +Task 10 ← Task 12 (orgs + teams + users) +Task 10 ← Task 13 (admin) +Task 10 ← Task 14-17 (extended services) +Task 14-17 ← Task 18 (remove stubs) +Task 18 ← Task 19 (forge push) +Task 19 ← Task 20 (wiki) +``` + +**Wave 1 (Tasks 1-6)**: Foundation — all independent once scaffolded +**Wave 2 (Tasks 7-9)**: Codegen — sequential (parser → generator → run) +**Wave 3 (Tasks 10-13)**: Core services — Task 10 first (creates Forge + stubs), then 11-13 parallel +**Wave 4 (Tasks 14-17)**: Extended services — all parallel after Task 10 +**Wave 5 (Task 18)**: Clean up — after all services done +**Wave 6 (Tasks 19-20)**: Ship — after clean up + +## Verification + +After all tasks: + +1. `cd /Users/snider/Code/go-forge && go test ./... -count=1` — all pass +2. `go build ./...` — compiles cleanly +3. `go vet ./...` — no issues +4. Verify `types/` contains generated files with `Repository`, `Issue`, `PullRequest`, etc. +5. Verify `NewForge()` creates client with all 17 services populated +6. Verify action methods exist (Fork, Merge, Pin, etc.) diff --git a/docs/plans/completed/cli-meta-package.md b/docs/plans/completed/cli-meta-package.md new file mode 100644 index 0000000..d88672b --- /dev/null +++ b/docs/plans/completed/cli-meta-package.md @@ -0,0 +1,30 @@ +# CLI Meta-Package Restructure — Completed + +**Completed:** 22 Feb 2026 + +## What Was Done + +`pkg/cli` was extracted from `core/go` into its own Go module at `forge.lthn.ai/core/cli`. This made the CLI SDK a first-class, independently versioned package rather than a subdirectory of the Go foundation repo. + +Following the extraction, an ecosystem-wide import path migration updated all consumers from the old path to the new one: + +- Old: `forge.lthn.ai/core/go/pkg/cli` +- New: `forge.lthn.ai/core/cli/pkg/cli` + +## Scope + +- **147+ files** updated across **10 repos** +- All repos build clean after migration + +## Repos Migrated + +`core/cli`, `core/go`, `go-devops`, `go-ai`, `go-agentic`, `go-crypt`, `go-rag`, `go-scm`, `go-api`, `go-update` + +## Key Outcomes + +- `forge.lthn.ai/core/cli/pkg/cli` is the single import for all CLI concerns across the ecosystem +- Domain repos are insulated from cobra, lipgloss, and bubbletea — only `pkg/cli` imports them +- Command registration uses the Core framework lifecycle via `cli.WithCommands()` — no `init()`, no global state +- `core/cli` is a thin assembly repo (~2K LOC) with 7 meta packages; all business logic lives in domain repos +- Variant binary pattern established: multiple `main.go` files can wire different `WithCommands` sets for targeted binaries (core-ci, core-mlx, core-ops, etc.) +- Command migration from the old `core/cli` monolith to domain repos was completed in full (13 command groups moved) diff --git a/docs/plans/completed/cli-sdk-expansion.md b/docs/plans/completed/cli-sdk-expansion.md new file mode 100644 index 0000000..a0a84a3 --- /dev/null +++ b/docs/plans/completed/cli-sdk-expansion.md @@ -0,0 +1,39 @@ +# CLI SDK Expansion — Completion Summary + +**Completed:** 21 February 2026 +**Module:** `forge.lthn.ai/core/go/pkg/cli` (later migrated to `forge.lthn.ai/core/cli`) +**Status:** Complete — all TUI primitives shipped, then extracted to core/cli + +## What Was Built + +Extended `pkg/cli` with charmbracelet TUI primitives so domain repos only +import `core/cli` for all CLI concerns. Charmbracelet dependencies (bubbletea, +bubbles, lipgloss) are encapsulated behind our own types. + +### Components added + +| Component | File | Purpose | +|-----------|------|---------| +| RunTUI | `runtui.go` | Escape hatch with `Model`/`Msg`/`Cmd`/`KeyMsg` types | +| Spinner | `spinner.go` | Async handle with `Update()`, `Done()`, `Fail()` | +| ProgressBar | `progressbar.go` | `Increment()`, `Set()`, `SetMessage()`, `Done()` | +| InteractiveList | `list.go` | Keyboard navigation with terminal fallback | +| TextInput | `textinput.go` | Placeholder, masking, validation | +| Viewport | `viewport.go` | Scrollable content for logs, diffs, docs | +| Form (stub) | `form.go` | Interface defined, bufio fallback | +| FilePicker (stub) | `filepicker.go` | Interface defined, bufio fallback | +| Tabs (stub) | `tabs.go` | Interface defined, simple fallback | + +### Subsequent migration + +On 22 February 2026, `pkg/cli` was extracted from `core/go` into its own +module at `forge.lthn.ai/core/cli` and all imports were updated. The TUI +primitives now live in the standalone CLI module. + +### Frame upgrade (follow-on) + +The Frame layout system was upgraded to implement `tea.Model` directly on +22 February 2026 (in `core/cli`), adding bubbletea lifecycle, `KeyMap` for +configurable bindings, `Navigate()`/`Back()` for panel switching, and +lipgloss-based HLCRF rendering. This was a separate plan +(`frame-bubbletea`) that built on the SDK expansion. diff --git a/docs/plans/completed/go-api.md b/docs/plans/completed/go-api.md new file mode 100644 index 0000000..86278a2 --- /dev/null +++ b/docs/plans/completed/go-api.md @@ -0,0 +1,57 @@ +# go-api — Completion Summary + +**Completed:** 21 February 2026 +**Module:** `forge.lthn.ai/core/go-api` +**Status:** Phases 1–3 complete, 176 tests passing + +## What Was Built + +### Phase 1 — Core Framework (20 Feb 2026) + +Gin-based HTTP engine with extensible middleware via `With*()` options. Key components: + +- `RouteGroup` / `StreamGroup` interfaces — subsystems register their own endpoints +- `Response[T]` envelope — `OK()`, `Fail()`, `Paginated()` generics +- `Engine` — `New()`, `Register()`, `Handler()`, `Serve()` with graceful shutdown +- Bearer auth, request ID, and CORS middleware +- WebSocket endpoint wrapping a `go-ws` Hub +- Swagger UI at `/swagger/` with runtime spec serving +- `/health` endpoint always available without auth +- First integration proof in `go-ml/api/` (3 endpoints, 12 tests) + +### Phase 2 — Gin Plugin Stack (20–21 Feb 2026) + +17 middleware plugins added across four waves, all as drop-in `With*()` options: + +| Wave | Plugins | +|------|---------| +| 1 — Gateway hardening | Authentik (OIDC + forward auth), secure headers, structured slog, timeouts, gzip, static files | +| 2 — Performance + auth | Brotli compression, in-memory response cache, server-side sessions, Casbin RBAC | +| 3 — Network + streaming | HTTP signature verification, SSE broker, reverse proxy detection, i18n locale, GraphQL | +| 4 — Observability | pprof, expvar, OpenTelemetry distributed tracing | + +### Phase 3 — OpenAPI + SDK Codegen (21 Feb 2026) + +Runtime spec generation (not swaggo annotations — incompatible with dynamic RouteGroups and `Response[T]` generics): + +- `DescribableGroup` interface — opt-in OpenAPI metadata for route groups +- `ToolBridge` — converts MCP tool descriptors into `POST /{tool_name}` REST endpoints +- `SpecBuilder` — assembles full OpenAPI 3.1 JSON from registered groups at runtime +- Spec export to JSON and YAML (`core api spec`) +- SDK codegen wrapper for openapi-generator-cli, 11 languages (`core api sdk --lang go`) +- `go-ai` `mcp/registry.go` — generic `addToolRecorded[In,Out]` captures types in closures +- `go-ai` `mcp/bridge.go` — `BridgeToAPI()` populates ToolBridge from MCP tool registry +- CLI commands: `core api spec`, `core api sdk` (in `core/cli` dev branch) + +## Key Outcomes + +- **176 tests** across go-api (143), go-ai bridge (10), and CLI commands (4), all passing +- Zero internal ecosystem dependencies — subsystems import go-api, not the reverse +- Authentik (OIDC) and bearer token auth coexist; Casbin adds RBAC on top +- Four-protocol access pattern established: REST, GraphQL, WebSocket, MCP — same handlers + +## Known Limitations + +- Subsystem MCP tools registered via `mcp.AddTool` directly are excluded from the REST bridge (only the 10 built-in tools appear). Fix: pass `*Service` to `RegisterTools` instead of `*mcp.Server`. +- `structSchema` reflection handles flat structs only; nested structs are not recursed. +- `core api spec` currently emits a spec with only `/health`; full MCP wiring into the CLI command is pending. diff --git a/docs/plans/completed/mcp-integration.md b/docs/plans/completed/mcp-integration.md new file mode 100644 index 0000000..7edf86e --- /dev/null +++ b/docs/plans/completed/mcp-integration.md @@ -0,0 +1,37 @@ +# MCP Integration — Completion Summary + +**Completed:** 2026-02-05 +**Plan:** `docs/plans/2026-02-05-mcp-integration.md` + +## What Was Built + +### RAG Tools (`pkg/mcp/tools_rag.go`) +Three MCP tools added to the existing `pkg/mcp` server: +- `rag_query` — semantic search against Qdrant vector DB +- `rag_ingest` — ingest a file or directory into a named collection +- `rag_collections` — list available Qdrant collections (with optional stats) + +### Metrics Tools (`pkg/mcp/tools_metrics.go`) +Two MCP tools for agent activity tracking: +- `metrics_record` — write a typed event (agent_id, repo, arbitrary data) to JSONL storage +- `metrics_query` — query events with aggregation by type, repo, and agent; supports human-friendly duration strings (7d, 24h) + +Also added `parseDuration()` helper for "Nd"/"Nh"/"Nm" duration strings. + +### `core mcp serve` Command (`internal/cmd/mcpcmd/cmd_mcp.go`) +New CLI sub-command registered via `cli.WithCommands()` (not `init()`). +- Runs `pkg/mcp` server over stdio by default +- TCP mode via `MCP_ADDR=:9000` environment variable +- `--workspace` flag to restrict file operations to a directory + +Registered in the full CLI variant. i18n strings added for all user-facing text. + +### Plugin Configuration +`.mcp.json` created for the `agentic-flows` Claude Code plugin, pointing to `core mcp serve`. Exposes all 15 tools to Claude Code agents via the `core-cli` MCP server name. + +## Key Outcomes + +- `core mcp serve` is the single entry point for all MCP tooling (file ops, RAG, metrics, language detection, process management, WebSocket, webview/CDP) +- MCP command moved to `go-ai/cmd/mcpcmd/` in final form; the plan's `internal/cmd/mcpcmd/` path reflects the pre-extraction location +- Registration pattern updated from `init()` + `RegisterCommands()` to `cli.WithCommands()` lifecycle hooks +- Services required at runtime: Qdrant (localhost:6333), Ollama with nomic-embed-text (localhost:11434) diff --git a/docs/plans/completed/qk-bone-orientation.md b/docs/plans/completed/qk-bone-orientation.md new file mode 100644 index 0000000..0cfcaa9 --- /dev/null +++ b/docs/plans/completed/qk-bone-orientation.md @@ -0,0 +1,62 @@ +# Q/K Bone Orientation — Completion Summary + +**Completed:** 23 February 2026 +**Repos:** go-inference, go-mlx, go-ml, LEM +**Status:** All 7 tasks complete, 14 files changed (+917 lines), all tests passing + +## What Was Built + +### go-inference — AttentionSnapshot types (Task 1) + +`AttentionSnapshot` struct and `AttentionInspector` optional interface. Backends expose attention data via type assertion — no breaking changes to `TextModel`. + +### go-mlx — KV cache extraction (Task 2) + +`InspectAttention` on `metalAdapter` runs a single prefill pass and extracts post-RoPE K vectors from each layer's KV cache. Tested against real Gemma3-1B (26 layers, 1 KV head via GQA, 256 head dim). + +### go-ml — Adapter pass-through (Task 3) + +`InspectAttention` on `InferenceAdapter` type-asserts the underlying `TextModel` to `AttentionInspector`. Returns clear error for unsupported backends. + +### LEM — Analysis engine (Task 4) + +Pure Go CPU math in `pkg/lem/attention.go`. Computes 5 BO metrics from raw K tensors: + +- **Mean Coherence** — pairwise cosine similarity of K vectors within each layer +- **Cross-Layer Alignment** — cosine similarity of mean K vectors between adjacent layers +- **Head Entropy** — normalised Shannon entropy of K vector magnitudes across positions +- **Phase-Lock Score** — fraction of head pairs above coherence threshold (0.7) +- **Joint Collapse Count** — layers where cross-alignment drops below threshold (0.5) + +Composite score: 30% coherence + 25% cross-alignment + 20% phase-lock + 15% entropy + 10% joint stability → 0-100 scale. + +### LEM — CLI command (Task 5) + +`lem score attention -model -prompt [-json]` loads a model, runs InspectAttention, and prints BO metrics. + +### LEM — Distill integration (Task 6) + +Opt-in attention scoring in the distill pipeline. Gated behind `scorer.attention: true` and `scorer.attention_min_score` in ai.yaml. Costs one extra prefill per probe. + +### LEM — Feature vectors (Task 7) + +19D full feature vector: 6D grammar + 8D heuristic + 5D attention (`mean_coherence`, `cross_alignment`, `head_entropy`, `phase_lock`, `joint_stability`). Ready for Poindexter KDTree spatial indexing. + +## Key Decisions + +- **Optional interface** — `AttentionInspector` via type assertion, not added to `TextModel` +- **Named `BOResult`** — avoids collision with `metal.AttentionResult` in go-mlx +- **Opt-in for distill** — extra prefill per probe is expensive, off by default +- **Pure Go analysis** — zero CGO deps in the analysis engine; GPU data extracted once via `.Floats()` + +## Commits + +| Repo | SHA | Message | +|------|-----|---------| +| go-inference | `0f7263f` | feat: add AttentionInspector optional interface | +| go-mlx | `c2177f7` | feat: implement AttentionInspector via KV cache extraction | +| go-ml | `45e9fed` | feat: add InspectAttention pass-through | +| LEM | `28309b2` | feat: add Q/K Bone Orientation analysis engine | +| LEM | `e333192` | feat: add 'lem score attention' CLI | +| LEM | `fbc636e` | feat: integrate attention scoring into distill pipeline | +| LEM | `b621baa` | feat: add 19D full feature vector |