docs: add ecosystem overview and historical design plans
Moved from core/go during docs cleanup — these belong with the CLI that orchestrates the ecosystem, not the DI framework. - ecosystem.md: full module inventory and dependency graph - 3 active plans (authentik-traefik, core-help design/plan) - 13 completed design plans (MCP, go-api, cli-meta, go-forge, etc.) Co-Authored-By: Virgil <virgil@lethean.io>
This commit is contained in:
Snider
parent
37a8ae8d31
commit
aa83cf77cc
17 changed files with 10422 additions and 0 deletions
457
docs/ecosystem.md
Normal file
457
docs/ecosystem.md
Normal file
|
|
@ -0,0 +1,457 @@
|
||||||
|
# Core Go Ecosystem
|
||||||
|
|
||||||
|
The Core Go ecosystem is a set of 19 standalone Go modules that form the infrastructure backbone for the host-uk platform and the Lethean network. All modules are hosted under the `forge.lthn.ai/core/` organisation. Each module has its own repository, independent versioning, and a `docs/` directory.
|
||||||
|
|
||||||
|
The CLI framework documented in the rest of this site (`forge.lthn.ai/core/cli`) is one node in this graph. The satellite packages listed here are separate repositories that the CLI imports or that stand alone as libraries.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Module Index
|
||||||
|
|
||||||
|
| Package | Module Path | Managed By |
|
||||||
|
|---------|-------------|-----------|
|
||||||
|
| [go-inference](#go-inference) | `forge.lthn.ai/core/go-inference` | Virgil |
|
||||||
|
| [go-mlx](#go-mlx) | `forge.lthn.ai/core/go-mlx` | Virgil |
|
||||||
|
| [go-rocm](#go-rocm) | `forge.lthn.ai/core/go-rocm` | Charon |
|
||||||
|
| [go-ml](#go-ml) | `forge.lthn.ai/core/go-ml` | Virgil |
|
||||||
|
| [go-ai](#go-ai) | `forge.lthn.ai/core/go-ai` | Virgil |
|
||||||
|
| [go-agentic](#go-agentic) | `forge.lthn.ai/core/go-agentic` | Charon |
|
||||||
|
| [go-rag](#go-rag) | `forge.lthn.ai/core/go-rag` | Charon |
|
||||||
|
| [go-i18n](#go-i18n) | `forge.lthn.ai/core/go-i18n` | Virgil |
|
||||||
|
| [go-html](#go-html) | `forge.lthn.ai/core/go-html` | Charon |
|
||||||
|
| [go-crypt](#go-crypt) | `forge.lthn.ai/core/go-crypt` | Virgil |
|
||||||
|
| [go-scm](#go-scm) | `forge.lthn.ai/core/go-scm` | Charon |
|
||||||
|
| [go-p2p](#go-p2p) | `forge.lthn.ai/core/go-p2p` | Charon |
|
||||||
|
| [go-devops](#go-devops) | `forge.lthn.ai/core/go-devops` | Virgil |
|
||||||
|
| [go-help](#go-help) | `forge.lthn.ai/core/go-help` | Charon |
|
||||||
|
| [go-ratelimit](#go-ratelimit) | `forge.lthn.ai/core/go-ratelimit` | Charon |
|
||||||
|
| [go-session](#go-session) | `forge.lthn.ai/core/go-session` | Charon |
|
||||||
|
| [go-store](#go-store) | `forge.lthn.ai/core/go-store` | Charon |
|
||||||
|
| [go-ws](#go-ws) | `forge.lthn.ai/core/go-ws` | Charon |
|
||||||
|
| [go-webview](#go-webview) | `forge.lthn.ai/core/go-webview` | Charon |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dependency Graph
|
||||||
|
|
||||||
|
The graph below shows import relationships. An arrow `A → B` means A imports B.
|
||||||
|
|
||||||
|
```
|
||||||
|
go-inference (no dependencies — foundation contract)
|
||||||
|
↑
|
||||||
|
├── go-mlx (CGO, Apple Silicon Metal GPU)
|
||||||
|
├── go-rocm (AMD ROCm, llama-server subprocess)
|
||||||
|
└── go-ml (scoring engine, backends, orchestrator)
|
||||||
|
↑
|
||||||
|
└── go-ai (MCP hub, 49 tools)
|
||||||
|
↑
|
||||||
|
└── go-agentic (service lifecycle, allowances)
|
||||||
|
|
||||||
|
go-rag (Qdrant + Ollama, standalone)
|
||||||
|
↑
|
||||||
|
└── go-ai
|
||||||
|
|
||||||
|
go-i18n (grammar engine, standalone; Phase 2a imports go-mlx)
|
||||||
|
|
||||||
|
go-crypt (standalone)
|
||||||
|
↑
|
||||||
|
├── go-p2p (UEPS wire protocol)
|
||||||
|
└── go-scm (AgentCI dispatch)
|
||||||
|
|
||||||
|
go-store (SQLite KV, standalone)
|
||||||
|
↑
|
||||||
|
├── go-ratelimit (sliding window limiter)
|
||||||
|
├── go-session (transcript parser)
|
||||||
|
└── go-agentic
|
||||||
|
|
||||||
|
go-ws (WebSocket hub, standalone)
|
||||||
|
↑
|
||||||
|
└── go-ai
|
||||||
|
|
||||||
|
go-webview (CDP client, standalone)
|
||||||
|
↑
|
||||||
|
└── go-ai
|
||||||
|
|
||||||
|
go-html (DOM compositor, standalone)
|
||||||
|
|
||||||
|
go-help (help catalogue, standalone)
|
||||||
|
|
||||||
|
go-devops (Ansible, build, infrastructure — imports go-scm)
|
||||||
|
```
|
||||||
|
|
||||||
|
The CLI framework (`forge.lthn.ai/core/cli`) has internal equivalents of several of these packages (`pkg/rag`, `pkg/ws`, `pkg/webview`, `pkg/i18n`) that were developed in parallel. The satellite packages are the canonical standalone versions intended for use outside the CLI binary.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Package Descriptions
|
||||||
|
|
||||||
|
### go-inference
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-inference`
|
||||||
|
|
||||||
|
Zero-dependency interface package that defines the common contract for all inference backends in the ecosystem:
|
||||||
|
|
||||||
|
- `TextModel` — the top-level model interface (`Generate`, `Stream`, `Close`)
|
||||||
|
- `Backend` — hardware/runtime abstraction (Metal, ROCm, CPU, remote)
|
||||||
|
- `Token` — streaming token type with metadata
|
||||||
|
|
||||||
|
No concrete implementations live here. Any package that needs to call inference without depending on a specific hardware library imports `go-inference` and receives an implementation at runtime.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-mlx
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-mlx`
|
||||||
|
|
||||||
|
Native Metal GPU inference for Apple Silicon using CGO bindings to `mlx-c` (the C API for Apple's MLX framework). Implements the `go-inference` interfaces.
|
||||||
|
|
||||||
|
Build requirements:
|
||||||
|
- macOS 13+ (Ventura) on Apple Silicon
|
||||||
|
- `mlx-c` installed (`brew install mlx`)
|
||||||
|
- CGO enabled: `CGO_CFLAGS` and `CGO_LDFLAGS` must reference the mlx-c headers and library
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Loads GGUF and MLX-format models
|
||||||
|
- Streaming token generation directly on GPU
|
||||||
|
- Quantised model support (Q4, Q8)
|
||||||
|
- Phase 4 backend abstraction in progress — will allow hot-swapping backends at runtime
|
||||||
|
|
||||||
|
Local path: `/Users/snider/Code/go-mlx`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-rocm
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-rocm`
|
||||||
|
|
||||||
|
AMD ROCm GPU inference for Linux. Rather than using CGO, this package manages a `llama-server` subprocess (from llama.cpp) compiled with ROCm support and communicates over its HTTP API.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Subprocess lifecycle management (start, health-check, restart on crash)
|
||||||
|
- OpenAI-compatible HTTP client wrapping llama-server's API
|
||||||
|
- Implements `go-inference` interfaces
|
||||||
|
- Targeted at the homelab RX 7800 XT running Ubuntu 24.04
|
||||||
|
|
||||||
|
Managed by Charon (Linux homelab).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-ml
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-ml`
|
||||||
|
|
||||||
|
Scoring engine, backend registry, and agent orchestration layer. The hub that connects models from `go-mlx`, `go-rocm`, and future backends into a unified interface.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Backend registry: register multiple inference backends, select by capability
|
||||||
|
- Scoring pipeline: evaluate model outputs against rubrics
|
||||||
|
- Agent orchestrator: coordinate multi-step inference tasks
|
||||||
|
- ~3.5K LOC
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-ai
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-ai`
|
||||||
|
|
||||||
|
MCP (Model Context Protocol) server hub with 49 registered tools. Acts as the primary facade for AI capabilities in the ecosystem.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- 49 MCP tools covering file operations, RAG, metrics, process management, WebSocket, and CDP/webview
|
||||||
|
- Imports `go-ml`, `go-rag`, `go-mlx`
|
||||||
|
- Can run as stdio MCP server or TCP MCP server
|
||||||
|
- AI usage metrics recorded to JSONL
|
||||||
|
|
||||||
|
Run the MCP server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# stdio (for Claude Desktop / Claude Code)
|
||||||
|
core mcp serve
|
||||||
|
|
||||||
|
# TCP
|
||||||
|
MCP_ADDR=:9000 core mcp serve
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-agentic
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-agentic`
|
||||||
|
|
||||||
|
Service lifecycle and allowance management for autonomous agents. Handles:
|
||||||
|
|
||||||
|
- Agent session tracking and state persistence
|
||||||
|
- Allowance system: budget constraints on tool calls, token usage, and wall-clock time
|
||||||
|
- Integration with `go-store` for persistence
|
||||||
|
- REST client for the PHP `core-agentic` backend
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-rag
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-rag`
|
||||||
|
|
||||||
|
Retrieval-Augmented Generation pipeline using Qdrant for vector storage and Ollama for embeddings.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- `ChunkMarkdown`: semantic splitting by H2 headers and paragraphs with overlap
|
||||||
|
- `Ingest`: crawl a directory of Markdown files, embed, and store in Qdrant
|
||||||
|
- `Query`: semantic search returning ranked `QueryResult` slices
|
||||||
|
- `FormatResultsContext`: formats results as XML tags for LLM prompt injection
|
||||||
|
- Clients: `QdrantClient` and `OllamaClient` wrapping their respective Go SDKs
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-i18n
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-i18n`
|
||||||
|
|
||||||
|
Grammar engine for natural-language generation. Goes beyond key-value lookup tables to handle pluralisation, verb conjugation, past tense, gerunds, and semantic sentence construction ("Subject verbed object").
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- `T(key, args...)` — main translation function
|
||||||
|
- `S(noun, value)` — semantic subject with grammatical context
|
||||||
|
- Language rules defined in JSON; algorithmic fallbacks for irregular verbs
|
||||||
|
- **GrammarImprint**: a linguistic hash (reversal of the grammar engine) used as a semantic fingerprint — part of the Lethean identity verification stack
|
||||||
|
- Phase 2a (imports `go-mlx` for language model-assisted reversal) currently blocked on `go-mlx` Phase 4
|
||||||
|
|
||||||
|
Local path: `/Users/snider/Code/go-i18n`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-html
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-html`
|
||||||
|
|
||||||
|
HLCRF DOM compositor — a programmatic HTML/DOM construction library targeting both server-side rendering and WASM (browser).
|
||||||
|
|
||||||
|
HLCRF stands for Header, Left, Content, Right, Footer — the region layout model used throughout the CLI's terminal UI and web rendering layer.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Composable region-based layout (mirrors the terminal `Composite` in `pkg/cli`)
|
||||||
|
- WASM build target: runs in the browser without JavaScript
|
||||||
|
- Used by the LEM Chat UI and web SDK generation
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-crypt
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-crypt`
|
||||||
|
|
||||||
|
Cryptographic primitives, authentication, and trust policy enforcement.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Password hashing (Argon2id with tuned parameters)
|
||||||
|
- Symmetric encryption (ChaCha20-Poly1305, AES-GCM)
|
||||||
|
- Key derivation (HKDF, Scrypt)
|
||||||
|
- OpenPGP challenge-response authentication
|
||||||
|
- Trust policies: define and evaluate access rules
|
||||||
|
- Foundation for the UEPS (User-controlled Encryption Policy System) wire protocol in `go-p2p`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-scm
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-scm`
|
||||||
|
|
||||||
|
Source control management and CI integration, including the AgentCI dispatch system.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Forgejo and Gitea API clients (typed wrappers)
|
||||||
|
- GitHub integration via the `gh` CLI
|
||||||
|
- `AgentCI`: dispatches AI work items to agent runners over SSH using Charm stack libraries (`soft-serve`, `keygen`, `melt`, `wishlist`)
|
||||||
|
- PR lifecycle management: create, review, merge, label
|
||||||
|
- JSONL job journal for audit trails
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-p2p
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-p2p`
|
||||||
|
|
||||||
|
Peer-to-peer mesh networking implementing the UEPS (User-controlled Encryption Policy System) wire protocol.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- UEPS: consent-gated TLV frames with Ed25519 consent tokens and an Intent-Broker
|
||||||
|
- Peer discovery and mesh routing
|
||||||
|
- Encrypted relay transport
|
||||||
|
- Integration with `go-crypt` for all cryptographic operations
|
||||||
|
|
||||||
|
This is a core component of the Lethean Web3 network layer.
|
||||||
|
|
||||||
|
Managed by Charon (Linux homelab).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-devops
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-devops`
|
||||||
|
|
||||||
|
Infrastructure automation, build tooling, and release pipeline utilities, intended as a standalone library form of what the Core CLI provides as commands.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Ansible-lite engine (native Go SSH playbook execution)
|
||||||
|
- LinuxKit image building and VM lifecycle
|
||||||
|
- Multi-target binary build and release
|
||||||
|
- Integration with `go-scm` for repository operations
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-help
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-help`
|
||||||
|
|
||||||
|
Embedded documentation catalogue with full-text search and an optional HTTP server for serving help content.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- YAML-frontmatter Markdown topic parsing
|
||||||
|
- In-memory reverse index with title/heading/body scoring
|
||||||
|
- Snippet extraction with keyword highlighting
|
||||||
|
- `HTTP server` mode: serve the catalogue as a documentation site
|
||||||
|
- Used by the `core pkg search` command and the `pkg/help` package inside the CLI
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-ratelimit
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-ratelimit`
|
||||||
|
|
||||||
|
Sliding-window rate limiter with a SQLite persistence backend.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Token bucket and sliding-window algorithms
|
||||||
|
- SQLite backend via `go-store` for durable rate state across restarts
|
||||||
|
- HTTP middleware helper
|
||||||
|
- Used by `go-ai` and `go-agentic` to enforce per-agent API quotas
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-session
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-session`
|
||||||
|
|
||||||
|
Claude Code JSONL transcript parser and visualisation toolkit (standalone version of `pkg/session` inside the CLI).
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- `ParseTranscript(path)`: reads `.jsonl` session files and reconstructs tool use timelines
|
||||||
|
- `ListSessions(dir)`: scans a Claude projects directory for session files
|
||||||
|
- `Search(dir, query)`: full-text search across sessions
|
||||||
|
- `RenderHTML(sess, path)`: single-file HTML visualisation
|
||||||
|
- `RenderMP4(sess, path)`: terminal video replay via VHS
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-store
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-store`
|
||||||
|
|
||||||
|
SQLite-backed key-value store with reactive change notification.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- `Get`, `Set`, `Delete`, `List` over typed keys
|
||||||
|
- `Watch(key, handler)`: register a callback that fires on change
|
||||||
|
- `OnChange(handler)`: subscribe to all changes
|
||||||
|
- Used by `go-ratelimit`, `go-session`, and `go-agentic` for lightweight persistence
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-ws
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-ws`
|
||||||
|
|
||||||
|
WebSocket hub with channel-based subscriptions and an optional Redis pub/sub bridge for multi-instance deployments.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Hub pattern: central registry of connected clients
|
||||||
|
- Channel routing: `SendToChannel(topic, msg)` delivers only to subscribers
|
||||||
|
- Redis bridge: publish messages from one instance, receive on all
|
||||||
|
- HTTP handler: `hub.Handler()` for embedding in any Go HTTP server
|
||||||
|
- `SendProcessOutput(id, line)`: convenience method for streaming process logs
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### go-webview
|
||||||
|
|
||||||
|
**Module:** `forge.lthn.ai/core/go-webview`
|
||||||
|
|
||||||
|
Chrome DevTools Protocol (CDP) client for browser automation, testing, and AI-driven web interaction (standalone version of `pkg/webview` inside the CLI).
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Navigation, click, type, screenshot
|
||||||
|
- `Evaluate(script)`: arbitrary JavaScript execution with result capture
|
||||||
|
- Console capture and filtering
|
||||||
|
- Angular-aware helpers: `WaitForAngular()`, `GetNgModel(selector)`
|
||||||
|
- `ActionSequence`: chain interactions into a single call
|
||||||
|
- Used by `go-ai` to expose browser tools to MCP agents
|
||||||
|
|
||||||
|
Managed by Charon.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Forge Repository Paths
|
||||||
|
|
||||||
|
All repositories are hosted at `forge.lthn.ai` (Forgejo). SSH access uses port 2223:
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-inference.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-mlx.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-rocm.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-ml.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-ai.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-agentic.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-rag.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-i18n.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-html.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-crypt.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-scm.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-p2p.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-devops.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-help.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-ratelimit.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-session.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-store.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-ws.git
|
||||||
|
ssh://git@forge.lthn.ai:2223/core/go-webview.git
|
||||||
|
```
|
||||||
|
|
||||||
|
HTTPS authentication is not available on Forge. Always use SSH remotes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Go Workspace Setup
|
||||||
|
|
||||||
|
The satellite packages can be used together in a Go workspace. After cloning the repositories you need:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go work init
|
||||||
|
go work use ./go-inference ./go-mlx ./go-rag ./go-ai # add as needed
|
||||||
|
go work sync
|
||||||
|
```
|
||||||
|
|
||||||
|
The CLI repository already uses a Go workspace that includes `cmd/core-gui`, `cmd/bugseti`, and `cmd/examples/*`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
- [index.md](index.md) — Main documentation hub
|
||||||
|
- [getting-started.md](getting-started.md) — CLI installation
|
||||||
|
- [configuration.md](configuration.md) — `repos.yaml` registry format
|
||||||
1163
docs/plans/2026-02-20-authentik-traefik-plan.md
Normal file
1163
docs/plans/2026-02-20-authentik-traefik-plan.md
Normal file
File diff suppressed because it is too large
Load diff
155
docs/plans/2026-02-21-core-help-design.md
Normal file
155
docs/plans/2026-02-21-core-help-design.md
Normal file
|
|
@ -0,0 +1,155 @@
|
||||||
|
# core.help Documentation Website — Design
|
||||||
|
|
||||||
|
**Date:** 2026-02-21
|
||||||
|
**Author:** Virgil
|
||||||
|
**Status:** Design approved
|
||||||
|
**Domain:** https://core.help
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Documentation is scattered across 39 repos (18 Go packages, 20 PHP packages, 1 CLI). There is no unified docs site. Developers need a single entry point to find CLI commands, Go package APIs, MCP tool references, and PHP module guides.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
A Hugo + Docsy static site at core.help, built from existing markdown docs aggregated by `core docs sync`. No new content — just collect and present what already exists across the ecosystem.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Stack
|
||||||
|
|
||||||
|
- **Hugo** — Go-native static site generator, sub-second builds
|
||||||
|
- **Docsy theme** — Purpose-built for technical docs (used by Kubernetes, gRPC, Knative)
|
||||||
|
- **BunnyCDN** — Static hosting with pull zone
|
||||||
|
- **`core docs sync --target hugo`** — Collects markdown from all repos into Hugo content tree
|
||||||
|
|
||||||
|
### Why Hugo + Docsy (not VitePress or mdBook)
|
||||||
|
|
||||||
|
- Go-native, no Node.js dependency
|
||||||
|
- Handles multi-section navigation (CLI, Go packages, PHP modules, MCP tools)
|
||||||
|
- Sub-second builds for ~250 markdown files
|
||||||
|
- Docsy has built-in search, versioned nav, API reference sections
|
||||||
|
|
||||||
|
## Content Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
docs-site/
|
||||||
|
├── hugo.toml
|
||||||
|
├── content/
|
||||||
|
│ ├── _index.md # Landing page
|
||||||
|
│ ├── getting-started/ # CLI top-level guides
|
||||||
|
│ │ ├── _index.md
|
||||||
|
│ │ ├── installation.md
|
||||||
|
│ │ ├── configuration.md
|
||||||
|
│ │ ├── user-guide.md
|
||||||
|
│ │ ├── troubleshooting.md
|
||||||
|
│ │ └── faq.md
|
||||||
|
│ ├── cli/ # CLI command reference (43 commands)
|
||||||
|
│ │ ├── _index.md
|
||||||
|
│ │ ├── dev/ # core dev commit, push, pull, etc.
|
||||||
|
│ │ ├── ai/ # core ai commands
|
||||||
|
│ │ ├── go/ # core go test, lint, etc.
|
||||||
|
│ │ └── ...
|
||||||
|
│ ├── go/ # Go ecosystem packages (18)
|
||||||
|
│ │ ├── _index.md # Ecosystem overview
|
||||||
|
│ │ ├── go-api/ # README + architecture/development/history
|
||||||
|
│ │ ├── go-ai/
|
||||||
|
│ │ ├── go-mlx/
|
||||||
|
│ │ ├── go-i18n/
|
||||||
|
│ │ └── ...
|
||||||
|
│ ├── mcp/ # MCP tool reference (49 tools)
|
||||||
|
│ │ ├── _index.md
|
||||||
|
│ │ ├── file-operations.md
|
||||||
|
│ │ ├── process-management.md
|
||||||
|
│ │ ├── rag.md
|
||||||
|
│ │ └── ...
|
||||||
|
│ ├── php/ # PHP packages (from core-php/docs/packages/)
|
||||||
|
│ │ ├── _index.md
|
||||||
|
│ │ ├── admin/
|
||||||
|
│ │ ├── tenant/
|
||||||
|
│ │ ├── commerce/
|
||||||
|
│ │ └── ...
|
||||||
|
│ └── kb/ # Knowledge base (wiki pages from go-mlx, go-i18n)
|
||||||
|
│ ├── _index.md
|
||||||
|
│ ├── mlx/
|
||||||
|
│ └── i18n/
|
||||||
|
├── static/ # Logos, favicons
|
||||||
|
├── layouts/ # Custom template overrides (minimal)
|
||||||
|
└── go.mod # Hugo modules (Docsy as module dep)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sync Pipeline
|
||||||
|
|
||||||
|
`core docs sync --target hugo --output site/content/` performs:
|
||||||
|
|
||||||
|
### Source Mapping
|
||||||
|
|
||||||
|
```
|
||||||
|
cli/docs/index.md → content/getting-started/_index.md
|
||||||
|
cli/docs/getting-started.md → content/getting-started/installation.md
|
||||||
|
cli/docs/user-guide.md → content/getting-started/user-guide.md
|
||||||
|
cli/docs/configuration.md → content/getting-started/configuration.md
|
||||||
|
cli/docs/troubleshooting.md → content/getting-started/troubleshooting.md
|
||||||
|
cli/docs/faq.md → content/getting-started/faq.md
|
||||||
|
|
||||||
|
core/docs/cmd/**/*.md → content/cli/**/*.md
|
||||||
|
|
||||||
|
go-*/README.md → content/go/{name}/_index.md
|
||||||
|
go-*/docs/*.md → content/go/{name}/*.md
|
||||||
|
go-*/KB/*.md → content/kb/{name-suffix}/*.md
|
||||||
|
|
||||||
|
core-*/docs/**/*.md → content/php/{name-suffix}/**/*.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### Front Matter Injection
|
||||||
|
|
||||||
|
If a markdown file doesn't start with `---`, prepend:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
title: "{derived from filename}"
|
||||||
|
linkTitle: "{short name}"
|
||||||
|
weight: {auto-incremented}
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
No other content transformations. Markdown stays as-is.
|
||||||
|
|
||||||
|
### Build & Deploy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core docs sync --target hugo --output docs-site/content/
|
||||||
|
cd docs-site && hugo build
|
||||||
|
hugo deploy --target bunnycdn
|
||||||
|
```
|
||||||
|
|
||||||
|
Hugo deploy config in `hugo.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[deployment]
|
||||||
|
[[deployment.targets]]
|
||||||
|
name = "bunnycdn"
|
||||||
|
URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto"
|
||||||
|
```
|
||||||
|
|
||||||
|
Credentials via env vars.
|
||||||
|
|
||||||
|
## Registry
|
||||||
|
|
||||||
|
All 39 repos registered in `.core/repos.yaml` with `docs: true`. Go repos use explicit `path:` fields since they live outside the PHP `base_path`. `FindRegistry()` checks `.core/repos.yaml` alongside `repos.yaml`.
|
||||||
|
|
||||||
|
## Prerequisites Completed
|
||||||
|
|
||||||
|
- [x] `.core/repos.yaml` created with all 39 repos
|
||||||
|
- [x] `FindRegistry()` updated to find `.core/repos.yaml`
|
||||||
|
- [x] `Repo.Path` supports explicit YAML override
|
||||||
|
- [x] go-api docs gap filled (architecture.md, development.md, history.md)
|
||||||
|
- [x] All 18 Go repos have standard docs trio
|
||||||
|
|
||||||
|
## What Remains (Implementation Plan)
|
||||||
|
|
||||||
|
1. Create docs-site repo with Hugo + Docsy scaffold
|
||||||
|
2. Extend `core docs sync` with `--target hugo` mode
|
||||||
|
3. Write section _index.md files (landing page, section intros)
|
||||||
|
4. Hugo config (navigation, search, theme colours)
|
||||||
|
5. BunnyCDN deployment config
|
||||||
|
6. CI pipeline on Forge (optional — can deploy manually initially)
|
||||||
642
docs/plans/2026-02-21-core-help-plan.md
Normal file
642
docs/plans/2026-02-21-core-help-plan.md
Normal file
|
|
@ -0,0 +1,642 @@
|
||||||
|
# core.help Hugo Documentation Site — Implementation Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||||
|
|
||||||
|
**Goal:** Build a Hugo + Docsy documentation site at core.help that aggregates markdown from 39 repos via `core docs sync --target hugo`.
|
||||||
|
|
||||||
|
**Architecture:** Hugo static site with Docsy theme, populated by extending `core docs sync` with a `--target hugo` flag that maps repo docs into Hugo's `content/` tree with auto-injected front matter. Deploy to BunnyCDN.
|
||||||
|
|
||||||
|
**Tech Stack:** Hugo (Go SSG), Docsy theme (Hugo module), BunnyCDN, `core docs sync` CLI
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The docs sync command lives in `/Users/snider/Code/host-uk/cli/cmd/docs/`. The site will be scaffolded at `/Users/snider/Code/host-uk/docs-site/`. The registry at `/Users/snider/Code/host-uk/.core/repos.yaml` already contains all 39 repos (20 PHP + 18 Go + 1 CLI) with explicit paths for Go repos.
|
||||||
|
|
||||||
|
Key files:
|
||||||
|
- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go` — sync command (modify)
|
||||||
|
- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go` — repo scanner (modify)
|
||||||
|
- `/Users/snider/Code/host-uk/docs-site/` — Hugo site (create)
|
||||||
|
|
||||||
|
## Task 1: Scaffold Hugo + Docsy site
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/hugo.toml`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/go.mod`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/_index.md`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/getting-started/_index.md`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/cli/_index.md`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/go/_index.md`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/mcp/_index.md`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/php/_index.md`
|
||||||
|
- Create: `/Users/snider/Code/host-uk/docs-site/content/kb/_index.md`
|
||||||
|
|
||||||
|
This is the one-time Hugo scaffolding. No tests — just files.
|
||||||
|
|
||||||
|
**`hugo.toml`:**
|
||||||
|
```toml
|
||||||
|
baseURL = "https://core.help/"
|
||||||
|
title = "Core Documentation"
|
||||||
|
languageCode = "en"
|
||||||
|
defaultContentLanguage = "en"
|
||||||
|
|
||||||
|
enableRobotsTXT = true
|
||||||
|
enableGitInfo = false
|
||||||
|
|
||||||
|
[outputs]
|
||||||
|
home = ["HTML", "JSON"]
|
||||||
|
section = ["HTML"]
|
||||||
|
|
||||||
|
[params]
|
||||||
|
description = "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools"
|
||||||
|
copyright = "Host UK — EUPL-1.2"
|
||||||
|
|
||||||
|
[params.ui]
|
||||||
|
sidebar_menu_compact = true
|
||||||
|
breadcrumb_disable = false
|
||||||
|
sidebar_search_disable = false
|
||||||
|
navbar_logo = false
|
||||||
|
|
||||||
|
[params.ui.readingtime]
|
||||||
|
enable = false
|
||||||
|
|
||||||
|
[module]
|
||||||
|
proxy = "direct"
|
||||||
|
|
||||||
|
[module.hugoVersion]
|
||||||
|
extended = true
|
||||||
|
min = "0.120.0"
|
||||||
|
|
||||||
|
[[module.imports]]
|
||||||
|
path = "github.com/google/docsy"
|
||||||
|
disable = false
|
||||||
|
|
||||||
|
[markup.goldmark.renderer]
|
||||||
|
unsafe = true
|
||||||
|
|
||||||
|
[menu]
|
||||||
|
[[menu.main]]
|
||||||
|
name = "Getting Started"
|
||||||
|
weight = 10
|
||||||
|
url = "/getting-started/"
|
||||||
|
[[menu.main]]
|
||||||
|
name = "CLI Reference"
|
||||||
|
weight = 20
|
||||||
|
url = "/cli/"
|
||||||
|
[[menu.main]]
|
||||||
|
name = "Go Packages"
|
||||||
|
weight = 30
|
||||||
|
url = "/go/"
|
||||||
|
[[menu.main]]
|
||||||
|
name = "MCP Tools"
|
||||||
|
weight = 40
|
||||||
|
url = "/mcp/"
|
||||||
|
[[menu.main]]
|
||||||
|
name = "PHP Packages"
|
||||||
|
weight = 50
|
||||||
|
url = "/php/"
|
||||||
|
[[menu.main]]
|
||||||
|
name = "Knowledge Base"
|
||||||
|
weight = 60
|
||||||
|
url = "/kb/"
|
||||||
|
```
|
||||||
|
|
||||||
|
**`go.mod`:**
|
||||||
|
```
|
||||||
|
module github.com/host-uk/docs-site
|
||||||
|
|
||||||
|
go 1.22
|
||||||
|
|
||||||
|
require github.com/google/docsy v0.11.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: Run `hugo mod get` after creating these files to populate `go.sum` and download Docsy.
|
||||||
|
|
||||||
|
**Section `_index.md` files** — each needs Hugo front matter:
|
||||||
|
|
||||||
|
`content/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "Core Documentation"
|
||||||
|
description: "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools"
|
||||||
|
---
|
||||||
|
|
||||||
|
Welcome to the Core ecosystem documentation.
|
||||||
|
|
||||||
|
## Sections
|
||||||
|
|
||||||
|
- [Getting Started](/getting-started/) — Installation, configuration, and first steps
|
||||||
|
- [CLI Reference](/cli/) — Command reference for `core` CLI
|
||||||
|
- [Go Packages](/go/) — Go ecosystem package documentation
|
||||||
|
- [MCP Tools](/mcp/) — Model Context Protocol tool reference
|
||||||
|
- [PHP Packages](/php/) — PHP module documentation
|
||||||
|
- [Knowledge Base](/kb/) — Wiki articles and deep dives
|
||||||
|
```
|
||||||
|
|
||||||
|
`content/getting-started/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "Getting Started"
|
||||||
|
linkTitle: "Getting Started"
|
||||||
|
weight: 10
|
||||||
|
description: "Installation, configuration, and first steps with the Core CLI"
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
`content/cli/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "CLI Reference"
|
||||||
|
linkTitle: "CLI Reference"
|
||||||
|
weight: 20
|
||||||
|
description: "Command reference for the core CLI tool"
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
`content/go/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "Go Packages"
|
||||||
|
linkTitle: "Go Packages"
|
||||||
|
weight: 30
|
||||||
|
description: "Documentation for the Go ecosystem packages"
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
`content/mcp/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "MCP Tools"
|
||||||
|
linkTitle: "MCP Tools"
|
||||||
|
weight: 40
|
||||||
|
description: "Model Context Protocol tool reference — file operations, RAG, ML inference, process management"
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
`content/php/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "PHP Packages"
|
||||||
|
linkTitle: "PHP Packages"
|
||||||
|
weight: 50
|
||||||
|
description: "Documentation for the PHP module ecosystem"
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
`content/kb/_index.md`:
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
title: "Knowledge Base"
|
||||||
|
linkTitle: "Knowledge Base"
|
||||||
|
weight: 60
|
||||||
|
description: "Wiki articles, deep dives, and reference material"
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify:** After creating files, run from `/Users/snider/Code/host-uk/docs-site/`:
|
||||||
|
```bash
|
||||||
|
hugo mod get
|
||||||
|
hugo server
|
||||||
|
```
|
||||||
|
The site should start and show the landing page with Docsy theme at `localhost:1313`.
|
||||||
|
|
||||||
|
**Commit:**
|
||||||
|
```bash
|
||||||
|
cd /Users/snider/Code/host-uk/docs-site
|
||||||
|
git init
|
||||||
|
git add .
|
||||||
|
git commit -m "feat: scaffold Hugo + Docsy documentation site"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Extend scanRepoDocs to collect KB/ and README
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go`
|
||||||
|
|
||||||
|
Currently `scanRepoDocs` only collects files from `docs/`. For the Hugo target we also need:
|
||||||
|
- `KB/**/*.md` files (wiki pages from go-mlx, go-i18n)
|
||||||
|
- `README.md` content (becomes the package _index.md)
|
||||||
|
|
||||||
|
Add a `KBFiles []string` field to `RepoDocInfo` and scan `KB/` alongside `docs/`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type RepoDocInfo struct {
|
||||||
|
Name string
|
||||||
|
Path string
|
||||||
|
HasDocs bool
|
||||||
|
Readme string
|
||||||
|
ClaudeMd string
|
||||||
|
Changelog string
|
||||||
|
DocsFiles []string // All files in docs/ directory (recursive)
|
||||||
|
KBFiles []string // All files in KB/ directory (recursive)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
In `scanRepoDocs`, after the `docs/` walk, add a second walk for `KB/`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Recursively scan KB/ directory for .md files
|
||||||
|
kbDir := filepath.Join(repo.Path, "KB")
|
||||||
|
if _, err := io.Local.List(kbDir); err == nil {
|
||||||
|
_ = filepath.WalkDir(kbDir, func(path string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if d.IsDir() || !strings.HasSuffix(d.Name(), ".md") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
relPath, _ := filepath.Rel(kbDir, path)
|
||||||
|
info.KBFiles = append(info.KBFiles, relPath)
|
||||||
|
info.HasDocs = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tests:** The existing tests should still pass. No new test file needed — this is a data-collection change.
|
||||||
|
|
||||||
|
**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...`
|
||||||
|
|
||||||
|
**Commit:**
|
||||||
|
```bash
|
||||||
|
git add cmd/docs/cmd_scan.go
|
||||||
|
git commit -m "feat(docs): scan KB/ directory alongside docs/"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Add `--target hugo` flag and Hugo sync logic
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go`
|
||||||
|
|
||||||
|
This is the main task. Add a `--target` flag (default `"php"`) and a new `runHugoSync` function that maps repos to Hugo's content tree.
|
||||||
|
|
||||||
|
**Add flag variable and registration:**
|
||||||
|
|
||||||
|
```go
|
||||||
|
var (
|
||||||
|
docsSyncRegistryPath string
|
||||||
|
docsSyncDryRun bool
|
||||||
|
docsSyncOutputDir string
|
||||||
|
docsSyncTarget string
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
docsSyncCmd.Flags().StringVar(&docsSyncRegistryPath, "registry", "", i18n.T("common.flag.registry"))
|
||||||
|
docsSyncCmd.Flags().BoolVar(&docsSyncDryRun, "dry-run", false, i18n.T("cmd.docs.sync.flag.dry_run"))
|
||||||
|
docsSyncCmd.Flags().StringVar(&docsSyncOutputDir, "output", "", i18n.T("cmd.docs.sync.flag.output"))
|
||||||
|
docsSyncCmd.Flags().StringVar(&docsSyncTarget, "target", "php", "Target format: php (default) or hugo")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Update RunE to pass target:**
|
||||||
|
```go
|
||||||
|
RunE: func(cmd *cli.Command, args []string) error {
|
||||||
|
return runDocsSync(docsSyncRegistryPath, docsSyncOutputDir, docsSyncDryRun, docsSyncTarget)
|
||||||
|
},
|
||||||
|
```
|
||||||
|
|
||||||
|
**Update `runDocsSync` signature and add target dispatch:**
|
||||||
|
```go
|
||||||
|
func runDocsSync(registryPath string, outputDir string, dryRun bool, target string) error {
|
||||||
|
reg, basePath, err := loadRegistry(registryPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
switch target {
|
||||||
|
case "hugo":
|
||||||
|
return runHugoSync(reg, basePath, outputDir, dryRun)
|
||||||
|
default:
|
||||||
|
return runPHPSync(reg, basePath, outputDir, dryRun)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rename current sync body to `runPHPSync`** — extract lines 67-159 of current `runDocsSync` into `runPHPSync(reg, basePath, outputDir string, dryRun bool) error`. This is a pure extract, no logic changes.
|
||||||
|
|
||||||
|
**Add `hugoOutputName` mapping function:**
|
||||||
|
```go
|
||||||
|
// hugoOutputName maps repo name to Hugo content section and folder.
|
||||||
|
// Returns (section, folder) where section is the top-level content dir.
|
||||||
|
func hugoOutputName(repoName string) (string, string) {
|
||||||
|
// CLI guides
|
||||||
|
if repoName == "cli" {
|
||||||
|
return "getting-started", ""
|
||||||
|
}
|
||||||
|
// Core CLI command docs
|
||||||
|
if repoName == "core" {
|
||||||
|
return "cli", ""
|
||||||
|
}
|
||||||
|
// Go packages
|
||||||
|
if strings.HasPrefix(repoName, "go-") {
|
||||||
|
return "go", repoName
|
||||||
|
}
|
||||||
|
// PHP packages
|
||||||
|
if strings.HasPrefix(repoName, "core-") {
|
||||||
|
return "php", strings.TrimPrefix(repoName, "core-")
|
||||||
|
}
|
||||||
|
return "go", repoName
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Add front matter injection helper:**
|
||||||
|
```go
|
||||||
|
// injectFrontMatter prepends Hugo front matter to markdown content if missing.
|
||||||
|
func injectFrontMatter(content []byte, title string, weight int) []byte {
|
||||||
|
// Already has front matter
|
||||||
|
if bytes.HasPrefix(bytes.TrimSpace(content), []byte("---")) {
|
||||||
|
return content
|
||||||
|
}
|
||||||
|
fm := fmt.Sprintf("---\ntitle: %q\nweight: %d\n---\n\n", title, weight)
|
||||||
|
return append([]byte(fm), content...)
|
||||||
|
}
|
||||||
|
|
||||||
|
// titleFromFilename derives a human-readable title from a filename.
|
||||||
|
func titleFromFilename(filename string) string {
|
||||||
|
name := strings.TrimSuffix(filepath.Base(filename), ".md")
|
||||||
|
name = strings.ReplaceAll(name, "-", " ")
|
||||||
|
name = strings.ReplaceAll(name, "_", " ")
|
||||||
|
// Title case
|
||||||
|
words := strings.Fields(name)
|
||||||
|
for i, w := range words {
|
||||||
|
if len(w) > 0 {
|
||||||
|
words[i] = strings.ToUpper(w[:1]) + w[1:]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return strings.Join(words, " ")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Add `runHugoSync` function:**
|
||||||
|
```go
|
||||||
|
func runHugoSync(reg *repos.Registry, basePath string, outputDir string, dryRun bool) error {
|
||||||
|
if outputDir == "" {
|
||||||
|
outputDir = filepath.Join(basePath, "docs-site", "content")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scan all repos
|
||||||
|
var docsInfo []RepoDocInfo
|
||||||
|
for _, repo := range reg.List() {
|
||||||
|
if repo.Name == "core-template" || repo.Name == "core-claude" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
info := scanRepoDocs(repo)
|
||||||
|
if info.HasDocs {
|
||||||
|
docsInfo = append(docsInfo, info)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(docsInfo) == 0 {
|
||||||
|
cli.Text("No documentation found")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cli.Print("\n Hugo sync: %d repos with docs → %s\n\n", len(docsInfo), outputDir)
|
||||||
|
|
||||||
|
// Show plan
|
||||||
|
for _, info := range docsInfo {
|
||||||
|
section, folder := hugoOutputName(info.Name)
|
||||||
|
target := section
|
||||||
|
if folder != "" {
|
||||||
|
target = section + "/" + folder
|
||||||
|
}
|
||||||
|
fileCount := len(info.DocsFiles) + len(info.KBFiles)
|
||||||
|
if info.Readme != "" {
|
||||||
|
fileCount++
|
||||||
|
}
|
||||||
|
cli.Print(" %s → %s/ (%d files)\n", repoNameStyle.Render(info.Name), target, fileCount)
|
||||||
|
}
|
||||||
|
|
||||||
|
if dryRun {
|
||||||
|
cli.Print("\n Dry run — no files written\n")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cli.Blank()
|
||||||
|
if !confirm("Sync to Hugo content directory?") {
|
||||||
|
cli.Text("Aborted")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cli.Blank()
|
||||||
|
var synced int
|
||||||
|
for _, info := range docsInfo {
|
||||||
|
section, folder := hugoOutputName(info.Name)
|
||||||
|
|
||||||
|
// Build destination path
|
||||||
|
destDir := filepath.Join(outputDir, section)
|
||||||
|
if folder != "" {
|
||||||
|
destDir = filepath.Join(destDir, folder)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy docs/ files
|
||||||
|
weight := 10
|
||||||
|
docsDir := filepath.Join(info.Path, "docs")
|
||||||
|
for _, f := range info.DocsFiles {
|
||||||
|
src := filepath.Join(docsDir, f)
|
||||||
|
dst := filepath.Join(destDir, f)
|
||||||
|
if err := copyWithFrontMatter(src, dst, weight); err != nil {
|
||||||
|
cli.Print(" %s %s: %s\n", errorStyle.Render("✗"), f, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
weight += 10
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy README.md as _index.md (if not CLI/core which use their own index)
|
||||||
|
if info.Readme != "" && folder != "" {
|
||||||
|
dst := filepath.Join(destDir, "_index.md")
|
||||||
|
if err := copyWithFrontMatter(info.Readme, dst, 1); err != nil {
|
||||||
|
cli.Print(" %s README: %s\n", errorStyle.Render("✗"), err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy KB/ files to kb/{suffix}/
|
||||||
|
if len(info.KBFiles) > 0 {
|
||||||
|
// Extract suffix: go-mlx → mlx, go-i18n → i18n
|
||||||
|
suffix := strings.TrimPrefix(info.Name, "go-")
|
||||||
|
kbDestDir := filepath.Join(outputDir, "kb", suffix)
|
||||||
|
kbDir := filepath.Join(info.Path, "KB")
|
||||||
|
kbWeight := 10
|
||||||
|
for _, f := range info.KBFiles {
|
||||||
|
src := filepath.Join(kbDir, f)
|
||||||
|
dst := filepath.Join(kbDestDir, f)
|
||||||
|
if err := copyWithFrontMatter(src, dst, kbWeight); err != nil {
|
||||||
|
cli.Print(" %s KB/%s: %s\n", errorStyle.Render("✗"), f, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
kbWeight += 10
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
cli.Print(" %s %s\n", successStyle.Render("✓"), info.Name)
|
||||||
|
synced++
|
||||||
|
}
|
||||||
|
|
||||||
|
cli.Print("\n Synced %d repos to Hugo content\n", synced)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// copyWithFrontMatter copies a markdown file, injecting front matter if missing.
|
||||||
|
func copyWithFrontMatter(src, dst string, weight int) error {
|
||||||
|
if err := io.Local.EnsureDir(filepath.Dir(dst)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
content, err := io.Local.Read(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
title := titleFromFilename(src)
|
||||||
|
result := injectFrontMatter([]byte(content), title, weight)
|
||||||
|
return io.Local.Write(dst, string(result))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Add imports** at top of file:
|
||||||
|
```go
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"forge.lthn.ai/core/go/pkg/cli"
|
||||||
|
"forge.lthn.ai/core/go/pkg/i18n"
|
||||||
|
"forge.lthn.ai/core/go/pkg/io"
|
||||||
|
"forge.lthn.ai/core/go/pkg/repos"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...`
|
||||||
|
|
||||||
|
**Commit:**
|
||||||
|
```bash
|
||||||
|
git add cmd/docs/cmd_sync.go
|
||||||
|
git commit -m "feat(docs): add --target hugo sync mode for core.help"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Test the full pipeline
|
||||||
|
|
||||||
|
**No code changes.** Run the pipeline end-to-end.
|
||||||
|
|
||||||
|
**Step 1:** Sync docs to Hugo:
|
||||||
|
```bash
|
||||||
|
cd /Users/snider/Code/host-uk
|
||||||
|
core docs sync --target hugo --dry-run
|
||||||
|
```
|
||||||
|
Verify all 39 repos appear with correct section mappings.
|
||||||
|
|
||||||
|
**Step 2:** Run actual sync:
|
||||||
|
```bash
|
||||||
|
core docs sync --target hugo
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3:** Build and preview:
|
||||||
|
```bash
|
||||||
|
cd /Users/snider/Code/host-uk/docs-site
|
||||||
|
hugo server
|
||||||
|
```
|
||||||
|
Open `localhost:1313` and verify:
|
||||||
|
- Landing page renders with section links
|
||||||
|
- Getting Started section has CLI guides
|
||||||
|
- CLI Reference section has command docs
|
||||||
|
- Go Packages section has 18 packages with architecture/development/history
|
||||||
|
- PHP Packages section has PHP module docs
|
||||||
|
- Knowledge Base has MLX and i18n wiki pages
|
||||||
|
- Navigation works, search works
|
||||||
|
|
||||||
|
**Step 4:** Fix any issues found during preview.
|
||||||
|
|
||||||
|
**Commit docs-site content:**
|
||||||
|
```bash
|
||||||
|
cd /Users/snider/Code/host-uk/docs-site
|
||||||
|
git add content/
|
||||||
|
git commit -m "feat: sync initial content from 39 repos"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: BunnyCDN deployment config
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `/Users/snider/Code/host-uk/docs-site/hugo.toml`
|
||||||
|
|
||||||
|
Add deployment target:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[deployment]
|
||||||
|
[[deployment.targets]]
|
||||||
|
name = "production"
|
||||||
|
URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto"
|
||||||
|
```
|
||||||
|
|
||||||
|
Add a `Taskfile.yml` for convenience:
|
||||||
|
|
||||||
|
**Create:** `/Users/snider/Code/host-uk/docs-site/Taskfile.yml`
|
||||||
|
```yaml
|
||||||
|
version: '3'
|
||||||
|
|
||||||
|
tasks:
|
||||||
|
dev:
|
||||||
|
desc: Start Hugo dev server
|
||||||
|
cmds:
|
||||||
|
- hugo server --buildDrafts
|
||||||
|
|
||||||
|
build:
|
||||||
|
desc: Build static site
|
||||||
|
cmds:
|
||||||
|
- hugo --minify
|
||||||
|
|
||||||
|
sync:
|
||||||
|
desc: Sync docs from all repos
|
||||||
|
dir: ..
|
||||||
|
cmds:
|
||||||
|
- core docs sync --target hugo
|
||||||
|
|
||||||
|
deploy:
|
||||||
|
desc: Build and deploy to BunnyCDN
|
||||||
|
cmds:
|
||||||
|
- task: sync
|
||||||
|
- task: build
|
||||||
|
- hugo deploy --target production
|
||||||
|
|
||||||
|
clean:
|
||||||
|
desc: Remove generated content (keeps _index.md files)
|
||||||
|
cmds:
|
||||||
|
- find content -name "*.md" ! -name "_index.md" -delete
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify:** `task dev` starts the site.
|
||||||
|
|
||||||
|
**Commit:**
|
||||||
|
```bash
|
||||||
|
git add hugo.toml Taskfile.yml
|
||||||
|
git commit -m "feat: add BunnyCDN deployment config and Taskfile"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dependency Sequencing
|
||||||
|
|
||||||
|
```
|
||||||
|
Task 1 (Hugo scaffold) — independent, do first
|
||||||
|
Task 2 (scan KB/) — independent, can parallel with Task 1
|
||||||
|
Task 3 (--target hugo) — depends on Task 2
|
||||||
|
Task 4 (test pipeline) — depends on Tasks 1 + 3
|
||||||
|
Task 5 (deploy config) — depends on Task 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
After all tasks:
|
||||||
|
1. `core docs sync --target hugo` populates `docs-site/content/` from all repos
|
||||||
|
2. `cd docs-site && hugo server` renders the full site
|
||||||
|
3. Navigation has 6 sections: Getting Started, CLI, Go, MCP, PHP, KB
|
||||||
|
4. All existing markdown renders correctly with auto-injected front matter
|
||||||
|
5. `hugo build` produces `public/` with no errors
|
||||||
851
docs/plans/completed/2026-02-05-mcp-integration-original.md
Normal file
851
docs/plans/completed/2026-02-05-mcp-integration-original.md
Normal file
|
|
@ -0,0 +1,851 @@
|
||||||
|
# MCP Integration Implementation Plan
|
||||||
|
|
||||||
|
> **Status:** Completed. MCP command now lives in `go-ai/cmd/mcpcmd/`. Code examples below use the old `init()` + `RegisterCommands()` pattern — the current approach uses `cli.WithCommands()` (see cli-meta-package-design.md).
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||||
|
|
||||||
|
**Goal:** Add `core mcp serve` command with RAG and metrics tools, then configure the agentic-flows plugin to use it.
|
||||||
|
|
||||||
|
**Architecture:** Create a new `mcp` command package that starts the pkg/mcp server with extended tools. RAG tools call the existing exported functions in internal/cmd/rag. Metrics tools call pkg/ai directly. The agentic-flows plugin gets a `.mcp.json` that spawns `core mcp serve`.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.25, github.com/modelcontextprotocol/go-sdk/mcp, pkg/rag, pkg/ai
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Add RAG tools to pkg/mcp
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `pkg/mcp/tools_rag.go`
|
||||||
|
- Modify: `pkg/mcp/mcp.go:99-101` (registerTools)
|
||||||
|
- Test: `pkg/mcp/tools_rag_test.go`
|
||||||
|
|
||||||
|
**Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Create `pkg/mcp/tools_rag_test.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRAGQueryTool_Good(t *testing.T) {
|
||||||
|
// This test verifies the tool is registered and callable.
|
||||||
|
// It doesn't require Qdrant/Ollama running - just checks structure.
|
||||||
|
s, err := New(WithWorkspaceRoot(""))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check that rag_query tool is registered
|
||||||
|
tools := s.Server().ListTools()
|
||||||
|
found := false
|
||||||
|
for _, tool := range tools {
|
||||||
|
if tool.Name == "rag_query" {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Error("rag_query tool not registered")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRAGQueryInput_Good(t *testing.T) {
|
||||||
|
input := RAGQueryInput{
|
||||||
|
Question: "how do I deploy?",
|
||||||
|
Collection: "hostuk-docs",
|
||||||
|
TopK: 5,
|
||||||
|
}
|
||||||
|
if input.Question == "" {
|
||||||
|
t.Error("Question should not be empty")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test -run TestRAGQueryTool ./pkg/mcp/... -v`
|
||||||
|
Expected: FAIL with "rag_query tool not registered"
|
||||||
|
|
||||||
|
**Step 3: Create tools_rag.go with types and tool registration**
|
||||||
|
|
||||||
|
Create `pkg/mcp/tools_rag.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
ragcmd "forge.lthn.ai/core/cli/internal/cmd/rag"
|
||||||
|
"forge.lthn.ai/core/cli/pkg/rag"
|
||||||
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RAG tool input/output types
|
||||||
|
|
||||||
|
// RAGQueryInput contains parameters for querying the vector database.
|
||||||
|
type RAGQueryInput struct {
|
||||||
|
Question string `json:"question"`
|
||||||
|
Collection string `json:"collection,omitempty"`
|
||||||
|
TopK int `json:"top_k,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RAGQueryOutput contains the query results.
|
||||||
|
type RAGQueryOutput struct {
|
||||||
|
Results []RAGResult `json:"results"`
|
||||||
|
Context string `json:"context"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RAGResult represents a single search result.
|
||||||
|
type RAGResult struct {
|
||||||
|
Content string `json:"content"`
|
||||||
|
Score float32 `json:"score"`
|
||||||
|
Source string `json:"source"`
|
||||||
|
Metadata map[string]string `json:"metadata,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RAGIngestInput contains parameters for ingesting documents.
|
||||||
|
type RAGIngestInput struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
Collection string `json:"collection,omitempty"`
|
||||||
|
Recreate bool `json:"recreate,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RAGIngestOutput contains the ingestion results.
|
||||||
|
type RAGIngestOutput struct {
|
||||||
|
Success bool `json:"success"`
|
||||||
|
Path string `json:"path"`
|
||||||
|
Chunks int `json:"chunks"`
|
||||||
|
Message string `json:"message,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RAGCollectionsInput contains parameters for listing collections.
|
||||||
|
type RAGCollectionsInput struct {
|
||||||
|
ShowStats bool `json:"show_stats,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RAGCollectionsOutput contains the list of collections.
|
||||||
|
type RAGCollectionsOutput struct {
|
||||||
|
Collections []CollectionInfo `json:"collections"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// CollectionInfo describes a Qdrant collection.
|
||||||
|
type CollectionInfo struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
PointsCount uint64 `json:"points_count,omitempty"`
|
||||||
|
Status string `json:"status,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// registerRAGTools adds RAG tools to the MCP server.
|
||||||
|
func (s *Service) registerRAGTools(server *mcp.Server) {
|
||||||
|
mcp.AddTool(server, &mcp.Tool{
|
||||||
|
Name: "rag_query",
|
||||||
|
Description: "Query the vector database for relevant documents using semantic search",
|
||||||
|
}, s.ragQuery)
|
||||||
|
|
||||||
|
mcp.AddTool(server, &mcp.Tool{
|
||||||
|
Name: "rag_ingest",
|
||||||
|
Description: "Ingest a file or directory into the vector database",
|
||||||
|
}, s.ragIngest)
|
||||||
|
|
||||||
|
mcp.AddTool(server, &mcp.Tool{
|
||||||
|
Name: "rag_collections",
|
||||||
|
Description: "List available vector database collections",
|
||||||
|
}, s.ragCollections)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) ragQuery(ctx context.Context, req *mcp.CallToolRequest, input RAGQueryInput) (*mcp.CallToolResult, RAGQueryOutput, error) {
|
||||||
|
s.logger.Info("MCP tool execution", "tool", "rag_query", "question", input.Question)
|
||||||
|
|
||||||
|
collection := input.Collection
|
||||||
|
if collection == "" {
|
||||||
|
collection = "hostuk-docs"
|
||||||
|
}
|
||||||
|
topK := input.TopK
|
||||||
|
if topK <= 0 {
|
||||||
|
topK = 5
|
||||||
|
}
|
||||||
|
|
||||||
|
results, err := ragcmd.QueryDocs(ctx, input.Question, collection, topK)
|
||||||
|
if err != nil {
|
||||||
|
return nil, RAGQueryOutput{}, fmt.Errorf("query failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert to output format
|
||||||
|
out := RAGQueryOutput{
|
||||||
|
Results: make([]RAGResult, 0, len(results)),
|
||||||
|
Context: rag.FormatResultsContext(results),
|
||||||
|
}
|
||||||
|
for _, r := range results {
|
||||||
|
out.Results = append(out.Results, RAGResult{
|
||||||
|
Content: r.Content,
|
||||||
|
Score: r.Score,
|
||||||
|
Source: r.Source,
|
||||||
|
Metadata: r.Metadata,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) ragIngest(ctx context.Context, req *mcp.CallToolRequest, input RAGIngestInput) (*mcp.CallToolResult, RAGIngestOutput, error) {
|
||||||
|
s.logger.Security("MCP tool execution", "tool", "rag_ingest", "path", input.Path)
|
||||||
|
|
||||||
|
collection := input.Collection
|
||||||
|
if collection == "" {
|
||||||
|
collection = "hostuk-docs"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if path is a file or directory
|
||||||
|
info, err := s.medium.Stat(input.Path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, RAGIngestOutput{}, fmt.Errorf("path not found: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if info.IsDir() {
|
||||||
|
err = ragcmd.IngestDirectory(ctx, input.Path, collection, input.Recreate)
|
||||||
|
if err != nil {
|
||||||
|
return nil, RAGIngestOutput{}, fmt.Errorf("ingest directory failed: %w", err)
|
||||||
|
}
|
||||||
|
return nil, RAGIngestOutput{
|
||||||
|
Success: true,
|
||||||
|
Path: input.Path,
|
||||||
|
Message: fmt.Sprintf("Ingested directory into collection %s", collection),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
chunks, err := ragcmd.IngestFile(ctx, input.Path, collection)
|
||||||
|
if err != nil {
|
||||||
|
return nil, RAGIngestOutput{}, fmt.Errorf("ingest file failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, RAGIngestOutput{
|
||||||
|
Success: true,
|
||||||
|
Path: input.Path,
|
||||||
|
Chunks: chunks,
|
||||||
|
Message: fmt.Sprintf("Ingested %d chunks into collection %s", chunks, collection),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) ragCollections(ctx context.Context, req *mcp.CallToolRequest, input RAGCollectionsInput) (*mcp.CallToolResult, RAGCollectionsOutput, error) {
|
||||||
|
s.logger.Info("MCP tool execution", "tool", "rag_collections")
|
||||||
|
|
||||||
|
client, err := rag.NewQdrantClient(rag.DefaultQdrantConfig())
|
||||||
|
if err != nil {
|
||||||
|
return nil, RAGCollectionsOutput{}, fmt.Errorf("connect to Qdrant: %w", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = client.Close() }()
|
||||||
|
|
||||||
|
names, err := client.ListCollections(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return nil, RAGCollectionsOutput{}, fmt.Errorf("list collections: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
out := RAGCollectionsOutput{
|
||||||
|
Collections: make([]CollectionInfo, 0, len(names)),
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, name := range names {
|
||||||
|
info := CollectionInfo{Name: name}
|
||||||
|
if input.ShowStats {
|
||||||
|
cinfo, err := client.CollectionInfo(ctx, name)
|
||||||
|
if err == nil {
|
||||||
|
info.PointsCount = cinfo.PointsCount
|
||||||
|
info.Status = cinfo.Status.String()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.Collections = append(out.Collections, info)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, out, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Update mcp.go to call registerRAGTools**
|
||||||
|
|
||||||
|
In `pkg/mcp/mcp.go`, modify the `registerTools` function (around line 104) to add:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (s *Service) registerTools(server *mcp.Server) {
|
||||||
|
// File operations (existing)
|
||||||
|
// ... existing code ...
|
||||||
|
|
||||||
|
// RAG operations
|
||||||
|
s.registerRAGTools(server)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 5: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test -run TestRAGQuery ./pkg/mcp/... -v`
|
||||||
|
Expected: PASS
|
||||||
|
|
||||||
|
**Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add pkg/mcp/tools_rag.go pkg/mcp/tools_rag_test.go pkg/mcp/mcp.go
|
||||||
|
git commit -m "feat(mcp): add RAG tools (query, ingest, collections)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Add metrics tools to pkg/mcp
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `pkg/mcp/tools_metrics.go`
|
||||||
|
- Modify: `pkg/mcp/mcp.go` (registerTools)
|
||||||
|
- Test: `pkg/mcp/tools_metrics_test.go`
|
||||||
|
|
||||||
|
**Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Create `pkg/mcp/tools_metrics_test.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMetricsRecordTool_Good(t *testing.T) {
|
||||||
|
s, err := New(WithWorkspaceRoot(""))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tools := s.Server().ListTools()
|
||||||
|
found := false
|
||||||
|
for _, tool := range tools {
|
||||||
|
if tool.Name == "metrics_record" {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Error("metrics_record tool not registered")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMetricsQueryTool_Good(t *testing.T) {
|
||||||
|
s, err := New(WithWorkspaceRoot(""))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tools := s.Server().ListTools()
|
||||||
|
found := false
|
||||||
|
for _, tool := range tools {
|
||||||
|
if tool.Name == "metrics_query" {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Error("metrics_query tool not registered")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test -run TestMetrics ./pkg/mcp/... -v`
|
||||||
|
Expected: FAIL
|
||||||
|
|
||||||
|
**Step 3: Create tools_metrics.go**
|
||||||
|
|
||||||
|
Create `pkg/mcp/tools_metrics.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"forge.lthn.ai/core/cli/pkg/ai"
|
||||||
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Metrics tool input/output types
|
||||||
|
|
||||||
|
// MetricsRecordInput contains parameters for recording a metric event.
|
||||||
|
type MetricsRecordInput struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
AgentID string `json:"agent_id,omitempty"`
|
||||||
|
Repo string `json:"repo,omitempty"`
|
||||||
|
Data map[string]any `json:"data,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// MetricsRecordOutput contains the result of recording.
|
||||||
|
type MetricsRecordOutput struct {
|
||||||
|
Success bool `json:"success"`
|
||||||
|
Timestamp time.Time `json:"timestamp"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// MetricsQueryInput contains parameters for querying metrics.
|
||||||
|
type MetricsQueryInput struct {
|
||||||
|
Since string `json:"since,omitempty"` // e.g., "7d", "24h"
|
||||||
|
}
|
||||||
|
|
||||||
|
// MetricsQueryOutput contains the query results.
|
||||||
|
type MetricsQueryOutput struct {
|
||||||
|
Total int `json:"total"`
|
||||||
|
ByType []MetricCount `json:"by_type"`
|
||||||
|
ByRepo []MetricCount `json:"by_repo"`
|
||||||
|
ByAgent []MetricCount `json:"by_agent"`
|
||||||
|
Events []MetricEventBrief `json:"events,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// MetricCount represents a count by key.
|
||||||
|
type MetricCount struct {
|
||||||
|
Key string `json:"key"`
|
||||||
|
Count int `json:"count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// MetricEventBrief is a simplified event for output.
|
||||||
|
type MetricEventBrief struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Timestamp time.Time `json:"timestamp"`
|
||||||
|
AgentID string `json:"agent_id,omitempty"`
|
||||||
|
Repo string `json:"repo,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// registerMetricsTools adds metrics tools to the MCP server.
|
||||||
|
func (s *Service) registerMetricsTools(server *mcp.Server) {
|
||||||
|
mcp.AddTool(server, &mcp.Tool{
|
||||||
|
Name: "metrics_record",
|
||||||
|
Description: "Record a metric event (AI task, security scan, job creation, etc.)",
|
||||||
|
}, s.metricsRecord)
|
||||||
|
|
||||||
|
mcp.AddTool(server, &mcp.Tool{
|
||||||
|
Name: "metrics_query",
|
||||||
|
Description: "Query recorded metrics with aggregation by type, repo, and agent",
|
||||||
|
}, s.metricsQuery)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) metricsRecord(ctx context.Context, req *mcp.CallToolRequest, input MetricsRecordInput) (*mcp.CallToolResult, MetricsRecordOutput, error) {
|
||||||
|
s.logger.Info("MCP tool execution", "tool", "metrics_record", "type", input.Type)
|
||||||
|
|
||||||
|
if input.Type == "" {
|
||||||
|
return nil, MetricsRecordOutput{}, fmt.Errorf("type is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
event := ai.Event{
|
||||||
|
Type: input.Type,
|
||||||
|
Timestamp: time.Now(),
|
||||||
|
AgentID: input.AgentID,
|
||||||
|
Repo: input.Repo,
|
||||||
|
Data: input.Data,
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := ai.Record(event); err != nil {
|
||||||
|
return nil, MetricsRecordOutput{}, fmt.Errorf("record event: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, MetricsRecordOutput{
|
||||||
|
Success: true,
|
||||||
|
Timestamp: event.Timestamp,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) metricsQuery(ctx context.Context, req *mcp.CallToolRequest, input MetricsQueryInput) (*mcp.CallToolResult, MetricsQueryOutput, error) {
|
||||||
|
s.logger.Info("MCP tool execution", "tool", "metrics_query", "since", input.Since)
|
||||||
|
|
||||||
|
since := input.Since
|
||||||
|
if since == "" {
|
||||||
|
since = "7d"
|
||||||
|
}
|
||||||
|
|
||||||
|
duration, err := parseDuration(since)
|
||||||
|
if err != nil {
|
||||||
|
return nil, MetricsQueryOutput{}, fmt.Errorf("invalid since value: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sinceTime := time.Now().Add(-duration)
|
||||||
|
events, err := ai.ReadEvents(sinceTime)
|
||||||
|
if err != nil {
|
||||||
|
return nil, MetricsQueryOutput{}, fmt.Errorf("read events: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
summary := ai.Summary(events)
|
||||||
|
|
||||||
|
out := MetricsQueryOutput{
|
||||||
|
Total: summary["total"].(int),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert by_type
|
||||||
|
if byType, ok := summary["by_type"].([]map[string]any); ok {
|
||||||
|
for _, entry := range byType {
|
||||||
|
out.ByType = append(out.ByType, MetricCount{
|
||||||
|
Key: entry["key"].(string),
|
||||||
|
Count: entry["count"].(int),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert by_repo
|
||||||
|
if byRepo, ok := summary["by_repo"].([]map[string]any); ok {
|
||||||
|
for _, entry := range byRepo {
|
||||||
|
out.ByRepo = append(out.ByRepo, MetricCount{
|
||||||
|
Key: entry["key"].(string),
|
||||||
|
Count: entry["count"].(int),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert by_agent
|
||||||
|
if byAgent, ok := summary["by_agent"].([]map[string]any); ok {
|
||||||
|
for _, entry := range byAgent {
|
||||||
|
out.ByAgent = append(out.ByAgent, MetricCount{
|
||||||
|
Key: entry["key"].(string),
|
||||||
|
Count: entry["count"].(int),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Include last 10 events for context
|
||||||
|
limit := 10
|
||||||
|
if len(events) < limit {
|
||||||
|
limit = len(events)
|
||||||
|
}
|
||||||
|
for i := len(events) - limit; i < len(events); i++ {
|
||||||
|
ev := events[i]
|
||||||
|
out.Events = append(out.Events, MetricEventBrief{
|
||||||
|
Type: ev.Type,
|
||||||
|
Timestamp: ev.Timestamp,
|
||||||
|
AgentID: ev.AgentID,
|
||||||
|
Repo: ev.Repo,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseDuration parses a human-friendly duration like "7d", "24h", "30d".
|
||||||
|
func parseDuration(s string) (time.Duration, error) {
|
||||||
|
if len(s) < 2 {
|
||||||
|
return 0, fmt.Errorf("invalid duration: %s", s)
|
||||||
|
}
|
||||||
|
|
||||||
|
unit := s[len(s)-1]
|
||||||
|
value := s[:len(s)-1]
|
||||||
|
|
||||||
|
var n int
|
||||||
|
if _, err := fmt.Sscanf(value, "%d", &n); err != nil {
|
||||||
|
return 0, fmt.Errorf("invalid duration: %s", s)
|
||||||
|
}
|
||||||
|
|
||||||
|
if n <= 0 {
|
||||||
|
return 0, fmt.Errorf("duration must be positive: %s", s)
|
||||||
|
}
|
||||||
|
|
||||||
|
switch unit {
|
||||||
|
case 'd':
|
||||||
|
return time.Duration(n) * 24 * time.Hour, nil
|
||||||
|
case 'h':
|
||||||
|
return time.Duration(n) * time.Hour, nil
|
||||||
|
case 'm':
|
||||||
|
return time.Duration(n) * time.Minute, nil
|
||||||
|
default:
|
||||||
|
return 0, fmt.Errorf("unknown unit %c in duration: %s", unit, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Update mcp.go to call registerMetricsTools**
|
||||||
|
|
||||||
|
In `pkg/mcp/mcp.go`, add to `registerTools`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (s *Service) registerTools(server *mcp.Server) {
|
||||||
|
// ... existing file operations ...
|
||||||
|
|
||||||
|
// RAG operations
|
||||||
|
s.registerRAGTools(server)
|
||||||
|
|
||||||
|
// Metrics operations
|
||||||
|
s.registerMetricsTools(server)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 5: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test -run TestMetrics ./pkg/mcp/... -v`
|
||||||
|
Expected: PASS
|
||||||
|
|
||||||
|
**Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add pkg/mcp/tools_metrics.go pkg/mcp/tools_metrics_test.go pkg/mcp/mcp.go
|
||||||
|
git commit -m "feat(mcp): add metrics tools (record, query)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Create `core mcp serve` command
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `internal/cmd/mcpcmd/cmd_mcp.go`
|
||||||
|
- Modify: `internal/variants/full.go` (add import)
|
||||||
|
- Test: Manual test via `core mcp serve`
|
||||||
|
|
||||||
|
**Step 1: Create the mcp command package**
|
||||||
|
|
||||||
|
Create `internal/cmd/mcpcmd/cmd_mcp.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package mcpcmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"syscall"
|
||||||
|
|
||||||
|
"forge.lthn.ai/core/cli/pkg/cli"
|
||||||
|
"forge.lthn.ai/core/cli/pkg/i18n"
|
||||||
|
"forge.lthn.ai/core/cli/pkg/mcp"
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
cli.RegisterCommands(AddMCPCommands)
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
mcpWorkspace string
|
||||||
|
)
|
||||||
|
|
||||||
|
var mcpCmd = &cli.Command{
|
||||||
|
Use: "mcp",
|
||||||
|
Short: i18n.T("cmd.mcp.short"),
|
||||||
|
Long: i18n.T("cmd.mcp.long"),
|
||||||
|
}
|
||||||
|
|
||||||
|
var serveCmd = &cli.Command{
|
||||||
|
Use: "serve",
|
||||||
|
Short: i18n.T("cmd.mcp.serve.short"),
|
||||||
|
Long: i18n.T("cmd.mcp.serve.long"),
|
||||||
|
RunE: func(cmd *cli.Command, args []string) error {
|
||||||
|
return runServe()
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
func AddMCPCommands(root *cli.Command) {
|
||||||
|
initMCPFlags()
|
||||||
|
mcpCmd.AddCommand(serveCmd)
|
||||||
|
root.AddCommand(mcpCmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
func initMCPFlags() {
|
||||||
|
serveCmd.Flags().StringVar(&mcpWorkspace, "workspace", "", i18n.T("cmd.mcp.serve.flag.workspace"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func runServe() error {
|
||||||
|
opts := []mcp.Option{}
|
||||||
|
|
||||||
|
if mcpWorkspace != "" {
|
||||||
|
opts = append(opts, mcp.WithWorkspaceRoot(mcpWorkspace))
|
||||||
|
} else {
|
||||||
|
// Default to unrestricted for MCP server
|
||||||
|
opts = append(opts, mcp.WithWorkspaceRoot(""))
|
||||||
|
}
|
||||||
|
|
||||||
|
svc, err := mcp.New(opts...)
|
||||||
|
if err != nil {
|
||||||
|
return cli.Wrap(err, "create MCP service")
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// Handle shutdown signals
|
||||||
|
sigCh := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
go func() {
|
||||||
|
<-sigCh
|
||||||
|
cancel()
|
||||||
|
}()
|
||||||
|
|
||||||
|
return svc.Run(ctx)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Add i18n strings**
|
||||||
|
|
||||||
|
Create or update `pkg/i18n/en.yaml` (if it exists) or add to the existing i18n mechanism:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
cmd.mcp.short: "MCP (Model Context Protocol) server"
|
||||||
|
cmd.mcp.long: "Start an MCP server for Claude Code integration with file, RAG, and metrics tools."
|
||||||
|
cmd.mcp.serve.short: "Start the MCP server"
|
||||||
|
cmd.mcp.serve.long: "Start the MCP server in stdio mode. Use MCP_ADDR env var for TCP mode."
|
||||||
|
cmd.mcp.serve.flag.workspace: "Restrict file operations to this directory (empty = unrestricted)"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Add import to full.go**
|
||||||
|
|
||||||
|
Modify `internal/variants/full.go` to add:
|
||||||
|
|
||||||
|
```go
|
||||||
|
import (
|
||||||
|
// ... existing imports ...
|
||||||
|
_ "forge.lthn.ai/core/cli/internal/cmd/mcpcmd"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Build and test**
|
||||||
|
|
||||||
|
Run: `go build && ./core mcp serve --help`
|
||||||
|
Expected: Help output showing the serve command
|
||||||
|
|
||||||
|
**Step 5: Test MCP server manually**
|
||||||
|
|
||||||
|
Run: `echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | ./core mcp serve`
|
||||||
|
Expected: JSON response listing all tools including rag_query, metrics_record, etc.
|
||||||
|
|
||||||
|
**Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/cmd/mcpcmd/cmd_mcp.go internal/variants/full.go
|
||||||
|
git commit -m "feat: add 'core mcp serve' command"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Configure agentic-flows plugin with .mcp.json
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`
|
||||||
|
- Modify: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.claude-plugin/plugin.json` (optional, add mcpServers)
|
||||||
|
|
||||||
|
**Step 1: Create .mcp.json**
|
||||||
|
|
||||||
|
Create `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"core-cli": {
|
||||||
|
"command": "core",
|
||||||
|
"args": ["mcp", "serve"],
|
||||||
|
"env": {
|
||||||
|
"MCP_WORKSPACE": ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Verify plugin loads**
|
||||||
|
|
||||||
|
Restart Claude Code and run `/mcp` to verify the core-cli server appears.
|
||||||
|
|
||||||
|
**Step 3: Test MCP tools**
|
||||||
|
|
||||||
|
Test that tools are available:
|
||||||
|
- `mcp__plugin_agentic-flows_core-cli__rag_query`
|
||||||
|
- `mcp__plugin_agentic-flows_core-cli__rag_ingest`
|
||||||
|
- `mcp__plugin_agentic-flows_core-cli__rag_collections`
|
||||||
|
- `mcp__plugin_agentic-flows_core-cli__metrics_record`
|
||||||
|
- `mcp__plugin_agentic-flows_core-cli__metrics_query`
|
||||||
|
- `mcp__plugin_agentic-flows_core-cli__file_read`
|
||||||
|
- etc.
|
||||||
|
|
||||||
|
**Step 4: Commit plugin changes**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/shared/hostuk/claude-plugins
|
||||||
|
git add plugins/agentic-flows/.mcp.json
|
||||||
|
git commit -m "feat(agentic-flows): add MCP server configuration for core-cli"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: Update documentation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `/home/claude/.claude/projects/-home-claude/memory/MEMORY.md`
|
||||||
|
- Modify: `/home/claude/.claude/projects/-home-claude/memory/plugin-dev-notes.md`
|
||||||
|
|
||||||
|
**Step 1: Update MEMORY.md**
|
||||||
|
|
||||||
|
Add under "Core CLI MCP Server" section:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
### Core CLI MCP Server
|
||||||
|
- **Command:** `core mcp serve` (stdio mode) or `MCP_ADDR=:9000 core mcp serve` (TCP)
|
||||||
|
- **Tools available:**
|
||||||
|
- File ops: file_read, file_write, file_edit, file_delete, file_rename, file_exists, dir_list, dir_create
|
||||||
|
- RAG: rag_query, rag_ingest, rag_collections
|
||||||
|
- Metrics: metrics_record, metrics_query
|
||||||
|
- Language: lang_detect, lang_list
|
||||||
|
- **Plugin config:** `plugins/agentic-flows/.mcp.json`
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Update plugin-dev-notes.md**
|
||||||
|
|
||||||
|
Add section:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## MCP Server (core mcp serve)
|
||||||
|
|
||||||
|
### Available Tools
|
||||||
|
| Tool | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| file_read | Read file contents |
|
||||||
|
| file_write | Write file contents |
|
||||||
|
| file_edit | Edit file (replace string) |
|
||||||
|
| file_delete | Delete file |
|
||||||
|
| file_rename | Rename/move file |
|
||||||
|
| file_exists | Check if file exists |
|
||||||
|
| dir_list | List directory contents |
|
||||||
|
| dir_create | Create directory |
|
||||||
|
| rag_query | Query vector DB |
|
||||||
|
| rag_ingest | Ingest file/directory |
|
||||||
|
| rag_collections | List collections |
|
||||||
|
| metrics_record | Record event |
|
||||||
|
| metrics_query | Query events |
|
||||||
|
| lang_detect | Detect file language |
|
||||||
|
| lang_list | List supported languages |
|
||||||
|
|
||||||
|
### Example .mcp.json
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"core-cli": {
|
||||||
|
"command": "core",
|
||||||
|
"args": ["mcp", "serve"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Commit documentation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add ~/.claude/projects/-home-claude/memory/*.md
|
||||||
|
git commit -m "docs: update memory with MCP server tools"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Task | Files | Purpose |
|
||||||
|
|------|-------|---------|
|
||||||
|
| 1 | `pkg/mcp/tools_rag.go` | RAG tools (query, ingest, collections) |
|
||||||
|
| 2 | `pkg/mcp/tools_metrics.go` | Metrics tools (record, query) |
|
||||||
|
| 3 | `internal/cmd/mcpcmd/cmd_mcp.go` | `core mcp serve` command |
|
||||||
|
| 4 | `plugins/agentic-flows/.mcp.json` | Plugin MCP configuration |
|
||||||
|
| 5 | Memory docs | Documentation updates |
|
||||||
|
|
||||||
|
## Services Required
|
||||||
|
|
||||||
|
- **Qdrant:** localhost:6333 (verified running)
|
||||||
|
- **Ollama:** localhost:11434 with nomic-embed-text (verified running)
|
||||||
|
- **InfluxDB:** localhost:8086 (optional, for future time-series metrics)
|
||||||
82
docs/plans/completed/2026-02-17-lem-chat-design.md
Normal file
82
docs/plans/completed/2026-02-17-lem-chat-design.md
Normal file
|
|
@ -0,0 +1,82 @@
|
||||||
|
# LEM Chat — Web Components Design
|
||||||
|
|
||||||
|
**Date**: 2026-02-17
|
||||||
|
**Status**: Approved
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Standalone chat UI built with vanilla Web Components (Custom Elements + Shadow DOM). Connects to the MLX inference server's OpenAI-compatible SSE streaming endpoint. Zero framework dependencies. Single JS file output, embeddable anywhere.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| Element | Purpose |
|
||||||
|
|---------|---------|
|
||||||
|
| `<lem-chat>` | Container. Conversation state, SSE connection, config via attributes |
|
||||||
|
| `<lem-messages>` | Scrollable message list with auto-scroll anchoring |
|
||||||
|
| `<lem-message>` | Single message bubble. Streams tokens for assistant messages |
|
||||||
|
| `<lem-input>` | Text input, Enter to send, Shift+Enter for newline |
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
User types in <lem-input>
|
||||||
|
→ dispatches 'lem-send' CustomEvent
|
||||||
|
→ <lem-chat> catches it
|
||||||
|
→ adds user message to <lem-messages>
|
||||||
|
→ POST /v1/chat/completions {stream: true, messages: [...history]}
|
||||||
|
→ reads SSE chunks via fetch + ReadableStream
|
||||||
|
→ appends tokens to streaming <lem-message>
|
||||||
|
→ on [DONE], finalises message
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```html
|
||||||
|
<lem-chat endpoint="http://localhost:8090" model="qwen3-8b"></lem-chat>
|
||||||
|
```
|
||||||
|
|
||||||
|
Attributes: `endpoint`, `model`, `system-prompt`, `max-tokens`, `temperature`
|
||||||
|
|
||||||
|
## Theming
|
||||||
|
|
||||||
|
Shadow DOM with CSS custom properties:
|
||||||
|
|
||||||
|
```css
|
||||||
|
--lem-bg: #1a1a1e;
|
||||||
|
--lem-msg-user: #2a2a3e;
|
||||||
|
--lem-msg-assistant: #1e1e2a;
|
||||||
|
--lem-accent: #5865f2;
|
||||||
|
--lem-text: #e0e0e0;
|
||||||
|
--lem-font: system-ui;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Markdown
|
||||||
|
|
||||||
|
Minimal inline parsing: fenced code blocks, inline code, bold, italic. No library.
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
lem-chat/
|
||||||
|
├── index.html # Demo page
|
||||||
|
├── src/
|
||||||
|
│ ├── lem-chat.ts # Main container + SSE client
|
||||||
|
│ ├── lem-messages.ts # Message list with scroll anchoring
|
||||||
|
│ ├── lem-message.ts # Single message with streaming
|
||||||
|
│ ├── lem-input.ts # Text input
|
||||||
|
│ ├── markdown.ts # Minimal markdown → HTML
|
||||||
|
│ └── styles.ts # CSS template literals
|
||||||
|
├── package.json # typescript + esbuild
|
||||||
|
└── tsconfig.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Build: `esbuild src/lem-chat.ts --bundle --outfile=dist/lem-chat.js`
|
||||||
|
|
||||||
|
## Not in v1
|
||||||
|
|
||||||
|
- Model selection UI
|
||||||
|
- Conversation persistence
|
||||||
|
- File/image upload
|
||||||
|
- Syntax highlighting
|
||||||
|
- Typing indicators
|
||||||
|
- User avatars
|
||||||
657
docs/plans/completed/2026-02-20-go-api-design-original.md
Normal file
657
docs/plans/completed/2026-02-20-go-api-design-original.md
Normal file
|
|
@ -0,0 +1,657 @@
|
||||||
|
# go-api Design — HTTP Gateway + OpenAPI SDK Generation
|
||||||
|
|
||||||
|
**Date:** 2026-02-20
|
||||||
|
**Author:** Virgil
|
||||||
|
**Status:** Phase 1 + Phase 2 + Phase 3 Complete (176 tests in go-api)
|
||||||
|
**Module:** `forge.lthn.ai/core/go-api`
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The Core Go ecosystem exposes 42+ tools via MCP (JSON-RPC), which is ideal for AI agents but inaccessible to regular HTTP clients, frontend applications, and third-party integrators. There is no unified HTTP gateway, no OpenAPI specification, and no generated SDKs.
|
||||||
|
|
||||||
|
Both external customers (Host UK products) and Lethean network peers need programmatic access to the same services. The gateway also serves web routes, static assets, and streaming endpoints — not just REST APIs.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
A `go-api` package that acts as the central HTTP gateway:
|
||||||
|
|
||||||
|
1. **Gin-based HTTP gateway** with extensible middleware via gin-contrib plugins
|
||||||
|
2. **RouteGroup interface** that subsystems implement to register their own endpoints (API, web, or both)
|
||||||
|
3. **WebSocket + SSE integration** for real-time streaming
|
||||||
|
4. **OpenAPI 3.1 spec generation** via runtime SpecBuilder (not swaggo annotations)
|
||||||
|
5. **SDK generation pipeline** targeting 11 languages via openapi-generator-cli
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Four-Protocol Access
|
||||||
|
|
||||||
|
Same backend services, four client protocols:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─── REST (go-api) POST /v1/ml/generate → JSON
|
||||||
|
│
|
||||||
|
├─── GraphQL (gqlgen) mutation { mlGenerate(...) { response } }
|
||||||
|
Client ────────────┤
|
||||||
|
├─── WebSocket (go-ws) subscribe ml.generate → streaming
|
||||||
|
│
|
||||||
|
└─── MCP (go-ai) ml_generate → JSON-RPC
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dependency Graph
|
||||||
|
|
||||||
|
```
|
||||||
|
go-api (Gin engine + middleware + OpenAPI)
|
||||||
|
↑ imported by (each registers its own routes)
|
||||||
|
├── go-ai/api/ → /v1/file/*, /v1/process/*, /v1/metrics/*
|
||||||
|
├── go-ml/api/ → /v1/ml/*
|
||||||
|
├── go-rag/api/ → /v1/rag/*
|
||||||
|
├── go-agentic/api/ → /v1/tasks/*
|
||||||
|
├── go-help/api/ → /v1/help/*
|
||||||
|
└── go-ws/api/ → /ws (WebSocket upgrade)
|
||||||
|
```
|
||||||
|
|
||||||
|
go-api has zero internal ecosystem dependencies. Subsystems import go-api, not the other way round.
|
||||||
|
|
||||||
|
### Subsystem Opt-In
|
||||||
|
|
||||||
|
Not every MCP tool becomes a REST endpoint. Each subsystem decides what to expose via a separate `RegisterAPI()` method, independent of MCP's `RegisterTools()`. A subsystem with 15 MCP tools might expose 5 REST endpoints.
|
||||||
|
|
||||||
|
## Package Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
forge.lthn.ai/core/go-api
|
||||||
|
├── api.go # Engine struct, New(), Serve(), Shutdown()
|
||||||
|
├── middleware.go # Auth, CORS, rate limiting, request logging, recovery
|
||||||
|
├── options.go # WithAddr, WithAuth, WithCORS, WithRateLimit, etc.
|
||||||
|
├── group.go # RouteGroup interface + registration
|
||||||
|
├── response.go # Envelope type, error responses, pagination
|
||||||
|
├── docs/ # Generated swagger docs (swaggo output)
|
||||||
|
├── sdk/ # SDK generation tooling / Makefile targets
|
||||||
|
└── go.mod # forge.lthn.ai/core/go-api
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Interface
|
||||||
|
|
||||||
|
```go
|
||||||
|
// RouteGroup registers API routes onto a Gin router group.
|
||||||
|
// Subsystems implement this to expose their endpoints.
|
||||||
|
type RouteGroup interface {
|
||||||
|
// Name returns the route group identifier (e.g. "ml", "rag", "tasks")
|
||||||
|
Name() string
|
||||||
|
// BasePath returns the URL prefix (e.g. "/v1/ml")
|
||||||
|
BasePath() string
|
||||||
|
// RegisterRoutes adds handlers to the provided router group
|
||||||
|
RegisterRoutes(rg *gin.RouterGroup)
|
||||||
|
}
|
||||||
|
|
||||||
|
// StreamGroup optionally declares WebSocket channels a subsystem publishes to.
|
||||||
|
type StreamGroup interface {
|
||||||
|
Channels() []string
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Subsystem Example (go-ml)
|
||||||
|
|
||||||
|
```go
|
||||||
|
// In go-ml/api/routes.go
|
||||||
|
package api
|
||||||
|
|
||||||
|
type Routes struct {
|
||||||
|
service *ml.Service
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewRoutes(svc *ml.Service) *Routes {
|
||||||
|
return &Routes{service: svc}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Routes) Name() string { return "ml" }
|
||||||
|
func (r *Routes) BasePath() string { return "/v1/ml" }
|
||||||
|
|
||||||
|
func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) {
|
||||||
|
rg.POST("/generate", r.Generate)
|
||||||
|
rg.POST("/score", r.Score)
|
||||||
|
rg.GET("/backends", r.Backends)
|
||||||
|
rg.GET("/status", r.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Routes) Channels() []string {
|
||||||
|
return []string{"ml.generate", "ml.status"}
|
||||||
|
}
|
||||||
|
|
||||||
|
// @Summary Generate text via ML backend
|
||||||
|
// @Tags ml
|
||||||
|
// @Accept json
|
||||||
|
// @Produce json
|
||||||
|
// @Param input body MLGenerateInput true "Generation parameters"
|
||||||
|
// @Success 200 {object} Response[MLGenerateOutput]
|
||||||
|
// @Router /v1/ml/generate [post]
|
||||||
|
func (r *Routes) Generate(c *gin.Context) {
|
||||||
|
var input MLGenerateInput
|
||||||
|
if err := c.ShouldBindJSON(&input); err != nil {
|
||||||
|
c.JSON(400, api.Fail("invalid_input", err.Error()))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
result, err := r.service.Generate(c.Request.Context(), input.Backend, input.Prompt, ml.GenOpts{
|
||||||
|
Temperature: input.Temperature,
|
||||||
|
MaxTokens: input.MaxTokens,
|
||||||
|
Model: input.Model,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
c.JSON(500, api.Fail("ml.generate_failed", err.Error()))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
c.JSON(200, api.OK(MLGenerateOutput{
|
||||||
|
Response: result,
|
||||||
|
Backend: input.Backend,
|
||||||
|
Model: input.Model,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Engine Wiring (in core CLI)
|
||||||
|
|
||||||
|
```go
|
||||||
|
engine := api.New(
|
||||||
|
api.WithAddr(":8080"),
|
||||||
|
api.WithCORS("*"),
|
||||||
|
api.WithAuth(api.BearerToken(cfg.APIKey)),
|
||||||
|
api.WithRateLimit(100, time.Minute),
|
||||||
|
api.WithWSHub(wsHub),
|
||||||
|
)
|
||||||
|
|
||||||
|
engine.Register(mlapi.NewRoutes(mlService))
|
||||||
|
engine.Register(ragapi.NewRoutes(ragService))
|
||||||
|
engine.Register(agenticapi.NewRoutes(agenticService))
|
||||||
|
|
||||||
|
engine.Serve(ctx) // Blocks until context cancelled
|
||||||
|
```
|
||||||
|
|
||||||
|
## Response Envelope
|
||||||
|
|
||||||
|
All endpoints return a consistent envelope:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Response[T any] struct {
|
||||||
|
Success bool `json:"success"`
|
||||||
|
Data T `json:"data,omitempty"`
|
||||||
|
Error *Error `json:"error,omitempty"`
|
||||||
|
Meta *Meta `json:"meta,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Error struct {
|
||||||
|
Code string `json:"code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
Details any `json:"details,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Meta struct {
|
||||||
|
RequestID string `json:"request_id"`
|
||||||
|
Duration string `json:"duration"`
|
||||||
|
Page int `json:"page,omitempty"`
|
||||||
|
PerPage int `json:"per_page,omitempty"`
|
||||||
|
Total int `json:"total,omitempty"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Helper functions:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func OK[T any](data T) Response[T]
|
||||||
|
func Fail(code, message string) Response[any]
|
||||||
|
func Paginated[T any](data T, page, perPage, total int) Response[T]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Middleware Stack
|
||||||
|
|
||||||
|
```go
|
||||||
|
api.New(
|
||||||
|
api.WithAddr(":8080"),
|
||||||
|
api.WithCORS(api.CORSConfig{...}), // gin-contrib/cors
|
||||||
|
api.WithAuth(api.BearerToken("...")), // Phase 1: simple bearer token
|
||||||
|
api.WithRateLimit(100, time.Minute), // Per-IP sliding window
|
||||||
|
api.WithRequestID(), // X-Request-ID header generation
|
||||||
|
api.WithRecovery(), // Panic recovery → 500 response
|
||||||
|
api.WithLogger(slog.Default()), // Structured request logging
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Auth evolution path: bearer token → API keys → Authentik (OIDC/forward auth). Middleware slot stays the same.
|
||||||
|
|
||||||
|
## WebSocket Integration
|
||||||
|
|
||||||
|
go-api wraps the existing go-ws Hub as a first-class transport:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Automatic registration:
|
||||||
|
// GET /ws → WebSocket upgrade (go-ws Hub)
|
||||||
|
|
||||||
|
// Client subscribes: {"type":"subscribe","channel":"ml.generate"}
|
||||||
|
// Events arrive: {"type":"event","channel":"ml.generate","data":{...}}
|
||||||
|
// Client unsubscribes: {"type":"unsubscribe","channel":"ml.generate"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Subsystems implementing `StreamGroup` declare which channels they publish to. This metadata feeds into the OpenAPI spec as documentation.
|
||||||
|
|
||||||
|
## OpenAPI + SDK Generation
|
||||||
|
|
||||||
|
### Runtime Spec Generation (SpecBuilder)
|
||||||
|
|
||||||
|
swaggo annotations were rejected because routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools already carry JSON Schema at runtime. Instead, a `SpecBuilder` constructs the full OpenAPI 3.1 spec from registered RouteGroups at runtime.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Groups that implement DescribableGroup contribute endpoint metadata
|
||||||
|
type DescribableGroup interface {
|
||||||
|
RouteGroup
|
||||||
|
Describe() []RouteDescription
|
||||||
|
}
|
||||||
|
|
||||||
|
// SpecBuilder assembles the spec from all groups
|
||||||
|
builder := &api.SpecBuilder{Title: "Core API", Description: "...", Version: "1.0.0"}
|
||||||
|
spec, _ := builder.Build(engine.Groups())
|
||||||
|
```
|
||||||
|
|
||||||
|
### MCP-to-REST Bridge (ToolBridge)
|
||||||
|
|
||||||
|
The `ToolBridge` converts MCP tool descriptors into REST POST endpoints and implements both `RouteGroup` and `DescribableGroup`. Each tool becomes `POST /{tool_name}`. Generic types are captured at MCP registration time via closures, enabling JSON unmarshalling to the correct input type at request time.
|
||||||
|
|
||||||
|
```go
|
||||||
|
bridge := api.NewToolBridge("/v1/tools")
|
||||||
|
mcp.BridgeToAPI(mcpService, bridge) // Populates bridge from MCP tool registry
|
||||||
|
engine.Register(bridge) // Registers REST endpoints + OpenAPI metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
### Swagger UI
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Built-in at GET /swagger/*any
|
||||||
|
// SpecBuilder output served via gin-swagger, cached via sync.Once
|
||||||
|
api.New(api.WithSwagger("Core API", "...", "1.0.0"))
|
||||||
|
```
|
||||||
|
|
||||||
|
### SDK Generation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Via openapi-generator-cli (11 languages supported)
|
||||||
|
core api sdk --lang go # Generate Go SDK
|
||||||
|
core api sdk --lang typescript-fetch,python # Multiple languages
|
||||||
|
core api sdk --lang rust --output ./sdk/ # Custom output dir
|
||||||
|
```
|
||||||
|
|
||||||
|
### CLI Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core api spec # Emit OpenAPI JSON to stdout
|
||||||
|
core api spec --format yaml # YAML variant
|
||||||
|
core api spec --output spec.json # Write to file
|
||||||
|
core api sdk --lang python # Generate Python SDK
|
||||||
|
core api sdk --lang go,rust # Multiple SDKs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
| Package | Purpose |
|
||||||
|
|---------|---------|
|
||||||
|
| `github.com/gin-gonic/gin` | HTTP framework |
|
||||||
|
| `github.com/swaggo/gin-swagger` | Swagger UI middleware |
|
||||||
|
| `github.com/gin-contrib/cors` | CORS middleware |
|
||||||
|
| `github.com/gin-contrib/secure` | Security headers |
|
||||||
|
| `github.com/gin-contrib/sessions` | Server-side sessions |
|
||||||
|
| `github.com/gin-contrib/authz` | Casbin authorisation |
|
||||||
|
| `github.com/gin-contrib/httpsign` | HTTP signature verification |
|
||||||
|
| `github.com/gin-contrib/slog` | Structured request logging |
|
||||||
|
| `github.com/gin-contrib/timeout` | Per-request timeouts |
|
||||||
|
| `github.com/gin-contrib/gzip` | Gzip compression |
|
||||||
|
| `github.com/gin-contrib/static` | Static file serving |
|
||||||
|
| `github.com/gin-contrib/pprof` | Runtime profiling |
|
||||||
|
| `github.com/gin-contrib/expvar` | Runtime metrics |
|
||||||
|
| `github.com/gin-contrib/location/v2` | Reverse proxy detection |
|
||||||
|
| `github.com/99designs/gqlgen` | GraphQL endpoint |
|
||||||
|
| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | Distributed tracing |
|
||||||
|
| `gopkg.in/yaml.v3` | YAML spec export |
|
||||||
|
| `forge.lthn.ai/core/go-ws` | WebSocket Hub (existing) |
|
||||||
|
|
||||||
|
## Estimated Size
|
||||||
|
|
||||||
|
| Component | LOC |
|
||||||
|
|-----------|-----|
|
||||||
|
| Engine + options | ~200 |
|
||||||
|
| Middleware | ~150 |
|
||||||
|
| Response envelope | ~80 |
|
||||||
|
| RouteGroup interface | ~30 |
|
||||||
|
| WebSocket integration | ~60 |
|
||||||
|
| Tests | ~300 |
|
||||||
|
| **Total go-api** | **~820** |
|
||||||
|
|
||||||
|
Each subsystem's `api/` package adds ~100-200 LOC per route group.
|
||||||
|
|
||||||
|
## Phase 1 — Implemented (20 Feb 2026)
|
||||||
|
|
||||||
|
**Commit:** `17ae945` on Forge (`core/go-api`)
|
||||||
|
|
||||||
|
| Component | Status | Tests |
|
||||||
|
|-----------|--------|-------|
|
||||||
|
| Response envelope (OK, Fail, Paginated) | Done | 9 |
|
||||||
|
| RouteGroup + StreamGroup interfaces | Done | 4 |
|
||||||
|
| Engine (New, Register, Handler, Serve) | Done | 9 |
|
||||||
|
| Bearer auth middleware | Done | 3 |
|
||||||
|
| Request ID middleware | Done | 2 |
|
||||||
|
| CORS middleware (gin-contrib/cors) | Done | 3 |
|
||||||
|
| WebSocket endpoint | Done | 3 |
|
||||||
|
| Swagger UI (gin-swagger) | Done | 2 |
|
||||||
|
| Health endpoint | Done | 1 |
|
||||||
|
| **Total** | **~840 LOC** | **36** |
|
||||||
|
|
||||||
|
**Integration proof:** go-ml/api/ registers 3 endpoints with 12 tests (`0c23858`).
|
||||||
|
|
||||||
|
## Phase 2 Wave 1 — Implemented (20 Feb 2026)
|
||||||
|
|
||||||
|
**Commits:** `6bb7195..daae6f7` on Forge (`core/go-api`)
|
||||||
|
|
||||||
|
| Component | Option | Dependency | Tests |
|
||||||
|
|-----------|--------|------------|-------|
|
||||||
|
| Authentik (forward auth + OIDC) | `WithAuthentik()` | `go-oidc/v3`, `oauth2` | 14 |
|
||||||
|
| Security headers (HSTS, CSP, etc.) | `WithSecure()` | `gin-contrib/secure` | 8 |
|
||||||
|
| Structured request logging | `WithSlog()` | `gin-contrib/slog` | 6 |
|
||||||
|
| Per-request timeouts | `WithTimeout()` | `gin-contrib/timeout` | 5 |
|
||||||
|
| Gzip compression | `WithGzip()` | `gin-contrib/gzip` | 5 |
|
||||||
|
| Static file serving | `WithStatic()` | `gin-contrib/static` | 5 |
|
||||||
|
| **Wave 1 Total** | | | **43** |
|
||||||
|
|
||||||
|
**Cumulative:** 76 tests (36 Phase 1 + 43 Wave 1 - 3 shared), all passing.
|
||||||
|
|
||||||
|
## Phase 2 Wave 2 — Implemented (20 Feb 2026)
|
||||||
|
|
||||||
|
**Commits:** `64a8b16..67dcc83` on Forge (`core/go-api`)
|
||||||
|
|
||||||
|
| Component | Option | Dependency | Tests | Notes |
|
||||||
|
|-----------|--------|------------|-------|-------|
|
||||||
|
| Brotli compression | `WithBrotli()` | `andybalholm/brotli` | 5 | Custom middleware; `gin-contrib/brotli` is empty stub |
|
||||||
|
| Response caching | `WithCache()` | none (in-memory) | 5 | Custom middleware; `gin-contrib/cache` is per-handler, not global |
|
||||||
|
| Server-side sessions | `WithSessions()` | `gin-contrib/sessions` | 5 | Cookie store, configurable name + secret |
|
||||||
|
| Casbin authorisation | `WithAuthz()` | `gin-contrib/authz`, `casbin/v2` | 5 | Subject via Basic Auth; RBAC policy model |
|
||||||
|
| **Wave 2 Total** | | | **20** | |
|
||||||
|
|
||||||
|
**Cumulative:** 102 passing tests (2 integration skipped), all green.
|
||||||
|
|
||||||
|
## Phase 2 Wave 3 — Implemented (20 Feb 2026)
|
||||||
|
|
||||||
|
**Commits:** `7b3f99e..d517fa2` on Forge (`core/go-api`)
|
||||||
|
|
||||||
|
| Component | Option | Dependency | Tests | Notes |
|
||||||
|
|-----------|--------|------------|-------|-------|
|
||||||
|
| HTTP signature verification | `WithHTTPSign()` | `gin-contrib/httpsign` | 5 | HMAC-SHA256; extensible via httpsign.Option |
|
||||||
|
| Server-Sent Events | `WithSSE()` | none (custom SSEBroker) | 6 | Channel filtering, multi-client broadcast, GET /events |
|
||||||
|
| Reverse proxy detection | `WithLocation()` | `gin-contrib/location/v2` | 5 | X-Forwarded-Host/Proto parsing |
|
||||||
|
| Locale detection | `WithI18n()` | `golang.org/x/text/language` | 5 | Accept-Language parsing, message lookup, GetLocale/GetMessage |
|
||||||
|
| GraphQL endpoint | `WithGraphQL()` | `99designs/gqlgen` | 5 | /graphql + optional /graphql/playground |
|
||||||
|
| **Wave 3 Total** | | | **26** | |
|
||||||
|
|
||||||
|
**Cumulative:** 128 passing tests (2 integration skipped), all green.
|
||||||
|
|
||||||
|
## Phase 2 Wave 4 — Implemented (21 Feb 2026)
|
||||||
|
|
||||||
|
**Commits:** `32b3680..8ba1716` on Forge (`core/go-api`)
|
||||||
|
|
||||||
|
| Component | Option | Dependency | Tests | Notes |
|
||||||
|
|-----------|--------|------------|-------|-------|
|
||||||
|
| Runtime profiling | `WithPprof()` | `gin-contrib/pprof` | 5 | /debug/pprof/* endpoints, flag-based mount |
|
||||||
|
| Runtime metrics | `WithExpvar()` | `gin-contrib/expvar` | 5 | /debug/vars endpoint, flag-based mount |
|
||||||
|
| Distributed tracing | `WithTracing()` | `otelgin` + OpenTelemetry SDK | 5 | W3C traceparent propagation, span attributes |
|
||||||
|
| **Wave 4 Total** | | | **15** | |
|
||||||
|
|
||||||
|
**Cumulative:** 143 passing tests (2 integration skipped), all green.
|
||||||
|
|
||||||
|
**Phase 2 complete.** All 4 waves implemented. Every planned plugin has a `With*()` option and tests.
|
||||||
|
|
||||||
|
## Phase 3 — OpenAPI Spec Generation + SDK Codegen (21 Feb 2026)
|
||||||
|
|
||||||
|
**Architecture:** Runtime OpenAPI generation via SpecBuilder (NOT swaggo annotations). Routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools carry JSON Schema at runtime. A `ToolBridge` converts tool descriptors into RouteGroup + OpenAPI metadata. A `SpecBuilder` constructs the full OpenAPI 3.1 spec. SDK codegen wraps `openapi-generator-cli`.
|
||||||
|
|
||||||
|
### Wave 1: go-api (Tasks 1-5)
|
||||||
|
|
||||||
|
**Commits:** `465bd60..1910aec` on Forge (`core/go-api`)
|
||||||
|
|
||||||
|
| Component | File | Tests | Notes |
|
||||||
|
|-----------|------|-------|-------|
|
||||||
|
| DescribableGroup interface | `group.go` | 5 | Opt-in OpenAPI metadata for RouteGroups |
|
||||||
|
| ToolBridge | `bridge.go` | 6 | Tool descriptors → POST endpoints + DescribableGroup |
|
||||||
|
| SpecBuilder | `openapi.go` | 6 | OpenAPI 3.1 JSON with Response[T] envelope wrapping |
|
||||||
|
| Swagger refactor | `swagger.go` | 5 | Replaced hardcoded empty spec with SpecBuilder |
|
||||||
|
| Spec export | `export.go` | 5 | JSON + YAML export to file/writer |
|
||||||
|
| SDK codegen | `codegen.go` | 5 | 11-language wrapper for openapi-generator-cli |
|
||||||
|
| **Wave 1 Total** | | **32** | |
|
||||||
|
|
||||||
|
### Wave 2: go-ai MCP bridge (Tasks 6-7)
|
||||||
|
|
||||||
|
**Commits:** `2107eda..c37e1cf` on Forge (`core/go-ai`)
|
||||||
|
|
||||||
|
| Component | File | Tests | Notes |
|
||||||
|
|-----------|------|-------|-------|
|
||||||
|
| Tool registry | `mcp/registry.go` | 5 | Generic `addToolRecorded[In,Out]` captures types in closures |
|
||||||
|
| BridgeToAPI | `mcp/bridge.go` | 5 | MCP tools → go-api ToolBridge, 10MB body limit, error classification |
|
||||||
|
| **Wave 2 Total** | | **10** | |
|
||||||
|
|
||||||
|
### Wave 3: CLI commands (Tasks 8-9)
|
||||||
|
|
||||||
|
**Commit:** `d6eec4d` on Forge (`core/cli` dev branch)
|
||||||
|
|
||||||
|
| Component | File | Tests | Notes |
|
||||||
|
|-----------|------|-------|-------|
|
||||||
|
| `core api spec` | `cmd/api/cmd_spec.go` | 2 | JSON/YAML export, --output/--format flags |
|
||||||
|
| `core api sdk` | `cmd/api/cmd_sdk.go` | 2 | --lang (required), --output, --spec, --package flags |
|
||||||
|
| **Wave 3 Total** | | **4** | |
|
||||||
|
|
||||||
|
**Cumulative go-api:** 176 passing tests. **Phase 3 complete.**
|
||||||
|
|
||||||
|
### Known Limitations
|
||||||
|
|
||||||
|
- **Subsystem tools excluded from bridge:** Subsystems call `mcp.AddTool` directly, bypassing `addToolRecorded`. Only the 10 built-in MCP tools appear in the REST bridge. Future: pass `*Service` to `RegisterTools` instead of `*mcp.Server`.
|
||||||
|
- **Flat schema only:** `structSchema` reflection handles flat structs but does not recurse into nested structs. Adequate for current tool inputs.
|
||||||
|
- **CLI spec produces empty bridge:** `core api spec` currently generates a spec with only `/health`. Full MCP integration requires wiring the MCP service into the CLI command.
|
||||||
|
|
||||||
|
## Phase 2 — Gin Plugin Roadmap (Complete)
|
||||||
|
|
||||||
|
All plugins drop in as `With*()` options on the Engine. No architecture changes needed.
|
||||||
|
|
||||||
|
### Security & Auth
|
||||||
|
|
||||||
|
| Plugin | Option | Purpose | Priority |
|
||||||
|
|--------|--------|---------|----------|
|
||||||
|
| ~~**Authentik**~~ | ~~`WithAuthentik()`~~ | ~~OIDC + forward auth integration.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/secure~~ | ~~`WithSecure()`~~ | ~~Security headers: HSTS, X-Frame-Options, X-Content-Type-Options, CSP.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/sessions~~ | ~~`WithSessions()`~~ | ~~Server-side sessions (cookie store). Web session management alongside Authentik tokens.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/authz~~ | ~~`WithAuthz()`~~ | ~~Casbin-based authorisation. Policy-driven access control via RBAC.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/httpsign~~ | ~~`WithHTTPSign()`~~ | ~~HTTP signature verification. HMAC-SHA256 with extensible options.~~ | ~~**Done**~~ |
|
||||||
|
|
||||||
|
### Performance & Reliability
|
||||||
|
|
||||||
|
| Plugin | Option | Purpose | Priority |
|
||||||
|
|--------|--------|---------|----------|
|
||||||
|
| ~~gin-contrib/cache~~ | ~~`WithCache()`~~ | ~~Response caching (in-memory). GET response caching with TTL, lazy eviction.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/timeout~~ | ~~`WithTimeout()`~~ | ~~Per-request timeouts.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/gzip~~ | ~~`WithGzip()`~~ | ~~Gzip response compression.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/brotli~~ | ~~`WithBrotli()`~~ | ~~Brotli compression via `andybalholm/brotli`. Custom middleware (gin-contrib stub empty).~~ | ~~**Done**~~ |
|
||||||
|
|
||||||
|
### Observability
|
||||||
|
|
||||||
|
| Plugin | Option | Purpose | Priority |
|
||||||
|
|--------|--------|---------|----------|
|
||||||
|
| ~~gin-contrib/slog~~ | ~~`WithSlog()`~~ | ~~Structured request logging via slog.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/pprof~~ | ~~`WithPprof()`~~ | ~~Runtime profiling endpoints at /debug/pprof/. Flag-based mount.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/expvar~~ | ~~`WithExpvar()`~~ | ~~Go runtime metrics at /debug/vars. Flag-based mount.~~ | ~~**Done**~~ |
|
||||||
|
| ~~otelgin~~ | ~~`WithTracing()`~~ | ~~OpenTelemetry distributed tracing. W3C traceparent propagation.~~ | ~~**Done**~~ |
|
||||||
|
|
||||||
|
### Content & Streaming
|
||||||
|
|
||||||
|
| Plugin | Option | Purpose | Priority |
|
||||||
|
|--------|--------|---------|----------|
|
||||||
|
| ~~gin-contrib/static~~ | ~~`WithStatic()`~~ | ~~Serve static files.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/sse~~ | ~~`WithSSE()`~~ | ~~Server-Sent Events. Custom SSEBroker with channel filtering, GET /events.~~ | ~~**Done**~~ |
|
||||||
|
| ~~gin-contrib/location~~ | ~~`WithLocation()`~~ | ~~Auto-detect scheme/host from X-Forwarded-* headers.~~ | ~~**Done**~~ |
|
||||||
|
|
||||||
|
### Query Layer
|
||||||
|
|
||||||
|
| Plugin | Option | Purpose | Priority |
|
||||||
|
|--------|--------|---------|----------|
|
||||||
|
| ~~99designs/gqlgen~~ | ~~`WithGraphQL()`~~ | ~~GraphQL endpoint at `/graphql` + optional playground. Accepts gqlgen ExecutableSchema.~~ | ~~**Done**~~ |
|
||||||
|
|
||||||
|
The GraphQL schema can be generated from the same Go Input/Output structs that define the REST endpoints. gqlgen produces an `http.Handler` that mounts directly on Gin. Subsystems opt-in via:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Subsystems that want GraphQL implement this alongside RouteGroup
|
||||||
|
type ResolverGroup interface {
|
||||||
|
// RegisterResolvers adds query/mutation resolvers to the GraphQL schema
|
||||||
|
RegisterResolvers(schema *graphql.Schema)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This means a subsystem like go-ml exposes:
|
||||||
|
- **REST:** `POST /v1/ml/generate` (existing)
|
||||||
|
- **GraphQL:** `mutation { mlGenerate(prompt: "...", backend: "mlx") { response, model } }` (same handler)
|
||||||
|
- **MCP:** `ml_generate` tool (existing)
|
||||||
|
|
||||||
|
Four protocols, one set of handlers.
|
||||||
|
|
||||||
|
### Ecosystem Integration
|
||||||
|
|
||||||
|
| Plugin | Option | Purpose | Priority |
|
||||||
|
|--------|--------|---------|----------|
|
||||||
|
| ~~gin-contrib/i18n~~ | ~~`WithI18n()`~~ | ~~Locale detection via Accept-Language. Custom middleware using `golang.org/x/text/language`.~~ | ~~**Done**~~ |
|
||||||
|
| [gin-contrib/graceful](https://github.com/gin-contrib/graceful) | — | Already implemented in Engine.Serve(). Could swap to this for more robust lifecycle management if needed. | — |
|
||||||
|
| [gin-contrib/requestid](https://github.com/gin-contrib/requestid) | — | Already implemented. Theirs uses UUID, ours uses hex. Could swap for standards compliance. | — |
|
||||||
|
|
||||||
|
### Implementation Order
|
||||||
|
|
||||||
|
**Wave 1 (gateway hardening):** ~~Authentik, secure, slog, timeout, gzip, static~~ **DONE** (20 Feb 2026)
|
||||||
|
**Wave 2 (performance + auth):** ~~cache, sessions, authz, brotli~~ **DONE** (20 Feb 2026)
|
||||||
|
**Wave 3 (network + streaming):** ~~httpsign, sse, location, i18n, gqlgen~~ **DONE** (20 Feb 2026)
|
||||||
|
**Wave 4 (observability):** ~~pprof, expvar, tracing~~ **DONE** (21 Feb 2026)
|
||||||
|
|
||||||
|
Each wave adds `With*()` options + tests. No breaking changes — existing code continues to work without any new options enabled.
|
||||||
|
|
||||||
|
## Authentik Integration
|
||||||
|
|
||||||
|
[Authentik](https://goauthentik.io/) is the identity provider and edge auth proxy. It handles user registration, login, MFA, social auth, SAML, and OIDC — so go-api doesn't have to.
|
||||||
|
|
||||||
|
### Two Integration Modes
|
||||||
|
|
||||||
|
**1. Forward Auth (web traffic)**
|
||||||
|
|
||||||
|
Traefik sits in front of go-api. For web routes, Traefik's `forwardAuth` middleware checks with Authentik before passing the request through. Authentik handles login flows, session cookies, and consent. go-api receives pre-authenticated requests with identity headers.
|
||||||
|
|
||||||
|
```
|
||||||
|
Browser → Traefik → Authentik (forward auth) → go-api
|
||||||
|
↓
|
||||||
|
Login page (if unauthenticated)
|
||||||
|
```
|
||||||
|
|
||||||
|
go-api reads trusted headers set by Authentik:
|
||||||
|
```
|
||||||
|
X-Authentik-Username: alice
|
||||||
|
X-Authentik-Groups: admins,developers
|
||||||
|
X-Authentik-Email: alice@example.com
|
||||||
|
X-Authentik-Uid: <uuid>
|
||||||
|
X-Authentik-Jwt: <signed token>
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. OIDC Token Validation (API traffic)**
|
||||||
|
|
||||||
|
API clients (SDKs, CLI tools, network peers) authenticate directly with Authentik's OAuth2 token endpoint, then send the JWT to go-api. go-api validates the JWT using Authentik's OIDC discovery endpoint (`.well-known/openid-configuration`).
|
||||||
|
|
||||||
|
```
|
||||||
|
SDK client → Authentik (token endpoint) → receives JWT
|
||||||
|
SDK client → go-api (Authorization: Bearer <jwt>) → validates via OIDC
|
||||||
|
```
|
||||||
|
|
||||||
|
### Implementation in go-api
|
||||||
|
|
||||||
|
```go
|
||||||
|
engine := api.New(
|
||||||
|
api.WithAuthentik(api.AuthentikConfig{
|
||||||
|
Issuer: "https://auth.lthn.ai/application/o/core-api/",
|
||||||
|
ClientID: "core-api",
|
||||||
|
TrustedProxy: true, // Trust X-Authentik-* headers from Traefik
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
`WithAuthentik()` adds middleware that:
|
||||||
|
1. Checks for `X-Authentik-Jwt` header (forward auth mode) — validates signature, extracts claims
|
||||||
|
2. Falls back to `Authorization: Bearer <jwt>` header (direct OIDC mode) — validates via JWKS
|
||||||
|
3. Populates `c.Set("user", AuthentikUser{...})` in the Gin context for handlers to use
|
||||||
|
4. Skips /health, /swagger, and any public paths
|
||||||
|
|
||||||
|
```go
|
||||||
|
// In any handler:
|
||||||
|
func (r *Routes) ListItems(c *gin.Context) {
|
||||||
|
user := api.GetUser(c) // Returns *AuthentikUser or nil
|
||||||
|
if user == nil {
|
||||||
|
c.JSON(401, api.Fail("unauthorised", "Authentication required"))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// user.Username, user.Groups, user.Email, user.UID available
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Auth Layers
|
||||||
|
|
||||||
|
```
|
||||||
|
Authentik (identity) → WHO is this? (user, groups, email)
|
||||||
|
↓
|
||||||
|
go-api middleware → IS their token valid? (JWT verification)
|
||||||
|
↓
|
||||||
|
Casbin authz (optional) → CAN they do this? (role → endpoint policies)
|
||||||
|
↓
|
||||||
|
Handler → DOES this (business logic)
|
||||||
|
```
|
||||||
|
|
||||||
|
Phase 1 bearer auth continues to work alongside Authentik — useful for service-to-service tokens, CI/CD, and development. `WithBearerAuth` and `WithAuthentik` can coexist.
|
||||||
|
|
||||||
|
### Authentik Deployment
|
||||||
|
|
||||||
|
Authentik runs as a Docker service alongside go-api, fronted by Traefik:
|
||||||
|
- **auth.lthn.ai** — Authentik UI + OIDC endpoints (production)
|
||||||
|
- **auth.leth.in** — Authentik for devnet/testnet
|
||||||
|
- Traefik routes `/outpost.goauthentik.io/` to Authentik's embedded outpost for forward auth
|
||||||
|
|
||||||
|
### Dependencies
|
||||||
|
|
||||||
|
| Package | Purpose |
|
||||||
|
|---------|---------|
|
||||||
|
| `github.com/coreos/go-oidc/v3` | OIDC discovery + JWT validation |
|
||||||
|
| `golang.org/x/oauth2` | OAuth2 token exchange (for server-side flows) |
|
||||||
|
|
||||||
|
Both are standard Go libraries with no heavy dependencies.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- gRPC gateway
|
||||||
|
- Built-in user registration/login (Authentik handles this)
|
||||||
|
- API versioning beyond /v1/ prefix
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
### Phase 1 (Done)
|
||||||
|
|
||||||
|
1. ~~`core api serve` starts a Gin server with registered subsystem routes~~
|
||||||
|
2. ~~WebSocket subscriptions work alongside REST~~
|
||||||
|
3. ~~Swagger UI accessible at `/swagger/`~~
|
||||||
|
4. ~~All endpoints return consistent Response envelope~~
|
||||||
|
5. ~~Bearer token auth protects all routes~~
|
||||||
|
6. ~~First subsystem integration (go-ml/api/) proves the pattern~~
|
||||||
|
|
||||||
|
### Phase 2 (Done)
|
||||||
|
|
||||||
|
7. ~~Security headers, compression, and caching active in production~~
|
||||||
|
8. ~~Session-based auth alongside bearer tokens~~
|
||||||
|
9. ~~HTTP signature verification for Lethean network peers~~
|
||||||
|
10. ~~Static file serving for docs site and SDK downloads~~
|
||||||
|
11. ~~GraphQL endpoint at `/graphql` with playground~~
|
||||||
|
|
||||||
|
### Phase 3 (Done)
|
||||||
|
|
||||||
|
12. ~~`core api spec` emits valid OpenAPI 3.1 JSON via runtime SpecBuilder~~
|
||||||
|
13. ~~`core api sdk` generates SDKs for 11 languages via openapi-generator-cli~~
|
||||||
|
14. ~~MCP tools bridged to REST endpoints via ToolBridge + BridgeToAPI~~
|
||||||
|
15. ~~OpenAPI spec includes Response[T] envelope wrapping~~
|
||||||
|
16. ~~Spec export to file in JSON and YAML formats~~
|
||||||
1503
docs/plans/completed/2026-02-20-go-api-plan-original.md
Normal file
1503
docs/plans/completed/2026-02-20-go-api-plan-original.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,128 @@
|
||||||
|
# CLI Meta-Package Restructure — Design
|
||||||
|
|
||||||
|
**Goal:** Transform `core/cli` from a 35K LOC monolith into a thin assembly repo that ships variant binaries. Domain repos own their commands. `go/pkg/cli` is the only import any domain package needs for CLI concerns.
|
||||||
|
|
||||||
|
**Architecture:** Commands register as framework services via `cli.WithCommands()`, passed to `cli.Main()`. Command code lives in the domain repos that own the business logic. The cli repo is a thin `main.go` that wires them together.
|
||||||
|
|
||||||
|
**Tech Stack:** go/pkg/cli (wraps cobra + charmbracelet), Core framework lifecycle, Taskfile
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. CLI SDK — The Single Import
|
||||||
|
|
||||||
|
`forge.lthn.ai/core/go/pkg/cli` is the **only** import domain packages use for CLI concerns. It wraps cobra, charmbracelet, and stdlib behind a stable API. If the underlying libraries change, only `go/pkg/cli` is touched — every domain repo is insulated.
|
||||||
|
|
||||||
|
### Already done
|
||||||
|
|
||||||
|
- **Cobra:** `Command` type alias, `NewCommand()`, `NewGroup()`, `NewRun()`, flag helpers (`StringFlag`, `BoolFlag`, `IntFlag`, `StringSliceFlag`), arg validators
|
||||||
|
- **Output:** `Success()`, `Error()`, `Warn()`, `Info()`, `Table`, `Section()`, `Label()`, `Task()`, `Hint()`
|
||||||
|
- **Prompts:** `Confirm()`, `Question()`, `Choose()`, `ChooseMulti()` with grammar-based action variants
|
||||||
|
- **Styles:** 17 pre-built styles, `AnsiStyle` builder, Tailwind colour constants (47 hex values)
|
||||||
|
- **Glyphs:** `:check:`, `:cross:`, `:warn:` etc. with Unicode/Emoji/ASCII themes
|
||||||
|
- **Layout:** HLCRF composite renderer (Header/Left/Content/Right/Footer)
|
||||||
|
- **Errors:** `Wrap()`, `WrapVerb()`, `ExitError`, `Is()`, `As()`
|
||||||
|
- **Logging:** `LogDebug()`, `LogInfo()`, `LogWarn()`, `LogError()`, `LogSecurity()`
|
||||||
|
- **TUI primitives:** `Spinner`, `ProgressBar`, `InteractiveList`, `TextInput`, `Viewport`, `RunTUI`
|
||||||
|
- **Command registration:** `WithCommands(name, fn)` — registers commands as framework services
|
||||||
|
|
||||||
|
### Stubbed for later (interface exists, returns simple fallback)
|
||||||
|
|
||||||
|
- `Form(fields []FormField) (map[string]string, error)` — multi-field form (backed by huh later)
|
||||||
|
- `FilePicker(opts ...FilePickerOption) (string, error)` — file browser
|
||||||
|
- `Tabs(items []TabItem) error` — tabbed content panes
|
||||||
|
|
||||||
|
### Rule
|
||||||
|
|
||||||
|
Domain packages import `forge.lthn.ai/core/go/pkg/cli` and **nothing else** for CLI concerns. No `cobra`, no `lipgloss`, no `bubbletea`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Command Registration — Framework Lifecycle
|
||||||
|
|
||||||
|
Commands register through the Core framework's service lifecycle, not through global state or `init()` functions.
|
||||||
|
|
||||||
|
### The contract
|
||||||
|
|
||||||
|
Each domain repo exports an `Add*Commands(root *cli.Command)` function. The CLI binary wires it in via `cli.WithCommands()`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// go-ai/cmd/daemon/cmd.go
|
||||||
|
package daemon
|
||||||
|
|
||||||
|
import "forge.lthn.ai/core/go/pkg/cli"
|
||||||
|
|
||||||
|
// AddDaemonCommand adds the 'daemon' command group to the root.
|
||||||
|
func AddDaemonCommand(root *cli.Command) {
|
||||||
|
daemonCmd := cli.NewGroup("daemon", "Manage the core daemon", "")
|
||||||
|
root.AddCommand(daemonCmd)
|
||||||
|
// subcommands...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
No `init()`. No blank imports. No `cli.RegisterCommands()`.
|
||||||
|
|
||||||
|
### How it works
|
||||||
|
|
||||||
|
`cli.WithCommands(name, fn)` wraps the registration function as a framework service implementing `Startable`. During `Core.ServiceStartup()`, the service's `OnStartup()` casts `Core.App` to `*cobra.Command` and calls the registration function. Core services (i18n, log, workspace) start first since they're registered before command services.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// cli/main.go
|
||||||
|
func main() {
|
||||||
|
cli.Main(
|
||||||
|
cli.WithCommands("config", config.AddConfigCommands),
|
||||||
|
cli.WithCommands("doctor", doctor.AddDoctorCommands),
|
||||||
|
// ...
|
||||||
|
)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Migration status (completed)
|
||||||
|
|
||||||
|
| Source | Destination | Status |
|
||||||
|
|--------|-------------|--------|
|
||||||
|
| `cmd/dev, setup, qa, docs, gitcmd, monitor` | `go-devops/cmd/` | Done |
|
||||||
|
| `cmd/lab` | `go-ai/cmd/` | Done |
|
||||||
|
| `cmd/workspace` | `go-agentic/cmd/` | Done |
|
||||||
|
| `cmd/go` | `core/go/cmd/gocmd` | Done |
|
||||||
|
| `cmd/vanity-import, community` | `go-devops/cmd/` | Done |
|
||||||
|
| `cmd/updater` | `go-update` | Done (own repo) |
|
||||||
|
| `cmd/daemon, mcpcmd, security` | `go-ai/cmd/` | Done |
|
||||||
|
| `cmd/crypt` | `go-crypt/cmd/` | Done |
|
||||||
|
| `cmd/rag` | `go-rag/cmd/` | Done |
|
||||||
|
| `cmd/unifi` | `go-netops/cmd/` | Done |
|
||||||
|
| `cmd/api` | `go-api/cmd/` | Done |
|
||||||
|
| `cmd/collect, forge, gitea` | `go-scm/cmd/` | Done |
|
||||||
|
| `cmd/deploy, prod, vm` | `go-devops/cmd/` | Done |
|
||||||
|
|
||||||
|
### Stays in cli/ (meta/framework commands)
|
||||||
|
|
||||||
|
`config`, `doctor`, `help`, `module`, `pkgcmd`, `plugin`, `session`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Variant Binaries (future)
|
||||||
|
|
||||||
|
The cli/ repo can produce variant binaries by creating multiple `main.go` files that wire different sets of commands.
|
||||||
|
|
||||||
|
```
|
||||||
|
cli/
|
||||||
|
├── main.go # Current — meta commands only
|
||||||
|
├── cmd/core-full/main.go # Full CLI — all ecosystem commands
|
||||||
|
├── cmd/core-ci/main.go # CI agent dispatch + SCM
|
||||||
|
├── cmd/core-mlx/main.go # ML inference subprocess
|
||||||
|
└── cmd/core-ops/main.go # DevOps + infra management
|
||||||
|
```
|
||||||
|
|
||||||
|
Each variant calls `cli.Main()` with its specific `cli.WithCommands()` set. No blank imports needed.
|
||||||
|
|
||||||
|
### Why variants matter
|
||||||
|
|
||||||
|
- `core-mlx` ships to the homelab as a ~10MB binary, not 50MB with devops/forge/netops
|
||||||
|
- `core-ci` deploys to agent machines without ML or CGO dependencies
|
||||||
|
- Adding a new variant = one new `main.go` with the right `WithCommands` calls
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Current State
|
||||||
|
|
||||||
|
cli/ has 7 meta packages, one `main.go`, and zero business logic. Everything else lives in the domain repos that own it. Total cli/ LOC is ~2K.
|
||||||
1724
docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md
Normal file
1724
docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md
Normal file
File diff suppressed because it is too large
Load diff
286
docs/plans/completed/2026-02-21-go-forge-design.md
Normal file
286
docs/plans/completed/2026-02-21-go-forge-design.md
Normal file
|
|
@ -0,0 +1,286 @@
|
||||||
|
# go-forge Design Document
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
**go-forge** is a full-coverage Go client for the Forgejo API (450 endpoints, 284 paths, 229 types). It uses a generic `Resource[T, C, U]` pattern for CRUD operations (91% of endpoints) and hand-written methods for 39 unique action endpoints. Types are generated from Forgejo's `swagger.v1.json` spec.
|
||||||
|
|
||||||
|
**Module path:** `forge.lthn.ai/core/go-forge`
|
||||||
|
|
||||||
|
**Origin:** Extracted from `go-scm/forge/` (45 methods covering 10% of API), expanded to full coverage.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
forge.lthn.ai/core/go-forge
|
||||||
|
├── client.go # HTTP client: auth, headers, rate limiting, context.Context
|
||||||
|
├── pagination.go # Generic paginated request helper
|
||||||
|
├── resource.go # Resource[T, C, U] generic CRUD (List/Get/Create/Update/Delete)
|
||||||
|
├── errors.go # Typed error handling (APIError, NotFound, Forbidden, etc.)
|
||||||
|
├── forge.go # Top-level Forge client aggregating all services
|
||||||
|
│
|
||||||
|
├── types/ # Generated from swagger.v1.json
|
||||||
|
│ ├── generate.go # //go:generate directive
|
||||||
|
│ ├── repo.go # Repository, CreateRepoOption, EditRepoOption
|
||||||
|
│ ├── issue.go # Issue, CreateIssueOption, EditIssueOption
|
||||||
|
│ ├── pr.go # PullRequest, CreatePullRequestOption
|
||||||
|
│ ├── user.go # User, CreateUserOption
|
||||||
|
│ ├── org.go # Organisation, CreateOrgOption
|
||||||
|
│ ├── team.go # Team, CreateTeamOption
|
||||||
|
│ ├── label.go # Label, CreateLabelOption
|
||||||
|
│ ├── release.go # Release, CreateReleaseOption
|
||||||
|
│ ├── branch.go # Branch, BranchProtection
|
||||||
|
│ ├── milestone.go # Milestone, CreateMilestoneOption
|
||||||
|
│ ├── hook.go # Hook, CreateHookOption
|
||||||
|
│ ├── key.go # DeployKey, PublicKey, GPGKey
|
||||||
|
│ ├── notification.go # NotificationThread, NotificationSubject
|
||||||
|
│ ├── package.go # Package, PackageFile
|
||||||
|
│ ├── action.go # ActionRunner, ActionSecret, ActionVariable
|
||||||
|
│ ├── commit.go # Commit, CommitStatus, CombinedStatus
|
||||||
|
│ ├── content.go # ContentsResponse, FileOptions
|
||||||
|
│ ├── wiki.go # WikiPage, WikiPageMetaData
|
||||||
|
│ ├── review.go # PullReview, PullReviewComment
|
||||||
|
│ ├── reaction.go # Reaction
|
||||||
|
│ ├── topic.go # TopicResponse
|
||||||
|
│ ├── misc.go # Markdown, License, GitignoreTemplate, NodeInfo
|
||||||
|
│ ├── admin.go # Cron, QuotaGroup, QuotaRule
|
||||||
|
│ ├── activity.go # Activity, Feed
|
||||||
|
│ └── common.go # Shared types: Permission, ExternalTracker, etc.
|
||||||
|
│
|
||||||
|
├── repos.go # RepoService: CRUD + fork, mirror, transfer, template
|
||||||
|
├── issues.go # IssueService: CRUD + pin, deadline, reactions, stopwatch
|
||||||
|
├── pulls.go # PullService: CRUD + merge, update, reviews, dismiss
|
||||||
|
├── orgs.go # OrgService: CRUD + members, avatar, block, hooks
|
||||||
|
├── users.go # UserService: CRUD + keys, followers, starred, settings
|
||||||
|
├── teams.go # TeamService: CRUD + members, repos
|
||||||
|
├── admin.go # AdminService: users, orgs, cron, runners, quota, unadopted
|
||||||
|
├── branches.go # BranchService: CRUD + protection rules
|
||||||
|
├── releases.go # ReleaseService: CRUD + assets
|
||||||
|
├── labels.go # LabelService: repo + org + issue labels
|
||||||
|
├── webhooks.go # WebhookService: CRUD + test hook
|
||||||
|
├── notifications.go # NotificationService: list, mark read
|
||||||
|
├── packages.go # PackageService: list, get, delete
|
||||||
|
├── actions.go # ActionsService: runners, secrets, variables, workflow dispatch
|
||||||
|
├── contents.go # ContentService: file read/write/delete via API
|
||||||
|
├── wiki.go # WikiService: pages
|
||||||
|
├── commits.go # CommitService: status, notes, diff
|
||||||
|
├── misc.go # MiscService: markdown, licenses, gitignore, nodeinfo
|
||||||
|
│
|
||||||
|
├── config.go # URL/token resolution: env → config file → flags
|
||||||
|
│
|
||||||
|
├── cmd/forgegen/ # Code generator: swagger.v1.json → types/*.go
|
||||||
|
│ ├── main.go
|
||||||
|
│ ├── parser.go # Parse OpenAPI 2.0 definitions
|
||||||
|
│ ├── generator.go # Render Go source files
|
||||||
|
│ └── templates/ # Go text/template files for codegen
|
||||||
|
│
|
||||||
|
└── testdata/
|
||||||
|
└── swagger.v1.json # Pinned spec for testing + generation
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Design Decisions
|
||||||
|
|
||||||
|
### 1. Generic Resource[T, C, U]
|
||||||
|
|
||||||
|
Three type parameters: T (resource type), C (create options), U (update options).
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Resource[T any, C any, U any] struct {
|
||||||
|
client *Client
|
||||||
|
path string // e.g. "/api/v1/repos/{owner}/{repo}/issues"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Resource[T, C, U]) List(ctx context.Context, params Params, opts ListOptions) ([]T, error)
|
||||||
|
func (r *Resource[T, C, U]) Get(ctx context.Context, params Params, id string) (*T, error)
|
||||||
|
func (r *Resource[T, C, U]) Create(ctx context.Context, params Params, body *C) (*T, error)
|
||||||
|
func (r *Resource[T, C, U]) Update(ctx context.Context, params Params, id string, body *U) (*T, error)
|
||||||
|
func (r *Resource[T, C, U]) Delete(ctx context.Context, params Params, id string) error
|
||||||
|
```
|
||||||
|
|
||||||
|
`Params` is `map[string]string` resolving path variables: `{"owner": "core", "repo": "go-forge"}`.
|
||||||
|
|
||||||
|
This covers 411 of 450 endpoints (91%).
|
||||||
|
|
||||||
|
### 2. Service Structs Embed Resource
|
||||||
|
|
||||||
|
```go
|
||||||
|
type IssueService struct {
|
||||||
|
Resource[types.Issue, types.CreateIssueOption, types.EditIssueOption]
|
||||||
|
}
|
||||||
|
|
||||||
|
// CRUD comes free. Actions are hand-written:
|
||||||
|
func (s *IssueService) Pin(ctx context.Context, owner, repo string, index int64) error
|
||||||
|
func (s *IssueService) SetDeadline(ctx context.Context, owner, repo string, index int64, deadline *time.Time) error
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Top-Level Forge Client
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Forge struct {
|
||||||
|
client *Client
|
||||||
|
Repos *RepoService
|
||||||
|
Issues *IssueService
|
||||||
|
Pulls *PullService
|
||||||
|
Orgs *OrgService
|
||||||
|
Users *UserService
|
||||||
|
Teams *TeamService
|
||||||
|
Admin *AdminService
|
||||||
|
Branches *BranchService
|
||||||
|
Releases *ReleaseService
|
||||||
|
Labels *LabelService
|
||||||
|
Webhooks *WebhookService
|
||||||
|
Notifications *NotificationService
|
||||||
|
Packages *PackageService
|
||||||
|
Actions *ActionsService
|
||||||
|
Contents *ContentService
|
||||||
|
Wiki *WikiService
|
||||||
|
Commits *CommitService
|
||||||
|
Misc *MiscService
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewForge(url, token string, opts ...Option) *Forge
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Codegen from swagger.v1.json
|
||||||
|
|
||||||
|
The `cmd/forgegen/` tool reads the OpenAPI 2.0 spec and generates:
|
||||||
|
- Go struct definitions with JSON tags and doc comments
|
||||||
|
- Enum constants
|
||||||
|
- Type mapping (OpenAPI → Go)
|
||||||
|
|
||||||
|
229 type definitions → ~25 grouped Go files in `types/`.
|
||||||
|
|
||||||
|
Type mapping rules:
|
||||||
|
| OpenAPI | Go |
|
||||||
|
|---------|-----|
|
||||||
|
| `string` | `string` |
|
||||||
|
| `string` + `date-time` | `time.Time` |
|
||||||
|
| `integer` + `int64` | `int64` |
|
||||||
|
| `integer` | `int` |
|
||||||
|
| `boolean` | `bool` |
|
||||||
|
| `array` of T | `[]T` |
|
||||||
|
| `$ref` | `*T` (pointer) |
|
||||||
|
| nullable | pointer type |
|
||||||
|
| `binary` | `[]byte` |
|
||||||
|
|
||||||
|
### 5. HTTP Client
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Client struct {
|
||||||
|
baseURL string
|
||||||
|
token string
|
||||||
|
httpClient *http.Client
|
||||||
|
userAgent string
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(url, token string, opts ...Option) *Client
|
||||||
|
|
||||||
|
func (c *Client) Get(ctx context.Context, path string, out any) error
|
||||||
|
func (c *Client) Post(ctx context.Context, path string, body, out any) error
|
||||||
|
func (c *Client) Patch(ctx context.Context, path string, body, out any) error
|
||||||
|
func (c *Client) Put(ctx context.Context, path string, body, out any) error
|
||||||
|
func (c *Client) Delete(ctx context.Context, path string) error
|
||||||
|
```
|
||||||
|
|
||||||
|
Options: `WithHTTPClient`, `WithUserAgent`, `WithRateLimit`, `WithLogger`.
|
||||||
|
|
||||||
|
### 6. Pagination
|
||||||
|
|
||||||
|
Forgejo uses `page` + `limit` query params and `X-Total-Count` response header.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type ListOptions struct {
|
||||||
|
Page int
|
||||||
|
Limit int // default 50, max configurable
|
||||||
|
}
|
||||||
|
|
||||||
|
type PagedResult[T any] struct {
|
||||||
|
Items []T
|
||||||
|
TotalCount int
|
||||||
|
Page int
|
||||||
|
HasMore bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListAll fetches all pages automatically.
|
||||||
|
func (r *Resource[T, C, U]) ListAll(ctx context.Context, params Params) ([]T, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7. Error Handling
|
||||||
|
|
||||||
|
```go
|
||||||
|
type APIError struct {
|
||||||
|
StatusCode int
|
||||||
|
Message string
|
||||||
|
URL string
|
||||||
|
}
|
||||||
|
|
||||||
|
func IsNotFound(err error) bool
|
||||||
|
func IsForbidden(err error) bool
|
||||||
|
func IsConflict(err error) bool
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8. Config Resolution (from go-scm/forge)
|
||||||
|
|
||||||
|
Priority: flags → environment → config file.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func NewFromConfig(flagURL, flagToken string) (*Forge, error)
|
||||||
|
func ResolveConfig(flagURL, flagToken string) (url, token string, err error)
|
||||||
|
func SaveConfig(url, token string) error
|
||||||
|
```
|
||||||
|
|
||||||
|
Env vars: `FORGE_URL`, `FORGE_TOKEN`. Config file: `~/.config/forge/config.json`.
|
||||||
|
|
||||||
|
## API Coverage
|
||||||
|
|
||||||
|
| Category | Endpoints | CRUD | Actions |
|
||||||
|
|----------|-----------|------|---------|
|
||||||
|
| Repository | 175 | 165 | 10 (fork, mirror, transfer, template, avatar, diffpatch) |
|
||||||
|
| User | 74 | 70 | 4 (avatar, GPG verify) |
|
||||||
|
| Issue | 67 | 57 | 10 (pin, deadline, reactions, stopwatch, labels) |
|
||||||
|
| Organisation | 63 | 59 | 4 (avatar, block/unblock) |
|
||||||
|
| Admin | 39 | 35 | 4 (cron run, rename, adopt, quota set) |
|
||||||
|
| Miscellaneous | 12 | 7 | 5 (markdown render, markup, nodeinfo) |
|
||||||
|
| Notification | 7 | 7 | 0 |
|
||||||
|
| ActivityPub | 6 | 3 | 3 (inbox POST) |
|
||||||
|
| Package | 4 | 4 | 0 |
|
||||||
|
| Settings | 4 | 4 | 0 |
|
||||||
|
| **Total** | **450** | **411** | **39** |
|
||||||
|
|
||||||
|
## Integration Points
|
||||||
|
|
||||||
|
### go-api
|
||||||
|
|
||||||
|
Services implement `DescribableGroup` from go-api Phase 3, enabling:
|
||||||
|
- REST endpoint generation via ToolBridge
|
||||||
|
- Auto-generated OpenAPI spec
|
||||||
|
- Multi-language SDK codegen
|
||||||
|
|
||||||
|
### go-scm
|
||||||
|
|
||||||
|
go-scm/forge/ becomes a thin adapter importing go-forge types. Existing go-scm users are unaffected — the multi-provider abstraction layer stays.
|
||||||
|
|
||||||
|
### go-ai/mcp
|
||||||
|
|
||||||
|
The MCP subsystem can register go-forge operations as MCP tools, giving AI agents full Forgejo API access.
|
||||||
|
|
||||||
|
## 39 Unique Action Methods
|
||||||
|
|
||||||
|
These require hand-written implementation:
|
||||||
|
|
||||||
|
**Repository:** migrate, fork, generate (template), transfer, accept/reject transfer, mirror sync, push mirror sync, avatar, diffpatch, contents (multi-file modify)
|
||||||
|
|
||||||
|
**Pull Requests:** merge, update (rebase), submit review, dismiss/undismiss review
|
||||||
|
|
||||||
|
**Issues:** pin, set deadline, add reaction, start/stop stopwatch, add issue labels
|
||||||
|
|
||||||
|
**Comments:** add reaction
|
||||||
|
|
||||||
|
**Admin:** run cron task, adopt unadopted, rename user, set quota groups
|
||||||
|
|
||||||
|
**Misc:** render markdown, render raw markdown, render markup, GPG key verify
|
||||||
|
|
||||||
|
**ActivityPub:** inbox POST (actor, repo, user)
|
||||||
|
|
||||||
|
**Actions:** dispatch workflow
|
||||||
|
|
||||||
|
**Git:** set note on commit, test webhook
|
||||||
2549
docs/plans/completed/2026-02-21-go-forge-plan.md
Normal file
2549
docs/plans/completed/2026-02-21-go-forge-plan.md
Normal file
File diff suppressed because it is too large
Load diff
30
docs/plans/completed/cli-meta-package.md
Normal file
30
docs/plans/completed/cli-meta-package.md
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
# CLI Meta-Package Restructure — Completed
|
||||||
|
|
||||||
|
**Completed:** 22 Feb 2026
|
||||||
|
|
||||||
|
## What Was Done
|
||||||
|
|
||||||
|
`pkg/cli` was extracted from `core/go` into its own Go module at `forge.lthn.ai/core/cli`. This made the CLI SDK a first-class, independently versioned package rather than a subdirectory of the Go foundation repo.
|
||||||
|
|
||||||
|
Following the extraction, an ecosystem-wide import path migration updated all consumers from the old path to the new one:
|
||||||
|
|
||||||
|
- Old: `forge.lthn.ai/core/go/pkg/cli`
|
||||||
|
- New: `forge.lthn.ai/core/cli/pkg/cli`
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- **147+ files** updated across **10 repos**
|
||||||
|
- All repos build clean after migration
|
||||||
|
|
||||||
|
## Repos Migrated
|
||||||
|
|
||||||
|
`core/cli`, `core/go`, `go-devops`, `go-ai`, `go-agentic`, `go-crypt`, `go-rag`, `go-scm`, `go-api`, `go-update`
|
||||||
|
|
||||||
|
## Key Outcomes
|
||||||
|
|
||||||
|
- `forge.lthn.ai/core/cli/pkg/cli` is the single import for all CLI concerns across the ecosystem
|
||||||
|
- Domain repos are insulated from cobra, lipgloss, and bubbletea — only `pkg/cli` imports them
|
||||||
|
- Command registration uses the Core framework lifecycle via `cli.WithCommands()` — no `init()`, no global state
|
||||||
|
- `core/cli` is a thin assembly repo (~2K LOC) with 7 meta packages; all business logic lives in domain repos
|
||||||
|
- Variant binary pattern established: multiple `main.go` files can wire different `WithCommands` sets for targeted binaries (core-ci, core-mlx, core-ops, etc.)
|
||||||
|
- Command migration from the old `core/cli` monolith to domain repos was completed in full (13 command groups moved)
|
||||||
39
docs/plans/completed/cli-sdk-expansion.md
Normal file
39
docs/plans/completed/cli-sdk-expansion.md
Normal file
|
|
@ -0,0 +1,39 @@
|
||||||
|
# CLI SDK Expansion — Completion Summary
|
||||||
|
|
||||||
|
**Completed:** 21 February 2026
|
||||||
|
**Module:** `forge.lthn.ai/core/go/pkg/cli` (later migrated to `forge.lthn.ai/core/cli`)
|
||||||
|
**Status:** Complete — all TUI primitives shipped, then extracted to core/cli
|
||||||
|
|
||||||
|
## What Was Built
|
||||||
|
|
||||||
|
Extended `pkg/cli` with charmbracelet TUI primitives so domain repos only
|
||||||
|
import `core/cli` for all CLI concerns. Charmbracelet dependencies (bubbletea,
|
||||||
|
bubbles, lipgloss) are encapsulated behind our own types.
|
||||||
|
|
||||||
|
### Components added
|
||||||
|
|
||||||
|
| Component | File | Purpose |
|
||||||
|
|-----------|------|---------|
|
||||||
|
| RunTUI | `runtui.go` | Escape hatch with `Model`/`Msg`/`Cmd`/`KeyMsg` types |
|
||||||
|
| Spinner | `spinner.go` | Async handle with `Update()`, `Done()`, `Fail()` |
|
||||||
|
| ProgressBar | `progressbar.go` | `Increment()`, `Set()`, `SetMessage()`, `Done()` |
|
||||||
|
| InteractiveList | `list.go` | Keyboard navigation with terminal fallback |
|
||||||
|
| TextInput | `textinput.go` | Placeholder, masking, validation |
|
||||||
|
| Viewport | `viewport.go` | Scrollable content for logs, diffs, docs |
|
||||||
|
| Form (stub) | `form.go` | Interface defined, bufio fallback |
|
||||||
|
| FilePicker (stub) | `filepicker.go` | Interface defined, bufio fallback |
|
||||||
|
| Tabs (stub) | `tabs.go` | Interface defined, simple fallback |
|
||||||
|
|
||||||
|
### Subsequent migration
|
||||||
|
|
||||||
|
On 22 February 2026, `pkg/cli` was extracted from `core/go` into its own
|
||||||
|
module at `forge.lthn.ai/core/cli` and all imports were updated. The TUI
|
||||||
|
primitives now live in the standalone CLI module.
|
||||||
|
|
||||||
|
### Frame upgrade (follow-on)
|
||||||
|
|
||||||
|
The Frame layout system was upgraded to implement `tea.Model` directly on
|
||||||
|
22 February 2026 (in `core/cli`), adding bubbletea lifecycle, `KeyMap` for
|
||||||
|
configurable bindings, `Navigate()`/`Back()` for panel switching, and
|
||||||
|
lipgloss-based HLCRF rendering. This was a separate plan
|
||||||
|
(`frame-bubbletea`) that built on the SDK expansion.
|
||||||
57
docs/plans/completed/go-api.md
Normal file
57
docs/plans/completed/go-api.md
Normal file
|
|
@ -0,0 +1,57 @@
|
||||||
|
# go-api — Completion Summary
|
||||||
|
|
||||||
|
**Completed:** 21 February 2026
|
||||||
|
**Module:** `forge.lthn.ai/core/go-api`
|
||||||
|
**Status:** Phases 1–3 complete, 176 tests passing
|
||||||
|
|
||||||
|
## What Was Built
|
||||||
|
|
||||||
|
### Phase 1 — Core Framework (20 Feb 2026)
|
||||||
|
|
||||||
|
Gin-based HTTP engine with extensible middleware via `With*()` options. Key components:
|
||||||
|
|
||||||
|
- `RouteGroup` / `StreamGroup` interfaces — subsystems register their own endpoints
|
||||||
|
- `Response[T]` envelope — `OK()`, `Fail()`, `Paginated()` generics
|
||||||
|
- `Engine` — `New()`, `Register()`, `Handler()`, `Serve()` with graceful shutdown
|
||||||
|
- Bearer auth, request ID, and CORS middleware
|
||||||
|
- WebSocket endpoint wrapping a `go-ws` Hub
|
||||||
|
- Swagger UI at `/swagger/` with runtime spec serving
|
||||||
|
- `/health` endpoint always available without auth
|
||||||
|
- First integration proof in `go-ml/api/` (3 endpoints, 12 tests)
|
||||||
|
|
||||||
|
### Phase 2 — Gin Plugin Stack (20–21 Feb 2026)
|
||||||
|
|
||||||
|
17 middleware plugins added across four waves, all as drop-in `With*()` options:
|
||||||
|
|
||||||
|
| Wave | Plugins |
|
||||||
|
|------|---------|
|
||||||
|
| 1 — Gateway hardening | Authentik (OIDC + forward auth), secure headers, structured slog, timeouts, gzip, static files |
|
||||||
|
| 2 — Performance + auth | Brotli compression, in-memory response cache, server-side sessions, Casbin RBAC |
|
||||||
|
| 3 — Network + streaming | HTTP signature verification, SSE broker, reverse proxy detection, i18n locale, GraphQL |
|
||||||
|
| 4 — Observability | pprof, expvar, OpenTelemetry distributed tracing |
|
||||||
|
|
||||||
|
### Phase 3 — OpenAPI + SDK Codegen (21 Feb 2026)
|
||||||
|
|
||||||
|
Runtime spec generation (not swaggo annotations — incompatible with dynamic RouteGroups and `Response[T]` generics):
|
||||||
|
|
||||||
|
- `DescribableGroup` interface — opt-in OpenAPI metadata for route groups
|
||||||
|
- `ToolBridge` — converts MCP tool descriptors into `POST /{tool_name}` REST endpoints
|
||||||
|
- `SpecBuilder` — assembles full OpenAPI 3.1 JSON from registered groups at runtime
|
||||||
|
- Spec export to JSON and YAML (`core api spec`)
|
||||||
|
- SDK codegen wrapper for openapi-generator-cli, 11 languages (`core api sdk --lang go`)
|
||||||
|
- `go-ai` `mcp/registry.go` — generic `addToolRecorded[In,Out]` captures types in closures
|
||||||
|
- `go-ai` `mcp/bridge.go` — `BridgeToAPI()` populates ToolBridge from MCP tool registry
|
||||||
|
- CLI commands: `core api spec`, `core api sdk` (in `core/cli` dev branch)
|
||||||
|
|
||||||
|
## Key Outcomes
|
||||||
|
|
||||||
|
- **176 tests** across go-api (143), go-ai bridge (10), and CLI commands (4), all passing
|
||||||
|
- Zero internal ecosystem dependencies — subsystems import go-api, not the reverse
|
||||||
|
- Authentik (OIDC) and bearer token auth coexist; Casbin adds RBAC on top
|
||||||
|
- Four-protocol access pattern established: REST, GraphQL, WebSocket, MCP — same handlers
|
||||||
|
|
||||||
|
## Known Limitations
|
||||||
|
|
||||||
|
- Subsystem MCP tools registered via `mcp.AddTool` directly are excluded from the REST bridge (only the 10 built-in tools appear). Fix: pass `*Service` to `RegisterTools` instead of `*mcp.Server`.
|
||||||
|
- `structSchema` reflection handles flat structs only; nested structs are not recursed.
|
||||||
|
- `core api spec` currently emits a spec with only `/health`; full MCP wiring into the CLI command is pending.
|
||||||
37
docs/plans/completed/mcp-integration.md
Normal file
37
docs/plans/completed/mcp-integration.md
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
# MCP Integration — Completion Summary
|
||||||
|
|
||||||
|
**Completed:** 2026-02-05
|
||||||
|
**Plan:** `docs/plans/2026-02-05-mcp-integration.md`
|
||||||
|
|
||||||
|
## What Was Built
|
||||||
|
|
||||||
|
### RAG Tools (`pkg/mcp/tools_rag.go`)
|
||||||
|
Three MCP tools added to the existing `pkg/mcp` server:
|
||||||
|
- `rag_query` — semantic search against Qdrant vector DB
|
||||||
|
- `rag_ingest` — ingest a file or directory into a named collection
|
||||||
|
- `rag_collections` — list available Qdrant collections (with optional stats)
|
||||||
|
|
||||||
|
### Metrics Tools (`pkg/mcp/tools_metrics.go`)
|
||||||
|
Two MCP tools for agent activity tracking:
|
||||||
|
- `metrics_record` — write a typed event (agent_id, repo, arbitrary data) to JSONL storage
|
||||||
|
- `metrics_query` — query events with aggregation by type, repo, and agent; supports human-friendly duration strings (7d, 24h)
|
||||||
|
|
||||||
|
Also added `parseDuration()` helper for "Nd"/"Nh"/"Nm" duration strings.
|
||||||
|
|
||||||
|
### `core mcp serve` Command (`internal/cmd/mcpcmd/cmd_mcp.go`)
|
||||||
|
New CLI sub-command registered via `cli.WithCommands()` (not `init()`).
|
||||||
|
- Runs `pkg/mcp` server over stdio by default
|
||||||
|
- TCP mode via `MCP_ADDR=:9000` environment variable
|
||||||
|
- `--workspace` flag to restrict file operations to a directory
|
||||||
|
|
||||||
|
Registered in the full CLI variant. i18n strings added for all user-facing text.
|
||||||
|
|
||||||
|
### Plugin Configuration
|
||||||
|
`.mcp.json` created for the `agentic-flows` Claude Code plugin, pointing to `core mcp serve`. Exposes all 15 tools to Claude Code agents via the `core-cli` MCP server name.
|
||||||
|
|
||||||
|
## Key Outcomes
|
||||||
|
|
||||||
|
- `core mcp serve` is the single entry point for all MCP tooling (file ops, RAG, metrics, language detection, process management, WebSocket, webview/CDP)
|
||||||
|
- MCP command moved to `go-ai/cmd/mcpcmd/` in final form; the plan's `internal/cmd/mcpcmd/` path reflects the pre-extraction location
|
||||||
|
- Registration pattern updated from `init()` + `RegisterCommands()` to `cli.WithCommands()` lifecycle hooks
|
||||||
|
- Services required at runtime: Qdrant (localhost:6333), Ollama with nomic-embed-text (localhost:11434)
|
||||||
62
docs/plans/completed/qk-bone-orientation.md
Normal file
62
docs/plans/completed/qk-bone-orientation.md
Normal file
|
|
@ -0,0 +1,62 @@
|
||||||
|
# Q/K Bone Orientation — Completion Summary
|
||||||
|
|
||||||
|
**Completed:** 23 February 2026
|
||||||
|
**Repos:** go-inference, go-mlx, go-ml, LEM
|
||||||
|
**Status:** All 7 tasks complete, 14 files changed (+917 lines), all tests passing
|
||||||
|
|
||||||
|
## What Was Built
|
||||||
|
|
||||||
|
### go-inference — AttentionSnapshot types (Task 1)
|
||||||
|
|
||||||
|
`AttentionSnapshot` struct and `AttentionInspector` optional interface. Backends expose attention data via type assertion — no breaking changes to `TextModel`.
|
||||||
|
|
||||||
|
### go-mlx — KV cache extraction (Task 2)
|
||||||
|
|
||||||
|
`InspectAttention` on `metalAdapter` runs a single prefill pass and extracts post-RoPE K vectors from each layer's KV cache. Tested against real Gemma3-1B (26 layers, 1 KV head via GQA, 256 head dim).
|
||||||
|
|
||||||
|
### go-ml — Adapter pass-through (Task 3)
|
||||||
|
|
||||||
|
`InspectAttention` on `InferenceAdapter` type-asserts the underlying `TextModel` to `AttentionInspector`. Returns clear error for unsupported backends.
|
||||||
|
|
||||||
|
### LEM — Analysis engine (Task 4)
|
||||||
|
|
||||||
|
Pure Go CPU math in `pkg/lem/attention.go`. Computes 5 BO metrics from raw K tensors:
|
||||||
|
|
||||||
|
- **Mean Coherence** — pairwise cosine similarity of K vectors within each layer
|
||||||
|
- **Cross-Layer Alignment** — cosine similarity of mean K vectors between adjacent layers
|
||||||
|
- **Head Entropy** — normalised Shannon entropy of K vector magnitudes across positions
|
||||||
|
- **Phase-Lock Score** — fraction of head pairs above coherence threshold (0.7)
|
||||||
|
- **Joint Collapse Count** — layers where cross-alignment drops below threshold (0.5)
|
||||||
|
|
||||||
|
Composite score: 30% coherence + 25% cross-alignment + 20% phase-lock + 15% entropy + 10% joint stability → 0-100 scale.
|
||||||
|
|
||||||
|
### LEM — CLI command (Task 5)
|
||||||
|
|
||||||
|
`lem score attention -model <path> -prompt <text> [-json]` loads a model, runs InspectAttention, and prints BO metrics.
|
||||||
|
|
||||||
|
### LEM — Distill integration (Task 6)
|
||||||
|
|
||||||
|
Opt-in attention scoring in the distill pipeline. Gated behind `scorer.attention: true` and `scorer.attention_min_score` in ai.yaml. Costs one extra prefill per probe.
|
||||||
|
|
||||||
|
### LEM — Feature vectors (Task 7)
|
||||||
|
|
||||||
|
19D full feature vector: 6D grammar + 8D heuristic + 5D attention (`mean_coherence`, `cross_alignment`, `head_entropy`, `phase_lock`, `joint_stability`). Ready for Poindexter KDTree spatial indexing.
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
|
||||||
|
- **Optional interface** — `AttentionInspector` via type assertion, not added to `TextModel`
|
||||||
|
- **Named `BOResult`** — avoids collision with `metal.AttentionResult` in go-mlx
|
||||||
|
- **Opt-in for distill** — extra prefill per probe is expensive, off by default
|
||||||
|
- **Pure Go analysis** — zero CGO deps in the analysis engine; GPU data extracted once via `.Floats()`
|
||||||
|
|
||||||
|
## Commits
|
||||||
|
|
||||||
|
| Repo | SHA | Message |
|
||||||
|
|------|-----|---------|
|
||||||
|
| go-inference | `0f7263f` | feat: add AttentionInspector optional interface |
|
||||||
|
| go-mlx | `c2177f7` | feat: implement AttentionInspector via KV cache extraction |
|
||||||
|
| go-ml | `45e9fed` | feat: add InspectAttention pass-through |
|
||||||
|
| LEM | `28309b2` | feat: add Q/K Bone Orientation analysis engine |
|
||||||
|
| LEM | `e333192` | feat: add 'lem score attention' CLI |
|
||||||
|
| LEM | `fbc636e` | feat: integrate attention scoring into distill pipeline |
|
||||||
|
| LEM | `b621baa` | feat: add 19D full feature vector |
|
||||||
Loading…
Add table
Reference in a new issue