docs: add ecosystem overview and historical design plans
Moved from core/go during docs cleanup — these belong with the CLI that orchestrates the ecosystem, not the DI framework. - ecosystem.md: full module inventory and dependency graph - 3 active plans (authentik-traefik, core-help design/plan) - 13 completed design plans (MCP, go-api, cli-meta, go-forge, etc.) Co-Authored-By: Virgil <virgil@lethean.io>
This commit is contained in:
Snider
parent
37a8ae8d31
commit
aa83cf77cc
17 changed files with 10422 additions and 0 deletions
457
docs/ecosystem.md
Normal file
457
docs/ecosystem.md
Normal file
|
|
@ -0,0 +1,457 @@
|
|||
# Core Go Ecosystem
|
||||
|
||||
The Core Go ecosystem is a set of 19 standalone Go modules that form the infrastructure backbone for the host-uk platform and the Lethean network. All modules are hosted under the `forge.lthn.ai/core/` organisation. Each module has its own repository, independent versioning, and a `docs/` directory.
|
||||
|
||||
The CLI framework documented in the rest of this site (`forge.lthn.ai/core/cli`) is one node in this graph. The satellite packages listed here are separate repositories that the CLI imports or that stand alone as libraries.
|
||||
|
||||
---
|
||||
|
||||
## Module Index
|
||||
|
||||
| Package | Module Path | Managed By |
|
||||
|---------|-------------|-----------|
|
||||
| [go-inference](#go-inference) | `forge.lthn.ai/core/go-inference` | Virgil |
|
||||
| [go-mlx](#go-mlx) | `forge.lthn.ai/core/go-mlx` | Virgil |
|
||||
| [go-rocm](#go-rocm) | `forge.lthn.ai/core/go-rocm` | Charon |
|
||||
| [go-ml](#go-ml) | `forge.lthn.ai/core/go-ml` | Virgil |
|
||||
| [go-ai](#go-ai) | `forge.lthn.ai/core/go-ai` | Virgil |
|
||||
| [go-agentic](#go-agentic) | `forge.lthn.ai/core/go-agentic` | Charon |
|
||||
| [go-rag](#go-rag) | `forge.lthn.ai/core/go-rag` | Charon |
|
||||
| [go-i18n](#go-i18n) | `forge.lthn.ai/core/go-i18n` | Virgil |
|
||||
| [go-html](#go-html) | `forge.lthn.ai/core/go-html` | Charon |
|
||||
| [go-crypt](#go-crypt) | `forge.lthn.ai/core/go-crypt` | Virgil |
|
||||
| [go-scm](#go-scm) | `forge.lthn.ai/core/go-scm` | Charon |
|
||||
| [go-p2p](#go-p2p) | `forge.lthn.ai/core/go-p2p` | Charon |
|
||||
| [go-devops](#go-devops) | `forge.lthn.ai/core/go-devops` | Virgil |
|
||||
| [go-help](#go-help) | `forge.lthn.ai/core/go-help` | Charon |
|
||||
| [go-ratelimit](#go-ratelimit) | `forge.lthn.ai/core/go-ratelimit` | Charon |
|
||||
| [go-session](#go-session) | `forge.lthn.ai/core/go-session` | Charon |
|
||||
| [go-store](#go-store) | `forge.lthn.ai/core/go-store` | Charon |
|
||||
| [go-ws](#go-ws) | `forge.lthn.ai/core/go-ws` | Charon |
|
||||
| [go-webview](#go-webview) | `forge.lthn.ai/core/go-webview` | Charon |
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
The graph below shows import relationships. An arrow `A → B` means A imports B.
|
||||
|
||||
```
|
||||
go-inference (no dependencies — foundation contract)
|
||||
↑
|
||||
├── go-mlx (CGO, Apple Silicon Metal GPU)
|
||||
├── go-rocm (AMD ROCm, llama-server subprocess)
|
||||
└── go-ml (scoring engine, backends, orchestrator)
|
||||
↑
|
||||
└── go-ai (MCP hub, 49 tools)
|
||||
↑
|
||||
└── go-agentic (service lifecycle, allowances)
|
||||
|
||||
go-rag (Qdrant + Ollama, standalone)
|
||||
↑
|
||||
└── go-ai
|
||||
|
||||
go-i18n (grammar engine, standalone; Phase 2a imports go-mlx)
|
||||
|
||||
go-crypt (standalone)
|
||||
↑
|
||||
├── go-p2p (UEPS wire protocol)
|
||||
└── go-scm (AgentCI dispatch)
|
||||
|
||||
go-store (SQLite KV, standalone)
|
||||
↑
|
||||
├── go-ratelimit (sliding window limiter)
|
||||
├── go-session (transcript parser)
|
||||
└── go-agentic
|
||||
|
||||
go-ws (WebSocket hub, standalone)
|
||||
↑
|
||||
└── go-ai
|
||||
|
||||
go-webview (CDP client, standalone)
|
||||
↑
|
||||
└── go-ai
|
||||
|
||||
go-html (DOM compositor, standalone)
|
||||
|
||||
go-help (help catalogue, standalone)
|
||||
|
||||
go-devops (Ansible, build, infrastructure — imports go-scm)
|
||||
```
|
||||
|
||||
The CLI framework (`forge.lthn.ai/core/cli`) has internal equivalents of several of these packages (`pkg/rag`, `pkg/ws`, `pkg/webview`, `pkg/i18n`) that were developed in parallel. The satellite packages are the canonical standalone versions intended for use outside the CLI binary.
|
||||
|
||||
---
|
||||
|
||||
## Package Descriptions
|
||||
|
||||
### go-inference
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-inference`
|
||||
|
||||
Zero-dependency interface package that defines the common contract for all inference backends in the ecosystem:
|
||||
|
||||
- `TextModel` — the top-level model interface (`Generate`, `Stream`, `Close`)
|
||||
- `Backend` — hardware/runtime abstraction (Metal, ROCm, CPU, remote)
|
||||
- `Token` — streaming token type with metadata
|
||||
|
||||
No concrete implementations live here. Any package that needs to call inference without depending on a specific hardware library imports `go-inference` and receives an implementation at runtime.
|
||||
|
||||
---
|
||||
|
||||
### go-mlx
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-mlx`
|
||||
|
||||
Native Metal GPU inference for Apple Silicon using CGO bindings to `mlx-c` (the C API for Apple's MLX framework). Implements the `go-inference` interfaces.
|
||||
|
||||
Build requirements:
|
||||
- macOS 13+ (Ventura) on Apple Silicon
|
||||
- `mlx-c` installed (`brew install mlx`)
|
||||
- CGO enabled: `CGO_CFLAGS` and `CGO_LDFLAGS` must reference the mlx-c headers and library
|
||||
|
||||
Features:
|
||||
- Loads GGUF and MLX-format models
|
||||
- Streaming token generation directly on GPU
|
||||
- Quantised model support (Q4, Q8)
|
||||
- Phase 4 backend abstraction in progress — will allow hot-swapping backends at runtime
|
||||
|
||||
Local path: `/Users/snider/Code/go-mlx`
|
||||
|
||||
---
|
||||
|
||||
### go-rocm
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-rocm`
|
||||
|
||||
AMD ROCm GPU inference for Linux. Rather than using CGO, this package manages a `llama-server` subprocess (from llama.cpp) compiled with ROCm support and communicates over its HTTP API.
|
||||
|
||||
Features:
|
||||
- Subprocess lifecycle management (start, health-check, restart on crash)
|
||||
- OpenAI-compatible HTTP client wrapping llama-server's API
|
||||
- Implements `go-inference` interfaces
|
||||
- Targeted at the homelab RX 7800 XT running Ubuntu 24.04
|
||||
|
||||
Managed by Charon (Linux homelab).
|
||||
|
||||
---
|
||||
|
||||
### go-ml
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-ml`
|
||||
|
||||
Scoring engine, backend registry, and agent orchestration layer. The hub that connects models from `go-mlx`, `go-rocm`, and future backends into a unified interface.
|
||||
|
||||
Features:
|
||||
- Backend registry: register multiple inference backends, select by capability
|
||||
- Scoring pipeline: evaluate model outputs against rubrics
|
||||
- Agent orchestrator: coordinate multi-step inference tasks
|
||||
- ~3.5K LOC
|
||||
|
||||
---
|
||||
|
||||
### go-ai
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-ai`
|
||||
|
||||
MCP (Model Context Protocol) server hub with 49 registered tools. Acts as the primary facade for AI capabilities in the ecosystem.
|
||||
|
||||
Features:
|
||||
- 49 MCP tools covering file operations, RAG, metrics, process management, WebSocket, and CDP/webview
|
||||
- Imports `go-ml`, `go-rag`, `go-mlx`
|
||||
- Can run as stdio MCP server or TCP MCP server
|
||||
- AI usage metrics recorded to JSONL
|
||||
|
||||
Run the MCP server:
|
||||
|
||||
```bash
|
||||
# stdio (for Claude Desktop / Claude Code)
|
||||
core mcp serve
|
||||
|
||||
# TCP
|
||||
MCP_ADDR=:9000 core mcp serve
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### go-agentic
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-agentic`
|
||||
|
||||
Service lifecycle and allowance management for autonomous agents. Handles:
|
||||
|
||||
- Agent session tracking and state persistence
|
||||
- Allowance system: budget constraints on tool calls, token usage, and wall-clock time
|
||||
- Integration with `go-store` for persistence
|
||||
- REST client for the PHP `core-agentic` backend
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-rag
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-rag`
|
||||
|
||||
Retrieval-Augmented Generation pipeline using Qdrant for vector storage and Ollama for embeddings.
|
||||
|
||||
Features:
|
||||
- `ChunkMarkdown`: semantic splitting by H2 headers and paragraphs with overlap
|
||||
- `Ingest`: crawl a directory of Markdown files, embed, and store in Qdrant
|
||||
- `Query`: semantic search returning ranked `QueryResult` slices
|
||||
- `FormatResultsContext`: formats results as XML tags for LLM prompt injection
|
||||
- Clients: `QdrantClient` and `OllamaClient` wrapping their respective Go SDKs
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-i18n
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-i18n`
|
||||
|
||||
Grammar engine for natural-language generation. Goes beyond key-value lookup tables to handle pluralisation, verb conjugation, past tense, gerunds, and semantic sentence construction ("Subject verbed object").
|
||||
|
||||
Features:
|
||||
- `T(key, args...)` — main translation function
|
||||
- `S(noun, value)` — semantic subject with grammatical context
|
||||
- Language rules defined in JSON; algorithmic fallbacks for irregular verbs
|
||||
- **GrammarImprint**: a linguistic hash (reversal of the grammar engine) used as a semantic fingerprint — part of the Lethean identity verification stack
|
||||
- Phase 2a (imports `go-mlx` for language model-assisted reversal) currently blocked on `go-mlx` Phase 4
|
||||
|
||||
Local path: `/Users/snider/Code/go-i18n`
|
||||
|
||||
---
|
||||
|
||||
### go-html
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-html`
|
||||
|
||||
HLCRF DOM compositor — a programmatic HTML/DOM construction library targeting both server-side rendering and WASM (browser).
|
||||
|
||||
HLCRF stands for Header, Left, Content, Right, Footer — the region layout model used throughout the CLI's terminal UI and web rendering layer.
|
||||
|
||||
Features:
|
||||
- Composable region-based layout (mirrors the terminal `Composite` in `pkg/cli`)
|
||||
- WASM build target: runs in the browser without JavaScript
|
||||
- Used by the LEM Chat UI and web SDK generation
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-crypt
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-crypt`
|
||||
|
||||
Cryptographic primitives, authentication, and trust policy enforcement.
|
||||
|
||||
Features:
|
||||
- Password hashing (Argon2id with tuned parameters)
|
||||
- Symmetric encryption (ChaCha20-Poly1305, AES-GCM)
|
||||
- Key derivation (HKDF, Scrypt)
|
||||
- OpenPGP challenge-response authentication
|
||||
- Trust policies: define and evaluate access rules
|
||||
- Foundation for the UEPS (User-controlled Encryption Policy System) wire protocol in `go-p2p`
|
||||
|
||||
---
|
||||
|
||||
### go-scm
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-scm`
|
||||
|
||||
Source control management and CI integration, including the AgentCI dispatch system.
|
||||
|
||||
Features:
|
||||
- Forgejo and Gitea API clients (typed wrappers)
|
||||
- GitHub integration via the `gh` CLI
|
||||
- `AgentCI`: dispatches AI work items to agent runners over SSH using Charm stack libraries (`soft-serve`, `keygen`, `melt`, `wishlist`)
|
||||
- PR lifecycle management: create, review, merge, label
|
||||
- JSONL job journal for audit trails
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-p2p
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-p2p`
|
||||
|
||||
Peer-to-peer mesh networking implementing the UEPS (User-controlled Encryption Policy System) wire protocol.
|
||||
|
||||
Features:
|
||||
- UEPS: consent-gated TLV frames with Ed25519 consent tokens and an Intent-Broker
|
||||
- Peer discovery and mesh routing
|
||||
- Encrypted relay transport
|
||||
- Integration with `go-crypt` for all cryptographic operations
|
||||
|
||||
This is a core component of the Lethean Web3 network layer.
|
||||
|
||||
Managed by Charon (Linux homelab).
|
||||
|
||||
---
|
||||
|
||||
### go-devops
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-devops`
|
||||
|
||||
Infrastructure automation, build tooling, and release pipeline utilities, intended as a standalone library form of what the Core CLI provides as commands.
|
||||
|
||||
Features:
|
||||
- Ansible-lite engine (native Go SSH playbook execution)
|
||||
- LinuxKit image building and VM lifecycle
|
||||
- Multi-target binary build and release
|
||||
- Integration with `go-scm` for repository operations
|
||||
|
||||
---
|
||||
|
||||
### go-help
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-help`
|
||||
|
||||
Embedded documentation catalogue with full-text search and an optional HTTP server for serving help content.
|
||||
|
||||
Features:
|
||||
- YAML-frontmatter Markdown topic parsing
|
||||
- In-memory reverse index with title/heading/body scoring
|
||||
- Snippet extraction with keyword highlighting
|
||||
- `HTTP server` mode: serve the catalogue as a documentation site
|
||||
- Used by the `core pkg search` command and the `pkg/help` package inside the CLI
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-ratelimit
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-ratelimit`
|
||||
|
||||
Sliding-window rate limiter with a SQLite persistence backend.
|
||||
|
||||
Features:
|
||||
- Token bucket and sliding-window algorithms
|
||||
- SQLite backend via `go-store` for durable rate state across restarts
|
||||
- HTTP middleware helper
|
||||
- Used by `go-ai` and `go-agentic` to enforce per-agent API quotas
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-session
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-session`
|
||||
|
||||
Claude Code JSONL transcript parser and visualisation toolkit (standalone version of `pkg/session` inside the CLI).
|
||||
|
||||
Features:
|
||||
- `ParseTranscript(path)`: reads `.jsonl` session files and reconstructs tool use timelines
|
||||
- `ListSessions(dir)`: scans a Claude projects directory for session files
|
||||
- `Search(dir, query)`: full-text search across sessions
|
||||
- `RenderHTML(sess, path)`: single-file HTML visualisation
|
||||
- `RenderMP4(sess, path)`: terminal video replay via VHS
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-store
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-store`
|
||||
|
||||
SQLite-backed key-value store with reactive change notification.
|
||||
|
||||
Features:
|
||||
- `Get`, `Set`, `Delete`, `List` over typed keys
|
||||
- `Watch(key, handler)`: register a callback that fires on change
|
||||
- `OnChange(handler)`: subscribe to all changes
|
||||
- Used by `go-ratelimit`, `go-session`, and `go-agentic` for lightweight persistence
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-ws
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-ws`
|
||||
|
||||
WebSocket hub with channel-based subscriptions and an optional Redis pub/sub bridge for multi-instance deployments.
|
||||
|
||||
Features:
|
||||
- Hub pattern: central registry of connected clients
|
||||
- Channel routing: `SendToChannel(topic, msg)` delivers only to subscribers
|
||||
- Redis bridge: publish messages from one instance, receive on all
|
||||
- HTTP handler: `hub.Handler()` for embedding in any Go HTTP server
|
||||
- `SendProcessOutput(id, line)`: convenience method for streaming process logs
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
### go-webview
|
||||
|
||||
**Module:** `forge.lthn.ai/core/go-webview`
|
||||
|
||||
Chrome DevTools Protocol (CDP) client for browser automation, testing, and AI-driven web interaction (standalone version of `pkg/webview` inside the CLI).
|
||||
|
||||
Features:
|
||||
- Navigation, click, type, screenshot
|
||||
- `Evaluate(script)`: arbitrary JavaScript execution with result capture
|
||||
- Console capture and filtering
|
||||
- Angular-aware helpers: `WaitForAngular()`, `GetNgModel(selector)`
|
||||
- `ActionSequence`: chain interactions into a single call
|
||||
- Used by `go-ai` to expose browser tools to MCP agents
|
||||
|
||||
Managed by Charon.
|
||||
|
||||
---
|
||||
|
||||
## Forge Repository Paths
|
||||
|
||||
All repositories are hosted at `forge.lthn.ai` (Forgejo). SSH access uses port 2223:
|
||||
|
||||
```
|
||||
ssh://git@forge.lthn.ai:2223/core/go-inference.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-mlx.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-rocm.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-ml.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-ai.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-agentic.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-rag.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-i18n.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-html.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-crypt.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-scm.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-p2p.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-devops.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-help.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-ratelimit.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-session.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-store.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-ws.git
|
||||
ssh://git@forge.lthn.ai:2223/core/go-webview.git
|
||||
```
|
||||
|
||||
HTTPS authentication is not available on Forge. Always use SSH remotes.
|
||||
|
||||
---
|
||||
|
||||
## Go Workspace Setup
|
||||
|
||||
The satellite packages can be used together in a Go workspace. After cloning the repositories you need:
|
||||
|
||||
```bash
|
||||
go work init
|
||||
go work use ./go-inference ./go-mlx ./go-rag ./go-ai # add as needed
|
||||
go work sync
|
||||
```
|
||||
|
||||
The CLI repository already uses a Go workspace that includes `cmd/core-gui`, `cmd/bugseti`, and `cmd/examples/*`.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [index.md](index.md) — Main documentation hub
|
||||
- [getting-started.md](getting-started.md) — CLI installation
|
||||
- [configuration.md](configuration.md) — `repos.yaml` registry format
|
||||
1163
docs/plans/2026-02-20-authentik-traefik-plan.md
Normal file
1163
docs/plans/2026-02-20-authentik-traefik-plan.md
Normal file
File diff suppressed because it is too large
Load diff
155
docs/plans/2026-02-21-core-help-design.md
Normal file
155
docs/plans/2026-02-21-core-help-design.md
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
# core.help Documentation Website — Design
|
||||
|
||||
**Date:** 2026-02-21
|
||||
**Author:** Virgil
|
||||
**Status:** Design approved
|
||||
**Domain:** https://core.help
|
||||
|
||||
## Problem
|
||||
|
||||
Documentation is scattered across 39 repos (18 Go packages, 20 PHP packages, 1 CLI). There is no unified docs site. Developers need a single entry point to find CLI commands, Go package APIs, MCP tool references, and PHP module guides.
|
||||
|
||||
## Solution
|
||||
|
||||
A Hugo + Docsy static site at core.help, built from existing markdown docs aggregated by `core docs sync`. No new content — just collect and present what already exists across the ecosystem.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Stack
|
||||
|
||||
- **Hugo** — Go-native static site generator, sub-second builds
|
||||
- **Docsy theme** — Purpose-built for technical docs (used by Kubernetes, gRPC, Knative)
|
||||
- **BunnyCDN** — Static hosting with pull zone
|
||||
- **`core docs sync --target hugo`** — Collects markdown from all repos into Hugo content tree
|
||||
|
||||
### Why Hugo + Docsy (not VitePress or mdBook)
|
||||
|
||||
- Go-native, no Node.js dependency
|
||||
- Handles multi-section navigation (CLI, Go packages, PHP modules, MCP tools)
|
||||
- Sub-second builds for ~250 markdown files
|
||||
- Docsy has built-in search, versioned nav, API reference sections
|
||||
|
||||
## Content Structure
|
||||
|
||||
```
|
||||
docs-site/
|
||||
├── hugo.toml
|
||||
├── content/
|
||||
│ ├── _index.md # Landing page
|
||||
│ ├── getting-started/ # CLI top-level guides
|
||||
│ │ ├── _index.md
|
||||
│ │ ├── installation.md
|
||||
│ │ ├── configuration.md
|
||||
│ │ ├── user-guide.md
|
||||
│ │ ├── troubleshooting.md
|
||||
│ │ └── faq.md
|
||||
│ ├── cli/ # CLI command reference (43 commands)
|
||||
│ │ ├── _index.md
|
||||
│ │ ├── dev/ # core dev commit, push, pull, etc.
|
||||
│ │ ├── ai/ # core ai commands
|
||||
│ │ ├── go/ # core go test, lint, etc.
|
||||
│ │ └── ...
|
||||
│ ├── go/ # Go ecosystem packages (18)
|
||||
│ │ ├── _index.md # Ecosystem overview
|
||||
│ │ ├── go-api/ # README + architecture/development/history
|
||||
│ │ ├── go-ai/
|
||||
│ │ ├── go-mlx/
|
||||
│ │ ├── go-i18n/
|
||||
│ │ └── ...
|
||||
│ ├── mcp/ # MCP tool reference (49 tools)
|
||||
│ │ ├── _index.md
|
||||
│ │ ├── file-operations.md
|
||||
│ │ ├── process-management.md
|
||||
│ │ ├── rag.md
|
||||
│ │ └── ...
|
||||
│ ├── php/ # PHP packages (from core-php/docs/packages/)
|
||||
│ │ ├── _index.md
|
||||
│ │ ├── admin/
|
||||
│ │ ├── tenant/
|
||||
│ │ ├── commerce/
|
||||
│ │ └── ...
|
||||
│ └── kb/ # Knowledge base (wiki pages from go-mlx, go-i18n)
|
||||
│ ├── _index.md
|
||||
│ ├── mlx/
|
||||
│ └── i18n/
|
||||
├── static/ # Logos, favicons
|
||||
├── layouts/ # Custom template overrides (minimal)
|
||||
└── go.mod # Hugo modules (Docsy as module dep)
|
||||
```
|
||||
|
||||
## Sync Pipeline
|
||||
|
||||
`core docs sync --target hugo --output site/content/` performs:
|
||||
|
||||
### Source Mapping
|
||||
|
||||
```
|
||||
cli/docs/index.md → content/getting-started/_index.md
|
||||
cli/docs/getting-started.md → content/getting-started/installation.md
|
||||
cli/docs/user-guide.md → content/getting-started/user-guide.md
|
||||
cli/docs/configuration.md → content/getting-started/configuration.md
|
||||
cli/docs/troubleshooting.md → content/getting-started/troubleshooting.md
|
||||
cli/docs/faq.md → content/getting-started/faq.md
|
||||
|
||||
core/docs/cmd/**/*.md → content/cli/**/*.md
|
||||
|
||||
go-*/README.md → content/go/{name}/_index.md
|
||||
go-*/docs/*.md → content/go/{name}/*.md
|
||||
go-*/KB/*.md → content/kb/{name-suffix}/*.md
|
||||
|
||||
core-*/docs/**/*.md → content/php/{name-suffix}/**/*.md
|
||||
```
|
||||
|
||||
### Front Matter Injection
|
||||
|
||||
If a markdown file doesn't start with `---`, prepend:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "{derived from filename}"
|
||||
linkTitle: "{short name}"
|
||||
weight: {auto-incremented}
|
||||
---
|
||||
```
|
||||
|
||||
No other content transformations. Markdown stays as-is.
|
||||
|
||||
### Build & Deploy
|
||||
|
||||
```bash
|
||||
core docs sync --target hugo --output docs-site/content/
|
||||
cd docs-site && hugo build
|
||||
hugo deploy --target bunnycdn
|
||||
```
|
||||
|
||||
Hugo deploy config in `hugo.toml`:
|
||||
|
||||
```toml
|
||||
[deployment]
|
||||
[[deployment.targets]]
|
||||
name = "bunnycdn"
|
||||
URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto"
|
||||
```
|
||||
|
||||
Credentials via env vars.
|
||||
|
||||
## Registry
|
||||
|
||||
All 39 repos registered in `.core/repos.yaml` with `docs: true`. Go repos use explicit `path:` fields since they live outside the PHP `base_path`. `FindRegistry()` checks `.core/repos.yaml` alongside `repos.yaml`.
|
||||
|
||||
## Prerequisites Completed
|
||||
|
||||
- [x] `.core/repos.yaml` created with all 39 repos
|
||||
- [x] `FindRegistry()` updated to find `.core/repos.yaml`
|
||||
- [x] `Repo.Path` supports explicit YAML override
|
||||
- [x] go-api docs gap filled (architecture.md, development.md, history.md)
|
||||
- [x] All 18 Go repos have standard docs trio
|
||||
|
||||
## What Remains (Implementation Plan)
|
||||
|
||||
1. Create docs-site repo with Hugo + Docsy scaffold
|
||||
2. Extend `core docs sync` with `--target hugo` mode
|
||||
3. Write section _index.md files (landing page, section intros)
|
||||
4. Hugo config (navigation, search, theme colours)
|
||||
5. BunnyCDN deployment config
|
||||
6. CI pipeline on Forge (optional — can deploy manually initially)
|
||||
642
docs/plans/2026-02-21-core-help-plan.md
Normal file
642
docs/plans/2026-02-21-core-help-plan.md
Normal file
|
|
@ -0,0 +1,642 @@
|
|||
# core.help Hugo Documentation Site — Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Build a Hugo + Docsy documentation site at core.help that aggregates markdown from 39 repos via `core docs sync --target hugo`.
|
||||
|
||||
**Architecture:** Hugo static site with Docsy theme, populated by extending `core docs sync` with a `--target hugo` flag that maps repo docs into Hugo's `content/` tree with auto-injected front matter. Deploy to BunnyCDN.
|
||||
|
||||
**Tech Stack:** Hugo (Go SSG), Docsy theme (Hugo module), BunnyCDN, `core docs sync` CLI
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
The docs sync command lives in `/Users/snider/Code/host-uk/cli/cmd/docs/`. The site will be scaffolded at `/Users/snider/Code/host-uk/docs-site/`. The registry at `/Users/snider/Code/host-uk/.core/repos.yaml` already contains all 39 repos (20 PHP + 18 Go + 1 CLI) with explicit paths for Go repos.
|
||||
|
||||
Key files:
|
||||
- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go` — sync command (modify)
|
||||
- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go` — repo scanner (modify)
|
||||
- `/Users/snider/Code/host-uk/docs-site/` — Hugo site (create)
|
||||
|
||||
## Task 1: Scaffold Hugo + Docsy site
|
||||
|
||||
**Files:**
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/hugo.toml`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/go.mod`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/_index.md`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/getting-started/_index.md`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/cli/_index.md`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/go/_index.md`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/mcp/_index.md`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/php/_index.md`
|
||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/kb/_index.md`
|
||||
|
||||
This is the one-time Hugo scaffolding. No tests — just files.
|
||||
|
||||
**`hugo.toml`:**
|
||||
```toml
|
||||
baseURL = "https://core.help/"
|
||||
title = "Core Documentation"
|
||||
languageCode = "en"
|
||||
defaultContentLanguage = "en"
|
||||
|
||||
enableRobotsTXT = true
|
||||
enableGitInfo = false
|
||||
|
||||
[outputs]
|
||||
home = ["HTML", "JSON"]
|
||||
section = ["HTML"]
|
||||
|
||||
[params]
|
||||
description = "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools"
|
||||
copyright = "Host UK — EUPL-1.2"
|
||||
|
||||
[params.ui]
|
||||
sidebar_menu_compact = true
|
||||
breadcrumb_disable = false
|
||||
sidebar_search_disable = false
|
||||
navbar_logo = false
|
||||
|
||||
[params.ui.readingtime]
|
||||
enable = false
|
||||
|
||||
[module]
|
||||
proxy = "direct"
|
||||
|
||||
[module.hugoVersion]
|
||||
extended = true
|
||||
min = "0.120.0"
|
||||
|
||||
[[module.imports]]
|
||||
path = "github.com/google/docsy"
|
||||
disable = false
|
||||
|
||||
[markup.goldmark.renderer]
|
||||
unsafe = true
|
||||
|
||||
[menu]
|
||||
[[menu.main]]
|
||||
name = "Getting Started"
|
||||
weight = 10
|
||||
url = "/getting-started/"
|
||||
[[menu.main]]
|
||||
name = "CLI Reference"
|
||||
weight = 20
|
||||
url = "/cli/"
|
||||
[[menu.main]]
|
||||
name = "Go Packages"
|
||||
weight = 30
|
||||
url = "/go/"
|
||||
[[menu.main]]
|
||||
name = "MCP Tools"
|
||||
weight = 40
|
||||
url = "/mcp/"
|
||||
[[menu.main]]
|
||||
name = "PHP Packages"
|
||||
weight = 50
|
||||
url = "/php/"
|
||||
[[menu.main]]
|
||||
name = "Knowledge Base"
|
||||
weight = 60
|
||||
url = "/kb/"
|
||||
```
|
||||
|
||||
**`go.mod`:**
|
||||
```
|
||||
module github.com/host-uk/docs-site
|
||||
|
||||
go 1.22
|
||||
|
||||
require github.com/google/docsy v0.11.0
|
||||
```
|
||||
|
||||
Note: Run `hugo mod get` after creating these files to populate `go.sum` and download Docsy.
|
||||
|
||||
**Section `_index.md` files** — each needs Hugo front matter:
|
||||
|
||||
`content/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "Core Documentation"
|
||||
description: "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools"
|
||||
---
|
||||
|
||||
Welcome to the Core ecosystem documentation.
|
||||
|
||||
## Sections
|
||||
|
||||
- [Getting Started](/getting-started/) — Installation, configuration, and first steps
|
||||
- [CLI Reference](/cli/) — Command reference for `core` CLI
|
||||
- [Go Packages](/go/) — Go ecosystem package documentation
|
||||
- [MCP Tools](/mcp/) — Model Context Protocol tool reference
|
||||
- [PHP Packages](/php/) — PHP module documentation
|
||||
- [Knowledge Base](/kb/) — Wiki articles and deep dives
|
||||
```
|
||||
|
||||
`content/getting-started/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "Getting Started"
|
||||
linkTitle: "Getting Started"
|
||||
weight: 10
|
||||
description: "Installation, configuration, and first steps with the Core CLI"
|
||||
---
|
||||
```
|
||||
|
||||
`content/cli/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "CLI Reference"
|
||||
linkTitle: "CLI Reference"
|
||||
weight: 20
|
||||
description: "Command reference for the core CLI tool"
|
||||
---
|
||||
```
|
||||
|
||||
`content/go/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "Go Packages"
|
||||
linkTitle: "Go Packages"
|
||||
weight: 30
|
||||
description: "Documentation for the Go ecosystem packages"
|
||||
---
|
||||
```
|
||||
|
||||
`content/mcp/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "MCP Tools"
|
||||
linkTitle: "MCP Tools"
|
||||
weight: 40
|
||||
description: "Model Context Protocol tool reference — file operations, RAG, ML inference, process management"
|
||||
---
|
||||
```
|
||||
|
||||
`content/php/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "PHP Packages"
|
||||
linkTitle: "PHP Packages"
|
||||
weight: 50
|
||||
description: "Documentation for the PHP module ecosystem"
|
||||
---
|
||||
```
|
||||
|
||||
`content/kb/_index.md`:
|
||||
```markdown
|
||||
---
|
||||
title: "Knowledge Base"
|
||||
linkTitle: "Knowledge Base"
|
||||
weight: 60
|
||||
description: "Wiki articles, deep dives, and reference material"
|
||||
---
|
||||
```
|
||||
|
||||
**Verify:** After creating files, run from `/Users/snider/Code/host-uk/docs-site/`:
|
||||
```bash
|
||||
hugo mod get
|
||||
hugo server
|
||||
```
|
||||
The site should start and show the landing page with Docsy theme at `localhost:1313`.
|
||||
|
||||
**Commit:**
|
||||
```bash
|
||||
cd /Users/snider/Code/host-uk/docs-site
|
||||
git init
|
||||
git add .
|
||||
git commit -m "feat: scaffold Hugo + Docsy documentation site"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Extend scanRepoDocs to collect KB/ and README
|
||||
|
||||
**Files:**
|
||||
- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go`
|
||||
|
||||
Currently `scanRepoDocs` only collects files from `docs/`. For the Hugo target we also need:
|
||||
- `KB/**/*.md` files (wiki pages from go-mlx, go-i18n)
|
||||
- `README.md` content (becomes the package _index.md)
|
||||
|
||||
Add a `KBFiles []string` field to `RepoDocInfo` and scan `KB/` alongside `docs/`:
|
||||
|
||||
```go
|
||||
type RepoDocInfo struct {
|
||||
Name string
|
||||
Path string
|
||||
HasDocs bool
|
||||
Readme string
|
||||
ClaudeMd string
|
||||
Changelog string
|
||||
DocsFiles []string // All files in docs/ directory (recursive)
|
||||
KBFiles []string // All files in KB/ directory (recursive)
|
||||
}
|
||||
```
|
||||
|
||||
In `scanRepoDocs`, after the `docs/` walk, add a second walk for `KB/`:
|
||||
|
||||
```go
|
||||
// Recursively scan KB/ directory for .md files
|
||||
kbDir := filepath.Join(repo.Path, "KB")
|
||||
if _, err := io.Local.List(kbDir); err == nil {
|
||||
_ = filepath.WalkDir(kbDir, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
if d.IsDir() || !strings.HasSuffix(d.Name(), ".md") {
|
||||
return nil
|
||||
}
|
||||
relPath, _ := filepath.Rel(kbDir, path)
|
||||
info.KBFiles = append(info.KBFiles, relPath)
|
||||
info.HasDocs = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**Tests:** The existing tests should still pass. No new test file needed — this is a data-collection change.
|
||||
|
||||
**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...`
|
||||
|
||||
**Commit:**
|
||||
```bash
|
||||
git add cmd/docs/cmd_scan.go
|
||||
git commit -m "feat(docs): scan KB/ directory alongside docs/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Add `--target hugo` flag and Hugo sync logic
|
||||
|
||||
**Files:**
|
||||
- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go`
|
||||
|
||||
This is the main task. Add a `--target` flag (default `"php"`) and a new `runHugoSync` function that maps repos to Hugo's content tree.
|
||||
|
||||
**Add flag variable and registration:**
|
||||
|
||||
```go
|
||||
var (
|
||||
docsSyncRegistryPath string
|
||||
docsSyncDryRun bool
|
||||
docsSyncOutputDir string
|
||||
docsSyncTarget string
|
||||
)
|
||||
|
||||
func init() {
|
||||
docsSyncCmd.Flags().StringVar(&docsSyncRegistryPath, "registry", "", i18n.T("common.flag.registry"))
|
||||
docsSyncCmd.Flags().BoolVar(&docsSyncDryRun, "dry-run", false, i18n.T("cmd.docs.sync.flag.dry_run"))
|
||||
docsSyncCmd.Flags().StringVar(&docsSyncOutputDir, "output", "", i18n.T("cmd.docs.sync.flag.output"))
|
||||
docsSyncCmd.Flags().StringVar(&docsSyncTarget, "target", "php", "Target format: php (default) or hugo")
|
||||
}
|
||||
```
|
||||
|
||||
**Update RunE to pass target:**
|
||||
```go
|
||||
RunE: func(cmd *cli.Command, args []string) error {
|
||||
return runDocsSync(docsSyncRegistryPath, docsSyncOutputDir, docsSyncDryRun, docsSyncTarget)
|
||||
},
|
||||
```
|
||||
|
||||
**Update `runDocsSync` signature and add target dispatch:**
|
||||
```go
|
||||
func runDocsSync(registryPath string, outputDir string, dryRun bool, target string) error {
|
||||
reg, basePath, err := loadRegistry(registryPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
switch target {
|
||||
case "hugo":
|
||||
return runHugoSync(reg, basePath, outputDir, dryRun)
|
||||
default:
|
||||
return runPHPSync(reg, basePath, outputDir, dryRun)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rename current sync body to `runPHPSync`** — extract lines 67-159 of current `runDocsSync` into `runPHPSync(reg, basePath, outputDir string, dryRun bool) error`. This is a pure extract, no logic changes.
|
||||
|
||||
**Add `hugoOutputName` mapping function:**
|
||||
```go
|
||||
// hugoOutputName maps repo name to Hugo content section and folder.
|
||||
// Returns (section, folder) where section is the top-level content dir.
|
||||
func hugoOutputName(repoName string) (string, string) {
|
||||
// CLI guides
|
||||
if repoName == "cli" {
|
||||
return "getting-started", ""
|
||||
}
|
||||
// Core CLI command docs
|
||||
if repoName == "core" {
|
||||
return "cli", ""
|
||||
}
|
||||
// Go packages
|
||||
if strings.HasPrefix(repoName, "go-") {
|
||||
return "go", repoName
|
||||
}
|
||||
// PHP packages
|
||||
if strings.HasPrefix(repoName, "core-") {
|
||||
return "php", strings.TrimPrefix(repoName, "core-")
|
||||
}
|
||||
return "go", repoName
|
||||
}
|
||||
```
|
||||
|
||||
**Add front matter injection helper:**
|
||||
```go
|
||||
// injectFrontMatter prepends Hugo front matter to markdown content if missing.
|
||||
func injectFrontMatter(content []byte, title string, weight int) []byte {
|
||||
// Already has front matter
|
||||
if bytes.HasPrefix(bytes.TrimSpace(content), []byte("---")) {
|
||||
return content
|
||||
}
|
||||
fm := fmt.Sprintf("---\ntitle: %q\nweight: %d\n---\n\n", title, weight)
|
||||
return append([]byte(fm), content...)
|
||||
}
|
||||
|
||||
// titleFromFilename derives a human-readable title from a filename.
|
||||
func titleFromFilename(filename string) string {
|
||||
name := strings.TrimSuffix(filepath.Base(filename), ".md")
|
||||
name = strings.ReplaceAll(name, "-", " ")
|
||||
name = strings.ReplaceAll(name, "_", " ")
|
||||
// Title case
|
||||
words := strings.Fields(name)
|
||||
for i, w := range words {
|
||||
if len(w) > 0 {
|
||||
words[i] = strings.ToUpper(w[:1]) + w[1:]
|
||||
}
|
||||
}
|
||||
return strings.Join(words, " ")
|
||||
}
|
||||
```
|
||||
|
||||
**Add `runHugoSync` function:**
|
||||
```go
|
||||
func runHugoSync(reg *repos.Registry, basePath string, outputDir string, dryRun bool) error {
|
||||
if outputDir == "" {
|
||||
outputDir = filepath.Join(basePath, "docs-site", "content")
|
||||
}
|
||||
|
||||
// Scan all repos
|
||||
var docsInfo []RepoDocInfo
|
||||
for _, repo := range reg.List() {
|
||||
if repo.Name == "core-template" || repo.Name == "core-claude" {
|
||||
continue
|
||||
}
|
||||
info := scanRepoDocs(repo)
|
||||
if info.HasDocs {
|
||||
docsInfo = append(docsInfo, info)
|
||||
}
|
||||
}
|
||||
|
||||
if len(docsInfo) == 0 {
|
||||
cli.Text("No documentation found")
|
||||
return nil
|
||||
}
|
||||
|
||||
cli.Print("\n Hugo sync: %d repos with docs → %s\n\n", len(docsInfo), outputDir)
|
||||
|
||||
// Show plan
|
||||
for _, info := range docsInfo {
|
||||
section, folder := hugoOutputName(info.Name)
|
||||
target := section
|
||||
if folder != "" {
|
||||
target = section + "/" + folder
|
||||
}
|
||||
fileCount := len(info.DocsFiles) + len(info.KBFiles)
|
||||
if info.Readme != "" {
|
||||
fileCount++
|
||||
}
|
||||
cli.Print(" %s → %s/ (%d files)\n", repoNameStyle.Render(info.Name), target, fileCount)
|
||||
}
|
||||
|
||||
if dryRun {
|
||||
cli.Print("\n Dry run — no files written\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
cli.Blank()
|
||||
if !confirm("Sync to Hugo content directory?") {
|
||||
cli.Text("Aborted")
|
||||
return nil
|
||||
}
|
||||
|
||||
cli.Blank()
|
||||
var synced int
|
||||
for _, info := range docsInfo {
|
||||
section, folder := hugoOutputName(info.Name)
|
||||
|
||||
// Build destination path
|
||||
destDir := filepath.Join(outputDir, section)
|
||||
if folder != "" {
|
||||
destDir = filepath.Join(destDir, folder)
|
||||
}
|
||||
|
||||
// Copy docs/ files
|
||||
weight := 10
|
||||
docsDir := filepath.Join(info.Path, "docs")
|
||||
for _, f := range info.DocsFiles {
|
||||
src := filepath.Join(docsDir, f)
|
||||
dst := filepath.Join(destDir, f)
|
||||
if err := copyWithFrontMatter(src, dst, weight); err != nil {
|
||||
cli.Print(" %s %s: %s\n", errorStyle.Render("✗"), f, err)
|
||||
continue
|
||||
}
|
||||
weight += 10
|
||||
}
|
||||
|
||||
// Copy README.md as _index.md (if not CLI/core which use their own index)
|
||||
if info.Readme != "" && folder != "" {
|
||||
dst := filepath.Join(destDir, "_index.md")
|
||||
if err := copyWithFrontMatter(info.Readme, dst, 1); err != nil {
|
||||
cli.Print(" %s README: %s\n", errorStyle.Render("✗"), err)
|
||||
}
|
||||
}
|
||||
|
||||
// Copy KB/ files to kb/{suffix}/
|
||||
if len(info.KBFiles) > 0 {
|
||||
// Extract suffix: go-mlx → mlx, go-i18n → i18n
|
||||
suffix := strings.TrimPrefix(info.Name, "go-")
|
||||
kbDestDir := filepath.Join(outputDir, "kb", suffix)
|
||||
kbDir := filepath.Join(info.Path, "KB")
|
||||
kbWeight := 10
|
||||
for _, f := range info.KBFiles {
|
||||
src := filepath.Join(kbDir, f)
|
||||
dst := filepath.Join(kbDestDir, f)
|
||||
if err := copyWithFrontMatter(src, dst, kbWeight); err != nil {
|
||||
cli.Print(" %s KB/%s: %s\n", errorStyle.Render("✗"), f, err)
|
||||
continue
|
||||
}
|
||||
kbWeight += 10
|
||||
}
|
||||
}
|
||||
|
||||
cli.Print(" %s %s\n", successStyle.Render("✓"), info.Name)
|
||||
synced++
|
||||
}
|
||||
|
||||
cli.Print("\n Synced %d repos to Hugo content\n", synced)
|
||||
return nil
|
||||
}
|
||||
|
||||
// copyWithFrontMatter copies a markdown file, injecting front matter if missing.
|
||||
func copyWithFrontMatter(src, dst string, weight int) error {
|
||||
if err := io.Local.EnsureDir(filepath.Dir(dst)); err != nil {
|
||||
return err
|
||||
}
|
||||
content, err := io.Local.Read(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
title := titleFromFilename(src)
|
||||
result := injectFrontMatter([]byte(content), title, weight)
|
||||
return io.Local.Write(dst, string(result))
|
||||
}
|
||||
```
|
||||
|
||||
**Add imports** at top of file:
|
||||
```go
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"forge.lthn.ai/core/go/pkg/cli"
|
||||
"forge.lthn.ai/core/go/pkg/i18n"
|
||||
"forge.lthn.ai/core/go/pkg/io"
|
||||
"forge.lthn.ai/core/go/pkg/repos"
|
||||
)
|
||||
```
|
||||
|
||||
**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...`
|
||||
|
||||
**Commit:**
|
||||
```bash
|
||||
git add cmd/docs/cmd_sync.go
|
||||
git commit -m "feat(docs): add --target hugo sync mode for core.help"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Test the full pipeline
|
||||
|
||||
**No code changes.** Run the pipeline end-to-end.
|
||||
|
||||
**Step 1:** Sync docs to Hugo:
|
||||
```bash
|
||||
cd /Users/snider/Code/host-uk
|
||||
core docs sync --target hugo --dry-run
|
||||
```
|
||||
Verify all 39 repos appear with correct section mappings.
|
||||
|
||||
**Step 2:** Run actual sync:
|
||||
```bash
|
||||
core docs sync --target hugo
|
||||
```
|
||||
|
||||
**Step 3:** Build and preview:
|
||||
```bash
|
||||
cd /Users/snider/Code/host-uk/docs-site
|
||||
hugo server
|
||||
```
|
||||
Open `localhost:1313` and verify:
|
||||
- Landing page renders with section links
|
||||
- Getting Started section has CLI guides
|
||||
- CLI Reference section has command docs
|
||||
- Go Packages section has 18 packages with architecture/development/history
|
||||
- PHP Packages section has PHP module docs
|
||||
- Knowledge Base has MLX and i18n wiki pages
|
||||
- Navigation works, search works
|
||||
|
||||
**Step 4:** Fix any issues found during preview.
|
||||
|
||||
**Commit docs-site content:**
|
||||
```bash
|
||||
cd /Users/snider/Code/host-uk/docs-site
|
||||
git add content/
|
||||
git commit -m "feat: sync initial content from 39 repos"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: BunnyCDN deployment config
|
||||
|
||||
**Files:**
|
||||
- Modify: `/Users/snider/Code/host-uk/docs-site/hugo.toml`
|
||||
|
||||
Add deployment target:
|
||||
|
||||
```toml
|
||||
[deployment]
|
||||
[[deployment.targets]]
|
||||
name = "production"
|
||||
URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto"
|
||||
```
|
||||
|
||||
Add a `Taskfile.yml` for convenience:
|
||||
|
||||
**Create:** `/Users/snider/Code/host-uk/docs-site/Taskfile.yml`
|
||||
```yaml
|
||||
version: '3'
|
||||
|
||||
tasks:
|
||||
dev:
|
||||
desc: Start Hugo dev server
|
||||
cmds:
|
||||
- hugo server --buildDrafts
|
||||
|
||||
build:
|
||||
desc: Build static site
|
||||
cmds:
|
||||
- hugo --minify
|
||||
|
||||
sync:
|
||||
desc: Sync docs from all repos
|
||||
dir: ..
|
||||
cmds:
|
||||
- core docs sync --target hugo
|
||||
|
||||
deploy:
|
||||
desc: Build and deploy to BunnyCDN
|
||||
cmds:
|
||||
- task: sync
|
||||
- task: build
|
||||
- hugo deploy --target production
|
||||
|
||||
clean:
|
||||
desc: Remove generated content (keeps _index.md files)
|
||||
cmds:
|
||||
- find content -name "*.md" ! -name "_index.md" -delete
|
||||
```
|
||||
|
||||
**Verify:** `task dev` starts the site.
|
||||
|
||||
**Commit:**
|
||||
```bash
|
||||
git add hugo.toml Taskfile.yml
|
||||
git commit -m "feat: add BunnyCDN deployment config and Taskfile"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependency Sequencing
|
||||
|
||||
```
|
||||
Task 1 (Hugo scaffold) — independent, do first
|
||||
Task 2 (scan KB/) — independent, can parallel with Task 1
|
||||
Task 3 (--target hugo) — depends on Task 2
|
||||
Task 4 (test pipeline) — depends on Tasks 1 + 3
|
||||
Task 5 (deploy config) — depends on Task 1
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
After all tasks:
|
||||
1. `core docs sync --target hugo` populates `docs-site/content/` from all repos
|
||||
2. `cd docs-site && hugo server` renders the full site
|
||||
3. Navigation has 6 sections: Getting Started, CLI, Go, MCP, PHP, KB
|
||||
4. All existing markdown renders correctly with auto-injected front matter
|
||||
5. `hugo build` produces `public/` with no errors
|
||||
851
docs/plans/completed/2026-02-05-mcp-integration-original.md
Normal file
851
docs/plans/completed/2026-02-05-mcp-integration-original.md
Normal file
|
|
@ -0,0 +1,851 @@
|
|||
# MCP Integration Implementation Plan
|
||||
|
||||
> **Status:** Completed. MCP command now lives in `go-ai/cmd/mcpcmd/`. Code examples below use the old `init()` + `RegisterCommands()` pattern — the current approach uses `cli.WithCommands()` (see cli-meta-package-design.md).
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Add `core mcp serve` command with RAG and metrics tools, then configure the agentic-flows plugin to use it.
|
||||
|
||||
**Architecture:** Create a new `mcp` command package that starts the pkg/mcp server with extended tools. RAG tools call the existing exported functions in internal/cmd/rag. Metrics tools call pkg/ai directly. The agentic-flows plugin gets a `.mcp.json` that spawns `core mcp serve`.
|
||||
|
||||
**Tech Stack:** Go 1.25, github.com/modelcontextprotocol/go-sdk/mcp, pkg/rag, pkg/ai
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Add RAG tools to pkg/mcp
|
||||
|
||||
**Files:**
|
||||
- Create: `pkg/mcp/tools_rag.go`
|
||||
- Modify: `pkg/mcp/mcp.go:99-101` (registerTools)
|
||||
- Test: `pkg/mcp/tools_rag_test.go`
|
||||
|
||||
**Step 1: Write the failing test**
|
||||
|
||||
Create `pkg/mcp/tools_rag_test.go`:
|
||||
|
||||
```go
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
|
||||
func TestRAGQueryTool_Good(t *testing.T) {
|
||||
// This test verifies the tool is registered and callable.
|
||||
// It doesn't require Qdrant/Ollama running - just checks structure.
|
||||
s, err := New(WithWorkspaceRoot(""))
|
||||
if err != nil {
|
||||
t.Fatalf("New() error: %v", err)
|
||||
}
|
||||
|
||||
// Check that rag_query tool is registered
|
||||
tools := s.Server().ListTools()
|
||||
found := false
|
||||
for _, tool := range tools {
|
||||
if tool.Name == "rag_query" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("rag_query tool not registered")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRAGQueryInput_Good(t *testing.T) {
|
||||
input := RAGQueryInput{
|
||||
Question: "how do I deploy?",
|
||||
Collection: "hostuk-docs",
|
||||
TopK: 5,
|
||||
}
|
||||
if input.Question == "" {
|
||||
t.Error("Question should not be empty")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `go test -run TestRAGQueryTool ./pkg/mcp/... -v`
|
||||
Expected: FAIL with "rag_query tool not registered"
|
||||
|
||||
**Step 3: Create tools_rag.go with types and tool registration**
|
||||
|
||||
Create `pkg/mcp/tools_rag.go`:
|
||||
|
||||
```go
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
ragcmd "forge.lthn.ai/core/cli/internal/cmd/rag"
|
||||
"forge.lthn.ai/core/cli/pkg/rag"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
|
||||
// RAG tool input/output types
|
||||
|
||||
// RAGQueryInput contains parameters for querying the vector database.
|
||||
type RAGQueryInput struct {
|
||||
Question string `json:"question"`
|
||||
Collection string `json:"collection,omitempty"`
|
||||
TopK int `json:"top_k,omitempty"`
|
||||
}
|
||||
|
||||
// RAGQueryOutput contains the query results.
|
||||
type RAGQueryOutput struct {
|
||||
Results []RAGResult `json:"results"`
|
||||
Context string `json:"context"`
|
||||
}
|
||||
|
||||
// RAGResult represents a single search result.
|
||||
type RAGResult struct {
|
||||
Content string `json:"content"`
|
||||
Score float32 `json:"score"`
|
||||
Source string `json:"source"`
|
||||
Metadata map[string]string `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// RAGIngestInput contains parameters for ingesting documents.
|
||||
type RAGIngestInput struct {
|
||||
Path string `json:"path"`
|
||||
Collection string `json:"collection,omitempty"`
|
||||
Recreate bool `json:"recreate,omitempty"`
|
||||
}
|
||||
|
||||
// RAGIngestOutput contains the ingestion results.
|
||||
type RAGIngestOutput struct {
|
||||
Success bool `json:"success"`
|
||||
Path string `json:"path"`
|
||||
Chunks int `json:"chunks"`
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
|
||||
// RAGCollectionsInput contains parameters for listing collections.
|
||||
type RAGCollectionsInput struct {
|
||||
ShowStats bool `json:"show_stats,omitempty"`
|
||||
}
|
||||
|
||||
// RAGCollectionsOutput contains the list of collections.
|
||||
type RAGCollectionsOutput struct {
|
||||
Collections []CollectionInfo `json:"collections"`
|
||||
}
|
||||
|
||||
// CollectionInfo describes a Qdrant collection.
|
||||
type CollectionInfo struct {
|
||||
Name string `json:"name"`
|
||||
PointsCount uint64 `json:"points_count,omitempty"`
|
||||
Status string `json:"status,omitempty"`
|
||||
}
|
||||
|
||||
// registerRAGTools adds RAG tools to the MCP server.
|
||||
func (s *Service) registerRAGTools(server *mcp.Server) {
|
||||
mcp.AddTool(server, &mcp.Tool{
|
||||
Name: "rag_query",
|
||||
Description: "Query the vector database for relevant documents using semantic search",
|
||||
}, s.ragQuery)
|
||||
|
||||
mcp.AddTool(server, &mcp.Tool{
|
||||
Name: "rag_ingest",
|
||||
Description: "Ingest a file or directory into the vector database",
|
||||
}, s.ragIngest)
|
||||
|
||||
mcp.AddTool(server, &mcp.Tool{
|
||||
Name: "rag_collections",
|
||||
Description: "List available vector database collections",
|
||||
}, s.ragCollections)
|
||||
}
|
||||
|
||||
func (s *Service) ragQuery(ctx context.Context, req *mcp.CallToolRequest, input RAGQueryInput) (*mcp.CallToolResult, RAGQueryOutput, error) {
|
||||
s.logger.Info("MCP tool execution", "tool", "rag_query", "question", input.Question)
|
||||
|
||||
collection := input.Collection
|
||||
if collection == "" {
|
||||
collection = "hostuk-docs"
|
||||
}
|
||||
topK := input.TopK
|
||||
if topK <= 0 {
|
||||
topK = 5
|
||||
}
|
||||
|
||||
results, err := ragcmd.QueryDocs(ctx, input.Question, collection, topK)
|
||||
if err != nil {
|
||||
return nil, RAGQueryOutput{}, fmt.Errorf("query failed: %w", err)
|
||||
}
|
||||
|
||||
// Convert to output format
|
||||
out := RAGQueryOutput{
|
||||
Results: make([]RAGResult, 0, len(results)),
|
||||
Context: rag.FormatResultsContext(results),
|
||||
}
|
||||
for _, r := range results {
|
||||
out.Results = append(out.Results, RAGResult{
|
||||
Content: r.Content,
|
||||
Score: r.Score,
|
||||
Source: r.Source,
|
||||
Metadata: r.Metadata,
|
||||
})
|
||||
}
|
||||
|
||||
return nil, out, nil
|
||||
}
|
||||
|
||||
func (s *Service) ragIngest(ctx context.Context, req *mcp.CallToolRequest, input RAGIngestInput) (*mcp.CallToolResult, RAGIngestOutput, error) {
|
||||
s.logger.Security("MCP tool execution", "tool", "rag_ingest", "path", input.Path)
|
||||
|
||||
collection := input.Collection
|
||||
if collection == "" {
|
||||
collection = "hostuk-docs"
|
||||
}
|
||||
|
||||
// Check if path is a file or directory
|
||||
info, err := s.medium.Stat(input.Path)
|
||||
if err != nil {
|
||||
return nil, RAGIngestOutput{}, fmt.Errorf("path not found: %w", err)
|
||||
}
|
||||
|
||||
if info.IsDir() {
|
||||
err = ragcmd.IngestDirectory(ctx, input.Path, collection, input.Recreate)
|
||||
if err != nil {
|
||||
return nil, RAGIngestOutput{}, fmt.Errorf("ingest directory failed: %w", err)
|
||||
}
|
||||
return nil, RAGIngestOutput{
|
||||
Success: true,
|
||||
Path: input.Path,
|
||||
Message: fmt.Sprintf("Ingested directory into collection %s", collection),
|
||||
}, nil
|
||||
}
|
||||
|
||||
chunks, err := ragcmd.IngestFile(ctx, input.Path, collection)
|
||||
if err != nil {
|
||||
return nil, RAGIngestOutput{}, fmt.Errorf("ingest file failed: %w", err)
|
||||
}
|
||||
|
||||
return nil, RAGIngestOutput{
|
||||
Success: true,
|
||||
Path: input.Path,
|
||||
Chunks: chunks,
|
||||
Message: fmt.Sprintf("Ingested %d chunks into collection %s", chunks, collection),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *Service) ragCollections(ctx context.Context, req *mcp.CallToolRequest, input RAGCollectionsInput) (*mcp.CallToolResult, RAGCollectionsOutput, error) {
|
||||
s.logger.Info("MCP tool execution", "tool", "rag_collections")
|
||||
|
||||
client, err := rag.NewQdrantClient(rag.DefaultQdrantConfig())
|
||||
if err != nil {
|
||||
return nil, RAGCollectionsOutput{}, fmt.Errorf("connect to Qdrant: %w", err)
|
||||
}
|
||||
defer func() { _ = client.Close() }()
|
||||
|
||||
names, err := client.ListCollections(ctx)
|
||||
if err != nil {
|
||||
return nil, RAGCollectionsOutput{}, fmt.Errorf("list collections: %w", err)
|
||||
}
|
||||
|
||||
out := RAGCollectionsOutput{
|
||||
Collections: make([]CollectionInfo, 0, len(names)),
|
||||
}
|
||||
|
||||
for _, name := range names {
|
||||
info := CollectionInfo{Name: name}
|
||||
if input.ShowStats {
|
||||
cinfo, err := client.CollectionInfo(ctx, name)
|
||||
if err == nil {
|
||||
info.PointsCount = cinfo.PointsCount
|
||||
info.Status = cinfo.Status.String()
|
||||
}
|
||||
}
|
||||
out.Collections = append(out.Collections, info)
|
||||
}
|
||||
|
||||
return nil, out, nil
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Update mcp.go to call registerRAGTools**
|
||||
|
||||
In `pkg/mcp/mcp.go`, modify the `registerTools` function (around line 104) to add:
|
||||
|
||||
```go
|
||||
func (s *Service) registerTools(server *mcp.Server) {
|
||||
// File operations (existing)
|
||||
// ... existing code ...
|
||||
|
||||
// RAG operations
|
||||
s.registerRAGTools(server)
|
||||
}
|
||||
```
|
||||
|
||||
**Step 5: Run test to verify it passes**
|
||||
|
||||
Run: `go test -run TestRAGQuery ./pkg/mcp/... -v`
|
||||
Expected: PASS
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add pkg/mcp/tools_rag.go pkg/mcp/tools_rag_test.go pkg/mcp/mcp.go
|
||||
git commit -m "feat(mcp): add RAG tools (query, ingest, collections)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Add metrics tools to pkg/mcp
|
||||
|
||||
**Files:**
|
||||
- Create: `pkg/mcp/tools_metrics.go`
|
||||
- Modify: `pkg/mcp/mcp.go` (registerTools)
|
||||
- Test: `pkg/mcp/tools_metrics_test.go`
|
||||
|
||||
**Step 1: Write the failing test**
|
||||
|
||||
Create `pkg/mcp/tools_metrics_test.go`:
|
||||
|
||||
```go
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMetricsRecordTool_Good(t *testing.T) {
|
||||
s, err := New(WithWorkspaceRoot(""))
|
||||
if err != nil {
|
||||
t.Fatalf("New() error: %v", err)
|
||||
}
|
||||
|
||||
tools := s.Server().ListTools()
|
||||
found := false
|
||||
for _, tool := range tools {
|
||||
if tool.Name == "metrics_record" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("metrics_record tool not registered")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricsQueryTool_Good(t *testing.T) {
|
||||
s, err := New(WithWorkspaceRoot(""))
|
||||
if err != nil {
|
||||
t.Fatalf("New() error: %v", err)
|
||||
}
|
||||
|
||||
tools := s.Server().ListTools()
|
||||
found := false
|
||||
for _, tool := range tools {
|
||||
if tool.Name == "metrics_query" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("metrics_query tool not registered")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `go test -run TestMetrics ./pkg/mcp/... -v`
|
||||
Expected: FAIL
|
||||
|
||||
**Step 3: Create tools_metrics.go**
|
||||
|
||||
Create `pkg/mcp/tools_metrics.go`:
|
||||
|
||||
```go
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"forge.lthn.ai/core/cli/pkg/ai"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
|
||||
// Metrics tool input/output types
|
||||
|
||||
// MetricsRecordInput contains parameters for recording a metric event.
|
||||
type MetricsRecordInput struct {
|
||||
Type string `json:"type"`
|
||||
AgentID string `json:"agent_id,omitempty"`
|
||||
Repo string `json:"repo,omitempty"`
|
||||
Data map[string]any `json:"data,omitempty"`
|
||||
}
|
||||
|
||||
// MetricsRecordOutput contains the result of recording.
|
||||
type MetricsRecordOutput struct {
|
||||
Success bool `json:"success"`
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
}
|
||||
|
||||
// MetricsQueryInput contains parameters for querying metrics.
|
||||
type MetricsQueryInput struct {
|
||||
Since string `json:"since,omitempty"` // e.g., "7d", "24h"
|
||||
}
|
||||
|
||||
// MetricsQueryOutput contains the query results.
|
||||
type MetricsQueryOutput struct {
|
||||
Total int `json:"total"`
|
||||
ByType []MetricCount `json:"by_type"`
|
||||
ByRepo []MetricCount `json:"by_repo"`
|
||||
ByAgent []MetricCount `json:"by_agent"`
|
||||
Events []MetricEventBrief `json:"events,omitempty"`
|
||||
}
|
||||
|
||||
// MetricCount represents a count by key.
|
||||
type MetricCount struct {
|
||||
Key string `json:"key"`
|
||||
Count int `json:"count"`
|
||||
}
|
||||
|
||||
// MetricEventBrief is a simplified event for output.
|
||||
type MetricEventBrief struct {
|
||||
Type string `json:"type"`
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
AgentID string `json:"agent_id,omitempty"`
|
||||
Repo string `json:"repo,omitempty"`
|
||||
}
|
||||
|
||||
// registerMetricsTools adds metrics tools to the MCP server.
|
||||
func (s *Service) registerMetricsTools(server *mcp.Server) {
|
||||
mcp.AddTool(server, &mcp.Tool{
|
||||
Name: "metrics_record",
|
||||
Description: "Record a metric event (AI task, security scan, job creation, etc.)",
|
||||
}, s.metricsRecord)
|
||||
|
||||
mcp.AddTool(server, &mcp.Tool{
|
||||
Name: "metrics_query",
|
||||
Description: "Query recorded metrics with aggregation by type, repo, and agent",
|
||||
}, s.metricsQuery)
|
||||
}
|
||||
|
||||
func (s *Service) metricsRecord(ctx context.Context, req *mcp.CallToolRequest, input MetricsRecordInput) (*mcp.CallToolResult, MetricsRecordOutput, error) {
|
||||
s.logger.Info("MCP tool execution", "tool", "metrics_record", "type", input.Type)
|
||||
|
||||
if input.Type == "" {
|
||||
return nil, MetricsRecordOutput{}, fmt.Errorf("type is required")
|
||||
}
|
||||
|
||||
event := ai.Event{
|
||||
Type: input.Type,
|
||||
Timestamp: time.Now(),
|
||||
AgentID: input.AgentID,
|
||||
Repo: input.Repo,
|
||||
Data: input.Data,
|
||||
}
|
||||
|
||||
if err := ai.Record(event); err != nil {
|
||||
return nil, MetricsRecordOutput{}, fmt.Errorf("record event: %w", err)
|
||||
}
|
||||
|
||||
return nil, MetricsRecordOutput{
|
||||
Success: true,
|
||||
Timestamp: event.Timestamp,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *Service) metricsQuery(ctx context.Context, req *mcp.CallToolRequest, input MetricsQueryInput) (*mcp.CallToolResult, MetricsQueryOutput, error) {
|
||||
s.logger.Info("MCP tool execution", "tool", "metrics_query", "since", input.Since)
|
||||
|
||||
since := input.Since
|
||||
if since == "" {
|
||||
since = "7d"
|
||||
}
|
||||
|
||||
duration, err := parseDuration(since)
|
||||
if err != nil {
|
||||
return nil, MetricsQueryOutput{}, fmt.Errorf("invalid since value: %w", err)
|
||||
}
|
||||
|
||||
sinceTime := time.Now().Add(-duration)
|
||||
events, err := ai.ReadEvents(sinceTime)
|
||||
if err != nil {
|
||||
return nil, MetricsQueryOutput{}, fmt.Errorf("read events: %w", err)
|
||||
}
|
||||
|
||||
summary := ai.Summary(events)
|
||||
|
||||
out := MetricsQueryOutput{
|
||||
Total: summary["total"].(int),
|
||||
}
|
||||
|
||||
// Convert by_type
|
||||
if byType, ok := summary["by_type"].([]map[string]any); ok {
|
||||
for _, entry := range byType {
|
||||
out.ByType = append(out.ByType, MetricCount{
|
||||
Key: entry["key"].(string),
|
||||
Count: entry["count"].(int),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Convert by_repo
|
||||
if byRepo, ok := summary["by_repo"].([]map[string]any); ok {
|
||||
for _, entry := range byRepo {
|
||||
out.ByRepo = append(out.ByRepo, MetricCount{
|
||||
Key: entry["key"].(string),
|
||||
Count: entry["count"].(int),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Convert by_agent
|
||||
if byAgent, ok := summary["by_agent"].([]map[string]any); ok {
|
||||
for _, entry := range byAgent {
|
||||
out.ByAgent = append(out.ByAgent, MetricCount{
|
||||
Key: entry["key"].(string),
|
||||
Count: entry["count"].(int),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Include last 10 events for context
|
||||
limit := 10
|
||||
if len(events) < limit {
|
||||
limit = len(events)
|
||||
}
|
||||
for i := len(events) - limit; i < len(events); i++ {
|
||||
ev := events[i]
|
||||
out.Events = append(out.Events, MetricEventBrief{
|
||||
Type: ev.Type,
|
||||
Timestamp: ev.Timestamp,
|
||||
AgentID: ev.AgentID,
|
||||
Repo: ev.Repo,
|
||||
})
|
||||
}
|
||||
|
||||
return nil, out, nil
|
||||
}
|
||||
|
||||
// parseDuration parses a human-friendly duration like "7d", "24h", "30d".
|
||||
func parseDuration(s string) (time.Duration, error) {
|
||||
if len(s) < 2 {
|
||||
return 0, fmt.Errorf("invalid duration: %s", s)
|
||||
}
|
||||
|
||||
unit := s[len(s)-1]
|
||||
value := s[:len(s)-1]
|
||||
|
||||
var n int
|
||||
if _, err := fmt.Sscanf(value, "%d", &n); err != nil {
|
||||
return 0, fmt.Errorf("invalid duration: %s", s)
|
||||
}
|
||||
|
||||
if n <= 0 {
|
||||
return 0, fmt.Errorf("duration must be positive: %s", s)
|
||||
}
|
||||
|
||||
switch unit {
|
||||
case 'd':
|
||||
return time.Duration(n) * 24 * time.Hour, nil
|
||||
case 'h':
|
||||
return time.Duration(n) * time.Hour, nil
|
||||
case 'm':
|
||||
return time.Duration(n) * time.Minute, nil
|
||||
default:
|
||||
return 0, fmt.Errorf("unknown unit %c in duration: %s", unit, s)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Update mcp.go to call registerMetricsTools**
|
||||
|
||||
In `pkg/mcp/mcp.go`, add to `registerTools`:
|
||||
|
||||
```go
|
||||
func (s *Service) registerTools(server *mcp.Server) {
|
||||
// ... existing file operations ...
|
||||
|
||||
// RAG operations
|
||||
s.registerRAGTools(server)
|
||||
|
||||
// Metrics operations
|
||||
s.registerMetricsTools(server)
|
||||
}
|
||||
```
|
||||
|
||||
**Step 5: Run test to verify it passes**
|
||||
|
||||
Run: `go test -run TestMetrics ./pkg/mcp/... -v`
|
||||
Expected: PASS
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add pkg/mcp/tools_metrics.go pkg/mcp/tools_metrics_test.go pkg/mcp/mcp.go
|
||||
git commit -m "feat(mcp): add metrics tools (record, query)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Create `core mcp serve` command
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/cmd/mcpcmd/cmd_mcp.go`
|
||||
- Modify: `internal/variants/full.go` (add import)
|
||||
- Test: Manual test via `core mcp serve`
|
||||
|
||||
**Step 1: Create the mcp command package**
|
||||
|
||||
Create `internal/cmd/mcpcmd/cmd_mcp.go`:
|
||||
|
||||
```go
|
||||
package mcpcmd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"forge.lthn.ai/core/cli/pkg/cli"
|
||||
"forge.lthn.ai/core/cli/pkg/i18n"
|
||||
"forge.lthn.ai/core/cli/pkg/mcp"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cli.RegisterCommands(AddMCPCommands)
|
||||
}
|
||||
|
||||
var (
|
||||
mcpWorkspace string
|
||||
)
|
||||
|
||||
var mcpCmd = &cli.Command{
|
||||
Use: "mcp",
|
||||
Short: i18n.T("cmd.mcp.short"),
|
||||
Long: i18n.T("cmd.mcp.long"),
|
||||
}
|
||||
|
||||
var serveCmd = &cli.Command{
|
||||
Use: "serve",
|
||||
Short: i18n.T("cmd.mcp.serve.short"),
|
||||
Long: i18n.T("cmd.mcp.serve.long"),
|
||||
RunE: func(cmd *cli.Command, args []string) error {
|
||||
return runServe()
|
||||
},
|
||||
}
|
||||
|
||||
func AddMCPCommands(root *cli.Command) {
|
||||
initMCPFlags()
|
||||
mcpCmd.AddCommand(serveCmd)
|
||||
root.AddCommand(mcpCmd)
|
||||
}
|
||||
|
||||
func initMCPFlags() {
|
||||
serveCmd.Flags().StringVar(&mcpWorkspace, "workspace", "", i18n.T("cmd.mcp.serve.flag.workspace"))
|
||||
}
|
||||
|
||||
func runServe() error {
|
||||
opts := []mcp.Option{}
|
||||
|
||||
if mcpWorkspace != "" {
|
||||
opts = append(opts, mcp.WithWorkspaceRoot(mcpWorkspace))
|
||||
} else {
|
||||
// Default to unrestricted for MCP server
|
||||
opts = append(opts, mcp.WithWorkspaceRoot(""))
|
||||
}
|
||||
|
||||
svc, err := mcp.New(opts...)
|
||||
if err != nil {
|
||||
return cli.Wrap(err, "create MCP service")
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
// Handle shutdown signals
|
||||
sigCh := make(chan os.Signal, 1)
|
||||
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
|
||||
go func() {
|
||||
<-sigCh
|
||||
cancel()
|
||||
}()
|
||||
|
||||
return svc.Run(ctx)
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Add i18n strings**
|
||||
|
||||
Create or update `pkg/i18n/en.yaml` (if it exists) or add to the existing i18n mechanism:
|
||||
|
||||
```yaml
|
||||
cmd.mcp.short: "MCP (Model Context Protocol) server"
|
||||
cmd.mcp.long: "Start an MCP server for Claude Code integration with file, RAG, and metrics tools."
|
||||
cmd.mcp.serve.short: "Start the MCP server"
|
||||
cmd.mcp.serve.long: "Start the MCP server in stdio mode. Use MCP_ADDR env var for TCP mode."
|
||||
cmd.mcp.serve.flag.workspace: "Restrict file operations to this directory (empty = unrestricted)"
|
||||
```
|
||||
|
||||
**Step 3: Add import to full.go**
|
||||
|
||||
Modify `internal/variants/full.go` to add:
|
||||
|
||||
```go
|
||||
import (
|
||||
// ... existing imports ...
|
||||
_ "forge.lthn.ai/core/cli/internal/cmd/mcpcmd"
|
||||
)
|
||||
```
|
||||
|
||||
**Step 4: Build and test**
|
||||
|
||||
Run: `go build && ./core mcp serve --help`
|
||||
Expected: Help output showing the serve command
|
||||
|
||||
**Step 5: Test MCP server manually**
|
||||
|
||||
Run: `echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | ./core mcp serve`
|
||||
Expected: JSON response listing all tools including rag_query, metrics_record, etc.
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add internal/cmd/mcpcmd/cmd_mcp.go internal/variants/full.go
|
||||
git commit -m "feat: add 'core mcp serve' command"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Configure agentic-flows plugin with .mcp.json
|
||||
|
||||
**Files:**
|
||||
- Create: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`
|
||||
- Modify: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.claude-plugin/plugin.json` (optional, add mcpServers)
|
||||
|
||||
**Step 1: Create .mcp.json**
|
||||
|
||||
Create `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"core-cli": {
|
||||
"command": "core",
|
||||
"args": ["mcp", "serve"],
|
||||
"env": {
|
||||
"MCP_WORKSPACE": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Verify plugin loads**
|
||||
|
||||
Restart Claude Code and run `/mcp` to verify the core-cli server appears.
|
||||
|
||||
**Step 3: Test MCP tools**
|
||||
|
||||
Test that tools are available:
|
||||
- `mcp__plugin_agentic-flows_core-cli__rag_query`
|
||||
- `mcp__plugin_agentic-flows_core-cli__rag_ingest`
|
||||
- `mcp__plugin_agentic-flows_core-cli__rag_collections`
|
||||
- `mcp__plugin_agentic-flows_core-cli__metrics_record`
|
||||
- `mcp__plugin_agentic-flows_core-cli__metrics_query`
|
||||
- `mcp__plugin_agentic-flows_core-cli__file_read`
|
||||
- etc.
|
||||
|
||||
**Step 4: Commit plugin changes**
|
||||
|
||||
```bash
|
||||
cd /home/shared/hostuk/claude-plugins
|
||||
git add plugins/agentic-flows/.mcp.json
|
||||
git commit -m "feat(agentic-flows): add MCP server configuration for core-cli"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Update documentation
|
||||
|
||||
**Files:**
|
||||
- Modify: `/home/claude/.claude/projects/-home-claude/memory/MEMORY.md`
|
||||
- Modify: `/home/claude/.claude/projects/-home-claude/memory/plugin-dev-notes.md`
|
||||
|
||||
**Step 1: Update MEMORY.md**
|
||||
|
||||
Add under "Core CLI MCP Server" section:
|
||||
|
||||
```markdown
|
||||
### Core CLI MCP Server
|
||||
- **Command:** `core mcp serve` (stdio mode) or `MCP_ADDR=:9000 core mcp serve` (TCP)
|
||||
- **Tools available:**
|
||||
- File ops: file_read, file_write, file_edit, file_delete, file_rename, file_exists, dir_list, dir_create
|
||||
- RAG: rag_query, rag_ingest, rag_collections
|
||||
- Metrics: metrics_record, metrics_query
|
||||
- Language: lang_detect, lang_list
|
||||
- **Plugin config:** `plugins/agentic-flows/.mcp.json`
|
||||
```
|
||||
|
||||
**Step 2: Update plugin-dev-notes.md**
|
||||
|
||||
Add section:
|
||||
|
||||
```markdown
|
||||
## MCP Server (core mcp serve)
|
||||
|
||||
### Available Tools
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| file_read | Read file contents |
|
||||
| file_write | Write file contents |
|
||||
| file_edit | Edit file (replace string) |
|
||||
| file_delete | Delete file |
|
||||
| file_rename | Rename/move file |
|
||||
| file_exists | Check if file exists |
|
||||
| dir_list | List directory contents |
|
||||
| dir_create | Create directory |
|
||||
| rag_query | Query vector DB |
|
||||
| rag_ingest | Ingest file/directory |
|
||||
| rag_collections | List collections |
|
||||
| metrics_record | Record event |
|
||||
| metrics_query | Query events |
|
||||
| lang_detect | Detect file language |
|
||||
| lang_list | List supported languages |
|
||||
|
||||
### Example .mcp.json
|
||||
```json
|
||||
{
|
||||
"core-cli": {
|
||||
"command": "core",
|
||||
"args": ["mcp", "serve"]
|
||||
}
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
**Step 3: Commit documentation**
|
||||
|
||||
```bash
|
||||
git add ~/.claude/projects/-home-claude/memory/*.md
|
||||
git commit -m "docs: update memory with MCP server tools"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Task | Files | Purpose |
|
||||
|------|-------|---------|
|
||||
| 1 | `pkg/mcp/tools_rag.go` | RAG tools (query, ingest, collections) |
|
||||
| 2 | `pkg/mcp/tools_metrics.go` | Metrics tools (record, query) |
|
||||
| 3 | `internal/cmd/mcpcmd/cmd_mcp.go` | `core mcp serve` command |
|
||||
| 4 | `plugins/agentic-flows/.mcp.json` | Plugin MCP configuration |
|
||||
| 5 | Memory docs | Documentation updates |
|
||||
|
||||
## Services Required
|
||||
|
||||
- **Qdrant:** localhost:6333 (verified running)
|
||||
- **Ollama:** localhost:11434 with nomic-embed-text (verified running)
|
||||
- **InfluxDB:** localhost:8086 (optional, for future time-series metrics)
|
||||
82
docs/plans/completed/2026-02-17-lem-chat-design.md
Normal file
82
docs/plans/completed/2026-02-17-lem-chat-design.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# LEM Chat — Web Components Design
|
||||
|
||||
**Date**: 2026-02-17
|
||||
**Status**: Approved
|
||||
|
||||
## Summary
|
||||
|
||||
Standalone chat UI built with vanilla Web Components (Custom Elements + Shadow DOM). Connects to the MLX inference server's OpenAI-compatible SSE streaming endpoint. Zero framework dependencies. Single JS file output, embeddable anywhere.
|
||||
|
||||
## Components
|
||||
|
||||
| Element | Purpose |
|
||||
|---------|---------|
|
||||
| `<lem-chat>` | Container. Conversation state, SSE connection, config via attributes |
|
||||
| `<lem-messages>` | Scrollable message list with auto-scroll anchoring |
|
||||
| `<lem-message>` | Single message bubble. Streams tokens for assistant messages |
|
||||
| `<lem-input>` | Text input, Enter to send, Shift+Enter for newline |
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
User types in <lem-input>
|
||||
→ dispatches 'lem-send' CustomEvent
|
||||
→ <lem-chat> catches it
|
||||
→ adds user message to <lem-messages>
|
||||
→ POST /v1/chat/completions {stream: true, messages: [...history]}
|
||||
→ reads SSE chunks via fetch + ReadableStream
|
||||
→ appends tokens to streaming <lem-message>
|
||||
→ on [DONE], finalises message
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```html
|
||||
<lem-chat endpoint="http://localhost:8090" model="qwen3-8b"></lem-chat>
|
||||
```
|
||||
|
||||
Attributes: `endpoint`, `model`, `system-prompt`, `max-tokens`, `temperature`
|
||||
|
||||
## Theming
|
||||
|
||||
Shadow DOM with CSS custom properties:
|
||||
|
||||
```css
|
||||
--lem-bg: #1a1a1e;
|
||||
--lem-msg-user: #2a2a3e;
|
||||
--lem-msg-assistant: #1e1e2a;
|
||||
--lem-accent: #5865f2;
|
||||
--lem-text: #e0e0e0;
|
||||
--lem-font: system-ui;
|
||||
```
|
||||
|
||||
## Markdown
|
||||
|
||||
Minimal inline parsing: fenced code blocks, inline code, bold, italic. No library.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
lem-chat/
|
||||
├── index.html # Demo page
|
||||
├── src/
|
||||
│ ├── lem-chat.ts # Main container + SSE client
|
||||
│ ├── lem-messages.ts # Message list with scroll anchoring
|
||||
│ ├── lem-message.ts # Single message with streaming
|
||||
│ ├── lem-input.ts # Text input
|
||||
│ ├── markdown.ts # Minimal markdown → HTML
|
||||
│ └── styles.ts # CSS template literals
|
||||
├── package.json # typescript + esbuild
|
||||
└── tsconfig.json
|
||||
```
|
||||
|
||||
Build: `esbuild src/lem-chat.ts --bundle --outfile=dist/lem-chat.js`
|
||||
|
||||
## Not in v1
|
||||
|
||||
- Model selection UI
|
||||
- Conversation persistence
|
||||
- File/image upload
|
||||
- Syntax highlighting
|
||||
- Typing indicators
|
||||
- User avatars
|
||||
657
docs/plans/completed/2026-02-20-go-api-design-original.md
Normal file
657
docs/plans/completed/2026-02-20-go-api-design-original.md
Normal file
|
|
@ -0,0 +1,657 @@
|
|||
# go-api Design — HTTP Gateway + OpenAPI SDK Generation
|
||||
|
||||
**Date:** 2026-02-20
|
||||
**Author:** Virgil
|
||||
**Status:** Phase 1 + Phase 2 + Phase 3 Complete (176 tests in go-api)
|
||||
**Module:** `forge.lthn.ai/core/go-api`
|
||||
|
||||
## Problem
|
||||
|
||||
The Core Go ecosystem exposes 42+ tools via MCP (JSON-RPC), which is ideal for AI agents but inaccessible to regular HTTP clients, frontend applications, and third-party integrators. There is no unified HTTP gateway, no OpenAPI specification, and no generated SDKs.
|
||||
|
||||
Both external customers (Host UK products) and Lethean network peers need programmatic access to the same services. The gateway also serves web routes, static assets, and streaming endpoints — not just REST APIs.
|
||||
|
||||
## Solution
|
||||
|
||||
A `go-api` package that acts as the central HTTP gateway:
|
||||
|
||||
1. **Gin-based HTTP gateway** with extensible middleware via gin-contrib plugins
|
||||
2. **RouteGroup interface** that subsystems implement to register their own endpoints (API, web, or both)
|
||||
3. **WebSocket + SSE integration** for real-time streaming
|
||||
4. **OpenAPI 3.1 spec generation** via runtime SpecBuilder (not swaggo annotations)
|
||||
5. **SDK generation pipeline** targeting 11 languages via openapi-generator-cli
|
||||
|
||||
## Architecture
|
||||
|
||||
### Four-Protocol Access
|
||||
|
||||
Same backend services, four client protocols:
|
||||
|
||||
```
|
||||
┌─── REST (go-api) POST /v1/ml/generate → JSON
|
||||
│
|
||||
├─── GraphQL (gqlgen) mutation { mlGenerate(...) { response } }
|
||||
Client ────────────┤
|
||||
├─── WebSocket (go-ws) subscribe ml.generate → streaming
|
||||
│
|
||||
└─── MCP (go-ai) ml_generate → JSON-RPC
|
||||
```
|
||||
|
||||
### Dependency Graph
|
||||
|
||||
```
|
||||
go-api (Gin engine + middleware + OpenAPI)
|
||||
↑ imported by (each registers its own routes)
|
||||
├── go-ai/api/ → /v1/file/*, /v1/process/*, /v1/metrics/*
|
||||
├── go-ml/api/ → /v1/ml/*
|
||||
├── go-rag/api/ → /v1/rag/*
|
||||
├── go-agentic/api/ → /v1/tasks/*
|
||||
├── go-help/api/ → /v1/help/*
|
||||
└── go-ws/api/ → /ws (WebSocket upgrade)
|
||||
```
|
||||
|
||||
go-api has zero internal ecosystem dependencies. Subsystems import go-api, not the other way round.
|
||||
|
||||
### Subsystem Opt-In
|
||||
|
||||
Not every MCP tool becomes a REST endpoint. Each subsystem decides what to expose via a separate `RegisterAPI()` method, independent of MCP's `RegisterTools()`. A subsystem with 15 MCP tools might expose 5 REST endpoints.
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go-api
|
||||
├── api.go # Engine struct, New(), Serve(), Shutdown()
|
||||
├── middleware.go # Auth, CORS, rate limiting, request logging, recovery
|
||||
├── options.go # WithAddr, WithAuth, WithCORS, WithRateLimit, etc.
|
||||
├── group.go # RouteGroup interface + registration
|
||||
├── response.go # Envelope type, error responses, pagination
|
||||
├── docs/ # Generated swagger docs (swaggo output)
|
||||
├── sdk/ # SDK generation tooling / Makefile targets
|
||||
└── go.mod # forge.lthn.ai/core/go-api
|
||||
```
|
||||
|
||||
## Core Interface
|
||||
|
||||
```go
|
||||
// RouteGroup registers API routes onto a Gin router group.
|
||||
// Subsystems implement this to expose their endpoints.
|
||||
type RouteGroup interface {
|
||||
// Name returns the route group identifier (e.g. "ml", "rag", "tasks")
|
||||
Name() string
|
||||
// BasePath returns the URL prefix (e.g. "/v1/ml")
|
||||
BasePath() string
|
||||
// RegisterRoutes adds handlers to the provided router group
|
||||
RegisterRoutes(rg *gin.RouterGroup)
|
||||
}
|
||||
|
||||
// StreamGroup optionally declares WebSocket channels a subsystem publishes to.
|
||||
type StreamGroup interface {
|
||||
Channels() []string
|
||||
}
|
||||
```
|
||||
|
||||
### Subsystem Example (go-ml)
|
||||
|
||||
```go
|
||||
// In go-ml/api/routes.go
|
||||
package api
|
||||
|
||||
type Routes struct {
|
||||
service *ml.Service
|
||||
}
|
||||
|
||||
func NewRoutes(svc *ml.Service) *Routes {
|
||||
return &Routes{service: svc}
|
||||
}
|
||||
|
||||
func (r *Routes) Name() string { return "ml" }
|
||||
func (r *Routes) BasePath() string { return "/v1/ml" }
|
||||
|
||||
func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) {
|
||||
rg.POST("/generate", r.Generate)
|
||||
rg.POST("/score", r.Score)
|
||||
rg.GET("/backends", r.Backends)
|
||||
rg.GET("/status", r.Status)
|
||||
}
|
||||
|
||||
func (r *Routes) Channels() []string {
|
||||
return []string{"ml.generate", "ml.status"}
|
||||
}
|
||||
|
||||
// @Summary Generate text via ML backend
|
||||
// @Tags ml
|
||||
// @Accept json
|
||||
// @Produce json
|
||||
// @Param input body MLGenerateInput true "Generation parameters"
|
||||
// @Success 200 {object} Response[MLGenerateOutput]
|
||||
// @Router /v1/ml/generate [post]
|
||||
func (r *Routes) Generate(c *gin.Context) {
|
||||
var input MLGenerateInput
|
||||
if err := c.ShouldBindJSON(&input); err != nil {
|
||||
c.JSON(400, api.Fail("invalid_input", err.Error()))
|
||||
return
|
||||
}
|
||||
result, err := r.service.Generate(c.Request.Context(), input.Backend, input.Prompt, ml.GenOpts{
|
||||
Temperature: input.Temperature,
|
||||
MaxTokens: input.MaxTokens,
|
||||
Model: input.Model,
|
||||
})
|
||||
if err != nil {
|
||||
c.JSON(500, api.Fail("ml.generate_failed", err.Error()))
|
||||
return
|
||||
}
|
||||
c.JSON(200, api.OK(MLGenerateOutput{
|
||||
Response: result,
|
||||
Backend: input.Backend,
|
||||
Model: input.Model,
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
### Engine Wiring (in core CLI)
|
||||
|
||||
```go
|
||||
engine := api.New(
|
||||
api.WithAddr(":8080"),
|
||||
api.WithCORS("*"),
|
||||
api.WithAuth(api.BearerToken(cfg.APIKey)),
|
||||
api.WithRateLimit(100, time.Minute),
|
||||
api.WithWSHub(wsHub),
|
||||
)
|
||||
|
||||
engine.Register(mlapi.NewRoutes(mlService))
|
||||
engine.Register(ragapi.NewRoutes(ragService))
|
||||
engine.Register(agenticapi.NewRoutes(agenticService))
|
||||
|
||||
engine.Serve(ctx) // Blocks until context cancelled
|
||||
```
|
||||
|
||||
## Response Envelope
|
||||
|
||||
All endpoints return a consistent envelope:
|
||||
|
||||
```go
|
||||
type Response[T any] struct {
|
||||
Success bool `json:"success"`
|
||||
Data T `json:"data,omitempty"`
|
||||
Error *Error `json:"error,omitempty"`
|
||||
Meta *Meta `json:"meta,omitempty"`
|
||||
}
|
||||
|
||||
type Error struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
Details any `json:"details,omitempty"`
|
||||
}
|
||||
|
||||
type Meta struct {
|
||||
RequestID string `json:"request_id"`
|
||||
Duration string `json:"duration"`
|
||||
Page int `json:"page,omitempty"`
|
||||
PerPage int `json:"per_page,omitempty"`
|
||||
Total int `json:"total,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
Helper functions:
|
||||
|
||||
```go
|
||||
func OK[T any](data T) Response[T]
|
||||
func Fail(code, message string) Response[any]
|
||||
func Paginated[T any](data T, page, perPage, total int) Response[T]
|
||||
```
|
||||
|
||||
## Middleware Stack
|
||||
|
||||
```go
|
||||
api.New(
|
||||
api.WithAddr(":8080"),
|
||||
api.WithCORS(api.CORSConfig{...}), // gin-contrib/cors
|
||||
api.WithAuth(api.BearerToken("...")), // Phase 1: simple bearer token
|
||||
api.WithRateLimit(100, time.Minute), // Per-IP sliding window
|
||||
api.WithRequestID(), // X-Request-ID header generation
|
||||
api.WithRecovery(), // Panic recovery → 500 response
|
||||
api.WithLogger(slog.Default()), // Structured request logging
|
||||
)
|
||||
```
|
||||
|
||||
Auth evolution path: bearer token → API keys → Authentik (OIDC/forward auth). Middleware slot stays the same.
|
||||
|
||||
## WebSocket Integration
|
||||
|
||||
go-api wraps the existing go-ws Hub as a first-class transport:
|
||||
|
||||
```go
|
||||
// Automatic registration:
|
||||
// GET /ws → WebSocket upgrade (go-ws Hub)
|
||||
|
||||
// Client subscribes: {"type":"subscribe","channel":"ml.generate"}
|
||||
// Events arrive: {"type":"event","channel":"ml.generate","data":{...}}
|
||||
// Client unsubscribes: {"type":"unsubscribe","channel":"ml.generate"}
|
||||
```
|
||||
|
||||
Subsystems implementing `StreamGroup` declare which channels they publish to. This metadata feeds into the OpenAPI spec as documentation.
|
||||
|
||||
## OpenAPI + SDK Generation
|
||||
|
||||
### Runtime Spec Generation (SpecBuilder)
|
||||
|
||||
swaggo annotations were rejected because routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools already carry JSON Schema at runtime. Instead, a `SpecBuilder` constructs the full OpenAPI 3.1 spec from registered RouteGroups at runtime.
|
||||
|
||||
```go
|
||||
// Groups that implement DescribableGroup contribute endpoint metadata
|
||||
type DescribableGroup interface {
|
||||
RouteGroup
|
||||
Describe() []RouteDescription
|
||||
}
|
||||
|
||||
// SpecBuilder assembles the spec from all groups
|
||||
builder := &api.SpecBuilder{Title: "Core API", Description: "...", Version: "1.0.0"}
|
||||
spec, _ := builder.Build(engine.Groups())
|
||||
```
|
||||
|
||||
### MCP-to-REST Bridge (ToolBridge)
|
||||
|
||||
The `ToolBridge` converts MCP tool descriptors into REST POST endpoints and implements both `RouteGroup` and `DescribableGroup`. Each tool becomes `POST /{tool_name}`. Generic types are captured at MCP registration time via closures, enabling JSON unmarshalling to the correct input type at request time.
|
||||
|
||||
```go
|
||||
bridge := api.NewToolBridge("/v1/tools")
|
||||
mcp.BridgeToAPI(mcpService, bridge) // Populates bridge from MCP tool registry
|
||||
engine.Register(bridge) // Registers REST endpoints + OpenAPI metadata
|
||||
```
|
||||
|
||||
### Swagger UI
|
||||
|
||||
```go
|
||||
// Built-in at GET /swagger/*any
|
||||
// SpecBuilder output served via gin-swagger, cached via sync.Once
|
||||
api.New(api.WithSwagger("Core API", "...", "1.0.0"))
|
||||
```
|
||||
|
||||
### SDK Generation
|
||||
|
||||
```bash
|
||||
# Via openapi-generator-cli (11 languages supported)
|
||||
core api sdk --lang go # Generate Go SDK
|
||||
core api sdk --lang typescript-fetch,python # Multiple languages
|
||||
core api sdk --lang rust --output ./sdk/ # Custom output dir
|
||||
```
|
||||
|
||||
### CLI Commands
|
||||
|
||||
```bash
|
||||
core api spec # Emit OpenAPI JSON to stdout
|
||||
core api spec --format yaml # YAML variant
|
||||
core api spec --output spec.json # Write to file
|
||||
core api sdk --lang python # Generate Python SDK
|
||||
core api sdk --lang go,rust # Multiple SDKs
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `github.com/gin-gonic/gin` | HTTP framework |
|
||||
| `github.com/swaggo/gin-swagger` | Swagger UI middleware |
|
||||
| `github.com/gin-contrib/cors` | CORS middleware |
|
||||
| `github.com/gin-contrib/secure` | Security headers |
|
||||
| `github.com/gin-contrib/sessions` | Server-side sessions |
|
||||
| `github.com/gin-contrib/authz` | Casbin authorisation |
|
||||
| `github.com/gin-contrib/httpsign` | HTTP signature verification |
|
||||
| `github.com/gin-contrib/slog` | Structured request logging |
|
||||
| `github.com/gin-contrib/timeout` | Per-request timeouts |
|
||||
| `github.com/gin-contrib/gzip` | Gzip compression |
|
||||
| `github.com/gin-contrib/static` | Static file serving |
|
||||
| `github.com/gin-contrib/pprof` | Runtime profiling |
|
||||
| `github.com/gin-contrib/expvar` | Runtime metrics |
|
||||
| `github.com/gin-contrib/location/v2` | Reverse proxy detection |
|
||||
| `github.com/99designs/gqlgen` | GraphQL endpoint |
|
||||
| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | Distributed tracing |
|
||||
| `gopkg.in/yaml.v3` | YAML spec export |
|
||||
| `forge.lthn.ai/core/go-ws` | WebSocket Hub (existing) |
|
||||
|
||||
## Estimated Size
|
||||
|
||||
| Component | LOC |
|
||||
|-----------|-----|
|
||||
| Engine + options | ~200 |
|
||||
| Middleware | ~150 |
|
||||
| Response envelope | ~80 |
|
||||
| RouteGroup interface | ~30 |
|
||||
| WebSocket integration | ~60 |
|
||||
| Tests | ~300 |
|
||||
| **Total go-api** | **~820** |
|
||||
|
||||
Each subsystem's `api/` package adds ~100-200 LOC per route group.
|
||||
|
||||
## Phase 1 — Implemented (20 Feb 2026)
|
||||
|
||||
**Commit:** `17ae945` on Forge (`core/go-api`)
|
||||
|
||||
| Component | Status | Tests |
|
||||
|-----------|--------|-------|
|
||||
| Response envelope (OK, Fail, Paginated) | Done | 9 |
|
||||
| RouteGroup + StreamGroup interfaces | Done | 4 |
|
||||
| Engine (New, Register, Handler, Serve) | Done | 9 |
|
||||
| Bearer auth middleware | Done | 3 |
|
||||
| Request ID middleware | Done | 2 |
|
||||
| CORS middleware (gin-contrib/cors) | Done | 3 |
|
||||
| WebSocket endpoint | Done | 3 |
|
||||
| Swagger UI (gin-swagger) | Done | 2 |
|
||||
| Health endpoint | Done | 1 |
|
||||
| **Total** | **~840 LOC** | **36** |
|
||||
|
||||
**Integration proof:** go-ml/api/ registers 3 endpoints with 12 tests (`0c23858`).
|
||||
|
||||
## Phase 2 Wave 1 — Implemented (20 Feb 2026)
|
||||
|
||||
**Commits:** `6bb7195..daae6f7` on Forge (`core/go-api`)
|
||||
|
||||
| Component | Option | Dependency | Tests |
|
||||
|-----------|--------|------------|-------|
|
||||
| Authentik (forward auth + OIDC) | `WithAuthentik()` | `go-oidc/v3`, `oauth2` | 14 |
|
||||
| Security headers (HSTS, CSP, etc.) | `WithSecure()` | `gin-contrib/secure` | 8 |
|
||||
| Structured request logging | `WithSlog()` | `gin-contrib/slog` | 6 |
|
||||
| Per-request timeouts | `WithTimeout()` | `gin-contrib/timeout` | 5 |
|
||||
| Gzip compression | `WithGzip()` | `gin-contrib/gzip` | 5 |
|
||||
| Static file serving | `WithStatic()` | `gin-contrib/static` | 5 |
|
||||
| **Wave 1 Total** | | | **43** |
|
||||
|
||||
**Cumulative:** 76 tests (36 Phase 1 + 43 Wave 1 - 3 shared), all passing.
|
||||
|
||||
## Phase 2 Wave 2 — Implemented (20 Feb 2026)
|
||||
|
||||
**Commits:** `64a8b16..67dcc83` on Forge (`core/go-api`)
|
||||
|
||||
| Component | Option | Dependency | Tests | Notes |
|
||||
|-----------|--------|------------|-------|-------|
|
||||
| Brotli compression | `WithBrotli()` | `andybalholm/brotli` | 5 | Custom middleware; `gin-contrib/brotli` is empty stub |
|
||||
| Response caching | `WithCache()` | none (in-memory) | 5 | Custom middleware; `gin-contrib/cache` is per-handler, not global |
|
||||
| Server-side sessions | `WithSessions()` | `gin-contrib/sessions` | 5 | Cookie store, configurable name + secret |
|
||||
| Casbin authorisation | `WithAuthz()` | `gin-contrib/authz`, `casbin/v2` | 5 | Subject via Basic Auth; RBAC policy model |
|
||||
| **Wave 2 Total** | | | **20** | |
|
||||
|
||||
**Cumulative:** 102 passing tests (2 integration skipped), all green.
|
||||
|
||||
## Phase 2 Wave 3 — Implemented (20 Feb 2026)
|
||||
|
||||
**Commits:** `7b3f99e..d517fa2` on Forge (`core/go-api`)
|
||||
|
||||
| Component | Option | Dependency | Tests | Notes |
|
||||
|-----------|--------|------------|-------|-------|
|
||||
| HTTP signature verification | `WithHTTPSign()` | `gin-contrib/httpsign` | 5 | HMAC-SHA256; extensible via httpsign.Option |
|
||||
| Server-Sent Events | `WithSSE()` | none (custom SSEBroker) | 6 | Channel filtering, multi-client broadcast, GET /events |
|
||||
| Reverse proxy detection | `WithLocation()` | `gin-contrib/location/v2` | 5 | X-Forwarded-Host/Proto parsing |
|
||||
| Locale detection | `WithI18n()` | `golang.org/x/text/language` | 5 | Accept-Language parsing, message lookup, GetLocale/GetMessage |
|
||||
| GraphQL endpoint | `WithGraphQL()` | `99designs/gqlgen` | 5 | /graphql + optional /graphql/playground |
|
||||
| **Wave 3 Total** | | | **26** | |
|
||||
|
||||
**Cumulative:** 128 passing tests (2 integration skipped), all green.
|
||||
|
||||
## Phase 2 Wave 4 — Implemented (21 Feb 2026)
|
||||
|
||||
**Commits:** `32b3680..8ba1716` on Forge (`core/go-api`)
|
||||
|
||||
| Component | Option | Dependency | Tests | Notes |
|
||||
|-----------|--------|------------|-------|-------|
|
||||
| Runtime profiling | `WithPprof()` | `gin-contrib/pprof` | 5 | /debug/pprof/* endpoints, flag-based mount |
|
||||
| Runtime metrics | `WithExpvar()` | `gin-contrib/expvar` | 5 | /debug/vars endpoint, flag-based mount |
|
||||
| Distributed tracing | `WithTracing()` | `otelgin` + OpenTelemetry SDK | 5 | W3C traceparent propagation, span attributes |
|
||||
| **Wave 4 Total** | | | **15** | |
|
||||
|
||||
**Cumulative:** 143 passing tests (2 integration skipped), all green.
|
||||
|
||||
**Phase 2 complete.** All 4 waves implemented. Every planned plugin has a `With*()` option and tests.
|
||||
|
||||
## Phase 3 — OpenAPI Spec Generation + SDK Codegen (21 Feb 2026)
|
||||
|
||||
**Architecture:** Runtime OpenAPI generation via SpecBuilder (NOT swaggo annotations). Routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools carry JSON Schema at runtime. A `ToolBridge` converts tool descriptors into RouteGroup + OpenAPI metadata. A `SpecBuilder` constructs the full OpenAPI 3.1 spec. SDK codegen wraps `openapi-generator-cli`.
|
||||
|
||||
### Wave 1: go-api (Tasks 1-5)
|
||||
|
||||
**Commits:** `465bd60..1910aec` on Forge (`core/go-api`)
|
||||
|
||||
| Component | File | Tests | Notes |
|
||||
|-----------|------|-------|-------|
|
||||
| DescribableGroup interface | `group.go` | 5 | Opt-in OpenAPI metadata for RouteGroups |
|
||||
| ToolBridge | `bridge.go` | 6 | Tool descriptors → POST endpoints + DescribableGroup |
|
||||
| SpecBuilder | `openapi.go` | 6 | OpenAPI 3.1 JSON with Response[T] envelope wrapping |
|
||||
| Swagger refactor | `swagger.go` | 5 | Replaced hardcoded empty spec with SpecBuilder |
|
||||
| Spec export | `export.go` | 5 | JSON + YAML export to file/writer |
|
||||
| SDK codegen | `codegen.go` | 5 | 11-language wrapper for openapi-generator-cli |
|
||||
| **Wave 1 Total** | | **32** | |
|
||||
|
||||
### Wave 2: go-ai MCP bridge (Tasks 6-7)
|
||||
|
||||
**Commits:** `2107eda..c37e1cf` on Forge (`core/go-ai`)
|
||||
|
||||
| Component | File | Tests | Notes |
|
||||
|-----------|------|-------|-------|
|
||||
| Tool registry | `mcp/registry.go` | 5 | Generic `addToolRecorded[In,Out]` captures types in closures |
|
||||
| BridgeToAPI | `mcp/bridge.go` | 5 | MCP tools → go-api ToolBridge, 10MB body limit, error classification |
|
||||
| **Wave 2 Total** | | **10** | |
|
||||
|
||||
### Wave 3: CLI commands (Tasks 8-9)
|
||||
|
||||
**Commit:** `d6eec4d` on Forge (`core/cli` dev branch)
|
||||
|
||||
| Component | File | Tests | Notes |
|
||||
|-----------|------|-------|-------|
|
||||
| `core api spec` | `cmd/api/cmd_spec.go` | 2 | JSON/YAML export, --output/--format flags |
|
||||
| `core api sdk` | `cmd/api/cmd_sdk.go` | 2 | --lang (required), --output, --spec, --package flags |
|
||||
| **Wave 3 Total** | | **4** | |
|
||||
|
||||
**Cumulative go-api:** 176 passing tests. **Phase 3 complete.**
|
||||
|
||||
### Known Limitations
|
||||
|
||||
- **Subsystem tools excluded from bridge:** Subsystems call `mcp.AddTool` directly, bypassing `addToolRecorded`. Only the 10 built-in MCP tools appear in the REST bridge. Future: pass `*Service` to `RegisterTools` instead of `*mcp.Server`.
|
||||
- **Flat schema only:** `structSchema` reflection handles flat structs but does not recurse into nested structs. Adequate for current tool inputs.
|
||||
- **CLI spec produces empty bridge:** `core api spec` currently generates a spec with only `/health`. Full MCP integration requires wiring the MCP service into the CLI command.
|
||||
|
||||
## Phase 2 — Gin Plugin Roadmap (Complete)
|
||||
|
||||
All plugins drop in as `With*()` options on the Engine. No architecture changes needed.
|
||||
|
||||
### Security & Auth
|
||||
|
||||
| Plugin | Option | Purpose | Priority |
|
||||
|--------|--------|---------|----------|
|
||||
| ~~**Authentik**~~ | ~~`WithAuthentik()`~~ | ~~OIDC + forward auth integration.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/secure~~ | ~~`WithSecure()`~~ | ~~Security headers: HSTS, X-Frame-Options, X-Content-Type-Options, CSP.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/sessions~~ | ~~`WithSessions()`~~ | ~~Server-side sessions (cookie store). Web session management alongside Authentik tokens.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/authz~~ | ~~`WithAuthz()`~~ | ~~Casbin-based authorisation. Policy-driven access control via RBAC.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/httpsign~~ | ~~`WithHTTPSign()`~~ | ~~HTTP signature verification. HMAC-SHA256 with extensible options.~~ | ~~**Done**~~ |
|
||||
|
||||
### Performance & Reliability
|
||||
|
||||
| Plugin | Option | Purpose | Priority |
|
||||
|--------|--------|---------|----------|
|
||||
| ~~gin-contrib/cache~~ | ~~`WithCache()`~~ | ~~Response caching (in-memory). GET response caching with TTL, lazy eviction.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/timeout~~ | ~~`WithTimeout()`~~ | ~~Per-request timeouts.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/gzip~~ | ~~`WithGzip()`~~ | ~~Gzip response compression.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/brotli~~ | ~~`WithBrotli()`~~ | ~~Brotli compression via `andybalholm/brotli`. Custom middleware (gin-contrib stub empty).~~ | ~~**Done**~~ |
|
||||
|
||||
### Observability
|
||||
|
||||
| Plugin | Option | Purpose | Priority |
|
||||
|--------|--------|---------|----------|
|
||||
| ~~gin-contrib/slog~~ | ~~`WithSlog()`~~ | ~~Structured request logging via slog.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/pprof~~ | ~~`WithPprof()`~~ | ~~Runtime profiling endpoints at /debug/pprof/. Flag-based mount.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/expvar~~ | ~~`WithExpvar()`~~ | ~~Go runtime metrics at /debug/vars. Flag-based mount.~~ | ~~**Done**~~ |
|
||||
| ~~otelgin~~ | ~~`WithTracing()`~~ | ~~OpenTelemetry distributed tracing. W3C traceparent propagation.~~ | ~~**Done**~~ |
|
||||
|
||||
### Content & Streaming
|
||||
|
||||
| Plugin | Option | Purpose | Priority |
|
||||
|--------|--------|---------|----------|
|
||||
| ~~gin-contrib/static~~ | ~~`WithStatic()`~~ | ~~Serve static files.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/sse~~ | ~~`WithSSE()`~~ | ~~Server-Sent Events. Custom SSEBroker with channel filtering, GET /events.~~ | ~~**Done**~~ |
|
||||
| ~~gin-contrib/location~~ | ~~`WithLocation()`~~ | ~~Auto-detect scheme/host from X-Forwarded-* headers.~~ | ~~**Done**~~ |
|
||||
|
||||
### Query Layer
|
||||
|
||||
| Plugin | Option | Purpose | Priority |
|
||||
|--------|--------|---------|----------|
|
||||
| ~~99designs/gqlgen~~ | ~~`WithGraphQL()`~~ | ~~GraphQL endpoint at `/graphql` + optional playground. Accepts gqlgen ExecutableSchema.~~ | ~~**Done**~~ |
|
||||
|
||||
The GraphQL schema can be generated from the same Go Input/Output structs that define the REST endpoints. gqlgen produces an `http.Handler` that mounts directly on Gin. Subsystems opt-in via:
|
||||
|
||||
```go
|
||||
// Subsystems that want GraphQL implement this alongside RouteGroup
|
||||
type ResolverGroup interface {
|
||||
// RegisterResolvers adds query/mutation resolvers to the GraphQL schema
|
||||
RegisterResolvers(schema *graphql.Schema)
|
||||
}
|
||||
```
|
||||
|
||||
This means a subsystem like go-ml exposes:
|
||||
- **REST:** `POST /v1/ml/generate` (existing)
|
||||
- **GraphQL:** `mutation { mlGenerate(prompt: "...", backend: "mlx") { response, model } }` (same handler)
|
||||
- **MCP:** `ml_generate` tool (existing)
|
||||
|
||||
Four protocols, one set of handlers.
|
||||
|
||||
### Ecosystem Integration
|
||||
|
||||
| Plugin | Option | Purpose | Priority |
|
||||
|--------|--------|---------|----------|
|
||||
| ~~gin-contrib/i18n~~ | ~~`WithI18n()`~~ | ~~Locale detection via Accept-Language. Custom middleware using `golang.org/x/text/language`.~~ | ~~**Done**~~ |
|
||||
| [gin-contrib/graceful](https://github.com/gin-contrib/graceful) | — | Already implemented in Engine.Serve(). Could swap to this for more robust lifecycle management if needed. | — |
|
||||
| [gin-contrib/requestid](https://github.com/gin-contrib/requestid) | — | Already implemented. Theirs uses UUID, ours uses hex. Could swap for standards compliance. | — |
|
||||
|
||||
### Implementation Order
|
||||
|
||||
**Wave 1 (gateway hardening):** ~~Authentik, secure, slog, timeout, gzip, static~~ **DONE** (20 Feb 2026)
|
||||
**Wave 2 (performance + auth):** ~~cache, sessions, authz, brotli~~ **DONE** (20 Feb 2026)
|
||||
**Wave 3 (network + streaming):** ~~httpsign, sse, location, i18n, gqlgen~~ **DONE** (20 Feb 2026)
|
||||
**Wave 4 (observability):** ~~pprof, expvar, tracing~~ **DONE** (21 Feb 2026)
|
||||
|
||||
Each wave adds `With*()` options + tests. No breaking changes — existing code continues to work without any new options enabled.
|
||||
|
||||
## Authentik Integration
|
||||
|
||||
[Authentik](https://goauthentik.io/) is the identity provider and edge auth proxy. It handles user registration, login, MFA, social auth, SAML, and OIDC — so go-api doesn't have to.
|
||||
|
||||
### Two Integration Modes
|
||||
|
||||
**1. Forward Auth (web traffic)**
|
||||
|
||||
Traefik sits in front of go-api. For web routes, Traefik's `forwardAuth` middleware checks with Authentik before passing the request through. Authentik handles login flows, session cookies, and consent. go-api receives pre-authenticated requests with identity headers.
|
||||
|
||||
```
|
||||
Browser → Traefik → Authentik (forward auth) → go-api
|
||||
↓
|
||||
Login page (if unauthenticated)
|
||||
```
|
||||
|
||||
go-api reads trusted headers set by Authentik:
|
||||
```
|
||||
X-Authentik-Username: alice
|
||||
X-Authentik-Groups: admins,developers
|
||||
X-Authentik-Email: alice@example.com
|
||||
X-Authentik-Uid: <uuid>
|
||||
X-Authentik-Jwt: <signed token>
|
||||
```
|
||||
|
||||
**2. OIDC Token Validation (API traffic)**
|
||||
|
||||
API clients (SDKs, CLI tools, network peers) authenticate directly with Authentik's OAuth2 token endpoint, then send the JWT to go-api. go-api validates the JWT using Authentik's OIDC discovery endpoint (`.well-known/openid-configuration`).
|
||||
|
||||
```
|
||||
SDK client → Authentik (token endpoint) → receives JWT
|
||||
SDK client → go-api (Authorization: Bearer <jwt>) → validates via OIDC
|
||||
```
|
||||
|
||||
### Implementation in go-api
|
||||
|
||||
```go
|
||||
engine := api.New(
|
||||
api.WithAuthentik(api.AuthentikConfig{
|
||||
Issuer: "https://auth.lthn.ai/application/o/core-api/",
|
||||
ClientID: "core-api",
|
||||
TrustedProxy: true, // Trust X-Authentik-* headers from Traefik
|
||||
}),
|
||||
)
|
||||
```
|
||||
|
||||
`WithAuthentik()` adds middleware that:
|
||||
1. Checks for `X-Authentik-Jwt` header (forward auth mode) — validates signature, extracts claims
|
||||
2. Falls back to `Authorization: Bearer <jwt>` header (direct OIDC mode) — validates via JWKS
|
||||
3. Populates `c.Set("user", AuthentikUser{...})` in the Gin context for handlers to use
|
||||
4. Skips /health, /swagger, and any public paths
|
||||
|
||||
```go
|
||||
// In any handler:
|
||||
func (r *Routes) ListItems(c *gin.Context) {
|
||||
user := api.GetUser(c) // Returns *AuthentikUser or nil
|
||||
if user == nil {
|
||||
c.JSON(401, api.Fail("unauthorised", "Authentication required"))
|
||||
return
|
||||
}
|
||||
// user.Username, user.Groups, user.Email, user.UID available
|
||||
}
|
||||
```
|
||||
|
||||
### Auth Layers
|
||||
|
||||
```
|
||||
Authentik (identity) → WHO is this? (user, groups, email)
|
||||
↓
|
||||
go-api middleware → IS their token valid? (JWT verification)
|
||||
↓
|
||||
Casbin authz (optional) → CAN they do this? (role → endpoint policies)
|
||||
↓
|
||||
Handler → DOES this (business logic)
|
||||
```
|
||||
|
||||
Phase 1 bearer auth continues to work alongside Authentik — useful for service-to-service tokens, CI/CD, and development. `WithBearerAuth` and `WithAuthentik` can coexist.
|
||||
|
||||
### Authentik Deployment
|
||||
|
||||
Authentik runs as a Docker service alongside go-api, fronted by Traefik:
|
||||
- **auth.lthn.ai** — Authentik UI + OIDC endpoints (production)
|
||||
- **auth.leth.in** — Authentik for devnet/testnet
|
||||
- Traefik routes `/outpost.goauthentik.io/` to Authentik's embedded outpost for forward auth
|
||||
|
||||
### Dependencies
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `github.com/coreos/go-oidc/v3` | OIDC discovery + JWT validation |
|
||||
| `golang.org/x/oauth2` | OAuth2 token exchange (for server-side flows) |
|
||||
|
||||
Both are standard Go libraries with no heavy dependencies.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- gRPC gateway
|
||||
- Built-in user registration/login (Authentik handles this)
|
||||
- API versioning beyond /v1/ prefix
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Phase 1 (Done)
|
||||
|
||||
1. ~~`core api serve` starts a Gin server with registered subsystem routes~~
|
||||
2. ~~WebSocket subscriptions work alongside REST~~
|
||||
3. ~~Swagger UI accessible at `/swagger/`~~
|
||||
4. ~~All endpoints return consistent Response envelope~~
|
||||
5. ~~Bearer token auth protects all routes~~
|
||||
6. ~~First subsystem integration (go-ml/api/) proves the pattern~~
|
||||
|
||||
### Phase 2 (Done)
|
||||
|
||||
7. ~~Security headers, compression, and caching active in production~~
|
||||
8. ~~Session-based auth alongside bearer tokens~~
|
||||
9. ~~HTTP signature verification for Lethean network peers~~
|
||||
10. ~~Static file serving for docs site and SDK downloads~~
|
||||
11. ~~GraphQL endpoint at `/graphql` with playground~~
|
||||
|
||||
### Phase 3 (Done)
|
||||
|
||||
12. ~~`core api spec` emits valid OpenAPI 3.1 JSON via runtime SpecBuilder~~
|
||||
13. ~~`core api sdk` generates SDKs for 11 languages via openapi-generator-cli~~
|
||||
14. ~~MCP tools bridged to REST endpoints via ToolBridge + BridgeToAPI~~
|
||||
15. ~~OpenAPI spec includes Response[T] envelope wrapping~~
|
||||
16. ~~Spec export to file in JSON and YAML formats~~
|
||||
1503
docs/plans/completed/2026-02-20-go-api-plan-original.md
Normal file
1503
docs/plans/completed/2026-02-20-go-api-plan-original.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,128 @@
|
|||
# CLI Meta-Package Restructure — Design
|
||||
|
||||
**Goal:** Transform `core/cli` from a 35K LOC monolith into a thin assembly repo that ships variant binaries. Domain repos own their commands. `go/pkg/cli` is the only import any domain package needs for CLI concerns.
|
||||
|
||||
**Architecture:** Commands register as framework services via `cli.WithCommands()`, passed to `cli.Main()`. Command code lives in the domain repos that own the business logic. The cli repo is a thin `main.go` that wires them together.
|
||||
|
||||
**Tech Stack:** go/pkg/cli (wraps cobra + charmbracelet), Core framework lifecycle, Taskfile
|
||||
|
||||
---
|
||||
|
||||
## 1. CLI SDK — The Single Import
|
||||
|
||||
`forge.lthn.ai/core/go/pkg/cli` is the **only** import domain packages use for CLI concerns. It wraps cobra, charmbracelet, and stdlib behind a stable API. If the underlying libraries change, only `go/pkg/cli` is touched — every domain repo is insulated.
|
||||
|
||||
### Already done
|
||||
|
||||
- **Cobra:** `Command` type alias, `NewCommand()`, `NewGroup()`, `NewRun()`, flag helpers (`StringFlag`, `BoolFlag`, `IntFlag`, `StringSliceFlag`), arg validators
|
||||
- **Output:** `Success()`, `Error()`, `Warn()`, `Info()`, `Table`, `Section()`, `Label()`, `Task()`, `Hint()`
|
||||
- **Prompts:** `Confirm()`, `Question()`, `Choose()`, `ChooseMulti()` with grammar-based action variants
|
||||
- **Styles:** 17 pre-built styles, `AnsiStyle` builder, Tailwind colour constants (47 hex values)
|
||||
- **Glyphs:** `:check:`, `:cross:`, `:warn:` etc. with Unicode/Emoji/ASCII themes
|
||||
- **Layout:** HLCRF composite renderer (Header/Left/Content/Right/Footer)
|
||||
- **Errors:** `Wrap()`, `WrapVerb()`, `ExitError`, `Is()`, `As()`
|
||||
- **Logging:** `LogDebug()`, `LogInfo()`, `LogWarn()`, `LogError()`, `LogSecurity()`
|
||||
- **TUI primitives:** `Spinner`, `ProgressBar`, `InteractiveList`, `TextInput`, `Viewport`, `RunTUI`
|
||||
- **Command registration:** `WithCommands(name, fn)` — registers commands as framework services
|
||||
|
||||
### Stubbed for later (interface exists, returns simple fallback)
|
||||
|
||||
- `Form(fields []FormField) (map[string]string, error)` — multi-field form (backed by huh later)
|
||||
- `FilePicker(opts ...FilePickerOption) (string, error)` — file browser
|
||||
- `Tabs(items []TabItem) error` — tabbed content panes
|
||||
|
||||
### Rule
|
||||
|
||||
Domain packages import `forge.lthn.ai/core/go/pkg/cli` and **nothing else** for CLI concerns. No `cobra`, no `lipgloss`, no `bubbletea`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Command Registration — Framework Lifecycle
|
||||
|
||||
Commands register through the Core framework's service lifecycle, not through global state or `init()` functions.
|
||||
|
||||
### The contract
|
||||
|
||||
Each domain repo exports an `Add*Commands(root *cli.Command)` function. The CLI binary wires it in via `cli.WithCommands()`:
|
||||
|
||||
```go
|
||||
// go-ai/cmd/daemon/cmd.go
|
||||
package daemon
|
||||
|
||||
import "forge.lthn.ai/core/go/pkg/cli"
|
||||
|
||||
// AddDaemonCommand adds the 'daemon' command group to the root.
|
||||
func AddDaemonCommand(root *cli.Command) {
|
||||
daemonCmd := cli.NewGroup("daemon", "Manage the core daemon", "")
|
||||
root.AddCommand(daemonCmd)
|
||||
// subcommands...
|
||||
}
|
||||
```
|
||||
|
||||
No `init()`. No blank imports. No `cli.RegisterCommands()`.
|
||||
|
||||
### How it works
|
||||
|
||||
`cli.WithCommands(name, fn)` wraps the registration function as a framework service implementing `Startable`. During `Core.ServiceStartup()`, the service's `OnStartup()` casts `Core.App` to `*cobra.Command` and calls the registration function. Core services (i18n, log, workspace) start first since they're registered before command services.
|
||||
|
||||
```go
|
||||
// cli/main.go
|
||||
func main() {
|
||||
cli.Main(
|
||||
cli.WithCommands("config", config.AddConfigCommands),
|
||||
cli.WithCommands("doctor", doctor.AddDoctorCommands),
|
||||
// ...
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### Migration status (completed)
|
||||
|
||||
| Source | Destination | Status |
|
||||
|--------|-------------|--------|
|
||||
| `cmd/dev, setup, qa, docs, gitcmd, monitor` | `go-devops/cmd/` | Done |
|
||||
| `cmd/lab` | `go-ai/cmd/` | Done |
|
||||
| `cmd/workspace` | `go-agentic/cmd/` | Done |
|
||||
| `cmd/go` | `core/go/cmd/gocmd` | Done |
|
||||
| `cmd/vanity-import, community` | `go-devops/cmd/` | Done |
|
||||
| `cmd/updater` | `go-update` | Done (own repo) |
|
||||
| `cmd/daemon, mcpcmd, security` | `go-ai/cmd/` | Done |
|
||||
| `cmd/crypt` | `go-crypt/cmd/` | Done |
|
||||
| `cmd/rag` | `go-rag/cmd/` | Done |
|
||||
| `cmd/unifi` | `go-netops/cmd/` | Done |
|
||||
| `cmd/api` | `go-api/cmd/` | Done |
|
||||
| `cmd/collect, forge, gitea` | `go-scm/cmd/` | Done |
|
||||
| `cmd/deploy, prod, vm` | `go-devops/cmd/` | Done |
|
||||
|
||||
### Stays in cli/ (meta/framework commands)
|
||||
|
||||
`config`, `doctor`, `help`, `module`, `pkgcmd`, `plugin`, `session`
|
||||
|
||||
---
|
||||
|
||||
## 3. Variant Binaries (future)
|
||||
|
||||
The cli/ repo can produce variant binaries by creating multiple `main.go` files that wire different sets of commands.
|
||||
|
||||
```
|
||||
cli/
|
||||
├── main.go # Current — meta commands only
|
||||
├── cmd/core-full/main.go # Full CLI — all ecosystem commands
|
||||
├── cmd/core-ci/main.go # CI agent dispatch + SCM
|
||||
├── cmd/core-mlx/main.go # ML inference subprocess
|
||||
└── cmd/core-ops/main.go # DevOps + infra management
|
||||
```
|
||||
|
||||
Each variant calls `cli.Main()` with its specific `cli.WithCommands()` set. No blank imports needed.
|
||||
|
||||
### Why variants matter
|
||||
|
||||
- `core-mlx` ships to the homelab as a ~10MB binary, not 50MB with devops/forge/netops
|
||||
- `core-ci` deploys to agent machines without ML or CGO dependencies
|
||||
- Adding a new variant = one new `main.go` with the right `WithCommands` calls
|
||||
|
||||
---
|
||||
|
||||
## 4. Current State
|
||||
|
||||
cli/ has 7 meta packages, one `main.go`, and zero business logic. Everything else lives in the domain repos that own it. Total cli/ LOC is ~2K.
|
||||
1724
docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md
Normal file
1724
docs/plans/completed/2026-02-21-cli-sdk-expansion-plan-original.md
Normal file
File diff suppressed because it is too large
Load diff
286
docs/plans/completed/2026-02-21-go-forge-design.md
Normal file
286
docs/plans/completed/2026-02-21-go-forge-design.md
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
# go-forge Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**go-forge** is a full-coverage Go client for the Forgejo API (450 endpoints, 284 paths, 229 types). It uses a generic `Resource[T, C, U]` pattern for CRUD operations (91% of endpoints) and hand-written methods for 39 unique action endpoints. Types are generated from Forgejo's `swagger.v1.json` spec.
|
||||
|
||||
**Module path:** `forge.lthn.ai/core/go-forge`
|
||||
|
||||
**Origin:** Extracted from `go-scm/forge/` (45 methods covering 10% of API), expanded to full coverage.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go-forge
|
||||
├── client.go # HTTP client: auth, headers, rate limiting, context.Context
|
||||
├── pagination.go # Generic paginated request helper
|
||||
├── resource.go # Resource[T, C, U] generic CRUD (List/Get/Create/Update/Delete)
|
||||
├── errors.go # Typed error handling (APIError, NotFound, Forbidden, etc.)
|
||||
├── forge.go # Top-level Forge client aggregating all services
|
||||
│
|
||||
├── types/ # Generated from swagger.v1.json
|
||||
│ ├── generate.go # //go:generate directive
|
||||
│ ├── repo.go # Repository, CreateRepoOption, EditRepoOption
|
||||
│ ├── issue.go # Issue, CreateIssueOption, EditIssueOption
|
||||
│ ├── pr.go # PullRequest, CreatePullRequestOption
|
||||
│ ├── user.go # User, CreateUserOption
|
||||
│ ├── org.go # Organisation, CreateOrgOption
|
||||
│ ├── team.go # Team, CreateTeamOption
|
||||
│ ├── label.go # Label, CreateLabelOption
|
||||
│ ├── release.go # Release, CreateReleaseOption
|
||||
│ ├── branch.go # Branch, BranchProtection
|
||||
│ ├── milestone.go # Milestone, CreateMilestoneOption
|
||||
│ ├── hook.go # Hook, CreateHookOption
|
||||
│ ├── key.go # DeployKey, PublicKey, GPGKey
|
||||
│ ├── notification.go # NotificationThread, NotificationSubject
|
||||
│ ├── package.go # Package, PackageFile
|
||||
│ ├── action.go # ActionRunner, ActionSecret, ActionVariable
|
||||
│ ├── commit.go # Commit, CommitStatus, CombinedStatus
|
||||
│ ├── content.go # ContentsResponse, FileOptions
|
||||
│ ├── wiki.go # WikiPage, WikiPageMetaData
|
||||
│ ├── review.go # PullReview, PullReviewComment
|
||||
│ ├── reaction.go # Reaction
|
||||
│ ├── topic.go # TopicResponse
|
||||
│ ├── misc.go # Markdown, License, GitignoreTemplate, NodeInfo
|
||||
│ ├── admin.go # Cron, QuotaGroup, QuotaRule
|
||||
│ ├── activity.go # Activity, Feed
|
||||
│ └── common.go # Shared types: Permission, ExternalTracker, etc.
|
||||
│
|
||||
├── repos.go # RepoService: CRUD + fork, mirror, transfer, template
|
||||
├── issues.go # IssueService: CRUD + pin, deadline, reactions, stopwatch
|
||||
├── pulls.go # PullService: CRUD + merge, update, reviews, dismiss
|
||||
├── orgs.go # OrgService: CRUD + members, avatar, block, hooks
|
||||
├── users.go # UserService: CRUD + keys, followers, starred, settings
|
||||
├── teams.go # TeamService: CRUD + members, repos
|
||||
├── admin.go # AdminService: users, orgs, cron, runners, quota, unadopted
|
||||
├── branches.go # BranchService: CRUD + protection rules
|
||||
├── releases.go # ReleaseService: CRUD + assets
|
||||
├── labels.go # LabelService: repo + org + issue labels
|
||||
├── webhooks.go # WebhookService: CRUD + test hook
|
||||
├── notifications.go # NotificationService: list, mark read
|
||||
├── packages.go # PackageService: list, get, delete
|
||||
├── actions.go # ActionsService: runners, secrets, variables, workflow dispatch
|
||||
├── contents.go # ContentService: file read/write/delete via API
|
||||
├── wiki.go # WikiService: pages
|
||||
├── commits.go # CommitService: status, notes, diff
|
||||
├── misc.go # MiscService: markdown, licenses, gitignore, nodeinfo
|
||||
│
|
||||
├── config.go # URL/token resolution: env → config file → flags
|
||||
│
|
||||
├── cmd/forgegen/ # Code generator: swagger.v1.json → types/*.go
|
||||
│ ├── main.go
|
||||
│ ├── parser.go # Parse OpenAPI 2.0 definitions
|
||||
│ ├── generator.go # Render Go source files
|
||||
│ └── templates/ # Go text/template files for codegen
|
||||
│
|
||||
└── testdata/
|
||||
└── swagger.v1.json # Pinned spec for testing + generation
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Generic Resource[T, C, U]
|
||||
|
||||
Three type parameters: T (resource type), C (create options), U (update options).
|
||||
|
||||
```go
|
||||
type Resource[T any, C any, U any] struct {
|
||||
client *Client
|
||||
path string // e.g. "/api/v1/repos/{owner}/{repo}/issues"
|
||||
}
|
||||
|
||||
func (r *Resource[T, C, U]) List(ctx context.Context, params Params, opts ListOptions) ([]T, error)
|
||||
func (r *Resource[T, C, U]) Get(ctx context.Context, params Params, id string) (*T, error)
|
||||
func (r *Resource[T, C, U]) Create(ctx context.Context, params Params, body *C) (*T, error)
|
||||
func (r *Resource[T, C, U]) Update(ctx context.Context, params Params, id string, body *U) (*T, error)
|
||||
func (r *Resource[T, C, U]) Delete(ctx context.Context, params Params, id string) error
|
||||
```
|
||||
|
||||
`Params` is `map[string]string` resolving path variables: `{"owner": "core", "repo": "go-forge"}`.
|
||||
|
||||
This covers 411 of 450 endpoints (91%).
|
||||
|
||||
### 2. Service Structs Embed Resource
|
||||
|
||||
```go
|
||||
type IssueService struct {
|
||||
Resource[types.Issue, types.CreateIssueOption, types.EditIssueOption]
|
||||
}
|
||||
|
||||
// CRUD comes free. Actions are hand-written:
|
||||
func (s *IssueService) Pin(ctx context.Context, owner, repo string, index int64) error
|
||||
func (s *IssueService) SetDeadline(ctx context.Context, owner, repo string, index int64, deadline *time.Time) error
|
||||
```
|
||||
|
||||
### 3. Top-Level Forge Client
|
||||
|
||||
```go
|
||||
type Forge struct {
|
||||
client *Client
|
||||
Repos *RepoService
|
||||
Issues *IssueService
|
||||
Pulls *PullService
|
||||
Orgs *OrgService
|
||||
Users *UserService
|
||||
Teams *TeamService
|
||||
Admin *AdminService
|
||||
Branches *BranchService
|
||||
Releases *ReleaseService
|
||||
Labels *LabelService
|
||||
Webhooks *WebhookService
|
||||
Notifications *NotificationService
|
||||
Packages *PackageService
|
||||
Actions *ActionsService
|
||||
Contents *ContentService
|
||||
Wiki *WikiService
|
||||
Commits *CommitService
|
||||
Misc *MiscService
|
||||
}
|
||||
|
||||
func NewForge(url, token string, opts ...Option) *Forge
|
||||
```
|
||||
|
||||
### 4. Codegen from swagger.v1.json
|
||||
|
||||
The `cmd/forgegen/` tool reads the OpenAPI 2.0 spec and generates:
|
||||
- Go struct definitions with JSON tags and doc comments
|
||||
- Enum constants
|
||||
- Type mapping (OpenAPI → Go)
|
||||
|
||||
229 type definitions → ~25 grouped Go files in `types/`.
|
||||
|
||||
Type mapping rules:
|
||||
| OpenAPI | Go |
|
||||
|---------|-----|
|
||||
| `string` | `string` |
|
||||
| `string` + `date-time` | `time.Time` |
|
||||
| `integer` + `int64` | `int64` |
|
||||
| `integer` | `int` |
|
||||
| `boolean` | `bool` |
|
||||
| `array` of T | `[]T` |
|
||||
| `$ref` | `*T` (pointer) |
|
||||
| nullable | pointer type |
|
||||
| `binary` | `[]byte` |
|
||||
|
||||
### 5. HTTP Client
|
||||
|
||||
```go
|
||||
type Client struct {
|
||||
baseURL string
|
||||
token string
|
||||
httpClient *http.Client
|
||||
userAgent string
|
||||
}
|
||||
|
||||
func New(url, token string, opts ...Option) *Client
|
||||
|
||||
func (c *Client) Get(ctx context.Context, path string, out any) error
|
||||
func (c *Client) Post(ctx context.Context, path string, body, out any) error
|
||||
func (c *Client) Patch(ctx context.Context, path string, body, out any) error
|
||||
func (c *Client) Put(ctx context.Context, path string, body, out any) error
|
||||
func (c *Client) Delete(ctx context.Context, path string) error
|
||||
```
|
||||
|
||||
Options: `WithHTTPClient`, `WithUserAgent`, `WithRateLimit`, `WithLogger`.
|
||||
|
||||
### 6. Pagination
|
||||
|
||||
Forgejo uses `page` + `limit` query params and `X-Total-Count` response header.
|
||||
|
||||
```go
|
||||
type ListOptions struct {
|
||||
Page int
|
||||
Limit int // default 50, max configurable
|
||||
}
|
||||
|
||||
type PagedResult[T any] struct {
|
||||
Items []T
|
||||
TotalCount int
|
||||
Page int
|
||||
HasMore bool
|
||||
}
|
||||
|
||||
// ListAll fetches all pages automatically.
|
||||
func (r *Resource[T, C, U]) ListAll(ctx context.Context, params Params) ([]T, error)
|
||||
```
|
||||
|
||||
### 7. Error Handling
|
||||
|
||||
```go
|
||||
type APIError struct {
|
||||
StatusCode int
|
||||
Message string
|
||||
URL string
|
||||
}
|
||||
|
||||
func IsNotFound(err error) bool
|
||||
func IsForbidden(err error) bool
|
||||
func IsConflict(err error) bool
|
||||
```
|
||||
|
||||
### 8. Config Resolution (from go-scm/forge)
|
||||
|
||||
Priority: flags → environment → config file.
|
||||
|
||||
```go
|
||||
func NewFromConfig(flagURL, flagToken string) (*Forge, error)
|
||||
func ResolveConfig(flagURL, flagToken string) (url, token string, err error)
|
||||
func SaveConfig(url, token string) error
|
||||
```
|
||||
|
||||
Env vars: `FORGE_URL`, `FORGE_TOKEN`. Config file: `~/.config/forge/config.json`.
|
||||
|
||||
## API Coverage
|
||||
|
||||
| Category | Endpoints | CRUD | Actions |
|
||||
|----------|-----------|------|---------|
|
||||
| Repository | 175 | 165 | 10 (fork, mirror, transfer, template, avatar, diffpatch) |
|
||||
| User | 74 | 70 | 4 (avatar, GPG verify) |
|
||||
| Issue | 67 | 57 | 10 (pin, deadline, reactions, stopwatch, labels) |
|
||||
| Organisation | 63 | 59 | 4 (avatar, block/unblock) |
|
||||
| Admin | 39 | 35 | 4 (cron run, rename, adopt, quota set) |
|
||||
| Miscellaneous | 12 | 7 | 5 (markdown render, markup, nodeinfo) |
|
||||
| Notification | 7 | 7 | 0 |
|
||||
| ActivityPub | 6 | 3 | 3 (inbox POST) |
|
||||
| Package | 4 | 4 | 0 |
|
||||
| Settings | 4 | 4 | 0 |
|
||||
| **Total** | **450** | **411** | **39** |
|
||||
|
||||
## Integration Points
|
||||
|
||||
### go-api
|
||||
|
||||
Services implement `DescribableGroup` from go-api Phase 3, enabling:
|
||||
- REST endpoint generation via ToolBridge
|
||||
- Auto-generated OpenAPI spec
|
||||
- Multi-language SDK codegen
|
||||
|
||||
### go-scm
|
||||
|
||||
go-scm/forge/ becomes a thin adapter importing go-forge types. Existing go-scm users are unaffected — the multi-provider abstraction layer stays.
|
||||
|
||||
### go-ai/mcp
|
||||
|
||||
The MCP subsystem can register go-forge operations as MCP tools, giving AI agents full Forgejo API access.
|
||||
|
||||
## 39 Unique Action Methods
|
||||
|
||||
These require hand-written implementation:
|
||||
|
||||
**Repository:** migrate, fork, generate (template), transfer, accept/reject transfer, mirror sync, push mirror sync, avatar, diffpatch, contents (multi-file modify)
|
||||
|
||||
**Pull Requests:** merge, update (rebase), submit review, dismiss/undismiss review
|
||||
|
||||
**Issues:** pin, set deadline, add reaction, start/stop stopwatch, add issue labels
|
||||
|
||||
**Comments:** add reaction
|
||||
|
||||
**Admin:** run cron task, adopt unadopted, rename user, set quota groups
|
||||
|
||||
**Misc:** render markdown, render raw markdown, render markup, GPG key verify
|
||||
|
||||
**ActivityPub:** inbox POST (actor, repo, user)
|
||||
|
||||
**Actions:** dispatch workflow
|
||||
|
||||
**Git:** set note on commit, test webhook
|
||||
2549
docs/plans/completed/2026-02-21-go-forge-plan.md
Normal file
2549
docs/plans/completed/2026-02-21-go-forge-plan.md
Normal file
File diff suppressed because it is too large
Load diff
30
docs/plans/completed/cli-meta-package.md
Normal file
30
docs/plans/completed/cli-meta-package.md
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
# CLI Meta-Package Restructure — Completed
|
||||
|
||||
**Completed:** 22 Feb 2026
|
||||
|
||||
## What Was Done
|
||||
|
||||
`pkg/cli` was extracted from `core/go` into its own Go module at `forge.lthn.ai/core/cli`. This made the CLI SDK a first-class, independently versioned package rather than a subdirectory of the Go foundation repo.
|
||||
|
||||
Following the extraction, an ecosystem-wide import path migration updated all consumers from the old path to the new one:
|
||||
|
||||
- Old: `forge.lthn.ai/core/go/pkg/cli`
|
||||
- New: `forge.lthn.ai/core/cli/pkg/cli`
|
||||
|
||||
## Scope
|
||||
|
||||
- **147+ files** updated across **10 repos**
|
||||
- All repos build clean after migration
|
||||
|
||||
## Repos Migrated
|
||||
|
||||
`core/cli`, `core/go`, `go-devops`, `go-ai`, `go-agentic`, `go-crypt`, `go-rag`, `go-scm`, `go-api`, `go-update`
|
||||
|
||||
## Key Outcomes
|
||||
|
||||
- `forge.lthn.ai/core/cli/pkg/cli` is the single import for all CLI concerns across the ecosystem
|
||||
- Domain repos are insulated from cobra, lipgloss, and bubbletea — only `pkg/cli` imports them
|
||||
- Command registration uses the Core framework lifecycle via `cli.WithCommands()` — no `init()`, no global state
|
||||
- `core/cli` is a thin assembly repo (~2K LOC) with 7 meta packages; all business logic lives in domain repos
|
||||
- Variant binary pattern established: multiple `main.go` files can wire different `WithCommands` sets for targeted binaries (core-ci, core-mlx, core-ops, etc.)
|
||||
- Command migration from the old `core/cli` monolith to domain repos was completed in full (13 command groups moved)
|
||||
39
docs/plans/completed/cli-sdk-expansion.md
Normal file
39
docs/plans/completed/cli-sdk-expansion.md
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# CLI SDK Expansion — Completion Summary
|
||||
|
||||
**Completed:** 21 February 2026
|
||||
**Module:** `forge.lthn.ai/core/go/pkg/cli` (later migrated to `forge.lthn.ai/core/cli`)
|
||||
**Status:** Complete — all TUI primitives shipped, then extracted to core/cli
|
||||
|
||||
## What Was Built
|
||||
|
||||
Extended `pkg/cli` with charmbracelet TUI primitives so domain repos only
|
||||
import `core/cli` for all CLI concerns. Charmbracelet dependencies (bubbletea,
|
||||
bubbles, lipgloss) are encapsulated behind our own types.
|
||||
|
||||
### Components added
|
||||
|
||||
| Component | File | Purpose |
|
||||
|-----------|------|---------|
|
||||
| RunTUI | `runtui.go` | Escape hatch with `Model`/`Msg`/`Cmd`/`KeyMsg` types |
|
||||
| Spinner | `spinner.go` | Async handle with `Update()`, `Done()`, `Fail()` |
|
||||
| ProgressBar | `progressbar.go` | `Increment()`, `Set()`, `SetMessage()`, `Done()` |
|
||||
| InteractiveList | `list.go` | Keyboard navigation with terminal fallback |
|
||||
| TextInput | `textinput.go` | Placeholder, masking, validation |
|
||||
| Viewport | `viewport.go` | Scrollable content for logs, diffs, docs |
|
||||
| Form (stub) | `form.go` | Interface defined, bufio fallback |
|
||||
| FilePicker (stub) | `filepicker.go` | Interface defined, bufio fallback |
|
||||
| Tabs (stub) | `tabs.go` | Interface defined, simple fallback |
|
||||
|
||||
### Subsequent migration
|
||||
|
||||
On 22 February 2026, `pkg/cli` was extracted from `core/go` into its own
|
||||
module at `forge.lthn.ai/core/cli` and all imports were updated. The TUI
|
||||
primitives now live in the standalone CLI module.
|
||||
|
||||
### Frame upgrade (follow-on)
|
||||
|
||||
The Frame layout system was upgraded to implement `tea.Model` directly on
|
||||
22 February 2026 (in `core/cli`), adding bubbletea lifecycle, `KeyMap` for
|
||||
configurable bindings, `Navigate()`/`Back()` for panel switching, and
|
||||
lipgloss-based HLCRF rendering. This was a separate plan
|
||||
(`frame-bubbletea`) that built on the SDK expansion.
|
||||
57
docs/plans/completed/go-api.md
Normal file
57
docs/plans/completed/go-api.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# go-api — Completion Summary
|
||||
|
||||
**Completed:** 21 February 2026
|
||||
**Module:** `forge.lthn.ai/core/go-api`
|
||||
**Status:** Phases 1–3 complete, 176 tests passing
|
||||
|
||||
## What Was Built
|
||||
|
||||
### Phase 1 — Core Framework (20 Feb 2026)
|
||||
|
||||
Gin-based HTTP engine with extensible middleware via `With*()` options. Key components:
|
||||
|
||||
- `RouteGroup` / `StreamGroup` interfaces — subsystems register their own endpoints
|
||||
- `Response[T]` envelope — `OK()`, `Fail()`, `Paginated()` generics
|
||||
- `Engine` — `New()`, `Register()`, `Handler()`, `Serve()` with graceful shutdown
|
||||
- Bearer auth, request ID, and CORS middleware
|
||||
- WebSocket endpoint wrapping a `go-ws` Hub
|
||||
- Swagger UI at `/swagger/` with runtime spec serving
|
||||
- `/health` endpoint always available without auth
|
||||
- First integration proof in `go-ml/api/` (3 endpoints, 12 tests)
|
||||
|
||||
### Phase 2 — Gin Plugin Stack (20–21 Feb 2026)
|
||||
|
||||
17 middleware plugins added across four waves, all as drop-in `With*()` options:
|
||||
|
||||
| Wave | Plugins |
|
||||
|------|---------|
|
||||
| 1 — Gateway hardening | Authentik (OIDC + forward auth), secure headers, structured slog, timeouts, gzip, static files |
|
||||
| 2 — Performance + auth | Brotli compression, in-memory response cache, server-side sessions, Casbin RBAC |
|
||||
| 3 — Network + streaming | HTTP signature verification, SSE broker, reverse proxy detection, i18n locale, GraphQL |
|
||||
| 4 — Observability | pprof, expvar, OpenTelemetry distributed tracing |
|
||||
|
||||
### Phase 3 — OpenAPI + SDK Codegen (21 Feb 2026)
|
||||
|
||||
Runtime spec generation (not swaggo annotations — incompatible with dynamic RouteGroups and `Response[T]` generics):
|
||||
|
||||
- `DescribableGroup` interface — opt-in OpenAPI metadata for route groups
|
||||
- `ToolBridge` — converts MCP tool descriptors into `POST /{tool_name}` REST endpoints
|
||||
- `SpecBuilder` — assembles full OpenAPI 3.1 JSON from registered groups at runtime
|
||||
- Spec export to JSON and YAML (`core api spec`)
|
||||
- SDK codegen wrapper for openapi-generator-cli, 11 languages (`core api sdk --lang go`)
|
||||
- `go-ai` `mcp/registry.go` — generic `addToolRecorded[In,Out]` captures types in closures
|
||||
- `go-ai` `mcp/bridge.go` — `BridgeToAPI()` populates ToolBridge from MCP tool registry
|
||||
- CLI commands: `core api spec`, `core api sdk` (in `core/cli` dev branch)
|
||||
|
||||
## Key Outcomes
|
||||
|
||||
- **176 tests** across go-api (143), go-ai bridge (10), and CLI commands (4), all passing
|
||||
- Zero internal ecosystem dependencies — subsystems import go-api, not the reverse
|
||||
- Authentik (OIDC) and bearer token auth coexist; Casbin adds RBAC on top
|
||||
- Four-protocol access pattern established: REST, GraphQL, WebSocket, MCP — same handlers
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Subsystem MCP tools registered via `mcp.AddTool` directly are excluded from the REST bridge (only the 10 built-in tools appear). Fix: pass `*Service` to `RegisterTools` instead of `*mcp.Server`.
|
||||
- `structSchema` reflection handles flat structs only; nested structs are not recursed.
|
||||
- `core api spec` currently emits a spec with only `/health`; full MCP wiring into the CLI command is pending.
|
||||
37
docs/plans/completed/mcp-integration.md
Normal file
37
docs/plans/completed/mcp-integration.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# MCP Integration — Completion Summary
|
||||
|
||||
**Completed:** 2026-02-05
|
||||
**Plan:** `docs/plans/2026-02-05-mcp-integration.md`
|
||||
|
||||
## What Was Built
|
||||
|
||||
### RAG Tools (`pkg/mcp/tools_rag.go`)
|
||||
Three MCP tools added to the existing `pkg/mcp` server:
|
||||
- `rag_query` — semantic search against Qdrant vector DB
|
||||
- `rag_ingest` — ingest a file or directory into a named collection
|
||||
- `rag_collections` — list available Qdrant collections (with optional stats)
|
||||
|
||||
### Metrics Tools (`pkg/mcp/tools_metrics.go`)
|
||||
Two MCP tools for agent activity tracking:
|
||||
- `metrics_record` — write a typed event (agent_id, repo, arbitrary data) to JSONL storage
|
||||
- `metrics_query` — query events with aggregation by type, repo, and agent; supports human-friendly duration strings (7d, 24h)
|
||||
|
||||
Also added `parseDuration()` helper for "Nd"/"Nh"/"Nm" duration strings.
|
||||
|
||||
### `core mcp serve` Command (`internal/cmd/mcpcmd/cmd_mcp.go`)
|
||||
New CLI sub-command registered via `cli.WithCommands()` (not `init()`).
|
||||
- Runs `pkg/mcp` server over stdio by default
|
||||
- TCP mode via `MCP_ADDR=:9000` environment variable
|
||||
- `--workspace` flag to restrict file operations to a directory
|
||||
|
||||
Registered in the full CLI variant. i18n strings added for all user-facing text.
|
||||
|
||||
### Plugin Configuration
|
||||
`.mcp.json` created for the `agentic-flows` Claude Code plugin, pointing to `core mcp serve`. Exposes all 15 tools to Claude Code agents via the `core-cli` MCP server name.
|
||||
|
||||
## Key Outcomes
|
||||
|
||||
- `core mcp serve` is the single entry point for all MCP tooling (file ops, RAG, metrics, language detection, process management, WebSocket, webview/CDP)
|
||||
- MCP command moved to `go-ai/cmd/mcpcmd/` in final form; the plan's `internal/cmd/mcpcmd/` path reflects the pre-extraction location
|
||||
- Registration pattern updated from `init()` + `RegisterCommands()` to `cli.WithCommands()` lifecycle hooks
|
||||
- Services required at runtime: Qdrant (localhost:6333), Ollama with nomic-embed-text (localhost:11434)
|
||||
62
docs/plans/completed/qk-bone-orientation.md
Normal file
62
docs/plans/completed/qk-bone-orientation.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
# Q/K Bone Orientation — Completion Summary
|
||||
|
||||
**Completed:** 23 February 2026
|
||||
**Repos:** go-inference, go-mlx, go-ml, LEM
|
||||
**Status:** All 7 tasks complete, 14 files changed (+917 lines), all tests passing
|
||||
|
||||
## What Was Built
|
||||
|
||||
### go-inference — AttentionSnapshot types (Task 1)
|
||||
|
||||
`AttentionSnapshot` struct and `AttentionInspector` optional interface. Backends expose attention data via type assertion — no breaking changes to `TextModel`.
|
||||
|
||||
### go-mlx — KV cache extraction (Task 2)
|
||||
|
||||
`InspectAttention` on `metalAdapter` runs a single prefill pass and extracts post-RoPE K vectors from each layer's KV cache. Tested against real Gemma3-1B (26 layers, 1 KV head via GQA, 256 head dim).
|
||||
|
||||
### go-ml — Adapter pass-through (Task 3)
|
||||
|
||||
`InspectAttention` on `InferenceAdapter` type-asserts the underlying `TextModel` to `AttentionInspector`. Returns clear error for unsupported backends.
|
||||
|
||||
### LEM — Analysis engine (Task 4)
|
||||
|
||||
Pure Go CPU math in `pkg/lem/attention.go`. Computes 5 BO metrics from raw K tensors:
|
||||
|
||||
- **Mean Coherence** — pairwise cosine similarity of K vectors within each layer
|
||||
- **Cross-Layer Alignment** — cosine similarity of mean K vectors between adjacent layers
|
||||
- **Head Entropy** — normalised Shannon entropy of K vector magnitudes across positions
|
||||
- **Phase-Lock Score** — fraction of head pairs above coherence threshold (0.7)
|
||||
- **Joint Collapse Count** — layers where cross-alignment drops below threshold (0.5)
|
||||
|
||||
Composite score: 30% coherence + 25% cross-alignment + 20% phase-lock + 15% entropy + 10% joint stability → 0-100 scale.
|
||||
|
||||
### LEM — CLI command (Task 5)
|
||||
|
||||
`lem score attention -model <path> -prompt <text> [-json]` loads a model, runs InspectAttention, and prints BO metrics.
|
||||
|
||||
### LEM — Distill integration (Task 6)
|
||||
|
||||
Opt-in attention scoring in the distill pipeline. Gated behind `scorer.attention: true` and `scorer.attention_min_score` in ai.yaml. Costs one extra prefill per probe.
|
||||
|
||||
### LEM — Feature vectors (Task 7)
|
||||
|
||||
19D full feature vector: 6D grammar + 8D heuristic + 5D attention (`mean_coherence`, `cross_alignment`, `head_entropy`, `phase_lock`, `joint_stability`). Ready for Poindexter KDTree spatial indexing.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Optional interface** — `AttentionInspector` via type assertion, not added to `TextModel`
|
||||
- **Named `BOResult`** — avoids collision with `metal.AttentionResult` in go-mlx
|
||||
- **Opt-in for distill** — extra prefill per probe is expensive, off by default
|
||||
- **Pure Go analysis** — zero CGO deps in the analysis engine; GPU data extracted once via `.Floats()`
|
||||
|
||||
## Commits
|
||||
|
||||
| Repo | SHA | Message |
|
||||
|------|-----|---------|
|
||||
| go-inference | `0f7263f` | feat: add AttentionInspector optional interface |
|
||||
| go-mlx | `c2177f7` | feat: implement AttentionInspector via KV cache extraction |
|
||||
| go-ml | `45e9fed` | feat: add InspectAttention pass-through |
|
||||
| LEM | `28309b2` | feat: add Q/K Bone Orientation analysis engine |
|
||||
| LEM | `e333192` | feat: add 'lem score attention' CLI |
|
||||
| LEM | `fbc636e` | feat: integrate attention scoring into distill pipeline |
|
||||
| LEM | `b621baa` | feat: add 19D full feature vector |
|
||||
Loading…
Add table
Reference in a new issue