docs: remove completed plan files
Co-Authored-By: Virgil <virgil@lethean.io>
This commit is contained in:
Snider
parent
135bb2f126
commit
ae5ebf2ff2
24 changed files with 0 additions and 13696 deletions
File diff suppressed because it is too large
Load diff
|
|
@ -1,155 +0,0 @@
|
||||||
# core.help Documentation Website — Design
|
|
||||||
|
|
||||||
**Date:** 2026-02-21
|
|
||||||
**Author:** Virgil
|
|
||||||
**Status:** Design approved
|
|
||||||
**Domain:** https://core.help
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Documentation is scattered across 39 repos (18 Go packages, 20 PHP packages, 1 CLI). There is no unified docs site. Developers need a single entry point to find CLI commands, Go package APIs, MCP tool references, and PHP module guides.
|
|
||||||
|
|
||||||
## Solution
|
|
||||||
|
|
||||||
A Hugo + Docsy static site at core.help, built from existing markdown docs aggregated by `core docs sync`. No new content — just collect and present what already exists across the ecosystem.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### Stack
|
|
||||||
|
|
||||||
- **Hugo** — Go-native static site generator, sub-second builds
|
|
||||||
- **Docsy theme** — Purpose-built for technical docs (used by Kubernetes, gRPC, Knative)
|
|
||||||
- **BunnyCDN** — Static hosting with pull zone
|
|
||||||
- **`core docs sync --target hugo`** — Collects markdown from all repos into Hugo content tree
|
|
||||||
|
|
||||||
### Why Hugo + Docsy (not VitePress or mdBook)
|
|
||||||
|
|
||||||
- Go-native, no Node.js dependency
|
|
||||||
- Handles multi-section navigation (CLI, Go packages, PHP modules, MCP tools)
|
|
||||||
- Sub-second builds for ~250 markdown files
|
|
||||||
- Docsy has built-in search, versioned nav, API reference sections
|
|
||||||
|
|
||||||
## Content Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
docs-site/
|
|
||||||
├── hugo.toml
|
|
||||||
├── content/
|
|
||||||
│ ├── _index.md # Landing page
|
|
||||||
│ ├── getting-started/ # CLI top-level guides
|
|
||||||
│ │ ├── _index.md
|
|
||||||
│ │ ├── installation.md
|
|
||||||
│ │ ├── configuration.md
|
|
||||||
│ │ ├── user-guide.md
|
|
||||||
│ │ ├── troubleshooting.md
|
|
||||||
│ │ └── faq.md
|
|
||||||
│ ├── cli/ # CLI command reference (43 commands)
|
|
||||||
│ │ ├── _index.md
|
|
||||||
│ │ ├── dev/ # core dev commit, push, pull, etc.
|
|
||||||
│ │ ├── ai/ # core ai commands
|
|
||||||
│ │ ├── go/ # core go test, lint, etc.
|
|
||||||
│ │ └── ...
|
|
||||||
│ ├── go/ # Go ecosystem packages (18)
|
|
||||||
│ │ ├── _index.md # Ecosystem overview
|
|
||||||
│ │ ├── go-api/ # README + architecture/development/history
|
|
||||||
│ │ ├── go-ai/
|
|
||||||
│ │ ├── go-mlx/
|
|
||||||
│ │ ├── go-i18n/
|
|
||||||
│ │ └── ...
|
|
||||||
│ ├── mcp/ # MCP tool reference (49 tools)
|
|
||||||
│ │ ├── _index.md
|
|
||||||
│ │ ├── file-operations.md
|
|
||||||
│ │ ├── process-management.md
|
|
||||||
│ │ ├── rag.md
|
|
||||||
│ │ └── ...
|
|
||||||
│ ├── php/ # PHP packages (from core-php/docs/packages/)
|
|
||||||
│ │ ├── _index.md
|
|
||||||
│ │ ├── admin/
|
|
||||||
│ │ ├── tenant/
|
|
||||||
│ │ ├── commerce/
|
|
||||||
│ │ └── ...
|
|
||||||
│ └── kb/ # Knowledge base (wiki pages from go-mlx, go-i18n)
|
|
||||||
│ ├── _index.md
|
|
||||||
│ ├── mlx/
|
|
||||||
│ └── i18n/
|
|
||||||
├── static/ # Logos, favicons
|
|
||||||
├── layouts/ # Custom template overrides (minimal)
|
|
||||||
└── go.mod # Hugo modules (Docsy as module dep)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Sync Pipeline
|
|
||||||
|
|
||||||
`core docs sync --target hugo --output site/content/` performs:
|
|
||||||
|
|
||||||
### Source Mapping
|
|
||||||
|
|
||||||
```
|
|
||||||
cli/docs/index.md → content/getting-started/_index.md
|
|
||||||
cli/docs/getting-started.md → content/getting-started/installation.md
|
|
||||||
cli/docs/user-guide.md → content/getting-started/user-guide.md
|
|
||||||
cli/docs/configuration.md → content/getting-started/configuration.md
|
|
||||||
cli/docs/troubleshooting.md → content/getting-started/troubleshooting.md
|
|
||||||
cli/docs/faq.md → content/getting-started/faq.md
|
|
||||||
|
|
||||||
core/docs/cmd/**/*.md → content/cli/**/*.md
|
|
||||||
|
|
||||||
go-*/README.md → content/go/{name}/_index.md
|
|
||||||
go-*/docs/*.md → content/go/{name}/*.md
|
|
||||||
go-*/KB/*.md → content/kb/{name-suffix}/*.md
|
|
||||||
|
|
||||||
core-*/docs/**/*.md → content/php/{name-suffix}/**/*.md
|
|
||||||
```
|
|
||||||
|
|
||||||
### Front Matter Injection
|
|
||||||
|
|
||||||
If a markdown file doesn't start with `---`, prepend:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
---
|
|
||||||
title: "{derived from filename}"
|
|
||||||
linkTitle: "{short name}"
|
|
||||||
weight: {auto-incremented}
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
No other content transformations. Markdown stays as-is.
|
|
||||||
|
|
||||||
### Build & Deploy
|
|
||||||
|
|
||||||
```bash
|
|
||||||
core docs sync --target hugo --output docs-site/content/
|
|
||||||
cd docs-site && hugo build
|
|
||||||
hugo deploy --target bunnycdn
|
|
||||||
```
|
|
||||||
|
|
||||||
Hugo deploy config in `hugo.toml`:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[deployment]
|
|
||||||
[[deployment.targets]]
|
|
||||||
name = "bunnycdn"
|
|
||||||
URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto"
|
|
||||||
```
|
|
||||||
|
|
||||||
Credentials via env vars.
|
|
||||||
|
|
||||||
## Registry
|
|
||||||
|
|
||||||
All 39 repos registered in `.core/repos.yaml` with `docs: true`. Go repos use explicit `path:` fields since they live outside the PHP `base_path`. `FindRegistry()` checks `.core/repos.yaml` alongside `repos.yaml`.
|
|
||||||
|
|
||||||
## Prerequisites Completed
|
|
||||||
|
|
||||||
- [x] `.core/repos.yaml` created with all 39 repos
|
|
||||||
- [x] `FindRegistry()` updated to find `.core/repos.yaml`
|
|
||||||
- [x] `Repo.Path` supports explicit YAML override
|
|
||||||
- [x] go-api docs gap filled (architecture.md, development.md, history.md)
|
|
||||||
- [x] All 18 Go repos have standard docs trio
|
|
||||||
|
|
||||||
## What Remains (Implementation Plan)
|
|
||||||
|
|
||||||
1. Create docs-site repo with Hugo + Docsy scaffold
|
|
||||||
2. Extend `core docs sync` with `--target hugo` mode
|
|
||||||
3. Write section _index.md files (landing page, section intros)
|
|
||||||
4. Hugo config (navigation, search, theme colours)
|
|
||||||
5. BunnyCDN deployment config
|
|
||||||
6. CI pipeline on Forge (optional — can deploy manually initially)
|
|
||||||
|
|
@ -1,642 +0,0 @@
|
||||||
# core.help Hugo Documentation Site — Implementation Plan
|
|
||||||
|
|
||||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
||||||
|
|
||||||
**Goal:** Build a Hugo + Docsy documentation site at core.help that aggregates markdown from 39 repos via `core docs sync --target hugo`.
|
|
||||||
|
|
||||||
**Architecture:** Hugo static site with Docsy theme, populated by extending `core docs sync` with a `--target hugo` flag that maps repo docs into Hugo's `content/` tree with auto-injected front matter. Deploy to BunnyCDN.
|
|
||||||
|
|
||||||
**Tech Stack:** Hugo (Go SSG), Docsy theme (Hugo module), BunnyCDN, `core docs sync` CLI
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
The docs sync command lives in `/Users/snider/Code/host-uk/cli/cmd/docs/`. The site will be scaffolded at `/Users/snider/Code/host-uk/docs-site/`. The registry at `/Users/snider/Code/host-uk/.core/repos.yaml` already contains all 39 repos (20 PHP + 18 Go + 1 CLI) with explicit paths for Go repos.
|
|
||||||
|
|
||||||
Key files:
|
|
||||||
- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go` — sync command (modify)
|
|
||||||
- `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go` — repo scanner (modify)
|
|
||||||
- `/Users/snider/Code/host-uk/docs-site/` — Hugo site (create)
|
|
||||||
|
|
||||||
## Task 1: Scaffold Hugo + Docsy site
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/hugo.toml`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/go.mod`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/_index.md`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/getting-started/_index.md`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/cli/_index.md`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/go/_index.md`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/mcp/_index.md`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/php/_index.md`
|
|
||||||
- Create: `/Users/snider/Code/host-uk/docs-site/content/kb/_index.md`
|
|
||||||
|
|
||||||
This is the one-time Hugo scaffolding. No tests — just files.
|
|
||||||
|
|
||||||
**`hugo.toml`:**
|
|
||||||
```toml
|
|
||||||
baseURL = "https://core.help/"
|
|
||||||
title = "Core Documentation"
|
|
||||||
languageCode = "en"
|
|
||||||
defaultContentLanguage = "en"
|
|
||||||
|
|
||||||
enableRobotsTXT = true
|
|
||||||
enableGitInfo = false
|
|
||||||
|
|
||||||
[outputs]
|
|
||||||
home = ["HTML", "JSON"]
|
|
||||||
section = ["HTML"]
|
|
||||||
|
|
||||||
[params]
|
|
||||||
description = "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools"
|
|
||||||
copyright = "Host UK — EUPL-1.2"
|
|
||||||
|
|
||||||
[params.ui]
|
|
||||||
sidebar_menu_compact = true
|
|
||||||
breadcrumb_disable = false
|
|
||||||
sidebar_search_disable = false
|
|
||||||
navbar_logo = false
|
|
||||||
|
|
||||||
[params.ui.readingtime]
|
|
||||||
enable = false
|
|
||||||
|
|
||||||
[module]
|
|
||||||
proxy = "direct"
|
|
||||||
|
|
||||||
[module.hugoVersion]
|
|
||||||
extended = true
|
|
||||||
min = "0.120.0"
|
|
||||||
|
|
||||||
[[module.imports]]
|
|
||||||
path = "github.com/google/docsy"
|
|
||||||
disable = false
|
|
||||||
|
|
||||||
[markup.goldmark.renderer]
|
|
||||||
unsafe = true
|
|
||||||
|
|
||||||
[menu]
|
|
||||||
[[menu.main]]
|
|
||||||
name = "Getting Started"
|
|
||||||
weight = 10
|
|
||||||
url = "/getting-started/"
|
|
||||||
[[menu.main]]
|
|
||||||
name = "CLI Reference"
|
|
||||||
weight = 20
|
|
||||||
url = "/cli/"
|
|
||||||
[[menu.main]]
|
|
||||||
name = "Go Packages"
|
|
||||||
weight = 30
|
|
||||||
url = "/go/"
|
|
||||||
[[menu.main]]
|
|
||||||
name = "MCP Tools"
|
|
||||||
weight = 40
|
|
||||||
url = "/mcp/"
|
|
||||||
[[menu.main]]
|
|
||||||
name = "PHP Packages"
|
|
||||||
weight = 50
|
|
||||||
url = "/php/"
|
|
||||||
[[menu.main]]
|
|
||||||
name = "Knowledge Base"
|
|
||||||
weight = 60
|
|
||||||
url = "/kb/"
|
|
||||||
```
|
|
||||||
|
|
||||||
**`go.mod`:**
|
|
||||||
```
|
|
||||||
module github.com/host-uk/docs-site
|
|
||||||
|
|
||||||
go 1.22
|
|
||||||
|
|
||||||
require github.com/google/docsy v0.11.0
|
|
||||||
```
|
|
||||||
|
|
||||||
Note: Run `hugo mod get` after creating these files to populate `go.sum` and download Docsy.
|
|
||||||
|
|
||||||
**Section `_index.md` files** — each needs Hugo front matter:
|
|
||||||
|
|
||||||
`content/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "Core Documentation"
|
|
||||||
description: "Documentation for the Core CLI, Go packages, PHP modules, and MCP tools"
|
|
||||||
---
|
|
||||||
|
|
||||||
Welcome to the Core ecosystem documentation.
|
|
||||||
|
|
||||||
## Sections
|
|
||||||
|
|
||||||
- [Getting Started](/getting-started/) — Installation, configuration, and first steps
|
|
||||||
- [CLI Reference](/cli/) — Command reference for `core` CLI
|
|
||||||
- [Go Packages](/go/) — Go ecosystem package documentation
|
|
||||||
- [MCP Tools](/mcp/) — Model Context Protocol tool reference
|
|
||||||
- [PHP Packages](/php/) — PHP module documentation
|
|
||||||
- [Knowledge Base](/kb/) — Wiki articles and deep dives
|
|
||||||
```
|
|
||||||
|
|
||||||
`content/getting-started/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "Getting Started"
|
|
||||||
linkTitle: "Getting Started"
|
|
||||||
weight: 10
|
|
||||||
description: "Installation, configuration, and first steps with the Core CLI"
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
`content/cli/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "CLI Reference"
|
|
||||||
linkTitle: "CLI Reference"
|
|
||||||
weight: 20
|
|
||||||
description: "Command reference for the core CLI tool"
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
`content/go/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "Go Packages"
|
|
||||||
linkTitle: "Go Packages"
|
|
||||||
weight: 30
|
|
||||||
description: "Documentation for the Go ecosystem packages"
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
`content/mcp/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "MCP Tools"
|
|
||||||
linkTitle: "MCP Tools"
|
|
||||||
weight: 40
|
|
||||||
description: "Model Context Protocol tool reference — file operations, RAG, ML inference, process management"
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
`content/php/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "PHP Packages"
|
|
||||||
linkTitle: "PHP Packages"
|
|
||||||
weight: 50
|
|
||||||
description: "Documentation for the PHP module ecosystem"
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
`content/kb/_index.md`:
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
title: "Knowledge Base"
|
|
||||||
linkTitle: "Knowledge Base"
|
|
||||||
weight: 60
|
|
||||||
description: "Wiki articles, deep dives, and reference material"
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verify:** After creating files, run from `/Users/snider/Code/host-uk/docs-site/`:
|
|
||||||
```bash
|
|
||||||
hugo mod get
|
|
||||||
hugo server
|
|
||||||
```
|
|
||||||
The site should start and show the landing page with Docsy theme at `localhost:1313`.
|
|
||||||
|
|
||||||
**Commit:**
|
|
||||||
```bash
|
|
||||||
cd /Users/snider/Code/host-uk/docs-site
|
|
||||||
git init
|
|
||||||
git add .
|
|
||||||
git commit -m "feat: scaffold Hugo + Docsy documentation site"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 2: Extend scanRepoDocs to collect KB/ and README
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_scan.go`
|
|
||||||
|
|
||||||
Currently `scanRepoDocs` only collects files from `docs/`. For the Hugo target we also need:
|
|
||||||
- `KB/**/*.md` files (wiki pages from go-mlx, go-i18n)
|
|
||||||
- `README.md` content (becomes the package _index.md)
|
|
||||||
|
|
||||||
Add a `KBFiles []string` field to `RepoDocInfo` and scan `KB/` alongside `docs/`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type RepoDocInfo struct {
|
|
||||||
Name string
|
|
||||||
Path string
|
|
||||||
HasDocs bool
|
|
||||||
Readme string
|
|
||||||
ClaudeMd string
|
|
||||||
Changelog string
|
|
||||||
DocsFiles []string // All files in docs/ directory (recursive)
|
|
||||||
KBFiles []string // All files in KB/ directory (recursive)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
In `scanRepoDocs`, after the `docs/` walk, add a second walk for `KB/`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Recursively scan KB/ directory for .md files
|
|
||||||
kbDir := filepath.Join(repo.Path, "KB")
|
|
||||||
if _, err := io.Local.List(kbDir); err == nil {
|
|
||||||
_ = filepath.WalkDir(kbDir, func(path string, d fs.DirEntry, err error) error {
|
|
||||||
if err != nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
if d.IsDir() || !strings.HasSuffix(d.Name(), ".md") {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
relPath, _ := filepath.Rel(kbDir, path)
|
|
||||||
info.KBFiles = append(info.KBFiles, relPath)
|
|
||||||
info.HasDocs = true
|
|
||||||
return nil
|
|
||||||
})
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Tests:** The existing tests should still pass. No new test file needed — this is a data-collection change.
|
|
||||||
|
|
||||||
**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...`
|
|
||||||
|
|
||||||
**Commit:**
|
|
||||||
```bash
|
|
||||||
git add cmd/docs/cmd_scan.go
|
|
||||||
git commit -m "feat(docs): scan KB/ directory alongside docs/"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 3: Add `--target hugo` flag and Hugo sync logic
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `/Users/snider/Code/host-uk/cli/cmd/docs/cmd_sync.go`
|
|
||||||
|
|
||||||
This is the main task. Add a `--target` flag (default `"php"`) and a new `runHugoSync` function that maps repos to Hugo's content tree.
|
|
||||||
|
|
||||||
**Add flag variable and registration:**
|
|
||||||
|
|
||||||
```go
|
|
||||||
var (
|
|
||||||
docsSyncRegistryPath string
|
|
||||||
docsSyncDryRun bool
|
|
||||||
docsSyncOutputDir string
|
|
||||||
docsSyncTarget string
|
|
||||||
)
|
|
||||||
|
|
||||||
func init() {
|
|
||||||
docsSyncCmd.Flags().StringVar(&docsSyncRegistryPath, "registry", "", i18n.T("common.flag.registry"))
|
|
||||||
docsSyncCmd.Flags().BoolVar(&docsSyncDryRun, "dry-run", false, i18n.T("cmd.docs.sync.flag.dry_run"))
|
|
||||||
docsSyncCmd.Flags().StringVar(&docsSyncOutputDir, "output", "", i18n.T("cmd.docs.sync.flag.output"))
|
|
||||||
docsSyncCmd.Flags().StringVar(&docsSyncTarget, "target", "php", "Target format: php (default) or hugo")
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Update RunE to pass target:**
|
|
||||||
```go
|
|
||||||
RunE: func(cmd *cli.Command, args []string) error {
|
|
||||||
return runDocsSync(docsSyncRegistryPath, docsSyncOutputDir, docsSyncDryRun, docsSyncTarget)
|
|
||||||
},
|
|
||||||
```
|
|
||||||
|
|
||||||
**Update `runDocsSync` signature and add target dispatch:**
|
|
||||||
```go
|
|
||||||
func runDocsSync(registryPath string, outputDir string, dryRun bool, target string) error {
|
|
||||||
reg, basePath, err := loadRegistry(registryPath)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
switch target {
|
|
||||||
case "hugo":
|
|
||||||
return runHugoSync(reg, basePath, outputDir, dryRun)
|
|
||||||
default:
|
|
||||||
return runPHPSync(reg, basePath, outputDir, dryRun)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rename current sync body to `runPHPSync`** — extract lines 67-159 of current `runDocsSync` into `runPHPSync(reg, basePath, outputDir string, dryRun bool) error`. This is a pure extract, no logic changes.
|
|
||||||
|
|
||||||
**Add `hugoOutputName` mapping function:**
|
|
||||||
```go
|
|
||||||
// hugoOutputName maps repo name to Hugo content section and folder.
|
|
||||||
// Returns (section, folder) where section is the top-level content dir.
|
|
||||||
func hugoOutputName(repoName string) (string, string) {
|
|
||||||
// CLI guides
|
|
||||||
if repoName == "cli" {
|
|
||||||
return "getting-started", ""
|
|
||||||
}
|
|
||||||
// Core CLI command docs
|
|
||||||
if repoName == "core" {
|
|
||||||
return "cli", ""
|
|
||||||
}
|
|
||||||
// Go packages
|
|
||||||
if strings.HasPrefix(repoName, "go-") {
|
|
||||||
return "go", repoName
|
|
||||||
}
|
|
||||||
// PHP packages
|
|
||||||
if strings.HasPrefix(repoName, "core-") {
|
|
||||||
return "php", strings.TrimPrefix(repoName, "core-")
|
|
||||||
}
|
|
||||||
return "go", repoName
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Add front matter injection helper:**
|
|
||||||
```go
|
|
||||||
// injectFrontMatter prepends Hugo front matter to markdown content if missing.
|
|
||||||
func injectFrontMatter(content []byte, title string, weight int) []byte {
|
|
||||||
// Already has front matter
|
|
||||||
if bytes.HasPrefix(bytes.TrimSpace(content), []byte("---")) {
|
|
||||||
return content
|
|
||||||
}
|
|
||||||
fm := fmt.Sprintf("---\ntitle: %q\nweight: %d\n---\n\n", title, weight)
|
|
||||||
return append([]byte(fm), content...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// titleFromFilename derives a human-readable title from a filename.
|
|
||||||
func titleFromFilename(filename string) string {
|
|
||||||
name := strings.TrimSuffix(filepath.Base(filename), ".md")
|
|
||||||
name = strings.ReplaceAll(name, "-", " ")
|
|
||||||
name = strings.ReplaceAll(name, "_", " ")
|
|
||||||
// Title case
|
|
||||||
words := strings.Fields(name)
|
|
||||||
for i, w := range words {
|
|
||||||
if len(w) > 0 {
|
|
||||||
words[i] = strings.ToUpper(w[:1]) + w[1:]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return strings.Join(words, " ")
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Add `runHugoSync` function:**
|
|
||||||
```go
|
|
||||||
func runHugoSync(reg *repos.Registry, basePath string, outputDir string, dryRun bool) error {
|
|
||||||
if outputDir == "" {
|
|
||||||
outputDir = filepath.Join(basePath, "docs-site", "content")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Scan all repos
|
|
||||||
var docsInfo []RepoDocInfo
|
|
||||||
for _, repo := range reg.List() {
|
|
||||||
if repo.Name == "core-template" || repo.Name == "core-claude" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
info := scanRepoDocs(repo)
|
|
||||||
if info.HasDocs {
|
|
||||||
docsInfo = append(docsInfo, info)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(docsInfo) == 0 {
|
|
||||||
cli.Text("No documentation found")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
cli.Print("\n Hugo sync: %d repos with docs → %s\n\n", len(docsInfo), outputDir)
|
|
||||||
|
|
||||||
// Show plan
|
|
||||||
for _, info := range docsInfo {
|
|
||||||
section, folder := hugoOutputName(info.Name)
|
|
||||||
target := section
|
|
||||||
if folder != "" {
|
|
||||||
target = section + "/" + folder
|
|
||||||
}
|
|
||||||
fileCount := len(info.DocsFiles) + len(info.KBFiles)
|
|
||||||
if info.Readme != "" {
|
|
||||||
fileCount++
|
|
||||||
}
|
|
||||||
cli.Print(" %s → %s/ (%d files)\n", repoNameStyle.Render(info.Name), target, fileCount)
|
|
||||||
}
|
|
||||||
|
|
||||||
if dryRun {
|
|
||||||
cli.Print("\n Dry run — no files written\n")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
cli.Blank()
|
|
||||||
if !confirm("Sync to Hugo content directory?") {
|
|
||||||
cli.Text("Aborted")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
cli.Blank()
|
|
||||||
var synced int
|
|
||||||
for _, info := range docsInfo {
|
|
||||||
section, folder := hugoOutputName(info.Name)
|
|
||||||
|
|
||||||
// Build destination path
|
|
||||||
destDir := filepath.Join(outputDir, section)
|
|
||||||
if folder != "" {
|
|
||||||
destDir = filepath.Join(destDir, folder)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Copy docs/ files
|
|
||||||
weight := 10
|
|
||||||
docsDir := filepath.Join(info.Path, "docs")
|
|
||||||
for _, f := range info.DocsFiles {
|
|
||||||
src := filepath.Join(docsDir, f)
|
|
||||||
dst := filepath.Join(destDir, f)
|
|
||||||
if err := copyWithFrontMatter(src, dst, weight); err != nil {
|
|
||||||
cli.Print(" %s %s: %s\n", errorStyle.Render("✗"), f, err)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
weight += 10
|
|
||||||
}
|
|
||||||
|
|
||||||
// Copy README.md as _index.md (if not CLI/core which use their own index)
|
|
||||||
if info.Readme != "" && folder != "" {
|
|
||||||
dst := filepath.Join(destDir, "_index.md")
|
|
||||||
if err := copyWithFrontMatter(info.Readme, dst, 1); err != nil {
|
|
||||||
cli.Print(" %s README: %s\n", errorStyle.Render("✗"), err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Copy KB/ files to kb/{suffix}/
|
|
||||||
if len(info.KBFiles) > 0 {
|
|
||||||
// Extract suffix: go-mlx → mlx, go-i18n → i18n
|
|
||||||
suffix := strings.TrimPrefix(info.Name, "go-")
|
|
||||||
kbDestDir := filepath.Join(outputDir, "kb", suffix)
|
|
||||||
kbDir := filepath.Join(info.Path, "KB")
|
|
||||||
kbWeight := 10
|
|
||||||
for _, f := range info.KBFiles {
|
|
||||||
src := filepath.Join(kbDir, f)
|
|
||||||
dst := filepath.Join(kbDestDir, f)
|
|
||||||
if err := copyWithFrontMatter(src, dst, kbWeight); err != nil {
|
|
||||||
cli.Print(" %s KB/%s: %s\n", errorStyle.Render("✗"), f, err)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
kbWeight += 10
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
cli.Print(" %s %s\n", successStyle.Render("✓"), info.Name)
|
|
||||||
synced++
|
|
||||||
}
|
|
||||||
|
|
||||||
cli.Print("\n Synced %d repos to Hugo content\n", synced)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// copyWithFrontMatter copies a markdown file, injecting front matter if missing.
|
|
||||||
func copyWithFrontMatter(src, dst string, weight int) error {
|
|
||||||
if err := io.Local.EnsureDir(filepath.Dir(dst)); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
content, err := io.Local.Read(src)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
title := titleFromFilename(src)
|
|
||||||
result := injectFrontMatter([]byte(content), title, weight)
|
|
||||||
return io.Local.Write(dst, string(result))
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Add imports** at top of file:
|
|
||||||
```go
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"fmt"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"forge.lthn.ai/core/go/pkg/cli"
|
|
||||||
"forge.lthn.ai/core/go/pkg/i18n"
|
|
||||||
"forge.lthn.ai/core/go/pkg/io"
|
|
||||||
"forge.lthn.ai/core/go/pkg/repos"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verify:** `cd /Users/snider/Code/host-uk/cli && GOWORK=off go build ./cmd/docs/...`
|
|
||||||
|
|
||||||
**Commit:**
|
|
||||||
```bash
|
|
||||||
git add cmd/docs/cmd_sync.go
|
|
||||||
git commit -m "feat(docs): add --target hugo sync mode for core.help"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 4: Test the full pipeline
|
|
||||||
|
|
||||||
**No code changes.** Run the pipeline end-to-end.
|
|
||||||
|
|
||||||
**Step 1:** Sync docs to Hugo:
|
|
||||||
```bash
|
|
||||||
cd /Users/snider/Code/host-uk
|
|
||||||
core docs sync --target hugo --dry-run
|
|
||||||
```
|
|
||||||
Verify all 39 repos appear with correct section mappings.
|
|
||||||
|
|
||||||
**Step 2:** Run actual sync:
|
|
||||||
```bash
|
|
||||||
core docs sync --target hugo
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 3:** Build and preview:
|
|
||||||
```bash
|
|
||||||
cd /Users/snider/Code/host-uk/docs-site
|
|
||||||
hugo server
|
|
||||||
```
|
|
||||||
Open `localhost:1313` and verify:
|
|
||||||
- Landing page renders with section links
|
|
||||||
- Getting Started section has CLI guides
|
|
||||||
- CLI Reference section has command docs
|
|
||||||
- Go Packages section has 18 packages with architecture/development/history
|
|
||||||
- PHP Packages section has PHP module docs
|
|
||||||
- Knowledge Base has MLX and i18n wiki pages
|
|
||||||
- Navigation works, search works
|
|
||||||
|
|
||||||
**Step 4:** Fix any issues found during preview.
|
|
||||||
|
|
||||||
**Commit docs-site content:**
|
|
||||||
```bash
|
|
||||||
cd /Users/snider/Code/host-uk/docs-site
|
|
||||||
git add content/
|
|
||||||
git commit -m "feat: sync initial content from 39 repos"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 5: BunnyCDN deployment config
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `/Users/snider/Code/host-uk/docs-site/hugo.toml`
|
|
||||||
|
|
||||||
Add deployment target:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[deployment]
|
|
||||||
[[deployment.targets]]
|
|
||||||
name = "production"
|
|
||||||
URL = "s3://core-help?endpoint=storage.bunnycdn.com®ion=auto"
|
|
||||||
```
|
|
||||||
|
|
||||||
Add a `Taskfile.yml` for convenience:
|
|
||||||
|
|
||||||
**Create:** `/Users/snider/Code/host-uk/docs-site/Taskfile.yml`
|
|
||||||
```yaml
|
|
||||||
version: '3'
|
|
||||||
|
|
||||||
tasks:
|
|
||||||
dev:
|
|
||||||
desc: Start Hugo dev server
|
|
||||||
cmds:
|
|
||||||
- hugo server --buildDrafts
|
|
||||||
|
|
||||||
build:
|
|
||||||
desc: Build static site
|
|
||||||
cmds:
|
|
||||||
- hugo --minify
|
|
||||||
|
|
||||||
sync:
|
|
||||||
desc: Sync docs from all repos
|
|
||||||
dir: ..
|
|
||||||
cmds:
|
|
||||||
- core docs sync --target hugo
|
|
||||||
|
|
||||||
deploy:
|
|
||||||
desc: Build and deploy to BunnyCDN
|
|
||||||
cmds:
|
|
||||||
- task: sync
|
|
||||||
- task: build
|
|
||||||
- hugo deploy --target production
|
|
||||||
|
|
||||||
clean:
|
|
||||||
desc: Remove generated content (keeps _index.md files)
|
|
||||||
cmds:
|
|
||||||
- find content -name "*.md" ! -name "_index.md" -delete
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verify:** `task dev` starts the site.
|
|
||||||
|
|
||||||
**Commit:**
|
|
||||||
```bash
|
|
||||||
git add hugo.toml Taskfile.yml
|
|
||||||
git commit -m "feat: add BunnyCDN deployment config and Taskfile"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Dependency Sequencing
|
|
||||||
|
|
||||||
```
|
|
||||||
Task 1 (Hugo scaffold) — independent, do first
|
|
||||||
Task 2 (scan KB/) — independent, can parallel with Task 1
|
|
||||||
Task 3 (--target hugo) — depends on Task 2
|
|
||||||
Task 4 (test pipeline) — depends on Tasks 1 + 3
|
|
||||||
Task 5 (deploy config) — depends on Task 1
|
|
||||||
```
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
After all tasks:
|
|
||||||
1. `core docs sync --target hugo` populates `docs-site/content/` from all repos
|
|
||||||
2. `cd docs-site && hugo server` renders the full site
|
|
||||||
3. Navigation has 6 sections: Getting Started, CLI, Go, MCP, PHP, KB
|
|
||||||
4. All existing markdown renders correctly with auto-injected front matter
|
|
||||||
5. `hugo build` produces `public/` with no errors
|
|
||||||
|
|
@ -1,271 +0,0 @@
|
||||||
# Core-IDE Job Runner Design
|
|
||||||
|
|
||||||
**Date:** 2026-02-05
|
|
||||||
**Status:** Approved
|
|
||||||
**Author:** @Snider + Claude
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Turn core-ide into an autonomous job runner that polls for actionable pipeline work, executes it via typed MCP tool handlers, captures JSONL training data, and self-updates. Supports 12 nodes running headless on servers and desktop on developer machines.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture Overview
|
|
||||||
|
|
||||||
```
|
|
||||||
+-------------------------------------------------+
|
|
||||||
| core-ide |
|
|
||||||
| |
|
|
||||||
| +----------+ +-----------+ +----------+ |
|
|
||||||
| | Poller |-->| Dispatcher|-->| Handler | |
|
|
||||||
| | (Source) | | (MCP route)| | Registry | |
|
|
||||||
| +----------+ +-----------+ +----------+ |
|
|
||||||
| | | | |
|
|
||||||
| | +----v----+ +---v-------+ |
|
|
||||||
| | | Journal | | JobSource | |
|
|
||||||
| | | (JSONL) | | (adapter) | |
|
|
||||||
| | +---------+ +-----------+ |
|
|
||||||
| +----v-----+ |
|
|
||||||
| | Updater | (existing internal/cmd/updater) |
|
|
||||||
| +----------+ |
|
|
||||||
+-------------------------------------------------+
|
|
||||||
```
|
|
||||||
|
|
||||||
**Three components:**
|
|
||||||
- **Poller** -- Periodic scan via pluggable JobSource adapters. Builds PipelineSignal structs from API responses. Never reads comment bodies (injection vector).
|
|
||||||
- **Dispatcher** -- Matches signals against handler registry in priority order. One action per signal per cycle (prevents cascades).
|
|
||||||
- **Journal** -- Appends JSONL after each completed action per issue-epic step 10 spec. Structural signals only -- IDs, SHAs, timestamps, cycle counts, instructions sent, automations performed.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Job Source Abstraction
|
|
||||||
|
|
||||||
GitHub is the first adapter. The platform's own Agentic API replaces it later. Handler logic is source-agnostic.
|
|
||||||
|
|
||||||
```go
|
|
||||||
type JobSource interface {
|
|
||||||
Name() string
|
|
||||||
Poll(ctx context.Context) ([]*PipelineSignal, error)
|
|
||||||
Report(ctx context.Context, result *ActionResult) error
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
| Adapter | When | Transport |
|
|
||||||
|-------------------|-------|----------------------------------------|
|
|
||||||
| `GitHubSource` | Now | REST API + conditional requests (ETag) |
|
|
||||||
| `HostUKSource` | Next | Agentic API (WebSocket or poll) |
|
|
||||||
| `HyperswarmSource`| Later | P2P encrypted channels via Holepunch |
|
|
||||||
|
|
||||||
**Multi-source:** Poller runs multiple sources concurrently. Own repos get priority. When idle (zero signals for N consecutive cycles), external project sources activate (WailsApp first).
|
|
||||||
|
|
||||||
**API budget:** 50% credit allocation for harvest mode is a config value on the source, not hardcoded.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Pipeline Signal
|
|
||||||
|
|
||||||
The structural snapshot passed to handlers. Never contains comment bodies or free text.
|
|
||||||
|
|
||||||
```go
|
|
||||||
type PipelineSignal struct {
|
|
||||||
EpicNumber int
|
|
||||||
ChildNumber int
|
|
||||||
PRNumber int
|
|
||||||
RepoOwner string
|
|
||||||
RepoName string
|
|
||||||
PRState string // OPEN, MERGED, CLOSED
|
|
||||||
IsDraft bool
|
|
||||||
Mergeable string // MERGEABLE, CONFLICTING, UNKNOWN
|
|
||||||
CheckStatus string // SUCCESS, FAILURE, PENDING
|
|
||||||
ThreadsTotal int
|
|
||||||
ThreadsResolved int
|
|
||||||
LastCommitSHA string
|
|
||||||
LastCommitAt time.Time
|
|
||||||
LastReviewAt time.Time
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Handler Registry
|
|
||||||
|
|
||||||
Each action from the issue-epic flow is a registered handler. All Go functions with typed inputs/outputs.
|
|
||||||
|
|
||||||
```go
|
|
||||||
type JobHandler interface {
|
|
||||||
Name() string
|
|
||||||
Match(signal *PipelineSignal) bool
|
|
||||||
Execute(ctx context.Context, signal *PipelineSignal) (*ActionResult, error)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
| Handler | Epic Stage | Input Signals | Action |
|
|
||||||
|--------------------|-----------|---------------------------------------------------|---------------------------------------------|
|
|
||||||
| `publish_draft` | 3 | PR draft=true, checks=SUCCESS | Mark PR as ready for review |
|
|
||||||
| `send_fix_command` | 4/6 | PR CONFLICTING or threads without fix commit | Comment "fix merge conflict" / "fix the code reviews" |
|
|
||||||
| `resolve_threads` | 5 | Unresolved threads, fix commit exists after review | Resolve all pre-commit threads |
|
|
||||||
| `enable_auto_merge`| 7 | PR MERGEABLE, checks passing, threads resolved | Enable auto-merge via API |
|
|
||||||
| `tick_parent` | 8 | Child PR merged | Update epic issue checklist |
|
|
||||||
| `close_child` | 9 | Child PR merged + parent ticked | Close child issue |
|
|
||||||
| `capture_journal` | 10 | Any completed action | Append JSONL entry |
|
|
||||||
|
|
||||||
**ActionResult** carries what was done -- action name, target IDs, success/failure, timestamps. Feeds directly into JSONL journal.
|
|
||||||
|
|
||||||
Handlers register at init time, same pattern as CLI commands in the existing codebase.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Headless vs Desktop Mode
|
|
||||||
|
|
||||||
Same binary, same handlers, different UI surface.
|
|
||||||
|
|
||||||
**Detection:**
|
|
||||||
|
|
||||||
```go
|
|
||||||
func hasDisplay() bool {
|
|
||||||
if runtime.GOOS == "windows" { return true }
|
|
||||||
return os.Getenv("DISPLAY") != "" || os.Getenv("WAYLAND_DISPLAY") != ""
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Headless mode** (Linux server, no display):
|
|
||||||
- Skip Wails window creation
|
|
||||||
- Start poller immediately
|
|
||||||
- Start MCP bridge (port 9877) for external tool access
|
|
||||||
- Log to stdout/file (structured JSON)
|
|
||||||
- Updater: check on startup, auto-apply + restart via watcher
|
|
||||||
- Managed by systemd: `Restart=always`
|
|
||||||
|
|
||||||
**Desktop mode** (display available):
|
|
||||||
- Full Wails system tray + webview panel
|
|
||||||
- Tray icon shows status: idle, polling, executing, error
|
|
||||||
- Tray menu: Start/Stop poller, Force update, Open journal, Configure sources
|
|
||||||
- Poller off by default (developer toggle)
|
|
||||||
- Same MCP bridge, same handlers, same journal
|
|
||||||
|
|
||||||
**CLI override:** `core-ide --headless` forces headless. `core-ide --desktop` forces GUI.
|
|
||||||
|
|
||||||
**Shared startup:**
|
|
||||||
|
|
||||||
```go
|
|
||||||
func main() {
|
|
||||||
// 1. Load config (repos, interval, channel, sources)
|
|
||||||
// 2. Build handler registry
|
|
||||||
// 3. Init journal
|
|
||||||
// 4. Init updater (check on startup)
|
|
||||||
// 5. Branch:
|
|
||||||
if hasDisplay() {
|
|
||||||
startDesktop() // Wails + tray + optional poller
|
|
||||||
} else {
|
|
||||||
startHeadless() // Poller + MCP bridge + signal handling
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Poller Configuration
|
|
||||||
|
|
||||||
```go
|
|
||||||
type PollerConfig struct {
|
|
||||||
Sources []JobSource
|
|
||||||
Handlers []JobHandler
|
|
||||||
Journal *Journal
|
|
||||||
PollInterval time.Duration // default: 60s
|
|
||||||
DryRun bool // log without executing
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rate limiting:** GitHub API allows 5000 req/hr with token. Full scan of 4 repos with ~30 PRs uses ~150 requests. Poller uses conditional requests (If-None-Match/ETag) to avoid counting unchanged responses. Backs off to 5min interval when idle.
|
|
||||||
|
|
||||||
**CLI flags:**
|
|
||||||
- `--poll-interval` (default: 60s)
|
|
||||||
- `--repos` (comma-separated: `host-uk/core,host-uk/core-php`)
|
|
||||||
- `--dry-run` (log actions without executing)
|
|
||||||
- `--headless` / `--desktop` (mode override)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Self-Update
|
|
||||||
|
|
||||||
Uses existing `internal/cmd/updater` package. Binary-safe replacement with platform-specific watcher process, SemVer channel selection (stable/beta/alpha/dev), automatic rollback on failure.
|
|
||||||
|
|
||||||
**Integration:**
|
|
||||||
- Headless: `CheckAndUpdateOnStartup` -- auto-apply + restart
|
|
||||||
- Desktop: `CheckOnStartup` -- notify via tray, user confirms
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Training Data (Journal)
|
|
||||||
|
|
||||||
JSONL format per issue-epic step 10. One record per completed action.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"ts": "2026-02-05T12:00:00Z",
|
|
||||||
"epic": 299,
|
|
||||||
"child": 212,
|
|
||||||
"pr": 316,
|
|
||||||
"repo": "host-uk/core",
|
|
||||||
"action": "publish_draft",
|
|
||||||
"signals": {
|
|
||||||
"pr_state": "OPEN",
|
|
||||||
"is_draft": true,
|
|
||||||
"check_status": "SUCCESS",
|
|
||||||
"mergeable": "UNKNOWN",
|
|
||||||
"threads_total": 0,
|
|
||||||
"threads_resolved": 0
|
|
||||||
},
|
|
||||||
"result": {
|
|
||||||
"success": true,
|
|
||||||
"duration_ms": 340
|
|
||||||
},
|
|
||||||
"cycle": 1
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rules:**
|
|
||||||
- NO content (no comments, no messages, no bodies)
|
|
||||||
- Structural signals only -- safe for training
|
|
||||||
- Append-only JSONL file per node
|
|
||||||
- File path: `~/.core/journal/<repo>/<date>.jsonl`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files Summary
|
|
||||||
|
|
||||||
| File | Action |
|
|
||||||
|------|--------|
|
|
||||||
| `pkg/jobrunner/types.go` | CREATE -- JobSource, JobHandler, PipelineSignal, ActionResult interfaces |
|
|
||||||
| `pkg/jobrunner/poller.go` | CREATE -- Poller, Dispatcher, multi-source orchestration |
|
|
||||||
| `pkg/jobrunner/journal.go` | CREATE -- JSONL writer, append-only, structured records |
|
|
||||||
| `pkg/jobrunner/github/source.go` | CREATE -- GitHubSource adapter, conditional requests |
|
|
||||||
| `pkg/jobrunner/github/signals.go` | CREATE -- PR/issue state extraction, signal building |
|
|
||||||
| `internal/core-ide/handlers/publish_draft.go` | CREATE -- Publish draft PR handler |
|
|
||||||
| `internal/core-ide/handlers/resolve_threads.go` | CREATE -- Resolve review threads handler |
|
|
||||||
| `internal/core-ide/handlers/send_fix_command.go` | CREATE -- Send fix command handler |
|
|
||||||
| `internal/core-ide/handlers/enable_auto_merge.go` | CREATE -- Enable auto-merge handler |
|
|
||||||
| `internal/core-ide/handlers/tick_parent.go` | CREATE -- Tick epic checklist handler |
|
|
||||||
| `internal/core-ide/handlers/close_child.go` | CREATE -- Close child issue handler |
|
|
||||||
| `internal/core-ide/main.go` | MODIFY -- Headless/desktop branching, poller integration |
|
|
||||||
| `internal/core-ide/mcp_bridge.go` | MODIFY -- Register job handlers as MCP tools |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What Doesn't Ship Yet
|
|
||||||
|
|
||||||
- HostUK Agentic API adapter (future -- replaces GitHub)
|
|
||||||
- Hyperswarm P2P adapter (future)
|
|
||||||
- External project scanning / harvest mode (future -- WailsApp first)
|
|
||||||
- LoRA training pipeline (separate concern -- reads JSONL journal)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Testing Strategy
|
|
||||||
|
|
||||||
- **Handlers:** Unit-testable. Mock PipelineSignal in, assert API calls out.
|
|
||||||
- **Poller:** httptest server returning fixture responses.
|
|
||||||
- **Journal:** Read back JSONL, verify schema.
|
|
||||||
- **Integration:** Dry-run mode against real repos, verify signals match expected state.
|
|
||||||
|
|
@ -1,851 +0,0 @@
|
||||||
# MCP Integration Implementation Plan
|
|
||||||
|
|
||||||
> **Status:** Completed. MCP command now lives in `go-ai/cmd/mcpcmd/`. Code examples below use the old `init()` + `RegisterCommands()` pattern — the current approach uses `cli.WithCommands()` (see cli-meta-package-design.md).
|
|
||||||
|
|
||||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
||||||
|
|
||||||
**Goal:** Add `core mcp serve` command with RAG and metrics tools, then configure the agentic-flows plugin to use it.
|
|
||||||
|
|
||||||
**Architecture:** Create a new `mcp` command package that starts the pkg/mcp server with extended tools. RAG tools call the existing exported functions in internal/cmd/rag. Metrics tools call pkg/ai directly. The agentic-flows plugin gets a `.mcp.json` that spawns `core mcp serve`.
|
|
||||||
|
|
||||||
**Tech Stack:** Go 1.25, github.com/modelcontextprotocol/go-sdk/mcp, pkg/rag, pkg/ai
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 1: Add RAG tools to pkg/mcp
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `pkg/mcp/tools_rag.go`
|
|
||||||
- Modify: `pkg/mcp/mcp.go:99-101` (registerTools)
|
|
||||||
- Test: `pkg/mcp/tools_rag_test.go`
|
|
||||||
|
|
||||||
**Step 1: Write the failing test**
|
|
||||||
|
|
||||||
Create `pkg/mcp/tools_rag_test.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
package mcp
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestRAGQueryTool_Good(t *testing.T) {
|
|
||||||
// This test verifies the tool is registered and callable.
|
|
||||||
// It doesn't require Qdrant/Ollama running - just checks structure.
|
|
||||||
s, err := New(WithWorkspaceRoot(""))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("New() error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check that rag_query tool is registered
|
|
||||||
tools := s.Server().ListTools()
|
|
||||||
found := false
|
|
||||||
for _, tool := range tools {
|
|
||||||
if tool.Name == "rag_query" {
|
|
||||||
found = true
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if !found {
|
|
||||||
t.Error("rag_query tool not registered")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRAGQueryInput_Good(t *testing.T) {
|
|
||||||
input := RAGQueryInput{
|
|
||||||
Question: "how do I deploy?",
|
|
||||||
Collection: "hostuk-docs",
|
|
||||||
TopK: 5,
|
|
||||||
}
|
|
||||||
if input.Question == "" {
|
|
||||||
t.Error("Question should not be empty")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Run test to verify it fails**
|
|
||||||
|
|
||||||
Run: `go test -run TestRAGQueryTool ./pkg/mcp/... -v`
|
|
||||||
Expected: FAIL with "rag_query tool not registered"
|
|
||||||
|
|
||||||
**Step 3: Create tools_rag.go with types and tool registration**
|
|
||||||
|
|
||||||
Create `pkg/mcp/tools_rag.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
package mcp
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
|
|
||||||
ragcmd "forge.lthn.ai/core/cli/internal/cmd/rag"
|
|
||||||
"forge.lthn.ai/core/cli/pkg/rag"
|
|
||||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
|
||||||
)
|
|
||||||
|
|
||||||
// RAG tool input/output types
|
|
||||||
|
|
||||||
// RAGQueryInput contains parameters for querying the vector database.
|
|
||||||
type RAGQueryInput struct {
|
|
||||||
Question string `json:"question"`
|
|
||||||
Collection string `json:"collection,omitempty"`
|
|
||||||
TopK int `json:"top_k,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// RAGQueryOutput contains the query results.
|
|
||||||
type RAGQueryOutput struct {
|
|
||||||
Results []RAGResult `json:"results"`
|
|
||||||
Context string `json:"context"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// RAGResult represents a single search result.
|
|
||||||
type RAGResult struct {
|
|
||||||
Content string `json:"content"`
|
|
||||||
Score float32 `json:"score"`
|
|
||||||
Source string `json:"source"`
|
|
||||||
Metadata map[string]string `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// RAGIngestInput contains parameters for ingesting documents.
|
|
||||||
type RAGIngestInput struct {
|
|
||||||
Path string `json:"path"`
|
|
||||||
Collection string `json:"collection,omitempty"`
|
|
||||||
Recreate bool `json:"recreate,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// RAGIngestOutput contains the ingestion results.
|
|
||||||
type RAGIngestOutput struct {
|
|
||||||
Success bool `json:"success"`
|
|
||||||
Path string `json:"path"`
|
|
||||||
Chunks int `json:"chunks"`
|
|
||||||
Message string `json:"message,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// RAGCollectionsInput contains parameters for listing collections.
|
|
||||||
type RAGCollectionsInput struct {
|
|
||||||
ShowStats bool `json:"show_stats,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// RAGCollectionsOutput contains the list of collections.
|
|
||||||
type RAGCollectionsOutput struct {
|
|
||||||
Collections []CollectionInfo `json:"collections"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// CollectionInfo describes a Qdrant collection.
|
|
||||||
type CollectionInfo struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
PointsCount uint64 `json:"points_count,omitempty"`
|
|
||||||
Status string `json:"status,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// registerRAGTools adds RAG tools to the MCP server.
|
|
||||||
func (s *Service) registerRAGTools(server *mcp.Server) {
|
|
||||||
mcp.AddTool(server, &mcp.Tool{
|
|
||||||
Name: "rag_query",
|
|
||||||
Description: "Query the vector database for relevant documents using semantic search",
|
|
||||||
}, s.ragQuery)
|
|
||||||
|
|
||||||
mcp.AddTool(server, &mcp.Tool{
|
|
||||||
Name: "rag_ingest",
|
|
||||||
Description: "Ingest a file or directory into the vector database",
|
|
||||||
}, s.ragIngest)
|
|
||||||
|
|
||||||
mcp.AddTool(server, &mcp.Tool{
|
|
||||||
Name: "rag_collections",
|
|
||||||
Description: "List available vector database collections",
|
|
||||||
}, s.ragCollections)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *Service) ragQuery(ctx context.Context, req *mcp.CallToolRequest, input RAGQueryInput) (*mcp.CallToolResult, RAGQueryOutput, error) {
|
|
||||||
s.logger.Info("MCP tool execution", "tool", "rag_query", "question", input.Question)
|
|
||||||
|
|
||||||
collection := input.Collection
|
|
||||||
if collection == "" {
|
|
||||||
collection = "hostuk-docs"
|
|
||||||
}
|
|
||||||
topK := input.TopK
|
|
||||||
if topK <= 0 {
|
|
||||||
topK = 5
|
|
||||||
}
|
|
||||||
|
|
||||||
results, err := ragcmd.QueryDocs(ctx, input.Question, collection, topK)
|
|
||||||
if err != nil {
|
|
||||||
return nil, RAGQueryOutput{}, fmt.Errorf("query failed: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Convert to output format
|
|
||||||
out := RAGQueryOutput{
|
|
||||||
Results: make([]RAGResult, 0, len(results)),
|
|
||||||
Context: rag.FormatResultsContext(results),
|
|
||||||
}
|
|
||||||
for _, r := range results {
|
|
||||||
out.Results = append(out.Results, RAGResult{
|
|
||||||
Content: r.Content,
|
|
||||||
Score: r.Score,
|
|
||||||
Source: r.Source,
|
|
||||||
Metadata: r.Metadata,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil, out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *Service) ragIngest(ctx context.Context, req *mcp.CallToolRequest, input RAGIngestInput) (*mcp.CallToolResult, RAGIngestOutput, error) {
|
|
||||||
s.logger.Security("MCP tool execution", "tool", "rag_ingest", "path", input.Path)
|
|
||||||
|
|
||||||
collection := input.Collection
|
|
||||||
if collection == "" {
|
|
||||||
collection = "hostuk-docs"
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check if path is a file or directory
|
|
||||||
info, err := s.medium.Stat(input.Path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, RAGIngestOutput{}, fmt.Errorf("path not found: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.IsDir() {
|
|
||||||
err = ragcmd.IngestDirectory(ctx, input.Path, collection, input.Recreate)
|
|
||||||
if err != nil {
|
|
||||||
return nil, RAGIngestOutput{}, fmt.Errorf("ingest directory failed: %w", err)
|
|
||||||
}
|
|
||||||
return nil, RAGIngestOutput{
|
|
||||||
Success: true,
|
|
||||||
Path: input.Path,
|
|
||||||
Message: fmt.Sprintf("Ingested directory into collection %s", collection),
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
chunks, err := ragcmd.IngestFile(ctx, input.Path, collection)
|
|
||||||
if err != nil {
|
|
||||||
return nil, RAGIngestOutput{}, fmt.Errorf("ingest file failed: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil, RAGIngestOutput{
|
|
||||||
Success: true,
|
|
||||||
Path: input.Path,
|
|
||||||
Chunks: chunks,
|
|
||||||
Message: fmt.Sprintf("Ingested %d chunks into collection %s", chunks, collection),
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *Service) ragCollections(ctx context.Context, req *mcp.CallToolRequest, input RAGCollectionsInput) (*mcp.CallToolResult, RAGCollectionsOutput, error) {
|
|
||||||
s.logger.Info("MCP tool execution", "tool", "rag_collections")
|
|
||||||
|
|
||||||
client, err := rag.NewQdrantClient(rag.DefaultQdrantConfig())
|
|
||||||
if err != nil {
|
|
||||||
return nil, RAGCollectionsOutput{}, fmt.Errorf("connect to Qdrant: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = client.Close() }()
|
|
||||||
|
|
||||||
names, err := client.ListCollections(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return nil, RAGCollectionsOutput{}, fmt.Errorf("list collections: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
out := RAGCollectionsOutput{
|
|
||||||
Collections: make([]CollectionInfo, 0, len(names)),
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, name := range names {
|
|
||||||
info := CollectionInfo{Name: name}
|
|
||||||
if input.ShowStats {
|
|
||||||
cinfo, err := client.CollectionInfo(ctx, name)
|
|
||||||
if err == nil {
|
|
||||||
info.PointsCount = cinfo.PointsCount
|
|
||||||
info.Status = cinfo.Status.String()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
out.Collections = append(out.Collections, info)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil, out, nil
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 4: Update mcp.go to call registerRAGTools**
|
|
||||||
|
|
||||||
In `pkg/mcp/mcp.go`, modify the `registerTools` function (around line 104) to add:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (s *Service) registerTools(server *mcp.Server) {
|
|
||||||
// File operations (existing)
|
|
||||||
// ... existing code ...
|
|
||||||
|
|
||||||
// RAG operations
|
|
||||||
s.registerRAGTools(server)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 5: Run test to verify it passes**
|
|
||||||
|
|
||||||
Run: `go test -run TestRAGQuery ./pkg/mcp/... -v`
|
|
||||||
Expected: PASS
|
|
||||||
|
|
||||||
**Step 6: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add pkg/mcp/tools_rag.go pkg/mcp/tools_rag_test.go pkg/mcp/mcp.go
|
|
||||||
git commit -m "feat(mcp): add RAG tools (query, ingest, collections)"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 2: Add metrics tools to pkg/mcp
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `pkg/mcp/tools_metrics.go`
|
|
||||||
- Modify: `pkg/mcp/mcp.go` (registerTools)
|
|
||||||
- Test: `pkg/mcp/tools_metrics_test.go`
|
|
||||||
|
|
||||||
**Step 1: Write the failing test**
|
|
||||||
|
|
||||||
Create `pkg/mcp/tools_metrics_test.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
package mcp
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestMetricsRecordTool_Good(t *testing.T) {
|
|
||||||
s, err := New(WithWorkspaceRoot(""))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("New() error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
tools := s.Server().ListTools()
|
|
||||||
found := false
|
|
||||||
for _, tool := range tools {
|
|
||||||
if tool.Name == "metrics_record" {
|
|
||||||
found = true
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if !found {
|
|
||||||
t.Error("metrics_record tool not registered")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMetricsQueryTool_Good(t *testing.T) {
|
|
||||||
s, err := New(WithWorkspaceRoot(""))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("New() error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
tools := s.Server().ListTools()
|
|
||||||
found := false
|
|
||||||
for _, tool := range tools {
|
|
||||||
if tool.Name == "metrics_query" {
|
|
||||||
found = true
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if !found {
|
|
||||||
t.Error("metrics_query tool not registered")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Run test to verify it fails**
|
|
||||||
|
|
||||||
Run: `go test -run TestMetrics ./pkg/mcp/... -v`
|
|
||||||
Expected: FAIL
|
|
||||||
|
|
||||||
**Step 3: Create tools_metrics.go**
|
|
||||||
|
|
||||||
Create `pkg/mcp/tools_metrics.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
package mcp
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"forge.lthn.ai/core/cli/pkg/ai"
|
|
||||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Metrics tool input/output types
|
|
||||||
|
|
||||||
// MetricsRecordInput contains parameters for recording a metric event.
|
|
||||||
type MetricsRecordInput struct {
|
|
||||||
Type string `json:"type"`
|
|
||||||
AgentID string `json:"agent_id,omitempty"`
|
|
||||||
Repo string `json:"repo,omitempty"`
|
|
||||||
Data map[string]any `json:"data,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// MetricsRecordOutput contains the result of recording.
|
|
||||||
type MetricsRecordOutput struct {
|
|
||||||
Success bool `json:"success"`
|
|
||||||
Timestamp time.Time `json:"timestamp"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// MetricsQueryInput contains parameters for querying metrics.
|
|
||||||
type MetricsQueryInput struct {
|
|
||||||
Since string `json:"since,omitempty"` // e.g., "7d", "24h"
|
|
||||||
}
|
|
||||||
|
|
||||||
// MetricsQueryOutput contains the query results.
|
|
||||||
type MetricsQueryOutput struct {
|
|
||||||
Total int `json:"total"`
|
|
||||||
ByType []MetricCount `json:"by_type"`
|
|
||||||
ByRepo []MetricCount `json:"by_repo"`
|
|
||||||
ByAgent []MetricCount `json:"by_agent"`
|
|
||||||
Events []MetricEventBrief `json:"events,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// MetricCount represents a count by key.
|
|
||||||
type MetricCount struct {
|
|
||||||
Key string `json:"key"`
|
|
||||||
Count int `json:"count"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// MetricEventBrief is a simplified event for output.
|
|
||||||
type MetricEventBrief struct {
|
|
||||||
Type string `json:"type"`
|
|
||||||
Timestamp time.Time `json:"timestamp"`
|
|
||||||
AgentID string `json:"agent_id,omitempty"`
|
|
||||||
Repo string `json:"repo,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// registerMetricsTools adds metrics tools to the MCP server.
|
|
||||||
func (s *Service) registerMetricsTools(server *mcp.Server) {
|
|
||||||
mcp.AddTool(server, &mcp.Tool{
|
|
||||||
Name: "metrics_record",
|
|
||||||
Description: "Record a metric event (AI task, security scan, job creation, etc.)",
|
|
||||||
}, s.metricsRecord)
|
|
||||||
|
|
||||||
mcp.AddTool(server, &mcp.Tool{
|
|
||||||
Name: "metrics_query",
|
|
||||||
Description: "Query recorded metrics with aggregation by type, repo, and agent",
|
|
||||||
}, s.metricsQuery)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *Service) metricsRecord(ctx context.Context, req *mcp.CallToolRequest, input MetricsRecordInput) (*mcp.CallToolResult, MetricsRecordOutput, error) {
|
|
||||||
s.logger.Info("MCP tool execution", "tool", "metrics_record", "type", input.Type)
|
|
||||||
|
|
||||||
if input.Type == "" {
|
|
||||||
return nil, MetricsRecordOutput{}, fmt.Errorf("type is required")
|
|
||||||
}
|
|
||||||
|
|
||||||
event := ai.Event{
|
|
||||||
Type: input.Type,
|
|
||||||
Timestamp: time.Now(),
|
|
||||||
AgentID: input.AgentID,
|
|
||||||
Repo: input.Repo,
|
|
||||||
Data: input.Data,
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := ai.Record(event); err != nil {
|
|
||||||
return nil, MetricsRecordOutput{}, fmt.Errorf("record event: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil, MetricsRecordOutput{
|
|
||||||
Success: true,
|
|
||||||
Timestamp: event.Timestamp,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *Service) metricsQuery(ctx context.Context, req *mcp.CallToolRequest, input MetricsQueryInput) (*mcp.CallToolResult, MetricsQueryOutput, error) {
|
|
||||||
s.logger.Info("MCP tool execution", "tool", "metrics_query", "since", input.Since)
|
|
||||||
|
|
||||||
since := input.Since
|
|
||||||
if since == "" {
|
|
||||||
since = "7d"
|
|
||||||
}
|
|
||||||
|
|
||||||
duration, err := parseDuration(since)
|
|
||||||
if err != nil {
|
|
||||||
return nil, MetricsQueryOutput{}, fmt.Errorf("invalid since value: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
sinceTime := time.Now().Add(-duration)
|
|
||||||
events, err := ai.ReadEvents(sinceTime)
|
|
||||||
if err != nil {
|
|
||||||
return nil, MetricsQueryOutput{}, fmt.Errorf("read events: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
summary := ai.Summary(events)
|
|
||||||
|
|
||||||
out := MetricsQueryOutput{
|
|
||||||
Total: summary["total"].(int),
|
|
||||||
}
|
|
||||||
|
|
||||||
// Convert by_type
|
|
||||||
if byType, ok := summary["by_type"].([]map[string]any); ok {
|
|
||||||
for _, entry := range byType {
|
|
||||||
out.ByType = append(out.ByType, MetricCount{
|
|
||||||
Key: entry["key"].(string),
|
|
||||||
Count: entry["count"].(int),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Convert by_repo
|
|
||||||
if byRepo, ok := summary["by_repo"].([]map[string]any); ok {
|
|
||||||
for _, entry := range byRepo {
|
|
||||||
out.ByRepo = append(out.ByRepo, MetricCount{
|
|
||||||
Key: entry["key"].(string),
|
|
||||||
Count: entry["count"].(int),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Convert by_agent
|
|
||||||
if byAgent, ok := summary["by_agent"].([]map[string]any); ok {
|
|
||||||
for _, entry := range byAgent {
|
|
||||||
out.ByAgent = append(out.ByAgent, MetricCount{
|
|
||||||
Key: entry["key"].(string),
|
|
||||||
Count: entry["count"].(int),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Include last 10 events for context
|
|
||||||
limit := 10
|
|
||||||
if len(events) < limit {
|
|
||||||
limit = len(events)
|
|
||||||
}
|
|
||||||
for i := len(events) - limit; i < len(events); i++ {
|
|
||||||
ev := events[i]
|
|
||||||
out.Events = append(out.Events, MetricEventBrief{
|
|
||||||
Type: ev.Type,
|
|
||||||
Timestamp: ev.Timestamp,
|
|
||||||
AgentID: ev.AgentID,
|
|
||||||
Repo: ev.Repo,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil, out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseDuration parses a human-friendly duration like "7d", "24h", "30d".
|
|
||||||
func parseDuration(s string) (time.Duration, error) {
|
|
||||||
if len(s) < 2 {
|
|
||||||
return 0, fmt.Errorf("invalid duration: %s", s)
|
|
||||||
}
|
|
||||||
|
|
||||||
unit := s[len(s)-1]
|
|
||||||
value := s[:len(s)-1]
|
|
||||||
|
|
||||||
var n int
|
|
||||||
if _, err := fmt.Sscanf(value, "%d", &n); err != nil {
|
|
||||||
return 0, fmt.Errorf("invalid duration: %s", s)
|
|
||||||
}
|
|
||||||
|
|
||||||
if n <= 0 {
|
|
||||||
return 0, fmt.Errorf("duration must be positive: %s", s)
|
|
||||||
}
|
|
||||||
|
|
||||||
switch unit {
|
|
||||||
case 'd':
|
|
||||||
return time.Duration(n) * 24 * time.Hour, nil
|
|
||||||
case 'h':
|
|
||||||
return time.Duration(n) * time.Hour, nil
|
|
||||||
case 'm':
|
|
||||||
return time.Duration(n) * time.Minute, nil
|
|
||||||
default:
|
|
||||||
return 0, fmt.Errorf("unknown unit %c in duration: %s", unit, s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 4: Update mcp.go to call registerMetricsTools**
|
|
||||||
|
|
||||||
In `pkg/mcp/mcp.go`, add to `registerTools`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (s *Service) registerTools(server *mcp.Server) {
|
|
||||||
// ... existing file operations ...
|
|
||||||
|
|
||||||
// RAG operations
|
|
||||||
s.registerRAGTools(server)
|
|
||||||
|
|
||||||
// Metrics operations
|
|
||||||
s.registerMetricsTools(server)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 5: Run test to verify it passes**
|
|
||||||
|
|
||||||
Run: `go test -run TestMetrics ./pkg/mcp/... -v`
|
|
||||||
Expected: PASS
|
|
||||||
|
|
||||||
**Step 6: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add pkg/mcp/tools_metrics.go pkg/mcp/tools_metrics_test.go pkg/mcp/mcp.go
|
|
||||||
git commit -m "feat(mcp): add metrics tools (record, query)"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 3: Create `core mcp serve` command
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `internal/cmd/mcpcmd/cmd_mcp.go`
|
|
||||||
- Modify: `internal/variants/full.go` (add import)
|
|
||||||
- Test: Manual test via `core mcp serve`
|
|
||||||
|
|
||||||
**Step 1: Create the mcp command package**
|
|
||||||
|
|
||||||
Create `internal/cmd/mcpcmd/cmd_mcp.go`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
package mcpcmd
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"os"
|
|
||||||
"os/signal"
|
|
||||||
"syscall"
|
|
||||||
|
|
||||||
"forge.lthn.ai/core/cli/pkg/cli"
|
|
||||||
"forge.lthn.ai/core/cli/pkg/i18n"
|
|
||||||
"forge.lthn.ai/core/cli/pkg/mcp"
|
|
||||||
)
|
|
||||||
|
|
||||||
func init() {
|
|
||||||
cli.RegisterCommands(AddMCPCommands)
|
|
||||||
}
|
|
||||||
|
|
||||||
var (
|
|
||||||
mcpWorkspace string
|
|
||||||
)
|
|
||||||
|
|
||||||
var mcpCmd = &cli.Command{
|
|
||||||
Use: "mcp",
|
|
||||||
Short: i18n.T("cmd.mcp.short"),
|
|
||||||
Long: i18n.T("cmd.mcp.long"),
|
|
||||||
}
|
|
||||||
|
|
||||||
var serveCmd = &cli.Command{
|
|
||||||
Use: "serve",
|
|
||||||
Short: i18n.T("cmd.mcp.serve.short"),
|
|
||||||
Long: i18n.T("cmd.mcp.serve.long"),
|
|
||||||
RunE: func(cmd *cli.Command, args []string) error {
|
|
||||||
return runServe()
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
func AddMCPCommands(root *cli.Command) {
|
|
||||||
initMCPFlags()
|
|
||||||
mcpCmd.AddCommand(serveCmd)
|
|
||||||
root.AddCommand(mcpCmd)
|
|
||||||
}
|
|
||||||
|
|
||||||
func initMCPFlags() {
|
|
||||||
serveCmd.Flags().StringVar(&mcpWorkspace, "workspace", "", i18n.T("cmd.mcp.serve.flag.workspace"))
|
|
||||||
}
|
|
||||||
|
|
||||||
func runServe() error {
|
|
||||||
opts := []mcp.Option{}
|
|
||||||
|
|
||||||
if mcpWorkspace != "" {
|
|
||||||
opts = append(opts, mcp.WithWorkspaceRoot(mcpWorkspace))
|
|
||||||
} else {
|
|
||||||
// Default to unrestricted for MCP server
|
|
||||||
opts = append(opts, mcp.WithWorkspaceRoot(""))
|
|
||||||
}
|
|
||||||
|
|
||||||
svc, err := mcp.New(opts...)
|
|
||||||
if err != nil {
|
|
||||||
return cli.Wrap(err, "create MCP service")
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
// Handle shutdown signals
|
|
||||||
sigCh := make(chan os.Signal, 1)
|
|
||||||
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
|
|
||||||
go func() {
|
|
||||||
<-sigCh
|
|
||||||
cancel()
|
|
||||||
}()
|
|
||||||
|
|
||||||
return svc.Run(ctx)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Add i18n strings**
|
|
||||||
|
|
||||||
Create or update `pkg/i18n/en.yaml` (if it exists) or add to the existing i18n mechanism:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
cmd.mcp.short: "MCP (Model Context Protocol) server"
|
|
||||||
cmd.mcp.long: "Start an MCP server for Claude Code integration with file, RAG, and metrics tools."
|
|
||||||
cmd.mcp.serve.short: "Start the MCP server"
|
|
||||||
cmd.mcp.serve.long: "Start the MCP server in stdio mode. Use MCP_ADDR env var for TCP mode."
|
|
||||||
cmd.mcp.serve.flag.workspace: "Restrict file operations to this directory (empty = unrestricted)"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 3: Add import to full.go**
|
|
||||||
|
|
||||||
Modify `internal/variants/full.go` to add:
|
|
||||||
|
|
||||||
```go
|
|
||||||
import (
|
|
||||||
// ... existing imports ...
|
|
||||||
_ "forge.lthn.ai/core/cli/internal/cmd/mcpcmd"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 4: Build and test**
|
|
||||||
|
|
||||||
Run: `go build && ./core mcp serve --help`
|
|
||||||
Expected: Help output showing the serve command
|
|
||||||
|
|
||||||
**Step 5: Test MCP server manually**
|
|
||||||
|
|
||||||
Run: `echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | ./core mcp serve`
|
|
||||||
Expected: JSON response listing all tools including rag_query, metrics_record, etc.
|
|
||||||
|
|
||||||
**Step 6: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add internal/cmd/mcpcmd/cmd_mcp.go internal/variants/full.go
|
|
||||||
git commit -m "feat: add 'core mcp serve' command"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 4: Configure agentic-flows plugin with .mcp.json
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`
|
|
||||||
- Modify: `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.claude-plugin/plugin.json` (optional, add mcpServers)
|
|
||||||
|
|
||||||
**Step 1: Create .mcp.json**
|
|
||||||
|
|
||||||
Create `/home/shared/hostuk/claude-plugins/plugins/agentic-flows/.mcp.json`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"core-cli": {
|
|
||||||
"command": "core",
|
|
||||||
"args": ["mcp", "serve"],
|
|
||||||
"env": {
|
|
||||||
"MCP_WORKSPACE": ""
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Verify plugin loads**
|
|
||||||
|
|
||||||
Restart Claude Code and run `/mcp` to verify the core-cli server appears.
|
|
||||||
|
|
||||||
**Step 3: Test MCP tools**
|
|
||||||
|
|
||||||
Test that tools are available:
|
|
||||||
- `mcp__plugin_agentic-flows_core-cli__rag_query`
|
|
||||||
- `mcp__plugin_agentic-flows_core-cli__rag_ingest`
|
|
||||||
- `mcp__plugin_agentic-flows_core-cli__rag_collections`
|
|
||||||
- `mcp__plugin_agentic-flows_core-cli__metrics_record`
|
|
||||||
- `mcp__plugin_agentic-flows_core-cli__metrics_query`
|
|
||||||
- `mcp__plugin_agentic-flows_core-cli__file_read`
|
|
||||||
- etc.
|
|
||||||
|
|
||||||
**Step 4: Commit plugin changes**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/shared/hostuk/claude-plugins
|
|
||||||
git add plugins/agentic-flows/.mcp.json
|
|
||||||
git commit -m "feat(agentic-flows): add MCP server configuration for core-cli"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Task 5: Update documentation
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `/home/claude/.claude/projects/-home-claude/memory/MEMORY.md`
|
|
||||||
- Modify: `/home/claude/.claude/projects/-home-claude/memory/plugin-dev-notes.md`
|
|
||||||
|
|
||||||
**Step 1: Update MEMORY.md**
|
|
||||||
|
|
||||||
Add under "Core CLI MCP Server" section:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
### Core CLI MCP Server
|
|
||||||
- **Command:** `core mcp serve` (stdio mode) or `MCP_ADDR=:9000 core mcp serve` (TCP)
|
|
||||||
- **Tools available:**
|
|
||||||
- File ops: file_read, file_write, file_edit, file_delete, file_rename, file_exists, dir_list, dir_create
|
|
||||||
- RAG: rag_query, rag_ingest, rag_collections
|
|
||||||
- Metrics: metrics_record, metrics_query
|
|
||||||
- Language: lang_detect, lang_list
|
|
||||||
- **Plugin config:** `plugins/agentic-flows/.mcp.json`
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 2: Update plugin-dev-notes.md**
|
|
||||||
|
|
||||||
Add section:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
## MCP Server (core mcp serve)
|
|
||||||
|
|
||||||
### Available Tools
|
|
||||||
| Tool | Description |
|
|
||||||
|------|-------------|
|
|
||||||
| file_read | Read file contents |
|
|
||||||
| file_write | Write file contents |
|
|
||||||
| file_edit | Edit file (replace string) |
|
|
||||||
| file_delete | Delete file |
|
|
||||||
| file_rename | Rename/move file |
|
|
||||||
| file_exists | Check if file exists |
|
|
||||||
| dir_list | List directory contents |
|
|
||||||
| dir_create | Create directory |
|
|
||||||
| rag_query | Query vector DB |
|
|
||||||
| rag_ingest | Ingest file/directory |
|
|
||||||
| rag_collections | List collections |
|
|
||||||
| metrics_record | Record event |
|
|
||||||
| metrics_query | Query events |
|
|
||||||
| lang_detect | Detect file language |
|
|
||||||
| lang_list | List supported languages |
|
|
||||||
|
|
||||||
### Example .mcp.json
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"core-cli": {
|
|
||||||
"command": "core",
|
|
||||||
"args": ["mcp", "serve"]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
```
|
|
||||||
|
|
||||||
**Step 3: Commit documentation**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add ~/.claude/projects/-home-claude/memory/*.md
|
|
||||||
git commit -m "docs: update memory with MCP server tools"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
| Task | Files | Purpose |
|
|
||||||
|------|-------|---------|
|
|
||||||
| 1 | `pkg/mcp/tools_rag.go` | RAG tools (query, ingest, collections) |
|
|
||||||
| 2 | `pkg/mcp/tools_metrics.go` | Metrics tools (record, query) |
|
|
||||||
| 3 | `internal/cmd/mcpcmd/cmd_mcp.go` | `core mcp serve` command |
|
|
||||||
| 4 | `plugins/agentic-flows/.mcp.json` | Plugin MCP configuration |
|
|
||||||
| 5 | Memory docs | Documentation updates |
|
|
||||||
|
|
||||||
## Services Required
|
|
||||||
|
|
||||||
- **Qdrant:** localhost:6333 (verified running)
|
|
||||||
- **Ollama:** localhost:11434 with nomic-embed-text (verified running)
|
|
||||||
- **InfluxDB:** localhost:8086 (optional, for future time-series metrics)
|
|
||||||
|
|
@ -1,150 +0,0 @@
|
||||||
# BugSETI HubService Design
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
A thin HTTP client service in the BugSETI desktop app that coordinates with the agentic portal's `/api/bugseti/*` endpoints. Prevents duplicate work across the 11 community testers, aggregates stats for leaderboard, and registers client instances.
|
|
||||||
|
|
||||||
## Decisions
|
|
||||||
|
|
||||||
| Decision | Choice | Rationale |
|
|
||||||
|----------|--------|-----------|
|
|
||||||
| Target | Direct to portal API | Endpoints built for this purpose |
|
|
||||||
| Auth | Auto-register via forge token | No manual key management for users |
|
|
||||||
| Sync strategy | Lazy/manual | User-triggered claims, manual stats sync |
|
|
||||||
| Offline mode | Offline-first | Queue failed writes, retry on reconnect |
|
|
||||||
| Approach | Thin HTTP client (net/http) | Matches existing patterns, no deps |
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
**File:** `internal/bugseti/hub.go` + `hub_test.go`
|
|
||||||
|
|
||||||
```
|
|
||||||
HubService
|
|
||||||
├── HTTP client (net/http, 10s timeout)
|
|
||||||
├── Auth: auto-register via forge token → cached ak_ token
|
|
||||||
├── Config: HubURL, HubToken, ClientID in ConfigService
|
|
||||||
├── Offline-first: queue failed writes, drain on next success
|
|
||||||
└── Lazy sync: user-triggered, no background goroutines
|
|
||||||
```
|
|
||||||
|
|
||||||
**Dependencies:** ConfigService only.
|
|
||||||
|
|
||||||
**Integration:**
|
|
||||||
- QueueService calls `hub.ClaimIssue()` when user picks an issue
|
|
||||||
- SubmitService calls `hub.UpdateStatus("completed")` after PR
|
|
||||||
- TrayService calls `hub.GetLeaderboard()` from UI
|
|
||||||
- main.go calls `hub.Register()` on startup
|
|
||||||
|
|
||||||
## Data Types
|
|
||||||
|
|
||||||
```go
|
|
||||||
type HubClient struct {
|
|
||||||
ClientID string // UUID, generated once, persisted in config
|
|
||||||
Name string // e.g. "Snider's MacBook"
|
|
||||||
Version string // bugseti.GetVersion()
|
|
||||||
OS string // runtime.GOOS
|
|
||||||
Arch string // runtime.GOARCH
|
|
||||||
}
|
|
||||||
|
|
||||||
type HubClaim struct {
|
|
||||||
IssueID string // "owner/repo#123"
|
|
||||||
Repo string
|
|
||||||
IssueNumber int
|
|
||||||
Title string
|
|
||||||
URL string
|
|
||||||
Status string // claimed|in_progress|completed|skipped
|
|
||||||
ClaimedAt time.Time
|
|
||||||
PRUrl string
|
|
||||||
PRNumber int
|
|
||||||
}
|
|
||||||
|
|
||||||
type LeaderboardEntry struct {
|
|
||||||
Rank int
|
|
||||||
ClientName string
|
|
||||||
IssuesCompleted int
|
|
||||||
PRsSubmitted int
|
|
||||||
PRsMerged int
|
|
||||||
CurrentStreak int
|
|
||||||
}
|
|
||||||
|
|
||||||
type GlobalStats struct {
|
|
||||||
TotalParticipants int
|
|
||||||
ActiveParticipants int
|
|
||||||
TotalIssuesCompleted int
|
|
||||||
TotalPRsMerged int
|
|
||||||
ActiveClaims int
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## API Mapping
|
|
||||||
|
|
||||||
| Method | HTTP | Endpoint | Trigger |
|
|
||||||
|--------|------|----------|---------|
|
|
||||||
| `Register()` | POST /register | App startup |
|
|
||||||
| `Heartbeat()` | POST /heartbeat | Manual / periodic if enabled |
|
|
||||||
| `ClaimIssue(issue)` | POST /issues/claim | User picks issue |
|
|
||||||
| `UpdateStatus(id, status)` | PATCH /issues/{id}/status | PR submitted, skip |
|
|
||||||
| `ReleaseClaim(id)` | DELETE /issues/{id}/claim | User abandons |
|
|
||||||
| `IsIssueClaimed(id)` | GET /issues/{id} | Before showing issue |
|
|
||||||
| `ListClaims(filters)` | GET /issues/claimed | UI active claims view |
|
|
||||||
| `SyncStats(stats)` | POST /stats/sync | Manual from UI |
|
|
||||||
| `GetLeaderboard(limit)` | GET /leaderboard | UI leaderboard view |
|
|
||||||
| `GetGlobalStats()` | GET /stats | UI stats dashboard |
|
|
||||||
|
|
||||||
## Auto-Register Flow
|
|
||||||
|
|
||||||
New endpoint on portal:
|
|
||||||
|
|
||||||
```
|
|
||||||
POST /api/bugseti/auth/forge
|
|
||||||
Body: { "forge_url": "https://forge.lthn.io", "forge_token": "..." }
|
|
||||||
```
|
|
||||||
|
|
||||||
Portal validates token against Forgejo API (`/api/v1/user`), creates an AgentApiKey with `bugseti.read` + `bugseti.write` scopes, returns `{ "api_key": "ak_..." }`.
|
|
||||||
|
|
||||||
HubService caches the `ak_` token in config.json. On 401, clears cached token and re-registers.
|
|
||||||
|
|
||||||
## Error Handling
|
|
||||||
|
|
||||||
| Error | Behaviour |
|
|
||||||
|-------|-----------|
|
|
||||||
| Network unreachable | Log, queue write ops, return cached reads |
|
|
||||||
| 401 Unauthorised | Clear token, re-register via forge |
|
|
||||||
| 409 Conflict (claim) | Return "already claimed" — not an error |
|
|
||||||
| 404 (claim not found) | Return nil |
|
|
||||||
| 429 Rate limited | Back off, queue the op |
|
|
||||||
| 5xx Server error | Log, queue write ops |
|
|
||||||
|
|
||||||
**Pending operations queue:**
|
|
||||||
- Failed writes stored in `[]PendingOp`, persisted to `$DataDir/hub_pending.json`
|
|
||||||
- Drained on next successful user-triggered call (no background goroutine)
|
|
||||||
- Each op has: method, path, body, created_at
|
|
||||||
|
|
||||||
## Config Changes
|
|
||||||
|
|
||||||
New fields in `Config` struct:
|
|
||||||
|
|
||||||
```go
|
|
||||||
HubURL string `json:"hubUrl,omitempty"` // portal API base URL
|
|
||||||
HubToken string `json:"hubToken,omitempty"` // cached ak_ token
|
|
||||||
ClientID string `json:"clientId,omitempty"` // UUID, generated once
|
|
||||||
ClientName string `json:"clientName,omitempty"` // display name
|
|
||||||
```
|
|
||||||
|
|
||||||
## Files Changed
|
|
||||||
|
|
||||||
| File | Action |
|
|
||||||
|------|--------|
|
|
||||||
| `internal/bugseti/hub.go` | New — HubService |
|
|
||||||
| `internal/bugseti/hub_test.go` | New — httptest-based tests |
|
|
||||||
| `internal/bugseti/config.go` | Edit — add Hub* + ClientID fields |
|
|
||||||
| `cmd/bugseti/main.go` | Edit — create + register HubService |
|
|
||||||
| `cmd/bugseti/tray.go` | Edit — leaderboard/stats menu items |
|
|
||||||
| Laravel: auth controller | New — `/api/bugseti/auth/forge` |
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
- `httptest.NewServer` mocks for all endpoints
|
|
||||||
- Test success, network error, 409 conflict, 401 re-auth flows
|
|
||||||
- Test pending ops queue: add when offline, drain on reconnect
|
|
||||||
- `_Good`, `_Bad`, `_Ugly` naming convention
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,82 +0,0 @@
|
||||||
# LEM Chat — Web Components Design
|
|
||||||
|
|
||||||
**Date**: 2026-02-17
|
|
||||||
**Status**: Approved
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Standalone chat UI built with vanilla Web Components (Custom Elements + Shadow DOM). Connects to the MLX inference server's OpenAI-compatible SSE streaming endpoint. Zero framework dependencies. Single JS file output, embeddable anywhere.
|
|
||||||
|
|
||||||
## Components
|
|
||||||
|
|
||||||
| Element | Purpose |
|
|
||||||
|---------|---------|
|
|
||||||
| `<lem-chat>` | Container. Conversation state, SSE connection, config via attributes |
|
|
||||||
| `<lem-messages>` | Scrollable message list with auto-scroll anchoring |
|
|
||||||
| `<lem-message>` | Single message bubble. Streams tokens for assistant messages |
|
|
||||||
| `<lem-input>` | Text input, Enter to send, Shift+Enter for newline |
|
|
||||||
|
|
||||||
## Data Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
User types in <lem-input>
|
|
||||||
→ dispatches 'lem-send' CustomEvent
|
|
||||||
→ <lem-chat> catches it
|
|
||||||
→ adds user message to <lem-messages>
|
|
||||||
→ POST /v1/chat/completions {stream: true, messages: [...history]}
|
|
||||||
→ reads SSE chunks via fetch + ReadableStream
|
|
||||||
→ appends tokens to streaming <lem-message>
|
|
||||||
→ on [DONE], finalises message
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
```html
|
|
||||||
<lem-chat endpoint="http://localhost:8090" model="qwen3-8b"></lem-chat>
|
|
||||||
```
|
|
||||||
|
|
||||||
Attributes: `endpoint`, `model`, `system-prompt`, `max-tokens`, `temperature`
|
|
||||||
|
|
||||||
## Theming
|
|
||||||
|
|
||||||
Shadow DOM with CSS custom properties:
|
|
||||||
|
|
||||||
```css
|
|
||||||
--lem-bg: #1a1a1e;
|
|
||||||
--lem-msg-user: #2a2a3e;
|
|
||||||
--lem-msg-assistant: #1e1e2a;
|
|
||||||
--lem-accent: #5865f2;
|
|
||||||
--lem-text: #e0e0e0;
|
|
||||||
--lem-font: system-ui;
|
|
||||||
```
|
|
||||||
|
|
||||||
## Markdown
|
|
||||||
|
|
||||||
Minimal inline parsing: fenced code blocks, inline code, bold, italic. No library.
|
|
||||||
|
|
||||||
## File Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
lem-chat/
|
|
||||||
├── index.html # Demo page
|
|
||||||
├── src/
|
|
||||||
│ ├── lem-chat.ts # Main container + SSE client
|
|
||||||
│ ├── lem-messages.ts # Message list with scroll anchoring
|
|
||||||
│ ├── lem-message.ts # Single message with streaming
|
|
||||||
│ ├── lem-input.ts # Text input
|
|
||||||
│ ├── markdown.ts # Minimal markdown → HTML
|
|
||||||
│ └── styles.ts # CSS template literals
|
|
||||||
├── package.json # typescript + esbuild
|
|
||||||
└── tsconfig.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Build: `esbuild src/lem-chat.ts --bundle --outfile=dist/lem-chat.js`
|
|
||||||
|
|
||||||
## Not in v1
|
|
||||||
|
|
||||||
- Model selection UI
|
|
||||||
- Conversation persistence
|
|
||||||
- File/image upload
|
|
||||||
- Syntax highlighting
|
|
||||||
- Typing indicators
|
|
||||||
- User avatars
|
|
||||||
|
|
@ -1,657 +0,0 @@
|
||||||
# go-api Design — HTTP Gateway + OpenAPI SDK Generation
|
|
||||||
|
|
||||||
**Date:** 2026-02-20
|
|
||||||
**Author:** Virgil
|
|
||||||
**Status:** Phase 1 + Phase 2 + Phase 3 Complete (176 tests in go-api)
|
|
||||||
**Module:** `forge.lthn.ai/core/go-api`
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
The Core Go ecosystem exposes 42+ tools via MCP (JSON-RPC), which is ideal for AI agents but inaccessible to regular HTTP clients, frontend applications, and third-party integrators. There is no unified HTTP gateway, no OpenAPI specification, and no generated SDKs.
|
|
||||||
|
|
||||||
Both external customers (Host UK products) and Lethean network peers need programmatic access to the same services. The gateway also serves web routes, static assets, and streaming endpoints — not just REST APIs.
|
|
||||||
|
|
||||||
## Solution
|
|
||||||
|
|
||||||
A `go-api` package that acts as the central HTTP gateway:
|
|
||||||
|
|
||||||
1. **Gin-based HTTP gateway** with extensible middleware via gin-contrib plugins
|
|
||||||
2. **RouteGroup interface** that subsystems implement to register their own endpoints (API, web, or both)
|
|
||||||
3. **WebSocket + SSE integration** for real-time streaming
|
|
||||||
4. **OpenAPI 3.1 spec generation** via runtime SpecBuilder (not swaggo annotations)
|
|
||||||
5. **SDK generation pipeline** targeting 11 languages via openapi-generator-cli
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### Four-Protocol Access
|
|
||||||
|
|
||||||
Same backend services, four client protocols:
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─── REST (go-api) POST /v1/ml/generate → JSON
|
|
||||||
│
|
|
||||||
├─── GraphQL (gqlgen) mutation { mlGenerate(...) { response } }
|
|
||||||
Client ────────────┤
|
|
||||||
├─── WebSocket (go-ws) subscribe ml.generate → streaming
|
|
||||||
│
|
|
||||||
└─── MCP (go-ai) ml_generate → JSON-RPC
|
|
||||||
```
|
|
||||||
|
|
||||||
### Dependency Graph
|
|
||||||
|
|
||||||
```
|
|
||||||
go-api (Gin engine + middleware + OpenAPI)
|
|
||||||
↑ imported by (each registers its own routes)
|
|
||||||
├── go-ai/api/ → /v1/file/*, /v1/process/*, /v1/metrics/*
|
|
||||||
├── go-ml/api/ → /v1/ml/*
|
|
||||||
├── go-rag/api/ → /v1/rag/*
|
|
||||||
├── go-agentic/api/ → /v1/tasks/*
|
|
||||||
├── go-help/api/ → /v1/help/*
|
|
||||||
└── go-ws/api/ → /ws (WebSocket upgrade)
|
|
||||||
```
|
|
||||||
|
|
||||||
go-api has zero internal ecosystem dependencies. Subsystems import go-api, not the other way round.
|
|
||||||
|
|
||||||
### Subsystem Opt-In
|
|
||||||
|
|
||||||
Not every MCP tool becomes a REST endpoint. Each subsystem decides what to expose via a separate `RegisterAPI()` method, independent of MCP's `RegisterTools()`. A subsystem with 15 MCP tools might expose 5 REST endpoints.
|
|
||||||
|
|
||||||
## Package Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
forge.lthn.ai/core/go-api
|
|
||||||
├── api.go # Engine struct, New(), Serve(), Shutdown()
|
|
||||||
├── middleware.go # Auth, CORS, rate limiting, request logging, recovery
|
|
||||||
├── options.go # WithAddr, WithAuth, WithCORS, WithRateLimit, etc.
|
|
||||||
├── group.go # RouteGroup interface + registration
|
|
||||||
├── response.go # Envelope type, error responses, pagination
|
|
||||||
├── docs/ # Generated swagger docs (swaggo output)
|
|
||||||
├── sdk/ # SDK generation tooling / Makefile targets
|
|
||||||
└── go.mod # forge.lthn.ai/core/go-api
|
|
||||||
```
|
|
||||||
|
|
||||||
## Core Interface
|
|
||||||
|
|
||||||
```go
|
|
||||||
// RouteGroup registers API routes onto a Gin router group.
|
|
||||||
// Subsystems implement this to expose their endpoints.
|
|
||||||
type RouteGroup interface {
|
|
||||||
// Name returns the route group identifier (e.g. "ml", "rag", "tasks")
|
|
||||||
Name() string
|
|
||||||
// BasePath returns the URL prefix (e.g. "/v1/ml")
|
|
||||||
BasePath() string
|
|
||||||
// RegisterRoutes adds handlers to the provided router group
|
|
||||||
RegisterRoutes(rg *gin.RouterGroup)
|
|
||||||
}
|
|
||||||
|
|
||||||
// StreamGroup optionally declares WebSocket channels a subsystem publishes to.
|
|
||||||
type StreamGroup interface {
|
|
||||||
Channels() []string
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Subsystem Example (go-ml)
|
|
||||||
|
|
||||||
```go
|
|
||||||
// In go-ml/api/routes.go
|
|
||||||
package api
|
|
||||||
|
|
||||||
type Routes struct {
|
|
||||||
service *ml.Service
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewRoutes(svc *ml.Service) *Routes {
|
|
||||||
return &Routes{service: svc}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *Routes) Name() string { return "ml" }
|
|
||||||
func (r *Routes) BasePath() string { return "/v1/ml" }
|
|
||||||
|
|
||||||
func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) {
|
|
||||||
rg.POST("/generate", r.Generate)
|
|
||||||
rg.POST("/score", r.Score)
|
|
||||||
rg.GET("/backends", r.Backends)
|
|
||||||
rg.GET("/status", r.Status)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *Routes) Channels() []string {
|
|
||||||
return []string{"ml.generate", "ml.status"}
|
|
||||||
}
|
|
||||||
|
|
||||||
// @Summary Generate text via ML backend
|
|
||||||
// @Tags ml
|
|
||||||
// @Accept json
|
|
||||||
// @Produce json
|
|
||||||
// @Param input body MLGenerateInput true "Generation parameters"
|
|
||||||
// @Success 200 {object} Response[MLGenerateOutput]
|
|
||||||
// @Router /v1/ml/generate [post]
|
|
||||||
func (r *Routes) Generate(c *gin.Context) {
|
|
||||||
var input MLGenerateInput
|
|
||||||
if err := c.ShouldBindJSON(&input); err != nil {
|
|
||||||
c.JSON(400, api.Fail("invalid_input", err.Error()))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
result, err := r.service.Generate(c.Request.Context(), input.Backend, input.Prompt, ml.GenOpts{
|
|
||||||
Temperature: input.Temperature,
|
|
||||||
MaxTokens: input.MaxTokens,
|
|
||||||
Model: input.Model,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
c.JSON(500, api.Fail("ml.generate_failed", err.Error()))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
c.JSON(200, api.OK(MLGenerateOutput{
|
|
||||||
Response: result,
|
|
||||||
Backend: input.Backend,
|
|
||||||
Model: input.Model,
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Engine Wiring (in core CLI)
|
|
||||||
|
|
||||||
```go
|
|
||||||
engine := api.New(
|
|
||||||
api.WithAddr(":8080"),
|
|
||||||
api.WithCORS("*"),
|
|
||||||
api.WithAuth(api.BearerToken(cfg.APIKey)),
|
|
||||||
api.WithRateLimit(100, time.Minute),
|
|
||||||
api.WithWSHub(wsHub),
|
|
||||||
)
|
|
||||||
|
|
||||||
engine.Register(mlapi.NewRoutes(mlService))
|
|
||||||
engine.Register(ragapi.NewRoutes(ragService))
|
|
||||||
engine.Register(agenticapi.NewRoutes(agenticService))
|
|
||||||
|
|
||||||
engine.Serve(ctx) // Blocks until context cancelled
|
|
||||||
```
|
|
||||||
|
|
||||||
## Response Envelope
|
|
||||||
|
|
||||||
All endpoints return a consistent envelope:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Response[T any] struct {
|
|
||||||
Success bool `json:"success"`
|
|
||||||
Data T `json:"data,omitempty"`
|
|
||||||
Error *Error `json:"error,omitempty"`
|
|
||||||
Meta *Meta `json:"meta,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type Error struct {
|
|
||||||
Code string `json:"code"`
|
|
||||||
Message string `json:"message"`
|
|
||||||
Details any `json:"details,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type Meta struct {
|
|
||||||
RequestID string `json:"request_id"`
|
|
||||||
Duration string `json:"duration"`
|
|
||||||
Page int `json:"page,omitempty"`
|
|
||||||
PerPage int `json:"per_page,omitempty"`
|
|
||||||
Total int `json:"total,omitempty"`
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Helper functions:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func OK[T any](data T) Response[T]
|
|
||||||
func Fail(code, message string) Response[any]
|
|
||||||
func Paginated[T any](data T, page, perPage, total int) Response[T]
|
|
||||||
```
|
|
||||||
|
|
||||||
## Middleware Stack
|
|
||||||
|
|
||||||
```go
|
|
||||||
api.New(
|
|
||||||
api.WithAddr(":8080"),
|
|
||||||
api.WithCORS(api.CORSConfig{...}), // gin-contrib/cors
|
|
||||||
api.WithAuth(api.BearerToken("...")), // Phase 1: simple bearer token
|
|
||||||
api.WithRateLimit(100, time.Minute), // Per-IP sliding window
|
|
||||||
api.WithRequestID(), // X-Request-ID header generation
|
|
||||||
api.WithRecovery(), // Panic recovery → 500 response
|
|
||||||
api.WithLogger(slog.Default()), // Structured request logging
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
Auth evolution path: bearer token → API keys → Authentik (OIDC/forward auth). Middleware slot stays the same.
|
|
||||||
|
|
||||||
## WebSocket Integration
|
|
||||||
|
|
||||||
go-api wraps the existing go-ws Hub as a first-class transport:
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Automatic registration:
|
|
||||||
// GET /ws → WebSocket upgrade (go-ws Hub)
|
|
||||||
|
|
||||||
// Client subscribes: {"type":"subscribe","channel":"ml.generate"}
|
|
||||||
// Events arrive: {"type":"event","channel":"ml.generate","data":{...}}
|
|
||||||
// Client unsubscribes: {"type":"unsubscribe","channel":"ml.generate"}
|
|
||||||
```
|
|
||||||
|
|
||||||
Subsystems implementing `StreamGroup` declare which channels they publish to. This metadata feeds into the OpenAPI spec as documentation.
|
|
||||||
|
|
||||||
## OpenAPI + SDK Generation
|
|
||||||
|
|
||||||
### Runtime Spec Generation (SpecBuilder)
|
|
||||||
|
|
||||||
swaggo annotations were rejected because routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools already carry JSON Schema at runtime. Instead, a `SpecBuilder` constructs the full OpenAPI 3.1 spec from registered RouteGroups at runtime.
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Groups that implement DescribableGroup contribute endpoint metadata
|
|
||||||
type DescribableGroup interface {
|
|
||||||
RouteGroup
|
|
||||||
Describe() []RouteDescription
|
|
||||||
}
|
|
||||||
|
|
||||||
// SpecBuilder assembles the spec from all groups
|
|
||||||
builder := &api.SpecBuilder{Title: "Core API", Description: "...", Version: "1.0.0"}
|
|
||||||
spec, _ := builder.Build(engine.Groups())
|
|
||||||
```
|
|
||||||
|
|
||||||
### MCP-to-REST Bridge (ToolBridge)
|
|
||||||
|
|
||||||
The `ToolBridge` converts MCP tool descriptors into REST POST endpoints and implements both `RouteGroup` and `DescribableGroup`. Each tool becomes `POST /{tool_name}`. Generic types are captured at MCP registration time via closures, enabling JSON unmarshalling to the correct input type at request time.
|
|
||||||
|
|
||||||
```go
|
|
||||||
bridge := api.NewToolBridge("/v1/tools")
|
|
||||||
mcp.BridgeToAPI(mcpService, bridge) // Populates bridge from MCP tool registry
|
|
||||||
engine.Register(bridge) // Registers REST endpoints + OpenAPI metadata
|
|
||||||
```
|
|
||||||
|
|
||||||
### Swagger UI
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Built-in at GET /swagger/*any
|
|
||||||
// SpecBuilder output served via gin-swagger, cached via sync.Once
|
|
||||||
api.New(api.WithSwagger("Core API", "...", "1.0.0"))
|
|
||||||
```
|
|
||||||
|
|
||||||
### SDK Generation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Via openapi-generator-cli (11 languages supported)
|
|
||||||
core api sdk --lang go # Generate Go SDK
|
|
||||||
core api sdk --lang typescript-fetch,python # Multiple languages
|
|
||||||
core api sdk --lang rust --output ./sdk/ # Custom output dir
|
|
||||||
```
|
|
||||||
|
|
||||||
### CLI Commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
core api spec # Emit OpenAPI JSON to stdout
|
|
||||||
core api spec --format yaml # YAML variant
|
|
||||||
core api spec --output spec.json # Write to file
|
|
||||||
core api sdk --lang python # Generate Python SDK
|
|
||||||
core api sdk --lang go,rust # Multiple SDKs
|
|
||||||
```
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
| Package | Purpose |
|
|
||||||
|---------|---------|
|
|
||||||
| `github.com/gin-gonic/gin` | HTTP framework |
|
|
||||||
| `github.com/swaggo/gin-swagger` | Swagger UI middleware |
|
|
||||||
| `github.com/gin-contrib/cors` | CORS middleware |
|
|
||||||
| `github.com/gin-contrib/secure` | Security headers |
|
|
||||||
| `github.com/gin-contrib/sessions` | Server-side sessions |
|
|
||||||
| `github.com/gin-contrib/authz` | Casbin authorisation |
|
|
||||||
| `github.com/gin-contrib/httpsign` | HTTP signature verification |
|
|
||||||
| `github.com/gin-contrib/slog` | Structured request logging |
|
|
||||||
| `github.com/gin-contrib/timeout` | Per-request timeouts |
|
|
||||||
| `github.com/gin-contrib/gzip` | Gzip compression |
|
|
||||||
| `github.com/gin-contrib/static` | Static file serving |
|
|
||||||
| `github.com/gin-contrib/pprof` | Runtime profiling |
|
|
||||||
| `github.com/gin-contrib/expvar` | Runtime metrics |
|
|
||||||
| `github.com/gin-contrib/location/v2` | Reverse proxy detection |
|
|
||||||
| `github.com/99designs/gqlgen` | GraphQL endpoint |
|
|
||||||
| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | Distributed tracing |
|
|
||||||
| `gopkg.in/yaml.v3` | YAML spec export |
|
|
||||||
| `forge.lthn.ai/core/go-ws` | WebSocket Hub (existing) |
|
|
||||||
|
|
||||||
## Estimated Size
|
|
||||||
|
|
||||||
| Component | LOC |
|
|
||||||
|-----------|-----|
|
|
||||||
| Engine + options | ~200 |
|
|
||||||
| Middleware | ~150 |
|
|
||||||
| Response envelope | ~80 |
|
|
||||||
| RouteGroup interface | ~30 |
|
|
||||||
| WebSocket integration | ~60 |
|
|
||||||
| Tests | ~300 |
|
|
||||||
| **Total go-api** | **~820** |
|
|
||||||
|
|
||||||
Each subsystem's `api/` package adds ~100-200 LOC per route group.
|
|
||||||
|
|
||||||
## Phase 1 — Implemented (20 Feb 2026)
|
|
||||||
|
|
||||||
**Commit:** `17ae945` on Forge (`core/go-api`)
|
|
||||||
|
|
||||||
| Component | Status | Tests |
|
|
||||||
|-----------|--------|-------|
|
|
||||||
| Response envelope (OK, Fail, Paginated) | Done | 9 |
|
|
||||||
| RouteGroup + StreamGroup interfaces | Done | 4 |
|
|
||||||
| Engine (New, Register, Handler, Serve) | Done | 9 |
|
|
||||||
| Bearer auth middleware | Done | 3 |
|
|
||||||
| Request ID middleware | Done | 2 |
|
|
||||||
| CORS middleware (gin-contrib/cors) | Done | 3 |
|
|
||||||
| WebSocket endpoint | Done | 3 |
|
|
||||||
| Swagger UI (gin-swagger) | Done | 2 |
|
|
||||||
| Health endpoint | Done | 1 |
|
|
||||||
| **Total** | **~840 LOC** | **36** |
|
|
||||||
|
|
||||||
**Integration proof:** go-ml/api/ registers 3 endpoints with 12 tests (`0c23858`).
|
|
||||||
|
|
||||||
## Phase 2 Wave 1 — Implemented (20 Feb 2026)
|
|
||||||
|
|
||||||
**Commits:** `6bb7195..daae6f7` on Forge (`core/go-api`)
|
|
||||||
|
|
||||||
| Component | Option | Dependency | Tests |
|
|
||||||
|-----------|--------|------------|-------|
|
|
||||||
| Authentik (forward auth + OIDC) | `WithAuthentik()` | `go-oidc/v3`, `oauth2` | 14 |
|
|
||||||
| Security headers (HSTS, CSP, etc.) | `WithSecure()` | `gin-contrib/secure` | 8 |
|
|
||||||
| Structured request logging | `WithSlog()` | `gin-contrib/slog` | 6 |
|
|
||||||
| Per-request timeouts | `WithTimeout()` | `gin-contrib/timeout` | 5 |
|
|
||||||
| Gzip compression | `WithGzip()` | `gin-contrib/gzip` | 5 |
|
|
||||||
| Static file serving | `WithStatic()` | `gin-contrib/static` | 5 |
|
|
||||||
| **Wave 1 Total** | | | **43** |
|
|
||||||
|
|
||||||
**Cumulative:** 76 tests (36 Phase 1 + 43 Wave 1 - 3 shared), all passing.
|
|
||||||
|
|
||||||
## Phase 2 Wave 2 — Implemented (20 Feb 2026)
|
|
||||||
|
|
||||||
**Commits:** `64a8b16..67dcc83` on Forge (`core/go-api`)
|
|
||||||
|
|
||||||
| Component | Option | Dependency | Tests | Notes |
|
|
||||||
|-----------|--------|------------|-------|-------|
|
|
||||||
| Brotli compression | `WithBrotli()` | `andybalholm/brotli` | 5 | Custom middleware; `gin-contrib/brotli` is empty stub |
|
|
||||||
| Response caching | `WithCache()` | none (in-memory) | 5 | Custom middleware; `gin-contrib/cache` is per-handler, not global |
|
|
||||||
| Server-side sessions | `WithSessions()` | `gin-contrib/sessions` | 5 | Cookie store, configurable name + secret |
|
|
||||||
| Casbin authorisation | `WithAuthz()` | `gin-contrib/authz`, `casbin/v2` | 5 | Subject via Basic Auth; RBAC policy model |
|
|
||||||
| **Wave 2 Total** | | | **20** | |
|
|
||||||
|
|
||||||
**Cumulative:** 102 passing tests (2 integration skipped), all green.
|
|
||||||
|
|
||||||
## Phase 2 Wave 3 — Implemented (20 Feb 2026)
|
|
||||||
|
|
||||||
**Commits:** `7b3f99e..d517fa2` on Forge (`core/go-api`)
|
|
||||||
|
|
||||||
| Component | Option | Dependency | Tests | Notes |
|
|
||||||
|-----------|--------|------------|-------|-------|
|
|
||||||
| HTTP signature verification | `WithHTTPSign()` | `gin-contrib/httpsign` | 5 | HMAC-SHA256; extensible via httpsign.Option |
|
|
||||||
| Server-Sent Events | `WithSSE()` | none (custom SSEBroker) | 6 | Channel filtering, multi-client broadcast, GET /events |
|
|
||||||
| Reverse proxy detection | `WithLocation()` | `gin-contrib/location/v2` | 5 | X-Forwarded-Host/Proto parsing |
|
|
||||||
| Locale detection | `WithI18n()` | `golang.org/x/text/language` | 5 | Accept-Language parsing, message lookup, GetLocale/GetMessage |
|
|
||||||
| GraphQL endpoint | `WithGraphQL()` | `99designs/gqlgen` | 5 | /graphql + optional /graphql/playground |
|
|
||||||
| **Wave 3 Total** | | | **26** | |
|
|
||||||
|
|
||||||
**Cumulative:** 128 passing tests (2 integration skipped), all green.
|
|
||||||
|
|
||||||
## Phase 2 Wave 4 — Implemented (21 Feb 2026)
|
|
||||||
|
|
||||||
**Commits:** `32b3680..8ba1716` on Forge (`core/go-api`)
|
|
||||||
|
|
||||||
| Component | Option | Dependency | Tests | Notes |
|
|
||||||
|-----------|--------|------------|-------|-------|
|
|
||||||
| Runtime profiling | `WithPprof()` | `gin-contrib/pprof` | 5 | /debug/pprof/* endpoints, flag-based mount |
|
|
||||||
| Runtime metrics | `WithExpvar()` | `gin-contrib/expvar` | 5 | /debug/vars endpoint, flag-based mount |
|
|
||||||
| Distributed tracing | `WithTracing()` | `otelgin` + OpenTelemetry SDK | 5 | W3C traceparent propagation, span attributes |
|
|
||||||
| **Wave 4 Total** | | | **15** | |
|
|
||||||
|
|
||||||
**Cumulative:** 143 passing tests (2 integration skipped), all green.
|
|
||||||
|
|
||||||
**Phase 2 complete.** All 4 waves implemented. Every planned plugin has a `With*()` option and tests.
|
|
||||||
|
|
||||||
## Phase 3 — OpenAPI Spec Generation + SDK Codegen (21 Feb 2026)
|
|
||||||
|
|
||||||
**Architecture:** Runtime OpenAPI generation via SpecBuilder (NOT swaggo annotations). Routes are dynamic via RouteGroup, Response[T] generics break swaggo, and MCP tools carry JSON Schema at runtime. A `ToolBridge` converts tool descriptors into RouteGroup + OpenAPI metadata. A `SpecBuilder` constructs the full OpenAPI 3.1 spec. SDK codegen wraps `openapi-generator-cli`.
|
|
||||||
|
|
||||||
### Wave 1: go-api (Tasks 1-5)
|
|
||||||
|
|
||||||
**Commits:** `465bd60..1910aec` on Forge (`core/go-api`)
|
|
||||||
|
|
||||||
| Component | File | Tests | Notes |
|
|
||||||
|-----------|------|-------|-------|
|
|
||||||
| DescribableGroup interface | `group.go` | 5 | Opt-in OpenAPI metadata for RouteGroups |
|
|
||||||
| ToolBridge | `bridge.go` | 6 | Tool descriptors → POST endpoints + DescribableGroup |
|
|
||||||
| SpecBuilder | `openapi.go` | 6 | OpenAPI 3.1 JSON with Response[T] envelope wrapping |
|
|
||||||
| Swagger refactor | `swagger.go` | 5 | Replaced hardcoded empty spec with SpecBuilder |
|
|
||||||
| Spec export | `export.go` | 5 | JSON + YAML export to file/writer |
|
|
||||||
| SDK codegen | `codegen.go` | 5 | 11-language wrapper for openapi-generator-cli |
|
|
||||||
| **Wave 1 Total** | | **32** | |
|
|
||||||
|
|
||||||
### Wave 2: go-ai MCP bridge (Tasks 6-7)
|
|
||||||
|
|
||||||
**Commits:** `2107eda..c37e1cf` on Forge (`core/go-ai`)
|
|
||||||
|
|
||||||
| Component | File | Tests | Notes |
|
|
||||||
|-----------|------|-------|-------|
|
|
||||||
| Tool registry | `mcp/registry.go` | 5 | Generic `addToolRecorded[In,Out]` captures types in closures |
|
|
||||||
| BridgeToAPI | `mcp/bridge.go` | 5 | MCP tools → go-api ToolBridge, 10MB body limit, error classification |
|
|
||||||
| **Wave 2 Total** | | **10** | |
|
|
||||||
|
|
||||||
### Wave 3: CLI commands (Tasks 8-9)
|
|
||||||
|
|
||||||
**Commit:** `d6eec4d` on Forge (`core/cli` dev branch)
|
|
||||||
|
|
||||||
| Component | File | Tests | Notes |
|
|
||||||
|-----------|------|-------|-------|
|
|
||||||
| `core api spec` | `cmd/api/cmd_spec.go` | 2 | JSON/YAML export, --output/--format flags |
|
|
||||||
| `core api sdk` | `cmd/api/cmd_sdk.go` | 2 | --lang (required), --output, --spec, --package flags |
|
|
||||||
| **Wave 3 Total** | | **4** | |
|
|
||||||
|
|
||||||
**Cumulative go-api:** 176 passing tests. **Phase 3 complete.**
|
|
||||||
|
|
||||||
### Known Limitations
|
|
||||||
|
|
||||||
- **Subsystem tools excluded from bridge:** Subsystems call `mcp.AddTool` directly, bypassing `addToolRecorded`. Only the 10 built-in MCP tools appear in the REST bridge. Future: pass `*Service` to `RegisterTools` instead of `*mcp.Server`.
|
|
||||||
- **Flat schema only:** `structSchema` reflection handles flat structs but does not recurse into nested structs. Adequate for current tool inputs.
|
|
||||||
- **CLI spec produces empty bridge:** `core api spec` currently generates a spec with only `/health`. Full MCP integration requires wiring the MCP service into the CLI command.
|
|
||||||
|
|
||||||
## Phase 2 — Gin Plugin Roadmap (Complete)
|
|
||||||
|
|
||||||
All plugins drop in as `With*()` options on the Engine. No architecture changes needed.
|
|
||||||
|
|
||||||
### Security & Auth
|
|
||||||
|
|
||||||
| Plugin | Option | Purpose | Priority |
|
|
||||||
|--------|--------|---------|----------|
|
|
||||||
| ~~**Authentik**~~ | ~~`WithAuthentik()`~~ | ~~OIDC + forward auth integration.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/secure~~ | ~~`WithSecure()`~~ | ~~Security headers: HSTS, X-Frame-Options, X-Content-Type-Options, CSP.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/sessions~~ | ~~`WithSessions()`~~ | ~~Server-side sessions (cookie store). Web session management alongside Authentik tokens.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/authz~~ | ~~`WithAuthz()`~~ | ~~Casbin-based authorisation. Policy-driven access control via RBAC.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/httpsign~~ | ~~`WithHTTPSign()`~~ | ~~HTTP signature verification. HMAC-SHA256 with extensible options.~~ | ~~**Done**~~ |
|
|
||||||
|
|
||||||
### Performance & Reliability
|
|
||||||
|
|
||||||
| Plugin | Option | Purpose | Priority |
|
|
||||||
|--------|--------|---------|----------|
|
|
||||||
| ~~gin-contrib/cache~~ | ~~`WithCache()`~~ | ~~Response caching (in-memory). GET response caching with TTL, lazy eviction.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/timeout~~ | ~~`WithTimeout()`~~ | ~~Per-request timeouts.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/gzip~~ | ~~`WithGzip()`~~ | ~~Gzip response compression.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/brotli~~ | ~~`WithBrotli()`~~ | ~~Brotli compression via `andybalholm/brotli`. Custom middleware (gin-contrib stub empty).~~ | ~~**Done**~~ |
|
|
||||||
|
|
||||||
### Observability
|
|
||||||
|
|
||||||
| Plugin | Option | Purpose | Priority |
|
|
||||||
|--------|--------|---------|----------|
|
|
||||||
| ~~gin-contrib/slog~~ | ~~`WithSlog()`~~ | ~~Structured request logging via slog.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/pprof~~ | ~~`WithPprof()`~~ | ~~Runtime profiling endpoints at /debug/pprof/. Flag-based mount.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/expvar~~ | ~~`WithExpvar()`~~ | ~~Go runtime metrics at /debug/vars. Flag-based mount.~~ | ~~**Done**~~ |
|
|
||||||
| ~~otelgin~~ | ~~`WithTracing()`~~ | ~~OpenTelemetry distributed tracing. W3C traceparent propagation.~~ | ~~**Done**~~ |
|
|
||||||
|
|
||||||
### Content & Streaming
|
|
||||||
|
|
||||||
| Plugin | Option | Purpose | Priority |
|
|
||||||
|--------|--------|---------|----------|
|
|
||||||
| ~~gin-contrib/static~~ | ~~`WithStatic()`~~ | ~~Serve static files.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/sse~~ | ~~`WithSSE()`~~ | ~~Server-Sent Events. Custom SSEBroker with channel filtering, GET /events.~~ | ~~**Done**~~ |
|
|
||||||
| ~~gin-contrib/location~~ | ~~`WithLocation()`~~ | ~~Auto-detect scheme/host from X-Forwarded-* headers.~~ | ~~**Done**~~ |
|
|
||||||
|
|
||||||
### Query Layer
|
|
||||||
|
|
||||||
| Plugin | Option | Purpose | Priority |
|
|
||||||
|--------|--------|---------|----------|
|
|
||||||
| ~~99designs/gqlgen~~ | ~~`WithGraphQL()`~~ | ~~GraphQL endpoint at `/graphql` + optional playground. Accepts gqlgen ExecutableSchema.~~ | ~~**Done**~~ |
|
|
||||||
|
|
||||||
The GraphQL schema can be generated from the same Go Input/Output structs that define the REST endpoints. gqlgen produces an `http.Handler` that mounts directly on Gin. Subsystems opt-in via:
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Subsystems that want GraphQL implement this alongside RouteGroup
|
|
||||||
type ResolverGroup interface {
|
|
||||||
// RegisterResolvers adds query/mutation resolvers to the GraphQL schema
|
|
||||||
RegisterResolvers(schema *graphql.Schema)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
This means a subsystem like go-ml exposes:
|
|
||||||
- **REST:** `POST /v1/ml/generate` (existing)
|
|
||||||
- **GraphQL:** `mutation { mlGenerate(prompt: "...", backend: "mlx") { response, model } }` (same handler)
|
|
||||||
- **MCP:** `ml_generate` tool (existing)
|
|
||||||
|
|
||||||
Four protocols, one set of handlers.
|
|
||||||
|
|
||||||
### Ecosystem Integration
|
|
||||||
|
|
||||||
| Plugin | Option | Purpose | Priority |
|
|
||||||
|--------|--------|---------|----------|
|
|
||||||
| ~~gin-contrib/i18n~~ | ~~`WithI18n()`~~ | ~~Locale detection via Accept-Language. Custom middleware using `golang.org/x/text/language`.~~ | ~~**Done**~~ |
|
|
||||||
| [gin-contrib/graceful](https://github.com/gin-contrib/graceful) | — | Already implemented in Engine.Serve(). Could swap to this for more robust lifecycle management if needed. | — |
|
|
||||||
| [gin-contrib/requestid](https://github.com/gin-contrib/requestid) | — | Already implemented. Theirs uses UUID, ours uses hex. Could swap for standards compliance. | — |
|
|
||||||
|
|
||||||
### Implementation Order
|
|
||||||
|
|
||||||
**Wave 1 (gateway hardening):** ~~Authentik, secure, slog, timeout, gzip, static~~ **DONE** (20 Feb 2026)
|
|
||||||
**Wave 2 (performance + auth):** ~~cache, sessions, authz, brotli~~ **DONE** (20 Feb 2026)
|
|
||||||
**Wave 3 (network + streaming):** ~~httpsign, sse, location, i18n, gqlgen~~ **DONE** (20 Feb 2026)
|
|
||||||
**Wave 4 (observability):** ~~pprof, expvar, tracing~~ **DONE** (21 Feb 2026)
|
|
||||||
|
|
||||||
Each wave adds `With*()` options + tests. No breaking changes — existing code continues to work without any new options enabled.
|
|
||||||
|
|
||||||
## Authentik Integration
|
|
||||||
|
|
||||||
[Authentik](https://goauthentik.io/) is the identity provider and edge auth proxy. It handles user registration, login, MFA, social auth, SAML, and OIDC — so go-api doesn't have to.
|
|
||||||
|
|
||||||
### Two Integration Modes
|
|
||||||
|
|
||||||
**1. Forward Auth (web traffic)**
|
|
||||||
|
|
||||||
Traefik sits in front of go-api. For web routes, Traefik's `forwardAuth` middleware checks with Authentik before passing the request through. Authentik handles login flows, session cookies, and consent. go-api receives pre-authenticated requests with identity headers.
|
|
||||||
|
|
||||||
```
|
|
||||||
Browser → Traefik → Authentik (forward auth) → go-api
|
|
||||||
↓
|
|
||||||
Login page (if unauthenticated)
|
|
||||||
```
|
|
||||||
|
|
||||||
go-api reads trusted headers set by Authentik:
|
|
||||||
```
|
|
||||||
X-Authentik-Username: alice
|
|
||||||
X-Authentik-Groups: admins,developers
|
|
||||||
X-Authentik-Email: alice@example.com
|
|
||||||
X-Authentik-Uid: <uuid>
|
|
||||||
X-Authentik-Jwt: <signed token>
|
|
||||||
```
|
|
||||||
|
|
||||||
**2. OIDC Token Validation (API traffic)**
|
|
||||||
|
|
||||||
API clients (SDKs, CLI tools, network peers) authenticate directly with Authentik's OAuth2 token endpoint, then send the JWT to go-api. go-api validates the JWT using Authentik's OIDC discovery endpoint (`.well-known/openid-configuration`).
|
|
||||||
|
|
||||||
```
|
|
||||||
SDK client → Authentik (token endpoint) → receives JWT
|
|
||||||
SDK client → go-api (Authorization: Bearer <jwt>) → validates via OIDC
|
|
||||||
```
|
|
||||||
|
|
||||||
### Implementation in go-api
|
|
||||||
|
|
||||||
```go
|
|
||||||
engine := api.New(
|
|
||||||
api.WithAuthentik(api.AuthentikConfig{
|
|
||||||
Issuer: "https://auth.lthn.ai/application/o/core-api/",
|
|
||||||
ClientID: "core-api",
|
|
||||||
TrustedProxy: true, // Trust X-Authentik-* headers from Traefik
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
`WithAuthentik()` adds middleware that:
|
|
||||||
1. Checks for `X-Authentik-Jwt` header (forward auth mode) — validates signature, extracts claims
|
|
||||||
2. Falls back to `Authorization: Bearer <jwt>` header (direct OIDC mode) — validates via JWKS
|
|
||||||
3. Populates `c.Set("user", AuthentikUser{...})` in the Gin context for handlers to use
|
|
||||||
4. Skips /health, /swagger, and any public paths
|
|
||||||
|
|
||||||
```go
|
|
||||||
// In any handler:
|
|
||||||
func (r *Routes) ListItems(c *gin.Context) {
|
|
||||||
user := api.GetUser(c) // Returns *AuthentikUser or nil
|
|
||||||
if user == nil {
|
|
||||||
c.JSON(401, api.Fail("unauthorised", "Authentication required"))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
// user.Username, user.Groups, user.Email, user.UID available
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Auth Layers
|
|
||||||
|
|
||||||
```
|
|
||||||
Authentik (identity) → WHO is this? (user, groups, email)
|
|
||||||
↓
|
|
||||||
go-api middleware → IS their token valid? (JWT verification)
|
|
||||||
↓
|
|
||||||
Casbin authz (optional) → CAN they do this? (role → endpoint policies)
|
|
||||||
↓
|
|
||||||
Handler → DOES this (business logic)
|
|
||||||
```
|
|
||||||
|
|
||||||
Phase 1 bearer auth continues to work alongside Authentik — useful for service-to-service tokens, CI/CD, and development. `WithBearerAuth` and `WithAuthentik` can coexist.
|
|
||||||
|
|
||||||
### Authentik Deployment
|
|
||||||
|
|
||||||
Authentik runs as a Docker service alongside go-api, fronted by Traefik:
|
|
||||||
- **auth.lthn.ai** — Authentik UI + OIDC endpoints (production)
|
|
||||||
- **auth.leth.in** — Authentik for devnet/testnet
|
|
||||||
- Traefik routes `/outpost.goauthentik.io/` to Authentik's embedded outpost for forward auth
|
|
||||||
|
|
||||||
### Dependencies
|
|
||||||
|
|
||||||
| Package | Purpose |
|
|
||||||
|---------|---------|
|
|
||||||
| `github.com/coreos/go-oidc/v3` | OIDC discovery + JWT validation |
|
|
||||||
| `golang.org/x/oauth2` | OAuth2 token exchange (for server-side flows) |
|
|
||||||
|
|
||||||
Both are standard Go libraries with no heavy dependencies.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
- gRPC gateway
|
|
||||||
- Built-in user registration/login (Authentik handles this)
|
|
||||||
- API versioning beyond /v1/ prefix
|
|
||||||
|
|
||||||
## Success Criteria
|
|
||||||
|
|
||||||
### Phase 1 (Done)
|
|
||||||
|
|
||||||
1. ~~`core api serve` starts a Gin server with registered subsystem routes~~
|
|
||||||
2. ~~WebSocket subscriptions work alongside REST~~
|
|
||||||
3. ~~Swagger UI accessible at `/swagger/`~~
|
|
||||||
4. ~~All endpoints return consistent Response envelope~~
|
|
||||||
5. ~~Bearer token auth protects all routes~~
|
|
||||||
6. ~~First subsystem integration (go-ml/api/) proves the pattern~~
|
|
||||||
|
|
||||||
### Phase 2 (Done)
|
|
||||||
|
|
||||||
7. ~~Security headers, compression, and caching active in production~~
|
|
||||||
8. ~~Session-based auth alongside bearer tokens~~
|
|
||||||
9. ~~HTTP signature verification for Lethean network peers~~
|
|
||||||
10. ~~Static file serving for docs site and SDK downloads~~
|
|
||||||
11. ~~GraphQL endpoint at `/graphql` with playground~~
|
|
||||||
|
|
||||||
### Phase 3 (Done)
|
|
||||||
|
|
||||||
12. ~~`core api spec` emits valid OpenAPI 3.1 JSON via runtime SpecBuilder~~
|
|
||||||
13. ~~`core api sdk` generates SDKs for 11 languages via openapi-generator-cli~~
|
|
||||||
14. ~~MCP tools bridged to REST endpoints via ToolBridge + BridgeToAPI~~
|
|
||||||
15. ~~OpenAPI spec includes Response[T] envelope wrapping~~
|
|
||||||
16. ~~Spec export to file in JSON and YAML formats~~
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,128 +0,0 @@
|
||||||
# CLI Meta-Package Restructure — Design
|
|
||||||
|
|
||||||
**Goal:** Transform `core/cli` from a 35K LOC monolith into a thin assembly repo that ships variant binaries. Domain repos own their commands. `go/pkg/cli` is the only import any domain package needs for CLI concerns.
|
|
||||||
|
|
||||||
**Architecture:** Commands register as framework services via `cli.WithCommands()`, passed to `cli.Main()`. Command code lives in the domain repos that own the business logic. The cli repo is a thin `main.go` that wires them together.
|
|
||||||
|
|
||||||
**Tech Stack:** go/pkg/cli (wraps cobra + charmbracelet), Core framework lifecycle, Taskfile
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. CLI SDK — The Single Import
|
|
||||||
|
|
||||||
`forge.lthn.ai/core/go/pkg/cli` is the **only** import domain packages use for CLI concerns. It wraps cobra, charmbracelet, and stdlib behind a stable API. If the underlying libraries change, only `go/pkg/cli` is touched — every domain repo is insulated.
|
|
||||||
|
|
||||||
### Already done
|
|
||||||
|
|
||||||
- **Cobra:** `Command` type alias, `NewCommand()`, `NewGroup()`, `NewRun()`, flag helpers (`StringFlag`, `BoolFlag`, `IntFlag`, `StringSliceFlag`), arg validators
|
|
||||||
- **Output:** `Success()`, `Error()`, `Warn()`, `Info()`, `Table`, `Section()`, `Label()`, `Task()`, `Hint()`
|
|
||||||
- **Prompts:** `Confirm()`, `Question()`, `Choose()`, `ChooseMulti()` with grammar-based action variants
|
|
||||||
- **Styles:** 17 pre-built styles, `AnsiStyle` builder, Tailwind colour constants (47 hex values)
|
|
||||||
- **Glyphs:** `:check:`, `:cross:`, `:warn:` etc. with Unicode/Emoji/ASCII themes
|
|
||||||
- **Layout:** HLCRF composite renderer (Header/Left/Content/Right/Footer)
|
|
||||||
- **Errors:** `Wrap()`, `WrapVerb()`, `ExitError`, `Is()`, `As()`
|
|
||||||
- **Logging:** `LogDebug()`, `LogInfo()`, `LogWarn()`, `LogError()`, `LogSecurity()`
|
|
||||||
- **TUI primitives:** `Spinner`, `ProgressBar`, `InteractiveList`, `TextInput`, `Viewport`, `RunTUI`
|
|
||||||
- **Command registration:** `WithCommands(name, fn)` — registers commands as framework services
|
|
||||||
|
|
||||||
### Stubbed for later (interface exists, returns simple fallback)
|
|
||||||
|
|
||||||
- `Form(fields []FormField) (map[string]string, error)` — multi-field form (backed by huh later)
|
|
||||||
- `FilePicker(opts ...FilePickerOption) (string, error)` — file browser
|
|
||||||
- `Tabs(items []TabItem) error` — tabbed content panes
|
|
||||||
|
|
||||||
### Rule
|
|
||||||
|
|
||||||
Domain packages import `forge.lthn.ai/core/go/pkg/cli` and **nothing else** for CLI concerns. No `cobra`, no `lipgloss`, no `bubbletea`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Command Registration — Framework Lifecycle
|
|
||||||
|
|
||||||
Commands register through the Core framework's service lifecycle, not through global state or `init()` functions.
|
|
||||||
|
|
||||||
### The contract
|
|
||||||
|
|
||||||
Each domain repo exports an `Add*Commands(root *cli.Command)` function. The CLI binary wires it in via `cli.WithCommands()`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
// go-ai/cmd/daemon/cmd.go
|
|
||||||
package daemon
|
|
||||||
|
|
||||||
import "forge.lthn.ai/core/go/pkg/cli"
|
|
||||||
|
|
||||||
// AddDaemonCommand adds the 'daemon' command group to the root.
|
|
||||||
func AddDaemonCommand(root *cli.Command) {
|
|
||||||
daemonCmd := cli.NewGroup("daemon", "Manage the core daemon", "")
|
|
||||||
root.AddCommand(daemonCmd)
|
|
||||||
// subcommands...
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
No `init()`. No blank imports. No `cli.RegisterCommands()`.
|
|
||||||
|
|
||||||
### How it works
|
|
||||||
|
|
||||||
`cli.WithCommands(name, fn)` wraps the registration function as a framework service implementing `Startable`. During `Core.ServiceStartup()`, the service's `OnStartup()` casts `Core.App` to `*cobra.Command` and calls the registration function. Core services (i18n, log, workspace) start first since they're registered before command services.
|
|
||||||
|
|
||||||
```go
|
|
||||||
// cli/main.go
|
|
||||||
func main() {
|
|
||||||
cli.Main(
|
|
||||||
cli.WithCommands("config", config.AddConfigCommands),
|
|
||||||
cli.WithCommands("doctor", doctor.AddDoctorCommands),
|
|
||||||
// ...
|
|
||||||
)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Migration status (completed)
|
|
||||||
|
|
||||||
| Source | Destination | Status |
|
|
||||||
|--------|-------------|--------|
|
|
||||||
| `cmd/dev, setup, qa, docs, gitcmd, monitor` | `go-devops/cmd/` | Done |
|
|
||||||
| `cmd/lab` | `go-ai/cmd/` | Done |
|
|
||||||
| `cmd/workspace` | `go-agentic/cmd/` | Done |
|
|
||||||
| `cmd/go` | `core/go/cmd/gocmd` | Done |
|
|
||||||
| `cmd/vanity-import, community` | `go-devops/cmd/` | Done |
|
|
||||||
| `cmd/updater` | `go-update` | Done (own repo) |
|
|
||||||
| `cmd/daemon, mcpcmd, security` | `go-ai/cmd/` | Done |
|
|
||||||
| `cmd/crypt` | `go-crypt/cmd/` | Done |
|
|
||||||
| `cmd/rag` | `go-rag/cmd/` | Done |
|
|
||||||
| `cmd/unifi` | `go-netops/cmd/` | Done |
|
|
||||||
| `cmd/api` | `go-api/cmd/` | Done |
|
|
||||||
| `cmd/collect, forge, gitea` | `go-scm/cmd/` | Done |
|
|
||||||
| `cmd/deploy, prod, vm` | `go-devops/cmd/` | Done |
|
|
||||||
|
|
||||||
### Stays in cli/ (meta/framework commands)
|
|
||||||
|
|
||||||
`config`, `doctor`, `help`, `module`, `pkgcmd`, `plugin`, `session`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Variant Binaries (future)
|
|
||||||
|
|
||||||
The cli/ repo can produce variant binaries by creating multiple `main.go` files that wire different sets of commands.
|
|
||||||
|
|
||||||
```
|
|
||||||
cli/
|
|
||||||
├── main.go # Current — meta commands only
|
|
||||||
├── cmd/core-full/main.go # Full CLI — all ecosystem commands
|
|
||||||
├── cmd/core-ci/main.go # CI agent dispatch + SCM
|
|
||||||
├── cmd/core-mlx/main.go # ML inference subprocess
|
|
||||||
└── cmd/core-ops/main.go # DevOps + infra management
|
|
||||||
```
|
|
||||||
|
|
||||||
Each variant calls `cli.Main()` with its specific `cli.WithCommands()` set. No blank imports needed.
|
|
||||||
|
|
||||||
### Why variants matter
|
|
||||||
|
|
||||||
- `core-mlx` ships to the homelab as a ~10MB binary, not 50MB with devops/forge/netops
|
|
||||||
- `core-ci` deploys to agent machines without ML or CGO dependencies
|
|
||||||
- Adding a new variant = one new `main.go` with the right `WithCommands` calls
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Current State
|
|
||||||
|
|
||||||
cli/ has 7 meta packages, one `main.go`, and zero business logic. Everything else lives in the domain repos that own it. Total cli/ LOC is ~2K.
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,286 +0,0 @@
|
||||||
# go-forge Design Document
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
**go-forge** is a full-coverage Go client for the Forgejo API (450 endpoints, 284 paths, 229 types). It uses a generic `Resource[T, C, U]` pattern for CRUD operations (91% of endpoints) and hand-written methods for 39 unique action endpoints. Types are generated from Forgejo's `swagger.v1.json` spec.
|
|
||||||
|
|
||||||
**Module path:** `forge.lthn.ai/core/go-forge`
|
|
||||||
|
|
||||||
**Origin:** Extracted from `go-scm/forge/` (45 methods covering 10% of API), expanded to full coverage.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
forge.lthn.ai/core/go-forge
|
|
||||||
├── client.go # HTTP client: auth, headers, rate limiting, context.Context
|
|
||||||
├── pagination.go # Generic paginated request helper
|
|
||||||
├── resource.go # Resource[T, C, U] generic CRUD (List/Get/Create/Update/Delete)
|
|
||||||
├── errors.go # Typed error handling (APIError, NotFound, Forbidden, etc.)
|
|
||||||
├── forge.go # Top-level Forge client aggregating all services
|
|
||||||
│
|
|
||||||
├── types/ # Generated from swagger.v1.json
|
|
||||||
│ ├── generate.go # //go:generate directive
|
|
||||||
│ ├── repo.go # Repository, CreateRepoOption, EditRepoOption
|
|
||||||
│ ├── issue.go # Issue, CreateIssueOption, EditIssueOption
|
|
||||||
│ ├── pr.go # PullRequest, CreatePullRequestOption
|
|
||||||
│ ├── user.go # User, CreateUserOption
|
|
||||||
│ ├── org.go # Organisation, CreateOrgOption
|
|
||||||
│ ├── team.go # Team, CreateTeamOption
|
|
||||||
│ ├── label.go # Label, CreateLabelOption
|
|
||||||
│ ├── release.go # Release, CreateReleaseOption
|
|
||||||
│ ├── branch.go # Branch, BranchProtection
|
|
||||||
│ ├── milestone.go # Milestone, CreateMilestoneOption
|
|
||||||
│ ├── hook.go # Hook, CreateHookOption
|
|
||||||
│ ├── key.go # DeployKey, PublicKey, GPGKey
|
|
||||||
│ ├── notification.go # NotificationThread, NotificationSubject
|
|
||||||
│ ├── package.go # Package, PackageFile
|
|
||||||
│ ├── action.go # ActionRunner, ActionSecret, ActionVariable
|
|
||||||
│ ├── commit.go # Commit, CommitStatus, CombinedStatus
|
|
||||||
│ ├── content.go # ContentsResponse, FileOptions
|
|
||||||
│ ├── wiki.go # WikiPage, WikiPageMetaData
|
|
||||||
│ ├── review.go # PullReview, PullReviewComment
|
|
||||||
│ ├── reaction.go # Reaction
|
|
||||||
│ ├── topic.go # TopicResponse
|
|
||||||
│ ├── misc.go # Markdown, License, GitignoreTemplate, NodeInfo
|
|
||||||
│ ├── admin.go # Cron, QuotaGroup, QuotaRule
|
|
||||||
│ ├── activity.go # Activity, Feed
|
|
||||||
│ └── common.go # Shared types: Permission, ExternalTracker, etc.
|
|
||||||
│
|
|
||||||
├── repos.go # RepoService: CRUD + fork, mirror, transfer, template
|
|
||||||
├── issues.go # IssueService: CRUD + pin, deadline, reactions, stopwatch
|
|
||||||
├── pulls.go # PullService: CRUD + merge, update, reviews, dismiss
|
|
||||||
├── orgs.go # OrgService: CRUD + members, avatar, block, hooks
|
|
||||||
├── users.go # UserService: CRUD + keys, followers, starred, settings
|
|
||||||
├── teams.go # TeamService: CRUD + members, repos
|
|
||||||
├── admin.go # AdminService: users, orgs, cron, runners, quota, unadopted
|
|
||||||
├── branches.go # BranchService: CRUD + protection rules
|
|
||||||
├── releases.go # ReleaseService: CRUD + assets
|
|
||||||
├── labels.go # LabelService: repo + org + issue labels
|
|
||||||
├── webhooks.go # WebhookService: CRUD + test hook
|
|
||||||
├── notifications.go # NotificationService: list, mark read
|
|
||||||
├── packages.go # PackageService: list, get, delete
|
|
||||||
├── actions.go # ActionsService: runners, secrets, variables, workflow dispatch
|
|
||||||
├── contents.go # ContentService: file read/write/delete via API
|
|
||||||
├── wiki.go # WikiService: pages
|
|
||||||
├── commits.go # CommitService: status, notes, diff
|
|
||||||
├── misc.go # MiscService: markdown, licenses, gitignore, nodeinfo
|
|
||||||
│
|
|
||||||
├── config.go # URL/token resolution: env → config file → flags
|
|
||||||
│
|
|
||||||
├── cmd/forgegen/ # Code generator: swagger.v1.json → types/*.go
|
|
||||||
│ ├── main.go
|
|
||||||
│ ├── parser.go # Parse OpenAPI 2.0 definitions
|
|
||||||
│ ├── generator.go # Render Go source files
|
|
||||||
│ └── templates/ # Go text/template files for codegen
|
|
||||||
│
|
|
||||||
└── testdata/
|
|
||||||
└── swagger.v1.json # Pinned spec for testing + generation
|
|
||||||
```
|
|
||||||
|
|
||||||
## Key Design Decisions
|
|
||||||
|
|
||||||
### 1. Generic Resource[T, C, U]
|
|
||||||
|
|
||||||
Three type parameters: T (resource type), C (create options), U (update options).
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Resource[T any, C any, U any] struct {
|
|
||||||
client *Client
|
|
||||||
path string // e.g. "/api/v1/repos/{owner}/{repo}/issues"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *Resource[T, C, U]) List(ctx context.Context, params Params, opts ListOptions) ([]T, error)
|
|
||||||
func (r *Resource[T, C, U]) Get(ctx context.Context, params Params, id string) (*T, error)
|
|
||||||
func (r *Resource[T, C, U]) Create(ctx context.Context, params Params, body *C) (*T, error)
|
|
||||||
func (r *Resource[T, C, U]) Update(ctx context.Context, params Params, id string, body *U) (*T, error)
|
|
||||||
func (r *Resource[T, C, U]) Delete(ctx context.Context, params Params, id string) error
|
|
||||||
```
|
|
||||||
|
|
||||||
`Params` is `map[string]string` resolving path variables: `{"owner": "core", "repo": "go-forge"}`.
|
|
||||||
|
|
||||||
This covers 411 of 450 endpoints (91%).
|
|
||||||
|
|
||||||
### 2. Service Structs Embed Resource
|
|
||||||
|
|
||||||
```go
|
|
||||||
type IssueService struct {
|
|
||||||
Resource[types.Issue, types.CreateIssueOption, types.EditIssueOption]
|
|
||||||
}
|
|
||||||
|
|
||||||
// CRUD comes free. Actions are hand-written:
|
|
||||||
func (s *IssueService) Pin(ctx context.Context, owner, repo string, index int64) error
|
|
||||||
func (s *IssueService) SetDeadline(ctx context.Context, owner, repo string, index int64, deadline *time.Time) error
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Top-Level Forge Client
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Forge struct {
|
|
||||||
client *Client
|
|
||||||
Repos *RepoService
|
|
||||||
Issues *IssueService
|
|
||||||
Pulls *PullService
|
|
||||||
Orgs *OrgService
|
|
||||||
Users *UserService
|
|
||||||
Teams *TeamService
|
|
||||||
Admin *AdminService
|
|
||||||
Branches *BranchService
|
|
||||||
Releases *ReleaseService
|
|
||||||
Labels *LabelService
|
|
||||||
Webhooks *WebhookService
|
|
||||||
Notifications *NotificationService
|
|
||||||
Packages *PackageService
|
|
||||||
Actions *ActionsService
|
|
||||||
Contents *ContentService
|
|
||||||
Wiki *WikiService
|
|
||||||
Commits *CommitService
|
|
||||||
Misc *MiscService
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewForge(url, token string, opts ...Option) *Forge
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Codegen from swagger.v1.json
|
|
||||||
|
|
||||||
The `cmd/forgegen/` tool reads the OpenAPI 2.0 spec and generates:
|
|
||||||
- Go struct definitions with JSON tags and doc comments
|
|
||||||
- Enum constants
|
|
||||||
- Type mapping (OpenAPI → Go)
|
|
||||||
|
|
||||||
229 type definitions → ~25 grouped Go files in `types/`.
|
|
||||||
|
|
||||||
Type mapping rules:
|
|
||||||
| OpenAPI | Go |
|
|
||||||
|---------|-----|
|
|
||||||
| `string` | `string` |
|
|
||||||
| `string` + `date-time` | `time.Time` |
|
|
||||||
| `integer` + `int64` | `int64` |
|
|
||||||
| `integer` | `int` |
|
|
||||||
| `boolean` | `bool` |
|
|
||||||
| `array` of T | `[]T` |
|
|
||||||
| `$ref` | `*T` (pointer) |
|
|
||||||
| nullable | pointer type |
|
|
||||||
| `binary` | `[]byte` |
|
|
||||||
|
|
||||||
### 5. HTTP Client
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Client struct {
|
|
||||||
baseURL string
|
|
||||||
token string
|
|
||||||
httpClient *http.Client
|
|
||||||
userAgent string
|
|
||||||
}
|
|
||||||
|
|
||||||
func New(url, token string, opts ...Option) *Client
|
|
||||||
|
|
||||||
func (c *Client) Get(ctx context.Context, path string, out any) error
|
|
||||||
func (c *Client) Post(ctx context.Context, path string, body, out any) error
|
|
||||||
func (c *Client) Patch(ctx context.Context, path string, body, out any) error
|
|
||||||
func (c *Client) Put(ctx context.Context, path string, body, out any) error
|
|
||||||
func (c *Client) Delete(ctx context.Context, path string) error
|
|
||||||
```
|
|
||||||
|
|
||||||
Options: `WithHTTPClient`, `WithUserAgent`, `WithRateLimit`, `WithLogger`.
|
|
||||||
|
|
||||||
### 6. Pagination
|
|
||||||
|
|
||||||
Forgejo uses `page` + `limit` query params and `X-Total-Count` response header.
|
|
||||||
|
|
||||||
```go
|
|
||||||
type ListOptions struct {
|
|
||||||
Page int
|
|
||||||
Limit int // default 50, max configurable
|
|
||||||
}
|
|
||||||
|
|
||||||
type PagedResult[T any] struct {
|
|
||||||
Items []T
|
|
||||||
TotalCount int
|
|
||||||
Page int
|
|
||||||
HasMore bool
|
|
||||||
}
|
|
||||||
|
|
||||||
// ListAll fetches all pages automatically.
|
|
||||||
func (r *Resource[T, C, U]) ListAll(ctx context.Context, params Params) ([]T, error)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7. Error Handling
|
|
||||||
|
|
||||||
```go
|
|
||||||
type APIError struct {
|
|
||||||
StatusCode int
|
|
||||||
Message string
|
|
||||||
URL string
|
|
||||||
}
|
|
||||||
|
|
||||||
func IsNotFound(err error) bool
|
|
||||||
func IsForbidden(err error) bool
|
|
||||||
func IsConflict(err error) bool
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8. Config Resolution (from go-scm/forge)
|
|
||||||
|
|
||||||
Priority: flags → environment → config file.
|
|
||||||
|
|
||||||
```go
|
|
||||||
func NewFromConfig(flagURL, flagToken string) (*Forge, error)
|
|
||||||
func ResolveConfig(flagURL, flagToken string) (url, token string, err error)
|
|
||||||
func SaveConfig(url, token string) error
|
|
||||||
```
|
|
||||||
|
|
||||||
Env vars: `FORGE_URL`, `FORGE_TOKEN`. Config file: `~/.config/forge/config.json`.
|
|
||||||
|
|
||||||
## API Coverage
|
|
||||||
|
|
||||||
| Category | Endpoints | CRUD | Actions |
|
|
||||||
|----------|-----------|------|---------|
|
|
||||||
| Repository | 175 | 165 | 10 (fork, mirror, transfer, template, avatar, diffpatch) |
|
|
||||||
| User | 74 | 70 | 4 (avatar, GPG verify) |
|
|
||||||
| Issue | 67 | 57 | 10 (pin, deadline, reactions, stopwatch, labels) |
|
|
||||||
| Organisation | 63 | 59 | 4 (avatar, block/unblock) |
|
|
||||||
| Admin | 39 | 35 | 4 (cron run, rename, adopt, quota set) |
|
|
||||||
| Miscellaneous | 12 | 7 | 5 (markdown render, markup, nodeinfo) |
|
|
||||||
| Notification | 7 | 7 | 0 |
|
|
||||||
| ActivityPub | 6 | 3 | 3 (inbox POST) |
|
|
||||||
| Package | 4 | 4 | 0 |
|
|
||||||
| Settings | 4 | 4 | 0 |
|
|
||||||
| **Total** | **450** | **411** | **39** |
|
|
||||||
|
|
||||||
## Integration Points
|
|
||||||
|
|
||||||
### go-api
|
|
||||||
|
|
||||||
Services implement `DescribableGroup` from go-api Phase 3, enabling:
|
|
||||||
- REST endpoint generation via ToolBridge
|
|
||||||
- Auto-generated OpenAPI spec
|
|
||||||
- Multi-language SDK codegen
|
|
||||||
|
|
||||||
### go-scm
|
|
||||||
|
|
||||||
go-scm/forge/ becomes a thin adapter importing go-forge types. Existing go-scm users are unaffected — the multi-provider abstraction layer stays.
|
|
||||||
|
|
||||||
### go-ai/mcp
|
|
||||||
|
|
||||||
The MCP subsystem can register go-forge operations as MCP tools, giving AI agents full Forgejo API access.
|
|
||||||
|
|
||||||
## 39 Unique Action Methods
|
|
||||||
|
|
||||||
These require hand-written implementation:
|
|
||||||
|
|
||||||
**Repository:** migrate, fork, generate (template), transfer, accept/reject transfer, mirror sync, push mirror sync, avatar, diffpatch, contents (multi-file modify)
|
|
||||||
|
|
||||||
**Pull Requests:** merge, update (rebase), submit review, dismiss/undismiss review
|
|
||||||
|
|
||||||
**Issues:** pin, set deadline, add reaction, start/stop stopwatch, add issue labels
|
|
||||||
|
|
||||||
**Comments:** add reaction
|
|
||||||
|
|
||||||
**Admin:** run cron task, adopt unadopted, rename user, set quota groups
|
|
||||||
|
|
||||||
**Misc:** render markdown, render raw markdown, render markup, GPG key verify
|
|
||||||
|
|
||||||
**ActivityPub:** inbox POST (actor, repo, user)
|
|
||||||
|
|
||||||
**Actions:** dispatch workflow
|
|
||||||
|
|
||||||
**Git:** set note on commit, test webhook
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,209 +0,0 @@
|
||||||
# Frame Bubbletea Upgrade Design
|
|
||||||
|
|
||||||
**Issue:** core/go#15
|
|
||||||
**Date:** 2026-02-22
|
|
||||||
**Status:** Approved
|
|
||||||
|
|
||||||
**Goal:** Upgrade `cli.Frame` from raw ANSI + `golang.org/x/term` to bubbletea internally, adding keyboard navigation, focus management, and lipgloss layout composition while preserving the existing public API.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
Single ownership model. Frame becomes the sole `tea.Model` wrapping a `tea.Program`. It owns the terminal (alt-screen, raw mode, resize events, input). Region models never touch the terminal directly.
|
|
||||||
|
|
||||||
Message routing:
|
|
||||||
- **Key messages** — routed to the focused region's `FrameModel.Update()` only
|
|
||||||
- **Tick/resize messages** — broadcast to all region `FrameModel.Update()` calls
|
|
||||||
- **Custom messages** — broadcast to all (enables cross-region communication)
|
|
||||||
|
|
||||||
Dual interface pattern:
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Existing — view-only, no changes
|
|
||||||
type Model interface {
|
|
||||||
View(width, height int) string
|
|
||||||
}
|
|
||||||
|
|
||||||
// New — interactive components
|
|
||||||
type FrameModel interface {
|
|
||||||
Model
|
|
||||||
Init() tea.Cmd
|
|
||||||
Update(tea.Msg) (FrameModel, tea.Cmd)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Frame wraps plain `Model` in a no-op adapter internally, so existing code (StatusLine, KeyHints, Breadcrumb, StaticModel, ModelFunc) works without changes.
|
|
||||||
|
|
||||||
Layout composition replaces the manual ANSI cursor/clear dance in `runLive()` with lipgloss `JoinVertical` and `JoinHorizontal`. The existing HLCRF variant parser and region size calculations stay, but rendering uses lipgloss instead of raw escape codes.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Focus Management
|
|
||||||
|
|
||||||
Focus ring. Frame maintains an ordered list of focusable regions (only regions with `FrameModel` components). Focus cycles through them.
|
|
||||||
|
|
||||||
Navigation:
|
|
||||||
- `Tab` / `Shift-Tab` — cycle focus forward/backward through the ring
|
|
||||||
- Arrow keys — spatial navigation (up to Header, down to Footer, left to Left sidebar, right to Right sidebar)
|
|
||||||
- Configurable via `KeyMap` struct with sensible defaults
|
|
||||||
|
|
||||||
```go
|
|
||||||
type KeyMap struct {
|
|
||||||
FocusNext key.Binding // Tab
|
|
||||||
FocusPrev key.Binding // Shift-Tab
|
|
||||||
FocusUp key.Binding // Up (to Header from Content)
|
|
||||||
FocusDown key.Binding // Down (to Footer from Content)
|
|
||||||
FocusLeft key.Binding // Left (to Left sidebar)
|
|
||||||
FocusRight key.Binding // Right (to Right sidebar)
|
|
||||||
Quit key.Binding // q, Ctrl-C
|
|
||||||
Back key.Binding // Esc (triggers Navigate back)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Visual feedback: focused region gets a subtle border highlight (configurable via lipgloss border styling). Unfocused regions render normally.
|
|
||||||
|
|
||||||
Key filtering: focus keys are consumed by Frame and never forwarded to region models. All other keys go to the focused region's `Update()`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Public API
|
|
||||||
|
|
||||||
### Preserved (no changes)
|
|
||||||
|
|
||||||
- `NewFrame(variant string) *Frame`
|
|
||||||
- `Header(m Model)`, `Left(m Model)`, `Content(m Model)`, `Right(m Model)`, `Footer(m Model)`
|
|
||||||
- `Navigate(m Model)`, `Back() bool`
|
|
||||||
- `Run()`, `RunFor(d time.Duration)`, `Stop()`
|
|
||||||
- `String()` — static render for non-TTY
|
|
||||||
- `ModelFunc`, `StaticModel`, `StatusLine`, `KeyHints`, `Breadcrumb`
|
|
||||||
|
|
||||||
### New additions
|
|
||||||
|
|
||||||
```go
|
|
||||||
// WithKeyMap sets custom key bindings for Frame navigation.
|
|
||||||
func (f *Frame) WithKeyMap(km KeyMap) *Frame
|
|
||||||
|
|
||||||
// Focused returns the currently focused region.
|
|
||||||
func (f *Frame) Focused() Region
|
|
||||||
|
|
||||||
// Focus sets focus to a specific region.
|
|
||||||
func (f *Frame) Focus(r Region)
|
|
||||||
|
|
||||||
// Send injects a message into the Frame's tea.Program.
|
|
||||||
// Useful for triggering updates from external goroutines.
|
|
||||||
func (f *Frame) Send(msg tea.Msg)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Behavioural changes
|
|
||||||
|
|
||||||
- `Run()` now starts a `tea.Program` in TTY mode (instead of raw ticker loop)
|
|
||||||
- Non-TTY path unchanged — still calls `String()` and returns
|
|
||||||
- `RunFor()` unchanged — uses `Stop()` after timer
|
|
||||||
|
|
||||||
### New dependencies
|
|
||||||
|
|
||||||
- `github.com/charmbracelet/bubbletea` (already in core/go)
|
|
||||||
- `github.com/charmbracelet/lipgloss` (already in core/go)
|
|
||||||
- `github.com/charmbracelet/bubbles/key` (key bindings)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Internal Implementation
|
|
||||||
|
|
||||||
Frame implements `tea.Model`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (f *Frame) Init() tea.Cmd
|
|
||||||
func (f *Frame) Update(tea.Msg) (tea.Model, tea.Cmd)
|
|
||||||
func (f *Frame) View() string
|
|
||||||
```
|
|
||||||
|
|
||||||
`Init()` collects `Init()` from all `FrameModel` regions via `tea.Batch()`.
|
|
||||||
|
|
||||||
`Update()` handles:
|
|
||||||
1. `tea.WindowSizeMsg` — update dimensions, broadcast to all FrameModels
|
|
||||||
2. `tea.KeyMsg` matching focus keys — advance/retreat focus ring
|
|
||||||
3. `tea.KeyMsg` matching quit — return `tea.Quit`
|
|
||||||
4. `tea.KeyMsg` matching back — call `Back()`, return nil
|
|
||||||
5. All other `tea.KeyMsg` — forward to focused region's `Update()`
|
|
||||||
6. All other messages — broadcast to all FrameModels
|
|
||||||
|
|
||||||
`View()` uses lipgloss composition:
|
|
||||||
|
|
||||||
```
|
|
||||||
header = renderRegion(H, width, 1)
|
|
||||||
footer = renderRegion(F, width, 1)
|
|
||||||
middleH = height - headerH - footerH
|
|
||||||
|
|
||||||
left = renderRegion(L, width/4, middleH)
|
|
||||||
right = renderRegion(R, width/4, middleH)
|
|
||||||
content = renderRegion(C, contentW, middleH)
|
|
||||||
|
|
||||||
middle = lipgloss.JoinHorizontal(Top, left, content, right)
|
|
||||||
output = lipgloss.JoinVertical(Left, header, middle, footer)
|
|
||||||
```
|
|
||||||
|
|
||||||
`Run()` change:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (f *Frame) Run() {
|
|
||||||
if !f.isTTY() {
|
|
||||||
fmt.Fprint(f.out, f.String())
|
|
||||||
return
|
|
||||||
}
|
|
||||||
p := tea.NewProgram(f, tea.WithAltScreen())
|
|
||||||
f.program = p
|
|
||||||
if _, err := p.Run(); err != nil {
|
|
||||||
Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Plain `Model` adapter:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type modelAdapter struct{ m Model }
|
|
||||||
func (a *modelAdapter) Init() tea.Cmd { return nil }
|
|
||||||
func (a *modelAdapter) Update(tea.Msg) (FrameModel, tea.Cmd) { return a, nil }
|
|
||||||
func (a *modelAdapter) View(w, h int) string { return a.m.View(w, h) }
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Testing Strategy
|
|
||||||
|
|
||||||
Existing 14 tests preserved. They use `bytes.Buffer` (non-TTY path), bypassing bubbletea.
|
|
||||||
|
|
||||||
New tests for interactive features:
|
|
||||||
- Focus cycling: Tab advances focus, Shift-Tab goes back
|
|
||||||
- Spatial navigation: arrow keys move focus to correct region
|
|
||||||
- Message routing: key events only reach focused model
|
|
||||||
- Tick broadcast: tick events reach all models
|
|
||||||
- Resize propagation: resize reaches all models
|
|
||||||
- FrameModel lifecycle: Init() called on Run(), Update() receives messages
|
|
||||||
- Adapter: plain Model wrapped correctly, receives no Update calls
|
|
||||||
- Navigate/Back with FrameModel: focus transfers correctly
|
|
||||||
- KeyMap customization: overridden bindings work
|
|
||||||
- Send(): external messages delivered to models
|
|
||||||
|
|
||||||
Testing approach: use bubbletea's `teatest` package for interactive tests. Non-TTY tests stay as-is with `bytes.Buffer`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files Affected
|
|
||||||
|
|
||||||
| File | Action | Purpose |
|
|
||||||
|------|--------|---------|
|
|
||||||
| `pkg/cli/frame.go` | modify | Add bubbletea tea.Model implementation, lipgloss layout, focus management |
|
|
||||||
| `pkg/cli/frame_model.go` | new | FrameModel interface, modelAdapter, KeyMap |
|
|
||||||
| `pkg/cli/frame_test.go` | modify | Add interactive tests alongside existing ones |
|
|
||||||
| `go.mod` | modify | Add bubbletea, lipgloss, bubbles dependencies |
|
|
||||||
|
|
||||||
## Design Decisions
|
|
||||||
|
|
||||||
1. **Frame as tea.Model, not wrapping separate tea.Model** — Frame IS the model, simplest ownership
|
|
||||||
2. **Dual interface (Model + FrameModel)** — backward compatible, existing components unchanged
|
|
||||||
3. **Lipgloss for layout** — replaces manual ANSI, consistent with bubbletea ecosystem
|
|
||||||
4. **Focus ring with spatial override** — Tab for cycling, arrows for direct spatial jumps
|
|
||||||
5. **Non-TTY path untouched** — `String()` and non-TTY `Run()` stay exactly as-is
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,57 +0,0 @@
|
||||||
# BugSETI HubService — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 13 February 2026
|
|
||||||
**Module:** `forge.lthn.ai/core/cli` (extracted to `core/bugseti` repo on 16 Feb 2026)
|
|
||||||
**Status:** Complete — all Go-side tasks implemented and wired into app lifecycle
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
Thin HTTP client service coordinating with the agentic portal's
|
|
||||||
`/api/bugseti/*` endpoints for issue claiming, stats sync, leaderboard,
|
|
||||||
and offline-first pending operations queue.
|
|
||||||
|
|
||||||
### Implementation (Tasks 1-8 from plan)
|
|
||||||
|
|
||||||
All 8 Go-side tasks were implemented across commits `a38ce05` through `177ce27`:
|
|
||||||
|
|
||||||
1. **Config fields** — HubURL, HubToken, ClientID, ClientName added to
|
|
||||||
ConfigService with getters/setters (`a38ce05`)
|
|
||||||
2. **HubService types + constructor** — HubService, PendingOp, HubClaim,
|
|
||||||
LeaderboardEntry, GlobalStats, ConflictError, NotFoundError (`a89acfa`)
|
|
||||||
3. **HTTP request helpers** — `doRequest()`, `doJSON()` with bearer auth,
|
|
||||||
error classification (401/404/409), and connection tracking (`ab7ef52`)
|
|
||||||
4. **AutoRegister** — exchange forge token for ak_ hub token via
|
|
||||||
`/auth/forge` endpoint (`21d5f5f`)
|
|
||||||
5. **Write operations** — Register, Heartbeat, ClaimIssue, UpdateStatus,
|
|
||||||
ReleaseClaim, SyncStats (`a6456e2`)
|
|
||||||
6. **Read operations** — IsIssueClaimed, ListClaims, GetLeaderboard,
|
|
||||||
GetGlobalStats (`7a92fe0`)
|
|
||||||
7. **Pending ops queue** — offline-first queue with disk persistence to
|
|
||||||
`hub_pending.json`, drain-on-reconnect (`a567568`)
|
|
||||||
8. **main.go integration** — HubService wired as Wails service with
|
|
||||||
auto-registration at startup (`177ce27`)
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
All operations tested with `httptest.NewServer` mocks covering success,
|
|
||||||
network error, 409 conflict, 401 re-auth, and pending ops persist/reload
|
|
||||||
scenarios. Hub test file: `internal/bugseti/hub_test.go`.
|
|
||||||
|
|
||||||
### Key files (before extraction)
|
|
||||||
|
|
||||||
- `internal/bugseti/hub.go` — HubService implementation (25 exported methods)
|
|
||||||
- `internal/bugseti/hub_test.go` — comprehensive httptest-based test suite
|
|
||||||
- `internal/bugseti/config.go` — hub config fields and accessors
|
|
||||||
- `cmd/bugseti/main.go` — lifecycle wiring
|
|
||||||
|
|
||||||
### Task 9 (Laravel endpoint)
|
|
||||||
|
|
||||||
The portal-side `/api/bugseti/auth/forge` endpoint (Task 9) lives in the
|
|
||||||
`agentic` repo, not in `core/cli`. It was designed in this plan but
|
|
||||||
implemented separately.
|
|
||||||
|
|
||||||
### Extraction
|
|
||||||
|
|
||||||
BugSETI was extracted to its own repo on 16 Feb 2026 (`8167f66`):
|
|
||||||
`internal/bugseti/` moved to `core/bugseti`, `cmd/bugseti/` moved to
|
|
||||||
`core/bugseti/cmd/`.
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
# CLI Meta-Package Restructure — Completed
|
|
||||||
|
|
||||||
**Completed:** 22 Feb 2026
|
|
||||||
|
|
||||||
## What Was Done
|
|
||||||
|
|
||||||
`pkg/cli` was extracted from `core/go` into its own Go module at `forge.lthn.ai/core/cli`. This made the CLI SDK a first-class, independently versioned package rather than a subdirectory of the Go foundation repo.
|
|
||||||
|
|
||||||
Following the extraction, an ecosystem-wide import path migration updated all consumers from the old path to the new one:
|
|
||||||
|
|
||||||
- Old: `forge.lthn.ai/core/go/pkg/cli`
|
|
||||||
- New: `forge.lthn.ai/core/cli/pkg/cli`
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
- **147+ files** updated across **10 repos**
|
|
||||||
- All repos build clean after migration
|
|
||||||
|
|
||||||
## Repos Migrated
|
|
||||||
|
|
||||||
`core/cli`, `core/go`, `go-devops`, `go-ai`, `go-agentic`, `go-crypt`, `go-rag`, `go-scm`, `go-api`, `go-update`
|
|
||||||
|
|
||||||
## Key Outcomes
|
|
||||||
|
|
||||||
- `forge.lthn.ai/core/cli/pkg/cli` is the single import for all CLI concerns across the ecosystem
|
|
||||||
- Domain repos are insulated from cobra, lipgloss, and bubbletea — only `pkg/cli` imports them
|
|
||||||
- Command registration uses the Core framework lifecycle via `cli.WithCommands()` — no `init()`, no global state
|
|
||||||
- `core/cli` is a thin assembly repo (~2K LOC) with 7 meta packages; all business logic lives in domain repos
|
|
||||||
- Variant binary pattern established: multiple `main.go` files can wire different `WithCommands` sets for targeted binaries (core-ci, core-mlx, core-ops, etc.)
|
|
||||||
- Command migration from the old `core/cli` monolith to domain repos was completed in full (13 command groups moved)
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
# CLI SDK Expansion — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 21 February 2026
|
|
||||||
**Module:** `forge.lthn.ai/core/go/pkg/cli` (later migrated to `forge.lthn.ai/core/cli`)
|
|
||||||
**Status:** Complete — all TUI primitives shipped, then extracted to core/cli
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
Extended `pkg/cli` with charmbracelet TUI primitives so domain repos only
|
|
||||||
import `core/cli` for all CLI concerns. Charmbracelet dependencies (bubbletea,
|
|
||||||
bubbles, lipgloss) are encapsulated behind our own types.
|
|
||||||
|
|
||||||
### Components added
|
|
||||||
|
|
||||||
| Component | File | Purpose |
|
|
||||||
|-----------|------|---------|
|
|
||||||
| RunTUI | `runtui.go` | Escape hatch with `Model`/`Msg`/`Cmd`/`KeyMsg` types |
|
|
||||||
| Spinner | `spinner.go` | Async handle with `Update()`, `Done()`, `Fail()` |
|
|
||||||
| ProgressBar | `progressbar.go` | `Increment()`, `Set()`, `SetMessage()`, `Done()` |
|
|
||||||
| InteractiveList | `list.go` | Keyboard navigation with terminal fallback |
|
|
||||||
| TextInput | `textinput.go` | Placeholder, masking, validation |
|
|
||||||
| Viewport | `viewport.go` | Scrollable content for logs, diffs, docs |
|
|
||||||
| Form (stub) | `form.go` | Interface defined, bufio fallback |
|
|
||||||
| FilePicker (stub) | `filepicker.go` | Interface defined, bufio fallback |
|
|
||||||
| Tabs (stub) | `tabs.go` | Interface defined, simple fallback |
|
|
||||||
|
|
||||||
### Subsequent migration
|
|
||||||
|
|
||||||
On 22 February 2026, `pkg/cli` was extracted from `core/go` into its own
|
|
||||||
module at `forge.lthn.ai/core/cli` and all imports were updated. The TUI
|
|
||||||
primitives now live in the standalone CLI module.
|
|
||||||
|
|
||||||
### Frame upgrade (follow-on)
|
|
||||||
|
|
||||||
The Frame layout system was upgraded to implement `tea.Model` directly on
|
|
||||||
22 February 2026 (in `core/cli`), adding bubbletea lifecycle, `KeyMap` for
|
|
||||||
configurable bindings, `Navigate()`/`Back()` for panel switching, and
|
|
||||||
lipgloss-based HLCRF rendering. This was a separate plan
|
|
||||||
(`frame-bubbletea`) that built on the SDK expansion.
|
|
||||||
|
|
@ -1,50 +0,0 @@
|
||||||
# Core-IDE Job Runner — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 9 February 2026
|
|
||||||
**Module:** `forge.lthn.ai/core/cli` (extracted to `core/ide` repo during monorepo split)
|
|
||||||
**Status:** Complete — all components built, tested, and operational before extraction
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
Autonomous job runner for core-ide that polls Forgejo for actionable pipeline
|
|
||||||
work, executes it via typed handler functions, captures JSONL training data,
|
|
||||||
and supports both headless (server) and desktop (Wails GUI) modes.
|
|
||||||
|
|
||||||
### Key components
|
|
||||||
|
|
||||||
- **`pkg/jobrunner/types.go`** — JobSource, JobHandler, PipelineSignal,
|
|
||||||
ActionResult interfaces and structs
|
|
||||||
- **`pkg/jobrunner/poller.go`** — multi-source poller with configurable
|
|
||||||
interval, ETag-based conditional requests, and idle backoff
|
|
||||||
- **`pkg/jobrunner/journal.go`** — append-only JSONL writer for training data
|
|
||||||
capture (structural signals only, no content)
|
|
||||||
- **`pkg/jobrunner/forgejo/source.go`** — ForgejoSource adapter (evolved from
|
|
||||||
original GitHubSource design to use pkg/forge SDK)
|
|
||||||
- **`pkg/jobrunner/forgejo/signals.go`** — PR/issue state extraction and
|
|
||||||
signal building from Forgejo API responses
|
|
||||||
|
|
||||||
### Handlers
|
|
||||||
|
|
||||||
All six handlers from the design were implemented with tests:
|
|
||||||
|
|
||||||
- `publish_draft` — mark draft PRs as ready when checks pass
|
|
||||||
- `send_fix_command` — comment fix instructions for conflicts/reviews
|
|
||||||
- `resolve_threads` — resolve pre-commit review threads after fix
|
|
||||||
- `enable_auto_merge` — enable auto-merge when all checks pass
|
|
||||||
- `tick_parent` — update epic issue checklist when child PR merges
|
|
||||||
- `dispatch` — SCP ticket delivery to agent machines via SSH (added beyond
|
|
||||||
original design)
|
|
||||||
|
|
||||||
### Headless / Desktop mode
|
|
||||||
|
|
||||||
- `hasDisplay()` detection for Linux/macOS/Windows
|
|
||||||
- `--headless` / `--desktop` CLI flag overrides
|
|
||||||
- Headless: poller + MCP bridge, signal handling, systemd-ready
|
|
||||||
- Desktop: Wails GUI with system tray, optional poller toggle
|
|
||||||
|
|
||||||
### Extraction
|
|
||||||
|
|
||||||
Code was fully operational and then extracted during the Feb 2026 monorepo
|
|
||||||
split (`abe74a1`). `pkg/jobrunner/` moved to `core/go`, `cmd/core-ide/` and
|
|
||||||
`internal/core-ide/` moved to `core/ide`. The agentci dispatch system
|
|
||||||
(`d9f3b72` through `886c67e`) built on top of the jobrunner before extraction.
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
# Frame Bubbletea Upgrade — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 22 February 2026
|
|
||||||
**Module:** `forge.lthn.ai/core/cli`
|
|
||||||
**Status:** Complete — Frame implements tea.Model with full bubbletea lifecycle
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
Upgraded the Frame layout system from a static HLCRF renderer to a full
|
|
||||||
bubbletea `tea.Model` with lifecycle management, keyboard handling, and
|
|
||||||
panel navigation.
|
|
||||||
|
|
||||||
### Key changes
|
|
||||||
|
|
||||||
- **Frame implements `tea.Model`** — `Init()`, `Update()`, `View()` lifecycle
|
|
||||||
- **`KeyMap`** — configurable keybindings with default set (quit, navigate,
|
|
||||||
help, focus cycling)
|
|
||||||
- **`Navigate(name)` / `Back()`** — panel switching with history stack
|
|
||||||
- **Focus management** — Tab/Shift-Tab cycles focus between visible models
|
|
||||||
- **lipgloss layout** — HLCRF regions (Header, Left, Content, Right, Footer)
|
|
||||||
rendered with lipgloss instead of raw ANSI
|
|
||||||
- **`FrameModel` interface** — models register with `Frame.Header()`,
|
|
||||||
`.Content()`, `.Footer()` etc., receiving focus/blur/resize messages
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
Navigate/Back stack tests, focus cycling, key dispatch, resize propagation.
|
|
||||||
All passing with `-race`.
|
|
||||||
|
|
||||||
### Dependencies
|
|
||||||
|
|
||||||
- `github.com/charmbracelet/bubbletea`
|
|
||||||
- `github.com/charmbracelet/lipgloss`
|
|
||||||
|
|
||||||
### Consumer
|
|
||||||
|
|
||||||
`go-blockchain/cmd/chain/` is the first consumer — TUI dashboard uses
|
|
||||||
Frame with StatusModel (header), ExplorerModel (content), KeyHintsModel
|
|
||||||
(footer).
|
|
||||||
|
|
@ -1,57 +0,0 @@
|
||||||
# go-api — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 21 February 2026
|
|
||||||
**Module:** `forge.lthn.ai/core/go-api`
|
|
||||||
**Status:** Phases 1–3 complete, 176 tests passing
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
### Phase 1 — Core Framework (20 Feb 2026)
|
|
||||||
|
|
||||||
Gin-based HTTP engine with extensible middleware via `With*()` options. Key components:
|
|
||||||
|
|
||||||
- `RouteGroup` / `StreamGroup` interfaces — subsystems register their own endpoints
|
|
||||||
- `Response[T]` envelope — `OK()`, `Fail()`, `Paginated()` generics
|
|
||||||
- `Engine` — `New()`, `Register()`, `Handler()`, `Serve()` with graceful shutdown
|
|
||||||
- Bearer auth, request ID, and CORS middleware
|
|
||||||
- WebSocket endpoint wrapping a `go-ws` Hub
|
|
||||||
- Swagger UI at `/swagger/` with runtime spec serving
|
|
||||||
- `/health` endpoint always available without auth
|
|
||||||
- First integration proof in `go-ml/api/` (3 endpoints, 12 tests)
|
|
||||||
|
|
||||||
### Phase 2 — Gin Plugin Stack (20–21 Feb 2026)
|
|
||||||
|
|
||||||
17 middleware plugins added across four waves, all as drop-in `With*()` options:
|
|
||||||
|
|
||||||
| Wave | Plugins |
|
|
||||||
|------|---------|
|
|
||||||
| 1 — Gateway hardening | Authentik (OIDC + forward auth), secure headers, structured slog, timeouts, gzip, static files |
|
|
||||||
| 2 — Performance + auth | Brotli compression, in-memory response cache, server-side sessions, Casbin RBAC |
|
|
||||||
| 3 — Network + streaming | HTTP signature verification, SSE broker, reverse proxy detection, i18n locale, GraphQL |
|
|
||||||
| 4 — Observability | pprof, expvar, OpenTelemetry distributed tracing |
|
|
||||||
|
|
||||||
### Phase 3 — OpenAPI + SDK Codegen (21 Feb 2026)
|
|
||||||
|
|
||||||
Runtime spec generation (not swaggo annotations — incompatible with dynamic RouteGroups and `Response[T]` generics):
|
|
||||||
|
|
||||||
- `DescribableGroup` interface — opt-in OpenAPI metadata for route groups
|
|
||||||
- `ToolBridge` — converts MCP tool descriptors into `POST /{tool_name}` REST endpoints
|
|
||||||
- `SpecBuilder` — assembles full OpenAPI 3.1 JSON from registered groups at runtime
|
|
||||||
- Spec export to JSON and YAML (`core api spec`)
|
|
||||||
- SDK codegen wrapper for openapi-generator-cli, 11 languages (`core api sdk --lang go`)
|
|
||||||
- `go-ai` `mcp/registry.go` — generic `addToolRecorded[In,Out]` captures types in closures
|
|
||||||
- `go-ai` `mcp/bridge.go` — `BridgeToAPI()` populates ToolBridge from MCP tool registry
|
|
||||||
- CLI commands: `core api spec`, `core api sdk` (in `core/cli` dev branch)
|
|
||||||
|
|
||||||
## Key Outcomes
|
|
||||||
|
|
||||||
- **176 tests** across go-api (143), go-ai bridge (10), and CLI commands (4), all passing
|
|
||||||
- Zero internal ecosystem dependencies — subsystems import go-api, not the reverse
|
|
||||||
- Authentik (OIDC) and bearer token auth coexist; Casbin adds RBAC on top
|
|
||||||
- Four-protocol access pattern established: REST, GraphQL, WebSocket, MCP — same handlers
|
|
||||||
|
|
||||||
## Known Limitations
|
|
||||||
|
|
||||||
- Subsystem MCP tools registered via `mcp.AddTool` directly are excluded from the REST bridge (only the 10 built-in tools appear). Fix: pass `*Service` to `RegisterTools` instead of `*mcp.Server`.
|
|
||||||
- `structSchema` reflection handles flat structs only; nested structs are not recursed.
|
|
||||||
- `core api spec` currently emits a spec with only `/health`; full MCP wiring into the CLI command is pending.
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
# MCP Integration — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 2026-02-05
|
|
||||||
**Plan:** `docs/plans/2026-02-05-mcp-integration.md`
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
### RAG Tools (`pkg/mcp/tools_rag.go`)
|
|
||||||
Three MCP tools added to the existing `pkg/mcp` server:
|
|
||||||
- `rag_query` — semantic search against Qdrant vector DB
|
|
||||||
- `rag_ingest` — ingest a file or directory into a named collection
|
|
||||||
- `rag_collections` — list available Qdrant collections (with optional stats)
|
|
||||||
|
|
||||||
### Metrics Tools (`pkg/mcp/tools_metrics.go`)
|
|
||||||
Two MCP tools for agent activity tracking:
|
|
||||||
- `metrics_record` — write a typed event (agent_id, repo, arbitrary data) to JSONL storage
|
|
||||||
- `metrics_query` — query events with aggregation by type, repo, and agent; supports human-friendly duration strings (7d, 24h)
|
|
||||||
|
|
||||||
Also added `parseDuration()` helper for "Nd"/"Nh"/"Nm" duration strings.
|
|
||||||
|
|
||||||
### `core mcp serve` Command (`internal/cmd/mcpcmd/cmd_mcp.go`)
|
|
||||||
New CLI sub-command registered via `cli.WithCommands()` (not `init()`).
|
|
||||||
- Runs `pkg/mcp` server over stdio by default
|
|
||||||
- TCP mode via `MCP_ADDR=:9000` environment variable
|
|
||||||
- `--workspace` flag to restrict file operations to a directory
|
|
||||||
|
|
||||||
Registered in the full CLI variant. i18n strings added for all user-facing text.
|
|
||||||
|
|
||||||
### Plugin Configuration
|
|
||||||
`.mcp.json` created for the `agentic-flows` Claude Code plugin, pointing to `core mcp serve`. Exposes all 15 tools to Claude Code agents via the `core-cli` MCP server name.
|
|
||||||
|
|
||||||
## Key Outcomes
|
|
||||||
|
|
||||||
- `core mcp serve` is the single entry point for all MCP tooling (file ops, RAG, metrics, language detection, process management, WebSocket, webview/CDP)
|
|
||||||
- MCP command moved to `go-ai/cmd/mcpcmd/` in final form; the plan's `internal/cmd/mcpcmd/` path reflects the pre-extraction location
|
|
||||||
- Registration pattern updated from `init()` + `RegisterCommands()` to `cli.WithCommands()` lifecycle hooks
|
|
||||||
- Services required at runtime: Qdrant (localhost:6333), Ollama with nomic-embed-text (localhost:11434)
|
|
||||||
|
|
@ -1,62 +0,0 @@
|
||||||
# Q/K Bone Orientation — Completion Summary
|
|
||||||
|
|
||||||
**Completed:** 23 February 2026
|
|
||||||
**Repos:** go-inference, go-mlx, go-ml, LEM
|
|
||||||
**Status:** All 7 tasks complete, 14 files changed (+917 lines), all tests passing
|
|
||||||
|
|
||||||
## What Was Built
|
|
||||||
|
|
||||||
### go-inference — AttentionSnapshot types (Task 1)
|
|
||||||
|
|
||||||
`AttentionSnapshot` struct and `AttentionInspector` optional interface. Backends expose attention data via type assertion — no breaking changes to `TextModel`.
|
|
||||||
|
|
||||||
### go-mlx — KV cache extraction (Task 2)
|
|
||||||
|
|
||||||
`InspectAttention` on `metalAdapter` runs a single prefill pass and extracts post-RoPE K vectors from each layer's KV cache. Tested against real Gemma3-1B (26 layers, 1 KV head via GQA, 256 head dim).
|
|
||||||
|
|
||||||
### go-ml — Adapter pass-through (Task 3)
|
|
||||||
|
|
||||||
`InspectAttention` on `InferenceAdapter` type-asserts the underlying `TextModel` to `AttentionInspector`. Returns clear error for unsupported backends.
|
|
||||||
|
|
||||||
### LEM — Analysis engine (Task 4)
|
|
||||||
|
|
||||||
Pure Go CPU math in `pkg/lem/attention.go`. Computes 5 BO metrics from raw K tensors:
|
|
||||||
|
|
||||||
- **Mean Coherence** — pairwise cosine similarity of K vectors within each layer
|
|
||||||
- **Cross-Layer Alignment** — cosine similarity of mean K vectors between adjacent layers
|
|
||||||
- **Head Entropy** — normalised Shannon entropy of K vector magnitudes across positions
|
|
||||||
- **Phase-Lock Score** — fraction of head pairs above coherence threshold (0.7)
|
|
||||||
- **Joint Collapse Count** — layers where cross-alignment drops below threshold (0.5)
|
|
||||||
|
|
||||||
Composite score: 30% coherence + 25% cross-alignment + 20% phase-lock + 15% entropy + 10% joint stability → 0-100 scale.
|
|
||||||
|
|
||||||
### LEM — CLI command (Task 5)
|
|
||||||
|
|
||||||
`lem score attention -model <path> -prompt <text> [-json]` loads a model, runs InspectAttention, and prints BO metrics.
|
|
||||||
|
|
||||||
### LEM — Distill integration (Task 6)
|
|
||||||
|
|
||||||
Opt-in attention scoring in the distill pipeline. Gated behind `scorer.attention: true` and `scorer.attention_min_score` in ai.yaml. Costs one extra prefill per probe.
|
|
||||||
|
|
||||||
### LEM — Feature vectors (Task 7)
|
|
||||||
|
|
||||||
19D full feature vector: 6D grammar + 8D heuristic + 5D attention (`mean_coherence`, `cross_alignment`, `head_entropy`, `phase_lock`, `joint_stability`). Ready for Poindexter KDTree spatial indexing.
|
|
||||||
|
|
||||||
## Key Decisions
|
|
||||||
|
|
||||||
- **Optional interface** — `AttentionInspector` via type assertion, not added to `TextModel`
|
|
||||||
- **Named `BOResult`** — avoids collision with `metal.AttentionResult` in go-mlx
|
|
||||||
- **Opt-in for distill** — extra prefill per probe is expensive, off by default
|
|
||||||
- **Pure Go analysis** — zero CGO deps in the analysis engine; GPU data extracted once via `.Floats()`
|
|
||||||
|
|
||||||
## Commits
|
|
||||||
|
|
||||||
| Repo | SHA | Message |
|
|
||||||
|------|-----|---------|
|
|
||||||
| go-inference | `0f7263f` | feat: add AttentionInspector optional interface |
|
|
||||||
| go-mlx | `c2177f7` | feat: implement AttentionInspector via KV cache extraction |
|
|
||||||
| go-ml | `45e9fed` | feat: add InspectAttention pass-through |
|
|
||||||
| LEM | `28309b2` | feat: add Q/K Bone Orientation analysis engine |
|
|
||||||
| LEM | `e333192` | feat: add 'lem score attention' CLI |
|
|
||||||
| LEM | `fbc636e` | feat: integrate attention scoring into distill pipeline |
|
|
||||||
| LEM | `b621baa` | feat: add 19D full feature vector |
|
|
||||||
Loading…
Add table
Reference in a new issue