core/go
8
0
Fork 0
generated from lthn/LEM
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
go/docs/ecosystem.md
Snider bcf2d3be48 docs: remove dispatch queue, add ecosystem map and docs index 
Delete TODO.md (satellite dispatch queue — all tasks complete) and
FINDINGS.md (go-i18n research — absorbed into go-i18n docs).
Add docs/ecosystem.md mapping all 19 satellite Go packages with
dependency graph and Forge SSH URLs. Rewrite docs/index.md as a
proper documentation hub with section organisation.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 15:10:56 +00:00

457 lines
15 KiB
Markdown
Raw Blame History

# Core Go Ecosystem
The Core Go ecosystem is a set of 19 standalone Go modules that form the infrastructure backbone for the host-uk platform and the Lethean network. All modules are hosted under the `forge.lthn.ai/core/` organisation. Each module has its own repository, independent versioning, and a `docs/` directory.
The CLI framework documented in the rest of this site (`forge.lthn.ai/core/cli`) is one node in this graph. The satellite packages listed here are separate repositories that the CLI imports or that stand alone as libraries.
---
## Module Index
| Package | Module Path | Managed By |
|---------|-------------|-----------|
| [go-inference](#go-inference) | `forge.lthn.ai/core/go-inference` | Virgil |
| [go-mlx](#go-mlx) | `forge.lthn.ai/core/go-mlx` | Virgil |
| [go-rocm](#go-rocm) | `forge.lthn.ai/core/go-rocm` | Charon |
| [go-ml](#go-ml) | `forge.lthn.ai/core/go-ml` | Virgil |
| [go-ai](#go-ai) | `forge.lthn.ai/core/go-ai` | Virgil |
| [go-agentic](#go-agentic) | `forge.lthn.ai/core/go-agentic` | Charon |
| [go-rag](#go-rag) | `forge.lthn.ai/core/go-rag` | Charon |
| [go-i18n](#go-i18n) | `forge.lthn.ai/core/go-i18n` | Virgil |
| [go-html](#go-html) | `forge.lthn.ai/core/go-html` | Charon |
| [go-crypt](#go-crypt) | `forge.lthn.ai/core/go-crypt` | Virgil |
| [go-scm](#go-scm) | `forge.lthn.ai/core/go-scm` | Charon |
| [go-p2p](#go-p2p) | `forge.lthn.ai/core/go-p2p` | Charon |
| [go-devops](#go-devops) | `forge.lthn.ai/core/go-devops` | Virgil |
| [go-help](#go-help) | `forge.lthn.ai/core/go-help` | Charon |
| [go-ratelimit](#go-ratelimit) | `forge.lthn.ai/core/go-ratelimit` | Charon |
| [go-session](#go-session) | `forge.lthn.ai/core/go-session` | Charon |
| [go-store](#go-store) | `forge.lthn.ai/core/go-store` | Charon |
| [go-ws](#go-ws) | `forge.lthn.ai/core/go-ws` | Charon |
| [go-webview](#go-webview) | `forge.lthn.ai/core/go-webview` | Charon |
---
## Dependency Graph
The graph below shows import relationships. An arrow `A → B` means A imports B.
```
go-inference (no dependencies — foundation contract)
↑
├── go-mlx (CGO, Apple Silicon Metal GPU)
├── go-rocm (AMD ROCm, llama-server subprocess)
└── go-ml (scoring engine, backends, orchestrator)
↑
└── go-ai (MCP hub, 49 tools)
↑
└── go-agentic (service lifecycle, allowances)
go-rag (Qdrant + Ollama, standalone)
↑
└── go-ai
go-i18n (grammar engine, standalone; Phase 2a imports go-mlx)
go-crypt (standalone)
↑
├── go-p2p (UEPS wire protocol)
└── go-scm (AgentCI dispatch)
go-store (SQLite KV, standalone)
↑
├── go-ratelimit (sliding window limiter)
├── go-session (transcript parser)
└── go-agentic
go-ws (WebSocket hub, standalone)
↑
└── go-ai
go-webview (CDP client, standalone)
↑
└── go-ai
go-html (DOM compositor, standalone)
go-help (help catalogue, standalone)
go-devops (Ansible, build, infrastructure — imports go-scm)
```
The CLI framework (`forge.lthn.ai/core/cli`) has internal equivalents of several of these packages (`pkg/rag`, `pkg/ws`, `pkg/webview`, `pkg/i18n`) that were developed in parallel. The satellite packages are the canonical standalone versions intended for use outside the CLI binary.
---
## Package Descriptions
### go-inference
**Module:** `forge.lthn.ai/core/go-inference`
Zero-dependency interface package that defines the common contract for all inference backends in the ecosystem:
- `TextModel` — the top-level model interface (`Generate`, `Stream`, `Close`)
- `Backend` — hardware/runtime abstraction (Metal, ROCm, CPU, remote)
- `Token` — streaming token type with metadata
No concrete implementations live here. Any package that needs to call inference without depending on a specific hardware library imports `go-inference` and receives an implementation at runtime.
---
### go-mlx
**Module:** `forge.lthn.ai/core/go-mlx`
Native Metal GPU inference for Apple Silicon using CGO bindings to `mlx-c` (the C API for Apple's MLX framework). Implements the `go-inference` interfaces.
Build requirements:
- macOS 13+ (Ventura) on Apple Silicon
- `mlx-c` installed (`brew install mlx`)
- CGO enabled: `CGO_CFLAGS` and `CGO_LDFLAGS` must reference the mlx-c headers and library
Features:
- Loads GGUF and MLX-format models
- Streaming token generation directly on GPU
- Quantised model support (Q4, Q8)
- Phase 4 backend abstraction in progress — will allow hot-swapping backends at runtime
Local path: `/Users/snider/Code/go-mlx`
---
### go-rocm
**Module:** `forge.lthn.ai/core/go-rocm`
AMD ROCm GPU inference for Linux. Rather than using CGO, this package manages a `llama-server` subprocess (from llama.cpp) compiled with ROCm support and communicates over its HTTP API.
Features:
- Subprocess lifecycle management (start, health-check, restart on crash)
- OpenAI-compatible HTTP client wrapping llama-server's API
- Implements `go-inference` interfaces
- Targeted at the homelab RX 7800 XT running Ubuntu 24.04
Managed by Charon (Linux homelab).
---
### go-ml
**Module:** `forge.lthn.ai/core/go-ml`
Scoring engine, backend registry, and agent orchestration layer. The hub that connects models from `go-mlx`, `go-rocm`, and future backends into a unified interface.
Features:
- Backend registry: register multiple inference backends, select by capability
- Scoring pipeline: evaluate model outputs against rubrics
- Agent orchestrator: coordinate multi-step inference tasks
- ~3.5K LOC
---
### go-ai
**Module:** `forge.lthn.ai/core/go-ai`
MCP (Model Context Protocol) server hub with 49 registered tools. Acts as the primary facade for AI capabilities in the ecosystem.
Features:
- 49 MCP tools covering file operations, RAG, metrics, process management, WebSocket, and CDP/webview
- Imports `go-ml`, `go-rag`, `go-mlx`
- Can run as stdio MCP server or TCP MCP server
- AI usage metrics recorded to JSONL
Run the MCP server:
```bash
# stdio (for Claude Desktop / Claude Code)
core mcp serve
# TCP
MCP_ADDR=:9000 core mcp serve
```
---
### go-agentic
**Module:** `forge.lthn.ai/core/go-agentic`
Service lifecycle and allowance management for autonomous agents. Handles:
- Agent session tracking and state persistence
- Allowance system: budget constraints on tool calls, token usage, and wall-clock time
- Integration with `go-store` for persistence
- REST client for the PHP `core-agentic` backend
Managed by Charon.
---
### go-rag
**Module:** `forge.lthn.ai/core/go-rag`
Retrieval-Augmented Generation pipeline using Qdrant for vector storage and Ollama for embeddings.
Features:
- `ChunkMarkdown`: semantic splitting by H2 headers and paragraphs with overlap
- `Ingest`: crawl a directory of Markdown files, embed, and store in Qdrant
- `Query`: semantic search returning ranked `QueryResult` slices
- `FormatResultsContext`: formats results as XML tags for LLM prompt injection
- Clients: `QdrantClient` and `OllamaClient` wrapping their respective Go SDKs
Managed by Charon.
---
### go-i18n
**Module:** `forge.lthn.ai/core/go-i18n`
Grammar engine for natural-language generation. Goes beyond key-value lookup tables to handle pluralisation, verb conjugation, past tense, gerunds, and semantic sentence construction ("Subject verbed object").
Features:
- `T(key, args...)` — main translation function
- `S(noun, value)` — semantic subject with grammatical context
- Language rules defined in JSON; algorithmic fallbacks for irregular verbs
- **GrammarImprint**: a linguistic hash (reversal of the grammar engine) used as a semantic fingerprint — part of the Lethean identity verification stack
- Phase 2a (imports `go-mlx` for language model-assisted reversal) currently blocked on `go-mlx` Phase 4
Local path: `/Users/snider/Code/go-i18n`
---
### go-html
**Module:** `forge.lthn.ai/core/go-html`
HLCRF DOM compositor — a programmatic HTML/DOM construction library targeting both server-side rendering and WASM (browser).
HLCRF stands for Header, Left, Content, Right, Footer — the region layout model used throughout the CLI's terminal UI and web rendering layer.
Features:
- Composable region-based layout (mirrors the terminal `Composite` in `pkg/cli`)
- WASM build target: runs in the browser without JavaScript
- Used by the LEM Chat UI and web SDK generation
Managed by Charon.
---
### go-crypt
**Module:** `forge.lthn.ai/core/go-crypt`
Cryptographic primitives, authentication, and trust policy enforcement.
Features:
- Password hashing (Argon2id with tuned parameters)
- Symmetric encryption (ChaCha20-Poly1305, AES-GCM)
- Key derivation (HKDF, Scrypt)
- OpenPGP challenge-response authentication
- Trust policies: define and evaluate access rules
- Foundation for the UEPS (User-controlled Encryption Policy System) wire protocol in `go-p2p`
---
### go-scm
**Module:** `forge.lthn.ai/core/go-scm`
Source control management and CI integration, including the AgentCI dispatch system.
Features:
- Forgejo and Gitea API clients (typed wrappers)
- GitHub integration via the `gh` CLI
- `AgentCI`: dispatches AI work items to agent runners over SSH using Charm stack libraries (`soft-serve`, `keygen`, `melt`, `wishlist`)
- PR lifecycle management: create, review, merge, label
- JSONL job journal for audit trails
Managed by Charon.
---
### go-p2p
**Module:** `forge.lthn.ai/core/go-p2p`
Peer-to-peer mesh networking implementing the UEPS (User-controlled Encryption Policy System) wire protocol.
Features:
- UEPS: consent-gated TLV frames with Ed25519 consent tokens and an Intent-Broker
- Peer discovery and mesh routing
- Encrypted relay transport
- Integration with `go-crypt` for all cryptographic operations
This is a core component of the Lethean Web3 network layer.
Managed by Charon (Linux homelab).
---
### go-devops
**Module:** `forge.lthn.ai/core/go-devops`
Infrastructure automation, build tooling, and release pipeline utilities, intended as a standalone library form of what the Core CLI provides as commands.
Features:
- Ansible-lite engine (native Go SSH playbook execution)
- LinuxKit image building and VM lifecycle
- Multi-target binary build and release
- Integration with `go-scm` for repository operations
---
### go-help
**Module:** `forge.lthn.ai/core/go-help`
Embedded documentation catalogue with full-text search and an optional HTTP server for serving help content.
Features:
- YAML-frontmatter Markdown topic parsing
- In-memory reverse index with title/heading/body scoring
- Snippet extraction with keyword highlighting
- `HTTP server` mode: serve the catalogue as a documentation site
- Used by the `core pkg search` command and the `pkg/help` package inside the CLI
Managed by Charon.
---
### go-ratelimit
**Module:** `forge.lthn.ai/core/go-ratelimit`
Sliding-window rate limiter with a SQLite persistence backend.
Features:
- Token bucket and sliding-window algorithms
- SQLite backend via `go-store` for durable rate state across restarts
- HTTP middleware helper
- Used by `go-ai` and `go-agentic` to enforce per-agent API quotas
Managed by Charon.
---
### go-session
**Module:** `forge.lthn.ai/core/go-session`
Claude Code JSONL transcript parser and visualisation toolkit (standalone version of `pkg/session` inside the CLI).
Features:
- `ParseTranscript(path)`: reads `.jsonl` session files and reconstructs tool use timelines
- `ListSessions(dir)`: scans a Claude projects directory for session files
- `Search(dir, query)`: full-text search across sessions
- `RenderHTML(sess, path)`: single-file HTML visualisation
- `RenderMP4(sess, path)`: terminal video replay via VHS
Managed by Charon.
---
### go-store
**Module:** `forge.lthn.ai/core/go-store`
SQLite-backed key-value store with reactive change notification.
Features:
- `Get`, `Set`, `Delete`, `List` over typed keys
- `Watch(key, handler)`: register a callback that fires on change
- `OnChange(handler)`: subscribe to all changes
- Used by `go-ratelimit`, `go-session`, and `go-agentic` for lightweight persistence
Managed by Charon.
---
### go-ws
**Module:** `forge.lthn.ai/core/go-ws`
WebSocket hub with channel-based subscriptions and an optional Redis pub/sub bridge for multi-instance deployments.
Features:
- Hub pattern: central registry of connected clients
- Channel routing: `SendToChannel(topic, msg)` delivers only to subscribers
- Redis bridge: publish messages from one instance, receive on all
- HTTP handler: `hub.Handler()` for embedding in any Go HTTP server
- `SendProcessOutput(id, line)`: convenience method for streaming process logs
Managed by Charon.
---
### go-webview
**Module:** `forge.lthn.ai/core/go-webview`
Chrome DevTools Protocol (CDP) client for browser automation, testing, and AI-driven web interaction (standalone version of `pkg/webview` inside the CLI).
Features:
- Navigation, click, type, screenshot
- `Evaluate(script)`: arbitrary JavaScript execution with result capture
- Console capture and filtering
- Angular-aware helpers: `WaitForAngular()`, `GetNgModel(selector)`
- `ActionSequence`: chain interactions into a single call
- Used by `go-ai` to expose browser tools to MCP agents
Managed by Charon.
---
## Forge Repository Paths
All repositories are hosted at `forge.lthn.ai` (Forgejo). SSH access uses port 2223:
```
ssh://git@forge.lthn.ai:2223/core/go-inference.git
ssh://git@forge.lthn.ai:2223/core/go-mlx.git
ssh://git@forge.lthn.ai:2223/core/go-rocm.git
ssh://git@forge.lthn.ai:2223/core/go-ml.git
ssh://git@forge.lthn.ai:2223/core/go-ai.git
ssh://git@forge.lthn.ai:2223/core/go-agentic.git
ssh://git@forge.lthn.ai:2223/core/go-rag.git
ssh://git@forge.lthn.ai:2223/core/go-i18n.git
ssh://git@forge.lthn.ai:2223/core/go-html.git
ssh://git@forge.lthn.ai:2223/core/go-crypt.git
ssh://git@forge.lthn.ai:2223/core/go-scm.git
ssh://git@forge.lthn.ai:2223/core/go-p2p.git
ssh://git@forge.lthn.ai:2223/core/go-devops.git
ssh://git@forge.lthn.ai:2223/core/go-help.git
ssh://git@forge.lthn.ai:2223/core/go-ratelimit.git
ssh://git@forge.lthn.ai:2223/core/go-session.git
ssh://git@forge.lthn.ai:2223/core/go-store.git
ssh://git@forge.lthn.ai:2223/core/go-ws.git
ssh://git@forge.lthn.ai:2223/core/go-webview.git
```
HTTPS authentication is not available on Forge. Always use SSH remotes.
---
## Go Workspace Setup
The satellite packages can be used together in a Go workspace. After cloning the repositories you need:
```bash
go work init
go work use ./go-inference ./go-mlx ./go-rag ./go-ai # add as needed
go work sync
```
The CLI repository already uses a Go workspace that includes `cmd/core-gui`, `cmd/bugseti`, and `cmd/examples/*`.
---
## See Also
- [index.md](index.md) — Main documentation hub
- [getting-started.md](getting-started.md) — CLI installation
- [configuration.md](configuration.md) — `repos.yaml` registry format
Reference in a new issue View git blame Copy permalink