docs: sync verified repo documentation to core.help
Some checks failed
Build and Deploy / deploy (push) Failing after 7s
Some checks failed
Build and Deploy / deploy (push) Failing after 7s
Update all 29 Go package pages, 4 tool pages (agent, mcp, ide, lint), TypeScript, and Go framework index with rich content from individual repo docs/. Add lint to Tools nav. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
5b5beaf36f
commit
e4f3c3e731
36 changed files with 4075 additions and 519 deletions
146
docs/go/index.md
146
docs/go/index.md
|
|
@ -1,98 +1,96 @@
|
|||
# Core Go
|
||||
---
|
||||
title: Core Go Framework
|
||||
description: Dependency injection and service lifecycle framework for Go.
|
||||
---
|
||||
|
||||
Core is a Go framework for the host-uk ecosystem - build, release, and deploy Go, Wails, PHP, and container workloads.
|
||||
# Core Go Framework
|
||||
|
||||
Core (`forge.lthn.ai/core/go`) is a dependency injection and service lifecycle framework for Go. It provides a typed service registry, lifecycle hooks, and a message-passing bus for decoupled communication between services.
|
||||
|
||||
This is the foundation layer of the ecosystem. It has no CLI, no GUI, and minimal dependencies.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Via Go (recommended)
|
||||
go install github.com/host-uk/core/cmd/core@latest
|
||||
|
||||
# Or download binary from releases
|
||||
curl -Lo core https://github.com/host-uk/core/releases/latest/download/core-$(go env GOOS)-$(go env GOARCH)
|
||||
chmod +x core && sudo mv core /usr/local/bin/
|
||||
|
||||
# Verify
|
||||
core doctor
|
||||
go get forge.lthn.ai/core/go
|
||||
```
|
||||
|
||||
See [Getting Started](getting-started.md) for all installation options including building from source.
|
||||
Requires Go 1.26 or later.
|
||||
|
||||
## Command Reference
|
||||
## What It Does
|
||||
|
||||
See [CLI](/build/cli/) for full command documentation.
|
||||
Core solves three problems that every non-trivial Go application eventually faces:
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| [go](/build/cli/go/) | Go development (test, fmt, lint, cov) |
|
||||
| [php](/build/cli/php/) | Laravel/PHP development |
|
||||
| [build](/build/cli/build/) | Build Go, Wails, Docker, LinuxKit projects |
|
||||
| [ci](/build/cli/ci/) | Publish releases (dry-run by default) |
|
||||
| [sdk](/build/cli/sdk/) | SDK generation and validation |
|
||||
| [dev](/build/cli/dev/) | Multi-repo workflow + dev environment |
|
||||
| [pkg](/build/cli/pkg/) | Package search and install |
|
||||
| [vm](/build/cli/vm/) | LinuxKit VM management |
|
||||
| [docs](/build/cli/docs/) | Documentation management |
|
||||
| [setup](/build/cli/setup/) | Clone repos from registry |
|
||||
| [doctor](/build/cli/doctor/) | Check development environment |
|
||||
1. **Service wiring** -- how do you register, retrieve, and type-check services without import cycles?
|
||||
2. **Lifecycle management** -- how do you start and stop services in the right order?
|
||||
3. **Decoupled communication** -- how do services talk to each other without knowing each other's types?
|
||||
|
||||
## Quick Start
|
||||
## Packages
|
||||
|
||||
```bash
|
||||
# Go development
|
||||
core go test # Run tests
|
||||
core go test --coverage # With coverage
|
||||
core go fmt # Format code
|
||||
core go lint # Lint code
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| [`pkg/core`](services.md) | DI container, service registry, lifecycle, message bus |
|
||||
| `pkg/log` | Structured logger service with Core integration |
|
||||
|
||||
# Build
|
||||
core build # Auto-detect and build
|
||||
core build --targets linux/amd64,darwin/arm64
|
||||
## Quick Example
|
||||
|
||||
# Release (dry-run by default)
|
||||
core ci # Preview release
|
||||
core ci --we-are-go-for-launch # Actually publish
|
||||
```go
|
||||
package main
|
||||
|
||||
# Multi-repo workflow
|
||||
core dev work # Status + commit + push
|
||||
core dev work --status # Just show status
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
# PHP development
|
||||
core php dev # Start dev environment
|
||||
core php test # Run tests
|
||||
"forge.lthn.ai/core/go/pkg/core"
|
||||
"forge.lthn.ai/core/go/pkg/log"
|
||||
)
|
||||
|
||||
func main() {
|
||||
c, err := core.New(
|
||||
core.WithName("log", log.NewService(log.Options{Level: log.LevelInfo})),
|
||||
core.WithServiceLock(), // Prevent late registration
|
||||
)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Start all services
|
||||
if err := c.ServiceStartup(context.Background(), nil); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Type-safe retrieval
|
||||
logger, err := core.ServiceFor[*log.Service](c, "log")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Println("Log level:", logger.Level())
|
||||
|
||||
// Shut down (reverse order)
|
||||
_ = c.ServiceShutdown(context.Background())
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
## Documentation
|
||||
|
||||
Core uses `.core/` directory for project configuration:
|
||||
| Page | Covers |
|
||||
|------|--------|
|
||||
| [Getting Started](getting-started.md) | Creating a Core app, registering your first service |
|
||||
| [Services](services.md) | Service registration, `ServiceRuntime`, factory pattern |
|
||||
| [Lifecycle](lifecycle.md) | `Startable`/`Stoppable` interfaces, startup/shutdown order |
|
||||
| [Messaging](messaging.md) | ACTION, QUERY, PERFORM -- the message bus |
|
||||
| [Configuration](configuration.md) | `WithService`, `WithName`, `WithAssets`, `WithServiceLock` options |
|
||||
| [Testing](testing.md) | Test naming conventions, test helpers, fuzz testing |
|
||||
| [Errors](errors.md) | `E()` helper, `Error` struct, unwrapping |
|
||||
|
||||
```
|
||||
.core/
|
||||
├── release.yaml # Release targets and settings
|
||||
├── build.yaml # Build configuration (optional)
|
||||
└── linuxkit/ # LinuxKit templates
|
||||
```
|
||||
## Dependencies
|
||||
|
||||
And `repos.yaml` in workspace root for multi-repo management.
|
||||
Core is deliberately minimal:
|
||||
|
||||
## Guides
|
||||
- `forge.lthn.ai/core/go-io` -- abstract storage (local, S3, SFTP, WebDAV)
|
||||
- `forge.lthn.ai/core/go-log` -- structured logging
|
||||
- `github.com/stretchr/testify` -- test assertions (test-only)
|
||||
|
||||
- [Getting Started](getting-started.md) - Installation and first steps
|
||||
- [Workflows](workflows.md) - Common task sequences
|
||||
- [Troubleshooting](troubleshooting.md) - When things go wrong
|
||||
- [Migration](migration.md) - Moving from legacy tools
|
||||
## Licence
|
||||
|
||||
## Reference
|
||||
|
||||
- [Configuration](configuration.md) - All config options
|
||||
- [Glossary](glossary.md) - Term definitions
|
||||
|
||||
## Claude Code Skill
|
||||
|
||||
Install the skill to teach Claude Code how to use the Core CLI:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/host-uk/core/main/.claude/skills/core/install.sh | bash
|
||||
```
|
||||
|
||||
See [skill/](skill/) for details.
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,36 +1,130 @@
|
|||
---
|
||||
title: go-ai Overview
|
||||
description: The AI integration hub for the Lethean Go ecosystem — MCP server, metrics, and facade.
|
||||
---
|
||||
|
||||
# go-ai
|
||||
|
||||
MCP (Model Context Protocol) hub for the Lethean AI stack.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-ai`
|
||||
**Language**: Go 1.26
|
||||
**Licence**: EUPL-1.2
|
||||
|
||||
Exposes 49 tools across file operations, directory management, language detection, RAG vector search, ML inference and scoring, process management, WebSocket streaming, browser automation via Chrome DevTools Protocol, JSONL metrics, and an IDE bridge to the Laravel core-agentic backend. The package is a pure library — the Core CLI (`core mcp serve`) imports it and handles transport selection (stdio, TCP, or Unix socket).
|
||||
go-ai is the **integration hub** for the Lethean AI stack. It imports specialised modules and exposes them as a unified MCP server with IDE bridge support, metrics recording, and a thin AI facade.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
AI Clients (Claude, Cursor, any MCP-capable IDE)
|
||||
| MCP JSON-RPC (stdio / TCP / Unix)
|
||||
v
|
||||
[ go-ai MCP Server ] <-- this module
|
||||
| | |
|
||||
| | +-- ide/ subsystem --> Laravel core-agentic (WebSocket)
|
||||
| +-- go-rag -----------------> Qdrant + Ollama
|
||||
+-- go-ml ---------------------------> inference backends (go-mlx, go-rocm, ...)
|
||||
|
||||
Core CLI (forge.lthn.ai/core/cli) bootstraps and wires everything
|
||||
```
|
||||
|
||||
go-ai is a pure library module. It contains no `main` package. The Core CLI (`core mcp serve`) imports `forge.lthn.ai/core/go-ai/mcp`, constructs a `mcp.Service`, and calls `Run()`.
|
||||
|
||||
## Package Layout
|
||||
|
||||
```
|
||||
go-ai/
|
||||
+-- ai/ # AI facade: RAG queries and JSONL metrics
|
||||
| +-- ai.go # Package documentation and composition overview
|
||||
| +-- rag.go # QueryRAGForTask() with graceful degradation
|
||||
| +-- metrics.go # Event, Record(), ReadEvents(), Summary()
|
||||
|
|
||||
+-- cmd/ # CLI command registrations
|
||||
| +-- daemon/ # core daemon (MCP server lifecycle)
|
||||
| +-- metrics/ # core ai metrics viewer
|
||||
| +-- rag/ # re-exports go-rag CLI commands
|
||||
| +-- security/ # security scanning tools (deps, alerts, secrets, scan, jobs)
|
||||
| +-- lab/ # homelab monitoring dashboard
|
||||
| +-- embed-bench/ # embedding model benchmark utility
|
||||
|
|
||||
+-- docs/ # This documentation
|
||||
```
|
||||
|
||||
The MCP server and all its tool subsystems are provided by the separate `forge.lthn.ai/core/mcp` module. go-ai wires that server together with the `ai/` facade and the CLI command registrations.
|
||||
|
||||
## Imported Modules
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/go-ml` | Inference backends, scoring engine |
|
||||
| `forge.lthn.ai/core/go-rag` | Vector search, embeddings |
|
||||
| `forge.lthn.ai/core/go-inference` | Shared TextModel/Backend interfaces |
|
||||
| `forge.lthn.ai/core/go-process` | Process lifecycle management |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logging with security levels |
|
||||
| `forge.lthn.ai/core/go-io` | Sandboxed filesystem abstraction |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalisation |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-ai/mcp"
|
||||
go-ai is not run directly. It is consumed by the Core CLI:
|
||||
|
||||
svc, err := mcp.New(
|
||||
mcp.WithWorkspaceRoot("/path/to/project"),
|
||||
mcp.WithProcessService(ps),
|
||||
)
|
||||
// Run as stdio server (default for AI client subprocess integration)
|
||||
err = svc.Run(ctx)
|
||||
// Or TCP: MCP_ADDR=127.0.0.1:9100 triggers ServeTCP automatically
|
||||
```bash
|
||||
# Start the MCP server on stdio (default)
|
||||
core mcp serve
|
||||
|
||||
# Start on TCP
|
||||
core mcp serve --mcp-transport tcp --mcp-addr 127.0.0.1:9100
|
||||
|
||||
# Run as a background daemon
|
||||
core daemon start
|
||||
|
||||
# View AI metrics
|
||||
core ai metrics --since 7d
|
||||
```
|
||||
|
||||
## Tool Categories
|
||||
## Documentation
|
||||
|
||||
| Category | Tools | Description |
|
||||
|----------|-------|-------------|
|
||||
| File | 8 | Read, write, search, glob, patch files |
|
||||
| Directory | 4 | List, create, move, tree |
|
||||
| Language | 3 | Detect, grammar, translate |
|
||||
| RAG | 5 | Ingest, search, embed, index, stats |
|
||||
| ML | 6 | Generate, score, probe, model management |
|
||||
| Process | 4 | Start, stop, list, logs |
|
||||
| WebSocket | 3 | Connect, send, subscribe |
|
||||
| Browser | 5 | Navigate, click, read, screenshot, evaluate |
|
||||
| Metrics | 3 | Write, query, dashboard |
|
||||
| IDE | 8 | Bridge to core-agentic Laravel backend |
|
||||
| Page | Description |
|
||||
|------|-------------|
|
||||
| [MCP Server](mcp-server.md) | Protocol implementation, transports, tool registration |
|
||||
| [ML Pipeline](ml-pipeline.md) | ML scoring, model management, inference backends |
|
||||
| [RAG Pipeline](rag.md) | Retrieval-augmented generation, vector search |
|
||||
| [Agentic Client](agentic.md) | Security scanning, metrics, CLI commands |
|
||||
| [IDE Bridge](ide-bridge.md) | IDE integration, WebSocket bridge to Laravel |
|
||||
|
||||
## Build and Test
|
||||
|
||||
```bash
|
||||
go test ./... # Run all tests
|
||||
go test -run TestName ./... # Run a single test
|
||||
go test -v -race ./... # Verbose with race detector
|
||||
go build ./... # Verify compilation (library -- no binary)
|
||||
go vet ./... # Vet
|
||||
```
|
||||
|
||||
Tests follow the `_Good`, `_Bad`, `_Ugly` suffix convention:
|
||||
|
||||
- `_Good` -- Happy path, valid input
|
||||
- `_Bad` -- Expected error conditions
|
||||
- `_Ugly` -- Panics and edge cases
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Direct
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (cobra-based command registration) |
|
||||
| `forge.lthn.ai/core/go-api` | API server framework |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalisation strings |
|
||||
| `forge.lthn.ai/core/go-inference` | Shared inference interfaces |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logging |
|
||||
| `forge.lthn.ai/core/go-ml` | ML scoring and inference |
|
||||
| `forge.lthn.ai/core/go-process` | Process lifecycle |
|
||||
| `forge.lthn.ai/core/go-rag` | RAG pipeline |
|
||||
| `github.com/modelcontextprotocol/go-sdk` | MCP Go SDK |
|
||||
| `github.com/gorilla/websocket` | WebSocket client (IDE bridge) |
|
||||
| `github.com/gin-gonic/gin` | HTTP router |
|
||||
|
||||
### Indirect (via go-ml and go-rag)
|
||||
|
||||
`go-mlx`, `go-rocm`, `go-duckdb`, `parquet-go`, `ollama`, `qdrant/go-client`, and the Arrow ecosystem are transitive dependencies not imported directly by go-ai.
|
||||
|
|
|
|||
|
|
@ -1,15 +1,159 @@
|
|||
---
|
||||
title: go-ansible
|
||||
description: Ansible executor for programmatic playbook and ad-hoc command execution
|
||||
description: A pure Go Ansible playbook engine -- parses YAML playbooks, inventories, and roles, then executes tasks on remote hosts via SSH without requiring Python.
|
||||
---
|
||||
|
||||
# go-ansible
|
||||
|
||||
`forge.lthn.ai/core/go-ansible`
|
||||
`forge.lthn.ai/core/go-ansible` is a pure Go implementation of an Ansible playbook engine. It parses standard Ansible YAML playbooks, inventories, and roles, then executes tasks against remote hosts over SSH -- with no dependency on Python or the upstream `ansible-playbook` binary.
|
||||
|
||||
Go wrapper around Ansible for executing playbooks and ad-hoc commands programmatically. Provides a clean API for running Ansible operations with structured output parsing and SSH client abstraction for testing.
|
||||
## Module Path
|
||||
|
||||
## Key Types
|
||||
```
|
||||
forge.lthn.ai/core/go-ansible
|
||||
```
|
||||
|
||||
- `Executor` — runs Ansible playbooks and ad-hoc commands, captures structured output
|
||||
- `MockSSHClient` — test double for SSH connections, enables unit testing without live hosts
|
||||
Requires **Go 1.26+**.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### As a Library
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
ansible "forge.lthn.ai/core/go-ansible"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create an executor rooted at the playbook directory
|
||||
executor := ansible.NewExecutor("/path/to/project")
|
||||
defer executor.Close()
|
||||
|
||||
// Load inventory
|
||||
if err := executor.SetInventory("/path/to/inventory.yml"); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Optionally set extra variables
|
||||
executor.SetVar("deploy_version", "1.2.3")
|
||||
|
||||
// Optionally limit to specific hosts
|
||||
executor.Limit = "web1"
|
||||
|
||||
// Set up callbacks for progress reporting
|
||||
executor.OnTaskStart = func(host string, task *ansible.Task) {
|
||||
fmt.Printf("TASK [%s] on %s\n", task.Name, host)
|
||||
}
|
||||
executor.OnTaskEnd = func(host string, task *ansible.Task, result *ansible.TaskResult) {
|
||||
if result.Failed {
|
||||
fmt.Printf(" FAILED: %s\n", result.Msg)
|
||||
} else if result.Changed {
|
||||
fmt.Printf(" changed\n")
|
||||
} else {
|
||||
fmt.Printf(" ok\n")
|
||||
}
|
||||
}
|
||||
|
||||
// Run the playbook
|
||||
ctx := context.Background()
|
||||
if err := executor.Run(ctx, "/path/to/playbook.yml"); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### As a CLI Command
|
||||
|
||||
The package ships with a CLI integration under `cmd/ansible/` that registers a `core ansible` subcommand:
|
||||
|
||||
```bash
|
||||
# Run a playbook
|
||||
core ansible playbooks/deploy.yml -i inventory/production.yml
|
||||
|
||||
# Limit to a single host
|
||||
core ansible site.yml -l web1
|
||||
|
||||
# Pass extra variables
|
||||
core ansible deploy.yml -e "version=1.2.3" -e "env=prod"
|
||||
|
||||
# Dry run (check mode)
|
||||
core ansible deploy.yml --check
|
||||
|
||||
# Increase verbosity
|
||||
core ansible deploy.yml -vvv
|
||||
|
||||
# Test SSH connectivity to a host
|
||||
core ansible test server.example.com -u root -i ~/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
**CLI flags:**
|
||||
|
||||
| Flag | Short | Description |
|
||||
|------|-------|-------------|
|
||||
| `--inventory` | `-i` | Inventory file or directory |
|
||||
| `--limit` | `-l` | Restrict execution to matching hosts |
|
||||
| `--tags` | `-t` | Only run tasks tagged with these values (comma-separated) |
|
||||
| `--skip-tags` | | Skip tasks tagged with these values |
|
||||
| `--extra-vars` | `-e` | Set additional variables (`key=value`, repeatable) |
|
||||
| `--verbose` | `-v` | Increase verbosity (stack for more: `-vvv`) |
|
||||
| `--check` | | Dry-run mode -- no changes are made |
|
||||
|
||||
## Package Layout
|
||||
|
||||
```
|
||||
go-ansible/
|
||||
types.go Core data types: Playbook, Play, Task, Inventory, Host, Facts
|
||||
parser.go YAML parser for playbooks, inventories, tasks, roles
|
||||
executor.go Execution engine: module dispatch, templating, conditions, loops
|
||||
modules.go 41 module implementations (shell, apt, docker-compose, etc.)
|
||||
ssh.go SSH client with key/password auth, become/sudo, file transfer
|
||||
types_test.go Tests for data types and YAML unmarshalling
|
||||
parser_test.go Tests for the YAML parser
|
||||
executor_test.go Tests for the executor engine
|
||||
ssh_test.go Tests for SSH client construction
|
||||
mock_ssh_test.go Mock SSH infrastructure for module tests
|
||||
modules_*_test.go Module-specific tests (cmd, file, svc, infra, adv)
|
||||
cmd/
|
||||
ansible/
|
||||
cmd.go CLI command registration
|
||||
ansible.go CLI implementation (flags, callbacks, output formatting)
|
||||
```
|
||||
|
||||
## Supported Modules
|
||||
|
||||
41 module handlers are implemented, covering the most commonly used Ansible modules:
|
||||
|
||||
| Category | Modules |
|
||||
|----------|---------|
|
||||
| **Command execution** | `shell`, `command`, `raw`, `script` |
|
||||
| **File operations** | `copy`, `template`, `file`, `lineinfile`, `blockinfile`, `stat`, `slurp`, `fetch`, `get_url` |
|
||||
| **Package management** | `apt`, `apt_key`, `apt_repository`, `package`, `pip` |
|
||||
| **Service management** | `service`, `systemd` |
|
||||
| **User and group** | `user`, `group` |
|
||||
| **HTTP** | `uri` |
|
||||
| **Source control** | `git` |
|
||||
| **Archive** | `unarchive` |
|
||||
| **System** | `hostname`, `sysctl`, `cron`, `reboot`, `setup` |
|
||||
| **Flow control** | `debug`, `fail`, `assert`, `set_fact`, `pause`, `wait_for`, `meta`, `include_vars` |
|
||||
| **Community** | `community.general.ufw`, `ansible.posix.authorized_key`, `community.docker.docker_compose` |
|
||||
|
||||
Both fully-qualified collection names (e.g. `ansible.builtin.shell`) and short-form names (e.g. `shell`) are accepted.
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (command registration, flags, styled output) |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logging and contextual error helper (`log.E()`) |
|
||||
| `golang.org/x/crypto` | SSH protocol implementation (`crypto/ssh`, `crypto/ssh/knownhosts`) |
|
||||
| `gopkg.in/yaml.v3` | YAML parsing for playbooks, inventories, and role files |
|
||||
| `github.com/stretchr/testify` | Test assertions (test-only) |
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,16 +1,173 @@
|
|||
---
|
||||
title: go-api
|
||||
description: REST framework with OpenAPI SDK generation, built on Gin
|
||||
description: Gin-based REST framework with OpenAPI generation, middleware composition, and SDK codegen for the Lethean Go ecosystem.
|
||||
---
|
||||
|
||||
<!-- SPDX-License-Identifier: EUPL-1.2 -->
|
||||
|
||||
# go-api
|
||||
|
||||
`forge.lthn.ai/core/go-api`
|
||||
**Module path:** `forge.lthn.ai/core/go-api`
|
||||
**Language:** Go 1.26
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
HTTP REST framework built on Gin with automatic OpenAPI spec generation and client SDK output. Provides a route group interface for modular endpoint registration and includes health check endpoints out of the box.
|
||||
go-api is a REST framework built on top of [Gin](https://github.com/gin-gonic/gin). It provides
|
||||
an `Engine` that subsystems plug into via the `RouteGroup` interface. Each ecosystem package
|
||||
(go-ai, go-ml, go-rag, and others) registers its own route group, and go-api handles the HTTP
|
||||
plumbing: middleware composition, response envelopes, WebSocket and SSE integration, GraphQL
|
||||
hosting, Authentik identity, OpenAPI 3.1 specification generation, and client SDK codegen.
|
||||
|
||||
## Key Types
|
||||
go-api is a library. It has no `main` package and produces no binary on its own. Callers
|
||||
construct an `Engine`, register route groups, and call `Serve()`.
|
||||
|
||||
- `Engine` — HTTP server wrapping Gin with OpenAPI integration and middleware pipeline
|
||||
- `healthGroup` — built-in health check route group for liveness and readiness probes
|
||||
- `RouteGroup` — interface for modular route registration
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
api "forge.lthn.ai/core/go-api"
|
||||
)
|
||||
|
||||
func main() {
|
||||
engine, _ := api.New(
|
||||
api.WithAddr(":8080"),
|
||||
api.WithBearerAuth("my-secret-token"),
|
||||
api.WithCORS("*"),
|
||||
api.WithRequestID(),
|
||||
api.WithSecure(),
|
||||
api.WithSlog(nil),
|
||||
api.WithSwagger("My API", "A service description", "1.0.0"),
|
||||
)
|
||||
|
||||
engine.Register(myRoutes) // any RouteGroup implementation
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
_ = engine.Serve(ctx) // blocks until context is cancelled, then shuts down gracefully
|
||||
}
|
||||
```
|
||||
|
||||
The default listen address is `:8080`. A built-in `GET /health` endpoint is always present.
|
||||
Every feature beyond panic recovery requires an explicit `With*()` option.
|
||||
|
||||
---
|
||||
|
||||
## Implementing a RouteGroup
|
||||
|
||||
Any type that satisfies the `RouteGroup` interface can register endpoints:
|
||||
|
||||
```go
|
||||
type Routes struct{ service *mypackage.Service }
|
||||
|
||||
func (r *Routes) Name() string { return "mypackage" }
|
||||
func (r *Routes) BasePath() string { return "/v1/mypackage" }
|
||||
|
||||
func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) {
|
||||
rg.GET("/items", r.listItems)
|
||||
rg.POST("/items", r.createItem)
|
||||
}
|
||||
|
||||
func (r *Routes) listItems(c *gin.Context) {
|
||||
items, _ := r.service.List(c.Request.Context())
|
||||
c.JSON(200, api.OK(items))
|
||||
}
|
||||
```
|
||||
|
||||
Register with the engine:
|
||||
|
||||
```go
|
||||
engine.Register(&Routes{service: svc})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Package Layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `api.go` | `Engine` struct, `New()`, `build()`, `Serve()`, `Handler()`, `Channels()` |
|
||||
| `options.go` | All `With*()` option functions (25 options) |
|
||||
| `group.go` | `RouteGroup`, `StreamGroup`, `DescribableGroup` interfaces; `RouteDescription` |
|
||||
| `response.go` | `Response[T]`, `Error`, `Meta`, `OK()`, `Fail()`, `FailWithDetails()`, `Paginated()` |
|
||||
| `middleware.go` | `bearerAuthMiddleware()`, `requestIDMiddleware()` |
|
||||
| `authentik.go` | `AuthentikUser`, `AuthentikConfig`, `GetUser()`, `RequireAuth()`, `RequireGroup()` |
|
||||
| `websocket.go` | `wrapWSHandler()` helper |
|
||||
| `sse.go` | `SSEBroker`, `NewSSEBroker()`, `Publish()`, `Handler()`, `Drain()`, `ClientCount()` |
|
||||
| `cache.go` | `cacheStore`, `cacheEntry`, `cacheWriter`, `cacheMiddleware()` |
|
||||
| `brotli.go` | `brotliHandler`, `newBrotliHandler()`; compression level constants |
|
||||
| `graphql.go` | `graphqlConfig`, `GraphQLOption`, `WithPlayground()`, `WithGraphQLPath()`, `mountGraphQL()` |
|
||||
| `i18n.go` | `I18nConfig`, `WithI18n()`, `i18nMiddleware()`, `GetLocale()`, `GetMessage()` |
|
||||
| `tracing.go` | `WithTracing()`, `NewTracerProvider()` |
|
||||
| `swagger.go` | `swaggerSpec`, `registerSwagger()`; sequence counter for multi-instance safety |
|
||||
| `openapi.go` | `SpecBuilder`, `Build()`, `buildPaths()`, `buildTags()`, `envelopeSchema()` |
|
||||
| `export.go` | `ExportSpec()`, `ExportSpecToFile()` |
|
||||
| `bridge.go` | `ToolDescriptor`, `ToolBridge`, `NewToolBridge()`, `Add()`, `Describe()`, `Tools()` |
|
||||
| `codegen.go` | `SDKGenerator`, `Generate()`, `Available()`, `SupportedLanguages()` |
|
||||
| `cmd/api/` | CLI subcommands: `core api spec` and `core api sdk` |
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Direct
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `github.com/gin-gonic/gin` | HTTP router and middleware engine |
|
||||
| `github.com/gin-contrib/cors` | CORS policy middleware |
|
||||
| `github.com/gin-contrib/secure` | Security headers (HSTS, X-Frame-Options, nosniff) |
|
||||
| `github.com/gin-contrib/gzip` | Gzip response compression |
|
||||
| `github.com/gin-contrib/slog` | Structured request logging via `log/slog` |
|
||||
| `github.com/gin-contrib/timeout` | Per-request deadline enforcement |
|
||||
| `github.com/gin-contrib/static` | Static file serving |
|
||||
| `github.com/gin-contrib/sessions` | Cookie-backed server sessions |
|
||||
| `github.com/gin-contrib/authz` | Casbin policy-based authorisation |
|
||||
| `github.com/gin-contrib/httpsign` | HTTP Signatures verification |
|
||||
| `github.com/gin-contrib/location/v2` | Reverse proxy header detection |
|
||||
| `github.com/gin-contrib/pprof` | Go profiling endpoints |
|
||||
| `github.com/gin-contrib/expvar` | Runtime metrics endpoint |
|
||||
| `github.com/casbin/casbin/v2` | Policy-based access control engine |
|
||||
| `github.com/coreos/go-oidc/v3` | OIDC provider discovery and JWT validation |
|
||||
| `github.com/andybalholm/brotli` | Brotli compression |
|
||||
| `github.com/gorilla/websocket` | WebSocket upgrade support |
|
||||
| `github.com/swaggo/gin-swagger` | Swagger UI handler |
|
||||
| `github.com/swaggo/files` | Swagger UI static assets |
|
||||
| `github.com/swaggo/swag` | Swagger spec registry |
|
||||
| `github.com/99designs/gqlgen` | GraphQL schema execution (gqlgen) |
|
||||
| `go.opentelemetry.io/otel` | OpenTelemetry tracing SDK |
|
||||
| `go.opentelemetry.io/contrib/.../otelgin` | OpenTelemetry Gin instrumentation |
|
||||
| `golang.org/x/text` | BCP 47 language tag matching |
|
||||
| `gopkg.in/yaml.v3` | YAML export of OpenAPI specs |
|
||||
| `forge.lthn.ai/core/cli` | CLI command registration (for `cmd/api/` subcommands) |
|
||||
|
||||
### Ecosystem position
|
||||
|
||||
go-api sits at the base of the Lethean HTTP stack. It has no imports from other Lethean
|
||||
ecosystem modules (beyond `core/cli` for the CLI subcommands). Other packages import go-api
|
||||
to expose their functionality as REST endpoints:
|
||||
|
||||
```
|
||||
Application main / Core CLI
|
||||
|
|
||||
v
|
||||
go-api Engine <-- this module
|
||||
| | |
|
||||
| | +-- OpenAPI spec --> SDKGenerator --> openapi-generator-cli
|
||||
| +-- ToolBridge --> go-ai / go-ml / go-rag route groups
|
||||
+-- RouteGroups ----------> any package implementing RouteGroup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- internals, key types, data flow, middleware stack
|
||||
- [Development](development.md) -- building, testing, contributing, coding standards
|
||||
|
|
|
|||
|
|
@ -1,42 +1,132 @@
|
|||
# go-blockchain
|
||||
---
|
||||
title: Lethean Go Blockchain
|
||||
description: Pure Go implementation of the Lethean CryptoNote/Zano-fork blockchain protocol.
|
||||
---
|
||||
|
||||
Pure Go implementation of the Lethean blockchain protocol.
|
||||
# Lethean Go Blockchain
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-blockchain`
|
||||
`go-blockchain` is a Go reimplementation of the Lethean blockchain protocol. It provides pure-Go implementations of chain logic, data structures, consensus rules, wallet operations, and networking, delegating only mathematically complex cryptographic operations (ring signatures, Bulletproofs+, Zarcanum proofs) to a cleaned C++ library via CGo.
|
||||
|
||||
Provides chain configuration, core cryptographic data types, CryptoNote wire serialisation, and LWMA difficulty adjustment for the Lethean CryptoNote/Zano-fork chain. Lineage: CryptoNote to IntenseCoin (2017) to Lethean to Zano rebase.
|
||||
**Module path:** `forge.lthn.ai/core/go-blockchain`
|
||||
|
||||
**Licence:** [European Union Public Licence (EUPL) version 1.2](https://joinup.ec.europa.eu/software/page/eupl/licence-eupl)
|
||||
|
||||
## Lineage
|
||||
|
||||
```
|
||||
CryptoNote (van Saberhagen, 2013)
|
||||
|
|
||||
IntenseCoin (2017)
|
||||
|
|
||||
Lethean (2017-present)
|
||||
|
|
||||
Zano rebase (2025) -- privacy upgrades: Zarcanum, CLSAG, Bulletproofs+, confidential assets
|
||||
|
|
||||
go-blockchain -- Go reimplementation of the Zano-fork protocol
|
||||
```
|
||||
|
||||
The Lethean mainnet launched on **2026-02-12** with genesis timestamp `1770897600` (12:00 UTC). The chain runs a hybrid PoW/PoS consensus with 120-second block targets.
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
go-blockchain/
|
||||
config/ Chain parameters (mainnet/testnet), hardfork schedule
|
||||
types/ Core data types: Hash, PublicKey, Address, Block, Transaction
|
||||
wire/ Binary serialisation (consensus-critical, bit-identical to C++)
|
||||
crypto/ CGo bridge to libcryptonote (ring sigs, BP+, Zarcanum, stealth)
|
||||
difficulty/ PoW + PoS difficulty adjustment (LWMA variant)
|
||||
consensus/ Three-layer block/transaction validation
|
||||
chain/ Blockchain storage, block/tx validation, mempool
|
||||
p2p/ Levin TCP protocol, peer discovery, handshake
|
||||
rpc/ Daemon and wallet JSON-RPC client
|
||||
wallet/ Key management, output scanning, tx construction
|
||||
mining/ Solo PoW miner (RandomX nonce grinding)
|
||||
tui/ Terminal dashboard (bubbletea + lipgloss)
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Consensus-critical code must be bit-identical** to the C++ implementation. The `wire/` package produces exactly the same binary output as the C++ serialisation for the same input.
|
||||
|
||||
2. **No global state.** Chain parameters are passed via `config.ChainConfig` structs, not package-level globals. `Mainnet` and `Testnet` are pre-defined instances.
|
||||
|
||||
3. **Interfaces at boundaries.** The `chain/` package defines interfaces for storage backends; the `wallet/` package uses Scanner, Signer, Builder, and RingSelector interfaces for v1/v2+ extensibility.
|
||||
|
||||
4. **Test against real chain data.** Wherever possible, tests use actual mainnet block and transaction hex blobs as test vectors, ensuring compatibility with the C++ node.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"forge.lthn.ai/core/go-blockchain/config"
|
||||
"forge.lthn.ai/core/go-blockchain/rpc"
|
||||
"forge.lthn.ai/core/go-blockchain/types"
|
||||
"forge.lthn.ai/core/go-blockchain/wire"
|
||||
"forge.lthn.ai/core/go-blockchain/difficulty"
|
||||
)
|
||||
|
||||
// Query the active hardfork version at a given block height
|
||||
version := config.VersionAtHeight(config.MainnetForks, 10081) // returns HF2
|
||||
// Query the daemon
|
||||
client := rpc.NewClient("http://localhost:36941")
|
||||
info, err := client.GetInfo()
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Printf("Height: %d, Difficulty: %d\n", info.Height, info.PowDifficulty)
|
||||
|
||||
// Encode and decode a Lethean address
|
||||
addr := &types.Address{SpendPublicKey: spendKey, ViewPublicKey: viewKey}
|
||||
encoded := addr.Encode(config.AddressPrefix)
|
||||
decoded, prefix, err := types.DecodeAddress(encoded)
|
||||
// Decode an address
|
||||
addr, prefix, err := types.DecodeAddress("iTHN...")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Printf("Spend key: %s\n", addr.SpendPublicKey)
|
||||
fmt.Printf("Auditable: %v\n", addr.IsAuditable())
|
||||
|
||||
// Varint encoding for the wire protocol
|
||||
buf := wire.EncodeVarint(0x1eaf7)
|
||||
val, n, err := wire.DecodeVarint(buf)
|
||||
|
||||
// Calculate next block difficulty
|
||||
nextDiff := difficulty.NextDifficulty(timestamps, cumulativeDiffs, 120)
|
||||
// Check hardfork version at a given height
|
||||
version := config.VersionAtHeight(config.MainnetForks, 15000)
|
||||
fmt.Printf("Active hardfork at height 15000: HF%d\n", version)
|
||||
```
|
||||
|
||||
## Packages
|
||||
## CGo Boundary
|
||||
|
||||
| Package | Description |
|
||||
|---------|-------------|
|
||||
| `config` | Hardfork schedule, network parameters, address prefixes |
|
||||
| `types` | Addresses, keys, hashes, amounts |
|
||||
| `wire` | CryptoNote binary serialisation, varint codec |
|
||||
| `difficulty` | LWMA difficulty adjustment algorithm |
|
||||
The `crypto/` package is the **only** package that crosses the CGo boundary. All other packages are pure Go.
|
||||
|
||||
```
|
||||
Go side C++ side (libcryptonote + librandomx)
|
||||
+---------+ +---------------------------+
|
||||
| crypto/ | --- CGo calls ---> | cn_fast_hash() |
|
||||
| | | generate_key_derivation |
|
||||
| | | generate_key_image |
|
||||
| | | check_ring_signature |
|
||||
| | | CLSAG_GG/GGX/GGXXG_verify|
|
||||
| | | bulletproof_plus_verify |
|
||||
| | | zarcanum_verify |
|
||||
| | | randomx_hash |
|
||||
+---------+ +---------------------------+
|
||||
```
|
||||
|
||||
When CGo is disabled, stub implementations return errors, allowing the rest of the codebase to compile and run tests that do not require real cryptographic operations.
|
||||
|
||||
## Development Phases
|
||||
|
||||
The project follows a 9-phase development plan. See the [wiki Development Phases page](https://forge.lthn.ai/core/go-blockchain/wiki/Development-Phases) for detailed phase descriptions.
|
||||
|
||||
| Phase | Scope | Status |
|
||||
|-------|-------|--------|
|
||||
| 0 | Config + Types | Complete |
|
||||
| 1 | Wire Serialisation | Complete |
|
||||
| 2 | CGo Crypto Bridge | Complete |
|
||||
| 3 | P2P Protocol | Complete |
|
||||
| 4 | RPC Client | Complete |
|
||||
| 5 | Chain Storage | Complete |
|
||||
| 6 | Wallet Core | Complete |
|
||||
| 7 | Consensus Rules | Complete |
|
||||
| 8 | Mining | Complete |
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- Package dependencies, CGo boundary, data structures
|
||||
- [Cryptography](cryptography.md) -- Crypto primitives, hashing, signatures, proofs
|
||||
- [Networking](networking.md) -- P2P protocol, peer discovery, message types
|
||||
- [RPC Reference](rpc.md) -- Daemon and wallet JSON-RPC API
|
||||
- [Chain Parameters](parameters.md) -- Tokenomics, emission, hardfork schedule
|
||||
|
|
|
|||
|
|
@ -1,16 +1,208 @@
|
|||
---
|
||||
title: go-build
|
||||
description: Build system, release publishers, and SDK generation
|
||||
description: Build system, release pipeline, and SDK generation for the Core ecosystem.
|
||||
---
|
||||
|
||||
# go-build
|
||||
|
||||
`forge.lthn.ai/core/go-build`
|
||||
`forge.lthn.ai/core/go-build` is the build, release, and SDK generation toolkit for Core projects. It provides:
|
||||
|
||||
Build system that auto-detects project types (Go, Wails, Docker, LinuxKit, C++) and produces cross-compiled binaries, archives, and checksums. Includes release publishers for GitHub, Docker registries, npm, Homebrew, Scoop, AUR, and Chocolatey. Also provides SDK generation for API clients.
|
||||
- **Auto-detecting builders** for Go, Wails, Docker, LinuxKit, C++, and Taskfile projects
|
||||
- **Cross-compilation** with per-target archiving (tar.gz, tar.xz, zip) and SHA-256 checksums
|
||||
- **Code signing** -- macOS codesign with notarisation, GPG detached signatures, Windows signtool (placeholder)
|
||||
- **Release automation** -- semantic versioning from git tags, conventional-commit changelogs, multi-target publishing
|
||||
- **SDK generation** -- OpenAPI spec diffing for breaking-change detection, code generation for TypeScript, Python, Go, and PHP
|
||||
- **CLI integration** -- registers `core build`, `core ci`, and `core sdk` commands via the Core CLI framework
|
||||
|
||||
## Packages
|
||||
## Module Path
|
||||
|
||||
- `pkg/build/` — project-type detection, cross-compilation, archiving
|
||||
- `pkg/release/` — changelog generation, publisher pipeline
|
||||
- `pkg/sdk/` — client SDK generation from OpenAPI specs
|
||||
```
|
||||
forge.lthn.ai/core/go-build
|
||||
```
|
||||
|
||||
Requires **Go 1.26+**.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Build a project
|
||||
|
||||
From any project directory containing a recognised marker file:
|
||||
|
||||
```bash
|
||||
core build # Auto-detect type, build for configured targets
|
||||
core build --targets linux/amd64 # Single target
|
||||
core build --ci # JSON output for CI pipelines
|
||||
core build --verbose # Detailed step-by-step output
|
||||
```
|
||||
|
||||
The builder is chosen by marker-file priority:
|
||||
|
||||
| Marker file | Builder |
|
||||
|-------------------|------------|
|
||||
| `wails.json` | Wails |
|
||||
| `go.mod` | Go |
|
||||
| `package.json` | Node (stub)|
|
||||
| `composer.json` | PHP (stub) |
|
||||
| `CMakeLists.txt` | C++ |
|
||||
| `Dockerfile` | Docker |
|
||||
| `linuxkit.yml` | LinuxKit |
|
||||
| `Taskfile.yml` | Taskfile |
|
||||
|
||||
### Release artifacts
|
||||
|
||||
```bash
|
||||
core build release --we-are-go-for-launch # Build + archive + checksum + publish
|
||||
core build release # Dry-run (default without the flag)
|
||||
core build release --draft --prerelease # Mark as draft pre-release
|
||||
```
|
||||
|
||||
### Publish pre-built artifacts
|
||||
|
||||
After `core build` has populated `dist/`:
|
||||
|
||||
```bash
|
||||
core ci # Dry-run publish from dist/
|
||||
core ci --we-are-go-for-launch # Actually publish
|
||||
core ci --version v1.2.3 # Override version
|
||||
```
|
||||
|
||||
### Generate changelogs
|
||||
|
||||
```bash
|
||||
core ci changelog # From latest tag to HEAD
|
||||
core ci changelog --from v0.1.0 --to v0.2.0
|
||||
core ci version # Show determined next version
|
||||
core ci init # Scaffold .core/release.yaml
|
||||
```
|
||||
|
||||
### SDK operations
|
||||
|
||||
```bash
|
||||
core build sdk # Generate SDKs for all configured languages
|
||||
core build sdk --lang typescript # Single language
|
||||
core sdk diff --base v1.0.0 --spec api/openapi.yaml # Breaking-change check
|
||||
core sdk validate # Validate OpenAPI spec
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go-build/
|
||||
|
|
||||
|-- cmd/
|
||||
| |-- build/ CLI commands for `core build` (build, from-path, pwa, sdk, release)
|
||||
| |-- ci/ CLI commands for `core ci` (init, changelog, version, publish)
|
||||
| +-- sdk/ CLI commands for `core sdk` (diff, validate)
|
||||
|
|
||||
+-- pkg/
|
||||
|-- build/ Core build types, config loading, discovery, archiving, checksums
|
||||
| |-- builders/ Builder implementations (Go, Wails, Docker, LinuxKit, C++, Taskfile)
|
||||
| +-- signing/ Code-signing implementations (macOS codesign, GPG, Windows stub)
|
||||
|
|
||||
|-- release/ Release orchestration, versioning, changelog, config
|
||||
| +-- publishers/ Publisher implementations (GitHub, Docker, npm, Homebrew, Scoop, AUR, Chocolatey, LinuxKit)
|
||||
|
|
||||
+-- sdk/ OpenAPI SDK generation and breaking-change diffing
|
||||
+-- generators/ Language generators (TypeScript, Python, Go, PHP)
|
||||
```
|
||||
|
||||
## Configuration Files
|
||||
|
||||
Build and release behaviour is driven by two YAML files in the `.core/` directory.
|
||||
|
||||
### `.core/build.yaml`
|
||||
|
||||
Controls compilation targets, flags, and signing:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
project:
|
||||
name: myapp
|
||||
description: My application
|
||||
main: ./cmd/myapp
|
||||
binary: myapp
|
||||
build:
|
||||
cgo: false
|
||||
flags: ["-trimpath"]
|
||||
ldflags: ["-s", "-w"]
|
||||
env: []
|
||||
targets:
|
||||
- os: linux
|
||||
arch: amd64
|
||||
- os: linux
|
||||
arch: arm64
|
||||
- os: darwin
|
||||
arch: arm64
|
||||
- os: windows
|
||||
arch: amd64
|
||||
sign:
|
||||
enabled: true
|
||||
gpg:
|
||||
key: $GPG_KEY_ID
|
||||
macos:
|
||||
identity: $CODESIGN_IDENTITY
|
||||
notarize: false
|
||||
apple_id: $APPLE_ID
|
||||
team_id: $APPLE_TEAM_ID
|
||||
app_password: $APPLE_APP_PASSWORD
|
||||
```
|
||||
|
||||
When no `.core/build.yaml` exists, sensible defaults apply (CGO off, `-trimpath -s -w`, four standard targets).
|
||||
|
||||
### `.core/release.yaml`
|
||||
|
||||
Controls versioning, changelog filtering, publishers, and SDK generation:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
project:
|
||||
name: myapp
|
||||
repository: owner/repo
|
||||
build:
|
||||
targets:
|
||||
- os: linux
|
||||
arch: amd64
|
||||
- os: darwin
|
||||
arch: arm64
|
||||
publishers:
|
||||
- type: github
|
||||
draft: false
|
||||
prerelease: false
|
||||
- type: homebrew
|
||||
tap: owner/homebrew-tap
|
||||
- type: docker
|
||||
registry: ghcr.io
|
||||
image: owner/myapp
|
||||
tags: ["latest", "{{.Version}}"]
|
||||
changelog:
|
||||
include: [feat, fix, perf, refactor]
|
||||
exclude: [chore, docs, style, test, ci]
|
||||
sdk:
|
||||
spec: api/openapi.yaml
|
||||
languages: [typescript, python, go, php]
|
||||
output: sdk
|
||||
diff:
|
||||
enabled: true
|
||||
fail_on_breaking: false
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Purpose |
|
||||
|---|---|
|
||||
| `forge.lthn.ai/core/cli` | CLI command registration and TUI styling |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction (`io.Medium`, `io.Local`) |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalised CLI labels |
|
||||
| `forge.lthn.ai/core/go-log` | Structured error logging |
|
||||
| `github.com/Snider/Borg` | XZ compression for tar.xz archives |
|
||||
| `github.com/getkin/kin-openapi` | OpenAPI spec loading and validation |
|
||||
| `github.com/oasdiff/oasdiff` | OpenAPI diff and breaking-change detection |
|
||||
| `gopkg.in/yaml.v3` | YAML config parsing |
|
||||
| `github.com/leaanthony/debme` | Embedded filesystem anchoring (PWA templates) |
|
||||
| `github.com/leaanthony/gosod` | Template extraction for PWA builds |
|
||||
| `golang.org/x/net` | HTML parsing for PWA manifest detection |
|
||||
| `golang.org/x/text` | Changelog section title casing |
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,15 +1,111 @@
|
|||
---
|
||||
title: go-cache
|
||||
description: In-memory cache with expiry support
|
||||
description: File-based caching with TTL expiry, storage-agnostic via the go-io Medium interface.
|
||||
---
|
||||
|
||||
# go-cache
|
||||
|
||||
`forge.lthn.ai/core/go-cache`
|
||||
`go-cache` is a lightweight, storage-agnostic caching library for Go. It stores
|
||||
JSON-serialised entries with automatic TTL expiry and path-traversal protection.
|
||||
|
||||
Lightweight in-memory cache for Go applications. Stores key-value pairs with optional expiry.
|
||||
**Module path:** `forge.lthn.ai/core/go-cache`
|
||||
|
||||
## Key Types
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
- `Cache` — thread-safe in-memory key-value store
|
||||
- `Entry` — single cached item with value and expiry metadata
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"forge.lthn.ai/core/go-cache"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a cache with default settings:
|
||||
// - storage: local filesystem (io.Local)
|
||||
// - directory: .core/cache/ in the working directory
|
||||
// - TTL: 1 hour
|
||||
c, err := cache.New(nil, "", 0)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Store a value
|
||||
err = c.Set("user/profile", map[string]string{
|
||||
"name": "Alice",
|
||||
"role": "admin",
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Retrieve it (returns false if missing or expired)
|
||||
var profile map[string]string
|
||||
found, err := c.Get("user/profile", &profile)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if found {
|
||||
fmt.Println(profile["name"]) // Alice
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Package Layout
|
||||
|
||||
| File | Purpose |
|
||||
|-----------------|-------------------------------------------------------------|
|
||||
| `cache.go` | Core types (`Cache`, `Entry`), CRUD operations, key helpers |
|
||||
| `cache_test.go` | Tests covering set/get, expiry, delete, clear, defaults |
|
||||
| `go.mod` | Module definition (Go 1.26) |
|
||||
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Version | Role |
|
||||
|-------------------------------|---------|---------------------------------------------|
|
||||
| `forge.lthn.ai/core/go-io` | v0.0.3 | Storage abstraction (`Medium` interface) |
|
||||
| `forge.lthn.ai/core/go-log` | v0.0.1 | Structured logging (indirect, via `go-io`) |
|
||||
|
||||
There are no other runtime dependencies. The test suite uses the standard
|
||||
library only (plus the `MockMedium` from `go-io`).
|
||||
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### Storage Backends
|
||||
|
||||
The cache does not read or write files directly. All I/O goes through the
|
||||
`io.Medium` interface defined in `go-io`. This means the same cache logic works
|
||||
against:
|
||||
|
||||
- **Local filesystem** (`io.Local`) -- the default
|
||||
- **SQLite KV store** (`store.Medium` from `go-io/store`)
|
||||
- **S3-compatible storage** (`go-io/s3`)
|
||||
- **In-memory mock** (`io.NewMockMedium()`) -- ideal for tests
|
||||
|
||||
Pass any `Medium` implementation as the first argument to `cache.New()`.
|
||||
|
||||
### TTL and Expiry
|
||||
|
||||
Every entry records both `cached_at` and `expires_at` timestamps. On `Get()`,
|
||||
if the current time is past `expires_at`, the entry is treated as a cache miss
|
||||
-- no stale data is ever returned. The default TTL is one hour
|
||||
(`cache.DefaultTTL`).
|
||||
|
||||
### GitHub Cache Keys
|
||||
|
||||
The package includes two helper functions that produce consistent cache keys
|
||||
for GitHub API data:
|
||||
|
||||
```go
|
||||
cache.GitHubReposKey("host-uk") // "github/host-uk/repos"
|
||||
cache.GitHubRepoKey("host-uk", "core") // "github/host-uk/core/meta"
|
||||
```
|
||||
|
||||
These are convenience helpers used by other packages in the ecosystem (such as
|
||||
`go-devops`) to avoid key duplication when caching GitHub responses.
|
||||
|
|
|
|||
|
|
@ -1,16 +1,141 @@
|
|||
---
|
||||
title: go-config
|
||||
description: Configuration service for the Core DI container
|
||||
description: Layered configuration management for the Core framework with file, environment, and in-memory resolution.
|
||||
---
|
||||
|
||||
# go-config
|
||||
|
||||
`forge.lthn.ai/core/go-config`
|
||||
`forge.lthn.ai/core/go-config` provides layered configuration management for applications built on the Core framework. It resolves values through a priority chain -- defaults, file, environment variables, flags -- so that the same codebase works identically across local development, CI, and production without code changes.
|
||||
|
||||
Configuration management service that integrates with the Core dependency injection container. Handles loading, merging, and accessing typed configuration values for services registered in the Core runtime.
|
||||
## Module Path
|
||||
|
||||
## Key Types
|
||||
```
|
||||
forge.lthn.ai/core/go-config
|
||||
```
|
||||
|
||||
- `Config` — configuration data holder with typed accessors
|
||||
- `Service` — Core-integrated service that manages configuration lifecycle
|
||||
- `ServiceOptions` — options for configuring the service registration
|
||||
Requires **Go 1.26+**.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Standalone usage
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
config "forge.lthn.ai/core/go-config"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg, err := config.New() // loads ~/.core/config.yaml if it exists
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Write a value and persist it
|
||||
_ = cfg.Set("dev.editor", "vim")
|
||||
_ = cfg.Commit()
|
||||
|
||||
// Read it back
|
||||
var editor string
|
||||
_ = cfg.Get("dev.editor", &editor)
|
||||
fmt.Println(editor) // "vim"
|
||||
}
|
||||
```
|
||||
|
||||
### As a Core framework service
|
||||
|
||||
```go
|
||||
import (
|
||||
config "forge.lthn.ai/core/go-config"
|
||||
"forge.lthn.ai/core/go/pkg/core"
|
||||
)
|
||||
|
||||
app, _ := core.New(
|
||||
core.WithService(config.NewConfigService),
|
||||
)
|
||||
// The config service loads automatically during OnStartup.
|
||||
// Retrieve it later via core.ServiceFor[*config.Service](app).
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| File | Purpose |
|
||||
|-----------------|----------------------------------------------------------------|
|
||||
| `config.go` | Core `Config` struct -- layered Get/Set, file load, commit |
|
||||
| `env.go` | Environment variable iteration and prefix-based loading |
|
||||
| `service.go` | Framework service wrapper with lifecycle (`Startable`) support |
|
||||
| `config_test.go`| Tests following the `_Good` / `_Bad` / `_Ugly` convention |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Role |
|
||||
|-----------------------------------|-----------------------------------------|
|
||||
| `forge.lthn.ai/core/go` | Core framework (`core.Config` interface, `ServiceRuntime`) |
|
||||
| `forge.lthn.ai/core/go-io` | Storage abstraction (`Medium` for reading/writing files) |
|
||||
| `forge.lthn.ai/core/go-log` | Contextual error helper (`E()`) |
|
||||
| `github.com/spf13/viper` | Underlying configuration engine |
|
||||
| `gopkg.in/yaml.v3` | YAML serialisation for `Commit()` |
|
||||
|
||||
## Configuration Priority
|
||||
|
||||
Values are resolved in ascending priority order:
|
||||
|
||||
1. **Defaults** -- hardcoded fallbacks (via `Set()` before any file load)
|
||||
2. **File** -- YAML loaded from `~/.core/config.yaml` (or a custom path)
|
||||
3. **Environment variables** -- prefixed with `CORE_CONFIG_` by default
|
||||
4. **Explicit Set()** -- in-memory overrides applied at runtime
|
||||
|
||||
Environment variables always override file values. An explicit `Set()` call overrides everything.
|
||||
|
||||
## Key Access
|
||||
|
||||
All keys use **dot notation** for nested values:
|
||||
|
||||
```go
|
||||
cfg.Set("a.b.c", "deep")
|
||||
|
||||
var val string
|
||||
cfg.Get("a.b.c", &val) // "deep"
|
||||
```
|
||||
|
||||
This maps to YAML structure:
|
||||
|
||||
```yaml
|
||||
a:
|
||||
b:
|
||||
c: deep
|
||||
```
|
||||
|
||||
## Environment Variable Mapping
|
||||
|
||||
Environment variables are mapped to dot-notation keys by:
|
||||
|
||||
1. Stripping the prefix (default `CORE_CONFIG_`)
|
||||
2. Lowercasing
|
||||
3. Replacing `_` with `.`
|
||||
|
||||
For example, `CORE_CONFIG_DEV_EDITOR=nano` resolves to key `dev.editor` with value `"nano"`.
|
||||
|
||||
You can change the prefix with `WithEnvPrefix`:
|
||||
|
||||
```go
|
||||
cfg, _ := config.New(config.WithEnvPrefix("MYAPP"))
|
||||
// MYAPP_SETTING=secret -> key "setting"
|
||||
```
|
||||
|
||||
## Persisting Changes
|
||||
|
||||
`Set()` only writes to memory. Call `Commit()` to flush changes to disk:
|
||||
|
||||
```go
|
||||
cfg.Set("dev.editor", "vim")
|
||||
cfg.Commit() // writes to ~/.core/config.yaml
|
||||
```
|
||||
|
||||
`Commit()` only persists values that were loaded from the file or explicitly set via `Set()`. Environment variable values are never leaked into the config file.
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,18 +1,146 @@
|
|||
---
|
||||
title: go-container
|
||||
description: Container runtime, LinuxKit builder, and dev environment management
|
||||
description: Container runtime, LinuxKit image builder, and portable development environment management for Go.
|
||||
---
|
||||
|
||||
# go-container
|
||||
|
||||
`forge.lthn.ai/core/go-container`
|
||||
`forge.lthn.ai/core/go-container` provides a container runtime built on LinuxKit and lightweight hypervisors. It manages the full lifecycle of LinuxKit virtual machines -- from building images with embedded templates, to running them via QEMU or Hyperkit, to offering a portable development environment with shell access, project mounting, test execution, and Claude AI integration.
|
||||
|
||||
Container runtime abstraction supporting multiple hypervisors for running LinuxKit-based images and managing development environments. Handles image building, VM lifecycle, and port forwarding across QEMU and HyperKit backends.
|
||||
This is **not** a Docker wrapper. It runs real VMs from LinuxKit images (ISO, qcow2, VMDK, raw) using platform-native acceleration (KVM on Linux, HVF on macOS, Hyperkit where available).
|
||||
|
||||
## Key Types
|
||||
|
||||
- `Container` — manages container lifecycle (create, start, stop, remove)
|
||||
- `RunOptions` — configuration for container execution (ports, volumes, environment)
|
||||
- `HypervisorOptions` — shared configuration across hypervisor backends
|
||||
- `QemuHypervisor` — QEMU-based VM backend
|
||||
- `HyperkitHypervisor` — HyperKit-based VM backend (macOS)
|
||||
## Module path
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go-container
|
||||
```
|
||||
|
||||
Requires **Go 1.26+**.
|
||||
|
||||
|
||||
## Quick start
|
||||
|
||||
### Run a VM from an image
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
container "forge.lthn.ai/core/go-container"
|
||||
"forge.lthn.ai/core/go-io"
|
||||
)
|
||||
|
||||
manager, err := container.NewLinuxKitManager(io.Local)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
c, err := manager.Run(ctx, "/path/to/image.qcow2", container.RunOptions{
|
||||
Name: "my-vm",
|
||||
Memory: 2048,
|
||||
CPUs: 2,
|
||||
SSHPort: 2222,
|
||||
Detach: true,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
fmt.Printf("Started container %s (PID %d)\n", c.ID, c.PID)
|
||||
```
|
||||
|
||||
### Use the development environment
|
||||
|
||||
```go
|
||||
import (
|
||||
"forge.lthn.ai/core/go-container/devenv"
|
||||
"forge.lthn.ai/core/go-io"
|
||||
)
|
||||
|
||||
dev, err := devenv.New(io.Local)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Boot the dev environment (downloads image if needed)
|
||||
ctx := context.Background()
|
||||
err = dev.Boot(ctx, devenv.DefaultBootOptions())
|
||||
|
||||
// Open an SSH shell
|
||||
err = dev.Shell(ctx, devenv.ShellOptions{})
|
||||
|
||||
// Run tests inside the VM
|
||||
err = dev.Test(ctx, "/path/to/project", devenv.TestOptions{})
|
||||
```
|
||||
|
||||
### Build and run from a LinuxKit template
|
||||
|
||||
```go
|
||||
import container "forge.lthn.ai/core/go-container"
|
||||
|
||||
// List available templates (built-in + user-defined)
|
||||
templates := container.ListTemplates()
|
||||
|
||||
// Apply variables to a template
|
||||
content, err := container.ApplyTemplate("core-dev", map[string]string{
|
||||
"SSH_KEY": "ssh-ed25519 AAAA...",
|
||||
"MEMORY": "4096",
|
||||
"HOSTNAME": "my-dev-box",
|
||||
})
|
||||
```
|
||||
|
||||
|
||||
## Package layout
|
||||
|
||||
| Package | Import path | Purpose |
|
||||
|---------|-------------|---------|
|
||||
| `container` (root) | `forge.lthn.ai/core/go-container` | Container struct, Manager interface, hypervisor abstraction, LinuxKit manager, state persistence, template engine |
|
||||
| `devenv` | `forge.lthn.ai/core/go-container/devenv` | Portable dev environment orchestration: boot, shell, serve, test, Claude sandbox, image management |
|
||||
| `sources` | `forge.lthn.ai/core/go-container/sources` | Image download backends: CDN and GitHub Releases with progress reporting |
|
||||
| `cmd/vm` | `forge.lthn.ai/core/go-container/cmd/vm` | CLI commands (`core vm run`, `core vm ps`, `core vm stop`, `core vm logs`, `core vm exec`, `core vm templates`) |
|
||||
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/go-io` | File system abstraction (`Medium` interface), process utilities |
|
||||
| `forge.lthn.ai/core/go-config` | Configuration loading (used by `devenv` for `~/.core/config.yaml`) |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalised UI strings (used by `cmd/vm`) |
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (used by `cmd/vm` for command registration) |
|
||||
| `github.com/stretchr/testify` | Test assertions |
|
||||
| `gopkg.in/yaml.v3` | YAML parsing for test configuration |
|
||||
|
||||
The root `container` package has only two direct dependencies: `go-io` and the standard library. The `devenv` and `cmd/vm` packages pull in the heavier dependencies.
|
||||
|
||||
|
||||
## CLI commands
|
||||
|
||||
When registered via `cmd/vm`, the following commands become available under `core vm`:
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `core vm run [image]` | Run a VM from an image file or `--template` |
|
||||
| `core vm ps` | List running VMs (`-a` for all including stopped) |
|
||||
| `core vm stop <id>` | Stop a running VM by ID or name (supports partial matching) |
|
||||
| `core vm logs <id>` | View VM logs (`-f` to follow) |
|
||||
| `core vm exec <id> <cmd>` | Execute a command inside the VM via SSH |
|
||||
| `core vm templates` | List available LinuxKit templates |
|
||||
| `core vm templates show <name>` | Display a template's full YAML |
|
||||
| `core vm templates vars <name>` | Show a template's required and optional variables |
|
||||
|
||||
|
||||
## Built-in templates
|
||||
|
||||
Two LinuxKit templates are embedded in the binary:
|
||||
|
||||
- **core-dev** -- Full development environment with Go, Node.js, PHP, Docker-in-LinuxKit, and SSH access
|
||||
- **server-php** -- Production PHP server with FrankenPHP, Caddy reverse proxy, and health checks
|
||||
|
||||
User-defined templates can be placed in `.core/linuxkit/` (workspace-relative) or `~/.core/linuxkit/` (global). They are discovered automatically and merged with the built-in set.
|
||||
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2. See [LICENSE](../LICENSE) for the full text.
|
||||
|
|
|
|||
|
|
@ -1,10 +1,19 @@
|
|||
---
|
||||
title: go-crypt
|
||||
description: Cryptographic primitives, authentication, and trust policy engine for the Lethean agent platform.
|
||||
---
|
||||
|
||||
# go-crypt
|
||||
|
||||
Cryptographic primitives, authentication, and trust policy engine.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-crypt`
|
||||
**Licence**: EUPL-1.2
|
||||
**Language**: Go 1.26
|
||||
|
||||
Provides symmetric encryption (ChaCha20-Poly1305 and AES-256-GCM with Argon2id KDF), OpenPGP challenge-response authentication with online and air-gapped courier modes, Argon2id password hashing, RSA-OAEP key generation, RFC-0004 deterministic content hashing, and a three-tier agent trust policy engine with an audit log and approval queue.
|
||||
Cryptographic primitives, authentication, and trust policy engine for the
|
||||
Lethean agent platform. Provides symmetric encryption, password hashing,
|
||||
OpenPGP authentication with both online and air-gapped modes, RSA key
|
||||
management, deterministic content hashing, and a three-tier agent access
|
||||
control system with an audit log and approval queue.
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
|
@ -14,25 +23,141 @@ import (
|
|||
"forge.lthn.ai/core/go-crypt/auth"
|
||||
"forge.lthn.ai/core/go-crypt/trust"
|
||||
)
|
||||
|
||||
// Encrypt with ChaCha20-Poly1305 + Argon2id KDF
|
||||
ciphertext, err := crypt.Encrypt(plaintext, passphrase)
|
||||
|
||||
// OpenPGP authentication
|
||||
a := auth.New(medium, auth.WithSessionStore(auth.NewSQLiteSessionStore(dbPath)))
|
||||
session, err := a.Login(userID, password)
|
||||
|
||||
// Trust policy evaluation
|
||||
engine := trust.NewPolicyEngine(registry)
|
||||
decision := engine.Evaluate("Charon", "repo.push", "core/go-crypt")
|
||||
```
|
||||
|
||||
## Packages
|
||||
### Encrypt and Decrypt Data
|
||||
|
||||
| Package | Description |
|
||||
|---------|-------------|
|
||||
| `crypt` | ChaCha20-Poly1305, AES-256-GCM, Argon2id KDF |
|
||||
| `auth` | OpenPGP challenge-response, session management |
|
||||
| `trust` | Three-tier policy engine, audit log, approval queue |
|
||||
| `hash` | RFC-0004 deterministic content hashing |
|
||||
| `keys` | RSA-OAEP key generation |
|
||||
The default cipher is XChaCha20-Poly1305 with Argon2id key derivation. A
|
||||
random salt and nonce are generated automatically and prepended to the
|
||||
ciphertext.
|
||||
|
||||
```go
|
||||
// Encrypt with ChaCha20-Poly1305 + Argon2id KDF
|
||||
ciphertext, err := crypt.Encrypt(plaintext, []byte("my passphrase"))
|
||||
|
||||
// Decrypt
|
||||
plaintext, err := crypt.Decrypt(ciphertext, []byte("my passphrase"))
|
||||
|
||||
// Or use AES-256-GCM instead
|
||||
ciphertext, err := crypt.EncryptAES(plaintext, []byte("my passphrase"))
|
||||
plaintext, err := crypt.DecryptAES(ciphertext, []byte("my passphrase"))
|
||||
```
|
||||
|
||||
### Hash and Verify Passwords
|
||||
|
||||
```go
|
||||
// Hash with Argon2id (recommended)
|
||||
hash, err := crypt.HashPassword("hunter2")
|
||||
// Returns: $argon2id$v=19$m=65536,t=3,p=4$<salt>$<hash>
|
||||
|
||||
// Verify (constant-time comparison)
|
||||
match, err := crypt.VerifyPassword("hunter2", hash)
|
||||
```
|
||||
|
||||
### OpenPGP Authentication
|
||||
|
||||
```go
|
||||
// Create an authenticator backed by a storage medium
|
||||
a := auth.New(medium,
|
||||
auth.WithSessionStore(sqliteStore),
|
||||
auth.WithSessionTTL(8 * time.Hour),
|
||||
)
|
||||
|
||||
// Register a user (generates PGP keypair, stores credentials)
|
||||
user, err := a.Register("alice", "password123")
|
||||
|
||||
// Password-based login (bypasses PGP challenge-response)
|
||||
session, err := a.Login(userID, "password123")
|
||||
|
||||
// Validate a session token
|
||||
session, err := a.ValidateSession(token)
|
||||
```
|
||||
|
||||
### Trust Policy Evaluation
|
||||
|
||||
```go
|
||||
// Set up a registry and register agents
|
||||
registry := trust.NewRegistry()
|
||||
registry.Register(trust.Agent{
|
||||
Name: "Athena",
|
||||
Tier: trust.TierFull,
|
||||
})
|
||||
registry.Register(trust.Agent{
|
||||
Name: "Clotho",
|
||||
Tier: trust.TierVerified,
|
||||
ScopedRepos: []string{"core/*"},
|
||||
})
|
||||
|
||||
// Evaluate capabilities
|
||||
engine := trust.NewPolicyEngine(registry)
|
||||
result := engine.Evaluate("Athena", trust.CapPushRepo, "core/go-crypt")
|
||||
// result.Decision == trust.Allow
|
||||
|
||||
result = engine.Evaluate("Clotho", trust.CapMergePR, "core/go-crypt")
|
||||
// result.Decision == trust.NeedsApproval
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Package | Import Path | Description |
|
||||
|---------|-------------|-------------|
|
||||
| `crypt` | `go-crypt/crypt` | High-level encrypt/decrypt (ChaCha20 + AES), password hashing, HMAC, checksums, key derivation |
|
||||
| `crypt/chachapoly` | `go-crypt/crypt/chachapoly` | Standalone ChaCha20-Poly1305 AEAD wrapper |
|
||||
| `crypt/lthn` | `go-crypt/crypt/lthn` | RFC-0004 quasi-salted deterministic hash for content identifiers |
|
||||
| `crypt/pgp` | `go-crypt/crypt/pgp` | OpenPGP key generation, encryption, decryption, signing, verification |
|
||||
| `crypt/rsa` | `go-crypt/crypt/rsa` | RSA-OAEP-SHA256 key generation and encryption (2048+ bit) |
|
||||
| `crypt/openpgp` | `go-crypt/crypt/openpgp` | Service wrapper implementing the `core.Crypt` interface with IPC support |
|
||||
| `auth` | `go-crypt/auth` | OpenPGP challenge-response authentication, session management, key rotation/revocation |
|
||||
| `trust` | `go-crypt/trust` | Agent trust model, policy engine, approval queue, audit log |
|
||||
| `cmd/crypt` | `go-crypt/cmd/crypt` | CLI commands: `crypt encrypt`, `crypt decrypt`, `crypt hash`, `crypt keygen`, `crypt checksum` |
|
||||
|
||||
## CLI Commands
|
||||
|
||||
The `cmd/crypt` package registers a `crypt` command group with the `core` CLI:
|
||||
|
||||
```bash
|
||||
# Encrypt a file (ChaCha20-Poly1305 by default)
|
||||
core crypt encrypt myfile.txt -p "passphrase"
|
||||
core crypt encrypt myfile.txt --aes -p "passphrase"
|
||||
|
||||
# Decrypt
|
||||
core crypt decrypt myfile.txt.enc -p "passphrase"
|
||||
|
||||
# Hash a password
|
||||
core crypt hash "my password" # Argon2id
|
||||
core crypt hash "my password" --bcrypt # Bcrypt
|
||||
|
||||
# Verify a password against a hash
|
||||
core crypt hash "my password" --verify '$argon2id$v=19$...'
|
||||
|
||||
# Generate a random key
|
||||
core crypt keygen # 32 bytes, hex
|
||||
core crypt keygen -l 64 --base64 # 64 bytes, base64
|
||||
|
||||
# Compute file checksums
|
||||
core crypt checksum myfile.txt # SHA-256
|
||||
core crypt checksum myfile.txt --sha512
|
||||
core crypt checksum myfile.txt --verify "abc123..."
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `forge.lthn.ai/core/go` | Framework: `core.E` error helper, `core.Crypt` interface, `io.Medium` storage abstraction |
|
||||
| `forge.lthn.ai/core/go-store` | SQLite KV store for persistent session storage |
|
||||
| `forge.lthn.ai/core/go-io` | `io.Medium` interface used by the auth package |
|
||||
| `forge.lthn.ai/core/go-log` | Contextual error wrapping via `core.E()` |
|
||||
| `forge.lthn.ai/core/cli` | CLI framework for the `cmd/crypt` commands |
|
||||
| `github.com/ProtonMail/go-crypto` | OpenPGP implementation (actively maintained, post-quantum research) |
|
||||
| `golang.org/x/crypto` | Argon2id, ChaCha20-Poly1305, scrypt, HKDF, bcrypt |
|
||||
| `github.com/stretchr/testify` | Test assertions (`assert`, `require`) |
|
||||
|
||||
No C toolchain or CGo is required. All cryptographic operations use pure Go
|
||||
implementations.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- internals, data flow, algorithm reference
|
||||
- [Development](development.md) -- building, testing, contributing
|
||||
- [History](history.md) -- completed phases, security audit findings, known limitations
|
||||
|
|
|
|||
|
|
@ -1,38 +1,146 @@
|
|||
---
|
||||
title: go-devops
|
||||
description: Build system, release publishers, infrastructure management, and DevOps tooling for the Lethean ecosystem.
|
||||
---
|
||||
|
||||
# go-devops
|
||||
|
||||
Infrastructure and build automation library for the Lethean ecosystem.
|
||||
`forge.lthn.ai/core/go-devops` is the build, release, and infrastructure automation library for the Lethean ecosystem. It replaces goreleaser with a native Go pipeline that auto-detects project types, cross-compiles, signs artefacts, generates changelogs, and publishes to eight distribution targets.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-devops`
|
||||
**Go**: 1.26
|
||||
**Licence**: EUPL-1.2
|
||||
|
||||
Provides a native Go Ansible playbook executor (~30 modules over SSH without shelling out), a multi-target build pipeline with project type auto-detection (Go, Wails, Docker, C++, LinuxKit, Taskfile), code signing (macOS codesign, GPG, Windows signtool), release orchestration with changelog generation and eight publisher backends (GitHub Releases, Docker, Homebrew, npm, AUR, Scoop, Chocolatey, LinuxKit), Hetzner Cloud and Robot API clients, CloudNS DNS management, container/VM management via QEMU and Hyperkit, an OpenAPI SDK generator (TypeScript, Python, Go, PHP), and a developer toolkit with cyclomatic complexity analysis, vulnerability scanning, and coverage trending.
|
||||
## What it does
|
||||
|
||||
## Quick Start
|
||||
| Area | Summary |
|
||||
|------|---------|
|
||||
| **Build system** | Auto-detect project type from marker files, cross-compile for multiple OS/arch targets, archive and checksum artefacts |
|
||||
| **Code signing** | macOS `codesign`, GPG detached signatures, Windows `signtool` |
|
||||
| **Release publishers** | GitHub Releases, Docker, Homebrew, npm, AUR, Scoop, Chocolatey, LinuxKit |
|
||||
| **SDK generation** | Generate typed API clients from OpenAPI specs (TypeScript, Python, Go, PHP) with breaking change detection |
|
||||
| **Ansible executor** | Native Go playbook runner with ~30 modules over SSH — no `ansible-playbook` shell-out |
|
||||
| **Infrastructure** | Hetzner Cloud/Robot provisioning, CloudNS DNS management |
|
||||
| **Container/VM** | LinuxKit-based VMs via QEMU (Linux) or Hyperkit (macOS) |
|
||||
| **Developer toolkit** | Cyclomatic complexity analysis, vulnerability scanning, coverage trending, secret scanning |
|
||||
| **Doc sync** | Collect documentation from multi-repo workspaces into a central location |
|
||||
|
||||
```go
|
||||
import (
|
||||
"forge.lthn.ai/core/go-devops/ansible"
|
||||
"forge.lthn.ai/core/go-devops/build"
|
||||
"forge.lthn.ai/core/go-devops/release"
|
||||
)
|
||||
## Package layout
|
||||
|
||||
// Run an Ansible playbook over SSH
|
||||
pb, _ := ansible.ParsePlaybook("playbooks/deploy.yml")
|
||||
inv, _ := ansible.ParseInventory("inventory.yml")
|
||||
pb.Run(ctx, inv)
|
||||
|
||||
// Build and release
|
||||
artifacts, _ := build.Build(ctx, ".")
|
||||
release.Publish(ctx, releaseCfg, false)
|
||||
```
|
||||
go-devops/
|
||||
├── ansible/ Ansible playbook execution engine (native Go, no shell-out)
|
||||
├── build/ Build system: project detection, archives, checksums
|
||||
│ ├── builders/ Builders: Go, Wails, Docker, C++, LinuxKit, Taskfile
|
||||
│ ├── signing/ Code signing: macOS codesign, GPG, Windows signtool
|
||||
│ └── buildcmd/ CLI handlers for core build / core release
|
||||
├── container/ LinuxKit VM management, hypervisor abstraction
|
||||
├── deploy/ Deployment integrations (Coolify PaaS, embedded Python)
|
||||
├── devkit/ Code quality, security, coverage trending
|
||||
├── devops/ Portable dev environment management
|
||||
│ └── sources/ Image download: GitHub Releases, S3/CDN
|
||||
├── infra/ Infrastructure APIs: Hetzner Cloud, Hetzner Robot, CloudNS
|
||||
├── release/ Release orchestration: version, changelog, publishing
|
||||
│ └── publishers/ 8 publisher backends
|
||||
├── sdk/ OpenAPI SDK generation and breaking change detection
|
||||
│ └── generators/ Language generators: TypeScript, Python, Go, PHP
|
||||
├── snapshot/ Frozen release manifest generation (core.json)
|
||||
└── cmd/ CLI command registrations
|
||||
├── dev/ Multi-repo workflow commands (work, health, commit, push, pull)
|
||||
├── docs/ Documentation sync and listing
|
||||
├── deploy/ Coolify deployment commands
|
||||
├── setup/ Repository and CI bootstrapping
|
||||
└── gitcmd/ Git helpers
|
||||
```
|
||||
|
||||
## Key Packages
|
||||
## CLI commands
|
||||
|
||||
| Package | Description |
|
||||
|---------|-------------|
|
||||
| `ansible` | Native Go Ansible executor (~30 modules) |
|
||||
| `build` | Multi-target build pipeline with auto-detection |
|
||||
| `release` | Changelog generation + 8 publisher backends |
|
||||
| `hetzner` | Hetzner Cloud and Robot API clients |
|
||||
| `cloudns` | CloudNS DNS management |
|
||||
| `sdk` | OpenAPI SDK generator |
|
||||
| `devkit` | Complexity analysis, vuln scanning, coverage |
|
||||
go-devops registers commands into the `core` CLI binary (built from `forge.lthn.ai/core/cli`). Key commands:
|
||||
|
||||
```bash
|
||||
# Build
|
||||
core build # Auto-detect project type, build for configured targets
|
||||
core build --ci # All targets, JSON output
|
||||
core build sdk # Generate SDKs from OpenAPI spec
|
||||
|
||||
# Release
|
||||
core build release # Build + changelog + publish (requires --we-are-go-for-launch)
|
||||
|
||||
# Multi-repo development
|
||||
core dev health # Quick summary across all repos
|
||||
core dev work # Combined status, commit, push workflow
|
||||
core dev commit # Claude-assisted commits for dirty repos
|
||||
core dev push # Push repos with unpushed commits
|
||||
core dev pull # Pull repos behind remote
|
||||
|
||||
# GitHub integration
|
||||
core dev issues # List open issues across repos
|
||||
core dev reviews # PRs needing review
|
||||
core dev ci # GitHub Actions status
|
||||
|
||||
# Documentation
|
||||
core docs list # Scan repos for docs
|
||||
core docs sync # Copy docs to central location
|
||||
core docs sync --target gohelp # Sync to go-help format
|
||||
|
||||
# Deployment
|
||||
core deploy servers # List Coolify servers
|
||||
core deploy apps # List Coolify applications
|
||||
|
||||
# Setup
|
||||
core setup repo # Generate .core/ configuration for a repo
|
||||
core setup ci # Bootstrap CI configuration
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Two YAML files in `.core/` at the project root control build and release behaviour:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `.core/build.yaml` | Project name, binary, build flags, cross-compilation targets |
|
||||
| `.core/release.yaml` | Repository, changelog rules, publisher configs, SDK settings |
|
||||
|
||||
See [Build System](build-system.md) and [Publishers](publishers.md) for full configuration reference.
|
||||
|
||||
## Core interfaces
|
||||
|
||||
Every extensible subsystem is defined by a small interface:
|
||||
|
||||
```go
|
||||
// Builder — project type plugin (build/builders/)
|
||||
type Builder interface {
|
||||
Name() string
|
||||
Detect(fs io.Medium, dir string) (bool, error)
|
||||
Build(ctx context.Context, cfg *Config, targets []Target) ([]Artifact, error)
|
||||
}
|
||||
|
||||
// Publisher — distribution target plugin (release/publishers/)
|
||||
type Publisher interface {
|
||||
Name() string
|
||||
Publish(ctx context.Context, release *Release, pubCfg PublisherConfig,
|
||||
relCfg ReleaseConfig, dryRun bool) error
|
||||
}
|
||||
|
||||
// Generator — SDK language generator (sdk/generators/)
|
||||
type Generator interface {
|
||||
Language() string
|
||||
Generate(ctx context.Context, spec, outputDir string, config *Config) error
|
||||
}
|
||||
|
||||
// Signer — code signing plugin (build/signing/)
|
||||
type Signer interface {
|
||||
Name() string
|
||||
Available() bool
|
||||
Sign(filePath, keyID string) ([]byte, error)
|
||||
}
|
||||
```
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Build System](build-system.md) — Builders, project detection, `.core/build.yaml` reference
|
||||
- [Publishers](publishers.md) — Release publishers, `.core/release.yaml` reference
|
||||
- [SDK Generation](sdk-generation.md) — OpenAPI client generation and breaking change detection
|
||||
- [Doc Sync](sync.md) — Documentation sync across multi-repo workspaces
|
||||
- [Architecture](architecture.md) — Full architecture deep-dive (Ansible, infra, devkit, containers)
|
||||
- [Development Guide](development.md) — Building, testing, coding standards
|
||||
|
|
|
|||
|
|
@ -1,18 +1,151 @@
|
|||
---
|
||||
title: go-forge
|
||||
description: Forgejo API client covering actions, admin, repos, branches, and more
|
||||
description: Full-coverage Go client for the Forgejo API with generics-based CRUD, pagination, and code-generated types.
|
||||
---
|
||||
|
||||
# go-forge
|
||||
|
||||
`forge.lthn.ai/core/go-forge`
|
||||
`forge.lthn.ai/core/go-forge` is a Go client library for the [Forgejo](https://forgejo.org) REST API. It provides typed access to 18 API domains (repositories, issues, pull requests, organisations, and more) through a single top-level `Forge` client. Types are generated directly from Forgejo's `swagger.v1.json` specification, keeping the library in lockstep with the server.
|
||||
|
||||
Comprehensive API client for Forgejo (Gitea-compatible) instances. Covers the full Forgejo REST API surface including repository management, CI/CD actions, admin operations, branch protection, and user management. Handles authentication, rate limiting, and pagination.
|
||||
**Module path:** `forge.lthn.ai/core/go-forge`
|
||||
**Go version:** 1.26+
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
## Key Types
|
||||
|
||||
- `ActionsService` — CI/CD actions and workflow management
|
||||
- `AdminService` — server administration operations
|
||||
- `BranchService` — branch creation, protection, and deletion
|
||||
- `APIError` — structured error from the Forgejo API
|
||||
- `RateLimit` — rate limit state tracking for API calls
|
||||
## Quick start
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"forge.lthn.ai/core/go-forge"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a client with your Forgejo URL and API token.
|
||||
f := forge.NewForge("https://forge.lthn.ai", "your-token")
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// List repositories for an organisation (first page, 50 per page).
|
||||
result, err := f.Repos.List(ctx, forge.Params{"org": "core"}, forge.DefaultList)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
for _, repo := range result.Items {
|
||||
fmt.Println(repo.Name)
|
||||
}
|
||||
|
||||
// Get a single repository.
|
||||
repo, err := f.Repos.Get(ctx, forge.Params{"owner": "core", "repo": "go-forge"})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Printf("%s — %s\n", repo.FullName, repo.Description)
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration from environment
|
||||
|
||||
If you prefer to resolve the URL and token from environment variables rather than hard-coding them, use `NewForgeFromConfig`:
|
||||
|
||||
```go
|
||||
// Priority: flags > env (FORGE_URL, FORGE_TOKEN) > defaults (http://localhost:3000)
|
||||
f, err := forge.NewForgeFromConfig("", "", forge.WithUserAgent("my-tool/1.0"))
|
||||
if err != nil {
|
||||
log.Fatal(err) // no token configured
|
||||
}
|
||||
```
|
||||
|
||||
Environment variables:
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|---------------|--------------------------------------|--------------------------|
|
||||
| `FORGE_URL` | Base URL of the Forgejo instance | `http://localhost:3000` |
|
||||
| `FORGE_TOKEN` | API token for authentication | (none -- required) |
|
||||
|
||||
|
||||
## Package layout
|
||||
|
||||
```
|
||||
go-forge/
|
||||
├── client.go HTTP client, auth, error handling, rate limits
|
||||
├── config.go Config resolution: flags > env > defaults
|
||||
├── forge.go Top-level Forge struct aggregating all 18 services
|
||||
├── resource.go Generic Resource[T, C, U] for CRUD operations
|
||||
├── pagination.go ListPage, ListAll, ListIter — paginated requests
|
||||
├── params.go Path variable resolution ({owner}/{repo} -> values)
|
||||
├── repos.go RepoService — repositories, forks, transfers, mirrors
|
||||
├── issues.go IssueService — issues, comments, labels, reactions
|
||||
├── pulls.go PullService — pull requests, merges, reviews
|
||||
├── orgs.go OrgService — organisations, members
|
||||
├── users.go UserService — users, followers, stars
|
||||
├── teams.go TeamService — teams, members, repositories
|
||||
├── admin.go AdminService — site admin, cron, user management
|
||||
├── branches.go BranchService — branches, branch protections
|
||||
├── releases.go ReleaseService — releases, assets, tags
|
||||
├── labels.go LabelService — repo and org labels
|
||||
├── webhooks.go WebhookService — repo and org webhooks
|
||||
├── notifications.go NotificationService — notifications, threads
|
||||
├── packages.go PackageService — package registry
|
||||
├── actions.go ActionsService — CI/CD secrets, variables, dispatches
|
||||
├── contents.go ContentService — file read/write/delete
|
||||
├── wiki.go WikiService — wiki pages
|
||||
├── commits.go CommitService — statuses, notes
|
||||
├── misc.go MiscService — markdown, licences, gitignore, version
|
||||
├── types/ 229 generated Go types from swagger.v1.json
|
||||
│ ├── generate.go go:generate directive
|
||||
│ ├── repo.go Repository, CreateRepoOption, EditRepoOption, ...
|
||||
│ ├── issue.go Issue, CreateIssueOption, ...
|
||||
│ ├── pr.go PullRequest, CreatePullRequestOption, ...
|
||||
│ └── ... (36 files total, grouped by domain)
|
||||
├── cmd/forgegen/ Code generator: swagger spec -> types/*.go
|
||||
│ ├── main.go CLI entry point
|
||||
│ ├── parser.go Swagger spec parsing, type extraction, CRUD pair detection
|
||||
│ └── generator.go Template-based Go source file generation
|
||||
└── testdata/
|
||||
└── swagger.v1.json Forgejo API specification (input for codegen)
|
||||
```
|
||||
|
||||
|
||||
## Services
|
||||
|
||||
The `Forge` struct exposes 18 service fields, each handling a different API domain:
|
||||
|
||||
| Service | Struct | Embedding | Domain |
|
||||
|-----------------|---------------------|----------------------------------|--------------------------------------|
|
||||
| `Repos` | `RepoService` | `Resource[Repository, ...]` | Repositories, forks, transfers |
|
||||
| `Issues` | `IssueService` | `Resource[Issue, ...]` | Issues, comments, labels, reactions |
|
||||
| `Pulls` | `PullService` | `Resource[PullRequest, ...]` | Pull requests, merges, reviews |
|
||||
| `Orgs` | `OrgService` | `Resource[Organization, ...]` | Organisations, members |
|
||||
| `Users` | `UserService` | `Resource[User, ...]` | Users, followers, stars |
|
||||
| `Teams` | `TeamService` | `Resource[Team, ...]` | Teams, members, repos |
|
||||
| `Admin` | `AdminService` | (standalone) | Site admin, cron, user management |
|
||||
| `Branches` | `BranchService` | `Resource[Branch, ...]` | Branches, protections |
|
||||
| `Releases` | `ReleaseService` | `Resource[Release, ...]` | Releases, assets, tags |
|
||||
| `Labels` | `LabelService` | (standalone) | Repo and org labels |
|
||||
| `Webhooks` | `WebhookService` | `Resource[Hook, ...]` | Repo and org webhooks |
|
||||
| `Notifications` | `NotificationService` | (standalone) | Notifications, threads |
|
||||
| `Packages` | `PackageService` | (standalone) | Package registry |
|
||||
| `Actions` | `ActionsService` | (standalone) | CI/CD secrets, variables, dispatches |
|
||||
| `Contents` | `ContentService` | (standalone) | File read/write/delete |
|
||||
| `Wiki` | `WikiService` | (standalone) | Wiki pages |
|
||||
| `Commits` | `CommitService` | (standalone) | Commit statuses, git notes |
|
||||
| `Misc` | `MiscService` | (standalone) | Markdown, licences, gitignore, version |
|
||||
|
||||
Services that embed `Resource[T, C, U]` inherit `List`, `ListAll`, `Iter`, `Get`, `Create`, `Update`, and `Delete` methods automatically. Standalone services have hand-written methods because their API endpoints are heterogeneous and do not fit a uniform CRUD pattern.
|
||||
|
||||
|
||||
## Dependencies
|
||||
|
||||
This module has **zero external dependencies**. It relies solely on the Go standard library (`net/http`, `encoding/json`, `context`, `iter`, etc.) and requires Go 1.26 or later.
|
||||
|
||||
```
|
||||
module forge.lthn.ai/core/go-forge
|
||||
|
||||
go 1.26.0
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,18 +1,125 @@
|
|||
---
|
||||
title: go-git
|
||||
description: Git operations helper for status, push, and repository queries
|
||||
description: Multi-repository Git operations library for Go with parallel status checking and Core framework integration.
|
||||
---
|
||||
|
||||
# go-git
|
||||
|
||||
`forge.lthn.ai/core/go-git`
|
||||
**Module:** `forge.lthn.ai/core/go-git`
|
||||
|
||||
Helper library for common Git operations. Provides structured output for repository status, push results, and error handling around Git CLI commands.
|
||||
**Go version:** 1.26+
|
||||
|
||||
## Key Types
|
||||
**Licence:** [EUPL-1.2](../LICENSE.md)
|
||||
|
||||
- `RepoStatus` — parsed state of a Git repository (branch, dirty files, ahead/behind)
|
||||
- `StatusOptions` — options controlling what status information to collect
|
||||
- `PushResult` — structured result of a push operation
|
||||
- `GitError` — typed error wrapping Git CLI failures with exit codes
|
||||
- `QueryStatus` — batch query result for multiple repositories
|
||||
## What it does
|
||||
|
||||
go-git is a Go library for orchestrating Git operations across multiple repositories. It was extracted from `forge.lthn.ai/core/go-scm/git/` into a standalone module.
|
||||
|
||||
The library provides two layers:
|
||||
|
||||
1. **Standalone functions** -- pure Git operations that depend only on the standard library.
|
||||
2. **Core service integration** -- a `Service` type that plugs into the Core DI framework, exposing Git operations via the query/task message bus.
|
||||
|
||||
Typical use cases include multi-repo status dashboards, batch push/pull workflows, and CI tooling that needs to inspect many repositories at once.
|
||||
|
||||
## Quick start
|
||||
|
||||
### Standalone usage (no framework)
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
git "forge.lthn.ai/core/go-git"
|
||||
)
|
||||
|
||||
func main() {
|
||||
statuses := git.Status(context.Background(), git.StatusOptions{
|
||||
Paths: []string{"/home/dev/repo-a", "/home/dev/repo-b"},
|
||||
Names: map[string]string{
|
||||
"/home/dev/repo-a": "repo-a",
|
||||
"/home/dev/repo-b": "repo-b",
|
||||
},
|
||||
})
|
||||
|
||||
for _, s := range statuses {
|
||||
if s.Error != nil {
|
||||
fmt.Printf("%s: error: %v\n", s.Name, s.Error)
|
||||
continue
|
||||
}
|
||||
fmt.Printf("%s [%s]: modified=%d untracked=%d staged=%d ahead=%d behind=%d\n",
|
||||
s.Name, s.Branch, s.Modified, s.Untracked, s.Staged, s.Ahead, s.Behind)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### With the Core framework
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"forge.lthn.ai/core/go/pkg/core"
|
||||
git "forge.lthn.ai/core/go-git"
|
||||
)
|
||||
|
||||
func main() {
|
||||
c, err := core.New(
|
||||
core.WithService(git.NewService(git.ServiceOptions{
|
||||
WorkDir: "/home/dev/projects",
|
||||
})),
|
||||
)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Query status via the message bus.
|
||||
result, err := c.Query(git.QueryStatus{
|
||||
Paths: []string{"/home/dev/projects/repo-a"},
|
||||
Names: map[string]string{"/home/dev/projects/repo-a": "repo-a"},
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
statuses := result.([]git.RepoStatus)
|
||||
for _, s := range statuses {
|
||||
fmt.Printf("%s: dirty=%v ahead=%v\n", s.Name, s.IsDirty(), s.HasUnpushed())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Package layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `git.go` | Standalone Git operations -- `Status`, `Push`, `Pull`, `PushMultiple`, error types. Zero framework dependencies. |
|
||||
| `service.go` | Core framework integration -- `Service`, query types (`QueryStatus`, `QueryDirtyRepos`, `QueryAheadRepos`), task types (`TaskPush`, `TaskPull`, `TaskPushMultiple`). |
|
||||
| `git_test.go` | Tests for standalone operations using real temporary Git repositories. |
|
||||
| `service_test.go` | Tests for `Service` filtering helpers (`DirtyRepos`, `AheadRepos`, iterators). |
|
||||
| `service_extra_test.go` | Integration tests for `Service` query/task handlers against the Core framework. |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Purpose |
|
||||
|------------|---------|
|
||||
| `forge.lthn.ai/core/go/pkg/core` | DI container, `ServiceRuntime`, query/task bus (used only by `service.go`). |
|
||||
| `github.com/stretchr/testify` | Assertions in tests (test-only). |
|
||||
|
||||
The standalone layer (`git.go`) uses only the Go standard library. It shells out to the system `git` binary -- there is no embedded Git implementation.
|
||||
|
||||
## Build targets
|
||||
|
||||
Defined in `.core/build.yaml`:
|
||||
|
||||
| OS | Architecture |
|
||||
|----|-------------|
|
||||
| Linux | amd64 |
|
||||
| Linux | arm64 |
|
||||
| Darwin | arm64 |
|
||||
| Windows | amd64 |
|
||||
|
|
|
|||
|
|
@ -1,35 +1,80 @@
|
|||
---
|
||||
title: go-html
|
||||
description: HLCRF DOM compositor with grammar pipeline integration for type-safe server-side HTML generation and optional WASM client rendering.
|
||||
---
|
||||
|
||||
# go-html
|
||||
|
||||
HLCRF DOM compositor with grammar pipeline integration.
|
||||
`go-html` is a pure-Go library for building HTML documents as type-safe node trees and rendering them to string output. It provides a five-slot layout compositor (Header, Left, Content, Right, Footer -- abbreviated HLCRF), a responsive multi-variant wrapper, a server-side grammar analysis pipeline, a Web Component code generator, and an optional WASM module for client-side rendering.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-html`
|
||||
|
||||
Provides a type-safe node tree (El, Text, Raw, If, Each, Switch, Entitled), a five-slot Header/Left/Content/Right/Footer layout compositor with deterministic `data-block` path IDs and ARIA roles, a responsive multi-variant wrapper, a server-side grammar pipeline (StripTags, GrammarImprint via go-i18n reversal, CompareVariants), a build-time Web Component codegen CLI, and a WASM module (2.90 MB raw, 842 KB gzip) exposing `renderToString()`.
|
||||
**Module path:** `forge.lthn.ai/core/go-html`
|
||||
**Go version:** 1.26
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-html"
|
||||
package main
|
||||
|
||||
page := html.NewLayout("HCF").
|
||||
H(html.El("nav", html.Text("i18n.label.navigation"))).
|
||||
C(html.El("main",
|
||||
html.El("h1", html.Text("i18n.label.welcome")),
|
||||
html.Each(items, func(item Item) html.Node {
|
||||
return html.El("li", html.Text(item.Name))
|
||||
}),
|
||||
)).
|
||||
F(html.El("footer", html.Text("i18n.label.copyright")))
|
||||
import html "forge.lthn.ai/core/go-html"
|
||||
|
||||
rendered := page.Render(html.NewContext("en-GB"))
|
||||
func main() {
|
||||
page := html.NewLayout("HCF").
|
||||
H(html.El("nav", html.Text("nav.label"))).
|
||||
C(html.El("article",
|
||||
html.El("h1", html.Text("page.title")),
|
||||
html.Each(items, func(item Item) html.Node {
|
||||
return html.El("li", html.Text(item.Name))
|
||||
}),
|
||||
)).
|
||||
F(html.El("footer", html.Text("footer.copyright")))
|
||||
|
||||
output := page.Render(html.NewContext())
|
||||
}
|
||||
```
|
||||
|
||||
## Layout Slots
|
||||
This builds a Header-Content-Footer layout with semantic HTML elements (`<header>`, `<main>`, `<footer>`), ARIA roles, and deterministic `data-block` path identifiers. Text nodes pass through the `go-i18n` translation layer and are HTML-escaped by default.
|
||||
|
||||
| Slot | Method | ARIA Role |
|
||||
|------|--------|-----------|
|
||||
| Header | `.H()` | `banner` |
|
||||
| Left | `.L()` | `navigation` |
|
||||
| Content | `.C()` | `main` |
|
||||
| Right | `.R()` | `complementary` |
|
||||
| Footer | `.F()` | `contentinfo` |
|
||||
## Package Layout
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `node.go` | `Node` interface and all node types: `El`, `Text`, `Raw`, `If`, `Unless`, `Each`, `EachSeq`, `Switch`, `Entitled` |
|
||||
| `layout.go` | HLCRF compositor with semantic HTML elements and ARIA roles |
|
||||
| `responsive.go` | Multi-variant breakpoint wrapper (`data-variant` containers) |
|
||||
| `context.go` | Rendering context: identity, locale, entitlements, i18n service |
|
||||
| `render.go` | `Render()` convenience function |
|
||||
| `path.go` | `ParseBlockID()` for decoding `data-block` path attributes |
|
||||
| `pipeline.go` | `StripTags`, `Imprint`, `CompareVariants` (server-side only, `!js` build tag) |
|
||||
| `codegen/codegen.go` | Web Component class generation (closed Shadow DOM) |
|
||||
| `cmd/codegen/main.go` | Build-time CLI: JSON slot map on stdin, JS bundle on stdout |
|
||||
| `cmd/wasm/main.go` | WASM entry point exporting `renderToString()` to JavaScript |
|
||||
|
||||
## Key Concepts
|
||||
|
||||
**Node tree** -- All renderable units implement `Node`, a single-method interface: `Render(ctx *Context) string`. The library composes nodes into trees using `El()` for elements, `Text()` for translated text, and control-flow constructors (`If`, `Unless`, `Each`, `Switch`, `Entitled`).
|
||||
|
||||
**HLCRF Layout** -- A five-slot compositor that maps to semantic HTML: `<header>` (H), `<aside>` (L/R), `<main>` (C), `<footer>` (F). The variant string controls which slots render: `"HLCRF"` for all five, `"HCF"` for three, `"C"` for content only. Layouts nest: placing a `Layout` inside another layout's slot produces hierarchical `data-block` paths like `L-0-C-0`.
|
||||
|
||||
**Responsive variants** -- `Responsive` wraps multiple `Layout` instances with named breakpoints (e.g. `"desktop"`, `"mobile"`). Each variant renders inside a `<div data-variant="name">` container for CSS or JavaScript targeting.
|
||||
|
||||
**Grammar pipeline** -- Server-side only. `Imprint()` renders a node tree to HTML, strips tags, tokenises the plain text via `go-i18n/reversal`, and returns a `GrammarImprint` for semantic analysis. `CompareVariants()` computes pairwise similarity scores across responsive variants.
|
||||
|
||||
**Web Component codegen** -- `cmd/codegen/` generates ES2022 Web Component classes with closed Shadow DOM from a JSON slot-to-tag mapping. This is a build-time tool, not used at runtime.
|
||||
|
||||
## Dependencies
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go-html
|
||||
forge.lthn.ai/core/go-i18n (direct, all builds)
|
||||
forge.lthn.ai/core/go-inference (indirect, via go-i18n)
|
||||
forge.lthn.ai/core/go-i18n/reversal (server builds only, !js)
|
||||
github.com/stretchr/testify (test only)
|
||||
```
|
||||
|
||||
Both `go-i18n` and `go-inference` must be present on the local filesystem. The `go.mod` uses `replace` directives pointing to sibling directories (`../go-i18n`, `../go-inference`).
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- Node interface, HLCRF layout internals, responsive compositor, grammar pipeline, WASM module, codegen CLI
|
||||
- [Development](development.md) -- Building, testing, benchmarks, WASM builds, coding standards, contribution guide
|
||||
|
|
|
|||
|
|
@ -1,37 +1,86 @@
|
|||
# go-i18n
|
||||
---
|
||||
title: go-i18n Grammar Engine
|
||||
description: Grammar-aware internationalisation for Go with forward composition and reverse decomposition.
|
||||
---
|
||||
|
||||
Grammar engine for Go.
|
||||
# go-i18n Grammar Engine
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-i18n`
|
||||
`forge.lthn.ai/core/go-i18n` is a **grammar engine** for Go. Unlike flat key-value translation systems, it composes grammatically correct output from verbs, nouns, and articles -- and can reverse the process, decomposing inflected text back into base forms with grammatical metadata.
|
||||
|
||||
Provides forward composition primitives (PastTense, Gerund, Pluralize, Article, composite progress and label functions), a `T()` translation entry point with namespace key handlers, and a reversal engine that recovers base forms and grammatical roles from inflected text. The reversal package produces `GrammarImprint` feature vectors for semantic similarity scoring, builds reference domain distributions, performs anomaly detection, and includes a 1B model pre-sort pipeline for training data classification.
|
||||
This is the foundation for the Poindexter classification pipeline and the LEM scoring system.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Layer | Package | Purpose |
|
||||
|-------|---------|---------|
|
||||
| Forward | Root (`i18n`) | Compose grammar-aware messages: `T()`, `PastTense()`, `Gerund()`, `Pluralize()`, `Article()` |
|
||||
| Reverse | `reversal/` | Decompose text back to base forms with tense/number metadata |
|
||||
| Imprint | `reversal/` | Lossy feature vector projection for grammar fingerprinting |
|
||||
| Multiply | `reversal/` | Deterministic training data augmentation |
|
||||
| Classify | Root (`i18n`) | 1B model domain classification pipeline |
|
||||
| Data | `locales/` | Grammar tables (JSON) -- only `gram.*` data |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-i18n"
|
||||
import i18n "forge.lthn.ai/core/go-i18n"
|
||||
|
||||
// Grammar primitives
|
||||
fmt.Println(i18n.PastTense("delete")) // "deleted"
|
||||
fmt.Println(i18n.Gerund("build")) // "building"
|
||||
fmt.Println(i18n.Pluralize("file", 3)) // "files"
|
||||
// Initialise the default service (uses embedded en.json)
|
||||
svc, err := i18n.New()
|
||||
i18n.SetDefault(svc)
|
||||
|
||||
// Translation with auto-composed output
|
||||
fmt.Println(i18n.T("i18n.progress.build")) // "Building..."
|
||||
fmt.Println(i18n.T("i18n.done.delete", "file")) // "File deleted"
|
||||
|
||||
// Reversal: recover grammar from text
|
||||
tokeniser := reversal.NewTokeniser()
|
||||
tokens := tokeniser.Tokenise("deleted the files")
|
||||
imprint := reversal.NewImprint(tokens)
|
||||
// Forward composition
|
||||
i18n.T("i18n.progress.build") // "Building..."
|
||||
i18n.T("i18n.done.delete", "cache") // "Cache deleted"
|
||||
i18n.T("i18n.count.file", 5) // "5 files"
|
||||
i18n.PastTense("commit") // "committed"
|
||||
i18n.Article("SSH") // "an"
|
||||
```
|
||||
|
||||
## Components
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-i18n/reversal"
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| Forward | PastTense, Gerund, Pluralize, Article, progress/label composers |
|
||||
| `T()` | Translation entry point with namespace key handlers |
|
||||
| Reversal | Base form recovery, grammatical role detection |
|
||||
| GrammarImprint | Feature vectors for semantic similarity scoring |
|
||||
| 1B Pipeline | Training data pre-sort and classification |
|
||||
// Reverse decomposition
|
||||
tok := reversal.NewTokeniser()
|
||||
tokens := tok.Tokenise("Deleted the configuration files")
|
||||
|
||||
// Grammar fingerprinting
|
||||
imp := reversal.NewImprint(tokens)
|
||||
sim := imp.Similar(otherImp) // 0.0-1.0
|
||||
|
||||
// Training data augmentation
|
||||
m := reversal.NewMultiplier()
|
||||
variants := m.Expand("Delete the file") // 4-7 grammatical variants
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Forward API](forward-api.md) -- `T()`, grammar primitives, namespace handlers, Subject builder
|
||||
- [Reversal Engine](reversal.md) -- 3-tier tokeniser, matching, morphology rules, round-trip verification
|
||||
- [GrammarImprint](grammar-imprint.md) -- Lossy feature vectors, weighted cosine similarity, reference distributions
|
||||
- [Locale JSON Schema](locale-schema.md) -- `en.json` structure, grammar table contract, sacred rules
|
||||
- [Multiplier](multiplier.md) -- Deterministic variant generation, case preservation, round-trip guarantee
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
**Grammar engine, not translation file manager.** Consumers bring their own translations. go-i18n provides the grammatical composition and decomposition primitives.
|
||||
|
||||
**3-tier lookup.** All grammar lookups follow the same pattern: JSON locale data (tier 1) takes precedence over irregular Go maps (tier 2), which take precedence over regular morphology rules (tier 3). This lets locale files override any built-in rule.
|
||||
|
||||
**Round-trip verification.** The reversal engine verifies tier 3 candidates by applying the forward function and checking the result matches the original. This eliminates phantom base forms like "walke" or "processe".
|
||||
|
||||
**Lossy imprints.** GrammarImprint intentionally discards content, preserving only grammatical structure. Two texts with similar grammar produce similar imprints regardless of subject matter. This is a privacy-preserving proxy for semantic similarity.
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
go test ./... # All tests
|
||||
go test -v ./reversal/ # Reversal engine tests
|
||||
go test -bench=. ./... # Benchmarks
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- **Phase 1** (Harden): Dual-class disambiguation -- design approved, implementation in progress
|
||||
- **Phase 2** (Reference Distributions): 1B pre-classification pipeline + imprint calibration
|
||||
- **Phase 3** (Multi-Language): French grammar tables
|
||||
|
|
|
|||
|
|
@ -1,45 +1,106 @@
|
|||
---
|
||||
title: go-inference
|
||||
description: Shared interfaces for text generation backends in the Core Go ecosystem.
|
||||
---
|
||||
|
||||
# go-inference
|
||||
|
||||
Shared interface contract for text generation backends.
|
||||
Module: `forge.lthn.ai/core/go-inference`
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-inference`
|
||||
go-inference defines the shared contract between GPU-specific inference backends and their consumers. It contains the interfaces, types, and registry that let a consumer load a model and generate text without knowing which GPU runtime is underneath.
|
||||
|
||||
Defines `TextModel`, `Backend`, `Token`, `Message`, and associated configuration types that GPU-specific backends implement and consumers depend on. Zero external dependencies — stdlib only — and compiles on all platforms regardless of GPU availability. The backend registry supports automatic selection (Metal preferred on macOS, ROCm on Linux) and explicit pinning.
|
||||
## Why it exists
|
||||
|
||||
## Quick Start
|
||||
The Core Go ecosystem has multiple inference backends:
|
||||
|
||||
- **go-mlx** — Apple Metal on macOS (darwin/arm64), native GPU memory access
|
||||
- **go-rocm** — AMD ROCm on Linux (linux/amd64), llama-server subprocess
|
||||
- **go-ml** — scoring engine, also wraps llama.cpp HTTP as a third backend path
|
||||
|
||||
And multiple consumers:
|
||||
|
||||
- **go-ai** — MCP hub exposing inference via 30+ agent tools
|
||||
- **go-i18n** — domain classification via Gemma3-1B
|
||||
- **go-ml** — training pipeline, scoring engine
|
||||
|
||||
Without a shared interface layer, every consumer would need to import every backend directly, dragging in CGO bindings, Metal frameworks, and ROCm libraries on platforms that cannot use them.
|
||||
|
||||
go-inference breaks that coupling. A backend imports go-inference and implements its interfaces. A consumer imports go-inference and programs against those interfaces. Neither needs to know about the other at compile time.
|
||||
|
||||
## Zero dependencies
|
||||
|
||||
The package imports only the Go standard library. The sole exception is `testify` in the test tree. This is a deliberate constraint — the package sits at the base of a dependency graph where backends pull in heavyweight GPU libraries. None of those concerns belong in the interface layer.
|
||||
|
||||
## Ecosystem position
|
||||
|
||||
```
|
||||
go-inference (this package)
|
||||
|
|
||||
|── implemented by ────────────────────────
|
||||
| |
|
||||
go-mlx go-rocm
|
||||
(darwin/arm64, Metal GPU) (linux/amd64, AMD ROCm)
|
||||
| |
|
||||
└──────────── consumed by ─────────────────┘
|
||||
|
|
||||
go-ml
|
||||
(scoring engine, llama.cpp HTTP)
|
||||
|
|
||||
go-ai
|
||||
(MCP hub, 30+ tools)
|
||||
|
|
||||
go-i18n
|
||||
(domain classification)
|
||||
```
|
||||
|
||||
## Package layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `inference.go` | `TextModel`, `Backend` interfaces, backend registry, `LoadModel()` entry point |
|
||||
| `options.go` | `GenerateConfig`, `LoadConfig`, functional options (`WithMaxTokens`, `WithBackend`, etc.) |
|
||||
| `training.go` | `TrainableModel`, `LoRAConfig`, `Adapter` interfaces, `LoadTrainable()` |
|
||||
| `discover.go` | `Discover()` scans directories for model files (config.json + *.safetensors) |
|
||||
|
||||
## Quick start
|
||||
|
||||
```go
|
||||
import (
|
||||
"forge.lthn.ai/core/go-inference"
|
||||
_ "forge.lthn.ai/core/go-mlx" // registers "metal" backend on darwin/arm64
|
||||
)
|
||||
import "forge.lthn.ai/core/go-inference"
|
||||
|
||||
model, err := inference.LoadModel("/path/to/safetensors/model/")
|
||||
defer model.Close()
|
||||
// Load a model (auto-detects the best available backend)
|
||||
m, err := inference.LoadModel("/path/to/model/")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer m.Close()
|
||||
|
||||
for tok := range model.Generate(ctx, "Hello", inference.WithMaxTokens(256)) {
|
||||
// Stream tokens
|
||||
ctx := context.Background()
|
||||
for tok := range m.Generate(ctx, "Once upon a time", inference.WithMaxTokens(128)) {
|
||||
fmt.Print(tok.Text)
|
||||
}
|
||||
```
|
||||
|
||||
## Key Interfaces
|
||||
|
||||
```go
|
||||
type TextModel interface {
|
||||
Generate(ctx context.Context, prompt string, opts ...Option) iter.Seq[Token]
|
||||
Close() error
|
||||
}
|
||||
|
||||
type Backend interface {
|
||||
LoadModel(path string) (TextModel, error)
|
||||
Name() string
|
||||
if err := m.Err(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
## Backend Registry
|
||||
## Further reading
|
||||
|
||||
| Backend | Platform | Registration |
|
||||
|---------|----------|-------------|
|
||||
| Metal (go-mlx) | darwin/arm64 | Auto via `init()` |
|
||||
| ROCm (go-rocm) | linux/amd64 | Auto via `init()` |
|
||||
| HTTP | All | Explicit via `NewHTTPBackend()` |
|
||||
- [Interfaces](interfaces.md) — `TextModel`, `Backend`, `TrainableModel`, `AttentionInspector`
|
||||
- [Types](types.md) — `Token`, `GenerateConfig`, `LoadConfig`, `LoRAConfig`, and all supporting structs
|
||||
- [Backends](backends.md) — How the registry works, how to implement a new backend
|
||||
|
||||
## Stability contract
|
||||
|
||||
This package is the shared contract. Changes here affect go-mlx, go-rocm, and go-ml simultaneously. The rules:
|
||||
|
||||
1. **Never change** existing method signatures on `TextModel` or `Backend`.
|
||||
2. **Only add** methods when two or more consumers have a concrete need.
|
||||
3. **New capability** is expressed as separate interfaces that embed `TextModel`, not by extending `TextModel` itself. Consumers opt in via type assertion.
|
||||
4. **New fields** on `GenerateConfig` or `LoadConfig` are safe — zero-value defaults preserve backwards compatibility.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Go 1.26+ (uses `iter.Seq`, `maps`, `slices`)
|
||||
- No CGO, no build tags, no platform constraints
|
||||
- Licence: EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,18 +1,122 @@
|
|||
---
|
||||
title: go-infra
|
||||
description: Infrastructure management with DNS provider integrations
|
||||
description: Infrastructure provider API clients and YAML-based configuration for managing production environments.
|
||||
---
|
||||
|
||||
# go-infra
|
||||
|
||||
`forge.lthn.ai/core/go-infra`
|
||||
`forge.lthn.ai/core/go-infra` provides typed Go clients for infrastructure provider APIs (Hetzner Cloud, Hetzner Robot, CloudNS) and a declarative YAML configuration layer for describing production topology. It also ships CLI commands for production management (`core prod`) and security monitoring (`core monitor`).
|
||||
|
||||
Infrastructure management library with provider integrations for DNS and cloud resources. Currently supports CloudNS as a DNS provider with full zone and record management. Includes retry logic and structured API error handling.
|
||||
The library has no framework dependencies beyond the Go standard library, YAML parsing, and testify for tests. All HTTP communication goes through a shared `APIClient` that handles retries, exponential backoff, and rate-limit compliance automatically.
|
||||
|
||||
## Key Types
|
||||
## Module Path
|
||||
|
||||
- `RetryConfig` — retry policy for transient API failures
|
||||
- `APIClient` — base HTTP client with authentication and retry support
|
||||
- `CloudNSClient` — CloudNS API client for DNS operations
|
||||
- `CloudNSZone` — DNS zone representation
|
||||
- `CloudNSRecord` — individual DNS record (A, AAAA, CNAME, MX, TXT, etc.)
|
||||
```
|
||||
forge.lthn.ai/core/go-infra
|
||||
```
|
||||
|
||||
Requires **Go 1.26+**.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using the API Clients Directly
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-infra"
|
||||
|
||||
// Hetzner Cloud -- list all servers
|
||||
hc := infra.NewHCloudClient(os.Getenv("HCLOUD_TOKEN"))
|
||||
servers, err := hc.ListServers(ctx)
|
||||
|
||||
// Hetzner Robot -- list dedicated servers
|
||||
hr := infra.NewHRobotClient(user, password)
|
||||
dedicated, err := hr.ListServers(ctx)
|
||||
|
||||
// CloudNS -- ensure a DNS record exists
|
||||
dns := infra.NewCloudNSClient(authID, authPassword)
|
||||
changed, err := dns.EnsureRecord(ctx, "example.com", "www", "A", "1.2.3.4", 300)
|
||||
```
|
||||
|
||||
### Loading Infrastructure Configuration
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-infra"
|
||||
|
||||
// Auto-discover infra.yaml by walking up from the current directory
|
||||
cfg, path, err := infra.Discover(".")
|
||||
|
||||
// Or load a specific file
|
||||
cfg, err := infra.Load("/path/to/infra.yaml")
|
||||
|
||||
// Query the configuration
|
||||
appServers := cfg.AppServers()
|
||||
for name, host := range appServers {
|
||||
fmt.Printf("%s: %s (%s)\n", name, host.IP, host.Role)
|
||||
}
|
||||
```
|
||||
|
||||
### CLI Commands
|
||||
|
||||
When registered with the `core` CLI binary, go-infra provides two command groups:
|
||||
|
||||
```bash
|
||||
# Production infrastructure management
|
||||
core prod status # Health check all hosts, services, and load balancer
|
||||
core prod setup # Phase 1 foundation: discover topology, create LB, configure DNS
|
||||
core prod setup --dry-run # Preview what setup would do
|
||||
core prod setup --step=dns # Run a single setup step
|
||||
core prod dns list # List DNS records for a zone
|
||||
core prod dns set www A 1.2.3.4 # Create or update a DNS record
|
||||
core prod lb status # Show load balancer status and target health
|
||||
core prod lb create # Create load balancer from infra.yaml
|
||||
core prod ssh noc # SSH into a named host
|
||||
|
||||
# Security monitoring (aggregates GitHub Security findings)
|
||||
core monitor # Scan current repo
|
||||
core monitor --all # Scan all repos in registry
|
||||
core monitor --repo core-php # Scan a specific repo
|
||||
core monitor --severity high # Filter by severity
|
||||
core monitor --json # JSON output
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `client.go` | Shared HTTP API client with retry, exponential backoff, and rate-limit handling |
|
||||
| `config.go` | YAML infrastructure configuration parser and typed config structs |
|
||||
| `hetzner.go` | Hetzner Cloud API (servers, load balancers, snapshots) and Hetzner Robot API (dedicated servers) |
|
||||
| `cloudns.go` | CloudNS DNS API (zones, records, ACME challenge helpers) |
|
||||
| `cmd/prod/` | CLI commands for production infrastructure management (`core prod`) |
|
||||
| `cmd/monitor/` | CLI commands for security finding aggregation (`core monitor`) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Direct
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (cobra-based command registration) |
|
||||
| `forge.lthn.ai/core/go-ansible` | SSH client used by `core prod status` for host health checks |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalisation strings for monitor command |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction used by monitor's registry lookup |
|
||||
| `forge.lthn.ai/core/go-log` | Structured error logging |
|
||||
| `forge.lthn.ai/core/go-scm` | Repository registry for multi-repo monitoring |
|
||||
| `gopkg.in/yaml.v3` | YAML parsing for `infra.yaml` |
|
||||
| `github.com/stretchr/testify` | Test assertions |
|
||||
|
||||
The core library types (`config.go`, `client.go`, `hetzner.go`, `cloudns.go`) only depend on the standard library and `gopkg.in/yaml.v3`. The heavier dependencies (`cli`, `go-ansible`, `go-scm`, etc.) are confined to the `cmd/` packages.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Used by | Description |
|
||||
|----------|---------|-------------|
|
||||
| `HCLOUD_TOKEN` | `prod setup`, `prod status`, `prod lb` | Hetzner Cloud API bearer token |
|
||||
| `HETZNER_ROBOT_USER` | `prod setup` | Hetzner Robot API username |
|
||||
| `HETZNER_ROBOT_PASS` | `prod setup` | Hetzner Robot API password |
|
||||
| `CLOUDNS_AUTH_ID` | `prod setup`, `prod dns` | CloudNS sub-auth user ID |
|
||||
| `CLOUDNS_AUTH_PASSWORD` | `prod setup`, `prod dns` | CloudNS auth password |
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,15 +1,121 @@
|
|||
---
|
||||
title: go-io
|
||||
description: File I/O abstractions and utilities
|
||||
description: Unified storage abstraction for Go with pluggable backends — local filesystem, S3, SQLite, in-memory, and key-value.
|
||||
---
|
||||
|
||||
# go-io
|
||||
|
||||
`forge.lthn.ai/core/go-io`
|
||||
`forge.lthn.ai/core/go-io` is a storage abstraction library that provides a single `Medium` interface for reading and writing files across different backends. Write your code against `Medium` once, then swap between local disk, S3, SQLite, or in-memory storage without changing a line of business logic.
|
||||
|
||||
File I/O abstraction layer providing platform-independent file and directory operations. Offers structured file metadata, directory traversal, and utility functions used as a foundation by other packages in the ecosystem.
|
||||
The library also includes `sigil`, a composable data-transformation pipeline for encoding, compression, hashing, and authenticated encryption.
|
||||
|
||||
## Key Types
|
||||
|
||||
- `FileInfo` — structured file metadata (size, permissions, timestamps)
|
||||
- `DirEntry` — directory entry for traversal operations
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import (
|
||||
io "forge.lthn.ai/core/go-io"
|
||||
"forge.lthn.ai/core/go-io/s3"
|
||||
"forge.lthn.ai/core/go-io/node"
|
||||
)
|
||||
|
||||
// Use the pre-initialised local filesystem (unsandboxed, rooted at "/").
|
||||
content, _ := io.Local.Read("/etc/hostname")
|
||||
|
||||
// Create a sandboxed medium restricted to a single directory.
|
||||
sandbox, _ := io.NewSandboxed("/var/data/myapp")
|
||||
_ = sandbox.Write("config.yaml", "key: value")
|
||||
|
||||
// In-memory filesystem with tar serialisation.
|
||||
mem := node.New()
|
||||
mem.AddData("hello.txt", []byte("world"))
|
||||
tarball, _ := mem.ToTar()
|
||||
|
||||
// S3 backend (requires an *s3.Client from the AWS SDK).
|
||||
bucket, _ := s3.New("my-bucket", s3.WithClient(awsClient), s3.WithPrefix("uploads/"))
|
||||
_ = bucket.Write("photo.jpg", rawData)
|
||||
```
|
||||
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Package | Import Path | Purpose |
|
||||
|---------|-------------|---------|
|
||||
| `io` (root) | `forge.lthn.ai/core/go-io` | `Medium` interface, helper functions, `MockMedium` for tests |
|
||||
| `local` | `forge.lthn.ai/core/go-io/local` | Local filesystem backend with path sandboxing and symlink-escape protection |
|
||||
| `s3` | `forge.lthn.ai/core/go-io/s3` | Amazon S3 / S3-compatible backend (Garage, MinIO, etc.) |
|
||||
| `sqlite` | `forge.lthn.ai/core/go-io/sqlite` | SQLite-backed virtual filesystem (pure Go driver, no CGO) |
|
||||
| `node` | `forge.lthn.ai/core/go-io/node` | In-memory filesystem implementing both `Medium` and `fs.FS`, with tar round-tripping |
|
||||
| `datanode` | `forge.lthn.ai/core/go-io/datanode` | Thread-safe in-memory `Medium` backed by Borg's DataNode, with snapshot/restore |
|
||||
| `store` | `forge.lthn.ai/core/go-io/store` | Group-namespaced key-value store (SQLite), with a `Medium` adapter and Go template rendering |
|
||||
| `sigil` | `forge.lthn.ai/core/go-io/sigil` | Composable data transformations: encoding, compression, hashing, XChaCha20-Poly1305 encryption |
|
||||
| `workspace` | `forge.lthn.ai/core/go-io/workspace` | Encrypted workspace service integrated with the Core DI container |
|
||||
|
||||
|
||||
## The Medium Interface
|
||||
|
||||
Every storage backend implements the same 18-method interface:
|
||||
|
||||
```go
|
||||
type Medium interface {
|
||||
// Content operations
|
||||
Read(path string) (string, error)
|
||||
Write(path, content string) error
|
||||
FileGet(path string) (string, error) // alias for Read
|
||||
FileSet(path, content string) error // alias for Write
|
||||
|
||||
// Streaming (for large files)
|
||||
ReadStream(path string) (io.ReadCloser, error)
|
||||
WriteStream(path string) (io.WriteCloser, error)
|
||||
Open(path string) (fs.File, error)
|
||||
Create(path string) (io.WriteCloser, error)
|
||||
Append(path string) (io.WriteCloser, error)
|
||||
|
||||
// Directory operations
|
||||
EnsureDir(path string) error
|
||||
List(path string) ([]fs.DirEntry, error)
|
||||
|
||||
// Metadata
|
||||
Stat(path string) (fs.FileInfo, error)
|
||||
Exists(path string) bool
|
||||
IsFile(path string) bool
|
||||
IsDir(path string) bool
|
||||
|
||||
// Mutation
|
||||
Delete(path string) error
|
||||
DeleteAll(path string) error
|
||||
Rename(oldPath, newPath string) error
|
||||
}
|
||||
```
|
||||
|
||||
All backends implement this interface fully. Backends where a method has no natural equivalent (e.g., `EnsureDir` on S3) provide a safe no-op.
|
||||
|
||||
|
||||
## Cross-Medium Operations
|
||||
|
||||
The root package provides helper functions that accept any `Medium`:
|
||||
|
||||
```go
|
||||
// Copy a file between any two backends.
|
||||
err := io.Copy(localMedium, "source.txt", s3Medium, "dest.txt")
|
||||
|
||||
// Read/Write wrappers that take an explicit medium.
|
||||
content, err := io.Read(medium, "path")
|
||||
err := io.Write(medium, "path", "content")
|
||||
```
|
||||
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Role |
|
||||
|------------|------|
|
||||
| `forge.lthn.ai/core/go-log` | Structured error helper (`E()`) |
|
||||
| `forge.lthn.ai/Snider/Borg` | DataNode in-memory FS (used by `datanode` package) |
|
||||
| `github.com/aws/aws-sdk-go-v2` | S3 client (used by `s3` package) |
|
||||
| `golang.org/x/crypto` | BLAKE2, SHA-3, RIPEMD-160, XChaCha20-Poly1305 (used by `sigil`) |
|
||||
| `modernc.org/sqlite` | Pure Go SQLite driver (used by `sqlite` and `store`) |
|
||||
| `github.com/stretchr/testify` | Test assertions |
|
||||
|
||||
Go version: **1.26.0**
|
||||
|
||||
Licence: **EUPL-1.2**
|
||||
|
|
|
|||
|
|
@ -1,17 +1,99 @@
|
|||
---
|
||||
title: go-log
|
||||
description: Structured logger with Core DI container integration
|
||||
description: Structured logging and error handling for Core applications
|
||||
---
|
||||
|
||||
# go-log
|
||||
|
||||
`forge.lthn.ai/core/go-log`
|
||||
`forge.lthn.ai/core/go-log` provides structured logging and contextual error
|
||||
handling for Go applications built on the Core framework. It is a small,
|
||||
zero-dependency library (only `testify` at test time) that replaces ad-hoc
|
||||
`fmt.Println` / `log.Printf` calls with level-filtered, key-value structured
|
||||
output and a rich error type that carries operation context through the call
|
||||
stack.
|
||||
|
||||
Structured logging library that integrates with the Core dependency injection container. Supports log rotation, configurable output formats, and error wrapping with contextual fields.
|
||||
## Quick Start
|
||||
|
||||
## Key Types
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-log"
|
||||
|
||||
- `Logger` — structured logger with levelled output and field support
|
||||
- `Err` — contextual error wrapper that attaches log fields
|
||||
- `RotationOptions` — configuration for log file rotation (size, age, count)
|
||||
- `Options` — logger configuration (level, format, output targets)
|
||||
// Use the package-level default logger straight away
|
||||
log.SetLevel(log.LevelDebug)
|
||||
log.Info("server started", "port", 8080)
|
||||
log.Warn("high latency", "ms", 320)
|
||||
log.Error("request failed", "err", err)
|
||||
|
||||
// Security events are always visible at Error level
|
||||
log.Security("brute force detected", "ip", "10.0.0.1", "attempts", 47)
|
||||
```
|
||||
|
||||
### Creating a Custom Logger
|
||||
|
||||
```go
|
||||
logger := log.New(log.Options{
|
||||
Level: log.LevelInfo,
|
||||
Output: os.Stdout,
|
||||
RedactKeys: []string{"password", "token", "secret"},
|
||||
})
|
||||
|
||||
logger.Info("login", "user", "admin", "password", "hunter2")
|
||||
// Output: 14:32:01 [INF] login user="admin" password="[REDACTED]"
|
||||
```
|
||||
|
||||
### Structured Errors
|
||||
|
||||
```go
|
||||
// Create an error with operational context
|
||||
err := log.E("db.Connect", "connection refused", underlyingErr)
|
||||
|
||||
// Wrap errors as they bubble up through layers
|
||||
err = log.Wrap(err, "user.Save", "failed to persist user")
|
||||
|
||||
// Inspect the chain
|
||||
log.Op(err) // "user.Save"
|
||||
log.Root(err) // the original underlyingErr
|
||||
log.StackTrace(err) // ["user.Save", "db.Connect"]
|
||||
log.FormatStackTrace(err) // "user.Save -> db.Connect"
|
||||
```
|
||||
|
||||
### Combined Log-and-Return
|
||||
|
||||
```go
|
||||
if err != nil {
|
||||
return log.LogError(err, "handler.Process", "request failed")
|
||||
// Logs at Error level AND returns a wrapped error -- one line instead of three
|
||||
}
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `log.go` | Logger type, log levels, key-value formatting, redaction, default logger, `Username()` helper |
|
||||
| `errors.go` | `Err` structured error type, creation helpers (`E`, `Wrap`, `WrapCode`, `NewCode`), introspection (`Op`, `ErrCode`, `Root`, `StackTrace`), combined log-and-return helpers (`LogError`, `LogWarn`, `Must`) |
|
||||
| `log_test.go` | Tests for the Logger: level filtering, key-value output, redaction, injection prevention, security logging |
|
||||
| `errors_test.go` | Tests for structured errors: creation, wrapping, code propagation, introspection, stack traces, log-and-return helpers |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| Go standard library only | Runtime -- no external dependencies |
|
||||
| `github.com/stretchr/testify` | Test assertions (test-only) |
|
||||
|
||||
The package deliberately avoids external runtime dependencies. Log rotation is
|
||||
supported through an optional `RotationWriterFactory` hook that can be wired up
|
||||
by `core/go-io` or any other provider -- go-log itself carries no file-rotation
|
||||
code.
|
||||
|
||||
## Module Path
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/go-log
|
||||
```
|
||||
|
||||
Requires **Go 1.26+** (uses `iter.Seq` from the standard library).
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,31 +1,120 @@
|
|||
---
|
||||
title: go-ml
|
||||
description: ML inference backends, scoring engine, and agent orchestrator for Go.
|
||||
---
|
||||
|
||||
# go-ml
|
||||
|
||||
ML inference backends, multi-suite scoring engine, and agent orchestrator.
|
||||
`forge.lthn.ai/core/go-ml` provides pluggable inference backends, a multi-suite scoring engine with ethics-aware probes, GGUF model management, and a concurrent worker pipeline for batch evaluation.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-ml`
|
||||
**Size**: ~7,500 LOC across 41 Go files, 6 test files
|
||||
**Licence**: EUPL-1.2
|
||||
|
||||
Provides pluggable backends (Apple Metal via go-mlx, managed llama-server subprocesses, and OpenAI-compatible HTTP APIs), a concurrent scoring engine that evaluates model responses across heuristic, semantic, content, and standard benchmark suites, 23 capability probes, GGUF model management, and an SSH-based agent orchestrator that streams checkpoint evaluation results to InfluxDB and DuckDB.
|
||||
## Core Capabilities
|
||||
|
||||
## Quick Start
|
||||
| Area | Description |
|
||||
|------|-------------|
|
||||
| **Inference backends** | MLX (Metal GPU), llama.cpp (subprocess), HTTP (Ollama/vLLM/OpenAI-compatible) |
|
||||
| **Scoring engine** | Heuristic (regex), semantic (LLM judge), exact match, ethics probes |
|
||||
| **Agent orchestrator** | Multi-model scoring runs with concurrent worker pool |
|
||||
| **Model management** | GGUF format parsing, MLX-to-PEFT conversion, Ollama model creation |
|
||||
| **Data pipeline** | DuckDB storage, Parquet I/O, InfluxDB metrics, HuggingFace publishing |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `core/go` | Framework services, lifecycle, process management |
|
||||
| `core/go-inference` | Shared `TextModel`/`Backend`/`Token` interfaces |
|
||||
| `core/go-mlx` | Native Metal GPU inference (darwin/arm64) |
|
||||
| `core/go-process` | Subprocess management for llama-server |
|
||||
| `core/go-log` | Structured error helpers |
|
||||
| `core/go-api` | REST API route registration |
|
||||
| `go-duckdb` | Embedded analytics database |
|
||||
| `parquet-go` | Columnar data format |
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The package is organised around four layers:
|
||||
|
||||
```
|
||||
Backend Layer — inference.go, backend_http.go, backend_llama.go, backend_mlx.go
|
||||
Pluggable backends behind a common Backend interface.
|
||||
|
||||
Scoring Layer — score.go, heuristic.go, judge.go, exact.go, probes.go
|
||||
Multi-suite concurrent scoring engine.
|
||||
|
||||
Agent Layer — agent_execute.go, agent_eval.go, agent_influx.go, agent_ssh.go
|
||||
Orchestrates checkpoint discovery, evaluation, and result publishing.
|
||||
|
||||
Data Layer — db.go, influx.go, parquet.go, export.go, io.go
|
||||
Storage, metrics, and training data pipeline.
|
||||
```
|
||||
|
||||
## Service Registration
|
||||
|
||||
`go-ml` integrates with the Core DI framework via `Service`:
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-ml"
|
||||
|
||||
// HTTP backend (Ollama, LM Studio, any OpenAI-compatible endpoint)
|
||||
backend := ml.NewHTTPBackend("http://localhost:11434", "qwen3:8b")
|
||||
resp, err := backend.Generate(ctx, "Hello", ml.GenOpts{MaxTokens: 256})
|
||||
|
||||
// Scoring engine
|
||||
engine := ml.NewEngine(backend, ml.Options{Suites: "heuristic,semantic", Concurrency: 4})
|
||||
scores := engine.ScoreAll(responses)
|
||||
c, _ := core.New(
|
||||
core.WithName("ml", ml.NewService(ml.Options{
|
||||
OllamaURL: "http://localhost:11434",
|
||||
JudgeURL: "http://localhost:11434",
|
||||
JudgeModel: "qwen3:8b",
|
||||
Suites: "all",
|
||||
Concurrency: 4,
|
||||
})),
|
||||
)
|
||||
```
|
||||
|
||||
## Components
|
||||
On startup, the service registers configured backends and initialises the scoring engine. It implements `Startable` and `Stoppable` for lifecycle integration.
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| Backends | Metal (go-mlx), llama-server subprocess, HTTP (OpenAI-compatible) |
|
||||
| Scoring | Heuristic, semantic, content, and benchmark suites |
|
||||
| Probes | 23 capability probes for model evaluation |
|
||||
| GGUF | Model management and metadata parsing |
|
||||
| Orchestrator | SSH-based agent dispatch with InfluxDB/DuckDB streaming |
|
||||
### Service Methods
|
||||
|
||||
```go
|
||||
svc.Generate(ctx, "ollama", "Explain LoRA", ml.DefaultGenOpts())
|
||||
svc.ScoreResponses(ctx, responses)
|
||||
svc.RegisterBackend("custom", myBackend)
|
||||
svc.Backend("ollama")
|
||||
svc.DefaultBackend()
|
||||
svc.Judge()
|
||||
svc.Engine()
|
||||
```
|
||||
|
||||
## REST API
|
||||
|
||||
The `api` sub-package provides Gin-based REST endpoints:
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/v1/ml/backends` | List registered backends with availability |
|
||||
| `GET` | `/v1/ml/status` | Service readiness, backend list, judge availability |
|
||||
| `POST` | `/v1/ml/generate` | Generate text against a named backend |
|
||||
|
||||
WebSocket channels: `ml.generate`, `ml.status`.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
// Create an HTTP backend pointing at Ollama
|
||||
backend := ml.NewHTTPBackend("http://localhost:11434", "qwen3:8b")
|
||||
|
||||
// Generate text
|
||||
result, err := backend.Generate(ctx, "What is LoRA?", ml.DefaultGenOpts())
|
||||
fmt.Println(result.Text)
|
||||
|
||||
// Score a batch of responses
|
||||
judge := ml.NewJudge(backend)
|
||||
engine := ml.NewEngine(judge, 4, "heuristic,semantic")
|
||||
scores := engine.ScoreAll(ctx, responses)
|
||||
```
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Scoring Engine](scoring.md) -- Heuristic analysis, LLM judge, probes, benchmarks
|
||||
- [Backends](backends.md) -- HTTP, llama.cpp, MLX, and the inference adapter
|
||||
- [Training Pipeline](training.md) -- Data export, LoRA conversion, adapter management
|
||||
- [Model Management](models.md) -- GGUF parsing, Ollama integration, checkpoint discovery
|
||||
|
|
|
|||
|
|
@ -1,34 +1,127 @@
|
|||
---
|
||||
title: go-mlx
|
||||
description: Native Metal GPU inference and training for Go on Apple Silicon.
|
||||
---
|
||||
|
||||
# go-mlx
|
||||
|
||||
Native Apple Metal GPU inference via mlx-c CGO bindings.
|
||||
`forge.lthn.ai/core/go-mlx` provides native Apple Metal GPU inference and LoRA fine-tuning for Go. It wraps Apple's [MLX](https://github.com/ml-explore/mlx) framework through the [mlx-c](https://github.com/ml-explore/mlx-c) C API, implementing the `inference.Backend` interface from `forge.lthn.ai/core/go-inference`.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-mlx`
|
||||
|
||||
Implements the `inference.Backend` and `inference.TextModel` interfaces from go-inference for Apple Silicon (M1-M4). Supports Gemma 3, Qwen 2/3, and Llama 3 architectures from HuggingFace safetensors format, with fused Metal kernels for RMSNorm, RoPE, and scaled dot-product attention, KV cache management, LoRA fine-tuning with AdamW, and batch inference. A Python subprocess backend (`mlxlm`) is provided as a CGO-free alternative.
|
||||
|
||||
!!! warning "Platform restriction"
|
||||
`darwin/arm64` only. A no-op stub compiles on all other platforms.
|
||||
**Platform:** darwin/arm64 only (Apple Silicon M1-M4). A stub provides `MetalAvailable() bool` returning false on all other platforms.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"forge.lthn.ai/core/go-inference"
|
||||
_ "forge.lthn.ai/core/go-mlx" // registers "metal" backend via init()
|
||||
_ "forge.lthn.ai/core/go-mlx" // registers "metal" backend via init()
|
||||
)
|
||||
|
||||
model, err := inference.LoadModel("/Volumes/Data/lem/safetensors/gemma-3-1b/")
|
||||
defer model.Close()
|
||||
func main() {
|
||||
m, err := inference.LoadModel("/path/to/safetensors/model/")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
defer m.Close()
|
||||
|
||||
for tok := range model.Generate(ctx, "Hello", inference.WithMaxTokens(256)) {
|
||||
fmt.Print(tok.Text)
|
||||
ctx := context.Background()
|
||||
for tok := range m.Generate(ctx, "What is 2+2?", inference.WithMaxTokens(128)) {
|
||||
fmt.Print(tok.Text)
|
||||
}
|
||||
if err := m.Err(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Supported Architectures
|
||||
The blank import (`_ "forge.lthn.ai/core/go-mlx"`) auto-registers the Metal backend. All interaction goes through the `go-inference` interfaces -- go-mlx itself exports only Metal-specific memory controls.
|
||||
|
||||
| Architecture | Models |
|
||||
|-------------|--------|
|
||||
| Gemma 3 | gemma-3-1b, gemma-3-4b |
|
||||
| Qwen 2/3 | qwen3-8b, qwen2.5-* |
|
||||
| Llama 3 | llama-3.2-*, llama-3.1-* |
|
||||
## Features
|
||||
|
||||
- **Streaming inference** -- token-by-token generation via `iter.Seq[Token]` (range-over-func)
|
||||
- **Multi-turn chat** -- native chat templates for Gemma 3, Qwen 2/3, and Llama 3
|
||||
- **Batch inference** -- `Classify` (prefill-only) and `BatchGenerate` (autoregressive) for multiple prompts
|
||||
- **LoRA fine-tuning** -- low-rank adaptation with AdamW optimiser and gradient checkpointing
|
||||
- **Quantisation** -- transparent support for 4-bit and 8-bit quantised models via `QuantizedMatmul`
|
||||
- **Attention inspection** -- extract post-RoPE K vectors from the KV cache for analysis
|
||||
- **Performance metrics** -- prefill/decode tokens per second, GPU memory usage
|
||||
|
||||
## Supported Models
|
||||
|
||||
Models must be in **HuggingFace safetensors format** (not GGUF). Architecture is auto-detected from `config.json`:
|
||||
|
||||
| Architecture | `model_type` values | Tested sizes |
|
||||
|-------------|---------------------|-------------|
|
||||
| Gemma 3 | `gemma3`, `gemma3_text`, `gemma2` | 1B, 4B, 27B |
|
||||
| Qwen 3 | `qwen3`, `qwen2` | 8B+ |
|
||||
| Llama 3 | `llama` | 8B+ |
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| Root (`mlx`) | Public API: Metal backend registration, memory controls, training type exports |
|
||||
| `internal/metal/` | All CGO code: array ops, model loaders, generation, training primitives |
|
||||
| `mlxlm/` | Alternative subprocess backend via Python's mlx-lm (no CGO required) |
|
||||
|
||||
## Metal Memory Controls
|
||||
|
||||
These control the Metal allocator directly, not individual models:
|
||||
|
||||
```go
|
||||
import mlx "forge.lthn.ai/core/go-mlx"
|
||||
|
||||
mlx.SetCacheLimit(4 << 30) // 4 GB cache limit
|
||||
mlx.SetMemoryLimit(32 << 30) // 32 GB hard limit
|
||||
mlx.ClearCache() // release cached memory between chat turns
|
||||
|
||||
fmt.Printf("active: %d MB, peak: %d MB\n",
|
||||
mlx.GetActiveMemory()/1024/1024,
|
||||
mlx.GetPeakMemory()/1024/1024)
|
||||
```
|
||||
|
||||
| Function | Purpose |
|
||||
|----------|---------|
|
||||
| `SetCacheLimit(bytes)` | Soft limit on the allocator cache |
|
||||
| `SetMemoryLimit(bytes)` | Hard ceiling on Metal memory |
|
||||
| `SetWiredLimit(bytes)` | Wired memory limit |
|
||||
| `GetActiveMemory()` | Current live allocations in bytes |
|
||||
| `GetPeakMemory()` | High-water mark since last reset |
|
||||
| `GetCacheMemory()` | Cached (not yet freed) memory |
|
||||
| `ClearCache()` | Release cached memory to the OS |
|
||||
| `ResetPeakMemory()` | Reset the high-water mark |
|
||||
| `GetDeviceInfo()` | Metal GPU hardware information |
|
||||
|
||||
## Performance Baseline
|
||||
|
||||
Measured on M3 Ultra (60-core GPU, 96 GB unified memory):
|
||||
|
||||
| Operation | Throughput |
|
||||
|-----------|-----------|
|
||||
| Gemma3-1B 4-bit prefill | 246 tok/s |
|
||||
| Gemma3-1B 4-bit decode | 82 tok/s |
|
||||
| Gemma3-1B 4-bit classify (4 prompts) | 152 prompts/s |
|
||||
| DeepSeek R1 7B 4-bit decode | 27 tok/s |
|
||||
| Llama 3.1 8B 4-bit decode | 30 tok/s |
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Architecture](architecture.md) -- CGO binding layer, lazy evaluation, memory model, attention, KV cache
|
||||
- [Models](models.md) -- model loading, supported architectures, tokenisation, chat templates
|
||||
- [Training](training.md) -- LoRA fine-tuning, gradient computation, AdamW optimiser, loss functions
|
||||
- [Build Guide](build.md) -- prerequisites, CMake setup, build tags, testing
|
||||
|
||||
## Downstream Consumers
|
||||
|
||||
| Package | Role |
|
||||
|---------|------|
|
||||
| `forge.lthn.ai/core/go-ml` | Imports go-inference + go-mlx for the Metal backend training loop |
|
||||
| `forge.lthn.ai/core/go-i18n` | Gemma3-1B domain classification (Phase 2a) |
|
||||
| `forge.lthn.ai/core/go-rocm` | Sibling AMD GPU backend, same go-inference interfaces |
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,12 +1,94 @@
|
|||
---
|
||||
title: go-p2p
|
||||
description: P2P mesh networking for Lethean with encrypted transport and intent routing
|
||||
title: go-p2p Overview
|
||||
description: P2P mesh networking layer for the Lethean network.
|
||||
---
|
||||
|
||||
# go-p2p
|
||||
|
||||
`forge.lthn.ai/core/go-p2p`
|
||||
P2P networking layer for the Lethean network. Encrypted WebSocket mesh with UEPS wire protocol.
|
||||
|
||||
Peer-to-peer mesh networking library for the Lethean network. Provides Ed25519 node identity, encrypted WebSocket transport with HMAC-SHA256 challenge-response authentication, and KD-tree based peer selection optimised for latency, hop count, geography, and reliability.
|
||||
**Module:** `forge.lthn.ai/core/go-p2p`
|
||||
**Go:** 1.26
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
Implements the UEPS wire protocol (RFC-021) with a TLV builder/reader, intent routing with a threat circuit breaker, and TIM deployment bundle encryption with Zip Slip and decompression-bomb defences.
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
go-p2p/
|
||||
├── node/ P2P mesh: identity, transport, peers, protocol, controller, dispatcher
|
||||
│ └── levin/ Levin binary protocol (header, storage, varint, connection)
|
||||
├── ueps/ UEPS wire protocol (RFC-021): TLV packet builder and stream reader
|
||||
└── logging/ Structured levelled logger with component scoping
|
||||
```
|
||||
|
||||
## What Each Piece Does
|
||||
|
||||
| Component | File(s) | Purpose |
|
||||
|-----------|---------|---------|
|
||||
| [Identity](identity.md) | `identity.go` | X25519 keypair, node ID derivation, HMAC-SHA256 challenge-response |
|
||||
| [Transport](transport.md) | `transport.go` | Encrypted WebSocket connections, SMSG encryption, rate limiting |
|
||||
| [Discovery](discovery.md) | `peer.go` | Peer registry, KD-tree selection, score tracking, allowlist auth |
|
||||
| [UEPS](ueps.md) | `ueps/packet.go`, `ueps/reader.go` | TLV wire protocol with HMAC integrity (RFC-021) |
|
||||
| [Routing](routing.md) | `dispatcher.go` | Intent-based packet routing with threat circuit breaker |
|
||||
| [TIM Bundles](tim.md) | `bundle.go` | Encrypted deployment bundles, tar extraction with Zip Slip defence |
|
||||
| Messages | `message.go` | Message envelope, payload types, protocol version negotiation |
|
||||
| Protocol | `protocol.go` | Response validation, structured error handling |
|
||||
| Controller | `controller.go` | Request-response correlation, remote peer operations |
|
||||
| Worker | `worker.go` | Incoming message dispatch, miner/profile management interfaces |
|
||||
| Buffer Pool | `bufpool.go` | `sync.Pool`-backed JSON encoding for hot paths |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/Snider/Borg` | STMF crypto (keypairs), SMSG encryption, TIM bundle format |
|
||||
| `forge.lthn.ai/Snider/Poindexter` | KD-tree peer scoring and nearest-neighbour selection |
|
||||
| `github.com/gorilla/websocket` | WebSocket transport |
|
||||
| `github.com/google/uuid` | Message and peer ID generation |
|
||||
| `github.com/adrg/xdg` | XDG base directory paths for key and config storage |
|
||||
|
||||
## Message Protocol
|
||||
|
||||
Every message is a JSON-encoded `Message` struct transported over WebSocket. After handshake, all messages are SMSG-encrypted using the X25519-derived shared secret.
|
||||
|
||||
```go
|
||||
type Message struct {
|
||||
ID string `json:"id"` // UUID v4
|
||||
Type MessageType `json:"type"` // Determines payload interpretation
|
||||
From string `json:"from"` // Sender node ID
|
||||
To string `json:"to"` // Recipient node ID
|
||||
Timestamp time.Time `json:"ts"`
|
||||
Payload json.RawMessage `json:"payload"` // Type-specific JSON
|
||||
ReplyTo string `json:"replyTo,omitempty"` // For request-response correlation
|
||||
}
|
||||
```
|
||||
|
||||
### Message Types
|
||||
|
||||
| Category | Types |
|
||||
|----------|-------|
|
||||
| Connection | `handshake`, `handshake_ack`, `ping`, `pong`, `disconnect` |
|
||||
| Operations | `get_stats`, `stats`, `start_miner`, `stop_miner`, `miner_ack` |
|
||||
| Deployment | `deploy`, `deploy_ack` |
|
||||
| Logs | `get_logs`, `logs` |
|
||||
| Error | `error` (codes 1000--1005) |
|
||||
|
||||
## Node Roles
|
||||
|
||||
```go
|
||||
const (
|
||||
RoleController NodeRole = "controller" // Orchestrates work distribution
|
||||
RoleWorker NodeRole = "worker" // Executes compute tasks
|
||||
RoleDual NodeRole = "dual" // Both controller and worker
|
||||
)
|
||||
```
|
||||
|
||||
## Architecture Layers
|
||||
|
||||
The stack has two distinct protocol layers:
|
||||
|
||||
1. **UEPS (low-level)** -- Binary TLV wire protocol with HMAC-SHA256 integrity, intent routing, and threat scoring. Operates beneath the mesh layer. See [ueps.md](ueps.md).
|
||||
|
||||
2. **Node mesh (high-level)** -- JSON-over-WebSocket with SMSG encryption. Handles identity, peer management, controller/worker operations, and deployment bundles.
|
||||
|
||||
The dispatcher bridges the two layers, routing verified UEPS packets to registered intent handlers whilst enforcing the threat circuit breaker.
|
||||
|
|
|
|||
|
|
@ -1,17 +1,124 @@
|
|||
---
|
||||
title: go-process
|
||||
description: Process and daemon management with PID files, health checks, and orchestration
|
||||
description: Process management with Core IPC integration for Go applications.
|
||||
---
|
||||
|
||||
# go-process
|
||||
|
||||
`forge.lthn.ai/core/go-process`
|
||||
`forge.lthn.ai/core/go-process` is a process management library that provides
|
||||
spawning, monitoring, and controlling external processes with real-time output
|
||||
streaming via the Core ACTION (IPC) system. It integrates directly with the
|
||||
[Core DI framework](https://forge.lthn.ai/core/go) as a first-class service.
|
||||
|
||||
Process and daemon management library. Handles PID file creation, health checks, process lifecycle events, and daemon orchestration through a registry. Communicates process state via Core's action/IPC system.
|
||||
## Features
|
||||
|
||||
## Key Types
|
||||
- Spawn and manage external processes with full lifecycle tracking
|
||||
- Real-time stdout/stderr streaming via Core IPC actions
|
||||
- Ring buffer output capture (default 1 MB, configurable)
|
||||
- Process pipeline runner with dependency graphs, sequential, and parallel modes
|
||||
- Daemon mode with PID file locking, health check HTTP server, and graceful shutdown
|
||||
- Daemon registry for tracking running instances across the system
|
||||
- Lightweight `exec` sub-package for one-shot command execution with logging
|
||||
- Thread-safe throughout; designed for concurrent use
|
||||
|
||||
- `ActionProcessStarted` — IPC event fired when a managed process starts
|
||||
- `ActionProcessOutput` — IPC event carrying stdout/stderr output from a process
|
||||
- `ActionProcessExited` — IPC event fired when a managed process exits
|
||||
- `RingBuffer` — bounded circular buffer for capturing recent process output
|
||||
## Quick Start
|
||||
|
||||
### Register with Core
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
framework "forge.lthn.ai/core/go/pkg/core"
|
||||
"forge.lthn.ai/core/go-process"
|
||||
)
|
||||
|
||||
// Create a Core instance with the process service
|
||||
c, err := framework.New(
|
||||
framework.WithName("process", process.NewService(process.Options{})),
|
||||
)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Retrieve the typed service
|
||||
svc, err := framework.ServiceFor[*process.Service](c, "process")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
### Run a Command
|
||||
|
||||
```go
|
||||
// Fire-and-forget (async)
|
||||
proc, err := svc.Start(ctx, "go", "test", "./...")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
<-proc.Done()
|
||||
fmt.Println(proc.Output())
|
||||
|
||||
// Synchronous convenience
|
||||
output, err := svc.Run(ctx, "echo", "hello world")
|
||||
```
|
||||
|
||||
### Listen for Events
|
||||
|
||||
Process lifecycle events are broadcast through Core's ACTION system:
|
||||
|
||||
```go
|
||||
c.RegisterAction(func(c *framework.Core, msg framework.Message) error {
|
||||
switch m := msg.(type) {
|
||||
case process.ActionProcessStarted:
|
||||
fmt.Printf("Started: %s (PID %d)\n", m.Command, m.PID)
|
||||
case process.ActionProcessOutput:
|
||||
fmt.Print(m.Line)
|
||||
case process.ActionProcessExited:
|
||||
fmt.Printf("Exit code: %d (%s)\n", m.ExitCode, m.Duration)
|
||||
case process.ActionProcessKilled:
|
||||
fmt.Printf("Killed with %s\n", m.Signal)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
```
|
||||
|
||||
### Global Convenience API
|
||||
|
||||
For applications that only need a single process service, a global singleton
|
||||
is available:
|
||||
|
||||
```go
|
||||
// Initialise once at startup
|
||||
process.Init(coreInstance)
|
||||
|
||||
// Then use package-level functions anywhere
|
||||
proc, _ := process.Start(ctx, "ls", "-la")
|
||||
output, _ := process.Run(ctx, "date")
|
||||
procs := process.List()
|
||||
running := process.Running()
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `*.go` (root) | Core process service, types, actions, runner, daemon, health, PID file, registry |
|
||||
| `exec/` | Lightweight command wrapper with fluent API and structured logging |
|
||||
|
||||
## Module Information
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Module path | `forge.lthn.ai/core/go-process` |
|
||||
| Go version | 1.26.0 |
|
||||
| Licence | EUPL-1.2 |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/go` | Core DI framework (`ServiceRuntime`, `Core.ACTION`, lifecycle interfaces) |
|
||||
| `github.com/stretchr/testify` | Test assertions (test-only) |
|
||||
|
||||
The package has no other runtime dependencies beyond the Go standard library
|
||||
and the Core framework.
|
||||
|
|
|
|||
|
|
@ -1,17 +1,95 @@
|
|||
---
|
||||
title: go-rag
|
||||
description: Retrieval-Augmented Generation with document chunking, embeddings, and vector search
|
||||
description: Retrieval-Augmented Generation library for Go — document chunking, Ollama embeddings, Qdrant vector storage, and formatted context retrieval for LLM prompt injection.
|
||||
---
|
||||
|
||||
# go-rag
|
||||
|
||||
`forge.lthn.ai/core/go-rag`
|
||||
`forge.lthn.ai/core/go-rag` is a Retrieval-Augmented Generation library for Go. It handles the full RAG pipeline: splitting documents into chunks, generating embeddings via Ollama, storing and searching vectors in Qdrant (gRPC), applying keyword boosting, and formatting results for human display or LLM prompt injection.
|
||||
|
||||
Retrieval-Augmented Generation pipeline. Splits documents using three-level Markdown-aware chunking, generates embeddings via Ollama, and stores/searches vectors in Qdrant over gRPC. Supports keyword boosting and multiple result formats (plain text, XML, JSON).
|
||||
The library is built around two core interfaces -- `Embedder` and `VectorStore` -- that decouple business logic from service implementations. You can swap backends, inject mocks for testing, or run the full pipeline against live services with the same API.
|
||||
|
||||
## Key Types
|
||||
**Module**: `forge.lthn.ai/core/go-rag`
|
||||
**Go version**: 1.26
|
||||
**Licence**: EUPL-1.2
|
||||
|
||||
- `ChunkConfig` — configuration for document splitting (sizes, overlap, strategy)
|
||||
- `Chunk` — individual text chunk with metadata and position information
|
||||
- `IngestConfig` — configuration for the document ingestion pipeline
|
||||
- `IngestStats` — statistics from a completed ingestion run (chunks created, embeddings generated)
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-rag"
|
||||
|
||||
// Ingest a directory of Markdown files into a Qdrant collection
|
||||
err := rag.IngestDirectory(ctx, "/path/to/docs", "my-collection", false)
|
||||
|
||||
// Query for relevant context (XML format, suitable for LLM prompt injection)
|
||||
context, err := rag.QueryDocsContext(ctx, "how does rate limiting work?", "my-collection", 5)
|
||||
|
||||
// For long-lived processes, construct clients once and use the *With variants
|
||||
qdrantClient, _ := rag.NewQdrantClient(rag.DefaultQdrantConfig())
|
||||
ollamaClient, _ := rag.NewOllamaClient(rag.DefaultOllamaConfig())
|
||||
|
||||
results, err := rag.QueryWith(ctx, qdrantClient, ollamaClient, "question", "collection", 5)
|
||||
```
|
||||
|
||||
The convenience wrappers (`IngestDirectory`, `QueryDocs`, etc.) create new connections on each call, which is fine for CLI usage. For server processes or loops, use the `*With` variants with pre-constructed clients to avoid per-call connection overhead.
|
||||
|
||||
## Package Layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `embedder.go` | `Embedder` interface -- `Embed`, `EmbedBatch`, `EmbedDimension` |
|
||||
| `vectorstore.go` | `VectorStore` interface -- collection management, upsert, search |
|
||||
| `chunk.go` | Markdown chunking with three-level splitting (sections, paragraphs, sentences) and configurable overlap |
|
||||
| `ollama.go` | `OllamaClient` -- implements `Embedder` via the Ollama HTTP API |
|
||||
| `qdrant.go` | `QdrantClient` -- implements `VectorStore` via the Qdrant gRPC API |
|
||||
| `ingest.go` | Ingestion pipeline -- walk directory, chunk files, embed, batch upsert |
|
||||
| `query.go` | Query pipeline -- embed query, vector search, threshold filter, format results |
|
||||
| `keyword.go` | Keyword boosting post-filter for re-ranking search results |
|
||||
| `collections.go` | Package-level collection management helpers |
|
||||
| `helpers.go` | Convenience wrappers -- `*With` variants and default-client functions |
|
||||
| `cmd/rag/` | CLI subcommands (`ingest`, `query`, `collections`) for the `core` binary |
|
||||
|
||||
## CLI Commands
|
||||
|
||||
The package provides CLI subcommands mounted under `core ai rag`:
|
||||
|
||||
```bash
|
||||
# Ingest a directory of Markdown files
|
||||
core ai rag ingest /path/to/docs --collection my-docs --recreate
|
||||
|
||||
# Query the vector database
|
||||
core ai rag query "how does the module system work?" --top 10 --format context
|
||||
|
||||
# List and manage collections
|
||||
core ai rag collections --stats
|
||||
core ai rag collections --delete old-collection
|
||||
```
|
||||
|
||||
All commands accept `--qdrant-host`, `--qdrant-port`, `--ollama-host`, `--ollama-port`, and `--model` flags, with defaults overridable via environment variables (`QDRANT_HOST`, `QDRANT_PORT`, `OLLAMA_HOST`, `OLLAMA_PORT`, `EMBEDDING_MODEL`).
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Role |
|
||||
|------------|------|
|
||||
| `forge.lthn.ai/core/go-log` | Structured error wrapping (`log.E`) |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalised CLI strings |
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (cobra-based commands) |
|
||||
| `github.com/ollama/ollama` | Ollama HTTP client for embedding generation |
|
||||
| `github.com/qdrant/go-client` | Qdrant gRPC client for vector storage and search |
|
||||
| `github.com/stretchr/testify` | Test assertions (test-only) |
|
||||
|
||||
Transitive dependencies include `google.golang.org/grpc`, `google.golang.org/protobuf`, and `github.com/google/uuid`.
|
||||
|
||||
## Service Defaults
|
||||
|
||||
| Service | Host | Port | Protocol |
|
||||
|---------|------|------|----------|
|
||||
| Qdrant | localhost | 6334 | gRPC |
|
||||
| Ollama | localhost | 11434 | HTTP |
|
||||
|
||||
The default embedding model is `nomic-embed-text` (768 dimensions). Other supported models include `mxbai-embed-large` (1024 dimensions) and `all-minilm` (384 dimensions).
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- interfaces, chunking strategy, ingestion pipeline, query pipeline, keyword boosting, performance characteristics
|
||||
- [Development](development.md) -- prerequisites, build commands, test patterns, coding standards, contribution guidelines
|
||||
|
|
|
|||
|
|
@ -1,17 +1,117 @@
|
|||
---
|
||||
title: go-ratelimit
|
||||
description: Provider-agnostic sliding window rate limiter for LLM API calls
|
||||
description: Provider-agnostic sliding window rate limiter for LLM API calls, with YAML and SQLite persistence backends.
|
||||
---
|
||||
|
||||
# go-ratelimit
|
||||
|
||||
`forge.lthn.ai/core/go-ratelimit`
|
||||
**Module**: `forge.lthn.ai/core/go-ratelimit`
|
||||
**Licence**: EUPL-1.2
|
||||
**Go version**: 1.26+
|
||||
|
||||
Provider-agnostic rate limiter for LLM API calls using sliding window counters. Enforces RPM (requests per minute), TPM (tokens per minute), and RPD (requests per day) quotas on a per-model basis. Ships with default profiles for Gemini, OpenAI, Anthropic, and local inference. Supports YAML or SQLite persistence for quota state.
|
||||
go-ratelimit enforces requests-per-minute (RPM), tokens-per-minute (TPM), and
|
||||
requests-per-day (RPD) quotas on a per-model basis using an in-memory sliding
|
||||
window. It ships with default quota profiles for Gemini, OpenAI, Anthropic, and
|
||||
a local inference provider. State persists across process restarts via YAML
|
||||
(single-process) or SQLite with WAL mode (multi-process). A YAML-to-SQLite
|
||||
migration helper is included.
|
||||
|
||||
## Key Types
|
||||
## Quick Start
|
||||
|
||||
- `ModelQuota` — per-model rate limits (RPM, TPM, RPD)
|
||||
- `ProviderProfile` — named provider with its model quotas
|
||||
- `Config` — rate limiter configuration (providers, persistence backend)
|
||||
- `UsageStats` — current usage counters and remaining quota
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-ratelimit"
|
||||
|
||||
// Create a limiter with Gemini defaults (YAML backend).
|
||||
rl, err := ratelimit.New()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Check capacity before sending.
|
||||
if rl.CanSend("gemini-2.0-flash", 1500) {
|
||||
// Make the API call...
|
||||
rl.RecordUsage("gemini-2.0-flash", 1000, 500) // promptTokens, outputTokens
|
||||
}
|
||||
|
||||
// Persist state to disk for recovery across restarts.
|
||||
if err := rl.Persist(); err != nil {
|
||||
log.Printf("persist failed: %v", err)
|
||||
}
|
||||
```
|
||||
|
||||
### Multi-provider configuration
|
||||
|
||||
```go
|
||||
rl, err := ratelimit.NewWithConfig(ratelimit.Config{
|
||||
Providers: []ratelimit.Provider{
|
||||
ratelimit.ProviderGemini,
|
||||
ratelimit.ProviderAnthropic,
|
||||
},
|
||||
Quotas: map[string]ratelimit.ModelQuota{
|
||||
// Override a specific model's limits.
|
||||
"gemini-3-pro-preview": {MaxRPM: 50, MaxTPM: 500000, MaxRPD: 200},
|
||||
// Add a custom model not in any profile.
|
||||
"llama-3.3-70b": {MaxRPM: 5, MaxTPM: 50000, MaxRPD: 0},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### SQLite backend (multi-process safe)
|
||||
|
||||
```go
|
||||
rl, err := ratelimit.NewWithSQLite("~/.core/ratelimits.db")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer rl.Close()
|
||||
|
||||
// Load persisted state.
|
||||
if err := rl.Load(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Use exactly as the YAML backend -- CanSend, RecordUsage, Persist, etc.
|
||||
```
|
||||
|
||||
### Blocking until capacity is available
|
||||
|
||||
```go
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
if err := rl.WaitForCapacity(ctx, "claude-opus-4", 2000); err != nil {
|
||||
log.Printf("timed out waiting for capacity: %v", err)
|
||||
return
|
||||
}
|
||||
// Capacity is available; proceed with the API call.
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
The module is a single package with no sub-packages.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `ratelimit.go` | Core types (`RateLimiter`, `ModelQuota`, `Config`, `Provider`), sliding window logic, provider profiles, YAML persistence, `CountTokens` helper |
|
||||
| `sqlite.go` | SQLite persistence backend (`sqliteStore`), schema creation, load/save operations |
|
||||
| `ratelimit_test.go` | Tests for core logic, provider profiles, concurrency, and benchmarks |
|
||||
| `sqlite_test.go` | Tests for SQLite backend, migration, and error recovery |
|
||||
| `error_test.go` | Tests for SQLite and YAML error paths |
|
||||
| `iter_test.go` | Tests for `Models()` and `Iter()` iterators, plus `CountTokens` edge cases |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Purpose | Category |
|
||||
|------------|---------|----------|
|
||||
| `gopkg.in/yaml.v3` | YAML serialisation for the legacy persistence backend | Direct |
|
||||
| `modernc.org/sqlite` | Pure Go SQLite driver (no CGO required) | Direct |
|
||||
| `github.com/stretchr/testify` | Test assertions (`assert`, `require`) | Test only |
|
||||
|
||||
All indirect dependencies are pulled in by `modernc.org/sqlite`. No C toolchain
|
||||
or system SQLite library is needed.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- sliding window algorithm, provider quotas, YAML and SQLite backends, concurrency model
|
||||
- [Development](development.md) -- build commands, test patterns, coding standards, commit conventions
|
||||
- [History](history.md) -- completed phases with commit hashes, known limitations
|
||||
|
|
|
|||
|
|
@ -1,41 +1,150 @@
|
|||
---
|
||||
title: go-scm
|
||||
description: SCM integration, AgentCI automation, and data collection for the Lethean ecosystem.
|
||||
---
|
||||
|
||||
# go-scm
|
||||
|
||||
SCM integration, AgentCI dispatch automation, and data collection.
|
||||
`go-scm` provides source control management integration for the Lethean ecosystem. It wraps the Forgejo and Gitea APIs behind ergonomic Go clients, runs an automated PR pipeline for AI agent workflows, collects data from external sources, and manages multi-repo workspaces via a declarative registry.
|
||||
|
||||
**Module**: `forge.lthn.ai/core/go-scm`
|
||||
|
||||
Provides a Forgejo API client and a Gitea client for the public mirror, multi-repo git operations with parallel status checks, the Clotho Protocol orchestrator for dual-run agent verification, a PR automation pipeline (poll -> dispatch -> journal) driven by epic issue task lists, and pluggable data collectors for BitcoinTalk, GitHub, market data, and research papers.
|
||||
**Module path:** `forge.lthn.ai/core/go-scm`
|
||||
**Go version:** 1.26
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
import (
|
||||
"forge.lthn.ai/core/go-scm/forge"
|
||||
"forge.lthn.ai/core/go-scm/git"
|
||||
"forge.lthn.ai/core/go-scm/jobrunner"
|
||||
)
|
||||
### Forgejo API Client
|
||||
|
||||
// Forgejo client
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-scm/forge"
|
||||
|
||||
// Create a client from config file / env / flags
|
||||
client, err := forge.NewFromConfig("", "")
|
||||
|
||||
// Multi-repo status
|
||||
statuses := git.Status(ctx, git.StatusOptions{Paths: repoPaths})
|
||||
// List open issues
|
||||
issues, err := client.ListIssues("core", "go-scm", forge.ListIssuesOpts{
|
||||
State: "open",
|
||||
})
|
||||
|
||||
// AgentCI dispatch loop
|
||||
// List repos in an organisation (paginated iterator)
|
||||
for repo, err := range client.ListOrgReposIter("core") {
|
||||
fmt.Println(repo.Name)
|
||||
}
|
||||
```
|
||||
|
||||
### Multi-Repo Git Status
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-scm/git"
|
||||
|
||||
statuses := git.Status(ctx, git.StatusOptions{
|
||||
Paths: []string{"/home/dev/core/go-scm", "/home/dev/core/go-ai"},
|
||||
Names: map[string]string{"/home/dev/core/go-scm": "go-scm"},
|
||||
})
|
||||
|
||||
for _, s := range statuses {
|
||||
if s.IsDirty() {
|
||||
fmt.Printf("%s: %d modified, %d untracked\n", s.Name, s.Modified, s.Untracked)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### AgentCI Poll-Dispatch Loop
|
||||
|
||||
```go
|
||||
import (
|
||||
"forge.lthn.ai/core/go-scm/jobrunner"
|
||||
"forge.lthn.ai/core/go-scm/jobrunner/forgejo"
|
||||
"forge.lthn.ai/core/go-scm/jobrunner/handlers"
|
||||
)
|
||||
|
||||
source := forgejo.New(forgejo.Config{Repos: []string{"core/go-scm"}}, forgeClient)
|
||||
poller := jobrunner.NewPoller(jobrunner.PollerConfig{
|
||||
Sources: []jobrunner.JobSource{forgejoSrc},
|
||||
Handlers: []jobrunner.JobHandler{dispatch, tickParent},
|
||||
Sources: []jobrunner.JobSource{source},
|
||||
Handlers: []jobrunner.JobHandler{
|
||||
handlers.NewDispatchHandler(forgeClient, forgeURL, token, spinner),
|
||||
handlers.NewTickParentHandler(forgeClient),
|
||||
handlers.NewEnableAutoMergeHandler(forgeClient),
|
||||
},
|
||||
PollInterval: 60 * time.Second,
|
||||
})
|
||||
poller.Run(ctx)
|
||||
```
|
||||
|
||||
## Components
|
||||
### Data Collection
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| `forge` | Forgejo API client |
|
||||
| `git` | Multi-repo operations, parallel status |
|
||||
| `jobrunner` | AgentCI dispatch pipeline |
|
||||
| `clotho` | Dual-run verification protocol |
|
||||
| `collectors` | BitcoinTalk, GitHub, market, research scrapers |
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-scm/collect"
|
||||
|
||||
cfg := collect.NewConfig("/tmp/collected")
|
||||
excavator := &collect.Excavator{
|
||||
Collectors: []collect.Collector{
|
||||
&collect.GitHubCollector{Org: "lethean-io"},
|
||||
&collect.MarketCollector{CoinID: "lethean", Historical: true},
|
||||
&collect.PapersCollector{Source: "all", Query: "cryptography VPN"},
|
||||
},
|
||||
Resume: true,
|
||||
}
|
||||
result, err := excavator.Run(ctx, cfg)
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Package | Import Path | Description |
|
||||
|---------|-------------|-------------|
|
||||
| `forge` | `go-scm/forge` | Forgejo API client -- repos, issues, PRs, labels, webhooks, organisations, PR metadata |
|
||||
| `gitea` | `go-scm/gitea` | Gitea API client -- repos, issues, PRs, mirroring, PR metadata |
|
||||
| `git` | `go-scm/git` | Multi-repo git operations -- parallel status checks, push, pull; Core DI service |
|
||||
| `jobrunner` | `go-scm/jobrunner` | AgentCI pipeline engine -- signal types, poller loop, JSONL audit journal |
|
||||
| `jobrunner/forgejo` | `go-scm/jobrunner/forgejo` | Forgejo job source -- polls epic issues for unchecked children, builds signals |
|
||||
| `jobrunner/handlers` | `go-scm/jobrunner/handlers` | Pipeline handlers -- dispatch, completion, auto-merge, publish-draft, dismiss-reviews, fix-command, tick-parent |
|
||||
| `agentci` | `go-scm/agentci` | Clotho Protocol orchestrator -- agent config, SSH security helpers, dual-run verification |
|
||||
| `collect` | `go-scm/collect` | Data collection framework -- collector interface, rate limiting, state tracking, event dispatch |
|
||||
| `manifest` | `go-scm/manifest` | Application manifest -- YAML parsing, ed25519 signing and verification |
|
||||
| `marketplace` | `go-scm/marketplace` | Module marketplace -- catalogue index, search, git-based installer with signature verification |
|
||||
| `plugin` | `go-scm/plugin` | CLI plugin system -- plugin interface, JSON registry, loader, GitHub-based installer |
|
||||
| `repos` | `go-scm/repos` | Workspace management -- `repos.yaml` registry, topological sorting, work config, git state, KB config |
|
||||
| `cmd/forge` | `go-scm/cmd/forge` | CLI commands for the `core forge` subcommand |
|
||||
| `cmd/gitea` | `go-scm/cmd/gitea` | CLI commands for the `core gitea` subcommand |
|
||||
| `cmd/collect` | `go-scm/cmd/collect` | CLI commands for data collection |
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Direct
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `codeberg.org/mvdkleijn/forgejo-sdk/forgejo/v2` | Forgejo API SDK |
|
||||
| `code.gitea.io/sdk/gitea` | Gitea API SDK |
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (Cobra, TUI) |
|
||||
| `forge.lthn.ai/core/go-config` | Layered config (`~/.core/config.yaml`) |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction (Medium, Sandbox, Store) |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logging and contextual error helper |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalisation |
|
||||
| `github.com/stretchr/testify` | Test assertions |
|
||||
| `golang.org/x/net` | HTML parsing for collectors |
|
||||
| `gopkg.in/yaml.v3` | YAML parsing for manifests and registries |
|
||||
|
||||
### Indirect
|
||||
|
||||
The module transitively pulls in `forge.lthn.ai/core/go` (DI framework) via `go-config`, plus `spf13/viper`, `spf13/cobra`, Charmbracelet TUI libraries, and Go standard library extensions.
|
||||
|
||||
## Configuration
|
||||
|
||||
Authentication for both Forgejo and Gitea is resolved through a three-tier priority chain:
|
||||
|
||||
1. **Config file** -- `~/.core/config.yaml` keys `forge.url`, `forge.token` (or `gitea.*`)
|
||||
2. **Environment variables** -- `FORGE_URL`, `FORGE_TOKEN` (or `GITEA_URL`, `GITEA_TOKEN`)
|
||||
3. **CLI flags** -- `--url`, `--token` (highest priority)
|
||||
|
||||
Set credentials once:
|
||||
|
||||
```bash
|
||||
core forge config --url https://forge.lthn.ai --token <your-token>
|
||||
core gitea config --url https://gitea.snider.dev --token <your-token>
|
||||
```
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- internal design, key types, data flow
|
||||
- [Development Guide](development.md) -- building, testing, contributing
|
||||
|
|
|
|||
|
|
@ -1,16 +1,99 @@
|
|||
---
|
||||
title: go-session
|
||||
description: Claude Code JSONL transcript parser, analytics engine, and HTML timeline renderer
|
||||
description: Claude Code JSONL transcript parser, analytics engine, and HTML timeline renderer for Go.
|
||||
---
|
||||
|
||||
# go-session
|
||||
|
||||
`forge.lthn.ai/core/go-session`
|
||||
`go-session` parses Claude Code JSONL session transcripts into structured event arrays, computes per-tool analytics, renders self-contained HTML timelines with client-side search, and generates VHS tape scripts for MP4 video output. It has no external runtime dependencies -- stdlib only.
|
||||
|
||||
Parser and analytics engine for Claude Code JSONL session transcripts. Extracts per-tool usage statistics, generates self-contained HTML timeline visualisations, and produces VHS tape scripts for terminal recording. Useful for understanding agent behaviour and tool usage patterns across sessions.
|
||||
**Module path:** `forge.lthn.ai/core/go-session`
|
||||
**Go version:** 1.26
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
## Key Types
|
||||
## Quick Start
|
||||
|
||||
- `SessionAnalytics` — aggregated statistics from one or more sessions (tool counts, durations, token usage)
|
||||
- `Event` — single parsed event from a JSONL transcript
|
||||
- `Session` — complete parsed session with ordered events and metadata
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-session"
|
||||
|
||||
// Parse a single session file
|
||||
sess, stats, err := session.ParseTranscript("/path/to/session.jsonl")
|
||||
|
||||
// Or parse from any io.Reader (streaming, in-memory, HTTP body, etc.)
|
||||
sess, stats, err := session.ParseTranscriptReader(reader, "my-session-id")
|
||||
|
||||
// Compute analytics
|
||||
analytics := session.Analyse(sess)
|
||||
fmt.Println(session.FormatAnalytics(analytics))
|
||||
|
||||
// Render an interactive HTML timeline
|
||||
err = session.RenderHTML(sess, "timeline.html")
|
||||
|
||||
// Search across all sessions in a directory
|
||||
results, err := session.Search("~/.claude/projects/my-project", "git commit")
|
||||
|
||||
// List sessions (newest first)
|
||||
sessions, err := session.ListSessions("~/.claude/projects/my-project")
|
||||
|
||||
// Prune old sessions
|
||||
deleted, err := session.PruneSessions("~/.claude/projects/my-project", 30*24*time.Hour)
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
The entire package lives in a single Go package (`session`) with five source files:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `parser.go` | Core types (`Event`, `Session`, `ParseStats`), JSONL parsing (`ParseTranscript`, `ParseTranscriptReader`), session listing (`ListSessions`, `ListSessionsSeq`), pruning (`PruneSessions`), fetching (`FetchSession`), tool input extraction |
|
||||
| `analytics.go` | `SessionAnalytics` type, `Analyse` (pure computation), `FormatAnalytics` (CLI-friendly text output) |
|
||||
| `html.go` | `RenderHTML` -- self-contained dark-theme HTML timeline with collapsible panels, search, and XSS protection |
|
||||
| `video.go` | `RenderMP4` -- VHS tape script generation and invocation for MP4 video output |
|
||||
| `search.go` | `Search` and `SearchSeq` -- case-insensitive cross-session search over tool call inputs and outputs |
|
||||
|
||||
Test files mirror the source files (`parser_test.go`, `analytics_test.go`, `html_test.go`, `video_test.go`, `search_test.go`) plus `bench_test.go` for benchmarks.
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Scope | Purpose |
|
||||
|------------|-------|---------|
|
||||
| Go standard library | Runtime | All parsing, HTML rendering, file I/O, JSON decoding |
|
||||
| `github.com/stretchr/testify` | Test only | Assertions and requirements in test files |
|
||||
| `vhs` (charmbracelet) | Optional external binary | Required only by `RenderMP4` for MP4 video generation |
|
||||
|
||||
The package has **zero runtime dependencies** beyond the Go standard library. `testify` is fetched automatically by `go test` and is never imported outside test files.
|
||||
|
||||
## Supported Tool Types
|
||||
|
||||
The parser recognises the following Claude Code tool types and formats their input for human readability:
|
||||
|
||||
| Tool | Input format | Example |
|
||||
|------|-------------|---------|
|
||||
| Bash | `command # description` | `ls -la # list files` |
|
||||
| Read | `file_path` | `/tmp/main.go` |
|
||||
| Edit | `file_path (edit)` | `/tmp/main.go (edit)` |
|
||||
| Write | `file_path (N bytes)` | `/tmp/out.txt (42 bytes)` |
|
||||
| Grep | `/pattern/ in path` | `/TODO/ in /src` |
|
||||
| Glob | `pattern` | `**/*.go` |
|
||||
| Task | `[subagent_type] description` | `[research] Code review` |
|
||||
| Any other (MCP tools, etc.) | Sorted top-level JSON keys | `body, repo, title` |
|
||||
|
||||
Unknown tools (including MCP tools like `mcp__forge__create_issue`) are handled gracefully by extracting and sorting the JSON field names from the raw input.
|
||||
|
||||
## Iterator Support
|
||||
|
||||
Several functions offer both slice-returning and iterator-based variants, using Go's `iter.Seq` type:
|
||||
|
||||
| Slice variant | Iterator variant | Description |
|
||||
|---------------|-----------------|-------------|
|
||||
| `ListSessions()` | `ListSessionsSeq()` | Enumerate sessions in a directory |
|
||||
| `Search()` | `SearchSeq()` | Search across sessions |
|
||||
| -- | `Session.EventsSeq()` | Iterate over events in a session |
|
||||
|
||||
The iterator variants avoid allocating the full result slice upfront and support early termination via `break` or `return` in `range` loops.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- JSONL format, parsing pipeline, event model, analytics, HTML rendering, XSS protection, data flow
|
||||
- [Development Guide](development.md) -- Prerequisites, build commands, test patterns, coding standards, how to add new tool types
|
||||
- [Project History](history.md) -- Completed phases, known limitations, future considerations
|
||||
|
|
|
|||
|
|
@ -1,17 +1,134 @@
|
|||
---
|
||||
title: go-store
|
||||
description: Group-namespaced SQLite key-value store with TTL, quotas, and reactive events
|
||||
description: Group-namespaced SQLite key-value store with TTL expiry, namespace isolation, quota enforcement, and reactive event hooks.
|
||||
---
|
||||
|
||||
# go-store
|
||||
|
||||
`forge.lthn.ai/core/go-store`
|
||||
`go-store` is a group-namespaced key-value store backed by SQLite. It provides persistent or in-memory storage with optional TTL expiry, namespace isolation for multi-tenant use, quota enforcement, and a reactive event system for observing mutations.
|
||||
|
||||
SQLite-backed key-value store with group namespacing, TTL expiry, and quota enforcement. Operates in WAL mode for concurrent read performance. Supports scoped stores for multi-tenant isolation, reactive Watch/Unwatch subscriptions, and OnChange callbacks. Designed as an integration point for go-ws real-time streaming.
|
||||
The package has a single runtime dependency -- a pure-Go SQLite driver (`modernc.org/sqlite`). No CGO is required. It compiles and runs on all platforms that Go supports.
|
||||
|
||||
**Module path:** `forge.lthn.ai/core/go-store`
|
||||
**Go version:** 1.26+
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"forge.lthn.ai/core/go-store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Open a store. Use ":memory:" for ephemeral data or a file path for persistence.
|
||||
st, err := store.New("/tmp/app.db")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
// Basic CRUD
|
||||
st.Set("config", "theme", "dark")
|
||||
val, _ := st.Get("config", "theme")
|
||||
fmt.Println(val) // "dark"
|
||||
|
||||
// TTL expiry -- key disappears after the duration elapses
|
||||
st.SetWithTTL("session", "token", "abc123", 24*time.Hour)
|
||||
|
||||
// Fetch all keys in a group
|
||||
all, _ := st.GetAll("config")
|
||||
fmt.Println(all) // map[theme:dark]
|
||||
|
||||
// Template rendering from stored values
|
||||
st.Set("mail", "host", "smtp.example.com")
|
||||
st.Set("mail", "port", "587")
|
||||
out, _ := st.Render(`{{ .host }}:{{ .port }}`, "mail")
|
||||
fmt.Println(out) // "smtp.example.com:587"
|
||||
|
||||
// Namespace isolation for multi-tenant use
|
||||
sc, _ := store.NewScoped(st, "tenant-42")
|
||||
sc.Set("prefs", "locale", "en-GB")
|
||||
// Stored internally as group "tenant-42:prefs", key "locale"
|
||||
|
||||
// Quota enforcement
|
||||
quota := store.QuotaConfig{MaxKeys: 100, MaxGroups: 5}
|
||||
sq, _ := store.NewScopedWithQuota(st, "tenant-99", quota)
|
||||
err = sq.Set("g", "k", "v") // returns store.ErrQuotaExceeded if limits are hit
|
||||
|
||||
// Watch for mutations via a buffered channel
|
||||
w := st.Watch("config", "*")
|
||||
defer st.Unwatch(w)
|
||||
go func() {
|
||||
for e := range w.Ch {
|
||||
fmt.Printf("event: %s %s/%s\n", e.Type, e.Group, e.Key)
|
||||
}
|
||||
}()
|
||||
|
||||
// Or register a synchronous callback
|
||||
unreg := st.OnChange(func(e store.Event) {
|
||||
fmt.Printf("changed: %s\n", e.Key)
|
||||
})
|
||||
defer unreg()
|
||||
}
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
The entire package lives in a single Go package (`package store`) with three source files:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `store.go` | Core `Store` type, CRUD operations (`Get`, `Set`, `SetWithTTL`, `Delete`, `DeleteGroup`), bulk queries (`GetAll`, `All`, `Count`, `CountAll`, `Groups`, `GroupsSeq`), string splitting helpers (`GetSplit`, `GetFields`), template rendering (`Render`), TTL expiry, background purge goroutine |
|
||||
| `events.go` | `EventType` constants, `Event` struct, `Watcher` type, `Watch`/`Unwatch` subscription management, `OnChange` callback registration, internal `notify` dispatch |
|
||||
| `scope.go` | `ScopedStore` wrapper for namespace isolation, `QuotaConfig` struct, `NewScoped`/`NewScopedWithQuota` constructors, quota enforcement logic |
|
||||
|
||||
Tests are organised in corresponding files:
|
||||
|
||||
| File | Covers |
|
||||
|------|--------|
|
||||
| `store_test.go` | CRUD, TTL, concurrency, edge cases, persistence, WAL verification |
|
||||
| `events_test.go` | Watch/Unwatch, OnChange, event dispatch, wildcard matching, buffer overflow |
|
||||
| `scope_test.go` | Namespace isolation, quota enforcement, cross-namespace behaviour |
|
||||
| `coverage_test.go` | Defensive error paths (scan errors, schema conflicts, database corruption) |
|
||||
| `bench_test.go` | Performance benchmarks for all major operations |
|
||||
|
||||
## Dependencies
|
||||
|
||||
**Runtime:**
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `modernc.org/sqlite` | Pure-Go SQLite driver (no CGO). Registered as a `database/sql` driver. |
|
||||
|
||||
**Test only:**
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `github.com/stretchr/testify` | Assertion helpers (`assert`, `require`) for tests. |
|
||||
|
||||
There are no other direct dependencies. The package uses only the Go standard library (`database/sql`, `context`, `sync`, `time`, `text/template`, `iter`, `errors`, `fmt`, `strings`, `regexp`, `slices`, `sync/atomic`) beyond the SQLite driver.
|
||||
|
||||
## Key Types
|
||||
|
||||
- `Event` — change event emitted on key create, update, or delete
|
||||
- `Watcher` — subscription handle for reactive change notifications
|
||||
- `QuotaConfig` — per-namespace limits (max keys, max value size, total size)
|
||||
- `ScopedStore` — namespace-isolated view of the store for multi-tenant use
|
||||
- **`Store`** -- the central type. Holds a `*sql.DB`, manages the background purge goroutine, and maintains the watcher/callback registry.
|
||||
- **`ScopedStore`** -- wraps a `*Store` with an auto-prefixed namespace. Provides the same API surface with group names transparently prefixed.
|
||||
- **`QuotaConfig`** -- configures per-namespace limits on total keys and distinct groups.
|
||||
- **`Event`** -- describes a single store mutation (type, group, key, value, timestamp).
|
||||
- **`Watcher`** -- a channel-based subscription to store events, created by `Watch`.
|
||||
- **`KV`** -- a simple key-value pair struct, used by the `All` iterator.
|
||||
|
||||
## Sentinel Errors
|
||||
|
||||
- **`ErrNotFound`** -- returned by `Get` when the requested key does not exist or has expired.
|
||||
- **`ErrQuotaExceeded`** -- returned by `ScopedStore.Set`/`SetWithTTL` when a namespace quota limit is reached.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- storage layer internals, TTL model, event system, concurrency design
|
||||
- [Development Guide](development.md) -- building, testing, benchmarks, contribution workflow
|
||||
|
|
|
|||
|
|
@ -1,17 +1,141 @@
|
|||
---
|
||||
title: go-webview
|
||||
description: Chrome DevTools Protocol client for browser automation and testing
|
||||
description: Chrome DevTools Protocol client for browser automation, testing, and scraping in Go.
|
||||
---
|
||||
|
||||
# go-webview
|
||||
|
||||
`forge.lthn.ai/core/go-webview`
|
||||
`go-webview` is a Go package that provides browser automation via the Chrome DevTools Protocol (CDP). It connects to an externally managed Chrome or Chromium instance running with `--remote-debugging-port=9222` and exposes a high-level API for navigation, DOM queries, input simulation, screenshot capture, console monitoring, and JavaScript evaluation.
|
||||
|
||||
Chrome DevTools Protocol (CDP) client for browser automation, testing, and scraping. Supports navigation, DOM queries, click/type interactions, console capture, JavaScript evaluation, screenshots, multi-tab management, and viewport emulation. Includes an ActionSequence builder for chaining operations and Angular-specific helpers.
|
||||
The package does not launch Chrome itself. The caller is responsible for starting the browser process before constructing a `Webview`.
|
||||
|
||||
## Key Types
|
||||
**Module path:** `forge.lthn.ai/core/go-webview`
|
||||
**Licence:** EUPL-1.2
|
||||
**Go version:** 1.26+
|
||||
**Dependencies:** `github.com/gorilla/websocket v1.5.3`
|
||||
|
||||
- `ClickAction` — clicks an element by selector
|
||||
- `TypeAction` — types text into an input element
|
||||
- `NavigateAction` — navigates to a URL and waits for load
|
||||
- `WaitAction` — waits for a selector, timeout, or network idle
|
||||
## Quick Start
|
||||
|
||||
Start Chrome with the remote debugging port enabled:
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
|
||||
--remote-debugging-port=9222
|
||||
|
||||
# Linux
|
||||
google-chrome --remote-debugging-port=9222
|
||||
|
||||
# Headless (suitable for CI)
|
||||
google-chrome --headless=new --remote-debugging-port=9222 --no-sandbox --disable-gpu
|
||||
```
|
||||
|
||||
Then use the package in Go:
|
||||
|
||||
```go
|
||||
import "forge.lthn.ai/core/go-webview"
|
||||
|
||||
// Connect to Chrome
|
||||
wv, err := webview.New(webview.WithDebugURL("http://localhost:9222"))
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer wv.Close()
|
||||
|
||||
// Navigate and interact
|
||||
if err := wv.Navigate("https://example.com"); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if err := wv.Click("#submit-button"); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
### Fluent Action Sequences
|
||||
|
||||
Chain multiple browser actions together with `ActionSequence`:
|
||||
|
||||
```go
|
||||
err := webview.NewActionSequence().
|
||||
Navigate("https://example.com").
|
||||
WaitForSelector("#login-form").
|
||||
Type("#email", "user@example.com").
|
||||
Type("#password", "secret").
|
||||
Click("#submit").
|
||||
Execute(ctx, wv)
|
||||
```
|
||||
|
||||
### Console Monitoring
|
||||
|
||||
Capture and filter browser console output:
|
||||
|
||||
```go
|
||||
cw := webview.NewConsoleWatcher(wv)
|
||||
cw.AddFilter(webview.ConsoleFilter{Type: "error"})
|
||||
|
||||
// ... perform browser actions ...
|
||||
|
||||
if cw.HasErrors() {
|
||||
for _, msg := range cw.Errors() {
|
||||
log.Printf("JS error: %s at %s:%d", msg.Text, msg.URL, msg.Line)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Screenshots
|
||||
|
||||
Capture the current page as PNG:
|
||||
|
||||
```go
|
||||
png, err := wv.Screenshot()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
os.WriteFile("screenshot.png", png, 0644)
|
||||
```
|
||||
|
||||
### Angular Applications
|
||||
|
||||
First-class support for Angular single-page applications:
|
||||
|
||||
```go
|
||||
ah := webview.NewAngularHelper(wv)
|
||||
|
||||
// Wait for Angular to stabilise
|
||||
if err := ah.WaitForAngular(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Navigate using Angular Router
|
||||
if err := ah.NavigateByRouter("/dashboard"); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Inspect component state (debug mode only)
|
||||
value, err := ah.GetComponentProperty("app-widget", "title")
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| File | Responsibility |
|
||||
|------|----------------|
|
||||
| `webview.go` | `Webview` struct, public API (navigate, click, type, screenshot, JS evaluation, DOM queries) |
|
||||
| `cdp.go` | `CDPClient` -- WebSocket transport, CDP message framing, event dispatch, tab management |
|
||||
| `actions.go` | `Action` interface, 19 concrete action types, `ActionSequence` fluent builder |
|
||||
| `console.go` | `ConsoleWatcher`, `ExceptionWatcher`, console log formatting |
|
||||
| `angular.go` | `AngularHelper` -- Zone.js stability, router navigation, component introspection, ngModel |
|
||||
| `webview_test.go` | Unit tests for structs, options, and action building |
|
||||
|
||||
## Configuration Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `WithDebugURL(url)` | *(required)* | Chrome DevTools HTTP debug endpoint, e.g. `http://localhost:9222` |
|
||||
| `WithTimeout(d)` | 30 seconds | Default timeout for all browser operations |
|
||||
| `WithConsoleLimit(n)` | 1000 | Maximum number of console messages retained in memory |
|
||||
|
||||
## Further Documentation
|
||||
|
||||
- [Architecture](architecture.md) -- internals, data flow, CDP protocol, type reference
|
||||
- [Development Guide](development.md) -- build, test, contribute, coding standards
|
||||
- [Project History](history.md) -- extraction origin, completed phases, known limitations
|
||||
|
|
|
|||
|
|
@ -1,17 +1,126 @@
|
|||
---
|
||||
title: go-ws
|
||||
description: WebSocket hub for real-time streaming with channel pub/sub and Redis bridge
|
||||
description: WebSocket hub for real-time streaming in Go, with channel pub/sub, token authentication, reconnecting clients, and a Redis bridge for multi-instance coordination.
|
||||
---
|
||||
|
||||
# go-ws
|
||||
|
||||
`forge.lthn.ai/core/go-ws`
|
||||
`go-ws` is a WebSocket hub library for Go. It implements centralised connection management using the hub pattern, named channel pub/sub, optional token-based authentication at upgrade time, a client-side reconnecting wrapper with exponential backoff, and a Redis pub/sub bridge that coordinates broadcasts across multiple hub instances.
|
||||
|
||||
WebSocket server implementing the hub pattern for real-time streaming. Provides named channel pub/sub, token-based authentication on the upgrade handshake, automatic reconnection with exponential backoff on the client side, and a Redis pub/sub bridge for multi-instance coordination.
|
||||
| | |
|
||||
|---|---|
|
||||
| **Module** | `forge.lthn.ai/core/go-ws` |
|
||||
| **Go version** | 1.26+ |
|
||||
| **Licence** | EUPL-1.2 |
|
||||
| **Repository** | `ssh://git@forge.lthn.ai:2223/core/go-ws.git` |
|
||||
|
||||
## Key Types
|
||||
## Quick Start
|
||||
|
||||
- `AuthResult` — outcome of a WebSocket authentication attempt
|
||||
- `APIKeyAuthenticator` — authenticates connections via API key header or query parameter
|
||||
- `BearerTokenAuth` — authenticates connections via Bearer token
|
||||
- `RedisConfig` — configuration for the Redis pub/sub bridge
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
|
||||
"forge.lthn.ai/core/go-ws"
|
||||
)
|
||||
|
||||
func main() {
|
||||
hub := ws.NewHub()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
go hub.Run(ctx)
|
||||
|
||||
http.HandleFunc("/ws", hub.Handler())
|
||||
http.ListenAndServe(":8080", nil)
|
||||
}
|
||||
```
|
||||
|
||||
Once running, clients connect via WebSocket and can subscribe to named channels. The server pushes messages to all connected clients or to subscribers of a specific channel.
|
||||
|
||||
### Sending Messages
|
||||
|
||||
```go
|
||||
// Broadcast to every connected client.
|
||||
hub.SendEvent("deploy:started", map[string]any{"env": "production"})
|
||||
|
||||
// Send process output to subscribers of "process:build-42".
|
||||
hub.SendProcessOutput("build-42", "Compiling main.go...")
|
||||
|
||||
// Send a process status change.
|
||||
hub.SendProcessStatus("build-42", "exited", 0)
|
||||
```
|
||||
|
||||
### Adding Authentication
|
||||
|
||||
```go
|
||||
auth := ws.NewAPIKeyAuth(map[string]string{
|
||||
"secret-key-1": "user-alice",
|
||||
"secret-key-2": "user-bob",
|
||||
})
|
||||
|
||||
hub := ws.NewHubWithConfig(ws.HubConfig{
|
||||
Authenticator: auth,
|
||||
OnAuthFailure: func(r *http.Request, result ws.AuthResult) {
|
||||
log.Printf("rejected connection from %s: %v", r.RemoteAddr, result.Error)
|
||||
},
|
||||
})
|
||||
go hub.Run(ctx)
|
||||
```
|
||||
|
||||
Clients connect with `Authorization: Bearer <key>`. Without a valid key, the upgrade is rejected with HTTP 401. When no `Authenticator` is set, all connections are accepted.
|
||||
|
||||
### Redis Bridge for Multi-Instance Deployments
|
||||
|
||||
```go
|
||||
bridge, err := ws.NewRedisBridge(hub, ws.RedisConfig{
|
||||
Addr: "localhost:6379",
|
||||
Prefix: "ws",
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if err := bridge.Start(ctx); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer bridge.Stop()
|
||||
|
||||
// Messages published via the bridge reach clients on all instances.
|
||||
bridge.PublishBroadcast(ws.Message{Type: ws.TypeEvent, Data: "hello from instance A"})
|
||||
bridge.PublishToChannel("process:build-42", ws.Message{
|
||||
Type: ws.TypeProcessOutput,
|
||||
Data: "output line",
|
||||
})
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
The entire library lives in a single Go package (`ws`). There are no sub-packages.
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `ws.go` | Hub, Client, Message, ReconnectingClient, connection pumps, channel subscription |
|
||||
| `auth.go` | Authenticator interface, AuthResult, APIKeyAuthenticator, BearerTokenAuth, QueryTokenAuth |
|
||||
| `errors.go` | Sentinel authentication errors |
|
||||
| `redis.go` | RedisBridge, RedisConfig, envelope pattern for loop prevention |
|
||||
| `ws_test.go` | Hub lifecycle, broadcast, channel, subscription, and integration tests |
|
||||
| `auth_test.go` | Authentication unit and integration tests |
|
||||
| `redis_test.go` | Redis bridge integration tests (skipped when Redis is unavailable) |
|
||||
| `ws_bench_test.go` | 9 benchmarks covering broadcast, channel send, subscribe/unsubscribe, fanout, and end-to-end WebSocket round-trips |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Version | Role |
|
||||
|---|---|---|
|
||||
| `github.com/gorilla/websocket` | v1.5.3 | WebSocket server and client implementation |
|
||||
| `github.com/redis/go-redis/v9` | v9.18.0 | Redis pub/sub bridge (runtime opt-in) |
|
||||
| `github.com/stretchr/testify` | v1.11.1 | Test assertions (test-only) |
|
||||
|
||||
The Redis dependency is a compile-time import but a runtime opt-in. Applications that never create a `RedisBridge` incur no Redis connections. There are no CGO requirements; the module builds cleanly on Linux, macOS, and Windows.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Architecture](architecture.md) -- Hub pattern, channels, authentication, Redis bridge, concurrency model
|
||||
- [Development Guide](development.md) -- Building, testing, coding standards, contribution workflow
|
||||
|
|
|
|||
|
|
@ -1,39 +1,186 @@
|
|||
---
|
||||
title: Agent
|
||||
description: Agent orchestration platform — Claude Code plugins, Go agent framework, and Laravel agent package
|
||||
title: Core Agent
|
||||
description: AI agent orchestration, Claude Code plugins, and lifecycle management for the Host UK platform — a polyglot Go + PHP repository.
|
||||
---
|
||||
|
||||
# Agent
|
||||
# Core Agent
|
||||
|
||||
`forge.lthn.ai/core/agent`
|
||||
Core Agent (`forge.lthn.ai/core/agent`) is a polyglot repository containing **Go libraries**, **CLI commands**, **MCP servers**, and a **Laravel PHP package** that together provide AI agent orchestration for the Host UK platform.
|
||||
|
||||
A monorepo containing the agent orchestration platform for Host UK. Combines Go and PHP into a single codebase that powers AI agent tooling across the stack.
|
||||
It answers three questions:
|
||||
|
||||
## What's Inside
|
||||
1. **How do agents get work?** -- The lifecycle package manages tasks, dispatching, and quota enforcement. The PHP side exposes a REST API for plans, sessions, and phases.
|
||||
2. **How do agents run?** -- The dispatch and jobrunner packages poll for work, clone repositories, invoke Claude/Codex/Gemini, and report results back to Forgejo.
|
||||
3. **How do agents collaborate?** -- Sessions, plans, and the OpenBrain vector store enable multi-agent handoff, replay, and persistent memory.
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Go (library / CLI commands)
|
||||
|
||||
The Go module is `forge.lthn.ai/core/agent`. It requires Go 1.26+.
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
core go test
|
||||
|
||||
# Full QA pipeline
|
||||
core go qa
|
||||
```
|
||||
|
||||
Key CLI commands (registered into the `core` binary via `cli.RegisterCommands`):
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `core ai tasks` | List available tasks from the agentic API |
|
||||
| `core ai task [id]` | View or claim a specific task |
|
||||
| `core ai task --auto` | Auto-select the highest-priority pending task |
|
||||
| `core ai agent list` | List configured AgentCI dispatch targets |
|
||||
| `core ai agent add <name> <host>` | Register a new agent machine |
|
||||
| `core ai agent fleet` | Show fleet status from the agent registry |
|
||||
| `core ai dispatch watch` | Poll the PHP API for work and execute phases |
|
||||
| `core ai dispatch run` | Process a single ticket from the local queue |
|
||||
|
||||
### PHP (Laravel package)
|
||||
|
||||
The PHP package is `lthn/agent` (Composer name). It depends on `lthn/php` (the foundation framework).
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
composer test
|
||||
|
||||
# Fix code style
|
||||
composer lint
|
||||
```
|
||||
|
||||
The package auto-registers via Laravel's service provider discovery (`Core\Mod\Agentic\Boot`).
|
||||
|
||||
|
||||
## Package Layout
|
||||
|
||||
### Go Packages
|
||||
|
||||
| Package | Path | Purpose |
|
||||
|---------|------|---------|
|
||||
| `lifecycle` | `pkg/lifecycle/` | Core domain: tasks, agents, dispatcher, allowance quotas, events, API client, brain (OpenBrain), embedded prompts |
|
||||
| `loop` | `pkg/loop/` | Autonomous agent loop: prompt-parse-execute cycle with tool calling against any `inference.TextModel` |
|
||||
| `orchestrator` | `pkg/orchestrator/` | Clotho protocol: dual-run verification, agent configuration, security helpers |
|
||||
| `jobrunner` | `pkg/jobrunner/` | Poll-dispatch engine: `Poller`, `Journal`, Forgejo source, pipeline handlers |
|
||||
| `plugin` | `pkg/plugin/` | Plugin contract tests |
|
||||
| `workspace` | `pkg/workspace/` | Workspace contract tests |
|
||||
|
||||
### Go Commands
|
||||
|
||||
| Directory | Registered As | Purpose |
|
||||
|-----------|---------------|---------|
|
||||
| `cmd/tasks/` | `core ai tasks`, `core ai task` | Task listing, viewing, claiming, updating |
|
||||
| `cmd/agent/` | `core ai agent` | AgentCI machine management (add, list, status, setup, fleet) |
|
||||
| `cmd/dispatch/` | `core ai dispatch` | Work queue processor (runs on agent machines) |
|
||||
| `cmd/workspace/` | `core workspace task`, `core workspace agent` | Isolated git-worktree workspaces for task execution |
|
||||
| `cmd/taskgit/` | *(internal)* | Git operations for task branches |
|
||||
| `cmd/mcp/` | Standalone binary | MCP server (stdio) with marketplace, ethics, and core CLI tools |
|
||||
|
||||
### MCP Servers
|
||||
|
||||
| Directory | Transport | Tools |
|
||||
|-----------|-----------|-------|
|
||||
| `cmd/mcp/` | stdio (mcp-go) | `marketplace_list`, `marketplace_plugin_info`, `core_cli`, `ethics_check` |
|
||||
| `google/mcp/` | HTTP (:8080) | `core_go_test`, `core_dev_health`, `core_dev_commit` |
|
||||
|
||||
### Claude Code Plugins
|
||||
|
||||
A collection of Claude Code plugins for the Host UK federated monorepo:
|
||||
| Plugin | Path | Commands |
|
||||
|--------|------|----------|
|
||||
| **code** | `claude/code/` | `/code:remember`, `/code:yes`, `/code:qa` |
|
||||
| **review** | `claude/review/` | `/review:review`, `/review:security`, `/review:pr` |
|
||||
| **verify** | `claude/verify/` | `/verify:verify`, `/verify:ready`, `/verify:tests` |
|
||||
| **qa** | `claude/qa/` | `/qa:qa`, `/qa:fix` |
|
||||
| **ci** | `claude/ci/` | `/ci:ci`, `/ci:workflow`, `/ci:fix`, `/ci:run`, `/ci:status` |
|
||||
|
||||
| Plugin | Description |
|
||||
|--------|-------------|
|
||||
| **code** | Core development — hooks, scripts, data collection |
|
||||
| **review** | Code review automation |
|
||||
Install all plugins: `claude plugin add host-uk/core-agent`
|
||||
|
||||
### Go Agent Framework
|
||||
### Codex Plugins
|
||||
|
||||
131 Go files providing the agent runtime, session management, and orchestration layer.
|
||||
The `codex/` directory mirrors the Claude plugin structure for OpenAI Codex, plus additional plugins for ethics, guardrails, performance, and issue management.
|
||||
|
||||
### Laravel Agent Package
|
||||
### PHP Package
|
||||
|
||||
231 PHP files providing the Laravel integration — agent sessions, plans, tool handlers, and the agentic portal backend.
|
||||
| Directory | Namespace | Purpose |
|
||||
|-----------|-----------|---------|
|
||||
| `src/php/` | `Core\Mod\Agentic\` | Laravel service provider, models, controllers, services |
|
||||
| `src/php/Actions/` | `...\Actions\` | Single-purpose business logic (Brain, Forge, Phase, Plan, Session, Task) |
|
||||
| `src/php/Controllers/` | `...\Controllers\` | REST API controllers for go-agentic client consumption |
|
||||
| `src/php/Models/` | `...\Models\` | Eloquent models: AgentPlan, AgentPhase, AgentSession, AgentApiKey, BrainMemory, Task, Prompt, WorkspaceState |
|
||||
| `src/php/Services/` | `...\Services\` | AgenticManager (multi-provider), BrainService (Ollama+Qdrant), ForgejoService, Claude/Gemini/OpenAI services |
|
||||
| `src/php/Mcp/` | `...\Mcp\` | MCP tool implementations: Brain, Content, Phase, Plan, Session, State, Task, Template |
|
||||
| `src/php/View/` | `...\View\` | Livewire admin components (Dashboard, Plans, Sessions, ApiKeys, Templates, ToolAnalytics) |
|
||||
| `src/php/Migrations/` | | 10 database migrations |
|
||||
| `src/php/tests/` | | Pest test suite |
|
||||
|
||||
## Repository
|
||||
|
||||
- **Source**: [forge.lthn.ai/core/agent](https://forge.lthn.ai/core/agent)
|
||||
- **Go module**: `forge.lthn.ai/core/agent`
|
||||
- **Composer**: `lthn/agentic`
|
||||
## Dependencies
|
||||
|
||||
## Absorbed Archives
|
||||
### Go
|
||||
|
||||
This repo consolidates code from the now-archived `go-agent`, `go-agentic`, and `php-agentic` packages. The `php-devops` functionality is also planned to be implemented here.
|
||||
| Dependency | Purpose |
|
||||
|------------|---------|
|
||||
| `forge.lthn.ai/core/go` | DI container and service lifecycle |
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (cobra + bubbletea TUI) |
|
||||
| `forge.lthn.ai/core/go-ai` | AI meta-hub (MCP facade) |
|
||||
| `forge.lthn.ai/core/go-config` | Configuration management (viper) |
|
||||
| `forge.lthn.ai/core/go-inference` | TextModel/Backend interfaces |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logging |
|
||||
| `forge.lthn.ai/core/go-ratelimit` | Rate limiting primitives |
|
||||
| `forge.lthn.ai/core/go-scm` | Source control (Forgejo client, repo registry) |
|
||||
| `forge.lthn.ai/core/go-store` | Key-value store abstraction |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalisation |
|
||||
| `github.com/mark3labs/mcp-go` | Model Context Protocol SDK |
|
||||
| `github.com/redis/go-redis/v9` | Redis client (registry + allowance backends) |
|
||||
| `modernc.org/sqlite` | Pure-Go SQLite (registry + allowance backends) |
|
||||
| `codeberg.org/mvdkleijn/forgejo-sdk` | Forgejo API SDK |
|
||||
|
||||
### PHP
|
||||
|
||||
| Dependency | Purpose |
|
||||
|------------|---------|
|
||||
| `lthn/php` | Foundation framework (events, modules, lifecycle) |
|
||||
| `livewire/livewire` | Admin panel reactive components |
|
||||
| `pestphp/pest` | Testing framework |
|
||||
| `orchestra/testbench` | Laravel package testing |
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
### Go Client (`~/.core/agentic.yaml`)
|
||||
|
||||
```yaml
|
||||
base_url: https://api.lthn.sh
|
||||
token: your-api-token
|
||||
default_project: my-project
|
||||
agent_id: cladius
|
||||
```
|
||||
|
||||
Environment variables override the YAML file:
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `AGENTIC_BASE_URL` | API base URL |
|
||||
| `AGENTIC_TOKEN` | Authentication token |
|
||||
| `AGENTIC_PROJECT` | Default project |
|
||||
| `AGENTIC_AGENT_ID` | Agent identifier |
|
||||
|
||||
### PHP (`.env`)
|
||||
|
||||
```env
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
GOOGLE_AI_API_KEY=...
|
||||
OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
The agentic module also reads `BRAIN_DB_*` for the dedicated brain database connection and Ollama/Qdrant URLs from `mcp.brain.*` config keys.
|
||||
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
|
|||
|
|
@ -1,28 +1,97 @@
|
|||
---
|
||||
title: IDE
|
||||
description: Native desktop IDE built with Wails 3 and Angular 20
|
||||
title: Core IDE
|
||||
description: A native desktop development environment and headless CI job runner built with Wails 3 and Angular, providing MCP-driven webview automation and Forgejo-integrated agent dispatch.
|
||||
---
|
||||
|
||||
# IDE
|
||||
# Core IDE
|
||||
|
||||
`forge.lthn.ai/core/ide`
|
||||
Core IDE is a native desktop application that serves two roles:
|
||||
|
||||
Native desktop IDE built with Wails 3 (Go backend) and Angular 20 (frontend). Includes a Claude Code bridge for AI-assisted development and an MCP bridge for tool integration.
|
||||
1. **GUI mode** -- a Wails 3 desktop application with an Angular frontend, system tray panel, and an embedded MCP HTTP server that exposes webview automation tools (DOM inspection, JavaScript execution, screenshots, network monitoring, and more).
|
||||
|
||||
## Stack
|
||||
2. **Headless mode** -- a daemon that polls Forgejo repositories for actionable signals (draft PRs, review state, fix commands) and dispatches work to AI agents via the Clotho spinner, with a minimal MCP HTTP interface for remote control.
|
||||
|
||||
- **Backend**: Go (Wails 3)
|
||||
- **Frontend**: Angular 20, Web Awesome, Font Awesome
|
||||
- **AI**: Claude Code bridge for in-IDE agent interaction
|
||||
- **Tools**: MCP bridge for protocol tool access
|
||||
Both modes listen on `127.0.0.1:9877` and expose `/health`, `/mcp`, `/mcp/tools`, and `/mcp/call` endpoints.
|
||||
|
||||
## Key Types
|
||||
## Quick start
|
||||
|
||||
- `ClaudeBridge` — connects the IDE to Claude Code sessions
|
||||
- `MCPBridge` — exposes MCP tools within the IDE
|
||||
- `GreetService` — Wails service example/template
|
||||
### Prerequisites
|
||||
|
||||
## Repository
|
||||
- Go 1.26+
|
||||
- Node.js 22+ and npm
|
||||
- Wails 3 CLI (`wails3`)
|
||||
- Access to the Go workspace at `~/Code/go.work` (the module uses `replace` directives for local siblings)
|
||||
|
||||
- **Source**: [forge.lthn.ai/core/ide](https://forge.lthn.ai/core/ide)
|
||||
- **Go module**: `forge.lthn.ai/core/ide`
|
||||
### Development
|
||||
|
||||
```bash
|
||||
cd /path/to/core/ide
|
||||
|
||||
# Install frontend dependencies and start dev mode (hot-reload)
|
||||
wails3 dev
|
||||
|
||||
# Or build a production binary
|
||||
core build # uses .core/build.yaml
|
||||
wails3 build # alternative, uses build/config.yml
|
||||
```
|
||||
|
||||
### Headless mode
|
||||
|
||||
```bash
|
||||
# Run as a headless daemon (no GUI required)
|
||||
./bin/core-ide --headless
|
||||
|
||||
# Dry-run mode -- logs what would happen without side-effects
|
||||
./bin/core-ide --headless --dry-run
|
||||
|
||||
# Specify which repos to poll (comma-separated)
|
||||
CORE_REPOS=host-uk/core,host-uk/core-php ./bin/core-ide --headless
|
||||
```
|
||||
|
||||
A systemd unit is provided at `build/linux/core-ide.service` for running headless mode as a system service.
|
||||
|
||||
## Package layout
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `main.go` | Entry point -- decides GUI or headless based on `--headless` flag / display availability |
|
||||
| `mcp_bridge.go` | `MCPBridge` -- Wails service that starts the MCP HTTP server, WebSocket hub, and webview tool dispatcher (GUI mode) |
|
||||
| `claude_bridge.go` | `ClaudeBridge` -- WebSocket relay between GUI clients and an upstream MCP core server |
|
||||
| `headless.go` | `startHeadless()` -- daemon with Forgejo poller, job handlers, agent dispatch, PID file, and health endpoint |
|
||||
| `headless_mcp.go` | `startHeadlessMCP()` -- minimal MCP HTTP server for headless mode (job status, dry-run toggle, manual poll trigger) |
|
||||
| `greetservice.go` | `GreetService` -- sample Wails-bound service |
|
||||
| `icons/` | Embedded PNG assets for system tray (macOS template icon, default icon) |
|
||||
| `frontend/` | Angular 20 application (standalone components, SCSS, Wails runtime bindings) |
|
||||
| `frontend/src/app/pages/tray/` | `TrayComponent` -- compact system tray panel (380x480 frameless window) |
|
||||
| `frontend/src/app/pages/ide/` | `IdeComponent` -- full IDE layout with sidebar, dashboard, file explorer, terminal placeholders |
|
||||
| `frontend/src/app/components/sidebar/` | `SidebarComponent` -- icon-based navigation rail |
|
||||
| `frontend/bindings/` | Auto-generated TypeScript bindings for Go services |
|
||||
| `build/` | Platform-specific build configuration (macOS plist, Windows NSIS/MSIX, Linux systemd/AppImage/nfpm) |
|
||||
| `.core/build.yaml` | `core build` configuration (Wails type, CGO enabled, cross-compilation targets) |
|
||||
| `build/config.yml` | Wails 3 project configuration (app metadata, dev mode settings, file associations) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Go modules
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `forge.lthn.ai/core/go` | Core framework -- config, forge client, job runner, agent CI |
|
||||
| `forge.lthn.ai/core/go-process` | Daemon utilities (PID file, health check endpoint) |
|
||||
| `forge.lthn.ai/core/gui` | WebView service (`pkg/webview`) and WebSocket hub (`pkg/ws`) |
|
||||
| `github.com/wailsapp/wails/v3` | Native desktop application framework |
|
||||
| `github.com/gorilla/websocket` | WebSocket client for the Claude bridge |
|
||||
|
||||
All three `forge.lthn.ai` dependencies are resolved via `replace` directives pointing to sibling directories (`../go`, `../go-process`, `../gui`).
|
||||
|
||||
### Frontend
|
||||
|
||||
| Package | Role |
|
||||
|---------|------|
|
||||
| `@angular/core` ^21 | Component framework |
|
||||
| `@wailsio/runtime` 3.0.0-alpha.72 | Go/JS bridge (events, method calls) |
|
||||
| `rxjs` ~7.8 | Reactive extensions |
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2. See the copyright notice in `build/config.yml` and `build/darwin/Info.plist`.
|
||||
|
|
|
|||
141
docs/tools/lint/index.md
Normal file
141
docs/tools/lint/index.md
Normal file
|
|
@ -0,0 +1,141 @@
|
|||
---
|
||||
title: core/lint
|
||||
description: Pattern catalog, regex-based code checker, and quality assurance toolkit for Go and PHP projects
|
||||
---
|
||||
|
||||
# core/lint
|
||||
|
||||
`forge.lthn.ai/core/lint` is a standalone pattern catalog and code quality toolkit. It ships a YAML-based rule catalog for detecting security issues, correctness bugs, and modernisation opportunities in Go source code. It also provides a full PHP quality assurance pipeline and a suite of developer tooling wrappers.
|
||||
|
||||
The library is designed to be embedded into other tools. The YAML rule files are compiled into the binary at build time via `go:embed`, so there are no runtime file dependencies.
|
||||
|
||||
## Module Path
|
||||
|
||||
```
|
||||
forge.lthn.ai/core/lint
|
||||
```
|
||||
|
||||
Requires Go 1.26+.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### As a Library
|
||||
|
||||
```go
|
||||
import (
|
||||
lint "forge.lthn.ai/core/lint"
|
||||
lintpkg "forge.lthn.ai/core/lint/pkg/lint"
|
||||
)
|
||||
|
||||
// Load the embedded rule catalog.
|
||||
cat, err := lint.LoadEmbeddedCatalog()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Filter rules for Go, severity medium and above.
|
||||
rules := cat.ForLanguage("go")
|
||||
filtered := (&lintpkg.Catalog{Rules: rules}).AtSeverity("medium")
|
||||
|
||||
// Create a scanner and scan a directory.
|
||||
scanner, err := lintpkg.NewScanner(filtered)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
findings, err := scanner.ScanDir("./src")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Output results.
|
||||
lintpkg.WriteText(os.Stdout, findings)
|
||||
```
|
||||
|
||||
### As a CLI
|
||||
|
||||
```bash
|
||||
# Build the binary
|
||||
core build # produces ./bin/core-lint
|
||||
|
||||
# Scan the current directory with all rules
|
||||
core-lint lint check
|
||||
|
||||
# Scan with filters
|
||||
core-lint lint check --lang go --severity high ./pkg/...
|
||||
|
||||
# Output as JSON
|
||||
core-lint lint check --format json .
|
||||
|
||||
# Browse the catalog
|
||||
core-lint lint catalog list
|
||||
core-lint lint catalog list --lang go
|
||||
core-lint lint catalog show go-sec-001
|
||||
```
|
||||
|
||||
### QA Commands
|
||||
|
||||
The `qa` command group provides workflow-level quality assurance:
|
||||
|
||||
```bash
|
||||
# Go-focused
|
||||
core qa watch # Monitor GitHub Actions after a push
|
||||
core qa review # PR review status with actionable next steps
|
||||
core qa health # Aggregate CI health across all repos
|
||||
core qa issues # Intelligent issue triage
|
||||
core qa docblock # Check Go docblock coverage
|
||||
|
||||
# PHP-focused
|
||||
core qa fmt # Format PHP code with Laravel Pint
|
||||
core qa stan # Run PHPStan/Larastan static analysis
|
||||
core qa psalm # Run Psalm static analysis
|
||||
core qa audit # Audit composer and npm dependencies
|
||||
core qa security # Security checks (.env, filesystem, deps)
|
||||
core qa rector # Automated code refactoring
|
||||
core qa infection # Mutation testing
|
||||
core qa test # Run Pest or PHPUnit tests
|
||||
```
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Package | Path | Description |
|
||||
|---------|------|-------------|
|
||||
| `lint` (root) | `lint.go` | Embeds YAML catalogs and exposes `LoadEmbeddedCatalog()` |
|
||||
| `pkg/lint` | `pkg/lint/` | Core library: Rule, Catalog, Matcher, Scanner, Report, Complexity, Coverage, VulnCheck, Toolkit |
|
||||
| `pkg/detect` | `pkg/detect/` | Project type detection (Go, PHP) by filesystem markers |
|
||||
| `pkg/php` | `pkg/php/` | PHP quality tools: format, analyse, audit, security, refactor, mutation, test, pipeline, runner |
|
||||
| `cmd/core-lint` | `cmd/core-lint/` | CLI binary (`core-lint lint check`, `core-lint lint catalog`) |
|
||||
| `cmd/qa` | `cmd/qa/` | QA workflow commands (watch, review, health, issues, docblock, PHP tools) |
|
||||
| `catalog/` | `catalog/` | YAML rule definitions (embedded at compile time) |
|
||||
|
||||
## Rule Catalogs
|
||||
|
||||
Three built-in YAML catalogs ship with the module:
|
||||
|
||||
| File | Rules | Focus |
|
||||
|------|-------|-------|
|
||||
| `go-security.yaml` | 6 | SQL injection, path traversal, XSS, timing attacks, log injection, secret leaks |
|
||||
| `go-correctness.yaml` | 7 | Unsynchronised goroutines, silent error swallowing, panics in library code, file deletion |
|
||||
| `go-modernise.yaml` | 5 | Replace legacy patterns with modern stdlib (`slices.Clone`, `slices.Sort`, `maps.Keys`, `errgroup`) |
|
||||
|
||||
Total: **18 rules** across 3 severity tiers (info, medium, high, critical). All rules target Go. The catalog is extensible -- add more YAML files to `catalog/` and they will be embedded automatically.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Direct dependencies:
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (`cli.Main()`, command registration, TUI styles) |
|
||||
| `forge.lthn.ai/core/go-i18n` | Internationalisation for CLI strings |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction for registry loading |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logging and error wrapping |
|
||||
| `forge.lthn.ai/core/go-scm` | Repository registry (`repos.yaml`) for multi-repo commands |
|
||||
| `github.com/stretchr/testify` | Test assertions |
|
||||
| `gopkg.in/yaml.v3` | YAML parsing for rule catalogs |
|
||||
|
||||
The `pkg/lint` sub-package has minimal dependencies (only `gopkg.in/yaml.v3` and standard library). The heavier CLI and SCM dependencies live in `cmd/`.
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2
|
||||
|
|
@ -1,30 +1,97 @@
|
|||
---
|
||||
title: MCP
|
||||
description: Model Context Protocol — Go MCP server and Laravel MCP package with 49 tools
|
||||
title: Core MCP
|
||||
description: Model Context Protocol server and tooling for AI agents -- Go binary + Laravel PHP package.
|
||||
---
|
||||
|
||||
# MCP
|
||||
# Core MCP
|
||||
|
||||
`forge.lthn.ai/core/mcp`
|
||||
`forge.lthn.ai/core/mcp` provides a complete Model Context Protocol (MCP)
|
||||
implementation spanning two languages:
|
||||
|
||||
Model Context Protocol server combining a Go MCP server with a Laravel MCP package. Produces the `core-mcp` binary. 49 MCP tools covering brain, RAG, ML, IDE bridge, and more.
|
||||
- **Go** -- a standalone MCP server binary (`core-mcp`) with file operations,
|
||||
ML inference, RAG, process management, webview automation, and WebSocket
|
||||
streaming.
|
||||
- **PHP** -- a Laravel package (`lthn/mcp`) that adds an HTTP MCP API, tool
|
||||
registry, SQL query tools, quota enforcement, circuit breakers, audit
|
||||
logging, and an admin panel to any Host UK application.
|
||||
|
||||
## What's Inside
|
||||
Both halves speak the same protocol and can bridge to one another via REST or
|
||||
WebSocket.
|
||||
|
||||
### Go MCP Server
|
||||
## Quick start
|
||||
|
||||
43 Go files in `pkg/mcp/` providing the MCP server implementation and `cmd/` entry point for the `core-mcp` binary.
|
||||
### Go binary
|
||||
|
||||
### Laravel MCP Package
|
||||
```bash
|
||||
# Build
|
||||
core build # produces ./core-mcp
|
||||
|
||||
145 PHP files in `src/php/` providing the Laravel integration — tool handlers, workspace management, database querying, analytics, quotas, and security.
|
||||
# Start on stdio (for Claude Code / IDE integration)
|
||||
./core-mcp mcp serve
|
||||
|
||||
## Repository
|
||||
# Start on TCP
|
||||
MCP_ADDR=127.0.0.1:9100 ./core-mcp mcp serve
|
||||
|
||||
- **Source**: [forge.lthn.ai/core/mcp](https://forge.lthn.ai/core/mcp)
|
||||
- **Go module**: `forge.lthn.ai/core/mcp`
|
||||
- **Composer**: `lthn/mcp`
|
||||
# Restrict file operations to a directory
|
||||
./core-mcp mcp serve --workspace /path/to/project
|
||||
```
|
||||
|
||||
## Absorbed Archives
|
||||
### PHP package
|
||||
|
||||
This repo consolidates code from the now-archived `php-mcp` package.
|
||||
Add the Composer dependency to a Laravel application:
|
||||
|
||||
```bash
|
||||
composer require lthn/mcp
|
||||
```
|
||||
|
||||
The package auto-registers via `Core\Front\Mcp\Boot`. No manual provider
|
||||
registration is needed. Run migrations, then visit the admin panel at
|
||||
`/admin/mcp`.
|
||||
|
||||
## Package layout
|
||||
|
||||
| Path | Language | Purpose |
|
||||
|------|----------|---------|
|
||||
| `pkg/mcp/` | Go | Core MCP server: `Service`, transports, tool registry, REST bridge |
|
||||
| `pkg/mcp/brain/` | Go | OpenBrain knowledge-store subsystem (remember/recall/forget/list) |
|
||||
| `pkg/mcp/ide/` | Go | IDE subsystem: Laravel WebSocket bridge, chat, builds, dashboard |
|
||||
| `cmd/core-mcp/` | Go | Binary entry point (`core-mcp`) |
|
||||
| `cmd/mcpcmd/` | Go | CLI command registration (`mcp serve`) |
|
||||
| `cmd/brain-seed/` | Go | Utility to import CLAUDE.md / MEMORY.md files into OpenBrain |
|
||||
| `src/php/src/Mcp/` | PHP | Laravel service provider, models, services, middleware, tools |
|
||||
| `src/php/src/Front/Mcp/` | PHP | MCP frontage: middleware group, `McpToolHandler` contract, `McpContext` |
|
||||
| `src/php/src/Website/Mcp/` | PHP | Public-facing MCP pages: playground, API explorer, metrics |
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Go
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `forge.lthn.ai/core/go` | DI container and service lifecycle |
|
||||
| `forge.lthn.ai/core/cli` | CLI framework (bubbletea TUI) |
|
||||
| `forge.lthn.ai/core/go-io` | Filesystem abstraction (`Medium`, sandboxing) |
|
||||
| `forge.lthn.ai/core/go-log` | Structured logger |
|
||||
| `forge.lthn.ai/core/go-ai` | AI event metrics |
|
||||
| `forge.lthn.ai/core/go-ml` | ML inference, scoring, capability probes |
|
||||
| `forge.lthn.ai/core/go-inference` | Backend registry (Ollama, MLX, ROCm) |
|
||||
| `forge.lthn.ai/core/go-rag` | Qdrant vector search and document ingestion |
|
||||
| `forge.lthn.ai/core/go-process` | External process lifecycle management |
|
||||
| `forge.lthn.ai/core/go-webview` | Chrome DevTools Protocol automation |
|
||||
| `forge.lthn.ai/core/go-ws` | WebSocket hub and channel messaging |
|
||||
| `forge.lthn.ai/core/go-api` | REST API framework and `ToolBridge` |
|
||||
| `github.com/modelcontextprotocol/go-sdk` | Official MCP Go SDK |
|
||||
| `github.com/gin-gonic/gin` | HTTP router (REST bridge) |
|
||||
| `github.com/gorilla/websocket` | WebSocket client for IDE bridge |
|
||||
|
||||
### PHP
|
||||
|
||||
| Package | Role |
|
||||
|---------|------|
|
||||
| `lthn/php` (core/php) | Foundation framework: lifecycle events, modules, actions |
|
||||
| Laravel 12 | Application framework |
|
||||
| Livewire / Flux Pro | Admin panel components |
|
||||
|
||||
## Licence
|
||||
|
||||
EUPL-1.2. See the `SPDX-License-Identifier` headers in each source file.
|
||||
|
|
|
|||
|
|
@ -1,23 +1,117 @@
|
|||
---
|
||||
title: CoreTS
|
||||
description: TypeScript runtime management — Go service wrapping Deno with lifecycle integration
|
||||
description: Go service that manages a Deno TypeScript runtime as a sandboxed sidecar process, providing permission-gated I/O over gRPC and Unix sockets.
|
||||
---
|
||||
|
||||
# CoreTS
|
||||
|
||||
`forge.lthn.ai/core/ts`
|
||||
CoreTS (`forge.lthn.ai/core/ts`) is a Go package that embeds a **Deno TypeScript runtime** as a managed sidecar process. It provides a bidirectional communication bridge between Go and Deno over Unix sockets, with fine-grained permission gating for filesystem, key-value store, and process operations.
|
||||
|
||||
Go service that manages a Deno TypeScript runtime as a sidecar process. Provides lifecycle integration with the Core framework, permission management, and a client interface for communicating with TypeScript modules from Go.
|
||||
The Go side exposes a **CoreService** gRPC server that Deno calls for I/O. The Deno side exposes a **DenoService** JSON-RPC server that Go calls for module lifecycle management. TypeScript modules run in isolated Deno Workers with per-module permission sandboxing.
|
||||
|
||||
## Key Types
|
||||
**Module path:** `forge.lthn.ai/core/ts`
|
||||
|
||||
- `Sidecar` — manages the Deno process lifecycle
|
||||
- `DenoClient` — communicates with running Deno instances
|
||||
- `Options` — configuration for the TypeScript runtime
|
||||
- `Permissions` — Deno permission mapping (network, read, write, env)
|
||||
- `ModulePermissions` — per-module permission scoping
|
||||
**Licence:** EUPL-1.2
|
||||
|
||||
## Repository
|
||||
## Quick Start
|
||||
|
||||
- **Source**: [forge.lthn.ai/core/ts](https://forge.lthn.ai/core/ts)
|
||||
- **Go module**: `forge.lthn.ai/core/ts`
|
||||
Register CoreTS as a service in a Core application:
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
core "forge.lthn.ai/core/go/pkg/core"
|
||||
ts "forge.lthn.ai/core/ts"
|
||||
)
|
||||
|
||||
opts := ts.Options{
|
||||
DenoPath: "deno",
|
||||
SocketPath: "/tmp/core/core.sock",
|
||||
AppRoot: "/app",
|
||||
SidecarArgs: []string{"run", "-A", "--unstable-worker-options", "runtime/main.ts"},
|
||||
}
|
||||
|
||||
app, err := core.New(core.WithService(ts.NewServiceFactory(opts)))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
```
|
||||
|
||||
On startup, the service will:
|
||||
|
||||
1. Create a sandboxed I/O medium scoped to `AppRoot`
|
||||
2. Open a SQLite-backed key-value store
|
||||
3. Start a gRPC server on a Unix socket
|
||||
4. Load and optionally verify the application manifest
|
||||
5. Launch the Deno sidecar process
|
||||
6. Wait for Deno's JSON-RPC server and connect as a client
|
||||
7. Auto-load any previously installed marketplace modules
|
||||
|
||||
## Package Layout
|
||||
|
||||
| Path | Language | Purpose |
|
||||
|------|----------|---------|
|
||||
| `*.go` | Go | Sidecar management, gRPC server, permission checks, service integration |
|
||||
| `proto/coredeno.proto` | Protobuf | Service definitions for CoreService and DenoService |
|
||||
| `proto/*.pb.go` | Go | Generated protobuf and gRPC stubs |
|
||||
| `runtime/main.ts` | TypeScript | Deno entry point -- boots the runtime, connects to Go |
|
||||
| `runtime/client.ts` | TypeScript | gRPC client that calls CoreService on the Go side |
|
||||
| `runtime/server.ts` | TypeScript | JSON-RPC server that implements DenoService for Go to call |
|
||||
| `runtime/modules.ts` | TypeScript | Module registry with Worker isolation and I/O bridge |
|
||||
| `runtime/worker-entry.ts` | TypeScript | Worker bootstrap -- loaded as entry point for every module Worker |
|
||||
| `runtime/polyfill.ts` | TypeScript | Patches for Deno 2.x http2/grpc-js compatibility issues |
|
||||
| `runtime/testdata/` | TypeScript | Test fixtures for integration tests |
|
||||
|
||||
## Go Source Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `coredeno.go` | `Options`, `Permissions`, `Sidecar` types and `NewSidecar()` constructor |
|
||||
| `lifecycle.go` | `Sidecar.Start()`, `Stop()`, `IsRunning()` -- process lifecycle |
|
||||
| `listener.go` | `ListenGRPC()` -- Unix socket gRPC listener with graceful shutdown |
|
||||
| `server.go` | `Server` -- CoreService gRPC implementation with permission gating |
|
||||
| `denoclient.go` | `DenoClient` -- JSON-RPC client for calling the Deno sidecar |
|
||||
| `permissions.go` | `CheckPath()`, `CheckNet()`, `CheckRun()` -- permission helpers |
|
||||
| `service.go` | `Service` -- framework integration (Startable/Stoppable lifecycle) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `forge.lthn.ai/core/go` | Core framework (DI container, `ServiceRuntime`, lifecycle interfaces) |
|
||||
| `forge.lthn.ai/core/go-io` | Sandboxed filesystem I/O (`Medium` interface, `MockMedium`) |
|
||||
| `forge.lthn.ai/core/go-io/store` | SQLite-backed key-value store |
|
||||
| `forge.lthn.ai/core/go-scm/manifest` | Module manifest loading and ed25519 verification |
|
||||
| `forge.lthn.ai/core/go-scm/marketplace` | Module installation from Git repositories |
|
||||
| `google.golang.org/grpc` | gRPC server and client |
|
||||
| `google.golang.org/protobuf` | Protocol buffer runtime |
|
||||
| `github.com/stretchr/testify` | Test assertions (dev only) |
|
||||
|
||||
The Deno runtime uses npm packages managed via `runtime/deno.json`:
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `@grpc/grpc-js` | gRPC client for calling CoreService |
|
||||
| `@grpc/proto-loader` | Dynamic protobuf loading |
|
||||
|
||||
## Configuration
|
||||
|
||||
The `Options` struct controls all behaviour:
|
||||
|
||||
```go
|
||||
type Options struct {
|
||||
DenoPath string // Path to deno binary (default: "deno")
|
||||
SocketPath string // Unix socket for Go's gRPC server
|
||||
DenoSocketPath string // Unix socket for Deno's JSON-RPC server
|
||||
AppRoot string // Application root directory (sandboxed I/O boundary)
|
||||
StoreDBPath string // SQLite path (default: AppRoot/.core/store.db)
|
||||
PublicKey ed25519.PublicKey // Ed25519 key for manifest verification (optional)
|
||||
SidecarArgs []string // Arguments passed to the Deno process
|
||||
}
|
||||
```
|
||||
|
||||
If `SocketPath` is not set, it defaults to `$XDG_RUNTIME_DIR/core/deno.sock` (or `/tmp/core/deno.sock` on macOS).
|
||||
|
||||
If `DenoSocketPath` is not set, it defaults to the same directory as `SocketPath` with filename `deno.sock`.
|
||||
|
||||
If `StoreDBPath` is not set and `AppRoot` is provided, it defaults to `AppRoot/.core/store.db`.
|
||||
|
|
|
|||
|
|
@ -211,6 +211,9 @@ nav = [
|
|||
{"IDE" = [
|
||||
"tools/ide/index.md",
|
||||
]},
|
||||
{"Lint" = [
|
||||
"tools/lint/index.md",
|
||||
]},
|
||||
{"CLI" = [
|
||||
"tools/cli/index.md",
|
||||
{"Development" = [
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue