Commit graph

42 commits

Author SHA1 Message Date
Snider
dff5fb23c5 refactor: apply go fix modernizers for Go 1.26
Automated fixes: interface{} → any, range-over-int, t.Context(),
wg.Go(), strings.SplitSeq, strings.Builder, slices.Contains,
maps helpers, min/max builtins.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-22 21:00:16 +00:00
Snider
6d07388407 chore: bump go directive to 1.26.0
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-22 20:33:48 +00:00
Snider
b2b4982e3c chore: pin forge deps to v0.0.1 tags for Go 1.26 compat
Go 1.26 rejects non-semver version strings (like 'main') in go.mod.
Tags v0.0.1 now exist on all forge repos — workspace still overrides
for local development.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-22 20:15:05 +00:00
Snider
ad865ab3b9 chore: use workspace-resolved versions, drop replace directives
Forge module versions now use main branch resolution via ~/Code/go.work
workspace. Removes local replace directives — the central go.work handles
all cross-repo resolution during development.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-22 19:49:36 +00:00
Snider
85e02297cc fix: update ml tool handlers for Result return type
go-ml's Backend.Generate() and Backend.Chat() now return ml.Result{Text, Metrics}
instead of (string, error). Updated mlGenerate and mlProbe handlers to unwrap
result.Text, and updated test mocks to match the new interface.

Added local replace directives for go-inference, go-ml, go-mlx while
these changes propagate through the dependency chain.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-22 19:24:32 +00:00
Snider
6298a84b8b refactor: remove init() + RegisterCommands pattern
Commands now register through Core framework lifecycle via
cli.WithCommands() instead of side-channel init() functions.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 22:07:10 +00:00
Snider
63d74568e2 feat: absorb lab command from CLI
LEM Lab launcher for local ML inference.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 21:46:59 +00:00
Snider
55e537a55c feat: migrate metrics command from CLI
Move AI metrics tracking command from core/cli cmd/ai into go-ai
where the metrics backend lives.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 20:42:17 +00:00
Snider
d962260b81 feat(cmd/rag): re-export go-rag CLI commands through go-ai
CLI should import RAG commands via go-ai (which already depends on
go-rag), not directly. This keeps the CLI's dependency tree clean.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 20:23:22 +00:00
Snider
c3de6c4935 feat: migrate daemon, mcpcmd, security commands from CLI
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 19:38:04 +00:00
Snider
bc9597c191 chore: resolve forge deps from registry, remove local replaces
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 19:22:32 +00:00
Snider
c37e1cf2de fix(mcp): harden REST bridge with body limit, error classification, sanitised messages
- Add 10MB body size limit via io.LimitReader
- Classify JSON parse errors as 400 Bad Request (not 500)
- Sanitise error messages to prevent path leakage
- Document nil CallToolRequest in RESTHandler closure

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 01:27:06 +00:00
Snider
d20688d50b feat(mcp): add BridgeToAPI for MCP-to-REST conversion
Bridge converts MCP ToolRecords into go-api ToolBridge entries,
making MCP tools accessible as REST POST endpoints with OpenAPI
metadata. Each tool's RESTHandler closure captures the generic
input type at registration time, enabling type-safe JSON
unmarshalling without runtime reflection.

Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 01:06:33 +00:00
Snider
fbf6421046 fix(mcp): add EUPL-1.2 header and fix empty struct schema
- Add missing SPDX license header to mcp.go
- Return empty schema object for zero-field structs instead of nil

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 01:01:13 +00:00
Snider
2107edaf79 feat(mcp): add parallel tool registry for REST bridge
Introduce ToolRecord type and addToolRecorded generic helper that
registers tools with the MCP SDK server while simultaneously recording
metadata (name, description, group, input/output JSON schemas) in a
parallel registry. This enables the upcoming REST bridge to enumerate
available tools and their schemas without depending on the SDK server's
unexported internals.

Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 00:59:13 +00:00
Snider
2210e0e298 docs: add README with quick start and docs links
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 15:11:19 +00:00
Snider
9016e2ee7d docs: graduate TODO/FINDINGS into production documentation
Replace internal task tracking (TODO.md, FINDINGS.md) with four
structured documentation files covering architecture, tool reference,
development guide, and project history. Trim CLAUDE.md to agent
instructions only — all detailed content now lives in docs/.

- docs/architecture.md: subsystem plugin model, transports, IDE bridge,
  AI facade, full package layout
- docs/tools.md: all 49 MCP tools with parameters and descriptions
- docs/development.md: prerequisites, test patterns, adding tools/subsystems
- docs/history.md: split history, 5 phases with commit hashes, known issues

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 14:54:05 +00:00
Claude
65ff6c6647
fix(go.mod): correct replace path for core/go
Was ../core (wrong), now ../go matching workspace layout.

Co-Authored-By: Charon <charon@lethean.io>
2026-02-20 14:20:33 +00:00
Snider
c5b17f8537 fix(mcp): eliminate data race in TCP transport diagnostic output
Replace bare os.Stderr writes with mutex-protected diagPrintf helper.
Test captures output via setDiagWriter instead of swapping os.Stderr,
preventing races with concurrent ServeTCP goroutines.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 12:30:16 +00:00
Snider
8e068ab867 fix(mcp): align rag tool with go-rag CollectionInfo API
PointsCount → PointCount (uint64, not pointer).
Status is already a string, remove .String() call.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 12:25:55 +00:00
Snider
28a574ad0a docs: mark Phase 4 complete in TODO.md
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 07:03:13 +00:00
Snider
8c0ef43dae feat(ide): Phase 4 — bridge auth, reconnect hardening, tool tests
4.1 Bridge reconnection: Fixed data race on callCount (atomic.Int32),
added exponential backoff verification test (HTTP 403 rejection path),
added server shutdown detection test.

4.2 Bridge auth: Added Token field to Config, WithToken option,
Authorization Bearer header in dial() when token is non-empty.
Tests verify header presence and absence.

4.3 Tool handler tests: 49 tests covering all 11 IDE tool handlers
(5 chat/session, 3 build, 3 dashboard) with nil bridge (error path)
and connected mock bridge (success path). JSON round-trip tests for
all input/output types. Added stub documentation comments to all
tool handler functions.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 07:02:50 +00:00
Snider
6f6844a8a5 test: Phase 5 — testing gaps (process/RAG/ML mocks, metrics bench)
Process tools CI tests:
- Full lifecycle tests using real process.Service with echo/sleep/cat/pwd/env
- Handler validation: empty command, empty ID, nonexistent ID, empty input
- Start → list → output → kill → list lifecycle test
- Working directory and environment variable passthrough tests
- stdin/stdout round-trip via cat process

RAG tools mock tests:
- Handler validation: empty question, empty path, nonexistent path
- Default collection and topK application verification
- Graceful error when Qdrant/Ollama unavailable (no panic)
- Expanded struct round-trip tests for all RAG types

ML tools mock tests:
- Mock ml.Backend for Generate/Chat without real inference
- Mock inference.Backend for registry testing
- Handler validation: empty prompt, empty response, missing backend
- Heuristic scoring without live services
- Semantic scoring fails gracefully without judge
- Content suite redirects to ml_probe
- Capability probes run against mock backend (23 probes)
- ml_backends lists mock inference registry entries

Metrics benchmarks:
- BenchmarkMetricsRecord: ~22μs/op single-threaded
- BenchmarkMetricsRecord_Parallel: ~13μs/op with 32 goroutines
- BenchmarkMetricsQuery_10K: ~15ms/op reading 10K JSONL events
- BenchmarkMetricsQuery_50K: ~75ms/op reading 50K JSONL events
- BenchmarkMetricsSummary_10K: ~235μs/op aggregating 10K events
- TestMetricsRecordAndRead_10K_Good: write+read+summarise 10K events

Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 06:48:31 +00:00
Snider
039bd11814 test: Phase 3 — MCP transport e2e tests
- TCP e2e: full JSON-RPC round-trip (initialize → tools/list → tools/call)
  including file_read, file_write, tool discovery, and error handling
- Unix socket e2e: connect + tool call via unix domain socket with
  short paths for macOS 104-byte sun_path limit
- Webview tools CI guard: skipIfShort() + TestWebviewToolHandlers_RequiresChrome
  verifies handler error paths without Chrome
- Stdio: documented skip (SDK binds os.Stdin/Stdout directly; protocol
  covered by TCP and Unix e2e tests)

Test count: 84 → 108 (8 new tests: 7 pass, 1 documented skip)

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 06:38:34 +00:00
Snider
43dabbe99f docs: add commit hash to Phase 2 TODO items
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 06:01:56 +00:00
Snider
4d73fa250c feat(ml): Phase 2 — migrate ml_backends to go-inference registry
- Make go-inference a direct dependency (was indirect)
- Rewrite mlBackends() to use inference.List()/Get()/Default()
  instead of ml.Service.Backends()/Backend()/DefaultBackend()
- Add documentation comments clarifying generation flow
- mlGenerate/mlScore/mlProbe unchanged (work via go-ml.Service)

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 06:01:30 +00:00
Snider
4665bea75e fix: post-split cleanup — remove test script, fix tests, update docs
- Delete test-mlx.go (standalone test script, not library code)
- Fix TestSandboxing_Symlinks_Followed to match security behaviour
  (renamed to TestSandboxing_Symlinks_Blocked — asserts sandbox
  correctly blocks symlinks escaping the workspace root)
- Fix TestNewTCPTransport_Warning by adding missing security warning
  to NewTCPTransport when binding to 0.0.0.0 (all interfaces)
- Update CLAUDE.md dependency table (go-mlx, duckdb, parquet, ollama,
  qdrant now indirect via go-ml/go-rag)
- Tidy go.mod (direct vs indirect aligned with actual imports)
- Verify go build, go vet, go test all pass

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 01:18:27 +00:00
Snider
a36466a870 docs: rewrite TODO.md with concrete file-level tasks
Replace vague Phase 1-4 with specific actionable items:
remove test-mlx.go, go-inference migration details,
MCP transport e2e testing, IDE bridge hardening.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 00:28:42 +00:00
Virgil
e777d84a46 docs: add CLAUDE.md, TODO.md, FINDINGS.md for fleet delegation
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 21:34:55 +00:00
Snider
0af152eaef refactor: extract ml/ to standalone core/go-ml module
go-ai is now a meta package: ai/ facade + mcp/ hub that imports
go-ml, go-rag, go-mlx, and go-agentic as separate modules.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 18:32:42 +00:00
Snider
2886ffaa4c refactor: extract rag/ to standalone core/go-rag module
Update ai/rag.go and mcp/tools_rag.go imports to reference new module.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 18:30:37 +00:00
Snider
f99ca10c6c refactor: extract agentic/ to standalone core/go-agentic module
Now lives at forge.lthn.ai/core/go-agentic.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 18:28:48 +00:00
Snider
34d0f9ce41 refactor: extract mlx/ to standalone core/go-mlx module
Remove mlx/ directory (now lives at forge.lthn.ai/core/go-mlx).
Update ml/backend_mlx.go imports to reference the new module.
Add replace directive for local development.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 18:24:32 +00:00
Snider
906a535899 chore: update module paths and add gitignore
Sync module imports across agentic, mcp, ml, and rag packages.
Add .gitignore for mlx build artifacts.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 19:19:38 +00:00
Snider
98749c66f2 fix(mlx): add DecodeToken for correct streaming word boundaries
The Decode method strips the SentencePiece leading space from every
token, which loses word boundaries during streaming. DecodeToken
preserves the space (it represents the word boundary) and only the
first token of each generation has its leading space stripped.

Fixes Gemma3 space prefix appearing in chat UI output.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 19:18:27 +00:00
Snider
2d870385f9 feat(mlx): LoRA injection into models + masked cross-entropy loss
Add LoRA field to Linear for transparent adapter injection via model's
Forward() path. ApplyLoRA() on Qwen3/Gemma3 wraps target projections.
Deterministic param ordering for adapter save/load consistency.
MaskedCrossEntropyLoss for training on assistant tokens only.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 17:37:44 +00:00
Snider
0eaf3d5a17 feat(mlx): add LoRA adapter layers and AdamW optimizer
LoRA: low-rank adaptation with trainable A/B matrices, Kaiming normal
init, safetensors save/load. AdamW: decoupled weight decay optimizer
with positional moment tracking for gradient-replaced params.

14 tests passing including end-to-end LoRA+AdamW training loop.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 17:25:42 +00:00
Snider
e9973aef3c feat(mlx): add autograd — VJP, JVP, ValueAndGrad, loss functions
Native Go bindings for MLX-C gradient computation on Apple Silicon.
Foundation for LoRA training without Python.

- VJP (reverse-mode autodiff) for backward pass
- JVP (forward-mode autodiff) for directional derivatives
- ValueAndGrad for combined loss + gradient computation
- Checkpoint for memory-efficient gradient recomputation
- CrossEntropyLoss (numerically stable via LogSumExp)
- MSELoss, Log, SumAll, MeanAll, OnesLike helpers
- TakeAlongAxis and LogSumExp ops
- Fix closure callback null vector bug (affects compile.go too)
- Fix Float() returning 0 for float32 arrays

14 tests passing on Metal GPU.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 17:18:47 +00:00
Snider
92c6282d50 refactor(mlx): drop mlx build tag, auto-enable on darwin/arm64
Remove the manual -tags mlx requirement. MLX is now automatically
compiled on darwin/arm64 via build constraints. Stubs remain for
other platforms. No functional change.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 16:57:41 +00:00
d0cbd5065e Merge pull request 'test: validate MLX inference and scoring pipeline on M3 Ultra' (#4) from test/mlx-validation-m3-ultra into main
Reviewed-on: #4
2026-02-16 17:47:16 +00:00
3916633f4d test: validate MLX inference and scoring pipeline on M3 Ultra
Fixes #2

- Run complete test suite: all 84 tests passing (100%)
- Verify Metal 4 GPU support and hardware capabilities
- Test scoring pipeline (heuristic + judge + engine)
- Confirm GGUF model directory with 9 models (40.43 GB)
- Document MLX backend build requirements
- Update module imports from forge.lthn.ai/core/go to forge.lthn.ai/core/cli
- Add comprehensive TEST-RESULTS.md with findings

Platform: M3 Ultra (60 GPU cores, 96GB RAM, Metal 4)
Results: All tests passing, scoring pipeline operational, MLX ready to build

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-16 17:24:36 +00:00
Claude
e84d6ad3c9
feat: extract AI/ML packages from core/go
LEM scoring pipeline, native MLX Metal bindings, Claude SDK wrapper,
RAG with Qdrant/Ollama, unified AI facade, and MCP protocol server.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 15:25:55 +00:00