Commit graph

858 commits

Author SHA1 Message Date
Snider
d7e5215618 docs: add core.help implementation plan
5 tasks: Hugo scaffold, KB scanning, --target hugo sync,
pipeline testing, BunnyCDN deployment config.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 02:13:14 +00:00
Snider
1e8a4131db docs: add core.help website design document
Hugo + Docsy static site aggregating docs from 39 repos.
Pipeline: core docs sync → Hugo build → BunnyCDN deploy.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 02:08:40 +00:00
Snider
df011ee42b feat: support .core/repos.yaml and explicit repo paths
- FindRegistry() now checks .core/repos.yaml alongside repos.yaml
- Repo.Path field accepts explicit path from YAML for repos outside base_path

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 01:58:08 +00:00
Snider
2d355f9223 docs: mark go-api Phase 3 complete in design doc
Phase 3 adds OpenAPI 3.1 runtime spec generation, MCP-to-REST bridge,
SDK codegen for 11 languages, and CLI commands. 176 tests in go-api.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 01:33:24 +00:00
Snider
db0c0adb65 docs: mark go-api Phase 2 complete — all 4 waves done
Wave 4 adds WithPprof, WithExpvar, WithTracing (OpenTelemetry).
143 tests passing across 21 With*() options. Phase 2 roadmap fully delivered.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 00:26:42 +00:00
Snider
ce12778561 docs: mark go-api Wave 3 complete in design doc
Wave 3 adds WithHTTPSign, WithSSE, WithLocation, WithI18n, WithGraphQL.
128 tests passing, 5 commits on forge.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 00:15:02 +00:00
Snider
44122f9ca6 docs: mark go-api Wave 2 complete in design doc
Wave 2 adds WithBrotli, WithCache, WithSessions, WithAuthz.
102 tests passing, 4 commits on forge.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 23:52:18 +00:00
Snider
b2e046f4c5 docs(go-api): mark Wave 1 complete — secure, slog, timeout, gzip, static
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 23:25:20 +00:00
Snider
3135352b2f docs: add Authentik + Traefik integration plan
10 tasks covering DevOps deployment (enable Authentik on de2,
add to prod playbook, configure OIDC app) and go-api middleware
(header extraction, JWT validation, RequireAuth/RequireGroup helpers).
~220 LOC, 16 new tests.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 16:30:52 +00:00
Snider
2bae1148bb docs(go-api): add Authentik integration and auth layers to design
Adds forward auth (Traefik) and direct OIDC validation modes,
auth layers diagram, deployment notes, and coreos/go-oidc dependency.
Authentik added to Wave 1 plugin roadmap.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 16:23:02 +00:00
Snider
cffd9d3929 docs(go-api): add GraphQL to Phase 2 roadmap
gqlgen (99designs) endpoint at /graphql with playground.
ResolverGroup interface for subsystem opt-in alongside RouteGroup.
Four-protocol access: REST + GraphQL + WebSocket + MCP.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 16:18:49 +00:00
Snider
cb0408db1d docs(go-api): add Phase 2 gin plugin roadmap
19 gin-contrib plugins mapped to With*() options across 4 waves:
security, performance, content/streaming, and observability.
Phase 1 marked as implemented with test counts.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 16:16:42 +00:00
Snider
e7f8ecb078 docs: add go-api implementation plan
10 sequential TDD tasks covering response envelope, RouteGroup interface,
Engine with options, middleware (auth/CORS/request-id), WebSocket
integration, Swagger UI, docs, Forge repo creation, and first
subsystem integration in go-ml.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 15:40:36 +00:00
Snider
1cdf92490a docs: add go-api REST framework design
REST projection of MCP tools via Gin + swaggo OpenAPI generation.
Subsystems implement RouteGroup interface to register their own
endpoints. Includes WebSocket integration via go-ws, response
envelope, middleware stack, and SDK generation pipeline.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 15:37:40 +00:00
Snider
bcf2d3be48 docs: remove dispatch queue, add ecosystem map and docs index
Delete TODO.md (satellite dispatch queue — all tasks complete) and
FINDINGS.md (go-i18n research — absorbed into go-i18n docs).
Add docs/ecosystem.md mapping all 19 satellite Go packages with
dependency graph and Forge SSH URLs. Rewrite docs/index.md as a
proper documentation hub with section organisation.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 15:10:56 +00:00
Snider
19521c8f18 chore: update gitignore for cmd/ build artifacts and landing page
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 12:16:55 +00:00
Snider
22121eae20 fix(i18n): skip completeness test when no T() calls exist in source
The test scanned for i18n.T("cmd.*") calls but none exist yet — CLI
commands haven't been wired to i18n. Changed require.NotEmpty to
t.Skip so the suite is green until translation keys are added.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 11:28:35 +00:00
Snider
b2e78bf29e fix: resolve IO migration test failures in node, cache, and cli
- pkg/io/node: implement ReadFile (fs.ReadFileFS), Walk with WalkOptions,
  CopyFile, FromTar constructor; fix Exists test calls to match bool return
- pkg/cache: add Medium DI parameter, use errors.Is for wrapped ErrNotExist
- pkg/cli: add Medium DI to PIDFile and DaemonOptions for testability
- TODO.md: mark go-i18n article/irregular validator complete

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-20 10:58:57 +00:00
Snider
94480ca38e docs: add LEM Lab conversational training pipeline design
Design doc for LEM's chat-driven training pipeline covering
prompt-response capture, DPO pair generation, and LoRA fine-tuning
flow for local MLX models.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 16:31:15 +00:00
Snider
3ff7b8a773 docs: add orchestration dispatch queue and research findings
TODO.md tracks tasks dispatched to satellite repos (go-i18n phases 1-3).
FINDINGS.md records go-i18n architecture assessment and CoreDeno PR #9 review.
Phase 2 expanded with 1B classification pipeline based on LEK benchmarks.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 15:51:54 +00:00
0192772ab5 Merge pull request 'feat(coredeno): Phase 4 foundation — Deno sidecar with marketplace install' (#9) from phase4-foundation into dev
Reviewed-on: #9
Reviewed-by: Snider <snider@noreply.forge.lthn.ai>
2026-02-19 14:44:08 +00:00
Snider
c1bc0dad5e merge: resolve conflicts with dev (PR #10 symlink fix)
Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 14:41:53 +00:00
Snider
19e3fd3af7 fix(coredeno): harden security and fix review issues
- Path traversal: CheckPath now requires separator after prefix match
- Store namespace: block reserved '_' prefixed groups
- StoreGet: distinguish ErrNotFound from real DB errors via sentinel
- Store: add rows.Err() checks in GetAll and Render
- gRPC leak: cleanupGRPC on all early-return error paths in OnStartup
- DenoClient: fix fmt.Sprint(nil) → type assertions
- Socket permissions: 0700 dirs, 0600 sockets (owner-only)
- Marketplace: persist SignKey, re-verify manifest on Update
- io/local: resolve symlinks in New() (macOS /var → /private/var)
- Tests: fix sun_path length overflow on macOS

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 14:39:56 +00:00
10f0ebaf22 Merge pull request 'fix(io/local): resolve symlinks on sandbox root' (#10) from fix/macos-sandbox-symlink into dev
Reviewed-on: #10
2026-02-19 14:22:27 +00:00
Snider
cbaa114bb2 fix(io/local): resolve symlinks on sandbox root to prevent false escape detection
Some checks failed
Auto Merge / merge (pull_request) Has been cancelled
CI / qa (pull_request) Has been cancelled
Coverage / coverage (pull_request) Has been cancelled
PR Build / build (amd64, linux, ubuntu-latest) (pull_request) Has been cancelled
PR Build / draft-release (pull_request) Has been cancelled
On macOS, /var is a symlink to /private/var. When New() stores the
unresolved root but validatePath() resolves child paths via EvalSymlinks,
the mismatch causes filepath.Rel to produce ".." prefixes — triggering
false SECURITY sandbox escape warnings on every file operation.

Fix: resolve symlinks on the root path in New() so both sides compare
like-for-like. Updates TestNew to compare against resolved paths.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-19 14:20:39 +00:00
Claude
9899398153
feat(coredeno): Tier 4 marketplace install pipeline — clone, verify, register, auto-load
Wire the marketplace to actually install modules from Git repos, verify
manifest signatures, track installations in the store, and auto-load them
as Workers at startup. A module goes from marketplace entry to running
Worker with Install() + LoadModule().

- Add Store.GetAll() for group-scoped key listing
- Create marketplace.Installer with Install/Remove/Update/Installed
- Export manifest.MarshalYAML for test fixtures
- Wire installer into Service with auto-load on startup (step 8)
- Expose Service.Installer() accessor
- Full integration test: install → load → verify store write → unload → remove

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 08:04:13 +00:00
Claude
ad6a466459
feat(coredeno): Tier 3 Worker isolation — sandboxed module loading with I/O bridge
Each module now runs in a real Deno Worker with per-module permission
sandboxing. The I/O bridge relays Worker postMessage calls through the
parent to CoreService gRPC, so modules can access store, files, and
processes without direct network/filesystem access.

- Worker bootstrap (worker-entry.ts): sets up RPC bridge, dynamically
  imports module, calls init(core) with typed I/O object
- ModuleRegistry rewritten: creates Workers with Deno permission
  constructor, handles LOADING → RUNNING → STOPPED lifecycle
- Structured ModulePermissions (read/write/net/run) replaces flat
  string array in Go→Deno JSON-RPC
- I/O bridge: Worker postMessage → parent dispatchRPC → CoreClient
  gRPC → response relayed back to Worker
- Test module proves end-to-end: Worker calls core.storeSet() →
  Go verifies value in store

40 unit tests + 3 integration tests (Tier 1 boot + Tier 2 bidir + Tier 3 Worker).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 00:48:16 +00:00
Claude
af98accc03
feat(coredeno): Tier 2 bidirectional bridge — Go↔Deno module lifecycle
Wire the CoreDeno sidecar into a fully bidirectional bridge:

- Deno→Go (gRPC): Deno connects as CoreService client via polyfilled
  @grpc/grpc-js over Unix socket. Polyfill patches Deno 2.x http2 gaps
  (getDefaultSettings, pre-connected socket handling, remoteSettings).
- Go→Deno (JSON-RPC): Go connects to Deno's newline-delimited JSON-RPC
  server for module lifecycle (LoadModule, UnloadModule, ModuleStatus).
  gRPC server direction avoided due to Deno http2.createServer limitations.
- ProcessStart/ProcessStop: gRPC handlers delegate to process.Service
  with manifest permission gating (run permissions).
- Deno runtime: main.ts boots DenoService server, connects CoreService
  client with retry + health-check round-trip, handles SIGTERM shutdown.

40 unit tests + 2 integration tests (Tier 1 boot + Tier 2 bidirectional).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 22:43:12 +00:00
Claude
2f246ad053
feat(coredeno): wire Tier 1 boot sequence — gRPC listener, manifest loading, sidecar launch
Service.OnStartup now creates sandboxed I/O medium, opens SQLite store,
starts gRPC listener on Unix socket, loads .core/view.yml manifest, and
launches Deno sidecar with CORE_SOCKET env var. Full shutdown in reverse.

New files: listener.go (Unix socket gRPC server), runtime/main.ts (Deno
entry point), integration_test.go (full boot with real Deno).

34 tests pass (33 unit + 1 integration).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:39:49 +00:00
Claude
7d047fbdcc
feat(coredeno): wire Service into framework DI with ServiceRuntime[T]
Service embeds ServiceRuntime[Options] for Core/Opts access.
NewServiceFactory returns factory for core.WithService registration.
Correct Startable/Stoppable signatures with context.Context.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:12:27 +00:00
Claude
e8695b72a6
feat(coredeno): gRPC server with permission-gated I/O fortress
Generated Go code from proto. Server implements CoreService with
FileRead/FileWrite/FileList/FileDelete/StoreGet/StoreSet — every
request checked against the calling module's manifest permissions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:09:40 +00:00
f0268d12bf Merge pull request 'phase4-foundation' (#8) from phase4-foundation into dev
Reviewed-on: #8
2026-02-17 21:03:50 +00:00
Claude
0681fba48e
feat(coredeno): framework service with Startable/Stoppable lifecycle
Service wraps Sidecar for DI registration. OnStartup/OnShutdown hooks
for framework lifecycle integration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
5b737a4933
feat(marketplace): Git-based module index parser and search
Module/Index types, ParseIndex from JSON, Search (fuzzy across code/name/
category), ByCategory filter, Find by code. Foundation for git-based
plugin marketplace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
f065c0a5be
feat(coredeno): permission engine for I/O fortress
CheckPath (prefix-based), CheckNet (exact match), CheckRun (exact match).
Empty allowed list = deny all. Secure by default.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
c490a05733
feat(coredeno): gRPC proto definitions for I/O fortress
CoreService (Go-side: file, store, process) and DenoService (Deno-side:
module lifecycle). Generated Go code pending protoc installation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
93be6c5ed2
feat(coredeno): sidecar Start/Stop/IsRunning lifecycle
Process launch with context cancellation, socket directory auto-creation,
channel-based stop synchronization. Uses sleep as fake Deno in tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
01924059ae
feat(coredeno): sidecar types, permission flags, socket path
Options, Permissions with Deno --allow-* flag generation,
DefaultSocketPath with XDG_RUNTIME_DIR support, Sidecar struct.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
262f0eb5d5
feat(store): group-namespaced key-value store with template rendering
SQLite-backed KV store with get/set/delete/count/deleteGroup/render.
Extracted from dAppServer object store pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
c7102826ba
feat(manifest): auto-discovery loader with signature verification
Load() reads .core/view.yml from any directory via io.Medium,
LoadVerified() adds ed25519 signature check. Uses MockMedium for tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:37 +00:00
Claude
ea63c3acae
feat(manifest): add ed25519 signing and verification
Sign() computes signature over canonical YAML (excluding sign field),
Verify() checks against public key. Tampered manifests are rejected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:36 +00:00
Claude
d2f2f0984c
feat(manifest): add .core/view.yml types and parser
Manifest struct, Permissions, Parse() from YAML, SlotNames() helper.
Foundation for Phase 4 module system.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 21:02:36 +00:00
fc8ebe53e1 refactor: strip 25 extracted packages, slim to foundation (#5)
## Summary
- Remove 25 packages extracted to domain repos (go-crypt, go-ai, go-devops, go-scm, go-netops)
- Make crypt service registration optional in CLI framework
- core/go is now a pure foundation library (~24K LOC, 17 packages)

## Remaining packages
cache, cli, config, framework, help, i18n, io, lab, log, plugin, process, ratelimit, repos, session, webview, workspace, ws

Co-authored-by: Claude <developers@lethean.io>
Reviewed-on: #5
Co-authored-by: Charon <charon@lthn.ai>
Co-committed-by: Charon <charon@lthn.ai>
2026-02-16 15:37:06 +00:00
Claude
ac2e83b88d refactor: add i18n-validate tool, remove bugseti plan docs
Move i18n-validate tool from core/cli internal/tools/ into
pkg/i18n/internal/validate/. Remove bugseti plan docs (now in
core/bugseti repo).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 14:57:04 +00:00
adaa4131f9 refactor: strip to pure package library (#3)
- Fix remaining 187 pkg/ files referencing core/cli → core/go
- Move SDK library code from internal/cmd/sdk/ → pkg/sdk/ (new package)
- Create pkg/rag/helpers.go with convenience functions from internal/cmd/rag/
- Fix pkg/mcp/tools_rag.go to use pkg/rag instead of internal/cmd/rag
- Fix pkg/build/buildcmd/cmd_sdk.go and pkg/release/sdk.go to use pkg/sdk
- Remove all non-library content: main.go, internal/, cmd/, docker/,
  scripts/, tasks/, tools/, .core/, .forgejo/, .woodpecker/, Taskfile.yml
- Run go mod tidy to trim unused dependencies

core/go is now a pure Go package suite (library only).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: Claude <developers@lethean.io>
Reviewed-on: #3
2026-02-16 14:23:45 +00:00
Snider
da5658c250 refactor: update package imports from cli to go
Some checks failed
Security Scan / Go Vulnerability Check (push) Has been cancelled
Security Scan / Secret Detection (push) Has been cancelled
Security Scan / Dependency & Config Scan (push) Has been cancelled
2026-02-16 13:57:29 +00:00
Snider
b3115a12a7 refactor: update package imports from cli to go
Some checks are pending
Security Scan / Go Vulnerability Check (push) Waiting to run
Security Scan / Secret Detection (push) Waiting to run
Security Scan / Dependency & Config Scan (push) Waiting to run
2026-02-16 13:53:54 +00:00
4eb1e02f5e feat/ml-integration (#2)
Some checks are pending
Security Scan / Go Vulnerability Check (push) Waiting to run
Security Scan / Secret Detection (push) Waiting to run
Security Scan / Dependency & Config Scan (push) Waiting to run
Co-authored-by: Charon (snider-linux) <charon@lethean.io>
Co-authored-by: Snider <snider@host.uk.com>
Co-authored-by: Virgil <virgil@lethean.io>
Co-authored-by: Claude <developers@lethean.io>
Reviewed-on: core/cli#2
Co-authored-by: Snider <snider@lethean.io>
Co-committed-by: Snider <snider@lethean.io>
2026-02-16 06:19:09 +00:00
Claude
5e9a9c2790 feat: integrate lab dashboard as core lab serve
Some checks failed
Security Scan / Go Vulnerability Check (push) Has been cancelled
Security Scan / Secret Detection (push) Has been cancelled
Security Scan / Dependency & Config Scan (push) Has been cancelled
Port the standalone lab dashboard (lab.lthn.io) into the core CLI as
pkg/lab/ with collectors, handlers, and HTML templates. The dashboard
monitors machines, Docker containers, Forgejo, HuggingFace models,
training runs, and InfluxDB metrics with SSE live updates.

New command: core lab serve --bind :8080

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 05:53:52 +00:00
Claude
1f3a1bcc47 feat: port 11 LEM data management commands into core ml
Ports all remaining LEM pipeline commands from pkg/lem into core ml,
eliminating the standalone LEM CLI dependency. Each command is split
into reusable business logic (pkg/ml/) and a thin cobra wrapper
(internal/cmd/ml/).

New commands: query, inventory, metrics, ingest, normalize, seed-influx,
consolidate, import-all, approve, publish, coverage.

Adds Path(), Exec(), QueryRowScan() convenience methods to DB type.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 05:53:52 +00:00