Commit graph

2 commits

Author SHA1 Message Date
Claude
fcd1758b7d
feat(ml): add format converters, data pipeline, and scoring agent
Some checks are pending
Security Scan / Go Vulnerability Check (push) Waiting to run
Security Scan / Secret Detection (push) Waiting to run
Security Scan / Dependency & Config Scan (push) Waiting to run
Port remaining lem-repo components into pkg/ml/:
- convert.go: safetensors reader/writer, MLX→PEFT converter
- gguf.go: GGUF v3 writer, MLX→GGUF LoRA converter
- export.go: training data JSONL export with split/filter
- parquet.go: Parquet export with snappy compression
- db.go: DuckDB wrapper for golden set and expansion prompts
- influx.go: InfluxDB v3 client for metrics/status
- ollama.go: Ollama model management (create/delete with adapters)
- status.go: training and generation status display
- expand.go: expansion generation pipeline (Backend interface)
- agent.go: scoring agent with probe running and InfluxDB push
- worker.go: distributed worker for LEM API task processing

Adds parquet-go and go-duckdb dependencies.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 23:46:24 +00:00
Claude
6f52e4e3ae
feat(ml): add ML inference and scoring engine from lem-repo
Port LEM scoring pipeline into CoreGo pkg/ml/:
- Backend interface abstracting HTTP, llama-server, and future backends
- HTTPBackend for OpenAI-compatible APIs with retry logic
- LlamaBackend managing llama-server via pkg/process
- Scoring engine with heuristic, semantic, content, and exact suites
- Judge for LLM-based multi-dimensional scoring
- 23 capability probes (math, logic, reasoning, code)
- 6 sovereignty content probes
- GGUF/PEFT format helpers, safetensors reader
- 37 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 23:34:54 +00:00