LEM/pkg/lem
Snider 1b742bf92c feat: native Metal distillation command + .core/ai config
Add `lem distill` — full Go pipeline for self-distillation using
go-mlx (native Metal inference) and go-i18n/reversal (v3 grammar
scoring). Replaces the Python distill.py bridge entirely.

New files:
- .core/ai/ai.yaml: global defaults (scorer, generation, distill)
- .core/ai/models/gemma3/{27b,1b}.yaml: model configs with paths,
  kernel, lessons, baselines
- .core/ai/probes.yaml: probe sets grouped by training phase
- pkg/lem/config.go: YAML config loaders for .core/ai/
- pkg/lem/grammar.go: in-process grammar scoring (ComputeGrammarScore,
  ComputeDelta, ScoreResponse) extracted from cmd/scorer
- pkg/lem/distill.go: RunDistill command — best-of-N generation,
  grammar quality gate, training JSONL output
- pkg/lem/backend_metal.go: blank import for go-mlx Metal registration

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-21 23:42:55 +00:00
..
agent.go feat: add scoring agent + 23 capability probes (replaces scoring_agent.py) 2026-02-15 17:22:40 +00:00
agent_test.go feat: add scoring agent + 23 capability probes (replaces scoring_agent.py) 2026-02-15 17:22:40 +00:00
approve.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
backend_metal.go feat: native Metal distillation command + .core/ai config 2026-02-21 23:42:55 +00:00
client.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
client_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
compare.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
compare_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
config.go feat: native Metal distillation command + .core/ai config 2026-02-21 23:42:55 +00:00
consolidate.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
conv.go feat: add lem conv command for conversational training data 2026-02-15 16:42:46 +00:00
conv_test.go refactor: remove Vi identity from calm conversations 2026-02-15 16:48:23 +00:00
conversations.go refactor: remove Vi identity from calm conversations 2026-02-15 16:48:23 +00:00
convert.go feat: add parquet, publish, metrics, convert commands 2026-02-15 17:05:08 +00:00
convert_test.go feat: add parquet, publish, metrics, convert commands 2026-02-15 17:05:08 +00:00
coverage.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
db.go feat: scaffold LEM Desktop app (Wails v3 system tray + Docker stack) 2026-02-15 17:43:19 +00:00
db_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
distill.go feat: native Metal distillation command + .core/ai config 2026-02-21 23:42:55 +00:00
engine.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
engine_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
exact.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
exact_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
expand.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
expand_status.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
expand_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
export.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
export_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
grammar.go feat: native Metal distillation command + .core/ai config 2026-02-21 23:42:55 +00:00
heuristic.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
heuristic_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
import.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
influx.go feat: add lem conv command for conversational training data 2026-02-15 16:42:46 +00:00
influx_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
ingest.go feat: add lem ingest command + go-huggingface dependency 2026-02-15 16:55:17 +00:00
ingest_test.go feat: add lem ingest command + go-huggingface dependency 2026-02-15 16:55:17 +00:00
inventory.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
io.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
io_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
judge.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
judge_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
metrics.go feat: add parquet, publish, metrics, convert commands 2026-02-15 17:05:08 +00:00
normalize.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
parquet.go feat: add parquet, publish, metrics, convert commands 2026-02-15 17:05:08 +00:00
parquet_test.go feat: add parquet, publish, metrics, convert commands 2026-02-15 17:05:08 +00:00
probe.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
probe_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
probes.go feat: add scoring agent + 23 capability probes (replaces scoring_agent.py) 2026-02-15 17:22:40 +00:00
probes_test.go feat: add scoring agent + 23 capability probes (replaces scoring_agent.py) 2026-02-15 17:22:40 +00:00
prompts.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
publish.go feat: add parquet, publish, metrics, convert commands 2026-02-15 17:05:08 +00:00
query.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
seed_influx.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
status.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
status_test.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
tier_score.go feat: convert all pipeline.py commands to Go 2026-02-15 17:12:03 +00:00
types.go refactor: move Go library to pkg/lem, thin main.go 2026-02-15 16:30:09 +00:00
worker.go feat: add lem worker command for distributed inference network 2026-02-15 18:10:59 +00:00
worker_test.go feat: add lem worker command for distributed inference network 2026-02-15 18:10:59 +00:00