Charon/LEM
1
0
Fork 0
forked from lthn/LEM
Code Pull requests Activity
18 commits 2 branches 0 tags 29 MiB
Commit graph

7 commits

Author SHA1 Message Date
Claude
9fac5749c2
feat: add scoring agent + 23 capability probes (replaces scoring_agent.py) 
Go scoring daemon that polls M3 for unscored LoRA checkpoints,
converts MLX→PEFT, runs 23 binary capability probes via OpenAI-
compatible API, and pushes results to InfluxDB. Zero Python deps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 17:22:40 +00:00
Claude
91ee389377
feat: convert all pipeline.py commands to Go 
Complete conversion of pipeline.py into Go `lem` CLI:
- import-all: bulk import all LEM data into DuckDB from M3
- consolidate: pull worker JSONLs, merge, deduplicate
- normalize: seeds → deduplicated expansion_prompts table
- approve: filter scored expansions → training JSONL
- tier-score: heuristic/judge tiered expansion scoring
- expand-status: expansion pipeline progress from DuckDB
- inventory: DuckDB table counts and summary
- coverage: seed coverage gap analysis
- seed-influx: bootstrap InfluxDB from DuckDB golden_gen
- query: ad-hoc SQL against DuckDB

22 commands total, 49 Go files. Replaces entire pipeline.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 17:12:03 +00:00
Claude
4eaf1bfb39
feat: add parquet, publish, metrics, convert commands 
- `lem parquet` — export JSONL training splits to Parquet (parquet-go)
- `lem publish` — push Parquet files to HuggingFace dataset repo
- `lem metrics` — push DuckDB golden set stats to InfluxDB
- `lem convert` — MLX LoRA adapter → HuggingFace PEFT format
  (pure Go safetensors read/write/transpose, no PyTorch needed)

Dependencies added: parquet-go, go-huggingface, go-rocm, go-pytorch, gotch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 17:05:08 +00:00
Claude
0afa5e9147
feat: add lem ingest command + go-huggingface dependency 
Ingests benchmark data (content scores, capability scores, training
curves) from JSONL files and mlx_lm logs into InfluxDB. Batched
writes, iteration extraction from checkpoint labels.

Also adds github.com/hupe1980/go-huggingface for future HF sync.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 16:55:17 +00:00
Claude
a18fd1c44e
refactor: remove Vi identity from calm conversations 
Vi identity is a separate training concern. Seed conversations now
contain only philosophical/mindfulness content for the R300 calm phase.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 16:48:23 +00:00
Claude
c4fb775298
feat: add lem conv command for conversational training data 
Ports conversational_training.py to Go with InfluxDB reporting.
24 built-in seed conversations (Vi identity, philosophy, mindfulness).
Supports extra JSONL files and golden set conversion to chat format.

Also fixes InfluxDB client to accept 204 No Content on writes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 16:42:46 +00:00
Claude
70dd18c065
refactor: move Go library to pkg/lem, thin main.go 
All scoring/influx/export/expand logic moves to pkg/lem as an
importable package. main.go is now a thin CLI dispatcher.

This lets new commands import the shared library directly —
ready for converting Python scripts to Go subcommands.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-15 16:30:09 +00:00