go-mlx/tokenizer
Snider 28b5c5bb47 test(sub-packages): add 33 tests for cache, sample, tokenizer
- cache_test.go (10): KVCache and RotatingKVCache — creation, single/multi
  update, bounded sliding window, reset, state access
- sample_test.go (8): greedy, temperature (high/low), topK (single/multi),
  chain composition, TopP/MinP stub pass-through
- tokenizer_test.go (15): Load (valid/missing/invalid), BOS/EOS detection,
  encode produces tokens, decode skips specials, decode regular tokens,
  DecodeToken (regular/special/space/unknown), FormatGemmaPrompt,
  GPT-2 byte maps (round-trip, printable ASCII, control chars)

Total test count: 29 → 148 across 10 test files.

Co-Authored-By: Virgil <virgil@lethean.io>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 18:40:49 +00:00
..
tokenizer.go feat: extract go-mlx from go-ai as standalone Metal inference package 2026-02-19 17:57:37 +00:00
tokenizer_test.go test(sub-packages): add 33 tests for cache, sample, tokenizer 2026-02-19 18:40:49 +00:00