core/go-ai
8
0
Fork 0
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity Actions
go-ai/mlx/tokenizer
History
Exact
Snider 98749c66f2 fix(mlx): add DecodeToken for correct streaming word boundaries 
The Decode method strips the SentencePiece leading space from every
token, which loses word boundaries during streaming. DecodeToken
preserves the space (it represents the word boundary) and only the
first token of each generation has its leading space stripped.

Fixes Gemma3 space prefix appearing in chat UI output.

Co-Authored-By: Virgil <virgil@lethean.io>
2026-02-17 19:18:27 +00:00
..
tokenizer.go fix(mlx): add DecodeToken for correct streaming word boundaries 2026-02-17 19:18:27 +00:00