Add !, ;, and , to splitTrailingPunct and matchPunctuation. Previously only ..., ?, and : were recognised. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| imprint.go | ||
| imprint_test.go | ||
| multiplier.go | ||
| multiplier_test.go | ||
| roundtrip_test.go | ||
| tokeniser.go | ||
| tokeniser_test.go | ||