Shared inference interfaces (TextModel, Backend, Token). Zero deps, compiles everywhere.
- Go 100%
254b391f31
Lifts the package-first request scheduler from go-mlx into a self-contained driver-neutral package. Symbols rename per the folder-taxonomy rule: ScheduledModel → scheduler.Model SchedulerConfig → scheduler.Config NewScheduledModel → scheduler.New scheduledJob → job (private) emitSchedulerProbe → (Model).emitProbe (private method) scheduledGenerateOptions → generateOptions (private) cloneSchedulerLabels → cloneLabels (private) scheduler.Model wraps an inference.TextModel with bounded queueing, cancellation, streaming backpressure, and ProbeEventScheduler probe emission. Worker pool sized by Config.MaxConcurrent; queue bounded by MaxQueue; per-request stream buffer set by StreamBuffer. Coverage: queue + latency probe, full-queue rejection, cancellation, Generate/Chat/Classify/BatchGenerate delegation, nil-scheduler defence paths, fallback cancel via inference.CancellableModel, Err propagation, generateOptions sampler conversion, cloneLabels defensive copy, millis helpers. Six tests, ten examples, all green. Co-Authored-By: Virgil <virgil@lethean.io> |
||
|---|---|---|
| .core | ||
| .forgejo/workflows | ||
| docs | ||
| external | ||
| go | ||
| .editorconfig | ||
| .gitignore | ||
| .gitmodules | ||
| .golangci.yml | ||
| .woodpecker.yml | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| CONTRIBUTING.md | ||
| go.work | ||
| LICENCE | ||
| README.md | ||
| sonar-project.properties | ||
go-inference
Shared interface contract for text generation backends in the Core Go ecosystem. Defines TextModel, Backend, Token, Message, and associated configuration types that GPU-specific backends implement and consumers depend on. Zero external dependencies — stdlib only — and compiles on all platforms regardless of GPU availability. The backend registry supports automatic selection (Metal preferred on macOS, ROCm on Linux) and explicit pinning.
Module: dappco.re/go/inference
Licence: EUPL-1.2
Language: Go 1.25
Quick Start
import (
"dappco.re/go/inference"
_ "forge.lthn.ai/core/go-mlx" // registers "metal" backend on darwin/arm64
)
model, err := inference.LoadModel("/path/to/safetensors/model/")
defer model.Close()
for tok := range model.Generate(ctx, "Hello", inference.WithMaxTokens(256)) {
fmt.Print(tok.Text)
}
Documentation
- Architecture — interfaces, registry, options, stability contract, ecosystem position
- Development Guide — prerequisites, build, test patterns, coding standards
- Project History — completed phases, commit log, known limitations
Build & Test
go test ./...
go vet ./...
go build ./...
Licence
European Union Public Licence 1.2 — see LICENCE for details.