30 lines
1.3 KiB
Markdown
30 lines
1.3 KiB
Markdown
|
# Phase 4: Performance Design
|
||
|
|
|
||
|
|
Approved 19 Feb 2026.
|
||
|
|
|
||
|
|
## 1. Benchmark Suite (`rocm_benchmark_test.go`)
|
||
|
|
|
||
|
|
Go benchmark tests (`testing.B`) with `b.ReportMetric()`, build-tagged `//go:build rocm`.
|
||
|
|
|
||
|
|
Three benchmarks per model:
|
||
|
|
|
||
|
|
- **BenchmarkDecode_{Model}** — Generate 128 tokens, report tok/s
|
||
|
|
- **BenchmarkTTFT_{Model}** — Time-to-first-token latency (ns)
|
||
|
|
- **BenchmarkConcurrent_{Model}** — N goroutines generating simultaneously, report aggregate tok/s
|
||
|
|
|
||
|
|
Models: Gemma3-4B-Q4_K_M, Llama-3.1-8B-Q4_K_M, Qwen-2.5-7B-Q4_K_M. Each loads once per benchmark function (load time excluded via `b.StopTimer`/`b.StartTimer`). VRAM logged after load via `GetVRAMInfo()`.
|
||
|
|
|
||
|
|
Run: `go test -tags rocm -bench . -benchtime 3x -timeout 600s`
|
||
|
|
|
||
|
|
## 2. Flash Attention (Manual)
|
||
|
|
|
||
|
|
No code changes. Build a second llama-server without `-DGGML_HIP_ROCWMMA_FATTN=ON`, run the benchmark suite twice with `ROCM_LLAMA_SERVER_PATH` pointing at each binary, record results in FINDINGS.md.
|
||
|
|
|
||
|
|
## 3. Parallel Slots
|
||
|
|
|
||
|
|
Add `ParallelSlots int` to go-inference's `LoadConfig` and `WithParallelSlots(n int) LoadOption`. Pass `--parallel N` to llama-server in `startServer` when > 0. Benchmark concurrent requests with 1 slot vs 4 slots.
|
||
|
|
|
||
|
|
## 4. Testing
|
||
|
|
|
||
|
|
Benchmarks are the tests — they produce the numbers. Existing integration tests remain as correctness checks. All results documented in FINDINGS.md.
|