fix: correct go.mod replace directive, prep contract files for fleet
- go.mod: replace ../core → ../go (actual framework location) - go mod tidy: resolve missing indirect deps - TODO.md: rewrite with phased task queue (0-4) - FINDINGS.md: add environment review, testability analysis Co-Authored-By: Charon <developers@lethean.io>
This commit is contained in:
Claude
parent
4999c8bdc3
commit
034b9da45f
4 changed files with 171 additions and 30 deletions
36
FINDINGS.md
36
FINDINGS.md
|
|
@ -19,3 +19,39 @@ Only chunk.go has tests. The Qdrant and Ollama clients are untested — they dep
|
|||
|
||||
- `go-ai/ai/rag.go` wraps this as `QueryRAGForTask()` facade
|
||||
- `go-ai/mcp/tools_rag.go` exposes RAG as MCP tools
|
||||
|
||||
---
|
||||
|
||||
## 2026-02-19: Environment Review (Charon)
|
||||
|
||||
### go.mod Fix
|
||||
|
||||
Replace directive was `../core` — should be `../go`. Fixed. Tests now pass.
|
||||
|
||||
### Coverage
|
||||
|
||||
```
|
||||
go-rag: 18.4% coverage (only chunk.go tested)
|
||||
```
|
||||
|
||||
### Infrastructure Status
|
||||
|
||||
| Service | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| Qdrant | **Not running** | Need `docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant` |
|
||||
| Ollama | **Not running locally** | M3 has Ollama at 10.69.69.108:11434, but local install preferred for tests |
|
||||
|
||||
### Testability Analysis
|
||||
|
||||
| File | Lines | Testable Without Services | Notes |
|
||||
|------|-------|---------------------------|-------|
|
||||
| chunk.go | 205 | Yes — pure functions | 8 tests exist, good coverage |
|
||||
| query.go | 163 | **Partially** — FormatResults* are pure | Query() needs Qdrant + Ollama |
|
||||
| qdrant.go | 226 | No — all gRPC calls | Need live Qdrant or mock interface |
|
||||
| ollama.go | 120 | **Partially** — EmbedDimension is pure | Embed() needs live Ollama |
|
||||
| ingest.go | 217 | No — orchestrates Qdrant + Ollama | Need mocks or live services |
|
||||
| helpers.go | 89 | **Partially** — QueryDocs/IngestDirectory are convenience wrappers | Same deps as query/ingest |
|
||||
|
||||
### Recommendation
|
||||
|
||||
Phase 1 should focus on pure-function tests (FormatResults*, EmbedDimension, defaults, valueToGo). Phase 2 extracts `Embedder` and `VectorStore` interfaces to enable mocked testing for ingest/query. Phase 3+ needs live services.
|
||||
|
|
|
|||
85
TODO.md
85
TODO.md
|
|
@ -1,41 +1,72 @@
|
|||
# TODO.md — go-rag Task Queue
|
||||
|
||||
## Phase 1: Post-Split
|
||||
|
||||
- [ ] **Verify tests pass standalone** — Run `go test ./...`. Confirm chunk_test.go passes.
|
||||
- [ ] **Add missing tests** — Only chunk.go has tests. Need: query, ingest, qdrant client, ollama client.
|
||||
- [ ] **Benchmark chunking** — No perf baselines. Add BenchmarkChunk for various document sizes.
|
||||
|
||||
## Phase 2: Enhancements
|
||||
|
||||
- [ ] **Embedding model selection** — Currently hardcoded to Ollama. Add backend interface for alternative embedding providers.
|
||||
- [ ] **Collection management** — Add list/delete collection operations for Qdrant.
|
||||
- [ ] **Hybrid search** — Combine vector similarity with keyword matching for better recall.
|
||||
- [ ] **Chunk overlap** — Current chunking may lose context at boundaries. Add configurable overlap.
|
||||
Dispatched from core/go orchestration. Pick up tasks in phase order.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0: Environment Setup
|
||||
|
||||
- [ ] **Fix go.mod replace directive** — Was `../core`, corrected to `../go`. Commit and push. (Charon, 19 Feb 2026)
|
||||
- [ ] **Run Qdrant locally** — Docker: `docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant`. Test with `curl http://localhost:6334/healthz`.
|
||||
- [ ] **Install Ollama** — `curl -fsSL https://ollama.com/install.sh | sh`. Pull embedding model: `ollama pull nomic-embed-text`.
|
||||
- [ ] **Verify both services** — `go test -v -run TestQdrant ./...` and `go test -v -run TestOllama ./...` (write these tests in Phase 1).
|
||||
|
||||
## Phase 1: Unit Tests (currently 18.4% coverage)
|
||||
|
||||
Only `chunk.go` has tests. Everything else is untested.
|
||||
|
||||
### Testable Without External Services
|
||||
|
||||
- [ ] **FormatResults tests** — FormatResultsText, FormatResultsContext, FormatResultsJSON with known QueryResult inputs. Pure string formatting, no deps.
|
||||
- [ ] **DefaultConfig tests** — Verify DefaultQdrantConfig, DefaultOllamaConfig, DefaultQueryConfig, DefaultChunkConfig, DefaultIngestConfig return expected values.
|
||||
- [ ] **EmbedDimension tests** — OllamaClient.EmbedDimension() for each model name (nomic-embed-text=768, mxbai-embed-large=1024, all-minilm=384, unknown=768).
|
||||
- [ ] **Point/SearchResult types** — Round-trip tests for Point struct and pointIDToString helper.
|
||||
- [ ] **valueToGo tests** — Qdrant value conversion for string, int, double, bool, list, struct, nil.
|
||||
- [ ] **Additional chunk tests** — Empty input, only headers no content, unicode/emoji, very long paragraph.
|
||||
|
||||
### Require External Services (use build tag `//go:build rag`)
|
||||
|
||||
- [ ] **Qdrant client tests** — Create collection, upsert, search, delete. Skip if Qdrant unavailable.
|
||||
- [ ] **Ollama client tests** — Embed single text, embed batch, verify model. Skip if Ollama unavailable.
|
||||
- [ ] **Query integration test** — Ingest a test doc, query it, verify results.
|
||||
|
||||
## Phase 2: Test Infrastructure
|
||||
|
||||
- [ ] **Interface extraction** — Extract `Embedder` interface (Embed, EmbedBatch, EmbedDimension) and `VectorStore` interface (Search, Upsert, etc.) so Qdrant and Ollama can be mocked in unit tests.
|
||||
- [ ] **Mock embedder** — Returns deterministic vectors for testing query/ingest without Ollama.
|
||||
- [ ] **Mock vector store** — In-memory implementation for testing ingest/query without Qdrant.
|
||||
- [ ] **Re-test with mocks** — Rewrite ingest + query tests using mock interfaces for fast, reliable CI.
|
||||
|
||||
## Phase 3: Enhancements
|
||||
|
||||
- [ ] **Chunk overlap** — Configurable overlap already in ChunkConfig but needs better boundary handling.
|
||||
- [ ] **Collection management** — List/delete collection operations (already in qdrant.go but untested).
|
||||
- [ ] **Hybrid search** — Combine vector similarity with keyword matching.
|
||||
- [ ] **Embedding model selection** — Backend interface for alternative providers (not just Ollama).
|
||||
|
||||
## Phase 4: GPU Embeddings
|
||||
|
||||
- [ ] **ROCm Ollama** — Test Ollama with ROCm on the RX 7800 XT. Measure embedding throughput.
|
||||
- [ ] **Batch optimisation** — EmbedBatch currently calls Embed sequentially. Ollama may support batch API.
|
||||
- [ ] **Benchmarks** — Chunking speed, embedding throughput, search latency.
|
||||
|
||||
---
|
||||
|
||||
## Linux Homelab Assignment (Virgil, 19 Feb 2026)
|
||||
## Known Issues
|
||||
|
||||
This package is assigned to the Linux homelab agent. The homelab can run Qdrant and Ollama natively, giving the dedicated Claude real vector DB and embedding infrastructure to test against.
|
||||
1. **go.mod had wrong replace path** — `../core` should be `../go`. Fixed by Charon.
|
||||
2. **Qdrant and Ollama not running on snider-linux** — Need docker setup for Qdrant, native install for Ollama.
|
||||
3. **No mocks/interfaces** — All external calls go directly to Qdrant/Ollama clients. Unit tests require live services until interfaces are extracted.
|
||||
4. **`log.E` returns error** — `forge.lthn.ai/core/go/pkg/log.E` wraps errors with component context. This is the framework's logging pattern.
|
||||
|
||||
### Linux-Specific Tasks
|
||||
## Platform
|
||||
|
||||
- [ ] **Run Qdrant locally** — Docker or native binary on the homelab. Test against real Qdrant instance, not mocks.
|
||||
- [ ] **Run Ollama locally** — Install Ollama on Linux. Test embedding generation with real models (nomic-embed-text, etc.).
|
||||
- [ ] **Full pipeline integration test** — Ingest → chunk → embed → store → query end-to-end with real Qdrant + Ollama.
|
||||
- [ ] **AMD GPU embeddings** — Ollama supports ROCm. Test embedding generation on the RX 7800 XT for faster processing.
|
||||
|
||||
### Platform
|
||||
|
||||
- **OS**: Ubuntu 24.04 (linux/amd64)
|
||||
- **GPU**: AMD RX 7800 XT (ROCm, for Ollama GPU acceleration)
|
||||
- **Co-located with**: go-rocm (AMD GPU inference), go-p2p (networking)
|
||||
- **OS**: Ubuntu (linux/amd64) — snider-linux
|
||||
- **Co-located with**: go-rocm, go-p2p
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Virgil in core/go writes tasks here after research
|
||||
2. This repo's session picks up tasks in phase order
|
||||
1. Charon dispatches tasks here after review
|
||||
2. Pick up tasks in phase order
|
||||
3. Mark `[x]` when done, note commit hash
|
||||
4. New discoveries → add notes, flag in FINDINGS.md
|
||||
|
|
|
|||
16
go.mod
16
go.mod
|
|
@ -10,11 +10,21 @@ require (
|
|||
)
|
||||
|
||||
require (
|
||||
github.com/bahlo/generic-list-go v0.2.0 // indirect
|
||||
github.com/buger/jsonparser v1.1.1 // indirect
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/mailru/easyjson v0.9.1 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251111163417-95abcf5c77ba // indirect
|
||||
google.golang.org/grpc v1.78.0 // indirect
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
|
||||
golang.org/x/crypto v0.48.0 // indirect
|
||||
golang.org/x/net v0.50.0 // indirect
|
||||
golang.org/x/sys v0.41.0 // indirect
|
||||
golang.org/x/text v0.34.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
|
||||
google.golang.org/grpc v1.79.1 // indirect
|
||||
google.golang.org/protobuf v1.36.11 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
replace forge.lthn.ai/core/go => ../core
|
||||
replace forge.lthn.ai/core/go => ../go
|
||||
|
|
|
|||
64
go.sum
Normal file
64
go.sum
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
|
||||
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
|
||||
github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
|
||||
github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/mailru/easyjson v0.9.1 h1:LbtsOm5WAswyWbvTEOqhypdPeZzHavpZx96/n553mR8=
|
||||
github.com/mailru/easyjson v0.9.1/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
|
||||
github.com/ollama/ollama v0.16.1 h1:DIxnLdS0om3hb7HheJqj6+ZnPCCMWmy/vyUxiQgRYoI=
|
||||
github.com/ollama/ollama v0.16.1/go.mod h1:FEk95NbAJJZk+t7cLh+bPGTul72j1O3PLLlYNV3FVZ0=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/qdrant/go-client v1.16.2 h1:UUMJJfvXTByhwhH1DwWdbkhZ2cTdvSqVkXSIfBrVWSg=
|
||||
github.com/qdrant/go-client v1.16.2/go.mod h1:I+EL3h4HRoRTeHtbfOd/4kDXwCukZfkd41j/9wryGkw=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc=
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
|
||||
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
|
||||
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
|
||||
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
|
||||
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
|
||||
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
|
||||
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
|
||||
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
|
||||
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
|
||||
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
|
||||
golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
|
||||
golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
|
||||
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
|
||||
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
|
||||
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 h1:gRkg/vSppuSQoDjxyiGfN4Upv/h/DQmIR10ZU8dh4Ww=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217/go.mod h1:7i2o+ce6H/6BluujYR+kqX3GKH+dChPTQU19wjRPiGk=
|
||||
google.golang.org/grpc v1.79.1 h1:zGhSi45ODB9/p3VAawt9a+O/MULLl9dpizzNNpq7flY=
|
||||
google.golang.org/grpc v1.79.1/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
|
||||
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
|
||||
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
Loading…
Add table
Reference in a new issue