core/go-api
8
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: add architecture, development, and history documentation

Browse source 
Standard docs trio matching the pattern used across all Go ecosystem repos.
Covers Engine architecture, 21 With*() options, RouteGroup interfaces,
OpenAPI generation pipeline, and three-phase development history (176 tests).

Co-Authored-By: Virgil <virgil@lethean.io>
This commit is contained in:
Snider 2026-02-21 01:57:44 +00:00
parent 1910aec1fe
commit 0d3479839d
3 changed files with 1137 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

498
docs/architecture.md Normal file
View file

@ -0,0 +1,498 @@
<!-- SPDX-License-Identifier: EUPL-1.2 -->
# go-api Architecture
**Module**: `forge.lthn.ai/core/go-api`
**Language**: Go 1.25
**Licence**: EUPL-1.2
**LOC**: ~2.1 K non-test, 176 tests passing
---
## 1. Overview
`go-api` is the Gin-based REST framework for the Lethean Go ecosystem. Subsystems across the
ecosystem plug into a central `Engine` via the `RouteGroup` interface, and go-api handles the HTTP
plumbing: middleware composition, response envelopes, WebSocket and SSE integration, GraphQL,
Authentik identity, OpenAPI spec generation, and SDK codegen.
### Position in the Lethean Stack
```
AI Clients / Browsers / SDK consumers
| HTTP / WebSocket / SSE
v
[ go-api Engine ] ← this module
| | |
| | └─ OpenAPI spec → SDKGenerator → openapi-generator-cli
| └─ ToolBridge ──→ go-ai / go-ml / go-rag route groups
└─ RouteGroups ─────────→ any package that implements RouteGroup
Core CLI or standalone main() bootstraps and wires everything
```
go-api is a library. It contains no `main` package. Callers construct an `Engine`, register route
groups, and call `Serve()`.
---
## 2. Engine
### 2.1 The Engine Struct
`Engine` is the central container. It owns the listen address, the ordered list of registered route
groups, the middleware chain, and all optional integrations:
```go
type Engine struct {
addr string
groups []RouteGroup
middlewares []gin.HandlerFunc
wsHandler http.Handler
sseBroker *SSEBroker
swaggerEnabled bool
swaggerTitle string
swaggerDesc string
swaggerVersion string
pprofEnabled bool
expvarEnabled bool
graphql *graphqlConfig
}
```
### 2.2 Construction
`New()` applies functional options and returns a configured engine. The default listen address is
`:8080`. No middleware is added automatically beyond Gin's built-in panic recovery; every feature
requires an explicit `With*()` option.
```go
engine, err := api.New(
api.WithAddr(":9000"),
api.WithBearerAuth("secret"),
api.WithCORS("*"),
api.WithRequestID(),
api.WithSlog(nil),
api.WithSwagger("My API", "Description", "1.0.0"),
)
engine.Register(myRoutes)
engine.Serve(ctx)
```
### 2.3 Build Sequence
`Engine.build()` (called internally by `Handler()` and `Serve()`) assembles a fresh `*gin.Engine`
each time. The order is fixed:
1. `gin.Recovery()` — panic recovery (always first).
2. User middleware in registration order — all `With*()` options that append to `e.middlewares`.
3. Built-in `GET /health` endpoint — always present.
4. Route groups — each mounted at its `BasePath()`.
5. WebSocket handler at `GET /ws` — when `WithWSHandler()` was called.
6. SSE broker at `GET /events` — when `WithSSE()` was called.
7. GraphQL endpoint — when `WithGraphQL()` was called.
8. Swagger UI at `GET /swagger/*any` — when `WithSwagger()` was called.
9. pprof endpoints at `GET /debug/pprof/*` — when `WithPprof()` was called.
10. expvar endpoint at `GET /debug/vars` — when `WithExpvar()` was called.
### 2.4 Graceful Shutdown
`Serve()` starts an `http.Server` in a goroutine and blocks on `ctx.Done()`. When the context is
cancelled, a 10-second `Shutdown` deadline is applied. In-flight requests complete or time out
before the server exits.
---
## 3. RouteGroup / DescribableGroup Interfaces
Three interfaces form the extension point model:
```go
// RouteGroup is the minimum interface. All subsystems implement this.
type RouteGroup interface {
Name() string
BasePath() string
RegisterRoutes(rg *gin.RouterGroup)
}
// StreamGroup is optionally implemented by groups that publish WebSocket channels.
type StreamGroup interface {
Channels() []string
}
// DescribableGroup extends RouteGroup with OpenAPI metadata.
// Groups implementing this will appear in the generated spec.
type DescribableGroup interface {
RouteGroup
Describe() []RouteDescription
}
```
`RouteDescription` carries the HTTP method, path, summary, description, tags, and JSON Schema
maps for the request body and response data. The `SpecBuilder` consumes these to generate the
OpenAPI 3.1 paths object.
`Engine.Channels()` iterates all registered groups and collects channel names from those that
implement `StreamGroup`. This list is used when initialising a WebSocket hub.
---
## 4. Middleware Stack — 21 With*() Options
All middleware options append to `Engine.middlewares` in the order they are passed to `New()`.
They are applied after `gin.Recovery()` but before any route handler.
| Option | Purpose | Key detail |
|--------|---------|-----------|
| `WithAddr(addr)` | Listen address | Default `:8080` |
| `WithBearerAuth(token)` | Static bearer token auth | Skips `/health` and `/swagger` |
| `WithRequestID()` | X-Request-ID propagation | Preserves client-supplied IDs |
| `WithCORS(origins...)` | CORS policy | `"*"` enables `AllowAllOrigins`; 12-hour `MaxAge` |
| `WithMiddleware(mw...)` | Arbitrary Gin middleware | Escape hatch |
| `WithStatic(prefix, root)` | Static file serving | Directory listing disabled |
| `WithWSHandler(h)` | WebSocket at `/ws` | Wraps any `http.Handler` |
| `WithAuthentik(cfg)` | Authentik forward-auth + OIDC | Permissive; sets context, never rejects |
| `WithSwagger(title, desc, ver)` | Swagger UI at `/swagger/` | Runtime spec via `SpecBuilder` |
| `WithPprof()` | Go profiling at `/debug/pprof/` | WARNING: do not expose in production |
| `WithExpvar()` | Runtime metrics at `/debug/vars` | WARNING: do not expose in production |
| `WithSecure()` | Security headers | HSTS 1yr, X-Frame-Options DENY, nosniff |
| `WithGzip(level...)` | Gzip response compression | Default compression if level omitted |
| `WithBrotli(level...)` | Brotli response compression | Default compression if level omitted |
| `WithSlog(logger)` | Structured request logging | Falls back to `slog.Default()` if nil |
| `WithTimeout(d)` | Per-request deadline | 504 error envelope on timeout |
| `WithCache(ttl)` | In-memory GET response caching | `X-Cache: HIT` on cache hits |
| `WithSessions(name, secret)` | Cookie-backed server sessions | gin-contrib/sessions |
| `WithAuthz(enforcer)` | Casbin policy-based authorisation | Subject from HTTP Basic Auth |
| `WithHTTPSign(secrets, opts...)` | HTTP Signatures verification | draft-cavage-http-signatures |
| `WithSSE(broker)` | Server-Sent Events at `/events` | `?channel=` filtering |
| `WithLocation()` | Reverse proxy header detection | X-Forwarded-Proto / X-Forwarded-Host |
| `WithI18n(cfg...)` | Accept-Language locale detection | BCP 47 matching via x/text/language |
| `WithTracing(name, opts...)` | OpenTelemetry distributed tracing | otelgin + W3C traceparent |
| `WithGraphQL(schema, opts...)` | GraphQL endpoint | gqlgen; optional playground UI |
---
## 5. Response[T] Envelope
All API responses share a single generic envelope:
```go
type Response[T any] struct {
Success bool `json:"success"`
Data T `json:"data,omitempty"`
Error *Error `json:"error,omitempty"`
Meta *Meta `json:"meta,omitempty"`
}
```
Constructor helpers:
| Helper | Usage |
|--------|-------|
| `OK(data)` | Successful response with typed data |
| `Fail(code, message)` | Error response with code and message |
| `FailWithDetails(code, message, details)` | Error with additional detail payload |
| `Paginated(data, page, perPage, total)` | Successful response with pagination meta |
`Meta` carries `request_id`, `duration`, `page`, `per_page`, and `total`. `Error` carries `code`,
`message`, and an optional `details` field for structured validation errors.
---
## 6. Authentik Integration
The `WithAuthentik()` option installs a permissive identity middleware that runs on every
non-public request. It has two extraction paths:
**Path 1 — Forward-auth headers (TrustedProxy: true):**
When Traefik or another reverse proxy is configured with Authentik forward-auth, it injects
`X-authentik-username`, `X-authentik-email`, `X-authentik-name`, `X-authentik-uid`,
`X-authentik-groups` (pipe-separated), `X-authentik-entitlements`, and `X-authentik-jwt` headers.
The middleware reads these and populates an `AuthentikUser` in the Gin context.
**Path 2 — OIDC JWT validation:**
For direct API clients that present a `Bearer` token, the middleware validates the JWT against
the configured OIDC issuer. Providers are cached by issuer URL to avoid repeated discovery requests.
In both paths, if extraction fails the request continues unauthenticated (fail-open). Handlers
check identity with:
```go
user := api.GetUser(c) // returns nil when unauthenticated
```
For protected routes, apply guards:
```go
rg.GET("/private", api.RequireAuth(), handler) // 401 if no user
rg.GET("/admin", api.RequireGroup("admins"), handler) // 403 if wrong group
```
`AuthentikConfig.PublicPaths` lists additional paths exempt from header extraction.
`/health` and paths starting with `/swagger` are always exempt.
---
## 7. WebSocket and SSE
### WebSocket
`WithWSHandler(h)` mounts any `http.Handler` at `GET /ws`. The intended pairing is a `go-ws` hub:
```go
hub := ws.NewHub()
go hub.Run(ctx)
engine, _ := api.New(api.WithWSHandler(hub.Handler()))
```
Groups implementing `StreamGroup` declare channel names, which `Engine.Channels()` aggregates.
### Server-Sent Events
`SSEBroker` manages persistent SSE connections. Clients connect to `GET /events` and optionally
subscribe to a named channel via `?channel=<name>`. Clients with no channel parameter receive
events on all channels.
```go
broker := api.NewSSEBroker()
engine, _ := api.New(api.WithSSE(broker))
// In a handler or background goroutine:
broker.Publish("deployments", "deploy.started", payload)
```
Each client has a 64-event buffer. Overflow events are dropped without blocking the publisher.
`SSEBroker.Drain()` signals all clients to disconnect on graceful shutdown.
---
## 8. GraphQL
`WithGraphQL()` mounts a gqlgen `ExecutableSchema` at `/graphql` (or a custom path via
`WithGraphQLPath()`). An optional `WithPlayground()` adds the interactive playground at
`{path}/playground`.
```go
engine, _ := api.New(
api.WithGraphQL(schema,
api.WithPlayground(),
api.WithGraphQLPath("/gql"),
),
)
```
The endpoint accepts all HTTP methods (POST for queries and mutations, GET for playground
redirects and introspection).
---
## 9. OpenAPI Spec Generation
### SpecBuilder
`SpecBuilder` generates an OpenAPI 3.1 JSON document from registered route groups:
```go
builder := &api.SpecBuilder{
Title: "My API",
Description: "...",
Version: "1.0.0",
}
data, _ := builder.Build(engine.Groups())
```
The built document includes:
- The `GET /health` endpoint under the `system` tag.
- One path entry per `RouteDescription` returned by `DescribableGroup.Describe()`.
- `#/components/schemas/Error` and `#/components/schemas/Meta` shared schemas.
- All response bodies wrapped in the `Response[T]` envelope schema.
- Tags derived from every registered group's `Name()`.
Groups that implement `RouteGroup` but not `DescribableGroup` contribute a tag but no paths.
### Export
Two convenience functions write the spec to an `io.Writer` or directly to a file:
```go
// Write JSON or YAML to a writer:
api.ExportSpec(os.Stdout, "yaml", builder, engine.Groups())
// Write to a file (parent dirs created automatically):
api.ExportSpecToFile("./api/openapi.yaml", "yaml", builder, engine.Groups())
```
### Swagger UI
When `WithSwagger()` is active, the spec is built lazily on first access by a `swaggerSpec`
wrapper and registered in the global `swag` registry with a unique sequence-based instance name.
Multiple `Engine` instances in the same process (common in tests) do not collide in the registry.
---
## 10. ToolBridge
`ToolBridge` converts tool descriptors — as produced by go-ai, go-ml, and similar packages — into
`POST /{tool_name}` REST endpoints, and implements `DescribableGroup` so those endpoints appear in
the generated OpenAPI spec.
```go
bridge := api.NewToolBridge("/v1/tools")
bridge.Add(api.ToolDescriptor{
Name: "file_read",
Description: "Read a file",
Group: "files",
InputSchema: map[string]any{"type": "object", "properties": ...},
}, fileReadHandler)
engine.Register(bridge)
```
Each registered tool becomes a `POST /v1/tools/{tool_name}` endpoint. `ToolBridge.Tools()` returns
the full list of descriptors, making it easy to enumerate tools from the outside.
---
## 11. SDK Codegen
`SDKGenerator` wraps `openapi-generator-cli` to generate client SDKs from an exported spec:
```go
gen := &api.SDKGenerator{
SpecPath: "./api/openapi.yaml",
OutputDir: "./sdk",
PackageName: "myapi",
}
if gen.Available() {
_ = gen.Generate(ctx, "typescript-fetch")
_ = gen.Generate(ctx, "python")
}
```
Supported target languages: `go`, `typescript-fetch`, `typescript-axios`, `python`, `java`,
`csharp`, `ruby`, `swift`, `kotlin`, `rust`, `php`.
`SupportedLanguages()` returns the full list in sorted order. `SDKGenerator.Available()` reports
whether `openapi-generator-cli` is on `PATH` without running it.
---
## 12. In-Memory Cache
`WithCache(ttl)` installs a URL-keyed in-memory response cache scoped to GET requests. Successful
2xx responses are stored. Cached responses are served with an `X-Cache: HIT` header and bypass all
downstream handlers. Expired entries are evicted lazily on the next access for the same key. The
cache is not shared across `Engine` instances and has no size limit.
---
## 13. OpenTelemetry Tracing
`WithTracing(serviceName)` adds otelgin middleware that creates a span for each request, tagged
with HTTP method, route template, and response status code. Trace context is propagated via the
W3C `traceparent` header.
`NewTracerProvider(exporter)` is a convenience helper for tests and simple single-service
deployments that constructs a synchronous `TracerProvider` and installs it globally. Production
deployments should build a batching provider with appropriate resource attributes.
---
## 14. Internationalisation
`WithI18n(cfg)` parses the `Accept-Language` header on every request and stores the resolved BCP 47
locale tag in the Gin context. The `golang.org/x/text/language` matcher handles quality-weighted
negotiation and script/region subtag matching against the configured supported locales.
Handlers retrieve the locale and optional localised messages:
```go
locale := api.GetLocale(c) // e.g. "en", "fr"
msg, ok := api.GetMessage(c, "greeting") // from configured Messages map
```
The built-in message map is a lightweight bridge. The `go-i18n` grammar engine is the intended
replacement for production-grade localisation.
---
## 15. Package Layout
```
go-api/
├── api.go — Engine struct, New(), build(), Serve(), Handler(), Addr(), Channels()
├── options.go — All With*() option functions
├── group.go — RouteGroup, StreamGroup, DescribableGroup interfaces; RouteDescription
├── response.go — Response[T], Error, Meta, OK(), Fail(), FailWithDetails(), Paginated()
├── middleware.go — bearerAuthMiddleware(), requestIDMiddleware()
├── authentik.go — AuthentikUser, AuthentikConfig, GetUser(), RequireAuth(), RequireGroup()
├── websocket.go — wrapWSHandler() helper
├── sse.go — SSEBroker, NewSSEBroker(), Publish(), Handler(), Drain()
├── cache.go — cacheStore, cacheEntry, cacheWriter, cacheMiddleware()
├── brotli.go — brotliHandler, newBrotliHandler(); BrotliDefault/BestSpeed/BestCompression constants
├── graphql.go — graphqlConfig, GraphQLOption, WithPlayground(), WithGraphQLPath(), mountGraphQL()
├── i18n.go — I18nConfig, WithI18n(), i18nMiddleware(), GetLocale(), GetMessage()
├── tracing.go — WithTracing(), NewTracerProvider()
├── swagger.go — swaggerSpec, registerSwagger(); swaggerSeq for multi-instance safety
├── openapi.go — SpecBuilder, Build(), buildPaths(), buildTags(), envelopeSchema()
├── export.go — ExportSpec(), ExportSpecToFile()
├── bridge.go — ToolDescriptor, ToolBridge, NewToolBridge(), Add(), Describe(), Tools()
└── codegen.go — SDKGenerator, Generate(), buildArgs(), Available(), SupportedLanguages()
```
---
## 16. Dependency Diagram
### Direct Dependencies
| Module | Role |
|--------|------|
| `github.com/gin-gonic/gin` | HTTP router and middleware engine |
| `github.com/gin-contrib/cors` | CORS policy middleware |
| `github.com/gin-contrib/secure` | Security headers |
| `github.com/gin-contrib/gzip` | Gzip compression |
| `github.com/gin-contrib/slog` | Structured logging |
| `github.com/gin-contrib/timeout` | Per-request deadlines |
| `github.com/gin-contrib/static` | Static file serving |
| `github.com/gin-contrib/sessions` | Cookie-backed sessions |
| `github.com/gin-contrib/authz` | Casbin authorisation |
| `github.com/gin-contrib/httpsign` | HTTP Signatures |
| `github.com/gin-contrib/location/v2` | Reverse proxy header detection |
| `github.com/gin-contrib/pprof` | Go profiling endpoints |
| `github.com/gin-contrib/expvar` | Runtime metrics endpoint |
| `github.com/casbin/casbin/v2` | Policy-based access control engine |
| `github.com/coreos/go-oidc/v3` | OIDC JWT validation for Authentik |
| `github.com/andybalholm/brotli` | Brotli compression |
| `github.com/gorilla/websocket` | WebSocket upgrade |
| `github.com/swaggo/gin-swagger` | Swagger UI handler |
| `github.com/swaggo/files` | Swagger UI static assets |
| `github.com/swaggo/swag` | Swagger spec registry |
| `github.com/99designs/gqlgen` | GraphQL schema execution |
| `go.opentelemetry.io/otel` | OpenTelemetry tracing |
| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | OTel + Gin |
| `golang.org/x/text/language` | BCP 47 language matching |
| `gopkg.in/yaml.v3` | YAML export of OpenAPI spec |
### What Imports go-api
go-api is imported by packages that need to expose REST endpoints. Within the Lethean ecosystem
the expected importers are:
```
Core CLI (forge.lthn.ai/core/go)
└─ go-api ← wires route groups from go-ai, go-ml, go-rag into an Engine
go-ai
└─ go-api ← ToolBridge converts MCP tool descriptors to REST endpoints
go-ml
└─ go-api ← may register inference/scoring route groups
Application main packages
└─ go-api ← top-level server bootstrap
```
go-api itself has no imports from other Lethean ecosystem modules. It is the stable base that
others import, not a consumer of them.

420
docs/development.md Normal file
View file

@ -0,0 +1,420 @@
<!-- SPDX-License-Identifier: EUPL-1.2 -->
# go-api Development Guide
go-api is the Gin-based REST framework and OpenAPI toolkit for the Lethean Go ecosystem. It
provides the `Engine` struct, 21 middleware options, the `Response[T]` envelope, `ToolBridge`,
`SpecBuilder`, and SDK codegen. This guide covers everything needed to build, test, extend, and
contribute to the repository.
Module path: `forge.lthn.ai/core/go-api`
Licence: EUPL-1.2
Language: Go 1.25
---
## Table of Contents
1. [Prerequisites](#prerequisites)
2. [Building](#building)
3. [Testing](#testing)
4. [Test Patterns](#test-patterns)
5. [Adding a New With*() Option](#adding-a-new-with-option)
6. [Adding a RouteGroup](#adding-a-routegroup)
7. [Adding a DescribableGroup](#adding-a-describablegroup)
8. [Coding Standards](#coding-standards)
---
## Prerequisites
### Go toolchain
Go 1.25 or later is required. Verify the installed version:
```bash
go version go1.25.x darwin/arm64
```
### No sibling dependencies
go-api has no `forge.lthn.ai/core/*` dependencies. It imports only external open-source packages.
There are no `replace` directives and no sibling repositories to check out. Cloning go-api alone
is sufficient to build and test it.
---
## Building
go-api is a library module with no `main` package. Build all packages to verify that everything
compiles cleanly:
```bash
go build ./...
```
Vet for suspicious constructs:
```bash
go vet ./...
```
Neither command produces a binary. If you need a runnable server for manual testing, create a
`main.go` in a temporary `cmd/` directory that imports go-api and calls `engine.Serve()`.
---
## Testing
### Run all tests
```bash
go test ./...
```
### Run a single test by name
```bash
go test -run TestName ./...
```
The `-run` flag accepts a regex:
```bash
go test -run TestToolBridge ./...
go test -run TestSpecBuilder_Good ./...
```
### Verbose output
```bash
go test -v ./...
```
### Race detector
Always run with `-race` before opening a pull request. The middleware layer uses concurrency
(SSE broker, cache store, Brotli writer), and the race detector catches data races reliably:
```bash
go test -race ./...
```
### Live Authentik integration tests
`authentik_integration_test.go` contains tests that require a live Authentik instance. These are
skipped automatically in environments without the `AUTHENTIK_ISSUER` and `AUTHENTIK_CLIENT_ID`
environment variables set. They do not run as part of `go test ./...` in standard CI.
---
## Test Patterns
### Naming convention
All test functions follow the `_Good`, `_Bad`, `_Ugly` suffix pattern:
| Suffix | Purpose |
|---------|---------|
| `_Good` | Happy path — the input is valid and the operation succeeds |
| `_Bad` | Expected error conditions — invalid input, missing config, wrong state |
| `_Ugly` | Panics and extreme edge cases — nil receivers, resource exhaustion, concurrent access |
Examples from the codebase:
```go
func TestSpecBuilder_Good_SingleDescribableGroup(t *testing.T) { ... }
func TestSpecBuilder_Good_MultipleGroups(t *testing.T) { ... }
func TestSDKGenerator_Bad_UnsupportedLanguage(t *testing.T) { ... }
func TestSDKGenerator_Bad_MissingSpecFile(t *testing.T) { ... }
```
### Engine test helpers
Tests that need a running HTTP server use `httptest.NewRecorder()` or `httptest.NewServer()`.
Build the engine handler directly rather than calling `Serve()`:
```go
engine, _ := api.New(api.WithBearerAuth("tok"))
engine.Register(&myGroup{})
handler := engine.Handler()
req := httptest.NewRequest(http.MethodGet, "/health", nil)
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", rec.Code)
}
```
### SSE tests
SSE handler tests open a real HTTP connection with `httptest.NewServer()` and read the
`text/event-stream` response body line by line. Publish from a goroutine and use a deadline
to avoid hanging indefinitely:
```go
srv := httptest.NewServer(engine.Handler())
defer srv.Close()
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, srv.URL+"/events", nil)
resp, _ := http.DefaultClient.Do(req)
// read lines from resp.Body
```
### Cache tests
Cache tests must use `httptest.NewServer()` rather than a recorder because the cache middleware
needs a proper response cycle to capture and replay responses. Verify the `X-Cache: HIT` header
on the second request to the same URL.
### Authentik tests
Authentik middleware tests use raw `httptest.NewRequest()` with `X-authentik-*` headers set
directly. No live Authentik instance is required for unit tests — the permissive middleware is
exercised entirely through header injection and context assertions.
---
## Adding a New With*() Option
With*() options are the primary extension point. All options follow an identical pattern.
### Step 1: Add the option function
Open `options.go` and add a new exported function that returns an `Option`:
```go
// WithRateLimit adds request rate limiting middleware.
// Requests exceeding limit per second per IP are rejected with 429 Too Many Requests.
func WithRateLimit(limit int) Option {
return func(e *Engine) {
e.middlewares = append(e.middlewares, rateLimitMiddleware(limit))
}
}
```
If the option stores state on `Engine` (like `swaggerEnabled` or `sseBroker`), add the
corresponding field to the `Engine` struct in `api.go` and reference it in `build()`.
### Step 2: Implement the middleware
If the option wraps a gin-contrib package, follow the existing pattern in `options.go` (inline).
For options with non-trivial logic, create a dedicated file (e.g. `ratelimit.go`):
```go
// SPDX-License-Identifier: EUPL-1.2
package api
import (
"net/http"
"sync"
"time"
"github.com/gin-gonic/gin"
)
func rateLimitMiddleware(limit int) gin.HandlerFunc {
// implementation
}
```
### Step 3: Add the dependency to go.mod
If the option relies on a new external package:
```bash
go get github.com/example/ratelimit
go mod tidy
```
### Step 4: Write tests
Create `ratelimit_test.go` following the `_Good`/`_Bad`/`_Ugly` naming convention. Test with
`httptest` rather than calling `Serve()`.
### Step 5: Update the Swagger UI build path
If the option adds a new built-in HTTP endpoint (like WebSocket at `/ws` or SSE at `/events`),
add it to the `build()` method in `api.go` after the GraphQL block but before Swagger.
---
## Adding a RouteGroup
`RouteGroup` is the standard way for subsystem packages to contribute REST endpoints.
### Minimum implementation
```go
// SPDX-License-Identifier: EUPL-1.2
package mypackage
import (
api "forge.lthn.ai/core/go-api"
"github.com/gin-gonic/gin"
"net/http"
)
type Routes struct {
service *Service
}
func NewRoutes(svc *Service) *Routes {
return &Routes{service: svc}
}
func (r *Routes) Name() string { return "mypackage" }
func (r *Routes) BasePath() string { return "/v1/mypackage" }
func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) {
rg.GET("/items", r.listItems)
rg.POST("/items", r.createItem)
}
func (r *Routes) listItems(c *gin.Context) {
items, err := r.service.List(c.Request.Context())
if err != nil {
c.JSON(http.StatusInternalServerError, api.Fail("internal", err.Error()))
return
}
c.JSON(http.StatusOK, api.OK(items))
}
```
Register with the engine:
```go
engine.Register(mypackage.NewRoutes(svc))
```
### Adding StreamGroup
If the group publishes WebSocket channels, implement `StreamGroup` as well:
```go
func (r *Routes) Channels() []string {
return []string{"mypackage.items.updated"}
}
```
---
## Adding a DescribableGroup
`DescribableGroup` extends `RouteGroup` with OpenAPI metadata. Implementing it ensures the group's
endpoints appear in the generated spec and Swagger UI.
```go
func (r *Routes) Describe() []api.RouteDescription {
return []api.RouteDescription{
{
Method: "GET",
Path: "/items",
Summary: "List items",
Tags: []string{"mypackage"},
Response: map[string]any{
"type": "array",
"items": map[string]any{"type": "object"},
},
},
{
Method: "POST",
Path: "/items",
Summary: "Create an item",
Tags: []string{"mypackage"},
RequestBody: map[string]any{
"type": "object",
"properties": map[string]any{
"name": map[string]any{"type": "string"},
},
"required": []string{"name"},
},
Response: map[string]any{"type": "object"},
},
}
}
```
Paths in `RouteDescription` are relative to `BasePath()`. The `SpecBuilder` concatenates them
when building the full OpenAPI path.
---
## Coding Standards
### Language
Use **UK English** in all comments, documentation, log messages, and user-facing strings:
`colour`, `organisation`, `centre`, `initialise`, `licence` (noun), `license` (verb).
### Licence header
Every new Go source file must carry the EUPL-1.2 SPDX identifier as the first line:
```go
// SPDX-License-Identifier: EUPL-1.2
package api
```
Do not add licence headers to test files.
### Error handling
- Always return errors rather than panicking.
- Wrap errors with context: `fmt.Errorf("component.Operation: what failed: %w", err)`.
- Do not discard errors with `_` unless the operation is genuinely fire-and-forget and the
reason is documented with a comment.
- Log errors at the point of handling, not at the point of wrapping, to avoid duplicate entries.
### Response envelope
All HTTP handlers must use the `api.OK()`, `api.Fail()`, `api.FailWithDetails()`, or
`api.Paginated()` helpers rather than constructing `Response[T]` directly. This ensures the
envelope structure is consistent across all route groups.
### Test naming
- Function names: `Test{Type}_{Suffix}_{Description}` where `{Suffix}` is `Good`, `Bad`, or `Ugly`.
- Helper constructors: `newTest{Type}(t *testing.T, ...) *Type`.
- Always call `t.Helper()` at the top of every test helper function.
### Formatting
The codebase uses `gofmt` defaults. Run before committing:
```bash
gofmt -l -w .
```
### Commits
Use [Conventional Commits](https://www.conventionalcommits.org/):
```
feat(api): add WithRateLimit per-IP rate limiting middleware
Adds configurable per-IP rate limiting. Requests exceeding the limit
per second are rejected with 429 Too Many Requests and a standard
Fail() error envelope.
Co-Authored-By: Virgil <virgil@lethean.io>
```
Types in use across the repository: `feat`, `fix`, `refactor`, `test`, `docs`, `chore`, `perf`.
### Middleware conventions
- Every `With*()` function must append to `e.middlewares`, not modify Gin routes directly.
Routes are only registered in `build()`.
- Options that require `Engine` struct fields (like `swaggerEnabled`) must be readable by `build()`,
not set inside `With*()` directly without a field.
- Middleware that exposes sensitive data (`WithPprof`, `WithExpvar`) must carry a `// WARNING:`
comment in the godoc directing users away from production exposure without authentication.

219
docs/history.md Normal file
View file

@ -0,0 +1,219 @@
<!-- SPDX-License-Identifier: EUPL-1.2 -->
# go-api — Project History and Known Limitations
Module: `forge.lthn.ai/core/go-api`
---
## Origins
`go-api` was created as the dedicated HTTP framework for the Lethean Go ecosystem. The motivation
was to give every Go package in the stack a consistent way to expose REST endpoints without each
package taking its own opinion on routing, middleware, response formatting, or OpenAPI generation.
It was scaffolded independently from the start — it was never extracted from a monolith — and has
no `forge.lthn.ai/core/*` dependencies. This keeps it at the bottom of the import graph: every
other package can import go-api, but go-api imports nothing from the ecosystem.
---
## Development Phases
### Phase 1 — Core Engine (36 tests)
Commits `889391a` through `22f8a69`
The initial phase established the foundational abstractions that all subsequent work builds on.
**Scaffold** (`889391a`):
Module path `forge.lthn.ai/core/go-api` created. `go.mod` initialised with Gin as the only
direct dependency.
**Response envelope** (`7835837`):
`Response[T]`, `Error`, and `Meta` types defined. `OK()`, `Fail()`, and `Paginated()` helpers
added. The generic envelope was established from the start rather than retrofitted.
**RouteGroup interface** (`6f5fb69`):
`RouteGroup` (Name, BasePath, RegisterRoutes) and `StreamGroup` (Channels) interfaces defined
in `group.go`. The interface-driven extension model was the core design decision of Phase 1.
**Engine** (`db75c88`):
`Engine` struct added with `New()`, `Register()`, `Handler()`, `Serve()`, and graceful shutdown.
Default listen address `:8080`. Built-in `GET /health` endpoint. Panic recovery via `gin.Recovery()`
always applied first.
**Bearer auth, request ID, CORS** (`d21734d`):
First three middleware options: `WithBearerAuth()`, `WithRequestID()`, `WithCORS()`.
The functional `Option` type and the `With*()` pattern were established here.
**Swagger UI** (`22f8a69`):
`WithSwagger()` added. The initial implementation served a static Swagger UI backed by a placeholder
spec; this was later replaced in Phase 3.
**WebSocket** (`22f8a69`):
`WithWSHandler()` added, mounting any `http.Handler` at `GET /ws`. `Engine.Channels()` added to
aggregate channel names from `StreamGroup` implementations.
By the end of Phase 1, the module had 36 tests covering the engine lifecycle, health endpoint,
response helpers, bearer auth, request ID propagation, CORS, and WebSocket mounting.
---
### Phase 2 — 21 Middleware Options (143 tests)
Commits `d760e77` through `8ba1716`
Phase 2 expanded the middleware library in four waves, reaching 21 `With*()` options total.
**Wave 1 — Security and Identity** (`d760e77` through `8f3e496`):
The Authentik integration was the most significant addition of this wave.
- `WithAuthentik()` — permissive forward-auth middleware. Reads `X-authentik-*` headers when
`TrustedProxy: true`, validates JWTs via OIDC discovery when `Issuer` and `ClientID` are set.
Fail-open: unauthenticated requests are never rejected by this middleware alone.
- `RequireAuth()`, `RequireGroup()` — explicit guards for protected routes, returning 401 and 403
respectively via the standard `Fail()` envelope.
- `GetUser()` — context accessor for the current `AuthentikUser`.
- `AuthentikUser` — carries Username, Email, Name, UID, Groups, Entitlements, and JWT. `HasGroup()`
convenience method added.
- Live integration tests added in `authentik_integration_test.go`, guarded by environment variables.
- `WithSecure()` — HSTS, X-Frame-Options DENY, X-Content-Type-Options nosniff, strict referrer
policy. SSL redirect deliberately omitted to work correctly behind a TLS-terminating proxy.
**Wave 2 — Compression and Logging** (`6521b90` through `6bb7195`):
- `WithTimeout(d)` — per-request deadline via gin-contrib/timeout. Returns 504 with the standard
Fail() envelope on expiry.
- `WithGzip(level...)` — gzip response compression; defaults to `gzip.DefaultCompression`.
- `WithBrotli(level...)` — Brotli compression via `andybalholm/brotli`. Custom `brotliHandler`
wrapping `brotli.HTTPCompressor`.
- `WithSlog(logger)` — structured request logging via gin-contrib/slog. Falls back to
`slog.Default()` when nil is passed.
- `WithStatic(prefix, root)` — static file serving via gin-contrib/static; directory listing
disabled.
**Wave 3 — Auth, Caching, Streaming** (`0ab962a` through `7b3f99e`):
- `WithCache(ttl)` — in-memory GET response cache. Custom `cacheWriter` intercepts the response
body without affecting the downstream handler. `X-Cache: HIT` on served cache entries.
- `WithSessions(name, secret)` — cookie-backed server sessions via gin-contrib/sessions.
- `WithAuthz(enforcer)` — Casbin policy-based authorisation via gin-contrib/authz. Subject from
HTTP Basic Auth.
- `WithHTTPSign(secrets, opts...)` — HTTP Signatures verification via gin-contrib/httpsign.
- `WithSSE(broker)` — Server-Sent Events at `GET /events`. `SSEBroker` added with `Publish()`,
channel filtering, 64-event per-client buffer, and `Drain()` for graceful shutdown.
**Wave 4 — Infrastructure and Protocol** (`a612d85` through `8ba1716`):
- `WithLocation()` — reverse proxy header detection via gin-contrib/location/v2.
- `WithI18n(cfg...)` — Accept-Language parsing and BCP 47 locale matching via
`golang.org/x/text/language`. `GetLocale()` and `GetMessage()` context accessors added.
- `WithGraphQL(schema, opts...)` — gqlgen `ExecutableSchema` mounting. `WithPlayground()` and
`WithGraphQLPath()` sub-options. Playground at `{path}/playground`.
- `WithPprof()` — Go runtime profiling at `/debug/pprof/`.
- `WithExpvar()` — expvar runtime metrics at `/debug/vars`.
- `WithTracing(name, opts...)` — OpenTelemetry distributed tracing via otelgin. `NewTracerProvider()`
convenience helper added. W3C `traceparent` propagation.
At the end of Phase 2, the module had 143 tests.
---
### Phase 3 — OpenAPI, ToolBridge, SDK Codegen (176 tests)
Commits `465bd60` through `1910aec`
Phase 3 upgraded the Swagger integration from a placeholder to a full runtime OpenAPI 3.1 pipeline
and added two new subsystems: `ToolBridge` and `SDKGenerator`.
**DescribableGroup interface** (`465bd60`):
`DescribableGroup` added to `group.go`, extending `RouteGroup` with `Describe() []RouteDescription`.
`RouteDescription` carries HTTP method, path, summary, description, tags, and JSON Schema maps
for request body and response data. This was the contract that `SpecBuilder` and `ToolBridge`
would both consume.
**ToolBridge** (`2b63c7b`):
`ToolBridge` added to `bridge.go`. Converts `ToolDescriptor` values into `POST /{tool_name}`
Gin routes and implements `DescribableGroup` so those routes appear in the OpenAPI spec. Designed
to bridge the MCP tool model (as used by go-ai) into the REST world. `Tools()` accessor added
for external enumeration.
**SpecBuilder** (`3e96f9b`):
`SpecBuilder` added to `openapi.go`. Generates a complete OpenAPI 3.1 JSON document from registered
`RouteGroup` and `DescribableGroup` values. Includes the built-in `GET /health` endpoint, shared
`Error` and `Meta` component schemas, and the `Response[T]` envelope schema wrapping every response
body. Tags are derived from all group names, not just describable ones.
**Spec export** (`e94283b`):
`ExportSpec()` and `ExportSpecToFile()` added to `export.go`. Supports `"json"` and `"yaml"`
output formats. YAML output is produced by unmarshalling the JSON then re-encoding with
`gopkg.in/yaml.v3` at 2-space indentation. Parent directories created automatically by
`ExportSpecToFile()`.
**Swagger refactor** (`303779f`):
`registerSwagger()` in `swagger.go` rewritten to use `SpecBuilder` rather than the previous
placeholder. A `swaggerSpec` wrapper satisfies the `swag.Spec` interface and builds the spec
lazily on first access via `sync.Once`. A `swaggerSeq` atomic counter assigns unique instance
names so multiple `Engine` instances in the same test binary do not collide in the global
`swag` registry.
**SDK codegen** (`a09a4e9`, `1910aec`):
`SDKGenerator` added to `codegen.go`. Wraps `openapi-generator-cli` to generate client SDKs
for 11 target languages. `SupportedLanguages()` returns the list in sorted order (the sort was
added in `1910aec` to ensure deterministic output in tests and documentation).
At the end of Phase 3, the module has 176 tests.
---
## Known Limitations
### 1. Cache has no size limit
`WithCache(ttl)` stores all successful GET responses in memory with no maximum entry count or
total size bound. For a server receiving requests to many distinct URLs, the cache will grow
without bound. A LRU eviction policy or a configurable maximum is the natural next step.
### 2. SDK codegen requires an external binary
`SDKGenerator.Generate()` shells out to `openapi-generator-cli`. This requires a JVM and the
openapi-generator JAR to be installed on the host. `Available()` checks whether the CLI is on
`PATH` but there is no programmatic fallback. Packaging `openapi-generator-cli` via a Docker
wrapper or replacing it with a pure-Go generator would remove this external dependency.
### 3. OpenAPI spec generation is build-time only
`SpecBuilder.Build()` generates the spec from `Describe()` return values, which are static at
the time of construction. Dynamic route generation (for example, routes registered after
`New()` returns) is not reflected in the spec. This matches the current design — all groups
must be registered before `Serve()` is called — but it would conflict with any future dynamic
route registration model.
### 4. i18n message map is a lightweight bridge only
`WithI18n()` accepts a `Messages map[string]map[string]string` for simple key-value lookups.
It does not support pluralisation, gender inflection, argument interpolation, or any of the
grammar features provided by `go-i18n`. Applications requiring production-grade localisation
should use `go-i18n` directly and use `GetLocale()` to pass the detected locale to it.
### 5. Authentik JWT validation performs OIDC discovery on first request
`getOIDCProvider()` performs an OIDC discovery request on first use and caches the resulting
`*oidc.Provider` by issuer URL. This is lazy — the first request to a non-public path will
incur a network round-trip to the issuer. A warm-up call during application startup would
eliminate this latency from the first real request.
### 6. ToolBridge has no input validation
`ToolBridge.Add()` accepts a `ToolDescriptor` with `InputSchema` and `OutputSchema` maps, but
these are used only for OpenAPI documentation. The registered `gin.HandlerFunc` is responsible
for its own input validation. There is no automatic binding or validation of incoming request
bodies against the declared JSON Schema.
### 7. SSEBroker.Drain() does not wait for clients to disconnect
`Drain()` closes all client event channels to signal disconnection but returns immediately
without waiting for client goroutines to exit. In a graceful shutdown sequence, there is a
brief window where client HTTP connections remain open. The engine's 10-second shutdown
deadline covers this window in practice, but there is no explicit coordination.