diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..0918fe1 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,498 @@ + + +# go-api Architecture + +**Module**: `forge.lthn.ai/core/go-api` +**Language**: Go 1.25 +**Licence**: EUPL-1.2 +**LOC**: ~2.1 K non-test, 176 tests passing + +--- + +## 1. Overview + +`go-api` is the Gin-based REST framework for the Lethean Go ecosystem. Subsystems across the +ecosystem plug into a central `Engine` via the `RouteGroup` interface, and go-api handles the HTTP +plumbing: middleware composition, response envelopes, WebSocket and SSE integration, GraphQL, +Authentik identity, OpenAPI spec generation, and SDK codegen. + +### Position in the Lethean Stack + +``` +AI Clients / Browsers / SDK consumers + | HTTP / WebSocket / SSE + v + [ go-api Engine ] ← this module + | | | + | | └─ OpenAPI spec → SDKGenerator → openapi-generator-cli + | └─ ToolBridge ──→ go-ai / go-ml / go-rag route groups + └─ RouteGroups ─────────→ any package that implements RouteGroup + + Core CLI or standalone main() bootstraps and wires everything +``` + +go-api is a library. It contains no `main` package. Callers construct an `Engine`, register route +groups, and call `Serve()`. + +--- + +## 2. Engine + +### 2.1 The Engine Struct + +`Engine` is the central container. It owns the listen address, the ordered list of registered route +groups, the middleware chain, and all optional integrations: + +```go +type Engine struct { + addr string + groups []RouteGroup + middlewares []gin.HandlerFunc + wsHandler http.Handler + sseBroker *SSEBroker + swaggerEnabled bool + swaggerTitle string + swaggerDesc string + swaggerVersion string + pprofEnabled bool + expvarEnabled bool + graphql *graphqlConfig +} +``` + +### 2.2 Construction + +`New()` applies functional options and returns a configured engine. The default listen address is +`:8080`. No middleware is added automatically beyond Gin's built-in panic recovery; every feature +requires an explicit `With*()` option. + +```go +engine, err := api.New( + api.WithAddr(":9000"), + api.WithBearerAuth("secret"), + api.WithCORS("*"), + api.WithRequestID(), + api.WithSlog(nil), + api.WithSwagger("My API", "Description", "1.0.0"), +) +engine.Register(myRoutes) +engine.Serve(ctx) +``` + +### 2.3 Build Sequence + +`Engine.build()` (called internally by `Handler()` and `Serve()`) assembles a fresh `*gin.Engine` +each time. The order is fixed: + +1. `gin.Recovery()` — panic recovery (always first). +2. User middleware in registration order — all `With*()` options that append to `e.middlewares`. +3. Built-in `GET /health` endpoint — always present. +4. Route groups — each mounted at its `BasePath()`. +5. WebSocket handler at `GET /ws` — when `WithWSHandler()` was called. +6. SSE broker at `GET /events` — when `WithSSE()` was called. +7. GraphQL endpoint — when `WithGraphQL()` was called. +8. Swagger UI at `GET /swagger/*any` — when `WithSwagger()` was called. +9. pprof endpoints at `GET /debug/pprof/*` — when `WithPprof()` was called. +10. expvar endpoint at `GET /debug/vars` — when `WithExpvar()` was called. + +### 2.4 Graceful Shutdown + +`Serve()` starts an `http.Server` in a goroutine and blocks on `ctx.Done()`. When the context is +cancelled, a 10-second `Shutdown` deadline is applied. In-flight requests complete or time out +before the server exits. + +--- + +## 3. RouteGroup / DescribableGroup Interfaces + +Three interfaces form the extension point model: + +```go +// RouteGroup is the minimum interface. All subsystems implement this. +type RouteGroup interface { + Name() string + BasePath() string + RegisterRoutes(rg *gin.RouterGroup) +} + +// StreamGroup is optionally implemented by groups that publish WebSocket channels. +type StreamGroup interface { + Channels() []string +} + +// DescribableGroup extends RouteGroup with OpenAPI metadata. +// Groups implementing this will appear in the generated spec. +type DescribableGroup interface { + RouteGroup + Describe() []RouteDescription +} +``` + +`RouteDescription` carries the HTTP method, path, summary, description, tags, and JSON Schema +maps for the request body and response data. The `SpecBuilder` consumes these to generate the +OpenAPI 3.1 paths object. + +`Engine.Channels()` iterates all registered groups and collects channel names from those that +implement `StreamGroup`. This list is used when initialising a WebSocket hub. + +--- + +## 4. Middleware Stack — 21 With*() Options + +All middleware options append to `Engine.middlewares` in the order they are passed to `New()`. +They are applied after `gin.Recovery()` but before any route handler. + +| Option | Purpose | Key detail | +|--------|---------|-----------| +| `WithAddr(addr)` | Listen address | Default `:8080` | +| `WithBearerAuth(token)` | Static bearer token auth | Skips `/health` and `/swagger` | +| `WithRequestID()` | X-Request-ID propagation | Preserves client-supplied IDs | +| `WithCORS(origins...)` | CORS policy | `"*"` enables `AllowAllOrigins`; 12-hour `MaxAge` | +| `WithMiddleware(mw...)` | Arbitrary Gin middleware | Escape hatch | +| `WithStatic(prefix, root)` | Static file serving | Directory listing disabled | +| `WithWSHandler(h)` | WebSocket at `/ws` | Wraps any `http.Handler` | +| `WithAuthentik(cfg)` | Authentik forward-auth + OIDC | Permissive; sets context, never rejects | +| `WithSwagger(title, desc, ver)` | Swagger UI at `/swagger/` | Runtime spec via `SpecBuilder` | +| `WithPprof()` | Go profiling at `/debug/pprof/` | WARNING: do not expose in production | +| `WithExpvar()` | Runtime metrics at `/debug/vars` | WARNING: do not expose in production | +| `WithSecure()` | Security headers | HSTS 1yr, X-Frame-Options DENY, nosniff | +| `WithGzip(level...)` | Gzip response compression | Default compression if level omitted | +| `WithBrotli(level...)` | Brotli response compression | Default compression if level omitted | +| `WithSlog(logger)` | Structured request logging | Falls back to `slog.Default()` if nil | +| `WithTimeout(d)` | Per-request deadline | 504 error envelope on timeout | +| `WithCache(ttl)` | In-memory GET response caching | `X-Cache: HIT` on cache hits | +| `WithSessions(name, secret)` | Cookie-backed server sessions | gin-contrib/sessions | +| `WithAuthz(enforcer)` | Casbin policy-based authorisation | Subject from HTTP Basic Auth | +| `WithHTTPSign(secrets, opts...)` | HTTP Signatures verification | draft-cavage-http-signatures | +| `WithSSE(broker)` | Server-Sent Events at `/events` | `?channel=` filtering | +| `WithLocation()` | Reverse proxy header detection | X-Forwarded-Proto / X-Forwarded-Host | +| `WithI18n(cfg...)` | Accept-Language locale detection | BCP 47 matching via x/text/language | +| `WithTracing(name, opts...)` | OpenTelemetry distributed tracing | otelgin + W3C traceparent | +| `WithGraphQL(schema, opts...)` | GraphQL endpoint | gqlgen; optional playground UI | + +--- + +## 5. Response[T] Envelope + +All API responses share a single generic envelope: + +```go +type Response[T any] struct { + Success bool `json:"success"` + Data T `json:"data,omitempty"` + Error *Error `json:"error,omitempty"` + Meta *Meta `json:"meta,omitempty"` +} +``` + +Constructor helpers: + +| Helper | Usage | +|--------|-------| +| `OK(data)` | Successful response with typed data | +| `Fail(code, message)` | Error response with code and message | +| `FailWithDetails(code, message, details)` | Error with additional detail payload | +| `Paginated(data, page, perPage, total)` | Successful response with pagination meta | + +`Meta` carries `request_id`, `duration`, `page`, `per_page`, and `total`. `Error` carries `code`, +`message`, and an optional `details` field for structured validation errors. + +--- + +## 6. Authentik Integration + +The `WithAuthentik()` option installs a permissive identity middleware that runs on every +non-public request. It has two extraction paths: + +**Path 1 — Forward-auth headers (TrustedProxy: true):** +When Traefik or another reverse proxy is configured with Authentik forward-auth, it injects +`X-authentik-username`, `X-authentik-email`, `X-authentik-name`, `X-authentik-uid`, +`X-authentik-groups` (pipe-separated), `X-authentik-entitlements`, and `X-authentik-jwt` headers. +The middleware reads these and populates an `AuthentikUser` in the Gin context. + +**Path 2 — OIDC JWT validation:** +For direct API clients that present a `Bearer` token, the middleware validates the JWT against +the configured OIDC issuer. Providers are cached by issuer URL to avoid repeated discovery requests. + +In both paths, if extraction fails the request continues unauthenticated (fail-open). Handlers +check identity with: + +```go +user := api.GetUser(c) // returns nil when unauthenticated +``` + +For protected routes, apply guards: + +```go +rg.GET("/private", api.RequireAuth(), handler) // 401 if no user +rg.GET("/admin", api.RequireGroup("admins"), handler) // 403 if wrong group +``` + +`AuthentikConfig.PublicPaths` lists additional paths exempt from header extraction. +`/health` and paths starting with `/swagger` are always exempt. + +--- + +## 7. WebSocket and SSE + +### WebSocket + +`WithWSHandler(h)` mounts any `http.Handler` at `GET /ws`. The intended pairing is a `go-ws` hub: + +```go +hub := ws.NewHub() +go hub.Run(ctx) +engine, _ := api.New(api.WithWSHandler(hub.Handler())) +``` + +Groups implementing `StreamGroup` declare channel names, which `Engine.Channels()` aggregates. + +### Server-Sent Events + +`SSEBroker` manages persistent SSE connections. Clients connect to `GET /events` and optionally +subscribe to a named channel via `?channel=`. Clients with no channel parameter receive +events on all channels. + +```go +broker := api.NewSSEBroker() +engine, _ := api.New(api.WithSSE(broker)) + +// In a handler or background goroutine: +broker.Publish("deployments", "deploy.started", payload) +``` + +Each client has a 64-event buffer. Overflow events are dropped without blocking the publisher. +`SSEBroker.Drain()` signals all clients to disconnect on graceful shutdown. + +--- + +## 8. GraphQL + +`WithGraphQL()` mounts a gqlgen `ExecutableSchema` at `/graphql` (or a custom path via +`WithGraphQLPath()`). An optional `WithPlayground()` adds the interactive playground at +`{path}/playground`. + +```go +engine, _ := api.New( + api.WithGraphQL(schema, + api.WithPlayground(), + api.WithGraphQLPath("/gql"), + ), +) +``` + +The endpoint accepts all HTTP methods (POST for queries and mutations, GET for playground +redirects and introspection). + +--- + +## 9. OpenAPI Spec Generation + +### SpecBuilder + +`SpecBuilder` generates an OpenAPI 3.1 JSON document from registered route groups: + +```go +builder := &api.SpecBuilder{ + Title: "My API", + Description: "...", + Version: "1.0.0", +} +data, _ := builder.Build(engine.Groups()) +``` + +The built document includes: + +- The `GET /health` endpoint under the `system` tag. +- One path entry per `RouteDescription` returned by `DescribableGroup.Describe()`. +- `#/components/schemas/Error` and `#/components/schemas/Meta` shared schemas. +- All response bodies wrapped in the `Response[T]` envelope schema. +- Tags derived from every registered group's `Name()`. + +Groups that implement `RouteGroup` but not `DescribableGroup` contribute a tag but no paths. + +### Export + +Two convenience functions write the spec to an `io.Writer` or directly to a file: + +```go +// Write JSON or YAML to a writer: +api.ExportSpec(os.Stdout, "yaml", builder, engine.Groups()) + +// Write to a file (parent dirs created automatically): +api.ExportSpecToFile("./api/openapi.yaml", "yaml", builder, engine.Groups()) +``` + +### Swagger UI + +When `WithSwagger()` is active, the spec is built lazily on first access by a `swaggerSpec` +wrapper and registered in the global `swag` registry with a unique sequence-based instance name. +Multiple `Engine` instances in the same process (common in tests) do not collide in the registry. + +--- + +## 10. ToolBridge + +`ToolBridge` converts tool descriptors — as produced by go-ai, go-ml, and similar packages — into +`POST /{tool_name}` REST endpoints, and implements `DescribableGroup` so those endpoints appear in +the generated OpenAPI spec. + +```go +bridge := api.NewToolBridge("/v1/tools") +bridge.Add(api.ToolDescriptor{ + Name: "file_read", + Description: "Read a file", + Group: "files", + InputSchema: map[string]any{"type": "object", "properties": ...}, +}, fileReadHandler) + +engine.Register(bridge) +``` + +Each registered tool becomes a `POST /v1/tools/{tool_name}` endpoint. `ToolBridge.Tools()` returns +the full list of descriptors, making it easy to enumerate tools from the outside. + +--- + +## 11. SDK Codegen + +`SDKGenerator` wraps `openapi-generator-cli` to generate client SDKs from an exported spec: + +```go +gen := &api.SDKGenerator{ + SpecPath: "./api/openapi.yaml", + OutputDir: "./sdk", + PackageName: "myapi", +} +if gen.Available() { + _ = gen.Generate(ctx, "typescript-fetch") + _ = gen.Generate(ctx, "python") +} +``` + +Supported target languages: `go`, `typescript-fetch`, `typescript-axios`, `python`, `java`, +`csharp`, `ruby`, `swift`, `kotlin`, `rust`, `php`. + +`SupportedLanguages()` returns the full list in sorted order. `SDKGenerator.Available()` reports +whether `openapi-generator-cli` is on `PATH` without running it. + +--- + +## 12. In-Memory Cache + +`WithCache(ttl)` installs a URL-keyed in-memory response cache scoped to GET requests. Successful +2xx responses are stored. Cached responses are served with an `X-Cache: HIT` header and bypass all +downstream handlers. Expired entries are evicted lazily on the next access for the same key. The +cache is not shared across `Engine` instances and has no size limit. + +--- + +## 13. OpenTelemetry Tracing + +`WithTracing(serviceName)` adds otelgin middleware that creates a span for each request, tagged +with HTTP method, route template, and response status code. Trace context is propagated via the +W3C `traceparent` header. + +`NewTracerProvider(exporter)` is a convenience helper for tests and simple single-service +deployments that constructs a synchronous `TracerProvider` and installs it globally. Production +deployments should build a batching provider with appropriate resource attributes. + +--- + +## 14. Internationalisation + +`WithI18n(cfg)` parses the `Accept-Language` header on every request and stores the resolved BCP 47 +locale tag in the Gin context. The `golang.org/x/text/language` matcher handles quality-weighted +negotiation and script/region subtag matching against the configured supported locales. + +Handlers retrieve the locale and optional localised messages: + +```go +locale := api.GetLocale(c) // e.g. "en", "fr" +msg, ok := api.GetMessage(c, "greeting") // from configured Messages map +``` + +The built-in message map is a lightweight bridge. The `go-i18n` grammar engine is the intended +replacement for production-grade localisation. + +--- + +## 15. Package Layout + +``` +go-api/ +├── api.go — Engine struct, New(), build(), Serve(), Handler(), Addr(), Channels() +├── options.go — All With*() option functions +├── group.go — RouteGroup, StreamGroup, DescribableGroup interfaces; RouteDescription +├── response.go — Response[T], Error, Meta, OK(), Fail(), FailWithDetails(), Paginated() +├── middleware.go — bearerAuthMiddleware(), requestIDMiddleware() +├── authentik.go — AuthentikUser, AuthentikConfig, GetUser(), RequireAuth(), RequireGroup() +├── websocket.go — wrapWSHandler() helper +├── sse.go — SSEBroker, NewSSEBroker(), Publish(), Handler(), Drain() +├── cache.go — cacheStore, cacheEntry, cacheWriter, cacheMiddleware() +├── brotli.go — brotliHandler, newBrotliHandler(); BrotliDefault/BestSpeed/BestCompression constants +├── graphql.go — graphqlConfig, GraphQLOption, WithPlayground(), WithGraphQLPath(), mountGraphQL() +├── i18n.go — I18nConfig, WithI18n(), i18nMiddleware(), GetLocale(), GetMessage() +├── tracing.go — WithTracing(), NewTracerProvider() +├── swagger.go — swaggerSpec, registerSwagger(); swaggerSeq for multi-instance safety +├── openapi.go — SpecBuilder, Build(), buildPaths(), buildTags(), envelopeSchema() +├── export.go — ExportSpec(), ExportSpecToFile() +├── bridge.go — ToolDescriptor, ToolBridge, NewToolBridge(), Add(), Describe(), Tools() +└── codegen.go — SDKGenerator, Generate(), buildArgs(), Available(), SupportedLanguages() +``` + +--- + +## 16. Dependency Diagram + +### Direct Dependencies + +| Module | Role | +|--------|------| +| `github.com/gin-gonic/gin` | HTTP router and middleware engine | +| `github.com/gin-contrib/cors` | CORS policy middleware | +| `github.com/gin-contrib/secure` | Security headers | +| `github.com/gin-contrib/gzip` | Gzip compression | +| `github.com/gin-contrib/slog` | Structured logging | +| `github.com/gin-contrib/timeout` | Per-request deadlines | +| `github.com/gin-contrib/static` | Static file serving | +| `github.com/gin-contrib/sessions` | Cookie-backed sessions | +| `github.com/gin-contrib/authz` | Casbin authorisation | +| `github.com/gin-contrib/httpsign` | HTTP Signatures | +| `github.com/gin-contrib/location/v2` | Reverse proxy header detection | +| `github.com/gin-contrib/pprof` | Go profiling endpoints | +| `github.com/gin-contrib/expvar` | Runtime metrics endpoint | +| `github.com/casbin/casbin/v2` | Policy-based access control engine | +| `github.com/coreos/go-oidc/v3` | OIDC JWT validation for Authentik | +| `github.com/andybalholm/brotli` | Brotli compression | +| `github.com/gorilla/websocket` | WebSocket upgrade | +| `github.com/swaggo/gin-swagger` | Swagger UI handler | +| `github.com/swaggo/files` | Swagger UI static assets | +| `github.com/swaggo/swag` | Swagger spec registry | +| `github.com/99designs/gqlgen` | GraphQL schema execution | +| `go.opentelemetry.io/otel` | OpenTelemetry tracing | +| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | OTel + Gin | +| `golang.org/x/text/language` | BCP 47 language matching | +| `gopkg.in/yaml.v3` | YAML export of OpenAPI spec | + +### What Imports go-api + +go-api is imported by packages that need to expose REST endpoints. Within the Lethean ecosystem +the expected importers are: + +``` +Core CLI (forge.lthn.ai/core/go) + └─ go-api ← wires route groups from go-ai, go-ml, go-rag into an Engine + +go-ai + └─ go-api ← ToolBridge converts MCP tool descriptors to REST endpoints + +go-ml + └─ go-api ← may register inference/scoring route groups + +Application main packages + └─ go-api ← top-level server bootstrap +``` + +go-api itself has no imports from other Lethean ecosystem modules. It is the stable base that +others import, not a consumer of them. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..81cb9ee --- /dev/null +++ b/docs/development.md @@ -0,0 +1,420 @@ + + +# go-api Development Guide + +go-api is the Gin-based REST framework and OpenAPI toolkit for the Lethean Go ecosystem. It +provides the `Engine` struct, 21 middleware options, the `Response[T]` envelope, `ToolBridge`, +`SpecBuilder`, and SDK codegen. This guide covers everything needed to build, test, extend, and +contribute to the repository. + +Module path: `forge.lthn.ai/core/go-api` +Licence: EUPL-1.2 +Language: Go 1.25 + +--- + +## Table of Contents + +1. [Prerequisites](#prerequisites) +2. [Building](#building) +3. [Testing](#testing) +4. [Test Patterns](#test-patterns) +5. [Adding a New With*() Option](#adding-a-new-with-option) +6. [Adding a RouteGroup](#adding-a-routegroup) +7. [Adding a DescribableGroup](#adding-a-describablegroup) +8. [Coding Standards](#coding-standards) + +--- + +## Prerequisites + +### Go toolchain + +Go 1.25 or later is required. Verify the installed version: + +```bash +go version go1.25.x darwin/arm64 +``` + +### No sibling dependencies + +go-api has no `forge.lthn.ai/core/*` dependencies. It imports only external open-source packages. +There are no `replace` directives and no sibling repositories to check out. Cloning go-api alone +is sufficient to build and test it. + +--- + +## Building + +go-api is a library module with no `main` package. Build all packages to verify that everything +compiles cleanly: + +```bash +go build ./... +``` + +Vet for suspicious constructs: + +```bash +go vet ./... +``` + +Neither command produces a binary. If you need a runnable server for manual testing, create a +`main.go` in a temporary `cmd/` directory that imports go-api and calls `engine.Serve()`. + +--- + +## Testing + +### Run all tests + +```bash +go test ./... +``` + +### Run a single test by name + +```bash +go test -run TestName ./... +``` + +The `-run` flag accepts a regex: + +```bash +go test -run TestToolBridge ./... +go test -run TestSpecBuilder_Good ./... +``` + +### Verbose output + +```bash +go test -v ./... +``` + +### Race detector + +Always run with `-race` before opening a pull request. The middleware layer uses concurrency +(SSE broker, cache store, Brotli writer), and the race detector catches data races reliably: + +```bash +go test -race ./... +``` + +### Live Authentik integration tests + +`authentik_integration_test.go` contains tests that require a live Authentik instance. These are +skipped automatically in environments without the `AUTHENTIK_ISSUER` and `AUTHENTIK_CLIENT_ID` +environment variables set. They do not run as part of `go test ./...` in standard CI. + +--- + +## Test Patterns + +### Naming convention + +All test functions follow the `_Good`, `_Bad`, `_Ugly` suffix pattern: + +| Suffix | Purpose | +|---------|---------| +| `_Good` | Happy path — the input is valid and the operation succeeds | +| `_Bad` | Expected error conditions — invalid input, missing config, wrong state | +| `_Ugly` | Panics and extreme edge cases — nil receivers, resource exhaustion, concurrent access | + +Examples from the codebase: + +```go +func TestSpecBuilder_Good_SingleDescribableGroup(t *testing.T) { ... } +func TestSpecBuilder_Good_MultipleGroups(t *testing.T) { ... } +func TestSDKGenerator_Bad_UnsupportedLanguage(t *testing.T) { ... } +func TestSDKGenerator_Bad_MissingSpecFile(t *testing.T) { ... } +``` + +### Engine test helpers + +Tests that need a running HTTP server use `httptest.NewRecorder()` or `httptest.NewServer()`. +Build the engine handler directly rather than calling `Serve()`: + +```go +engine, _ := api.New(api.WithBearerAuth("tok")) +engine.Register(&myGroup{}) +handler := engine.Handler() + +req := httptest.NewRequest(http.MethodGet, "/health", nil) +rec := httptest.NewRecorder() +handler.ServeHTTP(rec, req) + +if rec.Code != http.StatusOK { + t.Fatalf("expected 200, got %d", rec.Code) +} +``` + +### SSE tests + +SSE handler tests open a real HTTP connection with `httptest.NewServer()` and read the +`text/event-stream` response body line by line. Publish from a goroutine and use a deadline +to avoid hanging indefinitely: + +```go +srv := httptest.NewServer(engine.Handler()) +defer srv.Close() + +ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) +defer cancel() + +req, _ := http.NewRequestWithContext(ctx, http.MethodGet, srv.URL+"/events", nil) +resp, _ := http.DefaultClient.Do(req) +// read lines from resp.Body +``` + +### Cache tests + +Cache tests must use `httptest.NewServer()` rather than a recorder because the cache middleware +needs a proper response cycle to capture and replay responses. Verify the `X-Cache: HIT` header +on the second request to the same URL. + +### Authentik tests + +Authentik middleware tests use raw `httptest.NewRequest()` with `X-authentik-*` headers set +directly. No live Authentik instance is required for unit tests — the permissive middleware is +exercised entirely through header injection and context assertions. + +--- + +## Adding a New With*() Option + +With*() options are the primary extension point. All options follow an identical pattern. + +### Step 1: Add the option function + +Open `options.go` and add a new exported function that returns an `Option`: + +```go +// WithRateLimit adds request rate limiting middleware. +// Requests exceeding limit per second per IP are rejected with 429 Too Many Requests. +func WithRateLimit(limit int) Option { + return func(e *Engine) { + e.middlewares = append(e.middlewares, rateLimitMiddleware(limit)) + } +} +``` + +If the option stores state on `Engine` (like `swaggerEnabled` or `sseBroker`), add the +corresponding field to the `Engine` struct in `api.go` and reference it in `build()`. + +### Step 2: Implement the middleware + +If the option wraps a gin-contrib package, follow the existing pattern in `options.go` (inline). +For options with non-trivial logic, create a dedicated file (e.g. `ratelimit.go`): + +```go +// SPDX-License-Identifier: EUPL-1.2 + +package api + +import ( + "net/http" + "sync" + "time" + + "github.com/gin-gonic/gin" +) + +func rateLimitMiddleware(limit int) gin.HandlerFunc { + // implementation +} +``` + +### Step 3: Add the dependency to go.mod + +If the option relies on a new external package: + +```bash +go get github.com/example/ratelimit +go mod tidy +``` + +### Step 4: Write tests + +Create `ratelimit_test.go` following the `_Good`/`_Bad`/`_Ugly` naming convention. Test with +`httptest` rather than calling `Serve()`. + +### Step 5: Update the Swagger UI build path + +If the option adds a new built-in HTTP endpoint (like WebSocket at `/ws` or SSE at `/events`), +add it to the `build()` method in `api.go` after the GraphQL block but before Swagger. + +--- + +## Adding a RouteGroup + +`RouteGroup` is the standard way for subsystem packages to contribute REST endpoints. + +### Minimum implementation + +```go +// SPDX-License-Identifier: EUPL-1.2 + +package mypackage + +import ( + api "forge.lthn.ai/core/go-api" + "github.com/gin-gonic/gin" + "net/http" +) + +type Routes struct { + service *Service +} + +func NewRoutes(svc *Service) *Routes { + return &Routes{service: svc} +} + +func (r *Routes) Name() string { return "mypackage" } +func (r *Routes) BasePath() string { return "/v1/mypackage" } + +func (r *Routes) RegisterRoutes(rg *gin.RouterGroup) { + rg.GET("/items", r.listItems) + rg.POST("/items", r.createItem) +} + +func (r *Routes) listItems(c *gin.Context) { + items, err := r.service.List(c.Request.Context()) + if err != nil { + c.JSON(http.StatusInternalServerError, api.Fail("internal", err.Error())) + return + } + c.JSON(http.StatusOK, api.OK(items)) +} +``` + +Register with the engine: + +```go +engine.Register(mypackage.NewRoutes(svc)) +``` + +### Adding StreamGroup + +If the group publishes WebSocket channels, implement `StreamGroup` as well: + +```go +func (r *Routes) Channels() []string { + return []string{"mypackage.items.updated"} +} +``` + +--- + +## Adding a DescribableGroup + +`DescribableGroup` extends `RouteGroup` with OpenAPI metadata. Implementing it ensures the group's +endpoints appear in the generated spec and Swagger UI. + +```go +func (r *Routes) Describe() []api.RouteDescription { + return []api.RouteDescription{ + { + Method: "GET", + Path: "/items", + Summary: "List items", + Tags: []string{"mypackage"}, + Response: map[string]any{ + "type": "array", + "items": map[string]any{"type": "object"}, + }, + }, + { + Method: "POST", + Path: "/items", + Summary: "Create an item", + Tags: []string{"mypackage"}, + RequestBody: map[string]any{ + "type": "object", + "properties": map[string]any{ + "name": map[string]any{"type": "string"}, + }, + "required": []string{"name"}, + }, + Response: map[string]any{"type": "object"}, + }, + } +} +``` + +Paths in `RouteDescription` are relative to `BasePath()`. The `SpecBuilder` concatenates them +when building the full OpenAPI path. + +--- + +## Coding Standards + +### Language + +Use **UK English** in all comments, documentation, log messages, and user-facing strings: +`colour`, `organisation`, `centre`, `initialise`, `licence` (noun), `license` (verb). + +### Licence header + +Every new Go source file must carry the EUPL-1.2 SPDX identifier as the first line: + +```go +// SPDX-License-Identifier: EUPL-1.2 + +package api +``` + +Do not add licence headers to test files. + +### Error handling + +- Always return errors rather than panicking. +- Wrap errors with context: `fmt.Errorf("component.Operation: what failed: %w", err)`. +- Do not discard errors with `_` unless the operation is genuinely fire-and-forget and the + reason is documented with a comment. +- Log errors at the point of handling, not at the point of wrapping, to avoid duplicate entries. + +### Response envelope + +All HTTP handlers must use the `api.OK()`, `api.Fail()`, `api.FailWithDetails()`, or +`api.Paginated()` helpers rather than constructing `Response[T]` directly. This ensures the +envelope structure is consistent across all route groups. + +### Test naming + +- Function names: `Test{Type}_{Suffix}_{Description}` where `{Suffix}` is `Good`, `Bad`, or `Ugly`. +- Helper constructors: `newTest{Type}(t *testing.T, ...) *Type`. +- Always call `t.Helper()` at the top of every test helper function. + +### Formatting + +The codebase uses `gofmt` defaults. Run before committing: + +```bash +gofmt -l -w . +``` + +### Commits + +Use [Conventional Commits](https://www.conventionalcommits.org/): + +``` +feat(api): add WithRateLimit per-IP rate limiting middleware + +Adds configurable per-IP rate limiting. Requests exceeding the limit +per second are rejected with 429 Too Many Requests and a standard +Fail() error envelope. + +Co-Authored-By: Virgil +``` + +Types in use across the repository: `feat`, `fix`, `refactor`, `test`, `docs`, `chore`, `perf`. + +### Middleware conventions + +- Every `With*()` function must append to `e.middlewares`, not modify Gin routes directly. + Routes are only registered in `build()`. +- Options that require `Engine` struct fields (like `swaggerEnabled`) must be readable by `build()`, + not set inside `With*()` directly without a field. +- Middleware that exposes sensitive data (`WithPprof`, `WithExpvar`) must carry a `// WARNING:` + comment in the godoc directing users away from production exposure without authentication. diff --git a/docs/history.md b/docs/history.md new file mode 100644 index 0000000..823e360 --- /dev/null +++ b/docs/history.md @@ -0,0 +1,219 @@ + + +# go-api — Project History and Known Limitations + +Module: `forge.lthn.ai/core/go-api` + +--- + +## Origins + +`go-api` was created as the dedicated HTTP framework for the Lethean Go ecosystem. The motivation +was to give every Go package in the stack a consistent way to expose REST endpoints without each +package taking its own opinion on routing, middleware, response formatting, or OpenAPI generation. +It was scaffolded independently from the start — it was never extracted from a monolith — and has +no `forge.lthn.ai/core/*` dependencies. This keeps it at the bottom of the import graph: every +other package can import go-api, but go-api imports nothing from the ecosystem. + +--- + +## Development Phases + +### Phase 1 — Core Engine (36 tests) + +Commits `889391a` through `22f8a69` + +The initial phase established the foundational abstractions that all subsequent work builds on. + +**Scaffold** (`889391a`): +Module path `forge.lthn.ai/core/go-api` created. `go.mod` initialised with Gin as the only +direct dependency. + +**Response envelope** (`7835837`): +`Response[T]`, `Error`, and `Meta` types defined. `OK()`, `Fail()`, and `Paginated()` helpers +added. The generic envelope was established from the start rather than retrofitted. + +**RouteGroup interface** (`6f5fb69`): +`RouteGroup` (Name, BasePath, RegisterRoutes) and `StreamGroup` (Channels) interfaces defined +in `group.go`. The interface-driven extension model was the core design decision of Phase 1. + +**Engine** (`db75c88`): +`Engine` struct added with `New()`, `Register()`, `Handler()`, `Serve()`, and graceful shutdown. +Default listen address `:8080`. Built-in `GET /health` endpoint. Panic recovery via `gin.Recovery()` +always applied first. + +**Bearer auth, request ID, CORS** (`d21734d`): +First three middleware options: `WithBearerAuth()`, `WithRequestID()`, `WithCORS()`. +The functional `Option` type and the `With*()` pattern were established here. + +**Swagger UI** (`22f8a69`): +`WithSwagger()` added. The initial implementation served a static Swagger UI backed by a placeholder +spec; this was later replaced in Phase 3. + +**WebSocket** (`22f8a69`): +`WithWSHandler()` added, mounting any `http.Handler` at `GET /ws`. `Engine.Channels()` added to +aggregate channel names from `StreamGroup` implementations. + +By the end of Phase 1, the module had 36 tests covering the engine lifecycle, health endpoint, +response helpers, bearer auth, request ID propagation, CORS, and WebSocket mounting. + +--- + +### Phase 2 — 21 Middleware Options (143 tests) + +Commits `d760e77` through `8ba1716` + +Phase 2 expanded the middleware library in four waves, reaching 21 `With*()` options total. + +**Wave 1 — Security and Identity** (`d760e77` through `8f3e496`): + +The Authentik integration was the most significant addition of this wave. + +- `WithAuthentik()` — permissive forward-auth middleware. Reads `X-authentik-*` headers when + `TrustedProxy: true`, validates JWTs via OIDC discovery when `Issuer` and `ClientID` are set. + Fail-open: unauthenticated requests are never rejected by this middleware alone. +- `RequireAuth()`, `RequireGroup()` — explicit guards for protected routes, returning 401 and 403 + respectively via the standard `Fail()` envelope. +- `GetUser()` — context accessor for the current `AuthentikUser`. +- `AuthentikUser` — carries Username, Email, Name, UID, Groups, Entitlements, and JWT. `HasGroup()` + convenience method added. +- Live integration tests added in `authentik_integration_test.go`, guarded by environment variables. +- `WithSecure()` — HSTS, X-Frame-Options DENY, X-Content-Type-Options nosniff, strict referrer + policy. SSL redirect deliberately omitted to work correctly behind a TLS-terminating proxy. + +**Wave 2 — Compression and Logging** (`6521b90` through `6bb7195`): + +- `WithTimeout(d)` — per-request deadline via gin-contrib/timeout. Returns 504 with the standard + Fail() envelope on expiry. +- `WithGzip(level...)` — gzip response compression; defaults to `gzip.DefaultCompression`. +- `WithBrotli(level...)` — Brotli compression via `andybalholm/brotli`. Custom `brotliHandler` + wrapping `brotli.HTTPCompressor`. +- `WithSlog(logger)` — structured request logging via gin-contrib/slog. Falls back to + `slog.Default()` when nil is passed. +- `WithStatic(prefix, root)` — static file serving via gin-contrib/static; directory listing + disabled. + +**Wave 3 — Auth, Caching, Streaming** (`0ab962a` through `7b3f99e`): + +- `WithCache(ttl)` — in-memory GET response cache. Custom `cacheWriter` intercepts the response + body without affecting the downstream handler. `X-Cache: HIT` on served cache entries. +- `WithSessions(name, secret)` — cookie-backed server sessions via gin-contrib/sessions. +- `WithAuthz(enforcer)` — Casbin policy-based authorisation via gin-contrib/authz. Subject from + HTTP Basic Auth. +- `WithHTTPSign(secrets, opts...)` — HTTP Signatures verification via gin-contrib/httpsign. +- `WithSSE(broker)` — Server-Sent Events at `GET /events`. `SSEBroker` added with `Publish()`, + channel filtering, 64-event per-client buffer, and `Drain()` for graceful shutdown. + +**Wave 4 — Infrastructure and Protocol** (`a612d85` through `8ba1716`): + +- `WithLocation()` — reverse proxy header detection via gin-contrib/location/v2. +- `WithI18n(cfg...)` — Accept-Language parsing and BCP 47 locale matching via + `golang.org/x/text/language`. `GetLocale()` and `GetMessage()` context accessors added. +- `WithGraphQL(schema, opts...)` — gqlgen `ExecutableSchema` mounting. `WithPlayground()` and + `WithGraphQLPath()` sub-options. Playground at `{path}/playground`. +- `WithPprof()` — Go runtime profiling at `/debug/pprof/`. +- `WithExpvar()` — expvar runtime metrics at `/debug/vars`. +- `WithTracing(name, opts...)` — OpenTelemetry distributed tracing via otelgin. `NewTracerProvider()` + convenience helper added. W3C `traceparent` propagation. + +At the end of Phase 2, the module had 143 tests. + +--- + +### Phase 3 — OpenAPI, ToolBridge, SDK Codegen (176 tests) + +Commits `465bd60` through `1910aec` + +Phase 3 upgraded the Swagger integration from a placeholder to a full runtime OpenAPI 3.1 pipeline +and added two new subsystems: `ToolBridge` and `SDKGenerator`. + +**DescribableGroup interface** (`465bd60`): +`DescribableGroup` added to `group.go`, extending `RouteGroup` with `Describe() []RouteDescription`. +`RouteDescription` carries HTTP method, path, summary, description, tags, and JSON Schema maps +for request body and response data. This was the contract that `SpecBuilder` and `ToolBridge` +would both consume. + +**ToolBridge** (`2b63c7b`): +`ToolBridge` added to `bridge.go`. Converts `ToolDescriptor` values into `POST /{tool_name}` +Gin routes and implements `DescribableGroup` so those routes appear in the OpenAPI spec. Designed +to bridge the MCP tool model (as used by go-ai) into the REST world. `Tools()` accessor added +for external enumeration. + +**SpecBuilder** (`3e96f9b`): +`SpecBuilder` added to `openapi.go`. Generates a complete OpenAPI 3.1 JSON document from registered +`RouteGroup` and `DescribableGroup` values. Includes the built-in `GET /health` endpoint, shared +`Error` and `Meta` component schemas, and the `Response[T]` envelope schema wrapping every response +body. Tags are derived from all group names, not just describable ones. + +**Spec export** (`e94283b`): +`ExportSpec()` and `ExportSpecToFile()` added to `export.go`. Supports `"json"` and `"yaml"` +output formats. YAML output is produced by unmarshalling the JSON then re-encoding with +`gopkg.in/yaml.v3` at 2-space indentation. Parent directories created automatically by +`ExportSpecToFile()`. + +**Swagger refactor** (`303779f`): +`registerSwagger()` in `swagger.go` rewritten to use `SpecBuilder` rather than the previous +placeholder. A `swaggerSpec` wrapper satisfies the `swag.Spec` interface and builds the spec +lazily on first access via `sync.Once`. A `swaggerSeq` atomic counter assigns unique instance +names so multiple `Engine` instances in the same test binary do not collide in the global +`swag` registry. + +**SDK codegen** (`a09a4e9`, `1910aec`): +`SDKGenerator` added to `codegen.go`. Wraps `openapi-generator-cli` to generate client SDKs +for 11 target languages. `SupportedLanguages()` returns the list in sorted order (the sort was +added in `1910aec` to ensure deterministic output in tests and documentation). + +At the end of Phase 3, the module has 176 tests. + +--- + +## Known Limitations + +### 1. Cache has no size limit + +`WithCache(ttl)` stores all successful GET responses in memory with no maximum entry count or +total size bound. For a server receiving requests to many distinct URLs, the cache will grow +without bound. A LRU eviction policy or a configurable maximum is the natural next step. + +### 2. SDK codegen requires an external binary + +`SDKGenerator.Generate()` shells out to `openapi-generator-cli`. This requires a JVM and the +openapi-generator JAR to be installed on the host. `Available()` checks whether the CLI is on +`PATH` but there is no programmatic fallback. Packaging `openapi-generator-cli` via a Docker +wrapper or replacing it with a pure-Go generator would remove this external dependency. + +### 3. OpenAPI spec generation is build-time only + +`SpecBuilder.Build()` generates the spec from `Describe()` return values, which are static at +the time of construction. Dynamic route generation (for example, routes registered after +`New()` returns) is not reflected in the spec. This matches the current design — all groups +must be registered before `Serve()` is called — but it would conflict with any future dynamic +route registration model. + +### 4. i18n message map is a lightweight bridge only + +`WithI18n()` accepts a `Messages map[string]map[string]string` for simple key-value lookups. +It does not support pluralisation, gender inflection, argument interpolation, or any of the +grammar features provided by `go-i18n`. Applications requiring production-grade localisation +should use `go-i18n` directly and use `GetLocale()` to pass the detected locale to it. + +### 5. Authentik JWT validation performs OIDC discovery on first request + +`getOIDCProvider()` performs an OIDC discovery request on first use and caches the resulting +`*oidc.Provider` by issuer URL. This is lazy — the first request to a non-public path will +incur a network round-trip to the issuer. A warm-up call during application startup would +eliminate this latency from the first real request. + +### 6. ToolBridge has no input validation + +`ToolBridge.Add()` accepts a `ToolDescriptor` with `InputSchema` and `OutputSchema` maps, but +these are used only for OpenAPI documentation. The registered `gin.HandlerFunc` is responsible +for its own input validation. There is no automatic binding or validation of incoming request +bodies against the declared JSON Schema. + +### 7. SSEBroker.Drain() does not wait for clients to disconnect + +`Drain()` closes all client event channels to signal disconnection but returns immediately +without waiting for client goroutines to exit. In a graceful shutdown sequence, there is a +brief window where client HTTP connections remain open. The engine's 10-second shutdown +deadline covers this window in practice, but there is no explicit coordination.