go-api/docs/architecture.md

<!-- SPDX-License-Identifier: EUPL-1.2 -->

# go-api Architecture

**Module**: `forge.lthn.ai/core/go-api`
**Language**: Go 1.25
**Licence**: EUPL-1.2
**LOC**: ~2.1 K non-test, 176 tests passing

---

## 1. Overview

`go-api` is the Gin-based REST framework for the Lethean Go ecosystem. Subsystems across the
ecosystem plug into a central `Engine` via the `RouteGroup` interface, and go-api handles the HTTP
plumbing: middleware composition, response envelopes, WebSocket and SSE integration, GraphQL,
Authentik identity, OpenAPI spec generation, and SDK codegen.

### Position in the Lethean Stack

```
AI Clients / Browsers / SDK consumers
          |  HTTP / WebSocket / SSE
          v
  [ go-api Engine ]                ← this module
      |         |         |
      |         |         └─ OpenAPI spec → SDKGenerator → openapi-generator-cli
      |         └─ ToolBridge ──→ go-ai / go-ml / go-rag route groups
      └─ RouteGroups ─────────→ any package that implements RouteGroup

  Core CLI or standalone main() bootstraps and wires everything
```

go-api is a library. It contains no `main` package. Callers construct an `Engine`, register route
groups, and call `Serve()`.

---

## 2. Engine

### 2.1 The Engine Struct

`Engine` is the central container. It owns the listen address, the ordered list of registered route
groups, the middleware chain, and all optional integrations:

```go
type Engine struct {
    addr           string
    groups         []RouteGroup
    middlewares    []gin.HandlerFunc
    wsHandler      http.Handler
    sseBroker      *SSEBroker
    swaggerEnabled bool
    swaggerTitle   string
    swaggerDesc    string
    swaggerVersion string
    pprofEnabled   bool
    expvarEnabled  bool
    graphql        *graphqlConfig
}
```

### 2.2 Construction

`New()` applies functional options and returns a configured engine. The default listen address is
`:8080`. No middleware is added automatically beyond Gin's built-in panic recovery; every feature
requires an explicit `With*()` option.

```go
engine, err := api.New(
    api.WithAddr(":9000"),
    api.WithBearerAuth("secret"),
    api.WithCORS("*"),
    api.WithRequestID(),
    api.WithSlog(nil),
    api.WithSwagger("My API", "Description", "1.0.0"),
)
engine.Register(myRoutes)
engine.Serve(ctx)
```

### 2.3 Build Sequence

`Engine.build()` (called internally by `Handler()` and `Serve()`) assembles a fresh `*gin.Engine`
each time. The order is fixed:

1. `gin.Recovery()` — panic recovery (always first).
2. User middleware in registration order — all `With*()` options that append to `e.middlewares`.
3. Built-in `GET /health` endpoint — always present.
4. Route groups — each mounted at its `BasePath()`.
5. WebSocket handler at `GET /ws` — when `WithWSHandler()` was called.
6. SSE broker at `GET /events` — when `WithSSE()` was called.
7. GraphQL endpoint — when `WithGraphQL()` was called.
8. Swagger UI at `GET /swagger/*any` — when `WithSwagger()` was called.
9. pprof endpoints at `GET /debug/pprof/*` — when `WithPprof()` was called.
10. expvar endpoint at `GET /debug/vars` — when `WithExpvar()` was called.

### 2.4 Graceful Shutdown

`Serve()` starts an `http.Server` in a goroutine and blocks on `ctx.Done()`. When the context is
cancelled, a 10-second `Shutdown` deadline is applied. In-flight requests complete or time out
before the server exits.

---

## 3. RouteGroup / DescribableGroup Interfaces

Three interfaces form the extension point model:

```go
// RouteGroup is the minimum interface. All subsystems implement this.
type RouteGroup interface {
    Name() string
    BasePath() string
    RegisterRoutes(rg *gin.RouterGroup)
}

// StreamGroup is optionally implemented by groups that publish WebSocket channels.
type StreamGroup interface {
    Channels() []string
}

// DescribableGroup extends RouteGroup with OpenAPI metadata.
// Groups implementing this will appear in the generated spec.
type DescribableGroup interface {
    RouteGroup
    Describe() []RouteDescription
}
```

`RouteDescription` carries the HTTP method, path, summary, description, tags, and JSON Schema
maps for the request body and response data. The `SpecBuilder` consumes these to generate the
OpenAPI 3.1 paths object.

`Engine.Channels()` iterates all registered groups and collects channel names from those that
implement `StreamGroup`. This list is used when initialising a WebSocket hub.

---

## 4. Middleware Stack — 21 With*() Options

All middleware options append to `Engine.middlewares` in the order they are passed to `New()`.
They are applied after `gin.Recovery()` but before any route handler.

| Option | Purpose | Key detail |
|--------|---------|-----------|
| `WithAddr(addr)` | Listen address | Default `:8080` |
| `WithBearerAuth(token)` | Static bearer token auth | Skips `/health` and `/swagger` |
| `WithRequestID()` | X-Request-ID propagation | Preserves client-supplied IDs |
| `WithCORS(origins...)` | CORS policy | `"*"` enables `AllowAllOrigins`; 12-hour `MaxAge` |
| `WithMiddleware(mw...)` | Arbitrary Gin middleware | Escape hatch |
| `WithStatic(prefix, root)` | Static file serving | Directory listing disabled |
| `WithWSHandler(h)` | WebSocket at `/ws` | Wraps any `http.Handler` |
| `WithAuthentik(cfg)` | Authentik forward-auth + OIDC | Permissive; sets context, never rejects |
| `WithSwagger(title, desc, ver)` | Swagger UI at `/swagger/` | Runtime spec via `SpecBuilder` |
| `WithPprof()` | Go profiling at `/debug/pprof/` | WARNING: do not expose in production |
| `WithExpvar()` | Runtime metrics at `/debug/vars` | WARNING: do not expose in production |
| `WithSecure()` | Security headers | HSTS 1yr, X-Frame-Options DENY, nosniff |
| `WithGzip(level...)` | Gzip response compression | Default compression if level omitted |
| `WithBrotli(level...)` | Brotli response compression | Default compression if level omitted |
| `WithSlog(logger)` | Structured request logging | Falls back to `slog.Default()` if nil |
| `WithTimeout(d)` | Per-request deadline | 504 error envelope on timeout |
| `WithCache(ttl)` | In-memory GET response caching | `X-Cache: HIT` on cache hits |
| `WithSessions(name, secret)` | Cookie-backed server sessions | gin-contrib/sessions |
| `WithAuthz(enforcer)` | Casbin policy-based authorisation | Subject from HTTP Basic Auth |
| `WithHTTPSign(secrets, opts...)` | HTTP Signatures verification | draft-cavage-http-signatures |
| `WithSSE(broker)` | Server-Sent Events at `/events` | `?channel=` filtering |
| `WithLocation()` | Reverse proxy header detection | X-Forwarded-Proto / X-Forwarded-Host |
| `WithI18n(cfg...)` | Accept-Language locale detection | BCP 47 matching via x/text/language |
| `WithTracing(name, opts...)` | OpenTelemetry distributed tracing | otelgin + W3C traceparent |
| `WithGraphQL(schema, opts...)` | GraphQL endpoint | gqlgen; optional playground UI |

---

## 5. Response[T] Envelope

All API responses share a single generic envelope:

```go
type Response[T any] struct {
    Success bool   `json:"success"`
    Data    T      `json:"data,omitempty"`
    Error   *Error `json:"error,omitempty"`
    Meta    *Meta  `json:"meta,omitempty"`
}
```

Constructor helpers:

| Helper | Usage |
|--------|-------|
| `OK(data)` | Successful response with typed data |
| `Fail(code, message)` | Error response with code and message |
| `FailWithDetails(code, message, details)` | Error with additional detail payload |
| `Paginated(data, page, perPage, total)` | Successful response with pagination meta |

`Meta` carries `request_id`, `duration`, `page`, `per_page`, and `total`. `Error` carries `code`,
`message`, and an optional `details` field for structured validation errors.

---

## 6. Authentik Integration

The `WithAuthentik()` option installs a permissive identity middleware that runs on every
non-public request. It has two extraction paths:

**Path 1 — Forward-auth headers (TrustedProxy: true):**
When Traefik or another reverse proxy is configured with Authentik forward-auth, it injects
`X-authentik-username`, `X-authentik-email`, `X-authentik-name`, `X-authentik-uid`,
`X-authentik-groups` (pipe-separated), `X-authentik-entitlements`, and `X-authentik-jwt` headers.
The middleware reads these and populates an `AuthentikUser` in the Gin context.

**Path 2 — OIDC JWT validation:**
For direct API clients that present a `Bearer` token, the middleware validates the JWT against
the configured OIDC issuer. Providers are cached by issuer URL to avoid repeated discovery requests.

In both paths, if extraction fails the request continues unauthenticated (fail-open). Handlers
check identity with:

```go
user := api.GetUser(c)   // returns nil when unauthenticated
```

For protected routes, apply guards:

```go
rg.GET("/private", api.RequireAuth(), handler)          // 401 if no user
rg.GET("/admin",   api.RequireGroup("admins"), handler) // 403 if wrong group
```

`AuthentikConfig.PublicPaths` lists additional paths exempt from header extraction.
`/health` and paths starting with `/swagger` are always exempt.

---

## 7. WebSocket and SSE

### WebSocket

`WithWSHandler(h)` mounts any `http.Handler` at `GET /ws`. The intended pairing is a `go-ws` hub:

```go
hub := ws.NewHub()
go hub.Run(ctx)
engine, _ := api.New(api.WithWSHandler(hub.Handler()))
```

Groups implementing `StreamGroup` declare channel names, which `Engine.Channels()` aggregates.

### Server-Sent Events

`SSEBroker` manages persistent SSE connections. Clients connect to `GET /events` and optionally
subscribe to a named channel via `?channel=<name>`. Clients with no channel parameter receive
events on all channels.

```go
broker := api.NewSSEBroker()
engine, _ := api.New(api.WithSSE(broker))

// In a handler or background goroutine:
broker.Publish("deployments", "deploy.started", payload)
```

Each client has a 64-event buffer. Overflow events are dropped without blocking the publisher.
`SSEBroker.Drain()` signals all clients to disconnect on graceful shutdown.

---

## 8. GraphQL

`WithGraphQL()` mounts a gqlgen `ExecutableSchema` at `/graphql` (or a custom path via
`WithGraphQLPath()`). An optional `WithPlayground()` adds the interactive playground at
`{path}/playground`.

```go
engine, _ := api.New(
    api.WithGraphQL(schema,
        api.WithPlayground(),
        api.WithGraphQLPath("/gql"),
    ),
)
```

The endpoint accepts all HTTP methods (POST for queries and mutations, GET for playground
redirects and introspection).

---

## 9. OpenAPI Spec Generation

### SpecBuilder

`SpecBuilder` generates an OpenAPI 3.1 JSON document from registered route groups:

```go
builder := &api.SpecBuilder{
    Title:       "My API",
    Description: "...",
    Version:     "1.0.0",
}
data, _ := builder.Build(engine.Groups())
```

The built document includes:

- The `GET /health` endpoint under the `system` tag.
- One path entry per `RouteDescription` returned by `DescribableGroup.Describe()`.
- `#/components/schemas/Error` and `#/components/schemas/Meta` shared schemas.
- All response bodies wrapped in the `Response[T]` envelope schema.
- Tags derived from every registered group's `Name()`.

Groups that implement `RouteGroup` but not `DescribableGroup` contribute a tag but no paths.

### Export

Two convenience functions write the spec to an `io.Writer` or directly to a file:

```go
// Write JSON or YAML to a writer:
api.ExportSpec(os.Stdout, "yaml", builder, engine.Groups())

// Write to a file (parent dirs created automatically):
api.ExportSpecToFile("./api/openapi.yaml", "yaml", builder, engine.Groups())
```

### Swagger UI

When `WithSwagger()` is active, the spec is built lazily on first access by a `swaggerSpec`
wrapper and registered in the global `swag` registry with a unique sequence-based instance name.
Multiple `Engine` instances in the same process (common in tests) do not collide in the registry.

---

## 10. ToolBridge

`ToolBridge` converts tool descriptors — as produced by go-ai, go-ml, and similar packages — into
`POST /{tool_name}` REST endpoints, and implements `DescribableGroup` so those endpoints appear in
the generated OpenAPI spec.

```go
bridge := api.NewToolBridge("/v1/tools")
bridge.Add(api.ToolDescriptor{
    Name:        "file_read",
    Description: "Read a file",
    Group:       "files",
    InputSchema: map[string]any{"type": "object", "properties": ...},
}, fileReadHandler)

engine.Register(bridge)
```

Each registered tool becomes a `POST /v1/tools/{tool_name}` endpoint. `ToolBridge.Tools()` returns
the full list of descriptors, making it easy to enumerate tools from the outside.

---

## 11. SDK Codegen

`SDKGenerator` wraps `openapi-generator-cli` to generate client SDKs from an exported spec:

```go
gen := &api.SDKGenerator{
    SpecPath:    "./api/openapi.yaml",
    OutputDir:   "./sdk",
    PackageName: "myapi",
}
if gen.Available() {
    _ = gen.Generate(ctx, "typescript-fetch")
    _ = gen.Generate(ctx, "python")
}
```

Supported target languages: `go`, `typescript-fetch`, `typescript-axios`, `python`, `java`,
`csharp`, `ruby`, `swift`, `kotlin`, `rust`, `php`.

`SupportedLanguages()` returns the full list in sorted order. `SDKGenerator.Available()` reports
whether `openapi-generator-cli` is on `PATH` without running it.

---

## 12. In-Memory Cache

`WithCache(ttl)` installs a URL-keyed in-memory response cache scoped to GET requests. Successful
2xx responses are stored. Cached responses are served with an `X-Cache: HIT` header and bypass all
downstream handlers. Expired entries are evicted lazily on the next access for the same key. The
cache is not shared across `Engine` instances and has no size limit.

---

## 13. OpenTelemetry Tracing

`WithTracing(serviceName)` adds otelgin middleware that creates a span for each request, tagged
with HTTP method, route template, and response status code. Trace context is propagated via the
W3C `traceparent` header.

`NewTracerProvider(exporter)` is a convenience helper for tests and simple single-service
deployments that constructs a synchronous `TracerProvider` and installs it globally. Production
deployments should build a batching provider with appropriate resource attributes.

---

## 14. Internationalisation

`WithI18n(cfg)` parses the `Accept-Language` header on every request and stores the resolved BCP 47
locale tag in the Gin context. The `golang.org/x/text/language` matcher handles quality-weighted
negotiation and script/region subtag matching against the configured supported locales.

Handlers retrieve the locale and optional localised messages:

```go
locale := api.GetLocale(c)                    // e.g. "en", "fr"
msg, ok := api.GetMessage(c, "greeting")      // from configured Messages map
```

The built-in message map is a lightweight bridge. The `go-i18n` grammar engine is the intended
replacement for production-grade localisation.

---

## 15. Package Layout

```
go-api/
├── api.go           — Engine struct, New(), build(), Serve(), Handler(), Addr(), Channels()
├── options.go       — All With*() option functions
├── group.go         — RouteGroup, StreamGroup, DescribableGroup interfaces; RouteDescription
├── response.go      — Response[T], Error, Meta, OK(), Fail(), FailWithDetails(), Paginated()
├── middleware.go    — bearerAuthMiddleware(), requestIDMiddleware()
├── authentik.go     — AuthentikUser, AuthentikConfig, GetUser(), RequireAuth(), RequireGroup()
├── websocket.go     — wrapWSHandler() helper
├── sse.go           — SSEBroker, NewSSEBroker(), Publish(), Handler(), Drain()
├── cache.go         — cacheStore, cacheEntry, cacheWriter, cacheMiddleware()
├── brotli.go        — brotliHandler, newBrotliHandler(); BrotliDefault/BestSpeed/BestCompression constants
├── graphql.go       — graphqlConfig, GraphQLOption, WithPlayground(), WithGraphQLPath(), mountGraphQL()
├── i18n.go          — I18nConfig, WithI18n(), i18nMiddleware(), GetLocale(), GetMessage()
├── tracing.go       — WithTracing(), NewTracerProvider()
├── swagger.go       — swaggerSpec, registerSwagger(); swaggerSeq for multi-instance safety
├── openapi.go       — SpecBuilder, Build(), buildPaths(), buildTags(), envelopeSchema()
├── export.go        — ExportSpec(), ExportSpecToFile()
├── bridge.go        — ToolDescriptor, ToolBridge, NewToolBridge(), Add(), Describe(), Tools()
└── codegen.go       — SDKGenerator, Generate(), buildArgs(), Available(), SupportedLanguages()
```

---

## 16. Dependency Diagram

### Direct Dependencies

| Module | Role |
|--------|------|
| `github.com/gin-gonic/gin` | HTTP router and middleware engine |
| `github.com/gin-contrib/cors` | CORS policy middleware |
| `github.com/gin-contrib/secure` | Security headers |
| `github.com/gin-contrib/gzip` | Gzip compression |
| `github.com/gin-contrib/slog` | Structured logging |
| `github.com/gin-contrib/timeout` | Per-request deadlines |
| `github.com/gin-contrib/static` | Static file serving |
| `github.com/gin-contrib/sessions` | Cookie-backed sessions |
| `github.com/gin-contrib/authz` | Casbin authorisation |
| `github.com/gin-contrib/httpsign` | HTTP Signatures |
| `github.com/gin-contrib/location/v2` | Reverse proxy header detection |
| `github.com/gin-contrib/pprof` | Go profiling endpoints |
| `github.com/gin-contrib/expvar` | Runtime metrics endpoint |
| `github.com/casbin/casbin/v2` | Policy-based access control engine |
| `github.com/coreos/go-oidc/v3` | OIDC JWT validation for Authentik |
| `github.com/andybalholm/brotli` | Brotli compression |
| `github.com/gorilla/websocket` | WebSocket upgrade |
| `github.com/swaggo/gin-swagger` | Swagger UI handler |
| `github.com/swaggo/files` | Swagger UI static assets |
| `github.com/swaggo/swag` | Swagger spec registry |
| `github.com/99designs/gqlgen` | GraphQL schema execution |
| `go.opentelemetry.io/otel` | OpenTelemetry tracing |
| `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` | OTel + Gin |
| `golang.org/x/text/language` | BCP 47 language matching |
| `gopkg.in/yaml.v3` | YAML export of OpenAPI spec |

### What Imports go-api

go-api is imported by packages that need to expose REST endpoints. Within the Lethean ecosystem
the expected importers are:

```
Core CLI (forge.lthn.ai/core/go)
    └─ go-api  ← wires route groups from go-ai, go-ml, go-rag into an Engine

go-ai
    └─ go-api  ← ToolBridge converts MCP tool descriptors to REST endpoints

go-ml
    └─ go-api  ← may register inference/scoring route groups

Application main packages
    └─ go-api  ← top-level server bootstrap
```

go-api itself has no imports from other Lethean ecosystem modules. It is the stable base that
others import, not a consumer of them.