Snider 0d3479839d docs: add architecture, development, and history documentation

Standard docs trio matching the pattern used across all Go ecosystem repos.
Covers Engine architecture, 21 With*() options, RouteGroup interfaces,
OpenAPI generation pipeline, and three-phase development history (176 tests).

Co-Authored-By: Virgil <virgil@lethean.io>

2026-02-21 01:57:44 +00:00

18 KiB

Raw Blame History

go-api Architecture

Module: forge.lthn.ai/core/go-api Language: Go 1.25 Licence: EUPL-1.2 LOC: ~2.1 K non-test, 176 tests passing

1. Overview

go-api is the Gin-based REST framework for the Lethean Go ecosystem. Subsystems across the ecosystem plug into a central Engine via the RouteGroup interface, and go-api handles the HTTP plumbing: middleware composition, response envelopes, WebSocket and SSE integration, GraphQL, Authentik identity, OpenAPI spec generation, and SDK codegen.

Position in the Lethean Stack

AI Clients / Browsers / SDK consumers
          |  HTTP / WebSocket / SSE
          v
  [ go-api Engine ]                ← this module
      |         |         |
      |         |         └─ OpenAPI spec → SDKGenerator → openapi-generator-cli
      |         └─ ToolBridge ──→ go-ai / go-ml / go-rag route groups
      └─ RouteGroups ─────────→ any package that implements RouteGroup

  Core CLI or standalone main() bootstraps and wires everything

go-api is a library. It contains no main package. Callers construct an Engine, register route groups, and call Serve().

2. Engine

2.1 The Engine Struct

Engine is the central container. It owns the listen address, the ordered list of registered route groups, the middleware chain, and all optional integrations:

type Engine struct {
    addr           string
    groups         []RouteGroup
    middlewares    []gin.HandlerFunc
    wsHandler      http.Handler
    sseBroker      *SSEBroker
    swaggerEnabled bool
    swaggerTitle   string
    swaggerDesc    string
    swaggerVersion string
    pprofEnabled   bool
    expvarEnabled  bool
    graphql        *graphqlConfig
}

2.2 Construction

New() applies functional options and returns a configured engine. The default listen address is :8080. No middleware is added automatically beyond Gin's built-in panic recovery; every feature requires an explicit With*() option.

engine, err := api.New(
    api.WithAddr(":9000"),
    api.WithBearerAuth("secret"),
    api.WithCORS("*"),
    api.WithRequestID(),
    api.WithSlog(nil),
    api.WithSwagger("My API", "Description", "1.0.0"),
)
engine.Register(myRoutes)
engine.Serve(ctx)

2.3 Build Sequence

Engine.build() (called internally by Handler() and Serve()) assembles a fresh *gin.Engine each time. The order is fixed:

gin.Recovery() — panic recovery (always first).
User middleware in registration order — all With*() options that append to e.middlewares.
Built-in GET /health endpoint — always present.
Route groups — each mounted at its BasePath().
WebSocket handler at GET /ws — when WithWSHandler() was called.
SSE broker at GET /events — when WithSSE() was called.
GraphQL endpoint — when WithGraphQL() was called.
Swagger UI at GET /swagger/*any — when WithSwagger() was called.
pprof endpoints at GET /debug/pprof/* — when WithPprof() was called.
expvar endpoint at GET /debug/vars — when WithExpvar() was called.

2.4 Graceful Shutdown

Serve() starts an http.Server in a goroutine and blocks on ctx.Done(). When the context is cancelled, a 10-second Shutdown deadline is applied. In-flight requests complete or time out before the server exits.

3. RouteGroup / DescribableGroup Interfaces

Three interfaces form the extension point model:

// RouteGroup is the minimum interface. All subsystems implement this.
type RouteGroup interface {
    Name() string
    BasePath() string
    RegisterRoutes(rg *gin.RouterGroup)
}

// StreamGroup is optionally implemented by groups that publish WebSocket channels.
type StreamGroup interface {
    Channels() []string
}

// DescribableGroup extends RouteGroup with OpenAPI metadata.
// Groups implementing this will appear in the generated spec.
type DescribableGroup interface {
    RouteGroup
    Describe() []RouteDescription
}

RouteDescription carries the HTTP method, path, summary, description, tags, and JSON Schema maps for the request body and response data. The SpecBuilder consumes these to generate the OpenAPI 3.1 paths object.

Engine.Channels() iterates all registered groups and collects channel names from those that implement StreamGroup. This list is used when initialising a WebSocket hub.

4. Middleware Stack — 21 With*() Options

All middleware options append to Engine.middlewares in the order they are passed to New(). They are applied after gin.Recovery() but before any route handler.

Option	Purpose	Key detail
`WithAddr(addr)`	Listen address	Default `:8080`
`WithBearerAuth(token)`	Static bearer token auth	Skips `/health` and `/swagger`
`WithRequestID()`	X-Request-ID propagation	Preserves client-supplied IDs
`WithCORS(origins...)`	CORS policy	`"*"` enables `AllowAllOrigins`; 12-hour `MaxAge`
`WithMiddleware(mw...)`	Arbitrary Gin middleware	Escape hatch
`WithStatic(prefix, root)`	Static file serving	Directory listing disabled
`WithWSHandler(h)`	WebSocket at `/ws`	Wraps any `http.Handler`
`WithAuthentik(cfg)`	Authentik forward-auth + OIDC	Permissive; sets context, never rejects
`WithSwagger(title, desc, ver)`	Swagger UI at `/swagger/`	Runtime spec via `SpecBuilder`
`WithPprof()`	Go profiling at `/debug/pprof/`	WARNING: do not expose in production
`WithExpvar()`	Runtime metrics at `/debug/vars`	WARNING: do not expose in production
`WithSecure()`	Security headers	HSTS 1yr, X-Frame-Options DENY, nosniff
`WithGzip(level...)`	Gzip response compression	Default compression if level omitted
`WithBrotli(level...)`	Brotli response compression	Default compression if level omitted
`WithSlog(logger)`	Structured request logging	Falls back to `slog.Default()` if nil
`WithTimeout(d)`	Per-request deadline	504 error envelope on timeout
`WithCache(ttl)`	In-memory GET response caching	`X-Cache: HIT` on cache hits
`WithSessions(name, secret)`	Cookie-backed server sessions	gin-contrib/sessions
`WithAuthz(enforcer)`	Casbin policy-based authorisation	Subject from HTTP Basic Auth
`WithHTTPSign(secrets, opts...)`	HTTP Signatures verification	draft-cavage-http-signatures
`WithSSE(broker)`	Server-Sent Events at `/events`	`?channel=` filtering
`WithLocation()`	Reverse proxy header detection	X-Forwarded-Proto / X-Forwarded-Host
`WithI18n(cfg...)`	Accept-Language locale detection	BCP 47 matching via x/text/language
`WithTracing(name, opts...)`	OpenTelemetry distributed tracing	otelgin + W3C traceparent
`WithGraphQL(schema, opts...)`	GraphQL endpoint	gqlgen; optional playground UI

5. Response[T] Envelope

All API responses share a single generic envelope:

type Response[T any] struct {
    Success bool   `json:"success"`
    Data    T      `json:"data,omitempty"`
    Error   *Error `json:"error,omitempty"`
    Meta    *Meta  `json:"meta,omitempty"`
}

Constructor helpers:

Helper	Usage
`OK(data)`	Successful response with typed data
`Fail(code, message)`	Error response with code and message
`FailWithDetails(code, message, details)`	Error with additional detail payload
`Paginated(data, page, perPage, total)`	Successful response with pagination meta

Meta carries request_id, duration, page, per_page, and total. Error carries code, message, and an optional details field for structured validation errors.

6. Authentik Integration

The WithAuthentik() option installs a permissive identity middleware that runs on every non-public request. It has two extraction paths:

Path 1 — Forward-auth headers (TrustedProxy: true): When Traefik or another reverse proxy is configured with Authentik forward-auth, it injects X-authentik-username, X-authentik-email, X-authentik-name, X-authentik-uid, X-authentik-groups (pipe-separated), X-authentik-entitlements, and X-authentik-jwt headers. The middleware reads these and populates an AuthentikUser in the Gin context.

Path 2 — OIDC JWT validation: For direct API clients that present a Bearer token, the middleware validates the JWT against the configured OIDC issuer. Providers are cached by issuer URL to avoid repeated discovery requests.

In both paths, if extraction fails the request continues unauthenticated (fail-open). Handlers check identity with:

user := api.GetUser(c)   // returns nil when unauthenticated

For protected routes, apply guards:

rg.GET("/private", api.RequireAuth(), handler)          // 401 if no user
rg.GET("/admin",   api.RequireGroup("admins"), handler) // 403 if wrong group

AuthentikConfig.PublicPaths lists additional paths exempt from header extraction. /health and paths starting with /swagger are always exempt.

7. WebSocket and SSE

WebSocket

WithWSHandler(h) mounts any http.Handler at GET /ws. The intended pairing is a go-ws hub:

hub := ws.NewHub()
go hub.Run(ctx)
engine, _ := api.New(api.WithWSHandler(hub.Handler()))

Groups implementing StreamGroup declare channel names, which Engine.Channels() aggregates.

Server-Sent Events

SSEBroker manages persistent SSE connections. Clients connect to GET /events and optionally subscribe to a named channel via ?channel=<name>. Clients with no channel parameter receive events on all channels.

broker := api.NewSSEBroker()
engine, _ := api.New(api.WithSSE(broker))

// In a handler or background goroutine:
broker.Publish("deployments", "deploy.started", payload)

Each client has a 64-event buffer. Overflow events are dropped without blocking the publisher. SSEBroker.Drain() signals all clients to disconnect on graceful shutdown.

8. GraphQL

WithGraphQL() mounts a gqlgen ExecutableSchema at /graphql (or a custom path via WithGraphQLPath()). An optional WithPlayground() adds the interactive playground at {path}/playground.

engine, _ := api.New(
    api.WithGraphQL(schema,
        api.WithPlayground(),
        api.WithGraphQLPath("/gql"),
    ),
)

The endpoint accepts all HTTP methods (POST for queries and mutations, GET for playground redirects and introspection).

9. OpenAPI Spec Generation

SpecBuilder

SpecBuilder generates an OpenAPI 3.1 JSON document from registered route groups:

builder := &api.SpecBuilder{
    Title:       "My API",
    Description: "...",
    Version:     "1.0.0",
}
data, _ := builder.Build(engine.Groups())

The built document includes:

The GET /health endpoint under the system tag.
One path entry per RouteDescription returned by DescribableGroup.Describe().
#/components/schemas/Error and #/components/schemas/Meta shared schemas.
All response bodies wrapped in the Response[T] envelope schema.
Tags derived from every registered group's Name().

Groups that implement RouteGroup but not DescribableGroup contribute a tag but no paths.

Export

Two convenience functions write the spec to an io.Writer or directly to a file:

// Write JSON or YAML to a writer:
api.ExportSpec(os.Stdout, "yaml", builder, engine.Groups())

// Write to a file (parent dirs created automatically):
api.ExportSpecToFile("./api/openapi.yaml", "yaml", builder, engine.Groups())

Swagger UI

When WithSwagger() is active, the spec is built lazily on first access by a swaggerSpec wrapper and registered in the global swag registry with a unique sequence-based instance name. Multiple Engine instances in the same process (common in tests) do not collide in the registry.

10. ToolBridge

ToolBridge converts tool descriptors — as produced by go-ai, go-ml, and similar packages — into POST /{tool_name} REST endpoints, and implements DescribableGroup so those endpoints appear in the generated OpenAPI spec.

bridge := api.NewToolBridge("/v1/tools")
bridge.Add(api.ToolDescriptor{
    Name:        "file_read",
    Description: "Read a file",
    Group:       "files",
    InputSchema: map[string]any{"type": "object", "properties": ...},
}, fileReadHandler)

engine.Register(bridge)

Each registered tool becomes a POST /v1/tools/{tool_name} endpoint. ToolBridge.Tools() returns the full list of descriptors, making it easy to enumerate tools from the outside.

11. SDK Codegen

SDKGenerator wraps openapi-generator-cli to generate client SDKs from an exported spec:

gen := &api.SDKGenerator{
    SpecPath:    "./api/openapi.yaml",
    OutputDir:   "./sdk",
    PackageName: "myapi",
}
if gen.Available() {
    _ = gen.Generate(ctx, "typescript-fetch")
    _ = gen.Generate(ctx, "python")
}

Supported target languages: go, typescript-fetch, typescript-axios, python, java, csharp, ruby, swift, kotlin, rust, php.

SupportedLanguages() returns the full list in sorted order. SDKGenerator.Available() reports whether openapi-generator-cli is on PATH without running it.

12. In-Memory Cache

WithCache(ttl) installs a URL-keyed in-memory response cache scoped to GET requests. Successful 2xx responses are stored. Cached responses are served with an X-Cache: HIT header and bypass all downstream handlers. Expired entries are evicted lazily on the next access for the same key. The cache is not shared across Engine instances and has no size limit.

13. OpenTelemetry Tracing

WithTracing(serviceName) adds otelgin middleware that creates a span for each request, tagged with HTTP method, route template, and response status code. Trace context is propagated via the W3C traceparent header.

NewTracerProvider(exporter) is a convenience helper for tests and simple single-service deployments that constructs a synchronous TracerProvider and installs it globally. Production deployments should build a batching provider with appropriate resource attributes.

14. Internationalisation

WithI18n(cfg) parses the Accept-Language header on every request and stores the resolved BCP 47 locale tag in the Gin context. The golang.org/x/text/language matcher handles quality-weighted negotiation and script/region subtag matching against the configured supported locales.

Handlers retrieve the locale and optional localised messages:

locale := api.GetLocale(c)                    // e.g. "en", "fr"
msg, ok := api.GetMessage(c, "greeting")      // from configured Messages map

The built-in message map is a lightweight bridge. The go-i18n grammar engine is the intended replacement for production-grade localisation.

15. Package Layout

go-api/
├── api.go           — Engine struct, New(), build(), Serve(), Handler(), Addr(), Channels()
├── options.go       — All With*() option functions
├── group.go         — RouteGroup, StreamGroup, DescribableGroup interfaces; RouteDescription
├── response.go      — Response[T], Error, Meta, OK(), Fail(), FailWithDetails(), Paginated()
├── middleware.go    — bearerAuthMiddleware(), requestIDMiddleware()
├── authentik.go     — AuthentikUser, AuthentikConfig, GetUser(), RequireAuth(), RequireGroup()
├── websocket.go     — wrapWSHandler() helper
├── sse.go           — SSEBroker, NewSSEBroker(), Publish(), Handler(), Drain()
├── cache.go         — cacheStore, cacheEntry, cacheWriter, cacheMiddleware()
├── brotli.go        — brotliHandler, newBrotliHandler(); BrotliDefault/BestSpeed/BestCompression constants
├── graphql.go       — graphqlConfig, GraphQLOption, WithPlayground(), WithGraphQLPath(), mountGraphQL()
├── i18n.go          — I18nConfig, WithI18n(), i18nMiddleware(), GetLocale(), GetMessage()
├── tracing.go       — WithTracing(), NewTracerProvider()
├── swagger.go       — swaggerSpec, registerSwagger(); swaggerSeq for multi-instance safety
├── openapi.go       — SpecBuilder, Build(), buildPaths(), buildTags(), envelopeSchema()
├── export.go        — ExportSpec(), ExportSpecToFile()
├── bridge.go        — ToolDescriptor, ToolBridge, NewToolBridge(), Add(), Describe(), Tools()
└── codegen.go       — SDKGenerator, Generate(), buildArgs(), Available(), SupportedLanguages()

16. Dependency Diagram

Direct Dependencies

Module	Role
`github.com/gin-gonic/gin`	HTTP router and middleware engine
`github.com/gin-contrib/cors`	CORS policy middleware
`github.com/gin-contrib/secure`	Security headers
`github.com/gin-contrib/gzip`	Gzip compression
`github.com/gin-contrib/slog`	Structured logging
`github.com/gin-contrib/timeout`	Per-request deadlines
`github.com/gin-contrib/static`	Static file serving
`github.com/gin-contrib/sessions`	Cookie-backed sessions
`github.com/gin-contrib/authz`	Casbin authorisation
`github.com/gin-contrib/httpsign`	HTTP Signatures
`github.com/gin-contrib/location/v2`	Reverse proxy header detection
`github.com/gin-contrib/pprof`	Go profiling endpoints
`github.com/gin-contrib/expvar`	Runtime metrics endpoint
`github.com/casbin/casbin/v2`	Policy-based access control engine
`github.com/coreos/go-oidc/v3`	OIDC JWT validation for Authentik
`github.com/andybalholm/brotli`	Brotli compression
`github.com/gorilla/websocket`	WebSocket upgrade
`github.com/swaggo/gin-swagger`	Swagger UI handler
`github.com/swaggo/files`	Swagger UI static assets
`github.com/swaggo/swag`	Swagger spec registry
`github.com/99designs/gqlgen`	GraphQL schema execution
`go.opentelemetry.io/otel`	OpenTelemetry tracing
`go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin`	OTel + Gin
`golang.org/x/text/language`	BCP 47 language matching
`gopkg.in/yaml.v3`	YAML export of OpenAPI spec

What Imports go-api

go-api is imported by packages that need to expose REST endpoints. Within the Lethean ecosystem the expected importers are:

Core CLI (forge.lthn.ai/core/go)
    └─ go-api  ← wires route groups from go-ai, go-ml, go-rag into an Engine

go-ai
    └─ go-api  ← ToolBridge converts MCP tool descriptors to REST endpoints

go-ml
    └─ go-api  ← may register inference/scoring route groups

Application main packages
    └─ go-api  ← top-level server bootstrap

go-api itself has no imports from other Lethean ecosystem modules. It is the stable base that others import, not a consumer of them.

18 KiB Raw Blame History