From 3ed0cf4907fe7861dc0284b9fa346291add05886 Mon Sep 17 00:00:00 2001 From: Snider Date: Wed, 11 Mar 2026 13:02:40 +0000 Subject: [PATCH] docs: add human-friendly documentation Co-Authored-By: Claude Opus 4.6 --- docs/architecture.md | 271 +++++++++++++++++++++++++++++++++++++++++++ docs/development.md | 197 +++++++++++++++++++++++++++++++ docs/index.md | 146 +++++++++++++++++++++++ 3 files changed, 614 insertions(+) create mode 100644 docs/architecture.md create mode 100644 docs/development.md create mode 100644 docs/index.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..f87d516 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,271 @@ +--- +title: Architecture +description: Internal design of go-container -- types, data flow, hypervisor abstraction, state management, and template engine. +--- + +# Architecture + +go-container is organised into three packages with clear responsibilities. The root `container` package owns the core abstractions. The `devenv` package composes those abstractions into a higher-level development environment. The `sources` package provides pluggable image download backends. + +``` +container (root) + |-- Manager interface + LinuxKitManager implementation + |-- Hypervisor interface (QEMU, Hyperkit) + |-- State (persistent container registry) + |-- Template engine (embedded + user templates) + | + +-- devenv/ + | |-- DevOps orchestrator + | |-- ImageManager (download, manifest, update checks) + | |-- Shell, Serve, Test, Claude sessions + | +-- Config (from ~/.core/config.yaml) + | + +-- sources/ + |-- ImageSource interface + |-- CDNSource + +-- GitHubSource +``` + + +## Key types + +### Container + +The central data structure representing a running or stopped VM instance. + +```go +type Container struct { + ID string `json:"id"` + Name string `json:"name,omitempty"` + Image string `json:"image"` + Status Status `json:"status"` + PID int `json:"pid"` + StartedAt time.Time `json:"started_at"` + Ports map[int]int `json:"ports,omitempty"` + Memory int `json:"memory,omitempty"` + CPUs int `json:"cpus,omitempty"` +} +``` + +Each container gets a unique 8-character hex ID generated from `crypto/rand`. Status transitions follow `running -> stopped | error`. + + +### Manager interface + +The `Manager` interface defines the contract for container lifecycle management: + +```go +type Manager interface { + Run(ctx context.Context, image string, opts RunOptions) (*Container, error) + Stop(ctx context.Context, id string) error + List(ctx context.Context) ([]*Container, error) + Logs(ctx context.Context, id string, follow bool) (io.ReadCloser, error) + Exec(ctx context.Context, id string, cmd []string) error +} +``` + +The only implementation is `LinuxKitManager`, which delegates VM execution to a `Hypervisor` and tracks state in a `State` store. + + +### Hypervisor interface + +Abstracts the underlying virtualisation technology: + +```go +type Hypervisor interface { + Name() string + Available() bool + BuildCommand(ctx context.Context, image string, opts *HypervisorOptions) (*exec.Cmd, error) +} +``` + +Two implementations exist: + +| Implementation | Platform | Acceleration | Binary | +|----------------|----------|-------------|--------| +| `QemuHypervisor` | All | KVM (Linux), HVF (macOS) | `qemu-system-x86_64` | +| `HyperkitHypervisor` | macOS only | Native macOS hypervisor | `hyperkit` | + +`DetectHypervisor()` auto-selects the best available hypervisor. On macOS it prefers Hyperkit, falling back to QEMU. On Linux it uses QEMU with KVM if `/dev/kvm` is present. + + +## Data flow: running a container + +When `LinuxKitManager.Run()` is called, the following sequence occurs: + +1. **Validate** -- Checks the image file exists via `io.Medium` and detects its format from the file extension (`.iso`, `.qcow2`, `.vmdk`, `.raw`, `.img`). + +2. **Generate ID** -- Creates an 8-character hex identifier using `crypto/rand`. + +3. **Apply defaults** -- Memory defaults to 1024 MB, CPUs to 1, SSH port to 2222. + +4. **Build command** -- Delegates to the `Hypervisor.BuildCommand()` method, which constructs the full command line including: + - Memory and CPU allocation + - Disk image attachment (format-specific flags) + - Network with port forwarding (SSH + user-defined ports) + - 9p volume shares (QEMU only) + - Hardware acceleration flags + +5. **Start process** -- In **detached** mode, stdout/stderr are redirected to a log file under `~/.core/logs/.log`, and a background goroutine monitors the process for exit. In **foreground** mode, output is tee'd to both the log file and the terminal. + +6. **Persist state** -- The container record is written to `~/.core/containers.json` via the `State` store. + +7. **Monitor** -- For detached containers, `waitForExit()` runs in a goroutine, updating the container status to `stopped` or `error` when the process terminates. + + +## State management + +The `State` struct provides a thread-safe, JSON-persisted container registry: + +```go +type State struct { + Containers map[string]*Container `json:"containers"` + mu sync.RWMutex + filePath string +} +``` + +Key design decisions: + +- **Copy-on-read**: `Get()` and `All()` return copies of container structs to prevent data races when callers modify the returned values. +- **Write-through**: Every mutation (`Add`, `Update`, `Remove`) immediately persists to disk via `SaveState()`. +- **Auto-create**: `LoadState()` returns an empty state if the file does not exist, and `SaveState()` creates parent directories as needed. + +Default paths: + +| Path | Purpose | +|------|---------| +| `~/.core/containers.json` | Container state file | +| `~/.core/logs/.log` | Per-container log files | + + +## Stopping a container + +`LinuxKitManager.Stop()` performs a graceful shutdown: + +1. Sends `SIGTERM` to the hypervisor process. +2. Waits up to 10 seconds for the process to exit. +3. If the process does not exit in time, sends `SIGKILL`. +4. Respects context cancellation throughout -- if the context is cancelled, the process is killed immediately. + + +## Template engine + +LinuxKit templates are YAML files that define a complete VM image configuration (kernel, init, services, files). The template engine adds variable substitution on top. + +### Variable syntax + +Two forms are supported: + +- `${VAR}` -- Required variable. Produces an error if not provided. +- `${VAR:-default}` -- Optional variable with a default value. + +### Resolution order + +`GetTemplate(name)` searches for templates in this order: + +1. **Embedded templates** -- Compiled into the binary via `//go:embed templates/*.yml`. These always take precedence. +2. **Workspace templates** -- `.core/linuxkit/` relative to the current working directory. +3. **User templates** -- `~/.core/linuxkit/` in the user's home directory. + +User-defined templates that share a name with a built-in template are ignored (built-ins win). + +### Variable extraction + +`ExtractVariables(content)` parses a template and returns two collections: + +- A sorted slice of required variable names (those using `${VAR}` syntax with no default). +- A map of optional variable names to their default values (those using `${VAR:-default}` syntax). + +This powers the `core vm templates vars ` command. + + +## Development environment (devenv) + +The `DevOps` struct in the `devenv` package composes the lower-level primitives into a complete development workflow. + +### Boot sequence + +1. Checks the dev image is installed (platform-specific qcow2 file: `core-devops-{os}-{arch}.qcow2`). +2. Launches the image via `LinuxKitManager.Run()` in detached mode with 4096 MB RAM, 2 CPUs, and SSH on port 2222. +3. Polls for up to 60 seconds until the VM's SSH host key can be scanned, then writes it to `~/.core/known_hosts`. + +### Shell access + +Two modes are available: + +- **SSH** (default) -- Connects via `ssh -A -p 2222 root@localhost` with agent forwarding and strict host key checking against `~/.core/known_hosts`. +- **Serial console** -- Attaches to the QEMU serial console socket via `socat`. + +### Project mounting + +Projects are mounted into the VM at `/app` using a reverse SSHFS tunnel. The VM opens an SSH reverse tunnel back to the host (port 10000) and mounts the host directory via SSHFS. + +### Auto-detection + +Several operations auto-detect the project type by inspecting files on disk: + +| File detected | Serve command | Test command | +|---------------|---------------|--------------| +| `artisan` | `php artisan octane:start` | -- | +| `package.json` with `dev` script | `npm run dev -- --host 0.0.0.0` | `npm test` | +| `composer.json` with `test` script | `frankenphp php-server` | `composer test` | +| `go.mod` | `go run .` | `go test ./...` | +| `manage.py` | `python manage.py runserver` | -- | +| `pytest.ini` or `pyproject.toml` | -- | `pytest` | + +Auto-detection can be overridden with `.core/test.yaml` for tests or explicit `--command` flags. + +### Claude integration + +`DevOps.Claude()` starts a sandboxed Claude session inside the VM: + +1. Auto-boots the dev environment if not running. +2. Mounts the project directory at `/app`. +3. Forwards authentication credentials (Anthropic API key, GitHub CLI config, SSH agent, git identity) based on configurable options. +4. Launches `claude` inside the VM via SSH with agent forwarding. + + +## Image sources (sources) + +The `ImageSource` interface defines how dev environment images are downloaded: + +```go +type ImageSource interface { + Name() string + Available() bool + LatestVersion(ctx context.Context) (string, error) + Download(ctx context.Context, m io.Medium, dest string, progress func(downloaded, total int64)) error +} +``` + +| Source | Backend | Availability check | +|--------|---------|--------------------| +| `CDNSource` | HTTP GET from a configured CDN URL | CDN URL is configured | +| `GitHubSource` | `gh release download` via the GitHub CLI | `gh` is installed and authenticated | + +The `ImageManager` in `devenv` maintains a `manifest.json` in `~/.core/images/` that tracks installed image versions, SHA256 checksums, download timestamps, and source names. When the source is set to `"auto"` (the default), it tries GitHub first, then CDN. + + +## Configuration + +The `devenv` package reads `~/.core/config.yaml` via the `go-config` library: + +```yaml +version: 1 +images: + source: auto # auto | github | cdn + github: + repo: host-uk/core-images + registry: + image: ghcr.io/host-uk/core-devops + cdn: + url: https://cdn.example.com/images +``` + +If the file does not exist, sensible defaults are used. + + +## Licence + +EUPL-1.2 diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..a85ed64 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,197 @@ +--- +title: Development +description: How to build, test, and contribute to go-container. +--- + +# Development + +## Prerequisites + +- **Go 1.26+** -- The module uses Go 1.26 features. +- **Go workspace** -- This module is part of a Go workspace at `~/Code/go.work`. Local development of sibling modules (go-io, go-config, go-i18n, cli) requires the workspace file. + +Optional (for actually running VMs): + +- **QEMU** -- `qemu-system-x86_64` for running LinuxKit images on any platform. +- **Hyperkit** -- macOS-only alternative hypervisor. +- **LinuxKit** -- For building images from templates (`linuxkit build`). +- **GitHub CLI** (`gh`) -- For the GitHub image source. + + +## Running tests + +```bash +# All tests +go test ./... + +# With race detector +go test -race ./... + +# Single test by name +go test -run TestState_Add_Good ./... + +# Single package +go test ./sources/ +go test ./devenv/ +``` + +Tests use `testify` for assertions. Most tests are self-contained and do not require a running hypervisor -- they test command construction, state management, template parsing, and configuration loading in isolation. + + +## Test naming convention + +Tests follow a `_Good`, `_Bad`, `_Ugly` suffix pattern: + +| Suffix | Meaning | +|--------|---------| +| `_Good` | Happy path -- valid inputs, expected success | +| `_Bad` | Expected error conditions -- invalid inputs, missing resources | +| `_Ugly` | Edge cases, panics, and boundary conditions | + +Examples from the codebase: + +```go +func TestNewState_Good(t *testing.T) { /* creates state successfully */ } +func TestLoadState_Bad_InvalidJSON(t *testing.T) { /* handles corrupt state file */ } +func TestGetHypervisor_Bad_Unknown(t *testing.T) { /* rejects unknown hypervisor name */ } +``` + + +## Project structure + +``` +go-container/ + container.go # Container struct, Manager interface, Status, RunOptions, ImageFormat + hypervisor.go # Hypervisor interface, QemuHypervisor, HyperkitHypervisor, DetectHypervisor + hypervisor_test.go + linuxkit.go # LinuxKitManager (Manager implementation), followReader for log tailing + linuxkit_test.go + state.go # State persistence (containers.json), log paths + state_test.go + templates.go # Template listing, loading, variable substitution, user template scanning + templates_test.go + templates/ # Embedded LinuxKit YAML templates + core-dev.yml + server-php.yml + sources/ + source.go # ImageSource interface, SourceConfig + source_test.go + cdn.go # CDNSource implementation + cdn_test.go + github.go # GitHubSource implementation + github_test.go + devenv/ + devops.go # DevOps orchestrator, Boot, Stop, Status, ImageName, ImagePath + devops_test.go + config.go # Config, ImagesConfig, LoadConfig from ~/.core/config.yaml + config_test.go + images.go # ImageManager, Manifest, Install, CheckUpdate + images_test.go + shell.go # Shell (SSH and serial console) + shell_test.go + serve.go # Serve (mount project, auto-detect serve command) + serve_test.go + test.go # Test (auto-detect test command, .core/test.yaml) + test_test.go + claude.go # Claude sandbox session with auth forwarding + claude_test.go + ssh_utils.go # Host key scanning for ~/.core/known_hosts + cmd/vm/ + cmd_vm.go # CLI registration (init + AddVMCommands) + cmd_commands.go # Package doc + cmd_container.go # run, ps, stop, logs, exec commands + cmd_templates.go # templates, templates show, templates vars commands +``` + + +## Coding standards + +- **UK English** in all strings, comments, and documentation (colour, organisation, honour). +- **Strict typing** -- All function parameters and return values are typed. No `interface{}` without justification. +- **Error wrapping** -- Use `fmt.Errorf("context: %w", err)` for all error returns. +- **`io.Medium` abstraction** -- File system operations go through `io.Medium` (from `go-io`) rather than directly calling `os` functions. This enables testing with mock file systems. The `io.Local` singleton is used for real file system access. +- **Compile-time interface checks** -- Use `var _ Interface = (*Impl)(nil)` to verify implementations at compile time (see `sources/cdn.go` and `sources/github.go`). +- **Context propagation** -- All operations that might block accept a `context.Context` as their first parameter. + + +## Adding a new hypervisor + +1. Create a new struct implementing the `Hypervisor` interface in `hypervisor.go`: + +```go +type MyHypervisor struct { + Binary string +} + +func (h *MyHypervisor) Name() string { return "my-hypervisor" } +func (h *MyHypervisor) Available() bool { /* check if binary exists */ } +func (h *MyHypervisor) BuildCommand(ctx context.Context, image string, opts *HypervisorOptions) (*exec.Cmd, error) { + // Build and return exec.Cmd +} +``` + +2. Register it in `DetectHypervisor()` and `GetHypervisor()` in `hypervisor.go`. + +3. Add tests following the `_Good`/`_Bad` naming convention. + + +## Adding a new image source + +1. Create a new struct implementing `ImageSource` in the `sources/` package: + +```go +type MySource struct { + config SourceConfig +} + +var _ ImageSource = (*MySource)(nil) // Compile-time check + +func (s *MySource) Name() string { return "my-source" } +func (s *MySource) Available() bool { /* check prerequisites */ } +func (s *MySource) LatestVersion(ctx context.Context) (string, error) { /* fetch version */ } +func (s *MySource) Download(ctx context.Context, m io.Medium, dest string, progress func(downloaded, total int64)) error { + // Download image to dest +} +``` + +2. Wire it into `NewImageManager()` in `devenv/images.go` under the appropriate source selector. + + +## Adding a new LinuxKit template + +### Built-in template + +1. Create a `.yml` file in the `templates/` directory. +2. Add an entry to `builtinTemplates` in `templates.go`. +3. The file will be embedded via the `//go:embed templates/*.yml` directive. + +### User template + +Place a `.yml` file in either: + +- `.core/linuxkit/` relative to your project root (workspace-scoped) +- `~/.core/linuxkit/` in your home directory (global) + +The first comment line in the YAML file is extracted as the template description. + + +## File system paths + +All persistent data lives under `~/.core/`: + +| Path | Purpose | +|------|---------| +| `~/.core/containers.json` | Container state registry | +| `~/.core/logs/` | Per-container log files | +| `~/.core/images/` | Downloaded dev environment images | +| `~/.core/images/manifest.json` | Image version manifest | +| `~/.core/config.yaml` | Global configuration | +| `~/.core/known_hosts` | SSH host keys for dev VMs | +| `~/.core/linuxkit/` | User-defined LinuxKit templates | + +The `CORE_IMAGES_DIR` environment variable overrides the default images directory. + + +## Licence + +EUPL-1.2 diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..1afee3b --- /dev/null +++ b/docs/index.md @@ -0,0 +1,146 @@ +--- +title: go-container +description: Container runtime, LinuxKit image builder, and portable development environment management for Go. +--- + +# go-container + +`forge.lthn.ai/core/go-container` provides a container runtime built on LinuxKit and lightweight hypervisors. It manages the full lifecycle of LinuxKit virtual machines -- from building images with embedded templates, to running them via QEMU or Hyperkit, to offering a portable development environment with shell access, project mounting, test execution, and Claude AI integration. + +This is **not** a Docker wrapper. It runs real VMs from LinuxKit images (ISO, qcow2, VMDK, raw) using platform-native acceleration (KVM on Linux, HVF on macOS, Hyperkit where available). + + +## Module path + +``` +forge.lthn.ai/core/go-container +``` + +Requires **Go 1.26+**. + + +## Quick start + +### Run a VM from an image + +```go +import ( + "context" + container "forge.lthn.ai/core/go-container" + "forge.lthn.ai/core/go-io" +) + +manager, err := container.NewLinuxKitManager(io.Local) +if err != nil { + log.Fatal(err) +} + +ctx := context.Background() +c, err := manager.Run(ctx, "/path/to/image.qcow2", container.RunOptions{ + Name: "my-vm", + Memory: 2048, + CPUs: 2, + SSHPort: 2222, + Detach: true, +}) +if err != nil { + log.Fatal(err) +} + +fmt.Printf("Started container %s (PID %d)\n", c.ID, c.PID) +``` + +### Use the development environment + +```go +import ( + "forge.lthn.ai/core/go-container/devenv" + "forge.lthn.ai/core/go-io" +) + +dev, err := devenv.New(io.Local) +if err != nil { + log.Fatal(err) +} + +// Boot the dev environment (downloads image if needed) +ctx := context.Background() +err = dev.Boot(ctx, devenv.DefaultBootOptions()) + +// Open an SSH shell +err = dev.Shell(ctx, devenv.ShellOptions{}) + +// Run tests inside the VM +err = dev.Test(ctx, "/path/to/project", devenv.TestOptions{}) +``` + +### Build and run from a LinuxKit template + +```go +import container "forge.lthn.ai/core/go-container" + +// List available templates (built-in + user-defined) +templates := container.ListTemplates() + +// Apply variables to a template +content, err := container.ApplyTemplate("core-dev", map[string]string{ + "SSH_KEY": "ssh-ed25519 AAAA...", + "MEMORY": "4096", + "HOSTNAME": "my-dev-box", +}) +``` + + +## Package layout + +| Package | Import path | Purpose | +|---------|-------------|---------| +| `container` (root) | `forge.lthn.ai/core/go-container` | Container struct, Manager interface, hypervisor abstraction, LinuxKit manager, state persistence, template engine | +| `devenv` | `forge.lthn.ai/core/go-container/devenv` | Portable dev environment orchestration: boot, shell, serve, test, Claude sandbox, image management | +| `sources` | `forge.lthn.ai/core/go-container/sources` | Image download backends: CDN and GitHub Releases with progress reporting | +| `cmd/vm` | `forge.lthn.ai/core/go-container/cmd/vm` | CLI commands (`core vm run`, `core vm ps`, `core vm stop`, `core vm logs`, `core vm exec`, `core vm templates`) | + + +## Dependencies + +| Module | Purpose | +|--------|---------| +| `forge.lthn.ai/core/go-io` | File system abstraction (`Medium` interface), process utilities | +| `forge.lthn.ai/core/go-config` | Configuration loading (used by `devenv` for `~/.core/config.yaml`) | +| `forge.lthn.ai/core/go-i18n` | Internationalised UI strings (used by `cmd/vm`) | +| `forge.lthn.ai/core/cli` | CLI framework (used by `cmd/vm` for command registration) | +| `github.com/stretchr/testify` | Test assertions | +| `gopkg.in/yaml.v3` | YAML parsing for test configuration | + +The root `container` package has only two direct dependencies: `go-io` and the standard library. The `devenv` and `cmd/vm` packages pull in the heavier dependencies. + + +## CLI commands + +When registered via `cmd/vm`, the following commands become available under `core vm`: + +| Command | Description | +|---------|-------------| +| `core vm run [image]` | Run a VM from an image file or `--template` | +| `core vm ps` | List running VMs (`-a` for all including stopped) | +| `core vm stop ` | Stop a running VM by ID or name (supports partial matching) | +| `core vm logs ` | View VM logs (`-f` to follow) | +| `core vm exec ` | Execute a command inside the VM via SSH | +| `core vm templates` | List available LinuxKit templates | +| `core vm templates show ` | Display a template's full YAML | +| `core vm templates vars ` | Show a template's required and optional variables | + + +## Built-in templates + +Two LinuxKit templates are embedded in the binary: + +- **core-dev** -- Full development environment with Go, Node.js, PHP, Docker-in-LinuxKit, and SSH access +- **server-php** -- Production PHP server with FrankenPHP, Caddy reverse proxy, and health checks + +User-defined templates can be placed in `.core/linuxkit/` (workspace-relative) or `~/.core/linuxkit/` (global). They are discovered automatically and merged with the built-in set. + + +## Licence + +EUPL-1.2. See [LICENSE](../LICENSE) for the full text.