docs: add human-friendly documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Snider 2026-03-11 13:02:40 +00:00
parent a8e09bb03c
commit 3ed0cf4907
3 changed files with 614 additions and 0 deletions

271
docs/architecture.md Normal file
View file

@ -0,0 +1,271 @@
---
title: Architecture
description: Internal design of go-container -- types, data flow, hypervisor abstraction, state management, and template engine.
---
# Architecture
go-container is organised into three packages with clear responsibilities. The root `container` package owns the core abstractions. The `devenv` package composes those abstractions into a higher-level development environment. The `sources` package provides pluggable image download backends.
```
container (root)
|-- Manager interface + LinuxKitManager implementation
|-- Hypervisor interface (QEMU, Hyperkit)
|-- State (persistent container registry)
|-- Template engine (embedded + user templates)
|
+-- devenv/
| |-- DevOps orchestrator
| |-- ImageManager (download, manifest, update checks)
| |-- Shell, Serve, Test, Claude sessions
| +-- Config (from ~/.core/config.yaml)
|
+-- sources/
|-- ImageSource interface
|-- CDNSource
+-- GitHubSource
```
## Key types
### Container
The central data structure representing a running or stopped VM instance.
```go
type Container struct {
ID string `json:"id"`
Name string `json:"name,omitempty"`
Image string `json:"image"`
Status Status `json:"status"`
PID int `json:"pid"`
StartedAt time.Time `json:"started_at"`
Ports map[int]int `json:"ports,omitempty"`
Memory int `json:"memory,omitempty"`
CPUs int `json:"cpus,omitempty"`
}
```
Each container gets a unique 8-character hex ID generated from `crypto/rand`. Status transitions follow `running -> stopped | error`.
### Manager interface
The `Manager` interface defines the contract for container lifecycle management:
```go
type Manager interface {
Run(ctx context.Context, image string, opts RunOptions) (*Container, error)
Stop(ctx context.Context, id string) error
List(ctx context.Context) ([]*Container, error)
Logs(ctx context.Context, id string, follow bool) (io.ReadCloser, error)
Exec(ctx context.Context, id string, cmd []string) error
}
```
The only implementation is `LinuxKitManager`, which delegates VM execution to a `Hypervisor` and tracks state in a `State` store.
### Hypervisor interface
Abstracts the underlying virtualisation technology:
```go
type Hypervisor interface {
Name() string
Available() bool
BuildCommand(ctx context.Context, image string, opts *HypervisorOptions) (*exec.Cmd, error)
}
```
Two implementations exist:
| Implementation | Platform | Acceleration | Binary |
|----------------|----------|-------------|--------|
| `QemuHypervisor` | All | KVM (Linux), HVF (macOS) | `qemu-system-x86_64` |
| `HyperkitHypervisor` | macOS only | Native macOS hypervisor | `hyperkit` |
`DetectHypervisor()` auto-selects the best available hypervisor. On macOS it prefers Hyperkit, falling back to QEMU. On Linux it uses QEMU with KVM if `/dev/kvm` is present.
## Data flow: running a container
When `LinuxKitManager.Run()` is called, the following sequence occurs:
1. **Validate** -- Checks the image file exists via `io.Medium` and detects its format from the file extension (`.iso`, `.qcow2`, `.vmdk`, `.raw`, `.img`).
2. **Generate ID** -- Creates an 8-character hex identifier using `crypto/rand`.
3. **Apply defaults** -- Memory defaults to 1024 MB, CPUs to 1, SSH port to 2222.
4. **Build command** -- Delegates to the `Hypervisor.BuildCommand()` method, which constructs the full command line including:
- Memory and CPU allocation
- Disk image attachment (format-specific flags)
- Network with port forwarding (SSH + user-defined ports)
- 9p volume shares (QEMU only)
- Hardware acceleration flags
5. **Start process** -- In **detached** mode, stdout/stderr are redirected to a log file under `~/.core/logs/<id>.log`, and a background goroutine monitors the process for exit. In **foreground** mode, output is tee'd to both the log file and the terminal.
6. **Persist state** -- The container record is written to `~/.core/containers.json` via the `State` store.
7. **Monitor** -- For detached containers, `waitForExit()` runs in a goroutine, updating the container status to `stopped` or `error` when the process terminates.
## State management
The `State` struct provides a thread-safe, JSON-persisted container registry:
```go
type State struct {
Containers map[string]*Container `json:"containers"`
mu sync.RWMutex
filePath string
}
```
Key design decisions:
- **Copy-on-read**: `Get()` and `All()` return copies of container structs to prevent data races when callers modify the returned values.
- **Write-through**: Every mutation (`Add`, `Update`, `Remove`) immediately persists to disk via `SaveState()`.
- **Auto-create**: `LoadState()` returns an empty state if the file does not exist, and `SaveState()` creates parent directories as needed.
Default paths:
| Path | Purpose |
|------|---------|
| `~/.core/containers.json` | Container state file |
| `~/.core/logs/<id>.log` | Per-container log files |
## Stopping a container
`LinuxKitManager.Stop()` performs a graceful shutdown:
1. Sends `SIGTERM` to the hypervisor process.
2. Waits up to 10 seconds for the process to exit.
3. If the process does not exit in time, sends `SIGKILL`.
4. Respects context cancellation throughout -- if the context is cancelled, the process is killed immediately.
## Template engine
LinuxKit templates are YAML files that define a complete VM image configuration (kernel, init, services, files). The template engine adds variable substitution on top.
### Variable syntax
Two forms are supported:
- `${VAR}` -- Required variable. Produces an error if not provided.
- `${VAR:-default}` -- Optional variable with a default value.
### Resolution order
`GetTemplate(name)` searches for templates in this order:
1. **Embedded templates** -- Compiled into the binary via `//go:embed templates/*.yml`. These always take precedence.
2. **Workspace templates** -- `.core/linuxkit/` relative to the current working directory.
3. **User templates** -- `~/.core/linuxkit/` in the user's home directory.
User-defined templates that share a name with a built-in template are ignored (built-ins win).
### Variable extraction
`ExtractVariables(content)` parses a template and returns two collections:
- A sorted slice of required variable names (those using `${VAR}` syntax with no default).
- A map of optional variable names to their default values (those using `${VAR:-default}` syntax).
This powers the `core vm templates vars <name>` command.
## Development environment (devenv)
The `DevOps` struct in the `devenv` package composes the lower-level primitives into a complete development workflow.
### Boot sequence
1. Checks the dev image is installed (platform-specific qcow2 file: `core-devops-{os}-{arch}.qcow2`).
2. Launches the image via `LinuxKitManager.Run()` in detached mode with 4096 MB RAM, 2 CPUs, and SSH on port 2222.
3. Polls for up to 60 seconds until the VM's SSH host key can be scanned, then writes it to `~/.core/known_hosts`.
### Shell access
Two modes are available:
- **SSH** (default) -- Connects via `ssh -A -p 2222 root@localhost` with agent forwarding and strict host key checking against `~/.core/known_hosts`.
- **Serial console** -- Attaches to the QEMU serial console socket via `socat`.
### Project mounting
Projects are mounted into the VM at `/app` using a reverse SSHFS tunnel. The VM opens an SSH reverse tunnel back to the host (port 10000) and mounts the host directory via SSHFS.
### Auto-detection
Several operations auto-detect the project type by inspecting files on disk:
| File detected | Serve command | Test command |
|---------------|---------------|--------------|
| `artisan` | `php artisan octane:start` | -- |
| `package.json` with `dev` script | `npm run dev -- --host 0.0.0.0` | `npm test` |
| `composer.json` with `test` script | `frankenphp php-server` | `composer test` |
| `go.mod` | `go run .` | `go test ./...` |
| `manage.py` | `python manage.py runserver` | -- |
| `pytest.ini` or `pyproject.toml` | -- | `pytest` |
Auto-detection can be overridden with `.core/test.yaml` for tests or explicit `--command` flags.
### Claude integration
`DevOps.Claude()` starts a sandboxed Claude session inside the VM:
1. Auto-boots the dev environment if not running.
2. Mounts the project directory at `/app`.
3. Forwards authentication credentials (Anthropic API key, GitHub CLI config, SSH agent, git identity) based on configurable options.
4. Launches `claude` inside the VM via SSH with agent forwarding.
## Image sources (sources)
The `ImageSource` interface defines how dev environment images are downloaded:
```go
type ImageSource interface {
Name() string
Available() bool
LatestVersion(ctx context.Context) (string, error)
Download(ctx context.Context, m io.Medium, dest string, progress func(downloaded, total int64)) error
}
```
| Source | Backend | Availability check |
|--------|---------|--------------------|
| `CDNSource` | HTTP GET from a configured CDN URL | CDN URL is configured |
| `GitHubSource` | `gh release download` via the GitHub CLI | `gh` is installed and authenticated |
The `ImageManager` in `devenv` maintains a `manifest.json` in `~/.core/images/` that tracks installed image versions, SHA256 checksums, download timestamps, and source names. When the source is set to `"auto"` (the default), it tries GitHub first, then CDN.
## Configuration
The `devenv` package reads `~/.core/config.yaml` via the `go-config` library:
```yaml
version: 1
images:
source: auto # auto | github | cdn
github:
repo: host-uk/core-images
registry:
image: ghcr.io/host-uk/core-devops
cdn:
url: https://cdn.example.com/images
```
If the file does not exist, sensible defaults are used.
## Licence
EUPL-1.2

197
docs/development.md Normal file
View file

@ -0,0 +1,197 @@
---
title: Development
description: How to build, test, and contribute to go-container.
---
# Development
## Prerequisites
- **Go 1.26+** -- The module uses Go 1.26 features.
- **Go workspace** -- This module is part of a Go workspace at `~/Code/go.work`. Local development of sibling modules (go-io, go-config, go-i18n, cli) requires the workspace file.
Optional (for actually running VMs):
- **QEMU** -- `qemu-system-x86_64` for running LinuxKit images on any platform.
- **Hyperkit** -- macOS-only alternative hypervisor.
- **LinuxKit** -- For building images from templates (`linuxkit build`).
- **GitHub CLI** (`gh`) -- For the GitHub image source.
## Running tests
```bash
# All tests
go test ./...
# With race detector
go test -race ./...
# Single test by name
go test -run TestState_Add_Good ./...
# Single package
go test ./sources/
go test ./devenv/
```
Tests use `testify` for assertions. Most tests are self-contained and do not require a running hypervisor -- they test command construction, state management, template parsing, and configuration loading in isolation.
## Test naming convention
Tests follow a `_Good`, `_Bad`, `_Ugly` suffix pattern:
| Suffix | Meaning |
|--------|---------|
| `_Good` | Happy path -- valid inputs, expected success |
| `_Bad` | Expected error conditions -- invalid inputs, missing resources |
| `_Ugly` | Edge cases, panics, and boundary conditions |
Examples from the codebase:
```go
func TestNewState_Good(t *testing.T) { /* creates state successfully */ }
func TestLoadState_Bad_InvalidJSON(t *testing.T) { /* handles corrupt state file */ }
func TestGetHypervisor_Bad_Unknown(t *testing.T) { /* rejects unknown hypervisor name */ }
```
## Project structure
```
go-container/
container.go # Container struct, Manager interface, Status, RunOptions, ImageFormat
hypervisor.go # Hypervisor interface, QemuHypervisor, HyperkitHypervisor, DetectHypervisor
hypervisor_test.go
linuxkit.go # LinuxKitManager (Manager implementation), followReader for log tailing
linuxkit_test.go
state.go # State persistence (containers.json), log paths
state_test.go
templates.go # Template listing, loading, variable substitution, user template scanning
templates_test.go
templates/ # Embedded LinuxKit YAML templates
core-dev.yml
server-php.yml
sources/
source.go # ImageSource interface, SourceConfig
source_test.go
cdn.go # CDNSource implementation
cdn_test.go
github.go # GitHubSource implementation
github_test.go
devenv/
devops.go # DevOps orchestrator, Boot, Stop, Status, ImageName, ImagePath
devops_test.go
config.go # Config, ImagesConfig, LoadConfig from ~/.core/config.yaml
config_test.go
images.go # ImageManager, Manifest, Install, CheckUpdate
images_test.go
shell.go # Shell (SSH and serial console)
shell_test.go
serve.go # Serve (mount project, auto-detect serve command)
serve_test.go
test.go # Test (auto-detect test command, .core/test.yaml)
test_test.go
claude.go # Claude sandbox session with auth forwarding
claude_test.go
ssh_utils.go # Host key scanning for ~/.core/known_hosts
cmd/vm/
cmd_vm.go # CLI registration (init + AddVMCommands)
cmd_commands.go # Package doc
cmd_container.go # run, ps, stop, logs, exec commands
cmd_templates.go # templates, templates show, templates vars commands
```
## Coding standards
- **UK English** in all strings, comments, and documentation (colour, organisation, honour).
- **Strict typing** -- All function parameters and return values are typed. No `interface{}` without justification.
- **Error wrapping** -- Use `fmt.Errorf("context: %w", err)` for all error returns.
- **`io.Medium` abstraction** -- File system operations go through `io.Medium` (from `go-io`) rather than directly calling `os` functions. This enables testing with mock file systems. The `io.Local` singleton is used for real file system access.
- **Compile-time interface checks** -- Use `var _ Interface = (*Impl)(nil)` to verify implementations at compile time (see `sources/cdn.go` and `sources/github.go`).
- **Context propagation** -- All operations that might block accept a `context.Context` as their first parameter.
## Adding a new hypervisor
1. Create a new struct implementing the `Hypervisor` interface in `hypervisor.go`:
```go
type MyHypervisor struct {
Binary string
}
func (h *MyHypervisor) Name() string { return "my-hypervisor" }
func (h *MyHypervisor) Available() bool { /* check if binary exists */ }
func (h *MyHypervisor) BuildCommand(ctx context.Context, image string, opts *HypervisorOptions) (*exec.Cmd, error) {
// Build and return exec.Cmd
}
```
2. Register it in `DetectHypervisor()` and `GetHypervisor()` in `hypervisor.go`.
3. Add tests following the `_Good`/`_Bad` naming convention.
## Adding a new image source
1. Create a new struct implementing `ImageSource` in the `sources/` package:
```go
type MySource struct {
config SourceConfig
}
var _ ImageSource = (*MySource)(nil) // Compile-time check
func (s *MySource) Name() string { return "my-source" }
func (s *MySource) Available() bool { /* check prerequisites */ }
func (s *MySource) LatestVersion(ctx context.Context) (string, error) { /* fetch version */ }
func (s *MySource) Download(ctx context.Context, m io.Medium, dest string, progress func(downloaded, total int64)) error {
// Download image to dest
}
```
2. Wire it into `NewImageManager()` in `devenv/images.go` under the appropriate source selector.
## Adding a new LinuxKit template
### Built-in template
1. Create a `.yml` file in the `templates/` directory.
2. Add an entry to `builtinTemplates` in `templates.go`.
3. The file will be embedded via the `//go:embed templates/*.yml` directive.
### User template
Place a `.yml` file in either:
- `.core/linuxkit/` relative to your project root (workspace-scoped)
- `~/.core/linuxkit/` in your home directory (global)
The first comment line in the YAML file is extracted as the template description.
## File system paths
All persistent data lives under `~/.core/`:
| Path | Purpose |
|------|---------|
| `~/.core/containers.json` | Container state registry |
| `~/.core/logs/` | Per-container log files |
| `~/.core/images/` | Downloaded dev environment images |
| `~/.core/images/manifest.json` | Image version manifest |
| `~/.core/config.yaml` | Global configuration |
| `~/.core/known_hosts` | SSH host keys for dev VMs |
| `~/.core/linuxkit/` | User-defined LinuxKit templates |
The `CORE_IMAGES_DIR` environment variable overrides the default images directory.
## Licence
EUPL-1.2

146
docs/index.md Normal file
View file

@ -0,0 +1,146 @@
---
title: go-container
description: Container runtime, LinuxKit image builder, and portable development environment management for Go.
---
# go-container
`forge.lthn.ai/core/go-container` provides a container runtime built on LinuxKit and lightweight hypervisors. It manages the full lifecycle of LinuxKit virtual machines -- from building images with embedded templates, to running them via QEMU or Hyperkit, to offering a portable development environment with shell access, project mounting, test execution, and Claude AI integration.
This is **not** a Docker wrapper. It runs real VMs from LinuxKit images (ISO, qcow2, VMDK, raw) using platform-native acceleration (KVM on Linux, HVF on macOS, Hyperkit where available).
## Module path
```
forge.lthn.ai/core/go-container
```
Requires **Go 1.26+**.
## Quick start
### Run a VM from an image
```go
import (
"context"
container "forge.lthn.ai/core/go-container"
"forge.lthn.ai/core/go-io"
)
manager, err := container.NewLinuxKitManager(io.Local)
if err != nil {
log.Fatal(err)
}
ctx := context.Background()
c, err := manager.Run(ctx, "/path/to/image.qcow2", container.RunOptions{
Name: "my-vm",
Memory: 2048,
CPUs: 2,
SSHPort: 2222,
Detach: true,
})
if err != nil {
log.Fatal(err)
}
fmt.Printf("Started container %s (PID %d)\n", c.ID, c.PID)
```
### Use the development environment
```go
import (
"forge.lthn.ai/core/go-container/devenv"
"forge.lthn.ai/core/go-io"
)
dev, err := devenv.New(io.Local)
if err != nil {
log.Fatal(err)
}
// Boot the dev environment (downloads image if needed)
ctx := context.Background()
err = dev.Boot(ctx, devenv.DefaultBootOptions())
// Open an SSH shell
err = dev.Shell(ctx, devenv.ShellOptions{})
// Run tests inside the VM
err = dev.Test(ctx, "/path/to/project", devenv.TestOptions{})
```
### Build and run from a LinuxKit template
```go
import container "forge.lthn.ai/core/go-container"
// List available templates (built-in + user-defined)
templates := container.ListTemplates()
// Apply variables to a template
content, err := container.ApplyTemplate("core-dev", map[string]string{
"SSH_KEY": "ssh-ed25519 AAAA...",
"MEMORY": "4096",
"HOSTNAME": "my-dev-box",
})
```
## Package layout
| Package | Import path | Purpose |
|---------|-------------|---------|
| `container` (root) | `forge.lthn.ai/core/go-container` | Container struct, Manager interface, hypervisor abstraction, LinuxKit manager, state persistence, template engine |
| `devenv` | `forge.lthn.ai/core/go-container/devenv` | Portable dev environment orchestration: boot, shell, serve, test, Claude sandbox, image management |
| `sources` | `forge.lthn.ai/core/go-container/sources` | Image download backends: CDN and GitHub Releases with progress reporting |
| `cmd/vm` | `forge.lthn.ai/core/go-container/cmd/vm` | CLI commands (`core vm run`, `core vm ps`, `core vm stop`, `core vm logs`, `core vm exec`, `core vm templates`) |
## Dependencies
| Module | Purpose |
|--------|---------|
| `forge.lthn.ai/core/go-io` | File system abstraction (`Medium` interface), process utilities |
| `forge.lthn.ai/core/go-config` | Configuration loading (used by `devenv` for `~/.core/config.yaml`) |
| `forge.lthn.ai/core/go-i18n` | Internationalised UI strings (used by `cmd/vm`) |
| `forge.lthn.ai/core/cli` | CLI framework (used by `cmd/vm` for command registration) |
| `github.com/stretchr/testify` | Test assertions |
| `gopkg.in/yaml.v3` | YAML parsing for test configuration |
The root `container` package has only two direct dependencies: `go-io` and the standard library. The `devenv` and `cmd/vm` packages pull in the heavier dependencies.
## CLI commands
When registered via `cmd/vm`, the following commands become available under `core vm`:
| Command | Description |
|---------|-------------|
| `core vm run [image]` | Run a VM from an image file or `--template` |
| `core vm ps` | List running VMs (`-a` for all including stopped) |
| `core vm stop <id>` | Stop a running VM by ID or name (supports partial matching) |
| `core vm logs <id>` | View VM logs (`-f` to follow) |
| `core vm exec <id> <cmd>` | Execute a command inside the VM via SSH |
| `core vm templates` | List available LinuxKit templates |
| `core vm templates show <name>` | Display a template's full YAML |
| `core vm templates vars <name>` | Show a template's required and optional variables |
## Built-in templates
Two LinuxKit templates are embedded in the binary:
- **core-dev** -- Full development environment with Go, Node.js, PHP, Docker-in-LinuxKit, and SSH access
- **server-php** -- Production PHP server with FrankenPHP, Caddy reverse proxy, and health checks
User-defined templates can be placed in `.core/linuxkit/` (workspace-relative) or `~/.core/linuxkit/` (global). They are discovered automatically and merged with the built-in set.
## Licence
EUPL-1.2. See [LICENSE](../LICENSE) for the full text.