diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..4da33a4 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,348 @@ +--- +title: Architecture +description: Internals of go-process — key types, data flow, and design decisions. +--- + +# Architecture + +This document explains how `go-process` is structured, how data flows through +the system, and the role of each major component. + +## Overview + +The package is organised into four layers: + +1. **Service** — The Core-integrated service that owns processes and broadcasts events. +2. **Process** — An individual managed process with output capture and lifecycle state. +3. **Runner** — A pipeline orchestrator that runs multiple processes with dependency resolution. +4. **Daemon** — A higher-level abstraction for long-running services with PID files, health checks, and registry integration. + +A separate `exec/` sub-package provides a thin, fluent wrapper around `os/exec` +for simple one-shot commands. + +## Key Types + +### Status + +Process lifecycle is tracked as a state machine: + +``` +pending -> running -> exited + -> failed + -> killed +``` + +```go +type Status string + +const ( + StatusPending Status = "pending" + StatusRunning Status = "running" + StatusExited Status = "exited" + StatusFailed Status = "failed" + StatusKilled Status = "killed" +) +``` + +- **pending** — queued but not yet started (currently unused by the service, + reserved for future scheduling). +- **running** — actively executing. +- **exited** — completed; check `ExitCode` for success (0) or failure. +- **failed** — could not be started (e.g. binary not found). +- **killed** — terminated by signal or context cancellation. + +### Service + +`Service` is the central type. It embeds `core.ServiceRuntime[Options]` to +participate in the Core DI container and implements both `Startable` and +`Stoppable` lifecycle interfaces. + +```go +type Service struct { + *core.ServiceRuntime[Options] + processes map[string]*Process + mu sync.RWMutex + bufSize int + idCounter atomic.Uint64 +} +``` + +Key behaviours: + +- **OnStartup** — currently a no-op; reserved for future initialisation. +- **OnShutdown** — iterates all running processes and calls `Kill()` on each, + ensuring no orphaned child processes when the application exits. +- Process IDs are generated as `proc-N` using an atomic counter, guaranteeing + uniqueness without locks. + +#### Registration + +The service is registered with Core via a factory function: + +```go +process.NewService(process.Options{BufferSize: 2 * 1024 * 1024}) +``` + +`NewService` returns a `func(*core.Core) (any, error)` closure — the standard +Core service factory signature. The `Options` struct is captured by the closure +and applied when Core instantiates the service. + +### Process + +`Process` wraps an `os/exec.Cmd` with: + +- Thread-safe state (`sync.RWMutex` guards all mutable fields). +- A `RingBuffer` for output capture (configurable size, default 1 MB). +- A `done` channel that closes when the process exits, enabling `select`-based + coordination. +- Stdin pipe access via `SendInput()` and `CloseStdin()`. +- Context-based cancellation — cancelling the context kills the process. + +#### Info Snapshot + +`Process.Info()` returns an `Info` struct — a serialisable snapshot of the +process state, suitable for JSON APIs or UI display: + +```go +type Info struct { + ID string `json:"id"` + Command string `json:"command"` + Args []string `json:"args"` + Dir string `json:"dir"` + StartedAt time.Time `json:"startedAt"` + Status Status `json:"status"` + ExitCode int `json:"exitCode"` + Duration time.Duration `json:"duration"` + PID int `json:"pid"` +} +``` + +### RingBuffer + +A fixed-size circular buffer that overwrites the oldest data when full. +Thread-safe for concurrent reads and writes. + +```go +rb := process.NewRingBuffer(64 * 1024) // 64 KB +rb.Write([]byte("data")) +fmt.Println(rb.String()) // "data" +fmt.Println(rb.Len()) // 4 +fmt.Println(rb.Cap()) // 65536 +rb.Reset() +``` + +The ring buffer is used internally to capture process stdout and stderr. When +a process produces more output than the buffer capacity, the oldest data is +silently overwritten. This prevents unbounded memory growth for long-running +or verbose processes. + +### ACTION Messages + +Four IPC message types are broadcast through `Core.ACTION()`: + +| Type | When | Key Fields | +|------|------|------------| +| `ActionProcessStarted` | Process begins execution | `ID`, `Command`, `Args`, `Dir`, `PID` | +| `ActionProcessOutput` | Each line of stdout/stderr | `ID`, `Line`, `Stream` | +| `ActionProcessExited` | Process completes | `ID`, `ExitCode`, `Duration`, `Error` | +| `ActionProcessKilled` | Process is terminated | `ID`, `Signal` | + +The `Stream` type distinguishes stdout from stderr: + +```go +type Stream string + +const ( + StreamStdout Stream = "stdout" + StreamStderr Stream = "stderr" +) +``` + +## Data Flow + +When `Service.StartWithOptions()` is called: + +``` +1. Generate unique ID (atomic counter) +2. Create context with cancel +3. Build os/exec.Cmd with dir, env, pipes +4. Create RingBuffer (unless DisableCapture is set) +5. cmd.Start() +6. Store process in map +7. Broadcast ActionProcessStarted via Core.ACTION +8. Spawn 2 goroutines to stream stdout and stderr + - Each line is written to the RingBuffer + - Each line is broadcast as ActionProcessOutput +9. Spawn 1 goroutine to wait for process exit + - Waits for output goroutines to finish first + - Calls cmd.Wait() + - Updates process status and exit code + - Closes the done channel + - Broadcasts ActionProcessExited +``` + +The output streaming goroutines use `bufio.Scanner` with a 1 MB line buffer +to handle long lines without truncation. + +## Runner + +The `Runner` orchestrates multiple processes, defined as `RunSpec` values: + +```go +type RunSpec struct { + Name string + Command string + Args []string + Dir string + Env []string + After []string // dependency names + AllowFailure bool +} +``` + +Three execution strategies are available: + +### RunAll (dependency graph) + +Processes dependencies in waves. In each wave, all specs whose dependencies +are satisfied run in parallel. If a dependency fails (and `AllowFailure` is +false), its dependents are skipped. Circular dependencies are detected and +reported as skipped with an error. + +``` +Wave 1: [lint, vet] (no dependencies) +Wave 2: [test] (depends on lint, vet) +Wave 3: [build] (depends on test) +``` + +### RunSequential + +Executes specs one after another. Stops on the first failure unless +`AllowFailure` is set. Remaining specs are marked as skipped. + +### RunParallel + +Runs all specs concurrently, ignoring the `After` field entirely. Failures +do not affect other specs. + +All three strategies return a `RunAllResult` with aggregate counts: + +```go +type RunAllResult struct { + Results []RunResult + Duration time.Duration + Passed int + Failed int + Skipped int +} +``` + +## Daemon + +The `Daemon` type manages the full lifecycle of a long-running service: + +``` +NewDaemon(opts) -> Start() -> Run(ctx) -> Stop() +``` + +### PID File + +`PIDFile` provides single-instance enforcement. `Acquire()` writes the current +process PID to a file; if the file already exists and the recorded PID is still +alive (verified via `syscall.Signal(0)`), it returns an error. Stale PID files +from dead processes are automatically cleaned up. + +```go +pid := process.NewPIDFile("/var/run/myapp.pid") +err := pid.Acquire() // writes current PID, fails if another instance is live +defer pid.Release() // removes the file +``` + +### Health Server + +`HealthServer` exposes two HTTP endpoints: + +- **`/health`** — runs all registered `HealthCheck` functions. Returns 200 if + all pass, 503 if any fail. +- **`/ready`** — returns 200 or 503 based on the readiness flag, toggled via + `SetReady(bool)`. + +The server binds to a configurable address (use port `0` for ephemeral port +allocation in tests). `WaitForHealth()` is a polling utility that waits for +`/health` to return 200 within a timeout. + +### Registry + +`Registry` tracks running daemons via JSON files in a directory (default: +`~/.core/daemons/`). Each daemon is a `DaemonEntry`: + +```go +type DaemonEntry struct { + Code string `json:"code"` + Daemon string `json:"daemon"` + PID int `json:"pid"` + Health string `json:"health,omitempty"` + Project string `json:"project,omitempty"` + Binary string `json:"binary,omitempty"` + Started time.Time `json:"started"` +} +``` + +The registry automatically prunes entries with dead PIDs on `List()` and +`Get()`. When a `Daemon` is configured with a `Registry`, it auto-registers +on `Start()` and auto-unregisters on `Stop()`. + +File naming convention: `{code}-{daemon}.json` (slashes replaced with dashes). + +## exec Sub-Package + +The `exec` package (`forge.lthn.ai/core/go-process/exec`) provides a fluent +wrapper around `os/exec` for simple, one-shot commands that do not need Core +integration: + +```go +import "forge.lthn.ai/core/go-process/exec" + +// Fluent API +err := exec.Command(ctx, "go", "build", "./..."). + WithDir("/path/to/project"). + WithEnv([]string{"CGO_ENABLED=0"}). + WithLogger(myLogger). + Run() + +// Get output +out, err := exec.Command(ctx, "git", "status").Output() + +// Combined stdout + stderr +out, err := exec.Command(ctx, "make").CombinedOutput() + +// Quiet mode (suppresses stdout, includes stderr in error) +err := exec.RunQuiet(ctx, "go", "vet", "./...") +``` + +### Logging + +Commands are automatically logged at debug level before execution and at error +level on failure. The logger interface is minimal: + +```go +type Logger interface { + Debug(msg string, keyvals ...any) + Error(msg string, keyvals ...any) +} +``` + +A `NopLogger` (the default) discards all messages. Use `SetDefaultLogger()` to +set a package-wide logger, or `WithLogger()` for per-command overrides. + +## Thread Safety + +All public types are safe for concurrent use: + +- `Service` — `sync.RWMutex` protects the process map; atomic counter for IDs. +- `Process` — `sync.RWMutex` protects mutable state. +- `RingBuffer` — `sync.RWMutex` on all read/write operations. +- `PIDFile` — `sync.Mutex` on acquire/release. +- `HealthServer` — `sync.Mutex` on check list and readiness flag. +- `Registry` — filesystem-level isolation (one file per daemon). +- Global singleton — `atomic.Pointer` for lock-free reads. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..d11384f --- /dev/null +++ b/docs/development.md @@ -0,0 +1,164 @@ +--- +title: Development +description: How to build, test, and contribute to go-process. +--- + +# Development + +## Prerequisites + +- **Go 1.26+** (uses Go workspaces) +- **Core CLI** (`core` binary) for running tests and quality checks +- Access to `forge.lthn.ai` (private module registry) + +Ensure `GOPRIVATE` includes `forge.lthn.ai/*`: + +```bash +go env -w GOPRIVATE=forge.lthn.ai/* +``` + +## Go Workspace + +This module is part of the workspace defined at `~/Code/go.work`. After +cloning, run: + +```bash +go work sync +``` + +## Running Tests + +```bash +# All tests +core go test + +# Single test +core go test --run TestService_Start + +# With verbose output +core go test -v +``` + +Alternatively, using `go test` directly: + +```bash +go test ./... +go test -run TestRunner_RunAll ./... +go test -v -count=1 ./exec/... +``` + +## Quality Assurance + +```bash +# Format, vet, lint, test +core go qa + +# Full suite (includes race detector, vulnerability scan, security audit) +core go qa full +``` + +Individual commands: + +```bash +core go fmt # Format code +core go vet # Go vet +core go lint # Lint +core go cov # Generate coverage report +core go cov --open # Open coverage in browser +``` + +## Test Naming Convention + +Tests follow the `_Good`, `_Bad`, `_Ugly` suffix pattern used across the Core +ecosystem: + +- **`_Good`** — happy path, expected success. +- **`_Bad`** — expected error conditions, graceful handling. +- **`_Ugly`** — panics, edge cases, degenerate inputs. + +Where this pattern does not fit naturally, descriptive sub-test names are used +instead (e.g. `TestService_Start/echo_command`, `TestService_Start/context_cancellation`). + +## Project Structure + +``` +go-process/ + .core/ + build.yaml # Build configuration + release.yaml # Release configuration + exec/ + exec.go # Fluent command wrapper + exec_test.go # exec tests + logger.go # Logger interface and NopLogger + actions.go # IPC action message types + buffer.go # RingBuffer implementation + buffer_test.go # RingBuffer tests + daemon.go # Daemon lifecycle manager + daemon_test.go # Daemon tests + go.mod # Module definition + health.go # HTTP health check server + health_test.go # Health server tests + pidfile.go # PID file single-instance lock + pidfile_test.go # PID file tests + process.go # Process type and methods + process_global.go # Global singleton and convenience API + process_test.go # Process tests + global_test.go # Global API tests (concurrency) + registry.go # Daemon registry (JSON file store) + registry_test.go # Registry tests + runner.go # Pipeline runner (sequential, parallel, DAG) + runner_test.go # Runner tests + service.go # Core service (DI integration, lifecycle) + service_test.go # Service tests + types.go # Shared types (Status, Stream, RunOptions, Info) +``` + +## Adding a New Feature + +1. Write the implementation in the appropriate file (or create a new one if + the feature is clearly distinct). +2. Add tests following the naming conventions above. +3. If the feature introduces new IPC events, add the message types to + `actions.go`. +4. Run `core go qa` to verify formatting, linting, and tests pass. +5. Commit using conventional commits: `feat(process): add XYZ support`. + +## Coding Standards + +- **UK English** in documentation and comments (colour, organisation, centre). +- **`declare(strict_types=1)`-equivalent**: all functions have explicit + parameter and return types. +- **Error handling**: return errors rather than panicking. Use sentinel errors + (`ErrProcessNotFound`, `ErrProcessNotRunning`, `ErrStdinNotAvailable`) for + well-known conditions. +- **Thread safety**: all public types must be safe for concurrent use. Use + `sync.RWMutex` for read-heavy workloads, `sync.Mutex` where writes dominate. +- **Formatting**: `gofmt` / `goimports` via `core go fmt`. + +## Error Types + +| Error | Meaning | +|-------|---------| +| `ErrProcessNotFound` | No process with the given ID exists in the service | +| `ErrProcessNotRunning` | Operation requires a running process (e.g. SendInput, Signal) | +| `ErrStdinNotAvailable` | Stdin pipe is nil (already closed or never created) | +| `ErrServiceNotInitialized` | Global convenience function called before `process.Init()` | +| `ServiceError` | Wraps service-level errors with a message string | + +## Build Configuration + +The `.core/build.yaml` defines cross-compilation targets: + +| OS | Architecture | +|----|-------------| +| linux | amd64 | +| linux | arm64 | +| darwin | arm64 | +| windows | amd64 | + +Since this is a library (no binary), the build configuration is primarily +used for CI validation. The `binary` field is empty. + +## Licence + +EUPL-1.2. See the repository root for the full licence text. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..ddc0a5c --- /dev/null +++ b/docs/index.md @@ -0,0 +1,124 @@ +--- +title: go-process +description: Process management with Core IPC integration for Go applications. +--- + +# go-process + +`forge.lthn.ai/core/go-process` is a process management library that provides +spawning, monitoring, and controlling external processes with real-time output +streaming via the Core ACTION (IPC) system. It integrates directly with the +[Core DI framework](https://forge.lthn.ai/core/go) as a first-class service. + +## Features + +- Spawn and manage external processes with full lifecycle tracking +- Real-time stdout/stderr streaming via Core IPC actions +- Ring buffer output capture (default 1 MB, configurable) +- Process pipeline runner with dependency graphs, sequential, and parallel modes +- Daemon mode with PID file locking, health check HTTP server, and graceful shutdown +- Daemon registry for tracking running instances across the system +- Lightweight `exec` sub-package for one-shot command execution with logging +- Thread-safe throughout; designed for concurrent use + +## Quick Start + +### Register with Core + +```go +import ( + "context" + framework "forge.lthn.ai/core/go/pkg/core" + "forge.lthn.ai/core/go-process" +) + +// Create a Core instance with the process service +c, err := framework.New( + framework.WithName("process", process.NewService(process.Options{})), +) +if err != nil { + log.Fatal(err) +} + +// Retrieve the typed service +svc, err := framework.ServiceFor[*process.Service](c, "process") +if err != nil { + log.Fatal(err) +} +``` + +### Run a Command + +```go +// Fire-and-forget (async) +proc, err := svc.Start(ctx, "go", "test", "./...") +if err != nil { + return err +} +<-proc.Done() +fmt.Println(proc.Output()) + +// Synchronous convenience +output, err := svc.Run(ctx, "echo", "hello world") +``` + +### Listen for Events + +Process lifecycle events are broadcast through Core's ACTION system: + +```go +c.RegisterAction(func(c *framework.Core, msg framework.Message) error { + switch m := msg.(type) { + case process.ActionProcessStarted: + fmt.Printf("Started: %s (PID %d)\n", m.Command, m.PID) + case process.ActionProcessOutput: + fmt.Print(m.Line) + case process.ActionProcessExited: + fmt.Printf("Exit code: %d (%s)\n", m.ExitCode, m.Duration) + case process.ActionProcessKilled: + fmt.Printf("Killed with %s\n", m.Signal) + } + return nil +}) +``` + +### Global Convenience API + +For applications that only need a single process service, a global singleton +is available: + +```go +// Initialise once at startup +process.Init(coreInstance) + +// Then use package-level functions anywhere +proc, _ := process.Start(ctx, "ls", "-la") +output, _ := process.Run(ctx, "date") +procs := process.List() +running := process.Running() +``` + +## Package Layout + +| Path | Description | +|------|-------------| +| `*.go` (root) | Core process service, types, actions, runner, daemon, health, PID file, registry | +| `exec/` | Lightweight command wrapper with fluent API and structured logging | + +## Module Information + +| Field | Value | +|-------|-------| +| Module path | `forge.lthn.ai/core/go-process` | +| Go version | 1.26.0 | +| Licence | EUPL-1.2 | + +## Dependencies + +| Module | Purpose | +|--------|---------| +| `forge.lthn.ai/core/go` | Core DI framework (`ServiceRuntime`, `Core.ACTION`, lifecycle interfaces) | +| `github.com/stretchr/testify` | Test assertions (test-only) | + +The package has no other runtime dependencies beyond the Go standard library +and the Core framework.