cli/README.md
Vi 4debdc1449 feat: BugSETI app, WebSocket hub, browser automation, and MCP tools (#336)
* feat: add security logging and fix framework regressions

This commit implements comprehensive security event logging and resolves critical regressions in the core framework.

Security Logging:
- Enhanced `pkg/log` with a `Security` level and helper.
- Added `log.Username()` to consistently identify the executing user.
- Instrumented GitHub CLI auth, Agentic configuration, filesystem sandbox, MCP handlers, and MCP TCP transport with security logs.
- Added `SecurityStyle` to the CLI for consistent visual representation of security events.

UniFi Security (CodeQL):
- Refactored `pkg/unifi` to remove hardcoded `InsecureSkipVerify`, resolving a high-severity alert.
- Added a `--verify-tls` flag and configuration option to control TLS verification.
- Updated command handlers to support the new verification parameter.

Framework Fixes:
- Restored original signatures for `MustServiceFor`, `Config()`, and `Display()` in `pkg/framework/core`, which had been corrupted during a merge.
- Fixed `pkg/framework/framework.go` and `pkg/framework/core/runtime_pkg.go` to match the restored signatures.
- These fixes resolve project-wide compilation errors caused by the signature mismatches.

I encountered significant blockers due to a corrupted state of the `dev` branch after a merge, which introduced breaking changes in the core framework's DI system. I had to manually reconcile these signatures with the expected usage across the codebase to restore build stability.

* feat(mcp): add RAG tools (query, ingest, collections)

Add vector database tools to the MCP server for RAG operations:
- rag_query: Search for relevant documentation using semantic similarity
- rag_ingest: Ingest files or directories into the vector database
- rag_collections: List available collections

Uses existing internal/cmd/rag exports (QueryDocs, IngestDirectory, IngestFile)
and pkg/rag for Qdrant client access. Default collection is "hostuk-docs"
with topK=5 for queries.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(mcp): add metrics tools (record, query)

Add MCP tools for recording and querying AI/security metrics events.
The metrics_record tool writes events to daily JSONL files, and the
metrics_query tool provides aggregated statistics by type, repo, and agent.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add 'core mcp serve' command

Add CLI command to start the MCP server for AI tool integration.

- Create internal/cmd/mcpcmd package with serve subcommand
- Support --workspace flag for directory restriction
- Handle SIGINT/SIGTERM for clean shutdown
- Register in full.go build variant

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(ws): add WebSocket hub package for real-time streaming

Add pkg/ws package implementing a hub pattern for WebSocket connections:
- Hub manages client connections, broadcasts, and channel subscriptions
- Client struct represents connected WebSocket clients
- Message types: process_output, process_status, event, error, ping/pong
- Channel-based subscription system (subscribe/unsubscribe)
- SendProcessOutput and SendProcessStatus for process streaming integration
- Full test coverage including concurrency tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(mcp): add process management and WebSocket MCP tools

Add MCP tools for process management:
- process_start: Start a new external process
- process_stop: Gracefully stop a running process
- process_kill: Force kill a process
- process_list: List all managed processes
- process_output: Get captured process output
- process_input: Send input to process stdin

Add MCP tools for WebSocket:
- ws_start: Start WebSocket server for real-time streaming
- ws_info: Get hub statistics (clients, channels)

Update Service struct with optional process.Service and ws.Hub fields,
new WithProcessService and WithWSHub options, getter methods, and
Shutdown method for cleanup.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(webview): add browser automation package via Chrome DevTools Protocol

Add pkg/webview package for browser automation:
- webview.go: Main interface with Connect, Navigate, Click, Type, QuerySelector, Screenshot, Evaluate
- cdp.go: Chrome DevTools Protocol WebSocket client implementation
- actions.go: DOM action types (Click, Type, Hover, Scroll, etc.) and ActionSequence builder
- console.go: Console message capture and filtering with ConsoleWatcher and ExceptionWatcher
- angular.go: Angular-specific helpers for router navigation, component access, and Zone.js stability

Add MCP tools for webview:
- webview_connect/disconnect: Connection management
- webview_navigate: Page navigation
- webview_click/type/query/wait: DOM interaction
- webview_console: Console output capture
- webview_eval: JavaScript execution
- webview_screenshot: Screenshot capture

Add documentation:
- docs/mcp/angular-testing.md: Guide for Angular application testing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: document new packages and BugSETI application

- Update CLAUDE.md with documentation for:
  - pkg/ws (WebSocket hub for real-time streaming)
  - pkg/webview (Browser automation via CDP)
  - pkg/mcp (MCP server tools: process, ws, webview)
  - BugSETI application overview
- Add comprehensive README for BugSETI with:
  - Installation and configuration guide
  - Usage workflow documentation
  - Architecture overview
  - Contributing guidelines

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(bugseti): add BugSETI system tray app with auto-update

BugSETI - Distributed Bug Fixing like SETI@home but for code

Features:
- System tray app with Wails v3
- GitHub issue fetching with label filters
- Issue queue with priority management
- AI context seeding via seed-agent-developer skill
- Automated PR submission flow
- Stats tracking and leaderboard
- Cross-platform notifications
- Self-updating with stable/beta/nightly channels

Includes:
- cmd/bugseti: Main application with Angular frontend
- internal/bugseti: Core services (fetcher, queue, seeder, submit, config, stats, notify)
- internal/bugseti/updater: Auto-update system (checker, downloader, installer)
- .github/workflows/bugseti-release.yml: CI/CD for all platforms

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: resolve import cycle and code duplication

- Remove pkg/log import from pkg/io/local to break import cycle
  (pkg/log/rotation.go imports pkg/io, creating circular dependency)
- Use stderr logging for security events in sandbox escape detection
- Remove unused sync/atomic import from core.go
- Fix duplicate LogSecurity function declarations in cli/log.go
- Update workspace/service.go Crypt() call to match interface

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update tests for new function signatures and format code

- Update core_test.go: Config(), Display() now panic instead of returning error
- Update runtime_pkg_test.go: sr.Config() now panics instead of returning error
- Update MustServiceFor tests to use assert.Panics
- Format BugSETI, MCP tools, and webview packages with gofmt

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
Co-authored-by: Claude <developers@lethean.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 17:22:05 +00:00

443 lines
15 KiB
Markdown

# Core
[![codecov](https://codecov.io/gh/host-uk/core/branch/dev/graph/badge.svg)](https://codecov.io/gh/host-uk/core)
[![Go Test Coverage](https://github.com/host-uk/core/actions/workflows/coverage.yml/badge.svg)](https://github.com/host-uk/core/actions/workflows/coverage.yml)
[![Code Scanning](https://github.com/host-uk/core/actions/workflows/codescan.yml/badge.svg)](https://github.com/host-uk/core/actions/workflows/codescan.yml)
[![Go Version](https://img.shields.io/github/go-mod/go-version/host-uk/core)](https://go.dev/)
[![License](https://img.shields.io/badge/License-EUPL--1.2-blue.svg)](https://opensource.org/licenses/EUPL-1.2)
Core is a Web3 Framework, written in Go using Wails.io to replace Electron and the bloat of browsers that, at their core, still live in their mum's basement.
- Repo: https://github.com/host-uk/core
## Vision
Core is an **opinionated Web3 desktop application framework** providing:
1. **Service-Oriented Architecture** - Pluggable services with dependency injection
2. **Encrypted Workspaces** - Each workspace gets its own PGP keypair, files are obfuscated
3. **Cross-Platform Storage** - Abstract storage backends (local, SFTP, WebDAV) behind a `Medium` interface
4. **Multi-Brand Support** - Same codebase powers different "hub" apps (AdminHub, ServerHub, GatewayHub, DeveloperHub, ClientHub)
5. **Built-in Crypto** - PGP encryption/signing, hashing, checksums as first-class citizens
**Mental model:** A secure, encrypted workspace manager where each "workspace" is a cryptographically isolated environment. The framework handles windows, menus, trays, config, and i18n.
## CLI Quick Start
```bash
# 1. Install Core
go install github.com/host-uk/core/cmd/core@latest
# 2. Verify environment
core doctor
# 3. Run tests in any Go/PHP project
core go test # or core php test
# 4. Build and preview release
core build
core ci
```
For more details, see the [User Guide](docs/user-guide.md).
## Framework Quick Start (Go)
```go
import core "github.com/host-uk/core/pkg/framework/core"
app, err := core.New(
core.WithServiceLock(),
)
```
## Prerequisites
- [Go](https://go.dev/) 1.25+
- [Node.js](https://nodejs.org/)
- [Wails](https://wails.io/) v3
- [Task](https://taskfile.dev/)
## Development Workflow (TDD)
```bash
task test-gen # 1. Generate test stubs
task test # 2. Run tests (watch them fail)
# 3. Implement your feature
task test # 4. Run tests (watch them pass)
task review # 5. CodeRabbit review
```
## Building & Running
```bash
# GUI (Wails)
task gui:dev # Development with hot-reload
task gui:build # Production build
# CLI
task cli:build # Build to cmd/core/bin/core
task cli:run # Build and run
```
## Configuration
Core uses a layered configuration system where values are resolved in the following priority:
1. **Command-line flags** (if applicable)
2. **Environment variables**
3. **Configuration file**
4. **Default values**
### Configuration File
The default configuration file is located at `~/.core/config.yaml`.
#### Format
The file uses YAML format and supports nested structures.
```yaml
# ~/.core/config.yaml
dev:
editor: vim
debug: true
log:
level: info
```
### Environment Variables
#### Layered Configuration Mapping
Any configuration value can be overridden using environment variables with the `CORE_CONFIG_` prefix. After stripping the `CORE_CONFIG_` prefix, the remaining variable name is converted to lowercase and underscores are replaced with dots to map to the configuration hierarchy.
**Examples:**
- `CORE_CONFIG_DEV_EDITOR=nano` maps to `dev.editor: nano`
- `CORE_CONFIG_LOG_LEVEL=debug` maps to `log.level: debug`
#### Common Environment Variables
| Variable | Description |
|----------|-------------|
| `CORE_DAEMON` | Set to `1` to run the application in daemon mode. |
| `NO_COLOR` | If set (to any value), disables ANSI color output. |
| `MCP_ADDR` | Address for the MCP TCP server (e.g., `localhost:9100`). If not set, MCP uses Stdio. |
| `COOLIFY_TOKEN` | API token for Coolify deployments. |
| `AGENTIC_TOKEN` | API token for Agentic services. |
| `UNIFI_URL` | URL of the UniFi controller (e.g., `https://192.168.1.1`). |
| `UNIFI_INSECURE` | Set to `1` or `true` to skip UniFi TLS verification. |
## All Tasks
| Task | Description |
|------|-------------|
| `task test` | Run all Go tests |
| `task test-gen` | Generate test stubs for public API |
| `task check` | go mod tidy + tests + review |
| `task review` | CodeRabbit review |
| `task cov` | Run tests with coverage report |
| `task cov-view` | Open HTML coverage report |
| `task sync` | Update public API Go files |
---
## Architecture
### Project Structure
```
.
├── main.go # CLI application entry point
├── pkg/
│ ├── framework/core/ # Service container, DI, Runtime[T]
│ ├── crypt/ # Hashing, checksums, PGP
│ ├── io/ # Medium interface + backends
│ ├── help/ # In-app documentation
│ ├── i18n/ # Internationalization
│ ├── repos/ # Multi-repo registry & management
│ ├── agentic/ # AI agent task management
│ └── mcp/ # Model Context Protocol service
├── internal/
│ ├── cmd/ # CLI command implementations
│ └── variants/ # Build variants (full, minimal, etc.)
└── go.mod # Go module definition
```
### Service Pattern (Dual-Constructor DI)
Every service follows this pattern:
```go
// Static DI - standalone use/testing (no core.Runtime)
func New() (*Service, error)
// Dynamic DI - for core.WithService() registration
func Register(c *core.Core) (any, error)
```
Services embed `*core.Runtime[Options]` for access to `Core()` and `Config()`.
### IPC/Action System
Services implement `HandleIPCEvents(c *core.Core, msg core.Message) error` - auto-discovered via reflection. Handles typed actions like `core.ActionServiceStartup`.
---
## Wails v3 Frontend Bindings
Core uses [Wails v3](https://v3alpha.wails.io/) to expose Go methods to a WebView2 browser runtime. Wails automatically generates TypeScript bindings for registered services.
**Documentation:** [Wails v3 Method Bindings](https://v3alpha.wails.io/features/bindings/methods/)
### How It Works
1. **Go services** with exported methods are registered with Wails
2. Run `wails3 generate bindings` (or `wails3 dev` / `wails3 build`)
3. **TypeScript SDK** is generated in `frontend/bindings/`
4. Frontend calls Go methods with full type safety, no HTTP overhead
### Current Binding Architecture
```go
// cmd/core-gui/main.go
app.RegisterService(application.NewService(coreService)) // Only Core is registered
```
**Problem:** Only `Core` is registered with Wails. Sub-services (crypt, workspace, display, etc.) are internal to Core's service map - their methods aren't directly exposed to JS.
**Currently exposed** (see `cmd/core-gui/public/bindings/`):
```typescript
// From frontend:
import { ACTION, Config, Service } from './bindings/github.com/host-uk/core/pkg/core'
ACTION(msg) // Broadcast IPC message
Config() // Get config service reference
Service("workspace") // Get service by name (returns any)
```
**NOT exposed:** Direct calls like `workspace.CreateWorkspace()` or `crypt.Hash()`.
## Configuration Management
Core uses a **centralized configuration service** implemented in `pkg/config`, with YAML-based persistence and layered overrides.
The `pkg/config` package provides:
- YAML-backed persistence at `~/.core/config.yaml`
- Dot-notation key access (for example: `cfg.Set("dev.editor", "vim")`, `cfg.GetString("dev.editor")`)
- Environment variable overlay support (env vars can override persisted values)
- Thread-safe operations for concurrent reads/writes
Application code should treat `pkg/config` as the **primary configuration mechanism**. Direct reads/writes to YAML files should generally be avoided from application logic in favour of using this centralized service.
### Project and Service Configuration Files
In addition to the centralized configuration service, Core uses several YAML files for project-specific build/CI and service configuration. These live alongside (but are distinct from) the centralized configuration:
- **Project Configuration** (in the `.core/` directory of the project root):
- `build.yaml`: Build targets, flags, and project metadata.
- `release.yaml`: Release automation, changelog settings, and publishing targets.
- `ci.yaml`: CI pipeline configuration.
- **Global Configuration** (in the `~/.core/` directory):
- `config.yaml`: Centralized user/framework settings and defaults, managed via `pkg/config`.
- `agentic.yaml`: Configuration for agentic services (BaseURL, Token, etc.).
- **Registry Configuration** (`repos.yaml`, auto-discovered):
- Multi-repo registry definition.
- Searched in the current directory and its parent directories (walking up).
- Then in `~/Code/host-uk/repos.yaml`.
- Finally in `~/.config/core/repos.yaml`.
### Format
All persisted configuration files described above use **YAML** format for readability and nested structure support.
### The IPC Bridge Pattern (Chosen Architecture)
Sub-services are accessed via Core's **IPC/ACTION system**, not direct Wails bindings:
```typescript
// Frontend calls Core.ACTION() with typed messages
import { ACTION } from './bindings/github.com/host-uk/core/pkg/core'
// Open a window
ACTION({ action: "display.open_window", name: "settings", options: { Title: "Settings", Width: 800 } })
// Switch workspace
ACTION({ action: "workspace.switch_workspace", name: "myworkspace" })
```
Each service implements `HandleIPCEvents(c *core.Core, msg core.Message)` to process these messages:
```go
// pkg/display/display.go
func (s *Service) HandleIPCEvents(c *core.Core, msg core.Message) error {
switch m := msg.(type) {
case map[string]any:
if action, ok := m["action"].(string); ok && action == "display.open_window" {
return s.handleOpenWindowAction(m)
}
}
return nil
}
```
**Why this pattern:**
- Single Wails service (Core) = simpler binding generation
- Services remain decoupled from Wails
- Centralized message routing via `ACTION()`
- Services can communicate internally using same pattern
**Current gap:** Not all service methods have IPC handlers yet. See `HandleIPCEvents` in each service to understand what's wired up.
### Generating Bindings
Wails v3 bindings are typically generated in the GUI repository (e.g., `core-gui`).
```bash
wails3 generate bindings # Regenerate after Go changes
```
---
### Service Interfaces (`pkg/framework/core/interfaces.go`)
```go
type Config interface {
Get(key string, out any) error
Set(key string, v any) error
}
type Display interface {
OpenWindow(opts ...WindowOption) error
}
type Workspace interface {
CreateWorkspace(identifier, password string) (string, error)
SwitchWorkspace(name string) error
WorkspaceFileGet(filename string) (string, error)
WorkspaceFileSet(filename, content string) error
}
type Crypt interface {
EncryptPGP(writer io.Writer, recipientPath, data string, ...) (string, error)
DecryptPGP(recipientPath, message, passphrase string, ...) (string, error)
}
```
---
## Current State (Prototype)
### Working
| Package | Notes |
|---------|-------|
| `pkg/framework/core` | Service container, DI, thread-safe - solid |
| `pkg/config` | Layered YAML configuration, XDG paths - solid |
| `pkg/crypt` | Hashing, checksums, symmetric/asymmetric - solid, well-tested |
| `pkg/help` | Embedded docs, full-text search - solid |
| `pkg/i18n` | Multi-language with go-i18n - solid |
| `pkg/io` | Medium interface + local backend - solid |
| `pkg/repos` | Multi-repo registry & management - solid |
| `pkg/agentic` | AI agent task management - solid |
| `pkg/mcp` | Model Context Protocol service - solid |
---
## Package Deep Dives
### pkg/crypt
The crypt package provides a comprehensive suite of cryptographic primitives:
- **Hashing & Checksums**: SHA-256, SHA-512, and CRC32 support.
- **Symmetric Encryption**: AES-GCM and ChaCha20-Poly1305 for secure data at rest.
- **Key Derivation**: Argon2id for secure password hashing.
- **Asymmetric Encryption**: PGP implementation in the `pkg/crypt/openpgp` subpackage using `github.com/ProtonMail/go-crypto`.
### pkg/io - Storage Abstraction
```go
type Medium interface {
Read(path string) (string, error)
Write(path, content string) error
EnsureDir(path string) error
IsFile(path string) bool
FileGet(path string) (string, error)
FileSet(path, content string) error
}
```
Implementations: `local/`, `sftp/`, `webdav/`
---
## Future Work
### Phase 1: Core Stability
- [x] ~~Fix workspace medium injection (critical blocker)~~
- [x] ~~Initialize `io.Local` global~~
- [x] ~~Clean up dead code (orphaned vars, broken wrappers)~~
- [x] ~~Wire up IPC handlers for all services (config, crypt, display, help, i18n, workspace)~~
- [x] ~~Complete display menu handlers (New/List workspace)~~
- [x] ~~Tray icon setup with asset embedding~~
- [x] ~~Test coverage for io packages~~
- [ ] System tray brand-specific menus
### Phase 2: Multi-Brand Support
- [ ] Define brand configuration system (config? build flags?)
- [ ] Implement brand-specific tray menus (AdminHub, ServerHub, GatewayHub, DeveloperHub, ClientHub)
- [ ] Brand-specific theming/assets
- [ ] Per-brand default workspace configurations
### Phase 3: Remote Storage
- [ ] Complete SFTP backend (`pkg/io/sftp/`)
- [ ] Complete WebDAV backend (`pkg/io/webdav/`)
- [ ] Workspace sync across storage backends
- [ ] Conflict resolution for multi-device access
### Phase 4: Enhanced Crypto
- [ ] Key management UI (import/export, key rotation)
- [ ] Multi-recipient encryption
- [ ] Hardware key support (YubiKey, etc.)
- [ ] Encrypted workspace backup/restore
### Phase 5: Developer Experience
- [ ] TypeScript types for IPC messages (codegen from Go structs)
- [ ] Hot-reload for service registration
- [ ] Plugin system for third-party services
- [ ] CLI tooling for workspace management
### Phase 6: Distribution
- [ ] Auto-update mechanism
- [ ] Platform installers (DMG, MSI, AppImage)
- [ ] Signing and notarization
- [ ] Crash reporting integration
---
## Getting Help
- **[User Guide](docs/user-guide.md)**: Detailed usage and concepts.
- **[FAQ](docs/faq.md)**: Frequently asked questions.
- **[Workflows](docs/workflows.md)**: Common task sequences.
- **[Troubleshooting](docs/troubleshooting.md)**: Solving common issues.
- **[Configuration](docs/configuration.md)**: Config file reference.
```bash
# Check environment
core doctor
# Command help
core <command> --help
```
---
## For New Contributors
1. Run `task test` to verify all tests pass
2. Follow TDD: `task test-gen` creates stubs, implement to pass
3. The dual-constructor pattern is intentional: `New(deps)` for tests, `Register()` for runtime
4. IPC handlers in each service's `HandleIPCEvents()` are the frontend bridge