cli/docs/plans/2026-02-05-core-ide-job-runner-design.md

Core-IDE Job Runner Design

Date: 2026-02-05 Status: Approved Author: @Snider + Claude

---

Goal

Turn core-ide into an autonomous job runner that polls for actionable pipeline work, executes it via typed MCP tool handlers, captures JSONL training data, and self-updates. Supports 12 nodes running headless on servers and desktop on developer machines.

---

Architecture Overview

+-------------------------------------------------+
|                   core-ide                       |
|                                                  |
|  +----------+   +-----------+   +----------+    |
|  | Poller   |-->| Dispatcher|-->| Handler  |    |
|  | (Source) |   | (MCP route)|  | Registry |    |
|  +----------+   +-----------+   +----------+    |
|       |              |              |            |
|       |         +----v----+    +---v-------+    |
|       |         | Journal |    | JobSource |    |
|       |         | (JSONL) |    | (adapter) |    |
|       |         +---------+    +-----------+    |
|  +----v-----+                                   |
|  | Updater  |  (existing internal/cmd/updater)  |
|  +----------+                                   |
+-------------------------------------------------+

Three components:

• Poller -- Periodic scan via pluggable JobSource adapters. Builds PipelineSignal structs from API responses. Never reads comment bodies (injection vector).
• Dispatcher -- Matches signals against handler registry in priority order. One action per signal per cycle (prevents cascades).
• Journal -- Appends JSONL after each completed action per issue-epic step 10 spec. Structural signals only -- IDs, SHAs, timestamps, cycle counts, instructions sent, automations performed.

---

Job Source Abstraction

GitHub is the first adapter. The platform's own Agentic API replaces it later. Handler logic is source-agnostic.

type JobSource interface {
    Name() string
    Poll(ctx context.Context) ([]*PipelineSignal, error)
    Report(ctx context.Context, result *ActionResult) error
}

Adapter	When	Transport
GitHubSource	Now	REST API + conditional requests (ETag)
HostUKSource	Next	Agentic API (WebSocket or poll)
HyperswarmSource	Later	P2P encrypted channels via Holepunch

Multi-source: Poller runs multiple sources concurrently. Own repos get priority. When idle (zero signals for N consecutive cycles), external project sources activate (WailsApp first).

API budget: 50% credit allocation for harvest mode is a config value on the source, not hardcoded.

---

Pipeline Signal

The structural snapshot passed to handlers. Never contains comment bodies or free text.

type PipelineSignal struct {
    EpicNumber      int
    ChildNumber     int
    PRNumber        int
    RepoOwner       string
    RepoName        string
    PRState         string    // OPEN, MERGED, CLOSED
    IsDraft         bool
    Mergeable       string    // MERGEABLE, CONFLICTING, UNKNOWN
    CheckStatus     string    // SUCCESS, FAILURE, PENDING
    ThreadsTotal    int
    ThreadsResolved int
    LastCommitSHA   string
    LastCommitAt    time.Time
    LastReviewAt    time.Time
}

---

Handler Registry

Each action from the issue-epic flow is a registered handler. All Go functions with typed inputs/outputs.

type JobHandler interface {
    Name() string
    Match(signal *PipelineSignal) bool
    Execute(ctx context.Context, signal *PipelineSignal) (*ActionResult, error)
}

Handler	Epic Stage	Input Signals	Action
publish_draft	3	PR draft=true, checks=SUCCESS	Mark PR as ready for review
send_fix_command	4/6	PR CONFLICTING or threads without fix commit	Comment "fix merge conflict" / "fix the code reviews"
resolve_threads	5	Unresolved threads, fix commit exists after review	Resolve all pre-commit threads
enable_auto_merge	7	PR MERGEABLE, checks passing, threads resolved	Enable auto-merge via API
tick_parent	8	Child PR merged	Update epic issue checklist
close_child	9	Child PR merged + parent ticked	Close child issue
capture_journal	10	Any completed action	Append JSONL entry

ActionResult carries what was done -- action name, target IDs, success/failure, timestamps. Feeds directly into JSONL journal.

Handlers register at init time, same pattern as CLI commands in the existing codebase.

---

Headless vs Desktop Mode

Same binary, same handlers, different UI surface.

Detection:

func hasDisplay() bool {
    if runtime.GOOS == "windows" { return true }
    return os.Getenv("DISPLAY") != "" || os.Getenv("WAYLAND_DISPLAY") != ""
}

Headless mode (Linux server, no display):

• Skip Wails window creation
• Start poller immediately
• Start MCP bridge (port 9877) for external tool access
• Log to stdout/file (structured JSON)
• Updater: check on startup, auto-apply + restart via watcher
• Managed by systemd: Restart=always

Desktop mode (display available):

• Full Wails system tray + webview panel
• Tray icon shows status: idle, polling, executing, error
• Tray menu: Start/Stop poller, Force update, Open journal, Configure sources
• Poller off by default (developer toggle)
• Same MCP bridge, same handlers, same journal

CLI override: core-ide --headless forces headless. core-ide --desktop forces GUI.

Shared startup:

func main() {
    // 1. Load config (repos, interval, channel, sources)
    // 2. Build handler registry
    // 3. Init journal
    // 4. Init updater (check on startup)
    // 5. Branch:
    if hasDisplay() {
        startDesktop()  // Wails + tray + optional poller
    } else {
        startHeadless() // Poller + MCP bridge + signal handling
    }
}

---

Poller Configuration

type PollerConfig struct {
    Sources      []JobSource
    Handlers     []JobHandler
    Journal      *Journal
    PollInterval time.Duration // default: 60s
    DryRun       bool          // log without executing
}

Rate limiting: GitHub API allows 5000 req/hr with token. Full scan of 4 repos with ~30 PRs uses ~150 requests. Poller uses conditional requests (If-None-Match/ETag) to avoid counting unchanged responses. Backs off to 5min interval when idle.

CLI flags:

• --poll-interval (default: 60s)
• --repos (comma-separated: host-uk/core,host-uk/core-php)
• --dry-run (log actions without executing)
• --headless / --desktop (mode override)

---

Self-Update

Uses existing internal/cmd/updater package. Binary-safe replacement with platform-specific watcher process, SemVer channel selection (stable/beta/alpha/dev), automatic rollback on failure.

Integration:

• Headless: CheckAndUpdateOnStartup -- auto-apply + restart
• Desktop: CheckOnStartup -- notify via tray, user confirms

---

Training Data (Journal)

JSONL format per issue-epic step 10. One record per completed action.

{
  "ts": "2026-02-05T12:00:00Z",
  "epic": 299,
  "child": 212,
  "pr": 316,
  "repo": "host-uk/core",
  "action": "publish_draft",
  "signals": {
    "pr_state": "OPEN",
    "is_draft": true,
    "check_status": "SUCCESS",
    "mergeable": "UNKNOWN",
    "threads_total": 0,
    "threads_resolved": 0
  },
  "result": {
    "success": true,
    "duration_ms": 340
  },
  "cycle": 1
}

Rules:

• NO content (no comments, no messages, no bodies)
• Structural signals only -- safe for training
• Append-only JSONL file per node
• File path: ~/.core/journal/<repo>/<date>.jsonl

---

Files Summary

File	Action
pkg/jobrunner/types.go	CREATE -- JobSource, JobHandler, PipelineSignal, ActionResult interfaces
pkg/jobrunner/poller.go	CREATE -- Poller, Dispatcher, multi-source orchestration
pkg/jobrunner/journal.go	CREATE -- JSONL writer, append-only, structured records
pkg/jobrunner/github/source.go	CREATE -- GitHubSource adapter, conditional requests
pkg/jobrunner/github/signals.go	CREATE -- PR/issue state extraction, signal building
internal/core-ide/handlers/publish_draft.go	CREATE -- Publish draft PR handler
internal/core-ide/handlers/resolve_threads.go	CREATE -- Resolve review threads handler
internal/core-ide/handlers/send_fix_command.go	CREATE -- Send fix command handler
internal/core-ide/handlers/enable_auto_merge.go	CREATE -- Enable auto-merge handler
internal/core-ide/handlers/tick_parent.go	CREATE -- Tick epic checklist handler
internal/core-ide/handlers/close_child.go	CREATE -- Close child issue handler
internal/core-ide/main.go	MODIFY -- Headless/desktop branching, poller integration
internal/core-ide/mcp_bridge.go	MODIFY -- Register job handlers as MCP tools

---

What Doesn't Ship Yet

• HostUK Agentic API adapter (future -- replaces GitHub)
• Hyperswarm P2P adapter (future)
• External project scanning / harvest mode (future -- WailsApp first)
• LoRA training pipeline (separate concern -- reads JSONL journal)

---

Testing Strategy

• Handlers: Unit-testable. Mock PipelineSignal in, assert API calls out.
• Poller: httptest server returning fixture responses.
• Journal: Read back JSONL, verify schema.
• Integration: Dry-run mode against real repos, verify signals match expected state.