ts/docs/architecture.md

---
title: Architecture
description: How CoreTS manages a Deno sidecar with bidirectional gRPC/JSON-RPC communication, Worker isolation, and permission-gated I/O.
---

# Architecture

CoreTS follows a **sidecar pattern**: a Go process manages a Deno child process, and the two communicate over Unix domain sockets. This gives TypeScript modules access to Go-managed resources (filesystem, store, processes) whilst enforcing security boundaries at every layer.

## Overview

```
┌─────────────────────────────────────────────────────────────────┐
│ Go Process                                                      │
│                                                                 │
│  ┌──────────┐    ┌───────────┐    ┌──────────────────────────┐ │
│  │ Service   │───▸│ Sidecar   │    │ Server (CoreService)     │ │
│  │ (OnStart/ │    │ Start()   │    │ FileRead/Write/List/Del  │ │
│  │  OnStop)  │    │ Stop()    │    │ StoreGet/Set             │ │
│  └──────────┘    └───────────┘    │ ProcessStart/Stop        │ │
│       │                            └──────────────────────────┘ │
│       │                                       ▲                 │
│       ▼                                       │ gRPC            │
│  ┌──────────┐                         ┌───────┴───────┐        │
│  │DenoClient│──JSON-RPC──┐            │ Unix Socket   │        │
│  └──────────┘            │            │ (core.sock)   │        │
│                          │            └───────────────┘        │
└──────────────────────────│────────────────────────────────────┘
                           │
               ┌───────────┴─────────────────────────────────────┐
               │ Deno Process                                     │
               │                                                  │
               │  ┌────────────┐    ┌──────────────────────────┐ │
               │  │ CoreClient │───▸│ Go gRPC Server           │ │
               │  │ (gRPC)     │    │ (via core.sock)          │ │
               │  └────────────┘    └──────────────────────────┘ │
               │                                                  │
               │  ┌──────────────┐  ┌──────────────────────────┐ │
               │  │ DenoServer   │◂─│ Go DenoClient            │ │
               │  │ (JSON-RPC)   │  │ (via deno.sock)          │ │
               │  └──────────────┘  └──────────────────────────┘ │
               │                                                  │
               │  ┌──────────────────────────────────────────┐   │
               │  │ ModuleRegistry                           │   │
               │  │  ┌────────┐  ┌────────┐  ┌────────┐     │   │
               │  │  │Worker A│  │Worker B│  │Worker C│ ... │   │
               │  │  └────────┘  └────────┘  └────────┘     │   │
               │  └──────────────────────────────────────────┘   │
               └──────────────────────────────────────────────────┘
```

## Key Types

### Options

Configuration struct passed to `NewSidecar()` and `NewServiceFactory()`. Controls paths, security keys, and sidecar arguments. See [index.md](index.md#configuration) for field descriptions.

### Sidecar

Manages the Deno child process. Thread-safe via `sync.RWMutex`.

```go
type Sidecar struct {
    opts   Options
    mu     sync.RWMutex
    cmd    *exec.Cmd
    ctx    context.Context
    cancel context.CancelFunc
    done   chan struct{}
}
```

- `Start(ctx, args...)` -- launches `deno <args>` with `CORE_SOCKET` and `DENO_SOCKET` environment variables injected. Creates the socket directory with `0700` permissions. A background goroutine monitors the process and signals `done` on exit.
- `Stop()` -- cancels the context and blocks until the process exits.
- `IsRunning()` -- returns whether the child process is alive.

The sidecar refuses to start twice (returns an error if already running) and cleans up stale socket files before launch.

### Server (CoreService)

Implements the `CoreService` gRPC interface. Every I/O operation is gated by the calling module's declared permissions from its manifest.

```go
type Server struct {
    pb.UnimplementedCoreServiceServer
    medium    io.Medium        // Sandboxed filesystem
    store     *store.Store     // SQLite key-value store
    manifests map[string]*manifest.Manifest
    processes ProcessRunner    // Optional process management
}
```

**gRPC methods:**

| Method | Permission check | Description |
|--------|-----------------|-------------|
| `FileRead` | `CheckPath(path, manifest.Read)` | Read file content |
| `FileWrite` | `CheckPath(path, manifest.Write)` | Write file content |
| `FileList` | `CheckPath(path, manifest.Read)` | List directory entries |
| `FileDelete` | `CheckPath(path, manifest.Write)` | Delete a file |
| `StoreGet` | Reserved namespace (`_` prefix blocked) | Get a key-value pair |
| `StoreSet` | Reserved namespace (`_` prefix blocked) | Set a key-value pair |
| `ProcessStart` | `CheckRun(cmd, manifest.Run)` | Start a subprocess |
| `ProcessStop` | None (by process ID) | Stop a subprocess |

Store groups prefixed with `_` (e.g. `_coredeno`, `_modules`) are reserved for internal use and blocked from module access.

### DenoClient

Communicates with the Deno sidecar's JSON-RPC server over a Unix socket. Thread-safe via mutex (serialises requests over a single connection).

```go
type DenoClient struct {
    mu     sync.Mutex
    conn   net.Conn
    reader *bufio.Reader
}
```

**Methods:**
- `LoadModule(code, entryPoint, perms)` -- tells Deno to create a Worker for the module
- `UnloadModule(code)` -- terminates the module's Worker
- `ModuleStatus(code)` -- queries whether a module is LOADING, RUNNING, STOPPED, or ERRORED

The wire protocol is newline-delimited JSON over a raw Unix socket.

### Service

Wraps everything into a Core framework service with `Startable` and `Stoppable` lifecycle interfaces.

```go
type Service struct {
    *core.ServiceRuntime[Options]
    sidecar    *Sidecar
    grpcServer *Server
    store      *store.Store
    grpcCancel context.CancelFunc
    grpcDone   chan error
    denoClient *DenoClient
    installer  *marketplace.Installer
}
```

Register with the framework:

```go
core.New(core.WithService(ts.NewServiceFactory(opts)))
```

### Permissions

Three helper functions implement the permission model:

```go
// Prefix-based path matching with directory boundary checks.
// Empty allowed list = deny all (secure by default).
func CheckPath(path string, allowed []string) bool

// Exact match against allowed host:port list.
func CheckNet(addr string, allowed []string) bool

// Exact match against allowed command list.
func CheckRun(cmd string, allowed []string) bool
```

`CheckPath` cleans paths via `filepath.Clean` and verifies the separator boundary to prevent `"data"` from matching `"data-secrets"`.

## Startup Sequence

The `Service.OnStartup()` method orchestrates the full boot in order:

1. **Create sandboxed Medium** -- `io.NewSandboxed(AppRoot)` confines all filesystem operations to the application root. Falls back to `MockMedium` if no `AppRoot` is set.

2. **Open SQLite store** -- `store.New(dbPath)` opens the key-value database. Uses `:memory:` if no path is configured.

3. **Create gRPC server** -- `NewServer(medium, store)` wires up the CoreService implementation.

4. **Load manifest** -- reads `.core/view.yml` from `AppRoot`. If a `PublicKey` is configured, the manifest must pass ed25519 signature verification before being registered. Missing manifests are non-fatal.

5. **Start gRPC listener** -- `ListenGRPC()` runs in a background goroutine. Cleans up stale socket files, listens on a Unix socket, and sets `0600` permissions (owner-only).

6. **Launch sidecar** -- waits up to 5 seconds for the core socket to appear, then calls `Sidecar.Start()`. The child process receives `CORE_SOCKET` and `DENO_SOCKET` environment variables.

7. **Connect DenoClient** -- waits up to 10 seconds for the Deno socket to appear, then dials the JSON-RPC connection.

8. **Auto-load installed modules** -- if `AppRoot` is set, creates a `marketplace.Installer` and iterates over previously installed modules, calling `DenoClient.LoadModule()` for each.

If any step fails, earlier resources are cleaned up (gRPC listener cancelled, sidecar stopped) before the error is returned.

## Shutdown Sequence

`Service.OnShutdown()` tears down in reverse order:

1. Close the DenoClient connection
2. Stop the sidecar process (cancel context, wait for exit)
3. Cancel the gRPC listener context and wait for graceful stop
4. Close the SQLite store

## Deno Runtime Internals

### Entry Point (`runtime/main.ts`)

The Deno process boots through `main.ts`:

1. Reads `CORE_SOCKET` and `DENO_SOCKET` from environment (exits fatally if missing)
2. Creates a `ModuleRegistry`
3. Starts the DenoService JSON-RPC server on `DENO_SOCKET`
4. Connects to the Go CoreService gRPC server on `CORE_SOCKET` with retry (up to 20 attempts, 250ms apart)
5. Verifies connectivity by writing and reading back a health check value
6. Injects the CoreClient into the registry for I/O bridging
7. Listens for `SIGTERM` to initiate clean shutdown

### CoreClient (`runtime/client.ts`)

A gRPC client that dynamically loads the protobuf definition from `proto/coredeno.proto`. Provides typed methods for all CoreService operations (file read/write/list/delete, store get/set, process start/stop).

### DenoServer (`runtime/server.ts`)

A JSON-RPC server over a raw Unix socket (not gRPC -- Deno 2.x has broken http2 server support). Accepts newline-delimited JSON and dispatches to the ModuleRegistry:

- `LoadModule` -- create a Worker for a module
- `UnloadModule` -- terminate a module's Worker
- `ModuleStatus` -- query a module's current state

### ModuleRegistry (`runtime/modules.ts`)

Manages the lifecycle of TypeScript modules. Each module runs in its own Deno Worker with a tailored permission sandbox.

**Module states:** `UNKNOWN` | `LOADING` | `RUNNING` | `STOPPED` | `ERRORED`

When `load()` is called:

1. Any existing Worker for that module code is terminated
2. A new Worker is created from `worker-entry.ts` with Deno permissions derived from the module's declared permissions (read, write, net, run). Environment, system, and FFI access are always denied.
3. The Worker signals `ready`, and the registry responds with `{type: "load", url: "..."}` containing the module's entry point URL
4. The Worker dynamically imports the module and calls its `init(core)` function
5. The Worker signals `loaded` with success or error status

**I/O bridge:** Worker `postMessage` RPC calls are intercepted by the registry and relayed to the CoreClient. The registry injects the module's `code` into every gRPC call, so modules cannot spoof their identity.

### Worker Entry (`runtime/worker-entry.ts`)

The bootstrap script loaded as entry point for every module Worker. It:

1. Sets up request/response correlation for the postMessage-based RPC bridge
2. Exposes a `core` object with typed methods (`storeGet`, `storeSet`, `fileRead`, `fileWrite`, `processStart`, `processStop`)
3. Signals `ready` to the parent
4. On receiving `{type: "load"}`, dynamically imports the module URL and calls `init(core)` if the export exists

### Polyfill (`runtime/polyfill.ts`)

Must be imported before `@grpc/grpc-js`. Patches three Deno 2.x Node.js compatibility issues:

1. `http2.getDefaultSettings` is not implemented -- provides a stub
2. Already-connected Unix sockets never emit `connect`, causing http2 session hangs -- intercepts `net.connect` to create fresh sockets
3. Deno's http2 client never fires `remoteSettings` -- emits it synthetically after `connect`

## Module Manifest

Modules declare their identity and permissions in `.core/view.yml`:

```yaml
code: my-module
name: My Module
version: "1.0"
permissions:
  read: ["./data/"]
  write: ["./data/"]
  net: ["api.example.com:443"]
  run: ["ffmpeg"]
```

The manifest is loaded by `go-scm/manifest.Load()` and optionally verified with an ed25519 public key. Permissions from the manifest are enforced by the Go gRPC server on every I/O request.

## Marketplace Integration

When `AppRoot` is set, the service creates a `marketplace.Installer` backed by the `modules/` subdirectory. Modules are installed from Git repositories via `Installer.Install()` and automatically loaded into the Deno runtime on boot.

The marketplace flow:

1. `Installer.Install(ctx, module)` -- clones the Git repo into `AppRoot/modules/<code>/`
2. On next boot, `Service.OnStartup()` calls `Installer.Installed()` and loads each module
3. `DenoClient.LoadModule()` creates a Worker with the module's declared permissions
4. `Installer.Remove(code)` -- removes the module directory from disk

## Security Model

CoreTS enforces security at multiple layers:

- **Filesystem sandboxing** -- the `io.Medium` is scoped to `AppRoot`; no path escapes are possible
- **Permission gating** -- every gRPC call checks the module's manifest permissions before executing
- **Prefix matching with boundary checks** -- `CheckPath` prevents `"data"` from matching `"data-secrets"`
- **Reserved store namespaces** -- groups prefixed with `_` are blocked from module access
- **Worker isolation** -- each TypeScript module runs in its own Deno Worker with restricted permissions (no env, sys, or FFI access)
- **Identity injection** -- the Go side (via the ModuleRegistry I/O bridge) injects the module code into every gRPC call; modules cannot impersonate each other
- **Socket permissions** -- Unix sockets are created with `0600` (owner-only) and socket directories with `0700`
- **Manifest verification** -- optional ed25519 signature verification before registering a module