go-io/docs/architecture.md

title	description
Architecture	Internal design of go-io — the Medium interface, backend implementations, sigil transformation pipeline, and security model.

Architecture

This document explains how go-io is structured internally, how the key types relate to one another, and how data flows through the system.

Design Principles

1. One interface, many backends. All storage operations go through Medium. Business logic never imports a specific backend.
2. Path sandboxing by default. The local backend validates every path against symlink escapes. Sandboxed mediums cannot reach outside their root.
3. Composable transforms. The sigil package lets you chain encoding, compression, and encryption steps into reversible pipelines.
4. No CGO. The SQLite driver (modernc.org/sqlite) is pure Go. The entire module compiles without a C toolchain.

Core Types

Medium (root package)

The Medium interface is defined in io.go. It is the only type that consuming code needs to know about. The root package also provides:

• io.Local — a package-level variable initialised in init() via local.New("/"). This gives unsandboxed access to the host filesystem, mirroring the behaviour of the standard os package.
• io.NewSandboxed(root) — creates a local.Medium restricted to root. All path resolution is confined within that directory.
• io.Copy(src, srcPath, dst, dstPath) — copies a file between any two mediums by reading from one and writing to the other.
• io.NewMemoryMedium() — a fully functional in-memory implementation for unit tests. It tracks files, directories, and modification times in plain maps.

FileInfo and DirEntry (root package)

Simple struct implementations of fs.FileInfo and fs.DirEntry are exported from the root package for use in mocks and tests. Each backend also defines its own unexported equivalents internally.

Backend Implementations

local.Medium

File: local/medium.go

The local backend wraps the standard os package with two layers of path protection:

1. path(p string) — normalises paths using filepath.Clean("/" + p) before joining with the root. This neutralises .. traversal at the string level. When root is "/", absolute paths pass through unchanged and relative paths resolve against the working directory.

2. validatePath(p string) — walks each path component, calling filepath.EvalSymlinks at every step. If any resolved component lands outside the root, it logs a security event to stderr (including timestamp, root, attempted path, and OS username) and returns os.ErrPermission.

Delete operations refuse to remove "/" or the user's home directory as an additional safety rail.

caller -> path(p) -> validatePath(p) -> os.ReadFile / os.WriteFile / ...

s3.Medium

File: s3/s3.go

The S3 backend translates Medium operations into AWS SDK calls. Key design decisions:

• Key construction: key(p) uses path.Clean("/" + p) to sandbox traversal, then prepends the optional prefix. This means ../secret resolves to secret, not an escape.
• Directory semantics: S3 has no real directories. EnsureDir is a no-op. IsDir and Exists for directory-like paths use ListObjectsV2 with MaxKeys: 1 to check for objects under the prefix.
• Rename: Implemented as copy-then-delete, since S3 has no atomic rename.
• Append: Downloads existing content, appends in memory, re-uploads on Close(). This is the only viable approach given S3's immutable-object model.
• Testability: The Client interface abstracts the six SDK methods used. Tests inject a mockS3 that stores objects in a map[string][]byte with a sync.RWMutex.

sqlite.Medium

File: sqlite/sqlite.go

Stores files and directories as rows in a single SQLite table:

CREATE TABLE IF NOT EXISTS files (
    path    TEXT PRIMARY KEY,
    content BLOB NOT NULL,
    mode    INTEGER DEFAULT 420,   -- 0644
    is_dir  BOOLEAN DEFAULT FALSE,
    mtime   DATETIME DEFAULT CURRENT_TIMESTAMP
)

• WAL mode is enabled at connection time for better concurrent read performance.
• Path cleaning uses the same path.Clean("/" + p) pattern as other backends.
• Rename is transactional: it reads the source row, inserts at the destination, deletes the source, and moves all children (if it is a directory) within a single transaction.
• Custom tables are supported via sqlite.Options{Path: ":memory:", Table: "name"} to allow multiple logical filesystems in one database.
• :memory: databases work out of the box for tests.

node.Node

File: node/node.go

A pure in-memory filesystem that implements both Medium and the standard library's fs.FS, fs.StatFS, fs.ReadDirFS, and fs.ReadFileFS interfaces. Directories are implicit -- they exist whenever a stored file path contains a "/".

Key capabilities beyond Medium:

• ToTar() / FromTar() — serialise the entire tree to a tar archive and back. This enables snapshotting, transport, and archival.
• Walk() with WalkOptions — extends fs.WalkDir with MaxDepth, Filter, and SkipErrors controls.
• CopyFile(src, dst, perm) — copies a file from the in-memory tree to the real filesystem.
• CopyTo(target Medium, src, dst) — copies a file or directory tree to any other Medium.
• ReadFile(name) — returns a defensive copy of file content, preventing callers from mutating internal state.

datanode.Medium

File: datanode/medium.go

A thread-safe Medium backed by Borg's DataNode (an in-memory fs.FS with tar serialisation). It adds:

• sync.RWMutex on every operation for concurrent safety.
• Explicit directory tracking via a map[string]bool, since DataNode only stores files.
• Snapshot() / Restore(data) — serialise and deserialise the entire filesystem as a tarball.
• DataNode() — exposes the underlying Borg DataNode for integration with TIM containers.
• File deletion is handled by rebuilding the DataNode without the target file, since the upstream type does not expose a Remove method.

store Package

Files: store/store.go, store/medium.go

The store package provides two complementary APIs:

KeyValueStore (key-value)

A group-namespaced key-value store backed by SQLite:

CREATE TABLE IF NOT EXISTS kv (
    grp   TEXT NOT NULL,
    key   TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (grp, key)
)

Operations: Get, Set, Delete, Count, DeleteGroup, GetAll, Render.

The Render method loads all key-value pairs from a group into a map[string]string and executes a Go text/template against them:

keyValueStore, _ := store.New(store.Options{Path: ":memory:"})
keyValueStore.Set("user", "pool", "pool.lthn.io:3333")
keyValueStore.Set("user", "wallet", "iz...")
renderedText, _ := keyValueStore.Render(`{"pool":"{{ .pool }}"}`, "user")
assert.Equal(t, `{"pool":"pool.lthn.io:3333"}`, renderedText)

store.Medium (Medium adapter)

Wraps a KeyValueStore to satisfy the Medium interface. Paths are split as group/key:

• Read("config/theme") calls Get("config", "theme")
• List("") returns all groups as directories
• List("config") returns all keys in the config group as files
• IsDir("config") returns true if the group has entries

You can create it directly (store.NewMedium(store.Options{Path: ":memory:"})) or adapt an existing store (keyValueStore.AsMedium()).

sigil Package

Files: sigil/sigil.go, sigil/sigils.go, sigil/crypto_sigil.go

The sigil package implements composable, reversible data transformations.

Interface

type Sigil interface {
    In(data []byte) ([]byte, error)
    Out(data []byte) ([]byte, error)
}

Contracts:

• Reversible sigils: Out(In(x)) == x
• Irreversible sigils (hashes): Out returns input unchanged
• Symmetric sigils (reverse): In(x) == Out(x)
• nil input returns nil without error
• Empty input returns empty without error

Available Sigils

Created via NewSigil(name):

Name	Type	Reversible
reverse	Byte reversal	Yes (symmetric)
hex	Hexadecimal encoding	Yes
base64	Base64 encoding	Yes
gzip	Gzip compression	Yes
json	JSON compaction	No (Out is passthrough)
json-indent	JSON pretty-printing	No (Out is passthrough)
md4, md5, sha1	Legacy hashes	No
sha224 .. sha512	SHA-2 family	No
sha3-224 .. sha3-512	SHA-3 family	No
sha512-224, sha512-256	Truncated SHA-512	No
ripemd160	RIPEMD-160	No
blake2s-256	BLAKE2s	No
blake2b-256 .. blake2b-512	BLAKE2b	No

Pipeline Functions

encoded, _ := sigil.Transmute(data, []sigil.Sigil{gzipSigil, hexSigil})

original, _ := sigil.Untransmute(encoded, []sigil.Sigil{gzipSigil, hexSigil})

Authenticated Encryption: ChaChaPolySigil

ChaChaPolySigil provides XChaCha20-Poly1305 authenticated encryption with a pre-obfuscation layer. It implements the Sigil interface, so it composes naturally into pipelines.

Encryption flow:

plaintext -> obfuscate(nonce) -> XChaCha20-Poly1305 encrypt -> [nonce || ciphertext || tag]

Decryption flow:

[nonce || ciphertext || tag] -> decrypt -> deobfuscate(nonce) -> plaintext

The pre-obfuscation layer ensures that raw plaintext patterns are never sent directly to CPU encryption routines, providing defence-in-depth against side-channel attacks. Two obfuscators are provided:

• XORObfuscator (default) — XORs data with a SHA-256 counter-mode key stream derived from the nonce.
• ShuffleMaskObfuscator — applies XOR masking followed by a deterministic Fisher-Yates byte shuffle, making both value and position analysis more difficult.

key := make([]byte, 32)
rand.Read(key)

cipherSigil, _ := sigil.NewChaChaPolySigil(key, nil)
ciphertext, _ := cipherSigil.In([]byte("secret"))
plaintext, _ := cipherSigil.Out(ciphertext)

shuffleCipherSigil, _ := sigil.NewChaChaPolySigil(key, &sigil.ShuffleMaskObfuscator{})

Each call to In generates a fresh random nonce, so encrypting the same plaintext twice produces different ciphertexts.

workspace Package

File: workspace/service.go

A higher-level service that integrates with the Core DI container (forge.lthn.ai/core/go). It manages encrypted workspaces stored under ~/.core/workspaces/.

Each workspace:

• Is identified by a SHA-256 hash of the user-provided identifier
• Contains subdirectories: config/, log/, data/, files/, keys/
• Has a PGP keypair generated via the Core crypt service
• Supports file get/set operations on the files/ subdirectory
• Handles IPC events (workspace.create, workspace.switch) for integration with the Core message bus

The workspace service implements core.Workspace and uses io.Local as its storage medium.

Data Flow Summary

Application code
       |
       v
   io.Medium (interface)
       |
       +-- local.Medium  --> os package (with sandbox validation)
       +-- s3.Medium     --> AWS SDK S3 client
       +-- sqlite.Medium --> modernc.org/sqlite
       +-- node.Node     --> in-memory map + tar serialisation
       +-- datanode.Medium --> Borg DataNode + sync.RWMutex
       +-- store.Medium  --> store.KeyValueStore (SQLite KV) --> Medium adapter
       +-- MemoryMedium   --> map[string]string (for tests)

Every backend normalises paths using the same path.Clean("/" + p) pattern, ensuring consistent behaviour regardless of which backend is in use.