go-io/docs/index.md

title	description
go-io	Unified storage abstraction for Go with pluggable backends — local filesystem, S3, SQLite, in-memory, and key-value.

go-io

forge.lthn.ai/core/go-io is a storage abstraction library that provides a single Medium interface for reading and writing files across different backends. Write your code against Medium once, then swap between local disk, S3, SQLite, or in-memory storage without changing a line of business logic.

The library also includes sigil, a composable data-transformation pipeline for encoding, compression, hashing, and authenticated encryption.

Quick Start

import (
    io "forge.lthn.ai/core/go-io"
    "forge.lthn.ai/core/go-io/s3"
    "forge.lthn.ai/core/go-io/node"
)

content, _ := io.Local.Read("/etc/hostname")

sandboxMedium, _ := io.NewSandboxed("/var/data/myapp")
_ = sandboxMedium.Write("config.yaml", "key: value")

nodeTree := node.New()
nodeTree.AddData("hello.txt", []byte("world"))
tarball, _ := nodeTree.ToTar()

s3Medium, _ := s3.New(s3.Options{Bucket: "my-bucket", Client: awsClient, Prefix: "uploads/"})
_ = s3Medium.Write("photo.jpg", rawData)

Package Layout

Package	Import Path	Purpose
io (root)	forge.lthn.ai/core/go-io	Medium interface, helper functions, MemoryMedium for tests
local	forge.lthn.ai/core/go-io/local	Local filesystem backend with path sandboxing and symlink-escape protection
s3	forge.lthn.ai/core/go-io/s3	Amazon S3 / S3-compatible backend (Garage, MinIO, etc.)
sqlite	forge.lthn.ai/core/go-io/sqlite	SQLite-backed virtual filesystem (pure Go driver, no CGO)
node	forge.lthn.ai/core/go-io/node	In-memory filesystem implementing both Medium and fs.FS, with tar round-tripping
datanode	forge.lthn.ai/core/go-io/datanode	Thread-safe in-memory Medium backed by Borg's DataNode, with snapshot/restore
store	forge.lthn.ai/core/go-io/store	Group-namespaced key-value store (SQLite), with a Medium adapter and Go template rendering
sigil	forge.lthn.ai/core/go-io/sigil	Composable data transformations: encoding, compression, hashing, XChaCha20-Poly1305 encryption
workspace	forge.lthn.ai/core/go-io/workspace	Encrypted workspace service integrated with the Core DI container

The Medium Interface

Every storage backend implements the same 17-method interface:

type Medium interface {
    Read(path string) (string, error)
    Write(path, content string) error
    WriteMode(path, content string, mode fs.FileMode) error

    ReadStream(path string) (io.ReadCloser, error)
    WriteStream(path string) (io.WriteCloser, error)
    Open(path string) (fs.File, error)
    Create(path string) (io.WriteCloser, error)
    Append(path string) (io.WriteCloser, error)

    EnsureDir(path string) error
    List(path string) ([]fs.DirEntry, error)

    Stat(path string) (fs.FileInfo, error)
    Exists(path string) bool
    IsFile(path string) bool
    IsDir(path string) bool

    Delete(path string) error
    DeleteAll(path string) error
    Rename(oldPath, newPath string) error
}

All backends implement this interface fully. Backends where a method has no natural equivalent (e.g., EnsureDir on S3) provide a safe no-op.

Cross-Medium Operations

The root package provides helper functions that accept any Medium:

sourceMedium := io.Local
destinationMedium := io.NewMemoryMedium()
err := io.Copy(sourceMedium, "source.txt", destinationMedium, "dest.txt")

content, err := io.Read(destinationMedium, "path")
err = io.Write(destinationMedium, "path", "content")

Dependencies

Dependency	Role
forge.lthn.ai/core/go-log	Structured error helper (E())
forge.lthn.ai/Snider/Borg	DataNode in-memory FS (used by datanode package)
github.com/aws/aws-sdk-go-v2	S3 client (used by s3 package)
golang.org/x/crypto	BLAKE2, SHA-3, RIPEMD-160, XChaCha20-Poly1305 (used by sigil)
modernc.org/sqlite	Pure Go SQLite driver (used by sqlite and store)
github.com/stretchr/testify	Test assertions

Go version: 1.26.0

Licence: EUPL-1.2