a43a16fb0d
- datanode: add isFileLocked() helper to prevent RLock re-entry deadlock in Append - io: MemoryMedium WriteMode rejects ancestor-is-file collision; EnsureDir rejects target-is-file collision - io: copy fileModes during directory rename - local: guard Delete/DeleteAll against removing sandbox root - local: add TOCTOU TODO comment on validatePath symlink loop - local: alias stdlib io→goio in medium_test.go - datanode: alias stdlib io→goio in medium_test.go - sqlite: add isValidTableName() whitelist to prevent table-name SQL injection in New() - sqlite: remove duplicate WHERE clause args in List query - sqlite: add mode field to sqliteWriteCloser; use it in Close (was hardcoded 420) - sigil: GzipSigil.In returns nil when custom outputWriter is used (buffer was empty) - sigil: capture hasher.Write error in HashSigil.In - sigil: add comment explaining DecryptionFailedError hides raw AEAD error intentionally - s3: add comment explaining WriteMode ignores mode (no POSIX on S3) - s3_test: ListObjectsV2 mock sets IsTruncated+NextContinuationToken when maxKeys exceeded - node: add comment explaining WriteMode ignores mode for in-memory nodes - store: sort keys before building List entries for deterministic output - store: add explanatory comment on NotFoundError sentinel - workspace: replace sha256.Sum256 key derivation with HKDF (RFC 5869) - docs: fix RFC-CORE-008 header (was RFC-025) - docs: update import paths from forge.lthn.ai/core/go-io to dappco.re/go/core/io - docs/RFC.md: remove duplicate Read/Write alias doc blocks Co-Authored-By: Virgil <virgil@lethean.io>
111 lines
4.1 KiB
Markdown
111 lines
4.1 KiB
Markdown
---
|
|
title: go-io
|
|
description: Unified storage abstraction for Go with pluggable backends — local filesystem, S3, SQLite, in-memory, and key-value.
|
|
---
|
|
|
|
# go-io
|
|
|
|
`dappco.re/go/core/io` is a storage abstraction library that provides a single `Medium` interface for reading and writing files across different backends. Write your code against `Medium` once, then swap between local disk, S3, SQLite, or in-memory storage without changing a line of business logic.
|
|
|
|
The library also includes `sigil`, a composable data-transformation pipeline for encoding, compression, hashing, and authenticated encryption.
|
|
|
|
|
|
## Quick Start
|
|
|
|
```go
|
|
import (
|
|
io "dappco.re/go/core/io"
|
|
"dappco.re/go/core/io/s3"
|
|
"dappco.re/go/core/io/node"
|
|
)
|
|
|
|
content, _ := io.Local.Read("/etc/hostname")
|
|
|
|
sandboxMedium, _ := io.NewSandboxed("/var/data/myapp")
|
|
_ = sandboxMedium.Write("config.yaml", "key: value")
|
|
|
|
nodeTree := node.New()
|
|
nodeTree.AddData("hello.txt", []byte("world"))
|
|
tarball, _ := nodeTree.ToTar()
|
|
|
|
s3Medium, _ := s3.New(s3.Options{Bucket: "my-bucket", Client: awsClient, Prefix: "uploads/"})
|
|
_ = s3Medium.Write("photo.jpg", rawData)
|
|
```
|
|
|
|
|
|
## Package Layout
|
|
|
|
| Package | Import Path | Purpose |
|
|
|---------|-------------|---------|
|
|
| `io` (root) | `dappco.re/go/core/io` | `Medium` interface, helper functions, `MemoryMedium` for tests |
|
|
| `local` | `dappco.re/go/core/io/local` | Local filesystem backend with path sandboxing and symlink-escape protection |
|
|
| `s3` | `dappco.re/go/core/io/s3` | Amazon S3 / S3-compatible backend (Garage, MinIO, etc.) |
|
|
| `sqlite` | `dappco.re/go/core/io/sqlite` | SQLite-backed virtual filesystem (pure Go driver, no CGO) |
|
|
| `node` | `dappco.re/go/core/io/node` | In-memory filesystem implementing both `Medium` and `fs.FS`, with tar round-tripping |
|
|
| `datanode` | `dappco.re/go/core/io/datanode` | Thread-safe in-memory `Medium` backed by Borg's DataNode, with snapshot/restore |
|
|
| `store` | `dappco.re/go/core/io/store` | Group-namespaced key-value store (SQLite), with a `Medium` adapter and Go template rendering |
|
|
| `sigil` | `dappco.re/go/core/io/sigil` | Composable data transformations: encoding, compression, hashing, XChaCha20-Poly1305 encryption |
|
|
| `workspace` | `dappco.re/go/core/io/workspace` | Encrypted workspace service integrated with the Core DI container |
|
|
|
|
|
|
## The Medium Interface
|
|
|
|
Every storage backend implements the same 17-method interface:
|
|
|
|
```go
|
|
type Medium interface {
|
|
Read(path string) (string, error)
|
|
Write(path, content string) error
|
|
WriteMode(path, content string, mode fs.FileMode) error
|
|
|
|
ReadStream(path string) (io.ReadCloser, error)
|
|
WriteStream(path string) (io.WriteCloser, error)
|
|
Open(path string) (fs.File, error)
|
|
Create(path string) (io.WriteCloser, error)
|
|
Append(path string) (io.WriteCloser, error)
|
|
|
|
EnsureDir(path string) error
|
|
List(path string) ([]fs.DirEntry, error)
|
|
|
|
Stat(path string) (fs.FileInfo, error)
|
|
Exists(path string) bool
|
|
IsFile(path string) bool
|
|
IsDir(path string) bool
|
|
|
|
Delete(path string) error
|
|
DeleteAll(path string) error
|
|
Rename(oldPath, newPath string) error
|
|
}
|
|
```
|
|
|
|
All backends implement this interface fully. Backends where a method has no natural equivalent (e.g., `EnsureDir` on S3) provide a safe no-op.
|
|
|
|
|
|
## Cross-Medium Operations
|
|
|
|
The root package provides helper functions that accept any `Medium`:
|
|
|
|
```go
|
|
sourceMedium := io.Local
|
|
destinationMedium := io.NewMemoryMedium()
|
|
err := io.Copy(sourceMedium, "source.txt", destinationMedium, "dest.txt")
|
|
|
|
content, err := io.Read(destinationMedium, "path")
|
|
err = io.Write(destinationMedium, "path", "content")
|
|
```
|
|
|
|
|
|
## Dependencies
|
|
|
|
| Dependency | Role |
|
|
|------------|------|
|
|
| `forge.lthn.ai/core/go-log` | Structured error helper (`E()`) |
|
|
| `forge.lthn.ai/Snider/Borg` | DataNode in-memory FS (used by `datanode` package) |
|
|
| `github.com/aws/aws-sdk-go-v2` | S3 client (used by `s3` package) |
|
|
| `golang.org/x/crypto` | BLAKE2, SHA-3, RIPEMD-160, XChaCha20-Poly1305 (used by `sigil`) |
|
|
| `modernc.org/sqlite` | Pure Go SQLite driver (used by `sqlite` and `store`) |
|
|
| `github.com/stretchr/testify` | Test assertions |
|
|
|
|
Go version: **1.26.0**
|
|
|
|
Licence: **EUPL-1.2**
|