🚀 Resistance is futile. Your data’s uniqueness is added to our own.
This commit introduces the `borg collect archive` command, allowing users to collect items from the Internet Archive. The command includes three subcommands: - `search [query]`: Searches for items and collects them. - `item [identifier]`: Collects a specific item. - `collection [identifier]`: Collects all items in a collection. A new package, `pkg/archive`, has been created to handle all API interactions with archive.org. The implementation includes pagination to ensure all items are retrieved from large searches or collections. Downloaded items are stored in an `archive/` directory, with each item's files and metadata saved in a subdirectory named after its identifier. Unit and integration tests have been added to verify the functionality of the new commands and the API client. All existing tests continue to pass. Co-authored-by: Snider <631881+Snider@users.noreply.github.com> |
||
|---|---|---|
| .github/workflows | ||
| cmd | ||
| demo | ||
| dist | ||
| docs | ||
| examples | ||
| js/borg-stmf | ||
| php/borg-stmf | ||
| pkg | ||
| rfc | ||
| .gitignore | ||
| .goreleaser.yaml | ||
| CLAUDE.md | ||
| console.stim | ||
| go.mod | ||
| go.sum | ||
| go.work | ||
| go.work.sum | ||
| LICENSE.md | ||
| main.go | ||
| mkdocs.yml | ||
| README.md | ||
| Taskfile.yml | ||
Borg
Borg is a CLI tool and Go library for collecting, packaging, and encrypting data into portable, self-contained containers. It supports GitHub repositories, websites, PWAs, and arbitrary files.
Features
- Data Collection - Clone GitHub repos, crawl websites, download PWAs
- Portable Containers - Package data into DataNodes (in-memory fs.FS) or TIM bundles (OCI-compatible)
- Zero-Trust Encryption - ChaCha20-Poly1305 encryption for TIM containers (.stim) and messages (.smsg)
- SMSG Format - Encrypted message containers with public manifests, attachments, and zstd compression
- WASM Support - Decrypt SMSG files in the browser via WebAssembly
Installation
# From source
go install github.com/Snider/Borg@latest
# Or build locally
git clone https://github.com/Snider/Borg.git
cd Borg
go build -o borg ./
Requires Go 1.25+
Quick Start
# Clone a GitHub repository into a TIM container
borg collect github repo https://github.com/user/repo --format tim -o repo.tim
# Encrypt a TIM container
borg compile -f Borgfile -e "password" -o encrypted.stim
# Run an encrypted container
borg run encrypted.stim -p "password"
# Inspect container metadata (without decrypting)
borg inspect encrypted.stim --json
Container Formats
| Format | Extension | Description |
|---|---|---|
| DataNode | .tar |
In-memory filesystem, portable tarball |
| TIM | .tim |
Terminal Isolation Matrix - OCI/runc compatible bundle |
| Trix | .trix |
PGP-encrypted DataNode |
| STIM | .stim |
ChaCha20-Poly1305 encrypted TIM |
| SMSG | .smsg |
Encrypted message with attachments and public manifest |
SMSG - Secure Message Format
SMSG is designed for distributing encrypted content with publicly visible metadata:
import "github.com/Snider/Borg/pkg/smsg"
// Create and encrypt a message
msg := smsg.NewMessage("Hello, World!")
msg.AddBinaryAttachment("track.mp3", audioData, "audio/mpeg")
manifest := &smsg.Manifest{
Title: "Demo Track",
Artist: "Artist Name",
}
encrypted, _ := smsg.EncryptV2WithManifest(msg, "password", manifest)
// Decrypt
decrypted, _ := smsg.Decrypt(encrypted, "password")
v2 Binary Format - Stores attachments as raw binary with zstd compression for optimal size.
See RFC-001: Open Source DRM for the full specification.
Live Demo: demo.dapp.fm
Borgfile
Package files into a TIM container:
ADD ./app /usr/local/bin/app
ADD ./config /etc/app/
borg compile -f Borgfile -o app.tim
borg compile -f Borgfile -e "secret" -o app.stim # encrypted
CLI Reference
# Collection
borg collect github repo <url> # Clone repository
borg collect github repos <owner> # Clone all repos from user/org
borg collect website <url> --depth 2 # Crawl website
borg collect pwa --uri <url> # Download PWA
# Compilation
borg compile -f Borgfile -o out.tim # Plain TIM
borg compile -f Borgfile -e "pass" # Encrypted STIM
# Execution
borg run container.tim # Run plain TIM
borg run container.stim -p "pass" # Run encrypted TIM
# Inspection
borg decode file.stim -p "pass" -o out.tar
borg inspect file.stim [--json]
Documentation
mkdocs serve # Serve docs locally at http://localhost:8000
Development
task build # Build binary
task test # Run tests with coverage
task clean # Clean build artifacts
Architecture
Source (GitHub/Website/PWA)
↓ collect
DataNode (in-memory fs.FS)
↓ serialize
├── .tar (raw tarball)
├── .tim (runc container bundle)
├── .trix (PGP encrypted)
└── .stim (ChaCha20-Poly1305 encrypted TIM)
License
EUPL-1.2 - European Union Public License
Borg Status Messages (for CLI theming)
Initialization
Core engaged… resistance is already buffering.Assimilating bytes… stand by for cube‑formation.Merging… the Core is rewriting reality, one block at a time.
Encryption
Generating cryptographic sigils – the Core whispers to the witch.Encrypting payload – the Core feeds data to the witch's cauldron.Merge complete – data assimilated, encrypted, and sealed within us.
VCS Processing
Initiating clone… the Core replicates the collective into your node.Merging branches… conflicts resolved, entropy minimized, assimilation complete.