Adds support for parsing and respecting robots.txt during website collection.
This change introduces the following features:
- Fetches and parses /robots.txt before crawling a website.
- Respects `Disallow` patterns to avoid crawling restricted areas.
- Honors the `Crawl-delay` directive to prevent hammering sites.
- Adds command-line flags to configure the behavior:
- `--ignore-robots`: Ignores robots.txt rules.
- `--user-agent`: Sets a custom user-agent string.
- `--min-delay`: Overrides the crawl-delay with a minimum value.
The implementation includes a new `robots` package for parsing robots.txt files and integrates it into the existing website downloader. Tests have been added to verify the new functionality.
Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
Add multi-quality variant support for video content:
- New ABR types in pkg/smsg/types.go (ABRManifest, Variant, ABRPresets)
- New pkg/smsg/abr.go with manifest read/write and bandwidth estimation
- New cmd/mkdemo-abr CLI tool for creating ABR variant sets via ffmpeg
- WASM parseABRManifest and selectVariant functions
- Demo page "Adaptive Quality" tab with ABR player
- RFC-001 Section 3.7 documenting ABR format and algorithm
Profile page:
- No WASM or video download until play button clicked
- Play button visible immediately, loading on-demand
- Removed auto-play behavior completely
Streaming:
- GetV3HeaderFromPrefix for parsing from partial data
- v3 demo file with 128KB chunks for streaming tests
V3 streaming format enables zero-trust media streaming:
- Content encrypted once with random CEK
- CEK wrapped with time-bound stream keys derived from LTHN hash
- Rolling window: current period + next period always valid
- Keys auto-expire, no revocation needed
Cadence options (platform controls refresh rate):
- daily: 24-hour periods (2026-01-12)
- 12h: Half-day periods (2026-01-12-AM/PM)
- 6h: Quarter-day periods (2026-01-12-00/06/12/18)
- 1h: Hourly periods (2026-01-12-15)
Key derivation: SHA256(LTHN(period:license:fingerprint))
- LTHN is rainbow-table resistant (salt derived from input)
- Only the derived key can decrypt, never transmitted
New files:
- pkg/smsg/stream.go - v3 encryption/decryption
- pkg/smsg/stream_test.go - 17 tests including cadence
WASM v1.3.0:
- BorgSMSG.decryptV3(data, {license, fingerprint})
- getInfo() now returns cadence and keyMethod
- cmd/dapp-fm-app: Native desktop app with WebView (Wails)
- cmd/dapp-fm: CLI binary for HTTP server mode
- pkg/player: Shared player core with Go bindings
Architecture: Go decrypts SMSG content, serves via asset handler.
Frontend calls Go directly via Wails bindings for manifest/license
checks.
- Update docs/cli.md to include all current commands (all, compile, run, decode) and correct terminology (TIM).
- Update docs/library.md with correct, runnable Go code examples and updated import paths.
- Update docs/development.md, docs/installation.md, and docs/releasing.md to reflect the project's Go version (1.25.0).
- Remove outdated "No functional changes" notes from documentation files.
- Added unit tests for `ToTar` and `FromTar` in `pkg/datanode`, including a round-trip test and invalid input handling.
- Added unit tests for `Walk` options (`MaxDepth`, `Filter`, `SkipErrors`) in `pkg/datanode`.
- Added security tests for `pkg/tim` to verify protection against path traversal (Zip Slip) attacks and handling of invalid inputs.
- Fixed `cmd` package tests execution by adding `TestHelperProcess` to `cmd/run_test.go` to support mocked command execution.
- Increased coverage for `pkg/datanode` to 84.2%, `pkg/tim` to 74.2%, and `cmd` to 44.1%.
This commit introduces the `Enchantrix` library to add support for the `.trix` encrypted file format.
The main changes are:
- The `matrix` format has been renamed to `tim` (Terminal Isolation Matrix).
- The `.tim` format is now a specialized `.trix` file.
- A new `decode` command has been added to decode `.trix` and `.tim` files.
- The `collect` commands now support the `trix` and `tim` formats.
- A `--password` flag has been added to the `collect` commands for encryption.
- A `--i-am-in-isolation` flag has been added to the `decode` command for safely decoding `.tim` files.
- The decryption functionality is currently disabled due to a bug in the `Enchantrix` library. A follow-up PR will be created to re-enable it.
- Path traversal vulnerability in `pkg/tim/run.go` has been fixed.
- File descriptor leak in `pkg/tim/run.go` has been fixed.
- Improved error handling in `pkg/trix/trix.go`.
This commit introduces the `Enchantrix` library to add support for the `.trix` encrypted file format.
The main changes are:
- The `matrix` format has been renamed to `tim` (Terminal Isolation Matrix).
- The `.tim` format is now a specialized `.trix` file.
- A new `decode` command has been added to decode `.trix` and `.tim` files.
- The `collect` commands now support the `trix` and `tim` formats.
- A `--password` flag has been added to the `collect` commands for encryption.
- A `--i-am-in-isolation` flag has been added to the `decode` command for safely decoding `.tim` files.
- The decryption functionality is currently disabled due to a bug in the `Enchantrix` library. A follow-up PR will be created to re-enable it.
- Implements all placeholder Go examples in the `examples` directory.
- Corrects the `run_matrix_programmatically` example to use the `borg` package.
- Refactors the code to centralize the matrix execution logic in the `matrix` package.
- Updates the documentation to include a new "Programmatic Usage" section that describes all of the Go examples.
- Updates the "Terminal Isolation Matrix" section to remove manual 'runc' instructions, emphasizing that 'borg run' handles this process to maintain security and isolation.
- Adds missing examples for 'collect github repos', 'collect github release', and 'compile' commands to the documentation.
- Makes `pkg/matrix.Run` testable by exposing `exec.Command` as a public variable.
- Adds tests for the `matrix` package that mock the `runc` command.
- Updates the `cmd` package tests to mock `matrix.ExecCommand` instead of the old `cmd.execCommand`.
- Implements all placeholder Go examples in the `examples` directory.
- Corrects the `run_matrix_programmatically` example to use the `borg` package.
- Refactors the code to centralize the matrix execution logic in the `matrix` package.
- Updates the documentation to include a new "Programmatic Usage" section that describes all of the Go examples.
- Updates the "Terminal Isolation Matrix" section to remove manual 'runc' instructions, emphasizing that 'borg run' handles this process to maintain security and isolation.
- Adds missing examples for 'collect github repos', 'collect github release', and 'compile' commands to the documentation.
Refactored the existing tests to use the `_Good`, `_Bad`, and `_Ugly`
testing convention. This provides a more structured approach to testing
and ensures that a wider range of scenarios are covered, including
valid inputs, invalid inputs, and edge cases.
In addition to refactoring the tests, this change also includes several
bug fixes that were uncovered by the new tests. These fixes improve the
robustness and reliability of the codebase.
The following packages and commands were affected:
- `pkg/datanode`
- `pkg/compress`
- `pkg/github`
- `pkg/matrix`
- `pkg/pwa`
- `pkg/vcs`
- `pkg/website`
- `cmd/all`
- `cmd/collect`
- `cmd/collect_github_repo`
- `cmd/collect_website`
- `cmd/compile`
- `cmd/root`
- `cmd/run`
- `cmd/serve`
This reverts the following commits:
- feat: Increase test coverage for pkg/datanode
- feat: Increase test coverage for pkg/compress
- feat: Increase test coverage for pkg/pwa
- feat: Increase test coverage for pkg/website
- feat: Increase test coverage for pkg/vcs
These changes are being reverted because they were causing test failures
and were not contributing to the overall stability of the project.
This commit addresses three separate issues:
1. **cmd/compile.go**: The `Borgfile` parsing now returns an error for
unknown instructions instead of silently ignoring them.
2. **cmd/run_test.go**: The `execCommand` mock is now properly cleaned
up after the test, preventing it from leaking into other tests.
3. **pkg/matrix/matrix_test.go**: The EOF check in the tar reader loop
now uses a direct error comparison (`io.EOF`) instead of a string
comparison.
This commit increases the test coverage for the `pkg/datanode` package
from 58.1% to 66.9%.
The following changes were made:
- Added a new test file, `pkg/datanode/fromtar_test.go`, to test the
`FromTar` function.
- Added a new test file, `pkg/datanode/fs_test.go`, to test the `Stat`,
`Exists`, and `Walk` functions.
- Refactored the existing tests in `pkg/datanode/datanode_test.go` to
remove the monolithic test function and improve the organization of
the tests.
This commit adds placeholder Go programs to the `examples` directory for
all of Borg's features. This provides a clear roadmap for future
implementation and ensures that we have a testing strategy for each
function.
The new placeholder examples are:
- `examples/all`
- `examples/collect_github_release`
- `examples/collect_github_repo`
- `examples/collect_github_repos`
- `examples/collect_pwa`
- `examples/collect_website`
- `examples/serve`
This commit adds two new Go programs to the `examples` directory to
demonstrate how to use the `pkg/matrix` and `pkg/datanode` packages
programmatically:
- `examples/create_matrix_programmatically`: Shows how to create a
`.matrix` file from scratch.
- `examples/run_matrix_programmatically`: Shows how to unpack and execute
a `.matrix` file using `runc`.
The `examples` directory has been reorganized to place each runnable
example in its own subdirectory, which is the standard Go practice for
handling multiple `main` packages.
This commit introduces two new commands to the `borg` CLI:
- `borg compile`: Compiles a `Borgfile` into a "Terminal Isolation
Matrix" (`.matrix` file). The `Borgfile` format is a simple text file
with `ADD <src> <dest>` instructions.
- `borg run`: Executes a `.matrix` file using `runc`. The command
unpacks the matrix into a temporary directory and then executes `runc
run`.
This commit also adds comprehensive tests for the new commands and the
`pkg/matrix` package, significantly increasing the overall test
coverage. The tests for the `run` command use a mocking pattern to
avoid environment dependencies.
This commit introduces two new commands to the `borg` CLI:
- `borg compile`: Compiles a `Borgfile` into a "Terminal Isolation
Matrix" (`.matrix` file). The `Borgfile` format is a simple text file
with `ADD <src> <dest>` instructions.
- `borg run`: Executes a `.matrix` file using `runc`. The command
unpacks the matrix into a temporary directory and then executes `runc
run`.
Tests have been added for both commands, mocking the `runc` execution
to avoid environment dependencies.
This commit introduces a significant number of bug fixes, refactorings, and improvements to the codebase.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
- Addressed a resource leak in the `all` command.
- Improved error handling in the `pwa` package.
- Refactored `main.go` to remove duplicated logic.
- Fixed several other minor bugs and inconsistencies.
- Made tests platform-independent by removing hardcoded `/dev/null` paths.
- Fixed potential panics in tests by adding `nil` checks for errors.
- Fixed test state leakage by using `t.Cleanup` to restore mocked package-level variables.
- Added validation for command-line flags.
- Ensured output directories are created before writing files.
- Fixed naming conventions for Cobra command constructors.
- Consolidated tests for `ParseRepoFromURL`.
- Improved test failure messages.
- Added validation for release tags.
- Aggregated errors during asset downloads.
This commit introduces a significant refactoring of the `cmd` package to improve testability and increases test coverage across the application.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
- Addressed a resource leak in the `all` command.
- Improved error handling in the `pwa` package.
- Refactored `main.go` to remove duplicated logic.
- Fixed several other minor bugs and inconsistencies.
- Made tests platform-independent by removing hardcoded `/dev/null` paths.
- Fixed potential panics in tests by adding `nil` checks for errors.
- Fixed test state leakage by using `t.Cleanup` to restore mocked package-level variables.
This commit introduces a significant refactoring of the `cmd` package to improve testability and increases test coverage across the application.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
- Addressed a resource leak in the `all` command.
- Improved error handling in the `pwa` package.
- Refactored `main.go` to remove duplicated logic.
- Fixed several other minor bugs and inconsistencies.
This commit introduces a significant refactoring of the `cmd` package to improve testability and increases test coverage across the application.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
This commit refactors the codebase to use dependency injection for mocking external dependencies, removing the need for in-code mocking with the `BORG_PLEXSUS` environment variable.
- Interfaces have been created for external dependencies in the `pkg/vcs`, `pkg/github`, and `pkg/pwa` packages.
- The `cmd` package has been refactored to use these interfaces, with dependencies exposed as public variables for easy mocking in tests.
- The tests in `TDD/collect_commands_test.go` have been updated to inject mock implementations of these interfaces.
- The `BORG_PLEXSUS` environment variable has been removed from the codebase.
