This commit introduces a deduplication cache to avoid re-downloading files across multiple collection jobs.
Key changes include:
- A new `pkg/cache` package that provides content-addressable storage using SHA256 hashes of the file content.
- Integration of the cache into the `collect website` command. Downloads are now skipped if the content already exists in the cache.
- The addition of `--no-cache` and `--cache-dir` flags to give users control over the caching behavior.
- New `borg cache stats` and `borg cache clear` commands to allow users to manage the cache.
- A performance improvement to the cache implementation, which now only writes the URL-to-hash index file once at the end of the collection process, rather than on every file download.
- Centralized logic for determining the default cache directory, removing code duplication.
- Improved error handling and refactored duplicated cache-checking logic in the website collector.
- Added comprehensive unit tests for the new cache package and an integration test to verify that the website collector correctly uses the cache.
The implementation of cache size limiting and LRU eviction is still pending and will be addressed in a future commit.
Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
- cmd/dapp-fm-app: Native desktop app with WebView (Wails)
- cmd/dapp-fm: CLI binary for HTTP server mode
- pkg/player: Shared player core with Go bindings
Architecture: Go decrypts SMSG content, serves via asset handler.
Frontend calls Go directly via Wails bindings for manifest/license
checks.
This commit introduces the `Enchantrix` library to add support for the `.trix` encrypted file format.
The main changes are:
- The `matrix` format has been renamed to `tim` (Terminal Isolation Matrix).
- The `.tim` format is now a specialized `.trix` file.
- A new `decode` command has been added to decode `.trix` and `.tim` files.
- The `collect` commands now support the `trix` and `tim` formats.
- A `--password` flag has been added to the `collect` commands for encryption.
- A `--i-am-in-isolation` flag has been added to the `decode` command for safely decoding `.tim` files.
- The decryption functionality is currently disabled due to a bug in the `Enchantrix` library. A follow-up PR will be created to re-enable it.
- Implements all placeholder Go examples in the `examples` directory.
- Corrects the `run_matrix_programmatically` example to use the `borg` package.
- Refactors the code to centralize the matrix execution logic in the `matrix` package.
- Updates the documentation to include a new "Programmatic Usage" section that describes all of the Go examples.
- Updates the "Terminal Isolation Matrix" section to remove manual 'runc' instructions, emphasizing that 'borg run' handles this process to maintain security and isolation.
- Adds missing examples for 'collect github repos', 'collect github release', and 'compile' commands to the documentation.
This commit adds placeholder Go programs to the `examples` directory for
all of Borg's features. This provides a clear roadmap for future
implementation and ensures that we have a testing strategy for each
function.
The new placeholder examples are:
- `examples/all`
- `examples/collect_github_release`
- `examples/collect_github_repo`
- `examples/collect_github_repos`
- `examples/collect_pwa`
- `examples/collect_website`
- `examples/serve`
This commit adds two new Go programs to the `examples` directory to
demonstrate how to use the `pkg/matrix` and `pkg/datanode` packages
programmatically:
- `examples/create_matrix_programmatically`: Shows how to create a
`.matrix` file from scratch.
- `examples/run_matrix_programmatically`: Shows how to unpack and execute
a `.matrix` file using `runc`.
The `examples` directory has been reorganized to place each runnable
example in its own subdirectory, which is the standard Go practice for
handling multiple `main` packages.
This change introduces optional compression to the `collect` commands. Users can now specify `--compression` with `gz` or `xz` to compress the output. The `serve` command has also been enhanced to transparently decompress and serve these files.
This change enhances the 'serve' command to support serving files directly from a Terminal Isolation Matrix. It introduces a new 'pkg/tarfs' package that provides an http.FileSystem implementation for tar archives, allowing for a "passthrough" server that serves files directly from the Matrix bundle.
This change introduces the "Terminal Isolation Matrix", a new output format that creates a runc-compatible container bundle. This allows for the collected files to be run in an isolated environment. A --format flag has been added to all collect commands to support this new format.
This change adds a 'docs/README.md' file with command-line documentation and an 'examples' directory containing scripts to demonstrate the tool's usage. It also includes a Go program to inspect the contents of a DataNode.
This commit fixes a critical build issue where the application was being compiled as an archive instead of an executable. This was caused by the absence of a `main` package.
The following changes have been made to resolve this and improve the development process:
- A `main.go` file has been added to the root of the project to serve as the application's entry point.
- A `Taskfile.yml` has been introduced to standardize the build, run, and testing processes.
- The build process has been corrected to produce a runnable binary.
- An end-to-end test (`TestE2E`) has been added to the test suite. This test builds the application and runs it with the `--help` flag to ensure the binary is always executable, preventing similar build regressions in the future.
This commit adds a `--user` flag to the `collect git` command, allowing it to fetch all public repositories for a given GitHub user or organization.
- The `collect git` command was modified to accept a `--user` flag.
- The previous `all` command's functionality has been integrated into the `collect git` command.
- The documentation and examples have been updated to reflect the new flag.
This commit introduces several maintenance improvements to the repository.
- A `go.work` file has been added to define the workspace and make the project easier to work with.
- The module path in `go.mod` has been updated to use a GitHub URL, and all import paths have been updated accordingly.
- `examples` and `docs` directories have been created.
- The `examples` directory contains scripts that demonstrate the tool's functionality.
- The `docs` directory contains documentation for the project.
- Tests have been added to the `pkg/github` package following the `_Good`, `_Bad`, `_Ugly` convention.
- The missing `pkg/borg` package has been added to resolve a build error.
