This commit introduces a new feature to extract and index metadata from collected PDF files.
The following changes have been made:
- Added a new `pdf` command with a `metadata` subcommand to extract metadata from a single PDF file.
- Added a new `extract-metadata` command to extract metadata from all PDF files within a given archive and create an `INDEX.json` file.
- Added a `--extract-pdf-metadata` flag to the `collect website` command to extract metadata from downloaded PDF files.
- Created a new `pdf` package to encapsulate the PDF metadata extraction logic, which uses the `pdfinfo` command from the `poppler-utils` package.
- Added unit tests for the new `pdf` package, including mocking the `pdfinfo` command.
- Modified `Taskfile.yml` to install `poppler-utils` as a dependency.
Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
Add multi-quality variant support for video content:
- New ABR types in pkg/smsg/types.go (ABRManifest, Variant, ABRPresets)
- New pkg/smsg/abr.go with manifest read/write and bandwidth estimation
- New cmd/mkdemo-abr CLI tool for creating ABR variant sets via ffmpeg
- WASM parseABRManifest and selectVariant functions
- Demo page "Adaptive Quality" tab with ABR player
- RFC-001 Section 3.7 documenting ABR format and algorithm
Profile page:
- No WASM or video download until play button clicked
- Play button visible immediately, loading on-demand
- Removed auto-play behavior completely
Streaming:
- GetV3HeaderFromPrefix for parsing from partial data
- v3 demo file with 128KB chunks for streaming tests
- cmd/dapp-fm-app: Native desktop app with WebView (Wails)
- cmd/dapp-fm: CLI binary for HTTP server mode
- pkg/player: Shared player core with Go bindings
Architecture: Go decrypts SMSG content, serves via asset handler.
Frontend calls Go directly via Wails bindings for manifest/license
checks.
- Added unit tests for `ToTar` and `FromTar` in `pkg/datanode`, including a round-trip test and invalid input handling.
- Added unit tests for `Walk` options (`MaxDepth`, `Filter`, `SkipErrors`) in `pkg/datanode`.
- Added security tests for `pkg/tim` to verify protection against path traversal (Zip Slip) attacks and handling of invalid inputs.
- Fixed `cmd` package tests execution by adding `TestHelperProcess` to `cmd/run_test.go` to support mocked command execution.
- Increased coverage for `pkg/datanode` to 84.2%, `pkg/tim` to 74.2%, and `cmd` to 44.1%.
This commit introduces the `Enchantrix` library to add support for the `.trix` encrypted file format.
The main changes are:
- The `matrix` format has been renamed to `tim` (Terminal Isolation Matrix).
- The `.tim` format is now a specialized `.trix` file.
- A new `decode` command has been added to decode `.trix` and `.tim` files.
- The `collect` commands now support the `trix` and `tim` formats.
- A `--password` flag has been added to the `collect` commands for encryption.
- A `--i-am-in-isolation` flag has been added to the `decode` command for safely decoding `.tim` files.
- The decryption functionality is currently disabled due to a bug in the `Enchantrix` library. A follow-up PR will be created to re-enable it.
- Implements all placeholder Go examples in the `examples` directory.
- Corrects the `run_matrix_programmatically` example to use the `borg` package.
- Refactors the code to centralize the matrix execution logic in the `matrix` package.
- Updates the documentation to include a new "Programmatic Usage" section that describes all of the Go examples.
- Updates the "Terminal Isolation Matrix" section to remove manual 'runc' instructions, emphasizing that 'borg run' handles this process to maintain security and isolation.
- Adds missing examples for 'collect github repos', 'collect github release', and 'compile' commands to the documentation.
- Makes `pkg/matrix.Run` testable by exposing `exec.Command` as a public variable.
- Adds tests for the `matrix` package that mock the `runc` command.
- Updates the `cmd` package tests to mock `matrix.ExecCommand` instead of the old `cmd.execCommand`.
- Implements all placeholder Go examples in the `examples` directory.
- Corrects the `run_matrix_programmatically` example to use the `borg` package.
- Refactors the code to centralize the matrix execution logic in the `matrix` package.
- Updates the documentation to include a new "Programmatic Usage" section that describes all of the Go examples.
- Updates the "Terminal Isolation Matrix" section to remove manual 'runc' instructions, emphasizing that 'borg run' handles this process to maintain security and isolation.
- Adds missing examples for 'collect github repos', 'collect github release', and 'compile' commands to the documentation.
Refactored the existing tests to use the `_Good`, `_Bad`, and `_Ugly`
testing convention. This provides a more structured approach to testing
and ensures that a wider range of scenarios are covered, including
valid inputs, invalid inputs, and edge cases.
In addition to refactoring the tests, this change also includes several
bug fixes that were uncovered by the new tests. These fixes improve the
robustness and reliability of the codebase.
The following packages and commands were affected:
- `pkg/datanode`
- `pkg/compress`
- `pkg/github`
- `pkg/matrix`
- `pkg/pwa`
- `pkg/vcs`
- `pkg/website`
- `cmd/all`
- `cmd/collect`
- `cmd/collect_github_repo`
- `cmd/collect_website`
- `cmd/compile`
- `cmd/root`
- `cmd/run`
- `cmd/serve`
This reverts the following commits:
- feat: Increase test coverage for pkg/datanode
- feat: Increase test coverage for pkg/compress
- feat: Increase test coverage for pkg/pwa
- feat: Increase test coverage for pkg/website
- feat: Increase test coverage for pkg/vcs
These changes are being reverted because they were causing test failures
and were not contributing to the overall stability of the project.
This commit addresses three separate issues:
1. **cmd/compile.go**: The `Borgfile` parsing now returns an error for
unknown instructions instead of silently ignoring them.
2. **cmd/run_test.go**: The `execCommand` mock is now properly cleaned
up after the test, preventing it from leaking into other tests.
3. **pkg/matrix/matrix_test.go**: The EOF check in the tar reader loop
now uses a direct error comparison (`io.EOF`) instead of a string
comparison.
This commit introduces two new commands to the `borg` CLI:
- `borg compile`: Compiles a `Borgfile` into a "Terminal Isolation
Matrix" (`.matrix` file). The `Borgfile` format is a simple text file
with `ADD <src> <dest>` instructions.
- `borg run`: Executes a `.matrix` file using `runc`. The command
unpacks the matrix into a temporary directory and then executes `runc
run`.
This commit also adds comprehensive tests for the new commands and the
`pkg/matrix` package, significantly increasing the overall test
coverage. The tests for the `run` command use a mocking pattern to
avoid environment dependencies.
This commit introduces two new commands to the `borg` CLI:
- `borg compile`: Compiles a `Borgfile` into a "Terminal Isolation
Matrix" (`.matrix` file). The `Borgfile` format is a simple text file
with `ADD <src> <dest>` instructions.
- `borg run`: Executes a `.matrix` file using `runc`. The command
unpacks the matrix into a temporary directory and then executes `runc
run`.
Tests have been added for both commands, mocking the `runc` execution
to avoid environment dependencies.
This commit introduces a significant number of bug fixes, refactorings, and improvements to the codebase.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
- Addressed a resource leak in the `all` command.
- Improved error handling in the `pwa` package.
- Refactored `main.go` to remove duplicated logic.
- Fixed several other minor bugs and inconsistencies.
- Made tests platform-independent by removing hardcoded `/dev/null` paths.
- Fixed potential panics in tests by adding `nil` checks for errors.
- Fixed test state leakage by using `t.Cleanup` to restore mocked package-level variables.
- Added validation for command-line flags.
- Ensured output directories are created before writing files.
- Fixed naming conventions for Cobra command constructors.
- Consolidated tests for `ParseRepoFromURL`.
- Improved test failure messages.
- Added validation for release tags.
- Aggregated errors during asset downloads.
This commit introduces a significant refactoring of the `cmd` package to improve testability and increases test coverage across the application.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
- Addressed a resource leak in the `all` command.
- Improved error handling in the `pwa` package.
- Refactored `main.go` to remove duplicated logic.
- Fixed several other minor bugs and inconsistencies.
- Made tests platform-independent by removing hardcoded `/dev/null` paths.
- Fixed potential panics in tests by adding `nil` checks for errors.
- Fixed test state leakage by using `t.Cleanup` to restore mocked package-level variables.
This commit introduces a significant refactoring of the `cmd` package to improve testability and increases test coverage across the application.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
- Addressed a resource leak in the `all` command.
- Improved error handling in the `pwa` package.
- Refactored `main.go` to remove duplicated logic.
- Fixed several other minor bugs and inconsistencies.
This commit introduces a significant refactoring of the `cmd` package to improve testability and increases test coverage across the application.
Key changes include:
- Refactored Cobra commands to use `RunE` for better error handling and testing.
- Extracted business logic from command handlers into separate, testable functions.
- Added comprehensive unit tests for the `cmd`, `compress`, `github`, `logger`, and `pwa` packages.
- Added tests for missing command-line arguments, as requested.
- Implemented the `borg all` command to clone all public repositories for a GitHub user or organization.
- Restored and improved the `collect pwa` functionality.
- Removed duplicate code and fixed various bugs.
This commit refactors the codebase to use dependency injection for mocking external dependencies, removing the need for in-code mocking with the `BORG_PLEXSUS` environment variable.
- Interfaces have been created for external dependencies in the `pkg/vcs`, `pkg/github`, and `pkg/pwa` packages.
- The `cmd` package has been refactored to use these interfaces, with dependencies exposed as public variables for easy mocking in tests.
- The tests in `TDD/collect_commands_test.go` have been updated to inject mock implementations of these interfaces.
- The `BORG_PLEXSUS` environment variable has been removed from the codebase.
This commit introduces a TDD testing framework for the `collect` commands and resolves a build failure.
- A `TDD/` directory has been added to house tests for the `collect` commands.
- An environment variable `BORG_PLEXSUS=0` has been implemented to enable a mock mode, which prevents external network calls during testing.
- The `collect` commands have been updated to use the command's output streams, allowing for output capturing in tests.
- A `pkg/mocks` package has been added to provide mock implementations for testing.
- The `.gitignore` file has been updated to exclude generated `.datanode` files.
- The "flag redefined" build error has been fixed by refactoring the root command initialization in `cmd/root.go` to prevent duplicate flag definitions.
This commit introduces a TDD testing framework for the `collect` commands.
- A `TDD/` directory has been added to house the tests.
- An environment variable `BORG_PLEXSUS=0` has been implemented to enable a mock mode, which prevents external network calls during testing.
- The `collect` commands have been updated to use the command's output streams, allowing for output capturing in tests.
- A `pkg/mocks` package has been added to provide mock implementations for testing.
- The `.gitignore` file has been updated to exclude generated `.datanode` files.
This commit introduces a non-interactive mode for the `collect` commands. When running in a non-interactive session, the progress bar is replaced with a series of thematic Borg quotes, printed in matrix-green text.
The quotes are sourced from a `quotes.json` file, which is embedded into the binary using Go's `embed` package. The `pkg/ui` package now contains a `NonInteractivePrompter` that detects the session type and displays the quotes accordingly. The `collect` commands have been updated to use this new prompter, and the underlying `vcs` and `website` packages have been made more robust to handle cases where a progress bar is not provided.
This commit introduces a new `collect github repos` command to the CLI. This command allows users to fetch all public repositories for a given user or organization.
Key changes:
- Added a new command file `cmd/collect_github_repos.go`.
- Updated the documentation in `docs/index.md` to include the new command and the `collect github release` command.
This change introduces optional compression to the `collect` commands. Users can now specify `--compression` with `gz` or `xz` to compress the output. The `serve` command has also been enhanced to transparently decompress and serve these files.
This change enhances the 'serve' command to support serving files directly from a Terminal Isolation Matrix. It introduces a new 'pkg/tarfs' package that provides an http.FileSystem implementation for tar archives, allowing for a "passthrough" server that serves files directly from the Matrix bundle.
This change introduces the "Terminal Isolation Matrix", a new output format that creates a runc-compatible container bundle. This allows for the collected files to be run in an isolated environment. A --format flag has been added to all collect commands to support this new format.
This commit addresses several issues identified in a code review to improve the overall quality and robustness of the application.
Key changes include:
- Added safe type assertions with `nil` checks when retrieving the logger from the context to prevent panics.
- Moved the `bar.Finish()` call to be inside the loop in the `all` command, so each progress bar finishes after its corresponding repository is cloned.
- Added a check for context cancellation at the start of the pagination loop in the GitHub client to prevent unnecessary API calls.
- Ensured the authenticated client is used consistently, even when falling back to the organization endpoint.
- Added `nil` checks for the progress bar parameter in the `website` and `pwa` packages to prevent panics.
- Updated the `golang.org/x/oauth2` dependency to a patched release to address a reported vulnerability.
This commit improves the user experience of the application by adding progress bars to long-running operations.
The following commands now display a progress bar:
- `collect github repo`
- `collect website`
- `collect pwa`
The underlying packages (`pkg/vcs`, `pkg/website`, and `pkg/pwa`) have been updated to support progress reporting.
This commit introduces two key improvements to the application:
1. **Authenticated GitHub API Access:** The GitHub client now uses a personal access token (PAT) from the `GITHUB_TOKEN` environment variable if it is available. This increases the rate limit for GitHub API requests, making the tool more robust for users who need to collect a large number of repositories.
2. **Structured Logging:** The application now uses the standard library's `slog` package for structured logging. A `--verbose` flag has been added to the root command to control the log level, allowing for more detailed output when needed. This makes the application's output more consistent and easier to parse.
This commit addresses several issues identified in a code review to improve the overall quality and robustness of the application.
Key changes include:
- Refactored `cmd.Execute()` to return an error instead of calling `os.Exit`, making the application more testable.
- Fixed critical issues in `cmd/main_test.go`, including renaming `TestMain` to avoid conflicts and removing the brittle E2E test.
- Improved the GitHub API client in `pkg/github/github.go` by:
- Fixing a resource leak where an HTTP response body was not being closed.
- Restoring a parameterized function to improve testability.
- Adding support for `context.Context` and API pagination for robustness.
- Updated the `.github/workflows/go.yml` CI workflow to use the `Taskfile.yml` for building and testing, ensuring consistency.
- Added a `test` task to `Taskfile.yml`.
- Ran `go mod tidy` and fixed several unused import errors.
This commit fixes a failing end-to-end test (`TestE2E`) by implementing a more robust method for locating the `task` executable.
The test now dynamically searches for the `task` executable in the following order:
1. The system's `PATH`
2. The user's Go binary directory (`$HOME/go/bin`)
This change ensures that the test can run successfully in different environments, including CI, where the location of the `task` executable may not be consistent.
This commit fixes a critical build issue where the application was being compiled as an archive instead of an executable. This was caused by the absence of a `main` package.
The following changes have been made to resolve this and improve the development process:
- A `main.go` file has been added to the root of the project to serve as the application's entry point.
- A `Taskfile.yml` has been introduced to standardize the build, run, and testing processes.
- The build process has been corrected to produce a runnable binary.
- An end-to-end test (`TestE2E`) has been added to the test suite. This test builds the application and runs it with the `--help` flag to ensure the binary is always executable, preventing similar build regressions in the future.
This commit updates the `collect pwa` command to use the `--uri` and `--output` flags instead of positional arguments. This makes the command's usage more consistent with the other `collect` subcommands.
The documentation for the `collect pwa` command has also been updated to reflect the new flags and their usage.
This commit introduces a new `collect github` command with two subcommands:
- `repo`: Clones a GitHub repository. This functionality was previously under the `collect git` command.
- `release`: Downloads assets from the latest GitHub release of a repository.
The `release` subcommand supports the following features:
- Version checking against a provided version string using the `--version` flag.
- Downloading a specific file from the release using the `--file` flag.
- Downloading all assets from the release and packing them into a DataNode using the `--pack` flag.
- Specifying an output directory for the downloaded files using the `--output` flag.
This commit also resolves a merge conflict by restructuring the `collect` command and its subcommands. The `collect git` and `collect github-release` commands have been consolidated under the new `collect github` command to provide a more organized and user-friendly command structure.
This commit introduces a new command `collect github-release` that allows downloading assets from the latest GitHub release of a repository.
The command supports the following features:
- Downloading a specific file from the release using the `--file` flag.
- Downloading all assets from the release and packing them into a DataNode using the `--pack` flag.
- Specifying an output directory for the downloaded files using the `--output` flag.
This commit also includes a project-wide refactoring of the Go module path to `github.com/Snider/Borg` to align with Go's module system best practices.
This commit adds a `--user` flag to the `collect git` command, allowing it to fetch all public repositories for a given GitHub user or organization.
- The `collect git` command was modified to accept a `--user` flag.
- The previous `all` command's functionality has been integrated into the `collect git` command.
- The documentation and examples have been updated to reflect the new flag.
This commit introduces several maintenance improvements to the repository.
- A `go.work` file has been added to define the workspace and make the project easier to work with.
- The module path in `go.mod` has been updated to use a GitHub URL, and all import paths have been updated accordingly.
- `examples` and `docs` directories have been created.
- The `examples` directory contains scripts that demonstrate the tool's functionality.
- The `docs` directory contains documentation for the project.
- Tests have been added to the `pkg/github` package following the `_Good`, `_Bad`, `_Ugly` convention.
- The missing `pkg/borg` package has been added to resolve a build error.
This commit introduces a new `collect website` command that recursively downloads a website to a specified depth.
- A new `pkg/website` package contains the logic for the recursive download.
- A new `pkg/ui` package provides a progress bar for long-running operations, which is used by the website downloader.
- The `collect pwa` subcommand has been restored to be PWA-specific.
