This commit addresses several issues identified in a code review to improve the overall quality and robustness of the application.
Key changes include:
- Added safe type assertions with `nil` checks when retrieving the logger from the context to prevent panics.
- Moved the `bar.Finish()` call to be inside the loop in the `all` command, so each progress bar finishes after its corresponding repository is cloned.
- Added a check for context cancellation at the start of the pagination loop in the GitHub client to prevent unnecessary API calls.
- Ensured the authenticated client is used consistently, even when falling back to the organization endpoint.
- Added `nil` checks for the progress bar parameter in the `website` and `pwa` packages to prevent panics.
- Updated the `golang.org/x/oauth2` dependency to a patched release to address a reported vulnerability.
This commit introduces two key improvements to the application:
1. **Authenticated GitHub API Access:** The GitHub client now uses a personal access token (PAT) from the `GITHUB_TOKEN` environment variable if it is available. This increases the rate limit for GitHub API requests, making the tool more robust for users who need to collect a large number of repositories.
2. **Structured Logging:** The application now uses the standard library's `slog` package for structured logging. A `--verbose` flag has been added to the root command to control the log level, allowing for more detailed output when needed. This makes the application's output more consistent and easier to parse.
This commit fixes a critical build issue where the application was being compiled as an archive instead of an executable. This was caused by the absence of a `main` package.
The following changes have been made to resolve this and improve the development process:
- A `main.go` file has been added to the root of the project to serve as the application's entry point.
- A `Taskfile.yml` has been introduced to standardize the build, run, and testing processes.
- The build process has been corrected to produce a runnable binary.
- An end-to-end test (`TestE2E`) has been added to the test suite. This test builds the application and runs it with the `--help` flag to ensure the binary is always executable, preventing similar build regressions in the future.
This commit adds a `--user` flag to the `collect git` command, allowing it to fetch all public repositories for a given GitHub user or organization.
- The `collect git` command was modified to accept a `--user` flag.
- The previous `all` command's functionality has been integrated into the `collect git` command.
- The documentation and examples have been updated to reflect the new flag.
This commit introduces a new `collect website` command that recursively downloads a website to a specified depth.
- A new `pkg/website` package contains the logic for the recursive download.
- A new `pkg/ui` package provides a progress bar for long-running operations, which is used by the website downloader.
- The `collect pwa` subcommand has been restored to be PWA-specific.
This commit introduces two new commands: `pwa` and `serve`.
The `pwa` command downloads a Progressive Web Application (PWA) from a given URL. It discovers the PWA's manifest, downloads the assets referenced in the manifest (start URL and icons), and packages them into a single `.tar` file.
The `serve` command takes a `.tar` file created by the `pwa` command and serves its contents using a standard Go HTTP file server. It unpacks the tarball into an in-memory filesystem, making it a self-contained and efficient way to host the downloaded PWA.
This commit introduces the core functionality of the Borg Data Collector.
- Adds the `collect` command to clone a single Git repository and store it in a Trix cube.
- Adds the `collect all` command to clone all public repositories from a GitHub user or organization.
- Implements the Trix cube as a `tar` archive.
- Adds the `ingest` command to add files to a Trix cube.
- Adds the `cat` command to extract files from a Trix cube.
- Integrates Borg-themed status messages for a more engaging user experience.
