92ffddbc0a
This commit introduces a new `collect github repos` command to the CLI. This command allows users to fetch all public repositories for a given user or organization. Key changes: - Added a new command file `cmd/collect_github_repos.go`. - Updated the documentation in `docs/index.md` to include the new command and the `collect github release` command.
4.4 KiB
Borg Data Collector
Borg Data Collector is a command-line tool for collecting and managing data from various sources.
Commands
collect
This command is used to collect resources from different sources and store them in a DataNode.
collect github repo
Collects a single Git repository and stores it in a DataNode.
Usage:
borg collect github repo [repository-url] [flags]
Flags:
--output string: Output file for the DataNode (default "repo.dat")--format string: Output format (datanode or matrix) (default "datanode")--compression string: Compression format (none, gz, or xz) (default "none")
Example:
./borg collect github repo https://github.com/Snider/Borg --output borg.dat
collect github release
Download the latest release of a file from GitHub releases. If the file or URL has a version number, it will check for a higher version and download it if found.
Usage:
borg-data-collector collect github release [repository-url] [flags]
Flags:
--file string: The file to download from the release--output string: Output directory for the downloaded file (default ".")--pack: Pack all assets into a DataNode--version string: The version to check against
collect github repos
Collects all public repositories for a user or organization
Usage:
borg-data-collector collect github repos [user-or-org] [flags]
collect website
Collects a single website and stores it in a DataNode.
Usage:
borg collect website [url] [flags]
Flags:
--output string: Output file for the DataNode (default "website.dat")--depth int: Recursion depth for downloading (default 2)--format string: Output format (datanode or matrix) (default "datanode")--compression string: Compression format (none, gz, or xz) (default "none")
Example:
./borg collect website https://google.com --output website.dat --depth 1
collect pwa
Collects a single PWA and stores it in a DataNode.
Usage:
borg collect pwa [flags]
Flags:
--uri string: The URI of the PWA to collect--output string: Output file for the DataNode (default "pwa.dat")--format string: Output format (datanode or matrix) (default "datanode")--compression string: Compression format (none, gz, or xz) (default "none")
Example:
./borg collect pwa --uri https://squoosh.app --output squoosh.dat
serve
Serves the contents of a packaged DataNode or Terminal Isolation Matrix file using a static file server.
Usage:
borg serve [file] [flags]
Flags:
--port string: Port to serve the DataNode on (default "8080")
Example:
# Serve a DataNode
./borg serve squoosh.dat --port 8888
# Serve a Terminal Isolation Matrix
./borg serve borg.matrix --port 9999
Compression
All collect commands support optional compression. The following compression formats are available:
none: No compression (default)gz: Gzip compressionxz: XZ compression
To use compression, specify the desired format with the --compression flag. The output filename will be automatically updated with the appropriate extension (e.g., .gz, .xz).
Example:
./borg collect github repo https://github.com/Snider/Borg --compression gz
The serve command can transparently serve compressed files.
Terminal Isolation Matrix
The matrix format creates a runc compatible bundle. This bundle can be executed by runc to create a container with the collected files. This is useful for creating isolated environments for testing or analysis.
To create a Matrix, use the --format matrix flag with any of the collect subcommands.
Example:
./borg collect github repo https://github.com/Snider/Borg --output borg.matrix --format matrix
You can then execute the Matrix with runc:
# Create a directory for the bundle
mkdir borg-bundle
# Unpack the matrix into the bundle directory
tar -xf borg.matrix -C borg-bundle
# Run the bundle
cd borg-bundle
runc run borg
Inspecting a DataNode
The examples directory contains a Go program that can be used to inspect the contents of a .dat file.
Usage:
go run examples/inspect_datanode.go <path to .dat file>
Example:
# First, create a .dat file
./borg collect github repo https://github.com/Snider/Borg --output borg.dat
# Then, inspect it
go run examples/inspect_datanode.go borg.dat