This commit introduces support for routing website collection requests through HTTP/SOCKS5 proxies or Tor. New flags for the `borg collect website` command: - `--proxy <url>`: Route requests through a single HTTP or SOCKS5 proxy. - `--proxy-list <file>`: Select a random proxy from a file for all requests. - `--tor`: Route requests through a Tor SOCKS5 proxy (defaults to 127.0.0.1:9050). Key changes: - Created a new `pkg/httpclient` to centralize the creation of proxy-configured `http.Client` instances. - Refactored `pkg/website` to use a `Downloader` interface, allowing for dependency injection of the HTTP client. - Added validation to ensure the new proxy flags are mutually exclusive. - Implemented support for SOCKS5 authentication via credentials in the proxy URL. - Added comprehensive unit tests for the new functionality. Co-authored-by: Snider <631881+Snider@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| client.go | ||
| client_test.go | ||