Borg/pkg/website
google-labs-jules[bot] b36990cdec feat: Sitemap.xml parsing for website collection
This feature adds sitemap.xml parsing to the `borg collect website` command.

It introduces three new flags:
- `--use-sitemap`: Auto-detects and uses the sitemap in combination with crawling.
- `--sitemap-only`: Collects only the URLs found in the sitemap.
- `--sitemap`: Specifies an explicit URL for the sitemap.

The implementation supports standard sitemaps, sitemap indexes, and compressed sitemaps (.xml.gz).

Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
2026-02-02 00:48:52 +00:00
..
sitemap.go feat: Sitemap.xml parsing for website collection 2026-02-02 00:48:52 +00:00
sitemap_test.go feat: Sitemap.xml parsing for website collection 2026-02-02 00:48:52 +00:00
website.go feat: Sitemap.xml parsing for website collection 2026-02-02 00:48:52 +00:00
website_test.go feat: Sitemap.xml parsing for website collection 2026-02-02 00:48:52 +00:00