e90a84eaa0
Combines three repositories into a single workspace: - go-agent → pkg/orchestrator (Clotho), pkg/jobrunner, pkg/loop, cmd/ - go-agentic → pkg/lifecycle (allowance, sessions, plans, dispatch) - php-devops → repos.yaml, setup.sh, scripts/, .core/ Module path: forge.lthn.ai/core/agent All packages build, all tests pass. Co-Authored-By: Virgil <virgil@lethean.io>
55 lines
1.3 KiB
Markdown
55 lines
1.3 KiB
Markdown
---
|
|
name: website
|
|
description: Crawl and collect a website using Borg
|
|
args: <url> [--depth N] [--format stim|tim|tar] [-o output]
|
|
---
|
|
|
|
# Website Collection
|
|
|
|
Crawl and collect websites using Borg.
|
|
|
|
## Usage
|
|
|
|
```
|
|
/collect:website https://getmasari.org
|
|
/collect:website https://docs.lethean.io --depth 3
|
|
/collect:website https://graft.network --format stim -o graft-site.stim
|
|
```
|
|
|
|
## Action
|
|
|
|
Run Borg to crawl the website:
|
|
|
|
```bash
|
|
borg collect website <url> [--depth <N>] [--format <format>] [-o <output>]
|
|
```
|
|
|
|
Default depth is 2 levels.
|
|
|
|
## Options
|
|
|
|
| Option | Default | Description |
|
|
|--------|---------|-------------|
|
|
| `--depth` | 2 | How many levels deep to crawl |
|
|
| `--format` | tar | Output format (tar, tim, stim) |
|
|
| `-o` | auto | Output filename |
|
|
|
|
## Examples
|
|
|
|
```bash
|
|
# Basic crawl
|
|
borg collect website https://getmasari.org
|
|
|
|
# Deep crawl with encryption
|
|
borg collect website https://docs.lethean.io --depth 5 --format stim -o lethean-docs.stim
|
|
|
|
# Wayback Machine archive
|
|
borg collect website "https://web.archive.org/web/*/graft.network" --depth 3
|
|
```
|
|
|
|
## Use Cases
|
|
|
|
1. **Project Documentation** - Archive docs before they go offline
|
|
2. **Wayback Snapshots** - Collect historical versions
|
|
3. **Forum Threads** - Archive discussion pages
|
|
4. **PWA Collection** - Use `borg collect pwa` for progressive web apps
|