docs: add feature request issues for core CLI migration
12 issue files documenting features needed to replace shell scripts: Claude Code hooks: - 001: core ai session (state management) - 002: core ai context (fact capture) - 003: core ai hook (command validation) - 004: core qa debug (debug statement detection) Data collection: - 005: core collect github (issues/PRs archive) - 006: core collect bitcointalk (forum threads) - 007: core collect market (CMC/CoinGecko) - 008: core collect papers (whitepapers) - 009: core collect excavate (project archaeology) - 010: core collect process (HTML→MD) - 011: core collect dispatch (event hooks) 000: Overview tracking issue These will be submitted to host-uk/core when rate limit resets. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Snider
parent
39b2cbde09
commit
beb24f71d2
12 changed files with 788 additions and 0 deletions
66
claude/issues/000-overview.md
Normal file
66
claude/issues/000-overview.md
Normal file
|
|
@ -0,0 +1,66 @@
|
||||||
|
# feat: Replace shell scripts with core CLI commands
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
This tracking issue covers the migration from shell scripts to native `core` CLI commands in `core-agent`. Every shell script should be replaced by a `core` command, making the plugin purely configuration (JSON, MD) + calls to `core`.
|
||||||
|
|
||||||
|
## Philosophy
|
||||||
|
|
||||||
|
- Every agent using `core` tests the framework
|
||||||
|
- Missing features get raised as issues
|
||||||
|
- Code as if functionality exists (TDD approach)
|
||||||
|
- The CLI becomes battle-tested and bulletproof
|
||||||
|
|
||||||
|
## Issues
|
||||||
|
|
||||||
|
### Claude Code Hooks (core ai)
|
||||||
|
|
||||||
|
| Issue | Command | Replaces |
|
||||||
|
|-------|---------|----------|
|
||||||
|
| #001 | `core ai session` | pre-compact.sh, session-start.sh, suggest-compact.sh |
|
||||||
|
| #002 | `core ai context` | capture-context.sh, extract-actionables.sh |
|
||||||
|
| #003 | `core ai hook` | prefer-core.sh, block-docs.sh, post-commit-check.sh |
|
||||||
|
|
||||||
|
### Quality Assurance (core qa)
|
||||||
|
|
||||||
|
| Issue | Command | Replaces |
|
||||||
|
|-------|---------|----------|
|
||||||
|
| #004 | `core qa debug` | check-debug.sh |
|
||||||
|
|
||||||
|
### Data Collection (core collect)
|
||||||
|
|
||||||
|
| Issue | Command | Replaces |
|
||||||
|
|-------|---------|----------|
|
||||||
|
| #005 | `core collect github` | github-history/collect.sh |
|
||||||
|
| #006 | `core collect bitcointalk` | bitcointalk/collect.sh |
|
||||||
|
| #007 | `core collect market` | coinmarketcap/*.sh |
|
||||||
|
| #008 | `core collect papers` | ledger-papers/discover.sh, collect-whitepaper.sh |
|
||||||
|
| #009 | `core collect excavate` | project-archaeology/excavate.sh |
|
||||||
|
| #010 | `core collect process` | job-collector/process.sh |
|
||||||
|
| #011 | `core collect dispatch` | collection/dispatch.sh |
|
||||||
|
|
||||||
|
## Target State
|
||||||
|
|
||||||
|
After all issues are implemented:
|
||||||
|
|
||||||
|
```
|
||||||
|
claude/
|
||||||
|
├── hooks/
|
||||||
|
│ └── hooks.json # Calls core ai hook validate-*
|
||||||
|
├── commands/
|
||||||
|
│ └── remember.md # Calls core ai context add
|
||||||
|
├── collection/
|
||||||
|
│ └── hooks.json # Calls core collect dispatch
|
||||||
|
└── skills/
|
||||||
|
└── */
|
||||||
|
└── SKILL.md # Documentation only, calls core collect *
|
||||||
|
```
|
||||||
|
|
||||||
|
No shell scripts. Just JSON config + markdown docs + `core` CLI calls.
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
1. **Phase 1**: `core ai session` + `core ai context` (enables hooks to work)
|
||||||
|
2. **Phase 2**: `core ai hook` + `core qa debug` (safety + quality)
|
||||||
|
3. **Phase 3**: `core collect github` + `core collect bitcointalk` (most used)
|
||||||
|
4. **Phase 4**: Remaining collection commands
|
||||||
46
claude/issues/001-ai-session.md
Normal file
46
claude/issues/001-ai-session.md
Normal file
|
|
@ -0,0 +1,46 @@
|
||||||
|
# feat(ai): Add session state management for Claude Code hooks
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core ai session` subcommands to manage Claude Code session state, replacing shell scripts that currently handle this.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core ai session save # Save current session state (pre-compact)
|
||||||
|
core ai session restore # Restore session state (session-start)
|
||||||
|
core ai session stats # Track tool calls, suggest compaction
|
||||||
|
core ai session clear # Clear stale session data
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Scripts Being Replaced
|
||||||
|
|
||||||
|
- `claude/scripts/pre-compact.sh` - Saves state before auto-compact
|
||||||
|
- `claude/scripts/session-start.sh` - Restores context on startup
|
||||||
|
- `claude/scripts/suggest-compact.sh` - Suggests compaction at intervals
|
||||||
|
|
||||||
|
## State to Manage
|
||||||
|
|
||||||
|
- Working directory and git branch
|
||||||
|
- Git status (modified files)
|
||||||
|
- In-progress todos
|
||||||
|
- Context facts (decisions, actionables)
|
||||||
|
- Tool call counter per session
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
State stored in `~/.claude/sessions/`:
|
||||||
|
- `scratchpad.md` - Human-readable resume state
|
||||||
|
- `context.json` - Structured context facts
|
||||||
|
- `stats.json` - Session statistics
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"saved": true,
|
||||||
|
"path": "~/.claude/sessions/scratchpad.md",
|
||||||
|
"facts": 5,
|
||||||
|
"tool_calls": 47
|
||||||
|
}
|
||||||
|
```
|
||||||
53
claude/issues/002-ai-context.md
Normal file
53
claude/issues/002-ai-context.md
Normal file
|
|
@ -0,0 +1,53 @@
|
||||||
|
# feat(ai): Add context fact management for Claude Code
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core ai context` subcommands to capture and manage context facts that persist across compaction.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core ai context add <fact> [--source=X] # Save a context fact
|
||||||
|
core ai context list # List current facts
|
||||||
|
core ai context extract <output> # Extract actionables from tool output
|
||||||
|
core ai context clear # Clear old context (>3h)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Scripts Being Replaced
|
||||||
|
|
||||||
|
- `claude/scripts/capture-context.sh` - Stores context facts
|
||||||
|
- `claude/scripts/extract-actionables.sh` - Extracts actionables from core CLI output
|
||||||
|
|
||||||
|
## Context Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{"fact": "Use Action pattern not Service", "source": "user", "ts": 1234567890},
|
||||||
|
{"fact": "FAIL: TestFoo", "source": "core go test", "ts": 1234567891}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Extraction Patterns
|
||||||
|
|
||||||
|
The `extract` command should parse output from:
|
||||||
|
- `core go test` / `core go qa` / `core go lint` - Extract ERROR/WARN/FAIL lines
|
||||||
|
- `core php test` / `core php stan` - Extract FAIL/Error lines
|
||||||
|
- `core build` - Extract error/cannot/undefined lines
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
- `~/.claude/sessions/context.json`
|
||||||
|
- Max 20 items, auto-clear after 3 hours of inactivity
|
||||||
|
|
||||||
|
## Example Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Manual fact
|
||||||
|
core ai context add "User prefers UK English" --source=user
|
||||||
|
|
||||||
|
# Extract from piped output
|
||||||
|
core go test 2>&1 | core ai context extract
|
||||||
|
|
||||||
|
# List facts
|
||||||
|
core ai context list --json
|
||||||
|
```
|
||||||
60
claude/issues/003-ai-hook.md
Normal file
60
claude/issues/003-ai-hook.md
Normal file
|
|
@ -0,0 +1,60 @@
|
||||||
|
# feat(ai): Add hook validation for Claude Code PreToolUse
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core ai hook` subcommands to validate commands and file operations for Claude Code hooks.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core ai hook validate-command <cmd> # Check if command is safe/allowed
|
||||||
|
core ai hook validate-file <path> # Check if file creation is allowed
|
||||||
|
core ai hook post-commit # Check for uncommitted work after commit
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Scripts Being Replaced
|
||||||
|
|
||||||
|
- `claude/hooks/prefer-core.sh` - Blocks dangerous commands, enforces core CLI
|
||||||
|
- `claude/scripts/block-docs.sh` - Blocks random .md file creation
|
||||||
|
- `claude/scripts/post-commit-check.sh` - Warns about uncommitted work
|
||||||
|
|
||||||
|
## Command Validation Rules
|
||||||
|
|
||||||
|
Block these patterns:
|
||||||
|
- `rm -rf` / `rm -r` (except node_modules, vendor, .cache, dist, build)
|
||||||
|
- `mv`/`cp` with wildcards
|
||||||
|
- `xargs` with rm/mv/cp
|
||||||
|
- `find -exec` with file operations
|
||||||
|
- `sed -i` (in-place editing)
|
||||||
|
- `grep -l | ...` (mass file targeting)
|
||||||
|
- `perl -i`, `awk > file`
|
||||||
|
|
||||||
|
Redirect to core:
|
||||||
|
- `go test/build/fmt/mod` → suggest `core go *`
|
||||||
|
- `golangci-lint` → suggest `core go lint`
|
||||||
|
- `composer test` → suggest `core php test`
|
||||||
|
- `./vendor/bin/pint` → suggest `core php fmt`
|
||||||
|
- `php artisan serve` → suggest `core php dev`
|
||||||
|
|
||||||
|
## File Validation Rules
|
||||||
|
|
||||||
|
Allow:
|
||||||
|
- `README.md`, `CLAUDE.md`, `AGENTS.md`, `CONTRIBUTING.md`, `CHANGELOG.md`, `LICENSE.md`
|
||||||
|
- Files in `docs/` directory
|
||||||
|
|
||||||
|
Block:
|
||||||
|
- Other `.md` files (suggest using README.md or docs/)
|
||||||
|
|
||||||
|
## Output Format (JSON for hooks)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"decision": "approve"}
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"decision": "block", "message": "Use `core go test` instead of raw go test"}
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"decision": "warn", "message": "3 files remain uncommitted after commit"}
|
||||||
|
```
|
||||||
58
claude/issues/004-lint-debug.md
Normal file
58
claude/issues/004-lint-debug.md
Normal file
|
|
@ -0,0 +1,58 @@
|
||||||
|
# feat(qa): Add debug statement detection
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core qa debug` command to detect debug statements in code, for use in Claude Code PostToolUse hooks.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core qa debug <file> # Check single file for debug statements
|
||||||
|
core qa debug --staged # Check all staged files
|
||||||
|
core qa debug --changed # Check all changed files
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Script Being Replaced
|
||||||
|
|
||||||
|
- `claude/scripts/check-debug.sh` - Warns about debug statements after edits
|
||||||
|
|
||||||
|
## Detection Patterns
|
||||||
|
|
||||||
|
**Go files (*.go):**
|
||||||
|
- `fmt.Println`
|
||||||
|
- `fmt.Printf` (without format verbs suggesting actual logging)
|
||||||
|
- `log.Println`
|
||||||
|
- `log.Printf`
|
||||||
|
- `spew.Dump`
|
||||||
|
- `pp.Println`
|
||||||
|
|
||||||
|
**PHP files (*.php):**
|
||||||
|
- `dd(`
|
||||||
|
- `dump(`
|
||||||
|
- `var_dump(`
|
||||||
|
- `print_r(`
|
||||||
|
- `ray(`
|
||||||
|
|
||||||
|
**JavaScript/TypeScript (*.js, *.ts, *.tsx):**
|
||||||
|
- `console.log`
|
||||||
|
- `console.debug`
|
||||||
|
- `debugger`
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"file": "src/main.go",
|
||||||
|
"warnings": [
|
||||||
|
{"line": 42, "type": "fmt.Println", "content": "fmt.Println(\"debug\")"},
|
||||||
|
{"line": 87, "type": "log.Println", "content": "log.Println(err)"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
For hooks, output to stderr:
|
||||||
|
```
|
||||||
|
[qa] WARNING: Debug statements found in src/main.go
|
||||||
|
42: fmt.Println("debug")
|
||||||
|
87: log.Println(err)
|
||||||
|
```
|
||||||
71
claude/issues/005-collect-github.md
Normal file
71
claude/issues/005-collect-github.md
Normal file
|
|
@ -0,0 +1,71 @@
|
||||||
|
# feat(collect): Add GitHub history collection
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect github` command to archive GitHub issues and PRs from repositories and organisations.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect github <org/repo> # Collect issues + PRs from repo
|
||||||
|
core collect github <org> --org # Collect all repos in org
|
||||||
|
core collect github <org1,org2> --org # Batch collect multiple orgs
|
||||||
|
core collect github <repo> --issues-only # Issues only
|
||||||
|
core collect github <repo> --prs-only # PRs only
|
||||||
|
core collect github --check-rate # Show rate limit status
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Script Being Replaced
|
||||||
|
|
||||||
|
- `claude/skills/github-history/collect.sh` - 517 lines of bash
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
1. **Rate limit protection**
|
||||||
|
- Check every N calls
|
||||||
|
- Auto-pause at 25% remaining (75% used)
|
||||||
|
- Resume after reset + 10s buffer
|
||||||
|
|
||||||
|
2. **Incremental collection**
|
||||||
|
- Skip already-fetched issues/PRs
|
||||||
|
- Resume interrupted collections
|
||||||
|
|
||||||
|
3. **Reception scoring**
|
||||||
|
- ADDRESSED: Closed after discussion
|
||||||
|
- DISMISSED: Labeled wontfix/invalid
|
||||||
|
- IGNORED: Closed with no response
|
||||||
|
- STALE: Open with no replies
|
||||||
|
- ACTIVE: Open with discussion
|
||||||
|
- MERGED/REJECTED/PENDING for PRs
|
||||||
|
|
||||||
|
4. **Output structure**
|
||||||
|
```
|
||||||
|
repo/{org}/{repo}/
|
||||||
|
├── Issue/
|
||||||
|
│ ├── INDEX.md
|
||||||
|
│ ├── 001.md
|
||||||
|
│ └── 002.md
|
||||||
|
├── PR/
|
||||||
|
│ ├── INDEX.md
|
||||||
|
│ ├── 001.md
|
||||||
|
│ └── 002.md
|
||||||
|
└── .json/
|
||||||
|
├── issues-list.json
|
||||||
|
└── prs-list.json
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Index generation**
|
||||||
|
- Markdown tables with seq, GitHub #, title, score
|
||||||
|
- Score legend
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
Progress to stderr, final summary to stdout:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"repo": "host-uk/core",
|
||||||
|
"issues": 47,
|
||||||
|
"prs": 23,
|
||||||
|
"output": "repo/host-uk/core/"
|
||||||
|
}
|
||||||
|
```
|
||||||
64
claude/issues/006-collect-bitcointalk.md
Normal file
64
claude/issues/006-collect-bitcointalk.md
Normal file
|
|
@ -0,0 +1,64 @@
|
||||||
|
# feat(collect): Add BitcoinTalk thread collection
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect bitcointalk` command to archive BitcoinTalk forum threads.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect bitcointalk <topic-id> # Collect full thread
|
||||||
|
core collect bitcointalk <url> # Collect from URL
|
||||||
|
core collect bitcointalk <id> --pages=5 # Limit pages
|
||||||
|
core collect bitcointalk <id> --output=DIR # Custom output dir
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Script Being Replaced
|
||||||
|
|
||||||
|
- `claude/skills/bitcointalk/collect.sh` - 270 lines of bash + embedded Python
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
1. **Rate limiting**
|
||||||
|
- Respectful delay between requests (default 2s)
|
||||||
|
- Configurable via `--delay=N`
|
||||||
|
|
||||||
|
2. **Post type detection**
|
||||||
|
- ANN: Original announcement (post #1)
|
||||||
|
- UPDATE: Contains [UPDATE]/[RELEASE]/[ANNOUNCEMENT]
|
||||||
|
- QUESTION: Contains question mark in first 200 chars
|
||||||
|
- COMMUNITY: General discussion
|
||||||
|
|
||||||
|
3. **Output structure**
|
||||||
|
```
|
||||||
|
bitcointalk-{topic}/
|
||||||
|
├── INDEX.md
|
||||||
|
├── pages/
|
||||||
|
│ ├── page-0.html
|
||||||
|
│ └── page-20.html
|
||||||
|
└── posts/
|
||||||
|
├── POST-0001.md
|
||||||
|
└── POST-0002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Post metadata**
|
||||||
|
- Author
|
||||||
|
- Date
|
||||||
|
- Post type/score
|
||||||
|
- Original content
|
||||||
|
|
||||||
|
5. **Incremental collection**
|
||||||
|
- Resume interrupted collections
|
||||||
|
- Skip already-fetched pages
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"topic_id": "2769739",
|
||||||
|
"title": "Lethean - Privacy Blockchain VPN",
|
||||||
|
"posts": 1247,
|
||||||
|
"pages": 63,
|
||||||
|
"output": "bitcointalk-2769739/"
|
||||||
|
}
|
||||||
|
```
|
||||||
67
claude/issues/007-collect-market.md
Normal file
67
claude/issues/007-collect-market.md
Normal file
|
|
@ -0,0 +1,67 @@
|
||||||
|
# feat(collect): Add cryptocurrency market data collection
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect market` command to collect historical price/market data from CoinMarketCap, CoinGecko, etc.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect market <coin> # Collect current data
|
||||||
|
core collect market <coin> --historical # Include historical data
|
||||||
|
core collect market <coin> --from=2018-01-01 # Date range
|
||||||
|
core collect market <coin1,coin2> # Multiple coins
|
||||||
|
core collect market jobs <coin> # Generate job list
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Scripts Being Replaced
|
||||||
|
|
||||||
|
- `claude/skills/coinmarketcap/generate-jobs.sh` - Generates CMC collection jobs
|
||||||
|
- `claude/skills/coinmarketcap/process.sh` - Processes collected CMC data
|
||||||
|
|
||||||
|
## Data Sources
|
||||||
|
|
||||||
|
1. **CoinMarketCap**
|
||||||
|
- Main page (description, links)
|
||||||
|
- Markets/exchanges
|
||||||
|
- Historical data
|
||||||
|
- News
|
||||||
|
- API endpoints (where available)
|
||||||
|
|
||||||
|
2. **CoinGecko** (backup)
|
||||||
|
- Main page
|
||||||
|
- API for historical data
|
||||||
|
|
||||||
|
## Output Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
market/{coin}/
|
||||||
|
├── INDEX.md
|
||||||
|
├── cmc/
|
||||||
|
│ ├── main.json
|
||||||
|
│ ├── markets.json
|
||||||
|
│ └── historical.json
|
||||||
|
├── coingecko/
|
||||||
|
│ ├── detail.json
|
||||||
|
│ └── history.json
|
||||||
|
└── summary.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Job Format
|
||||||
|
|
||||||
|
For `jobs` subcommand (pipe to collector):
|
||||||
|
```
|
||||||
|
URL|FILENAME|TYPE|METADATA
|
||||||
|
https://coinmarketcap.com/currencies/lethean/|cmc-lethean-main.html|cmc-main|coin=lethean
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"coin": "lethean",
|
||||||
|
"sources": ["cmc", "coingecko"],
|
||||||
|
"historical_days": 2190,
|
||||||
|
"output": "market/lethean/"
|
||||||
|
}
|
||||||
|
```
|
||||||
68
claude/issues/008-collect-papers.md
Normal file
68
claude/issues/008-collect-papers.md
Normal file
|
|
@ -0,0 +1,68 @@
|
||||||
|
# feat(collect): Add research paper/whitepaper collection
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect papers` command to discover and collect distributed ledger research papers and whitepapers.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect papers --all # All known papers
|
||||||
|
core collect papers --category=cryptography # Filter by category
|
||||||
|
core collect papers --topic=bulletproofs # Filter by topic
|
||||||
|
core collect papers --project=monero # Filter by project
|
||||||
|
core collect papers --search-iacr # Search IACR eprint
|
||||||
|
core collect papers queue <url> # Queue single paper
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Scripts Being Replaced
|
||||||
|
|
||||||
|
- `claude/skills/ledger-papers/discover.sh` - Paper discovery
|
||||||
|
- `claude/collection/collect-whitepaper.sh` - Queue whitepaper collection
|
||||||
|
- `claude/collection/update-index.sh` - Update paper index
|
||||||
|
|
||||||
|
## Categories
|
||||||
|
|
||||||
|
- genesis (Bitcoin, b-money, hashcash, bit gold)
|
||||||
|
- cryptonote (CryptoNote v2.0, CNS standards)
|
||||||
|
- mrl (Monero Research Lab papers)
|
||||||
|
- privacy (Zcash, Dash, Mimblewimble, Lelantus)
|
||||||
|
- smart-contracts (Ethereum, Solana, Cardano)
|
||||||
|
- layer2 (Lightning, Plasma, Rollups)
|
||||||
|
- consensus (PBFT, Tendermint, HotStuff)
|
||||||
|
- cryptography (Bulletproofs, CLSAG, PLONK)
|
||||||
|
- defi (Uniswap, Aave, Compound)
|
||||||
|
- storage (IPFS, Filecoin, Arweave)
|
||||||
|
- identity (DIDs, Verifiable Credentials)
|
||||||
|
- attacks (Selfish mining, eclipse, traceability)
|
||||||
|
|
||||||
|
## Registry
|
||||||
|
|
||||||
|
Papers defined in `registry.json`:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "bulletproofs",
|
||||||
|
"title": "Bulletproofs: Short Proofs for Confidential Transactions",
|
||||||
|
"year": 2017,
|
||||||
|
"url": "https://eprint.iacr.org/2017/1066.pdf",
|
||||||
|
"topics": ["range-proofs", "zero-knowledge"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
Job list for collection:
|
||||||
|
```
|
||||||
|
URL|FILENAME|TYPE|METADATA
|
||||||
|
https://bitcoin.org/bitcoin.pdf|bitcoin.pdf|paper|category=genesis,title=Bitcoin...
|
||||||
|
```
|
||||||
|
|
||||||
|
Summary:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"papers": 91,
|
||||||
|
"categories": 15,
|
||||||
|
"queued": 3,
|
||||||
|
"output": "papers/"
|
||||||
|
}
|
||||||
|
```
|
||||||
92
claude/issues/009-collect-excavate.md
Normal file
92
claude/issues/009-collect-excavate.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
# feat(collect): Add project archaeology/excavation
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect excavate` command to perform deep excavation of abandoned cryptocurrency projects before data is lost.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect excavate <project> # Full excavation
|
||||||
|
core collect excavate <project> --scan-only # Check what's accessible
|
||||||
|
core collect excavate <project> --resume # Resume interrupted dig
|
||||||
|
core collect excavate <project> --only=github,btt # Specific collectors
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Script Being Replaced
|
||||||
|
|
||||||
|
- `claude/skills/project-archaeology/excavate.sh` - 312 lines of bash
|
||||||
|
|
||||||
|
## Excavation Pipeline
|
||||||
|
|
||||||
|
Runs collectors in priority order:
|
||||||
|
|
||||||
|
| Phase | Source | Collector | Priority | Notes |
|
||||||
|
|-------|--------|-----------|----------|-------|
|
||||||
|
| 1 | GitHub repos | `collect github` | P1 | Often deleted first |
|
||||||
|
| 1 | GitHub releases | `collect github` | P1 | Binaries disappear |
|
||||||
|
| 2 | BitcoinTalk ANN | `collect bitcointalk` | P2 | Usually persists |
|
||||||
|
| 2 | Website (Wayback) | `collect wayback` | P2 | Snapshots exist |
|
||||||
|
| 3 | Block explorer | `collect explorer` | P3 | Chain data |
|
||||||
|
| 3 | CoinMarketCap | `collect market` | P3 | Historical prices |
|
||||||
|
| 4 | Reddit | `collect reddit` | P4 | Community context |
|
||||||
|
| 4 | Medium posts | `collect medium` | P4 | Announcements |
|
||||||
|
|
||||||
|
## Project Registry
|
||||||
|
|
||||||
|
Projects defined in `cryptonote-discovery/registry.json`:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "Masari",
|
||||||
|
"symbol": "MSR",
|
||||||
|
"status": "abandoned",
|
||||||
|
"github": ["masari-project"],
|
||||||
|
"bitcointalk": "2769739",
|
||||||
|
"website": "getmasari.org",
|
||||||
|
"explorer": "explorer.getmasari.org",
|
||||||
|
"cmc": "masari"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
digs/{project}/
|
||||||
|
├── EXCAVATION.md # Dig log with timestamps
|
||||||
|
├── SALVAGE-REPORT.md # What's worth keeping
|
||||||
|
├── LESSONS.md # Post-mortem analysis
|
||||||
|
├── github/ # All repo history
|
||||||
|
├── releases/ # Wallet binaries, checksums
|
||||||
|
├── bitcointalk/ # Thread archive
|
||||||
|
├── website/ # Wayback snapshots
|
||||||
|
├── explorer/ # Chain data samples
|
||||||
|
├── market/ # Price history
|
||||||
|
├── papers/ # Whitepapers, docs
|
||||||
|
└── community/ # Reddit, Medium, etc
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scan Mode
|
||||||
|
|
||||||
|
`--scan-only` checks accessibility without downloading:
|
||||||
|
- GitHub org accessible?
|
||||||
|
- BitcoinTalk thread exists?
|
||||||
|
- Wayback snapshots available?
|
||||||
|
- Block explorer online?
|
||||||
|
- CMC page exists?
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"project": "masari",
|
||||||
|
"status": "abandoned",
|
||||||
|
"accessible": {
|
||||||
|
"github": true,
|
||||||
|
"bitcointalk": true,
|
||||||
|
"wayback": true,
|
||||||
|
"explorer": false,
|
||||||
|
"cmc": true
|
||||||
|
},
|
||||||
|
"output": "digs/masari/"
|
||||||
|
}
|
||||||
|
```
|
||||||
70
claude/issues/010-collect-process.md
Normal file
70
claude/issues/010-collect-process.md
Normal file
|
|
@ -0,0 +1,70 @@
|
||||||
|
# feat(collect): Add collected data processing
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect process` command to convert collected HTML/JSON files into clean markdown.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect process <source> <downloads-dir> # Process downloaded files
|
||||||
|
core collect process bitcointalk ./downloads # BitcoinTalk HTML → MD
|
||||||
|
core collect process reddit ./downloads # Reddit JSON → MD
|
||||||
|
core collect process wayback ./downloads # Wayback HTML → MD
|
||||||
|
core collect process medium ./downloads # Medium RSS → MD
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Script Being Replaced
|
||||||
|
|
||||||
|
- `claude/skills/job-collector/process.sh` - 243 lines of bash + embedded Python
|
||||||
|
|
||||||
|
## Supported Sources
|
||||||
|
|
||||||
|
1. **bitcointalk** / **btt**
|
||||||
|
- Input: HTML pages
|
||||||
|
- Extract: posts, authors, dates
|
||||||
|
- Output: POST-NNNN.md files
|
||||||
|
|
||||||
|
2. **reddit**
|
||||||
|
- Input: JSON from Reddit API
|
||||||
|
- Extract: posts, comments, scores
|
||||||
|
- Output: REDDIT-NNNN.md files
|
||||||
|
|
||||||
|
3. **wayback**
|
||||||
|
- Input: HTML from Wayback Machine
|
||||||
|
- Extract: title, body text
|
||||||
|
- Output: {basename}.md files
|
||||||
|
|
||||||
|
4. **medium**
|
||||||
|
- Input: RSS/XML feed
|
||||||
|
- Extract: title, author, date, content
|
||||||
|
- Output: MEDIUM-NNNN.md files
|
||||||
|
|
||||||
|
## Output Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
processed/
|
||||||
|
├── INDEX.md
|
||||||
|
└── posts/
|
||||||
|
├── POST-0001.md
|
||||||
|
├── POST-0002.md
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Index Generation
|
||||||
|
|
||||||
|
Auto-generates INDEX.md with:
|
||||||
|
- Source metadata
|
||||||
|
- Post count
|
||||||
|
- Links to all posts
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"source": "bitcointalk",
|
||||||
|
"input_files": 15,
|
||||||
|
"posts_extracted": 347,
|
||||||
|
"output": "processed/"
|
||||||
|
}
|
||||||
|
```
|
||||||
73
claude/issues/011-collect-dispatch.md
Normal file
73
claude/issues/011-collect-dispatch.md
Normal file
|
|
@ -0,0 +1,73 @@
|
||||||
|
# feat(collect): Add event hook dispatch system
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Add `core collect dispatch` command to fire events during data collection, allowing modular hook handling.
|
||||||
|
|
||||||
|
## Required Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
core collect dispatch <event> [args...] # Fire an event
|
||||||
|
core collect dispatch on_url_found <url> # URL discovered
|
||||||
|
core collect dispatch on_file_collected <file> <type> # File downloaded
|
||||||
|
core collect dispatch on_collection_complete # Batch finished
|
||||||
|
core collect hooks list # List registered hooks
|
||||||
|
core collect hooks register <event> <handler> # Register new hook
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Shell Script Being Replaced
|
||||||
|
|
||||||
|
- `claude/collection/dispatch.sh` - Hook dispatcher
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Trigger | Args |
|
||||||
|
|-------|---------|------|
|
||||||
|
| `on_url_found` | URL discovered during collection | url |
|
||||||
|
| `on_file_collected` | File successfully downloaded | file, type |
|
||||||
|
| `on_collection_complete` | Job batch finishes | - |
|
||||||
|
|
||||||
|
## Hook Registration
|
||||||
|
|
||||||
|
Hooks defined in `hooks.json`:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"hooks": {
|
||||||
|
"on_url_found": [
|
||||||
|
{
|
||||||
|
"name": "whitepaper-collector",
|
||||||
|
"pattern": "\\.pdf$",
|
||||||
|
"handler": "collect papers queue",
|
||||||
|
"priority": 10,
|
||||||
|
"enabled": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pattern Matching
|
||||||
|
|
||||||
|
- Hooks can specify regex pattern for filtering
|
||||||
|
- Only matching events trigger the handler
|
||||||
|
- Multiple hooks can handle same event (priority ordering)
|
||||||
|
|
||||||
|
## Handler Types
|
||||||
|
|
||||||
|
1. **Core commands**: `collect papers queue`
|
||||||
|
2. **Shell scripts**: `./my-handler.sh`
|
||||||
|
3. **External binaries**: `/usr/local/bin/handler`
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"event": "on_url_found",
|
||||||
|
"args": ["https://example.com/paper.pdf"],
|
||||||
|
"handlers_fired": 2,
|
||||||
|
"results": [
|
||||||
|
{"handler": "whitepaper-collector", "status": "ok"},
|
||||||
|
{"handler": "archive-notifier", "status": "ok"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
Loading…
Add table
Reference in a new issue