docs: add feature request issues for core CLI migration
12 issue files documenting features needed to replace shell scripts: Claude Code hooks: - 001: core ai session (state management) - 002: core ai context (fact capture) - 003: core ai hook (command validation) - 004: core qa debug (debug statement detection) Data collection: - 005: core collect github (issues/PRs archive) - 006: core collect bitcointalk (forum threads) - 007: core collect market (CMC/CoinGecko) - 008: core collect papers (whitepapers) - 009: core collect excavate (project archaeology) - 010: core collect process (HTML→MD) - 011: core collect dispatch (event hooks) 000: Overview tracking issue These will be submitted to host-uk/core when rate limit resets. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
39b2cbde09
commit
beb24f71d2
12 changed files with 788 additions and 0 deletions
66
claude/issues/000-overview.md
Normal file
66
claude/issues/000-overview.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# feat: Replace shell scripts with core CLI commands
|
||||
|
||||
## Summary
|
||||
|
||||
This tracking issue covers the migration from shell scripts to native `core` CLI commands in `core-agent`. Every shell script should be replaced by a `core` command, making the plugin purely configuration (JSON, MD) + calls to `core`.
|
||||
|
||||
## Philosophy
|
||||
|
||||
- Every agent using `core` tests the framework
|
||||
- Missing features get raised as issues
|
||||
- Code as if functionality exists (TDD approach)
|
||||
- The CLI becomes battle-tested and bulletproof
|
||||
|
||||
## Issues
|
||||
|
||||
### Claude Code Hooks (core ai)
|
||||
|
||||
| Issue | Command | Replaces |
|
||||
|-------|---------|----------|
|
||||
| #001 | `core ai session` | pre-compact.sh, session-start.sh, suggest-compact.sh |
|
||||
| #002 | `core ai context` | capture-context.sh, extract-actionables.sh |
|
||||
| #003 | `core ai hook` | prefer-core.sh, block-docs.sh, post-commit-check.sh |
|
||||
|
||||
### Quality Assurance (core qa)
|
||||
|
||||
| Issue | Command | Replaces |
|
||||
|-------|---------|----------|
|
||||
| #004 | `core qa debug` | check-debug.sh |
|
||||
|
||||
### Data Collection (core collect)
|
||||
|
||||
| Issue | Command | Replaces |
|
||||
|-------|---------|----------|
|
||||
| #005 | `core collect github` | github-history/collect.sh |
|
||||
| #006 | `core collect bitcointalk` | bitcointalk/collect.sh |
|
||||
| #007 | `core collect market` | coinmarketcap/*.sh |
|
||||
| #008 | `core collect papers` | ledger-papers/discover.sh, collect-whitepaper.sh |
|
||||
| #009 | `core collect excavate` | project-archaeology/excavate.sh |
|
||||
| #010 | `core collect process` | job-collector/process.sh |
|
||||
| #011 | `core collect dispatch` | collection/dispatch.sh |
|
||||
|
||||
## Target State
|
||||
|
||||
After all issues are implemented:
|
||||
|
||||
```
|
||||
claude/
|
||||
├── hooks/
|
||||
│ └── hooks.json # Calls core ai hook validate-*
|
||||
├── commands/
|
||||
│ └── remember.md # Calls core ai context add
|
||||
├── collection/
|
||||
│ └── hooks.json # Calls core collect dispatch
|
||||
└── skills/
|
||||
└── */
|
||||
└── SKILL.md # Documentation only, calls core collect *
|
||||
```
|
||||
|
||||
No shell scripts. Just JSON config + markdown docs + `core` CLI calls.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Phase 1**: `core ai session` + `core ai context` (enables hooks to work)
|
||||
2. **Phase 2**: `core ai hook` + `core qa debug` (safety + quality)
|
||||
3. **Phase 3**: `core collect github` + `core collect bitcointalk` (most used)
|
||||
4. **Phase 4**: Remaining collection commands
|
||||
46
claude/issues/001-ai-session.md
Normal file
46
claude/issues/001-ai-session.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# feat(ai): Add session state management for Claude Code hooks
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core ai session` subcommands to manage Claude Code session state, replacing shell scripts that currently handle this.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core ai session save # Save current session state (pre-compact)
|
||||
core ai session restore # Restore session state (session-start)
|
||||
core ai session stats # Track tool calls, suggest compaction
|
||||
core ai session clear # Clear stale session data
|
||||
```
|
||||
|
||||
## Current Shell Scripts Being Replaced
|
||||
|
||||
- `claude/scripts/pre-compact.sh` - Saves state before auto-compact
|
||||
- `claude/scripts/session-start.sh` - Restores context on startup
|
||||
- `claude/scripts/suggest-compact.sh` - Suggests compaction at intervals
|
||||
|
||||
## State to Manage
|
||||
|
||||
- Working directory and git branch
|
||||
- Git status (modified files)
|
||||
- In-progress todos
|
||||
- Context facts (decisions, actionables)
|
||||
- Tool call counter per session
|
||||
|
||||
## Storage
|
||||
|
||||
State stored in `~/.claude/sessions/`:
|
||||
- `scratchpad.md` - Human-readable resume state
|
||||
- `context.json` - Structured context facts
|
||||
- `stats.json` - Session statistics
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"saved": true,
|
||||
"path": "~/.claude/sessions/scratchpad.md",
|
||||
"facts": 5,
|
||||
"tool_calls": 47
|
||||
}
|
||||
```
|
||||
53
claude/issues/002-ai-context.md
Normal file
53
claude/issues/002-ai-context.md
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
# feat(ai): Add context fact management for Claude Code
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core ai context` subcommands to capture and manage context facts that persist across compaction.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core ai context add <fact> [--source=X] # Save a context fact
|
||||
core ai context list # List current facts
|
||||
core ai context extract <output> # Extract actionables from tool output
|
||||
core ai context clear # Clear old context (>3h)
|
||||
```
|
||||
|
||||
## Current Shell Scripts Being Replaced
|
||||
|
||||
- `claude/scripts/capture-context.sh` - Stores context facts
|
||||
- `claude/scripts/extract-actionables.sh` - Extracts actionables from core CLI output
|
||||
|
||||
## Context Format
|
||||
|
||||
```json
|
||||
[
|
||||
{"fact": "Use Action pattern not Service", "source": "user", "ts": 1234567890},
|
||||
{"fact": "FAIL: TestFoo", "source": "core go test", "ts": 1234567891}
|
||||
]
|
||||
```
|
||||
|
||||
## Extraction Patterns
|
||||
|
||||
The `extract` command should parse output from:
|
||||
- `core go test` / `core go qa` / `core go lint` - Extract ERROR/WARN/FAIL lines
|
||||
- `core php test` / `core php stan` - Extract FAIL/Error lines
|
||||
- `core build` - Extract error/cannot/undefined lines
|
||||
|
||||
## Storage
|
||||
|
||||
- `~/.claude/sessions/context.json`
|
||||
- Max 20 items, auto-clear after 3 hours of inactivity
|
||||
|
||||
## Example Usage
|
||||
|
||||
```bash
|
||||
# Manual fact
|
||||
core ai context add "User prefers UK English" --source=user
|
||||
|
||||
# Extract from piped output
|
||||
core go test 2>&1 | core ai context extract
|
||||
|
||||
# List facts
|
||||
core ai context list --json
|
||||
```
|
||||
60
claude/issues/003-ai-hook.md
Normal file
60
claude/issues/003-ai-hook.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# feat(ai): Add hook validation for Claude Code PreToolUse
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core ai hook` subcommands to validate commands and file operations for Claude Code hooks.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core ai hook validate-command <cmd> # Check if command is safe/allowed
|
||||
core ai hook validate-file <path> # Check if file creation is allowed
|
||||
core ai hook post-commit # Check for uncommitted work after commit
|
||||
```
|
||||
|
||||
## Current Shell Scripts Being Replaced
|
||||
|
||||
- `claude/hooks/prefer-core.sh` - Blocks dangerous commands, enforces core CLI
|
||||
- `claude/scripts/block-docs.sh` - Blocks random .md file creation
|
||||
- `claude/scripts/post-commit-check.sh` - Warns about uncommitted work
|
||||
|
||||
## Command Validation Rules
|
||||
|
||||
Block these patterns:
|
||||
- `rm -rf` / `rm -r` (except node_modules, vendor, .cache, dist, build)
|
||||
- `mv`/`cp` with wildcards
|
||||
- `xargs` with rm/mv/cp
|
||||
- `find -exec` with file operations
|
||||
- `sed -i` (in-place editing)
|
||||
- `grep -l | ...` (mass file targeting)
|
||||
- `perl -i`, `awk > file`
|
||||
|
||||
Redirect to core:
|
||||
- `go test/build/fmt/mod` → suggest `core go *`
|
||||
- `golangci-lint` → suggest `core go lint`
|
||||
- `composer test` → suggest `core php test`
|
||||
- `./vendor/bin/pint` → suggest `core php fmt`
|
||||
- `php artisan serve` → suggest `core php dev`
|
||||
|
||||
## File Validation Rules
|
||||
|
||||
Allow:
|
||||
- `README.md`, `CLAUDE.md`, `AGENTS.md`, `CONTRIBUTING.md`, `CHANGELOG.md`, `LICENSE.md`
|
||||
- Files in `docs/` directory
|
||||
|
||||
Block:
|
||||
- Other `.md` files (suggest using README.md or docs/)
|
||||
|
||||
## Output Format (JSON for hooks)
|
||||
|
||||
```json
|
||||
{"decision": "approve"}
|
||||
```
|
||||
|
||||
```json
|
||||
{"decision": "block", "message": "Use `core go test` instead of raw go test"}
|
||||
```
|
||||
|
||||
```json
|
||||
{"decision": "warn", "message": "3 files remain uncommitted after commit"}
|
||||
```
|
||||
58
claude/issues/004-lint-debug.md
Normal file
58
claude/issues/004-lint-debug.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# feat(qa): Add debug statement detection
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core qa debug` command to detect debug statements in code, for use in Claude Code PostToolUse hooks.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core qa debug <file> # Check single file for debug statements
|
||||
core qa debug --staged # Check all staged files
|
||||
core qa debug --changed # Check all changed files
|
||||
```
|
||||
|
||||
## Current Shell Script Being Replaced
|
||||
|
||||
- `claude/scripts/check-debug.sh` - Warns about debug statements after edits
|
||||
|
||||
## Detection Patterns
|
||||
|
||||
**Go files (*.go):**
|
||||
- `fmt.Println`
|
||||
- `fmt.Printf` (without format verbs suggesting actual logging)
|
||||
- `log.Println`
|
||||
- `log.Printf`
|
||||
- `spew.Dump`
|
||||
- `pp.Println`
|
||||
|
||||
**PHP files (*.php):**
|
||||
- `dd(`
|
||||
- `dump(`
|
||||
- `var_dump(`
|
||||
- `print_r(`
|
||||
- `ray(`
|
||||
|
||||
**JavaScript/TypeScript (*.js, *.ts, *.tsx):**
|
||||
- `console.log`
|
||||
- `console.debug`
|
||||
- `debugger`
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"file": "src/main.go",
|
||||
"warnings": [
|
||||
{"line": 42, "type": "fmt.Println", "content": "fmt.Println(\"debug\")"},
|
||||
{"line": 87, "type": "log.Println", "content": "log.Println(err)"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
For hooks, output to stderr:
|
||||
```
|
||||
[qa] WARNING: Debug statements found in src/main.go
|
||||
42: fmt.Println("debug")
|
||||
87: log.Println(err)
|
||||
```
|
||||
71
claude/issues/005-collect-github.md
Normal file
71
claude/issues/005-collect-github.md
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
# feat(collect): Add GitHub history collection
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect github` command to archive GitHub issues and PRs from repositories and organisations.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect github <org/repo> # Collect issues + PRs from repo
|
||||
core collect github <org> --org # Collect all repos in org
|
||||
core collect github <org1,org2> --org # Batch collect multiple orgs
|
||||
core collect github <repo> --issues-only # Issues only
|
||||
core collect github <repo> --prs-only # PRs only
|
||||
core collect github --check-rate # Show rate limit status
|
||||
```
|
||||
|
||||
## Current Shell Script Being Replaced
|
||||
|
||||
- `claude/skills/github-history/collect.sh` - 517 lines of bash
|
||||
|
||||
## Features
|
||||
|
||||
1. **Rate limit protection**
|
||||
- Check every N calls
|
||||
- Auto-pause at 25% remaining (75% used)
|
||||
- Resume after reset + 10s buffer
|
||||
|
||||
2. **Incremental collection**
|
||||
- Skip already-fetched issues/PRs
|
||||
- Resume interrupted collections
|
||||
|
||||
3. **Reception scoring**
|
||||
- ADDRESSED: Closed after discussion
|
||||
- DISMISSED: Labeled wontfix/invalid
|
||||
- IGNORED: Closed with no response
|
||||
- STALE: Open with no replies
|
||||
- ACTIVE: Open with discussion
|
||||
- MERGED/REJECTED/PENDING for PRs
|
||||
|
||||
4. **Output structure**
|
||||
```
|
||||
repo/{org}/{repo}/
|
||||
├── Issue/
|
||||
│ ├── INDEX.md
|
||||
│ ├── 001.md
|
||||
│ └── 002.md
|
||||
├── PR/
|
||||
│ ├── INDEX.md
|
||||
│ ├── 001.md
|
||||
│ └── 002.md
|
||||
└── .json/
|
||||
├── issues-list.json
|
||||
└── prs-list.json
|
||||
```
|
||||
|
||||
5. **Index generation**
|
||||
- Markdown tables with seq, GitHub #, title, score
|
||||
- Score legend
|
||||
|
||||
## Output Format
|
||||
|
||||
Progress to stderr, final summary to stdout:
|
||||
```json
|
||||
{
|
||||
"repo": "host-uk/core",
|
||||
"issues": 47,
|
||||
"prs": 23,
|
||||
"output": "repo/host-uk/core/"
|
||||
}
|
||||
```
|
||||
64
claude/issues/006-collect-bitcointalk.md
Normal file
64
claude/issues/006-collect-bitcointalk.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
# feat(collect): Add BitcoinTalk thread collection
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect bitcointalk` command to archive BitcoinTalk forum threads.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect bitcointalk <topic-id> # Collect full thread
|
||||
core collect bitcointalk <url> # Collect from URL
|
||||
core collect bitcointalk <id> --pages=5 # Limit pages
|
||||
core collect bitcointalk <id> --output=DIR # Custom output dir
|
||||
```
|
||||
|
||||
## Current Shell Script Being Replaced
|
||||
|
||||
- `claude/skills/bitcointalk/collect.sh` - 270 lines of bash + embedded Python
|
||||
|
||||
## Features
|
||||
|
||||
1. **Rate limiting**
|
||||
- Respectful delay between requests (default 2s)
|
||||
- Configurable via `--delay=N`
|
||||
|
||||
2. **Post type detection**
|
||||
- ANN: Original announcement (post #1)
|
||||
- UPDATE: Contains [UPDATE]/[RELEASE]/[ANNOUNCEMENT]
|
||||
- QUESTION: Contains question mark in first 200 chars
|
||||
- COMMUNITY: General discussion
|
||||
|
||||
3. **Output structure**
|
||||
```
|
||||
bitcointalk-{topic}/
|
||||
├── INDEX.md
|
||||
├── pages/
|
||||
│ ├── page-0.html
|
||||
│ └── page-20.html
|
||||
└── posts/
|
||||
├── POST-0001.md
|
||||
└── POST-0002.md
|
||||
```
|
||||
|
||||
4. **Post metadata**
|
||||
- Author
|
||||
- Date
|
||||
- Post type/score
|
||||
- Original content
|
||||
|
||||
5. **Incremental collection**
|
||||
- Resume interrupted collections
|
||||
- Skip already-fetched pages
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"topic_id": "2769739",
|
||||
"title": "Lethean - Privacy Blockchain VPN",
|
||||
"posts": 1247,
|
||||
"pages": 63,
|
||||
"output": "bitcointalk-2769739/"
|
||||
}
|
||||
```
|
||||
67
claude/issues/007-collect-market.md
Normal file
67
claude/issues/007-collect-market.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# feat(collect): Add cryptocurrency market data collection
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect market` command to collect historical price/market data from CoinMarketCap, CoinGecko, etc.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect market <coin> # Collect current data
|
||||
core collect market <coin> --historical # Include historical data
|
||||
core collect market <coin> --from=2018-01-01 # Date range
|
||||
core collect market <coin1,coin2> # Multiple coins
|
||||
core collect market jobs <coin> # Generate job list
|
||||
```
|
||||
|
||||
## Current Shell Scripts Being Replaced
|
||||
|
||||
- `claude/skills/coinmarketcap/generate-jobs.sh` - Generates CMC collection jobs
|
||||
- `claude/skills/coinmarketcap/process.sh` - Processes collected CMC data
|
||||
|
||||
## Data Sources
|
||||
|
||||
1. **CoinMarketCap**
|
||||
- Main page (description, links)
|
||||
- Markets/exchanges
|
||||
- Historical data
|
||||
- News
|
||||
- API endpoints (where available)
|
||||
|
||||
2. **CoinGecko** (backup)
|
||||
- Main page
|
||||
- API for historical data
|
||||
|
||||
## Output Structure
|
||||
|
||||
```
|
||||
market/{coin}/
|
||||
├── INDEX.md
|
||||
├── cmc/
|
||||
│ ├── main.json
|
||||
│ ├── markets.json
|
||||
│ └── historical.json
|
||||
├── coingecko/
|
||||
│ ├── detail.json
|
||||
│ └── history.json
|
||||
└── summary.md
|
||||
```
|
||||
|
||||
## Job Format
|
||||
|
||||
For `jobs` subcommand (pipe to collector):
|
||||
```
|
||||
URL|FILENAME|TYPE|METADATA
|
||||
https://coinmarketcap.com/currencies/lethean/|cmc-lethean-main.html|cmc-main|coin=lethean
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"coin": "lethean",
|
||||
"sources": ["cmc", "coingecko"],
|
||||
"historical_days": 2190,
|
||||
"output": "market/lethean/"
|
||||
}
|
||||
```
|
||||
68
claude/issues/008-collect-papers.md
Normal file
68
claude/issues/008-collect-papers.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# feat(collect): Add research paper/whitepaper collection
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect papers` command to discover and collect distributed ledger research papers and whitepapers.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect papers --all # All known papers
|
||||
core collect papers --category=cryptography # Filter by category
|
||||
core collect papers --topic=bulletproofs # Filter by topic
|
||||
core collect papers --project=monero # Filter by project
|
||||
core collect papers --search-iacr # Search IACR eprint
|
||||
core collect papers queue <url> # Queue single paper
|
||||
```
|
||||
|
||||
## Current Shell Scripts Being Replaced
|
||||
|
||||
- `claude/skills/ledger-papers/discover.sh` - Paper discovery
|
||||
- `claude/collection/collect-whitepaper.sh` - Queue whitepaper collection
|
||||
- `claude/collection/update-index.sh` - Update paper index
|
||||
|
||||
## Categories
|
||||
|
||||
- genesis (Bitcoin, b-money, hashcash, bit gold)
|
||||
- cryptonote (CryptoNote v2.0, CNS standards)
|
||||
- mrl (Monero Research Lab papers)
|
||||
- privacy (Zcash, Dash, Mimblewimble, Lelantus)
|
||||
- smart-contracts (Ethereum, Solana, Cardano)
|
||||
- layer2 (Lightning, Plasma, Rollups)
|
||||
- consensus (PBFT, Tendermint, HotStuff)
|
||||
- cryptography (Bulletproofs, CLSAG, PLONK)
|
||||
- defi (Uniswap, Aave, Compound)
|
||||
- storage (IPFS, Filecoin, Arweave)
|
||||
- identity (DIDs, Verifiable Credentials)
|
||||
- attacks (Selfish mining, eclipse, traceability)
|
||||
|
||||
## Registry
|
||||
|
||||
Papers defined in `registry.json`:
|
||||
```json
|
||||
{
|
||||
"id": "bulletproofs",
|
||||
"title": "Bulletproofs: Short Proofs for Confidential Transactions",
|
||||
"year": 2017,
|
||||
"url": "https://eprint.iacr.org/2017/1066.pdf",
|
||||
"topics": ["range-proofs", "zero-knowledge"]
|
||||
}
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
Job list for collection:
|
||||
```
|
||||
URL|FILENAME|TYPE|METADATA
|
||||
https://bitcoin.org/bitcoin.pdf|bitcoin.pdf|paper|category=genesis,title=Bitcoin...
|
||||
```
|
||||
|
||||
Summary:
|
||||
```json
|
||||
{
|
||||
"papers": 91,
|
||||
"categories": 15,
|
||||
"queued": 3,
|
||||
"output": "papers/"
|
||||
}
|
||||
```
|
||||
92
claude/issues/009-collect-excavate.md
Normal file
92
claude/issues/009-collect-excavate.md
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
# feat(collect): Add project archaeology/excavation
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect excavate` command to perform deep excavation of abandoned cryptocurrency projects before data is lost.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect excavate <project> # Full excavation
|
||||
core collect excavate <project> --scan-only # Check what's accessible
|
||||
core collect excavate <project> --resume # Resume interrupted dig
|
||||
core collect excavate <project> --only=github,btt # Specific collectors
|
||||
```
|
||||
|
||||
## Current Shell Script Being Replaced
|
||||
|
||||
- `claude/skills/project-archaeology/excavate.sh` - 312 lines of bash
|
||||
|
||||
## Excavation Pipeline
|
||||
|
||||
Runs collectors in priority order:
|
||||
|
||||
| Phase | Source | Collector | Priority | Notes |
|
||||
|-------|--------|-----------|----------|-------|
|
||||
| 1 | GitHub repos | `collect github` | P1 | Often deleted first |
|
||||
| 1 | GitHub releases | `collect github` | P1 | Binaries disappear |
|
||||
| 2 | BitcoinTalk ANN | `collect bitcointalk` | P2 | Usually persists |
|
||||
| 2 | Website (Wayback) | `collect wayback` | P2 | Snapshots exist |
|
||||
| 3 | Block explorer | `collect explorer` | P3 | Chain data |
|
||||
| 3 | CoinMarketCap | `collect market` | P3 | Historical prices |
|
||||
| 4 | Reddit | `collect reddit` | P4 | Community context |
|
||||
| 4 | Medium posts | `collect medium` | P4 | Announcements |
|
||||
|
||||
## Project Registry
|
||||
|
||||
Projects defined in `cryptonote-discovery/registry.json`:
|
||||
```json
|
||||
{
|
||||
"name": "Masari",
|
||||
"symbol": "MSR",
|
||||
"status": "abandoned",
|
||||
"github": ["masari-project"],
|
||||
"bitcointalk": "2769739",
|
||||
"website": "getmasari.org",
|
||||
"explorer": "explorer.getmasari.org",
|
||||
"cmc": "masari"
|
||||
}
|
||||
```
|
||||
|
||||
## Output Structure
|
||||
|
||||
```
|
||||
digs/{project}/
|
||||
├── EXCAVATION.md # Dig log with timestamps
|
||||
├── SALVAGE-REPORT.md # What's worth keeping
|
||||
├── LESSONS.md # Post-mortem analysis
|
||||
├── github/ # All repo history
|
||||
├── releases/ # Wallet binaries, checksums
|
||||
├── bitcointalk/ # Thread archive
|
||||
├── website/ # Wayback snapshots
|
||||
├── explorer/ # Chain data samples
|
||||
├── market/ # Price history
|
||||
├── papers/ # Whitepapers, docs
|
||||
└── community/ # Reddit, Medium, etc
|
||||
```
|
||||
|
||||
## Scan Mode
|
||||
|
||||
`--scan-only` checks accessibility without downloading:
|
||||
- GitHub org accessible?
|
||||
- BitcoinTalk thread exists?
|
||||
- Wayback snapshots available?
|
||||
- Block explorer online?
|
||||
- CMC page exists?
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"project": "masari",
|
||||
"status": "abandoned",
|
||||
"accessible": {
|
||||
"github": true,
|
||||
"bitcointalk": true,
|
||||
"wayback": true,
|
||||
"explorer": false,
|
||||
"cmc": true
|
||||
},
|
||||
"output": "digs/masari/"
|
||||
}
|
||||
```
|
||||
70
claude/issues/010-collect-process.md
Normal file
70
claude/issues/010-collect-process.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# feat(collect): Add collected data processing
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect process` command to convert collected HTML/JSON files into clean markdown.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect process <source> <downloads-dir> # Process downloaded files
|
||||
core collect process bitcointalk ./downloads # BitcoinTalk HTML → MD
|
||||
core collect process reddit ./downloads # Reddit JSON → MD
|
||||
core collect process wayback ./downloads # Wayback HTML → MD
|
||||
core collect process medium ./downloads # Medium RSS → MD
|
||||
```
|
||||
|
||||
## Current Shell Script Being Replaced
|
||||
|
||||
- `claude/skills/job-collector/process.sh` - 243 lines of bash + embedded Python
|
||||
|
||||
## Supported Sources
|
||||
|
||||
1. **bitcointalk** / **btt**
|
||||
- Input: HTML pages
|
||||
- Extract: posts, authors, dates
|
||||
- Output: POST-NNNN.md files
|
||||
|
||||
2. **reddit**
|
||||
- Input: JSON from Reddit API
|
||||
- Extract: posts, comments, scores
|
||||
- Output: REDDIT-NNNN.md files
|
||||
|
||||
3. **wayback**
|
||||
- Input: HTML from Wayback Machine
|
||||
- Extract: title, body text
|
||||
- Output: {basename}.md files
|
||||
|
||||
4. **medium**
|
||||
- Input: RSS/XML feed
|
||||
- Extract: title, author, date, content
|
||||
- Output: MEDIUM-NNNN.md files
|
||||
|
||||
## Output Structure
|
||||
|
||||
```
|
||||
processed/
|
||||
├── INDEX.md
|
||||
└── posts/
|
||||
├── POST-0001.md
|
||||
├── POST-0002.md
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Index Generation
|
||||
|
||||
Auto-generates INDEX.md with:
|
||||
- Source metadata
|
||||
- Post count
|
||||
- Links to all posts
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "bitcointalk",
|
||||
"input_files": 15,
|
||||
"posts_extracted": 347,
|
||||
"output": "processed/"
|
||||
}
|
||||
```
|
||||
73
claude/issues/011-collect-dispatch.md
Normal file
73
claude/issues/011-collect-dispatch.md
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
# feat(collect): Add event hook dispatch system
|
||||
|
||||
## Summary
|
||||
|
||||
Add `core collect dispatch` command to fire events during data collection, allowing modular hook handling.
|
||||
|
||||
## Required Commands
|
||||
|
||||
```bash
|
||||
core collect dispatch <event> [args...] # Fire an event
|
||||
core collect dispatch on_url_found <url> # URL discovered
|
||||
core collect dispatch on_file_collected <file> <type> # File downloaded
|
||||
core collect dispatch on_collection_complete # Batch finished
|
||||
core collect hooks list # List registered hooks
|
||||
core collect hooks register <event> <handler> # Register new hook
|
||||
```
|
||||
|
||||
## Current Shell Script Being Replaced
|
||||
|
||||
- `claude/collection/dispatch.sh` - Hook dispatcher
|
||||
|
||||
## Events
|
||||
|
||||
| Event | Trigger | Args |
|
||||
|-------|---------|------|
|
||||
| `on_url_found` | URL discovered during collection | url |
|
||||
| `on_file_collected` | File successfully downloaded | file, type |
|
||||
| `on_collection_complete` | Job batch finishes | - |
|
||||
|
||||
## Hook Registration
|
||||
|
||||
Hooks defined in `hooks.json`:
|
||||
```json
|
||||
{
|
||||
"hooks": {
|
||||
"on_url_found": [
|
||||
{
|
||||
"name": "whitepaper-collector",
|
||||
"pattern": "\\.pdf$",
|
||||
"handler": "collect papers queue",
|
||||
"priority": 10,
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Pattern Matching
|
||||
|
||||
- Hooks can specify regex pattern for filtering
|
||||
- Only matching events trigger the handler
|
||||
- Multiple hooks can handle same event (priority ordering)
|
||||
|
||||
## Handler Types
|
||||
|
||||
1. **Core commands**: `collect papers queue`
|
||||
2. **Shell scripts**: `./my-handler.sh`
|
||||
3. **External binaries**: `/usr/local/bin/handler`
|
||||
|
||||
## Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "on_url_found",
|
||||
"args": ["https://example.com/paper.pdf"],
|
||||
"handlers_fired": 2,
|
||||
"results": [
|
||||
{"handler": "whitepaper-collector", "status": "ok"},
|
||||
{"handler": "archive-notifier", "status": "ok"}
|
||||
]
|
||||
}
|
||||
```
|
||||
Loading…
Add table
Reference in a new issue