docs: add chain state observations from testnet testing

Real operational notes: PoS difficulty spikes, sequence limits,
tx pool behaviour, API server crashes, alias registration rates.
Knowledge base for operators and future debugging.

Co-Authored-By: Charon <charon@lethean.io>

Chain-Observations.md

@@ -0,0 +1,88 @@
+# Chain State Observations
+
+Operational notes from testnet (April 2026). These document real chain conditions encountered during testing — what they look like, what causes them, and how to recover.
+
+## PoS Difficulty Spike
+
+**Symptom:** PoS wallet logs `status: NOT_FOUND` on every iteration despite having balance.
+
+**Cause:** A burst of consecutive PoS blocks causes PoS difficulty to ramp exponentially. At 311+ quadrillion difficulty, the wallet's 2,000 LTHN balance can't find a valid kernel.
+
+**Recovery:** Wait for PoW blocks to land. The hybrid difficulty adjustment lowers PoS difficulty when PoW blocks arrive. The chain self-corrects but may stall for minutes if PoW hashrate is low.
+
+**Lesson:** A testnet with a single PoW miner and a single PoS staker can get stuck if PoS mints too many consecutive blocks. Production networks with multiple miners won't hit this.
+
+## PoS Sequence Factor Limit
+
+**Symptom:** Wallet logs `PoS sequence factor is too high, waiting for a PoW block...`
+
+**Cause:** The daemon enforces a maximum number of consecutive PoS blocks (`BLOCK_POS_STRICT_SEQUENCE_LIMIT`). When reached, PoS blocks are rejected until a PoW block breaks the sequence.
+
+**Recovery:** PoW miner must produce a block. If no external miner, the daemon's `--start-mining` flag with CPU threads can break the deadlock.
+
+**Lesson:** Hybrid PoW/PoS requires both active. A chain with only stakers will stall at the sequence limit.
+
+## POS MINING NOT ALLOWED YET
+
+**Symptom:** Wallet repeatedly logs `POS MINING NOT ALLOWED YET` despite having coins and `--do-pos-mining` enabled.
+
+**Cause:** The daemon's `get_pos_mining_details` RPC checks `p2p.get_connections_count()` — if zero peers and `--rpc-ignore-offline` is NOT set, it returns `DISCONNECTED` status which the wallet interprets as "not allowed".
+
+**Fix:** Add `--rpc-ignore-offline` to daemon startup flags.
+
+**Root code:** `core_rpc_server.cpp:654` — the peer count check gates PoS mining details.
+
+## Transaction Rejected by Daemon
+
+**Symptom:** `transaction was rejected by daemon` on alias registration.
+
+**Possible causes:**
+1. **"already in transaction pool by it's address"** — one pending alias per address. Wait for current tx to confirm before submitting next.
+2. **Stale UTXO** — outputs used in the tx were already spent. Wallet needs to refresh.
+3. **Duplicate alias** — alias already registered on-chain.
+
+**Fix:** For batch registration, wait for tx pool to clear between submissions. Use multiple wallets for parallel registration.
+
+## Block Failed Verification (PoW)
+
+**Symptom:** `mined block has failed to pass verification: id ... @ height N prev_id: ...`
+
+**Cause:** The miner's block template references an old `prev_id` that no longer matches the chain tip. Happens when the daemon was restarted while the miner had a stale template, or when another block was found between template fetch and submission.
+
+**Recovery:** Self-corrects — the miner fetches a new template on the next attempt. No action needed unless it persists.
+
+## Swagger/API Server Crash (SIGABRT)
+
+**Symptom:** Daemon crashes immediately with `oatpp::swagger::Resources::loadFromFile(): Can't load file '../share/api_server/res/favicon-16x16.png'`
+
+**Cause:** The embedded API server (oatpp/swagger) loads static resources from `../share/api_server/res/` relative to the current working directory (NOT the binary location).
+
+**Fix:** Either:
+- Run the daemon from its own directory: `cd /opt/lethean/testnet && ./lethean-testnet-chain-node`
+- OR ensure `../share/api_server/res/` exists relative to CWD
+- Docker: the Dockerfile must copy the `share/` directory alongside the binary
+
+## API Port Conflict (Mainnet + Testnet)
+
+**Symptom:** Testnet daemon crashes with `oatpp::network::tcp::server::ConnectionProvider: Couldn't bind. Address already in use`
+
+**Cause:** Both mainnet and testnet daemons default to API port 36943. If mainnet is running, testnet can't bind.
+
+**Fix:** Pass `--api-bind-port 46943` to the testnet daemon to use a different API port.
+
+## Alias Registration Rate
+
+**Observed rate:** 1 alias per block cycle (~120 seconds) per wallet.
+
+**Bottleneck:** Daemon rejects multiple alias registrations from the same address if one is already in the tx pool.
+
+**Scaling:** Use N wallets in parallel for N registrations per block. 10 wallets = 10 aliases per ~120 seconds = ~7,200/day.
+
+## PoW CPU Mining Feasibility
+
+**Observation:** CPU mining (2 threads) at ProgPoWZ difficulty 474K produced a block after ~10 minutes. This is viable for testnet block production but not economically viable for mainnet where GPU miners dominate.
+
+**Use case:** `--start-mining` with CPU threads is useful for:
+- Breaking PoS sequence deadlocks on testnet
+- Solo mining on isolated test networks
+- Keeping a private testnet alive without external miners

Home.md

@@ -11,6 +11,7 @@ Lethean blockchain — Zano fork with ProgPoWZ mining, confidential assets, and
 - [Mining Guide](Mining-Guide) — mine with your GPU
 
 ## Reference
+- [Chain Observations](Chain-Observations) — operational notes, error recovery, testnet findings
 
 - [Getting Started](Getting-Started) — what Lethean is, specifications, and tokenomics
 - [Ecosystem](Ecosystem) — all repos, images, ports, and architecture