From dfc743c825569a2eec797dfe9b1b1bc26bb8174c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Apr 2026 15:32:29 +0100 Subject: [PATCH] docs: add chain state observations from testnet testing Real operational notes: PoS difficulty spikes, sequence limits, tx pool behaviour, API server crashes, alias registration rates. Knowledge base for operators and future debugging. Co-Authored-By: Charon --- Chain-Observations.md | 88 +++++++++++++++++++++++++++++++++++++++++++ Home.md | 1 + 2 files changed, 89 insertions(+) create mode 100644 Chain-Observations.md diff --git a/Chain-Observations.md b/Chain-Observations.md new file mode 100644 index 0000000..97c3c4e --- /dev/null +++ b/Chain-Observations.md @@ -0,0 +1,88 @@ +# Chain State Observations + +Operational notes from testnet (April 2026). These document real chain conditions encountered during testing — what they look like, what causes them, and how to recover. + +## PoS Difficulty Spike + +**Symptom:** PoS wallet logs `status: NOT_FOUND` on every iteration despite having balance. + +**Cause:** A burst of consecutive PoS blocks causes PoS difficulty to ramp exponentially. At 311+ quadrillion difficulty, the wallet's 2,000 LTHN balance can't find a valid kernel. + +**Recovery:** Wait for PoW blocks to land. The hybrid difficulty adjustment lowers PoS difficulty when PoW blocks arrive. The chain self-corrects but may stall for minutes if PoW hashrate is low. + +**Lesson:** A testnet with a single PoW miner and a single PoS staker can get stuck if PoS mints too many consecutive blocks. Production networks with multiple miners won't hit this. + +## PoS Sequence Factor Limit + +**Symptom:** Wallet logs `PoS sequence factor is too high, waiting for a PoW block...` + +**Cause:** The daemon enforces a maximum number of consecutive PoS blocks (`BLOCK_POS_STRICT_SEQUENCE_LIMIT`). When reached, PoS blocks are rejected until a PoW block breaks the sequence. + +**Recovery:** PoW miner must produce a block. If no external miner, the daemon's `--start-mining` flag with CPU threads can break the deadlock. + +**Lesson:** Hybrid PoW/PoS requires both active. A chain with only stakers will stall at the sequence limit. + +## POS MINING NOT ALLOWED YET + +**Symptom:** Wallet repeatedly logs `POS MINING NOT ALLOWED YET` despite having coins and `--do-pos-mining` enabled. + +**Cause:** The daemon's `get_pos_mining_details` RPC checks `p2p.get_connections_count()` — if zero peers and `--rpc-ignore-offline` is NOT set, it returns `DISCONNECTED` status which the wallet interprets as "not allowed". + +**Fix:** Add `--rpc-ignore-offline` to daemon startup flags. + +**Root code:** `core_rpc_server.cpp:654` — the peer count check gates PoS mining details. + +## Transaction Rejected by Daemon + +**Symptom:** `transaction was rejected by daemon` on alias registration. + +**Possible causes:** +1. **"already in transaction pool by it's address"** — one pending alias per address. Wait for current tx to confirm before submitting next. +2. **Stale UTXO** — outputs used in the tx were already spent. Wallet needs to refresh. +3. **Duplicate alias** — alias already registered on-chain. + +**Fix:** For batch registration, wait for tx pool to clear between submissions. Use multiple wallets for parallel registration. + +## Block Failed Verification (PoW) + +**Symptom:** `mined block has failed to pass verification: id ... @ height N prev_id: ...` + +**Cause:** The miner's block template references an old `prev_id` that no longer matches the chain tip. Happens when the daemon was restarted while the miner had a stale template, or when another block was found between template fetch and submission. + +**Recovery:** Self-corrects — the miner fetches a new template on the next attempt. No action needed unless it persists. + +## Swagger/API Server Crash (SIGABRT) + +**Symptom:** Daemon crashes immediately with `oatpp::swagger::Resources::loadFromFile(): Can't load file '../share/api_server/res/favicon-16x16.png'` + +**Cause:** The embedded API server (oatpp/swagger) loads static resources from `../share/api_server/res/` relative to the current working directory (NOT the binary location). + +**Fix:** Either: +- Run the daemon from its own directory: `cd /opt/lethean/testnet && ./lethean-testnet-chain-node` +- OR ensure `../share/api_server/res/` exists relative to CWD +- Docker: the Dockerfile must copy the `share/` directory alongside the binary + +## API Port Conflict (Mainnet + Testnet) + +**Symptom:** Testnet daemon crashes with `oatpp::network::tcp::server::ConnectionProvider: Couldn't bind. Address already in use` + +**Cause:** Both mainnet and testnet daemons default to API port 36943. If mainnet is running, testnet can't bind. + +**Fix:** Pass `--api-bind-port 46943` to the testnet daemon to use a different API port. + +## Alias Registration Rate + +**Observed rate:** 1 alias per block cycle (~120 seconds) per wallet. + +**Bottleneck:** Daemon rejects multiple alias registrations from the same address if one is already in the tx pool. + +**Scaling:** Use N wallets in parallel for N registrations per block. 10 wallets = 10 aliases per ~120 seconds = ~7,200/day. + +## PoW CPU Mining Feasibility + +**Observation:** CPU mining (2 threads) at ProgPoWZ difficulty 474K produced a block after ~10 minutes. This is viable for testnet block production but not economically viable for mainnet where GPU miners dominate. + +**Use case:** `--start-mining` with CPU threads is useful for: +- Breaking PoS sequence deadlocks on testnet +- Solo mining on isolated test networks +- Keeping a private testnet alive without external miners diff --git a/Home.md b/Home.md index 84fb1cf..fcbdb0c 100644 --- a/Home.md +++ b/Home.md @@ -11,6 +11,7 @@ Lethean blockchain — Zano fork with ProgPoWZ mining, confidential assets, and - [Mining Guide](Mining-Guide) — mine with your GPU ## Reference +- [Chain Observations](Chain-Observations) — operational notes, error recovery, testnet findings - [Getting Started](Getting-Started) — what Lethean is, specifications, and tokenomics - [Ecosystem](Ecosystem) — all repos, images, ports, and architecture