I spent two days debugging why my from-scratch Layer 1 blockchain committed every block with zero transactions, despite the mempool accepting them perfectly.
This is the story of how I found it, what it taught me, and why silent failures are the hardest bugs in distributed systems.
The setup
I'm building NOVAI , a Layer 1 blockchain in Rust with HotStuff BFT consensus. No forks, no frameworks. Every crate written from scratch. Four validators running on a local devnet, producing blocks at around 75 per second.
The chain worked perfectly. Blocks committed. QCs formed. Validators voted. Everything was green.
Then I added transaction support. And every single block committed with tx_count=0.
The symptoms
The RPC endpoint accepted transactions. Submitted, accepted, zero rejected. The mempool inserted them. drain_ready pulled them into proposed blocks. But every committed block: zero transactions.
No error logs. No panics. No warnings.…