Performance optimizations to the NOMT storage engine while preserving
correctness (all triecompare cross-validation tests pass at 10K+ scale):
- Pool SHA256 hashers via sync.Pool in HashInternal and HashStem
- Replace allStems map with sorted slice + O(N+M) merge (in-place fast
path for incremental updates avoids allocation entirely)
- Add UpdateSorted to db.DB, skipping redundant sort of pre-sorted ops
- Simplify canonicalRoot to use pre-sorted allStems directly
- Optimize StemSharedBits with byte-level XOR + bits.LeadingZeros8
- Replace stemLess loops with bytes.Compare in all locations
- Eliminate per-stem map alloc in groupAndHashStems (use [256]bool dirty)
- Use stack-allocated [248]bool for downBits in BuildInternalTree
- Remove unused stemPathCmp function
BenchmarkHash/10000/nomt: 9.8ms → 8.2ms (-16%)
BenchmarkBlockWorkload/nomt: 7.7ms → 6.6ms (-14%)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add trie/triecompare/ package with realistic state generation and cross-
validation tests proving NOMT produces identical roots as bintrie at scale
(10K+ accounts, PowerLaw/Uniform/Exponential distributions, multi-block).
Fix a subtle bug in groupAndHashStems: sort.Slice was used instead of
sort.SliceStable, causing non-deterministic results when the same account
is mutated twice in a single block (duplicate stem+suffix entries need
last-writer-wins ordering preserved).
Tests: 5 correctness tests + 4 benchmarks + storage footprint comparison.
All pass with race detector clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two root causes of hash mismatch fixed:
1. Canonical root computation: Hash() now uses BuildInternalTree(skip=0)
over all known stems instead of the page tree's depth-7 internal root.
The page tree is still updated for persistent storage, but the canonical
root bypasses its depth-7 worker split that added extra wrapping levels.
2. Code chunk grouping: UpdateContractCode now matches bintrie's group-based
key derivation exactly — computing the stem key only at group boundaries
and using groupOffset as the suffix, instead of computing a separate
GetBinaryTreeKey per chunk (which produced different stems).
Cross-validation tests (compat_test.go) assert strict equality:
- TestSingleAccountRootMatch
- TestMultiAccountRootMatch
- TestStorageRootMatch
- TestCodeChunkRootMatch
- TestMixedOpsRootMatch
- 4 BuildInternalTree vs bintrie diagnostic tests
48 tests passing, race-detector clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add stem-level flat state storage and the groupAndHashStems pipeline
that bridges NomtTrie's per-slot updates to NOMT's page-tree:
- stemValueDBKey/stemValueDBPrefix: ethdb key format (0x03||stem||suffix)
- loadStemValues: prefix-iteration loader for all 256 slots of a stem
- writeStemValues: batch writer with nil=delete semantics
- groupAndHashStems: sort→group→load→merge→hash→write pipeline
producing sorted []core.StemKeyValue for the page tree
Cross-validated against bintrie's StemNode.Hash algorithm.
11 new tests, all passing with -race.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add stem-aware key encoding wrappers delegating to bintrie for identical
SHA256 key derivation. Add packBasicData/packStorageValue matching
bintrie's exact encoding layout. Stub trie.go with stemUpdate type
pending Phase E implementation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire the NOMT binary merkle trie engine into geth's triedb/state
framework. This adds two new packages:
- triedb/nomtdb: backend implementing triedb.backend interface, manages
flat state persistence in ethdb and delegates trie ops to nomt/db
- trie/nomttrie: NomtTrie implementing state.Trie, accumulates LeafOps
during block execution and flushes to NOMT engine on Hash()/Commit()
Key design choices:
- Single flat keyspace: accounts use keccak256(addr), storage uses
keccak256(keccak256(addr) || keccak256(slot)) as 256-bit trie paths
- OpenStorageTrie returns the account trie itself (no separate tries)
- Flat state (account/storage values) stored in ethdb with prefixed keys
- NOMT trie stores only hashes; reads delegate to ethdb flat state
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
