Parallelize the PageWalker trie update across multiple goroutines by
partitioning sorted operations by the root page's 64 child subtrees
(first 6 bits of each key path).
Each worker runs an independent PageWalker constrained to child pages
below the root (using parentPage mechanism), producing ChildPageRoots.
After all workers complete, a root walker places the child roots using
AdvanceAndPlaceNode and concludes with the final trie root.
Workers operate on disjoint page subtrees so no synchronization is
needed during computation — only sync.WaitGroup for goroutine join.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire the NOMT binary merkle trie engine into geth's triedb/state
framework. This adds two new packages:
- triedb/nomtdb: backend implementing triedb.backend interface, manages
flat state persistence in ethdb and delegates trie ops to nomt/db
- trie/nomttrie: NomtTrie implementing state.Trie, accumulates LeafOps
during block execution and flushes to NOMT engine on Hash()/Commit()
Key design choices:
- Single flat keyspace: accounts use keccak256(addr), storage uses
keccak256(keccak256(addr) || keccak256(slot)) as 256-bit trie paths
- OpenStorageTrie returns the account trie itself (no separate tries)
- Flat state (account/storage values) stored in ethdb with prefixed keys
- NOMT trie stores only hashes; reads delegate to ethdb flat state
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement crash-safe persistence for the Bitbox hash table:
- wal.go: WAL format with START/CLEAR/UPDATE/END entries, builder/reader
- sync.go: 3-phase sync protocol (BeginSync → WriteWAL → CommitSync)
- recover.go: WAL replay for crash recovery
The WAL records page diffs (not full pages) for compact logging. The
3-phase protocol ensures: WAL fsynced before HT modification, HT fsynced
before WAL truncation, providing at-least-once delivery of page updates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement the on-disk open-addressing hash table for storing trie pages:
- htfile.go: HT file layout with header, meta pages, and data pages
- metamap.go: in-memory meta byte map with dirty page tracking
- probe.go: triangular probing with xxhash64 page ID hashing
- db.go: Bitbox DB with StorePage, LoadPage, DeletePage, FlushMeta, Sync
The hash table uses 1-byte meta tags (top 7 bits of hash) for fast
filtering before reading full 4096-byte data pages. Triangular probing
with power-of-2 capacity guarantees all buckets are visited.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
modify keyToPath
add comment
add credit
clean up
more clean up
moree clean up
chore
moreee clean up
delete more unused methods
# Conflicts:
# trie/bintrie/expired_node.go
# trie/bintrie/expired_node_test.go
This fixes two cases where `Iterator.Err()` was misused. The method will
only return an error after `Next()` has returned false, so it makes no
sense to check for the error within the loop itself.
Most uses of the iterator are like this:
it, _ := rlp.NewListIterator(data)
for it.Next() {
do(it.Value())
}
This doesn't require the iterator to be a pointer and it's better to
have it stack-allocated. AFAIK the compiler cannot prove it is OK to
stack-allocate when it is returned as a pointer because the methods of
`Iterator` use pointer receiver and also mutate the object.
The iterator type was not exported until very recently, so I think it is
still OK to change this API.
GetStorage and DeleteStorage used GetBinaryTreeKey to compute the tree
key, while UpdateStorage used GetBinaryTreeKeyStorageSlot. The latter
applies storage slot remapping (header offset for slots <64, main
storage prefix for the rest), so reads and deletes were targeting
different tree locations than writes.
Replace GetBinaryTreeKey with GetBinaryTreeKeyStorageSlot in both
GetStorage and DeleteStorage to match UpdateStorage. Add a regression
test that verifies the write→read→delete→read round-trip for main
storage slots.
The `decodeRef` function used `size > hashLen` to reject oversized
embedded nodes, but this incorrectly allowed nodes of exactly 32 bytes
through. The encoding side (hasher.go, stacktrie.go) consistently uses
`len(enc) < 32` to decide whether to embed a node inline, meaning nodes
of 32+ bytes are always hash-referenced. The error message itself
already stated `want size < 32`, confirming the intended threshold.
Changed `size > hashLen` to `size >= hashLen` in `decodeRef` to align
the decoding validation with the encoding logic, the Yellow Paper spec,
and the surrounding comments.
This PR fixes a panic in a corner case situation when a `ChainEvent` is
received by `eth.Ethereum.updateFilterMapsHeads()` but the given chain
section does not exist in `BlockChain` any more. This can happen during
chain rewind because chain events are processed asynchronously. Ignoring
the event in this case is ok, the final event will point to the final
rewound head and the indexer will be updated.
Note that similar issues will not happen once we transition to
https://github.com/ethereum/go-ethereum/pull/32292 and the new indexer
built on top of this. Until then, the current fix should be fine.
This PR makes `TestEIP8024_Execution` verify explicit error types (e.g.,
`ErrStackUnderflow` vs `ErrInvalidOpCode`) rather than accepting any
error. It also fails fast on unexpected opcodes in the mini-interpreter
to avoid false positives from missing opcode handling.
Here is a draft for the New EraE implementation. The code follows along
with the spec listed at https://hackmd.io/pIZlxnitSciV5wUgW6W20w.
---------
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
### Problem
`HasBody` and `HasReceipts` returned `true` for pruned blocks because
they only checked `isCanon()` which verifies the hash table — but
hash/header tables have `prunable: false` while body/receipt tables have
`prunable: true`.
After `TruncateTail()`, hashes still exist but bodies/receipts are gone.
This caused inconsistency: `HasBody()` returns `true`, but `ReadBody()`
returns `nil`.
### Changes
Both functions now check `db.Tail()` when the block is in ancient store.
If `number < tail`, the data has been pruned and the function correctly
returns `false`.
This aligns `HasBody`/`HasReceipts` behavior with
`ReadBody`/`ReadReceipts` and fixes potential issues in
`skeleton.linked()` which relies on these checks during sync.
Follow-up to https://github.com/ethereum/go-ethereum/pull/33748
Same issue - ResettingTimer can be registered via loadOrRegister() but
GetAll() silently drops it during JSON export. The prometheus exporter
handles it fine (collector.go:70), so this is just an oversight in the
JSON path.
Note: ResettingTimer.Snapshot() resets the timer by design, which is
consistent with how the prometheus exporter uses it.
This adds a new type wrapper that decodes as a list, but does not
actually decode the contents of the list. The type parameter exists as a
marker, and enables decoding the elements lazily. RawList can also be
used for building a list incrementally.
The upstream libray has removed the assembly-based implementation of
keccak. We need to maintain our own library to avoid a peformance
regression.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
kzg4844.Blob is 131072 bytes. Using `for _, blob := range` copies the
entire blob on each iteration. With up to 6 blobs per transaction, this
wastes ~768KB of memory copies.
Switch to index-based iteration and pass pointers directly.
The `Witness` method was not implemented for the binary tree, which
caused `debug_excutionWitness` to panic. This PR fixes that.
Note that the `TransitionTrie` version isn't implemented, and that's on
purpose: more thought must be given to what should go in the global
witness.
I recently went on a longer flight and started profiling the geth block
production pipeline.
This PR contains a bunch of individual fixes split into separate
commits.
I can drop some if necessary.
Benchmarking is not super easy, the benchmark I wrote is a bit
non-deterministic.
I will try to write a better benchmark later
```
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/miner
cpu: Intel(R) Core(TM) Ultra 7 155U
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
BuildPayload-14 141.5µ ± 3% 146.0µ ± 6% ~ (p=0.346 n=200)
│ /tmp/old.txt │ /tmp/new.txt │
│ B/op │ B/op vs base │
BuildPayload-14 188.2Ki ± 4% 177.4Ki ± 4% -5.71% (p=0.018 n=200)
│ /tmp/old.txt │ /tmp/new.txt │
│ allocs/op │ allocs/op vs base │
BuildPayload-14 2.703k ± 4% 2.453k ± 5% -9.25% (p=0.000 n=200)
```
Preallocates slices with known capacity in `stateSet.encode()` and
`StateSetWithOrigin.encode()` methods to eliminate redundant
reallocations during serialization.
Preallocate capacity for `keyOffsets` and `valOffsets` slices in
`decodeRestartTrailer` since the exact size (`nRestarts`) is known
upfront.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
core/state: add bounds check in heap eviction loop
Add len(h) > 0 check before accessing h[0] to prevent potential panic
and align with existing heap access patterns in txpool, p2p, and mclock
packages.
Fix ECIES invalid-curve handling in RLPx handshake (reject invalid
ephemeral pubkeys early)
- Add curve validation in crypto/ecies.GenerateShared to reject invalid
public keys before ECDH.
- Update RLPx PoC test to assert invalid curve points fail with
ErrInvalidPublicKey.
Motivation / Context
RLPx handshake uses ECIES decryption on unauthenticated network input.
Prior to this change, an invalid-curve ephemeral public key would
proceed into ECDH and only fail at MAC verification, returning
ErrInvalidMessage. This allows an oracle on decrypt success/failure and
leaves the code path vulnerable to invalid-curve/small-subgroup attacks.
The fix enforces IsOnCurve validation up front.
Heartbeats are used to drop non-executable transactions from the queue.
The timeout mechanism was not clearly documented, and it was updates
also when not necessary.
This PR restores the previous Pebble configuration, disabling seek compaction.
This feature is still needed by hash mode archive node, mitigating the
overhead of frequent compaction.
