The request-latency category scores peers by the reciprocal of their
RequestLatencyEMA, but that EMA is only updated by NotifyRequestLatency
— which only fires when the tx fetcher sends a request to the peer.
A peer can serve a burst of fast replies to build a strong EMA, stop
announcing transactions so we never request from them again, and
retain latency protection indefinitely with a frozen score.
Record LastLatencySample (wall-clock time) per peer alongside the EMA
update. In the dropper's scoring function, return 0 when the last
sample is older than MaxLatencyStaleness (10 minutes). Fresh samples
reset the clock, so peers that resume activity become eligible again.
Timestamps rather than block counts: real-time is what we actually
care about (10 minutes idle), not a block count that varies with
chain pace, and the EMA itself is a time.Duration so measuring
staleness in the same domain stays consistent.
Tests cover the timestamp update on NotifyRequestLatency, the timestamp
advancing on successive samples, and the dropper rejecting a stale
peer whose EMA and sample count would otherwise qualify.
Require substantially more samples before a peer's request-latency EMA
becomes eligible for protection. A 10-sample floor was too low: a peer
hitting 10 fast replies in a short burst could earn protection before
the slow alpha=0.01 EMA had moved meaningfully away from the bootstrap
value. At ~70-sample EMA half-life, a 100-sample floor means the EMA
has been refined through several half-lives before it can affect
dropping decisions.
Updates the dropper tests that previously used RequestSamples=50 to
use peerstats.MinLatencySamples so they stay robust to future value
changes. Design notes and a test comment reference the new value.
Introduces a new eth/peerstats package as the single home for per-peer
quality metrics consumed by the dropper. txtracker shrinks to a pure
tx-lifecycle role: it maps tx hash to deliverer, subscribes to chain
heads, computes per-block per-peer inclusion and finalization deltas,
and emits them to a StatsConsumer.
peerstats owns the aggregates: inclusion EMA, finalized counter,
latency EMA, sample counter, and the MinLatencySamples bootstrap
constant the dropper uses to filter under-sampled peers. It's a
plain struct with a mutex — no goroutine of its own, no lifecycle
management. The fetcher's onRequestLatency callback now flows to
peerStats.NotifyRequestLatency, the handler's unregisterPeer cleans
up via peerStats.NotifyPeerDrop, and the dropper reads its snapshot
via peerStats.GetAllPeerStats.
txtracker.handleChainHead computes deltas under its own lock, then
releases the lock before calling the consumer, which avoids any
cross-package lock ordering. Tests are split along the same line:
tracker tests use a mock consumer to assert what signals are emitted,
peerstats tests cover EMA math and aggregation semantics directly.
Adds a third protection category to the dropper, scoring peers by
per-peer tx-request response latency. Fast peers are harder to drop;
peers that chronically time out (their EMA drifts toward the 5s
timeout sample) score low and are normal drop candidates.
PeerInclusionStats gains RequestLatencyEMA (time.Duration) and
RequestSamples (int64). The stats adapter in backend.go copies them
from txtracker.PeerStats. The scoring function returns 1/EMA once
the peer has >= MinLatencySamples (10) recorded samples — an
under-sampled peer scores 0 and is filtered by the existing
"score <= 0" rule, preventing a single lucky-fast reply from
displacing established peers.
Adds three unit tests via protectedPeersByPool for the basic
top-N selection, the bootstrap guard, and per-pool independence.
Plugs the tx fetcher's new onRequestLatency callback into the
txtracker's per-peer latency EMA. Tx-request round-trip and timeout
samples now flow into Tracker state and become available to the
dropper as a per-peer PeerInclusionStats.RequestLatencyEMA signal.
Adds an optional onRequestLatency(peer, latency) callback to the tx
fetcher constructor, fired exactly once per request:
- On in-time delivery: the actual round-trip latency (clock.Now - req.time).
- On timeout (req.time + txFetchTimeout exceeded): the timeout value
itself, so slow peers contribute samples instead of being silently
omitted from the downstream EMA.
Late deliveries for requests already counted as timeouts do not
double-record. Existing callers (handler.go, fuzzer, tests) pass nil
for the new parameter; handler wiring to txTracker follows in a
separate commit.
Adds NotifyRequestLatency(peer, latency) and a slow per-peer EMA
(alpha=0.01, ~70-sample half-life) that the dropper will use as a
new protection signal. The first sample seeds the EMA directly so
fresh peers don't ramp up from zero. RequestSamples is exposed
alongside the EMA so consumers can apply a minimum-samples bootstrap
guard before trusting the value.
Includes design notes for the broader peerdrop-latency feature.
The total-finalized protection category ranked peers by a monotonic
cumulative count, so a peer that had been productive in the past kept
a high score forever — even if they had since gone silent — and held
a protected slot without contributing.
Replace txtracker.PeerStats.Finalized (int64 cumulative) with
RecentFinalized (float64 EMA). On each chain head, finalization
credits accumulated over the newly-finalized range are folded into a
slow EMA (alpha=0.0001, half-life ~6930 blocks ≈ 23 hours on 12s
mainnet blocks). Peers that continue contributing keep a high score;
peers that stop decay toward zero over roughly a day.
The dropper category renames to "recent-finalized" accordingly. The
type's docstring is rewritten to describe both categories as EMAs
with different time horizons (slow finalized, fast included).
Refactors checkFinalization to return a per-peer credits map rather
than mutating state directly, so both EMAs update in the same loop
over tracked peers.
PeerInclusionStats was declared identically to txtracker.PeerStats as a
decoupling abstraction: any stats provider could implement the dropper's
callback by returning this shape. In practice there's one provider and
the two types were kept in sync by a rote copy adapter in backend.go.
Delete PeerInclusionStats, have the dropper consume txtracker.PeerStats
directly via getPeerStatsFunc. backend.go now passes
txTracker.GetAllPeerStats as the callback with no adapter.
If a second stats provider ever appears, the abstraction can come back;
until then, one fewer type and 8 fewer lines of ceremony.
The skipped_protected metric (added earlier on this branch) counted the
subset of drop skips where inclusion protection was the cause. The
signal can be inferred from rising dropSkipped rate plus the existing
"Protecting high-value peers" debug log, which wasn't worth the second
metric, the causality-check loop over the protected set, and the
baseNotDrop closure extracted solely to share the predicate.
Collapse baseNotDrop back into selectDoNotDrop and remove the metric.
dropSkipped still fires on every skip (fast-path headroom + all-filtered).
The protection feature promises top-N per inbound/dialed pool, but
every existing test constructed peers via p2p.NewPeer (which produces
no-flag peers), so all test peers landed in the dialed pool and the
per-pool split was never validated.
Extract the selection logic from protectedPeers into a pure helper
protectedPeersByPool(inbound, dialed, stats) that accepts pre-split
pools. This sidesteps the unexported p2p.connFlag types and makes the
interesting behavior directly testable. Add three tests covering:
- exact top-N selected independently in each pool
- cross-category union with overlap deduplication
- per-pool independence: top dialed peers stay protected even when
every inbound peer scores higher globally
handleChainHead resolves the head block via GetBlock(hash, number) so
that a stale head event after a reorg cannot credit transactions from
the wrong block. The existing mockChain ignored the hash argument, so a
regression to GetBlockByNumber would have gone undetected.
Make mockChain hash-aware: store blocks keyed by hash with a separate
canonical-by-number index for the finalization path, and have sendHead
emit the real block's hash. Add TestReorgSafety with two blocks at the
same height to exercise the hash selector directly.
The original assertions read stats["peerA"].Finalized and .RecentIncluded,
both of which return zero for a missing key — so the test would pass even
if NotifyAccepted were a complete no-op, contradicting its stated purpose.
Assert that exactly one peer entry exists, peerA is present, and the
internal txs map and FIFO order slice are populated as NotifyAccepted is
meant to do.
Rename eth/dropper/protected to eth/dropper/skipped and mark it on
every skip (fast-path headroom, all candidates un-droppable, or
protection emptied the list). Add eth/dropper/skipped_protected to
count the subset of skips where at least one otherwise-droppable
peer was kept only because of inclusion protection.
The pair lets operators see both the total churn-miss rate and how
often peer protection specifically is the cause.
When neither the dialed nor inbound peer pool is close to capacity,
every non-trusted/non-static peer is already marked do-not-drop by
the pool-threshold rules in selectDoNotDrop, so the droppable set is
guaranteed empty regardless of inclusion protection.
Return early in that case to avoid the wasted peerStatsFunc call,
per-direction split, and per-category sort in protectedPeers.
Fix goimports violations: align callback field comments in
tx_fetcher.go, sort import of eth/txtracker after eth/protocols
in handler.go. Add missing onAccepted parameter (nil) to the
txfetcher fuzzer.
Tests used time.Sleep(50ms) to wait for async chain head processing,
making the suite slow (~2s) and flaky under CI load.
Add a step channel (buffered 1) to the Tracker, sent after each
event in the loop. Tests wait on the channel with a 1s timeout.
Suite now completes in <10ms.
NotifyPeerDrop deleted t.peers[peer] but left t.txs entries pointing
to that peer. When those txs later finalized, checkFinalization
recreated the peer entry, and the EMA loop decayed it forever.
Fix: create peer entries in NotifyAccepted (when txs are first
accepted), not in handleChainHead or checkFinalization. Both chain
event handlers now skip peers with no entry — disconnected peers
whose entries were deleted by NotifyPeerDrop stay deleted.
order = order[1:] reslices without releasing the backing array.
After N total insertions the array retains N hashes (32 bytes each)
but only the last maxTracked are live. On a long-running node
processing ~100 txs/s this leaks ~275 MB/day.
Compact by copying to a fresh array when capacity exceeds 2×maxTracked.
Remove the custom topN generic function. Use slices.SortedFunc
(creates a sorted copy from an iterator) + slices.DeleteFunc (filters
score <= 0) from the standard library. No custom generics needed.
Replace peerWithStats wrapper, manual slice copying, and protectTopN
closure with a generic topN[T] function that sorts by score and
returns top elements. protectedPeers now works directly with
[]*p2p.Peer slices, building per-category score functions that close
over the stats map.
Compute the protected peer set once in dropRandomPeer via
protectedPeers(), then include protection as a condition in
selectDoNotDrop alongside trusted/static/recent checks. This
eliminates the separate filterProtectedPeers post-pass and the
awkward "all protected → skip" branch.
Rename filterProtectedPeers to protectedPeers, returning
map[*p2p.Peer]bool instead of filtering a slice. The map is
checked directly in selectDoNotDrop via protected[p].
enqueueAndTrack used pool.Has() after Enqueue to determine accepted
txs. Under concurrent delivery of the same tx from two peers, both
could see Has()==true, making attribution non-deterministic.
Add an onAccepted callback to the fetcher, called from Enqueue with
(peer, acceptedHashes) immediately after pool.Add returns for each
batch. Attribution happens atomically inside Enqueue using the per-tx
error from addTxs (nil = accepted), before another goroutine can
race.
Remove the enqueueAndTrack helper from handler_eth.go — the fetcher
now handles notification directly.
protectTopN used maxPeers (configured capacity) to compute the
number of peers to protect. With small droppable sets this could
protect everyone, permanently disabling churn.
Use len(entries) (current droppable count in each category) instead.
With 20 droppable dialed peers and 10% fraction, 2 are protected.
With 3 droppable peers, 0 are protected — churn is never blocked.
Peer stats were never pruned, so the peers map grew with every peer
ever seen. The EMA decay loop and stats copy iterated all historical
peers on every block/query.
Add NotifyPeerDrop(peer) that deletes the peer's stats entry. Called
from handler.unregisterPeer alongside txFetcher.Drop.
handleChainHead fetched the block by number only. If the tracker
goroutine lagged and that height was reorged before processing,
the EMA was computed from the wrong canonical block.
Use GetBlock(hash, number) with the header hash from the event to
fetch the exact block the event refers to, not whatever is currently
canonical at that height.
NotifyReceived was called before pool validation, allowing a peer
to claim deliverer credit by replaying already-included txs or
sending invalid packets.
Rename to NotifyAccepted (takes hashes, not full txs). Call it from
a new enqueueAndTrack helper in handler_eth.go that runs after
Enqueue and checks pool.Has to identify accepted txs. Only accepted
txs are credited to the delivering peer.
lastFinalNum started at 0, so the first checkFinalization after
startup iterated from block 1 to the current finalized head (~20M
blocks on mainnet) under the mutex, stalling the tracker and
potentially awarding bogus credit for ancient txs whose hashes
happened to match recently-received ones.
Seed lastFinalNum from chain.CurrentFinalBlock() in Start() so only
blocks finalized after startup are processed.
txtracker tests (7 tests):
- NotifyReceived: stats empty before chain events
- InclusionEMA: EMA increases on inclusion, decays on empty blocks
- Finalization: Finalized counter credited after finalization
- MultiplePeers: each peer credited for own txs only
- FirstDelivererWins: duplicate delivery ignored
- NoFinalizationCredit: no credit without finalization
- EMADecay: EMA approaches zero after 30 empty blocks
dropper tests (6 tests):
- FilterProtectedNoStats: nil stats → all droppable
- FilterProtectedEmptyStats: empty map → all droppable
- FilterProtectedTopPeer: top-scored peers removed from droppable
- FilterProtectedZeroScore: zero scores → no protection
- FilterProtectedOverlap: peer top in both categories → counted once
- FilterProtectedAllProtected: all droppable protected → empty list
Also fix: create peer entries during EMA update for peers with
inclusions in the current block (previously only created during
finalization, so EMA was not tracked before first finalization).
Expand the txtracker package doc to describe the tracking flow
(NotifyReceived → chain head → finalization → peer credit) and its
role as stats provider for the dropper.
Rewrite the dropper struct comment to document the full behavior
including the inclusion-based peer protection: two scoring categories
(total finalized + recent EMA), top 10% per pool, union of protected
sets.
Change the long-term protection category from total inclusions to
total finalized inclusions. Finalized txs are harder to game (require
actual block finality, not just inclusion) and represent confirmed
on-chain value.
The recent-inclusion EMA stays on chain head inclusions for
responsiveness — a peer delivering txs that appear in the latest
blocks gets quick protection without waiting for finalization.
The tracker now checks CurrentFinalBlock() on each chain head event
and credits delivering peers for all newly finalized blocks since
the last check.
Minimal txtracker that records which peer delivered each transaction
and credits peers when their transactions appear on chain. Provides
the PeerInclusionStats needed by the dropper's protection logic.
Design:
- NotifyReceived(peer, txs): records deliverer per tx hash (called
from handler_eth.go when tx bodies arrive via P2P)
- Subscribes to ChainHeadEvent, fetches block txs, credits the
delivering peer for each included tx
- Per-peer EMA of recent inclusions (alpha=0.05), updated every block
- LRU eviction at 262K entries to bound memory
- Mutex-based (not channel-based) for simplicity — the hot path
(NotifyReceived) is a fast map insert
Wired into the dropper via an adapter callback in backend.go that
converts txtracker.PeerStats to the dropper's PeerInclusionStats.
The dropper periodically disconnects random peers to create churn.
This was blind to peer quality. Add inclusion-based peer protection
using two categories:
1. Total inclusions: protects peers with the highest cumulative
count of delivered txs that were included on chain
2. Recent inclusions (EMA): protects peers with the best recent
inclusion rate, giving newly productive peers faster protection
Each category independently protects the top 10% of inbound and
top 10% of dialed peers. The union of both sets is protected. Only
peers with positive scores qualify.
The dropper defines its own PeerInclusionStats struct and callback
type (getPeerInclusionStatsFunc) so any stats provider (e.g. a
transaction tracker) can plug in without a package dependency. The
callback is nil by default (protection disabled until wired).
The protectionCategories slice is designed for easy extension —
adding a new category requires only appending a struct with a name,
scoring function, and protection fraction.
This tool is designed for the offline translation of an MPT database to
a binary trie. This is to be used for users who e.g. want to prove
equivalence of a binary tree chain shadowing the MPT chain.
It adds a `bintrie` command, cleanly separating the concerns.
This PR fixes https://github.com/ethereum/go-ethereum/issues/34623 by
changing the `vm.StateDB` interface:
Instead of `EmitLogsForBurnAccounts()` emitting burn logs, `LogsForBurnAccounts()
[]*types.Log` just returns these logs which are then emitted by the caller.
This way when tracing is used, `hookedStateDB.AddLog` will be used
automatically and there is no need to duplicate either the burn log
logic or the `OnLog` tracing hook.
ProcessBeaconBlockRoot (EIP-4788) and processRequestsSystemCall
(EIP-7002/7251) do not merge the EVM access events into the state after
execution. ProcessParentBlockHash (EIP-2935) already does this correctly
at line 290-291.
Without this merge, the Verkle witness will be missing the storage
accesses from the beacon root and request system calls, leading to
incomplete witnesses and potential consensus issues when Verkle
activates.
PathDB keys diff layers by state root, not by block hash. That means a
side-chain block can legitimately collide with an existing canonical diff layer
when both blocks produce the same post-state (for example same parent,
same coinbase, no txs).
Today `layerTree.add` blindly inserts that second layer. If the root
already exists, this overwrites `tree.layers[root]` and appends the same
root to the mutation lookup again. Later account/storage lookups resolve
that root to the wrong diff layer, which can corrupt reads for descendant
canonical states.
At runtime, the corruption is silent: no error is logged and no invariant check
fires. State reads against affected descendants simply return stale data
from the wrong diff layer (for example, an account balance that reflects one
fewer block reward), which can propagate into RPC responses and block
validation.
This change makes duplicate-root inserts idempotent. A second layer with
the same state root does not add any new retrievable state to a tree that is
already keyed by root; keeping the original layer preserves the existing parent
chain and avoids polluting the lookup history with duplicate roots.
The regression test imports a canonical chain of two layers followed by
a fork layer at height 1 with the same state root but a different block hash.
Before the fix, account and storage lookups at the head resolve the fork
layer instead of the canonical one. After the fix, the duplicate insert is
skipped and lookups remain correct.
This PR refactors the encoding rules for `AccessListsPacket` in the wire
protocol. Specifically:
- The response is now encoded as a list of `rlp.RawValue`
- `rlp.EmptyString` is used as a placeholder for unavailable BAL objects
This PR adds `GenerateTrie(db, scheme, root)` to the `triedb` package,
which rebuilds all tries from flat snapshot KV data. This is needed by
snap/2 sync so it can rebuild the trie after downloading the flat state.
The shared trie generation pipeline from `pathdb/verifier.go` was moved
into `triedb/internal/conversion.go` so both `GenerateTrie` and
`VerifyState` reuse the same code.
👋
This PR makes it possible to run "Amsterdam" in statetests. I'm aware
that they'll be failing and not in consensus with other clients, yet,
but it's nice to be able to run tests and see what works and what
doesn't
Before the change:
```
$ go run ./cmd/evm statetest ./amsterdam.json
[
{
"name": "00000019-mixed-1",
"pass": false,
"fork": "Amsterdam",
"error": "unexpected error: unsupported fork \"Amsterdam\""
}
]
```
After
```
$ go run ./cmd/evm statetest ./amsterdam.json
{"stateRoot": "0x25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e"}
[
{
"name": "00000019-mixed-1",
"pass": false,
"stateRoot": "0x25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e",
"fork": "Amsterdam",
"error": "post state root mismatch: got 25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e, want 0000000000000000000000000000000000000000000000000000000000000000"
}
]
```
