Three files had goimports drift from resolving rebase conflicts
(eth/dropper_test.go, eth/fetcher/tx_fetcher.go, eth/handler.go) —
re-run goimports.
Also remove an unused mockConsumer.count() helper in
eth/txtracker/tracker_test.go that no test calls. The method was
left in during the peerstats split and never needed.
Replace NotifyRequestLatency(peer, latency) with
NotifyRequestResult(peer, latency, timeout). The new timeout bool
tells peerstats whether the request was answered or timed out.
Per-peer RequestSuccesses and RequestTimeouts counters replace the
single RequestSamples field — any two of the three are derivable, so
we keep the two primary counters and derive the total
(successes + timeouts) where needed (e.g. the MinLatencySamples
guard in the dropper).
The latency EMA continues to use the timeout value for timed-out
requests, penalizing slow peers as before. The success/timeout
counters are exposed as statistics only — no protection category
uses them yet.
The request-latency category scores peers by the reciprocal of their
RequestLatencyEMA, but that EMA is only updated by NotifyRequestLatency
— which only fires when the tx fetcher sends a request to the peer.
A peer can serve a burst of fast replies to build a strong EMA, stop
announcing transactions so we never request from them again, and
retain latency protection indefinitely with a frozen score.
Record LastLatencySample (wall-clock time) per peer alongside the EMA
update. In the dropper's scoring function, return 0 when the last
sample is older than MaxLatencyStaleness (10 minutes). Fresh samples
reset the clock, so peers that resume activity become eligible again.
Timestamps rather than block counts: real-time is what we actually
care about (10 minutes idle), not a block count that varies with
chain pace, and the EMA itself is a time.Duration so measuring
staleness in the same domain stays consistent.
Tests cover the timestamp update on NotifyRequestLatency, the timestamp
advancing on successive samples, and the dropper rejecting a stale
peer whose EMA and sample count would otherwise qualify.
Require substantially more samples before a peer's request-latency EMA
becomes eligible for protection. A 10-sample floor was too low: a peer
hitting 10 fast replies in a short burst could earn protection before
the slow alpha=0.01 EMA had moved meaningfully away from the bootstrap
value. At ~70-sample EMA half-life, a 100-sample floor means the EMA
has been refined through several half-lives before it can affect
dropping decisions.
Updates the dropper tests that previously used RequestSamples=50 to
use peerstats.MinLatencySamples so they stay robust to future value
changes. Design notes and a test comment reference the new value.
Introduces a new eth/peerstats package as the single home for per-peer
quality metrics consumed by the dropper. txtracker shrinks to a pure
tx-lifecycle role: it maps tx hash to deliverer, subscribes to chain
heads, computes per-block per-peer inclusion and finalization deltas,
and emits them to a StatsConsumer.
peerstats owns the aggregates: inclusion EMA, finalized counter,
latency EMA, sample counter, and the MinLatencySamples bootstrap
constant the dropper uses to filter under-sampled peers. It's a
plain struct with a mutex — no goroutine of its own, no lifecycle
management. The fetcher's onRequestLatency callback now flows to
peerStats.NotifyRequestLatency, the handler's unregisterPeer cleans
up via peerStats.NotifyPeerDrop, and the dropper reads its snapshot
via peerStats.GetAllPeerStats.
txtracker.handleChainHead computes deltas under its own lock, then
releases the lock before calling the consumer, which avoids any
cross-package lock ordering. Tests are split along the same line:
tracker tests use a mock consumer to assert what signals are emitted,
peerstats tests cover EMA math and aggregation semantics directly.
Adds a third protection category to the dropper, scoring peers by
per-peer tx-request response latency. Fast peers are harder to drop;
peers that chronically time out (their EMA drifts toward the 5s
timeout sample) score low and are normal drop candidates.
PeerInclusionStats gains RequestLatencyEMA (time.Duration) and
RequestSamples (int64). The stats adapter in backend.go copies them
from txtracker.PeerStats. The scoring function returns 1/EMA once
the peer has >= MinLatencySamples (10) recorded samples — an
under-sampled peer scores 0 and is filtered by the existing
"score <= 0" rule, preventing a single lucky-fast reply from
displacing established peers.
Adds three unit tests via protectedPeersByPool for the basic
top-N selection, the bootstrap guard, and per-pool independence.
Plugs the tx fetcher's new onRequestLatency callback into the
txtracker's per-peer latency EMA. Tx-request round-trip and timeout
samples now flow into Tracker state and become available to the
dropper as a per-peer PeerInclusionStats.RequestLatencyEMA signal.
Adds an optional onRequestLatency(peer, latency) callback to the tx
fetcher constructor, fired exactly once per request:
- On in-time delivery: the actual round-trip latency (clock.Now - req.time).
- On timeout (req.time + txFetchTimeout exceeded): the timeout value
itself, so slow peers contribute samples instead of being silently
omitted from the downstream EMA.
Late deliveries for requests already counted as timeouts do not
double-record. Existing callers (handler.go, fuzzer, tests) pass nil
for the new parameter; handler wiring to txTracker follows in a
separate commit.
Adds NotifyRequestLatency(peer, latency) and a slow per-peer EMA
(alpha=0.01, ~70-sample half-life) that the dropper will use as a
new protection signal. The first sample seeds the EMA directly so
fresh peers don't ramp up from zero. RequestSamples is exposed
alongside the EMA so consumers can apply a minimum-samples bootstrap
guard before trusting the value.
Includes design notes for the broader peerdrop-latency feature.
The total-finalized protection category ranked peers by a monotonic
cumulative count, so a peer that had been productive in the past kept
a high score forever — even if they had since gone silent — and held
a protected slot without contributing.
Replace txtracker.PeerStats.Finalized (int64 cumulative) with
RecentFinalized (float64 EMA). On each chain head, finalization
credits accumulated over the newly-finalized range are folded into a
slow EMA (alpha=0.0001, half-life ~6930 blocks ≈ 23 hours on 12s
mainnet blocks). Peers that continue contributing keep a high score;
peers that stop decay toward zero over roughly a day.
The dropper category renames to "recent-finalized" accordingly. The
type's docstring is rewritten to describe both categories as EMAs
with different time horizons (slow finalized, fast included).
Refactors checkFinalization to return a per-peer credits map rather
than mutating state directly, so both EMAs update in the same loop
over tracked peers.
PeerInclusionStats was declared identically to txtracker.PeerStats as a
decoupling abstraction: any stats provider could implement the dropper's
callback by returning this shape. In practice there's one provider and
the two types were kept in sync by a rote copy adapter in backend.go.
Delete PeerInclusionStats, have the dropper consume txtracker.PeerStats
directly via getPeerStatsFunc. backend.go now passes
txTracker.GetAllPeerStats as the callback with no adapter.
If a second stats provider ever appears, the abstraction can come back;
until then, one fewer type and 8 fewer lines of ceremony.
The skipped_protected metric (added earlier on this branch) counted the
subset of drop skips where inclusion protection was the cause. The
signal can be inferred from rising dropSkipped rate plus the existing
"Protecting high-value peers" debug log, which wasn't worth the second
metric, the causality-check loop over the protected set, and the
baseNotDrop closure extracted solely to share the predicate.
Collapse baseNotDrop back into selectDoNotDrop and remove the metric.
dropSkipped still fires on every skip (fast-path headroom + all-filtered).
The protection feature promises top-N per inbound/dialed pool, but
every existing test constructed peers via p2p.NewPeer (which produces
no-flag peers), so all test peers landed in the dialed pool and the
per-pool split was never validated.
Extract the selection logic from protectedPeers into a pure helper
protectedPeersByPool(inbound, dialed, stats) that accepts pre-split
pools. This sidesteps the unexported p2p.connFlag types and makes the
interesting behavior directly testable. Add three tests covering:
- exact top-N selected independently in each pool
- cross-category union with overlap deduplication
- per-pool independence: top dialed peers stay protected even when
every inbound peer scores higher globally
handleChainHead resolves the head block via GetBlock(hash, number) so
that a stale head event after a reorg cannot credit transactions from
the wrong block. The existing mockChain ignored the hash argument, so a
regression to GetBlockByNumber would have gone undetected.
Make mockChain hash-aware: store blocks keyed by hash with a separate
canonical-by-number index for the finalization path, and have sendHead
emit the real block's hash. Add TestReorgSafety with two blocks at the
same height to exercise the hash selector directly.
The original assertions read stats["peerA"].Finalized and .RecentIncluded,
both of which return zero for a missing key — so the test would pass even
if NotifyAccepted were a complete no-op, contradicting its stated purpose.
Assert that exactly one peer entry exists, peerA is present, and the
internal txs map and FIFO order slice are populated as NotifyAccepted is
meant to do.
Rename eth/dropper/protected to eth/dropper/skipped and mark it on
every skip (fast-path headroom, all candidates un-droppable, or
protection emptied the list). Add eth/dropper/skipped_protected to
count the subset of skips where at least one otherwise-droppable
peer was kept only because of inclusion protection.
The pair lets operators see both the total churn-miss rate and how
often peer protection specifically is the cause.
When neither the dialed nor inbound peer pool is close to capacity,
every non-trusted/non-static peer is already marked do-not-drop by
the pool-threshold rules in selectDoNotDrop, so the droppable set is
guaranteed empty regardless of inclusion protection.
Return early in that case to avoid the wasted peerStatsFunc call,
per-direction split, and per-category sort in protectedPeers.
Fix goimports violations: align callback field comments in
tx_fetcher.go, sort import of eth/txtracker after eth/protocols
in handler.go. Add missing onAccepted parameter (nil) to the
txfetcher fuzzer.
Tests used time.Sleep(50ms) to wait for async chain head processing,
making the suite slow (~2s) and flaky under CI load.
Add a step channel (buffered 1) to the Tracker, sent after each
event in the loop. Tests wait on the channel with a 1s timeout.
Suite now completes in <10ms.
NotifyPeerDrop deleted t.peers[peer] but left t.txs entries pointing
to that peer. When those txs later finalized, checkFinalization
recreated the peer entry, and the EMA loop decayed it forever.
Fix: create peer entries in NotifyAccepted (when txs are first
accepted), not in handleChainHead or checkFinalization. Both chain
event handlers now skip peers with no entry — disconnected peers
whose entries were deleted by NotifyPeerDrop stay deleted.
order = order[1:] reslices without releasing the backing array.
After N total insertions the array retains N hashes (32 bytes each)
but only the last maxTracked are live. On a long-running node
processing ~100 txs/s this leaks ~275 MB/day.
Compact by copying to a fresh array when capacity exceeds 2×maxTracked.
Remove the custom topN generic function. Use slices.SortedFunc
(creates a sorted copy from an iterator) + slices.DeleteFunc (filters
score <= 0) from the standard library. No custom generics needed.
Replace peerWithStats wrapper, manual slice copying, and protectTopN
closure with a generic topN[T] function that sorts by score and
returns top elements. protectedPeers now works directly with
[]*p2p.Peer slices, building per-category score functions that close
over the stats map.
Compute the protected peer set once in dropRandomPeer via
protectedPeers(), then include protection as a condition in
selectDoNotDrop alongside trusted/static/recent checks. This
eliminates the separate filterProtectedPeers post-pass and the
awkward "all protected → skip" branch.
Rename filterProtectedPeers to protectedPeers, returning
map[*p2p.Peer]bool instead of filtering a slice. The map is
checked directly in selectDoNotDrop via protected[p].
enqueueAndTrack used pool.Has() after Enqueue to determine accepted
txs. Under concurrent delivery of the same tx from two peers, both
could see Has()==true, making attribution non-deterministic.
Add an onAccepted callback to the fetcher, called from Enqueue with
(peer, acceptedHashes) immediately after pool.Add returns for each
batch. Attribution happens atomically inside Enqueue using the per-tx
error from addTxs (nil = accepted), before another goroutine can
race.
Remove the enqueueAndTrack helper from handler_eth.go — the fetcher
now handles notification directly.
protectTopN used maxPeers (configured capacity) to compute the
number of peers to protect. With small droppable sets this could
protect everyone, permanently disabling churn.
Use len(entries) (current droppable count in each category) instead.
With 20 droppable dialed peers and 10% fraction, 2 are protected.
With 3 droppable peers, 0 are protected — churn is never blocked.
Peer stats were never pruned, so the peers map grew with every peer
ever seen. The EMA decay loop and stats copy iterated all historical
peers on every block/query.
Add NotifyPeerDrop(peer) that deletes the peer's stats entry. Called
from handler.unregisterPeer alongside txFetcher.Drop.
handleChainHead fetched the block by number only. If the tracker
goroutine lagged and that height was reorged before processing,
the EMA was computed from the wrong canonical block.
Use GetBlock(hash, number) with the header hash from the event to
fetch the exact block the event refers to, not whatever is currently
canonical at that height.
NotifyReceived was called before pool validation, allowing a peer
to claim deliverer credit by replaying already-included txs or
sending invalid packets.
Rename to NotifyAccepted (takes hashes, not full txs). Call it from
a new enqueueAndTrack helper in handler_eth.go that runs after
Enqueue and checks pool.Has to identify accepted txs. Only accepted
txs are credited to the delivering peer.
lastFinalNum started at 0, so the first checkFinalization after
startup iterated from block 1 to the current finalized head (~20M
blocks on mainnet) under the mutex, stalling the tracker and
potentially awarding bogus credit for ancient txs whose hashes
happened to match recently-received ones.
Seed lastFinalNum from chain.CurrentFinalBlock() in Start() so only
blocks finalized after startup are processed.
txtracker tests (7 tests):
- NotifyReceived: stats empty before chain events
- InclusionEMA: EMA increases on inclusion, decays on empty blocks
- Finalization: Finalized counter credited after finalization
- MultiplePeers: each peer credited for own txs only
- FirstDelivererWins: duplicate delivery ignored
- NoFinalizationCredit: no credit without finalization
- EMADecay: EMA approaches zero after 30 empty blocks
dropper tests (6 tests):
- FilterProtectedNoStats: nil stats → all droppable
- FilterProtectedEmptyStats: empty map → all droppable
- FilterProtectedTopPeer: top-scored peers removed from droppable
- FilterProtectedZeroScore: zero scores → no protection
- FilterProtectedOverlap: peer top in both categories → counted once
- FilterProtectedAllProtected: all droppable protected → empty list
Also fix: create peer entries during EMA update for peers with
inclusions in the current block (previously only created during
finalization, so EMA was not tracked before first finalization).
Expand the txtracker package doc to describe the tracking flow
(NotifyReceived → chain head → finalization → peer credit) and its
role as stats provider for the dropper.
Rewrite the dropper struct comment to document the full behavior
including the inclusion-based peer protection: two scoring categories
(total finalized + recent EMA), top 10% per pool, union of protected
sets.
Change the long-term protection category from total inclusions to
total finalized inclusions. Finalized txs are harder to game (require
actual block finality, not just inclusion) and represent confirmed
on-chain value.
The recent-inclusion EMA stays on chain head inclusions for
responsiveness — a peer delivering txs that appear in the latest
blocks gets quick protection without waiting for finalization.
The tracker now checks CurrentFinalBlock() on each chain head event
and credits delivering peers for all newly finalized blocks since
the last check.
Minimal txtracker that records which peer delivered each transaction
and credits peers when their transactions appear on chain. Provides
the PeerInclusionStats needed by the dropper's protection logic.
Design:
- NotifyReceived(peer, txs): records deliverer per tx hash (called
from handler_eth.go when tx bodies arrive via P2P)
- Subscribes to ChainHeadEvent, fetches block txs, credits the
delivering peer for each included tx
- Per-peer EMA of recent inclusions (alpha=0.05), updated every block
- LRU eviction at 262K entries to bound memory
- Mutex-based (not channel-based) for simplicity — the hot path
(NotifyReceived) is a fast map insert
Wired into the dropper via an adapter callback in backend.go that
converts txtracker.PeerStats to the dropper's PeerInclusionStats.
The dropper periodically disconnects random peers to create churn.
This was blind to peer quality. Add inclusion-based peer protection
using two categories:
1. Total inclusions: protects peers with the highest cumulative
count of delivered txs that were included on chain
2. Recent inclusions (EMA): protects peers with the best recent
inclusion rate, giving newly productive peers faster protection
Each category independently protects the top 10% of inbound and
top 10% of dialed peers. The union of both sets is protected. Only
peers with positive scores qualify.
The dropper defines its own PeerInclusionStats struct and callback
type (getPeerInclusionStatsFunc) so any stats provider (e.g. a
transaction tracker) can plug in without a package dependency. The
callback is nil by default (protection disabled until wired).
The protectionCategories slice is designed for easy extension —
adding a new category requires only appending a struct with a name,
scoring function, and protection fraction.
This PR refactors the encoding rules for `AccessListsPacket` in the wire
protocol. Specifically:
- The response is now encoded as a list of `rlp.RawValue`
- `rlp.EmptyString` is used as a placeholder for unavailable BAL objects
In this PR, the Database interface in `core/state` has been extended
with one more function:
```go
// Iteratee returns a state iteratee associated with the specified state root,
// through which the account iterator and storage iterator can be created.
Iteratee(root common.Hash) (Iteratee, error)
```
With this additional abstraction layer, the implementation details can be hidden
behind the interface. For example, state traversal can now operate directly on
the flat state for Verkle or binary trees, which do not natively support traversal.
Moreover, state dumping will now prefer using the flat state iterator as
the primary option, offering better efficiency.
Edit: this PR also fixes a tiny issue in the state dump, marshalling the
next field in the correct way.
This is a breaking change in the opcode (structLog) tracer. Several fields
will have a slight formatting difference to conform to the newly established
spec at: https://github.com/ethereum/execution-apis/pull/762. The differences
include:
- `memory`: words will have the 0x prefix. Also last word of memory will be padded to 32-bytes.
- `storage`: keys and values will have the 0x prefix.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
In this PR, we add support for protocol version eth/70, defined by EIP-7975.
Overall changes:
- Each response is buffered in the peer’s receipt buffer when the
`lastBlockIncomplete` field is true.
- Continued request uses the same request id of its original
request(`RequestPartialReceipts`).
- Partial responses are verified in `validateLastBlockReceipt`.
- Even if all receipts for partial blocks of the request are collected,
those partial results are not sinked to the downloader, to avoid
complexity. This assumes that partial response and buffering occur only
in exceptional cases.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR introduces a new type HistoryPolicy which captures user intent
as opposed to pruning point stored in the blockchain which persists the
actual tail of data in the database.
It is in preparation for the rolling history expiry feature.
It comes with a semantic change: if database was pruned and geth is
running without a history mode flag (or explicit keep all flag) geth
will emit a warning but continue running as opposed to stopping the
world.
