commitPivotBlock updated committed without pivotLock, while other pivot
transitions serialize through that lock. Take the lock around the store
to keep pivot commitment consistent with pivotHeader updates.
This PR coordinates the prefetcher with the main tx executor. Block
processing publishes the index of the transaction it is executing,
prefetch workers skip anything already reached and transactions above 1M
gas are promoted to the front of the prefetch queue while the rest keeps
block order.
The deep-reorg check used depth >= maxReorgDepth, rejecting reorgs at
exactly the configured limit. Use > so a depth equal to maxReorgDepth is
still accepted.
When a peer drop removes the last waiter for a hash, waitlist and
waittime were cleaned up but partial could retain a stale entry. Delete
it too so the hash is fully forgotten.
This PR improves the sync mode management, disabling the snap mode once
the pivot block is committed.
Originally the mode will only be flipped from snap to full once the entire sync
cycle is completed. However, it's theoretically possible to cancel the sync cycle
after committing the pivot but before inserting the remaining blocks. It drags
the following sync cycle back to the snap mode.
The snap mode should only be used if the head state is missing and unrecoverable.
Enabling the snap mode for multiple times should be strictly prevented.
What's more, it brings a fix to pivot block management, that once the
pivot block is committed, the pivot block marker should be no longer advanced,
preventing the situation that the real pivot state is below the pivot marker.
blobFetcherQueueingHashes used len(announces), which is the number of
peers with pending announces, not the number of queued hashes. Count
unique hashes across peers instead.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
Commit 1f87331fb moved the known-transaction marking in
`sendPooledTransactionHashes` to after a successful send, so hashes are
not marked known to the peer if the announcement fails to go out.
The sparse blobpool change (d91b71fb3) reintroduced the original
track-before-send ordering when adding the eth/72 packet variant,
causing failed announcements to suppress future re-announcements of the
same hashes to that peer.
This restores the send-first ordering for both eth/71 and eth/72 packet
versions, and adds a regression test covering success and failure paths
on both protocol versions.
## Checklist
- [x] Restored mark-known-after-send for ETH71 and ETH72
- [x] Added `TestSendPooledTransactionHashes` covering success and
closed-pipe failure
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
This is the implementation of EIP-8070 Sparse Blobpool. It introduces
protocol version eth/72 which relays blob transaction cells instead of
full blobs.
The blobpool now store 'incomplete' transactions, where only some of
the cells are provided. The stored cell indexes are taken from the
custody bitmap, which is provided by the consensus layer in
forkchoiceUpdatedV4. This method will be called once Glamsterdam
activates, and the default custody is full custody, so for now there
is no change in the amount of stored cells for now.
The main entities added are the BlobBuffer and BlobFetcher, which work
together to track and fetch missing cells from connected peers. The
partial transactions become available for inclusion in blocks when
they are covered by enough peers that hold all cells.
This change also introduces engine_getBlobsV4, which allows for
cell-based responses (and partial blobs with only some of the cells).
We maintain backward compatibility with getBlobsV3 which expects full
blob responses by recovering the blob from available cells. Since this
process is resource-intensive, we proactively cache the conversion so
it is ready in time for getBlobsV3 calls. This mechanism will be
removed once support for getBlobsV4 is universal across all consensus
layer implementations.
devp2p tests for eth/72 are not part of this initial change. This is
to avoid breaking test success status for execution clients that do
not have eth/72 implemented yet. The tests will be added in a
subsequent change.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
- Move `msg.Discard` defer ahead of the max message size check in
`handleMessage`.
- Ensures oversized messages are released when the handler returns
early.
The peer dropper periodically disconnects random peers to create churn.
This was previously blind to peer quality.
This PR adds peer-score based peer protection, handling the
multi-dimensionality problem of peer scoring through the concept of
protected peer pools.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: healthykim <bsbs8645@snu.ac.kr>
This PR does a few things:
- reject `debug_setHead` if the target is even before the pivot block
(if non-nil)
- reject `debug_setHead` if in path mode, the target is not recoverable
- decouple the chain rewinding and state recovery in path mode and
recover the state in one shot
---------
Co-authored-by: jonny rhea <5555162+jrhea@users.noreply.github.com>
This PR addresses an issue in the eth71 `BlockAccessListsMsg` handler,
specifically:
- if the requested bal is not accessible in the server side, 0x80
(EmptyString) will be returned as the marker
- at the client side, old message definition
`rlp.RawList[RawBlockAccessList]` assumes all the elements are List
- the message with 0x80 (kind = string) won't be decoded correctly
- the peer will be disconnected
The message definition has been changed to `rlp.RawList[rlp.RawValue]`,
which is aligned with the one in SNAP/2 protocol.
This PR addresses the panic in tests. As the eventLoop is spun up when
the downloader was closed, the sub will be nil and make the panic
happens.
```
goroutine 421 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*DownloaderAPI).eventLoop(0xcb0e4d0)
/opt/actions-runner/_work/go-ethereum/go-ethereum/eth/downloader/api.go:91 +0x127
created by github.com/ethereum/go-ethereum/eth/downloader.NewDownloaderAPI in goroutine 352
/opt/actions-runner/_work/go-ethereum/go-ethereum/eth/downloader/api.go:50 +0xf2
```
This PR improves the block download used by snap sync. Specifically,
blocks and their associated data (receipts and canonical hash mappings)
are now written directly to the database without checking existence.
The current implementation could fail in cases where the block header
and body were already present (has.Block returns true), but the
corresponding canonical hash mapping was missing. One possible scenario
is when a newPayload event is processed without a subsequent
forkChoiceUpdate.
It is still unclear why Geth may re-enter snap sync after Engine API
events have been processed after the sync. Anyway, bypassing the
existence is a reasonable change.
What's more, in the downloader, the presence of canonical hash is also
considered for deciding the range of blocks to be downloaded.
Specifically:
- in the full sync, the block with header and body available but
canonical hash missing will be re-inserted;
- in the snap sync, the block with header, body and receipt available
but canonical hash missing will be re-inserted;
This PR drops support for v0 blob sidecar in blobpool. Since the
osaka fork activation time has passed, these code paths are
now unused. It is assumed that only v1 transactions exist in
the blobpool.
Adds `testing_commitBlockV1`. It is the write companion of `testing_buildBlockV1`:
it builds a block from the provided payload attributes and transactions on
top of the current canonical head, inserts it, and sets it as the new
head, returning the new head hash.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This PR fixes an issue where flat states are continuously persisted
during downloadState, while the sync journal is only persisted at the
end of Sync.
As a result, an unclean shutdown can leave the on-disk flat state ahead
of the journal markers. Some persisted entries may be stale (storage
slots that should have been deleted), and these dangling entries are not
detected or fixed by subsequent state downloads.
To address this, this PR introduces a cleanup step before state
downloading begins. It removes all state entries that are not covered by
the persisted journal markers.
This PR introduces a new condition that if the local node falls behind
too much and the required BAL for catching up is very likely to be
unavailable, the entire snap sync will be restarting from scratch.
As the defined BAL retention window is weak-subjective-period which is
calculated dynamically. A more conservative threshold is used (90K
blocks) for robustness.
Apart from that, the BAL catchup will be divided into several spans and
apply one by one. It's essential to prevent the potential out-of-memory
panic of placing the entire BAL set in memory.
This PR improves the slot reservation logic in the context of snap/2.
Geth has the mechanism to reserve roughly half the peer slots for peers
supporting the snap protocol if snap syncing is needed by local node.
With the context of snap/2, this mechanism should be changed that:
we reserve the slot for the "usable snap peer", not blindly for peer
with snap extension enabled (such as legacy snap/1, which can't serve
the snap/2).
This PR fixes an issue that when peers legitimately lack a requested
BAL, empty (0x80) is delivered and this BAL entry will be refetched
over and over again.
A `refused` tracker is added and catchUp will fail if this BAL is
unavailable against the entire peerset.
Implements https://eips.ethereum.org/EIPS/eip-8037
mainly done in order to judge the complexity of the EIP
and to act as a jumping off point, since the eip will likely
change.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR introduces a cache for GetBlobs request.
The main purpose of this PR is to reduce the getBlobs latency by reading and
decoding blobs from the pool in advance of the actual query. This is important
especially in the context of a sparse blobpool, since it may be necessary to
recover blobs from cells on a getBlobs request.
Previously, the Engine API read and decoded blobs from the pool on every call.
Now those calls check the cache and only fall back to the pool on a miss.
The cache has two modes:
- In topK mode (default), it wakes up periodically, picks the most profitable
pending blob transactions up to the current fork's maxBlobsPerBlock, and loads
their blobs. The selection logic is shared with the miner's block-building
logic. The selection size is derived from eip4844.MaxBlobsPerBlock at the
current head.
- When the CL calls HasBlobs, the cache switches to hasBlobs mode and tries to
pin the set it just reported as available. Cache updates (read, decode, and
optionally conversion in the future) run in background goroutines.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds snap/2 (EIP-8189), a block-access-list (BAL) based state sync, and
wires it to run side by side with snap/1. It's opt-in (for now) behind a
new --snap.v2 flag and chosen at startup.
https://eips.ethereum.org/EIPS/eip-8189
---------
Co-authored-by: Toni Wahrstätter <info@toniwahrstaetter.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This is a PR that removes all correctly flagged typos, in order to stop
an onslaught of slop PRs in its tracks. It should be followed by #34994
but the latter needs more configuration work and I want to limit the
stem of PRs right now.
The per-call SERVER span ended inside `handleCall()`, so the JSON-RPC
response write happened after the span closed. For large responses like
`engine_getBlobsV*`, that write time was missing from traces.
- Extend the SERVER span past `writeJSON`.
- For batches, add a top-level `jsonrpc.batch` SERVER span (with `rpc.batch.size`) covering the whole batch including `callBuffer.write`.
- Add `rpc.writeJSON` span around the non-batch response write.
- Add `rpc.writeJSONBatch` span around the batch response write.
- Add `rpc.httpWrite` span around the actual HTTP write, separating JSON encoding from network write.
- Add additional telemetry helpers.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is a prerequisite for landing snap v2, the BAL-healing snap sync
algorithm.
It duplicates much of the snap v1 skeleton, which is expected to be
deprecated once v2 is enabled. The code duplication is acceptable as a
short-term tradeoff, simplifying development and reducing integration
complexity.
fixes#32672
This is kind of a band aid solution since it fixes the issue by
bypassing the snap sync expectations of an empty db and attempting to
import the new payload if we're at block 1. The next FCU will set the
status to synced.
Will continue looking to better understand how the above issue arises
and find a more thorough solution.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
