beaconBackfiller.resume() already returns early when partialSyncComplete
is set, so in normal CL-driven operation the downloader never reaches
synchronise after the initial partial-state sync finishes. Add the same
guard at the synchronise entry point as defense in depth: any future
caller of synchronise (tests, other wiring) inherits the invariant
that partial-state nodes do not run full downloader cycles after
initial sync, even if the resume path is bypassed.
The check is cheap (one atomic.Load) and sits on the cold path, so the
impact on normal full-sync users is nil.
d.partialSyncComplete is consulted by beaconBackfiller.resume() to skip
redundant downloader cycles after the initial partial-state sync has
finished. It was an in-memory atomic.Bool, so every process restart
reset it to false, and the next forkchoiceUpdated from the CL would
re-enter the sync loop.
Persist the flag in leveldb via a new PartialSyncComplete marker:
- Add ReadPartialSyncComplete / WritePartialSyncComplete /
DeletePartialSyncComplete accessors in core/rawdb/accessors_chain.go
backed by a single-byte value under the PartialSyncComplete key.
- Write the marker in the downloader right after AdvancePartialHead
succeeds (same spot we flip the in-memory flag).
- Rehydrate the in-memory flag from leveldb in Downloader.New() so a
freshly-started process with a completed partial-state sync keeps
the resume short-circuit active from the first beacon forkchoice.
Without this, the restart invariant relied on HasState(header.Root)
accidentally returning false to reroute the downloader back to
SnapSync; with this the resume guard is the primary protection
regardless of how header-root convergence evolves.
The second state sync (pivot→HEAD) determines its target using
CurrentSnapBlock(), which may equal CurrentBlock() if no afterP blocks
were processed before the queue drained. This is a timing-dependent
race: with rate-limited pivot advances, the pivot ends up close to
the CL head, so the final batch may contain zero afterP blocks,
causing CurrentSnapBlock == CurrentBlock. The check
`snapHead.Hash() != currentHead.Hash()` then fails and the second
sync is skipped entirely. Without the second sync, disableSnap()
is never called, ConfigSyncMode() stays SnapSync, and ALL subsequent
newPayload calls are delayed forever.
Fix: use the skeleton head (beacon chain tip) as the second sync
target instead of CurrentSnapBlock(). The skeleton head is always
available and correctly reflects the CL's latest finalized target,
independent of queue draining timing.
Also removes the fragile "snap head too old" and "snap head too far
behind" guards which could abort the second sync prematurely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix several interacting issues that prevented partial state nodes from
syncing and following the chain on bal-devnet-2:
1. Stale pivot deadlock: Replace unconditional pivot suppression with
rate-limited advances (2-minute cooldown). This prevents the restart
loop bug while allowing recovery when the initial pivot is too stale
for peers to serve.
2. Storage root resolution: Add snap-based resolver that queries peers
for untracked contracts' storage roots during BAL processing. This
lets the computed state root converge toward the header root.
3. SetCanonical for partial state: When the computed root differs from
the header root (expected when untracked contracts have unresolved
storage roots), check HasState(partialState.Root()) instead of only
HasState(block.Root()). Guard against zero root during snap sync.
4. Canonical hash backfill: AdvancePartialHead now writes canonical
hashes for all blocks between the pivot and snap head, fixing the
"final block not in canonical chain" error caused by
InsertReceiptChain skipping blocks whose bodies already exist.
5. Gap block processing: After snap sync completes, process accumulated
blocks between the sync head and chain tip using their persisted BALs
before entering steady-state chain following.
6. Computed root chaining: Use partialState.Root() (actual computed root)
as parentRoot for subsequent blocks, not the header root. This ensures
correct trie chaining when computed != header root.
Tested end-to-end on bal-devnet-2: snap sync completes, gap blocks
processed, canonical head advances at chain tip (~1 block/12s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Freeze the pivot header for partial state nodes to ensure stable state
sync progress:
- Suppress pivot movement in fetchHeaders() (beaconsync.go)
- Suppress pivot movement in processSnapSyncContent() (downloader.go)
- Reuse existing pivot across sync cycle restarts in syncToHead()
After initial snap sync completes, bridge the gap from pivot to HEAD:
- Import post-pivot blocks with receipts (no execution needed since
untracked contracts have empty storage tries)
- Run second state sync to download HEAD state root
- Add AdvancePartialHead to update currentBlock without re-execution
Guard the backfiller for partial state mode:
- suspend() skips Cancel() during active snap sync to prevent
constant cancel/restart cycles from beacon head updates
- resume() skips new sync cycles after partial sync completes
Add chain retention for partial state mode: only the most recent N blocks
(default 1024) retain bodies and receipts. During sync, older blocks are
skipped entirely. After sync, the freezer enforces a rolling window.
Add engine API support for Block Access Lists (EIP-7928): NewPayloadV5
accepts BAL data alongside execution payloads, enabling partial state
nodes to receive per-block storage access information from the CL.
Fix beacon backfilling failure caused by dynamic chain cutoff not
clearing the cutoff hash (which remained at the genesis hash).
Add partial state awareness to eth_call/eth_estimateGas to return clear
errors when accessing untracked contract storage.
Passes the partial statefulness filter from Ethereum backend through
the handler config and into the downloader. The filter is then passed
to the snap syncer to enable selective storage/code syncing.
Updates downloader tests to accommodate the new filter parameter.
Part of partial statefulness Phase 2.
In this PR, we add support for protocol version eth/70, defined by EIP-7975.
Overall changes:
- Each response is buffered in the peer’s receipt buffer when the
`lastBlockIncomplete` field is true.
- Continued request uses the same request id of its original
request(`RequestPartialReceipts`).
- Partial responses are verified in `validateLastBlockReceipt`.
- Even if all receipts for partial blocks of the request are collected,
those partial results are not sinked to the downloader, to avoid
complexity. This assumes that partial response and buffering occur only
in exceptional cases.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR contains two changes:
Firstly, the finalized header will be resolved from local chain if it's
not recently announced via the `engine_newPayload`.
What's more importantly is, in the downloader, originally there are two
code paths to push forward the pivot point block, one in the beacon
header fetcher (`fetchHeaders`), and another one is in the snap content
processer (`processSnapSyncContent`).
Usually if there are new blocks and local pivot block becomes stale, it
will firstly be detected by the `fetchHeaders`. `processSnapSyncContent`
is fully driven by the beacon headers and will only detect the stale pivot
block after synchronizing the corresponding chain segment. I think the
detection here is redundant and useless.
With this, we are dropping support for protocol version eth/68. The only supported
version is eth/69 now. The p2p receipt encoding logic can be simplified a lot, and
processing of receipts during sync gets a little faster because we now transform
the network encoding into the database encoding directly, without decoding the
receipts first.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the p2p protocol handlers to delay message decoding. It's
the first part of a larger change that will delay decoding all the way
through message processing. For responses, we delay the decoding until
it is confirmed that the response matches an active request and does not
exceed its limits.
In order to make this work, all messages have been changed to use
rlp.RawList instead of a slice of the decoded item type. For block
bodies specifically, the decoding has been delayed all the way until
after verification of the response hash.
The role of p2p/tracker.Tracker changes significantly in this PR. The
Tracker's original purpose was to maintain metrics about requests and
responses in the peer-to-peer protocols. Each protocol maintained a
single global Tracker instance. As of this change, the Tracker is now
always active (regardless of metrics collection), and there is a
separate instance of it for each peer. Whenever a response arrives, it
is first verified that a request exists for it in the tracker. The
tracker is also the place where limits are kept.
In this PR, two things have been fixed:
---
(a) truncate the stale beacon headers with latest snap block
Originally, b.filled is used as the indicator for deleting stale beacon headers.
This field is set only after synchronization has been scheduled, under the
assumption that the skeleton chain is already linked to the local chain.
However, the local chain can be mutated via `debug_setHead`, which may
cause `b.filled` outdated. For instance, `b.filled` refers to the last head snap block
in the last sync cycle while after `debug_setHead`, the head snap block has been
rewounded to 1.
As a result, Geth can enter an unintended loop: it repeatedly downloads
the missing beacon headers for the skeleton chain and attempts to schedule the
actual synchronization, but in the final step, all recently fetched headers are removed
by `cleanStales` due to the stale `b.filled` value.
This issue is addressed by always using the latest snap block as the indicator,
without relying on any cached value. However, note that before the skeleton
chain is linked to the local chain, the latest snap block will always be below
skeleton.tail, and this condition should not be treated as an error.
---
(b) merge the subchains once the skeleton chain links to local chain
Once the skeleton chain links with local one, it will try to schedule the
synchronization by fetching the missing blocks and import them then.
It's possible the last subchain already overwrites the previous subchain and
results in having two subchains leftover. As a result, an error log will printed
https://github.com/ethereum/go-ethereum/blob/master/eth/downloader/skeleton.go#L1074
This moves the tracking of the current syncmode into the downloader, fixing an
issue where the syncmode being requested through the engine API could go
out-of-sync with the actual mode being performed by downloader.
Fixes#32629
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
The GetHeader function was incorrectly returning an error when
encountering nil peers in the peers list, which contradicted the comment
"keep retrying if none are yet available".
Changed the logic to skip nil peers with 'continue' instead of returning
an error, allowing the function to properly iterate through all
available peers and attempt to retrieve the target header from each valid peer.
This ensures the function behaves as intended - trying all available
peers before giving up, rather than failing on the first nil peer encountered.
The previous comment stated that every 3rd block has a tx and every 5th
has an uncle.
The implementation actually adds one transaction to every second block
and does not add uncles.
Updated the comment to reflect the real behavior to avoid confusion when
reading tests.
This pull request fixes an issue in disabling direct-ancient mode in
snap sync.
Specifically, if `origin >= frozen && origin != 0`, it implies a part of
chain data has been written into the key-value store, all the following
writes into ancient store scheduled by downloader will be rejected
with error
`ERROR[07-10|03:46:57.924] Error importing chain data to ancients
err="can't add block 1166 hash: the append operation is out-order: have
1166 want 0"`.
This issue is detected by the https://github.com/ethpandaops/kurtosis-sync-test,
which initiates the first snap sync cycle without the finalized header and
implicitly disables the direct-ancient mode. A few seconds later the second
snap sync cycle is initiated with the finalized information and direct-ancient mode
is enabled incorrectly.
Small update for logs when syncing with blsync. Downgrades the "latest
filled block is not available" to warn.
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
This pull request tracks the state indexing progress in eth_syncing
RPC response, i.e. we will return non-null syncing status until indexing
has finished.
If Geth is engaged in a long-run block synchronization, such as a full
syncing over a large number of blocks, invoking `debug_setHead` will
cause `downloader.Cancel` to wait for all fetchers to stop first.
This can be time-consuming, particularly for the block processing
thread.
To address this, we manually call `blockchain.StopInsert` to interrupt
the blocking processing thread and allow it to exit immediately, and
after that call `blockchain.ResumeInsert` to resume the block
downloading process.
Additionally, we add a sanity check for the input block number of
`debug_setHead` to ensure its validity.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
In this pull request, the original `CacheConfig` has been renamed to `BlockChainConfig`.
Over time, more fields have been added to `CacheConfig` to support
blockchain configuration. Such as `ChainHistoryMode`, which clearly extends
beyond just caching concerns.
Additionally, adding new parameters to the blockchain constructor has
become increasingly complicated, since it’s initialized across multiple
places in the codebase. A natural solution is to consolidate these arguments
into a dedicated configuration struct.
As a result, the existing `CacheConfig` has been redefined as `BlockChainConfig`.
Some parameters, such as `VmConfig`, `TxLookupLimit`, and `ChainOverrides`
have been moved into `BlockChainConfig`. Besides, a few fields in `BlockChainConfig`
were renamed, specifically:
- `TrieCleanNoPrefetch` -> `NoPrefetch`
- `TrieDirtyDisabled` -> `ArchiveMode`
Notably, this change won't affect the command line flags or the toml
configuration file. It's just an internal refactoring and fully backward-compatible.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a regression introduced by #29158 where receipts of empty blocks
were stored into the database as an empty byte array, instead of an RLP empty list.
Fixes#31938
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This implements a backing store for chain history based on era1 files.
The new store is integrated with the freezer. Queries for blocks and receipts
below the current freezer tail are handled by the era store.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR implements eth/69. This protocol version drops the bloom filter
from receipts messages, reducing the amount of data needed for a sync
by ~530GB (2.3B txs * 256 byte) uncompressed. Compressed this will
be reduced to ~100GB
The new version also changes the Status message and introduces the
BlockRangeUpdate message to relay information about the available history
range.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This pull request introduces new sync logic for pruning mode. The downloader will now skip
insertion of block bodies and receipts before the configured history cutoff point.
Originally, in snap sync, the header chain and other components (bodies and receipts) were
inserted separately. However, in Proof-of-Stake, this separation is unnecessary since the
sync target is already verified by the CL.
To simplify the process, this pull request modifies `InsertReceiptChain` to insert headers
along with block bodies and receipts together. Besides, `InsertReceiptChain` doesn't have
the notion of reorg, as the common ancestor is always be found before the sync and extra
side chain is truncated at the beginning if they fall in the ancient store. The stale
canonical chain flags will always be rewritten by the new chain. Explicit reorg logic is
no longer required in `InsertReceiptChain`.
The total difficulty is the sum of all block difficulties from genesis
to a certain block. This value was used in PoW for deciding which chain
is heavier, and thus which chain to select. Since PoS has a different
fork selection algorithm, all blocks since the merge have a difficulty
of 0, and all total difficulties are the same for the past 2 years.
Whilst the TDs are mostly useless nowadays, there was never really a
reason to mess around removing them since they are so tiny. This
reasoning changes when we go down the path of pruned chain history. In
order to reconstruct any TD, we **must** retrieve all the headers from
chain head to genesis and then iterate all the difficulties to compute
the TD.
In a world where we completely prune past chain segments (bodies,
receipts, headers), it is not possible to reconstruct the TD at all. In
a world where we still keep chain headers and prune only the rest,
reconstructing it possible as long as we process (or download) the chain
forward from genesis, but trying to snap sync the head first and
backfill later hits the same issue, the TD becomes impossible to
calculate until genesis is backfilled.
All in all, the TD is a messy out-of-state, out-of-consensus computed
field that is overall useless nowadays, but code relying on it forces
the client into certain modes of operation and prevents other modes or
other optimizations. This PR completely nukes out the TD from the node.
It doesn't compute it, it doesn't operate on it, it's as if it didn't
even exist.
Caveats:
- Whenever we have APIs that return TD (devp2p handshake, tracer, etc.)
we return a TD of 0.
- For era files, we recompute the TD during export time (fairly quick)
to retain the format content.
- It is not possible to "verify" the merge point (i.e. with TD gone, TTD
is useless). Since we're not verifying PoW any more, just blindly trust
it, not verifying but blindly trusting the many year old merge point
seems just the same trust model.
- Our tests still need to be able to generate pre and post merge blocks,
so they need a new way to split the merge without TTD. The PR introduces
a settable ttdBlock field on the consensus object which is used by tests
as the block where originally the TTD happened. This is not needed for
live nodes, we never want to generate old blocks.
- One merge transition consensus test was disabled. With a
non-operational TD, testing how the client reacts to TTD is useless, it
cannot react.
Questions:
- Should we also drop total terminal difficulty from the genesis json?
It's a number we cannot react on any more, so maybe it would be cleaner
to get rid of even more concepts.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR modifies how the metrics library handles `Enabled`: previously,
the package `init` decided whether to serve real metrics or just
dummy-types.
This has several drawbacks:
- During pkg init, we need to determine whether metrics are enabled or
not. So we first hacked in a check if certain geth-specific
commandline-flags were enabled. Then we added a similar check for
geth-env-vars. Then we almost added a very elaborate check for
toml-config-file, plus toml parsing.
- Using "real" types and dummy types interchangeably means that
everything is hidden behind interfaces. This has a performance penalty,
and also it just adds a lot of code.
This PR removes the interface stuff, uses concrete types, and allows for
the setting of Enabled to happen later. It is still assumed that
`metrics.Enable()` is invoked early on.
The somewhat 'heavy' operations, such as ticking meters and exp-decay,
now checks the enable-flag to prevent resource leak.
The change may be large, but it's mostly pretty trivial, and from the
last time I gutted the metrics, I ensured that we have fairly good test
coverage.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Lots of packages depend on eth/downloader just for the SyncMode type.
Since we have a dedicated package for eth protocol configuration, it
makes more sense to define SyncMode there, turning eth/downloader into
more of a leaf package.
Changelog: https://golangci-lint.run/product/changelog/#1610
Removes `exportloopref` (no longer needed), replaces it with
`copyloopvar` which is basically the opposite.
Also adds:
- `durationcheck`
- `gocheckcompilerdirectives`
- `reassign`
- `mirror`
- `tenv`
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This implements recent changes to EIP-7685, EIP-6110, and
execution-apis.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: Shude Li <islishude@gmail.com>
This PR changes how sidechains are handled.
Before the merge, it was possible to import a chain with lower td and not set it as canonical. After the merge, we expect every chain that we get via InsertChain to be canonical. Non-canonical blocks can still be inserted
with InsertBlockWIthoutSetHead.
If during the InsertChain, the existing chain is not canonical anymore, we mark it as a sidechain and send the SideChainEvents normally.
Fixes a slight miscalculation in the downloader queue, which was not accurately taking block withdrawals into account when calculating the size of the items in the queue
This PR fixes an issue in the setMode method of beaconBackfiller where the
log message was not displaying the previous mode correctly. The log message
now shows both the old and new sync modes.
This pull request fixes the flay test TestSkeletonSyncRetrievals. In this test, we first
trigger a sync cycle and wait for it to meet certain expectations. We then inject a new
head and potentially also a new peer, then perform another final sync. The test now
performs the newPeer addition before launching the final sync, and waits a bit for that
peer to get registered. This fixes the logic race that made the test fail sometimes.
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
