beaconBackfiller.resume() already returns early when partialSyncComplete
is set, so in normal CL-driven operation the downloader never reaches
synchronise after the initial partial-state sync finishes. Add the same
guard at the synchronise entry point as defense in depth: any future
caller of synchronise (tests, other wiring) inherits the invariant
that partial-state nodes do not run full downloader cycles after
initial sync, even if the resume path is bypassed.
The check is cheap (one atomic.Load) and sits on the cold path, so the
impact on normal full-sync users is nil.
d.partialSyncComplete is consulted by beaconBackfiller.resume() to skip
redundant downloader cycles after the initial partial-state sync has
finished. It was an in-memory atomic.Bool, so every process restart
reset it to false, and the next forkchoiceUpdated from the CL would
re-enter the sync loop.
Persist the flag in leveldb via a new PartialSyncComplete marker:
- Add ReadPartialSyncComplete / WritePartialSyncComplete /
DeletePartialSyncComplete accessors in core/rawdb/accessors_chain.go
backed by a single-byte value under the PartialSyncComplete key.
- Write the marker in the downloader right after AdvancePartialHead
succeeds (same spot we flip the in-memory flag).
- Rehydrate the in-memory flag from leveldb in Downloader.New() so a
freshly-started process with a completed partial-state sync keeps
the resume short-circuit active from the first beacon forkchoice.
Without this, the restart invariant relied on HasState(header.Root)
accidentally returning false to reroute the downloader back to
SnapSync; with this the resume guard is the primary protection
regardless of how header-root convergence evolves.
The stateless block check in forkchoiceUpdated was calling BeaconSync()
on every FCU (~12 seconds) during active snap sync, restarting the
entire sync cycle each time. This prevented state download from ever
completing.
Guard the check with ConfigSyncMode: during active snap sync, the
downloader is already working, so just return STATUS_SYNCING without
restarting. Only trigger BeaconSync for stateless blocks after snap
sync has completed (FullSync mode).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The second state sync (pivot→HEAD) determines its target using
CurrentSnapBlock(), which may equal CurrentBlock() if no afterP blocks
were processed before the queue drained. This is a timing-dependent
race: with rate-limited pivot advances, the pivot ends up close to
the CL head, so the final batch may contain zero afterP blocks,
causing CurrentSnapBlock == CurrentBlock. The check
`snapHead.Hash() != currentHead.Hash()` then fails and the second
sync is skipped entirely. Without the second sync, disableSnap()
is never called, ConfigSyncMode() stays SnapSync, and ALL subsequent
newPayload calls are delayed forever.
Fix: use the skeleton head (beacon chain tip) as the second sync
target instead of CurrentSnapBlock(). The skeleton head is always
available and correctly reflects the CL's latest finalized target,
independent of queue draining timing.
Also removes the fragile "snap head too old" and "snap head too far
behind" guards which could abort the second sync prematurely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Live testing on bal-devnet-2 confirmed that computed roots DO diverge from
header roots. Block 75315 computed root 0xe909c7.. vs header root
0x9acbbe.. — untracked contracts' storage roots in the local trie are from
snap sync time and differ from the actual current roots, even when the
storage root resolver successfully queries peers.
This means subsequent blocks must chain off the computed root (via
partialState.Root()), not the header root (via parent.Root()). Restore
the stateRoot field using atomic.Pointer[common.Hash] instead of the
previous sync.RWMutex for lock-free concurrent access.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apply review fixes: BAL iterator start (Fix 2), fatal root mismatch when
all storage resolved (Fix 3), WriteBlockWithoutState error handling (Fix 4),
contract filter construction order (Fix 5), canonical hash backfill (Fix 6),
underflow guard in gap processing (Fix 8), O(n²) prepend fix (Fix 9),
ReadBALHistory corruption detection (Fix 11), incomplete resolution error
(Fix 13), RLP encode panic (Fix 14), gap processing log level (Fix 16),
TriggerPartialResync message (Fix 18), and comment accuracy fixes.
Remove the stateRoot field and sync.RWMutex from PartialState entirely.
Since partial state maintains the full account trie, the computed root
always matches the header root (assuming storage root resolution succeeds).
ProcessBlockWithBAL now derives parent root from parent.Root() directly,
matching how full nodes derive state root from currentBlock headers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move partial state CLI flags into their own "PARTIAL STATE" help category
(matching BEACON CHAIN, DEVELOPER CHAIN patterns), improve Usage strings
with examples and constraint descriptions, expand PartialStateConfig doc
comments to explain EIP-7928 implications, and raise BAL retention minimum
from 64 to 256 (required by BLOCKHASH opcode).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix several interacting issues that prevented partial state nodes from
syncing and following the chain on bal-devnet-2:
1. Stale pivot deadlock: Replace unconditional pivot suppression with
rate-limited advances (2-minute cooldown). This prevents the restart
loop bug while allowing recovery when the initial pivot is too stale
for peers to serve.
2. Storage root resolution: Add snap-based resolver that queries peers
for untracked contracts' storage roots during BAL processing. This
lets the computed state root converge toward the header root.
3. SetCanonical for partial state: When the computed root differs from
the header root (expected when untracked contracts have unresolved
storage roots), check HasState(partialState.Root()) instead of only
HasState(block.Root()). Guard against zero root during snap sync.
4. Canonical hash backfill: AdvancePartialHead now writes canonical
hashes for all blocks between the pivot and snap head, fixing the
"final block not in canonical chain" error caused by
InsertReceiptChain skipping blocks whose bodies already exist.
5. Gap block processing: After snap sync completes, process accumulated
blocks between the sync head and chain tip using their persisted BALs
before entering steady-state chain following.
6. Computed root chaining: Use partialState.Root() (actual computed root)
as parentRoot for subsequent blocks, not the header root. This ensures
correct trie chaining when computed != header root.
Tested end-to-end on bal-devnet-2: snap sync completes, gap blocks
processed, canonical head advances at chain tip (~1 block/12s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix the post-sync deadlock where blocks validated via BAL in newPayload
were never written to the database, causing ForkchoiceUpdated to fail
finding them and triggering infinite sync cycles.
Changes:
- Export WriteBlockWithoutState and call it after ProcessBlockWithBAL
in newPayload, so FCU can find blocks via GetBlockByHash
- Guard SetCanonical against recoverAncestors for partial state nodes
(they can't re-execute blocks, only apply BAL diffs)
- Auto-disable log indexing when partial state is enabled (no receipts)
- Fix BAL type field accesses to match upstream bal-devnet-2 types
(StorageChanges, CodeChanges, BalanceChanges, Validate signature)
- Update newPayload signature (BAL now comes from ExecutableData params)
- Add partial sync scripts and documentation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Geth has two independent snapshot tiers, each with its own disable
mechanism:
1. In-memory snapshot cache: controlled by SnapshotLimit (derived from
ethconfig.SnapshotCache). Setting SnapshotCache=0 disables it.
2. On-disk snapshot generator: a background goroutine in pathdb that
iterates the entire state trie to build flat key-value snapshots.
Controlled by pathdb.Config.SnapshotNoBuild.
The partial state configuration (cmd/utils/flags.go) already set
SnapshotCache=0 to disable the in-memory cache. However, SnapshotNoBuild
was never set, so pathdb.Enable() — called after snap sync completes —
still launched the background generator goroutine.
This generator immediately hits missing storage tries for untracked
contracts (whose storage was intentionally skipped during partial sync),
logs "Trie missing, snapshotting paused", and blocks forever on its
abort channel — a permanent goroutine leak with no recovery path.
Additionally, BlockChainConfig.SnapshotNoBuild was never propagated to
pathdb.Config.SnapshotNoBuild in the triedbConfig() conversion. The
field only reached the hash-scheme snapshot module (core/blockchain.go
setupSnapshot), which is already skipped for path-scheme databases. This
plumbing gap meant pathdb.Config.SnapshotNoBuild was never set in
production code — only in tests.
Fix both issues:
- Set SnapshotNoBuild=true when partial state is enabled
- Propagate BlockChainConfig.SnapshotNoBuild into pathdb.Config
The statelessPeers map permanently blacklists peers that return empty
responses for the entire Sync() cycle. In partial state mode, the faster
account advancement (due to skipping storage/code for non-tracked
contracts) creates bursty request patterns that can trigger transient
empty responses. Combined with the permanent blacklist, this causes a
cascade where all peers get banned and sync stalls permanently.
Replace the permanent map[string]struct{} with map[string]time.Time to
track when each peer was marked. For partial state mode, peers are given
a 30-second cooldown instead of permanent banishment. After the cooldown
expires, the peer is eligible for task assignment again. Full sync mode
behavior is unchanged (permanent blacklist preserved).
Freeze the pivot header for partial state nodes to ensure stable state
sync progress:
- Suppress pivot movement in fetchHeaders() (beaconsync.go)
- Suppress pivot movement in processSnapSyncContent() (downloader.go)
- Reuse existing pivot across sync cycle restarts in syncToHead()
After initial snap sync completes, bridge the gap from pivot to HEAD:
- Import post-pivot blocks with receipts (no execution needed since
untracked contracts have empty storage tries)
- Run second state sync to download HEAD state root
- Add AdvancePartialHead to update currentBlock without re-execution
Guard the backfiller for partial state mode:
- suspend() skips Cancel() during active snap sync to prevent
constant cancel/restart cycles from beacon head updates
- resume() skips new sync cycles after partial sync completes
Refactor partial state filter from DB skip markers to direct filter
checks via shouldSyncStorage()/shouldSyncCode(), avoiding stale marker
issues across sync cycles.
Additional fixes:
- Skip WriteAccountSnapshot/WriteStorageSnapshot in partial mode
(forwardAccountTask, processStorageResponse, onHealState)
- Guard against negative ETA in reportSyncProgress when sync restarts
with persisted progress counters
- Add break after forwardAccountTask in cleanStorageTasks to prevent
nil pointer when task.res is cleared
- Add diagnostic log in assignAccountTasks when no idle peers available
Partial state nodes don't need snapshots since account data is read
directly from the trie (which is small enough for fast lookups) and
BAL-based block processing never uses snapshots.
- Set SnapshotCache to 0 when partial state is enabled (flags.go)
- Allow snap sync without snapshots for partial state mode (handler.go)
- Add nil-check for Snapshots() in snap request handlers to prevent
panics when serving HashScheme peers (snap/handler.go)
LoadPartialStateContracts() was only called from Validate() which was
never invoked, causing the contracts file to never be loaded. Call it
directly during Ethereum node initialization when partial state is
enabled.
Add chain retention for partial state mode: only the most recent N blocks
(default 1024) retain bodies and receipts. During sync, older blocks are
skipped entirely. After sync, the freezer enforces a rolling window.
Add engine API support for Block Access Lists (EIP-7928): NewPayloadV5
accepts BAL data alongside execution payloads, enabling partial state
nodes to receive per-block storage access information from the CL.
Fix beacon backfilling failure caused by dynamic chain cutoff not
clearing the cutoff hash (which remained at the genesis hash).
Add partial state awareness to eth_call/eth_estimateGas to return clear
errors when accessing untracked contract storage.
Add partial state mode support to the RPC API. In partial state mode:
- Account queries (balance, nonce, account proofs) work for ALL accounts
- Storage/code queries only work for tracked contracts
- Clear error codes help clients understand limitations
Changes:
- New error types: StorageNotTrackedError (-32001), CodeNotTrackedError (-32002)
- Backend interface: PartialStateEnabled(), IsContractTracked()
- Modified RPCs: GetStorageAt, GetCode, GetProof check tracked status
- 7 new tests verify correct behavior for tracked/untracked contracts
Comprehensive integration tests using mock peers that verify partial
sync behavior end-to-end:
- TestPartialSyncIntegration: Full sync with 20 accounts, 2 tracked
- TestPartialSyncAllAccounts: Verifies complete account trie synced
- TestPartialSyncSkipMarkers: Verifies skip markers written correctly
- TestPartialSyncNoStorageForUntracked: No storage for skipped accounts
- TestPartialSyncRequestCount: Diagnostic showing request filtering
- TestPartialSyncVsFullSync: Compares full vs partial, shows 83% reduction
Level 2 validation was also performed using a two-node local devnet
(full node + partial node) to verify database size reduction and
correct RPC responses. The mock peer tests provide equivalent coverage
with faster execution and CI compatibility.
Part of partial statefulness Phase 2.
Adds partial sync mode to the snap syncer that filters which contracts
have their storage and bytecode synced based on the configured filter.
Key changes:
- Syncer accepts optional ContractFilter for partial mode
- Skip markers (SnapSkipped prefix) track intentionally skipped accounts
- processAccountResponse checks filter before requesting storage/code
- Healing phase uses NewPartialStateSync to respect skip markers
- Helper functions for skip marker persistence (mark/check/delete)
When partial sync is active, only tracked contracts have their storage
synced, reducing sync size from ~1TB+ to ~30-40GB while maintaining
a complete account trie for balance queries.
Part of partial statefulness Phase 2.
Passes the partial statefulness filter from Ethereum backend through
the handler config and into the downloader. The filter is then passed
to the snap syncer to enable selective storage/code syncing.
Updates downloader tests to accommodate the new filter parameter.
Part of partial statefulness Phase 2.
Implements EIP-7928 BAL-based partial statefulness infrastructure:
- Add PartialStateConfig to eth/ethconfig with CLI flags
- Add ContractFilter interface in core/state/partial/
- Add BAL history database accessors in core/rawdb/
- Add PartialState and BALHistory managers
This enables nodes to track only configured contracts' storage
while maintaining full account trie integrity.
Wire up slotnum for `testing_buildBlockV1` for `bal-devnet-3` branch
We are experimenting testing block building through hive via EELS (PR
[here](https://github.com/ethereum/execution-specs/pull/2679)). This is
the only change needed to test against `bal-devnet-3` - missing slotnum.
* add method on StateReaderTracker to clear the accumulated reads
* don't factor the BAL size into the payload size during construction in the miner
* simplify miner code for constructing payloads-with-BALs via the use of aformentioned StateReaderTracker clear method
* clean up the configuration of the BAL execution mode based on the preset flag specified
This is a breaking change in the opcode (structLog) tracer. Several fields
will have a slight formatting difference to conform to the newly established
spec at: https://github.com/ethereum/execution-apis/pull/762. The differences
include:
- `memory`: words will have the 0x prefix. Also last word of memory will be padded to 32-bytes.
- `storage`: keys and values will have the 0x prefix.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
In this PR, we add support for protocol version eth/70, defined by EIP-7975.
Overall changes:
- Each response is buffered in the peer’s receipt buffer when the
`lastBlockIncomplete` field is true.
- Continued request uses the same request id of its original
request(`RequestPartialReceipts`).
- Partial responses are verified in `validateLastBlockReceipt`.
- Even if all receipts for partial blocks of the request are collected,
those partial results are not sinked to the downloader, to avoid
complexity. This assumes that partial response and buffering occur only
in exceptional cases.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR introduces a new type HistoryPolicy which captures user intent
as opposed to pruning point stored in the blockchain which persists the
actual tail of data in the database.
It is in preparation for the rolling history expiry feature.
It comes with a semantic change: if database was pruned and geth is
running without a history mode flag (or explicit keep all flag) geth
will emit a warning but continue running as opposed to stopping the
world.
`TestSubscribePendingTxHashes` hangs indefinitely because pending tx
events are permanently missed due to a race condition in
`NewPendingTransactions` (and `NewHeads`). Both handlers called their
event subscription functions (`SubscribePendingTxs`,
`SubscribeNewHeads`) inside goroutines, so the RPC handler returned the
subscription ID to the client before the filter was installed in the
event loop. When the client then sent a transaction, the event fired but
no filter existed to catch it — the event was silently lost.
- Move `SubscribePendingTxs` and `SubscribeNewHeads` calls out of
goroutines so filters are installed synchronously before the RPC
response is sent, matching the pattern already used by `Logs` and
`TransactionReceipts`
<!-- START COPILOT CODING AGENT TIPS -->
---
💬 We'd love your input! Share your thoughts on Copilot coding agent in
our [2 minute survey](https://gh.io/copilot-coding-agent-survey).
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: s1na <1591639+s1na@users.noreply.github.com>
This PR contains two changes:
Firstly, the finalized header will be resolved from local chain if it's
not recently announced via the `engine_newPayload`.
What's more importantly is, in the downloader, originally there are two
code paths to push forward the pivot point block, one in the beacon
header fetcher (`fetchHeaders`), and another one is in the snap content
processer (`processSnapSyncContent`).
Usually if there are new blocks and local pivot block becomes stale, it
will firstly be detected by the `fetchHeaders`. `processSnapSyncContent`
is fully driven by the beacon headers and will only detect the stale pivot
block after synchronizing the corresponding chain segment. I think the
detection here is redundant and useless.
This PR fixes a regression introduced in https://github.com/ethereum/go-ethereum/pull/33836/changes
Before PR 33836, running mainnet would automatically bump the cache size
to 4GB and trigger a cache re-calculation, specifically setting the key-value
database cache to 2GB.
After PR 33836, this logic was removed, and the cache value is no longer
recomputed if no command line flags are specified. The default key-value
database cache is 512MB.
This PR bumps the default key-value database cache size alongside the
default cache size for other components (such as snapshot) accordingly.
I observed failing tests in Hive `engine-withdrawals`:
-
https://hive.ethpandaops.io/#/test/generic/1772351960-ad3e3e460605c670efe1b4f4178eb422?testnumber=146
-
https://hive.ethpandaops.io/#/test/generic/1772351960-ad3e3e460605c670efe1b4f4178eb422?testnumber=147
```shell
DEBUG (Withdrawals Fork on Block 2): NextPayloadID before getPayloadV2:
id=0x01487547e54e8abe version=1
>> engine_getPayloadV2("0x01487547e54e8abe")
<< error: {"code":-38005,"message":"Unsupported fork"}
FAIL: Expected no error on EngineGetPayloadV2: error=Unsupported fork
```
The same failure pattern occurred for Block 3.
Per Shanghai engine_getPayloadV2 spec, pre-Shanghai payloads should be
accepted via V2 and returned as ExecutionPayloadV1:
- executionPayload: ExecutionPayloadV1 | ExecutionPayloadV2
- ExecutionPayloadV1 MUST be returned if payload timestamp < Shanghai
timestamp
- ExecutionPayloadV2 MUST be returned if payload timestamp >= Shanghai
timestamp
Reference:
-
https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md#engine_getpayloadv2
Current implementation only allows GetPayloadV2 on the Shanghai fork
window (`[]forks.Fork{forks.Shanghai}`), so pre-Shanghai payloads are
rejected with Unsupported fork.
If my interpretation of the spec is incorrect, please let me know and I
can adjust accordingly.
---------
Co-authored-by: muzry.li <muzry.li1@ambergroup.io>
Eth currently has a flaky test, related to the tx fetcher.
The issue seems to happen when Unsubscribe is called while sub is nil.
It seems that chain.Stop() may be invoked before the loop starts in some
tests, but the exact cause is still under investigation through repeated
runs. I think this change will at least prevent the error.
With this, we are dropping support for protocol version eth/68. The only supported
version is eth/69 now. The p2p receipt encoding logic can be simplified a lot, and
processing of receipts during sync gets a little faster because we now transform
the network encoding into the database encoding directly, without decoding the
receipts first.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
fix the flaky test found in
https://ci.appveyor.com/project/ethereum/go-ethereum/builds/53601688/job/af5ccvufpm9usq39
1. increase the timeout from 3+1s to 15s, and use timer instead of
sleep(in the CI env, it may need more time to sync the 1024 blocks)
2. add `synced.Load()` to ensure the full async chain is finished
Signed-off-by: Delweng <delweng@gmail.com>
Previously, handshake timeouts were recorded as generic peer errors
instead of timeout errors. waitForHandshake passed a raw
p2p.DiscReadTimeout into markError, but markError classified errors only
via errors.Unwrap(err), which returns nil for non-wrapped errors. As a
result, the timeoutError meter was never incremented and all such
failures fell into the peerError bucket.
This change makes markError switch on the base error, using
errors.Unwrap(err) when available and falling back to the original error
otherwise. With this adjustment, p2p.DiscReadTimeout is correctly mapped
to timeoutError, while existing behaviour for the other wrapped sentinel
errors remains unchanged
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
The fetcher should not fetch transactions that are already on chain.
Until now we were only checking in the txpool, but that does not have
the old transaction. This was leading to extra fetches of transactions
that were announced by a peer but are already on chain.
Here we extend the check to the chain as well.
All five `revert*Request` functions (account, bytecode, storage,
trienode heal, bytecode heal) remove the request from the tracked set
but never restore the peer to its corresponding idle pool. When a
request times out and no response arrives, the peer is permanently lost
from the idle pool, preventing new work from being assigned to it.
In normal operation mode (snap-sync full state) this bug is masked by
pivot movement (which resets idle pools via new Sync() cycles every ~15
minutes) and peer churn (reconnections re-add peers via Register()).
However in scenarios like the one I have running my (partial-stateful
node)[https://github.com/ethereum/go-ethereum/pull/33764] with
long-running sync cycles and few peers, all peers can eventually leak
out of the idle pools, stalling sync entirely.
Fix: after deleting from the request map, restore the peer to its idle
pool if it is still registered (guards against the peer-drop path where
Unregister already removed the peer). This mirrors the pattern used in
all five On* response handlers.
This only seems to manifest in peer-thirstly scenarios as where I find
myself when testing snapsync for the partial-statefull node).
Still, thought was at least good to raise this point. Unsure if required
to discuss or not
