This PR aligns the `Cells` and `GetCells` message implementations with
the spec: https://github.com/ethereum/devp2p/blob/master/caps/eth.md#getcells-0x14
Previously, `GetCellsPacket` and `CellsPacket` embedded `GetCellsRequest`
and `CellsResponse`. This caused them to be encoded as nested lists, which
does not match the wire format defined by the spec. This PR inlines their fields
to flatten the RLP layout of `Cells` and `GetCells`.
Blocks heavy in precompile calls (e.g. Aztec's proof settlement txs)
spend most of their processing time on operations (ECMUL, pairings, KZG
point evaluation, MODEXP) that the state prefetcher has already computed
and thrown away.
This PR adds a precompile result cache shared between the prefetcher and
block processing (and the miner), following the JumpDestCache pattern.
Note that cached precompiles are keyed by address and input, with
entries partitioned by the active precompile set, so a fork that changes
the behavior behind an address can never be served results from before
it.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR fixes the incorrect size calculation for blob sidecar. The
original formula is for legacy sidecar without the version tag. As the
legacy version has been deprecated and no longer supported by the Geth's
blobPool, the size calculation should also be flipped to sidecar v1.
commitPivotBlock updated committed without pivotLock, while other pivot
transitions serialize through that lock. Take the lock around the store
to keep pivot commitment consistent with pivotHeader updates.
This PR coordinates the prefetcher with the main tx executor. Block
processing publishes the index of the transaction it is executing,
prefetch workers skip anything already reached and transactions above 1M
gas are promoted to the front of the prefetch queue while the rest keeps
block order.
The deep-reorg check used depth >= maxReorgDepth, rejecting reorgs at
exactly the configured limit. Use > so a depth equal to maxReorgDepth is
still accepted.
When a peer drop removes the last waiter for a hash, waitlist and
waittime were cleaned up but partial could retain a stale entry. Delete
it too so the hash is fully forgotten.
This PR improves the sync mode management, disabling the snap mode once
the pivot block is committed.
Originally the mode will only be flipped from snap to full once the entire sync
cycle is completed. However, it's theoretically possible to cancel the sync cycle
after committing the pivot but before inserting the remaining blocks. It drags
the following sync cycle back to the snap mode.
The snap mode should only be used if the head state is missing and unrecoverable.
Enabling the snap mode for multiple times should be strictly prevented.
What's more, it brings a fix to pivot block management, that once the
pivot block is committed, the pivot block marker should be no longer advanced,
preventing the situation that the real pivot state is below the pivot marker.
Restarting a node mid-snap-sync can crash-loop during startup chain
repair. On a restart mid-sync the pathdb is disabled (`waitSync`), so
repair rewinds the stateless head to genesis and asks `StateRecoverable`
if it can roll back there. `Recoverable` doesn't check `waitSync`, so it
reports genesis as recoverable, but `Recover` does and refuses with
`waiting for sync`, tripping a `log.Crit`. The flag is persisted, so the
node crash-loops on every restart.
CRIT Failed to recover state number=0 hash=37fcdc..6edf0b err="waiting
for sync"
The fix is to make `Recoverable` return false while a sync is running,
so it agrees with `Recover`. Repair then falls through to the existing
"wait state sync" path.
This PR adds the CLI flag gogc for twisting the garbage collection
target. The default value is chosen as the 50, balancing the
performance gain and potential memory peak.
blobFetcherQueueingHashes used len(announces), which is the number of
peers with pending announces, not the number of queued hashes. Count
unique hashes across peers instead.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
Now with 8037, there are transactions that fail AFTER intrinsic gas but
BEFORE Call or Create operation. These will currently result in a panic
in tracing, since they produce a receipt
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
`conversionQueue` is meant to run billy/legacy migrations serially on a
single worker. On each `startConversion`, it currently always calls
`runNextTask()`, which overwrites `taskDone` and starts another
goroutine even when a migration is already running.
That breaks two invariants during an upgrade with legacy entries in both
the main store and limbo:
1. Migrations intended to be serial can run concurrently.
2. `close()` only waits for the most recently launched task, so shutdown
can close the stores while a forgotten migration is still using them.
Only start the next queued migration when no task is active (`taskDone
== nil`). The existing `taskDone` completion path then advances the
queue, and `close` reliably waits for the active task.
Commit 1f87331fb moved the known-transaction marking in
`sendPooledTransactionHashes` to after a successful send, so hashes are
not marked known to the peer if the announcement fails to go out.
The sparse blobpool change (d91b71fb3) reintroduced the original
track-before-send ordering when adding the eth/72 packet variant,
causing failed announcements to suppress future re-announcements of the
same hashes to that peer.
This restores the send-first ordering for both eth/71 and eth/72 packet
versions, and adds a regression test covering success and failure paths
on both protocol versions.
## Checklist
- [x] Restored mark-known-after-send for ETH71 and ETH72
- [x] Added `TestSendPooledTransactionHashes` covering success and
closed-pipe failure
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
Follow-up to #34850. The miner's payload EVM previously used a private
per-EVM JUMPDEST map, so every ~2s payload rebuild re-analyzed the code
bitmap of every contract the block touches.
`Chain.GetHeaders` steps by `1 + Skip` in the forward branch but by `1 -
Skip` in the reverse branch. `Skip` is a `uint64`, so the reverse stride
is wrong for every `Skip > 0`:
| Skip | reverse step | result |
| --- | --- | --- |
| 0 | `-= 1` | correct |
| 1 | `-= 0` | block number never moves, the same header is returned
`Amount` times |
| >= 2 | `1 - Skip` underflows | walks *forward* instead of back (`Skip
= 2` on block 100 lands on 101) |
Per the `GetBlockHeadersRequest` definition, `Skip` is "Blocks to skip
between consecutive headers", so the stride is `Skip + 1` in whichever
direction the query runs. Subtracting `1 + Skip` makes the reverse
branch mirror the forward one.
The existing table test covers forward + `Skip: 1` and reverse + `Skip:
0` (which works by accident, since `1 - 0 == 1`). Added the missing
reverse + `Skip: 1` case; it fails on master:
```
--- FAIL: TestChainGetHeaders/3
Test: TestChainGetHeaders/3
FAIL github.com/ethereum/go-ethereum/cmd/devp2p/internal/ethtest
```
and passes with the fix (4/4).
No caller sends a reverse request with `Skip > 0` today, so nothing is
broken in the current suite. The helper computes the expected headers
that responses are checked against, though, so the moment a reverse+skip
case is added it would silently assert the wrong headers rather than
fail loudly.
This is the implementation of EIP-8070 Sparse Blobpool. It introduces
protocol version eth/72 which relays blob transaction cells instead of
full blobs.
The blobpool now store 'incomplete' transactions, where only some of
the cells are provided. The stored cell indexes are taken from the
custody bitmap, which is provided by the consensus layer in
forkchoiceUpdatedV4. This method will be called once Glamsterdam
activates, and the default custody is full custody, so for now there
is no change in the amount of stored cells for now.
The main entities added are the BlobBuffer and BlobFetcher, which work
together to track and fetch missing cells from connected peers. The
partial transactions become available for inclusion in blocks when
they are covered by enough peers that hold all cells.
This change also introduces engine_getBlobsV4, which allows for
cell-based responses (and partial blobs with only some of the cells).
We maintain backward compatibility with getBlobsV3 which expects full
blob responses by recovering the blob from available cells. Since this
process is resource-intensive, we proactively cache the conversion so
it is ready in time for getBlobsV3 calls. This mechanism will be
removed once support for getBlobsV4 is universal across all consensus
layer implementations.
devp2p tests for eth/72 are not part of this initial change. This is
to avoid breaking test success status for execution clients that do
not have eth/72 implemented yet. The tests will be added in a
subsequent change.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
- Move `msg.Discard` defer ahead of the max message size check in
`handleMessage`.
- Ensures oversized messages are released when the handler returns
early.
The DNS discovery crawler (ethereum/discv4-crawl) builds signed enrtree
lists by crawling the network and running the collected records through
`devp2p nodeset filter` and `devp2p dns sign`. Those records come from
external nodes, and `enr.Record` keeps entries it can't decode as raw
RLP, so a self-signed record with an out-of-range port (EIP-778 defines
ports only as "big endian integer", not `uint16`) can round-trip
verbatim into a signed tree. Consumers that decode ports strictly then
fail to decode that record.
Two publish-side checks:
- `MakeTree` rejects a record if a present port entry
(`tcp`/`tcp6`/`udp`/`udp6`/`quic`/`quic6`) does not decode as a
`uint16`. This is an invariant at the signing boundary: geth won't sign
a tree containing a record with an undecodable port, regardless of how
the node list was produced. Absent and zero ports are unaffected.
- `devp2p nodeset filter -dialable` keeps only nodes advertising a
usable RLPx port (non-zero `tcp`/`tcp6`/`quic`/`quic6`), letting the
crawler drop discovery-only and unreachable nodes so consumers aren't
handed peers they can't connect to.
The two are deliberately separate: the `MakeTree` check is a correctness
guard that always applies, while `-dialable` is an opt-in selection
filter for the crawler pipeline. A follow-up will add `-dialable` to
discv4-crawl's `filter_list`.
When we are deploying library dependencies, if the `TransactOpts` nonce
is unset we will choose the nonce of the next deployment transaction
based on the pending nonce of the sender. If a previous library
deployment transaction was made but not yet accepted into the pool, the
pending nonce will not be updated for the the next deployment
transaction.
This PR introduces a new method to the bind API `WaitAccepted` which
poll until a submitted transaction hash is accepted into the pool or
rejected. The bindings for v1 are updated to invoke this method after
deploying each library dependency.
## Summary
- Only apply user-configured `MaxBlobsPerBlock` when it is strictly
below the protocol-defined maximum.
- Prevents the miner from building blocks that exceed the consensus blob
limit.
## Test plan
- [x] `go test -short ./miner/`
WithAttrs and terminalFormat appended to h.attrs directly, which can
mutate the shared slice when it has spare capacity. Clone attrs first to
avoid corrupting parent handler state.
The peer dropper periodically disconnects random peers to create churn.
This was previously blind to peer quality.
This PR adds peer-score based peer protection, handling the
multi-dimensionality problem of peer scoring through the concept of
protected peer pools.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: healthykim <bsbs8645@snu.ac.kr>
## Summary
Sanity fixes surfaced by running the EELS `tests@v20.0.0` fixture
release (63,109 blockchain tests) through `evm blocktest`.
Also bumps CI to consume the new release: `build/checksums.txt` now
points at `tests@v20.0.0` / `fixtures.tar.gz` from
`ethereum/execution-specs` (supersedes the archived EEST repo's
`fixtures_develop`).
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR does a few things:
- reject `debug_setHead` if the target is even before the pivot block
(if non-nil)
- reject `debug_setHead` if in path mode, the target is not recoverable
- decouple the chain rewinding and state recovery in path mode and
recover the state in one shot
---------
Co-authored-by: jonny rhea <5555162+jrhea@users.noreply.github.com>
Closes#35314.
### Problem
`TransactionArgs.ToTransaction` selects `SetCodeTxType` when an
`authorizationList` is present, but then unconditionally downgrades to
`LegacyTxType` when `gasPrice` is set:
```go
case args.AuthorizationList != nil || defaultType == types.SetCodeTxType:
usedType = types.SetCodeTxType
...
if args.GasPrice != nil {
usedType = types.LegacyTxType
}
```
A legacy transaction cannot carry an EIP-7702 authorization list, so the
list is silently dropped. `eth_sendTransaction`, `eth_signTransaction`
and `eth_fillTransaction` can then return a plain legacy
transaction/hash even though the requested delegation update or
revocation was never included.
The downgrade also masks a latent issue: `setFeeDefaults` returns early
once `gasPrice` is set (without `authorizationList`) and never fills
`MaxFeePerGas`/`MaxPriorityFeePerGas`. Building the `SetCodeTx` without
the downgrade would hit
`uint256.MustFromBig((*big.Int)(args.MaxFeePerGas))` with a nil fee cap
and panic. Erroring early is therefore the correct behavior.
### Fix
Reject the `gasPrice` + `authorizationList` combination in
`setFeeDefaults`, mirroring the existing `gasPrice` / EIP-1559 guard, so
the request fails explicitly instead of producing a transaction that
omits the authorization list.
### Test
Added a `setFeeDefaults` case asserting the new error. Verified
fail-on-main: with the guard reverted the test fails (`expected error:
both gasPrice and authorizationList specified`); with the guard it
passes. Full `internal/ethapi` package, `go vet` and `gofmt` are clean.
### Note
The same silent-drop applies to `gasPrice` + `blobVersionedHashes` (blob
transactions also cannot be legacy). I scoped this PR to the reported
`authorizationList` case; happy to extend the guard to blob hashes in
the same spot if preferred.
This is an edge case found by @weiihann.
Under 8038, the cold storage access cost is increased to 3,000 gas,
which exceeds the sentry check threshold. Therefore, the sentry check no
longer guarantees that the remaining gas is sufficient to cover a cold
slot access.
Therefore, an additional access affordability check is added to
eliminate the potential DoS vector.
This PR clears `journal.writer` immediately after `Close()` in `rotate` and `setupWriter`,
before checking the error. This prevents leaving a stale file handle that could be reused
if close fails.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
Passing `--fuzz=false` currently still selects blocktest fuzz output
mode because the command checks whether the flag was set.
This switches the blocktest fuzz-mode checks to use the boolean flag
value, so explicit false keeps the normal output path while `--fuzz`
still enables fuzz-oriented reporting.
The HTTP JSON-RPC server caps request bodies at a hardcoded 5 MB
(`defaultBodyLimit`), with no way to raise it. EEST bloatnet depth
benchmarks post worst-case batch requests larger than that during
fill-stateful, which the server rejects with `413 Request Entity Too
Large`.
This adds `node.Config.HTTPBodyLimit` (default 5 MB), wired through to
the HTTP/WS server config, and exposes it via `--rpc.http-body-limit`
(value in **megabytes**, default 5). Default behaviour is unchanged.
## Summary
- Add the missing `forks.Amsterdam` case to `ChainConfig.Timestamp`.
- Callers can now look up the Amsterdam fork activation time.
## Test plan
- [x] `go test -short ./params/`
This PR fixes five remaining differences between master and
glamsterdam-devnet-6:
- RegularCost for Auths has been increased, since 8037 increases the
WARM_ACCESS cost
- Floor is anchored to the transaction base cost, not the intrinsic cost
- Auth destinations need to be recorded in the BAL before the call is
executed
- Change the place where intrinsic gas is verified in calls to delegated
addresses
- Refund state gas directly to reservoir in outer tx frame
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
`FairMix.AddSource` takes ownership of the supplied iterator, but if the
mixer has already been closed it currently returns without closing that
iterator. Close the rejected source in this shutdown path so callers
racing with `Close` do not leak iterator resources.
The drop helper uses sort.Search on the sorted wallet list, but the
returned index can point at the next greater wallet when the target URL
is missing. Fix this by rewriting the drop to use slices.DeleteFunc and a map.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Increment `dropCount` on every TALKREQ dropped due to overload timeout.
Previously the counter was only updated when the throttled warning log
was emitted.
This PR addresses an issue in the eth71 `BlockAccessListsMsg` handler,
specifically:
- if the requested bal is not accessible in the server side, 0x80
(EmptyString) will be returned as the marker
- at the client side, old message definition
`rlp.RawList[RawBlockAccessList]` assumes all the elements are List
- the message with 0x80 (kind = string) won't be decoded correctly
- the peer will be disconnected
The message definition has been changed to `rlp.RawList[rlp.RawValue]`,
which is aligned with the one in SNAP/2 protocol.
EIP-7928 requires that each address appears exactly once in the block
access list.
In the `BlockAccessList.Validate`, the strict `isStrictlySortedFunc` is
required to detect the duplicated accounts.
The LookupRandom RPC handler obtains a node iterator from
RandomNodes() but never closes it.
RandomNodes() returns a lookupIterator backed by a cancelable
context derived from the listener's lifetime context
(newLookupIterator -> context.WithCancel). The iterator's
Close() is what calls the cancel func, so each LookupRandom call
that never closes leaks the cancel func (and any in-flight lookup
goroutine) until the discv4 listener shuts down. When the listener
serves the RPC API (discv4 --rpc) it is long-lived, so repeated
LookupRandom calls accumulate the leak.
Close the iterator when the handler returns. This matches how the
crawler already releases the same RandomNodes() iterators
(crawl.go closes every iterator it consumes).
SignTx populated BlobHashes and the sidecar fields (Blobs/Commitments/
Proofs) for a blob transaction but never set args.BlobFeeCap. As a
result the external (clef) signer received maxFeePerBlobGas:null and
signed a transaction inconsistent with the one passed in, silently
dropping the blob fee cap.
Set args.BlobFeeCap from tx.BlobGasFeeCap() so the signing request
faithfully reflects the input transaction. This mirrors the existing
handling of the other blob-tx fields.
Include the actual and expected request IDs in the devp2p test
response failure message. This also fixes the typo in the diagnostic so
mismatched responses report a useful error.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
Print the expected freezer close sync error before the actual value in
TestFreezerCloseSync. The assertion already compared have and want
correctly, but the failure message reported them in the wrong order.
This PR adds blobTxForPool migration support in limbo. Previously, there
was no conversion path from limbo entries containing types.Transaction.
Now that we have the new blobTxForPool type, this PR adds migration
logic between both types. New test code (limbo_test.go) to test this
conversion is added.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is related to the recent bug reported in #35210.
While trying to reproduce the error, I found that when the head state is
missing (e.g. unclean shutdown), we attempt to truncate the head to the
most recent block with state across all chain freezer tables.
However, for newly added tables such as the bal table, both the head and
tail are initialized to the minimum head of the existing chain freezer
tables. As a result, the `truncateHead` fails with the “truncate below
tail” error.
This PR fixes the issue by resetting newly added empty tables with
`items` as the tail when `truncateHead(items)` is called on them.
Clone the existing terminal handler attrs before appending new attrs.
This avoids a potential attr memory overwrite when append reuses the
backing array shared with the parent handler.
`devp2p discv4 listen` / `discv5 listen` is the supported replacement
for the removed bootnode tool, but it bound IPv4-only and `-extaddr`
took a single address, so it couldn't run a dual-stack bootnode.
This binds the listener dual-stack (falling back to IPv4-only where IPv6
is unavailable) and lets `-extaddr` take a comma-separated IPv4/IPv6
pair. A single node can then advertise both `ip` and `ip6` in its ENR
over one UDP port:
```
devp2p discv4 listen --nodekey <key> --addr [::]:30301 \
--extaddr 203.0.113.10:30301,[2001:db8::1]:30301
```
The fallback IP is only derived from the listener when no `-extaddr` is
given, so a v4- or v6-only `-extaddr` no longer leaks a loopback entry.
All addresses must share one UDP port (single socket).
This feature is an optimization used in the BAL, mostly for experimental
purpose.
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
## Summary
- Release the storage iterator after iterating slots in `geth snapshot
dump`, matching the existing account iterator cleanup.
## Test plan
- [x] `go build ./cmd/geth/...`
- [ ] Manual: run `geth snapshot dump` on a node with storage data and
verify output is unchanged
This PR addresses the panic in tests. As the eventLoop is spun up when
the downloader was closed, the sub will be nil and make the panic
happens.
```
goroutine 421 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*DownloaderAPI).eventLoop(0xcb0e4d0)
/opt/actions-runner/_work/go-ethereum/go-ethereum/eth/downloader/api.go:91 +0x127
created by github.com/ethereum/go-ethereum/eth/downloader.NewDownloaderAPI in goroutine 352
/opt/actions-runner/_work/go-ethereum/go-ethereum/eth/downloader/api.go:50 +0xf2
```
This PR improves the block download used by snap sync. Specifically,
blocks and their associated data (receipts and canonical hash mappings)
are now written directly to the database without checking existence.
The current implementation could fail in cases where the block header
and body were already present (has.Block returns true), but the
corresponding canonical hash mapping was missing. One possible scenario
is when a newPayload event is processed without a subsequent
forkChoiceUpdate.
It is still unclear why Geth may re-enter snap sync after Engine API
events have been processed after the sync. Anyway, bypassing the
existence is a reasonable change.
What's more, in the downloader, the presence of canonical hash is also
considered for deciding the range of blocks to be downloaded.
Specifically:
- in the full sync, the block with header and body available but
canonical hash missing will be re-inserted;
- in the snap sync, the block with header, body and receipt available
but canonical hash missing will be re-inserted;
Implements spec change https://github.com/ethereum/EIPs/pull/11807
This PR resolves the conflict between the EIP-7928 and EIP-8037.
Specifically in contract deployment, EIP-7928 requires to not resolve
the deployed account until it's accessed, while in EIP-8037, the early
access is required to determine if the account-creation should
be charged or not.
This PR addresses this conflict by changing the EIP-8037 a bit,
unconditionally charge the account creation in CREATE Family
(CreateTx, Create/Create2 opcode) and refunds the associated
gas cost if the account creation doesn't happen ultimately.
Checkout https://hackmd.io/@bFEBbZiVSAO0IURh9qzEFg/BJmFYqCeGl for more
details
What's more, now the LIFO mechanism is used for refilling the state cost
in frame revert, frame halt, state opcode refunds.
When ancient history is pruned, geth serves old block bodies and receipts
back from era files on disk. Until now that fallback only worked for .era1
files (pre-merge), so requests for post-merge blocks backed by .ere files
failed even though the data was present. This PR generalizes the era store
to open both formats.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR adds the support of Pebble v2, details as below:
- Pebble V2 will be used if database is empty
- Pebble V1 will be used if database is not empty and the format is old
- Upgrade command (geth db pebble-upgrade) is provided to upgrade the
format to v2 offline
Mirror the guard applied to (*UDPv4).Dial in #34916: when the target
node has no usable UDP endpoint, return errNoUDPEndpoint instead of
silently sending the ENRRequest to an invalid AddrPort and waiting for a
timeout.
The other UDPEndpoint-using request paths in this file already do this:
ping v4_udp.go:215 errNoUDPEndpoint
Ping v4_udp.go:228 errNoUDPEndpoint
newLookup v4_udp.go:309 errNoUDPEndpoint
RequestENR v4_udp.go:358 addr, _ := n.UDPEndpoint() <-- outlier
RequestENR is reachable from external callers like cmd/devp2p/crawl.go,
which feeds in arbitrary nodes that may not have a UDP port set. Before
this change, such nodes burn one full RPC timeout; after it, the caller
gets a clean error immediately.
The added test fails on master with "RPC timeout" and the trace logs
"PING/v4 addr=invalid AddrPort", confirming packets are being written to
an unspecified address; with the fix it returns errNoUDPEndpoint without
doing any I/O.
This PR inlines the gas deduction by getting rid of the tracer and use
`chargeRegularOnly` for the non-state opcode.
It fixes a performance regression introduced by EIP-8037 PR.
```
throughput MGas/s | 184.4 (±0.3%) | 193.1 (±1.0%) | +4.7% ▲
-- | -- | -- | --
mean newPayload | 164.2 ms (±0.3%) | 156.9 ms (±1.0%) | -4.5% ▲
p50 newPayload | 154.6 ms (±0.1%) | 147.6 ms (±0.7%) | -4.5% ▲
p95 newPayload | 273.3 ms (±2.3%) | 261.6 ms (±2.4%) | -4.3% ≈ noise
p99 newPayload | 403.6 ms (±4.4%) | 380.9 ms (±4.0%) | -5.6% ≈ noise
```
This PR drops support for v0 blob sidecar in blobpool. Since the
osaka fork activation time has passed, these code paths are
now unused. It is assumed that only v1 transactions exist in
the blobpool.
Since go 1.18 reflect has `reflect.Pointer` which replaces
`reflect.Ptr`. Newer versions of `govet` will alert.
See also: https://pkg.go.dev/reflect#pkg-constants
Adds `testing_commitBlockV1`. It is the write companion of `testing_buildBlockV1`:
it builds a block from the provided payload attributes and transactions on
top of the current canonical head, inserts it, and sets it as the new
head, returning the new head hash.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This PR fixes an issue where flat states are continuously persisted
during downloadState, while the sync journal is only persisted at the
end of Sync.
As a result, an unclean shutdown can leave the on-disk flat state ahead
of the journal markers. Some persisted entries may be stale (storage
slots that should have been deleted), and these dangling entries are not
detected or fixed by subsequent state downloads.
To address this, this PR introduces a cleanup step before state
downloading begins. It removes all state entries that are not covered by
the persisted journal markers.
This PR introduces a new condition that if the local node falls behind
too much and the required BAL for catching up is very likely to be
unavailable, the entire snap sync will be restarting from scratch.
As the defined BAL retention window is weak-subjective-period which is
calculated dynamically. A more conservative threshold is used (90K
blocks) for robustness.
Apart from that, the BAL catchup will be divided into several spans and
apply one by one. It's essential to prevent the potential out-of-memory
panic of placing the entire BAL set in memory.
This PR improves the slot reservation logic in the context of snap/2.
Geth has the mechanism to reserve roughly half the peer slots for peers
supporting the snap protocol if snap syncing is needed by local node.
With the context of snap/2, this mechanism should be changed that:
we reserve the slot for the "usable snap peer", not blindly for peer
with snap extension enabled (such as legacy snap/1, which can't serve
the snap/2).
sendInvalidTxs's *eth.TransactionsPacket case iterated `txs` — the
locally-sent invalid transactions, every one of which is in `invalids`
by construction — instead of the transactions actually carried by the
received packet. As a result the loop returned "received bad tx" on the
very first TransactionsPacket the peer sent, regardless of its contents,
and never inspected what was really propagated.
Iterate msg.Items() (the decoded contents of the received packet) so the
"node must not propagate invalid txs" conformance check tests the real
condition instead of producing a false negative.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
The stack primitives pop by value: pop() returns the 32-byte value
itself, so every popped operand is copied out of the stack arena before
it is used. The result side was already in place, peek returns a pointer
and binary ops write into the new stack top. This PR fixes the operand
side: pointer-returning primitives (popPtr, popPtrPeek, etc), with the
handlers rewritten to read operands directly from their arena slots.
Every popped operand paid the copy, whatever the op went on to do with
it, so this optimization covers the arithmetic and comparison ops as
much as JUMP, MSTORE, SSTORE and RETURN.
The copy is visible in the assembly. On arm64, master's opLt spends four
instructions moving the popped value through the frame, and the
comparison then reads it back from there:
LDP (R5), (R6, R7) ; load words 0 and 1 of the popped value from the
arena
LDP 16(R5), (R5, R8) ; load words 2 and 3
STP (R6, R7), vm.~r0-64(SP) ; store words 0 and 1 into a frame slot
STP (R5, R8), vm.~r0-48(SP) ; store words 2 and 3
With popPtrPeek those four instructions are gone, the frame shrinks from
locals=0x58 to locals=0x18, and the function from 336 to 288 bytes. The
compiler cannot remove the copy itself: uint256.Int is a four-element
array, and Go's SSA does not promote arrays longer than one element to
registers, so a by-value pop pays this round trip no matter how far
inlining gets, for LT exactly as for ADD.
The CALL and CREATE families are deliberately not converted: a child
frame reuses the same stack arena, so parent pointers into popped slots
die when the child pushes. The rule is recorded on the primitives:
pointers stay valid until the next push or any sub call. Converting the
call family safely means materializing scalars before the child call,
left for later work with a call-heavy benchmark to justify it.
### Benchmarks
Measured with the benchmark suite from #35144 (the evm-bench contract
workloads and the block import benchmark), which is not part of this
PR's diff. Apple M4 Max, fixed iteration counts, n=10, all p=0.000. B/op
and allocs/op are statistically identical on every benchmark:
| benchmark | master | PR | vs master |
|---|---|---|---|
| Snailtracer | 60.0 ms | 54.1 ms | -9.8% |
| TenThousandHashes | 13.2 ms | 12.2 ms | -7.8% |
| ERC20Transfer | 11.7 ms | 11.0 ms | -5.5% |
| ERC20Mint | 7.49 ms | 7.02 ms | -6.2% |
| ERC20ApprovalTransfer | 8.92 ms | 8.44 ms | -5.4% |
This PR is independent of #35144 but plays nicely with it: the generated
dispatch there splices these handler bodies, so the in-place forms land
in its fast path too, where they measure larger.
### Testing
The rewritten handlers run on the interpreter's only execution path, so
correctness rests on references outside the change:
- **Consensus fixtures.** The full tests package passes: state tests,
the execution-spec families, blockchain tests.
- **Opcode testcases.** The JSON testcases compare individual opcode
results against committed expected values.
- **Tracer fixtures.** The tracetest reference files pin exact log and
return data shapes, covering the rewritten LOG and RETURN paths.
- **Cross-build differential.** A goevmlab campaign running this
branch's evm against master's evm over generated state tests across four
forks (Prague, Cancun, London, Osaka) with full trace comparison:
160,566 tests, zero divergences.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
`TestTracingHTTPTimeout` still flakes in CI after #35101, failing at the
POST:
--- FAIL: TestTracingHTTPTimeout (0.26s)
tracing_test.go:633: request: Post "http://127.0.0.1:43497": EOF
The test sets a short server `WriteTimeout` and posts a blocking call.
`ContextRequestTimeout` leaves a fixed 100ms for the server to write its
timeout response before the HTTP write deadline cuts the connection.
I can't repro it locally, but my theory is that under load that write
can miss the window, so the connection is dropped and the client POST
returns `EOF`, failing the test before it inspects the span. This is the
only test exposed to it because it is the only one that configures a
`WriteTimeout`.
The EOF is benign: the server sets the timeout error on the SERVER span
before attempting the write, independent of whether the client receives
the response. Since that span status is all the test asserts,
`tryPostJSONRPC` tolerates the transport error instead of failing on it.
This PR fixes an issue that when peers legitimately lack a requested
BAL, empty (0x80) is delivered and this BAL entry will be refetched
over and over again.
A `refused` tracker is added and catchUp will fail if this BAL is
unavailable against the entire peerset.
Implements https://eips.ethereum.org/EIPS/eip-8037
mainly done in order to judge the complexity of the EIP
and to act as a jumping off point, since the eip will likely
change.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Currently geth ignores the docker `--memory` directive and doesn't
adjust its cache size downward when necessary, potentially running into
OOM.
while gopsutil has functions like `docker/CgroupMem()` they are rather
for reading cgroup memory limit of a container from the host.
This PR adds an optimization to the `findNodeByID` function in
`p2p/discover`. There is already an open PR (#33205) for similar
improvements, and I have further optimized the function to get better
performance. I have attached the benchmark results comparing the current
`main` branch with my `optimized version`, and the results show clear
improvements.
---------
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR introduces a cache for GetBlobs request.
The main purpose of this PR is to reduce the getBlobs latency by reading and
decoding blobs from the pool in advance of the actual query. This is important
especially in the context of a sparse blobpool, since it may be necessary to
recover blobs from cells on a getBlobs request.
Previously, the Engine API read and decoded blobs from the pool on every call.
Now those calls check the cache and only fall back to the pool on a miss.
The cache has two modes:
- In topK mode (default), it wakes up periodically, picks the most profitable
pending blob transactions up to the current fork's maxBlobsPerBlock, and loads
their blobs. The selection logic is shared with the miner's block-building
logic. The selection size is derived from eip4844.MaxBlobsPerBlock at the
current head.
- When the CL calls HasBlobs, the cache switches to hasBlobs mode and tries to
pin the set it just reported as available. Cache updates (read, decode, and
optionally conversion in the future) run in background goroutines.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds snap/2 (EIP-8189), a block-access-list (BAL) based state sync, and
wires it to run side by side with snap/1. It's opt-in (for now) behind a
new --snap.v2 flag and chosen at startup.
https://eips.ethereum.org/EIPS/eip-8189
---------
Co-authored-by: Toni Wahrstätter <info@toniwahrstaetter.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
In old code, mu is struct not pointer, it caused create new mutex event
with same writer. Change to use pointer to sync.Mutex, so that the mutex
is shared between handler with same writer.
This adds a client option to configure trace context propagation via the
`traceparent` HTTP header.
I'm adding this so that prysm can enable distributed tracing on their
engine API client.
This is a PR that removes all correctly flagged typos, in order to stop
an onslaught of slop PRs in its tracks. It should be followed by #34994
but the latter needs more configuration work and I want to limit the
stem of PRs right now.
Fixes an issue where we would falsely return 400 when we should return
200 because the request was served successfully
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is trying to stem further slop PRs by going over all incorrect
strings and fixing them.
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
## Overview
This PR fixes a race condition during blockchain shutdown where snapshot
generation could continue accessing the trie database after it has been
closed, leading to iterator errors. We noticed this in one of our nodes
on https://github.com/ava-labs/avalanchego, which relies on an older
version of geth with the same issue (so this behavior does happen!).
During node shutdown, the following sequence occurs:
1. `BlockChain.Stop()` calls `snaps.Release()` to clean up snapshot
resources
2. `Release()` only resets the cache but doesn't stop the generator
goroutine
3. The trie database is then closed via `triedb.Close()`
4. The still-running generator attempts to iterate storage tries
5. Iterator fails because the database is closed (`"Generator failed to
iterate storage trie"`)
## Problem
There are three related bugs:
1. `Release()` doesn't stop generation: The `diskLayer.Release()` method
only resets the cache without stopping ongoing snapshot generation,
leaving the generator goroutine running after database closure.
2. `stopGeneration()` has an incorrect completion check: The
`stopGeneration()` method checks `genMarker != nil` to determine if
generation is running. However, `genMarker` is set to nil when
generation completes successfully, even though the generator goroutine
is still waiting for the abort signal at the end of `generate()`. See
line 705 in `generate.go`:
eaaa5b716d/core/state/snapshot/generate.go (L699-L707)
This means `stopGeneration()` returns early without sending the abort
signal.
3. Node shutdown doesn't stop generation: During shutdown, no code path
calls `stopGeneration()` or sends the abort signal to the generator,
causing the generator to access a closed database and error.
## Fix
- Modified `diskLayer.Release()` to call `stopGeneration()` before
releasing resources
- Added cancelation architecture, removing reliance on someone having to
wait
- Fixed `stopGeneration()` to properly and safely stop snapshot
generation
- Added `TestGenerateGoroutineLeak` to verify the fix and prevent
regression. The test fails without the fix and passes with it.
- The test creates a snapshot with active generation, waits for
completion, then calls `Release()`, and uses `go.uber.org/goleak` to
assert no generator goroutine survives.
- Without the fix, the test fails: `Release()` returns without stopping
the generator, which stays parked at `generate.go:705` waiting for an
abort signal that never comes:
```
--- FAIL: TestGenerateGoroutineLeak (0.88s)
generate_test.go: found unexpected goroutines:
[Goroutine 6 in state chan receive, with
core/state/snapshot.(*diskLayer).generate on top of the stack:
core/state/snapshot.(*diskLayer).generate(...)
core/state/snapshot/generate.go:705
created by core/state/snapshot.generateSnapshot
core/state/snapshot/generate.go:79 ]
```
- With the fix, the test passes: `Release()` -> `stopGeneration()`
blocks until the generator goroutine has fully exited, so nothing leaks
Note that this fix follows the same pattern used in `Tree.Disable()` in
https://github.com/ethereum/go-ethereum/pull/30040, which introduced
`stopGeneration()` for use in `Disable()` and `Rebuild()` but didn't
address the shutdown path.
The test follows the same pattern used in
`TestCheckSimBackendGoroutineLeak`
### Summary
`TestServerWebsocketReadLimit/limit_with_large_request_-_should_fail` is
flaky on `windows/amd64` (see [run
25364841576](https://github.com/ethereum/go-ethereum/actions/runs/25364841576/job/74378334589)
referenced in #34877):
```
--- FAIL: TestServerWebsocketReadLimit/limit_with_large_request_-_should_fail (0.02s)
server_test.go:279: unexpected error for read limit violation: read tcp 127.0.0.1:56703->127.0.0.1:56700:
wsarecv: An existing connection was forcibly closed by the remote host.
```
When the server enforces the read limit and tears the connection down,
the client's read can race the close frame. On Windows the OS surfaces
that race as `wsarecv: An existing connection was forcibly closed by the
remote host` instead of the gorilla `CloseError(1009)`,
`websocket.ErrReadLimit`, or the POSIX `connection reset by peer` the
test already tolerates.
This change adds `"forcibly closed"` to the set of acceptable error
substrings for the failure case, so the Windows reset is recognized as a
valid signal that the server enforced the limit.
### Fixes
#34877
### Test plan
- [x] `go test -count=5 -run TestServerWebsocketReadLimit ./rpc/`
(darwin/arm64) — pass
- [x] `go test ./rpc/...` — pass
- [x] `go vet ./rpc/...` / `gofmt -l rpc/server_test.go` — clean
- [ ] CI on `windows/amd64` confirms the flake no longer trips
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
The checksum count during EraE import is off by one when `checksums.txt`
ends its last line on a newline, as the pandaops file does. The current
code would result in one empty string after the final `\n`, something
like
```
[]string{
"line1",
"line2",
"line3",
"",
}
```
Trim off the final `\n`, if it exists: `return
strings.Split(strings.TrimRight(string(b), "\n"), "\n"), nil`
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
`clef` is a great tool, however:
* It is no longer maintained
* No one else in the team can pronounce it properly
We are however receiving some slop PRs for it, so I think it's time -
with infinite sadness - to say goodbye.
Make the `Block` parameter optional on the six state-reading methods,
defaulting to `latest` when omitted:
- `eth_getBalance`
- `eth_getCode`
- `eth_getStorageAt`
- `eth_getTransactionCount`
- `eth_getProof`
- `eth_getStorageValues`
This implements the behavior proposed in https://github.com/ethereum/execution-apis/pull/812.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
Update the EraE (ere) reader and builder to the latest e2store ere spec
(https://github.com/eth-clients/e2store-format-specs/pull/16).
The reader now derives the component layout from the on-disk e2store type
tags via the dynamic block index, rather than assuming fixed slot positions.
This makes the optional components (receipts, td, proof) resolvable in any
supported subset.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Rewrites triedb.GenerateTrie as a single partitioned pass that
reconciles stale account.Root fields and rebuilds the trie at the same
time, with 16-way parallelism and crash resume baked in.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The per-call SERVER span ended inside `handleCall()`, so the JSON-RPC
response write happened after the span closed. For large responses like
`engine_getBlobsV*`, that write time was missing from traces.
- Extend the SERVER span past `writeJSON`.
- For batches, add a top-level `jsonrpc.batch` SERVER span (with `rpc.batch.size`) covering the whole batch including `callBuffer.write`.
- Add `rpc.writeJSON` span around the non-batch response write.
- Add `rpc.writeJSONBatch` span around the batch response write.
- Add `rpc.httpWrite` span around the actual HTTP write, separating JSON encoding from network write.
- Add additional telemetry helpers.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is a prerequisite for landing snap v2, the BAL-healing snap sync
algorithm.
It duplicates much of the snap v1 skeleton, which is expected to be
deprecated once v2 is enabled. The code duplication is acceptable as a
short-term tradeoff, simplifying development and reducing integration
complexity.
This PR implements flat-file storage for finalized block access lists,
specifically:
* The freezer is extended with the notion of tail groups, allowing
different groups within a single freezer instance to maintain
independent tails, while all tables within the same group remain
tail-aligned.
* The freezer can now dynamically attach new tables to an existing
freezer instance, with both the table head and tail initialized to the
freezer's common head.
* A new freezer table, **bals**, has been added to the chain freezer
with its own dedicated tail group, preserving the flexibility to deploy
a tail-pruning policy different from the main chain data group.
Additionally, the BALs in the key-value store will be migrated to the
freezer instance once they are finalized or there are at least 90K block
confirmations on top acting as a "soft finalization". This freezing
policy is same with all chain segment data.
Checks the Ledger Ethereum app version before sending typed transactions
that require newer app support.
EIP-2930/EIP-1559 transactions now require Ledger app v1.9.0 or newer,
and EIP-7702 transactions require v1.17.0 or newer. Older apps now
return the same kind of local update error already used for earlier
Ledger feature gates instead of sending an unsupported transaction to
the device.
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Because the UBT doesn't differentiate slots from accounts, the content
of the tree can not be exported as a `GenesisAlloc`, which means that
`evm t8n` can not intergrate it. We have tried integrating the new
format into execution-specs, but this is very hard to maintain because
the team doesn't see it as a priority and their own repository is seeing
a lot of churn. This PR adds the ability to capture the structure of
what is being inserted in the tree, so that the information isn't lost
and it can be dumped in the t8n context.
---------
Co-authored-by: felipe <fselmo2@gmail.com>
The server already rejects empty batches with -32600. On the client
side, calling BatchCallContext with a zero-length slice on inproc/WS/IPC
transports registers no request IDs but the server still replies with an
error message whose id is null. The dispatch loop has no requestOp to
match it to, so op.resp is never written and op.wait blocks until ctx
deadline.
Short-circuit on len(b) == 0 with the same invalidRequestError the
server uses, so all transports return immediately with -32600.
Add a disableGzip parameter to NewHTTPHandlerStack and httpConfig.
initAuth sets it true so compression is disabled in the engine api.
Public HTTP RPC behavior is unchanged.
This is another one of my slop-PRs, aimed at reducing the amount of
future slop PRs by doing it all in one go.
All of the deprecated cli flags have been in that state for over a year.
It's time to remove them, especially since they are ineffective.
Note that I kept the code to report and manage deprecated cli flags, as
I assume we will be deprecating more flags in the future.
### Summary
Closes#34621.
`github.com/pion/dtls/v2` is affected by
[CVE-2026-26014](https://nvd.nist.gov/vuln/detail/CVE-2026-26014); the
fix lives in `github.com/pion/dtls/v3`. In this tree, dtls/v2 is pulled
in indirectly via `github.com/pion/stun/v2 v2.0.0` (declared at
`go.mod:53`), which is the only direct consumer — `p2p/nat/stun.go` is
the sole call site.
`github.com/pion/stun/v3` already uses dtls/v3, so bumping `stun`
upgrades the vulnerable dependency without touching `pion/dtls`
directly.
### API check
The v3 surface used by `p2p/nat/stun.go` is byte-identical in shape to
v2:
| Symbol | v2 | v3 |
|---|---|---|
| `Dial` | `func Dial(network, address string) (*Client, error)` | same
|
| `Build` | `func Build(setters ...Setter) (*Message, error)` | same |
| `TransactionID` | `var TransactionID Setter` | same |
| `BindingRequest` | `var BindingRequest = NewType(MethodBinding,
ClassRequest)` | same |
| `Event` | `type Event struct` | same |
| `XORMappedAddress` | `type XORMappedAddress struct { …
GetFrom(*Message) error }` | same |
| `DefaultPort` | `const DefaultPort = 3478` | same |
So the code change is just the import rename plus an alias rename to
keep the local label honest (`stunV2` → `stunV3`).
### Change
`go.mod` / `go.sum`:
- Replace direct `github.com/pion/stun/v2 v2.0.0` with
`github.com/pion/stun/v3 v3.0.1`.
- `go mod tidy` drops every `pion/dtls/v2` and `pion/stun/v2` entry from
`go.sum` and pulls `pion/dtls/v3 v3.0.7`, `pion/stun/v3 v3.0.1`,
`pion/transport/v3 v3.0.8` as the new indirect set.
`p2p/nat/stun.go`:
- Update the import path and rename the alias from `stunV2` to `stunV3`.
### Verification
- `go build ./p2p/nat/` clean.
- `go test ./p2p/nat/ -count=1` passes (26s).
- `grep 'pion/dtls/v2\|pion/stun/v2' go.sum` returns zero matches.
### Notes
- `pion/dtls` is not imported directly anywhere in the tree, so no other
code needs touching.
- `pion/transport/v3` was already in the dependency graph (the `stun/v3`
upgrade just bumps the patch from v3.0.1 → v3.0.8); the v2 transport
drops out cleanly.
fixes#32672
This is kind of a band aid solution since it fixes the issue by
bypassing the snap sync expectations of an empty db and attempting to
import the new payload if we're at block 1. The next FCU will set the
status to synced.
Will continue looking to better understand how the above issue arises
and find a more thorough solution.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Fixes#35033
## Problem
The GraphQL HTTP handler decoded request bodies directly before
executing the query. Unlike the JSON-RPC HTTP path, `/graphql` did not
have an explicit request body limit before JSON decoding.
A single `Decode` also stops after the first JSON value, so the handler
now requires EOF after the GraphQL request object to ensure oversized
trailing request data is not ignored.
## Changes
- Limit GraphQL request bodies to 5 MiB, matching the existing JSON-RPC
default body limit.
- Return `413 Request Entity Too Large` when the limit is exceeded.
- Require EOF after the request JSON object.
- Add regression coverage for oversized query bodies and oversized
trailing request data.
- Fix an existing GraphQL test fixture that had an unintended trailing
quote after the JSON object.
## Validation
- `gofmt -w graphql/service.go graphql/graphql_test.go`
- `go run golang.org/x/tools/cmd/goimports@latest -w graphql/service.go
graphql/graphql_test.go`
- `go test ./graphql -run TestGraphQLHTTPBodyLimit -count=1`
- `go test ./graphql -count=1`
There is currently no way for JSON-RPC clients to discover which
historical data a node can serve without probing with trial-and-error
calls and interpreting opaque error messages (`pruned history
unavailable`).
This makes it hard to build robust tooling on top of nodes that prune
their history, for example nodes started with `--history.chain
postmerge`
or with reduced `TransactionHistory`, `LogHistory`, or `StateHistory`
windows.
This PR implements `eth_capabilities` as defined in
ethereum/execution-apis#755. The method takes no parameters and returns
the current head plus six per-resource capability records:
- `state`
- `tx`
- `logs`
- `receipts`
- `blocks`
- `stateproofs`
Closes#33828
Archive nodes store the full history of transactions in the index. This PR
fixes a bug for users who provided the NoPruning field in a YAML config file.
Now geth correctly stores full transaction history if archive is configured via
YAML.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Each commit on a PR kicks off a CI run. Those CI jobs run to the finish
regardless, even when new commits have been pushed which make them stale
and useless. This change attempts to cancel any previously running job
for the same PR.
Passing `--state.size-tracking=false` currently cannot disable state
size tracking when it was enabled by the config file because the CLI
path only turns the config value on.
---------
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
The first NODES response sets total = min(int(response.RespCount),
totalNodesResponseLimit), With RespCount=0, total=0 but receive become
1; receive == count is never satisfied.
Return empty raw bytes when the GraphQL `Block.raw` resolver cannot load
the block body. This matches the nil handling used by the other
block-body-backed resolvers and avoids exposing RLP empty-list bytes as
raw block data.
The changes here enable us to fill tests with Amsterdam using geth EVM
bin.
This will be useful for block builder tests using `testing_buildBlockV1`
endpoint and for filling benchmarking compute and stateful tests as
Python is too slow for benchmark tests.
Tested in
[ethereum/execution-specs](https://github.com/ethereum/execution-specs)
with:
```
uv run fill --clean --fork=Amsterdam tests/amsterdam/eip7928_block_level_access_lists/test_block_access_lists.py --evm-bin=$GETH_EVM_PATH
```
- Adds tracing to the `GetBlobsV1/V2/V3`
- Adds `blobs.requested` and `blobs.filled` attributes to
`GetBlobsV1/V2/V3` spans.
- Adds tracing to `BlobTxPool().GetBlobs()`
This PR:
- Adds `engine_newPayloadWithWitnessV5`. The codebase already supports
the previous `VX`, so only `V5` was missing.
- Make the consensus witness format use the field [ordering defined in
the
spec](8d7e68f4b7/src/ethereum/forks/amsterdam/stateless_host_exec_witness.py (L175-L176))
to make it canonical.
cc @gballet
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Fixes a regression where nil results from getBlobs were encoded as an empty array instead of null.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds a fast path for ExecutionPayloadEnvelope and BlobAndProofListV*
that bypasses encoding/json's reflection and re-validation, which are
expensive for large payloads with many blobs. Also hand-rolls the
jsonrpcMessage wire encoding in the RPC codec to avoid a second
re-validation pass when writing responses to the connection.
Resolves#33814
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
Updates the static validation logic to cover additional edge cases
(reflecting the state of the latest devnet branch, except cleaned up
slightly).
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR implements the serving side of the eth71 BAL exchange messages.
Until commit 4cd7092 also contained the requesting side, but since that
part still needs more work, I'm splitting it out into a separate PR.
The test injects BALs directly into rawdb. This can be removed once BAL
generation is integrated into the chain maker.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR finally lands EIP-7928, collecting the block accessList during
the block execution and verifying against the block header.
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR fixes a bug in the current blobpool `Reset` function where it
used the Transaction type instead of blobTxForPool.
Decoding transactions fetched from the pool as Transaction type
caused an error because the blobpool stores blobTxForPool types.
## Summary
The `--rpc.telemetry.sample-ratio` flag declares `Value: 1.0` and `geth
--help` advertises `(default: 1)`. In practice, however, omitting the
flag produces a sample ratio of `0`, causing
`sdktrace.TraceIDRatioBased(0)` to drop 100% of spans. Users who enable
`--rpc.telemetry` see the `OpenTelemetry trace export enabled` log line
and a clean startup, but no traces ever leave the process.
The root cause is the interaction between two pieces of code:
1. `cmd/utils/flags.go:setOpenTelemetry` (added in #34062) only copies
the flag value when `ctx.IsSet(...)` returns true:
```go
if ctx.IsSet(RPCTelemetrySampleRatioFlag.Name) {
tcfg.SampleRatio = ctx.Float64(RPCTelemetrySampleRatioFlag.Name)
}
```
That is the right pattern for "don't clobber a config-file value with
the CLI default," but it implies that something else must initialise the
field when neither source sets it.
2. `node/defaults.go:DefaultConfig` never initialises
`OpenTelemetry.SampleRatio`, leaving it at the float64 zero value.
The result for the common CLI-only user (no TOML config) is `SampleRatio
= 0` → every span is silently dropped, despite the documented default of
1.
## Change
Seed `OpenTelemetry: OpenTelemetryConfig{SampleRatio: 1.0}` in
`node.DefaultConfig` so the documented default matches runtime behavior
and the `ctx.IsSet` guard in `setOpenTelemetry` continues to do what it
was designed to do.
This PR extends the journal to track the pre-transaction values of
mutated balances, nonces, and code.
At the end of the transaction, these values are used to filter out no-op
changes, such as balance transitions from a-> b->a. These changes are
excluded from the block-level access list.
Additionally, there is a dedicated `bal.ConstructionBlockAccessList`
objects for gathering the state reads and writes within the current
transaction. These state writes will be keyed by the block accessList
index.
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This PR introduces OnGasChangeV2 tracing hook, as the pre-requisite for landing
EIP-8037.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
Passing `--v2=false` currently still selects the v2 binding generator
because the command checks whether the flag was set.
This switches generation to use the boolean flag value, so explicit
false continues to generate legacy bindings while `--v2` keeps selecting
v2.
This PR introduces a separate transaction pool type for sparse blobpool.
In sparse blobpool, PooledTransactions message delivers transactions without
blobs, partial or full cells are downloaded by Cells message. Blobpool no longer
stores transactions with complete sidecars, and it stores transactions without
blobs, along with the corresponding cells. Because of this, a dedicated type
distinct from types.Transaction is required.
This PR introduces a type called `BlobTxForPool` and stores each sidecar field
independently, in order to bypass the assumption that a sidecar always exists as
a complete unit.
Reintroducing the conversion queue was considered, but was ultimately omitted
because type conversion should be sufficiently fast. With sparse blobpool, blob
-> cell computation would take about ~13ms per blob. Not sure whether this is
fast enough, but otherwise we can add the conversion queue later on the sparse
blobpool branch.
In b2843a11d, metrics check len(res) == len(hashes) but res is
pre-allocated with make(), so length is always equal. Partial hit metric
never fires. Count non-nil elements instead.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
This is a refactoring PR to wrap all pre/post-execution system calls as
the exported functions, eliminating the duplicated system calls across
the codebase.
There are a few things unchanged but worths highlight:
- ChainMaker is left as unchanged, a significant rewrite is required
- BeaconRoot in header should be non-nil if Cancun is enabled
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
In the --create path, execFunc returns gasLeft as the second return
value, but the rest of the code treats this value as "gas used" (printed
as such, and compared in timedExec). This makes gas reporting incorrect
and can cause benchmark consistency checks to fail.
Every tracer that implements Stop/GetResult held a `reason error` field
that is written by Stop (called from the trace-timeout watchdog
goroutine in api.go) and read by GetResult (called by the RPC handler
main goroutine). These accesses were unsynchronized.
Passing `--dev=false` currently still enters the dev-mode startup path
because a couple of branches check whether the flag was set, not its
boolean value.
This switches those branches to use `ctx.Bool`, so explicit false does
not start dev mode or emit a dev genesis, while `--dev` keeps its
existing behavior.
Removes the appveyor.yml since we moved to github runners.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
I was tracing a signature verification issue in a nocgo build and found
that `VerifySignature` doesn't validate hash length. #33104 added the
check to `Sign` and `sigToPub` but missed this one. The cgo path in
`secp256k1/secp256.go` already rejects non-32-byte hashes, so the nocgo
path should do the same — otherwise a wrong-length hash gets passed to
decred's `Verify` and silently gives a bogus result.
Passing `--graphql=false` currently still registers the GraphQL handler
because the startup path checks whether the flag was set, not its
boolean value.
This switches the registration condition to use `ctx.Bool`, so explicit
false disables GraphQL while the default behavior remains unchanged.
`GetBlobs` returned early when `CellProofsAt` reported
corrupted/out-of-bounds proofs, dropping every blob already collected
and aborting the remaining hashes — a single bad sidecar killed the
whole Engine API batch for consensus clients. Replaced the `return nil,
nil, nil, err` with `log.Error + continue` so the slot stays `nil` per
the sparse-array contract, matching the store/RLP/nil-sidecar branches a
few lines above.
Fixes#34881
This fixes a hang in `Table.waitForNodes`. It is a replacement for PRs
#34890, #33665 which tried to fix the same issue in a different way.
- #34890 doesn't really fix the issue, just makes it less likely
- #33665 tries to fix it by moving the feed send outside of the lock
I created this PR because I want to keep the synchronous node feed
sending in `Table.nodeAdded`.
---------
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This fixes an issue where packets send to the `Unhandled` channel
configured on discv4 could be corrupted when the packet buffer gets
reused.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR makes a small update to the `Pending()` method in the legacy
pool. By changing the lock from exclusive to read-only, it aims to
improve concurrency performance.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This is an alternative PR for
https://github.com/ethereum/go-ethereum/pull/34746.
This PR implements the second approach among the two possible solutions
mentioned in the above PR.
Requests for unavailable items are possible when the peer is following a
different fork from us. However this is not expected to happen
frequently. Considering the amount of complexity added to the codebase,
the simpler approach (this PR) can be preferred.
This PR adds `AdoptSyncedState()` alongside `Enable()`. It does the same
pathdb bookkeeping (now factored into a shared `resetForReactivation()`
helper), but skips the regeneration. The wiring/calling code lands in
#34626
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
When `io.Copy` succeeds but the buffered `Close` fails (e.g. disk full
on `Flush`), the error was swallowed and verification reported a
misleading hash mismatch instead of the real I/O failure. Keep the
`Close` error when `io.Copy` didn't already produce one.
---------
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
## Summary
Fixes#31917.
`geth era-download` now only prints `is stale` when an existing
downloaded file fails checksum verification. Missing files are still
downloaded normally, but no longer get mislabeled as stale.
## Why
`DownloadFile` used `verifyHash` for both missing files and checksum
mismatches, then printed `is stale` for any error. This made first-time
downloads look like corrupt or outdated files.
## Validation
- `make all`
- `go run ./build/ci.go test`
- `go run ./build/ci.go lint`
- `go run ./build/ci.go check_generate`
- `go run ./build/ci.go check_baddeps`
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
The refactor from `for el := plist.Front(); ...; el = el.Next()` to the
new `iterList` iterator in #34743 silently dropped two things needed by
resetTimeout:
1. `nextTimeout = el.Value.(*replyMatcher)` at the top of the loop. This
assignment is what gives `nextTimeout` its documented meaning ("head of
plist when timeout was last reset"), and what makes the early-return
optimization at the top of resetTimeout work. Without it, nextTimeout is
only ever written to nil, so `nextTimeout == plist.Front().Value` is
always false and the optimization is dead.
2. `nextTimeout.errc <- errClockWarp` in the clock-warp branch now reads
a stale or nil pointer. Prior to the refactor, the inner assignment kept
nextTimeout pointing at the current matcher so its errc was the right
channel to receive the errClockWarp signal. After the refactor, on first
entry into the clock-warp branch nextTimeout is nil, which panics the
UDPv4 loop goroutine with a nil pointer deref and takes discv4 down.
Re-assign `nextTimeout = p` at the head of the loop (restoring the
documented invariant) and send the clock-warp error on `p.errc` rather
than the now-stale `nextTimeout.errc`.
The clock-warp branch triggers only when the system clock jumps backward
after a deadline is assigned (deadline - time.Now() >= 2*respTimeout,
i.e. at least ~500ms backward jump), which is why this regression
slipped past CI - it is not exercised by any existing unit test, and
writing one would require plumbing a clock through the loop.
The reconstruct callback indexes parallel response slices (bodies,
receipts). Passing the accept counter used the wrong element when an
earlier header in the same batch hit a stale slot.
the gapped queue cap was effectively per-sender rather than total — a
sender pool spread across enough distinct addresses could grow
`p.gapped` well past `maxGapped`, defeating the resource bound.
`maxGapped` was being compared against `len(p.gapped)`, which is a
`map[address][]tx` and counts unique senders, not queued txs. Switched
the check to `len(p.gappedSource)` (keyed by tx hash, so its length is
the real total). Also wired up a `blobpool/gapped/count` gauge plus
`promoted`, `evicted`, and `gappedfull` meters so queue size and churn
are actually observable in prod.
Disables the recently added log indexer from a simulated backend.
In most cases the log indexer is not required and unindexed search
should be fast enough.
Fixes https://github.com/ethereum/go-ethereum/issues/32552.
This updates the typed `ethclient` model for `eth_simulateV1` call
results to include `maxUsedGas`, matching the field already returned by
the server-side RPC response.
Follow-up to #32789.
The mux tracer fanned out every standard hook to its children but never
forwarded OnSystemCall{Start,End}. Tracers that rely on these - like
`logger.jsonLogger`, which uses the start hook to silence its opcode
hook for the duration of a system call - never got the signal when
wrapped behind a mux.
In evm t8n, combining `--trace` with `--opcode-count` (default for geth
with exec specs) produces exactly that wrapping. The first system call
(e.g. `ProcessBeaconBlockRoot`) then fires `OnOpcode` on the json logger
before any `OnTxStart` has run, dereferencing a nil env and crashing
t8n.
Forward both hooks through the mux. The V2 fan-out falls back to V1 for
children that only implement the legacy hook, mirroring the precedence
already used in `core/state_processor.go`.
This PR addresses one of the biggest performance issue with binary
tries: storing each internal node individually bloats the index, the
disk, and triggers a lot of write amplifications. To fix this issue,
this PR serializes groups of nodes together.
Because we are still looking for the ideal group size, the "depth" of
the group tree is made a parameter, but that will be removed in the
future, once the perfect size is known.
This is a rebase of #33658
---------
Co-authored-by: Copilot <copilot@github.com>
Next() function in RawIterator returned true on decompression errors.Now it
returns false on those cases. Redundant error check on cmd/era/main.go is also
removed.
---------
Co-authored-by: Bosul Mun <bsbs8645@snu.ac.kr>
The layer-5 diff condition used `i > 50 || i < 85`, which is true for
almost all keys in the 0..255 loop. Use `i > 50 && i < 85` so layer 5
only covers the intended band (51..84), consistent with the snapshot
iterator test fix.
- Fixes an error shadowing issue in the deliver() function, where a
stale result from GetDeliverySlot caused the original failure to be
overwritten by errStaleDelivery.
- Adds errInvalidBody and errInvalidReceipt to the downloader error
checks to properly drop peers who sent invalid responses.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
EIP-7825 caps the transaction gas limit at `MaxTxGas`, but after
Amsterdam/EIP-8037 the transaction gas limit can include state gas
reservoir in addition to the regular gas dimension. Applying the Osaka
cap to the full `tx.Gas()` rejects otherwise valid Amsterdam
transactions that need more than `MaxTxGas` total gas because of state
gas, while their regular gas use remains within the intended limit.
This changes geth to stop applying the full transaction gas cap once
Amsterdam is active:
- txpool stateless validation no longer rejects `tx.Gas() > MaxTxGas`
under Amsterdam
- legacy pool reorg cleanup does not purge high-total-gas transactions
at the Osaka transition if Amsterdam is also active
- execution precheck mirrors the txpool behavior and does not reject
high-total-gas messages under Amsterdam
The block gas limit check remains in place, so transactions still cannot
request more total gas than the current block gas limit.
Validation run:
```
go test ./core/txpool ./core/txpool/legacypool
go test ./core -run TestStateProcessorErrors
```
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Here, we change the EVM stack implementation to use an 'arena', i.e.
a shared allocation pool for sub-call stacks. The stack is now more
GC-friendly, since it is a slice of uint256 values instead of a slice of pointers.
Code that pushes an item to the stack has been changed to get() the top
item, then overwrite it.
The PR is a rewrite/rebase of #30362.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Save `el.Next()` before calling `plist.Remove(el)` so iteration
continues correctly. Previously the loop exited after removing the first
expired matcher because `Remove` invalidates the element's links.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
`StateSetWithOrigin.decode()` was missing size computation after
deserializing origin data, causing `size` to remain zero after journal
reload. Added the same calculation logic used in
`NewStateSetWithOrigin()`.
This PR updates the BAL structure definition to the latest the spec,
- Balance has been changed from [16]byte to uint256
- Storage key and value has been changed from [32]byte to uint256
- BlockAccessList has been changed from a struct to a slice of
AccountChanges
- TxIndex has been changed from uint16 to uint32
The stateReadList field introduced by #34776 to track the state access
footprint for EIP-7928 was not propagated by StateDB.Copy. Every other
per-transaction field that lives alongside it (accessList,
transientStorage, journal, witness, accessEvents) is copied explicitly,
so this field was simply missed.
After Copy the copy's stateReadList is nil while the original keeps its
entries, so the nil-safe guards on StateAccessList.AddAccount / AddState
silently drop every access recorded on the copy. For any post-Amsterdam
code path that copies a prepared state and keeps reading from the copy,
the BAL footprint becomes incomplete.
Add a Copy method on bal.StateAccessList and invoke it from
StateDB.Copy, matching the pattern used for accessList and accessEvents.
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
The testPeer request counters (nAccountRequests, nStorageRequests,
nBytecodeRequests, nTrienodeRequests) were plain int fields incremented
with ++. These increments happen in Request* methods that are invoked
concurrently by the Syncer from multiple goroutines
(assignBytecodeTasks, assignStorageTasks, etc.), causing a data race
reliably detected by go test -race.
Change the counters to atomic.Int64 so increments and reads are
synchronized without introducing a mutex.
Fixes races detected in TestMultiSyncManyUseless,
TestMultiSyncManyUselessWithLowTimeout,
TestMultiSyncManyUnresponsive, TestSyncWithStorageAndOneCappedPeer,
TestSyncWithStorageAndCorruptPeer, and
TestSyncWithStorageAndNonProvingPeer.
scheduleFetches.func1 is the biggest allocator in the long-duration
profile of node (11% of total alloc_space).
Each peer-iteration pre-allocated make([]common.Hash, 0, maxTxRetrievals),
even for peers that end up collecting no new hashes (all their announces
were already being fetched by someone else).
Defer the slice allocation to the first append. Peers that collect zero hashes
now pay zero allocation, which is the common case on the timeoutTrigger
path where all peers with any announces are iterated.
When `rpc.Client.Close()` is called, the TCP connection is torn down
without sending a WebSocket Close frame. The server sees `websocket:
close 1006 (abnormal closure): unexpected EOF` instead of a clean 1000
(normal closure).
### Root cause
`websocketCodec.close()` delegates to `jsonCodec.close()` which calls
`c.conn.Close()` — gorilla/websocket's `Conn.Close` explicitly "[closes
the underlying network connection without sending or waiting for a close
message](https://pkg.go.dev/github.com/gorilla/websocket#Conn.Close)"
(per RFC 6455).
### Fix
Send a WebSocket Close control frame (opcode 0x8, status 1000) before
closing the underlying connection. Uses `WriteControl` with the same
`encMu` mutex pattern already used by `pingLoop` for write
serialization, and reuses the existing `wsPingWriteTimeout` (5s)
constant.
`WriteControl` errors are safe to ignore — the connection may already be
broken by the time we attempt the close frame.
Fixes#30482
This PR adds three cell-level kzg functions required for the sparse
blobpool (eth/72).
- VerifyCells: Verifies cells corresponding to proofs. This is used to
verify cells received from eth/72 peers.
- ComputeCells: Computes cells from blobs. This is needed because user
submissions and eth/71 transaction deliveries contain blobs, while
eth/72 peers expect cells.
- RecoverBlobs: Recovers blobs from partial cells. This is needed to
support both eth/71 and eth/72
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
scheduleFetches.func1 is the single biggest allocator in the Pyroscope
profile of a busy node (~13.5 GB/hr, 8% of total alloc_space). Each
peer-iteration pre-allocated 'make([]common.Hash, 0, maxTxRetrievals)'
= 8 KB, even for peers that end up collecting no new hashes (all their
announces were already being fetched by someone else).
Defer the slice allocation to the first append. Peers that collect zero
hashes now pay zero allocation, which is the common case on the
timeoutTrigger path where all peers with any announces are iterated.
New benchmarks BenchmarkScheduleFetches_{100peers_10new,
100peers_allFetching, 500peers_3new} (benchstat, 6 samples):
scenario ns/op B/op allocs/op
100p/10new unchanged unchanged unchanged (fast path)
100p/allFetching -62% -92% -20%
500p/3new -22% -44% -7%
geomean -33% -65% -9%
The rlpx ping command mishandled disconnect responses on two counts:
the error return from rlp.DecodeBytes was ignored, so decode failures
silently produced an "invalid disconnect message" error with no context;
and the decoder assumed the spec-compliant list form exclusively, while
older geth and some other implementations send the reason as a bare
byte.
Accept both wire forms (matching the legacy-tolerant behavior already
in p2p.decodeDisconnectMessage), and on decode failure include the raw
payload so operators can see exactly what the peer sent. Add a unit
test for the decoder covering both forms plus the empty-payload error
path.
This PR reverts the last change to the freebsd build, and it fixes the
_direct_ FreeBSD build.
Here, we change the upstream of github.com/karalabe/hid to its new home,
github.com/ethereum/hid. The new dependency includes a dummy.go file
that makes `go mod vendor` work.
##### Origin of the problem
Enrique is maintaining the FreeBSD ports, and FreeBSD ports only support
vendored go modules. It turns out that `go mod vendor` will not include
C files if there is no `.go` file in the directory. Since the C files
were missing for `karalabe/hid`, the ports maintainer tried to use the
version of `hidapi` that is provided by the ports. To do so, he had to
modify the way things are included. This broke the _out of ports_
FreeBSD build.
Difference to Appveyor:
- Missing 386 build. Hit some issue because user-space memory there is
around 2Gbs. Also seems generally extremely niche.
- Not doing the archive step and NSIS installer and uploads (those are
done on the builder).
This PR removes `FinalizeAndAssemble` from the consensus engine
interface
and relocates block assembly logic outside of the consensus engine.
Block assembly is consensus-agnostic. Most validations can be performed
by the caller. For example:
- Withdrawals must be nil prior to Shanghai
- After Shanghai upgrade, withdrawals must be non-nil, even if empty.
The only notable consensus-specific validation is related to uncles. In
clique,
the concept of uncles does not exist, and any block containing uncles
should
be considered invalid.
Within the block production package, the policy is to produce blocks
according
to the latest chain specification. As a result, Clique-specific block
production
is no longer supported. This tradeoff is considered acceptable.
The nodes were named using the byte representation of the path, instead
of the binary representation. This was confusing to other client devs
trying to achieve interop.
## Summary
- Add `grpc://` and `grpcs://` URL scheme support for OTLP trace export
alongside existing `http://`/`https://`
- The OTLP spec defines two transports: HTTP (port 4318) and gRPC (port
4317). Many observability backends (Jaeger, Tempo, Datadog) prefer gRPC
for lower overhead
- Both `otlptracehttp` and `otlptracegrpc` return `*otlptrace.Exporter`,
so only exporter construction changes — everything downstream (batch
processor, tracer provider, lifecycle) is untouched
- Update flag usage strings to be transport-agnostic
## Example usage
```
geth --rpc.telemetry --rpc.telemetry.endpoint grpc://localhost:4317
geth --rpc.telemetry --rpc.telemetry.endpoint grpcs://tempo-grpc.example.com:443
```
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
clarify that `ReadLastPivotNumber` returns `nil` only when snap sync has
never been attempted, since the marker is written during snap sync and
never cleared.
In the recent refactoring, the state commit logic has been abstracted,
making it more flexible to design state databases for various use cases.
For example, execution-only modes where state mutation is disabled.
As part of this change, the database interface was extended with a
Commit function. However, it currently accepts an unexported struct
`stateUpdate`, which prevents downstream projects from customizing
the state commit behavior.
To address this limitation, the stateUpdate type is now exported.
This PR separates the trie reader to mptTrieReader and ubtTrieReader for
improved readability and extensibility.
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
## Summary
Replace the `BinaryNode` interface with `NodeRef uint32` indices into
typed arena pools, eliminating GC-scanned pointers from binary trie
nodes.
Inspired by [fjl's
observation](https://github.com/ethereum/go-ethereum/pull/34034#issuecomment-4075176446):
> *"if the binary trie produces such a large graph, it should probably
be changed so that the trie node type does not contain pointers. The
runtime does not scan objects that do not contain pointers, so it can
really help with the performance to build it this way."*
### The problem
CPU profiling of the binary trie (EIP-7864) showed **44% of CPU time in
garbage collection**. Each `InternalNode` held two `BinaryNode`
interface values (2 pointer-words each), and the GC scanned every one.
With ~25K `InternalNode`s in memory during block processing, this
created enormous GC pressure.
### The solution
`NodeRef` is a compact `uint32` (2-bit kind tag + 30-bit pool index).
`NodeStore` manages chunked typed pools per node kind:
- **InternalNode pool**: ZERO Go pointers (children are `NodeRef`, hash
is `[32]byte`) → noscan spans
- **HashedNode pool**: ZERO Go pointers → noscan spans
- **StemNode pool**: retains `Values [][]byte` (matching existing
format)
The serialization format is unchanged — flat InternalNode
`[type][leftHash][rightHash]` = 65 bytes.
## Benchmark: Apple M4 Pro (`--benchtime=10s --count=3`, on top of
#34021)
| Metric | Baseline | Arena | Delta |
|--------|----------|-------|-------|
| Approve (Mgas/s) | 374 | 382 | **+2.1%** |
| BalanceOf (Mgas/s) | 885 | 901 | **+1.8%** |
| Approve allocs/op | 775K | **607K** | **-21.7%** |
| BalanceOf allocs/op | 265K | **228K** | **-14.0%** |
## Benchmark: AMD EPYC 48-core (50GB state, execution-specs ERC-20, on
top of #34021 + #34032)
| Benchmark | Baseline | Arena | Delta |
|-----------|----------|-------|-------|
| erc20_approve (write) | 22.4 Mgas/s | **27.0 Mgas/s** | **+20.5%** |
| mixed_sload_sstore | 62.9 Mgas/s | **97.3 Mgas/s** | **+54.7%** |
| erc20_balanceof (read) | 180.8 Mgas/s | 167.6 Mgas/s | -7.3% (cold
cache variance) |
The arena benefit scales with heap size — the EPYC (larger heap, more GC
pressure) shows much larger gains than the M4 Pro (efficient unified
memory). The mixed workload baseline was unstable (62.9 vs 16.3 Mgas/s
between runs due to GC-induced throughput collapse); the arena
eliminates this entirely (95-97 Mgas/s, stable).
## Dependencies
Benchmarked with #34021 (H01 N+1 fix) + #34032 (R14 parallel hashing).
No code dependency — applies independently to master.
All test suites pass (`trie/bintrie` with `-race`, `core/state`,
`triedb/pathdb`, `cmd/geth`).
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
This PR introduces a gasBudget struct to track the available gas for EVM
execution.
With the upcoming EIP-8037, multi-dimensional gas accounting will be
introduced, requiring multiple gas budget counters to be tracked
simultaneously. To support this, the counters are grouped into a gasBudget
structure.
This change is a prerequisite for internal refactoring in preparation
for EIP-8037.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
In openFreezerFileForAppend, if Seek fails after the file is
successfully opened, the file handle is not closed, leaking a
descriptor.
Similarly in newTable, if opening the meta file fails, the
already-opened index file is not closed. And if newMetadata fails, both
the index and meta files are leaked.
Under repeated error conditions (e.g., corrupted filesystem), these
leaks accumulate and may exhaust the OS file descriptor limit, causing
cascading failures.
## Problem
`mustCopyTrie` in `core/state/database.go` panics on any trie type not
in its type switch:
```go
func mustCopyTrie(t Trie) Trie {
switch t := t.(type) {
case *trie.StateTrie:
return t.Copy()
case *transitiontrie.TransitionTrie:
return t.Copy()
default:
panic(fmt.Errorf("unknown trie type %T", t))
}
}
```
On UBT-backed databases (`state.NewUBTDatabase(...)`, used by
`blockchain.go:2124` when the triedb is configured for binary trie),
`StateDB.trie` is `*bintrie.BinaryTrie` — so every `StateDB.Copy()` call
(hit from `statedb.go:699` and the `*trie.StateTrie` branch of
`state_object.go:546`) crashes with `unknown trie type
*bintrie.BinaryTrie`.
## Fix
Add the `*bintrie.BinaryTrie` case. `BinaryTrie.Copy()` already exists
at `trie/bintrie/trie.go:372` and produces a correct deep copy — this
just wires it into the switch.
## Problem
`BinaryTrie.Commit` unconditionally walked every resolved in-memory node
and flushed it into the `NodeSet`, producing one Pebble write per
resolved internal + stem node on every block — even when the node's
on-disk blob was bitwise identical to the previous commit. On a warm
400M-state workload this meant tens of thousands of redundant 65-byte
writes per block, compounding Pebble compaction pressure on every
commit.
The existing `mustRecompute` flag tracks *hash* staleness, not
*disk-blob* staleness: after `Hash()` completes, `mustRecompute` is
cleared even though the fresh blob has not been persisted. It is
therefore insufficient for a skip-flush optimization.
## Fix
Mirror the MPT committer pattern (`trie/committer.go:51-56`) by adding a
`dirty` flag on `InternalNode` and `StemNode` with the semantics *the
on-disk blob is stale*. The flag is:
- set to `true` wherever the node is created or structurally modified
(the same call sites that already set `mustRecompute = true`);
- set to `false` only after the node has been passed to the `flushfn`
inside `CollectNodes`;
- left `false` on nodes produced by `DeserializeNodeWithHash`, matching
the *loaded from disk, already persisted* semantics.
`CollectNodes` short-circuits on `!dirty` subtrees. The propagation
invariant (an ancestor of any dirty node is itself dirty) is already
maintained by the existing `InsertValuesAtStem` / `Insert` paths, which
now mirror every `mustRecompute = true` setter with a `dirty = true`
setter.
## Benchmark
New `BenchmarkCollectNodes_SparseWrite` measures commit cost when only
one leaf changes between blocks — the common case for state updates.
10,000-stem trie, one-leaf modification + Commit per iteration, Apple M4
Pro:
| | before | after | delta |
|---|---|---|---|
| time / op | 12,653,000 ns | 7,336 ns | **~1,725×** |
| bytes / op | 107,224,740 B | 37,774 B | **~2,839×** |
| allocs / op | 80,953 | 134 | **~604×** |
End-to-end impact on a real workload depends on the
resolved-footprint-to-dirty-path ratio; the new
`TestBinaryTrieCommitIncremental` provides a structural regression guard
(asserts that a Commit following a single-leaf modification flushes a
root-to-leaf path, not the whole tree).
---
Found all of this stuff while bloating my #34706 DB to make some
benchmarks. And saw we were spending A LOT OF TIME on hashing.
Hope this helps the perf a bit. Will rebase the flat-state PR on top of
this once merged.
`timedExec` compares errors by direct interface inequality (haveErr !=
err). If execFunc returns newly constructed errors with the same message
each run, this will panic even though behavior is equivalent.
Adds a 'code' exporter to 'geth db export' that iterates over all
contract bytecode entries (CodePrefix + code_hash -> bytecode).
Usage: geth --datadir <dir> db export code code.rlp
This enables exporting contract bytecode.
This Pr implements some prerequisite changes for #34004 : split the
`CachingDB` into a `MerkleDB` and a `UBTDB`, so that very different
behaviors don't clash as much.
The transition isn't handled by this PR, but after talking to Gary we
agreed that `UBTDB` should receive another `triedb`, which will only be
loaded if the `Ended` flag is set to false in the conversion contract.
If this is too hard to achieve, it makes sense to load it regardless,
and then loading can be prevented at a later stage by adding a
`UBTTransitionFinalizationTime` in `ChainConfig`.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
`trace.noreturndata` is documented as "enable return data output" but
the flag name/value imply it disables return data. This is confusing for
users and likely inverted wording. Update the Usage string to reflect
the actual behavior (disable return data output).
Changes the log handler to check for vmodule level overrides
even for messages above the current level. This enables the user to selectively
hide messages from certain packages, among other things.
Also fixes a bug where handler instances created by WithAttr would not follow
the level setting anymore. The WithAttrs method is calledd by slog.Logger.With,
which we also use in go-ethereum to create context specific loggers with
pre-filled attributes. Under the previous implementation of WithAttrs, if the
application created a long-lived logger (for example, for a specific peer), then
that logger would not be affected by later level changes done on the top-level
logger, leading to potentially missed events.
Closes: #30717
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
Auto-enable logic for `StatelessSelfValidation` was reading CLI flag
directly via `ctx.Bool()`, bypassing the merged `cfg.EnableWitnessStats`
value. Now uses `cfg.EnableWitnessStats` so config file settings trigger
the same auto-enable behavior as CLI flags.
StateDB.Commit first commits all storage changes into the storage trie,
then updates the account metadata with the new storage root into the
account trie.
Within StateDB.Commit, the new storage trie root has already been
computed and applied as the storage root. This PR explicitly skips the
redundant storage trie root assignment for readability.
This is a copy of #34721 but against `master` (rather than
`bal-devnet-3`), as requested by @jwasinger, since the slotnum logic now
exists on `master` as well.
This PR simplifies the implementation of EIP-7610 by eliminating the
need to check storage emptiness during contract deployment.
EIP-7610 specifies that contract creation must be rejected if the
destination account has a non-zero nonce, non-empty runtime code, or
**non-empty storage**.
After EIP-161, all newly deployed contracts are initialized with a nonce
of one. As a result, such accounts are no longer eligible as deployment
targets unless they are explicitly cleared.
However, prior to EIP-161, contracts were initialized with a nonce of
zero. This made it possible to end up with accounts that have:
- zero nonce
- empty runtime code
- non-empty storage (created during constructor execution)
- non-zero balance
These edge-case accounts complicate the storage emptiness check.
In practice, contract addresses are derived using one of the following
formulas:
- `Keccak256(rlp({sender, nonce}))[12:]`
- `Keccak256([]byte{0xff}, sender, salt[:], initHash)[12:]`
As such, an existing address is not selected as a deployment target
unless a collision occurs, which is extremely unlikely.
---
Previously, verifying storage emptiness relied on GetStorageRoot.
However, with the transition to the block-based access list (BAL),
the storage root is no longer available, as computing it would require
reconstructing the full storage trie from all mutations of preceding
transactions.
To address this, this PR introduces a simplified approach: it hardcodes
the set of known accounts that have zero nonce, empty runtime code,
but non-empty storage and non-zero balance. During contract deployment,
if the destination address belongs to this set, the deployment is
rejected.
This check is applied retroactively back to genesis. Since no address
collision events have occurred in Ethereum’s history, this change does
not
alter existing behavior. Instead, it serves as a safeguard for future
state
transitions.
This fixes the remaining Hive discv5/FindnodeResults failures in the
cmd/devp2p/internal/v5test fixture.
The issue was in the simulator-side bystander behavior, not in
production discovery logic. The existing fixture could get bystanders
inserted into the remote table, but under current geth behavior they
were not stable enough to remain valid FINDNODE results. In
particular, the fixture still had a few protocol/behavior mismatches:
- incomplete WHOAREYOU recovery
- replies not consistently following the UDP envelope source
- incorrect endpoint echoing in PONG
- fixture-originated PING using the wrong ENR sequence
- bystanders answering background FINDNODE with empty NODES
That last point was important because current lookup accounting can
treat repeatedly unhelpful FINDNODE interactions as failures. As a
result, a bystander could become live via PING/PONG and still later be
dropped from the table before the final FindnodeResults assertion.
This change updates the fixture so that bystanders behave more like
stable discv5 peers:
- perform one explicit initial handshake, then switch to passive response handling
- resend the exact challenged packet when handling WHOAREYOU
- reply to the actual UDP packet source and mirror that source in PONG.ToIP / PONG.ToPort
- use the bystander’s own ENR sequence in fixture-originated PING
- prefill each bystander with the bystander ENR set and answer FINDNODE from that set
The result is that the fixture now forms a small self-consistent lookup
environment instead of a set of peers that are live but systematically
poor lookup participants.
Fixes#34108
The UDPv5 test harness (`newUDPV5Test`) uses the default `PingInterval`
of 3 seconds. When tests like `TestUDPv5_findnodeHandling` insert nodes
into the routing table via `fillTable`, the table's revalidation loop
may schedule PING packets for those nodes. Under the race detector or on
slow CI runners, the test runs long enough for revalidation to fire,
causing background pings to be written to the test pipe. The `close()`
method then finds these as unmatched packets and fails.
The fix sets `PingInterval` to a very large value in the test harness so
revalidation never fires during tests.
Verified locally: 100 iterations with `-race -count=100` pass reliably,
where previously the test would fail within ~50 iterations.
runtime.setDefaults was unconditionally assigning cfg.Random =
&common.Hash{}, which silently overwrote any caller-provided Random
value. This made it impossible to simulate a specific PREVRANDAO and
also forced post-merge rules whenever London was active, regardless of
the intended environment.
This change only initializes cfg.Random when it is nil, matching how
other fields in Config are defaulted. Existing callers that did not set
Random keep the same behavior (a non-nil zero hash still enables
post-merge semantics), while callers that explicitly set Random now get
their value respected.
This fixes a truncation bug that results in an invalid serialization of
empty EIP712.
For example:
```json
{
"method": "eth_signTypedData_v4",
"request": {
"types": {
"EIP712Domain": [
{
"name": "version",
"type": "string"
}
],
"Empty": []
},
"primaryType": "Empty",
"domain": {
"version": "0"
},
"message": {}
}
}
```
When calculating the type-hash for the stuct-hash, it will incorrectly
use `Empty)` instead of `Empty()`
# Summary
Replaces the inline `errors.New("event signature mismatch")` in
generated `UnpackXxxEvent` methods with per-event package-level sentinel
errors (e.g. `ErrTransferSignatureMismatch`,
`ErrApprovalSignatureMismatch`), allowing callers to reliably
distinguish a topic mismatch from a genuine decoding failure via
`errors.Is`.
Each event gets its own sentinel, generated via the abigen template:
```go
var ErrTransferSignatureMismatch = errors.New("event signature mismatch")
```
This scoping is intentional — it allows callers to be precise about
*which* event was mismatched, which is useful when routing logs across
multiple unpackers.
# Motivation
Previously, all errors returned from `UnpackXxxEvent` were
indistinguishable without string matching. This is especially
problematic when processing logs sourced from `eth_getBlockReceipts`,
where a caller receives the full set of logs for a block across all
contracts and event types. In that context, a signature mismatch is
expected and should be skipped, while any other error (malformed data,
topic parsing failure) indicates something is genuinely wrong and should
halt execution:
```go
for _, log := range blockLogs {
event, err := contract.UnpackTransferEvent(log)
if errors.Is(err, gen.ErrTransferSignatureMismatch) {
continue // not our event, expected
}
if err != nil {
return fmt.Errorf("unexpected decode failure: %w", err) // alert
}
// process event
}
```
**Changes:**
- `abigen` template: generates a `ErrXxxSignatureMismatch` sentinel per
event and returns it on topic mismatch instead of an inline error
- Existing generated bindings & testdata: regenerated to reflect the
update
Implements #34075
Return ErrInvalidOpCode with the executing opcode and offending
immediate for forbidden DUPN, SWAPN, and EXCHANGE operands. Extend
TestEIP8024_Execution to assert both opcode and operand for all
invalid-immediate paths.
TestUpdatedKeyfileContents was intermittently failing with:
- Emptying account file failed
- wasn't notified of new accounts
Root cause: waitForAccounts required the account list match and an
immediately readable ks.changes notification in the same instant,
creating a timing race between cache update visibility and channel
delivery.
This change keeps the same timeout window but waits until both
conditions are observed, which preserves test intent while removing the
flaky timing dependency.
Validation:
- go test ./accounts/keystore -run '^TestUpdatedKeyfileContents$'
-count=100
The comment formula showed (i+3) but the code multiplies by 9 (Lsh 3 +
add = 8+1).
This was a error when porting from upstream golang.org/x/crypto/bn256
where ξ=i+3.
Go-ethereum changed the constant to ξ=i+9 but forgot to update the inner
formula.
Two fixes for `testing_buildBlockV1`:
1. Add `omitempty` to `SlotNumber` in `ExecutableData` so it is omitted
for pre-Amsterdam payloads. The spec defines the response as
`ExecutionPayloadV3` which does not include `slotNumber`.
2. Pass `res.fees` instead of `new(big.Int)` in `BuildTestingPayload` so
`blockValue` reflects actual priority fees instead of always being zero.
Corresponding fixture update: ethereum/execution-apis#783
Fix `GetAccount` returning **wrong account data** for non-existent
addresses when the trie root is a `StemNode` (single-account trie) — the
`StemNode` branch returned `r.Values` without verifying the queried
address's stem matches.
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
`BinaryTrie.DeleteAccount` was a no-op, silently ignoring the caller's
deletion request and leaving the old `BasicData` and `CodeHash` in the
trie.
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
This tool is designed for the offline translation of an MPT database to
a binary trie. This is to be used for users who e.g. want to prove
equivalence of a binary tree chain shadowing the MPT chain.
It adds a `bintrie` command, cleanly separating the concerns.
This PR fixes https://github.com/ethereum/go-ethereum/issues/34623 by
changing the `vm.StateDB` interface:
Instead of `EmitLogsForBurnAccounts()` emitting burn logs, `LogsForBurnAccounts()
[]*types.Log` just returns these logs which are then emitted by the caller.
This way when tracing is used, `hookedStateDB.AddLog` will be used
automatically and there is no need to duplicate either the burn log
logic or the `OnLog` tracing hook.
ProcessBeaconBlockRoot (EIP-4788) and processRequestsSystemCall
(EIP-7002/7251) do not merge the EVM access events into the state after
execution. ProcessParentBlockHash (EIP-2935) already does this correctly
at line 290-291.
Without this merge, the Verkle witness will be missing the storage
accesses from the beacon root and request system calls, leading to
incomplete witnesses and potential consensus issues when Verkle
activates.
PathDB keys diff layers by state root, not by block hash. That means a
side-chain block can legitimately collide with an existing canonical diff layer
when both blocks produce the same post-state (for example same parent,
same coinbase, no txs).
Today `layerTree.add` blindly inserts that second layer. If the root
already exists, this overwrites `tree.layers[root]` and appends the same
root to the mutation lookup again. Later account/storage lookups resolve
that root to the wrong diff layer, which can corrupt reads for descendant
canonical states.
At runtime, the corruption is silent: no error is logged and no invariant check
fires. State reads against affected descendants simply return stale data
from the wrong diff layer (for example, an account balance that reflects one
fewer block reward), which can propagate into RPC responses and block
validation.
This change makes duplicate-root inserts idempotent. A second layer with
the same state root does not add any new retrievable state to a tree that is
already keyed by root; keeping the original layer preserves the existing parent
chain and avoids polluting the lookup history with duplicate roots.
The regression test imports a canonical chain of two layers followed by
a fork layer at height 1 with the same state root but a different block hash.
Before the fix, account and storage lookups at the head resolve the fork
layer instead of the canonical one. After the fix, the duplicate insert is
skipped and lookups remain correct.
This PR refactors the encoding rules for `AccessListsPacket` in the wire
protocol. Specifically:
- The response is now encoded as a list of `rlp.RawValue`
- `rlp.EmptyString` is used as a placeholder for unavailable BAL objects
This PR adds `GenerateTrie(db, scheme, root)` to the `triedb` package,
which rebuilds all tries from flat snapshot KV data. This is needed by
snap/2 sync so it can rebuild the trie after downloading the flat state.
The shared trie generation pipeline from `pathdb/verifier.go` was moved
into `triedb/internal/conversion.go` so both `GenerateTrie` and
`VerifyState` reuse the same code.
👋
This PR makes it possible to run "Amsterdam" in statetests. I'm aware
that they'll be failing and not in consensus with other clients, yet,
but it's nice to be able to run tests and see what works and what
doesn't
Before the change:
```
$ go run ./cmd/evm statetest ./amsterdam.json
[
{
"name": "00000019-mixed-1",
"pass": false,
"fork": "Amsterdam",
"error": "unexpected error: unsupported fork \"Amsterdam\""
}
]
```
After
```
$ go run ./cmd/evm statetest ./amsterdam.json
{"stateRoot": "0x25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e"}
[
{
"name": "00000019-mixed-1",
"pass": false,
"stateRoot": "0x25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e",
"fork": "Amsterdam",
"error": "post state root mismatch: got 25b78260b76493a783c77c513125c8b0c5d24e058b4e87130bbe06f1d8b9419e, want 0000000000000000000000000000000000000000000000000000000000000000"
}
]
```
In this PR, the Database interface in `core/state` has been extended
with one more function:
```go
// Iteratee returns a state iteratee associated with the specified state root,
// through which the account iterator and storage iterator can be created.
Iteratee(root common.Hash) (Iteratee, error)
```
With this additional abstraction layer, the implementation details can be hidden
behind the interface. For example, state traversal can now operate directly on
the flat state for Verkle or binary trees, which do not natively support traversal.
Moreover, state dumping will now prefer using the flat state iterator as
the primary option, offering better efficiency.
Edit: this PR also fixes a tiny issue in the state dump, marshalling the
next field in the correct way.
This PR implements the missing functionality for archive nodes by
pruning stale index data.
The current mechanism is relatively simple but sufficient for now:
it periodically iterates over index entries and deletes outdated data
on a per-block basis.
The pruning process is triggered every 90,000 new blocks (approximately
every 12 days), and the iteration typically takes ~30 minutes on a
mainnet node.
This mechanism is only applied with `gcmode=archive` enabled, having
no impact on normal full node.
This PR changes the blsync checkpoint init logic so that even if the
initialization fails with a certain server and an error log message is
printed, the server goes back to its initial state and is allowed to
retry initialization after the failure delay period. The previous logic
had an `ssDone` server state that did put the server in a permanently
unusable state once the checkpoint init failed for an apparently
permanent reason. This was not the correct behavior because different
servers behave differently in case of overload and sometimes the
response to a permanently missing item is not clearly distinguishable
from an overload response. A safer logic is to never assume anything to
be permanent and always give a chance to retry.
The failure delay formula is also fixed; now it is properly capped at
`maxFailureDelay`. The previous formula did allow the delay to grow
unlimited if a retry was attempted immediately after each delay period.
This is a breaking change in the opcode (structLog) tracer. Several fields
will have a slight formatting difference to conform to the newly established
spec at: https://github.com/ethereum/execution-apis/pull/762. The differences
include:
- `memory`: words will have the 0x prefix. Also last word of memory will be padded to 32-bytes.
- `storage`: keys and values will have the 0x prefix.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
Add persistent storage for Block Access Lists (BALs) in `core/rawdb/`.
This provides read/write/delete accessors for BALs in the active
key-value store.
---------
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
In this PR, we add support for protocol version eth/70, defined by EIP-7975.
Overall changes:
- Each response is buffered in the peer’s receipt buffer when the
`lastBlockIncomplete` field is true.
- Continued request uses the same request id of its original
request(`RequestPartialReceipts`).
- Partial responses are verified in `validateLastBlockReceipt`.
- Even if all receipts for partial blocks of the request are collected,
those partial results are not sinked to the downloader, to avoid
complexity. This assumes that partial response and buffering occur only
in exceptional cases.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
`pool.signer.Sender(tx)` bypasses the sender cache used by types.Sender,
which can force an extra signature recovery for every promotable tx
(promotion runs frequently). Use `types.Sender(pool.signer, tx)` here to
keep sender derivation cached and consistent.
This PR enables the block validation of keeper in the womir/openvm zkvm.
It also fixes some issues related to building the executables in CI.
Namely, it activates the build which was actually disabled, and also
resolves some resulting build conflicts by fixing the tags.
Co-authored-by: Leo <leo@powdrlabs.com>
Improve speed of import-history command by two orders of magnitude.
Rework ImportHistory to collect up to 2500 blocks per flush instead of
flushing after each block, reducing database commit overhead.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
conn.read() used the actual UDP packet source address for
codec.Decode(), but conn.write() always used tc.remoteAddr. When the
remote node is reachable via multiple Docker networks, the packet source
IP differs from tc.remoteAddr, causing a session key lookup failure in
the codec.
Use tc.remoteAddr.String() consistently in conn.read() so the session
cache key matches what was used during Encode.
In `setOpenTelemetry`, all other fields (Enabled, Endpoint, AuthUser,
AuthPassword, InstanceID, Tags) are guarded by `ctx.IsSet()` checks, so
they only override the config file when explicitly set via CLI flags.
`SampleRatio` was the only field missing this guard, causing the flag
default (`1.0`) to always overwrite whatever was loaded from the config
file.
- Fix OpenTelemetry `SampleRatio` being unconditionally overwritten by
the CLI flag default value (`1.0`), even when the user did not pass
`--rpc.telemetry.sample-ratio`
- This caused config file values for `SampleRatio` to be silently
ignored
Problem:
The max-initcode sentinel moved from core to vm, but RPC pre-check
mapping still depended on core.ErrMaxInitCodeSizeExceeded. This mismatch
could surface inconsistent error mapping when oversized initcode is
submitted through JSON-RPC.
Solution:
- Remove core.ErrMaxInitCodeSizeExceeded from the core pre-check error
set.
- Map max-initcode validation errors in RPC from
vm.ErrMaxInitCodeSizeExceeded.
- Keep the RPC error code mapping unchanged (-38025).
Impact:
- Restores consistent max-initcode error mapping after the sentinel
move.
- Preserves existing JSON-RPC client expectations for error code -38025.
- No consensus, state, or protocol behavior changes.
Fix incorrect key length calculation for `numHashPairings` in
`InspectDatabase`, introduced in #34000.
The `headerHashKey` format is `headerPrefix + num + headerHashSuffix`
(10 bytes), but the check incorrectly included `common.HashLength`,
expecting 42 bytes.
This caused all number -- hash entries to be misclassified as
unaccounted data.
Fix three issues in the binary trie NodeIterator:
1. Empty nodes now properly backtrack to parent and continue iteration
instead of terminating the entire walk early.
2. `HashedNode` resolver handles `nil` data (all-zeros hash) gracefully
by treating it as Empty rather than panicking.
3. Parent update after node resolution guards against stack underflow
when resolving the root node itself.
---------
Co-authored-by: tellabg <249254436+tellabg@users.noreply.github.com>
## Summary
In binary trie mode, `IntermediateRoot` calls `updateTrie()` once per
dirty account. But with the binary trie there is only one unified trie
(`OpenStorageTrie` returns `self`), so each call redundantly does
per-account trie setup: `getPrefetchedTrie`, `getTrie`, slice
allocations for deletions/used, and `prefetcher.used` — all for the same
trie pointer.
This PR replaces the per-account `updateTrie()` calls with a single flat
loop that applies all storage updates directly to `s.trie`. The MPT path
is unchanged. The prefetcher trie replacement is guarded to avoid
overwriting the binary trie that received updates.
This is the phase-1 counterpart to #34021 (H01). H01 fixes the commit
phase (`trie.Commit()` called N+1 times). This PR fixes the update phase
(`updateTrie()` called N times with redundant setup). Same root cause —
unified binary trie operated on per-account — different phases.
## Benchmark (Apple M4 Pro, 500K entries, `--benchtime=10s --count=3`,
on top of #34021)
| Metric | H01 baseline | H01 + this PR | Delta |
|--------|:------------:|:-------------:|:-----:|
| Approve (Mgas/s) | 368 | **414** | **+12.5%** |
| BalanceOf (Mgas/s) | 870 | 875 | +0.6% |
Should be rebased after #34021 is merged.
EIP-7928 brings state reads into consensus by recording accounts and storage accessed during execution in the block access list. As part of the spec, we need to check that there is enough gas available to cover the cost component which doesn't depend on looking up state. If this component can't be covered by the available gas, we exit immediately.
The portion of the call dynamic cost which doesn't depend on state look ups:
- EIP2929 call costs
- value transfer cost
- memory expansion cost
This PR:
- breaks up the "inner" gas calculation for each call variant into a pair of stateless/stateful cost methods
- modifies the gas calculation logic of calls to check stateless cost component first, and go out of gas immediately if it is not covered.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
\`oss-fuzz.sh\` line 38 writes \`#/bin/sh\` instead of \`#!/bin/sh\` as
the shebang of generated fuzz test runner scripts.
\`\`\`diff
-#/bin/sh
+#!/bin/sh
\`\`\`
Without the \`!\`, the kernel does not recognize the interpreter
directive.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This PR introduces a new type HistoryPolicy which captures user intent
as opposed to pruning point stored in the blockchain which persists the
actual tail of data in the database.
It is in preparation for the rolling history expiry feature.
It comes with a semantic change: if database was pruned and geth is
running without a history mode flag (or explicit keep all flag) geth
will emit a warning but continue running as opposed to stopping the
world.
## Summary
At tree depths below `log2(NumCPU)` (clamped to [2, 8]), hash the left
subtree in a goroutine while hashing the right subtree inline. This
exploits available CPU cores for the top levels of the tree where
subtree hashing is most expensive. On single-core machines, the parallel
path is disabled entirely.
Deeper nodes use sequential hashing with the existing `sync.Pool` hasher
where goroutine overhead would exceed the hash computation cost. The
parallel path uses `sha256.Sum256` with a stack-allocated buffer to
avoid pool contention across goroutines.
**Safety:**
- Left/right subtrees are disjoint — no shared mutable state
- `sync.WaitGroup` provides happens-before guarantee for the result
- `defer wg.Done()` + `recover()` prevents goroutine panics from
crashing the process
- `!bt.mustRecompute` early return means clean nodes never enter the
parallel path
- Hash results are deterministic regardless of computation order — no
consensus risk
## Benchmark (AMD EPYC 48-core, 500K entries, `--benchtime=10s
--count=3`, post-H01 baseline)
| Metric | Baseline | Parallel | Delta |
|--------|----------|----------|-------|
| Approve (Mgas/s) | 224.5 ± 7.1 | **259.6 ± 2.4** | **+15.6%** |
| BalanceOf (Mgas/s) | 982.9 ± 5.1 | 954.3 ± 10.8 | -2.9% (noise, clean
nodes skip parallel path) |
| Allocs/op (approve) | ~810K | ~700K | -13.6% |
Found this bug while implementing the Amsterdam changes t8n changes for
benchmark test filling in EELS.
Prefixes were incorrectly being stripped on requests over t8n and this
was leading to `fill` correctly catching hash mismatches on the EELS
side for some BAL tests. Though this was caught there, I think this
change might as well be cherry-picked there instead and merged to
`master`.
This PR brings this behavior to parity with EELS for Osaka filling.
There are still some quirks with regards to invalid block tests but I
did not investigate this further.
This PR improves the pbss archive mode. Initial sync
of an archive mode which has the --gcmode archive
flag enabled will be significantly sped up.
It achieves that with the following changes:
The indexer now attempts to process histories in batch whenever
possible.
Batch indexing is enforced when the node is still syncing and the local
chain
head is behind the network chain head.
In this scenario, instead of scheduling indexing frequently alongside
block
insertion, the indexer waits until a sufficient amount of history has
accumulated
and then processes it in a batch, which is significantly more efficient.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
## Summary
**Bug fix.** In Verkle mode, all state objects share a single unified
trie (`OpenStorageTrie` returns `self`). During `stateDB.commit()`, the
main account trie is committed via `s.trie.Commit(true)`, which calls
`CollectNodes` to traverse and serialize the entire tree. However, each
dirty account's `obj.commit()` also calls `s.trie.Commit(false)` on the
**same trie object**, redundantly traversing and serializing the full
tree once per dirty account.
With N dirty accounts per block, this causes **N+1 full-tree
traversals** instead of 1. On a write-heavy workload (2250 SSTOREs),
this produces ~131 GB of allocations per block from duplicate NodeSet
creation and serialization. It also causes a latent data race from N+1
goroutines concurrently calling `CollectNodes` on shared `InternalNode`
objects.
This commit adds an `IsVerkle()` early return in `stateObject.commit()`
to skip the redundant `trie.Commit()` call.
## Benchmark (AMD EPYC 48-core, 500K entries, `--benchtime=10s
--count=3`)
| Metric | Baseline | Fixed | Delta |
|--------|----------|-------|-------|
| Approve (Mgas/s) | 4.16 ± 0.37 | **220.2 ± 10.1** | **+5190%** |
| BalanceOf (Mgas/s) | 966.2 ± 8.1 | 971.0 ± 3.0 | +0.5% |
| Allocs/op (approve) | 136.4M | 792K | **-99.4%** |
Resolves the TODO in statedb.go about the account trie commit being
"very heavy" and "something's wonky".
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
cgo builds have been broken in FreeBSD ports because of the hid lib.
@enriquefynn has made a temporary patch, but the fix has been merged in
the master branch, so let's reflect that here.
The signify flag in `doWindowsInstaller` was defined as "signify key"
(with a space), making it impossible to pass via CLI (`-signify
<value>`). This meant the Windows installer signify signing was silently
never executed.
Fix by renaming the flag to "signify", consistent with `doArchive` and
`doKeeperArchive`.
The slotNumber field was being passed as a raw *uint64 to the JSON
marshaler, which serializes it as a plain decimal integer (e.g. 159).
All Ethereum JSON-RPC quantity fields must be hex-encoded per spec.
Wrap with hexutil.Uint64 to match the encoding of other numeric header
fields like blobGasUsed and excessBlobGas.
Co-authored-by: qu0b <stefan@starflinger.eu>
This PR improves `db inspect` classification accuracy in
`core/rawdb/database.go` by tightening key-shape checks for:
- `Block number->hash`
- `Difficulties (deprecated)`
Previously, both categories used prefix/suffix heuristics and could
mis-bucket unrelated entries.
Binary tree hashing is quite slow, owing to many factors. One of them is
the GC pressure that is the consequence of allocating many hashers, as a
binary tree has 4x the size of an MPT. This PR introduces an
optimization that already exists for the MPT: keep a pool of hashers, in
order to reduce the amount of allocations.
Implement EIP7954, This PR raises the maximum contract code size
to 32KiB and initcode size to 64KiB , following https://eips.ethereum.org/EIPS/eip-7954
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
`TestSubscribePendingTxHashes` hangs indefinitely because pending tx
events are permanently missed due to a race condition in
`NewPendingTransactions` (and `NewHeads`). Both handlers called their
event subscription functions (`SubscribePendingTxs`,
`SubscribeNewHeads`) inside goroutines, so the RPC handler returned the
subscription ID to the client before the filter was installed in the
event loop. When the client then sent a transaction, the event fired but
no filter existed to catch it — the event was silently lost.
- Move `SubscribePendingTxs` and `SubscribeNewHeads` calls out of
goroutines so filters are installed synchronously before the RPC
response is sent, matching the pattern already used by `Logs` and
`TransactionReceipts`
<!-- START COPILOT CODING AGENT TIPS -->
---
💬 We'd love your input! Share your thoughts on Copilot coding agent in
our [2 minute survey](https://gh.io/copilot-coding-agent-survey).
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: s1na <1591639+s1na@users.noreply.github.com>
Fixes https://github.com/ethereum/go-ethereum/issues/33907
Notably there is a behavioral change:
- Previously Geth will refuse to restart if the existing trienode
history is gapped with the state data
- With this PR, the gapped trienode history will be entirely reset and
being constructed from scratch
This PR adds a cmd tool fetchpayload which connects to a
node and gets all the information in order to create a serialized
payload that can then be passed to the zkvm.
This PR allows users to prune their nodes up to the Prague fork. It
indirectly depends on #32157 and can't really be merged before eraE
files are widely available for download.
The `--history.chain` flag becomes mandatory for `prune-history`
command. Here I've listed all the edge cases that can happen and how we
behave:
## prune-history Behavior
| From | To | Result |
|-------------|--------------|--------------------------|
| full | postmerge | ✅ prunes |
| full | postprague | ✅ prunes |
| postmerge | postprague | ✅ prunes further |
| postprague | postmerge | ❌ can't unprune |
| any | all | ❌ use import-history |
## Node Startup Behavior
| DB State | Flag | Result |
|-------------|--------------|----------------------------------------------------------------|
| fresh | postprague | ✅ syncs from Prague |
| full | postprague | ❌ "run prune-history first" |
| postmerge | postprague | ❌ "run prune-history first" |
| postprague | postmerge | ❌ "can't unprune, use import-history or fix
flag" |
| pruned | all | ✅ accepts known prune points |
The `--remove.chain` flag incorrectly described itself as selecting
"state data" for removal, which could mislead operators into removing
the wrong data category. This corrects the description to accurately
reflect that the flag targets chain data (block bodies and receipts).
This PR contains two changes:
Firstly, the finalized header will be resolved from local chain if it's
not recently announced via the `engine_newPayload`.
What's more importantly is, in the downloader, originally there are two
code paths to push forward the pivot point block, one in the beacon
header fetcher (`fetchHeaders`), and another one is in the snap content
processer (`processSnapSyncContent`).
Usually if there are new blocks and local pivot block becomes stale, it
will firstly be detected by the `fetchHeaders`. `processSnapSyncContent`
is fully driven by the beacon headers and will only detect the stale pivot
block after synchronizing the corresponding chain segment. I think the
detection here is redundant and useless.
We got a report for a bug in the tracing journal which has the
responsibility to emit events for all state that must be reverted.
The edge case is as follows: on CREATE operations the nonce is
incremented. When a create frame reverts, the nonce increment associated
with it does **not** revert. This works fine on master. Now one step
further: if the parent frame reverts tho, the nonce **should** revert
and there is the bug.
This PR fixes a regression introduced in https://github.com/ethereum/go-ethereum/pull/33836/changes
Before PR 33836, running mainnet would automatically bump the cache size
to 4GB and trigger a cache re-calculation, specifically setting the key-value
database cache to 2GB.
After PR 33836, this logic was removed, and the cache value is no longer
recomputed if no command line flags are specified. The default key-value
database cache is 512MB.
This PR bumps the default key-value database cache size alongside the
default cache size for other components (such as snapshot) accordingly.
I observed failing tests in Hive `engine-withdrawals`:
-
https://hive.ethpandaops.io/#/test/generic/1772351960-ad3e3e460605c670efe1b4f4178eb422?testnumber=146
-
https://hive.ethpandaops.io/#/test/generic/1772351960-ad3e3e460605c670efe1b4f4178eb422?testnumber=147
```shell
DEBUG (Withdrawals Fork on Block 2): NextPayloadID before getPayloadV2:
id=0x01487547e54e8abe version=1
>> engine_getPayloadV2("0x01487547e54e8abe")
<< error: {"code":-38005,"message":"Unsupported fork"}
FAIL: Expected no error on EngineGetPayloadV2: error=Unsupported fork
```
The same failure pattern occurred for Block 3.
Per Shanghai engine_getPayloadV2 spec, pre-Shanghai payloads should be
accepted via V2 and returned as ExecutionPayloadV1:
- executionPayload: ExecutionPayloadV1 | ExecutionPayloadV2
- ExecutionPayloadV1 MUST be returned if payload timestamp < Shanghai
timestamp
- ExecutionPayloadV2 MUST be returned if payload timestamp >= Shanghai
timestamp
Reference:
-
https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md#engine_getpayloadv2
Current implementation only allows GetPayloadV2 on the Shanghai fork
window (`[]forks.Fork{forks.Shanghai}`), so pre-Shanghai payloads are
rejected with Unsupported fork.
If my interpretation of the spec is incorrect, please let me know and I
can adjust accordingly.
---------
Co-authored-by: muzry.li <muzry.li1@ambergroup.io>
Updates go-eth-kzg to
https://github.com/crate-crypto/go-eth-kzg/releases/tag/v1.5.0
Significantly reduces the allocations in VerifyCellProofBatch which is
around ~5% of all allocations on my node
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Reduce allocations in calculation of tx cost.
---------
Co-authored-by: weixie.cui <weixie.cui@okg.com>
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
`GenerateChain` commits trie nodes asynchronously, and it can happen
that some nodes aren't making it to the db in time for `GenerateChain`
to open it and find the data it is looking for.
This is an optimization that existed for verkle and the MPT, but that
got dropped during the rebase.
Mark the nodes that were modified as needing recomputation, and skip the
hash computation if this is not needed. Otherwise, the whole tree is
hashed, which kills performance.
The computation of `MAIN_STORAGE_OFFSET` was incorrect, causing the last
byte of the stem to be dropped. This means that there would be a
collision in the hash computation (at the preimage level, not a hash
collision of course) if two keys were only differing at byte 31.
Eth currently has a flaky test, related to the tx fetcher.
The issue seems to happen when Unsubscribe is called while sub is nil.
It seems that chain.Stop() may be invoked before the loop starts in some
tests, but the exact cause is still under investigation through repeated
runs. I think this change will at least prevent the error.
The BatchSpanProcessor queue size was incorrectly set to
DefaultMaxExportBatchSize (512) instead of DefaultMaxQueueSize (2048).
I noticed the issue on bloatnet when analyzing the block building
traces. During a particular run, the miner was including 1000
transactions in a single block. When telemetry is enabled, the miner
creates a span for each transaction added to the block. With the queue
capped at 512, spans were silently dropped when production outpaced the
span export, resulting in incomplete traces with orphaned spans. While
this doesn't eliminate the possibility of drops under extreme
load, using the correct default restores the 4x buffer between queue
capacity and export batch size that the SDK was designed around.
Pebble maintains a batch pool to recycle the batch object. Unfortunately
batch object must be
explicitly returned via `batch.Close` function. This PR extends the
batch interface by adding
the close function and also invoke batch.Close in some critical code
paths.
Memory allocation must be measured before merging this change. What's
more, it's an open
question that whether we should apply batch.Close as much as possible in
every invocation.
For bal-devnet-3 we need to update the EIP-8024 implementation to the
latest spec changes: https://github.com/ethereum/EIPs/pull/11306
> Note: I deleted tests not specified in the EIP bc maintaining them
through EIP changes is too error prone.
Return the Amsterdam instruction set from `LookupInstructionSet` when
`IsAmsterdam` is true, so Amsterdam rules no longer fall through to the
Osaka jump table.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
In `buildPayload()`, the background goroutine uses a `select` to wait on
the recommit timer, the stop channel, and the end timer. When both
`timer.C` and `payload.stop` are ready simultaneously, Go's `select`
picks a case non-deterministically. This means the loop can enter the
`timer.C` case and perform an unnecessary `generateWork` call even after
the payload has been resolved.
Add a non-blocking check of `payload.stop` at the top of the `timer.C`
case to exit immediately when the payload has already been delivered.
We got a report that after v1.17.0 a geth-teku node starts to time out
on engine_getBlobsV2 after around 3h of operation. The culprit seems to
be our optional http2 service which Teku attempts first. The exact cause
of the timeout is still unclear.
This PR is more of a workaround than proper fix until we figure out the
underlying issue. But I don't expect http2 to particularly benefit
engine API throughput and latency. Hence it should be fine to disable it
for now.
The payload rebuild loop resets the timer with the full Recommit
duration after generateWork returns, making the actual interval
generateWork_elapsed + Recommit instead of Recommit alone.
Since fillTransactions uses Recommit (2s) as its timeout ceiling, the
effective rebuild interval can reach ~4s under heavy blob workloads —
only 1–2 rebuilds in a 6s half-slot window instead of the intended 3.
Fix by subtracting elapsed time from the timer reset.
### Before this fix
```
t=0s timer fires, generateWork starts
t=2s fillTransactions times out, timer.Reset(2s)
t=4s second rebuild starts
t=6s CL calls getPayload — gets the t=2s result (1 effective rebuild)
```
### After
```
t=0s timer fires, generateWork starts
t=2s fillTransactions times out, timer.Reset(2s - 2s = 0)
t=2s second rebuild starts immediately
t=4s timer.Reset(0), third rebuild starts
t=6s CL calls getPayload — gets the t=4s result (3 effective rebuilds)
```
This PR introduces a threshold (relative to current market base fees),
below which we suppress the diffusion of low fee transactions. Once base
fees go down, and if the transactions were not evicted in the meantime,
we release these transactions.
The PR also updates the bucketing logic to be more sensitive, removing
the extra logarithm. Blobpool description is also
updated to reflect the new behavior.
EIP-7918 changed the maximim blob fee decrease that can happen in a
slot. The PR also updates fee jump calculation to reflect this.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
With this, we are dropping support for protocol version eth/68. The only supported
version is eth/69 now. The p2p receipt encoding logic can be simplified a lot, and
processing of receipts during sync gets a little faster because we now transform
the network encoding into the database encoding directly, without decoding the
receipts first.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
fix the flaky test found in
https://ci.appveyor.com/project/ethereum/go-ethereum/builds/53601688/job/af5ccvufpm9usq39
1. increase the timeout from 3+1s to 15s, and use timer instead of
sleep(in the CI env, it may need more time to sync the 1024 blocks)
2. add `synced.Load()` to ensure the full async chain is finished
Signed-off-by: Delweng <delweng@gmail.com>
We didn't upgrade to 1.25, so this jumps over one version. I want to
upgrade all builds to Go 1.26 soon, but let's start with the Docker
build to get a sense of any possible issues.
Previously, handshake timeouts were recorded as generic peer errors
instead of timeout errors. waitForHandshake passed a raw
p2p.DiscReadTimeout into markError, but markError classified errors only
via errors.Unwrap(err), which returns nil for non-wrapped errors. As a
result, the timeoutError meter was never incremented and all such
failures fell into the peerError bucket.
This change makes markError switch on the base error, using
errors.Unwrap(err) when available and falling back to the original error
otherwise. With this adjustment, p2p.DiscReadTimeout is correctly mapped
to timeoutError, while existing behaviour for the other wrapped sentinel
errors remains unchanged
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
inside tx.GasPrice()/GasFeeCap()/GasTipCap() already new a big.Int.
bench result:
```
goos: darwin
goarch: arm64
pkg: github.com/ethereum/go-ethereum/core
cpu: Apple M4
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
TransactionToMessage-10 240.1n ± 7% 175.1n ± 7% -27.09% (p=0.000 n=10)
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
TransactionToMessage-10 544.0 ± 0% 424.0 ± 0% -22.06% (p=0.000 n=10)
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
TransactionToMessage-10 17.00 ± 0% 11.00 ± 0% -35.29% (p=0.000 n=10)
```
benchmark code:
```
// Copyright 2025 The go-ethereum Authors
// This file is part of the go-ethereum library.
//
// The go-ethereum library is free software: you can redistribute it and/or modify
// it under the terms of the GNU Lesser General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// The go-ethereum library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public License
// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
package core
import (
"math/big"
"testing"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/params"
)
// BenchmarkTransactionToMessage benchmarks the TransactionToMessage function.
func BenchmarkTransactionToMessage(b *testing.B) {
key, _ := crypto.GenerateKey()
signer := types.LatestSigner(params.TestChainConfig)
to := common.HexToAddress("0x000000000000000000000000000000000000dead")
// Create a DynamicFeeTx transaction
txdata := &types.DynamicFeeTx{
ChainID: big.NewInt(1),
Nonce: 42,
GasTipCap: big.NewInt(1000000000), // 1 gwei
GasFeeCap: big.NewInt(2000000000), // 2 gwei
Gas: 21000,
To: &to,
Value: big.NewInt(1000000000000000000), // 1 ether
Data: []byte{0x12, 0x34, 0x56, 0x78},
AccessList: types.AccessList{
types.AccessTuple{
Address: common.HexToAddress("0x0000000000000000000000000000000000000001"),
StorageKeys: []common.Hash{
common.HexToHash("0x0000000000000000000000000000000000000000000000000000000000000001"),
},
},
},
}
tx, _ := types.SignNewTx(key, signer, txdata)
baseFee := big.NewInt(1500000000) // 1.5 gwei
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
_, err := TransactionToMessage(tx, signer, baseFee)
if err != nil {
b.Fatal(err)
}
}
}
l
```
From the https://eips.ethereum.org/EIPS/eip-7928
> SELFDESTRUCT (in-transaction): Accounts destroyed within a transaction
MUST be included in AccountChanges without nonce or code changes.
However, if the account had a positive balance pre-transaction, the
balance change to zero MUST be recorded. Storage keys within the self-destructed
contracts that were modified or read MUST be included as a storage_reads
entry.
The storage read against the empty contract (zero storage) should also
be recorded in the BAL's readlist.
https://eips.ethereum.org/EIPS/eip-7928 spec:
> Precompiled contracts: Precompiles MUST be included when accessed.
If a precompile receives value, it is recorded with a balance change.
Otherwise, it is included with empty change lists.
The precompiled contracts are not explicitly touched when they are
invoked since Amsterdam fork.
The fetcher should not fetch transactions that are already on chain.
Until now we were only checking in the txpool, but that does not have
the old transaction. This was leading to extra fetches of transactions
that were announced by a peer but are already on chain.
Here we extend the check to the chain as well.
All five `revert*Request` functions (account, bytecode, storage,
trienode heal, bytecode heal) remove the request from the tracked set
but never restore the peer to its corresponding idle pool. When a
request times out and no response arrives, the peer is permanently lost
from the idle pool, preventing new work from being assigned to it.
In normal operation mode (snap-sync full state) this bug is masked by
pivot movement (which resets idle pools via new Sync() cycles every ~15
minutes) and peer churn (reconnections re-add peers via Register()).
However in scenarios like the one I have running my (partial-stateful
node)[https://github.com/ethereum/go-ethereum/pull/33764] with
long-running sync cycles and few peers, all peers can eventually leak
out of the idle pools, stalling sync entirely.
Fix: after deleting from the request map, restore the peer to its idle
pool if it is still registered (guards against the peer-drop path where
Unregister already removed the peer). This mirrors the pattern used in
all five On* response handlers.
This only seems to manifest in peer-thirstly scenarios as where I find
myself when testing snapsync for the partial-statefull node).
Still, thought was at least good to raise this point. Unsure if required
to discuss or not
Implements the new eth_getStorageValues method. It returns storage
values for a list of contracts.
Spec: https://github.com/ethereum/execution-apis/pull/756
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
The PR exposes the InfuxDB reporting interval as a CLI parameter, which
was previously fixed 10s. Default is still kept at 10s.
Note that decreasing the interval comes with notable extra traffic and
load on InfluxDB.
This changes the challenge resend logic again to use the existing
`ChallengeData` field of `v5wire.Whoareyou` instead of storing a second
copy of the packet in `Whoareyou.Encoded`. It's more correct this way
since `ChallengeData` is supposed to be the data that is used by the ID
verification procedure.
Also adapts the cross-client test to verify this behavior.
Follow-up to #31543
In src/ethereum/forks/amsterdam/vm/interpreter.py:299-304, the caller
address is
only tracked for block level accessList when there's a value transfer:
```python
if message.should_transfer_value and message.value != 0:
# Track value transfer
sender_balance = get_account(state, message.caller).balance
recipient_balance = get_account(state, message.current_target).balance
track_address(message.state_changes, message.caller) # Line 304
```
Since system transactions have should_transfer_value=False and value=0,
this condition is never met, so the caller (SYSTEM_ADDRESS) is not
tracked.
This condition is applied for the syscall in the geth implementation,
aligning with the spec of EIP7928.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds `--opcode.count=<file>` flag to `evm t8n` that writes per-opcode
execution frequency counts to a JSON file (relative to
`--output.basedir`).
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This changes the p2p protocol handlers to delay message decoding. It's
the first part of a larger change that will delay decoding all the way
through message processing. For responses, we delay the decoding until
it is confirmed that the response matches an active request and does not
exceed its limits.
In order to make this work, all messages have been changed to use
rlp.RawList instead of a slice of the decoded item type. For block
bodies specifically, the decoding has been delayed all the way until
after verification of the response hash.
The role of p2p/tracker.Tracker changes significantly in this PR. The
Tracker's original purpose was to maintain metrics about requests and
responses in the peer-to-peer protocols. Each protocol maintained a
single global Tracker instance. As of this change, the Tracker is now
always active (regardless of metrics collection), and there is a
separate instance of it for each peer. Whenever a response arrives, it
is first verified that a request exists for it in the tracker. The
tracker is also the place where limits are kept.
This PR adds OpenTelemetry tracing configuration to geth via
command-line flags. When enabled, geth initializes the global
OpenTelemetry TracerProvider and installs standard trace context
propagation. When disabled (the default), tracing remains a no-op and
behavior is unchanged.
Co-authored-by: Felix Lange <fjl@twurst.com>
The endSpan closure accepted error by value, meaning deferred calls like
defer spanEnd(err) captured the error at defer-time (always nil), not at
function-return time. This meant errors were never recorded on spans.
- Changed endSpan to accept *error
- Updated all call sites in rpc/handler.go to pass error pointers, and
adjusted handleCall to avoid propagating child-span errors to the parent
- Added TestTracingHTTPErrorRecording to verify that errors from RPC
methods are properly recorded on the rpc.runMethod span
I removed `Iterator.Count` in #33840, because it appeared to be unused
and did not provide the documented invariant: the returned count should
always be an upper bound on the number of iterations allowed by `Next`.
In order to make `Count` work, the semantics of `CountValues` has to
change to return the number of items up and including the invalid one. I
have reviewed all callsites of `CountValues` to assess if changing this
is safe. There aren't that many, and the only call that doesn't check
the error and return is in the trie node parser,
`trie.decodeNodeUnsafe`. There, we distinguish the node type based on
the number of items, and it previously returned an error for item count
zero. In order to avoid any potential issue that could result from this
change, I'm adding an error check in that function, though it isn't
necessary.
This changes `RawList` to ensure the count of items is always valid.
Lists with invalid structure, i.e. ones where an element exceeds the
size of the container, are now detected during decoding of the `RawList`
and thus cannot exist.
Also remove `RawList.Empty` since it is now fully redundant, and
`Iterator.Count` since it returns incorrect results in the presence of
invalid input. There are no callers of these methods (yet).
The reasoning for using the cleartext format here is that the JSON-RPC
API is internal only. Providers which expose it publicly already put it
behind a proxy which handles also the encryption.
This fixes two cases where `Iterator.Err()` was misused. The method will
only return an error after `Next()` has returned false, so it makes no
sense to check for the error within the loop itself.
Most uses of the iterator are like this:
it, _ := rlp.NewListIterator(data)
for it.Next() {
do(it.Value())
}
This doesn't require the iterator to be a pointer and it's better to
have it stack-allocated. AFAIK the compiler cannot prove it is OK to
stack-allocate when it is returned as a pointer because the methods of
`Iterator` use pointer receiver and also mutate the object.
The iterator type was not exported until very recently, so I think it is
still OK to change this API.
GetStorage and DeleteStorage used GetBinaryTreeKey to compute the tree
key, while UpdateStorage used GetBinaryTreeKeyStorageSlot. The latter
applies storage slot remapping (header offset for slots <64, main
storage prefix for the rest), so reads and deletes were targeting
different tree locations than writes.
Replace GetBinaryTreeKey with GetBinaryTreeKeyStorageSlot in both
GetStorage and DeleteStorage to match UpdateStorage. Add a regression
test that verifies the write→read→delete→read round-trip for main
storage slots.
The `decodeRef` function used `size > hashLen` to reject oversized
embedded nodes, but this incorrectly allowed nodes of exactly 32 bytes
through. The encoding side (hasher.go, stacktrie.go) consistently uses
`len(enc) < 32` to decide whether to embed a node inline, meaning nodes
of 32+ bytes are always hash-referenced. The error message itself
already stated `want size < 32`, confirming the intended threshold.
Changed `size > hashLen` to `size >= hashLen` in `decodeRef` to align
the decoding validation with the encoding logic, the Yellow Paper spec,
and the surrounding comments.
This PR fixes a panic in a corner case situation when a `ChainEvent` is
received by `eth.Ethereum.updateFilterMapsHeads()` but the given chain
section does not exist in `BlockChain` any more. This can happen during
chain rewind because chain events are processed asynchronously. Ignoring
the event in this case is ok, the final event will point to the final
rewound head and the indexer will be updated.
Note that similar issues will not happen once we transition to
https://github.com/ethereum/go-ethereum/pull/32292 and the new indexer
built on top of this. Until then, the current fix should be fine.
This PR makes `TestEIP8024_Execution` verify explicit error types (e.g.,
`ErrStackUnderflow` vs `ErrInvalidOpCode`) rather than accepting any
error. It also fails fast on unexpected opcodes in the mini-interpreter
to avoid false positives from missing opcode handling.
Here is a draft for the New EraE implementation. The code follows along
with the spec listed at https://hackmd.io/pIZlxnitSciV5wUgW6W20w.
---------
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
### Problem
`HasBody` and `HasReceipts` returned `true` for pruned blocks because
they only checked `isCanon()` which verifies the hash table — but
hash/header tables have `prunable: false` while body/receipt tables have
`prunable: true`.
After `TruncateTail()`, hashes still exist but bodies/receipts are gone.
This caused inconsistency: `HasBody()` returns `true`, but `ReadBody()`
returns `nil`.
### Changes
Both functions now check `db.Tail()` when the block is in ancient store.
If `number < tail`, the data has been pruned and the function correctly
returns `false`.
This aligns `HasBody`/`HasReceipts` behavior with
`ReadBody`/`ReadReceipts` and fixes potential issues in
`skeleton.linked()` which relies on these checks during sync.
Follow-up to https://github.com/ethereum/go-ethereum/pull/33748
Same issue - ResettingTimer can be registered via loadOrRegister() but
GetAll() silently drops it during JSON export. The prometheus exporter
handles it fine (collector.go:70), so this is just an oversight in the
JSON path.
Note: ResettingTimer.Snapshot() resets the timer by design, which is
consistent with how the prometheus exporter uses it.
This adds a new type wrapper that decodes as a list, but does not
actually decode the contents of the list. The type parameter exists as a
marker, and enables decoding the elements lazily. RawList can also be
used for building a list incrementally.
The upstream libray has removed the assembly-based implementation of
keccak. We need to maintain our own library to avoid a peformance
regression.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
kzg4844.Blob is 131072 bytes. Using `for _, blob := range` copies the
entire blob on each iteration. With up to 6 blobs per transaction, this
wastes ~768KB of memory copies.
Switch to index-based iteration and pass pointers directly.
The `Witness` method was not implemented for the binary tree, which
caused `debug_excutionWitness` to panic. This PR fixes that.
Note that the `TransitionTrie` version isn't implemented, and that's on
purpose: more thought must be given to what should go in the global
witness.
I recently went on a longer flight and started profiling the geth block
production pipeline.
This PR contains a bunch of individual fixes split into separate
commits.
I can drop some if necessary.
Benchmarking is not super easy, the benchmark I wrote is a bit
non-deterministic.
I will try to write a better benchmark later
```
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/miner
cpu: Intel(R) Core(TM) Ultra 7 155U
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
BuildPayload-14 141.5µ ± 3% 146.0µ ± 6% ~ (p=0.346 n=200)
│ /tmp/old.txt │ /tmp/new.txt │
│ B/op │ B/op vs base │
BuildPayload-14 188.2Ki ± 4% 177.4Ki ± 4% -5.71% (p=0.018 n=200)
│ /tmp/old.txt │ /tmp/new.txt │
│ allocs/op │ allocs/op vs base │
BuildPayload-14 2.703k ± 4% 2.453k ± 5% -9.25% (p=0.000 n=200)
```
Preallocates slices with known capacity in `stateSet.encode()` and
`StateSetWithOrigin.encode()` methods to eliminate redundant
reallocations during serialization.
Preallocate capacity for `keyOffsets` and `valOffsets` slices in
`decodeRestartTrailer` since the exact size (`nRestarts`) is known
upfront.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
core/state: add bounds check in heap eviction loop
Add len(h) > 0 check before accessing h[0] to prevent potential panic
and align with existing heap access patterns in txpool, p2p, and mclock
packages.
Fix ECIES invalid-curve handling in RLPx handshake (reject invalid
ephemeral pubkeys early)
- Add curve validation in crypto/ecies.GenerateShared to reject invalid
public keys before ECDH.
- Update RLPx PoC test to assert invalid curve points fail with
ErrInvalidPublicKey.
Motivation / Context
RLPx handshake uses ECIES decryption on unauthenticated network input.
Prior to this change, an invalid-curve ephemeral public key would
proceed into ECDH and only fail at MAC verification, returning
ErrInvalidMessage. This allows an oracle on decrypt success/failure and
leaves the code path vulnerable to invalid-curve/small-subgroup attacks.
The fix enforces IsOnCurve validation up front.
Heartbeats are used to drop non-executable transactions from the queue.
The timeout mechanism was not clearly documented, and it was updates
also when not necessary.
This PR restores the previous Pebble configuration, disabling seek compaction.
This feature is still needed by hash mode archive node, mitigating the
overhead of frequent compaction.
Implement standardized JSON format for slow block logging to enable
cross-client performance analysis and protocol research.
This change is part of the Cross-Client Execution Metrics initiative
proposed by Gary Rong: https://hackmd.io/dg7rizTyTXuCf2LSa2LsyQ
The standardized metrics enabled data-driven analysis like the EIP-7907
research: https://ethresear.ch/t/data-driven-analysis-on-eip-7907/23850
JSON format includes:
- block: number, hash, gas_used, tx_count
- timing: execution_ms, total_ms
- throughput: mgas_per_sec
- state_reads: accounts, storage_slots, bytecodes, code_bytes
- state_writes: accounts, storage_slots, bytecodes
- cache: account/storage/code hits, misses, hit_rate
This should come after merging #33522
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Based on [EIP-7864](https://eips.ethereum.org/EIPS/eip-7864), the tree
index should be 32 bytes instead of 31 bytes.
```
def get_tree_key(address: Address32, tree_index: int, sub_index: int):
# Assumes STEM_SUBTREE_WIDTH = 256
return tree_hash(address + tree_index.to_bytes(32, "little"))[:31] + bytes(
[sub_index]
)
```
This is a tweak to the wasm build, that expects the `geth_io` namespace
to expect a `geth_io` module, providing a `len` and `read` methods. This
will be provided by the WASM interface in sp1. This forces an API change
on the OpenVM side, but the interface on their side is still being
designed, so we should proceed with this change, and we'll make a
different tag for OpenVM if this can't work for them.
Co-authored-by: wakabat <wakabat@protonmail.com>
This PR optimizes the historical trie node reader by reworking how data
is accessed and memory is managed, reducing allocation overhead
significantly.
Specifically:
- Instead of decoding an entire history object to locate a specific trie node,
the reader now searches directly within the history.
- Besides, slice pre-allocation can avoid unnecessary deep-copy significantly.
This PR optimizes memory allocation in StateTrie.PrefetchAccount() and
StateTrie.PrefetchStorage() by preallocating slice capacity when the
final size is known.
This PR extends the statistics of contract code read by adding these
fields:
- **CacheHitBytes**: the total number of bytes served by cache
- **CacheMissBytes**: the total number of bytes read on cache miss
- **CodeReadBytes**: the total number of bytes for contract code read
Calling `pool.priced.Removed` is needed to keep is sync with
`pool.all.Remove`.
It was called in other occurances, but not here.
The counter is used for internal heap management. It was working even without this, just not calling reheap at the intended frequency.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR adds metrics that count the number of accounts having transactions
in the txpool. Together with the transaction count this can be used as a
simple indicator of the diversity of transactions in the pool.
Note: as an alternative implementation, we could use a periodic or event
driven update of these Gauges using len.
I've preferred this implementation to match what we have for the pool
sizes.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Fixes#33630
Sort self-destructed addresses before emitting hooks in Finalise() to
ensure deterministic ordering and fix flaky test
TestHooks_OnCodeChangeV2.
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This adds support for Grafana Pyroscope, a continuous profiling solution.
The client is configured similarly to metrics, i.e. run
geth --pyroscope --pyroscope.server=https://...
This commit is a resubmit of #33261 with some changes.
---------
Co-authored-by: Carlos Bermudez Porto <cbermudez.dev@gmail.com>
This PR reverts a part of changes brought by https://github.com/ethereum/go-ethereum/pull/33281/changes
Specifically, read-only protection should always be enforced at the opcode level,
regardless of whether the check has already been performed during gas metering.
It should act as a gatekeeper, otherwise, it is easy to introduce errors by adding
new gas measurement logic without consistently applying the read-only protection.
Adding an RPC flag to limit the block range size for eth_getLogs and
eth_newFilter requests.
closing https://github.com/ethereum/go-ethereum/issues/24508
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
The core part of this PR that we need to adopt is to move the code and
nonce change hook invocations to occur at tx finalization, instead of
when the selfdestruct opcode is called.
Additionally:
* remove `SelfDestruct6780` now that it is essentially the same as
`SelfDestruct` just gated by `is new contract`
* don't duplicate `BalanceIncreaseSelfdestruct` (transfer to recipient
of selfdestruct) in the hooked statedb and in the opcode handler for the
selfdestruct opcode.
* balance is burned immediately when the beneficiary of the selfdestruct
is the sender, and the contract was created in the same transaction.
Previously we emit two balance increases to the recipient (see above
point), and a balance decrease from the sender.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
There's no need to perform the subsequent state access on the target if
we already know that we are out of gas.
This aligns the state access behavior of selfdestruct with EIP-7928
This PR causes execution to terminate at the gas handler in the case of
sstore/call if they are invoked in a static execution context.
This aligns the behavior with EIP 7928 by ensuring that we don't record
any state reads in the access list from an SSTORE/CALL in this
circumstance.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Remove a large amount of duplicate code from the tx_fetcher tests.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
The bitmap is used in compact-encoded trie nodes to indicate which elements
have been modified. The bitmap format has been updated to use big-endian
encoding.
Bit positions are numbered from 0 to 15, where position 0 corresponds to
the most significant bit of b[0], and position 15 corresponds to the least
significant bit of b[1].
This PR adds support for the extraction of OpenTelemetry trace context
from incoming JSON-RPC request headers, allowing geth spans to be linked
to upstream traces when present.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Add Open Telemetry tracing inside the RPC server to help attribute runtime costs within `handler.handleCall()`. In particular, it allows us to distinguish time spent decoding arguments, invoking methods via reflection, and actually executing the method and constructing/encoding JSON responses.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Updated the `avail` calculation to correctly compute remaining capacity:
`buf.limit - len(buf.output)`, ensuring the buffer never exceeds its
configured limit regardless of how many times `Write()` is called.
The coverage build path was generating go test commands with a bogus
-tags flag that held the coverpkg value, so the run kept failing. I
switched coverbuild to treat the optional argument as an override for
-coverpkg and stopped passing coverpkg from the caller. Now the script
emits a clean go test invocation that should actually succeed.
This PR fixes an issue where the tx indexer would repeatedly try to
“unindex” a block with a missing body, causing a spike in CPU usage.
This change skips these blocks and advances the index tail. The fix was
verified both manually on a local development chain and with a new test.
resolves#33371
This PR fixes an issue where `evm statetest` would not verify the
post-state root hash if the test case expected an exception (e.g.
invalid transaction).
The fix involves:
1. Modifying `tests/state_test_util.go` in the `Run` method.
2. When an expected error occurs (`err != nil`), we now check if
`post.Root` is defined.
3. If defined, we recalculate the intermediate root from the current
state (which is reverted to the pre-transaction snapshot upon error).
4. We use `GetChainConfig` and `IsEIP158` to ensure the correct state
clearing rules are applied when calculating the root, avoiding
regressions on forks that require EIP-158 state clearing.
5. If the calculated root mismatches the expected root, the test now
fails.
This ensures that state tests are strictly verified against their
expected post-state, even for failure scenarios.
Fixes issue #33527
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
- pass `rpc.BlockNumberOrHash` directly to `eth_getBlockReceipts` so
`requireCanonical` and other fields survive
- aligns `BlockReceipts` with other `ethclient` methods and re-enables
canonical-only receipt queries
Allow the blobpool to accept blobs out of nonce order
Previously, we were dropping blobs that arrived out-of-order. However,
since fetch decisions are done on receiver side,
out-of-order delivery can happen, leading to inefficiencies.
This PR:
- adds an in-memory blob tx storage, similar to the queue in the
legacypool
- a limited number of received txs can be added to this per account
- txs waiting in the gapped queue are not processed further and not
propagated further until they are unblocked by adding the previos nonce
to the blobpool
The size of the in-memory storage is currently limited per account,
following a slow-start logic.
An overall size limit, and a TTL is also enforced for DoS protection.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This pull request introduces a mechanism to compress trienode history by
storing only the node diffs between consecutive versions.
- For full nodes, only the modified children are recorded in the history;
- For short nodes, only the modified value is stored;
If the node type has changed, or if the node is newly created or
deleted, the entire node value is stored instead.
To mitigate the overhead of reassembling nodes from diffs during history
reads, checkpoints are introduced by periodically storing full node values.
The current checkpoint interval is set to every 16 mutations, though
this parameter may be made configurable in the future.
Fixes#33369
This omits "topics" and "addresses" from the filter when they are unspecified.
It is required for interoperability with some server implementations that cannot
handle `null` for these fields.
It's a PR based on #33303 and introduces an approach for trienode
history indexing.
---
In the current archive node design, resolving a historical trie node at
a specific block
involves the following steps:
- Look up the corresponding trie node index and locate the first entry
whose state ID
is greater than the target state ID.
- Resolve the trie node from the associated trienode history object.
A naive approach would be to store mutation records for every trie node,
similar to
how flat state mutations are recorded. However, the total number of trie
nodes is
extremely large (approximately 2.4 billion), and the vast majority of
them are rarely
modified. Creating an index entry for each individual trie node would be
very wasteful
in both storage and indexing overhead. To address this, we aggregate
multiple trie
nodes into chunks and index mutations at the chunk level instead.
---
For a storage trie, the trie is vertically partitioned into multiple sub
tries, each spanning
three consecutive levels. The top three levels (1 + 16 + 256 nodes) form
the first chunk,
and every subsequent three-level segment forms another chunk.
```
Original trie structure
Level 0 [ ROOT ] 1 node
Level 1 [0] [1] [2] ... [f] 16 nodes
Level 2 [00] [01] ... [0f] [10] ... [ff] 256 nodes
Level 3 [000] [001] ... [00f] [010] ... [fff] 4096 nodes
Level 4 [0000] ... [000f] [0010] ... [001f] ... [ffff] 65536 nodes
Vertical split into chunks (3 levels per chunk)
Level0 [ ROOT ] 1 chunk
Level3 [000] ... [fff] 4096 chunks
Level6 [000000] ... [fffffff] 16777216 chunks
```
Within each chunk, there are 273 nodes in total, regardless of the
chunk's depth in the trie.
```
Level 0 [ 0 ] 1 node
Level 1 [ 1 ] … [ 16 ] 16 nodes
Level 2 [ 17 ] … … [ 272 ] 256 nodes
```
Each chunk is uniquely identified by the path prefix of the root node of
its corresponding
sub-trie. Within a chunk, nodes are identified by a numeric index
ranging from 0 to 272.
For example, suppose that at block 100, the nodes with paths `[]`,
`[0]`, `[f]`, `[00]`, and `[ff]`
are modified. The mutation record for chunk 0 is then appended with the
following entry:
`[100 → [0, 1, 16, 17, 272]]`, `272` is the numeric ID of path `[ff]`.
Furthermore, due to the structural properties of the Merkle Patricia
Trie, if a child node
is modified, all of its ancestors along the same path must also be
updated. As a result,
in the above example, recording mutations for nodes `00` and `ff` alone
is sufficient,
as this implicitly indicates that their ancestor nodes `[]`, `[0]` and
`[f]` were also
modified at block 100.
---
Query processing is slightly more complicated. Since trie nodes are
indexed at the chunk
level, each individual trie node lookup requires an additional filtering
step to ensure that
a given mutation record actually corresponds to the target trie node.
As mentioned earlier, mutation records store only the numeric
identifiers of leaf nodes,
while ancestor nodes are omitted for storage efficiency. Consequently,
when querying
an ancestor node, additional checks are required to determine whether
the mutation
record implicitly represents a modification to that ancestor.
Moreover, since trie nodes are indexed at the chunk level, some trie
nodes may be
updated frequently, causing their mutation records to dominate the
index. Queries
targeting rarely modified trie nodes would then scan a large amount of
irrelevant
index data, significantly degrading performance.
To address this issue, a bitmap is introduced for each index block and
stored in the
chunk's metadata. Before loading a specific index block, the bitmap is
checked to
determine whether the block contains mutation records relevant to the
target trie node.
If the bitmap indicates that the block does not contain such records,
the block is skipped entirely.
Adds BlobTxType and SetCodeTxType to GasPrice switch case, aligning with
`MaxFeePerGas` and `MaxPriorityFeePerGas` handling.
Co-authored-by: m6xwzzz <maskk.weller@gmail.com>
### Description
Add a new `OnStateUpdate` hook which gets invoked after state is
committed.
### Rationale
For our particular use case, we need to obtain the state size metrics at
every single block when fuly syncing from genesis. With the current
state sizer, whenever the node is stopped, the background process must
be freshly initialized. During this re-initialization, it can skip some
blocks while the node continues executing blocks, causing gaps in the
recorded metrics.
Using this state update hook allows us to customize our own data
persistence logic, and we would never skip blocks upon node restart.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Adds missing trienode freezer case to InspectFreezerTable, making it
consistent with InspectFreezer which already supports it.
Co-authored-by: m6xwzzz <maskk.weller@gmail.com>
Fix#33390
`setHeadBeyondRoot` was failing to invalidate finalized blocks because
it compared against the original head instead of the rewound root. This
fix updates the comparison to use the post-rewind block number,
preventing the node from reporting a finalized block that no longer
exists. Also added relevant test cases for it.
This PR removes the version-check command and its associated logic as
discussed in issue #31222.
Removed versionCheckCommand from misccmd.go and main.go.
Deleted version_check.go and its corresponding tests.
Cleaned up testdata/vcheck directory (~800 lines of JSON/signatures
removed).
Verified build with make geth
HeadSync kept reqFinalityEpoch entries for servers after receiving
EvUnregistered, while other per-server maps were cleared. This left
stale request.Server keys reachable from HeadSync, which can lead to a
slow memory leak in setups that dynamically register and unregister
servers.
The fix adds deletion of the reqFinalityEpoch entry in the
EvUnregistered handler. This aligns HeadSync with the cleanup pattern
used by other sync modules and keeps the finality request bookkeeping
strictly limited to currently registered servers.
This pull request optimizes history indexing by splitting a single large
database
batch into multiple smaller chunks.
Originally, the indexer will resolve a batch of state histories and
commit all
corresponding index entries atomically together with the indexing
marker.
While indexing more state histories in a single batch improves
efficiency, excessively
large batches can cause significant memory issues.
To mitigate this, the pull request splits the mega-batch into several
smaller batches
and flushes them independently during indexing. However, this introduces
a potential
inconsistency that some index entries may be flushed while the indexing
marker is not,
and an unclean shutdown may leave the database in a partially updated
state.
This can corrupt index data.
To address this, head truncation is introduced. After a restart, any
excessive index
entries beyond the expected indexing marker are removed, ensuring the
index remains
consistent after an unclean shutdown.
This is a new step in my crusade against the braindead fad of starting
PR titles with a word that is completely redundant with github labels,
thus wasting prime first-line real-estate for something that isn't
necessary.
I noticed that every single one of these PRs are low-quality AI-slop, so
I think there is a strong case to be made for these PRs to be
auto-closed. A message is added before closing the PR, redirecting to
our contribution guidelines, so I expect quality first-time contributors
to read them and reopen the PR. In the case of spam PRs, the author is
unlikely to revisit a given PR, and so auto-closing might have a
positive impact. That's an experiment worth trying, imo.
In order to reduce the amount of code that is embedded into the keeper
binary, I am removing all the verkle code that uses go-verkle and
go-ipa. This will be followed by further PRs that are more like stubs to
replace code when the keeper build is detected.
I'm keeping the binary tree of course. This means that you will still
see `isVerkle` variables all over the codebase, but they will be renamed
when code is touched (i.e. this is not an invitation for 30+ AI slop
PRs).
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
In this PR, two things have been fixed:
---
(a) truncate the stale beacon headers with latest snap block
Originally, b.filled is used as the indicator for deleting stale beacon headers.
This field is set only after synchronization has been scheduled, under the
assumption that the skeleton chain is already linked to the local chain.
However, the local chain can be mutated via `debug_setHead`, which may
cause `b.filled` outdated. For instance, `b.filled` refers to the last head snap block
in the last sync cycle while after `debug_setHead`, the head snap block has been
rewounded to 1.
As a result, Geth can enter an unintended loop: it repeatedly downloads
the missing beacon headers for the skeleton chain and attempts to schedule the
actual synchronization, but in the final step, all recently fetched headers are removed
by `cleanStales` due to the stale `b.filled` value.
This issue is addressed by always using the latest snap block as the indicator,
without relying on any cached value. However, note that before the skeleton
chain is linked to the local chain, the latest snap block will always be below
skeleton.tail, and this condition should not be treated as an error.
---
(b) merge the subchains once the skeleton chain links to local chain
Once the skeleton chain links with local one, it will try to schedule the
synchronization by fetching the missing blocks and import them then.
It's possible the last subchain already overwrites the previous subchain and
results in having two subchains leftover. As a result, an error log will printed
https://github.com/ethereum/go-ethereum/blob/master/eth/downloader/skeleton.go#L1074
Blobs are stored per transaction in the pool, so we need billy to handle
up to the per-tx limit, not to the per-block limit.
The per-block limit was larger than the per-tx limit, so it not a bug,
we just created and handled a few billy files for no reason.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR removes the legacy sidecar conversion logic.
After the Osaka fork, the blobpool will accept only blob sidecar version
1.
Any remaining version 0 blob transactions, if they still exist, will no
longer
be eligible for inclusion.
Note that conversion at the RPC layer is still supported, and version 0
blob
transactions will be automatically converted to version 1 there.
When iterating over a map with value types in Go, the loop variable is a
copy. In `markCodeExistence`, assigning to `code.exists` modified only
the local copy, not the actual map entry, causing the existence flag to
always remain false.
This resulted in overcounting contract codes in state size statistics,
as codes that already existed in the database were incorrectly counted
as new.
Fix by changing `codes` from `map[common.Address]contractCode` to
`map[common.Address]*contractCode`, so mutations apply directly to the
struct.
The fuzz test file has been broken for a while - it doesn't compile with
the `gofuzz` build tag.
Two issues:
- Line 59: called `SignifySignFile` which doesn't exist (should be
`SignFile`)
- Line 71: used `:=` instead of `=` for already declared `err` variable
This PR fixes the bug reported in #33365.
The impact of the bug is not catastrophic. After a transaction is
ultimately fetched, validation and propagation will be performed based
on the fetched body, and any response with a mismatched type is treated
as a protocol violation. An attacker could only waste the limited
portion of victim’s bandwidth at most.
However, the reasons for submitting this PR are as follows
1. Fetching a transaction announced with an arbitrary type is a weird
behavior.
2. It aligns with efforts such as EIP-8077 and #33119 to make the
fetcher smarter and reduce bandwidth waste.
Regarding the `FilterType` function, it could potentially be implemented
by modifying the Filter function's parameter itself, but I wasn’t sure
whether changing that function is acceptable, so I left it as is.
The simulator computed active precompiles from the base header, which is
incorrect when simulations cross fork boundaries. This change selects
precompiles using the current simulated header so the precompile set
matches the block’s number/time. It brings simulate in line with doCall,
tracing, and mining, and keeps precompile state overrides applied on the
correct epoch set.
## Description
This PR fixes incorrect contract code state metrics by ensuring
duplicate codes are not counted towards the reported results.
## Rationale
The contract code metrics don't consider database deduplication. The
current implementation assumes that the results are only **slightly
inaccurate**, but this is not true, especially for data collection
efforts that started from the genesis block.
Fixes an issue where HashFolder skipped the root directory upon hitting
the first file in the excludes list. This happened because the walk function
returned SkipDir even for regular files.
This moves the tracking of the current syncmode into the downloader, fixing an
issue where the syncmode being requested through the engine API could go
out-of-sync with the actual mode being performed by downloader.
Fixes#32629
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
The EIP says to increment PC by 2 _instead of_ the standard increment by
1. The opcode handlers added in #33095 result in incrementing PC by 3,
because they ignored the increment already present in `interpreter.go`.
Does this need to be better specified in the EIP? I've added a [new test
case](https://github.com/ethereum/EIPs/pull/10859) for it anyway.
Found by @0xriptide.
XORBytes was added to package crypto/subtle in Go 1.20, and it's faster
than our bitutil.XORBytes. There is only one use of this function
across go-ethereum so we can simply deprecate the custom implementation.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
The random-port retry loop in addAnyPortMapping shadowed the err
variable, causing the function to return (0, nil) when all attempts
failed. This change removes the shadowing and preserves the last error
across both the fixed-port and random-port retries, ensuring failures
are reported to callers correctly.
This PR changes the Pebble configurations as below:
- increase the MemTableStopWritesThreshold for handling temporary spike
- decrease the L0CompactionConcurrency and CompactionDebtConcurrency to
scale up compaction readily
The original condition `deleted && !logPrinted || time.Since(...)` was
incorrectly grouping due to operator precedence, causing logs to print
every 10 seconds even when no deletion was happening (deleted=false).
According to SafeDeleteRange documentation, the 'deleted' parameter is
"true if entries have actually been deleted already". The logging should
only happen when deletion is active.
Fixed by adding parentheses: `deleted && (!logPrinted ||
time.Since(...))`Now logs print only when items are being deleted AND
either it's the first log or 10+ seconds have passed since the last one.
This improves the error code for cases where invalid query parameters
are submitted to `eth_getLogs`. I also improved the error message that
is emitted when querying into the future.
This is to benchmark how much the internal parts of GetBlobsV2 take.
This is not an RPC-level benchmark, so JSON-RPC overhead is not
included.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR exposes the state size statistics to the metrics, making them
easier to demonstrate.
Note that the contract code included in the metrics is not
de-duplicated, so the reported size
will appear larger than the actual storage footprint.
This introduces two main changes to Pebble's configuration:
(a) Remove the Bloom filter at Level 6
The Bloom filter is never used at the bottom-most level, so keeping it
serves no purpose. Removing it saves storage without affecting read
performance.
(b) Re-enable read-sampling compaction
Read-sampling compaction was previously disabled in the hash-based
scheme because all data was identified by hashes and basically no data
overwrite. Read sampling compaction makes no sense.
After switching to the path-based scheme, data overwrites are much more
common, making read-sampling compaction beneficial and reasonable to re-enable.
This PR introduces a new debug feature, logging the slow blocks with
detailed performance statistics, such as state read, EVM execution and
so on.
Notably, the detailed performance statistics of slow blocks won't be
logged during the sync to not overwhelm users. Specifically, the statistics
are only logged if there is a single block processed.
Example output
```
########## SLOW BLOCK #########
Block: 23537063 (0xa7f878611c2dd27f245fc41107d12ebcf06b4e289f1d6acf44d49a169554ee09) txs: 248, mgasps: 202.99
EVM execution: 63.295ms
Validation: 1.130ms
Account read: 6.634ms(648)
Storage read: 17.391ms(1434)
State hash: 6.722ms
DB commit: 3.260ms
Block write: 1.954ms
Total: 99.094ms
State read cache: account (hit: 622, miss: 26), storage (hit: 1325, miss: 109)
##############################
```
We still default to legacy txes for methods like eth_sendTransaction,
eth_signTransaction. We can default to 0x2 and if someone would like to
stay on legacy they can do so by setting the `gasPrice` field.
cc @deffrian
Recently in #31630 we removed support for overriding the network id in
preset networks. While this feature is niche, it is useful for shadow
forks. This PR proposes we add the functionality back, but in a simpler
way.
Instead of checking whether the flag is set in each branch of the
network switch statement, simply apply the network flag after the switch
statement is complete. This retains the following behavior:
1. Auto network id based on chain id still works, because `IsSet` only
returns true if the flag is _actually_ set. Not if it just has a default
set.
2. The preset networks will set their network id directly and only if
the network id flag is set is it overridden. This, combined with the
override genesis flag is what allows the shadow forks.
3. Setting the network id to the same network id that the preset _would
have_ set causes no issues and simply emits the `WARN` that the flag is
being set explicitly. I don't think people explicitly set the network id
flag often.
```
WARN [10-22|09:36:15.052] Setting network id with flag id=10
```
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds checks into getPayload to ensure the correct version is called
for the fork which applies to the payload.
---------
Co-authored-by: jsvisa <delweng@gmail.com>
This was found because other clients are failing RPC tests generated by
Geth. Nethermind and Besu return the correct error code, -32602, in this
situation.
Enable blocktest to read filenames from stdin when no path argument is
provided, matching the existing statetest behavior. This allows
efficient batch processing of blockchain tests.
Usage:
- Single file: evm blocktest <path>
- Batch mode: find tests/ -name "*.json" | evm blocktest
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
No matter what value of P2P.DiscoveryV4 or DiscoveryV5 is set in config file,
it will be overwritten by the CLI flag, even if the flag is not set. This fixes it
to apply the flag only if set.
Fixes error messages to print the actual blob gas value instead of the
pointer address by dereferencing `ExcessBlobGas`, `BlobGasUsed` and
`ParentBeaconRoot`.
Bumps
[github.com/consensys/gnark-crypto](https://github.com/consensys/gnark-crypto)
from 0.18.0 to 0.18.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/consensys/gnark-crypto/releases">github.com/consensys/gnark-crypto's
releases</a>.</em></p>
<blockquote>
<h2>v0.18.1</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/Consensys/gnark-crypto/compare/v0.18.0...v0.18.1">https://github.com/Consensys/gnark-crypto/compare/v0.18.0...v0.18.1</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/Consensys/gnark-crypto/blob/master/CHANGELOG.md">github.com/consensys/gnark-crypto's
changelog</a>.</em></p>
<blockquote>
<h2>[v0.18.1] - 2025-10-28</h2>
<h3>Docs</h3>
<ul>
<li>add CHANGELOG for 0.18.1</li>
</ul>
<h3>Perf</h3>
<ul>
<li>limit memory allocation during Vector deserialization (<a
href="https://redirect.github.com/Consensys/gnark-crypto/issues/759">#759</a>)</li>
</ul>
<p><!-- raw HTML omitted --><!-- raw HTML omitted --></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="fb04e95c3b"><code>fb04e95</code></a>
docs: add CHANGELOG for 0.18.1</li>
<li><a
href="0a4d04ae62"><code>0a4d04a</code></a>
perf: limit memory allocation during Vector deserialization (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/759">#759</a>)</li>
<li>See full diff in <a
href="https://github.com/consensys/gnark-crypto/compare/v0.18.0...v0.18.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ethereum/go-ethereum/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This change fixes a stall in the legacy blob sidecar conversion pipeline
where tasks that arrived during an active batch could remain unprocessed
indefinitely after that batch completed, unless a new external event
arrived.
The root cause was that the loop did not restart processing in
the case <-done: branch even when txTasks had accumulated work, relying
instead on a future event to retrigger the scheduler. This behavior is
inconsistent with the billy task pipeline, which immediately chains to
the next task via runNextBillyTask() without requiring an external trigger.
The fix adds a symmetric restart path in `case <-done`: that checks
`len(txTasks) > 0`, clones the accumulated tasks, clears the queue, and
launches a new run with a fresh done and interrupt.
This preserves batching semantics, prevents indefinite blocking of callers
of convert(), and remains safe during shutdown since the quit path
still interrupts and awaits the active batch. No public interfaces or logging
were changed.
Fixes error messages to print the actual blob gas value instead of the
pointer address by dereferencing `ExcessBlobGas`, `BlobGasUsed` and
`ParentBeaconRoot`.
A new pointless fad appeared recently where people just create a fairly
low information tag at the beginning of their github PR titles.
Something like `feat` or other keywords.
This seems to originate from the angular community and to be used for
automation scripts over there. We do not use any of those scripts and if
we did we would be using the github labels, which offer strictly
equivalent functionalities without wasting useful PR title space.
In order for these keywords to fail the validation, I am adding a check
that these directories listed indeed exist in the repository.
Looks like (in some very EVM specific tests) we spent a lot of time
resizing memory. If the underlying array is big enough, we can speed it
up a bit by simply slicing the memory.
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/core/vm
cpu: Intel(R) Core(TM) Ultra 7 155U
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
Resize-14 6.145n ± 9% 1.854n ± 14% -69.83% (p=0.000 n=10)
│ /tmp/old.txt │ /tmp/new.txt │
│ B/op │ B/op vs base │
Resize-14 5.000 ± 0% 5.000 ± 0% ~ (p=1.000 n=10)
│ /tmp/old.txt │ /tmp/new.txt │
│ allocs/op │ allocs/op vs base │
Resize-14 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
From the blocktest benchmark:
620ms 10.93s (flat, cum) 9.92% of Total
. . 80:func (m *Memory) Resize(size uint64) {
30ms 60ms 81: if uint64(m.Len()) < size {
590ms 10.87s 82: m.store = append(m.store, make([]byte, size-uint64(m.Len()))...)
. . 83: }
. . 84:}
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
EIP-8024: Backward compatible SWAPN, DUPN, EXCHANGE
Introduces additional instructions for manipulating the stack which
allow accessing the stack at higher depths. This is an initial implementation
of the EIP, which is still in Review stage.
Adds a flag to specify how many blobs a node is willing to include in
their locally build block as specified in
https://eips.ethereum.org/EIPS/eip-7872
I deviated from the EIP in one case, I allowed for specifying 0 as the
minimum blobs/block
The list iterator previously returned true on parse errors without
advancing the input, which could lead to non-advancing infinite loops
for callers that do not check Err() inside the loop; to make iteration
safe while preserving error visibility, Next() now marks the iterator as
finished when readKind fails, returning true for the error step so
existing users that check Err() can handle it, and then false on
subsequent calls, and the function comment was updated to document this
behavior and the need to check Err().
This PR adds the "FULU" beacon chain config entries for all networks and
fixes the select statements that choose the appropriate engine API call
versions (no new version there but the "default" was always the first
version; now it's the latest version so no need to change unless there
is actually a new version).
New beacon checkpoints are also added for mainnet, sepolia and hoodi
(not for holesky because it's not finalizing at the moment).
Note that though unrelated to fusaka, the log indexer checkpoints are
also updated for mainnet (not for the other testnets, mainly because I
only have mainnet synced here on my travel SSD; this should be fine
though because the index is also reverse generated for a year by default
so it does not really affect the indexing time)
Links for the new checkpoints:
https://beaconcha.in/slot/13108192https://light-sepolia.beaconcha.in/slot/9032384https://hoodi.beaconcha.in/slot/1825728
This change introduces an iterator for the history index in the pathdb.
It provides sequential access to historical entries, enabling efficient
scanning and future features built on top of historical state traversal.
Fix#33212.
This PR remove `github.com/olekukonko/tablewriter` from dependencies and
use a naive stub implementation.
`github.com/olekukonko/tablewriter` is used to format database inspection
output neatly. However, it requires custom adjustments for TinyGo and is
incompatible with the latest version.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This is broken off of #31730 to only focus on testing networks that
start with verkle at genesis.
The PR has seen a lot of work since its creation, and it now targets
creating and re-executing tests for a binary tree testnet without the
transition (so it starts at genesis). The transition tree has been moved
to its own package. It also replaces verkle with the binary tree for
this specific application.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The iterator loop in findTxInBlockBody returned the outer-scoped err
when iter.Err() was non-nil, which could incorrectly propagate a nil or
stale error and hide actual RLP decoding issues. This patch returns
iter.Err() as intended by the rlp list iterator API, matching
established patterns elsewhere in the codebase and improving diagnostics
when encountering malformed transaction entries.
While updating to latest Geth, I noticed `OnCodeChangeV2` was not
properly handled in `SelfDestruct/6780`, this PR fixes this and bring a
unit test. Let me know if it's deemed more approriate to merge the tests
with the other one.
[powdr](github.com/powdr-labs/powdr) has tested keeper in their womir
system and managed to get it to work. This PR adds wasm as a keeper
target. There's another plan by the zkevm team to support wasm with wasi
as well, so these PR adds both targets.
These currently uses the `example` tag, as there is no precompile
intefrace defined for either target yet. Nonetheless, this is useful for
testing these zkvms so it makes sense to support these experimental
targets already.
The periodic sealing loop failed to reset its timer when sealBlock
returned an error, causing the timer to never fire again and effectively
halting block production in developer periodic mode after the first
failure. This is a bug because the loop relies on the timer to trigger
subsequent sealing attempts, and transient errors (e.g., pool races or
chain rewinds) should not permanently stop the loop. The change moves
timer.Reset after the sealing attempt unconditionally, ensuring the loop
continues ticking and retrying even when sealing fails, which matches
how other periodic timers in the codebase behave and preserves forward
progress.
The version check incorrectly used `&&` instead of `||`, causing
versions like v1.0.x through v1.4.x to be allowed when they should be
rejected. These versions don't support EIP-712 signing which was
introduced in firmware v1.5.0.
Because the map iteration is unstable, we need to order logs by tx index
and keep the same order with receipts and their logs, so we can still
get the same `LogsHash` across runs.
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This pull request updates `PrettyAge.String` so that the age formatter
now treats exact unit boundaries (like a full day or week) as that unit
instead of spilling into smaller components, keeping duration output
aligned with human expectations.
This adds two new CI targets. One is for building all supported keeper
executables, the other is for running unit tests on 32-bit Linux.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
## Description
- Summary: Correct the JS timer callback argument forwarding to match
standard JS semantics.
- What changed: In `internal/jsre/jsre.go`, the callback is now invoked
with only the arguments after the callback and delay.
- Why: Previously, the callback received the function and delay as
parameters, causing unexpected behavior and logic bugs for consumers.
Equal is called every time the transaction sender is accessed,
even when the sender is cached, so it is worth optimizing.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In this PR, several changes have been made:
(a) restructure the trienode history header section
Previously, the offsets of the key and value sections were recorded before
encoding data into these sections. As a result, these offsets referred to the
start position of each chunk rather than the end position.
This caused an issue where the end position of the last chunk was
unknown, making it incompatible with the freezer partial-read APIs.
With this update, all offsets now refer to the end position, and the
start position of the first chunk is always 0.
(b) Enable partial freezer read for trienode data retrieval
The partial freezer read feature is now utilized in trienode data
retrieval, improving efficiency.
At the time keeper support was added into ci.go, we were using a go.work
file to make ./cmd/keeper accessible from within the main go-ethereum
module. The workspace file has since been removed, so we need to build
keeper from within its own module instead.
Found in
https://github.com/ethereum/go-ethereum/actions/runs/17803828253/job/50611300621?pr=32585
```
--- FAIL: TestClientCancelWebsocket (0.33s)
panic: read tcp 127.0.0.1:36048->127.0.0.1:38643: read: connection reset by peer [recovered, repanicked]
goroutine 15 [running]:
testing.tRunner.func1.2({0x98dd20, 0xc0005b0100})
/opt/actions-runner/_work/_tool/go/1.25.1/x64/src/testing/testing.go:1872 +0x237
testing.tRunner.func1()
/opt/actions-runner/_work/_tool/go/1.25.1/x64/src/testing/testing.go:1875 +0x35b
panic({0x98dd20?, 0xc0005b0100?})
/opt/actions-runner/_work/_tool/go/1.25.1/x64/src/runtime/panic.go:783 +0x132
github.com/ethereum/go-ethereum/rpc.httpTestClient(0xc0001dc1c0?, {0x9d5e40, 0x2}, 0xc0002bc1c0)
/opt/actions-runner/_work/go-ethereum/go-ethereum/rpc/client_test.go:932 +0x2b1
github.com/ethereum/go-ethereum/rpc.testClientCancel({0x9d5e40, 0x2}, 0xc0001dc1c0)
/opt/actions-runner/_work/go-ethereum/go-ethereum/rpc/client_test.go:356 +0x15f
github.com/ethereum/go-ethereum/rpc.TestClientCancelWebsocket(0xc0001dc1c0?)
/opt/actions-runner/_work/go-ethereum/go-ethereum/rpc/client_test.go:319 +0x25
testing.tRunner(0xc0001dc1c0, 0xa07370)
/opt/actions-runner/_work/_tool/go/1.25.1/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
/opt/actions-runner/_work/_tool/go/1.25.1/x64/src/testing/testing.go:1997 +0x465
FAIL github.com/ethereum/go-ethereum/rpc 0.371s
```
In `testClientCancel` we wrap the server listener in `flakeyListener`,
which schedules an unconditional close of every accepted connection
after a random delay, if the random delay is zero then the timer fires
immediately, and then the http client paniced of connection reset by
peer.
Here we add a minimum 10ms to ensure the timeout won't fire immediately.
Signed-off-by: jsvisa <delweng@gmail.com>
This PR is an alternative to #32556.
Instead of trying to be smart and reuse `geth init`, we can introduce a
new flag `--genesis` that loads the `genesis.json` from file into the
`Genesis` object in the same path that the other network flags currently
work in.
Question: is something like `--genesis` enough to start deprecating
`geth init`?
--
```console
$ geth --datadir data --hoodi
..
INFO [10-06|22:37:11.202] - BPO2: @1762955544
..
$ geth --datadir data --genesis genesis.json
..
INFO [10-06|22:37:27.988] - BPO2: @1862955544
..
```
Pull the genesis [from the
specs](https://raw.githubusercontent.com/eth-clients/hoodi/refs/heads/main/metadata/genesis.json)
and modify one of the BPO timestamps to simulate a shadow fork.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
In this PR, the database batch for writing the history index data is
pre-allocated.
It's observed that database batch repeatedly grows the size of the
mega-batch,
causing significant memory allocation pressure. This approach can
effectively
mitigate the overhead.
This PR prevents the SetCode hook from being called when the contract
code
remains unchanged.
This situation can occur in the following cases:
- The deployed runtime code has zero length
- An EIP-7702 authorization attempt tries to unset a non-delegated
account
- An EIP-7702 authorization attempt tries to delegate to the same
account
Previously, the journal writer is nil until the first time rejournal
(default 1h), which means during this period, txs submitted to this node
are not written into journal file (transactions.rlp). If this node is
shutdown before the first time rejournal, then txs in pending or queue
will get lost.
Here, this PR initializes the journal writer soon after launch to solve
this issue.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Uses the go module's `replace` directive to delegate keccak computation
to precompiles.
This is still in draft because it needs more testing. Also, it relies on
a PR that I created, that hasn't been merged yet.
_Note that this PR doesn't implement the stateful keccak state
structure, and it reverts to the current behavior. This is a bit silly
since this is what is used in the tree root computation. The runtime
doesn't currently export the sponge. I will see if I can fix that in a
further PR, but it is going to take more time. In the meantime, this is
a useful first step_
This PR removes dangling peers in `alternates` map
In the current code, a dropped peer is removed from alternates for only
the specific transaction hash it was requesting. If that peer is listed
as an alternate for other transaction hashes, those entries still stick
around in alternates/announced even though that peer already got
dropped.
This PR introduces two new metrics to monitor slow peers
- One tracks the number of slow peers.
- The other measures the time it takes for those peers to become
"unfrozen"
These metrics help with monitoring and evaluating the need for future
optimization of the transaction fetcher and peer management, for example i
n peer scoring and prioritization.
Additionally, this PR moves the fetcher metrics into a separate file,
`eth/fetcher/metrics.go`.
This change addresses critical issues in the state object duplication
process specific to Verkle trie implementations. Without these
modifications, updates to state objects fail to propagate correctly
through the trie structure after a statedb copy operation, leading to
inaccuracies in the computation of the state root hash.
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
New RPC method eth_sendRawTransactionSync(rawTx, timeoutMs?) that
submits a signed tx and blocks until a receipt is available or a timeout
elapses.
Two CLI flags to tune server-side limits:
--rpc.txsync.defaulttimeout (default wait window)
--rpc.txsync.maxtimeout (upper bound; requests are clamped)
closes https://github.com/ethereum/go-ethereum/issues/32094
---------
Co-authored-by: aodhgan <gawnieg@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Adds ethclient support for the eth_simulateV1 RPC method, which allows
simulating transactions on top of a base state without making changes to
the blockchain.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Fix logging in the verkle dump path to report the actual key being
processed.
Previously, the loop always logged keylist[0], which misled users when
expanding multiple keys and made debugging harder. This change aligns
the
log with the key passed to root.Get, improving traceability and
diagnostics.
This fixes a regression introduced in #32518. In that PR, we removed the
slowdown logic that would throttle lookups when the table runs empty.
Said logic was originally added in #20389.
Usually it's fine, but there exist pathological cases, such as hive
tests, where the node can only discover one other node, so it can only
ever query that node and won't get any results. In cases like these, we
need to throttle the creation of lookups to avoid crazy CPU usage.
Drop peer if sending the same transaction multiple times in a single message.
Fixes https://github.com/ethereum/go-ethereum/issues/32724
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This adds a temporary conversion path for blob transactions with legacy
proof sidecar. This feature will activate after Fusaka. We will phase
this out when the fork has sufficiently settled and client side
libraries have been upgraded to send the new proofs.
This pr addresses a few issues brought by the #32270
- Add updates to pricedList after dropping transactions.
- Remove redundant deletions in queue.evictList, since
pool.removeTx(hash, true, true) already performs the removal.
- Prevent duplicate addresses during promotion when Reset is not nil.
The limit check for `MaxUint32` is done after the cast to `int`. On 64
bits machines, that will work without a problem. On 32 bits machines,
that will always fail. The compiler catches it and refuses to build.
Note that this only fixes the compiler build. ~~If the limit is above
`MaxInt32` but strictly below `MaxUint32` then this will fail at runtime
and we have another issue.~~ I checked and this should not happen during
regular execution, although it might happen in tests.
This PR adds a `filterfuzz` subcommand to the workload tester that
generates requests similarly to `filtergen` (though with a much smaller
block length limit) and also verifies the results by retrieving all
block receipts in the range and locally filtering out relevant results.
Unlike `filtergen` that operates on the finalized chain range only,
`filterfuzz` does check the head region, actually it seeds a new query
at every new chain head.
This PR move the queue out of the main transaction pool.
For now there should be no functional changes.
I see this as a first step to refactor the legacypool and make the queue
a fully separate concept from the main pending pool.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR implements the partial read functionalities in the freezer, optimizing
the state history reader by resolving less data from freezer.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
invalidTxMeter was counting txs, while validTxMeter was counting
accounts. Better make the two comparable.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
- Introduce a new subscription kind `transactionReceipts` to allow clients to
receive transaction receipts over WebSocket as soon as they are available.
- Accept optional `transactionHashes` filter to subscribe to receipts for specific
transactions; an empty or omitted filter subscribes to all receipts.
- Preserve the same receipt format as returned by `eth_getTransactionReceipt`.
- Avoid additional HTTP polling, reducing RPC load and latency.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Fixes issue #32793. When the pending tx subscription ends, the filter
is removed from `api.filters`, but it is not terminated. There is no other
way to terminate it, so the subscription will leak, and potentially block
the producer side.
In both `TestSimultaneousRequests` and `TestSameRequestID`, we send two
concurrent requests. The client under test is free to respond in either
order, so we need to handle responses both ways.
Also fixes an issue where some generated blob transactions didn't have
any blobs.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR does a few things:
- Sets the gh actions runner sizes for lint (s) and test (l) workflows
- Runs the tests on gh actions in parallel
- Skips fetching the spec tests when unnecessary (on windows in
appveyor)
- Removes ubuntu appveyor runner since it's essentially duplicate of the
gh action workflow now
The gh test seems to go down from ~35min to ~13min.
This pr implements https://github.com/ethereum/go-ethereum/issues/32733
to make StateProcessor more customisable.
## Compatibility notes
This introduces a breaking change to users using geth EVM as a library.
The `NewStateProcessor` function now takes one parameter which has the
chainConfig embedded instead of 2 parameters.
Description:
We found a occasionally node hang issue on BSC, I think Geth may
also have the issue, so pick the fix patch here.
The fix on BSC repo: https://github.com/bnb-chain/bsc/pull/3347
When the hang occurs, there are two routines stuck.
- routine 1: AsyncFilter(...)
On node start, it will run part of the DiscoveryV4 protocol, which could
take considerable time, here is its hang callstack:
```
goroutine 9711 [chan receive]: // this routine was stuck on read channel: `<-f.slots`
github.com/ethereum/go-ethereum/p2p/enode.AsyncFilter.func1()
github.com/ethereum/go-ethereum/p2p/enode/iter.go:206 +0x125
created by github.com/ethereum/go-ethereum/p2p/enode.AsyncFilter in goroutine 1
github.com/ethereum/go-ethereum/p2p/enode/iter.go:192 +0x205
```
- Routine 2: Node Stop
It is the main routine to shutdown the process, but it got stuck when it
tries to shutdown the discovery components, as it tries to drain the
channel of `<-f.slots`, but the extra 1 slot will never have chance to
be resumed.
```
goroutine 11796 [chan receive]:
github.com/ethereum/go-ethereum/p2p/enode.(*asyncFilterIter).Close.func1()
github.com/ethereum/go-ethereum/p2p/enode/iter.go:248 +0x5c
sync.(*Once).doSlow(0xc032a97cb8?, 0xc032a97d18?)
sync/once.go:78 +0xab
sync.(*Once).Do(...)
sync/once.go:69
github.com/ethereum/go-ethereum/p2p/enode.(*asyncFilterIter).Close(0xc092ff8d00?)
github.com/ethereum/go-ethereum/p2p/enode/iter.go:244 +0x36
github.com/ethereum/go-ethereum/p2p/enode.(*bufferIter).Close.func1()
github.com/ethereum/go-ethereum/p2p/enode/iter.go:299 +0x24
sync.(*Once).doSlow(0x11a175f?, 0x2bfe63e?)
sync/once.go:78 +0xab
sync.(*Once).Do(...)
sync/once.go:69
github.com/ethereum/go-ethereum/p2p/enode.(*bufferIter).Close(0x30?)
github.com/ethereum/go-ethereum/p2p/enode/iter.go:298 +0x36
github.com/ethereum/go-ethereum/p2p/enode.(*FairMix).Close(0xc0004bfea0)
github.com/ethereum/go-ethereum/p2p/enode/iter.go:379 +0xb7
github.com/ethereum/go-ethereum/eth.(*Ethereum).Stop(0xc000997b00)
github.com/ethereum/go-ethereum/eth/backend.go:960 +0x4a
github.com/ethereum/go-ethereum/node.(*Node).stopServices(0xc0001362a0, {0xc012e16330, 0x1, 0xc000111410?})
github.com/ethereum/go-ethereum/node/node.go:333 +0xb3
github.com/ethereum/go-ethereum/node.(*Node).Close(0xc0001362a0)
github.com/ethereum/go-ethereum/node/node.go:263 +0x167
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1.1 in goroutine 9729
github.com/ethereum/go-ethereum/cmd/utils/cmd.go:101 +0x78
```
The rootcause of the hang is caused by the extra 1 slot, which was
designed to make sure the routines in `AsyncFilter(...)` can be
finished. This PR fixes it by making sure the extra 1 shot can always be
resumed when node shutdown.
This PR updates the `payloadVersion` function in `simulated_beacon.go`
to handle additional following forks used during development and testing
phases after Osaka.
This change ensures that the simulated beacon correctly resolves the
payload version for these forks, enabling consistent and valid execution
payload handling during local testing or simulation.
The TxPool.signer field was never read and each subpool (legacy/blob)
maintains its own signer instance. This field remained after txpool
refactoring into subpools and is dead code. Removing it reduces
confusion and simplifies the constructor.
- Replace outdated NewFreezer doc that referenced map[string]bool/snappy
toggle with accurate description of -map[string]freezerTableConfig
(noSnappy, prunable).
- Fix misleading field comment on freezerTable.config that spoke as if
it were a boolean (“if true”), clarifying it’s a struct and noting
compression is non-retroactive.
These functions were previously ignoring the error returned by both
`statedb.Commit()` and the subsequent `state.New()`,
which could silently fail and cause panics later when the `statedb` is
used.
This change adds proper error checking and panics with a descriptive
error
message if state creation fails.
While unlikely in normal operation, this can occur if there are database
corruption issues or if invalid root hashes are provided, making
debugging
significantly easier when such issues do occur.
This issue was encountered and fixed in
https://github.com/gballet/go-ethereum/pull/552
where the error handling proved essential for debugging
cc: @gballet as this was discussed in a call already.
Introduces a new tracer which returns the preimages
of evm KECCAK256 hashes.
See #32570.
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Bail out of decodeHash when the raw hex string is longer than 32 byte before actually decoding.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This is a small improvement on #32656 in case Add was called with
multiple type 3 transactions, adding transactions to the pool one-by-one
as they are converted.
Announcement to peers is still done in a batch.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Fixes race in WaitDeploy test where the backend is closed before goroutine using it wraps up.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Replace time.After with a long‑lived time.Ticker in KeyStore.updater, preventing per‑iteration timer allocations and potential timer buildup.
Co-authored-by: lightclient <lightclient@protonmail.com>
- Correct the error message in TestOneElementProof to expect 'v' instead
of 'k'.
- The trie is updated with key "k" and value "v"; on mismatch the
expected value must be 'v'.
- Aligns the message with the actual test logic and other similar checks
in this file, reducing confusion during test failures. No behavioral
changes.
This implements the conversion of existing blob transactions to the new proof
version. Conversion is triggered at the Osaka fork boundary. The conversion is
designed to be idempotent, and may be triggered multiple times in case of a reorg
around the fork boundary.
This change is the last missing piece that completes our strategy for the blobpool
conversion. After the Osaka fork,
- new transactions will be converted on-the-fly upon entry to the pool
- reorged transactions will be converted while being reinjected
- (this change) existing transactions will be converted in the background
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
This fixes `go run build/ci.go install`. It was failing because we
resolved all main packages by parsing sources, which fails when the
source directory contains multiple modules.
The parent header was missing the BaseFee field when calculating the
reserve price for EIP-7918 in the Osaka fork, causing a nil pointer
dereference. This fix ensures BaseFee is properly set from ParentBaseFee
in the environment.
Added regression test case 34 to verify Osaka fork blob gas calculation
works correctly with parent base fee.
This change replaces wrapping a stale outer err with the iterator’s own
error after Next(), and switches the post-BlockAndReceipts() check to
use the returned err. According to internal/era iterator contract,
Error() should be consulted immediately after Next() to surface
iteration errors, while decoding errors from Block/Receipts are returned
directly. The previous code could hide the real failure (using nil or
unrelated err), leading to misleading diagnostics and missed iteration
errors.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
https://go.dev/ref/mod#go-work-file advises against checking `go.work`
files because they can interfere with local development. We added the
workspace file in order to make `go test` and other tools work across
multiple modules. But it seems to cause weird issues with the
`go.work.sum` file being modified, etc.
So with this PR, we instead run all the `ci.go` commands for all modules
in the workspace manually.
This pull request introduces a queue for legacy sidecar conversion to
handle transactions that persist after the Osaka fork. Simply dropping
these transactions would significantly harm the user experience.
To balance usability with system complexity, we have introduced a
conversion time window of two hours post Osaka fork. During this period,
the system will accept legacy blob transactions and convert them in a
background process.
After the window, all legacy transactions will be rejected. Notably, all
the blob transactions will be validated statically before the conversion,
and also all conversion are performed in a single thread, minimize the risk
of being DoS.
We believe this two hour window provides sufficient time to process
in-flight legacy transactions and allows submitters to migrate to the
new format.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Fix the t.Fatalf format arguments in TestBadBlockStorage to match the
intended #index output. Previously, the left number used i+1 and the
right index used the block number, producing misleading diagnostics.
Correct mapping improves test failure clarity and debuggability.
Remove redundant duplicate test vectors. The two entries were identical
and back-to-back, providing no additional coverage while adding noise.
Keeping a single instance maintains test intent and clarity without
altering behavior.
Fix typo in test error message where "MustParseBig" was incorrectly
used instead of "MustParseUint64" in the TestMustParseUint64Panic
function.
The test still functions correctly, but now the error message
accurately reflects the function being tested.
before:
go test -run=^$ -bench=. ./crypto/... 94.83s user 2.68s system 138% cpu
1:10.55 tota
after:
go test -run=^$ -bench=. ./crypto/... 75.43s user 2.58s system 123% cpu
1:03.01 total
before:
go test -run=^$ -bench=. ./core/types 47.80s user 2.18s system 102% cpu
48.936 tota
after:
go test -run=^$ -bench=. ./core/types 42.42s user 2.27s system 112% cpu
39.593 total
before:
go test -run=^$ -bench=. ./core/vm/... -timeout=1h 1841.87s user 40.96s
system 124% cpu 25:15.76 total
after:
go test -run=^$ -bench=. ./core/vm/... -timeout=1h 1588.65s user 33.79s
system 123% cpu 21:53.25 total
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
before:
go test -run=^$ -bench=. ./log -timeout=1h 12.19s user 2.19s system 89%
cpu 16.025 total
after:
go test -run=^$ -bench=. ./log -timeout=1h 10.64s user 1.53s system 89%
cpu 13.607 total
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
before:
go test -run=^$ -bench=. ./eth/... 827.57s user 23.80s system 361% cpu
3:55.49 total
after:
go test -run=^$ -bench=. ./eth/... 281.62s user 13.62s system 245% cpu
2:00.49 total
before:
go test -run=^$ -bench=. ./core/state/... 120.85s user 7.96s system 129%
cpu 1:39.13 tota
after:
go test -run=^$ -bench=. ./core/state/... 21.32s user 2.12s system 97%
cpu 24.006 total
This PR improves `TestBlockSync` so that it also tests the finality
update validation.
Note: to this date four long and complex (at least partly AI generated)
PRs arrived that did something related to testing finality but honestly
we do not need a bloated "comprehensive" test to test a trivial feature
because maintaining these tests can also be a pain over the long term.
This PR adds some sufficient sanity checks to detect if finality ever
gets broken by a future change.
This disables the tx gaslimit cap for eth_call and related RPC operations.
I don't like how this fix works. Ideally we'd be checking the tx
gaslimit somewhere else, like in the block validator, or any other place
that considers block transactions. Doing the check in StateTransition
means it affects all possible ways of executing a message.
The challenge is finding a place for this check that also triggers
correctly in tests where it is wanted. So for now, we are just combining
this with the EOA sender check for transactions. Both are disabled for
call-type messages.
Fixes a crash when loading the beacon chain config if new fields like
`BLOB_SCHEDULE: []` are present.
Previously, the config loader assumed all values were strings, causing
errors such as:
```
Fatal: Could not load beacon chain config '/network-configs/config.yaml': failed to parse beacon chain config file: yaml: unmarshal errors:
line 242: cannot unmarshal !!seq into string
```
This PR updates the parsing logic to handle non-string values correctly
and adds explicit validation for fork fields.
Add cli configurable limit for the number of addresses allowed in
eth_getLogs filter criteria:
https://github.com/ethereum/go-ethereum/issues/32264
Key changes:
- Added --rpc.getlogmaxaddrs CLI flag (default: 1000) to configure the
maximum number of addresses
- Updated ethconfig.Config with FilterMaxAddresses field for
configuration management
- Modified filter system to use the configurable limit instead of the
hardcoded maxAddresses constant
- Enhanced test coverage with new test cases for address limit
validation
- Removed hardcoded validation from JSON unmarshaling, moving it to
runtime validation
Please notice that I remove the check at FilterCriteria UnmarshalJSON
because the runtime config can not pass into this validation.
Please help review this change!
---------
Co-authored-by: zsfelfoldi <zsfelfoldi@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Addresses https://github.com/ethereum/go-ethereum/issues/32630
This pull request enables the stateless engine APIs for Osaka and the
following BPOs. Apart from that, a few more descriptions have been added
in the engine APIs, making it easier to follow the spec change.
https://github.com/ethereum/execution-spec-tests/releases/tag/v5.0.0
As of this release, execution-spec-tests also contains all state tests
that were previously in ethereum/tests. We can probably remove the tests
submodule now. However, this would mean we are missing the pre-cancun
tests. Still need to figure out how to resolve this.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
When I implemented in #31340 I didn't expect multiple forks to be
configured at once, but this is exactly how BPOs are defined. This
updates the method to determine the next scheduled fork rather than the
last fork.
ZKVMs are constrained environments that liberally allocate memory and
never release it. In this context, using the GC is only going to cause
issues down the road, and slow things down in any case.
- Adds `NodeIteratorWithPrefix()` method to support iterating only nodes
within a specific key prefix
- Adds `NodeIteratorWithRange()` method to support iterating only nodes
within a specific key range
Current `NodeIterator` always traverses the entire remaining trie from a
start position. For non-ethereum applications using the trie implementation,
there's no way to limit iteration to just a subtree with a specific prefix.
**Usage:**
```go
// Only iterate nodes with prefix "key1"
iter, err := trie.NodeIteratorWithPrefix([]byte("key1"))
```
Testing: Comprehensive test suite covering edge cases and boundary conditions.
Closes#32484
---------
Co-authored-by: gballet <guillaume.ballet@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This pull request is based on #32306 , is the second part for shipping
trienode history.
Specifically, this pull request generalize the existing index mechanism,
making is usable
by both state history and trienode history in the near future.
The format that is currently reported by the chain isn't very useful, as
it gives an average for ALL the nodes, and not only the leaves, which
skews the results.
Also, until now there was no way to activate the reporting of errors.
We also decided that metrics weren't the right tool to report this data,
so we decided to dump it to the console if the flag is enabled. A better
system should be built, but for now, printing to the logs does the job.
This improves the latency of lookups in small networks and test setups. When the local node table runs empty, the lookupIterator will trigger refresh to try and fill the table again.
The behaviour of lookup in case of an empty table is changed:
- Previously, lookup waited fixed 1 second before trying to continue the lookup
- Now, lookup on an empty table returns immediately, and a better wait implementation is part of the LookupIterator. It reinitialises the table, and continues the interator as soon as a node becomes available.
This PR adds a new RPC call, which re-executes a block with stateless
mode activated, so that the witness data are collected and returned.
They are `debug_executionWitnessByHash` which takes in a block hash
and `debug_executionWitness` which takes in a block number.
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
ApplyTransaction calls the hooks and builds the receipt, so some
duplicated code can be removed from t8ntool. Test cases have been
changed to add the `blockNumber` and `blockHash` in receipts, since
they were previously not filled in.
Keeper is a zmvm guest program that runs the block transition.
It relies on the zkvm maker implementing `getInput`. For now, we only
provide a single implementation for the 'ziren' VM.
Why keeper?
In the _Mass Effect_ lore, the keepers are animals (?) who maintain the
citadel. Nothing is known from them, and attempts at tampering with them
have failed, as they self-destruct upon inquiry. They have a secret,
nefarious purpose that is only revealed later in the game series, don't
want any spoilers so I didn't dig deeper. All in all, a good metaphor
for zkvms.
---------
Co-authored-by: weilzkm <140377101+weilzkm@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Implements a migration path for the blobpool slotter
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change ensures TransitionState.Copy preserves BaseRoot. During a
Verkle transition, ts.BaseRoot is required to construct the overlay MPT
when ts.InTransition() is true. Previously, copies dropped BaseRoot,
risking an invalid zero-hash base trie and inconsistent behavior.
Fixes an issue I accidentally introduced in #32579. Essentially, because
we gate the engine methods based on particular forks and I did not add
the BPOs as allowed forks to the method.
Ensure Database.namespace is initialized in pebble.New(...). Without
this, the write-stall metrics registered in onWriteStallBegin/End are
emitted without the intended namespace prefix, while other Pebble
metrics use the provided constructor parameter. This aligns stall
metrics with the rest of the Pebble metric set and fixes inconsistent
metric naming.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
The lookup would add self into the replyBuffer if returned by another node.
Avoid doing that by marking self as seen.
With the changed initialization behavior of lookup, the lookupIterator needs to yield the
buffer right after creation. This fixes the smallNetConvergence test, where all results
are straight out of the local table.
Refresh is doing some lookups and thus it could block for some time. We
do not want the initializer of an iterator to block. If there is
something blocking, it should happen when calling Next.
Here, next will start a lookup, which will wait if needed (no nodes),
making sure the iterator's Next is not creating a busy loop.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR removes the conversion of legacy sidecars after Osaka and instead rejects them to the pool.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This pull request addresses the corrupted path database with log
indicating:
`history head truncation out of range, tail: 122557, head: 212208,
target: 212557`
This is a rare edge case where the in-memory layers, including the write
buffer
in the disk layer, are fully persisted (e.g., written to file), but the
state history
freezer is not properly closed (e.g., Geth is terminated after
journaling but
before freezer.Close). In this situation, the recent state history
writes will be
truncated on the next startup, while the in-memory layers resolve
correctly.
As a result, the state history falls behind the disk layer (including
the write buffer).
In this pull request, the state history freezer is always synced before
journal,
ensuring the state history writes are always persisted before the
others.
Edit:
It's confirmed that devops team has 10s container termination setting.
It
explains why Geth didn't finish the entire termination without state
history
being closed.
https://github.com/ethpandaops/fusaka-devnets/pull/63/files
TestBlobTxWithoutSidecar test could run infinitely in case of a client
not requesting the good transaction. This adds a timeout to make the
test fail in this case.
Fixes https://github.com/ethereum/go-ethereum/issues/32422
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Add state size tracking and retrieve api, start geth with `--state.size-tracking`,
the initial bootstrap is required (around 1h on mainnet), after the bootstrap,
use `debug_stateSize()` RPC to retrieve the state size:
```
> debug.stateSize()
{
accountBytes: "0x39681967b",
accountTrienodeBytes: "0xc57939f0c",
accountTrienodes: "0x198b36ac",
accounts: "0x129da14a",
blockNumber: "0x1635e90",
contractCodeBytes: "0x2b63ef481",
contractCodes: "0x1c7b45",
stateRoot: "0x9c36a3ec3745d72eea8700bd27b90dcaa66de0494b187c5600750044151e620a",
storageBytes: "0x18a6e7d3f1",
storageTrienodeBytes: "0x2e7f53fae6",
storageTrienodes: "0x6e49a234",
storages: "0x517859c5"
}
```
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
As in #32060 we introduced the file based journal path, for the other
sub command(eg: snapshot, db), we should also pass the directory to the
triedb, else the subcommand(eg: `geth snapshot`) failed to run:
```bash
geth snapshot verify-state --datadir /geth-data
...
INFO [09-02|02:12:29.493] Allocated cache and file handles database=/geth-data/geth/chaindata cache=512.00MiB handles=524,288
INFO [09-02|02:12:32.746] Opened ancient database database=/geth-data/geth/chaindata/ancient/chain readonly=true
INFO [09-02|02:12:32.746] Opened Era store datadir=/geth-data/geth/chaindata/ancient/chain/era
INFO [09-02|02:12:32.758] State scheme set to already existing scheme=path
INFO [09-02|02:12:32.760] Load database journal from disk
INFO [09-02|02:12:32.764] Failed to load journal, discard it err="journal not found"
INFO [09-02|02:12:32.789] Opened ancient database database=/geth-data/geth/chaindata/ancient/state readonly=true
INFO [09-02|02:12:32.790] Initialized path database readonly=true triecache=0.00B statecache=0.00B buffer=0.00B history="entire chain"
ERROR[09-02|02:12:32.791] Failed to verify state root=c5458d..4cc785 err="unknown layer: c5458d476da0136a67ef24a93b909aa5c29efa5c5b885dbd1fbaed4e784cc785"
```
This PR is the first step in the trienode history series.
It introduces the `nodeWithOrigin` struct in the path database, which tracks
the original values of dirty nodes to support trienode history construction.
Note, the original value is always empty in this PR, so it won't break the
existing journal for encoding and decoding. The compatibility of journal
should be handled in the following PR.
Another getBlobs PR on top of
https://github.com/ethereum/go-ethereum/pull/32190 to avoid some minor
regressions.
- bring back more log messages from before
- continue processing also on some internal errors
- ensure v2 complies with spec even if there are internal errors
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This pull request fixes a regression, introduced in #32190
Specifically, in GetBlobsV1 engine API, if any blob is missing, the null
should be placed in
response, unfortunately a behavioral change was introduced in #32190,
returning an error
instead.
What's more, a more comprehensive test suite is added to cover both
`GetBlobsV1` and
`GetBlobsV2` endpoints.
Switches to using counters so that the gauges don't cause any
information to be lost. Counters can be used to calculate all sorts of
metrics on Grafana. Which is also why min/avg/max logic is removed to
make things simple and small here.
~Will probably be mostly supplanted by #32224, but this should do for
now for devnet 3.~
Seems like #32224 is going to take some more time, so I have completed
the implementation of eth_config here. It is quite a bit simpler to
implement now that the config hashing was removed.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Implement the binary tree as specified in [eip-7864](https://eips.ethereum.org/EIPS/eip-7864).
This will gradually replace verkle trees in the codebase. This is only
running the tests and will not be executed in production, but will help
me rebase some of my work, so that it doesn't bitrot as much.
---------
Signed-off-by: Guillaume Ballet
Co-authored-by: Parithosh Jayanthi <parithosh.jayanthi@ethereum.org>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Filtering for leaf nodes was missing from #32388, which means that even
the root done was reported, which made little sense for the bloatnet
data processing we want to do.
### Summary
Fixes long-standing ETA calculation errors in progress indicators that
have been present since February 2021. The current implementation
produces increasingly inaccurate estimates due to integer division
precision loss.
### Problem
3aeccadd04/triedb/pathdb/history_indexer.go (L541-L553)
The ETA calculation has two critical issues:
1. **Integer division precision loss**: `speed` is calculated as
`uint64`
2. **Off-by-one**: `speed` uses `+ 1`(2 times) to avoid division by
zero, however it makes mistake in the final calculation
This results in wildly inaccurate time estimates that don't improve as
progress continues.
### Example
Current output during state history indexing:
```
lvl=info msg="Indexing state history" processed=16858580 left=41802252 elapsed=18h22m59.848s eta=11h36m42.252s
```
**Expected calculation:**
- Speed: 16858580 ÷ 66179848ms = 0.255 blocks/ms
- ETA: 41802252 ÷ 0.255 = ~45.6 hours
**Current buggy calculation:**
- Speed: rounds to 1 block/ms
- ETA: 41802252 ÷ 1 = ~11.6 hours ❌
### Solution
- Created centralized `CalculateETA()` function in common package
- Replaced all 8 duplicate code copies across the codebase
### Testing
Verified accurate ETA calculations during archive node reindexing with
significantly improved time estimates.
`db inspect` on the full database currently takes **30min+**, because
the db iterate was run in one thread, propose to split the key-space to
256 sub range, and assign them to the worker pool to speed up.
After the change, the time of running `db inspect --workers 16` reduced
to **10min**(the keyspace is not evenly distributed).
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This pull request preserves the root->ID mappings in the path database
even after the associated state histories are truncated, regardless of
whether the truncation occurs at the head or the tail.
The motivation is to support an additional history type, trienode history.
Since the root->ID mappings are shared between two history instances,
they must not be removed by either one.
As a consequence, the root->ID mappings remain in the database even
after the corresponding histories are pruned. While these mappings may
become dangling, it is safe and cheap to keep them.
Additionally, this pull request enhances validation during historical
reader construction, ensuring that only canonical historical state will be
served.
When maxPeers was just above some perfect square, and a few peers
dropped for some reason, we changed the peer selection function.
When new peers were acquired, we changed again.
This PR improves the selection function, in two ways. First, it will always select
sqrt(peers) to broadcast to. Second, the selection now uses siphash with a secret
key, to guard against information leaks about tx source.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a prestateTracer test case covering 7702 delegation.
---------
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This pull request implements #32235 , constructing blob sidecar in new
format (cell proof)
if the Osaka has been activated.
Apart from that, it introduces a pre-conversion step in the blob pool
before adding the txs.
This mechanism is essential for handling the remote **legacy** blob txs
from the network.
One thing is still missing and probably is worthy being highlighted
here: the blobpool may
contain several legacy blob txs before the Osaka and these txs should be
converted once
Osaka is activated. While the `GetBlob` API in blobpool is capable for
generating cell proofs
at the runtime, converting legacy txs at one time is much cheaper
overall.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: lightclient <lightclient@protonmail.com>
Replace hardcoded 5-second sleep with polling loop that actively checks
snap sync state. This approach is already used in other project tests
(like account_cache_test.go) and provides better reliability by:
- Reducing flaky behavior on slower systems
- Finishing early when sync completes quickly
- Using 1-second timeout with 100ms polling intervals
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
closes#32240#32232
The main cause for the time out is the slow json encoding of large data.
In #32240 they tried to resolve the issue by reducing the size of the
test. However as Felix pointed out, the test is still kind of confusing.
I've refactored the test so it is more understandable and have reduced
the amount of data needed to be json encoded. I think it is still
important to ensure that the default read limit is not active, so I have
retained one large (~32 MB) test case, but it's at least smaller than
the existing ~64 MB test case.
This pull request refactors the internal implementation in path database
a bit, specifically:
- purge the state index data in batch
- simplify the logic of state history construction and index, make it more readable
This is a internal refactoring PR, renaming the history to stateHistory.
It's a pre-requisite PR for merging trienode history, avoid the name
conflict.
The TestTraceChain function was missing a defer backend.teardown() call,
which is required to properly release blockchain resources after test
completion.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Full disclosure: this has been generated by AI. The goal is to have a
quick check that the PR format is correct, before we merge it. This is
to avoid the periodical case when someone forgets to add a milestone or
check the title matches our preferred format.
Supersedes #32470.
### What
- snap: shorten stall watchdog in `eth/protocols/snap/sync_test.go` from
1m to 10s.
- discover/v5: consolidate FINDNODE negative tests into a single
table-driven test:
- `TestUDPv5_findnodeCall_InvalidNodes` covers:
- invalid IP (unspecified `0.0.0.0`) → ignored
- low UDP port (`<=1024`) → ignored
### Why
- Addresses TODOs:
- “Make tests smaller” (reduce long 1m timeout).
- “check invalid IPs”; also cover low port per `verifyResponseNode`
rules (UDP must be >1024).
### How it’s validated
- Test-only changes; no production code touched.
- Local runs:
- `go test ./p2p/discover -count=1 -timeout=300s` → ok
- `go test ./eth/protocols/snap -count=1 -timeout=600s` → ok
- Lint:
- `go run build/ci.go lint` → 0 issues on modified files.
### Notes
- The test harness uses `enode.ValidSchemesForTesting` (which includes
the “null” scheme), so records signed with `enode.SignNull` are
signature-valid; failures here are due to IP/port validation in
`verifyResponseNode` and `netutil.CheckRelayAddr`.
- Tests are written as a single table-driven function for clarity; no
helpers or environment switching.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR should reduce overall allocations of a running node by ~10
percent. Since most allocations are coming from the re-heaping of the
transaction pool.
```
(pprof) list EffectiveGasTipCmp
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core/types.(*Transaction).EffectiveGasTipCmp in github.com/ethereum/go-ethereum/core/types/transaction.go
0 3766837369 (flat, cum) 9.86% of Total
. . 386:func (tx *Transaction) EffectiveGasTipCmp(other *Transaction, baseFee *big.Int) int {
. . 387: if baseFee == nil {
. . 388: return tx.GasTipCapCmp(other)
. . 389: }
. . 390: // Use more efficient internal method.
. . 391: txTip, otherTip := new(big.Int), new(big.Int)
. 1796172553 392: tx.calcEffectiveGasTip(txTip, baseFee)
. 1970664816 393: other.calcEffectiveGasTip(otherTip, baseFee)
. . 394: return txTip.Cmp(otherTip)
. . 395:}
. . 396:
. . 397:// EffectiveGasTipIntCmp compares the effective gasTipCap of a transaction to the given gasTipCap.
. . 398:func (tx *Transaction) EffectiveGasTipIntCmp(other *big.Int, baseFee *big.Int) int {
```
This PR reduces the allocations for comparing two transactions from 2 to
0:
```
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/core/types
cpu: Intel(R) Core(TM) Ultra 7 155U
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
EffectiveGasTipCmp/Original-14 64.67n ± 2% 25.13n ± 9% -61.13% (p=0.000 n=10)
│ /tmp/old.txt │ /tmp/new.txt │
│ B/op │ B/op vs base │
EffectiveGasTipCmp/Original-14 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10)
│ /tmp/old.txt │ /tmp/new.txt │
│ allocs/op │ allocs/op vs base │
EffectiveGasTipCmp/Original-14 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
```
It also speeds up the process by ~60%
There are two minor caveats with this PR:
- We change the API for `EffectiveGasTipCmp` and `EffectiveGasTipIntCmp`
(which are probably not used by much)
- We slightly change the behavior of `tx.EffectiveGasTip` when it
returns an error. It would previously return a negative number on error,
now it does not (since uint256 does not allow for negative numbers)
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
## Description
This PR fixes a bug in the Ledger hardware wallet version validation
logic for EIP-155 transaction signing. The original condition
incorrectly allowed older versions that don't support EIP-155 such as
0.9.9 and 0.1.5 to proceed.
This pull introduces a `Prefetch` operation in the trie to prefetch trie
nodes in parallel. It is used by the `triePrefetcher` to accelerate state
loading and improve overall chain processing performance.
## Summary
This PR addresses a DoS vulnerability in the GraphQL service by
implementing a maximum query depth limit. While #26026 introduced
timeout handling, it didn't fully mitigate the attack vector where
deeply nested queries can still consume excessive CPU and memory
resources before the timeout is reached.
## Changes
- Added `maxQueryDepth` constant (set to 20) to limit the maximum
nesting depth of GraphQL queries
- Applied the depth limit using `graphql.MaxDepth()` option when parsing
the schema
- Added test case `TestGraphQLMaxDepth` to verify that queries exceeding
the depth limit are properly rejected
## Security Impact
Without query depth limits, malicious actors could craft deeply nested
queries that:
- Consume excessive CPU cycles during query parsing and execution
- Allocate large amounts of memory for nested result structures
- Potentially cause service degradation or outages even with timeout
protection
This fix complements the existing timeout mechanism by preventing
resource-intensive queries from being executed in the first place.
## Testing
Added `TestGraphQLMaxDepth` which verifies that queries with nesting
depth > 20 are rejected with a `MaxDepthExceeded` error.
## References
- Original issue: #26026
- Related security best practices:
https://www.howtographql.com/advanced/4-security/
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Exposing the public method to setReadLimits for Websocket RPC to
prevent OOM.
Current, Geth Server is using a default 32MB max read limit (message
size) for websocket, which is prune to being attacked for OOM. Any one
can easily launch a client to send a bunch of concurrent large request
to cause the node to crash for OOM. One example of such script that can
easily crash a Geth node running websocket server is like this:
ec830979ac/poc.go
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Continuation of https://github.com/ethereum/go-ethereum/issues/32022
tablewriter assumes unix or windows, which may not be the case for
embedded targets.
For v0.0.5 of tablewriter, it is noted in table.go: "The protocols were
written in pure Go and works on windows and unix systems"
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
The order of the checks was wrong which would have allowed a call to
modexp with `baseLen == 0 && modLen == 0` post fusaka.
Also handles an edge case where base/mod/exp length >= 2**64
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the implementation to resolve the blob parameters according
to the current header timestamp. This matters for EIP-7918, where we
would previously resolve the UpdateFraction according to the parent
header fork, leading to a confusing situation at the fork transition
block.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This add some of the changes that were missing from #31634. It
introduces the `TransitionTrie`, which is a façade pattern between the
current MPT trie and the overlay tree.
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
**Problem:** Including full account code in prestateTracer response
significantly increases response payload size.
**Solution:** Add codeHash field to the response. This will allow
client-side bytecode caching and is a non-breaking change.
**Note:** codeHash for EoAs is excluded to save space.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
## Description
Correct symmetric tolerance in gas limit validation:
Replace ambiguous "+-=" with standard "+/-" in the error message.
Logic rejects when |header − parent| ≥ limit, so allowed range is |Δ| ≤
limit − 1.
No logic or functionality has been modified.
These changes made in the PR should be highlighted here
The trie tracer is split into two distinct structs: opTracer and prevalueTracer.
The former is specific to MPT, while the latter is generic and applicable to all
trie implementations.
The original values of dirty nodes are tracked in a NodeSet. This serves
as the foundation for both full archive node implementations and the state live
tracer.
The GetHeader function was incorrectly returning an error when
encountering nil peers in the peers list, which contradicted the comment
"keep retrying if none are yet available".
Changed the logic to skip nil peers with 'continue' instead of returning
an error, allowing the function to properly iterate through all
available peers and attempt to retrieve the target header from each valid peer.
This ensures the function behaves as intended - trying all available
peers before giving up, rather than failing on the first nil peer encountered.
The previous comment stated that every 3rd block has a tx and every 5th
has an uncle.
The implementation actually adds one transaction to every second block
and does not add uncles.
Updated the comment to reflect the real behavior to avoid confusion when
reading tests.
Add missing it.Error() check after iteration in Database.DeleteRange to
avoid silently ignoring iterator errors before writing the batch.
Aligns behavior with batch.DeleteRange, which already validates iterator
errors. No other functional changes; existing tests pass (TestLevelDB).
This PR makes 2 changes to how
[EIP-7825](https://github.com/ethereum/go-ethereum/pull/31824) behaves.
When `eth_estimateGas` or `eth_createAccessList` is called without any
gas limit in the payload, geth will choose the block's gas limit or the
`RPCGasCap`, which can be larger than the `maxTxGas`.
When this happens for `estimateGas`, the gas estimation just errors out
and ends, when it should continue doing binary search to find the lowest
possible gas limit.
This PR will:
- Add a check to see if `hi` is larger than `maxTxGas` and cap it to
`maxTxGas` if it's larger. And add a special case handling for gas
estimation execute when it errs with `ErrGasLimitTooHigh`
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
- If all the `vhashes` are in the same `sidecar`, then it will load the
same blob tx many times. This PR aims to upgrade this.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This is the first part of #31532
It maintains a series of conversion maker which are to be updated by the
conversion code (in a follow-up PR, this is a breakdown of a larger PR
to make things easier to review). They can be used in this way:
- During the conversion, by storing the conversion markers when the
block has been processed. This is meant to be written in a function that
isn't currently present, hence [this
TODO](https://github.com/ethereum/go-ethereum/pull/31634/files#diff-89272f61e115723833d498a0acbe59fa2286e3dc7276a676a7f7816f21e248b7R384).
Part of https://github.com/ethereum/go-ethereum/issues/31583
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This introduces an error when the filter has both `blockHash` and
`fromBlock`/`toBlock`, since these are mutually exclusive. Seems the
tests were actually returning `not found` error, which went undetected
since there was no check on the actual returned error in the test.
This adds a method on vm.EVM to set the jumpdest cache implementation.
It can be used to maintain an analysis cache across VM invocations, to improve
performance by skipping the analysis for already known contracts.
---------
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This pull request optimizes trie hashing by reducing memory allocation
overhead. Specifically:
- define a fullNodeEncoder pool to reuse encoders and avoid memory
allocations.
- simplify the encoding logic for shortNode and fullNode by getting rid
of the Go interfaces.
This PR addresses a flakiness in the rollback test discussed in
https://github.com/ethereum/go-ethereum/issues/32252
I found `nonce` collision caused transactions occasionally fail to send.
I tried to change error message in the failed test like:
```
if err = client.SendTransaction(ctx, signedTx); err != nil {
t.Fatalf("failed to send transaction: %v, nonce: %d", err, signedTx.Nonce())
}
```
and I occasionally got test failure with this message:
```
=== CONT TestFlakyFunction/Run_#100
rollback_test.go:44: failed to send transaction: already known, nonce: 0
--- FAIL: TestFlakyFunction/Run_#100 (0.07s)
```
Although `nonces` are obtained via `PendingNonceAt`, we observed that,
in rare cases (approximately 1 in 1000), two transactions from the same
sender end up with the same nonce. This likely happens because `tx0` has
not yet propagated to the transaction pool before `tx1` requests its
nonce. When the test succeeds, `tx0` and `tx1` have nonces `0` and `1`,
respectively. However, in rare failures, both transactions end up with
nonce `0`.
We modified the test to explicitly assign nonces to each transaction. By
controlling the nonce values manually, we eliminated the race condition
and ensured consistent behavior. After several thousand runs, the
flakiness was no longer reproducible in my local environment.
Reduced internal polling interval in `pendingStateHasTx()` to speed up
test execution without impacting stability. It reduces test time for
`TestTransactionRollbackBehavior` from about 7 seconds to 2 seconds.
Correct the error message in the ExecuteStatelessPayloadV4 function to
reference newPayloadV4 and the Prague fork, instead of incorrectly
referencing newPayloadV3 and Cancun.
This improves clarity during debugging and aligns the error message with
the actual function and fork being validated. No logic is changed.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Seems the `signal.result` was not sent back in shorten case, this will
cause a deadlock.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Replace manual byte-by-byte XOR implementation with the optimized
bitutil.XORBytes function. This improves performance by using word-sized
operations on supported architectures while maintaining the same
functionality. The optimized version processes data in bulk rather than
one byte at a time
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
`binary.AppendUvarint` offers better performance than using append
directly, because it avoids unnecessary memory allocation and copying.
In our case, it can increase the performance by +35.8% for the
`blockWriter.append` function:
```
benchmark old ns/op new ns/op delta
BenchmarkBlockWriterAppend-8 5.97 3.83 -35.80%
```
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
[EIP-7594](https://eips.ethereum.org/EIPS/eip-7594) defines a limit of
max 6 blobs per transaction. We need to enforce this limit during block
processing.
> Additionally, a limit of 6 blobs per transaction is introduced.
Clients MUST enforce this limit when validating blob transactions at
submission time, when received from the network, and during block
production and processing.
The main purpose of this change is to enforce the version setting when
constructing the blobSidecar, avoiding creating sidecar with wrong/default
version tag.
The implementation of `parseIndexBlock` used a reverse loop with slice
appends to build the restart points, which was less cache-friendly and
involved unnecessary allocations and operations. In this PR we change
the implementation to read and validate the restart points in one single
forward loop.
Here is the benchmark test:
```bash
go test -benchmem -bench=BenchmarkParseIndexBlock ./triedb/pathdb/
```
The result as below:
```
benchmark old ns/op new ns/op delta
BenchmarkParseIndexBlock-8 52.9 37.5 -29.05%
```
about 29% improvements
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Fixes#32175.
This fixes the scenario where the blockhash opcode would return 0x0
during RPC simulations when using BlockOverrides with a future block
number. The root cause was that BlockOverrides.Apply() only modified the
vm.BlockContext, but GetHashFn() depends on the actual
types.Header.Number to resolve valid historical block hashes. This
caused a mismatch and resulted in incorrect behavior during trace and
call simulations.
---------
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
The root cause of the flaky test was a nonce conflict caused by async
contract deployments.
This solution defines a custom deployer with automatic nonce management.
This is something interesting I came across during my benchmarks, we
spent ~3.8% of all allocations allocating the header number on the heap.
```
(pprof) list GetHeaderByHash
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core.(*BlockChain).GetHeaderByHash in github.com/ethereum/go-ethereum/core/blockchain_reader.go
0 5786566117 (flat, cum) 15.15% of Total
. . 79:func (bc *BlockChain) GetHeaderByHash(hash common.Hash) *types.Header {
. 5786566117 80: return bc.hc.GetHeaderByHash(hash)
. . 81:}
. . 82:
. . 83:// GetHeaderByNumber retrieves a block header from the database by number,
. . 84:// caching it (associated with its hash) if found.
. . 85:func (bc *BlockChain) GetHeaderByNumber(number uint64) *types.Header {
ROUTINE ======================== github.com/ethereum/go-ethereum/core.(*HeaderChain).GetHeaderByHash in github.com/ethereum/go-ethereum/core/headerchain.go
0 5786566117 (flat, cum) 15.15% of Total
. . 404:func (hc *HeaderChain) GetHeaderByHash(hash common.Hash) *types.Header {
. 1471264309 405: number := hc.GetBlockNumber(hash)
. . 406: if number == nil {
. . 407: return nil
. . 408: }
. 4315301808 409: return hc.GetHeader(hash, *number)
. . 410:}
. . 411:
. . 412:// HasHeader checks if a block header is present in the database or not.
. . 413:// In theory, if header is present in the database, all relative components
. . 414:// like td and hash->number should be present too.
(pprof) list GetBlockNumber
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core.(*HeaderChain).GetBlockNumber in github.com/ethereum/go-ethereum/core/headerchain.go
94438817 1471264309 (flat, cum) 3.85% of Total
. . 100:func (hc *HeaderChain) GetBlockNumber(hash common.Hash) *uint64 {
94438817 94438817 101: if cached, ok := hc.numberCache.Get(hash); ok {
. . 102: return &cached
. . 103: }
. 1376270828 104: number := rawdb.ReadHeaderNumber(hc.chainDb, hash)
. . 105: if number != nil {
. 554664 106: hc.numberCache.Add(hash, *number)
. . 107: }
. . 108: return number
. . 109:}
. . 110:
. . 111:type headerWriteResult struct {
(pprof) list ReadHeaderNumber
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core/rawdb.ReadHeaderNumber in github.com/ethereum/go-ethereum/core/rawdb/accessors_chain.go
204606513 1376270828 (flat, cum) 3.60% of Total
. . 146:func ReadHeaderNumber(db ethdb.KeyValueReader, hash common.Hash) *uint64 {
109577863 1281242178 147: data, _ := db.Get(headerNumberKey(hash))
. . 148: if len(data) != 8 {
. . 149: return nil
. . 150: }
95028650 95028650 151: number := binary.BigEndian.Uint64(data)
. . 152: return &number
. . 153:}
. . 154:
. . 155:// WriteHeaderNumber stores the hash->number mapping.
. . 156:func WriteHeaderNumber(db ethdb.KeyValueWriter, hash common.Hash, number uint64) {
```
Opening this to discuss the idea, I know that rawdb.EmptyNumber is not a
great name for the variable, open to suggestions
---
**Description:**
- Replaced outdated GitHub wiki links with current, official
documentation URLs.
- Removed links that redirect or are no longer relevant.
- Ensured all references point to up-to-date and reliable sources.
---
This pull request slightly improves the freezer fsync mechanism by scheduling
the Sync operation based on the number of uncommitted items and original
time interval.
Originally, freezer.Sync was triggered every 30 seconds, which worked well during
active chain synchronization. However, once the initial state sync is complete,
the fixed interval causes Sync to be scheduled too frequently.
To address this, the scheduling logic has been improved to consider both the time
interval and the number of uncommitted items. This additional condition helps
avoid unnecessary Sync operations when the chain is idle.
Introduce file-based state journal in path database, fixing
the Pebble restriction when the journal size exceeds 4GB.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR fixes an issue in the tx_fetcher DoS prevention logic where the
code keeps the overflow amount (`want - maxTxAnnounces`) instead of the
allowed amount (`maxTxAnnounces - used`). The specific changes are:
- Correct slice indexing in the announcement drop logic
- Extend the overflow test case to cover the inversion scenario
This is a resubmit of https://github.com/ethereum/go-ethereum/pull/31820
against the `master` branch.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
---
**Description:**
- Replaced outdated GitHub wiki links with the official Ethereum
documentation for Web3 Secret Storage.
- Updated references in `keystore.go` and `passphrase.go` for improved
accuracy and reliability.
---
This pull request fixes an issue in disabling direct-ancient mode in
snap sync.
Specifically, if `origin >= frozen && origin != 0`, it implies a part of
chain data has been written into the key-value store, all the following
writes into ancient store scheduled by downloader will be rejected
with error
`ERROR[07-10|03:46:57.924] Error importing chain data to ancients
err="can't add block 1166 hash: the append operation is out-order: have
1166 want 0"`.
This issue is detected by the https://github.com/ethpandaops/kurtosis-sync-test,
which initiates the first snap sync cycle without the finalized header and
implicitly disables the direct-ancient mode. A few seconds later the second
snap sync cycle is initiated with the finalized information and direct-ancient mode
is enabled incorrectly.
This adds the SSZ types from the
[EIP-7928](https://eips.ethereum.org/EIPS/eip-7928) and also adds
encoder/decoder generation using https://github.com/ferranbt/fastssz.
The fastssz dependency is updated because the generation will not work
properly with the master branch version due to a bug in fastssz.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR updates the outdated documentation URL from docs.gnosis.io to
the new official docs.safe.global domain. The change reflects the
rebranding from Gnosis Safe to Safe and ensures that users are directed
to the current API documentation for transaction service reference.
This PR adds a block validation check for the maximum block size, as required by
EIP-7934, and also applies a slightly lower size limit during block building.
---------
Co-authored-by: spencer-tb <spencer@spencertaylorbrown.uk>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
alternate approach to https://github.com/ethereum/go-ethereum/pull/31328
suggested by @MariusVanDerWijden . This prevents Geth from outputting a
lot of logs when trying to commit on-demand dev mode blocks while the
client is shutting down.
The issue is hard to reproduce, but I've seen it myself and it is
annoying when it happens. I think this is a reasonable simple solution,
and we can revisit if we find that the output is still too large (i.e.
there is a large delay between initiating shut down and the simulated
beacon receiving the signal, while in this loop).
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This change enables more tests to run on GitHub actions. First, it
removes the `-short` flag passed to `go test`, unskipping some longer
running tests. We also enable the full consensus tests to run by
enabling submodules during git clone.
The EF now operates org wide runners with the `self-hosted-ghr` label.
These are auto-scaling runners which should ideally allow us to process
any amount of testing load we throw at them. The new runners have `HOME`
configured differently from the actual user home directory, so our
internal test for resolving `~` had to be adapted to work in this scenario.
This is a follow up PR after #32128 , Seems I've missed to add
--txlookuplimit as hidden. In hte meanwhile, I also add the other
deprecated flags into the output of `show-deprecated-flags`
Improves the SSTORE gas calculation a bit. Previously we would pull up
the state object twice. This is okay for existing objects, since they
are cached, however non-existing objects are not cached, thus we needed
to go through all 128 diff layers as well as the disk layer twice, just
for the gas calculation
```
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/core/vm
cpu: AMD Ryzen 9 5900X 12-Core Processor
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
Interpreter-24 1118.0n ± 2% 602.8n ± 1% -46.09% (p=0.000 n=10)
```
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Small update for logs when syncing with blsync. Downgrades the "latest
filled block is not available" to warn.
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
1. Fix the error return format.
**todo**: ~~`bindtype` needs more complex logic to fix it.~~
`
if err != nil {
return nil, err
}
if err == nil {
return obj, nil
}
`
2. ~~Return pointer type object to avoid copying the whole struct
content.~~
3. Give the panic decision to the user.
4. Fix empty line at the end of function.
**TODO**: ~~fix some related test cases.~~
---------
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
geth cmd: `geth --dev --dev.period 5`
call: `debug.setHead` to rollback several blocks.
If the `debug.setHead` call is delayed, it will trigger a panic with a
small probability, due to using the null point of
`fcResponse.PayloadID`.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This pull request refines the filtermap implementation, defining key
APIs for map and
epoch calculations to improve readability.
This pull request doesn't change any logic, it's a pure cleanup.
---------
Co-authored-by: zsfelfoldi <zsfelfoldi@gmail.com>
## Summary
This PR resolves Issue #31929 by reducing log noise generated by the log
indexer after `debug_setHead` operations.
## Problem Description
When `debug_setHead` is called to rewind the blockchain, blocks are
removed from the database. However, the log indexer's `ChainView`
objects may still hold references to these deleted blocks. When
`extendNonCanonical()` attempts to access these missing headers, it
results in:
1. **Repeated ERROR logs**: `Header not found number=X hash=0x...`
2. **Log noise** that can mask other important errors
3. **User confusion** about whether this indicates a real problem
## Root Cause Analysis
The issue occurs because:
- `debug_setHead` removes blocks from the blockchain database
- Log indexer's `ChainView` may still reference deleted block hashes
- `extendNonCanonical()` in `core/filtermaps/chain_view.go` tries to
fetch these missing headers
- The existing `return false` logic properly handles the error, but logs
at ERROR level
## Solution
This is a **logging improvement only** - no functional logic changes:
### Changes Made
1. **Log level**: Changed from `ERROR` to `DEBUG`
2. **Log message**: Enhanced with descriptive context about chain view
extension
3. **Comments**: Added explanation for when this situation occurs
4. **Behavior**: Maintains existing error handling (`return false` was
already present)
### Code Changes
```go
// Before
log.Error("Header not found", "number", number, "hash", hash)
return false
// After
// Header not found - this can happen after debug_setHead operations
// where blocks have been deleted. Return false to indicate the chain view
// is no longer valid rather than logging repeated errors.
log.Debug("Header not found during chain view extension", "number", number, "hash", hash)
return false
```
## Testing
### Automated Tests
- ✅ All existing filtermaps tests pass: `go test ./core/filtermaps -v`
- ✅ No regressions in related functionality
### Manual Verification
1. **Before fix**: Started geth in dev mode, generated blocks, called
`debug_setHead(3)` → **5 repeated ERROR logs**
2. **After fix**: Same scenario → **4 DEBUG logs, no ERROR noise**
### Test Environment
```bash
# Setup test environment
rm -rf ./dev-test-data
./build/bin/geth --dev --datadir ./dev-test-data --http --http.api debug,eth,net,web3 --verbosity 4
# Generate test blocks and trigger issue
curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"debug_setHead","params":["0x3"],"id":1}' http://localhost:8545
```
## Related Issues
- Fixes#31929
## Additional Context
This issue was reported as spurious error messages appearing after
`debug_setHead` operations. The investigation revealed that while the
error handling was functionally correct, the ERROR log level was
inappropriate for this expected scenario in development/debugging
workflows.
The fix maintains full compatibility while significantly improving the
debugging experience for developers using `debug_setHead`.
---------
Co-authored-by: Sun Tae, Kim <38067691+humblefirm@users.noreply.github.com>
Co-authored-by: zsfelfoldi <zsfelfoldi@gmail.com>
The address filter was never checked against a maximum limit, which can
be somewhat abusive for API nodes. This PR adds a limit similar to
topics
## Description (AI generated)
This pull request introduces a new validation to enforce a maximum limit
on the number of addresses allowed in filter criteria for Ethereum logs.
It includes updates to the `FilterAPI` and `EventSystem` logic, as well
as corresponding test cases to ensure the new constraint is properly
enforced.
### Core functionality changes:
* **Validation for maximum addresses in filter criteria**:
- Added a new constant, `maxAddresses`, set to 100, to define the
maximum allowable addresses in a filter.
- Introduced a new error, `errExceedMaxAddresses`, to handle cases where
the number of addresses exceeds the limit.
- Updated the `GetLogs` method in `FilterAPI` to validate the number of
addresses against `maxAddresses`.
- Modified the `UnmarshalJSON` method to return an error if the number
of addresses in the input JSON exceeds `maxAddresses`.
- Added similar validation to the `SubscribeLogs` method in
`EventSystem`.
### Test updates:
* **New test cases for address limit validation**:
- Added a test in `TestUnmarshalJSONNewFilterArgs` to verify that
exceeding the maximum number of addresses triggers the
`errExceedMaxAddresses` error.
- Updated `TestInvalidLogFilterCreation` to include a test case for an
invalid filter with more than `maxAddresses` addresses.
- Updated `TestInvalidGetLogsRequest` to test for invalid log requests
with excessive addresses.
These changes ensure that the system enforces a reasonable limit on the
number of addresses in filter criteria, improving robustness and
preventing potential performance issues.
---------
Co-authored-by: zsfelfoldi <zsfelfoldi@gmail.com>
Fix the issue after initial snap sync with `gcmode=archive` enabled.
```
NewPayload: inserting block failed error="history indexing is out of order, last: null, requested: 1"
```
---------
Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
core.BlockChainConfig.VmConfig is not a pointer, so setting the Tracer
on the `vmConfig` object after it was passed to options does *not* apply
it to options.VmConfig
This fixes the issue by setting the value directly inside the `options`
object and removing the confusing `vmConfig` variable to prevent further
mistakes.
This pull request tracks the state indexing progress in eth_syncing
RPC response, i.e. we will return non-null syncing status until indexing
has finished.
This pull request fixes a flaw in the PBSS state iterator, which
could return empty account or storage data.
In PBSS, multiple in-memory diff layers and a write buffer are
maintained. These layers are persisted to the database and reloaded after
node restarts. However, since the state data is encoded using RLP, the
distinction between nil and an empty byte slice is lost during the encode/decode
process. As a result, invalid state values such as `[]byte{}` can appear in PBSS
and ultimately be returned by the state iterator.
Checkout
https://github.com/ethereum/go-ethereum/blob/master/triedb/pathdb/iterator_fast.go#L270
for more iterator details.
It's a long-term existent issue and now be activated since the snapshot
integration.
The error `err="range contains deletion"` will occur when Geth tries to
serve other
peers with SNAP protocol request.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This pull request fixes a flaw in PBSS archive mode that significantly
degrades performance when the mode is enabled.
Originally, in hash mode, the dirty trie cache is completely disabled
when archive mode is active, in order to disable the in-memory garbage
collection mechanism. However, the internal logic in path mode differs
significantly, and the dirty trie node cache is essential for maintaining
chain insertion performance. Therefore, the cache is now retained in
path mode.
Previously, the account trie for a given state root was resolved immediately
when the stateDB was created, implying that the trie was always required
by the stateDB.
However, this assumption no longer holds, especially for path archive nodes,
where historical states can be accessed even if the corresponding trie data
does not exist.
Downloading from a range was failing because it would return and error
early with an error misinterpreting "start-end".
---------
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Replaced the outdated and broken link to the Web3 Secret Storage
Definition with the current official URL from ethereum.org in the Clef
README. This ensures users have access to up-to-date and accurate
documentation for the keystore file format.
If Geth is engaged in a long-run block synchronization, such as a full
syncing over a large number of blocks, invoking `debug_setHead` will
cause `downloader.Cancel` to wait for all fetchers to stop first.
This can be time-consuming, particularly for the block processing
thread.
To address this, we manually call `blockchain.StopInsert` to interrupt
the blocking processing thread and allow it to exit immediately, and
after that call `blockchain.ResumeInsert` to resume the block
downloading process.
Additionally, we add a sanity check for the input block number of
`debug_setHead` to ensure its validity.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Previously, PathDB used a single buffer to aggregate database writes,
which needed to be flushed atomically. However, flushing large amounts
of data (e.g., 256MB) caused significant overhead, often blocking the
system for around 3 seconds during the flush.
To mitigate this overhead and reduce performance spikes, a double-buffer
mechanism is introduced. When the active buffer fills up, it is marked
as frozen and a background flushing process is triggered. Meanwhile, a
new buffer is allocated for incoming writes, allowing operations to
continue uninterrupted.
This approach reduces system blocking times and provides flexibility in
adjusting buffer parameters for improved performance.
This pull request introduces a mechanism to expose statistics from the
state reader, specifically related to cache utilization during state prefetching.
To improve state access performance, a pair of state readers is constructed
with a shared local cache. One reader to execute transactions ahead of time
to warm up the cache. The other reader is used by the actual chain processing
logic, which can benefit from the prefetched states.
This PR adds visibility into how effective the cache is by exposing relevant
usage statistics.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR improves the IsOnCurve methods for BN254 G2 points by:
* Clarifying its behavior the docstring, making it explicit that it
verifies both the point being on the curve and in the correct subgroup.
* Adding an in-line comment explaining the subgroup membership check
(c.Mul(Order)).
* Minor wording adjustments for readability and consistency.
The optimization tried to defer allocating the cache map until it was used for the
first time. It's a relic from earlier times, when tries were copied often. This seems
unnecessary now, so we can just create the map when the trie is created.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In this pull request, the original `CacheConfig` has been renamed to `BlockChainConfig`.
Over time, more fields have been added to `CacheConfig` to support
blockchain configuration. Such as `ChainHistoryMode`, which clearly extends
beyond just caching concerns.
Additionally, adding new parameters to the blockchain constructor has
become increasingly complicated, since it’s initialized across multiple
places in the codebase. A natural solution is to consolidate these arguments
into a dedicated configuration struct.
As a result, the existing `CacheConfig` has been redefined as `BlockChainConfig`.
Some parameters, such as `VmConfig`, `TxLookupLimit`, and `ChainOverrides`
have been moved into `BlockChainConfig`. Besides, a few fields in `BlockChainConfig`
were renamed, specifically:
- `TrieCleanNoPrefetch` -> `NoPrefetch`
- `TrieDirtyDisabled` -> `ArchiveMode`
Notably, this change won't affect the command line flags or the toml
configuration file. It's just an internal refactoring and fully backward-compatible.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Since we have the effective gas price in the message, we can compute tip by
simply subtracting the basefee. No need to recompute the effective price.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Reading a single transaction out of a block shouldn't need decoding the
entire body
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
As https://github.com/ethereum/go-ethereum/pull/31769 defined a global
hash pool, so we can reuse it, and also remove the unnecessary
KeccakState buffering
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Adds marshaling fuzzing for G1 and G2 to oss-fuzz.
Also aligns the behavior of the google library to that of gnark and
cloudflare, which only ever read the first 64 / 128 bytes of the input,
regardless of how long the input is
Fixes a data race on the `wallets` slice when closing account Manager.
At the moment, there is a data race between a go-routine calling the
Manager's `Close` function and the background go-routine handling most
operations on the `Manager`. The `Manager`'s `wallets` field is accessed
without proper synchronization.
By moving the closing of wallets from the `Close()` function into the
background thread, this issue can be resolved.
This pull request reduces the threshold for triggering compaction at
level0, leading to less compaction debt. This change is helpful in the
case of heavy write-load, mitigating the case of heavy write stalls
caused by compaction.
closes https://github.com/ethereum/go-ethereum/issues/31830
When `GetKey` is called, a missing preimage can cause the function to return a `nil`
key. This, in turn, makes `account.Storage` persist an incorrect value.
This fixes a data race when accessing the `httpConfig.prefix` field.
This field can be modified while the server is running through
`enableRPC`. The fix is storing the prefix in the handler, which is
accessed through the atomic pointer.
alternative to #32035
fixes https://github.com/ethereum/go-ethereum/issues/32019
The prestateTracer had the intention of excluding accounts that were
empty prior to execution from the prestate. This was being done only for
created contracts. This PR makes it so all such empty accounts are
excluded. This behavior is configurable using the `includeEmpty: true`
flag introduced in #31855.
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR introduces a flag that enables returning of newly created state
objects in the prestateTracer.
**Rationale**
Having this information is useful because local execution can more
easily distinguish between newly created objects and system contracts.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
As the preimage will only be stored if `t.preimages != nil`, so no need
to save them into local cache if not enabled. This will reduce the memory
wasted to copy the bytes
---------
Signed-off-by: jsvisa <delweng@gmail.com>
This is a followup to #31753.
A cumulative counter is more useful when we need to measure / aggregate
the metric over a longer period of time. It also means we won't miss data,
e.g. our prometheus scrapes every 30 seconds, and so may miss a transient
spike in the pre-aggregated mgas/s.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
With EOF removed from the Osaka fork, and no longer being tested, the
implementation will now just be bitrotting. I'm opting to remove it so
it doesn't get in the way of other changes.
This pull request introduces a new test suite in workload framework, for
transaction tracing.
**test generation**
`go run . tracegen --trace-tests trace-test.json http://host:8545`
and you can choose to store the trace result in a specific folder
`go run . tracegen --trace-tests trace-test.json --trace-output
./trace-result http://host:8545`
**test run**
`./workload test -run Trace/Transaction --trace-invalid ./trace-invalid
http://host:8545`
The mismatched trace result will be saved in the specific folder for
further investigation.
This pull request adjusts the number of allowed memory tables in Pebble.
Pebble allows configuring an arbitrary number of memory tables to hold
unflushed data. When the current memtable becomes full, it is scheduled
for flushing, and a new memtable is allocated to accept subsequent
writes. However, if too many memtables accumulate and are waiting to be
flushed, subsequent writes will stall.
Originally, only two memtables were configured, each with a size of 512
MB for Ethereum mainnet. While this setup works well under normal
conditions, it becomes problematic under heavy write loads. In such scenarios,
flushing is only triggered when more than 512 MB of data is pending, which may
not be responsive enough. Even worse, if compactions are running
concurrently, flushing memtables can become slow due to the heavy IO
overhead, leading to write stalls across the system.
This pull request tries to mitigate the performance degradation by having
more memory tables but with a smaller size. In this case, the pending
writes can be flushed more smoothly and responsively.
This changes the era1 downloader to place the files into the correct
location where geth will actually use them. Also adds integration with
the new --datadir.era flag.
This PR improves the speed of Disc/v4 and Disc/v5 based discovery by
adding a prefetch buffer to discovery sources, eliminating slowdowns
due to timeouts and rate mismatch between the two processes.
Since we now want to filter the discv4 nodes iterator, it is being removed
from the default discovery mix in p2p.Server. To keep backwards-compatibility,
the default unfiltered discovery iterator will be utilized by the server when
no protocol-specific discovery is configured.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This fixes a regression introduced by #29158 where receipts of empty blocks
were stored into the database as an empty byte array, instead of an RLP empty list.
Fixes#31938
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This is an alternative approach to
https://github.com/ethereum/go-ethereum/pull/31607 , that doesn't break
backwards-compatibility with abigen.
Note that this does change the behavior of `Argument.Pack`: previously,
packing negative values for a `uint` parameter would cause them to be
represented in signed binary representation via two's complement. Now,
it will fail explicitly in this case.
However, I don't see a reason to support this functionality. The ABI
already explicitly supports signed integers. There's no reason that a
smart contract author would choose to store signed values in a `uint`
afaict.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Fixes an issue where querying logs for block ranges starting from 0 would fail with an irrelevant
error on a pruned node. Now the correct "history is pruned" error will be returned.
This situation was failing quietly for me recently when I had a partial
data corruption issue. Changing the log level to Error would increase
visibility for me.
This PR changes the database access of the base part of filter rows that
are stored in groups of 32 adjacent maps for improved database storage
size and data access efficiency.
Before this grouped storage was introduced, filter rows were not cached
because the access pattern of either the index rendering or the search
does not really benefit from caching. Also no mutex was necessary for
filter row access. Storing adjacent rows in groups complicated the
situation as a search typically required reading all or most of adjacent
rows of a group, so in order to implement the single row read operation
without having to read the entire group up to 32 times, a cache for the
base row groups was added. This also introduced data race issues for
concurrenct read/write in the same group which was avoided by locking
the `indexLock` mutex. Unfortunately this also led to slowed down or
temporarily blocked search operations when indexing was in progress.
This PR returns to the original concept of uncached, no-mutex filter map
access by increasing read efficiency in a better way; similiarly to
write operations that already operate on groups of filter maps, now
`getFilterMapRow` is also replaced by `getFilterMapRows` that accepts a
single `rowIndex` and a list of `mapIndices`. It slightly complicates
`singleMatcherInstance.getMatchesForLayer` which now has to collect
groups of map indices accessed in the same row, but in exchange it
guarantees maximum read efficiency while avoiding read/write mutex
interference.
Note: a follow-up refactoring is WIP that further changes the database
access scheme by prodiving an immutable index view to the matcher, makes
the whole indexer more straightforward with no callbacks, and entirely
removes the concept of matcher syncing with `validBlocks` and the
resulting multiple retry logic in `eth/filters/filter.go`. This might
take a bit longer to finish though and in the meantime this change could
hopefully already solve the blocked request issues.
This implements a backing store for chain history based on era1 files.
The new store is integrated with the freezer. Queries for blocks and receipts
below the current freezer tail are handled by the era store.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
This changes the API backend to return null for not-found blocks. This behavior
is required by the RPC When `BlockByNumberOrHash` always returned an error
for this case ever since being added in https://github.com/ethereum/go-ethereum/pull/19491.
The backend method has a couple of call sites, and all of them handle a `nil`
block result because `BlockByNumber` returns `nil` for not-found.
The only case where this makes a real difference is for `eth_getBlockReceipts`,
which was changed in #31361 to actually forward the error from `BlockByNumberOrHash`
to the caller.
Release artefact building has been migrated to another system (Gitea),
so we can finally stop using Travis CI. However, in order to have a
fail-safe for the release, I'm leaving the config in and it will still
trigger builds on Travis for tagged releases. That way, if our new
system fails to work for the next release, we will still have the option
of using Travis.
This PR modifies the disclaimer/banner that is printed when starting up
Geth in dev mode:
* if the client is spun up in ephemeral dev mode with a keystore
override, the address of the first (prefunded) account is printed.
* if the client is spun up in ephemeral mode without a keystore
override, the genesis allocation contains a single static prefunded EOA
account. It's address and private key are logged.
* the banner is printed at the end of client initialization to make it
more prominent. Previously, it was logged towards the beginning of
client initialization and subsequent logging from startup filled the
terminal, pushing it out of view of the user.
Other change is that we now use a static prefunded dev account instead
of generating a random one when instantiating a new dev mode chain.
This is an example of what the banner looks like:
```
WARN [05-28|23:05:16.475] You are running Geth in --dev mode. Please note the following:
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] 1. This mode is only intended for fast, iterative development without assumptions on
WARN [05-28|23:05:16.475] security or persistence.
WARN [05-28|23:05:16.475] 2. The database is created in memory unless specified otherwise. Therefore, shutting down
WARN [05-28|23:05:16.475] your computer or losing power will wipe your entire block data and chain state for
WARN [05-28|23:05:16.475] your dev environment.
WARN [05-28|23:05:16.475] 3. A random, pre-allocated developer account will be available and unlocked as
WARN [05-28|23:05:16.475] eth.coinbase, which can be used for testing. The random dev account is temporary,
WARN [05-28|23:05:16.475] stored on a ramdisk, and will be lost if your machine is restarted.
WARN [05-28|23:05:16.475] 4. Mining is enabled by default. However, the client will only seal blocks if transactions
WARN [05-28|23:05:16.475] are pending in the mempool. The miner's minimum accepted gas price is 1.
WARN [05-28|23:05:16.475] 5. Networking is disabled; there is no listen-address, the maximum number of peers is set
WARN [05-28|23:05:16.475] to 0, and discovery is disabled.
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] Running in ephemeral mode. The following account has been prefunded in the genesis:
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] Account
WARN [05-28|23:05:16.475] ------------------
WARN [05-28|23:05:16.475] 0x71562b71999873db5b286df957af199ec94617f7 (10^49 ETH)
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] Private Key
WARN [05-28|23:05:16.475] ------------------
WARN [05-28|23:05:16.475] 0xb71c71a67e1177ad4e901695e1b4b9ee17ae16c6668d313eac2f96dbcda3f291
WARN [05-28|23:05:16.475]
```
closes#31796
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This pull request introduces a mechanism to improve state lookup
efficiency in pathdb by maintaining a lookup structure that eliminates
unnecessary iteration over diff layers.
The core idea is to track a mutation history for each dirty state entry
residing in the diff layers. This history records the state roots of all layers
in which the entry was modified, sorted from oldest to newest.
During state lookup, this mutation history is queried to find the most
recent layer whose state root either matches the target root or is a
descendant of it. This allows us to quickly identify the layer containing
the relevant data, avoiding the need to iterate through all diff layers from
top to bottom.
Besides, the overhead for state lookup is constant, no matter how many
diff layers are retained in the pathdb, which unlocks the potential to hold
more diff layers.
Of course, maintaining this lookup structure introduces some overhead.
For each state transition, we need to:
(a) update the mutation records for the modified state entries, and
(b) remove stale mutation records associated with outdated layers.
On our benchmark machine, it will introduce around 1ms overhead which is
acceptable.
This adds support for the Github actions environment in the build tool.
Information from environment variables, like the build number and
branch/tag name, is used to make decisions about uploads and package
filenames.
Updated reference URL in accumulator.go comment to point to the correct
location of the historical-hashes-accumulator documentation in the
Ethereum portal network specs
We deleted outdated pectra-devnet-6@v1.0.0 release by mistake, so this
PR updates the referenced EEST release to the correct latest version.
@s1na I removed the TODO comment because I think this solves it, unless
it meant something else.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This PR introduces a new native tracer for AA bundlers. Bundlers participating in the alternative
mempool will need to validate userops. This tracer will return sufficient information for them to
decide whether griefing is possible. Resolves#30546
---------
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
Some tests involving transactions near the txMaxSize limit were flaky.
This was due to ECDSA signatures occasionally having leading zeros,
which are omitted during RLP encoding — making the final transaction
size 1 byte smaller than expected.
To address this, a new helper function pricedDataTransactionWithFixedSignature
was added. It ensures both r and s are exactly 32 bytes (i.e., no leading zeros),
producing transactions with deterministic size.
I've updated the broken link to point to the current official Ethereum
JSON-RPC API documentation at
https://ethereum.org/en/developers/docs/apis/json-rpc/. This is the
correct and up-to-date location for the Ethereum Execution Layer APIs
documentation. The link should now work properly.
I added a test for BlockRangeUpdate in #29158 but forgot to enable it.
Here I'm adding two more tests for it. Also applied a small refactoring
to combine calls to `dial()` and `peer()` into a single function, since
these two calls are duplicated in each test.
This PR implements eth/69. This protocol version drops the bloom filter
from receipts messages, reducing the amount of data needed for a sync
by ~530GB (2.3B txs * 256 byte) uncompressed. Compressed this will
be reduced to ~100GB
The new version also changes the Status message and introduces the
BlockRangeUpdate message to relay information about the available history
range.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In this pull request, snapshot generation in pathdb has been ported from
the legacy state snapshot implementation. Additionally, when running in
path mode, legacy state snapshot data is now managed by the pathdb
based snapshot logic.
Note: Existing snapshot data will be re-generated, regardless of whether
it was previously fully constructed.
This adds a geth subcommand for downloading era1 files and placing them into
the correct location. The tool can be used even while geth is already running
on the datadir. Downloads are checked against a hard-coded list of checksums
for mainnet and sepolia.
```
./geth download-era --server $SERVER --block 333333
./geth download-era --server $SERVER --block 333333-444444
./geth download-era --server $SERVER --epoch 0-10
./geth download-era --server $SERVER --all
```
The implementation reuses the file downloader we already had for
fetching build tools. I've done some refactoring on it to make sure it
can support the new use case, and there are some changes to the build
here as well.
Adding values to the witness introduces a new class of issues for
computing gas: if there is not enough gas to cover adding an item to the
witness, then the item should not be added to the witness.
The problem happens when several items are added together, and that
process runs out of gas. The witness gas computation needs a way to
signal that not enough gas was provided. These values can not be
hardcoded, however, as they are context dependent, i.e. two calls to the
same function with the same parameters can give two different results.
The approach is to return both the gas that was actually consumed, and
the gas that was necessary. If the values don't match, then a witness
update OOG'd. The caller should then charge the `consumed` value
(remaining gas will be 0) and error out.
Why not return a boolean instead of the wanted value? Because when
several items are touched, we want to distinguish which item lacked gas.
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
This adds support for naming the source iterators of FairMix, like so:
mix.AddSource(enode.WithSourceName("mySource", iter))
The source that produced the latest node is returned by the new NodeSource method.
This PR contains three refactors:
- refactor the latest fork check that we use quite extensively
- refactor the nil checks in NewPayloads
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
I saw in https://github.com/ethereum/go-ethereum/pull/31378 introduced
github.com/crate-crypto/go-eth-kzg to calculate the kzg hash, and
github.com/crate-crypto/go-kzg-4844 was only used in the test files, so
propose to drop it with go-eth-kzg instead
This adds a metric called `chain/mgasps`, which records how many million
gas per second are being used during block insertion.
The value is calculated as `usedGas * 1000 / elapsed`, and it's updated
in the `insertStats.report` method. Also cleaned up the log output to
reuse the same value instead of recalculating it.
Useful for monitoring block processing throughput.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR introduces an allocation-free version of the
Transaction.EffectiveGasTip method to improve performance by reducing
memory allocations.
## Changes
- Added a new `EffectiveGasTipInto` method that accepts a destination
parameter to avoid memory allocations
- Refactored the existing `EffectiveGasTip` method to use the new
allocation-free implementation
- Updated related methods (`EffectiveGasTipValue`, `EffectiveGasTipCmp`,
`EffectiveGasTipIntCmp`) to use the allocation-free approach
- Added tests and benchmarks to verify correctness and measure
performance improvements
## Motivation
In high-transaction-volume environments, the `EffectiveGasTip` method is
called frequently. Reducing memory allocations in this method decreases
garbage collection pressure and improves overall system performance.
## Benchmark Results
As-Is
BenchmarkEffectiveGasTip/Original-10 42089140 27.45 ns/op 8 B/op 1
allocs/op
To-Be
BenchmarkEffectiveGasTip/IntoMethod-10 72353263 16.73 ns/op 0 B/op 0
allocs/op
## Summary of Improvements
- **Performance**: ~39% faster execution (27.45 ns/op → 16.73 ns/op)
- **Memory**: Eliminated all allocations (8 B/op → 0 B/op)
- **Allocation count**: Reduced from 1 to 0 allocations per operation
This optimization follows the same pattern successfully applied to other
methods in the codebase, maintaining API compatibility while improving
performance.
## Safety & Compatibility
This optimization has no side effects or adverse impacts because:
- It maintains functional equivalence as confirmed by comprehensive
tests
- It preserves API compatibility with existing callers
- It follows clear memory ownership patterns with the destination
parameter
- It maintains thread safety by only modifying the caller-provided
destination parameter
This optimization follows the same pattern successfully applied to other
methods in the codebase, providing better performance without
compromising stability or correctness.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR creates a global hasher pool that can be used by all packages.
It also removes a bunch of the package local pools.
It also updates a few locations to use available hashers or the global
hashing pool to reduce allocations all over the codebase.
This change should reduce global allocation count by ~1%
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This pull request enhances the block prefetcher by executing transactions
in parallel to warm the cache alongside the main block processor.
Unlike the original prefetcher, which only executes the next block and
is limited to chain syncing, the new implementation can be applied to any
block. This makes it useful not only during chain sync but also for regular
block insertion after the initial sync.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR fixes an issue that could lead to data corruption.
Writing the state history may fail due to insufficient disk space or
other potential errors. With this change, the entire state insertion
will be aborted instead of silently ignoring the error.
Without this fix, state transitions would continue while the associated
state history is lost. After a restart, the resulting gap would be detected,
making recovery impossible.
This pull request introduces a SyncKeyValue function to the
ethdb.KeyValueStore
interface, providing the ability to forcibly flush all previous writes
to disk.
This functionality is critical for go-ethereum, which internally uses
two independent
database engines: a key-value store (such as Pebble, LevelDB, or
memoryDB for
testing) and a flat-file–based freezer. To ensure write-order
consistency between
these engines, the key-value store must be explicitly synced before
writing to the
freezer and vice versa.
Fixes
- https://github.com/ethereum/go-ethereum/issues/31405
- https://github.com/ethereum/go-ethereum/issues/29819
The function `BacktraceAt` has been removed in #28187 . But the API
end-point `debug_backtraceAt` is not removed from the file
`internal/web3ext/web3ext.go`.
Fixes methods debug_standardTraceBlockToFile
and debug_standardTraceBadBlockToFile which were
outputting empty files.
---------
Co-authored-by: maskpp <maskpp266@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change adds a limit for RPC method names to prevent potential abuse
where large method names could lead to large response sizes.
The limit is enforced in:
- handleCall for regular RPC method calls
- handleSubscribe for subscription method calls
Added tests in websocket_test.go to verify the length limit
functionality for both regular method calls and subscriptions.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Issue statement: when user requests eth_simulateV1 to return full
transaction objects, these objects always had an empty `from` field. The
reason is we lose the sender when translation the message into a
types.Transaction which is then later on serialized.
I did think of an alternative but opted to keep with this approach as it
keeps complexity at the edge. The alternative would be to pass down a
signer object to RPCMarshal* methods and define a custom signer which
keeps the senders in its state and doesn't attempt the signature
recovery.
updates the log entries in `core/filtermaps/indexer.go` to remove double
quotes around keys like "first block" and "last block", changing them to
`firstblock` and `lastblock`. This brings them in line with the general
logging style used across the codebase, where log keys are unquoted
single words.
For example, the log:
` INFO [...] "first block"=..., "last block"=...`
Is now rendered as:
` INFO [...] firstblock=..., lastblock=...`
This change improves readability and maintains consistency with logs
such as:
` INFO [...] number=2 sealhash=... uncles=0 txs=0 ...`
No functional behavior is changed — this is purely a formatting cleanup
for better developer experience.
Fixes https://github.com/ethereum/go-ethereum/issues/31732.
This logic was removed in the recent refactoring in the txindexer to
handle history cutoff (#31393). It was first introduced in this PR:
https://github.com/ethereum/go-ethereum/pull/28908.
I have tested it and it works as an alternative to #31745.
This PR packs 3 changes to the flow of fetching txs from the API:
- It caches the indexer tail after each run is over to avoid hitting the
db all the time as was done originally in #28908.
- Changes `backend.GetTransaction`. It doesn't return an error anymore
when tx indexer is in progress. It shifts the responsibility to the
caller to check the progress. The reason is that in most cases we anyway
check the txpool for the tx. If it was indeed a pending tx we can avoid
the indexer progress check.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
`DefaultBlobSchedule` is actually used downstream to calculate blob fees
(e.g.,
[src](601a380e47/op-service/eth/blob.go (L301))),
this PR makes it explicit that these params are for `Ethereum prod`
instead of `test chains`.
This PR fixes an initialization bug that in some cases caused the map
renderer to leave the last, partially rendered map as is and resume
rendering from the next map. At initialization we check whether the
existing rendered maps are consistent with the current chain view and
revert them if necessary. Until now this happened through an ugly hacky
solution, a "limited" chain view that was supposed to trigger a rollback
of some maps in the renderer logic if necessary. This whole setup worked
under assumptions that just weren't true any more. As a result it always
tried to revert the last map but also it did not shorten the indexed
range, only set `headIndexed` to false which indicated to the renderer
logic that the last map is fully populated (which it wasn't).
Now an explicit rollback of any unusable (reorged) maps happens at
startup, which also means that no hacky chain view is necessary, as soon
as the new `FilterMaps` is returned, the indexed range and view are
consistent with each other.
In the first commit an extra check is also added to `writeFinishedMaps`
so that if there is ever again a bug that would result in a gapped index
then it will not break the db with writing the incomplete data. Instead
it will return an indexing error which causes the indexer to revert to
unindexed mode and print an error log instantly. Hopefully this will not
ever happen in the future, but in order to test this safeguard check I
manually triggered the bug with only the first commit enabled, which
caused an indexing error as expected. With the second commit added (the
actual fix) the same operation succeeded without any issues.
Note that the database version is also bumped in this PR in order to
enforce a full reindexing as any existing database might be potentially
broken.
Fixes https://github.com/ethereum/go-ethereum/issues/31729
This PR fixes the out-of-range block number logic of `getBlockLvPointer`
which sometimes caused searches to fail if the head was updated in the
wrong moment. This logic ensures that querying the pointer of a future
block returns the pointer after the last fully indexed block (instead of
failing) and therefore an async range update will not cause the search
to fail. Earier this behaviour only worked when `headIndexed` was true
and `headDelimiter` pointed to the end of the indexed range. Now it also
works for an unfinished index.
This logic is also moved from `FilterMaps.getBlockLvPointer` to
`FilterMapsMatcherBackend.GetBlockLvPointer` because it is only required
by the search anyways. `FilterMaps.getBlockLvPointer` now only returns a
pointer for existing blocks, consistently with how it is used in the
indexer/renderer.
Note that this unhandled case has been present in the code for a long
time but went unnoticed because either one of two previously fixed bugs
did prevent it from being triggered; the incorrectly positive
`tempRange.headIndexed` (fixed in
https://github.com/ethereum/go-ethereum/pull/31680), though caused other
problems, prevented this one from being triggered as with a positive
`headIndexed` no database read was triggered in `getBlockLvPointer`.
Also, the unnecessary `indexLock` in `synced()` (fixed in
https://github.com/ethereum/go-ethereum/pull/31708) usually did prevent
the search seeing the temp range and therefore avoided noticeable
issues.
The functions `rpcRequest` and `batchRpcRequest` call `baseRpcRequest`.
And `resp.Body` will be closed in the function `baseRpcRequest` later by
`t.Cleanup`:
```go
func baseRpcRequest(t *testing.T, url, bodyStr string, extraHeaders ...string) *http.Response {
// ......
t.Cleanup(func() { resp.Body.Close() })
return resp
}
```
Add tests for GetBlockHeaders that verify client does not disconnect when unlikely block numbers are requested, e.g. max uint64.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Since the block hash is not returned for pending blocks, ethclient cannot unmarshal into RPC block. This makes hash optional on rpc block and compute the hash locally for pending blocks to correctly key the tx sender cache.
a82303f4e3/internal/ethapi/api.go (L500-L504)
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This changes the filtermaps to only pull up the raw receipts, not the
derived receipts which saves a lot of allocations.
During normal execution this will reduce the allocations of the whole
geth node by ~15%.
For PeerDAS, we need to compute cell proofs. Both ckzg and gokzg support
computing these cell proofs.
This PR does the following:
- Update the go-kzg library from "github.com/crate-crypto/go-kzg-4844"
to "github.com/crate-crypto/go-eth-kzg" which will be the new upstream
for go-kzg moving forward
- Update ckzg from v1.0.0 to v2.0.1 and switch to /v2
- Updates the trusted setup to contain the g1 points both in lagrange
and monomial form
- Expose `ComputeCells` to compute the cell proofs
This PR applies the config overrides to the new config as well,
otherwise they will not be applied to defined configs, making
shadowforks impossible.
To test:
```
> ./build/bin/geth --override.prague 123 --dev --datadir /tmp/geth
INFO [04-28|21:20:47.009] - Prague: @123
> ./build/bin/geth --override.prague 321 --dev --datadir /tmp/geth
INFO [04-28|21:23:59.760] - Prague: @321
``
This PR adds checking for an edgecase which theoretically can happen in
the range-prover. Right now, we check that a key does not overwrite a
previous one by checking that the key is increasing. However, if keys
are of different lengths, it is possible to create a key which is
increasing _and_ overwrites the previous key. Example: `0xaabbcc`
followed by `0xaabbccdd`.
This can not happen in go-ethereum, which always uses fixed-size paths
for accounts and storage slot paths in the trie, but it might happen if
the range prover is used without guaranteed fixed-size keys.
This PR also adds some testcases for the errors that are expected.
TruncatePending shows up bright red on our nodes, because it computes
the length of a map multiple times.
I don't know why this is so expensive, but around 20% of our time is
spent on this, which is super weird.
```
//PR: BenchmarkTruncatePending-24 17498 69397 ns/op 32872 B/op 3 allocs/op
//Master: BenchmarkTruncatePending-24 9960 123954 ns/op 32872 B/op 3 allocs/op
```
```
benchmark old ns/op new ns/op delta
BenchmarkTruncatePending-24 123954 69397 -44.01%
benchmark old allocs new allocs delta
BenchmarkTruncatePending-24 3 3 +0.00%
benchmark old bytes new bytes delta
BenchmarkTruncatePending-24 32872 32872 +0.00%
```
This simple PR is a 44% improvement over the old state
```
OUTINE ======================== github.com/ethereum/go-ethereum/core/txpool/legacypool.(*LegacyPool).truncatePending in github.com/ethereum/go-ethereum/core/txpool/legacypool/legacypool.go
1.96s 18.02s (flat, cum) 19.57% of Total
. . 1495:func (pool *LegacyPool) truncatePending() {
. . 1496: pending := uint64(0)
60ms 2.99s 1497: for _, list := range pool.pending {
250ms 5.48s 1498: pending += uint64(list.Len())
. . 1499: }
. . 1500: if pending <= pool.config.GlobalSlots {
. . 1501: return
. . 1502: }
. . 1503:
. . 1504: pendingBeforeCap := pending
. . 1505: // Assemble a spam order to penalize large transactors first
. 510ms 1506: spammers := prque.New[int64, common.Address](nil)
140ms 2.50s 1507: for addr, list := range pool.pending {
. . 1508: // Only evict transactions from high rollers
50ms 5.08s 1509: if uint64(list.Len()) > pool.config.AccountSlots {
. . 1510: spammers.Push(addr, int64(list.Len()))
. . 1511: }
. . 1512: }
. . 1513: // Gradually drop transactions from offenders
. . 1514: offenders := []common.Address{}
```
```go
// Benchmarks the speed of batch transaction insertion in case of multiple accounts.
func BenchmarkTruncatePending(b *testing.B) {
// Generate a batch of transactions to enqueue into the pool
pool, _ := setupPool()
defer pool.Close()
b.ReportAllocs()
batches := make(types.Transactions, 4096+1024+1)
for i := range len(batches) {
key, _ := crypto.GenerateKey()
account := crypto.PubkeyToAddress(key.PublicKey)
pool.currentState.AddBalance(account, uint256.NewInt(1000000), tracing.BalanceChangeUnspecified)
tx := transaction(uint64(0), 100000, key)
batches[i] = tx
}
for _, tx := range batches {
pool.addRemotesSync([]*types.Transaction{tx})
}
b.ResetTimer()
// benchmark truncating the pending
for range b.N {
pool.truncatePending()
}
}
```
This PR fixes a deadlock situation is deleteTailEpoch that might arise
when
range delete is running in iterator based fallback mode (either using
leveldb
database or the hashdb state storage scheme).
In this case a stopCb callback is called periodically that does check
events,
including matcher sync requests, in which case it tries to acquire
indexLock
for read access, while deleteTailEpoch already held it for write access.
This pull request removes the indexLock acquiring in
`FilterMapsMatcherBackend.synced`
as this function is only called in the indexLoop.
Fixes https://github.com/ethereum/go-ethereum/issues/31700
This PR adds the `AuthorizationList` field to the `CallMsg` interface to support `eth_call`
and `eth_estimateGas` of set-code transactions.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR ensures that caching a slice or a slice of slices will never
affect the original version by always cloning a slice fetched from cache
if it is not used in a guaranteed read only way.
This PR changes the chain view update mechanism of the log filter.
Previously the head updates were all wired through the indexer, even in
unindexed mode. This was both a bit weird and also unsafe as the
indexer's chain view was updates asynchronously with some delay, making
some log related tests flaky. Also, the reorg safety of the indexed
search was integrated with unindexed search in a weird way, relying on
`syncRange.ValidBlocks` in the unindexed case too, with a special
condition added to only consider the head of the valid range but not the
tail in the unindexed case.
In this PR the current chain view is directly accessible through the
filter backend and unindexed search is also chain view based, making it
inherently safe. The matcher sync mechanism is now only used for indexed
search as originally intended, removing a few ugly special conditions.
The PR is currently based on top of
https://github.com/ethereum/go-ethereum/pull/31642
Together they fix https://github.com/ethereum/go-ethereum/issues/31518
and replace https://github.com/ethereum/go-ethereum/pull/31542
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The API `eth_feeHistory` returns
`{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"json:
unsupported value: NaN"}}`, when we query `eth_feeHistory` with a old
block that without a blob, or when the field
`config.blobSchedule.cancun.max` in genesis.config is 0 (that happens
for some projects fork geth but they don't have blob).
So here we specially handle the case when maxBlobGas == 0 to prevent
this issue from happening.
This PR makes `filtermaps.ChainView` thread safe because it is used
concurrently both by the indexer and multiple matcher threads. Even
though it represents an immutable view of the chain, adding a mutex lock
to the `blockHash` function is necessary because it does so by extending
its list of non-canonical hashes if the underlying blockchain is
changed.
The unsafe concurrency did cause a panic once after running the unit
tests for several hours and it could also happen during live operation.
This PR makes the conditions for using a map rendering snapshot stricter
so that whenever a reorg happens, only a snapshot of a common ancestor
block can be used. The issue fixed in
https://github.com/ethereum/go-ethereum/pull/31642 originated from using
a snapshot that wasn't a common ancestor. For example in the following
reorg scenario: `A->B`, then `A->B2`, then `A->B2->C2`, then `A->B->C`
the last reorg triggered a render from snapshot `B` saved earlier. Now
this is possible under certain conditions but extra care is needed, for
example if block `B` crosses a map boundary then it should not be
allowed. With the latest fix the checks are sufficient but I realized I
would just feel safer if we disallowed this rare and risky scenario
altogether and just render from snapshot `A` after the last reorg in the
example above. The performance difference if a few milliseconds and it
occurs rarely (about once a day on Holesky, probably much more rare on
Mainnet).
Note that this PR only makes the snapshot conditions stricter and
`TestIndexerRandomRange` does check that snapshots are still used
whenever it's obviously possible (adding blocks after the current head
without a reorg) so this change can be considered safe. Also I am
running the unit tests and the fuzzer and everything seems to be fine.
This pull request improves error handling for local transaction submissions.
Specifically, if a transaction fails with a temporary error but might be
accepted later, the error will not be returned to the user; instead, the
transaction will be tracked locally for resubmission.
However, if the transaction fails with a permanent error (e.g., invalid
transaction or insufficient balance), the error will be propagated to the user.
These errors returned in the legacyPool are regarded as temporary failure:
- `ErrOutOfOrderTxFromDelegated`
- `txpool.ErrInflightTxLimitReached`
- `ErrAuthorityReserved`
- `txpool.ErrUnderpriced`
- `ErrTxPoolOverflow`
- `ErrFutureReplacePending`
Notably, InsufficientBalance is also treated as a permanent error, as
it’s highly unlikely that users will transfer funds into the sender account
after submitting the transaction. Otherwise, users may be confused—seeing
their transaction submitted but unaware that the sender lacks sufficient funds—and
continue waiting for it to be included.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
closes#31401
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
closes#31310
This has been requested a few times in the past and I think it is a nice
quality-of-life improvement for users. At a predetermined interval,
there will now be a "Fork ready" log when a future fork is scheduled,
but not yet active.
It can only possibly print after block import, which kinda avoids the
scenario where the client isn't progressing or is syncing and the user
thinks it's "ready" because it sees a ready log.
New output:
```console
INFO [03-08|21:32:57.472] Imported new potential chain segment number=7 hash=aa24ee..f09e62 blocks=1 txs=0 mgas=0.000 elapsed="874.916µs" mgasps=0.000 snapdiffs=973.00B triediffs=7.05KiB triedirty=0.00B
INFO [03-08|21:32:57.473] Ready for fork activation fork=Prague date="18 Mar 25 19:29 CET" remaining=237h57m0s timestamp=1,742,322,597
INFO [03-08|21:32:57.475] Chain head was updated number=7 hash=aa24ee..f09e62 root=19b0de..8d32f2 elapsed="129.125µs"
```
Easiest way to verify this behavior is to apply this patch and run `geth
--dev --dev.period=12`
```patch
diff --git a/params/config.go b/params/config.go
index 9c7719d901..030c4f80e7 100644
--- a/params/config.go
+++ b/params/config.go
@@ -174,7 +174,7 @@ var (
ShanghaiTime: newUint64(0),
CancunTime: newUint64(0),
TerminalTotalDifficulty: big.NewInt(0),
- PragueTime: newUint64(0),
+ PragueTime: newUint64(uint64(time.Now().Add(time.Hour * 300).Unix())),
BlobScheduleConfig: &BlobScheduleConfig{
Cancun: DefaultCancunBlobConfig,
Prague: DefaultPragueBlobConfig,
```
BroadcastTransactions needs the Sender address to route message flows
from the same Sender address consistently to the same random subset of
peers. It however spent considerable time calculating the Sender
addresses, even if the Sender address was already calculated and cached
in other parts of the code.
Since we only need the mapping, we can use any signer, and the one that
had already been used is a better choice because of cache reuse.
This is an attempt at fixing #31601. I think what happens is the startup
logic will try to get the full block body (it's `bc.loadLastState`) and
fail because genesis block has been pruned from the freezer. This will
cause it to keep repeating the reset logic, causing a deadlock.
This can happen when due to an unsuccessful sync we don't have the state
for the head (or any other state) fully, and try to redo the snap sync.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This fixes an issue where running geth with `--history.chain postmerge`
would not work on an empty database.
```
ERROR[04-16|23:11:12.913] Chain history database is pruned to unknown block tail=0
Fatal: Failed to register the Ethereum service: unexpected database tail
```
This PR fixes a bug in the map renderer that sometimes used an obsolete
block log value pointer to initialize the iterator for rendering from a
snapshot. This bug was triggered by chain reorgs and sometimes caused
indexing errors and invalid search results. A few other conditions are
also made safer that were not reported to cause issues yet but could
potentially be unsafe in some corner cases. A new unit test is also
added that reproduced the bug but passes with the new fixes.
Fixes https://github.com/ethereum/go-ethereum/issues/31593
Might also fix https://github.com/ethereum/go-ethereum/issues/31589
though this issue has not been reproduced yet, but it appears to be
related to a log index database corruption around a specific block,
similarly to the other issue.
Note that running this branch resets and regenerates the log index
database. For this purpose a `Version` field has been added to
`rawdb.FilterMapsRange` which will also make this easier in the future
if a breaking database change is needed or the existing one is
considered potentially broken due to a bug, like in this case.
Our metrics related to dial errors were off. The original error was not
wrapped, so the caller function had no chance of picking it up.
Therefore the most common error, which is "TooManyPeers", was not
correctly counted.
The metrics were originally introduced in
https://github.com/ethereum/go-ethereum/pull/27621
I was thinking of various possible solutions.
- the one proposed here wraps both the new error and the origial error.
It is not a pattern we use in other parts of the code, but works. This
is maybe the smallest possible change.
- as an alternate, I could write a proper `errProtoHandshakeError` with
it's own wrapped error
- finally, I'm not even sure we need `errProtoHandshakeError`, maybe we
could just pass up the original error.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Thank you, @holiman, for being an integral part of the Go-Ethereum
and for your invaluable contributions over the years.
This will always be your home and you're welcome back anytime!
I added the history mode configuration in eth/ethconfig initially, since
it seemed like the logical place. But it turns out we need access to the
intended pruning setting at a deeper level, and it actually needs to be
integrated with the blockchain startup procedure.
With this change applied, if a node previously had its history pruned,
and is subsequently restarted **without** the `--history.chain
postmerge` flag, the `BlockChain` initialization code will now verify
the freezer tail against the known pruning point of the predefined
network and will restore pruning status. Note that this logic is quite
restrictive, we allow non-zero tail only for known networks, and only
for the specific pruning point that is defined.
As of now, Geth disconnects peers only on protocol error or timeout,
meaning once connection slots are filled, the peerset is largely fixed.
As mentioned in https://github.com/ethereum/go-ethereum/issues/31321,
Geth should occasionally disconnect peers to ensure some churn.
What/when to disconnect could depend on:
- the state of geth (e.g. sync or not)
- current number of peers
- peer level metrics
This PR adds a very slow churn using a random drop.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Our previous success metrics gave success even if a peer disconnected
right after connection. These metrics only count peers that stayed
connected for at least 1 min. The 1 min limit is an arbitrary choice. We do
not use this for decision logic, only statistics.
When we instantiate a sub-logger via
`go-ethereum/internal/testlog/logger.With`, we copy the reference to the
`bufHandler` from the parent logger. However, internally,
`go-ethereum/internal/testlog/logger.With` calls `log/slog/Logger.With`
which creates a new handler instance (via
`internal/bufHandler.WithAttrs`).
This PR modifies sub-logger instantiation to use the newly-instantiated
handler, instead of copying the reference from the parent instance. The
type cast from `slog.Handler` to `*bufHandler` in
`internal/testlog/Logger.With` is safe here because a
`internal/testlog/Logger` can only be instantiated with a `*bufHandler`
as the underlying handler type.
Note, that I've also removed a pre-existing method that broke the above
assumption. However, this method is not used in our codebase.
I'm not sure if the assumption holds for forks of geth (e.g. optimism
has modified the testlogger somewhat allowing test loggers to accept
arbitrary handler types), but it seems okay to break API compatibility
given that this is in the `internal` package.
closes https://github.com/ethereum/go-ethereum/issues/31533
before this changes, this will result in numerous test failures:
```
> go test -run=Eth2AssembleBlock -c
> stress ./catalyst.test
```
The reason is that after creating/inserting the test chain, there is a
race between the txpool head reset and the promotion of txs added from
tests.
Ensuring that the txpool state is up to date with the head of the chain
before proceeding fixes these flaky tests.
This fix allows Trezor to support full 32bit chainId in geth, with the
next version of firmware.
For `chainId > 2147483630` case, Trezor returns signature bit only.
- Trezor returns only signature parity for `chainId > 2147483630` case.
- for `chainId == 2147483630` case, Trezor returns `MAX_UINT32` or `0`,
but it doesn't matter.
(`2147483630 * 2 + 35` = `4294967295`(`MAX_UINT32`))
chainId | returned signature_v | compatible issue
---------|------------------------|--------------------
0 < chainId <= 255 | chainId * 2 + 35 + v | no issue (firmware `1.6.2`
for Trezor one)
255 < chainId <= 2147483630 | chainId * 2 + 35 + v | ***fixed.***
*firmware `1.6.3`*
chainId > 2147483630 | v | *firmware `1.6.3`*
Please see also: full 32bit chainId support for Trezor
- Trezor one: https://github.com/trezor/trezor-mcu/pull/399 ***merged***
- Trezor model T: https://github.com/trezor/trezor-core/pull/311
***merged***
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Fixes#31169
The TestTransactionForgotten test was flaky due to real time
dependencies. This PR:
- Replaces real time with mock clock for deterministic timing control
- Adds precise state checks at timeout boundaries
- Verifies underpriced cache states and cleanup
- Improves test reliability by controlling transaction timestamps
- Adds checks for transaction re-enqueueing behavior
The changes ensure consistent test behavior without timing-related
flakiness.
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
This PR proposes a change to the authorizations' validation introduced
in commit cdb66c8. These changes make the expected behavior independent
of the order of admission of authorizations, improving the
predictability of the resulting state and the usability of the system
with it.
The current implementation behavior is dependent on the transaction
submission order: This issue is related to authorities and the sender of
a transaction, and can be reproduced respecting the normal nonce rules.
The issue can be reproduced by the two following cases:
**First case**
- Given an empty pool.
- Submit transaction `{ from: B, auths [ A ] }`: is accepted.
- Submit transaction `{ from: A }`: Is accepted: it becomes the one
in-flight transaction allowed.
**Second case**
- Given an empty pool.
- Submit transaction `{ from: A }`: is accepted
- Submit transaction `{ from: B, auths [ A ] }`: is rejected since there
is already a queued/pending transaction from A.
The expected behavior is that both sequences of events would lead to the
same sets of accepted and rejected transactions.
**Proposed changes**
The queued/pending transactions issued from any authority of the
transaction being validated have to be counted, allowing one transaction
from accounts submitting an authorization.
- Notice that the expected behavior was explicitly forbidden in the case
"reject-delegation-from-pending-account", I believe that this behavior
conflicts to the definition of the limitation, and it is removed in this
PR. The expected behavior is tested in
"accept-authorization-from-sender-of-one-inflight-tx".
- Replacement tests have been separated to improve readability of the
acceptance test.
- The test "allow-more-than-one-tx-from-replaced-authority" has been
extended with one extra transaction, since the system would always have
accepted one transaction (but not two).
- The test "accept-one-inflight-tx-of-delegated-account" is extended to
clean-up state, avoiding leaking the delegation used into the other
tests. Additionally, replacement check is removed to be tested in its
own test case.
**Expected behavior**
The expected behavior of the authorizations' validation shall be as
follows:
Notice that replacement shall be allowed, and behavior shall remain
coherent with the table, according to the replaced transaction.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Make UPnP more robust
- Once a random port was mapped, we try to stick to it even if a UPnP
refresh fails. Previously we were immediately moving back to try the
default port, leading to frequent ENR changes.
- We were deleting port mappings before refresh as a possible
workaround. This created issues in some UPnP servers. The UPnP (and PMP)
specification is explicit about the refresh requirements, and delete is
clearly not needed (see
https://github.com/ethereum/go-ethereum/pull/30265#issuecomment-2766987859).
From now on we only delete when closing.
- We were trying to add port mappings only once, and then moved on to
random ports. Now we insist a bit more, so that a simple failed request
won't lead to ENR changes.
Fixes https://github.com/ethereum/go-ethereum/issues/31418
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
During my benchmarks on Holesky, around 10% of all CPU time was spent in
PUSH2
```
ROUTINE ======================== github.com/ethereum/go-ethereum/core/vm.newFrontierInstructionSet.makePush.func1 in github.com/ethereum/go-ethereum/core/vm/instructions.go
16.38s 20.35s (flat, cum) 10.31% of Total
740ms 740ms 976: return func(pc *uint64, interpreter *EVMInterpreter, scope *ScopeContext) ([]byte, error) {
. . 977: var (
40ms 40ms 978: codeLen = len(scope.Contract.Code)
970ms 970ms 979: start = min(codeLen, int(*pc+1))
200ms 200ms 980: end = min(codeLen, start+pushByteSize)
. . 981: )
670ms 2.39s 982: a := new(uint256.Int).SetBytes(scope.Contract.Code[start:end])
. . 983:
. . 984: // Missing bytes: pushByteSize - len(pushData)
410ms 410ms 985: if missing := pushByteSize - (end - start); missing > 0 {
. . 986: a.Lsh(a, uint(8*missing))
. . 987: }
12.69s 14.94s 988: scope.Stack.push2(*a)
10ms 10ms 989: *pc += size
650ms 650ms 990: return nil, nil
. . 991: }
. . 992:}
```
Which is quite crazy. We have a handwritten encoder for PUSH1 already,
this PR adds one for PUSH2.
PUSH2 is the second most used opcode as shown here:
https://gist.github.com/shemnon/fb9b292a103abb02d98d64df6fbd35c8 since
it is used by solidity quite significantly. Its used ~20 times as much
as PUSH20 and PUSH32.
# Benchmarks
```
BenchmarkPush/makePush-14 94196547 12.27 ns/op 0 B/op 0 allocs/op
BenchmarkPush/push-14 429976924 2.829 ns/op 0 B/op 0 allocs/op
```
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This pull request introduces two constraints in the blobPool:
(a) If the sender has a pending authorization or delegation, only one
in-flight
executable transaction can be cached.
(b) If the authority address in a SetCode transaction is already
reserved by
the blobPool, the transaction will be rejected.
These constraints mitigate an attack where an attacker spams the pool
with
numerous blob transactions, evicts other transactions, and then cancels
all
pending blob transactions by draining the sender’s funds if they have a
delegation.
Note, because there is no exclusive lock held between different subpools
when processing transactions, it's totally possible the SetCode
transaction
and blob transactions with conflict sender and authorities are accepted
simultaneously. I think it's acceptable as it's very hard to be
exploited.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR improves error handling in the remotedb package by fixing two
issues:
1. In the `Has` method, we now properly propagate errors instead of
silently returning false. This makes the behavior more predictable and
helps clients better understand when there are connection issues.
2. In the `New` constructor, we add a nil check for the client parameter
to prevent potential panics. This follows Go best practices for
constructor functions.
These changes make the code more robust and follow Go's error handling
idioms without requiring any changes to other parts of the codebase.
Changes:
- Modified `Has` method to return errors instead of silently returning
false
- Added nil check in `New` constructor
- Fixed field name in constructor to match struct definition
Co-authored-by: lightclient <lightclient@protonmail.com>
Add GetHeaderByNumber and GetReceiptsByNumber to allow more efficient API request filling from Era files.
Here we are modifying the port mapping logic so that existing port
mappings will only be removed when they were previously created by geth.
The AddAnyPortMapping functionality has been adapted to work consistently
between the IGDv1 and IGDv2 backends.
This PR adds a new `--beacon.checkpoint.file` config flag to geth and
blsync which specifies a checkpoint import/export file. If a file with
an existing checkpoint is specified, it is used for initialization
instead of the hardcoded one (except when `--beacon.checkpoint` is also
specified simultaneously). Whenever the client encounters a new valid
finality update with a suitable finalized beacon block root at an epoch
boundary, it saves the block root in hex format to the checkpoint file.
This adds the test description text to the output, instead of keeping it
as a Go comment. Logs are visible in the hive UI where these tests run,
while Go comments are not.
This pull request introduces new sync logic for pruning mode. The downloader will now skip
insertion of block bodies and receipts before the configured history cutoff point.
Originally, in snap sync, the header chain and other components (bodies and receipts) were
inserted separately. However, in Proof-of-Stake, this separation is unnecessary since the
sync target is already verified by the CL.
To simplify the process, this pull request modifies `InsertReceiptChain` to insert headers
along with block bodies and receipts together. Besides, `InsertReceiptChain` doesn't have
the notion of reorg, as the common ancestor is always be found before the sync and extra
side chain is truncated at the beginning if they fall in the ancient store. The stale
canonical chain flags will always be rewritten by the new chain. Explicit reorg logic is
no longer required in `InsertReceiptChain`.
This is for the implementation of Portal Network in the Shisui client.
Their handler needs access to the node object in order to send further
calls to the requesting node. This is a breaking API change but it
should be fine, since there are basically no known users of TALKREQ
outside of Portal network.
---------
Signed-off-by: thinkAfCod <q315xia@163.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
When resending the WHOAREYOU packet, a new nonce and random IV should not
be generated. The sent packet needs to match the previously-sent one exactly
in order to make the handshake retry work.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This is an alternative to #31309
With eth/68, transaction announcement must have transaction type and
size. So in announceTransactions, we need to query the transaction from
transaction pool with its hash. This creates overhead in case of blob
transaction which needs to load data from billy and RLP decode. This
commit creates a lightweight lookup from transaction hash to transaction
size and a function GetMetadata to query transaction type and
transaction size given the transaction hash.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR refactors the `nodeSet` structure in the path database to use
separate maps for account and storage trie nodes, resulting in
performance improvements. The change maintains the same API while
optimizing the internal data structure.
It introduces a new variable to store the external port returned by the
addAnyPortMapping function and ensures that the correct external port is
returned even in case of an error.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds `rawdb.SafeDeleteRange` and uses it for range deletion in
`core/filtermaps`. This includes deleting the old bloombits database,
resetting the log index database and removing index data for unindexed
tail epochs (which previously weren't properly implemented for the
fallback case).
`SafeDeleteRange` either calls `ethdb.DeleteRange` if the node uses the
new path based state scheme or uses an iterator based fallback method
that safely skips trie nodes in the range if the old hash based state
scheme is used. Note that `ethdb.DeleteRange` also has its own iterator
based fallback implementation in `ethdb/leveldb`. If a path based state
scheme is used and the backing db is pebble (as it is on the majority of
new nodes) then `rawdb.SafeDeleteRange` uses the fast native range
delete.
Also note that `rawdb.SafeDeleteRange` has different semantics from
`ethdb.DeleteRange`, it does not automatically return if the operation
takes a long time. Instead it receives a `stopCallback` that can
interrupt the process if necessary. This is because in the safe mode
potentially a lot of entries are iterated without being deleted (this is
definitely the case when deleting the old bloombits database which has a
single byte prefix) and therefore restarting the process every time a
fixed number of entries have been iterated would result in a quadratic
run time in the number of skipped entries.
When running in safe mode, unindexing an epoch takes about a second,
removing bloombits takes around 10s while resetting a full log index
might take a few minutes. If a range delete operation takes a
significant amount of time then log messages are printed. Also, any
range delete operation can be interrupted by shutdown (tail uinindexing
can also be interrupted by head indexing, similarly to how tail indexing
works). If the last unindexed epoch might have "dirty" index data left
then the indexed map range points to the first valid epoch and
`cleanedEpochsBefore` points to the previous, potentially dirty one. At
startup it is always assumed that the epoch before the first fully
indexed one might be dirty. New tail maps are never rendered and also no
further maps are unindexed before the previous unindexing is properly
cleaned up.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR changes log indexer error handling so that if an indexing error
happens then it disables the indexer and reverts to unindexed more
without resetting the database (except in case of a failed database
init).
Resetting the database on the first error would probably be overkill as
a client update might fix this without having to reindex the entire
history. It would also make debugging very hard. On the other hand,
these errors do not resolve themselves automatically so constantly
retrying makes no sense either. With these changes a new attempt to
resume indexing is made every time the client is restarted.
The PR also fixes https://github.com/ethereum/go-ethereum/issues/31491
which originated from the tail indexer trying to resume processing a
failed map renderer.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Add support for state overrides in eth_createAccessList. This will make the method consistent
with other execution methods.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This pull request removes the node copy operation to reduce memory
allocation. Key Changes as below:
**(a) Use `decodeNodeUnsafe` for decoding nodes retrieved from the trie
node reader**
In the current implementation of the MPT, once a trie node blob is
retrieved, it is passed to `decodeNode` for decoding. However,
`decodeNode` assumes the supplied byte slice might be mutated later, so
it performs a deep copy internally before parsing the node.
Given that the node reader is implemented by the path database and the
hash database, both of which guarantee the immutability of the returned
byte slice. By restricting the node reader interface to explicitly
guarantee that the returned byte slice will not be modified, we can
safely replace `decodeNode` with `decodeNodeUnsafe`. This eliminates the
need for a redundant byte copy during each node resolution.
**(b) Modify the trie in place**
In the current implementation of the MPT, a copy of a trie node is
created before any modifications are made. These modifications include:
- Node resolution: Converting the value from a hash to the actual node.
- Node hashing: Tagging the hash into its cache.
- Node commit: Replacing the children with its hash.
- Structural changes: For example, adding a new child to a fullNode or
replacing a child of a shortNode.
This mechanism ensures that modifications only affect the live tree,
leaving all previously created copies unaffected.
Unfortunately, this property leads to a huge memory allocation
requirement. For example, if we want to modify the fullNode for n times,
the node will be copied for n times.
In this pull request, all the trie modifications are made in place. In
order to make sure all previously created copies are unaffected, the
`Copy` function now will deep-copy all the live nodes rather than the
root node itself.
With this change, while the `Copy` function becomes more expensive, it's
totally acceptable as it's not a frequently used one. For the normal
trie operations (Get, GetNode, Hash, Commit, Insert, Delete), the node
copy is not required anymore.
text:"SA1019: cfg.TxLookupLimit is deprecated: use 'TransactionHistory' instead."
- path:cmd/utils/flags.go
text:"SA1019: ethconfig.Defaults.TxLookupLimit is deprecated: use 'TransactionHistory' instead."
- path:internal/build/pgp.go
text: 'SA1019:"golang.org/x/crypto/openpgp"is deprecated:this package is unmaintained except for security fixes.'
- path:core/vm/contracts.go
text: 'SA1019:"golang.org/x/crypto/ripemd160"is deprecated:RIPEMD-160 is a legacy hash and should not be used for new applications.'
exclude:
- 'SA1019: event.TypeMux is deprecated:use Feed'
- 'SA1019:strings.Title is deprecated'
- 'SA1019: strings.Title has been deprecated since Go 1.18 and an alternative has been available since Go 1.0:The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead.'
- 'SA1029:should not use built-in type string as key for value'
- linters:
- deadcode
- staticcheck
path:crypto/bn256/cloudflare/optate.go
- linters:
- revive
path:crypto/bn256/
- path:cmd/utils/flags.go
text:"SA1019: cfg.TxLookupLimit is deprecated: use 'TransactionHistory' instead."
- path:cmd/utils/flags.go
text:"SA1019: ethconfig.Defaults.TxLookupLimit is deprecated: use 'TransactionHistory' instead."
- path:internal/build/pgp.go
text: 'SA1019:"golang.org/x/crypto/openpgp"is deprecated:this package is unmaintained except for security fixes.'
- path:core/vm/contracts.go
text: 'SA1019:"golang.org/x/crypto/ripemd160"is deprecated:RIPEMD-160 is a legacy hash and should not be used for new applications.'
- path:(.+)\.go$
text: 'SA1019: event.TypeMux is deprecated:use Feed'
- path:(.+)\.go$
text: 'SA1019:strings.Title is deprecated'
- path:(.+)\.go$
text: 'SA1019: strings.Title has been deprecated since Go 1.18 and an alternative has been available since Go 1.0:The rule Title uses for word boundaries does not handle Unicode punctuation properly. Use golang.org/x/text/cases instead.'
- path:(.+)\.go$
text: 'SA1029:should not use built-in type string as key for value'
- **Keep changes minimal and focused.** Only modify code directly related to the task at hand. Do not refactor unrelated code, rename existing variables or functions for style, or bundle unrelated fixes into the same commit or PR.
- **Do not add, remove, or update dependencies** unless the task explicitly requires it.
## Pre-Commit Checklist
Before every commit, run **all** of the following checks and ensure they pass:
### 1. Formatting
Before committing, always run `gofmt` and `goimports` on all modified files:
```sh
gofmt -w <modifiedfiles>
goimports -w <modifiedfiles>
```
### 2. Build All Commands
Verify that all tools compile successfully:
```sh
make all
```
This builds all executables under `cmd/`, including `keeper` which has special build requirements.
### 3. Tests
While iterating during development, use `-short` for faster feedback:
```sh
go run ./build/ci.go test -short
```
Before committing, run the full test suite **without**`-short` to ensure all tests pass, including the Ethereum execution-spec tests and all state/block test permutations:
```sh
go run ./build/ci.go test
```
### 4. Linting
```sh
go run ./build/ci.go lint
```
This runs additional style checks. Fix any issues before committing.
### 5. Generated Code
```sh
go run ./build/ci.go check_generate
```
Ensures that all generated files (e.g., `gen_*.go`) are up to date. If this fails, first install the required code generators by running `make devtools`, then run the appropriate `go generate` commands and include the updated files in your commit.
### 6. Dependency Hygiene
```sh
go run ./build/ci.go check_baddeps
```
Verifies that no forbidden dependencies have been introduced.
## What to include in commits
Do not commit binaries, whether they are produced by the main build or byproducts of investigations.
## Commit Message Format
Commit messages must be prefixed with the package(s) they modify, followed by a short lowercase description:
```
<package(s)>: description
```
Examples:
- `core/vm: fix stack overflow in PUSH instruction`
- `eth, rpc: make trace configs optional`
- `cmd/geth: add new flag for sync mode`
Use comma-separated package names when multiple areas are affected. Keep the description concise.
## Pull Request Title Format
PR titles follow the same convention as commit messages:
```
<listofmodifiedpaths>: description
```
Examples:
- `core/vm: fix stack overflow in PUSH instruction`
- `trie/archiver: streaming subtree archival to fix OOM`
Use the top-level package paths, comma-separated if multiple areas are affected. Only mention the directories with functional changes, interface changes that trickle all over the codebase should not generate an exhaustive list. The description should be a short, lowercase summary of the change.
| **`geth`** | Our main Ethereum CLI client. It is the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default), archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. `geth --help` and the [CLI page](https://geth.ethereum.org/docs/fundamentals/command-line-options) for command line options. |
| `clef` | Stand-alone signing tool, which can be used as a backend signer for `geth`. |
| `devp2p` | Utilities to interact with nodes on the networking layer, without running a full blockchain. |
| `abigen` | Source code generator to convert Ethereum contract definitions into easy-to-use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://docs.soliditylang.org/en/develop/abi-spec.html) with expanded functionality if the contract bytecode is also available. However, it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://geth.ethereum.org/docs/developers/dapp-developer/native-bindings) page for details. |
| `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow isolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug run`). |
@ -162,7 +162,7 @@ accessible from the outside.
As a developer, sooner rather than later you'll want to start interacting with `geth` and the
Ethereum network via your own programs and not manually through the console. To aid
this, `geth` has built-in support for a JSON-RPC based APIs ([standard APIs](https://ethereum.github.io/execution-apis/api-documentation/)
this, `geth` has built-in support for a JSON-RPC based APIs ([standard APIs](https://ethereum.org/en/developers/docs/apis/json-rpc/)
and [`geth` specific APIs](https://geth.ethereum.org/docs/interacting-with-geth/rpc)).
These can be exposed via HTTP, WebSockets and IPC (UNIX sockets on UNIX based
@ -21,7 +20,6 @@ Audit reports are published in the `docs` folder: https://github.com/ethereum/go
To find out how to disclose a vulnerability in Ethereum visit [https://bounty.ethereum.org](https://bounty.ethereum.org) or email bounty@ethereum.org. Please read the [disclosure page](https://github.com/ethereum/go-ethereum/security/advisories?state=published) for more information about publicly disclosed security vulnerabilities.
Use the built-in `geth version-check` feature to check whether the software is affected by any known vulnerability. This command will fetch the latest [`vulnerabilities.json`](https://geth.ethereum.org/docs/vulnerabilities/vulnerabilities.json) file which contains known security vulnerabilities concerning `geth`, and cross-check the data against its own version number.
The following key may be used to communicate sensitive information to developers.
//lint:ignore ST1005 brand name displayed on the console
returncommon.Address{},nil,fmt.Errorf("Ledger version >= 1.9.0 required for EIP-2930/EIP-1559 signing (found version v%d.%d.%d)",w.version[0],w.version[1],w.version[2])
}
casetypes.SetCodeTxType:
ifledgerVersionLessThan(w.version,1,17,0){
//lint:ignore ST1005 brand name displayed on the console
returncommon.Address{},nil,fmt.Errorf("Ledger version >= 1.17.0 required for EIP-7702 signing (found version v%d.%d.%d)",w.version[0],w.version[1],w.version[2])
//lint:ignore ST1005 brand name displayed on the console
returncommon.Address{},nil,fmt.Errorf("Ledger v%d.%d.%d doesn't support signing this transaction, please update to v1.0.3 at least",w.version[0],w.version[1],w.version[2])
}
@ -184,7 +206,7 @@ func (w *ledgerDriver) SignTypedMessage(path accounts.DerivationPath, domainHash
returnnil,accounts.ErrWalletClosed
}
// Ensure the wallet is capable of signing the given transaction
ifw.version[0]<1&&w.version[1]<5{
ifledgerVersionLessThan(w.version,1,5,0){
//lint:ignore ST1005 brand name displayed on the console
returnnil,fmt.Errorf("Ledger version >= 1.5.0 required for EIP-712 signing (found version v%d.%d.%d)",w.version[0],w.version[1],w.version[2])
