Implement standardized JSON format for slow block logging to enable
cross-client performance analysis and protocol research.
This change is part of the Cross-Client Execution Metrics initiative
proposed by Gary Rong: https://hackmd.io/dg7rizTyTXuCf2LSa2LsyQ
The standardized metrics enabled data-driven analysis like the EIP-7907
research: https://ethresear.ch/t/data-driven-analysis-on-eip-7907/23850
JSON format includes:
- block: number, hash, gas_used, tx_count
- timing: execution_ms, total_ms
- throughput: mgas_per_sec
- state_reads: accounts, storage_slots, bytecodes, code_bytes
- state_writes: accounts, storage_slots, bytecodes
- cache: account/storage/code hits, misses, hit_rate
This should come after merging #33522
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Based on [EIP-7864](https://eips.ethereum.org/EIPS/eip-7864), the tree
index should be 32 bytes instead of 31 bytes.
```
def get_tree_key(address: Address32, tree_index: int, sub_index: int):
# Assumes STEM_SUBTREE_WIDTH = 256
return tree_hash(address + tree_index.to_bytes(32, "little"))[:31] + bytes(
[sub_index]
)
```
This is a tweak to the wasm build, that expects the `geth_io` namespace
to expect a `geth_io` module, providing a `len` and `read` methods. This
will be provided by the WASM interface in sp1. This forces an API change
on the OpenVM side, but the interface on their side is still being
designed, so we should proceed with this change, and we'll make a
different tag for OpenVM if this can't work for them.
Co-authored-by: wakabat <wakabat@protonmail.com>
This PR optimizes the historical trie node reader by reworking how data
is accessed and memory is managed, reducing allocation overhead
significantly.
Specifically:
- Instead of decoding an entire history object to locate a specific trie node,
the reader now searches directly within the history.
- Besides, slice pre-allocation can avoid unnecessary deep-copy significantly.
This PR optimizes memory allocation in StateTrie.PrefetchAccount() and
StateTrie.PrefetchStorage() by preallocating slice capacity when the
final size is known.
This PR extends the statistics of contract code read by adding these
fields:
- **CacheHitBytes**: the total number of bytes served by cache
- **CacheMissBytes**: the total number of bytes read on cache miss
- **CodeReadBytes**: the total number of bytes for contract code read
Calling `pool.priced.Removed` is needed to keep is sync with
`pool.all.Remove`.
It was called in other occurances, but not here.
The counter is used for internal heap management. It was working even without this, just not calling reheap at the intended frequency.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
This PR adds metrics that count the number of accounts having transactions
in the txpool. Together with the transaction count this can be used as a
simple indicator of the diversity of transactions in the pool.
Note: as an alternative implementation, we could use a periodic or event
driven update of these Gauges using len.
I've preferred this implementation to match what we have for the pool
sizes.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Fixes#33630
Sort self-destructed addresses before emitting hooks in Finalise() to
ensure deterministic ordering and fix flaky test
TestHooks_OnCodeChangeV2.
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This adds support for Grafana Pyroscope, a continuous profiling solution.
The client is configured similarly to metrics, i.e. run
geth --pyroscope --pyroscope.server=https://...
This commit is a resubmit of #33261 with some changes.
---------
Co-authored-by: Carlos Bermudez Porto <cbermudez.dev@gmail.com>
This PR reverts a part of changes brought by https://github.com/ethereum/go-ethereum/pull/33281/changes
Specifically, read-only protection should always be enforced at the opcode level,
regardless of whether the check has already been performed during gas metering.
It should act as a gatekeeper, otherwise, it is easy to introduce errors by adding
new gas measurement logic without consistently applying the read-only protection.
Adding an RPC flag to limit the block range size for eth_getLogs and
eth_newFilter requests.
closing https://github.com/ethereum/go-ethereum/issues/24508
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
The core part of this PR that we need to adopt is to move the code and
nonce change hook invocations to occur at tx finalization, instead of
when the selfdestruct opcode is called.
Additionally:
* remove `SelfDestruct6780` now that it is essentially the same as
`SelfDestruct` just gated by `is new contract`
* don't duplicate `BalanceIncreaseSelfdestruct` (transfer to recipient
of selfdestruct) in the hooked statedb and in the opcode handler for the
selfdestruct opcode.
* balance is burned immediately when the beneficiary of the selfdestruct
is the sender, and the contract was created in the same transaction.
Previously we emit two balance increases to the recipient (see above
point), and a balance decrease from the sender.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
There's no need to perform the subsequent state access on the target if
we already know that we are out of gas.
This aligns the state access behavior of selfdestruct with EIP-7928
This PR causes execution to terminate at the gas handler in the case of
sstore/call if they are invoked in a static execution context.
This aligns the behavior with EIP 7928 by ensuring that we don't record
any state reads in the access list from an SSTORE/CALL in this
circumstance.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Remove a large amount of duplicate code from the tx_fetcher tests.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
The bitmap is used in compact-encoded trie nodes to indicate which elements
have been modified. The bitmap format has been updated to use big-endian
encoding.
Bit positions are numbered from 0 to 15, where position 0 corresponds to
the most significant bit of b[0], and position 15 corresponds to the least
significant bit of b[1].
This PR adds support for the extraction of OpenTelemetry trace context
from incoming JSON-RPC request headers, allowing geth spans to be linked
to upstream traces when present.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Add Open Telemetry tracing inside the RPC server to help attribute runtime costs within `handler.handleCall()`. In particular, it allows us to distinguish time spent decoding arguments, invoking methods via reflection, and actually executing the method and constructing/encoding JSON responses.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Updated the `avail` calculation to correctly compute remaining capacity:
`buf.limit - len(buf.output)`, ensuring the buffer never exceeds its
configured limit regardless of how many times `Write()` is called.
The coverage build path was generating go test commands with a bogus
-tags flag that held the coverpkg value, so the run kept failing. I
switched coverbuild to treat the optional argument as an override for
-coverpkg and stopped passing coverpkg from the caller. Now the script
emits a clean go test invocation that should actually succeed.
This PR fixes an issue where the tx indexer would repeatedly try to
“unindex” a block with a missing body, causing a spike in CPU usage.
This change skips these blocks and advances the index tail. The fix was
verified both manually on a local development chain and with a new test.
resolves#33371
This PR fixes an issue where `evm statetest` would not verify the
post-state root hash if the test case expected an exception (e.g.
invalid transaction).
The fix involves:
1. Modifying `tests/state_test_util.go` in the `Run` method.
2. When an expected error occurs (`err != nil`), we now check if
`post.Root` is defined.
3. If defined, we recalculate the intermediate root from the current
state (which is reverted to the pre-transaction snapshot upon error).
4. We use `GetChainConfig` and `IsEIP158` to ensure the correct state
clearing rules are applied when calculating the root, avoiding
regressions on forks that require EIP-158 state clearing.
5. If the calculated root mismatches the expected root, the test now
fails.
This ensures that state tests are strictly verified against their
expected post-state, even for failure scenarios.
Fixes issue #33527
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
- pass `rpc.BlockNumberOrHash` directly to `eth_getBlockReceipts` so
`requireCanonical` and other fields survive
- aligns `BlockReceipts` with other `ethclient` methods and re-enables
canonical-only receipt queries
Allow the blobpool to accept blobs out of nonce order
Previously, we were dropping blobs that arrived out-of-order. However,
since fetch decisions are done on receiver side,
out-of-order delivery can happen, leading to inefficiencies.
This PR:
- adds an in-memory blob tx storage, similar to the queue in the
legacypool
- a limited number of received txs can be added to this per account
- txs waiting in the gapped queue are not processed further and not
propagated further until they are unblocked by adding the previos nonce
to the blobpool
The size of the in-memory storage is currently limited per account,
following a slow-start logic.
An overall size limit, and a TTL is also enforced for DoS protection.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This pull request introduces a mechanism to compress trienode history by
storing only the node diffs between consecutive versions.
- For full nodes, only the modified children are recorded in the history;
- For short nodes, only the modified value is stored;
If the node type has changed, or if the node is newly created or
deleted, the entire node value is stored instead.
To mitigate the overhead of reassembling nodes from diffs during history
reads, checkpoints are introduced by periodically storing full node values.
The current checkpoint interval is set to every 16 mutations, though
this parameter may be made configurable in the future.
Fixes#33369
This omits "topics" and "addresses" from the filter when they are unspecified.
It is required for interoperability with some server implementations that cannot
handle `null` for these fields.
It's a PR based on #33303 and introduces an approach for trienode
history indexing.
---
In the current archive node design, resolving a historical trie node at
a specific block
involves the following steps:
- Look up the corresponding trie node index and locate the first entry
whose state ID
is greater than the target state ID.
- Resolve the trie node from the associated trienode history object.
A naive approach would be to store mutation records for every trie node,
similar to
how flat state mutations are recorded. However, the total number of trie
nodes is
extremely large (approximately 2.4 billion), and the vast majority of
them are rarely
modified. Creating an index entry for each individual trie node would be
very wasteful
in both storage and indexing overhead. To address this, we aggregate
multiple trie
nodes into chunks and index mutations at the chunk level instead.
---
For a storage trie, the trie is vertically partitioned into multiple sub
tries, each spanning
three consecutive levels. The top three levels (1 + 16 + 256 nodes) form
the first chunk,
and every subsequent three-level segment forms another chunk.
```
Original trie structure
Level 0 [ ROOT ] 1 node
Level 1 [0] [1] [2] ... [f] 16 nodes
Level 2 [00] [01] ... [0f] [10] ... [ff] 256 nodes
Level 3 [000] [001] ... [00f] [010] ... [fff] 4096 nodes
Level 4 [0000] ... [000f] [0010] ... [001f] ... [ffff] 65536 nodes
Vertical split into chunks (3 levels per chunk)
Level0 [ ROOT ] 1 chunk
Level3 [000] ... [fff] 4096 chunks
Level6 [000000] ... [fffffff] 16777216 chunks
```
Within each chunk, there are 273 nodes in total, regardless of the
chunk's depth in the trie.
```
Level 0 [ 0 ] 1 node
Level 1 [ 1 ] … [ 16 ] 16 nodes
Level 2 [ 17 ] … … [ 272 ] 256 nodes
```
Each chunk is uniquely identified by the path prefix of the root node of
its corresponding
sub-trie. Within a chunk, nodes are identified by a numeric index
ranging from 0 to 272.
For example, suppose that at block 100, the nodes with paths `[]`,
`[0]`, `[f]`, `[00]`, and `[ff]`
are modified. The mutation record for chunk 0 is then appended with the
following entry:
`[100 → [0, 1, 16, 17, 272]]`, `272` is the numeric ID of path `[ff]`.
Furthermore, due to the structural properties of the Merkle Patricia
Trie, if a child node
is modified, all of its ancestors along the same path must also be
updated. As a result,
in the above example, recording mutations for nodes `00` and `ff` alone
is sufficient,
as this implicitly indicates that their ancestor nodes `[]`, `[0]` and
`[f]` were also
modified at block 100.
---
Query processing is slightly more complicated. Since trie nodes are
indexed at the chunk
level, each individual trie node lookup requires an additional filtering
step to ensure that
a given mutation record actually corresponds to the target trie node.
As mentioned earlier, mutation records store only the numeric
identifiers of leaf nodes,
while ancestor nodes are omitted for storage efficiency. Consequently,
when querying
an ancestor node, additional checks are required to determine whether
the mutation
record implicitly represents a modification to that ancestor.
Moreover, since trie nodes are indexed at the chunk level, some trie
nodes may be
updated frequently, causing their mutation records to dominate the
index. Queries
targeting rarely modified trie nodes would then scan a large amount of
irrelevant
index data, significantly degrading performance.
To address this issue, a bitmap is introduced for each index block and
stored in the
chunk's metadata. Before loading a specific index block, the bitmap is
checked to
determine whether the block contains mutation records relevant to the
target trie node.
If the bitmap indicates that the block does not contain such records,
the block is skipped entirely.
