This pull request enhances the unit test, avoiding unnecessary failure
in CI.
```
--- FAIL: TestSimulatedBeaconSendWithdrawals (12.08s)
simulated_beacon_test.go:139: timed out without including all withdrawals/txs
FAIL
```
Here I am adding a config option and geth flag (`--history.chain`) for
configuring history pruning. There are two options available:
- `--history.chain all` is the default and will keep all history like
before.
- `--history.chain postmerge` will configure the history cutoff point to
the merge block.
The option doesn't actually do anything right now, but we need it as a
precursor for other history pruning changes.
This ensures that if we receive a blob transaction announcement where we cannot
link the tx to the sidecar commitments, we will drop the sending peer. This check
is added in the protocol handler for the PooledTransactions message.
Tests for this have also been added in the cross-client "eth" protocol test suite.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In transaction-sending APIs such as `eth_sendRawTransaction`, a submitted transaction
failing the configured txpool validation rules (i.e. fee too low) would cause an error to be
returned, even though the transaction was successfully added into the locals tracker.
Once added there, the transaction may even be included into the chain at a later time,
when fee market conditions change.
This change improves on this by performing the validation in the locals tracker, basically
skipping some of the validation rules for local transactions. We still try to add the tx to the
main pool immediately, but an error will only be returned for transactions which are
fundamentally invalid.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
I ran into this while trying to debug a discv5 thing. I tried to disable
DNS discovery using `--discovery.dns=false`, which doesn't work.
Annoyingly, geth started anyway and discarded the error silently. I
eventually found my mistake, but it took way longer than it should have.
Also including a small change to the error message for invalid DNS URLs
here. The user actually needs to see the URL to make sense of the error.
This PR does a few things including:
- Remove `ContractRef` interface
- Remove `vm.AccountRef` which implements `ContractRef` interface
- Maintain the `jumpDests` struct in EVM for sharing between call frames
- Simplify the delegateCall context initialization
Currently, when calculating block's bloom, we loop through all the
receipt logs to calculate the hash value. However, normally, after going
through applyTransaction, the receipt's bloom is already calculated
based on the receipt log, so the block's bloom can be calculated by just
ORing these receipt's blooms.
```
goos: darwin
goarch: arm64
pkg: github.com/ethereum/go-ethereum/core/types
cpu: Apple M1 Pro
BenchmarkCreateBloom
BenchmarkCreateBloom/small
BenchmarkCreateBloom/small-10 810922 1481 ns/op 104 B/op 5 allocs/op
BenchmarkCreateBloom/large
BenchmarkCreateBloom/large-10 8173 143764 ns/op 9614 B/op 401 allocs/op
BenchmarkCreateBloom/small-mergebloom
BenchmarkCreateBloom/small-mergebloom-10 5178918 232.0 ns/op 0 B/op 0 allocs/op
BenchmarkCreateBloom/large-mergebloom
BenchmarkCreateBloom/large-mergebloom-10 54110 22207 ns/op 0 B/op 0 allocs/op
```
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
This removes the method `TestingTTDBlock` introduced by #30744. It was
added to make the beacon consensus engine aware of the merge block in
tests without relying on the total difficulty. However, tracking the
merge block this way is very annoying. We usually configure forks in the
`ChainConfig`, but the method is on the consensus engine, which isn't
always created in the same place. By sidestepping the `ChainConfig` we
don't get the usual fork-order checking, so it's possible to enable the
merge before the London fork, for example. This in turn can lead to very
hard-to-debug outputs and validation errors.
So here I'm changing the consensus engine to check the
`MergeNetsplitBlock` instead. Alternatively, we assume a network is
merged if it has a `TerminalTotalDifficulty` of zero, which is a very
common configuration in tests.
Here we add some more changes for live tracing API v1.1:
- Hook `OnSystemCallStartV2` was introduced with `VMContext` as parameter.
- Hook `OnBlockHashRead` was introduced.
- `GetCodeHash` was added to the state interface
- The new `WrapWithJournal` construction helps with tracking EVM reverts in the tracer.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR changes the signature of `CalcExcessBlobGas` to take in just
the header timestamp instead of the whole object. It also adds a sanity
check for the parent->child block order to `VerifyEIP4844Header`.
A clarification was made to EIP-7691 stating that at the fork boundary
it is required to use the target blob count associated with the head
block, rather than the parent as implemented here.
See for more: https://github.com/ethereum/EIPs/pull/9249
Replaces #29297, descendant from #27535
---------
This PR removes `locals` as a concept from transaction pools. Therefore,
the pool now acts as very a good simulation/approximation of how our
peers' pools behave. What this PR does instead, is implement a
locals-tracker, which basically is a little thing which, from time to
time, asks the pool "did you forget this transaction?". If it did, the
tracker resubmits it.
If the txpool _had_ forgotten it, chances are that the peers had also
forgotten it. It will be propagated again.
Doing this change means that we can simplify the pool internals, quite a
lot.
### The semantics of `local`
Historically, there has been two features, or usecases, that has been
combined into the concept of `locals`.
1. "I want my local node to remember this transaction indefinitely, and
resubmit to the network occasionally"
2. "I want this (valid) transaction included to be top-prio for my
miner"
This PR splits these features up, let's call it `1: local` and `2:
prio`. The `prio` is not actually individual transaction, but rather a
set of `address`es to prioritize.
The attribute `local` means it will be tracked, and `prio` means it will
be prioritized by miner.
For `local`: anything transaction received via the RPC is marked as
`local`, and tracked by the tracker.
For `prio`: any transactions from this sender is included first, when
building a block. The existing commandline-flag `--txpool.locals` sets
the set of `prio` addresses.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR defines the Osaka fork. An easy first step to start our work on
the next hardfork
(This is needed for EOF testing as well)
---------
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
The total difficulty is the sum of all block difficulties from genesis
to a certain block. This value was used in PoW for deciding which chain
is heavier, and thus which chain to select. Since PoS has a different
fork selection algorithm, all blocks since the merge have a difficulty
of 0, and all total difficulties are the same for the past 2 years.
Whilst the TDs are mostly useless nowadays, there was never really a
reason to mess around removing them since they are so tiny. This
reasoning changes when we go down the path of pruned chain history. In
order to reconstruct any TD, we **must** retrieve all the headers from
chain head to genesis and then iterate all the difficulties to compute
the TD.
In a world where we completely prune past chain segments (bodies,
receipts, headers), it is not possible to reconstruct the TD at all. In
a world where we still keep chain headers and prune only the rest,
reconstructing it possible as long as we process (or download) the chain
forward from genesis, but trying to snap sync the head first and
backfill later hits the same issue, the TD becomes impossible to
calculate until genesis is backfilled.
All in all, the TD is a messy out-of-state, out-of-consensus computed
field that is overall useless nowadays, but code relying on it forces
the client into certain modes of operation and prevents other modes or
other optimizations. This PR completely nukes out the TD from the node.
It doesn't compute it, it doesn't operate on it, it's as if it didn't
even exist.
Caveats:
- Whenever we have APIs that return TD (devp2p handshake, tracer, etc.)
we return a TD of 0.
- For era files, we recompute the TD during export time (fairly quick)
to retain the format content.
- It is not possible to "verify" the merge point (i.e. with TD gone, TTD
is useless). Since we're not verifying PoW any more, just blindly trust
it, not verifying but blindly trusting the many year old merge point
seems just the same trust model.
- Our tests still need to be able to generate pre and post merge blocks,
so they need a new way to split the merge without TTD. The PR introduces
a settable ttdBlock field on the consensus object which is used by tests
as the block where originally the TTD happened. This is not needed for
live nodes, we never want to generate old blocks.
- One merge transition consensus test was disabled. With a
non-operational TD, testing how the client reacts to TTD is useless, it
cannot react.
Questions:
- Should we also drop total terminal difficulty from the genesis json?
It's a number we cannot react on any more, so maybe it would be cleaner
to get rid of even more concepts.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This pull request delivers the new version of the state history, where
the raw storage key is used instead of the hash.
Before the cancun fork, it's supported by protocol to destruct a
specific account and therefore, all the storage slot owned by it should
be wiped in the same transition.
Technically, storage wiping should be performed through storage
iteration, and only the storage key hash will be available for traversal
if the state snapshot is not available. Therefore, the storage key hash
is chosen as the identifier in the old version state history.
Fortunately, account self-destruction has been deprecated by the
protocol since the Cancun fork, and there are no empty accounts eligible
for deletion under EIP-158. Therefore, we can conclude that no storage
wiping should occur after the Cancun fork. In this case, it makes no
sense to keep using hash.
Besides, another big reason for making this change is the current format
state history is unusable if verkle is activated. Verkle tree has a
different key derivation scheme (merkle uses keccak256), the preimage of
key hash must be provided in order to make verkle rollback functional.
This pull request is a prerequisite for landing verkle.
Additionally, the raw storage key is more human-friendly for those who
want to manually check the history, even though Solidity already
performs some hashing to derive the storage location.
---
This pull request doesn't bump the database version, as I believe the
database should still be compatible if users degrade from the new geth
version to old one, the only side effect is the persistent new version
state history will be unusable.
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Fix the error comparison in tracer to prevent dropping revert reason data
---------
Co-authored-by: Martin <mrscdevel@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This commit makes it so that the struct logger will not emit logs while
system calls are being executed. This will make it consistent with
the JSON and MD loggers. It is as it stands hard to distinguish when
system calls are being processed vs when a tx is being processed.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR implements EIP-7702: "Set EOA account code".
Specification: https://eips.ethereum.org/EIPS/eip-7702
> Add a new transaction type that adds a list of `[chain_id, address,
nonce, y_parity, r, s]` authorization tuples. For each tuple, write a
delegation designator `(0xef0100 ++ address)` to the signing account’s
code. All code reading operations must load the code pointed to by the
designator.
---------
Co-authored-by: Mario Vega <marioevz@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR modifies how the metrics library handles `Enabled`: previously,
the package `init` decided whether to serve real metrics or just
dummy-types.
This has several drawbacks:
- During pkg init, we need to determine whether metrics are enabled or
not. So we first hacked in a check if certain geth-specific
commandline-flags were enabled. Then we added a similar check for
geth-env-vars. Then we almost added a very elaborate check for
toml-config-file, plus toml parsing.
- Using "real" types and dummy types interchangeably means that
everything is hidden behind interfaces. This has a performance penalty,
and also it just adds a lot of code.
This PR removes the interface stuff, uses concrete types, and allows for
the setting of Enabled to happen later. It is still assumed that
`metrics.Enable()` is invoked early on.
The somewhat 'heavy' operations, such as ticking meters and exp-decay,
now checks the enable-flag to prevent resource leak.
The change may be large, but it's mostly pretty trivial, and from the
last time I gutted the metrics, I ensured that we have fairly good test
coverage.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR attempts to clean up some ambiguities and quirks from recent
changes to evm flag handling.
This PR mainly focuses on `evm run` subcommand, to use the same flags
for configuring tracing/output options as `statetest/blocktest` does.
Additionally, it adds quite a lot of tests for expected outputs of the
various subcommands, to avoid accidental changes.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR refactors the structlog a bit, making it so that it can be used
in a streaming mode.
-------------
OBS: this PR makes a change in the input `config` config, the third
input-parem field to `debug.traceCall`. Previously, seteting it to e.g.
` {"enableMemory": true, "limit": 1024}` would mean that the response
was limited to `1024` items. Since an 'item' may include both memory and
storage, the actual size of the response was undertermined.
After this change, the response will be limited to `1024` __`bytes`__
(or thereabouts).
-----------
The commandline usage of structlog now uses the streaming mode, leaving
the non-streaming mode of operation for the eth_Call.
There are two benefits of streaming mode
1. Not have to maintain a long list of operations,
2. Not have to duplicate / n-plicate data, e.g. memory / stack /
returndata so that each entry has their own private slice.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Lots of packages depend on eth/downloader just for the SyncMode type.
Since we have a dedicated package for eth protocol configuration, it
makes more sense to define SyncMode there, turning eth/downloader into
more of a leaf package.
* unify `staterunner` and `blockrunner` CLI flags, especially around
tracing
* added support for struct logger or json logging (although having issue
#30658)
* new --cross-check flag to validate the stateless witness collection
/ execution matches stateful
* adds support for tracing the stateless execution when a tracer is set
(to more easily debug differences)
* --human for more readable test summary
* directory or file input, so if you pass tests/spec-tests/fixtures/blockchain_tests it will execute all
blockchain tests
When a tx/block was being traced through the API the state hooks weren't
being called as they should. This is due to #30745 moving the hooked
statedb one level up in the state processor. This PR fixes that.
---------
Co-authored-by: Martin HS <martin@swende.se>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change relocates the EVM tx context switching to the ApplyMessage function.
With this change, we can remove a lot of EVM.SetTxContext calls before
message execution.
### Tracing API changes
- This PR replaces the `GasPrice` field of the `VMContext` struct with
`BaseFee`. Users may instead take the effective gas price from
`tx.EffectiveGasTipValue(env.BaseFee)`.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR introduces a `ContractCodeReader` interface with functions defined:
type ContractCodeReader interface {
Code(addr common.Address, codeHash common.Hash) ([]byte, error)
CodeSize(addr common.Address, codeHash common.Hash) (int, error)
}
This interface can be implemented in various ways. Although the codebase
currently includes only one implementation, additional implementations
could be created for different purposes and scenarios, such as a code
reader designed for the Verkle tree approach or one that reads code from
the witness.
*Notably, this interface modifies the function’s semantics. If the
contract code is not found, no error will be returned. An error should
only be returned in the event of an unexpected issue, primarily for
future implementations.*
The original state.Reader interface is extended with ContractCodeReader
methods, it gives us more flexibility to manipulate the reader with additional
logic on top, e.g. Hooks.
type Reader interface {
ContractCodeReader
StateReader
}
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This updates the message you get when trying to initialize Geth with
genesis.json that doesn't have `terminalTotalDifficulty`. The previous
message was a bit obscure, I had to check the code to find out what the
problem was.
This PR improves the output of the markdown logger a bit.
- Remove `RStack` field,
- Move `Stack` last, since it may have very large vertical expansion
- Make the pre- and post-exec metadata structured into a bullet-list
This pull request refactors the EVM constructor by removing the
TxContext parameter.
The EVM object is frequently overused. Ideally, only a single EVM
instance should be created and reused throughout the entire state
transition of a block, with the transaction context switched as needed
by calling evm.SetTxContext.
Unfortunately, in some parts of the code, the EVM object is repeatedly
created, resulting in unnecessary complexity. This pull request is the
first step towards gradually improving and simplifying this setup.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This adds an API method `DropTransactions` to legacy pool, blob pool and
txpool interface. This method removes all txs currently tracked in the
pools.
It modifies the simulated beacon to use the new method in `Rollback`
which removes previous hacky implementation that also erroneously reset
the gas tip to 1 gwei.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
`flatCallTracer` will now specify the type of a create in the action
via the `creationMethod` field.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR fixes some issues with benchmarks
- [x] Removes log output from a log-test
- [x] Avoids a `nil`-defer in `triedb/pathdb`
- [x] Fixes some crashes re tracers
- [x] Refactors a very resource-expensive benchmark for blobpol.
**NOTE**: this rewrite touches live production code (a little bit), as
it makes the validator-function used by the blobpool configurable.
- [x] Switch some benches over to use pebble over leveldb
- [x] reduce mem overhead in the setup-phase of some tests
- [x] Marks some tests with a long setup-phase to be skipped if `-short`
is specified (where long is on the order of tens of seconds). Ideally,
in my opinion, one should be able to run with `-benchtime 10ms -short`
and sanity-check all tests very quickly.
- [x] Drops some metrics-bechmark which times the speed of `copy`.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This test depends on a 100ms timer, which fails quite often, messing up
our pipelines. Hook directly into the internal version of getPayload
which has the capacity to wait for the full payload before returning.
This might not be absolutely correct from a test perspective, but it
beats failing ci. The alternative would be to expose the full build hook
into the outside, but it might be a bit overkill for this scenario.
When using the prestateTracer, in some cases users are only concerned
with balances or nonce information, and are not interested in the lengthy
contract code or storage data.
Therefore, this PR introduces two new configuration options in the
`prestateTracerConfig` structure:
- `disableCode`
- `disableStorage`
These options allow users to control whether the tracer returns contract
code and storage data during execution tracing. By setting these
options, users can more flexibly customize their needs and focus on
obtaining information that is more critical and relevant to their
specific use cases.
These options work with the default mode as well as `diffMode: true`.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>
rebased https://github.com/ethereum/go-ethereum/pull/29766 . The
downstream branch appears to have been deleted and I don't have perms to
push to that fork.
`TerminalTotalDifficultyPassed` is removed. `TerminalTotalDifficulty`
must now be non-nil, and it is expected that networks are already
merged: we can only import PoW/Clique chains, not produce blocks on
them.
---------
Co-authored-by: stevemilk <wangpeculiar@gmail.com>
This PR moves the logging/tracing-facilities out of `*state.StateDB`,
in to a wrapping struct which implements `vm.StateDB` instead.
In most places, it is a pretty straight-forward change:
- First, hoisting the invocations from state objects up to the statedb.
- Then making the mutation-methods simply return the previous value, so
that the external logging layer could log everything.
Some internal code uses the direct object-accessors to mutate the state,
particularly in testing and in setting up state overrides, which means
that these changes are unobservable for the hooked layer. Thus, configuring
the overrides are not necessarily part of the API we want to publish.
The trickiest part about the layering is that when the selfdestructs are
finally deleted during `Finalise`, there's the possibility that someone
sent some ether to it, which is burnt at that point, and thus needs to
be logged. The hooked layer reaches into the inner layer to figure out
these events.
In package `vm`, the conversion from `state.StateDB + hooks` into a
hooked `vm.StateDB` is performed where needed.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
While looking at some mem profiles from `evm` runs, I noticed that
`goja` compilation of the bigint library was present. The bigint library
compilation happens in a package `init`, whenever the package
`eth/tracers/js` is loaded. This PR changes it to load lazily when
needed.
It becomes slightly faster with this change, and slightly less alloc:y.
Non-scientific benchmark with 100 executions:
```
time for i in {1..100}; do ./evm --code 6040 run; done;
```
current `master`:
```
real 0m6.634s
user 0m5.213s
sys 0m2.277s
```
Without compiling bigint
```
real 0m5.802s
user 0m4.191s
sys 0m1.965s
```
Breaking changes:
- The ChainConfig was exposed to tracers via VMContext passed in
`OnTxStart`. This is unnecessary specially looking through the lens of
live tracers as chain config remains the same throughout the lifetime of
the program. It was there so that native API-invoked tracers could
access it. So instead we moved it to the constructor of API tracers.
Non-breaking:
- Change the default config of the tracers to be `{}` instead of nil.
This way an extra nil check can be avoided.
Refactoring:
- Rename `supply` struct to `supplyTracer`.
- Un-export some hook definitions.
This change brings geth into compliance with the current engine API
specification for the Prague fork. I have moved the assignment of
ExecutionPayloadEnvelope.Requests into BlockToExecutableData to ensure
there is a single place where the type is removed.
While doing so, I noticed that handling of requests in the miner was not
quite correct for the empty payload. It would return `nil` requests for
the empty payload even for blocks after the Prague fork. To fix this, I
have added the emptyRequests field in miner.Payload.
Changelog: https://golangci-lint.run/product/changelog/#1610
Removes `exportloopref` (no longer needed), replaces it with
`copyloopvar` which is basically the opposite.
Also adds:
- `durationcheck`
- `gocheckcompilerdirectives`
- `reassign`
- `mirror`
- `tenv`
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This is a redo of #29052 based on newer specs. Here we implement EIPs
scheduled for the Prague fork:
- EIP-7002: Execution layer triggerable withdrawals
- EIP-7251: Increase the MAX_EFFECTIVE_BALANCE
Co-authored-by: lightclient <lightclient@protonmail.com>
A couple of tests set the debug level to `TRACE` on stdout,
and all subsequent tests in the same package are also affected
by that, resulting in outputs of tens of megabytes.
This PR removes such calls from two packages where it was prevalent.
This makes getting a summary of failing tests simpler, and possibly
reduces some strain from the CI pipeline.
This implements recent changes to EIP-7685, EIP-6110, and
execution-apis.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: Shude Li <islishude@gmail.com>
Reverts ethereum/go-ethereum#30495
You are free to create a proper Clear method if that's the best way. But
one that does a proper cleanup, not some hacky call to set gas which
screws up logs, metrics and everything along the way. Also doesn't work
for legacy pool local transactions.
The current code had a hack in the simulated code, now we have a hack in
live txpooling code. No, that's not acceptable. I want the live code to
be proper, meaningful API, meaningful comments, meaningful
implementation.
Here we move the method that drops all transactions by temporarily increasing the fee
into the TxPool itself. It's better to have it there because we can set it back to the
configured value afterwards. This resolves a TODO in the simulated backend.
This PR integrates witness-enabled block production, witness-creating
payload execution and stateless cross-validation into the `engine` API.
The purpose of the PR is to enable the following use-cases (for API
details, please see next section):
- Cross validating locally created blocks:
- Call `forkchoiceUpdatedWithWitness` instead of `forkchoiceUpdated` to
trigger witness creation too.
- Call `getPayload` as before to retrieve the new block and also the
above created witness.
- Call `executeStatelessPayload` against another client to
cross-validate the block.
- Cross validating locally processed blocks:
- Call `newPayloadWithWitness` instead of `newPayload` to trigger
witness creation too.
- Call `executeStatelessPayload` against another client to
cross-validate the block.
- Block production for stateless clients (local or MEV builders):
- Call `forkchoiceUpdatedWithWitness` instead of `forkchoiceUpdated` to
trigger witness creation too.
- Call `getPayload` as before to retrieve the new block and also the
above created witness.
- Propagate witnesses across the consensus libp2p network for stateless
Ethereum.
- Stateless validator validation:
- Call `executeStatelessPayload` with the propagated witness to
statelessly validate the block.
*Note, the various `WithWitness` methods could also *just be* an
additional boolean flag on the base methods, but this PR wanted to keep
the methods separate until a final consensus is reached on how to
integrate in production.*
---
The following `engine` API types are introduced:
```go
// StatelessPayloadStatusV1 is the result of a stateless payload execution.
type StatelessPayloadStatusV1 struct {
Status string `json:"status"`
StateRoot common.Hash `json:"stateRoot"`
ReceiptsRoot common.Hash `json:"receiptsRoot"`
ValidationError *string `json:"validationError"`
}
```
- Add `forkchoiceUpdatedWithWitnessV1,2,3` with same params and returns
as `forkchoiceUpdatedV1,2,3`, but triggering a stateless witness
building if block production is requested.
- Extend `getPayloadV2,3` to return `executionPayloadEnvelope` with an
additional `witness` field of type `bytes` iff created via
`forkchoiceUpdatedWithWitnessV2,3`.
- Add `newPayloadWithWitnessV1,2,3,4` with same params and returns as
`newPayloadV1,2,3,4`, but triggering a stateless witness creation during
payload execution to allow cross validating it.
- Extend `payloadStatusV1` with a `witness` field of type `bytes` if
returned by `newPayloadWithWitnessV1,2,3,4`.
- Add `executeStatelessPayloadV1,2,3,4` with same base params as
`newPayloadV1,2,3,4` and one more additional param (`witness`) of type
`bytes`. The method returns `statelessPayloadStatusV1`, which mirrors
`payloadStatusV1` but replaces `latestValidHash` with `stateRoot` and
`receiptRoot`.
h/t @MariusVanDerWijden for finding and fixing this on devnet 3.
I made the mistake of thinking `PayloadVersion` was correlated with the
`GetPayloadVX` method, but it actually tracks which version of
`PayloadAttributes` were passed to `forkchoiceUpdated`. So far, Prague
does not necessitate a new version of fcu, so there is no need for
`PayloadV4`.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR fixes an issue with blob transaction propagation due to the blob
transation txpool rejecting transactions with gapped nonces. The
specific changes are:
- fetch transactions from a peer in the order they were announced to
minimize nonce-gaps (which cause blob txs to be rejected
- don't wait on fetching blob transactions after announcement is
received, since they are not broadcast
Testing:
- unit tests updated to reflect that fetch order should always match tx
announcement order
- unit test added to confirm blob transactions are scheduled immediately
for fetching
- running the PR on an eth mainnet full node without incident so far
---------
Signed-off-by: Roberto Bayardo <bayardo@alum.mit.edu>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This is a successor PR to #25743. This PR is based on a new iteration of
the spec: https://github.com/ethereum/execution-apis/pull/484.
`eth_multicall` takes in a list of blocks, each optionally overriding
fields like number, timestamp, etc. of a base block. Each block can
include calls. At each block users can override the state. There are
extra features, such as:
- Include ether transfers as part of the logs
- Overriding precompile codes with evm bytecode
- Redirecting accounts to another address
## Breaking changes
This PR includes the following breaking changes:
- Block override fields of eth_call and debug_traceCall have had the
following fields renamed
- `coinbase` -> `feeRecipient`
- `random` -> `prevRandao`
- `baseFee` -> `baseFeePerGas`
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This pull request introduces a state.Reader interface for state
accessing.
The interface could be implemented in various ways. It can be pure trie
only reader, or the combination of trie and state snapshot. What's more,
this interface allows us to have more flexibility in the future, e.g.
the
archive reader (for accessing archive state).
Additionally, this pull request removes the following metrics
- `chain/snapshot/account/reads`
- `chain/snapshot/storage/reads`
This PR changes how sidechains are handled.
Before the merge, it was possible to import a chain with lower td and not set it as canonical. After the merge, we expect every chain that we get via InsertChain to be canonical. Non-canonical blocks can still be inserted
with InsertBlockWIthoutSetHead.
If during the InsertChain, the existing chain is not canonical anymore, we mark it as a sidechain and send the SideChainEvents normally.
Make tracers more robust by handling `nil` receipt as input.
Also pass in a receipt with gas used in the state test runner.
Closes https://github.com/ethereum/go-ethereum/issues/30117.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
closes#29475, replaces #29657, #30104
Fixes two issues. First is a deadlock where the txpool attempts to reorg, but can't complete because there are no readers left for the new txs subscription. Second, resolves a problem with on demand mode where txs may be left pending when there are more pending txs than block space.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Here I am adding a discv5 nodes source into the p2p dial iterator. It's
an improved version of #29533.
Unlike discv4, the discv5 random nodes iterator will always provide full
ENRs. This means we can apply filtering to the results and will only try
dialing nodes which explictly opt into the eth protocol with a matching
chain.
I have also removed the dial iterator from snap. We don't have an
official DNS list for snap anymore, and I doubt anyone else is running
one. While we could potentially filter for snap on discv5, there will be
very few nodes announcing it, and the extra iterator would just stall
the dialer.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Add coinbase address to javascript tracer context.
This PR adds the `coinbase` address to `jsTracer.ctx`, allowing access
to the coinbase address (fee receipient) in custom JavaScript tracers.
Example usage:
```javascript
result: function(ctx) {
return toAddress(ctx.coinbase);
}
```
This change enables custom tracers to access coinbase address,
previously unavailable, enhancing their capabilities to match built-in
tracers.
This pull request drops the legacy transaction retrieval support from before
eth68, adding the restrictions that transaction metadata must be provided
along with the transaction announment.
This pull request fixes#30229.
During snap sync, large storage will be split into several pieces and
synchronized concurrently. Unfortunately, the tradeoff is that the respective
merkle trie of each storage chunk will be incomplete due to the incomplete
boundaries. The trie nodes on these boundaries will be discarded, and any
dangling nodes on disk will also be removed if they fall on these paths,
ensuring the state healer won't be blocked.
However, the dangling account trie nodes on the path from the root to the
associated account are left untouched. This means the dangling account trie
nodes could potentially stop the state healing and break the assumption that the
entire subtrie should exist if the subtrie root exists. We should consider the
account trie node as the ancestor of the corresponding storage trie node.
In the scenarios described in the above ticket, the state corruption could occur
if there is a dangling account trie node while some storage trie nodes are
removed due to synchronization redo.
The fixing idea is pretty straightforward, the trie nodes on the path from root
to account should all be explicitly removed if an incomplete storage trie
occurs. Therefore, a `delete` operation has been added into `gentrie` to
explicitly clear the account along with all nodes on this path. The special
thing is that it's a cross-trie clearing. In theory, there may be a dangling
node at any position on this account key and we have to clear all of them.
Fixes a slight miscalculation in the downloader queue, which was not accurately taking block withdrawals into account when calculating the size of the items in the queue
This PR fixes an issue in the setMode method of beaconBackfiller where the
log message was not displaying the previous mode correctly. The log message
now shows both the old and new sync modes.
* avoid unnecessary copy
* delete the never used function ProofList
* eth/protocols/snap, trie/trienode: polish the code
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Introduces the first built-in live tracer. The supply tracer tracks ETH supply changes across blocks
and writes the output to disk. This will need to be enabled through CLI using the `--vmtrace supply` flag.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This should fix an occasional test failure in ethclient/simulated.TestForkResendTx.
Inspection of logs revealed the cause of the failure to be that the txpool was not done
reorganizing by the time Fork is called.
This pull request fixes the flay test TestSkeletonSyncRetrievals. In this test, we first
trigger a sync cycle and wait for it to meet certain expectations. We then inject a new
head and potentially also a new peer, then perform another final sync. The test now
performs the newPeer addition before launching the final sync, and waits a bit for that
peer to get registered. This fixes the logic race that made the test fail sometimes.
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
This fixes an issue for `debug_traceBlock*` methods where the BASEFEE opcode was returning always 0. This caused the method return invalid results.
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* all: refactor so NewBlock(..) and WithBody(..) take a types.Body
* core: fixup comments, remove txs != receipts panic
* core/types: add empty withdrawls to body if len == 0
This change adds a testcase and fixes a corner-case in the skeleton sync.
With this change, when doing the skeleton cleanup, we check if the filled header is acually within the range of what we were meant to backfill. If not, it means the backfill was a noop (possibly because we started and stopped it so quickly that it didn't have time to do any meaningful work). In that case, just don't clean up anything.
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
The beacon root when applied in `state_processor.go` is performed right before executing transaction. That means that contract reliying on this value would query the same value found in the block header.
In that spirit, it means that any tracing/operation relying on state data which touches transaction must have updated the beacon root before any transaction processing.
This change fixes three flaky tests `TestEth2AssembleBlock`,`TestEth2NewBlock`, `TestEth2PrepareAndGetPayload` and `TestDisable`.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change removes support for subscribing to pending logs.
"Pending logs" were always an odd feature, because it can never be fully reliable. When support for it was added many years ago, the intention was for this to be used by wallet apps to show the 'potential future token balance' of accounts, i.e. as a way of notifying the user of incoming transfers before they were mined. In order to generate the pending logs, the node must pick a subset of all public mempool transactions, execute them in the EVM, and then dispatch the resulting logs to API consumers.
This pull request defines a gentrie for snap sync purpose.
The stackTrie is used to generate the merkle tree nodes upon receiving a state batch. Several additional options have been added into stackTrie to handle incomplete states (either missing states before or after).
In this pull request, these options have been relocated from stackTrie to genTrie, which serves as a wrapper for stackTrie specifically for snap sync purposes.
Further, the logic for managing incomplete state has been enhanced in this change. Originally, there are two cases handled:
- boundary node filtering
- internal (covered by extension node) node clearing
This changes adds one more:
- Clearing leftover nodes on the boundaries.
This feature is necessary if there are leftover trie nodes in database, otherwise node inconsistency may break the state healing.
time.After is equivalent to NewTimer(d).C, and does not call Stop if the timer is no longer needed. This can cause memory leaks. This change changes many such occations to use NewTimer instead, and calling Stop once the timer is no longer needed.
* use generic atomic types in tx caches
* use generic atomic types in block caches
* eth/catalyst: avoid copying tx in test
---------
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds a flag `--trace.callframes` to t8n which will log info when entering or exiting a call frame in addition to the execution steps.
---------
Co-authored-by: Mario Vega <marioevz@gmail.com>
Here we add a Go API for running tracing plugins within the main block import process.
As an advanced user of geth, you can now create a Go file in eth/tracers/live/, and within
that file register your custom tracer implementation. Then recompile geth and select your tracer
on the command line. Hooks defined in the tracer will run whenever a block is processed.
The hook system is defined in package core/tracing. It uses a struct with callbacks, instead of
requiring an interface, for several reasons:
- We plan to keep this API stable long-term. The core/tracing hook API does not depend on
on deep geth internals.
- There are a lot of hooks, and tracers will only need some of them. Using a struct allows you
to implement only the hooks you want to actually use.
All existing tracers in eth/tracers/native have been rewritten to use the new hook system.
This change breaks compatibility with the vm.EVMLogger interface that we used to have.
If you are a user of vm.EVMLogger, please migrate to core/tracing, and sorry for breaking
your stuff. But we just couldn't have both the old and new tracing APIs coexist in the EVM.
---------
Co-authored-by: Matthieu Vachon <matthieu.o.vachon@gmail.com>
Co-authored-by: Delweng <delweng@gmail.com>
Co-authored-by: Martin HS <martin@swende.se>
This adds support for the Deneb beacon chain fork, and fork handling
in general, to the beacon chain light client implementation.
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Here we add a beacon chain light client for use by geth.
Geth can now be configured to run against a beacon chain API endpoint,
without pointing a CL to it. To set this up, use the `--beacon.api` flag. Information
provided by the beacon chain is verified, i.e. geth does not blindly trust the beacon
API endpoint in this mode. The root of trust are the beacon chain 'sync committees'.
The configured beacon API endpoint must provide light client data. At this time, only
Lodestar and Nimbus provide the necessary APIs.
There is also a standalone tool, cmd/blsync, which uses the beacon chain light client
to drive any EL implementation via its engine API.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
* miner: untangle miner
* miner: use common.hash instead of *types.header
* cmd/geth: deprecate --mine
* eth: get rid of most miner api
* console: get rid of coinbase in welcome message
* miner/stress: get rid of the miner stress test
* eth: get rid of miner.setEtherbase
* ethstats: remove miner and hashrate flags
* ethstats: remove miner and hashrate flags
* cmd: rename pendingBlockProducer to miner.pending.feeRecipient flag
* miner: use pendingFeeRecipient instead of etherbase
* miner: add mutex to protect the pending block
* miner: add mutex to protect the pending block
* eth: get rid of etherbase mentions
* miner: no need to lock the coinbase
* eth, miner: fix linter
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* eth: drop support for forward sync triggers and head block packets
* consensus, eth: enforce always merged network
* eth: fix tx looper startup and shutdown
* cmd, core: fix some tests
* core: remove notion of future blocks
* core, eth: drop unused methods and types
The prestateTracer was reporting an inaccurate nonce for the contract being created in
post EIP-158 transactions. Correct nonce is 0, due to the issue nonce was being reported as 1.
* eth: make transaction propagation paths in the network deterministic
* eth: avoid potential division by 0
* eth: make tx propagation dependent on local node id too
* eth: fix review comments
eth_call and debug_traceCall allow users to override various block fields, among them base fee. However the overriden base fee was not considered for computing the effective gas price of that message, and instead base fee of the base block was used. This has been fixed in this commit.
* core/txpool, miner: speed up blob pool pending retrievals
* miner: fix test merge issue
* eth: same same
* core/txpool/blobpool: speed up blobtx creation in benchmark a bit
* core/txpool/blobpool: fix linter
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change makes the legacy transaction pool use of `uint256.Int` instead of `big.Int`. The changes are made primarily only on the internal functions of legacypool.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth, miner: fix enforcing the minimum miner tip
* ethclient/simulated: fix failing test due the min tip change
* accounts/abi/bind: fix simulater gas tip issue
* eth/downloader: fix skeleton cleanup
* eth/downloader: short circuit if nothing to delete
* eth/downloader: polish the logic in cleanup
* eth/downloader: address comments
At some point, `ForkchoiceUpdatedV2` stopped working for `PayloadAttributesV1` while `paris` was active. This was causing a few failures in hive. This PR fixes that, and also adds a gate in `ForkchoiceUpdatedV1` to disallow `PayloadAttributesV3`.
GetPayloadVX should only return payloads which match its version. GetPayloadV2 is a special snowflake that supports v1 and v2 payloads. This change uses a a version-specific prefix within in the payload id, basically a namespace for the version number.
This PR fixes an issues in the new simulated backend. The root cause is the fact that the transaction pool has an internal reset operation that runs on a background thread.
When a new transaction is added to the pool via the RPC, the transaction is added to a non-executable queue and will be moved to its final location on a background thread. If the machine is overloaded (or simply due to timing issues), it can happen that the simulated backend will try to produce the next block, whilst the pool has not yet marked the newly added transaction executable. This will cause the block to not contain the transaction. This is an issue because we want determinism from the simulator: add a tx, mine a block. It should be in there.
The PR fixes it by adding a Sync function to the txpool, which waits for the current reset operation (if any) to finish, and then runs an entire round of reset on top. The new round is needed because resets are only triggered by new head events, so newly added transactions will not trigger the outer resets that we can wait on. The transaction pool would eventually internally do a reset even on transaction addition, but there's no easy way to wait on that and there's no meaningful reason to bubble that across everything. A clean outer reset will at worse be a small noop goroutine.
This PR introduces a few changes with respect to payload verification in fcu and new payload requests:
* First of all, it undoes the `verifyPayloadAttributes(..)` simplification I attempted in #27872.
* Adds timestamp validation to fcu payload attributes [as required](https://github.com/ethereum/execution-apis/blob/main/src/engine/cancun.md#specification-1) (section 2) by the Engine API spec.
* For the new payload methods, I also update the verification of the executable data. For `newPayloadV2`, it does not currently ensure that cancun values are `nil`. Which could make it possible to submit cancun payloads through it.
* On `newPayloadV3` the same types of checks are added. All shanghai and cancun related fields in the executable data must be non-nil, with the addition that the timestamp is _only_ with cancun.
* Finally it updates a newly failing catalyst test to call the correct fcu and new payload methods depending on the fork.
This change makes use of uin256 to represent balance in state. It touches primarily upon statedb, stateobject and state processing, trying to avoid changes in transaction pools, core types, rpc and tracers.
This change simplifies the logic for indexing transactions and enhances the UX when transaction is not found by returning more information to users.
Transaction indexing is now considered as a part of the initial sync, and `eth.syncing` will thus be `true` if transaction indexing is not yet finished. API consumers can use the syncing status to determine if the node is ready to serve users.
The code to compute a versioned hash was duplicated a couple times, and also had a small
issue: if we ever change params.BlobTxHashVersion, it will most likely also cause changes
to the actual hash computation. So it's a bit useless to have this constant in params.
In the tracing tests, the base fee was generally set to nil. This commit changes this to pass the proper base instead, and fixes the few tests which become broken by the change.
This is a rewrite of the 'simulated backend', an implementation of the ethclient interfaces
which is backed by a simulated blockchain. It was getting annoying to maintain the old
version of the simulated backend feature because there was a lot of code duplication with
the main client.
The new version is built using parts that we already have: an in-memory geth node instance
running in developer mode provides the chain, while the Go API is provided by ethclient.
A backwards-compatibility wrapper is provided, but the simulated backend has also moved to
a more sensible import path: github.com/ethereum/go-ethereum/ethclient/simulated
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.
Main changes are as follows:
* removes any log handlers that were unused in the Geth codebase.
* Json, logfmt, and terminal formatters are now slog handlers.
* Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options.
* `--log.backtraceat` and `--log.debug` are removed.
The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged.
A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed.
----
For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did
```golang
log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true))))
```
You now instead need to do
```golang
log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true)))
```
See more about reasoning here: https://github.com/ethereum/go-ethereum/issues/28558#issuecomment-1820606613
* eth/gasestimator: early exit for plain transfer and error allowance
* core, eth/gasestimator: hard guess at a possible required gas
* internal/ethapi: update estimation tests with the error ratio
* eth/gasestimator: I hate you linter
* graphql: fix gas estimation test
---------
Co-authored-by: Oren <orenyomtov@users.noreply.github.com>
This change fixes two type-inconsistencies in the JS tracer:
- In most places we return byte arrays as a `Uint8Array` to the tracer. However it seems we missed doing the conversion for `ctx` fields which are passed to the tracer during `result`. They are passed as simple arrays. I think Uint8Arrays are more suitable and we should change this inconsistency. Note: this will be a breaking-change. But I believe the effect is small. If we look at our tracers we see that these fields (`ctx.from`, `ctx.to`, etc.) are used in 2 ways. Passed to `toHex` which takes both array or buffer. Or the length was measured which is the same for both types.
- The `slice` taking in `int, int` params versus `memory.slice` taking `int64, int64` params. I suggest changing `slice` types to `int64`. This should have no effect almost in any case.
There were several problems related to dumping state.
- If a preimage was missing, even if we had set the `OnlyWithAddresses` to `false`, to export them anyway, the way the mapping was constructed (using `common.Address` as key) made the entries get lost anyway. Concerns both state- and blockchain tests.
- Blockchain test execution was not configured to store preimages.
This changes makes it so that the block test executor takes a callback, just like the state test executor already does. This callback can be used to examine the post-execution state, e.g. to aid debugging of test failures.
* cmd, les, tests: remove light client code
This commit removes the light client (LES) code.
Since the merge the light client has been broken and
it is hard to maintain it alongside the normal client.
We decided it would be best to remove it for now and
maybe rework and reintroduce it in the future.
* cmd, eth: remove some more mentions of light mode
* cmd: re-add flags and mark as deprecated
* cmd: warn the user about deprecated flags
* eth: better error message
geth --dev can be used with an existing data directory and genesis block. Since
dev mode only works with PoS, we need to verify that the merge has happened.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR moves our fuzzers from tests/fuzzers into whatever their respective 'native' package is.
The historical reason why they were placed in an external location, is that when they were based on go-fuzz, they could not be "hidden" via the _test.go prefix. So in order to shove them away from the go-ethereum "production code", they were put aside.
But now we've rewritten them to be based on golang testing, and thus can be brought back. I've left (in tests/) the ones that are not production (bls128381), require non-standard imports (secp requires btcec, bn256 requires gnark/google/cloudflare deps).
This PR also adds a fuzzer for precompiled contracts, because why not.
This PR utilizes a newly rewritten replacement for go-118-fuzz-build, namely gofuzz-shim, which utilises the inputs from the fuzzing engine better.
Currently, geth's will return `[]` for any `len(topics) > 4` log filter. The EVM only supports up to four logs, via LOG4 opcode, so larger criterias fail. This change makes the filter query exit early in those cases.
