This pull request removes the node copy operation to reduce memory
allocation. Key Changes as below:
**(a) Use `decodeNodeUnsafe` for decoding nodes retrieved from the trie
node reader**
In the current implementation of the MPT, once a trie node blob is
retrieved, it is passed to `decodeNode` for decoding. However,
`decodeNode` assumes the supplied byte slice might be mutated later, so
it performs a deep copy internally before parsing the node.
Given that the node reader is implemented by the path database and the
hash database, both of which guarantee the immutability of the returned
byte slice. By restricting the node reader interface to explicitly
guarantee that the returned byte slice will not be modified, we can
safely replace `decodeNode` with `decodeNodeUnsafe`. This eliminates the
need for a redundant byte copy during each node resolution.
**(b) Modify the trie in place**
In the current implementation of the MPT, a copy of a trie node is
created before any modifications are made. These modifications include:
- Node resolution: Converting the value from a hash to the actual node.
- Node hashing: Tagging the hash into its cache.
- Node commit: Replacing the children with its hash.
- Structural changes: For example, adding a new child to a fullNode or
replacing a child of a shortNode.
This mechanism ensures that modifications only affect the live tree,
leaving all previously created copies unaffected.
Unfortunately, this property leads to a huge memory allocation
requirement. For example, if we want to modify the fullNode for n times,
the node will be copied for n times.
In this pull request, all the trie modifications are made in place. In
order to make sure all previously created copies are unaffected, the
`Copy` function now will deep-copy all the live nodes rather than the
root node itself.
With this change, while the `Copy` function becomes more expensive, it's
totally acceptable as it's not a frequently used one. For the normal
trie operations (Get, GetNode, Hash, Commit, Insert, Delete), the node
copy is not required anymore.
This PR adds an extra condition to the log indexer initialization in
order to avoid initializing with block 0 as target head. Previously this
caused the indexer to initialize without a checkpoint. Later, when the
real chain head was set, it indexed the entire history, then unindexed
most of it if only the recent history was supposed to be indexed. Now
the init only happens when there is an actual synced chain head and
therefore the index is initialized at the most recent checkpoint and
only the last year is indexed according to the default parameters.
During checkpoint initialization the best available checkpoint is also
checked against the history cutoff point and fails if the indexing would
have to start from a block older than the cutoff. If initialization
fails then the indexer reverts to unindexed mode instead of retrying
because the the failure conditions cannot be expected to recover later.
Instead of reporting all filtermaps stuff in one line, I'm breaking it
down into the three separate kinds of entries here.
```
+-----------------------+-----------------------------+------------+------------+
| DATABASE | CATEGORY | SIZE | ITEMS |
+-----------------------+-----------------------------+------------+------------+
| Key-Value store | Log index filter-map rows | 59.21 GiB | 616077345 |
| Key-Value store | Log index last-block-of-map | 12.35 MiB | 269755 |
| Key-Value store | Log index block-lv | 421.70 MiB | 22109169 |
```
Also added some other changes to make it easier to debug:
- restored bloombits into the inspect output, so we notice if it doesn't
get deleted for some reason
- tracking of unaccounted key examples
This adds a new subcommand 'geth prune-history' that removes the pre-merge history
on supported networks. Geth is not fully ready to work in this mode, please do not run
this command on your production node.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In #31384 we unindex TXes prior to the merge block. However when the
node starts up it will try to re-index those back if the config is to index the
whole chain. This change makes the indexer aware of the history cutoff block,
avoiding reindexing in that segment.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This pull request improves the protection mechanism in the txpool for
senders with delegation. A sender with either delegation or pending
delegation is now limited to a maximum of one in-flight executable
transaction, while gapped transactions will be rejected.
Reason:
If nonce-gapped transaction from delegated/pending-delegated senders
can be acceptable, then it's no-longer possible to send another
"executable" transaction with correct nonce due to the policy of at most
one inflight tx. The gapped transaction will be stuck in the txpool, with no
meaningful way to unlock the sender.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This fixes the handshake in a scenario where the remote end sends two unknown
packets in a row. When this happens, we would previously respond to both with
a WHOAREYOU challenge, but keep only the latest sent challenge. Transmission is
assumed to be unreliable, so any client that sends two request packets simultaneously
has to be prepared to follow up on whichever request leads to a handshake. With
this fix, we force them to do the handshake that we can actually complete.
Fixes#30581
This PR changes the matcher syncing conditions so that it is possible to
run a search while head indexing is in progress. Previously it was a
requirement to have the head indexed in order to perform matcher sync
before and after a search. This was unnecessarily strict as the purpose
was just to avoid syncing the valid range with the temporary shortened
indexed range applied while updating existing head maps. Now the sync
condition explicitly checks whether the indexer has a temporary indexed
range with some head maps being partially updated.
It also fixes a deadlock that happened when matcher synchronization was
attempted in the event handler called from the `writeFinishedMaps`
periodical callback.
This pull request fixes a broken unit test
```
=== CONT TestTracingWithOverrides
api_test.go:1012: result: {"gas":21167,"failed":false,"returnValue":"0x0000000000000000000000000000000000000000000000000000000000000002","structLogs":[{"pc":0,"op":"PUSH1","gas":24978860,"gasCost":3,"depth":1,"stack":[]},{"pc":2,"op":"CALLDATALOAD","gas":24978857,"gasCost":3,"depth":1,"stack":["0x0"]},{"pc":3,"op":"PUSH1","gas":24978854,"gasCost":3,"depth":1,"stack":["0x1"]},{"pc":5,"op":"ADD","gas":24978851,"gasCost":3,"depth":1,"stack":["0x1","0x1"]},{"pc":6,"op":"PUSH1","gas":24978848,"gasCost":3,"depth":1,"stack":["0x2"]},{"pc":8,"op":"MSTORE","gas":24978845,"gasCost":6,"depth":1,"stack":["0x2","0x0"]},{"pc":9,"op":"PUSH1","gas":24978839,"gasCost":3,"depth":1,"stack":[]},{"pc":11,"op":"PUSH1","gas":24978836,"gasCost":3,"depth":1,"stack":["0x20"]},{"pc":13,"op":"RETURN","gas":24978833,"gasCost":0,"depth":1,"stack":["0x20","0x0"]}]}
api_test.go:1013: test 10, result mismatch, have
{21167 false 0x0000000000000000000000000000000000000000000000000000000000000002}
, want
{21167 false 0000000000000000000000000000000000000000000000000000000000000002}
api_test.go:1012: result: {"gas":25664,"failed":false,"returnValue":"0x000000000000000000000000c6e93f4c1920eaeaa1e699f76a7a8c18e3056074","structLogs":[]}
api_test.go:1013: test 11, result mismatch, have
{25664 false 0x000000000000000000000000c6e93f4c1920eaeaa1e699f76a7a8c18e3056074}
, want
{25664 false 000000000000000000000000c6e93f4c1920eaeaa1e699f76a7a8c18e3056074}
```
This is a **breaking change** to the opcode tracer. The top-level
`returnValue` field of a trace will be now hex-encoded. If the return
data is empty, this field will contain "0x".
Fixes#31196
This PR fixes the broken request error handling of the workload filter
tests. Until now `validateHistoryPruneErr` was invoked with `fq.Err` as
an input which was always nil and a timeout or http error was reported
as a result content mismatch.
Also, in case of `errPrunedHistory` it is wrong to return here without
setting an error because then it will look like a valid empty result and
the check will later fail. So instead `errPrunedHistory` is always
returned now (without printing an error message) and the callers of
`run` should handle this special case (typically ignore silently).
This removes the signer type-train in favor of defining a single object
that can handle all tx types. Supported types are enabled via a map.
Notably, the new signer also supports disabling legacy transactions.
This PR roughly halves the number of allocations needed to compute the
sigHash for a transaction.
This sigHash is used whenever we recover a signature of a transaction,
so quite often. During a recent benchmark full syncing on Holesky,
roughly 2.8% of all allocations were happening here because the fields
from the transaction would be copied multiple times.
```
66168733 153175654 (flat, cum) 2.80% of Total
. . 368:func (s londonSigner) Hash(tx *Transaction) common.Hash {
. . 369: if tx.Type() != DynamicFeeTxType {
. . 370: return s.eip2930Signer.Hash(tx)
. . 371: }
. 19169966 372: return prefixedRlpHash(
. . 373: tx.Type(),
26442187 26442187 374: []interface{}{
. . 375: s.chainId,
6848616 6848616 376: tx.Nonce(),
. 19694077 377: tx.GasTipCap(),
. 18956774 378: tx.GasFeeCap(),
6357089 6357089 379: tx.Gas(),
. 12321050 380: tx.To(),
. 16865054 381: tx.Value(),
13435187 13435187 382: tx.Data(),
13085654 13085654 383: tx.AccessList(),
. . 384: })
. . 385:}
```
This PR reduces the allocations and speeds up the computation of the
sigHash by ~22%, which is quite significantly given that this operation
involves a call to Keccak
```
// BenchmarkHash-8 440082 2639 ns/op 384 B/op 13 allocs/op
// BenchmarkHash-8 493566 2033 ns/op 240 B/op 6 allocs/op
```
```
Hash-8 2.691µ ± 8% 2.097µ ± 9% -22.07% (p=0.000 n=10)
```
It also kinda cleans up stuff in my opinion, since the transaction
should itself know best how to compute the sighash
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Currently, when answering GetPooledTransaction request, txpool.Get() is
used. When the requested hash is blob transaction, blobpool.Get() is
called. This function loads the RLP-encoded transaction from limbo then
decodes and returns. Later, in answerGetPooledTransactions, we need to
RLP encode again. This decode then encode is wasteful. This commit adds
GetRLP to transaction pool interface so that answerGetPooledTransactions
can use the RLP-encoded from limbo directly.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The main use case I see of this is that it allows users to estimate gas
against the same state that they query for their nonce, and the same
state they base the data of their transaction against. This helps ensure
that gas estimation won't fail and the transaction won't revert on-chain
because of a mismatch between the state used for gas estimation and the
state used to generate the inputs to gas estimation or the transaction's
nonce when submitted to the mempool.
This PR also updates the EstimateGas comment based on the new geth
`eth_estimateGas` default of using latest state as of v1.12.0:
https://github.com/ethereum/go-ethereum/pull/24363
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
When I'm running `geth import --metrics`, the metrics is different to
normal `geth --metrics`, so the grafana dashboard needs to be updated,
eg: `eth_db_chaindata_disk_read` vs `disk_read`.
So I think we should always set the name to `eth/db/chaindata` for more
convenient.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
This PR fixes a bug in the `lastMapBoundaryBefore` logic that resulted
in incorrect checkpoint initialization (started rendering from the
previous epoch boundary which caused the `needTailEpoch` check to fail).
Apparently the bug was present before but went unnoticed because
`needTailEpoch` behaved differently.
Fixes https://github.com/ethereum/go-ethereum/issues/31413
Here we add the notion of prunable tables for the `TruncateTail` operation
in the freezer. TruncateTail for the chain freezer now only truncates the body and
receipts tables, leaving headers and hashes as-is.
This change also requires changing the validation/repair at startup to allow for
tables with different tail. For the header and hash tables, we now require them to start
at number zero.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR implements a new version of the abigen utility (v2) which exists
along with the pre-existing v1 version.
Abigen is a utility command provided by go-ethereum that, given a
solidity contract ABI definition, will generate Go code to transact/call
the contract methods, converting the method parameters/results and
structures defined in the contract into corresponding Go types. This is
useful for preventing the need to write custom boilerplate code for
contract interactions.
Methods in the generated bindings perform encoding between Go types and
Solidity ABI-encoded packed bytecode, as well as some action (e.g.
`eth_call` or creating and submitting a transaction). This limits the
flexibility of how the generated bindings can be used, and prevents
easily adding new functionality, as it will make the generated bindings
larger for each feature added.
Abigen v2 was conceived of by the observation that the only
functionality that generated Go bindings ought to perform is conversion
between Go types and ABI-encoded packed data. Go-ethereum already
provides various APIs which in conjunction with conversion methods
generated in v2 bindings can cover all functionality currently provided
by v1, and facilitate all other previously-desired use-cases.
## Generating Bindings
To generate contract bindings using abigen v2, invoke the `abigen`
command with the `--v2` flag. The functionality of all other flags is
preserved between the v2 and v1 versions.
## What is Generated in the Bindings
The execution of `abigen --v2` generates Go code containing methods
which convert between Go types and corresponding ABI-encoded data
expected by the contract. For each input-accepting contract method and
the constructor, a "packing" method is generated in the binding which
converts from Go types to the corresponding packed solidity expected by
the contract. If a method returns output, an "unpacking" method is
generated to convert this output from ABI-encoded data to the
corresponding Go types.
For contracts which emit events, an unpacking method is defined for each
event to unpack the corresponding raw log to the Go type that it
represents.
Likewise, where custom errors are defined by contracts, an unpack method
is generated to unpack raw error data into a Go type.
## Using the Generated Bindings
For a smooth user-experience, abigen v2 comes with a number of utility
functions to be used in conjunction with the generated bindings for
performing common contract interaction use-cases. These include:
* filtering for historical logs of a given topic
* watching the chain for emission of logs with a given topic
* contract deployment methods
* Call/Transact methods
https://geth.ethereum.org will be updated to include a new tutorial page
for abigen v2 with full code examples. The page currently exists in a
PR: https://github.com/ethereum/go-ethereum/pull/31390 .
There are also extensive examples of interactions with contract bindings
in [test
cases](cc855c7ede/accounts/abi/bind/v2/lib_test.go)
provided with this PR.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
this adds 2 features to improve `geth --dev` experience.
1. we don't need to use `dev_SetFeeRecipient` to set initial coinbase
address. it was a pain.
2. we don't need to unlock keystore if we don't use it. we had it
because of clique.
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.34.0 to
0.36.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="85d1d54551"><code>85d1d54</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="cde1dda944"><code>cde1dda</code></a>
proxy, http/httpproxy: do not mismatch IPv6 zone ids against hosts</li>
<li><a
href="fe7f0391aa"><code>fe7f039</code></a>
publicsuffix: spruce up code gen and speed up PublicSuffix</li>
<li><a
href="459513d1f8"><code>459513d</code></a>
internal/http3: move more common stream processing to genericConn</li>
<li><a
href="aad0180cad"><code>aad0180</code></a>
http2: fix flakiness from t.Log when GOOS=js</li>
<li><a
href="b73e5746f6"><code>b73e574</code></a>
http2: don't log expected errors from writing invalid trailers</li>
<li><a
href="5f45c776a9"><code>5f45c77</code></a>
internal/http3: make read-data tests usable for server handlers</li>
<li><a
href="43c2540165"><code>43c2540</code></a>
http2, internal/httpcommon: reject userinfo in :authority</li>
<li><a
href="1d78a08500"><code>1d78a08</code></a>
http2, internal/httpcommon: factor out server header logic for
h2/h3</li>
<li><a
href="0d7dc54a59"><code>0d7dc54</code></a>
quic: add Conn.ConnectionState</li>
<li>Additional commits viewable in <a
href="https://github.com/golang/net/compare/v0.34.0...v0.36.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ethereum/go-ethereum/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This pull request enhances the unit test, avoiding unnecessary failure
in CI.
```
--- FAIL: TestSimulatedBeaconSendWithdrawals (12.08s)
simulated_beacon_test.go:139: timed out without including all withdrawals/txs
FAIL
```
Fixes `evm statetest` for state test fixtures with multiple fork entries
in their `post` field (e.g.,
[chainId.json](81862e4848/GeneralStateTests/stChainId/chainId.json (L39))).
When these re-activated flags aren't exposed, `statetest` only executes
the fixture for a single fork entry instead of all of the forks as
expected.
This only affects ethereum/tests state test fixtures, not
ethereum/execution-spec-tests (EEST) state tests. EEST writes a separate
fixture/test case (i.e. a separate top-level dict entry in the .json)
for each fork configuration as apposed to combining multiple forks in
one fixture test case: New EEST state tests targeting Prague behavior
are not affected.
Currently, even though it takes in a `Logger` interface,
`log.SetDefualt` enforces that the concrete type of the provided logger
is `*logger` because:
1. in `init` `root.Store` is called with a `*logger`
2. `atomic.Value` panics if the concrete type provided in `Store` is not
consistent across calls.
([ref](https://pkg.go.dev/sync/atomic#Value.Store))
> All calls to Store for a given Value must use values of the same
concrete type.
This PR changes to use `sync.RWMutex` and adds a test that panics on
`master`.
