In order to reduce the amount of code that is embedded into the keeper
binary, I am removing all the verkle code that uses go-verkle and
go-ipa. This will be followed by further PRs that are more like stubs to
replace code when the keeper build is detected.
I'm keeping the binary tree of course. This means that you will still
see `isVerkle` variables all over the codebase, but they will be renamed
when code is touched (i.e. this is not an invitation for 30+ AI slop
PRs).
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR introduces a new debug feature, logging the slow blocks with
detailed performance statistics, such as state read, EVM execution and
so on.
Notably, the detailed performance statistics of slow blocks won't be
logged during the sync to not overwhelm users. Specifically, the statistics
are only logged if there is a single block processed.
Example output
```
########## SLOW BLOCK #########
Block: 23537063 (0xa7f878611c2dd27f245fc41107d12ebcf06b4e289f1d6acf44d49a169554ee09) txs: 248, mgasps: 202.99
EVM execution: 63.295ms
Validation: 1.130ms
Account read: 6.634ms(648)
Storage read: 17.391ms(1434)
State hash: 6.722ms
DB commit: 3.260ms
Block write: 1.954ms
Total: 99.094ms
State read cache: account (hit: 622, miss: 26), storage (hit: 1325, miss: 109)
##############################
```
Recently in #31630 we removed support for overriding the network id in
preset networks. While this feature is niche, it is useful for shadow
forks. This PR proposes we add the functionality back, but in a simpler
way.
Instead of checking whether the flag is set in each branch of the
network switch statement, simply apply the network flag after the switch
statement is complete. This retains the following behavior:
1. Auto network id based on chain id still works, because `IsSet` only
returns true if the flag is _actually_ set. Not if it just has a default
set.
2. The preset networks will set their network id directly and only if
the network id flag is set is it overridden. This, combined with the
override genesis flag is what allows the shadow forks.
3. Setting the network id to the same network id that the preset _would
have_ set causes no issues and simply emits the `WARN` that the flag is
being set explicitly. I don't think people explicitly set the network id
flag often.
```
WARN [10-22|09:36:15.052] Setting network id with flag id=10
```
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Adds a flag to specify how many blobs a node is willing to include in
their locally build block as specified in
https://eips.ethereum.org/EIPS/eip-7872
I deviated from the EIP in one case, I allowed for specifying 0 as the
minimum blobs/block
Fix#33212.
This PR remove `github.com/olekukonko/tablewriter` from dependencies and
use a naive stub implementation.
`github.com/olekukonko/tablewriter` is used to format database inspection
output neatly. However, it requires custom adjustments for TinyGo and is
incompatible with the latest version.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
This PR is an alternative to #32556.
Instead of trying to be smart and reuse `geth init`, we can introduce a
new flag `--genesis` that loads the `genesis.json` from file into the
`Genesis` object in the same path that the other network flags currently
work in.
Question: is something like `--genesis` enough to start deprecating
`geth init`?
--
```console
$ geth --datadir data --hoodi
..
INFO [10-06|22:37:11.202] - BPO2: @1762955544
..
$ geth --datadir data --genesis genesis.json
..
INFO [10-06|22:37:27.988] - BPO2: @1862955544
..
```
Pull the genesis [from the
specs](https://raw.githubusercontent.com/eth-clients/hoodi/refs/heads/main/metadata/genesis.json)
and modify one of the BPO timestamps to simulate a shadow fork.
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
New RPC method eth_sendRawTransactionSync(rawTx, timeoutMs?) that
submits a signed tx and blocks until a receipt is available or a timeout
elapses.
Two CLI flags to tune server-side limits:
--rpc.txsync.defaulttimeout (default wait window)
--rpc.txsync.maxtimeout (upper bound; requests are clamped)
closes https://github.com/ethereum/go-ethereum/issues/32094
---------
Co-authored-by: aodhgan <gawnieg@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Fix logging in the verkle dump path to report the actual key being
processed.
Previously, the loop always logged keylist[0], which misled users when
expanding multiple keys and made debugging harder. This change aligns
the
log with the key passed to root.Get, improving traceability and
diagnostics.
Add cli configurable limit for the number of addresses allowed in
eth_getLogs filter criteria:
https://github.com/ethereum/go-ethereum/issues/32264
Key changes:
- Added --rpc.getlogmaxaddrs CLI flag (default: 1000) to configure the
maximum number of addresses
- Updated ethconfig.Config with FilterMaxAddresses field for
configuration management
- Modified filter system to use the configurable limit instead of the
hardcoded maxAddresses constant
- Enhanced test coverage with new test cases for address limit
validation
- Removed hardcoded validation from JSON unmarshaling, moving it to
runtime validation
Please notice that I remove the check at FilterCriteria UnmarshalJSON
because the runtime config can not pass into this validation.
Please help review this change!
---------
Co-authored-by: zsfelfoldi <zsfelfoldi@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
The format that is currently reported by the chain isn't very useful, as
it gives an average for ALL the nodes, and not only the leaves, which
skews the results.
Also, until now there was no way to activate the reporting of errors.
We also decided that metrics weren't the right tool to report this data,
so we decided to dump it to the console if the flag is enabled. A better
system should be built, but for now, printing to the logs does the job.
Add state size tracking and retrieve api, start geth with `--state.size-tracking`,
the initial bootstrap is required (around 1h on mainnet), after the bootstrap,
use `debug_stateSize()` RPC to retrieve the state size:
```
> debug.stateSize()
{
accountBytes: "0x39681967b",
accountTrienodeBytes: "0xc57939f0c",
accountTrienodes: "0x198b36ac",
accounts: "0x129da14a",
blockNumber: "0x1635e90",
contractCodeBytes: "0x2b63ef481",
contractCodes: "0x1c7b45",
stateRoot: "0x9c36a3ec3745d72eea8700bd27b90dcaa66de0494b187c5600750044151e620a",
storageBytes: "0x18a6e7d3f1",
storageTrienodeBytes: "0x2e7f53fae6",
storageTrienodes: "0x6e49a234",
storages: "0x517859c5"
}
```
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
As in #32060 we introduced the file based journal path, for the other
sub command(eg: snapshot, db), we should also pass the directory to the
triedb, else the subcommand(eg: `geth snapshot`) failed to run:
```bash
geth snapshot verify-state --datadir /geth-data
...
INFO [09-02|02:12:29.493] Allocated cache and file handles database=/geth-data/geth/chaindata cache=512.00MiB handles=524,288
INFO [09-02|02:12:32.746] Opened ancient database database=/geth-data/geth/chaindata/ancient/chain readonly=true
INFO [09-02|02:12:32.746] Opened Era store datadir=/geth-data/geth/chaindata/ancient/chain/era
INFO [09-02|02:12:32.758] State scheme set to already existing scheme=path
INFO [09-02|02:12:32.760] Load database journal from disk
INFO [09-02|02:12:32.764] Failed to load journal, discard it err="journal not found"
INFO [09-02|02:12:32.789] Opened ancient database database=/geth-data/geth/chaindata/ancient/state readonly=true
INFO [09-02|02:12:32.790] Initialized path database readonly=true triecache=0.00B statecache=0.00B buffer=0.00B history="entire chain"
ERROR[09-02|02:12:32.791] Failed to verify state root=c5458d..4cc785 err="unknown layer: c5458d476da0136a67ef24a93b909aa5c29efa5c5b885dbd1fbaed4e784cc785"
```
This is something interesting I came across during my benchmarks, we
spent ~3.8% of all allocations allocating the header number on the heap.
```
(pprof) list GetHeaderByHash
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core.(*BlockChain).GetHeaderByHash in github.com/ethereum/go-ethereum/core/blockchain_reader.go
0 5786566117 (flat, cum) 15.15% of Total
. . 79:func (bc *BlockChain) GetHeaderByHash(hash common.Hash) *types.Header {
. 5786566117 80: return bc.hc.GetHeaderByHash(hash)
. . 81:}
. . 82:
. . 83:// GetHeaderByNumber retrieves a block header from the database by number,
. . 84:// caching it (associated with its hash) if found.
. . 85:func (bc *BlockChain) GetHeaderByNumber(number uint64) *types.Header {
ROUTINE ======================== github.com/ethereum/go-ethereum/core.(*HeaderChain).GetHeaderByHash in github.com/ethereum/go-ethereum/core/headerchain.go
0 5786566117 (flat, cum) 15.15% of Total
. . 404:func (hc *HeaderChain) GetHeaderByHash(hash common.Hash) *types.Header {
. 1471264309 405: number := hc.GetBlockNumber(hash)
. . 406: if number == nil {
. . 407: return nil
. . 408: }
. 4315301808 409: return hc.GetHeader(hash, *number)
. . 410:}
. . 411:
. . 412:// HasHeader checks if a block header is present in the database or not.
. . 413:// In theory, if header is present in the database, all relative components
. . 414:// like td and hash->number should be present too.
(pprof) list GetBlockNumber
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core.(*HeaderChain).GetBlockNumber in github.com/ethereum/go-ethereum/core/headerchain.go
94438817 1471264309 (flat, cum) 3.85% of Total
. . 100:func (hc *HeaderChain) GetBlockNumber(hash common.Hash) *uint64 {
94438817 94438817 101: if cached, ok := hc.numberCache.Get(hash); ok {
. . 102: return &cached
. . 103: }
. 1376270828 104: number := rawdb.ReadHeaderNumber(hc.chainDb, hash)
. . 105: if number != nil {
. 554664 106: hc.numberCache.Add(hash, *number)
. . 107: }
. . 108: return number
. . 109:}
. . 110:
. . 111:type headerWriteResult struct {
(pprof) list ReadHeaderNumber
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core/rawdb.ReadHeaderNumber in github.com/ethereum/go-ethereum/core/rawdb/accessors_chain.go
204606513 1376270828 (flat, cum) 3.60% of Total
. . 146:func ReadHeaderNumber(db ethdb.KeyValueReader, hash common.Hash) *uint64 {
109577863 1281242178 147: data, _ := db.Get(headerNumberKey(hash))
. . 148: if len(data) != 8 {
. . 149: return nil
. . 150: }
95028650 95028650 151: number := binary.BigEndian.Uint64(data)
. . 152: return &number
. . 153:}
. . 154:
. . 155:// WriteHeaderNumber stores the hash->number mapping.
. . 156:func WriteHeaderNumber(db ethdb.KeyValueWriter, hash common.Hash, number uint64) {
```
Opening this to discuss the idea, I know that rawdb.EmptyNumber is not a
great name for the variable, open to suggestions
Downloading from a range was failing because it would return and error
early with an error misinterpreting "start-end".
---------
Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This changes the era1 downloader to place the files into the correct
location where geth will actually use them. Also adds integration with
the new --datadir.era flag.
This implements a backing store for chain history based on era1 files.
The new store is integrated with the freezer. Queries for blocks and receipts
below the current freezer tail are handled by the era store.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR modifies the disclaimer/banner that is printed when starting up
Geth in dev mode:
* if the client is spun up in ephemeral dev mode with a keystore
override, the address of the first (prefunded) account is printed.
* if the client is spun up in ephemeral mode without a keystore
override, the genesis allocation contains a single static prefunded EOA
account. It's address and private key are logged.
* the banner is printed at the end of client initialization to make it
more prominent. Previously, it was logged towards the beginning of
client initialization and subsequent logging from startup filled the
terminal, pushing it out of view of the user.
Other change is that we now use a static prefunded dev account instead
of generating a random one when instantiating a new dev mode chain.
This is an example of what the banner looks like:
```
WARN [05-28|23:05:16.475] You are running Geth in --dev mode. Please note the following:
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] 1. This mode is only intended for fast, iterative development without assumptions on
WARN [05-28|23:05:16.475] security or persistence.
WARN [05-28|23:05:16.475] 2. The database is created in memory unless specified otherwise. Therefore, shutting down
WARN [05-28|23:05:16.475] your computer or losing power will wipe your entire block data and chain state for
WARN [05-28|23:05:16.475] your dev environment.
WARN [05-28|23:05:16.475] 3. A random, pre-allocated developer account will be available and unlocked as
WARN [05-28|23:05:16.475] eth.coinbase, which can be used for testing. The random dev account is temporary,
WARN [05-28|23:05:16.475] stored on a ramdisk, and will be lost if your machine is restarted.
WARN [05-28|23:05:16.475] 4. Mining is enabled by default. However, the client will only seal blocks if transactions
WARN [05-28|23:05:16.475] are pending in the mempool. The miner's minimum accepted gas price is 1.
WARN [05-28|23:05:16.475] 5. Networking is disabled; there is no listen-address, the maximum number of peers is set
WARN [05-28|23:05:16.475] to 0, and discovery is disabled.
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] Running in ephemeral mode. The following account has been prefunded in the genesis:
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] Account
WARN [05-28|23:05:16.475] ------------------
WARN [05-28|23:05:16.475] 0x71562b71999873db5b286df957af199ec94617f7 (10^49 ETH)
WARN [05-28|23:05:16.475]
WARN [05-28|23:05:16.475] Private Key
WARN [05-28|23:05:16.475] ------------------
WARN [05-28|23:05:16.475] 0xb71c71a67e1177ad4e901695e1b4b9ee17ae16c6668d313eac2f96dbcda3f291
WARN [05-28|23:05:16.475]
```
closes#31796
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This adds a geth subcommand for downloading era1 files and placing them into
the correct location. The tool can be used even while geth is already running
on the datadir. Downloads are checked against a hard-coded list of checksums
for mainnet and sepolia.
```
./geth download-era --server $SERVER --block 333333
./geth download-era --server $SERVER --block 333333-444444
./geth download-era --server $SERVER --epoch 0-10
./geth download-era --server $SERVER --all
```
The implementation reuses the file downloader we already had for
fetching build tools. I've done some refactoring on it to make sure it
can support the new use case, and there are some changes to the build
here as well.
I added the history mode configuration in eth/ethconfig initially, since
it seemed like the logical place. But it turns out we need access to the
intended pruning setting at a deeper level, and it actually needs to be
integrated with the blockchain startup procedure.
With this change applied, if a node previously had its history pruned,
and is subsequently restarted **without** the `--history.chain
postmerge` flag, the `BlockChain` initialization code will now verify
the freezer tail against the known pruning point of the predefined
network and will restore pruning status. Note that this logic is quite
restrictive, we allow non-zero tail only for known networks, and only
for the specific pruning point that is defined.
This PR adds a new `--beacon.checkpoint.file` config flag to geth and
blsync which specifies a checkpoint import/export file. If a file with
an existing checkpoint is specified, it is used for initialization
instead of the hardcoded one (except when `--beacon.checkpoint` is also
specified simultaneously). Whenever the client encounters a new valid
finality update with a suitable finalized beacon block root at an epoch
boundary, it saves the block root in hex format to the checkpoint file.
This pull request introduces new sync logic for pruning mode. The downloader will now skip
insertion of block bodies and receipts before the configured history cutoff point.
Originally, in snap sync, the header chain and other components (bodies and receipts) were
inserted separately. However, in Proof-of-Stake, this separation is unnecessary since the
sync target is already verified by the CL.
To simplify the process, this pull request modifies `InsertReceiptChain` to insert headers
along with block bodies and receipts together. Besides, `InsertReceiptChain` doesn't have
the notion of reorg, as the common ancestor is always be found before the sync and extra
side chain is truncated at the beginning if they fall in the ancient store. The stale
canonical chain flags will always be rewritten by the new chain. Explicit reorg logic is
no longer required in `InsertReceiptChain`.
This adds a new subcommand 'geth prune-history' that removes the pre-merge history
on supported networks. Geth is not fully ready to work in this mode, please do not run
this command on your production node.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
this adds 2 features to improve `geth --dev` experience.
1. we don't need to use `dev_SetFeeRecipient` to set initial coinbase
address. it was a pain.
2. we don't need to unlock keystore if we don't use it. we had it
because of clique.
Here I am adding a config option and geth flag (`--history.chain`) for
configuring history pruning. There are two options available:
- `--history.chain all` is the default and will keep all history like
before.
- `--history.chain postmerge` will configure the history cutoff point to
the merge block.
The option doesn't actually do anything right now, but we need it as a
precursor for other history pruning changes.
When I press Ctrl-C during the import of multiple files, the import
process will still attempt to import the subsequent files. However, in
normal circumstances, users would expect the import to stop immediately
upon pressing Ctrl-C.
And because the current file was not finished importing, subsequent
import tasks often fail due to an `unknown ancestor` error.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This pull request delivers the new version of the state history, where
the raw storage key is used instead of the hash.
Before the cancun fork, it's supported by protocol to destruct a
specific account and therefore, all the storage slot owned by it should
be wiped in the same transition.
Technically, storage wiping should be performed through storage
iteration, and only the storage key hash will be available for traversal
if the state snapshot is not available. Therefore, the storage key hash
is chosen as the identifier in the old version state history.
Fortunately, account self-destruction has been deprecated by the
protocol since the Cancun fork, and there are no empty accounts eligible
for deletion under EIP-158. Therefore, we can conclude that no storage
wiping should occur after the Cancun fork. In this case, it makes no
sense to keep using hash.
Besides, another big reason for making this change is the current format
state history is unusable if verkle is activated. Verkle tree has a
different key derivation scheme (merkle uses keccak256), the preimage of
key hash must be provided in order to make verkle rollback functional.
This pull request is a prerequisite for landing verkle.
Additionally, the raw storage key is more human-friendly for those who
want to manually check the history, even though Solidity already
performs some hashing to derive the storage location.
---
This pull request doesn't bump the database version, as I believe the
database should still be compatible if users degrade from the new geth
version to old one, the only side effect is the persistent new version
state history will be unusable.
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
