Add state size tracking and retrieve api, start geth with `--state.size-tracking`,
the initial bootstrap is required (around 1h on mainnet), after the bootstrap,
use `debug_stateSize()` RPC to retrieve the state size:
```
> debug.stateSize()
{
accountBytes: "0x39681967b",
accountTrienodeBytes: "0xc57939f0c",
accountTrienodes: "0x198b36ac",
accounts: "0x129da14a",
blockNumber: "0x1635e90",
contractCodeBytes: "0x2b63ef481",
contractCodes: "0x1c7b45",
stateRoot: "0x9c36a3ec3745d72eea8700bd27b90dcaa66de0494b187c5600750044151e620a",
storageBytes: "0x18a6e7d3f1",
storageTrienodeBytes: "0x2e7f53fae6",
storageTrienodes: "0x6e49a234",
storages: "0x517859c5"
}
```
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
As in #32060 we introduced the file based journal path, for the other
sub command(eg: snapshot, db), we should also pass the directory to the
triedb, else the subcommand(eg: `geth snapshot`) failed to run:
```bash
geth snapshot verify-state --datadir /geth-data
...
INFO [09-02|02:12:29.493] Allocated cache and file handles database=/geth-data/geth/chaindata cache=512.00MiB handles=524,288
INFO [09-02|02:12:32.746] Opened ancient database database=/geth-data/geth/chaindata/ancient/chain readonly=true
INFO [09-02|02:12:32.746] Opened Era store datadir=/geth-data/geth/chaindata/ancient/chain/era
INFO [09-02|02:12:32.758] State scheme set to already existing scheme=path
INFO [09-02|02:12:32.760] Load database journal from disk
INFO [09-02|02:12:32.764] Failed to load journal, discard it err="journal not found"
INFO [09-02|02:12:32.789] Opened ancient database database=/geth-data/geth/chaindata/ancient/state readonly=true
INFO [09-02|02:12:32.790] Initialized path database readonly=true triecache=0.00B statecache=0.00B buffer=0.00B history="entire chain"
ERROR[09-02|02:12:32.791] Failed to verify state root=c5458d..4cc785 err="unknown layer: c5458d476da0136a67ef24a93b909aa5c29efa5c5b885dbd1fbaed4e784cc785"
```
This PR is the first step in the trienode history series.
It introduces the `nodeWithOrigin` struct in the path database, which tracks
the original values of dirty nodes to support trienode history construction.
Note, the original value is always empty in this PR, so it won't break the
existing journal for encoding and decoding. The compatibility of journal
should be handled in the following PR.
Another getBlobs PR on top of
https://github.com/ethereum/go-ethereum/pull/32190 to avoid some minor
regressions.
- bring back more log messages from before
- continue processing also on some internal errors
- ensure v2 complies with spec even if there are internal errors
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This pull request fixes a regression, introduced in #32190
Specifically, in GetBlobsV1 engine API, if any blob is missing, the null
should be placed in
response, unfortunately a behavioral change was introduced in #32190,
returning an error
instead.
What's more, a more comprehensive test suite is added to cover both
`GetBlobsV1` and
`GetBlobsV2` endpoints.
Switches to using counters so that the gauges don't cause any
information to be lost. Counters can be used to calculate all sorts of
metrics on Grafana. Which is also why min/avg/max logic is removed to
make things simple and small here.
~Will probably be mostly supplanted by #32224, but this should do for
now for devnet 3.~
Seems like #32224 is going to take some more time, so I have completed
the implementation of eth_config here. It is quite a bit simpler to
implement now that the config hashing was removed.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Implement the binary tree as specified in [eip-7864](https://eips.ethereum.org/EIPS/eip-7864).
This will gradually replace verkle trees in the codebase. This is only
running the tests and will not be executed in production, but will help
me rebase some of my work, so that it doesn't bitrot as much.
---------
Signed-off-by: Guillaume Ballet
Co-authored-by: Parithosh Jayanthi <parithosh.jayanthi@ethereum.org>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Filtering for leaf nodes was missing from #32388, which means that even
the root done was reported, which made little sense for the bloatnet
data processing we want to do.
### Summary
Fixes long-standing ETA calculation errors in progress indicators that
have been present since February 2021. The current implementation
produces increasingly inaccurate estimates due to integer division
precision loss.
### Problem
3aeccadd04/triedb/pathdb/history_indexer.go (L541-L553)
The ETA calculation has two critical issues:
1. **Integer division precision loss**: `speed` is calculated as
`uint64`
2. **Off-by-one**: `speed` uses `+ 1`(2 times) to avoid division by
zero, however it makes mistake in the final calculation
This results in wildly inaccurate time estimates that don't improve as
progress continues.
### Example
Current output during state history indexing:
```
lvl=info msg="Indexing state history" processed=16858580 left=41802252 elapsed=18h22m59.848s eta=11h36m42.252s
```
**Expected calculation:**
- Speed: 16858580 ÷ 66179848ms = 0.255 blocks/ms
- ETA: 41802252 ÷ 0.255 = ~45.6 hours
**Current buggy calculation:**
- Speed: rounds to 1 block/ms
- ETA: 41802252 ÷ 1 = ~11.6 hours ❌
### Solution
- Created centralized `CalculateETA()` function in common package
- Replaced all 8 duplicate code copies across the codebase
### Testing
Verified accurate ETA calculations during archive node reindexing with
significantly improved time estimates.
`db inspect` on the full database currently takes **30min+**, because
the db iterate was run in one thread, propose to split the key-space to
256 sub range, and assign them to the worker pool to speed up.
After the change, the time of running `db inspect --workers 16` reduced
to **10min**(the keyspace is not evenly distributed).
---------
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This pull request preserves the root->ID mappings in the path database
even after the associated state histories are truncated, regardless of
whether the truncation occurs at the head or the tail.
The motivation is to support an additional history type, trienode history.
Since the root->ID mappings are shared between two history instances,
they must not be removed by either one.
As a consequence, the root->ID mappings remain in the database even
after the corresponding histories are pruned. While these mappings may
become dangling, it is safe and cheap to keep them.
Additionally, this pull request enhances validation during historical
reader construction, ensuring that only canonical historical state will be
served.
When maxPeers was just above some perfect square, and a few peers
dropped for some reason, we changed the peer selection function.
When new peers were acquired, we changed again.
This PR improves the selection function, in two ways. First, it will always select
sqrt(peers) to broadcast to. Second, the selection now uses siphash with a secret
key, to guard against information leaks about tx source.
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a prestateTracer test case covering 7702 delegation.
---------
Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This pull request implements #32235 , constructing blob sidecar in new
format (cell proof)
if the Osaka has been activated.
Apart from that, it introduces a pre-conversion step in the blob pool
before adding the txs.
This mechanism is essential for handling the remote **legacy** blob txs
from the network.
One thing is still missing and probably is worthy being highlighted
here: the blobpool may
contain several legacy blob txs before the Osaka and these txs should be
converted once
Osaka is activated. While the `GetBlob` API in blobpool is capable for
generating cell proofs
at the runtime, converting legacy txs at one time is much cheaper
overall.
---------
Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
Co-authored-by: lightclient <lightclient@protonmail.com>
Replace hardcoded 5-second sleep with polling loop that actively checks
snap sync state. This approach is already used in other project tests
(like account_cache_test.go) and provides better reliability by:
- Reducing flaky behavior on slower systems
- Finishing early when sync completes quickly
- Using 1-second timeout with 100ms polling intervals
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
closes#32240#32232
The main cause for the time out is the slow json encoding of large data.
In #32240 they tried to resolve the issue by reducing the size of the
test. However as Felix pointed out, the test is still kind of confusing.
I've refactored the test so it is more understandable and have reduced
the amount of data needed to be json encoded. I think it is still
important to ensure that the default read limit is not active, so I have
retained one large (~32 MB) test case, but it's at least smaller than
the existing ~64 MB test case.
This pull request refactors the internal implementation in path database
a bit, specifically:
- purge the state index data in batch
- simplify the logic of state history construction and index, make it more readable
This is a internal refactoring PR, renaming the history to stateHistory.
It's a pre-requisite PR for merging trienode history, avoid the name
conflict.
The TestTraceChain function was missing a defer backend.teardown() call,
which is required to properly release blockchain resources after test
completion.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Full disclosure: this has been generated by AI. The goal is to have a
quick check that the PR format is correct, before we merge it. This is
to avoid the periodical case when someone forgets to add a milestone or
check the title matches our preferred format.
Supersedes #32470.
### What
- snap: shorten stall watchdog in `eth/protocols/snap/sync_test.go` from
1m to 10s.
- discover/v5: consolidate FINDNODE negative tests into a single
table-driven test:
- `TestUDPv5_findnodeCall_InvalidNodes` covers:
- invalid IP (unspecified `0.0.0.0`) → ignored
- low UDP port (`<=1024`) → ignored
### Why
- Addresses TODOs:
- “Make tests smaller” (reduce long 1m timeout).
- “check invalid IPs”; also cover low port per `verifyResponseNode`
rules (UDP must be >1024).
### How it’s validated
- Test-only changes; no production code touched.
- Local runs:
- `go test ./p2p/discover -count=1 -timeout=300s` → ok
- `go test ./eth/protocols/snap -count=1 -timeout=600s` → ok
- Lint:
- `go run build/ci.go lint` → 0 issues on modified files.
### Notes
- The test harness uses `enode.ValidSchemesForTesting` (which includes
the “null” scheme), so records signed with `enode.SignNull` are
signature-valid; failures here are due to IP/port validation in
`verifyResponseNode` and `netutil.CheckRelayAddr`.
- Tests are written as a single table-driven function for clarity; no
helpers or environment switching.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR should reduce overall allocations of a running node by ~10
percent. Since most allocations are coming from the re-heaping of the
transaction pool.
```
(pprof) list EffectiveGasTipCmp
Total: 38197204475
ROUTINE ======================== github.com/ethereum/go-ethereum/core/types.(*Transaction).EffectiveGasTipCmp in github.com/ethereum/go-ethereum/core/types/transaction.go
0 3766837369 (flat, cum) 9.86% of Total
. . 386:func (tx *Transaction) EffectiveGasTipCmp(other *Transaction, baseFee *big.Int) int {
. . 387: if baseFee == nil {
. . 388: return tx.GasTipCapCmp(other)
. . 389: }
. . 390: // Use more efficient internal method.
. . 391: txTip, otherTip := new(big.Int), new(big.Int)
. 1796172553 392: tx.calcEffectiveGasTip(txTip, baseFee)
. 1970664816 393: other.calcEffectiveGasTip(otherTip, baseFee)
. . 394: return txTip.Cmp(otherTip)
. . 395:}
. . 396:
. . 397:// EffectiveGasTipIntCmp compares the effective gasTipCap of a transaction to the given gasTipCap.
. . 398:func (tx *Transaction) EffectiveGasTipIntCmp(other *big.Int, baseFee *big.Int) int {
```
This PR reduces the allocations for comparing two transactions from 2 to
0:
```
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/core/types
cpu: Intel(R) Core(TM) Ultra 7 155U
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
EffectiveGasTipCmp/Original-14 64.67n ± 2% 25.13n ± 9% -61.13% (p=0.000 n=10)
│ /tmp/old.txt │ /tmp/new.txt │
│ B/op │ B/op vs base │
EffectiveGasTipCmp/Original-14 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10)
│ /tmp/old.txt │ /tmp/new.txt │
│ allocs/op │ allocs/op vs base │
EffectiveGasTipCmp/Original-14 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
```
It also speeds up the process by ~60%
There are two minor caveats with this PR:
- We change the API for `EffectiveGasTipCmp` and `EffectiveGasTipIntCmp`
(which are probably not used by much)
- We slightly change the behavior of `tx.EffectiveGasTip` when it
returns an error. It would previously return a negative number on error,
now it does not (since uint256 does not allow for negative numbers)
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
## Description
This PR fixes a bug in the Ledger hardware wallet version validation
logic for EIP-155 transaction signing. The original condition
incorrectly allowed older versions that don't support EIP-155 such as
0.9.9 and 0.1.5 to proceed.
