This PR implements eth/69. This protocol version drops the bloom filter
from receipts messages, reducing the amount of data needed for a sync
by ~530GB (2.3B txs * 256 byte) uncompressed. Compressed this will
be reduced to ~100GB
The new version also changes the Status message and introduces the
BlockRangeUpdate message to relay information about the available history
range.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In this pull request, snapshot generation in pathdb has been ported from
the legacy state snapshot implementation. Additionally, when running in
path mode, legacy state snapshot data is now managed by the pathdb
based snapshot logic.
Note: Existing snapshot data will be re-generated, regardless of whether
it was previously fully constructed.
This adds a geth subcommand for downloading era1 files and placing them into
the correct location. The tool can be used even while geth is already running
on the datadir. Downloads are checked against a hard-coded list of checksums
for mainnet and sepolia.
```
./geth download-era --server $SERVER --block 333333
./geth download-era --server $SERVER --block 333333-444444
./geth download-era --server $SERVER --epoch 0-10
./geth download-era --server $SERVER --all
```
The implementation reuses the file downloader we already had for
fetching build tools. I've done some refactoring on it to make sure it
can support the new use case, and there are some changes to the build
here as well.
Adding values to the witness introduces a new class of issues for
computing gas: if there is not enough gas to cover adding an item to the
witness, then the item should not be added to the witness.
The problem happens when several items are added together, and that
process runs out of gas. The witness gas computation needs a way to
signal that not enough gas was provided. These values can not be
hardcoded, however, as they are context dependent, i.e. two calls to the
same function with the same parameters can give two different results.
The approach is to return both the gas that was actually consumed, and
the gas that was necessary. If the values don't match, then a witness
update OOG'd. The caller should then charge the `consumed` value
(remaining gas will be 0) and error out.
Why not return a boolean instead of the wanted value? Because when
several items are touched, we want to distinguish which item lacked gas.
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
This adds support for naming the source iterators of FairMix, like so:
mix.AddSource(enode.WithSourceName("mySource", iter))
The source that produced the latest node is returned by the new NodeSource method.
This PR contains three refactors:
- refactor the latest fork check that we use quite extensively
- refactor the nil checks in NewPayloads
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
I saw in https://github.com/ethereum/go-ethereum/pull/31378 introduced
github.com/crate-crypto/go-eth-kzg to calculate the kzg hash, and
github.com/crate-crypto/go-kzg-4844 was only used in the test files, so
propose to drop it with go-eth-kzg instead
This adds a metric called `chain/mgasps`, which records how many million
gas per second are being used during block insertion.
The value is calculated as `usedGas * 1000 / elapsed`, and it's updated
in the `insertStats.report` method. Also cleaned up the log output to
reuse the same value instead of recalculating it.
Useful for monitoring block processing throughput.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR introduces an allocation-free version of the
Transaction.EffectiveGasTip method to improve performance by reducing
memory allocations.
## Changes
- Added a new `EffectiveGasTipInto` method that accepts a destination
parameter to avoid memory allocations
- Refactored the existing `EffectiveGasTip` method to use the new
allocation-free implementation
- Updated related methods (`EffectiveGasTipValue`, `EffectiveGasTipCmp`,
`EffectiveGasTipIntCmp`) to use the allocation-free approach
- Added tests and benchmarks to verify correctness and measure
performance improvements
## Motivation
In high-transaction-volume environments, the `EffectiveGasTip` method is
called frequently. Reducing memory allocations in this method decreases
garbage collection pressure and improves overall system performance.
## Benchmark Results
As-Is
BenchmarkEffectiveGasTip/Original-10 42089140 27.45 ns/op 8 B/op 1
allocs/op
To-Be
BenchmarkEffectiveGasTip/IntoMethod-10 72353263 16.73 ns/op 0 B/op 0
allocs/op
## Summary of Improvements
- **Performance**: ~39% faster execution (27.45 ns/op → 16.73 ns/op)
- **Memory**: Eliminated all allocations (8 B/op → 0 B/op)
- **Allocation count**: Reduced from 1 to 0 allocations per operation
This optimization follows the same pattern successfully applied to other
methods in the codebase, maintaining API compatibility while improving
performance.
## Safety & Compatibility
This optimization has no side effects or adverse impacts because:
- It maintains functional equivalence as confirmed by comprehensive
tests
- It preserves API compatibility with existing callers
- It follows clear memory ownership patterns with the destination
parameter
- It maintains thread safety by only modifying the caller-provided
destination parameter
This optimization follows the same pattern successfully applied to other
methods in the codebase, providing better performance without
compromising stability or correctness.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR creates a global hasher pool that can be used by all packages.
It also removes a bunch of the package local pools.
It also updates a few locations to use available hashers or the global
hashing pool to reduce allocations all over the codebase.
This change should reduce global allocation count by ~1%
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This pull request enhances the block prefetcher by executing transactions
in parallel to warm the cache alongside the main block processor.
Unlike the original prefetcher, which only executes the next block and
is limited to chain syncing, the new implementation can be applied to any
block. This makes it useful not only during chain sync but also for regular
block insertion after the initial sync.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR fixes an issue that could lead to data corruption.
Writing the state history may fail due to insufficient disk space or
other potential errors. With this change, the entire state insertion
will be aborted instead of silently ignoring the error.
Without this fix, state transitions would continue while the associated
state history is lost. After a restart, the resulting gap would be detected,
making recovery impossible.
This pull request introduces a SyncKeyValue function to the
ethdb.KeyValueStore
interface, providing the ability to forcibly flush all previous writes
to disk.
This functionality is critical for go-ethereum, which internally uses
two independent
database engines: a key-value store (such as Pebble, LevelDB, or
memoryDB for
testing) and a flat-file–based freezer. To ensure write-order
consistency between
these engines, the key-value store must be explicitly synced before
writing to the
freezer and vice versa.
Fixes
- https://github.com/ethereum/go-ethereum/issues/31405
- https://github.com/ethereum/go-ethereum/issues/29819
The function `BacktraceAt` has been removed in #28187 . But the API
end-point `debug_backtraceAt` is not removed from the file
`internal/web3ext/web3ext.go`.
Fixes methods debug_standardTraceBlockToFile
and debug_standardTraceBadBlockToFile which were
outputting empty files.
---------
Co-authored-by: maskpp <maskpp266@gmail.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This change adds a limit for RPC method names to prevent potential abuse
where large method names could lead to large response sizes.
The limit is enforced in:
- handleCall for regular RPC method calls
- handleSubscribe for subscription method calls
Added tests in websocket_test.go to verify the length limit
functionality for both regular method calls and subscriptions.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Issue statement: when user requests eth_simulateV1 to return full
transaction objects, these objects always had an empty `from` field. The
reason is we lose the sender when translation the message into a
types.Transaction which is then later on serialized.
I did think of an alternative but opted to keep with this approach as it
keeps complexity at the edge. The alternative would be to pass down a
signer object to RPCMarshal* methods and define a custom signer which
keeps the senders in its state and doesn't attempt the signature
recovery.
updates the log entries in `core/filtermaps/indexer.go` to remove double
quotes around keys like "first block" and "last block", changing them to
`firstblock` and `lastblock`. This brings them in line with the general
logging style used across the codebase, where log keys are unquoted
single words.
For example, the log:
` INFO [...] "first block"=..., "last block"=...`
Is now rendered as:
` INFO [...] firstblock=..., lastblock=...`
This change improves readability and maintains consistency with logs
such as:
` INFO [...] number=2 sealhash=... uncles=0 txs=0 ...`
No functional behavior is changed — this is purely a formatting cleanup
for better developer experience.
Fixes https://github.com/ethereum/go-ethereum/issues/31732.
This logic was removed in the recent refactoring in the txindexer to
handle history cutoff (#31393). It was first introduced in this PR:
https://github.com/ethereum/go-ethereum/pull/28908.
I have tested it and it works as an alternative to #31745.
This PR packs 3 changes to the flow of fetching txs from the API:
- It caches the indexer tail after each run is over to avoid hitting the
db all the time as was done originally in #28908.
- Changes `backend.GetTransaction`. It doesn't return an error anymore
when tx indexer is in progress. It shifts the responsibility to the
caller to check the progress. The reason is that in most cases we anyway
check the txpool for the tx. If it was indeed a pending tx we can avoid
the indexer progress check.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
`DefaultBlobSchedule` is actually used downstream to calculate blob fees
(e.g.,
[src](601a380e47/op-service/eth/blob.go (L301))),
this PR makes it explicit that these params are for `Ethereum prod`
instead of `test chains`.
This PR fixes an initialization bug that in some cases caused the map
renderer to leave the last, partially rendered map as is and resume
rendering from the next map. At initialization we check whether the
existing rendered maps are consistent with the current chain view and
revert them if necessary. Until now this happened through an ugly hacky
solution, a "limited" chain view that was supposed to trigger a rollback
of some maps in the renderer logic if necessary. This whole setup worked
under assumptions that just weren't true any more. As a result it always
tried to revert the last map but also it did not shorten the indexed
range, only set `headIndexed` to false which indicated to the renderer
logic that the last map is fully populated (which it wasn't).
Now an explicit rollback of any unusable (reorged) maps happens at
startup, which also means that no hacky chain view is necessary, as soon
as the new `FilterMaps` is returned, the indexed range and view are
consistent with each other.
In the first commit an extra check is also added to `writeFinishedMaps`
so that if there is ever again a bug that would result in a gapped index
then it will not break the db with writing the incomplete data. Instead
it will return an indexing error which causes the indexer to revert to
unindexed mode and print an error log instantly. Hopefully this will not
ever happen in the future, but in order to test this safeguard check I
manually triggered the bug with only the first commit enabled, which
caused an indexing error as expected. With the second commit added (the
actual fix) the same operation succeeded without any issues.
Note that the database version is also bumped in this PR in order to
enforce a full reindexing as any existing database might be potentially
broken.
Fixes https://github.com/ethereum/go-ethereum/issues/31729
This PR fixes the out-of-range block number logic of `getBlockLvPointer`
which sometimes caused searches to fail if the head was updated in the
wrong moment. This logic ensures that querying the pointer of a future
block returns the pointer after the last fully indexed block (instead of
failing) and therefore an async range update will not cause the search
to fail. Earier this behaviour only worked when `headIndexed` was true
and `headDelimiter` pointed to the end of the indexed range. Now it also
works for an unfinished index.
This logic is also moved from `FilterMaps.getBlockLvPointer` to
`FilterMapsMatcherBackend.GetBlockLvPointer` because it is only required
by the search anyways. `FilterMaps.getBlockLvPointer` now only returns a
pointer for existing blocks, consistently with how it is used in the
indexer/renderer.
Note that this unhandled case has been present in the code for a long
time but went unnoticed because either one of two previously fixed bugs
did prevent it from being triggered; the incorrectly positive
`tempRange.headIndexed` (fixed in
https://github.com/ethereum/go-ethereum/pull/31680), though caused other
problems, prevented this one from being triggered as with a positive
`headIndexed` no database read was triggered in `getBlockLvPointer`.
Also, the unnecessary `indexLock` in `synced()` (fixed in
https://github.com/ethereum/go-ethereum/pull/31708) usually did prevent
the search seeing the temp range and therefore avoided noticeable
issues.
The functions `rpcRequest` and `batchRpcRequest` call `baseRpcRequest`.
And `resp.Body` will be closed in the function `baseRpcRequest` later by
`t.Cleanup`:
```go
func baseRpcRequest(t *testing.T, url, bodyStr string, extraHeaders ...string) *http.Response {
// ......
t.Cleanup(func() { resp.Body.Close() })
return resp
}
```
Add tests for GetBlockHeaders that verify client does not disconnect when unlikely block numbers are requested, e.g. max uint64.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
Since the block hash is not returned for pending blocks, ethclient cannot unmarshal into RPC block. This makes hash optional on rpc block and compute the hash locally for pending blocks to correctly key the tx sender cache.
a82303f4e3/internal/ethapi/api.go (L500-L504)
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This changes the filtermaps to only pull up the raw receipts, not the
derived receipts which saves a lot of allocations.
During normal execution this will reduce the allocations of the whole
geth node by ~15%.
For PeerDAS, we need to compute cell proofs. Both ckzg and gokzg support
computing these cell proofs.
This PR does the following:
- Update the go-kzg library from "github.com/crate-crypto/go-kzg-4844"
to "github.com/crate-crypto/go-eth-kzg" which will be the new upstream
for go-kzg moving forward
- Update ckzg from v1.0.0 to v2.0.1 and switch to /v2
- Updates the trusted setup to contain the g1 points both in lagrange
and monomial form
- Expose `ComputeCells` to compute the cell proofs
