This changes the filtermaps to only pull up the raw receipts, not the
derived receipts which saves a lot of allocations.
During normal execution this will reduce the allocations of the whole
geth node by ~15%.
This PR applies the config overrides to the new config as well,
otherwise they will not be applied to defined configs, making
shadowforks impossible.
To test:
```
> ./build/bin/geth --override.prague 123 --dev --datadir /tmp/geth
INFO [04-28|21:20:47.009] - Prague: @123
> ./build/bin/geth --override.prague 321 --dev --datadir /tmp/geth
INFO [04-28|21:23:59.760] - Prague: @321
``
TruncatePending shows up bright red on our nodes, because it computes
the length of a map multiple times.
I don't know why this is so expensive, but around 20% of our time is
spent on this, which is super weird.
```
//PR: BenchmarkTruncatePending-24 17498 69397 ns/op 32872 B/op 3 allocs/op
//Master: BenchmarkTruncatePending-24 9960 123954 ns/op 32872 B/op 3 allocs/op
```
```
benchmark old ns/op new ns/op delta
BenchmarkTruncatePending-24 123954 69397 -44.01%
benchmark old allocs new allocs delta
BenchmarkTruncatePending-24 3 3 +0.00%
benchmark old bytes new bytes delta
BenchmarkTruncatePending-24 32872 32872 +0.00%
```
This simple PR is a 44% improvement over the old state
```
OUTINE ======================== github.com/ethereum/go-ethereum/core/txpool/legacypool.(*LegacyPool).truncatePending in github.com/ethereum/go-ethereum/core/txpool/legacypool/legacypool.go
1.96s 18.02s (flat, cum) 19.57% of Total
. . 1495:func (pool *LegacyPool) truncatePending() {
. . 1496: pending := uint64(0)
60ms 2.99s 1497: for _, list := range pool.pending {
250ms 5.48s 1498: pending += uint64(list.Len())
. . 1499: }
. . 1500: if pending <= pool.config.GlobalSlots {
. . 1501: return
. . 1502: }
. . 1503:
. . 1504: pendingBeforeCap := pending
. . 1505: // Assemble a spam order to penalize large transactors first
. 510ms 1506: spammers := prque.New[int64, common.Address](nil)
140ms 2.50s 1507: for addr, list := range pool.pending {
. . 1508: // Only evict transactions from high rollers
50ms 5.08s 1509: if uint64(list.Len()) > pool.config.AccountSlots {
. . 1510: spammers.Push(addr, int64(list.Len()))
. . 1511: }
. . 1512: }
. . 1513: // Gradually drop transactions from offenders
. . 1514: offenders := []common.Address{}
```
```go
// Benchmarks the speed of batch transaction insertion in case of multiple accounts.
func BenchmarkTruncatePending(b *testing.B) {
// Generate a batch of transactions to enqueue into the pool
pool, _ := setupPool()
defer pool.Close()
b.ReportAllocs()
batches := make(types.Transactions, 4096+1024+1)
for i := range len(batches) {
key, _ := crypto.GenerateKey()
account := crypto.PubkeyToAddress(key.PublicKey)
pool.currentState.AddBalance(account, uint256.NewInt(1000000), tracing.BalanceChangeUnspecified)
tx := transaction(uint64(0), 100000, key)
batches[i] = tx
}
for _, tx := range batches {
pool.addRemotesSync([]*types.Transaction{tx})
}
b.ResetTimer()
// benchmark truncating the pending
for range b.N {
pool.truncatePending()
}
}
```
This PR fixes a deadlock situation is deleteTailEpoch that might arise
when
range delete is running in iterator based fallback mode (either using
leveldb
database or the hashdb state storage scheme).
In this case a stopCb callback is called periodically that does check
events,
including matcher sync requests, in which case it tries to acquire
indexLock
for read access, while deleteTailEpoch already held it for write access.
This pull request removes the indexLock acquiring in
`FilterMapsMatcherBackend.synced`
as this function is only called in the indexLoop.
Fixes https://github.com/ethereum/go-ethereum/issues/31700
This PR adds the `AuthorizationList` field to the `CallMsg` interface to support `eth_call`
and `eth_estimateGas` of set-code transactions.
---------
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR ensures that caching a slice or a slice of slices will never
affect the original version by always cloning a slice fetched from cache
if it is not used in a guaranteed read only way.
This PR changes the chain view update mechanism of the log filter.
Previously the head updates were all wired through the indexer, even in
unindexed mode. This was both a bit weird and also unsafe as the
indexer's chain view was updates asynchronously with some delay, making
some log related tests flaky. Also, the reorg safety of the indexed
search was integrated with unindexed search in a weird way, relying on
`syncRange.ValidBlocks` in the unindexed case too, with a special
condition added to only consider the head of the valid range but not the
tail in the unindexed case.
In this PR the current chain view is directly accessible through the
filter backend and unindexed search is also chain view based, making it
inherently safe. The matcher sync mechanism is now only used for indexed
search as originally intended, removing a few ugly special conditions.
The PR is currently based on top of
https://github.com/ethereum/go-ethereum/pull/31642
Together they fix https://github.com/ethereum/go-ethereum/issues/31518
and replace https://github.com/ethereum/go-ethereum/pull/31542
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR makes `filtermaps.ChainView` thread safe because it is used
concurrently both by the indexer and multiple matcher threads. Even
though it represents an immutable view of the chain, adding a mutex lock
to the `blockHash` function is necessary because it does so by extending
its list of non-canonical hashes if the underlying blockchain is
changed.
The unsafe concurrency did cause a panic once after running the unit
tests for several hours and it could also happen during live operation.
This PR makes the conditions for using a map rendering snapshot stricter
so that whenever a reorg happens, only a snapshot of a common ancestor
block can be used. The issue fixed in
https://github.com/ethereum/go-ethereum/pull/31642 originated from using
a snapshot that wasn't a common ancestor. For example in the following
reorg scenario: `A->B`, then `A->B2`, then `A->B2->C2`, then `A->B->C`
the last reorg triggered a render from snapshot `B` saved earlier. Now
this is possible under certain conditions but extra care is needed, for
example if block `B` crosses a map boundary then it should not be
allowed. With the latest fix the checks are sufficient but I realized I
would just feel safer if we disallowed this rare and risky scenario
altogether and just render from snapshot `A` after the last reorg in the
example above. The performance difference if a few milliseconds and it
occurs rarely (about once a day on Holesky, probably much more rare on
Mainnet).
Note that this PR only makes the snapshot conditions stricter and
`TestIndexerRandomRange` does check that snapshots are still used
whenever it's obviously possible (adding blocks after the current head
without a reorg) so this change can be considered safe. Also I am
running the unit tests and the fuzzer and everything seems to be fine.
This pull request improves error handling for local transaction submissions.
Specifically, if a transaction fails with a temporary error but might be
accepted later, the error will not be returned to the user; instead, the
transaction will be tracked locally for resubmission.
However, if the transaction fails with a permanent error (e.g., invalid
transaction or insufficient balance), the error will be propagated to the user.
These errors returned in the legacyPool are regarded as temporary failure:
- `ErrOutOfOrderTxFromDelegated`
- `txpool.ErrInflightTxLimitReached`
- `ErrAuthorityReserved`
- `txpool.ErrUnderpriced`
- `ErrTxPoolOverflow`
- `ErrFutureReplacePending`
Notably, InsufficientBalance is also treated as a permanent error, as
it’s highly unlikely that users will transfer funds into the sender account
after submitting the transaction. Otherwise, users may be confused—seeing
their transaction submitted but unaware that the sender lacks sufficient funds—and
continue waiting for it to be included.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
closes#31310
This has been requested a few times in the past and I think it is a nice
quality-of-life improvement for users. At a predetermined interval,
there will now be a "Fork ready" log when a future fork is scheduled,
but not yet active.
It can only possibly print after block import, which kinda avoids the
scenario where the client isn't progressing or is syncing and the user
thinks it's "ready" because it sees a ready log.
New output:
```console
INFO [03-08|21:32:57.472] Imported new potential chain segment number=7 hash=aa24ee..f09e62 blocks=1 txs=0 mgas=0.000 elapsed="874.916µs" mgasps=0.000 snapdiffs=973.00B triediffs=7.05KiB triedirty=0.00B
INFO [03-08|21:32:57.473] Ready for fork activation fork=Prague date="18 Mar 25 19:29 CET" remaining=237h57m0s timestamp=1,742,322,597
INFO [03-08|21:32:57.475] Chain head was updated number=7 hash=aa24ee..f09e62 root=19b0de..8d32f2 elapsed="129.125µs"
```
Easiest way to verify this behavior is to apply this patch and run `geth
--dev --dev.period=12`
```patch
diff --git a/params/config.go b/params/config.go
index 9c7719d901..030c4f80e7 100644
--- a/params/config.go
+++ b/params/config.go
@@ -174,7 +174,7 @@ var (
ShanghaiTime: newUint64(0),
CancunTime: newUint64(0),
TerminalTotalDifficulty: big.NewInt(0),
- PragueTime: newUint64(0),
+ PragueTime: newUint64(uint64(time.Now().Add(time.Hour * 300).Unix())),
BlobScheduleConfig: &BlobScheduleConfig{
Cancun: DefaultCancunBlobConfig,
Prague: DefaultPragueBlobConfig,
```
This is an attempt at fixing #31601. I think what happens is the startup
logic will try to get the full block body (it's `bc.loadLastState`) and
fail because genesis block has been pruned from the freezer. This will
cause it to keep repeating the reset logic, causing a deadlock.
This can happen when due to an unsuccessful sync we don't have the state
for the head (or any other state) fully, and try to redo the snap sync.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This fixes an issue where running geth with `--history.chain postmerge`
would not work on an empty database.
```
ERROR[04-16|23:11:12.913] Chain history database is pruned to unknown block tail=0
Fatal: Failed to register the Ethereum service: unexpected database tail
```
This PR fixes a bug in the map renderer that sometimes used an obsolete
block log value pointer to initialize the iterator for rendering from a
snapshot. This bug was triggered by chain reorgs and sometimes caused
indexing errors and invalid search results. A few other conditions are
also made safer that were not reported to cause issues yet but could
potentially be unsafe in some corner cases. A new unit test is also
added that reproduced the bug but passes with the new fixes.
Fixes https://github.com/ethereum/go-ethereum/issues/31593
Might also fix https://github.com/ethereum/go-ethereum/issues/31589
though this issue has not been reproduced yet, but it appears to be
related to a log index database corruption around a specific block,
similarly to the other issue.
Note that running this branch resets and regenerates the log index
database. For this purpose a `Version` field has been added to
`rawdb.FilterMapsRange` which will also make this easier in the future
if a breaking database change is needed or the existing one is
considered potentially broken due to a bug, like in this case.
I added the history mode configuration in eth/ethconfig initially, since
it seemed like the logical place. But it turns out we need access to the
intended pruning setting at a deeper level, and it actually needs to be
integrated with the blockchain startup procedure.
With this change applied, if a node previously had its history pruned,
and is subsequently restarted **without** the `--history.chain
postmerge` flag, the `BlockChain` initialization code will now verify
the freezer tail against the known pruning point of the predefined
network and will restore pruning status. Note that this logic is quite
restrictive, we allow non-zero tail only for known networks, and only
for the specific pruning point that is defined.
This PR proposes a change to the authorizations' validation introduced
in commit cdb66c8. These changes make the expected behavior independent
of the order of admission of authorizations, improving the
predictability of the resulting state and the usability of the system
with it.
The current implementation behavior is dependent on the transaction
submission order: This issue is related to authorities and the sender of
a transaction, and can be reproduced respecting the normal nonce rules.
The issue can be reproduced by the two following cases:
**First case**
- Given an empty pool.
- Submit transaction `{ from: B, auths [ A ] }`: is accepted.
- Submit transaction `{ from: A }`: Is accepted: it becomes the one
in-flight transaction allowed.
**Second case**
- Given an empty pool.
- Submit transaction `{ from: A }`: is accepted
- Submit transaction `{ from: B, auths [ A ] }`: is rejected since there
is already a queued/pending transaction from A.
The expected behavior is that both sequences of events would lead to the
same sets of accepted and rejected transactions.
**Proposed changes**
The queued/pending transactions issued from any authority of the
transaction being validated have to be counted, allowing one transaction
from accounts submitting an authorization.
- Notice that the expected behavior was explicitly forbidden in the case
"reject-delegation-from-pending-account", I believe that this behavior
conflicts to the definition of the limitation, and it is removed in this
PR. The expected behavior is tested in
"accept-authorization-from-sender-of-one-inflight-tx".
- Replacement tests have been separated to improve readability of the
acceptance test.
- The test "allow-more-than-one-tx-from-replaced-authority" has been
extended with one extra transaction, since the system would always have
accepted one transaction (but not two).
- The test "accept-one-inflight-tx-of-delegated-account" is extended to
clean-up state, avoiding leaking the delegation used into the other
tests. Additionally, replacement check is removed to be tested in its
own test case.
**Expected behavior**
The expected behavior of the authorizations' validation shall be as
follows:
Notice that replacement shall be allowed, and behavior shall remain
coherent with the table, according to the replaced transaction.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
During my benchmarks on Holesky, around 10% of all CPU time was spent in
PUSH2
```
ROUTINE ======================== github.com/ethereum/go-ethereum/core/vm.newFrontierInstructionSet.makePush.func1 in github.com/ethereum/go-ethereum/core/vm/instructions.go
16.38s 20.35s (flat, cum) 10.31% of Total
740ms 740ms 976: return func(pc *uint64, interpreter *EVMInterpreter, scope *ScopeContext) ([]byte, error) {
. . 977: var (
40ms 40ms 978: codeLen = len(scope.Contract.Code)
970ms 970ms 979: start = min(codeLen, int(*pc+1))
200ms 200ms 980: end = min(codeLen, start+pushByteSize)
. . 981: )
670ms 2.39s 982: a := new(uint256.Int).SetBytes(scope.Contract.Code[start:end])
. . 983:
. . 984: // Missing bytes: pushByteSize - len(pushData)
410ms 410ms 985: if missing := pushByteSize - (end - start); missing > 0 {
. . 986: a.Lsh(a, uint(8*missing))
. . 987: }
12.69s 14.94s 988: scope.Stack.push2(*a)
10ms 10ms 989: *pc += size
650ms 650ms 990: return nil, nil
. . 991: }
. . 992:}
```
Which is quite crazy. We have a handwritten encoder for PUSH1 already,
this PR adds one for PUSH2.
PUSH2 is the second most used opcode as shown here:
https://gist.github.com/shemnon/fb9b292a103abb02d98d64df6fbd35c8 since
it is used by solidity quite significantly. Its used ~20 times as much
as PUSH20 and PUSH32.
# Benchmarks
```
BenchmarkPush/makePush-14 94196547 12.27 ns/op 0 B/op 0 allocs/op
BenchmarkPush/push-14 429976924 2.829 ns/op 0 B/op 0 allocs/op
```
---------
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This pull request introduces two constraints in the blobPool:
(a) If the sender has a pending authorization or delegation, only one
in-flight
executable transaction can be cached.
(b) If the authority address in a SetCode transaction is already
reserved by
the blobPool, the transaction will be rejected.
These constraints mitigate an attack where an attacker spams the pool
with
numerous blob transactions, evicts other transactions, and then cancels
all
pending blob transactions by draining the sender’s funds if they have a
delegation.
Note, because there is no exclusive lock held between different subpools
when processing transactions, it's totally possible the SetCode
transaction
and blob transactions with conflict sender and authorities are accepted
simultaneously. I think it's acceptable as it's very hard to be
exploited.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This pull request introduces new sync logic for pruning mode. The downloader will now skip
insertion of block bodies and receipts before the configured history cutoff point.
Originally, in snap sync, the header chain and other components (bodies and receipts) were
inserted separately. However, in Proof-of-Stake, this separation is unnecessary since the
sync target is already verified by the CL.
To simplify the process, this pull request modifies `InsertReceiptChain` to insert headers
along with block bodies and receipts together. Besides, `InsertReceiptChain` doesn't have
the notion of reorg, as the common ancestor is always be found before the sync and extra
side chain is truncated at the beginning if they fall in the ancient store. The stale
canonical chain flags will always be rewritten by the new chain. Explicit reorg logic is
no longer required in `InsertReceiptChain`.
This is an alternative to #31309
With eth/68, transaction announcement must have transaction type and
size. So in announceTransactions, we need to query the transaction from
transaction pool with its hash. This creates overhead in case of blob
transaction which needs to load data from billy and RLP decode. This
commit creates a lightweight lookup from transaction hash to transaction
size and a function GetMetadata to query transaction type and
transaction size given the transaction hash.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR adds `rawdb.SafeDeleteRange` and uses it for range deletion in
`core/filtermaps`. This includes deleting the old bloombits database,
resetting the log index database and removing index data for unindexed
tail epochs (which previously weren't properly implemented for the
fallback case).
`SafeDeleteRange` either calls `ethdb.DeleteRange` if the node uses the
new path based state scheme or uses an iterator based fallback method
that safely skips trie nodes in the range if the old hash based state
scheme is used. Note that `ethdb.DeleteRange` also has its own iterator
based fallback implementation in `ethdb/leveldb`. If a path based state
scheme is used and the backing db is pebble (as it is on the majority of
new nodes) then `rawdb.SafeDeleteRange` uses the fast native range
delete.
Also note that `rawdb.SafeDeleteRange` has different semantics from
`ethdb.DeleteRange`, it does not automatically return if the operation
takes a long time. Instead it receives a `stopCallback` that can
interrupt the process if necessary. This is because in the safe mode
potentially a lot of entries are iterated without being deleted (this is
definitely the case when deleting the old bloombits database which has a
single byte prefix) and therefore restarting the process every time a
fixed number of entries have been iterated would result in a quadratic
run time in the number of skipped entries.
When running in safe mode, unindexing an epoch takes about a second,
removing bloombits takes around 10s while resetting a full log index
might take a few minutes. If a range delete operation takes a
significant amount of time then log messages are printed. Also, any
range delete operation can be interrupted by shutdown (tail uinindexing
can also be interrupted by head indexing, similarly to how tail indexing
works). If the last unindexed epoch might have "dirty" index data left
then the indexed map range points to the first valid epoch and
`cleanedEpochsBefore` points to the previous, potentially dirty one. At
startup it is always assumed that the epoch before the first fully
indexed one might be dirty. New tail maps are never rendered and also no
further maps are unindexed before the previous unindexing is properly
cleaned up.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR changes log indexer error handling so that if an indexing error
happens then it disables the indexer and reverts to unindexed more
without resetting the database (except in case of a failed database
init).
Resetting the database on the first error would probably be overkill as
a client update might fix this without having to reindex the entire
history. It would also make debugging very hard. On the other hand,
these errors do not resolve themselves automatically so constantly
retrying makes no sense either. With these changes a new attempt to
resume indexing is made every time the client is restarted.
The PR also fixes https://github.com/ethereum/go-ethereum/issues/31491
which originated from the tail indexer trying to resume processing a
failed map renderer.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR adds an extra condition to the log indexer initialization in
order to avoid initializing with block 0 as target head. Previously this
caused the indexer to initialize without a checkpoint. Later, when the
real chain head was set, it indexed the entire history, then unindexed
most of it if only the recent history was supposed to be indexed. Now
the init only happens when there is an actual synced chain head and
therefore the index is initialized at the most recent checkpoint and
only the last year is indexed according to the default parameters.
During checkpoint initialization the best available checkpoint is also
checked against the history cutoff point and fails if the indexing would
have to start from a block older than the cutoff. If initialization
fails then the indexer reverts to unindexed mode instead of retrying
because the the failure conditions cannot be expected to recover later.
Instead of reporting all filtermaps stuff in one line, I'm breaking it
down into the three separate kinds of entries here.
```
+-----------------------+-----------------------------+------------+------------+
| DATABASE | CATEGORY | SIZE | ITEMS |
+-----------------------+-----------------------------+------------+------------+
| Key-Value store | Log index filter-map rows | 59.21 GiB | 616077345 |
| Key-Value store | Log index last-block-of-map | 12.35 MiB | 269755 |
| Key-Value store | Log index block-lv | 421.70 MiB | 22109169 |
```
Also added some other changes to make it easier to debug:
- restored bloombits into the inspect output, so we notice if it doesn't
get deleted for some reason
- tracking of unaccounted key examples
This adds a new subcommand 'geth prune-history' that removes the pre-merge history
on supported networks. Geth is not fully ready to work in this mode, please do not run
this command on your production node.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In #31384 we unindex TXes prior to the merge block. However when the
node starts up it will try to re-index those back if the config is to index the
whole chain. This change makes the indexer aware of the history cutoff block,
avoiding reindexing in that segment.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This pull request improves the protection mechanism in the txpool for
senders with delegation. A sender with either delegation or pending
delegation is now limited to a maximum of one in-flight executable
transaction, while gapped transactions will be rejected.
Reason:
If nonce-gapped transaction from delegated/pending-delegated senders
can be acceptable, then it's no-longer possible to send another
"executable" transaction with correct nonce due to the policy of at most
one inflight tx. The gapped transaction will be stuck in the txpool, with no
meaningful way to unlock the sender.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR changes the matcher syncing conditions so that it is possible to
run a search while head indexing is in progress. Previously it was a
requirement to have the head indexed in order to perform matcher sync
before and after a search. This was unnecessarily strict as the purpose
was just to avoid syncing the valid range with the temporary shortened
indexed range applied while updating existing head maps. Now the sync
condition explicitly checks whether the indexer has a temporary indexed
range with some head maps being partially updated.
It also fixes a deadlock that happened when matcher synchronization was
attempted in the event handler called from the `writeFinishedMaps`
periodical callback.
This removes the signer type-train in favor of defining a single object
that can handle all tx types. Supported types are enabled via a map.
Notably, the new signer also supports disabling legacy transactions.
This PR roughly halves the number of allocations needed to compute the
sigHash for a transaction.
This sigHash is used whenever we recover a signature of a transaction,
so quite often. During a recent benchmark full syncing on Holesky,
roughly 2.8% of all allocations were happening here because the fields
from the transaction would be copied multiple times.
```
66168733 153175654 (flat, cum) 2.80% of Total
. . 368:func (s londonSigner) Hash(tx *Transaction) common.Hash {
. . 369: if tx.Type() != DynamicFeeTxType {
. . 370: return s.eip2930Signer.Hash(tx)
. . 371: }
. 19169966 372: return prefixedRlpHash(
. . 373: tx.Type(),
26442187 26442187 374: []interface{}{
. . 375: s.chainId,
6848616 6848616 376: tx.Nonce(),
. 19694077 377: tx.GasTipCap(),
. 18956774 378: tx.GasFeeCap(),
6357089 6357089 379: tx.Gas(),
. 12321050 380: tx.To(),
. 16865054 381: tx.Value(),
13435187 13435187 382: tx.Data(),
13085654 13085654 383: tx.AccessList(),
. . 384: })
. . 385:}
```
This PR reduces the allocations and speeds up the computation of the
sigHash by ~22%, which is quite significantly given that this operation
involves a call to Keccak
```
// BenchmarkHash-8 440082 2639 ns/op 384 B/op 13 allocs/op
// BenchmarkHash-8 493566 2033 ns/op 240 B/op 6 allocs/op
```
```
Hash-8 2.691µ ± 8% 2.097µ ± 9% -22.07% (p=0.000 n=10)
```
It also kinda cleans up stuff in my opinion, since the transaction
should itself know best how to compute the sighash
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Currently, when answering GetPooledTransaction request, txpool.Get() is
used. When the requested hash is blob transaction, blobpool.Get() is
called. This function loads the RLP-encoded transaction from limbo then
decodes and returns. Later, in answerGetPooledTransactions, we need to
RLP encode again. This decode then encode is wasteful. This commit adds
GetRLP to transaction pool interface so that answerGetPooledTransactions
can use the RLP-encoded from limbo directly.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR fixes a bug in the `lastMapBoundaryBefore` logic that resulted
in incorrect checkpoint initialization (started rendering from the
previous epoch boundary which caused the `needTailEpoch` check to fail).
Apparently the bug was present before but went unnoticed because
`needTailEpoch` behaved differently.
Fixes https://github.com/ethereum/go-ethereum/issues/31413
Here we add the notion of prunable tables for the `TruncateTail` operation
in the freezer. TruncateTail for the chain freezer now only truncates the body and
receipts tables, leaving headers and hashes as-is.
This change also requires changing the validation/repair at startup to allow for
tables with different tail. For the header and hash tables, we now require them to start
at number zero.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR moves the updating of the `blockProcFeed` event feed from
`InsertChain` to `insertChain` in order to ensure that the feed
subscribers are notified whenever block processing happens.
Note that this event is not subscribed to anywhere in our codebase at
the moment, earlier it was used by the LES server to avoid slowing down
block processing. Now I want to do the same with the log indexer, the
problem is that back then every block insertion was done by
`InsertChain`, now the regular payload insertion is done by
`InsertBlockWithoutSetHead`. Both of these (and also `SetCanonical` if
needed) calls `insertChain` so I moved the feed update there.
Here I am adding a config option and geth flag (`--history.chain`) for
configuring history pruning. There are two options available:
- `--history.chain all` is the default and will keep all history like
before.
- `--history.chain postmerge` will configure the history cutoff point to
the merge block.
The option doesn't actually do anything right now, but we need it as a
precursor for other history pruning changes.
This error log in `legacypool.go` isn't necessary, since even though the
behavior is unexpected, it is handled correctly. A discussion on issue
#22301 concluded that this should instead be a warning log.
1. The metric of preimage/hits are always the same as preimage/total, prefer to replace
the hits with miss instead.
2. For the state/read/accounts metric, follow the same naming of others,
change into singuar.
This resolves a situation on the Sepolia testnet, which has a different
deposit contract. The contract on that network emits two kinds of logs,
instead of only deposit events like the deposit contract on mainnet. So
we need to skip events with mismatched topics.
This ensures that if we receive a blob transaction announcement where we cannot
link the tx to the sidecar commitments, we will drop the sending peer. This check
is added in the protocol handler for the PooledTransactions message.
Tests for this have also been added in the cross-client "eth" protocol test suite.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
In transaction-sending APIs such as `eth_sendRawTransaction`, a submitted transaction
failing the configured txpool validation rules (i.e. fee too low) would cause an error to be
returned, even though the transaction was successfully added into the locals tracker.
Once added there, the transaction may even be included into the chain at a later time,
when fee market conditions change.
This change improves on this by performing the validation in the locals tracker, basically
skipping some of the validation rules for local transactions. We still try to add the tx to the
main pool immediately, but an error will only be returned for transactions which are
fundamentally invalid.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This is a not-particularly-important "cleanliness" PR. It removes the
last remnants of the `x/exp` package, where we used the `maps.Keys`
function.
The original returned the keys in a slice, but when it became 'native'
the signature changed to return an iterator, so the new idiom is
`slices.Collect(maps.Keys(theMap))`, unless of course the raw iterator
can be used instead.
In some cases, where we previously collect into slice and then sort, we
can now instead do `slices.SortXX` on the iterator instead, making the
code a bit more concise.
This PR might be _slighly_ less optimal, because the original `x/exp`
implementation allocated the slice at the correct size off the bat,
which I suppose the new code won't.
Putting it up for discussion.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
when remove an non-SetCodeTxType transaction, error logs flood
```
t=2025-02-25T03:11:06+0000 lvl=error msg="Authority with untracked tx" addr=0xD5bf9221fCB1C31Cd1EE477a60c148d40dD63DC1 hash=0x626fdf205a5b1619deb2f9e51fed567353f80acbd522265b455daa0821c571d9
```
in this PR, only try to removeAuthorities for txs with SetCodeTxType
in addition, the performance of removeAuthorities improved a lot,
because no need range all `t.auths` now.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
In this PR, several improvements have been made:
Authorization-related validations have been moved to legacyPool.
Previously, these checks were part of the standard validation procedure,
which applies common validations across different pools. Since these
checks are specific to SetCode transactions, relocating them to
legacyPool
is a more reasonable choice.
Additionally, authorization conflict checks are now performed regardless
of whether the transaction is a replacement or not.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
I maintain an improved version of the go-ethereum assembler at
https://github.com/fjl/geas. We don't really use core/asm in our tests,
and it has some bugs that prevent it from being useful, so I'm removing
the package.
This PR does a few things including:
- Remove `ContractRef` interface
- Remove `vm.AccountRef` which implements `ContractRef` interface
- Maintain the `jumpDests` struct in EVM for sharing between call frames
- Simplify the delegateCall context initialization
Currently, when calculating block's bloom, we loop through all the
receipt logs to calculate the hash value. However, normally, after going
through applyTransaction, the receipt's bloom is already calculated
based on the receipt log, so the block's bloom can be calculated by just
ORing these receipt's blooms.
```
goos: darwin
goarch: arm64
pkg: github.com/ethereum/go-ethereum/core/types
cpu: Apple M1 Pro
BenchmarkCreateBloom
BenchmarkCreateBloom/small
BenchmarkCreateBloom/small-10 810922 1481 ns/op 104 B/op 5 allocs/op
BenchmarkCreateBloom/large
BenchmarkCreateBloom/large-10 8173 143764 ns/op 9614 B/op 401 allocs/op
BenchmarkCreateBloom/small-mergebloom
BenchmarkCreateBloom/small-mergebloom-10 5178918 232.0 ns/op 0 B/op 0 allocs/op
BenchmarkCreateBloom/large-mergebloom
BenchmarkCreateBloom/large-mergebloom-10 54110 22207 ns/op 0 B/op 0 allocs/op
```
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
This PR addresses a flaw in the freezer table upgrade path.
In v1.15.0, freezer table v2 was introduced, including an additional
field (`flushOffset`) maintained in the metadata file. To ensure
backward compatibility, an upgrade path was implemented for legacy
freezer tables by setting `flushOffset` to the size of the index file.
However, if the freezer table is opened in read-only mode, this file
write operation is rejected, causing Geth to shut down entirely.
Given that invalid items in the freezer index file can be detected and
truncated, all items in freezer v0 index files are guaranteed to be
complete. Therefore, when operating in read-only mode, it is safe to
use the freezer data without performing an upgrade.
This is to prevent a crash on startup with a custom genesis configuration.
With this change in place, upgrading a chain created by geth v1.14.x and
below will now print an error instead of crashing:
Fatal: Failed to register the Ethereum service: invalid chain configuration: missing entry for fork "cancun" in blobSchedule
Arguably this is not great, and it should just auto-upgrade the config.
We'll address this in a follow-up PR for geth v1.15.2
The new SetCode transaction type introduces some additional complexity
when handling the transaction pool.
This complexity stems from two new account behaviors:
1. The balance and nonce of an account can change during regular
transaction execution *when they have a deployed delegation*.
2. The nonce and code of an account can change without any EVM execution
at all. This is the "set code" mechanism introduced by EIP-7702.
The first issue has already been considered extensively during the design
of ERC-4337, and we're relatively confident in the solution of simply
limiting the number of in-flight pending transactions an account can have
to one. This puts a reasonable bound on transaction cancellation. Normally
to cancel, you would need to spend 21,000 gas. Now it's possible to cancel
for around the cost of warming the account and sending value
(`2,600+9,000=11,600`). So 50% cheaper.
The second issue is more novel and needs further consideration.
Since authorizations are not bound to a specific transaction, we
cannot drop transactions with conflicting authorizations. Otherwise,
it might be possible to cherry-pick authorizations from txs and front
run them with different txs at much lower fee amounts, effectively DoSing
the authority. Fortunately, conflicting authorizations do not affect the
underlying validity of the transaction so we can just accept both.
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
Agreed to the following fork dates for Holesky and Sepolia on ACDC 150
Holesky slot: 3710976 (Mon, Feb 24 at 21:55:12 UTC)
Sepolia slot: 7118848 (Wed, Mar 5 at 07:29:36 UTC)
Here we add some more changes for live tracing API v1.1:
- Hook `OnSystemCallStartV2` was introduced with `VMContext` as parameter.
- Hook `OnBlockHashRead` was introduced.
- `GetCodeHash` was added to the state interface
- The new `WrapWithJournal` construction helps with tracking EVM reverts in the tracer.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR changes the signature of `CalcExcessBlobGas` to take in just
the header timestamp instead of the whole object. It also adds a sanity
check for the parent->child block order to `VerifyEIP4844Header`.
A clarification was made to EIP-7691 stating that at the fork boundary
it is required to use the target blob count associated with the head
block, rather than the parent as implemented here.
See for more: https://github.com/ethereum/EIPs/pull/9249
Replaces #29297, descendant from #27535
---------
This PR removes `locals` as a concept from transaction pools. Therefore,
the pool now acts as very a good simulation/approximation of how our
peers' pools behave. What this PR does instead, is implement a
locals-tracker, which basically is a little thing which, from time to
time, asks the pool "did you forget this transaction?". If it did, the
tracker resubmits it.
If the txpool _had_ forgotten it, chances are that the peers had also
forgotten it. It will be propagated again.
Doing this change means that we can simplify the pool internals, quite a
lot.
### The semantics of `local`
Historically, there has been two features, or usecases, that has been
combined into the concept of `locals`.
1. "I want my local node to remember this transaction indefinitely, and
resubmit to the network occasionally"
2. "I want this (valid) transaction included to be top-prio for my
miner"
This PR splits these features up, let's call it `1: local` and `2:
prio`. The `prio` is not actually individual transaction, but rather a
set of `address`es to prioritize.
The attribute `local` means it will be tracked, and `prio` means it will
be prioritized by miner.
For `local`: anything transaction received via the RPC is marked as
`local`, and tracked by the tracker.
For `prio`: any transactions from this sender is included first, when
building a block. The existing commandline-flag `--txpool.locals` sets
the set of `prio` addresses.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
I hit this case while trying something with the simulated backend. The
EVM only enables instruction set forks after the merge when 'Random' is
set. In the simulated backend, the random value will be set via the
engine API for all blocks after genesis. But for the genesis block
itself, the random value will not be assigned in the vm.BlockContext
because the genesis has a non-zero difficulty. For my case, this meant
that estimateGas did not work for the first transaction sent on the
simulated chain, since the contract contained a PUSH0 instruction.
This could also be fixed by explicitly configuring a zero difficulty in
the simulated backend. However, I think that zero difficulty is a better
default these days.
---------
Co-authored-by: lightclient <lightclient@protonmail.com>
This PR defines the Osaka fork. An easy first step to start our work on
the next hardfork
(This is needed for EOF testing as well)
---------
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
This is a follow-up PR to #29792 to get rid of the data file sync.
**This is a non-backward compatible change, which increments the
database version from 8 to 9**.
We introduce a flushOffset for each freezer table, which tracks the position
of the most recently fsync’d item in the index file. When this offset moves
forward, it indicates that all index entries below it, along with their corresponding
data items, have been properly persisted to disk. The offset can also be moved
backward when truncating from either the head or tail of the file.
Previously, the data file required an explicit fsync after every mutation, which
was highly inefficient. With the introduction of the flush offset, the synchronization
strategy becomes more flexible, allowing the freezer to sync every 30 seconds
instead.
The data items above the flush offset are regarded volatile and callers must ensure
they are recoverable after the unclean shutdown, or explicitly sync the freezer
before any proceeding operations.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR builds on #29040 and updates it to the new version of the spec.
I filled the EEST tests and they pass.
Link to spec: https://eips.ethereum.org/EIPS/eip-7623
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
Same as #31015 but requires the contract to exist. Not compatible with
any verkle testnet up to now.
This adds a `isSytemCall` flag so that it is possible to detect when a
system call is executed, so that the code execution and other locations
are not added to the witness.
---------
Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
The total difficulty is the sum of all block difficulties from genesis
to a certain block. This value was used in PoW for deciding which chain
is heavier, and thus which chain to select. Since PoS has a different
fork selection algorithm, all blocks since the merge have a difficulty
of 0, and all total difficulties are the same for the past 2 years.
Whilst the TDs are mostly useless nowadays, there was never really a
reason to mess around removing them since they are so tiny. This
reasoning changes when we go down the path of pruned chain history. In
order to reconstruct any TD, we **must** retrieve all the headers from
chain head to genesis and then iterate all the difficulties to compute
the TD.
In a world where we completely prune past chain segments (bodies,
receipts, headers), it is not possible to reconstruct the TD at all. In
a world where we still keep chain headers and prune only the rest,
reconstructing it possible as long as we process (or download) the chain
forward from genesis, but trying to snap sync the head first and
backfill later hits the same issue, the TD becomes impossible to
calculate until genesis is backfilled.
All in all, the TD is a messy out-of-state, out-of-consensus computed
field that is overall useless nowadays, but code relying on it forces
the client into certain modes of operation and prevents other modes or
other optimizations. This PR completely nukes out the TD from the node.
It doesn't compute it, it doesn't operate on it, it's as if it didn't
even exist.
Caveats:
- Whenever we have APIs that return TD (devp2p handshake, tracer, etc.)
we return a TD of 0.
- For era files, we recompute the TD during export time (fairly quick)
to retain the format content.
- It is not possible to "verify" the merge point (i.e. with TD gone, TTD
is useless). Since we're not verifying PoW any more, just blindly trust
it, not verifying but blindly trusting the many year old merge point
seems just the same trust model.
- Our tests still need to be able to generate pre and post merge blocks,
so they need a new way to split the merge without TTD. The PR introduces
a settable ttdBlock field on the consensus object which is used by tests
as the block where originally the TTD happened. This is not needed for
live nodes, we never want to generate old blocks.
- One merge transition consensus test was disabled. With a
non-operational TD, testing how the client reacts to TTD is useless, it
cannot react.
Questions:
- Should we also drop total terminal difficulty from the genesis json?
It's a number we cannot react on any more, so maybe it would be cleaner
to get rid of even more concepts.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
As part of trying to make the inputs and outputs of the evm subcommands
more streamlined and aligned, this PR modifies how `evm t8n` manages
output-files.
Previously, we do a kind of wonky thing where between each transaction,
we invoke a `getTracer` closure. In that closure, we create a new
output-file, a tracer, and then make the tracer stream output to the
file. We also fiddle a bit to ensure that the file becomes properly
closed.
It is a kind of hacky solution we have in place. This PR changes it, so
that from the execution-pipeline point of view, we have just a regular
tracer. No fiddling with re-setting it or closing files.
That particular tracer, however, is a bit special: it takes care of
creating new files per transaction (in the tx-start-hook) and closing
(on tx-end-hook). Also instantiating the right type of underlying
tracer, which can be a json-logger or a custom tracer.
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
