rebased https://github.com/ethereum/go-ethereum/pull/29766 . The
downstream branch appears to have been deleted and I don't have perms to
push to that fork.
`TerminalTotalDifficultyPassed` is removed. `TerminalTotalDifficulty`
must now be non-nil, and it is expected that networks are already
merged: we can only import PoW/Clique chains, not produce blocks on
them.
---------
Co-authored-by: stevemilk <wangpeculiar@gmail.com>
Changelog: https://golangci-lint.run/product/changelog/#1610
Removes `exportloopref` (no longer needed), replaces it with
`copyloopvar` which is basically the opposite.
Also adds:
- `durationcheck`
- `gocheckcompilerdirectives`
- `reassign`
- `mirror`
- `tenv`
---------
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
This PR integrates witness-enabled block production, witness-creating
payload execution and stateless cross-validation into the `engine` API.
The purpose of the PR is to enable the following use-cases (for API
details, please see next section):
- Cross validating locally created blocks:
- Call `forkchoiceUpdatedWithWitness` instead of `forkchoiceUpdated` to
trigger witness creation too.
- Call `getPayload` as before to retrieve the new block and also the
above created witness.
- Call `executeStatelessPayload` against another client to
cross-validate the block.
- Cross validating locally processed blocks:
- Call `newPayloadWithWitness` instead of `newPayload` to trigger
witness creation too.
- Call `executeStatelessPayload` against another client to
cross-validate the block.
- Block production for stateless clients (local or MEV builders):
- Call `forkchoiceUpdatedWithWitness` instead of `forkchoiceUpdated` to
trigger witness creation too.
- Call `getPayload` as before to retrieve the new block and also the
above created witness.
- Propagate witnesses across the consensus libp2p network for stateless
Ethereum.
- Stateless validator validation:
- Call `executeStatelessPayload` with the propagated witness to
statelessly validate the block.
*Note, the various `WithWitness` methods could also *just be* an
additional boolean flag on the base methods, but this PR wanted to keep
the methods separate until a final consensus is reached on how to
integrate in production.*
---
The following `engine` API types are introduced:
```go
// StatelessPayloadStatusV1 is the result of a stateless payload execution.
type StatelessPayloadStatusV1 struct {
Status string `json:"status"`
StateRoot common.Hash `json:"stateRoot"`
ReceiptsRoot common.Hash `json:"receiptsRoot"`
ValidationError *string `json:"validationError"`
}
```
- Add `forkchoiceUpdatedWithWitnessV1,2,3` with same params and returns
as `forkchoiceUpdatedV1,2,3`, but triggering a stateless witness
building if block production is requested.
- Extend `getPayloadV2,3` to return `executionPayloadEnvelope` with an
additional `witness` field of type `bytes` iff created via
`forkchoiceUpdatedWithWitnessV2,3`.
- Add `newPayloadWithWitnessV1,2,3,4` with same params and returns as
`newPayloadV1,2,3,4`, but triggering a stateless witness creation during
payload execution to allow cross validating it.
- Extend `payloadStatusV1` with a `witness` field of type `bytes` if
returned by `newPayloadWithWitnessV1,2,3,4`.
- Add `executeStatelessPayloadV1,2,3,4` with same base params as
`newPayloadV1,2,3,4` and one more additional param (`witness`) of type
`bytes`. The method returns `statelessPayloadStatusV1`, which mirrors
`payloadStatusV1` but replaces `latestValidHash` with `stateRoot` and
`receiptRoot`.
This pull request introduces a state.Reader interface for state
accessing.
The interface could be implemented in various ways. It can be pure trie
only reader, or the combination of trie and state snapshot. What's more,
this interface allows us to have more flexibility in the future, e.g.
the
archive reader (for accessing archive state).
Additionally, this pull request removes the following metrics
- `chain/snapshot/account/reads`
- `chain/snapshot/storage/reads`
* cmd/geth, ethdb/pebble: polish method naming and code comment
* implement db stat for pebble
* cmd, core, ethdb, internal, trie: remove db property selector
* cmd, core, ethdb: fix function description
---------
Co-authored-by: prpeh <prpeh@proton.me>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Here we add a Go API for running tracing plugins within the main block import process.
As an advanced user of geth, you can now create a Go file in eth/tracers/live/, and within
that file register your custom tracer implementation. Then recompile geth and select your tracer
on the command line. Hooks defined in the tracer will run whenever a block is processed.
The hook system is defined in package core/tracing. It uses a struct with callbacks, instead of
requiring an interface, for several reasons:
- We plan to keep this API stable long-term. The core/tracing hook API does not depend on
on deep geth internals.
- There are a lot of hooks, and tracers will only need some of them. Using a struct allows you
to implement only the hooks you want to actually use.
All existing tracers in eth/tracers/native have been rewritten to use the new hook system.
This change breaks compatibility with the vm.EVMLogger interface that we used to have.
If you are a user of vm.EVMLogger, please migrate to core/tracing, and sorry for breaking
your stuff. But we just couldn't have both the old and new tracing APIs coexist in the EVM.
---------
Co-authored-by: Matthieu Vachon <matthieu.o.vachon@gmail.com>
Co-authored-by: Delweng <delweng@gmail.com>
Co-authored-by: Martin HS <martin@swende.se>
This pull request introduces a database tool for inspecting the state history.
It can be used for either account history or storage slot history, within a
specific block range.
The state output format can be chosen either with
- the "rlp-encoded" values (those inserted into the merkle trie)
- the "rlp-decoded" value (the raw state value)
The latter one needs --raw flag.
This adds support for the Deneb beacon chain fork, and fork handling
in general, to the beacon chain light client implementation.
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Here we add a beacon chain light client for use by geth.
Geth can now be configured to run against a beacon chain API endpoint,
without pointing a CL to it. To set this up, use the `--beacon.api` flag. Information
provided by the beacon chain is verified, i.e. geth does not blindly trust the beacon
API endpoint in this mode. The root of trust are the beacon chain 'sync committees'.
The configured beacon API endpoint must provide light client data. At this time, only
Lodestar and Nimbus provide the necessary APIs.
There is also a standalone tool, cmd/blsync, which uses the beacon chain light client
to drive any EL implementation via its engine API.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
* miner: untangle miner
* miner: use common.hash instead of *types.header
* cmd/geth: deprecate --mine
* eth: get rid of most miner api
* console: get rid of coinbase in welcome message
* miner/stress: get rid of the miner stress test
* eth: get rid of miner.setEtherbase
* ethstats: remove miner and hashrate flags
* ethstats: remove miner and hashrate flags
* cmd: rename pendingBlockProducer to miner.pending.feeRecipient flag
* miner: use pendingFeeRecipient instead of etherbase
* miner: add mutex to protect the pending block
* miner: add mutex to protect the pending block
* eth: get rid of etherbase mentions
* miner: no need to lock the coinbase
* eth, miner: fix linter
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* eth: drop support for forward sync triggers and head block packets
* consensus, eth: enforce always merged network
* eth: fix tx looper startup and shutdown
* cmd, core: fix some tests
* core: remove notion of future blocks
* core, eth: drop unused methods and types
* all: implement era format, add history importer/export
* internal/era/e2store: refactor e2store to provide ReadAt interface
* internal/era/e2store: export HeaderSize
* internal/era: refactor era to use ReadAt interface
* internal/era: elevate anonymous func to named
* cmd/utils: don't store entire era file in-memory during import / export
* internal/era: better abstraction between era and e2store
* cmd/era: properly close era files
* cmd/era: don't let defers stack
* cmd/geth: add description for import-history
* cmd/utils: better bytes buffer
* internal/era: error if accumulator has more records than max allowed
* internal/era: better doc comment
* internal/era/e2store: rm superfluous reader, rm superfluous testcases, add fuzzer
* internal/era: avoid some repetition
* internal/era: simplify clauses
* internal/era: unexport things
* internal/era,cmd/utils,cmd/era: change to iterator interface for reading era entries
* cmd/utils: better defer handling in history test
* internal/era,cmd: add number method to era iterator to get the current block number
* internal/era/e2store: avoid double allocation during write
* internal/era,cmd/utils: fix lint issues
* internal/era: add ReaderAt func so entry value can be read lazily
Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* internal/era: improve iterator interface
* internal/era: fix rlp decode of header and correctly read total difficulty
* cmd/era: fix rebase errors
* cmd/era: clearer comments
* cmd,internal: fix comment typos
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
When managing geth, it is sometimes desirable to do a partial wipe; deleting state but retaining freezer data. A partial wipe can be somewhat tricky to accomplish.
This change implements the ability to perform partial wipe by making it possible to run geth removedb non-interactive, using command line options instead.
Original problem was caused by #28595, where we made it so that as soon as we start to sync, the root of the disk layer is deleted. That is not wrong per se, but another part of the code uses the "presence of the root" as an init-check for the pathdb. And, since the init-check now failed, the code tried to re-initialize it which failed since a sync was already ongoing.
The total impact being: after a state-sync has begun, if the node for some reason is is shut down, it will refuse to start up again, with the error message: `Fatal: Failed to register the Ethereum service: waiting for sync.`.
This change also modifies how `geth removedb` works, so that the user is prompted for two things: `state data` and `ancient chain`. The former includes both the chaindb aswell as any state history stored in ancients.
---------
Co-authored-by: Martin HS <martin@swende.se>
This change
- Removes interface `log.Format`,
- Removes method `log.FormatFunc`,
- unexports `TerminalHandler.TerminalFormat` formatting methods (renamed to `TerminalHandler.format`)
- removes the notion of `log.Lazy` values
The lazy handler was useful in the old log package, since it
could defer the evaluation of costly attributes until later in the
log pipeline: thus, if the logging was done at 'Trace', we could
skip evaluation if logging only was set to 'Info'.
With the move to slog, this way of deferring evaluation is no longer
needed, since slog introduced 'Enabled': the caller can thus do
the evaluate-or-not decision at the callsite, which is much more
straight-forward than dealing with lazy reflect-based evaluation.
Also, lazy evaluation would not work with 'native' slog, as in, these
two statements would be evaluated differently:
```golang
log.Info("foo", "my lazy", lazyObj)
slog.Info("foo", "my lazy", lazyObj)
```
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.
Main changes are as follows:
* removes any log handlers that were unused in the Geth codebase.
* Json, logfmt, and terminal formatters are now slog handlers.
* Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options.
* `--log.backtraceat` and `--log.debug` are removed.
The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged.
A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed.
----
For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did
```golang
log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true))))
```
You now instead need to do
```golang
log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true)))
```
See more about reasoning here: https://github.com/ethereum/go-ethereum/issues/28558#issuecomment-1820606613
There were several problems related to dumping state.
- If a preimage was missing, even if we had set the `OnlyWithAddresses` to `false`, to export them anyway, the way the mapping was constructed (using `common.Address` as key) made the entries get lost anyway. Concerns both state- and blockchain tests.
- Blockchain test execution was not configured to store preimages.
This changes makes it so that the block test executor takes a callback, just like the state test executor already does. This callback can be used to examine the post-execution state, e.g. to aid debugging of test failures.
* cmd, les, tests: remove light client code
This commit removes the light client (LES) code.
Since the merge the light client has been broken and
it is hard to maintain it alongside the normal client.
We decided it would be best to remove it for now and
maybe rework and reintroduce it in the future.
* cmd, eth: remove some more mentions of light mode
* cmd: re-add flags and mark as deprecated
* cmd: warn the user about deprecated flags
* eth: better error message
Adds a subcommand: `geth snapshot export-preimages`, to export preimages of every hash found during a snapshot enumeration: that is, it exports _only the active state_, and not _all_ preimages that have been used but are no longer part of the state.
This tool is needed for the verkle transition, in order to distribute the preimages needed for the conversion. Since only the 'active' preimages are exported, the output is shrunk from ~70GB to ~4GB.
The order of the output is the order used by the snapshot enumeration, which avoids database thrashing. However, it also means that storage-slot preimages are not deduplicated.
This change allows the creation of a genesis block for verkle testnets. This makes for a chunk of code that is easier to review and still touches many discussion points.
* cmd/geth: more testcases for logging
This adds more edgecases around logging, particularly around handling of different types of nil-values
as concrete types and within interfaces.
Also adds tests with 'reserved' values which breaks json/logfmt formats. The json output is checked in,
but not actively used by any testcase at the moment.
* cmd/geth/testdata: remove timestamps
a little copying is better than a little dependency
-- go proverb
We have this dependency on docker, a.k.a moby: a gigantic library, and we only need ~70 LOC,
so here I tried moving it inline instead.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is a bit in preparation for the slog work in #28187 .
Our current test re logging mostly test the internals, but we have no real end-to-end test of the logging output. This PR introduces a simple reexec-based log tester. This also relies upon a special mode in geth, which can be made to eject a set of predefined log messages (only available if the build-tag `integrationtests` is used
e.g. go run --tags=integrationtests ./cmd/geth --log.format terminal logtest
While working on this, I also noticed a quirk in the setup: when geth was configured to use a file output, then two separate handlers were used (one handler for the file, one handler for the console). Using two separate handlers means that two formatters are used, thus the formatting of any/all records happened twice. This PR changes the mechanism to use two separate io.Writers instead, which is both more optimal and fixes a bug which occurs due to a global statefulness in the formatter.
* cmd, core: resolve scheme from a read-write database
* cmd, core, eth: move the scheme check in the ethereum constructor
* cmd/geth: dump should in ro mode
* cmd: reverts
This updates minisign to the latest version. One new thing is that minisign (not go-minisign) has started to prehash the file, and in order to make geth pass the version-check, we need to sign the file in legacy-mode.
This chang creates a GaugeInfo metrics type for registering informational (textual) metrics, e.g. geth version number. It also improves the testing for backend-exporters, and uses a shared subpackage in 'internal' to provide sample datasets and ordered registry.
Implements #21783
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/blobpool: implement txpool for blob txs
* core/txpool: track address reservations to notice any weird bugs
* core/txpool/blobpool: add support for in-memory operation for tests
* core/txpool/blobpool: fix heap updating after SetGasTip if account is evicted
* core/txpool/blobpool: fix eviction order if cheap leading txs are included
* core/txpool/blobpool: add note as to why the eviction fields are not inited in reinject
* go.mod: pull in inmem billy form upstream
* core/txpool/blobpool: fix review commens
* core/txpool/blobpool: make heap and heap test deterministic
* core/txpool/blobpool: luv u linter
* core/txpool: limit blob transactions to 16 per account
* core/txpool/blobpool: fix rebase errors
* core/txpool/blobpool: luv you linter
* go.mod: revert some strange crypto package dep updates
This change removes PoW header syncing related code from LES and also deletes
duplicated packages les/catalyst, les/downloader and les/fetcher. These package copies
were created because people wanted to make changes in their eth/ counterparts, but weren't
able to adapt LES code to the API changes.
It is usually best to set GOMAXPROCS to the number of available CPU cores. However, setting
it like that does not work well when the process is quota-limited to a certain number of CPUs.
The automaxprocs library configures GOMAXPROCS, taking such limits into account.
This simplifies the code that initializes the discovery a bit, and
adds new flags for enabling/disabling discv4 and discv5 separately.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This change adds back the 'geth --dev' mode of operation, using a cl-mocker.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
This PR adds server-side limits for JSON-RPC batch requests. Before this change, batches
were limited only by processing time. The server would pick calls from the batch and
answer them until the response timeout occurred, then stop processing the remaining batch
items.
Here, we are adding two additional limits which can be configured:
- the 'item limit': batches can have at most N items
- the 'response size limit': batches can contain at most X response bytes
These limits are optional in package rpc. In Geth, we set a default limit of 1000 items
and 25MB response size.
When a batch goes over the limit, an error response is returned to the client. However,
doing this correctly isn't always possible. In JSON-RPC, only method calls with a valid
`id` can be responded to. Since batches may also contain non-call messages or
notifications, the best effort thing we can do to report an error with the batch itself is
reporting the limit violation as an error for the first method call in the batch. If a batch is
too large, but contains only notifications and responses, the error will be reported with
a null `id`.
The RPC client was also changed so it can deal with errors resulting from too large
batches. An older client connected to the server code in this PR could get stuck
until the request timeout occurred when the batch is too large. **Upgrading to a version
of the RPC client containing this change is strongly recommended to avoid timeout issues.**
For some weird reason, when writing the original client implementation, @fjl worked off of
the assumption that responses could be distributed across batches arbitrarily. So for a
batch request containing requests `[A B C]`, the server could respond with `[A B C]` but
also with `[A B] [C]` or even `[A] [B] [C]` and it wouldn't make a difference to the
client.
So in the implementation of BatchCallContext, the client waited for all requests in the
batch individually. If the server didn't respond to some of the requests in the batch, the
client would eventually just time out (if a context was used).
With the addition of batch limits into the server, we anticipate that people will hit this
kind of error way more often. To handle this properly, the client now waits for a single
response batch and expects it to contain all responses to the requests.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/txpool: abstraction prep work for secondary pools (blob pool)
* core/txpool: leave subpool concepts to a followup pr
* les: fix tests using hard coded errors
* core/txpool: use bitmaps instead of maps for tx type filtering
* cryto/kzg4844: pull in the C and Go libs for KZG cryptography
* go.mod: pull in the KZG libraries
* crypto/kzg4844: add basic becnhmarks for ballpark numbers
* cmd, crypto: integrate both CKZG and GoKZG all the time, add flag
* cmd/utils, crypto/kzg4844: run library init on startup
* crypto/kzg4844: make linter happy
* crypto/kzg4844: push missing file
* crypto/kzg4844: fully disable CKZG but leave in the sources
* build, crypto/kzg4844, internal: link CKZG by default and with portable mode
* crypto/kzg4844: drop verifying the trusted setup in gokzg
* internal/build: yolo until it works?
* cmd/utils: make flag description friendlier
Co-authored-by: Martin Holst Swende <martin@swende.se>
* crypto/ckzg: no need for double availability check
* build: tiny flag cleanup nitpick
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
rename parameter
In this case, the naming of "extapi" might create some confusion. Although it represents an External Signer Backend, its name could be mistaken for an API. In reality, it is a backend instance used for communicating with external signers. A better naming choice could be "extBackend" or "externalBackend" to more accurately describe that it is a backend instance rather than an API.
Makes the `geth account ... ` commands usable even if a geth-process is already executing, since the account commands do not read the chaindata, it was not required for those to use the same locking mechanism.
---
Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
* cmd/utils, node: switch to Pebble as the default db if none exists
* node: fall back to LevelDB on platforms not supporting Pebble
* core/rawdb, node: default to Pebble at the node level
* cmd/geth: fix some tests explicitly using leveldb
* ethdb/pebble: allow double closes, makes tests simpler
This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
