* Add CLI flags to config LevelDB table/total sizes
I wired up CLI flags to allow configuring LevelDB table and total sizes:
- `--leveldb.compaction.table.size`, LevelDB SSTable file size factor in MiB (default: 2)
- `--leveldb.compaction.table.multiplier`, multiplier on LevelDB SSTable file size (default: 1)
- `--leveldb.compaction.total.size`, total size factor in MiB of LevelDB levels (default: 10)
- `--leveldb.compaction.total.multiplier`, multiplier on LevelDB total level size (default: 10)
N.B. that the default values for these configs are exactly the same as
before this changset and so Bor behavior should not change unless these
flags are deliberately overridden. Bor/Geth inherited the default values
from [the `goleveldb`
defaults](126854af5e/leveldb/opt/options.go).
We (Alchemy) found it necessary to override these configs as follows to
keep Bor archive nodes tracking the canonical chain:
- `--leveldb.compaction.table.size=4`
- `--leveldb.compaction.total.size=20`
These overrides double the size of LevelDB SSTable files (2 MiB -> 4
MiB) and also the total amount of data in each level (100 MiB -> 200
MiB, 1,000 MiB -> 2,000 MiB, etc.). The idea is to have LevelDB read and
write data in larger chunks while keeping the proportional frequency of
compaction operations the same as in the original defaults defined by
Dean and Ghemawat.
Without these overrides we found that our archive nodes would tend to
fall into a "LevelDB compaction loop of death" where the incoming stream
of blockchain data could not be flowed into LevelDB's structure quickly
enough, resulting in the node blocking writes for long periods of time
while LevelDB's single-threaded compaction organized the data. Over
time the nodes would fall farther and farther behind the canonical chain
head, metaphorically dying a slow node's death.
These configs can be changed on existing node databases (resyncing is
not necessary). LevelDB appears to work correctly with SSTable files of
different sizes. Note that the database does not undergo any sort of
migration when changing these configs. Only newly-written files (due to
new data or compaction) are affected by these configs.
* Update docs
* Adjust line spacing for linter
* Replace map with `ExtraDBConfig`
* Rename `LevelDbConfig` to `ExtraDBConfig`
* Regenerate docs
This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* eth: cmd: deprecate personal namespace
* eth: cmd: move deprecation to node
* node: disable toml of enablepersonal
* node: disable personal on ipc as well
* Update node/node.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* console: error -> warn
* node: less roulette
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR changes the API so that uint64 is used for fork timestamps.
It's a good choice because types.Header also uses uint64 for time.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change introduces a breaking change to miner.etherbase is configured.
Previously, users did not need to explicitly set the etherbase address via flag, since 'first' local account was used as etherbase automatically. This change removes the "default first account" feature.
In Proof-of-stake world, the fee recipient address is provided by CL, and not configured in Geth any more - meaning that miner.etherbase is mostly for legacy networks(pow, clique networks etc).
This PR does a few things.
It fixes a shutdown-order flaw in the chainfreezer. Previously, the chain-freezer would shutdown the freezer backend first, and then signal for the loop to exit. This can lead to a scenario where the freezer tries to fsync closed files, which is an error-conditon that could lead to exit via log.Crit.
It also makes the printout more detailed when truncating 'dangling' items, by showing the exact number instead of approximate MB.
This PR also adds calls to fsync files before closing them, and also makes the `db inspect` command slightly more robust.
--syncTarget is a feature for development purpose in post-merge world. Previously
it's added into eth.Config. But it turns out that's a stupid idea.
- syncTarget is a block object, which is hard to be put in config file(large)
- syncTarget is just a dev feature, doesn't make too much sense to add it in config file
So I remove it from the eth config object. And it also fixes the #26328
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.
As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
This PR adds a parameter to startup, --synctarget. The synctarget flag is a developer-flag, that can be useful in some scenarios as a replacement for a CL node. It defines a fixed block sync target:
geth --syncmode=full --synctarget=./block_15816882.hex_rlp
The --synctarget is only made available during syncmode=full
* changed default of maxpeers from 200 to 50
* docs: update additional notes for new-cli
* docs: update additional notes for new-cli
* small bug fix in the script to fix shopt issue
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
This PR fixes a regression causing snapshots not to be generated in "geth --import" mode. It also fixes the geth export command to be truly readonly, and adds a new test for geth export.
This update resolves an issue where StringSliceFlag would not be
rendered correctly in help output + mention that -H can be used multiple times
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR makes it possible to set custom headers, in particular for two scenarios:
- geth attach
- geth commands which can use --remotedb, e..g geth db inspect
The ability to use custom headers is typically useful for connecting to cloud-apis, e.g. providing an infura- or alchemy key, or for that matter access-keys for environments behind cloudflare.
Co-authored-by: Felix Lange <fjl@twurst.com>
`geth dumpgenesis` currently does not respect the content of the data directory. Instead, it outputs the genesis block created by command-line flags. This PR fixes it to read the genesis from the database, if the database already exists.
Co-authored-by: Martin Holst Swende <martin@swende.se>
