* Add CLI flags to config LevelDB table/total sizes
I wired up CLI flags to allow configuring LevelDB table and total sizes:
- `--leveldb.compaction.table.size`, LevelDB SSTable file size factor in MiB (default: 2)
- `--leveldb.compaction.table.multiplier`, multiplier on LevelDB SSTable file size (default: 1)
- `--leveldb.compaction.total.size`, total size factor in MiB of LevelDB levels (default: 10)
- `--leveldb.compaction.total.multiplier`, multiplier on LevelDB total level size (default: 10)
N.B. that the default values for these configs are exactly the same as
before this changset and so Bor behavior should not change unless these
flags are deliberately overridden. Bor/Geth inherited the default values
from [the `goleveldb`
defaults](126854af5e/leveldb/opt/options.go).
We (Alchemy) found it necessary to override these configs as follows to
keep Bor archive nodes tracking the canonical chain:
- `--leveldb.compaction.table.size=4`
- `--leveldb.compaction.total.size=20`
These overrides double the size of LevelDB SSTable files (2 MiB -> 4
MiB) and also the total amount of data in each level (100 MiB -> 200
MiB, 1,000 MiB -> 2,000 MiB, etc.). The idea is to have LevelDB read and
write data in larger chunks while keeping the proportional frequency of
compaction operations the same as in the original defaults defined by
Dean and Ghemawat.
Without these overrides we found that our archive nodes would tend to
fall into a "LevelDB compaction loop of death" where the incoming stream
of blockchain data could not be flowed into LevelDB's structure quickly
enough, resulting in the node blocking writes for long periods of time
while LevelDB's single-threaded compaction organized the data. Over
time the nodes would fall farther and farther behind the canonical chain
head, metaphorically dying a slow node's death.
These configs can be changed on existing node databases (resyncing is
not necessary). LevelDB appears to work correctly with SSTable files of
different sizes. Note that the database does not undergo any sort of
migration when changing these configs. Only newly-written files (due to
new data or compaction) are affected by these configs.
* Update docs
* Adjust line spacing for linter
* Replace map with `ExtraDBConfig`
* Rename `LevelDbConfig` to `ExtraDBConfig`
* Regenerate docs
* Milestone Implementation
* Merge branch 'POS-347' into reciept-e2e-test
* Changes for testing, will be removed after testing
* Debugged the error
* Just for testing purpose
* refactor debug api methods, rename whitelist -> checkpoint
* remove first iteration based vars
* fix linters
* Rewind Changes
* Error changes
* RewindBack function in bor_checkpoint_verifier
* Testcases
* Added the fetch test for milestone and checkpoint
* Debugged the lint changes
* Debugged the lint changes
* Debugged the lint changes
* Debugged the lint changes
* Improved the error in Miner test file
* Improved the error of pointing to the wrong function
* Locking the sprint after the vote has been made on it.
* Adding more logs for testing
* Adding more logs for testing
* Adding more logs for testing
* Implemented the NoAckMilestone fetching mechanism
* Testcases for milestone implementation
* Testing code for fetchNoAckMilestone and fetchLastNoAckMilestone
* Testing changes
* refactor else-if
* Corrected the number of params in bor_ext.go
* Dummy API for testing
* Defined the GetVoteOnRootHash in interface
* Defined the GetVoteOnRootHash in interface
* Made changes in the web3ext file
* Added the GetVoteOnRootHash in PublicBlockChain API
* Added the GetVoteOnRootHash in filterBackend
* Added the log of Root and RooHash
* Removed the 0x from rootHash
* Just for testing purpose
* "GetVoteOnRootHash" mock implementation
* bor_test.go
* Added the test for milestone implementation
* Added service for fetching milestone by ID
* Improved the comments
* Removed the duplicate code
* use setter for borVerifier
* use setter for borVerifier
* refactor handleNoAckMilestone
* remove code repetition with retry function
* Converged the repetitive code
* after CR
* persistence
* persistence implementation
* feature flag
* Persistence Changes
* cr
* initial
* fix
* fix
* Whitelist Flag
* 1 Add:Included the milestone flag 2.Add:Hardlimit the rewind to maximum of 255 blocks
* Chg:Updated go.mod file
* Remove:Dubai Hardfork code
* Add:checked errors for call functions to the Db, Rmv: Remote Header variable from the IsValidPeer() function
* Fix:Linting issues'
* Add:MilestoneGRPC functions
* Fix:Lint issues
* Fix:Lint issues
* Fix: TestFetchMilestoneFromMockHeimdall
* Fix:Integrations tests
* Add:Test for sprint length and milestone changes
* Add:Functionality to fetch the finalized block
* Chg:Changed default val of TriesInmemory to 1024
* fix:Some functions of heimdallGRPC client
* Restored the GRPC functionality, was commented out for developing purpose
* Fix:Bor_checkpoint_Verfier function
* Test:Added the chain Rewinding test
* Test:Added the Sprint Length + Milestone merge test
* Add:Implemented the future milestone
* Add:Future milestone changes
* Add:Future milestone changes
* Chg: Voting on endBlockHash rather than rootHash
* Chg: Changed the logic of future milestone from rootHash checking to endBlockHash checking
* Fix:Using endBockHash while verifying the incoming milestone
* Chg:Variable names for better readiblity
* Fix:Testing changes
* Add:metrics for milestone implementation
* Add:Metrics for milestone implementatian
* Fix:Order of statements in a function for better optimization
* Chg:Removed unrequired file
* Fix:new variable intialization
* Add:Comment to increase readiblity
* Fix:Logs
* Chg:Name of GetVoteOnRootHash to GetVoteOnHash
* Fix:Linting issues
* Fixed linting issues
* Rmv: Unnecessary logs and Add:Skip test for long tests
* Fix:Checking current chain with whitelisted milestone or checkpoint in Finalized block function
* Fix:Test
* Fix:Whitelisting of Milestone and Checkpoint process
* Fix: Milestone JSON structure
* Chg:Testcases changes
* Fix:Change from VoteOnRootHash to VoteOnHash
* Fix:Variable name fix
* Fix:Finalized API
* internal/jsre/deps: update web3.js bundle
* Fix:milestone verifier
* Chg:Handling the long future chain import issue
* Fix:Lint issues
* Fix:TestLowDiffLongChain and TestPrunedImportSide tests, used hardcoded value 128 instead of DefaultTriesInMemory value
* Chg:Testcode for producing metrics
* Chg:Milestong polling value to 32 secs
* Add:Testcases
* Add:Implemented the check to fetch the milestoneId from heimdall before locking the fork
* Added GRPC method for FetchMilestoneID
* Fix:lint issue
* Fix:lint issue
* Skiped out the tests which were mainly used to produce the supporting data
* remove vcs build when running snyk
* Add:Improved the logs and comments
* fix linters
* Skipped some test as they are panic due to timeout issue in github
* Chg:Variable name LockerSprintNumber to LockedMilestoneNumber for better readablity and clarity
* Chg:Conflicting variable names in milestone test file
* Chg:Conflicting function names in milestone test file
* fix : minor fix in TestInsertingSpanSizeBlocks
* Fix:Mocking issue in TestInsertingSpanSizeBlocks
* Fix:GRPC Polyproto Version
* eth/downloader: skip peer drop due to whitelisting err
* eth, tests/bor: bug fixes and minor refactor
* Add:Implemented the milestone related functions in the HeimdallApp
* Fix:Lint Errors & Remove:Redundant Code
* Fix:Testing Errors
* Fix:Bor integeration tests
* Fix:Test errors
* update heimdall client mock files
* remove unused arguments
* remove redundant code
* Chg:Changed the milestone polling intervals
* Add: added block finality from whitelisted checkpoint
* skip future chain validation
* Add:confirmation check of 16 blocks over the end block while voting for the milestone in GetVoteHash() function
* Chg:Included endBlockNum in UnlockMutex function
* Add:Property based test for milestone
* Fix:Opening the lock while processing future milestone
* Add:Property based test for futureMilestone
* Defined the value of TempTriesInMemory
* Fixed the finalized api
* Fixed lint issues
* eth: add logs while fetching and rewinding
* fix linters: use default returns instead of recursive calls
* Fix:Milestone intergration test
* Add:GetVoteHash fn in mock backend
* tests/bor: fix mock span
* tests/bor: remove t.Parallel()
* use bor namespace in ethclient, fix mock function
---------
Co-authored-by: Vaibhav Jindal <vaibhavjindal29@gmail.com>
Co-authored-by: VaibhavJindal <74560896+VAIBHAVJINDAL3012@users.noreply.github.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
This change removes a chainconfig parameter passed into rawdb.ReadLogs, which is not used nor needed.
It also modifies the filter loop slightly, avoiding a labeled break and instead using a method.
This change does not modify any behaviour.
* all: implement path-based state scheme
* all: edits from review
* core/rawdb, trie/triedb/pathdb: review changes
* core, light, trie, eth, tests: reimplement pbss history
* core, trie/triedb/pathdb: track block number in state history
* trie/triedb/pathdb: add history documentation
* core, trie/triedb/pathdb: address comments from Peter's review
Important changes to list:
- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated
* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache
* core/rawdb: fix tests
* trie/triedb/pathdb: rename metrics, change clean cache key
* trie/triedb: manage the clean cache inside of disk layer
* trie/triedb/pathdb: move journal function
* trie/triedb/path: fix tests
* trie/triedb/pathdb: fix journal
* trie/triedb/pathdb: fix history
* trie/triedb/pathdb: try to fix tests on windows
* core, trie: address comments
* trie/triedb/pathdb: fix test issues
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change adds the ability to perform reads from freezer without size limitation. This can be useful in cases where callers are certain that out-of-memory will not happen (e.g. reading only a few elements).
The previous API was designed to behave both optimally and secure while servicing a request from a peer, whereas this change should _not_ be used when an untrusted peer can influence the query size.
* cmd/utils, node: switch to Pebble as the default db if none exists
* node: fall back to LevelDB on platforms not supporting Pebble
* core/rawdb, node: default to Pebble at the node level
* cmd/geth: fix some tests explicitly using leveldb
* ethdb/pebble: allow double closes, makes tests simpler
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)
* Added changes per discussion in PR, more changes may be necessary
* Adding prerelease true
* Disabling goreleaser
* Removing README swap file
* change bor_dir and add bor user for v0.3.0 release
* rollback bor user and use root
* metrics: handle equal to separated config flag (#596)
* metrics: handle based config path
* internal/cli/server: add more context to logs
* use space separated flag and value in bor.service
* fixed static-nodes related buf (os independent) (#598)
* fixed static-nodes related buf (os independent)
* taking static-nodes as input if default not present
* Update default flags (#600)
* internal/cli/server: use geth's default for txpool.pricelimit and add comments
* builder/files: update config.toml for mainnet
* packaging/templates: update defaults for mainnet and mumbai
* internal/cli/server: skip overriding cache
* packaging/templates: update cache value for mainnet
* packaging/templates: update gcmode for archive mumbai node
* metrics: handle nil telemetry config (#601)
* resolve merge conflicts
* update go version in release.yml
* update goversion in makefile
* update Docker login for goreleaser-cross v1.19
* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist
* added a toml configuration file with comments describing each flag (#607)
* added a toml configuration file with comments describing each flag
* internal/cli/server: update flag description
* docs/cli: update example config and description of flags
* docs: update new-cli docs
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update
* added ancient datadir flag and toml field, need to decide on default value and update the conversion script
* updated toml files with ancient field
* Add support for new flags in new config.toml, which were present in old config.toml (#612)
* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml
* added RAW fields for these time.Duration flags
* updated the conversion script to support these extra 4 flags
* removed hcl and json config tests as we are only supporting toml config files
* updated toml files with cache.timeout field
* updated toml files with jsonrpc.timeouts field
* tests/bor: expect a call for latest checkpoint
* tests/bor: expect a call for latest checkpoint
* packaging/templates: update cache values for archive nodes
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* remove unwanted code
* Fix docker publish authentication issue
In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.
* Revert "update Docker login for goreleaser-cross v1.19"
This reverts commit 4d19cf5342.
* Bump version to stable
* Revert "Merge pull request #435 from maticnetwork/POS-553"
This reverts commit 657d262def, reversing
changes made to 88dbfa1c13.
* revert change for release for go1.19
* Add default values to CLI helper and docs
This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.
* Add a summary of new CLI in docs
* Updating packager as binutils changed version so that apt-get installs current versions
* Add state pruning to new CLI
* Minor wording fix in prune state description
* Bumping control file versions
* Mainnet Delhi fork
* Set version to stable
* change delhi hardfork block number
* handle future chain import and skip peer drop (#650)
* handle future chain import and skip peer drop
* add block import metric
* params: bump version to v0.3.3-stable
* Bump bor version in control files for v0.3.3 mainnet release
* Added checks to RPC requests and introduced new flags to customise the parameters (#657)
* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)
* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)
* updated the config files and docs
* chg : trieTimeout from 60 to 10 mins (#692)
* chg : trieTimeout from 60 to 10 mins
* chg : cache.timout to 10m from 1h in configs
* internal/cli/server : fix : added triesInMemory in config (#691)
* changed version from 0.3.0 to 0.3.4-beta (#693)
* fix nil state-sync issue, increase grpc limit (#695)
* Increase grpc message size limit in pprof
* consensus/bor/bor.go : stateSyncs init fixed [Fix#686]
* eth/filters: handle nil state-sync before notify
* eth/filters: update check
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
* core, tests/bor: add more tests for state-sync validation (#710)
* core: add get state sync function for tests
* tests/bor: add validation for state sync events post consensus
* Arpit/temp bor sync (#701)
* Increase grpc message size limit in pprof
* ReadBorReceipts improvements
* use internal function
* fix tests
* fetch geth upstread for ReadBorReceiptRLP
* Only query bor receipt when the query index is equal to # tx in block body
This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.
* Revert "fetch geth upstread for ReadBorReceiptRLP"
This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.
* Restore ReadBorReceiptRLP
* fix bor receipts
* remove unused
* fix lints
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)
This reverts commit 241843c7e7.
* Arpit/add execution pool 2 (#719)
* initial
* linters
* linters
* remove timeout
* update pool
* change pool size function
* check nil
* check nil
* fix tests
* Use execution pool from server in all handlers
* simplify things
* test fix
* add support for cli, config
* add to cli and config
* merge base branch
* debug statements
* fix bug
* atomic pointer timeout
* add apis
* update workerpool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* fix tests
* mutex
* refactor flag and value names
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* debug statements
* fix bug
* update workerpool
* atomic pointer timeout
* add apis
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* merge base branch
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* mutex
* fix tests
* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* Change default size of execution pool to 40
* refactor flag and value names
* fix merge conflicts
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* fix linters
* fix go.mod
* change sec to ms
* change default value for ep timeout
* fix node api calls
* comment setter for ep timeout
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* version change (#721)
* Event based pprof (#732)
* feature
* Save pprof to /tmp
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
* Cherry-pick changes from develop (#738)
* Check if block is nil to prevent panic (#736)
* miner: use env for tracing instead of block object (#728)
---------
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* add max code init size check in txpool (#739)
* Revert "Event based pprof" and update version (#742)
* Revert "Event based pprof (#732)"
This reverts commit 22fa4033e8.
* params: update version to 0.3.4-beta3
* packaging/templates: update bor version
* internal/ethapi :: Fix : newRPCTransactionFromBlockIndex
* Merge master to qa (#813)
* Merge qa to master (#750)
* Added checks to RPC requests and introduced new flags to customise the parameters (#657)
* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)
* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)
* updated the config files and docs
* chg : trieTimeout from 60 to 10 mins (#692)
* chg : trieTimeout from 60 to 10 mins
* chg : cache.timout to 10m from 1h in configs
* internal/cli/server : fix : added triesInMemory in config (#691)
* changed version from 0.3.0 to 0.3.4-beta (#693)
* fix nil state-sync issue, increase grpc limit (#695)
* Increase grpc message size limit in pprof
* consensus/bor/bor.go : stateSyncs init fixed [Fix#686]
* eth/filters: handle nil state-sync before notify
* eth/filters: update check
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
* core, tests/bor: add more tests for state-sync validation (#710)
* core: add get state sync function for tests
* tests/bor: add validation for state sync events post consensus
* Arpit/temp bor sync (#701)
* Increase grpc message size limit in pprof
* ReadBorReceipts improvements
* use internal function
* fix tests
* fetch geth upstread for ReadBorReceiptRLP
* Only query bor receipt when the query index is equal to # tx in block body
This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.
* Revert "fetch geth upstread for ReadBorReceiptRLP"
This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.
* Restore ReadBorReceiptRLP
* fix bor receipts
* remove unused
* fix lints
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)
This reverts commit 241843c7e7.
* Arpit/add execution pool 2 (#719)
* initial
* linters
* linters
* remove timeout
* update pool
* change pool size function
* check nil
* check nil
* fix tests
* Use execution pool from server in all handlers
* simplify things
* test fix
* add support for cli, config
* add to cli and config
* merge base branch
* debug statements
* fix bug
* atomic pointer timeout
* add apis
* update workerpool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* fix tests
* mutex
* refactor flag and value names
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* debug statements
* fix bug
* update workerpool
* atomic pointer timeout
* add apis
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* merge base branch
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* mutex
* fix tests
* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* Change default size of execution pool to 40
* refactor flag and value names
* fix merge conflicts
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* fix linters
* fix go.mod
* change sec to ms
* change default value for ep timeout
* fix node api calls
* comment setter for ep timeout
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* version change (#721)
* Event based pprof (#732)
* feature
* Save pprof to /tmp
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
* Cherry-pick changes from develop (#738)
* Check if block is nil to prevent panic (#736)
* miner: use env for tracing instead of block object (#728)
---------
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* add max code init size check in txpool (#739)
* Revert "Event based pprof" and update version (#742)
* Revert "Event based pprof (#732)"
This reverts commit 22fa4033e8.
* params: update version to 0.3.4-beta3
* packaging/templates: update bor version
* params, packaging/templates: update bor version
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* core, miner: add sub-spans for tracing (#753)
* core, miner: add sub-spans for tracing
* fix linters
* core: add logs for debugging
* core: add more logs to print tdd while reorg
* fix linters
* core: minor fix
* core: remove debug logs
* core: use different span for write block and set head
* core: use internal context for sending traces (#755)
* core: add : impossible reorg block dump (#754)
* add : impossible reorg block dump
* chg : 3 seperate files for impossoble reorg dump
* add : use exportBlocks method and RLP blocks before writing
* chg : small changes
* bump : go version from 1.19 to 1.20.1 (#761)
* Revert "bump : go version from 1.19 to 1.20.1 (#761)"
This reverts commit 4561012af9.
* core/vm: use optimized bigint (#26021)
* Add holiman/big
* Fix linter
* Bump version to v0.3.5
* fix lints from develop (few lints decided to appear from code that was untouched, weird)
* upgrade crypto lib version (#770)
* bump dep : github.com/Masterminds/goutils to v1.1.1 (#769)
* mardizzone/pos-1313: bump crypto dependency (#772)
* dev: chg: bumd net dependency
* dev: chg: bump crypto dependency
* dev: chg: bump crypto dependency
* bump dep : golang.org/x/net to v0.8.0 (#771)
* Verify validator set against local contract on receiving an end-of-sprint block (#768)
* Verify validator set against local contract on receiving an end-of-sprint block
* Fix tests
* Respect error returned by ParseValidators
* Keep going back until a parent block presents
* core/txpool: implement DoS defenses from geth (#778)
* Hotfixes and deps bump (#776)
* dev: chg: bump deps
* internal/cli/server, rpc: lower down http readtimeout to 10s
* dev: chg: get p2p adapter
* dev: chg: lower down jsonrpc readtimeout to 10s
* cherry-pick txpool optimisation changes
* add check for empty lists in txpool (#704)
* add check
* linters
* core, miner: add empty instrumentation name for tracing
---------
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* packaging,params: bump to v0.3.6 (#782)
* v0.3.6 fix (#787)
* Fix get validator set in header verifier
* chg : commit tx logs from info to debug (#673)
* chg : commit tx logs from info to debug
* fix : minor changes
* chg : miner : commitTransactions-stats moved from info to debug
* lint : fix linters
* refactor logging
* miner : chg : UnauthorizedSignerError to debug
* lint : fix lint
* fix : log.Logger interface compatibility
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Remove unnecessary sorting of valset from header in verification
* dev: chg: version bump
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
* core: improve locks in txpool (#807)
* added a write lock to the txs.filter method and a read lock to the txs.reheap method - both of which are called by Filter during reorg adjustments to txpool
* txpool reorg locks
* more locks
* locks
* linters
* params, packaging: update version for v0.3.8-beta release
* core: add logs in reheap
---------
Co-authored-by: Alex <dalexwatts@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Merge qa to master (#808)
* Added checks to RPC requests and introduced new flags to customise the parameters (#657)
* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)
* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)
* updated the config files and docs
* chg : trieTimeout from 60 to 10 mins (#692)
* chg : trieTimeout from 60 to 10 mins
* chg : cache.timout to 10m from 1h in configs
* internal/cli/server : fix : added triesInMemory in config (#691)
* changed version from 0.3.0 to 0.3.4-beta (#693)
* fix nil state-sync issue, increase grpc limit (#695)
* Increase grpc message size limit in pprof
* consensus/bor/bor.go : stateSyncs init fixed [Fix#686]
* eth/filters: handle nil state-sync before notify
* eth/filters: update check
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
* core, tests/bor: add more tests for state-sync validation (#710)
* core: add get state sync function for tests
* tests/bor: add validation for state sync events post consensus
* Arpit/temp bor sync (#701)
* Increase grpc message size limit in pprof
* ReadBorReceipts improvements
* use internal function
* fix tests
* fetch geth upstread for ReadBorReceiptRLP
* Only query bor receipt when the query index is equal to # tx in block body
This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.
* Revert "fetch geth upstread for ReadBorReceiptRLP"
This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.
* Restore ReadBorReceiptRLP
* fix bor receipts
* remove unused
* fix lints
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)
This reverts commit 241843c7e7.
* Arpit/add execution pool 2 (#719)
* initial
* linters
* linters
* remove timeout
* update pool
* change pool size function
* check nil
* check nil
* fix tests
* Use execution pool from server in all handlers
* simplify things
* test fix
* add support for cli, config
* add to cli and config
* merge base branch
* debug statements
* fix bug
* atomic pointer timeout
* add apis
* update workerpool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* fix tests
* mutex
* refactor flag and value names
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* debug statements
* fix bug
* update workerpool
* atomic pointer timeout
* add apis
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* merge base branch
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* mutex
* fix tests
* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* Change default size of execution pool to 40
* refactor flag and value names
* fix merge conflicts
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* fix linters
* fix go.mod
* change sec to ms
* change default value for ep timeout
* fix node api calls
* comment setter for ep timeout
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* version change (#721)
* Event based pprof (#732)
* feature
* Save pprof to /tmp
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
* Cherry-pick changes from develop (#738)
* Check if block is nil to prevent panic (#736)
* miner: use env for tracing instead of block object (#728)
---------
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* add max code init size check in txpool (#739)
* Revert "Event based pprof" and update version (#742)
* Revert "Event based pprof (#732)"
This reverts commit 22fa4033e8.
* params: update version to 0.3.4-beta3
* packaging/templates: update bor version
* internal/ethapi :: Fix : newRPCTransactionFromBlockIndex
* fix: remove assignment for bor receipt
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* Setting up bor to use hosted 18.04 runner as ubuntu provided 18.04 runner is end of life
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Alex <dalexwatts@gmail.com>
Co-authored-by: Daniel Jones <djones@polygon.technology>
* core: remove duplicate tests
* miner: use get validators by hash in tests
---------
Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Alex <dalexwatts@gmail.com>
The meter for "for measuring the effective amount of data read" within the freezertable was never updated. This change remedies that.
---------
Signed-off-by: jsvisa <delweng@gmail.com>
This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
This PR is a (superior) alternative to https://github.com/ethereum/go-ethereum/pull/26708, it handles deprecation, primarily two specific cases.
`rand.Seed` is typically used in two ways
- `rand.Seed(time.Now().UnixNano())` -- we seed it, just to be sure to get some random, and not always get the same thing on every run. This is not needed, with global seeding, so those are just removed.
- `rand.Seed(1)` this is typically done to ensure we have a stable test. If we rely on this, we need to fix up the tests to use a deterministic prng-source. A few occurrences like this has been replaced with a proper custom source.
`rand.Read` has been replaced by `crypto/rand`.`Read` in this PR.
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
* common, core, eth, les, trie: make prque generic
* les/vflux/server: fixed issues in priorityPool
* common, core, eth, les, trie: make priority also generic in prque
* les/flowcontrol: add test case for priority accumulator overflow
* les/flowcontrol: avoid priority value overflow
* common/prque: use int priority in some tests
No need to convert to int64 when we can just change the type used by the
queue.
* common/prque: remove comment about int64 range
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This change implements withdrawals as specified in EIP-4895.
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: marioevz <marioevz@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR does a few things.
It fixes a shutdown-order flaw in the chainfreezer. Previously, the chain-freezer would shutdown the freezer backend first, and then signal for the loop to exit. This can lead to a scenario where the freezer tries to fsync closed files, which is an error-conditon that could lead to exit via log.Crit.
It also makes the printout more detailed when truncating 'dangling' items, by showing the exact number instead of approximate MB.
This PR also adds calls to fsync files before closing them, and also makes the `db inspect` command slightly more robust.
This PR fixes an issue which might result in data lost in freezer.
Whenever mutation happens in freezer, all data will be written into head data file
and it will be rotated with a new one in case the size of file reaches the threshold.
Theoretically, the rotated old data file should be fsync'd to prevent data loss.
In freezer.Sync function, we only fsync: (1) index file (2) meta file and (3) head
data file. So this PR forcibly fsync the head data file if mutation happens in the
boundary of data file.
This PR implements resettable freezer by adding a ResettableFreezer wrapper.
The resettable freezer wraps the original freezer in a way that makes it possible to ensure atomic resets. Implementation wise, it relies on the os.Rename and os.RemoveAll to atomically delete the original freezer data and re-create a new one from scratch.
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.
As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
While investigating #22374, I noticed that the Sync operation of the
freezer does not take the table lock. It also doesn't call sync for all files
if there is an error with one of them. I doubt this will fix anything, but
didn't want to drop the fix on the floor either.
This PR ports a few changes from PBSS:
- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
This PR reworks tx indexer a bit. Compared to the original version, one scenario is no longer handled - upgrading from legacy geth without indexer support.
The tx indexer was introduced in 2020 and have been present through hardforks, so it can be assumed that all Geth nodes have tx indexer already. So we can simplify the tx indexer logic a bit:
- If the tail flag is not present, it means node is just initialized may or may not with an ancient store attached. In this case all blocks are regarded as unindexed
- If the tail flag is present, it means blocks below tail are unindexed, blocks above tail are indexed
This change also address some weird cornercases that could make the indexer not work after a crash.
core/blockchain: downgrade tx indexing and unindexing logs from info to debug
If a user has a finite tx lookup limit, they will see an "unindexing" info level log each time a block is imported. This information might help a user understand that they are removing the index each block and some txs may not be retrievable by hash, but overall it is generally more of a nuisance than a benefit. This change downgrades the log to a debug log.
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.
This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
* merge geth v1.10.15
* fix: Removed FastSync from cli server
* fix: TestHeadersRLPStorage
* Added t.skip(ETH2 in bor)
* fix: flow in create consensus engine
* bumped version
* Fix typo
* increase block time
* remove file
* bumped version
* merge gethv1.10.17
* bumped version
* fix failing tests
* Bump Go version to v1.18 (#368)
* Bump Go version to v1.18.1
* Build using netgo tag
This will create a static build using Go native networking stack.
Checked and it works stable for all archs and distros.
* Fix meta
* Bump version
* Meta as stable
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Sandeep Sreenath <sandeep.sreenath@gmail.com>
Co-authored-by: Victor Castell <victor@victorcastell.com>
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but t.TempDir handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* cmd,core: add simple legacy receipt converter
core/rawdb: use forEach in migrate
core/rawdb: batch reads in forEach
core/rawdb: make forEach anonymous fn
cmd/geth: check for legacy receipts on node startup
fix err msg
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix log
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix some review comments
add warning to cmd
drop isLegacy fn from migrateTable params
add test for windows rename
test replacing in windows case
* minor fix
* sanity check for tail-deletion
* add log before moving files around
* speed-up hack for mainnet
* fix mainnet check, use networkid instead
* check mainnet genesis
* review fixes
* resume previous migration attempt
* core/rawdb: lint fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
