* eth: drop support for forward sync triggers and head block packets
* consensus, eth: enforce always merged network
* eth: fix tx looper startup and shutdown
* cmd, core: fix some tests
* core: remove notion of future blocks
* core, eth: drop unused methods and types
This pull request resets the txlookup cache if chain reorg happens,
preventing them from remaining reachable. It addresses failures in
the hive tests.
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This change simplifies the logic for indexing transactions and enhances the UX when transaction is not found by returning more information to users.
Transaction indexing is now considered as a part of the initial sync, and `eth.syncing` will thus be `true` if transaction indexing is not yet finished. API consumers can use the syncing status to determine if the node is ready to serve users.
This chang creates a GaugeInfo metrics type for registering informational (textual) metrics, e.g. geth version number. It also improves the testing for backend-exporters, and uses a shared subpackage in 'internal' to provide sample datasets and ordered registry.
Implements #21783
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* Milestone Implementation
* Merge branch 'POS-347' into reciept-e2e-test
* Changes for testing, will be removed after testing
* Debugged the error
* Just for testing purpose
* refactor debug api methods, rename whitelist -> checkpoint
* remove first iteration based vars
* fix linters
* Rewind Changes
* Error changes
* RewindBack function in bor_checkpoint_verifier
* Testcases
* Added the fetch test for milestone and checkpoint
* Debugged the lint changes
* Debugged the lint changes
* Debugged the lint changes
* Debugged the lint changes
* Improved the error in Miner test file
* Improved the error of pointing to the wrong function
* Locking the sprint after the vote has been made on it.
* Adding more logs for testing
* Adding more logs for testing
* Adding more logs for testing
* Implemented the NoAckMilestone fetching mechanism
* Testcases for milestone implementation
* Testing code for fetchNoAckMilestone and fetchLastNoAckMilestone
* Testing changes
* refactor else-if
* Corrected the number of params in bor_ext.go
* Dummy API for testing
* Defined the GetVoteOnRootHash in interface
* Defined the GetVoteOnRootHash in interface
* Made changes in the web3ext file
* Added the GetVoteOnRootHash in PublicBlockChain API
* Added the GetVoteOnRootHash in filterBackend
* Added the log of Root and RooHash
* Removed the 0x from rootHash
* Just for testing purpose
* "GetVoteOnRootHash" mock implementation
* bor_test.go
* Added the test for milestone implementation
* Added service for fetching milestone by ID
* Improved the comments
* Removed the duplicate code
* use setter for borVerifier
* use setter for borVerifier
* refactor handleNoAckMilestone
* remove code repetition with retry function
* Converged the repetitive code
* after CR
* persistence
* persistence implementation
* feature flag
* Persistence Changes
* cr
* initial
* fix
* fix
* Whitelist Flag
* 1 Add:Included the milestone flag 2.Add:Hardlimit the rewind to maximum of 255 blocks
* Chg:Updated go.mod file
* Remove:Dubai Hardfork code
* Add:checked errors for call functions to the Db, Rmv: Remote Header variable from the IsValidPeer() function
* Fix:Linting issues'
* Add:MilestoneGRPC functions
* Fix:Lint issues
* Fix:Lint issues
* Fix: TestFetchMilestoneFromMockHeimdall
* Fix:Integrations tests
* Add:Test for sprint length and milestone changes
* Add:Functionality to fetch the finalized block
* Chg:Changed default val of TriesInmemory to 1024
* fix:Some functions of heimdallGRPC client
* Restored the GRPC functionality, was commented out for developing purpose
* Fix:Bor_checkpoint_Verfier function
* Test:Added the chain Rewinding test
* Test:Added the Sprint Length + Milestone merge test
* Add:Implemented the future milestone
* Add:Future milestone changes
* Add:Future milestone changes
* Chg: Voting on endBlockHash rather than rootHash
* Chg: Changed the logic of future milestone from rootHash checking to endBlockHash checking
* Fix:Using endBockHash while verifying the incoming milestone
* Chg:Variable names for better readiblity
* Fix:Testing changes
* Add:metrics for milestone implementation
* Add:Metrics for milestone implementatian
* Fix:Order of statements in a function for better optimization
* Chg:Removed unrequired file
* Fix:new variable intialization
* Add:Comment to increase readiblity
* Fix:Logs
* Chg:Name of GetVoteOnRootHash to GetVoteOnHash
* Fix:Linting issues
* Fixed linting issues
* Rmv: Unnecessary logs and Add:Skip test for long tests
* Fix:Checking current chain with whitelisted milestone or checkpoint in Finalized block function
* Fix:Test
* Fix:Whitelisting of Milestone and Checkpoint process
* Fix: Milestone JSON structure
* Chg:Testcases changes
* Fix:Change from VoteOnRootHash to VoteOnHash
* Fix:Variable name fix
* Fix:Finalized API
* internal/jsre/deps: update web3.js bundle
* Fix:milestone verifier
* Chg:Handling the long future chain import issue
* Fix:Lint issues
* Fix:TestLowDiffLongChain and TestPrunedImportSide tests, used hardcoded value 128 instead of DefaultTriesInMemory value
* Chg:Testcode for producing metrics
* Chg:Milestong polling value to 32 secs
* Add:Testcases
* Add:Implemented the check to fetch the milestoneId from heimdall before locking the fork
* Added GRPC method for FetchMilestoneID
* Fix:lint issue
* Fix:lint issue
* Skiped out the tests which were mainly used to produce the supporting data
* remove vcs build when running snyk
* Add:Improved the logs and comments
* fix linters
* Skipped some test as they are panic due to timeout issue in github
* Chg:Variable name LockerSprintNumber to LockedMilestoneNumber for better readablity and clarity
* Chg:Conflicting variable names in milestone test file
* Chg:Conflicting function names in milestone test file
* fix : minor fix in TestInsertingSpanSizeBlocks
* Fix:Mocking issue in TestInsertingSpanSizeBlocks
* Fix:GRPC Polyproto Version
* eth/downloader: skip peer drop due to whitelisting err
* eth, tests/bor: bug fixes and minor refactor
* Add:Implemented the milestone related functions in the HeimdallApp
* Fix:Lint Errors & Remove:Redundant Code
* Fix:Testing Errors
* Fix:Bor integeration tests
* Fix:Test errors
* update heimdall client mock files
* remove unused arguments
* remove redundant code
* Chg:Changed the milestone polling intervals
* Add: added block finality from whitelisted checkpoint
* skip future chain validation
* Add:confirmation check of 16 blocks over the end block while voting for the milestone in GetVoteHash() function
* Chg:Included endBlockNum in UnlockMutex function
* Add:Property based test for milestone
* Fix:Opening the lock while processing future milestone
* Add:Property based test for futureMilestone
* Defined the value of TempTriesInMemory
* Fixed the finalized api
* Fixed lint issues
* eth: add logs while fetching and rewinding
* fix linters: use default returns instead of recursive calls
* Fix:Milestone intergration test
* Add:GetVoteHash fn in mock backend
* tests/bor: fix mock span
* tests/bor: remove t.Parallel()
* use bor namespace in ethclient, fix mock function
---------
Co-authored-by: Vaibhav Jindal <vaibhavjindal29@gmail.com>
Co-authored-by: VaibhavJindal <74560896+VAIBHAVJINDAL3012@users.noreply.github.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
Currently, we trigger the logic to (un)index transactions when the node receives a new
block. However, in some cases the node may not receive new blocks (eg, when the Geth node
is configured without peer discovery, or when it acts as an RPC node for historical-only
data).
In these situations, the Geth node user may not have previously configured txlookuplimit
(i.e. the default of around one year), but later realizes they need to index all
historical blocks. However, adding txlookuplimit=0 and restarting geth has no effect. This
change makes it check for required indexing work once, on startup, to fix the issue.
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR removes the newly added txpool.Transaction wrapper type, and instead adds a way
of keeping the blob sidecar within types.Transaction. It's better this way because most
code in go-ethereum does not care about blob transactions, and probably never will. This
will start mattering especially on the client side of RPC, where all APIs are based on
types.Transaction. Users need to be able to use the same signing flows they already
have.
However, since blobs are only allowed in some places but not others, we will now need to
add checks to avoid creating invalid blocks. I'm still trying to figure out the best place
to do some of these. The way I have it currently is as follows:
- In block validation (import), txs are verified not to have a blob sidecar.
- In miner, we strip off the sidecar when committing the transaction into the block.
- In TxPool validation, txs must have a sidecar to be added into the blobpool.
- Note there is a special case here: when transactions are re-added because of a chain
reorg, we cannot use the transactions gathered from the old chain blocks as-is,
because they will be missing their blobs. This was previously handled by storing the
blobs into the 'blobpool limbo'. The code has now changed to store the full
transaction in the limbo instead, but it might be confusing for code readers why we're
not simply adding the types.Transaction we already have.
Code changes summary:
- txpool.Transaction removed and all uses replaced by types.Transaction again
- blobpool now stores types.Transaction instead of defining its own blobTx format for storage
- the blobpool limbo now stores types.Transaction instead of storing only the blobs
- checks to validate the presence/absence of the blob sidecar added in certain critical places
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
* all: implement path-based state scheme
* all: edits from review
* core/rawdb, trie/triedb/pathdb: review changes
* core, light, trie, eth, tests: reimplement pbss history
* core, trie/triedb/pathdb: track block number in state history
* trie/triedb/pathdb: add history documentation
* core, trie/triedb/pathdb: address comments from Peter's review
Important changes to list:
- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated
* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache
* core/rawdb: fix tests
* trie/triedb/pathdb: rename metrics, change clean cache key
* trie/triedb: manage the clean cache inside of disk layer
* trie/triedb/pathdb: move journal function
* trie/triedb/path: fix tests
* trie/triedb/pathdb: fix journal
* trie/triedb/pathdb: fix history
* trie/triedb/pathdb: try to fix tests on windows
* core, trie: address comments
* trie/triedb/pathdb: fix test issues
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
The logs in this function are pulled straight from disk in rawdb.ReadRawReceipts and
also modified in receipts.DeriveFields, so removing the copy should be fine.
* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)
* Added changes per discussion in PR, more changes may be necessary
* Adding prerelease true
* Disabling goreleaser
* Removing README swap file
* change bor_dir and add bor user for v0.3.0 release
* rollback bor user and use root
* metrics: handle equal to separated config flag (#596)
* metrics: handle based config path
* internal/cli/server: add more context to logs
* use space separated flag and value in bor.service
* fixed static-nodes related buf (os independent) (#598)
* fixed static-nodes related buf (os independent)
* taking static-nodes as input if default not present
* Update default flags (#600)
* internal/cli/server: use geth's default for txpool.pricelimit and add comments
* builder/files: update config.toml for mainnet
* packaging/templates: update defaults for mainnet and mumbai
* internal/cli/server: skip overriding cache
* packaging/templates: update cache value for mainnet
* packaging/templates: update gcmode for archive mumbai node
* metrics: handle nil telemetry config (#601)
* resolve merge conflicts
* update go version in release.yml
* update goversion in makefile
* update Docker login for goreleaser-cross v1.19
* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist
* added a toml configuration file with comments describing each flag (#607)
* added a toml configuration file with comments describing each flag
* internal/cli/server: update flag description
* docs/cli: update example config and description of flags
* docs: update new-cli docs
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update
* added ancient datadir flag and toml field, need to decide on default value and update the conversion script
* updated toml files with ancient field
* Add support for new flags in new config.toml, which were present in old config.toml (#612)
* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml
* added RAW fields for these time.Duration flags
* updated the conversion script to support these extra 4 flags
* removed hcl and json config tests as we are only supporting toml config files
* updated toml files with cache.timeout field
* updated toml files with jsonrpc.timeouts field
* tests/bor: expect a call for latest checkpoint
* tests/bor: expect a call for latest checkpoint
* packaging/templates: update cache values for archive nodes
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* remove unwanted code
* Fix docker publish authentication issue
In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.
* Revert "update Docker login for goreleaser-cross v1.19"
This reverts commit 4d19cf5342.
* Bump version to stable
* Revert "Merge pull request #435 from maticnetwork/POS-553"
This reverts commit 657d262def, reversing
changes made to 88dbfa1c13.
* revert change for release for go1.19
* Add default values to CLI helper and docs
This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.
* Add a summary of new CLI in docs
* Updating packager as binutils changed version so that apt-get installs current versions
* Add state pruning to new CLI
* Minor wording fix in prune state description
* Bumping control file versions
* Mainnet Delhi fork
* Set version to stable
* change delhi hardfork block number
* handle future chain import and skip peer drop (#650)
* handle future chain import and skip peer drop
* add block import metric
* params: bump version to v0.3.3-stable
* Bump bor version in control files for v0.3.3 mainnet release
* Added checks to RPC requests and introduced new flags to customise the parameters (#657)
* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)
* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)
* updated the config files and docs
* chg : trieTimeout from 60 to 10 mins (#692)
* chg : trieTimeout from 60 to 10 mins
* chg : cache.timout to 10m from 1h in configs
* internal/cli/server : fix : added triesInMemory in config (#691)
* changed version from 0.3.0 to 0.3.4-beta (#693)
* fix nil state-sync issue, increase grpc limit (#695)
* Increase grpc message size limit in pprof
* consensus/bor/bor.go : stateSyncs init fixed [Fix#686]
* eth/filters: handle nil state-sync before notify
* eth/filters: update check
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
* core, tests/bor: add more tests for state-sync validation (#710)
* core: add get state sync function for tests
* tests/bor: add validation for state sync events post consensus
* Arpit/temp bor sync (#701)
* Increase grpc message size limit in pprof
* ReadBorReceipts improvements
* use internal function
* fix tests
* fetch geth upstread for ReadBorReceiptRLP
* Only query bor receipt when the query index is equal to # tx in block body
This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.
* Revert "fetch geth upstread for ReadBorReceiptRLP"
This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.
* Restore ReadBorReceiptRLP
* fix bor receipts
* remove unused
* fix lints
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)
This reverts commit 241843c7e7.
* Arpit/add execution pool 2 (#719)
* initial
* linters
* linters
* remove timeout
* update pool
* change pool size function
* check nil
* check nil
* fix tests
* Use execution pool from server in all handlers
* simplify things
* test fix
* add support for cli, config
* add to cli and config
* merge base branch
* debug statements
* fix bug
* atomic pointer timeout
* add apis
* update workerpool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* fix tests
* mutex
* refactor flag and value names
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* debug statements
* fix bug
* update workerpool
* atomic pointer timeout
* add apis
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* merge base branch
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* mutex
* fix tests
* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* Change default size of execution pool to 40
* refactor flag and value names
* fix merge conflicts
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* fix linters
* fix go.mod
* change sec to ms
* change default value for ep timeout
* fix node api calls
* comment setter for ep timeout
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* version change (#721)
* Event based pprof (#732)
* feature
* Save pprof to /tmp
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
* Cherry-pick changes from develop (#738)
* Check if block is nil to prevent panic (#736)
* miner: use env for tracing instead of block object (#728)
---------
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* add max code init size check in txpool (#739)
* Revert "Event based pprof" and update version (#742)
* Revert "Event based pprof (#732)"
This reverts commit 22fa4033e8.
* params: update version to 0.3.4-beta3
* packaging/templates: update bor version
* internal/ethapi :: Fix : newRPCTransactionFromBlockIndex
* Merge master to qa (#813)
* Merge qa to master (#750)
* Added checks to RPC requests and introduced new flags to customise the parameters (#657)
* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)
* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)
* updated the config files and docs
* chg : trieTimeout from 60 to 10 mins (#692)
* chg : trieTimeout from 60 to 10 mins
* chg : cache.timout to 10m from 1h in configs
* internal/cli/server : fix : added triesInMemory in config (#691)
* changed version from 0.3.0 to 0.3.4-beta (#693)
* fix nil state-sync issue, increase grpc limit (#695)
* Increase grpc message size limit in pprof
* consensus/bor/bor.go : stateSyncs init fixed [Fix#686]
* eth/filters: handle nil state-sync before notify
* eth/filters: update check
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
* core, tests/bor: add more tests for state-sync validation (#710)
* core: add get state sync function for tests
* tests/bor: add validation for state sync events post consensus
* Arpit/temp bor sync (#701)
* Increase grpc message size limit in pprof
* ReadBorReceipts improvements
* use internal function
* fix tests
* fetch geth upstread for ReadBorReceiptRLP
* Only query bor receipt when the query index is equal to # tx in block body
This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.
* Revert "fetch geth upstread for ReadBorReceiptRLP"
This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.
* Restore ReadBorReceiptRLP
* fix bor receipts
* remove unused
* fix lints
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)
This reverts commit 241843c7e7.
* Arpit/add execution pool 2 (#719)
* initial
* linters
* linters
* remove timeout
* update pool
* change pool size function
* check nil
* check nil
* fix tests
* Use execution pool from server in all handlers
* simplify things
* test fix
* add support for cli, config
* add to cli and config
* merge base branch
* debug statements
* fix bug
* atomic pointer timeout
* add apis
* update workerpool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* fix tests
* mutex
* refactor flag and value names
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* debug statements
* fix bug
* update workerpool
* atomic pointer timeout
* add apis
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* merge base branch
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* mutex
* fix tests
* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* Change default size of execution pool to 40
* refactor flag and value names
* fix merge conflicts
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* fix linters
* fix go.mod
* change sec to ms
* change default value for ep timeout
* fix node api calls
* comment setter for ep timeout
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* version change (#721)
* Event based pprof (#732)
* feature
* Save pprof to /tmp
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
* Cherry-pick changes from develop (#738)
* Check if block is nil to prevent panic (#736)
* miner: use env for tracing instead of block object (#728)
---------
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* add max code init size check in txpool (#739)
* Revert "Event based pprof" and update version (#742)
* Revert "Event based pprof (#732)"
This reverts commit 22fa4033e8.
* params: update version to 0.3.4-beta3
* packaging/templates: update bor version
* params, packaging/templates: update bor version
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* core, miner: add sub-spans for tracing (#753)
* core, miner: add sub-spans for tracing
* fix linters
* core: add logs for debugging
* core: add more logs to print tdd while reorg
* fix linters
* core: minor fix
* core: remove debug logs
* core: use different span for write block and set head
* core: use internal context for sending traces (#755)
* core: add : impossible reorg block dump (#754)
* add : impossible reorg block dump
* chg : 3 seperate files for impossoble reorg dump
* add : use exportBlocks method and RLP blocks before writing
* chg : small changes
* bump : go version from 1.19 to 1.20.1 (#761)
* Revert "bump : go version from 1.19 to 1.20.1 (#761)"
This reverts commit 4561012af9.
* core/vm: use optimized bigint (#26021)
* Add holiman/big
* Fix linter
* Bump version to v0.3.5
* fix lints from develop (few lints decided to appear from code that was untouched, weird)
* upgrade crypto lib version (#770)
* bump dep : github.com/Masterminds/goutils to v1.1.1 (#769)
* mardizzone/pos-1313: bump crypto dependency (#772)
* dev: chg: bumd net dependency
* dev: chg: bump crypto dependency
* dev: chg: bump crypto dependency
* bump dep : golang.org/x/net to v0.8.0 (#771)
* Verify validator set against local contract on receiving an end-of-sprint block (#768)
* Verify validator set against local contract on receiving an end-of-sprint block
* Fix tests
* Respect error returned by ParseValidators
* Keep going back until a parent block presents
* core/txpool: implement DoS defenses from geth (#778)
* Hotfixes and deps bump (#776)
* dev: chg: bump deps
* internal/cli/server, rpc: lower down http readtimeout to 10s
* dev: chg: get p2p adapter
* dev: chg: lower down jsonrpc readtimeout to 10s
* cherry-pick txpool optimisation changes
* add check for empty lists in txpool (#704)
* add check
* linters
* core, miner: add empty instrumentation name for tracing
---------
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* packaging,params: bump to v0.3.6 (#782)
* v0.3.6 fix (#787)
* Fix get validator set in header verifier
* chg : commit tx logs from info to debug (#673)
* chg : commit tx logs from info to debug
* fix : minor changes
* chg : miner : commitTransactions-stats moved from info to debug
* lint : fix linters
* refactor logging
* miner : chg : UnauthorizedSignerError to debug
* lint : fix lint
* fix : log.Logger interface compatibility
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Remove unnecessary sorting of valset from header in verification
* dev: chg: version bump
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
* core: improve locks in txpool (#807)
* added a write lock to the txs.filter method and a read lock to the txs.reheap method - both of which are called by Filter during reorg adjustments to txpool
* txpool reorg locks
* more locks
* locks
* linters
* params, packaging: update version for v0.3.8-beta release
* core: add logs in reheap
---------
Co-authored-by: Alex <dalexwatts@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Merge qa to master (#808)
* Added checks to RPC requests and introduced new flags to customise the parameters (#657)
* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)
* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)
* updated the config files and docs
* chg : trieTimeout from 60 to 10 mins (#692)
* chg : trieTimeout from 60 to 10 mins
* chg : cache.timout to 10m from 1h in configs
* internal/cli/server : fix : added triesInMemory in config (#691)
* changed version from 0.3.0 to 0.3.4-beta (#693)
* fix nil state-sync issue, increase grpc limit (#695)
* Increase grpc message size limit in pprof
* consensus/bor/bor.go : stateSyncs init fixed [Fix#686]
* eth/filters: handle nil state-sync before notify
* eth/filters: update check
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
* core, tests/bor: add more tests for state-sync validation (#710)
* core: add get state sync function for tests
* tests/bor: add validation for state sync events post consensus
* Arpit/temp bor sync (#701)
* Increase grpc message size limit in pprof
* ReadBorReceipts improvements
* use internal function
* fix tests
* fetch geth upstread for ReadBorReceiptRLP
* Only query bor receipt when the query index is equal to # tx in block body
This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.
* Revert "fetch geth upstread for ReadBorReceiptRLP"
This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.
* Restore ReadBorReceiptRLP
* fix bor receipts
* remove unused
* fix lints
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)
This reverts commit 241843c7e7.
* Arpit/add execution pool 2 (#719)
* initial
* linters
* linters
* remove timeout
* update pool
* change pool size function
* check nil
* check nil
* fix tests
* Use execution pool from server in all handlers
* simplify things
* test fix
* add support for cli, config
* add to cli and config
* merge base branch
* debug statements
* fix bug
* atomic pointer timeout
* add apis
* update workerpool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* fix tests
* mutex
* refactor flag and value names
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* debug statements
* fix bug
* update workerpool
* atomic pointer timeout
* add apis
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* fix issues
* change params
* fix issues
* fix ipc issue
* remove execution pool from IPC
* revert
* merge base branch
* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* mutex
* fix tests
* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool
* Change default size of execution pool to 40
* refactor flag and value names
* fix merge conflicts
* ordering fix
* refactor flag and value names
* update default ep size to 40
* fix bor start issues
* revert file changes
* fix linters
* fix go.mod
* change sec to ms
* change default value for ep timeout
* fix node api calls
* comment setter for ep timeout
---------
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* version change (#721)
* Event based pprof (#732)
* feature
* Save pprof to /tmp
---------
Co-authored-by: Jerry <jerrycgh@gmail.com>
* Cherry-pick changes from develop (#738)
* Check if block is nil to prevent panic (#736)
* miner: use env for tracing instead of block object (#728)
---------
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* add max code init size check in txpool (#739)
* Revert "Event based pprof" and update version (#742)
* Revert "Event based pprof (#732)"
This reverts commit 22fa4033e8.
* params: update version to 0.3.4-beta3
* packaging/templates: update bor version
* internal/ethapi :: Fix : newRPCTransactionFromBlockIndex
* fix: remove assignment for bor receipt
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
* Setting up bor to use hosted 18.04 runner as ubuntu provided 18.04 runner is end of life
---------
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Alex <dalexwatts@gmail.com>
Co-authored-by: Daniel Jones <djones@polygon.technology>
* core: remove duplicate tests
* miner: use get validators by hash in tests
---------
Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Alex <dalexwatts@gmail.com>
This PR removes the Debug field from vmconfig, making it so that if a tracer is set, debug=true is implied.
---------
Co-authored-by: 0xTylerHolmes <tyler@ethereum.org>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Makes clear the distinction between Finalize and FinalizedAndAssemble:
- In Finalize function, a series of state operations are applied according to consensus rules. The statedb is mutated and the root hash can be checked and compared afterwards.
This function should be used in block processing(receive afrom network and apply it locally) but not block generation.
- In FinalizeAndAssemble function, after applying state mutations, the block is also to be assembled with the latest
state root computed, updating the header.
This function should be used in block generation only.
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
* common, core, eth, les, trie: make prque generic
* les/vflux/server: fixed issues in priorityPool
* common, core, eth, les, trie: make priority also generic in prque
* les/flowcontrol: add test case for priority accumulator overflow
* les/flowcontrol: avoid priority value overflow
* common/prque: use int priority in some tests
No need to convert to int64 when we can just change the type used by the
queue.
* common/prque: remove comment about int64 range
---------
Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)
* Added changes per discussion in PR, more changes may be necessary
* Adding prerelease true
* Disabling goreleaser
* Removing README swap file
* change bor_dir and add bor user for v0.3.0 release
* rollback bor user and use root
* metrics: handle equal to separated config flag (#596)
* metrics: handle based config path
* internal/cli/server: add more context to logs
* use space separated flag and value in bor.service
* fixed static-nodes related buf (os independent) (#598)
* fixed static-nodes related buf (os independent)
* taking static-nodes as input if default not present
* Update default flags (#600)
* internal/cli/server: use geth's default for txpool.pricelimit and add comments
* builder/files: update config.toml for mainnet
* packaging/templates: update defaults for mainnet and mumbai
* internal/cli/server: skip overriding cache
* packaging/templates: update cache value for mainnet
* packaging/templates: update gcmode for archive mumbai node
* metrics: handle nil telemetry config (#601)
* resolve merge conflicts
* update go version in release.yml
* update goversion in makefile
* update Docker login for goreleaser-cross v1.19
* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist
* added a toml configuration file with comments describing each flag (#607)
* added a toml configuration file with comments describing each flag
* internal/cli/server: update flag description
* docs/cli: update example config and description of flags
* docs: update new-cli docs
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update
* added ancient datadir flag and toml field, need to decide on default value and update the conversion script
* updated toml files with ancient field
* Add support for new flags in new config.toml, which were present in old config.toml (#612)
* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml
* added RAW fields for these time.Duration flags
* updated the conversion script to support these extra 4 flags
* removed hcl and json config tests as we are only supporting toml config files
* updated toml files with cache.timeout field
* updated toml files with jsonrpc.timeouts field
* tests/bor: expect a call for latest checkpoint
* tests/bor: expect a call for latest checkpoint
* packaging/templates: update cache values for archive nodes
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
* remove unwanted code
* Fix docker publish authentication issue
In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.
* Revert "update Docker login for goreleaser-cross v1.19"
This reverts commit 4d19cf5342.
* Bump version to stable
* Revert "Merge pull request #435 from maticnetwork/POS-553"
This reverts commit 657d262def, reversing
changes made to 88dbfa1c13.
* revert change for release for go1.19
* Add default values to CLI helper and docs
This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.
* Add a summary of new CLI in docs
* Updating packager as binutils changed version so that apt-get installs current versions
* Add state pruning to new CLI
* Minor wording fix in prune state description
* Bumping control file versions
* Mainnet Delhi fork
* Set version to stable
* change delhi hardfork block number
* handle future chain import and skip peer drop (#650)
* handle future chain import and skip peer drop
* add block import metric
* params: bump version to v0.3.3-stable
* Bump bor version in control files for v0.3.3 mainnet release
Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
While investigating another issue, I found that all callers of collectLogs have the
complete block available. rawdb.ReadReceipts loads the block from the database,
so it is better to use ReadRawReceipts here, and derive the receipt information using
the block which is already in memory.
This PR makes it possible to modify the flush interval time via RPC. On one extreme, `0s`, it would act as an archive node. If set to `1h`, means that after one hour of effective block processing time, the trie would be flushed. If one block takes 200ms, this means that a flush would occur every `5*3600=18000` blocks -- however, if the memory size of the cached states grows too large, it will flush sooner.
Essentially, this makes it possible to configure the node to be more or less "archive:ish", and without restarting the node while reconfiguring it.
The gcproc field tracks the amount of time spent processing blocks,
and is used to trigger a state flush to disk when a certain threshold is
reached. After the merge, single block insertion by CL is the most
common source of block processing time, but this time was not added
into gcproc.
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
It seems there is no fully typed library implementation of an LRU cache.
So I wrote one. Method names are the same as github.com/hashicorp/golang-lru,
and the new type can be used as a drop-in replacement.
Two reasons to do this:
- It's much easier to understand what a cache is for when the types are right there.
- Performance: the new implementation is slightly faster and performs zero memory
allocations in Add when the cache is at capacity. Overall, memory usage of the cache
is much reduced because keys are values are no longer wrapped in interface.
This PR ports a few changes from PBSS:
- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
* ethclient/gethclient: improve time-sensitive flaky test
* eth/catalyst: fix (?) flaky test
* core: stop blockchains in tests after use
* core: fix dangling blockchain instances
* core: rm whitespace
* eth/gasprice, eth/tracers, consensus/clique: stop dangling blockchains in tests
* all: address review concerns
* core: goimports
* eth/catalyst: fix another time-sensitive test
* consensus/clique: add snapshot test run function
* core: rename stop() to stopWithoutSaving()
Co-authored-by: Felix Lange <fjl@twurst.com>
* base setup for miner opentel
* version change
* modify ctx passing
* add attributes
* update fill txs span attributes
* fix: use common attributes for remote and local txs
* pass context in seal
* fix
* fix
* add traces to finalize and assemble
* fix
* use task ctx in result loop
* only start parent span if no error
* send nil tracer from Finalize
* clean up
* add sub function timings in span attribute
* modify span attributes
* set time attribute to milliseconds
* linters fix
* fix linters for consensus
* add nolint to worker
* fix testcase
* Added fillTransactions subTraces
* add traces in intermediate root hash function
* add traces in WriteBlockAndSetHead function, fix linters
* fix: linting errors
* fix: test cases
* fix: go.mod
* fix: testcase
* extract tracing package
* linters fix
* debug
* Revert "debug"
This reverts commit 2d68b7c7b1105080563a4e1a6949dabc10acaff8.
* fix: panic in NewTransactionsByPriceAndNonce iteration
* change heimdall version to develop
* miner/worker: fix duplicate call to tx ordering
* miner/worker: refactor tracing
* consensus/bor: use tracing package
* tracing: add more abstraction for spans
* tracing: set all attributes at once
* remove nested tracing from blockchain.WriteBlockAndSetHead function
* remove nested tracing from statedb.IntermediateRoot function
* handle end span in bor.Seal function
* fix: typo
* minor fixes
* fix: linters
* fix: remove nolint
* go mod tidy
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
This PR reworks tx indexer a bit. Compared to the original version, one scenario is no longer handled - upgrading from legacy geth without indexer support.
The tx indexer was introduced in 2020 and have been present through hardforks, so it can be assumed that all Geth nodes have tx indexer already. So we can simplify the tx indexer logic a bit:
- If the tail flag is not present, it means node is just initialized may or may not with an ancient store attached. In this case all blocks are regarded as unindexed
- If the tail flag is present, it means blocks below tail are unindexed, blocks above tail are indexed
This change also address some weird cornercases that could make the indexer not work after a crash.
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
This shortens the chain config summary in bad block reports,
and adds go-ethereum version information as well.
Co-authored-by: Felix Lange <fjl@twurst.com>
