go-ethereum/internal/cli/server/service.go
marcello33 8667ac17cd
merge v0.3.9-alpha to qa (#824)
* commit logs

* CI: test launch devnet without hardcoded sleep time (pos-534)

* CI: try using checked out bor path

* CI: fix missing ;

* CI: fix assignment operator

* CI: echo peers and block no.

* CI: cleanup

* minor chg: add new line

* dev: add: pos-944: snyk and govuln integration (#578)

* dev: add: pos-944 security ci and readme

* dev: add: pos-944 remove linters as this is included already in build ci

* dev: chg: pos-947 dependencies upgrade to solve snyk security issues

* dev: chg: update security-ci

* dev: chg: remove linter to allow replacements for security issues

* dev: add: pos-944 verify path when updating metrics from config

* dev: add: pos-944 fix linter

* dev: add: pos-944 add .snyk policy file / fix snyk code vulnerabilities

* dev: fix: pos-944 import common package / gitignore snyk dccache file

* dev: fix: pos-944 verify canonical path for crashers

* dev: fix: pos-944 linter

* dev: add: pos-976 add govuln check

* dev: add: pos-976 test upload with permissions

* dev: add: pos-976 remove duplicated upload

* dev: add: pos-976 report upload

* dev: add: pos-976 remove upload

* dev: fix: pos-944 fix govuln action

* dev: fix: pos-944 move govulncheck to security-ci

* dev: fix: pos-944 bump golvun action and golang versions

* dev: fix: pos-944 remove persmissions and fix conflicts

* dev: chg: restore err msg

* dev: chg: remove duplicated function

* dev: chg: sort import

* dev: chg: fix linter

* dev: add: use common VerifyCrasher function to avoid duplications / replace deprecated ioutils.ReadFile

* dev: fix: typo

* fix linters

* upgrade grpc version

* add ignore rule for net/http2

* Shivam/txpool tracing (#604)

* lock, unlock to rlock, runlock

* add : tracing Pending() and Locals()

* Log time spent in committing a tx during mining

* Remove data from logging

* Move log into case where a tx completes without error

* profile fillTransactions

* fix conflict

* bug fixes

* add logs

* txpool: add tracing in Pending()

* rearrange tracing

* add attributes

* fix

* fix

* log error in profiling

* update file mode and file path for profiling

* full profiling

* fix log

* fix log

* less wait

* fix

* fix

* logs

* worker: use block number for prof files

* initial

* txList add

* fix gas calculation

* fix

* green tests

* linters

* prettify

* allocate less

* no locks between pending and reorg

* no locks

* no locks on locals

* more tests

* linters

* less allocs

* comment

* optimize errors

* linters

* fix

* fix

* Linters

* linters

* linters

* simplify errors

* atomics for transactions

* fix

* optimize workers

* fix copy

* linters

* txpool tracing

* linters

* fix tracing

* duration in mcs

* locks

* metrics

* fix

* cache hit/miss

* less locks on evict

* remove once

* remove once

* switch off pprof

* fix data race

* fix data race

* add : sealed total/empty blocks metric gauge

* add : RPC debug_getTraceStack

* fix : RPC debug_getTraceStack

* fix : RPC debug_getTraceStack for all go-routines

* linters

* add data race test on txpool

* fix concurrency

* noleak

* increase batch size

* prettify

* tests

* baseFee mutex

* panic fix

* linters

* fix gas fee data race

* linters

* more transactions

* debug

* debug

* fix ticker

* fix test

* add cacheMu

* more tests

* fix test panic

* linters

* add statistics

* add statistics

* txitems data race

* fix tx list Has

* fix : lint

Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Reduce txArriveTimeout to 100ms

* init : remove exit on keystore err

* add : multiple keystore tolerance

* lint : fix linters

* chg : use standard logging

* chg : logging strings

* Added flags to run heimdall as a child process (#597)

* Added flags to run heimdall as a child process

* Fix: Lint

* Fix btcd package dependency for CI

* Update btcd package version

* Try removing ambigious importts

* dev: fix: lint fix for parallel tests

* remove delete for ambigious import

* Remove unwanted space

* go mod tidy

* try replace

* try replace

* use vendor

* rename vendor

* tidy

* vendor btcec

* clean up

* remove submodule

* remove submodule

* remove submodule

* remove submodule

* remove vendor & added replacer in test

* go mod tidy

* added replacer

* Update replacer

* Update replacer

* Merge branch 'develop' into run-heimdall-flags

* Skip TestGolangBindings

* Typo fix

* Remove unwanted changes

Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* dev: chg: update PR template to include nodes audience check (#641)

* dev: chg: update PR template to include nodes audience check

* dev: chg: better description

* dev: chg: add entry to changes too

* sonarqube integration (#658)

* dev: add: sonarqube integration into security-ci

* dev: add: exclude java files from sonarqube analysis

* Merge branch 'qa' and 'master' into develop (#663)

* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)

* Added changes per discussion in PR, more changes may be necessary

* Adding prerelease true

* Disabling goreleaser

* Removing README swap file

* change bor_dir and add bor user for v0.3.0 release

* rollback bor user and use root

* metrics: handle equal to separated config flag (#596)

* metrics: handle  based config path

* internal/cli/server: add more context to logs

* use space separated flag and value in bor.service

* fixed static-nodes related buf (os independent) (#598)

* fixed static-nodes related buf (os independent)

* taking static-nodes as input if default not present

* Update default flags (#600)

* internal/cli/server: use geth's default for txpool.pricelimit and add comments

* builder/files: update config.toml for mainnet

* packaging/templates: update defaults for mainnet and mumbai

* internal/cli/server: skip overriding cache

* packaging/templates: update cache value for mainnet

* packaging/templates: update gcmode for archive mumbai node

* metrics: handle nil telemetry config (#601)

* resolve merge conflicts

* update go version in release.yml

* update goversion in makefile

* update Docker login for goreleaser-cross v1.19

* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist

* added a toml configuration file with comments describing each flag (#607)

* added a toml configuration file with comments describing each flag

* internal/cli/server: update flag description

* docs/cli: update example config and description of flags

* docs: update new-cli docs

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update

* added ancient datadir flag and toml field, need to decide on default value and update the conversion script

* updated toml files with ancient field

* Add support for new flags in new config.toml, which were present in old config.toml (#612)

* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml

* added RAW fields for these time.Duration flags

* updated the conversion script to support these extra 4 flags

* removed hcl and json config tests as we are only supporting toml config files

* updated toml files with cache.timeout field

* updated toml files with jsonrpc.timeouts field

* tests/bor: expect a call for latest checkpoint

* tests/bor: expect a call for latest checkpoint

* packaging/templates: update cache values for archive nodes

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* remove unwanted code

* Fix docker publish authentication issue

In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.

* Revert "update Docker login for goreleaser-cross v1.19"

This reverts commit 4d19cf5342.

* Bump version to stable

* Revert "Merge pull request #435 from maticnetwork/POS-553"

This reverts commit 657d262def, reversing
changes made to 88dbfa1c13.

* revert change for release for go1.19

* Add default values to CLI helper and docs

This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.

* Add a summary of new CLI in docs

* Updating packager as binutils changed version so that apt-get installs current versions

* Add state pruning to new CLI

* Minor wording fix in prune state description

* Bumping control file versions

* Mainnet Delhi fork

* Set version to stable

* change delhi hardfork block number

* handle future chain import and skip peer drop (#650)

* handle future chain import and skip peer drop

* add block import metric

* params: bump version to v0.3.3-stable

* Bump bor version in control files for v0.3.3 mainnet release

Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>

* CI: use matic-cli master branch

* trigger ci

* internal/cli/server : fix : added triesInMemory in config (#677)

* update requirements in README (#681)

* consensus/bor : add : devFakeAuthor flag

* core,eth,internal/cli,internal/ethapi: add --rpc.allow-unprotected-txs flag to allow txs to get replayed (for shadow node)

* internal/cli: add `skiptrace` flag for profiling (#715)

* internal/cli: add skiptrace flag

* docs: update docs for skiptrace flag

* Added flag in Status to wait for backend, and fixed panic issue. (#708)

* checking if backend is available during status call, and added a flag to wait if backend is not available

* added test for status command (does not cover the whole code)

* Revert "Reduce txArriveTimeout to 100ms" (#707)

This reverts commit 243d231fe4.

* consensus/bor : add : devFakeAuthor flag (#697)

* add check for empty lists in txpool (#704)

* add check

* linters

* dev: chg: POS-215 move sonarqube to own ci (#733)

* Added verbosity flag, supports log-level as well, but will remove that in future. (#722)

* changed log-level flag back to verbosity, and updated the conversion script

* supporting both verbosity and log-level, with a message to deprecat log-level

* converted verbosity to a int value

* Check if block is nil to prevent panic (#736)

* miner: use env for tracing instead of block object (#728)

* Add : mutex pprof profile (#731)

* add : mutex pprof profile

* rm : remove trace from default pprof profiles

* chg : commit tx logs from info to debug (#673)

* chg : commit tx logs from info to debug

* fix : minor changes

* chg : miner : commitTransactions-stats moved from info to debug

* lint : fix linters

* refactor logging

* miner : chg : UnauthorizedSignerError to debug

* lint : fix lint

* fix : log.Logger interface compatibility

---------

Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Add : commit details to bor version (#730)

* add : commit details to bor version

* fix : MAKEFILE

* undo : rm test-txpool-race

* rm : params/build_date

* rm : params/gitbranch and params/gitdate

* core,docs/cli,internal/cli/server: make docs

* builder,docs/cli,packaging: update toml files

* mardizzone/hotfix-snyk: remove vcs build when running snyk (#745)

* Feat : SetMaxPeers (#726)

* init : add admin.setMaxPeers and getMaxPeers

* lint : fix lint

* add : some comments

* Update wiki link (#762)

* Heimdall App implementation (#646)

* added support for miner.recommit flag (#743)

* interrupting commit experiment (#556)

* initial

* delete

* linters

* big benchmark

* benchmark big ints

* delay

* fix generate

* remove debug

* miner : chg : remove noempty check

* fix lints

* consensus/bor: handle unauthorized signer in consensus.Prepare (#651)

* fix : break loop fix

* lint : fix

* lint : more lint fix

* fix : skip TestEmptyWorkEthash and TestEmptyWorkClique

* add : metrics commitInterruptCounter

---------

Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* internal/cli: added missing flags (#744)

* minor comment update

* added support for rpc.evmtimeout flag

* added support for vmdebug (EnablePreimageRecording) flag

* added support for jsonrpc.auth.(jwtsecret, addr, port, vhosts) flags

* added support for miner.recommit flag

* added support for gpo.maxheaderhistory and gpo.maxblockhistory flag

* Revert "added support for miner.recommit flag"

This reverts commit fcb722e4bf5bdda23d4c91da40ab95c9023a80c3.

* added pprof related flags (expect --pprof.cpuprofile - Write CPU profile to the given file)

* added support for --dev.gaslimit flag

* added support for --fdlimit flag

* added support for --netrestrict flag

* added support for --nodekey and --nodekeyhex flag

* added support for --vmodule, --log.json, --log.backtrace, and --log.debug flags

* fixed related lint errors

* fix lints from develop (few lints decided to appear from code that was untouched, weird)

* more weird lints from develop

* small precautionary fix

* small bug ;) fix in NetRestrict

* weird lints

* change vmdebug = true to vmdebug = false

* internal/ethapi: set rpc gas cap as default gas limit when creating access list

* add : testcase for CommitInterruptExperiment (#792)

* add : TestCommitInterruptExperimentBor

* lint : fix lint

* chg : add waitgroup

* dev: chg: continue on error when uploading snyk results to GH (#795)

* consensus/bor: revert handle unauthorized signer in consensus.Prepare

* Revert "consensus/bor: handle unauthorized signer in consensus.Prepare (#651)"

This reverts commit 9ce8c7de75a1b5021bb6f2b8086e813c8244b22c.

* eth/downloader/whitelist: skip future chain validation (#796)

* eth/downloader/whitelist: skip future chain validation

* eth/downloader/whitelist: fix tests for future chain import

* Setting up bor to use hosted 18.04 runner as ubuntu provided 18.04 runner is end of life

* Merge qa to develop (#814)

* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)

* Added changes per discussion in PR, more changes may be necessary

* Adding prerelease true

* Disabling goreleaser

* Removing README swap file

* change bor_dir and add bor user for v0.3.0 release

* rollback bor user and use root

* metrics: handle equal to separated config flag (#596)

* metrics: handle  based config path

* internal/cli/server: add more context to logs

* use space separated flag and value in bor.service

* fixed static-nodes related buf (os independent) (#598)

* fixed static-nodes related buf (os independent)

* taking static-nodes as input if default not present

* Update default flags (#600)

* internal/cli/server: use geth's default for txpool.pricelimit and add comments

* builder/files: update config.toml for mainnet

* packaging/templates: update defaults for mainnet and mumbai

* internal/cli/server: skip overriding cache

* packaging/templates: update cache value for mainnet

* packaging/templates: update gcmode for archive mumbai node

* metrics: handle nil telemetry config (#601)

* resolve merge conflicts

* update go version in release.yml

* update goversion in makefile

* update Docker login for goreleaser-cross v1.19

* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist

* added a toml configuration file with comments describing each flag (#607)

* added a toml configuration file with comments describing each flag

* internal/cli/server: update flag description

* docs/cli: update example config and description of flags

* docs: update new-cli docs

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update

* added ancient datadir flag and toml field, need to decide on default value and update the conversion script

* updated toml files with ancient field

* Add support for new flags in new config.toml, which were present in old config.toml (#612)

* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml

* added RAW fields for these time.Duration flags

* updated the conversion script to support these extra 4 flags

* removed hcl and json config tests as we are only supporting toml config files

* updated toml files with cache.timeout field

* updated toml files with jsonrpc.timeouts field

* tests/bor: expect a call for latest checkpoint

* tests/bor: expect a call for latest checkpoint

* packaging/templates: update cache values for archive nodes

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* remove unwanted code

* Fix docker publish authentication issue

In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.

* Revert "update Docker login for goreleaser-cross v1.19"

This reverts commit 4d19cf5342.

* Bump version to stable

* Revert "Merge pull request #435 from maticnetwork/POS-553"

This reverts commit 657d262def, reversing
changes made to 88dbfa1c13.

* revert change for release for go1.19

* Add default values to CLI helper and docs

This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.

* Add a summary of new CLI in docs

* Updating packager as binutils changed version so that apt-get installs current versions

* Add state pruning to new CLI

* Minor wording fix in prune state description

* Bumping control file versions

* Mainnet Delhi fork

* Set version to stable

* change delhi hardfork block number

* handle future chain import and skip peer drop (#650)

* handle future chain import and skip peer drop

* add block import metric

* params: bump version to v0.3.3-stable

* Bump bor version in control files for v0.3.3 mainnet release

* Added checks to RPC requests and introduced new flags to customise the parameters (#657)

* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)

* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)

* updated the config files and docs

* chg : trieTimeout from 60 to 10 mins (#692)

* chg : trieTimeout from 60 to 10 mins

* chg : cache.timout to 10m from 1h in configs

* internal/cli/server : fix : added triesInMemory in config (#691)

* changed version from 0.3.0 to 0.3.4-beta (#693)

* fix nil state-sync issue, increase grpc limit (#695)

* Increase grpc message size limit in pprof

* consensus/bor/bor.go : stateSyncs init fixed [Fix #686]

* eth/filters: handle nil state-sync before notify

* eth/filters: update check

Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>

* core, tests/bor: add more tests for state-sync validation (#710)

* core: add get state sync function for tests

* tests/bor: add validation for state sync events post consensus

* Arpit/temp bor sync (#701)

* Increase grpc message size limit in pprof

* ReadBorReceipts improvements

* use internal function

* fix tests

* fetch geth upstread for ReadBorReceiptRLP

* Only query bor receipt when the query index is equal to # tx in block body

This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.

* Revert "fetch geth upstread for ReadBorReceiptRLP"

This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.

* Restore ReadBorReceiptRLP

* fix bor receipts

* remove unused

* fix lints

---------

Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)

This reverts commit 241843c7e7.

* Arpit/add execution pool 2 (#719)

* initial

* linters

* linters

* remove timeout

* update pool

* change pool size function

* check nil

* check nil

* fix tests

* Use execution pool from server in all handlers

* simplify things

* test fix

* add support for cli, config

* add to cli and config

* merge base branch

* debug statements

* fix bug

* atomic pointer timeout

* add apis

* update workerpool

* fix issues

* change params

* fix issues

* fix ipc issue

* remove execution pool from IPC

* revert

* fix tests

* mutex

* refactor flag and value names

* ordering fix

* refactor flag and value names

* update default ep size to 40

* fix bor start issues

* revert file changes

* debug statements

* fix bug

* update workerpool

* atomic pointer timeout

* add apis

* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* fix issues

* change params

* fix issues

* fix ipc issue

* remove execution pool from IPC

* revert

* merge base branch

* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* mutex

* fix tests

* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* Change default size of execution pool to 40

* refactor flag and value names

* fix merge conflicts

* ordering fix

* refactor flag and value names

* update default ep size to 40

* fix bor start issues

* revert file changes

* fix linters

* fix go.mod

* change sec to ms

* change default value for ep timeout

* fix node api calls

* comment setter for ep timeout

---------

Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* version change (#721)

* Event based pprof (#732)

* feature

* Save pprof to /tmp

---------

Co-authored-by: Jerry <jerrycgh@gmail.com>

* Cherry-pick changes from develop (#738)

* Check if block is nil to prevent panic (#736)

* miner: use env for tracing instead of block object (#728)

---------

Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>

* add max code init size check in txpool (#739)

* Revert "Event based pprof" and update version (#742)

* Revert "Event based pprof (#732)"

This reverts commit 22fa4033e8.

* params: update version to 0.3.4-beta3

* packaging/templates: update bor version

* internal/ethapi :: Fix : newRPCTransactionFromBlockIndex

* Merge master to qa (#813)

* Merge qa to master (#750)

* Added checks to RPC requests and introduced new flags to customise the parameters (#657)

* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)

* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)

* updated the config files and docs

* chg : trieTimeout from 60 to 10 mins (#692)

* chg : trieTimeout from 60 to 10 mins

* chg : cache.timout to 10m from 1h in configs

* internal/cli/server : fix : added triesInMemory in config (#691)

* changed version from 0.3.0 to 0.3.4-beta (#693)

* fix nil state-sync issue, increase grpc limit (#695)

* Increase grpc message size limit in pprof

* consensus/bor/bor.go : stateSyncs init fixed [Fix #686]

* eth/filters: handle nil state-sync before notify

* eth/filters: update check

Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>

* core, tests/bor: add more tests for state-sync validation (#710)

* core: add get state sync function for tests

* tests/bor: add validation for state sync events post consensus

* Arpit/temp bor sync (#701)

* Increase grpc message size limit in pprof

* ReadBorReceipts improvements

* use internal function

* fix tests

* fetch geth upstread for ReadBorReceiptRLP

* Only query bor receipt when the query index is equal to # tx in block body

This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.

* Revert "fetch geth upstread for ReadBorReceiptRLP"

This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.

* Restore ReadBorReceiptRLP

* fix bor receipts

* remove unused

* fix lints

---------

Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)

This reverts commit 241843c7e7.

* Arpit/add execution pool 2 (#719)

* initial

* linters

* linters

* remove timeout

* update pool

* change pool size function

* check nil

* check nil

* fix tests

* Use execution pool from server in all handlers

* simplify things

* test fix

* add support for cli, config

* add to cli and config

* merge base branch

* debug statements

* fix bug

* atomic pointer timeout

* add apis

* update workerpool

* fix issues

* change params

* fix issues

* fix ipc issue

* remove execution pool from IPC

* revert

* fix tests

* mutex

* refactor flag and value names

* ordering fix

* refactor flag and value names

* update default ep size to 40

* fix bor start issues

* revert file changes

* debug statements

* fix bug

* update workerpool

* atomic pointer timeout

* add apis

* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* fix issues

* change params

* fix issues

* fix ipc issue

* remove execution pool from IPC

* revert

* merge base branch

* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* mutex

* fix tests

* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* Change default size of execution pool to 40

* refactor flag and value names

* fix merge conflicts

* ordering fix

* refactor flag and value names

* update default ep size to 40

* fix bor start issues

* revert file changes

* fix linters

* fix go.mod

* change sec to ms

* change default value for ep timeout

* fix node api calls

* comment setter for ep timeout

---------

Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* version change (#721)

* Event based pprof (#732)

* feature

* Save pprof to /tmp

---------

Co-authored-by: Jerry <jerrycgh@gmail.com>

* Cherry-pick changes from develop (#738)

* Check if block is nil to prevent panic (#736)

* miner: use env for tracing instead of block object (#728)

---------

Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>

* add max code init size check in txpool (#739)

* Revert "Event based pprof" and update version (#742)

* Revert "Event based pprof (#732)"

This reverts commit 22fa4033e8.

* params: update version to 0.3.4-beta3

* packaging/templates: update bor version

* params, packaging/templates: update bor version

---------

Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>

* core, miner: add sub-spans for tracing (#753)

* core, miner: add sub-spans for tracing

* fix linters

* core: add logs for debugging

* core: add more logs to print tdd while reorg

* fix linters

* core: minor fix

* core: remove debug logs

* core: use different span for write block and set head

* core: use internal context for sending traces (#755)

* core: add : impossible reorg block dump (#754)

* add : impossible reorg block dump

* chg : 3 seperate files for impossoble reorg dump

* add : use exportBlocks method and RLP blocks before writing

* chg : small changes

* bump : go version from 1.19 to 1.20.1 (#761)

* Revert "bump : go version from 1.19 to 1.20.1 (#761)"

This reverts commit 4561012af9.

* core/vm: use optimized bigint (#26021)

* Add holiman/big

* Fix linter

* Bump version to v0.3.5

* fix lints from develop (few lints decided to appear from code that was untouched, weird)

* upgrade crypto lib version (#770)

* bump dep : github.com/Masterminds/goutils to v1.1.1 (#769)

* mardizzone/pos-1313: bump crypto dependency (#772)

* dev: chg: bumd net dependency

* dev: chg: bump crypto dependency

* dev: chg: bump crypto dependency

* bump dep : golang.org/x/net to v0.8.0 (#771)

* Verify validator set against local contract on receiving an end-of-sprint block (#768)

* Verify validator set against local contract on receiving an end-of-sprint block

* Fix tests

* Respect error returned by ParseValidators

* Keep going back until a parent block presents

* core/txpool: implement DoS defenses from geth (#778)

* Hotfixes and deps bump (#776)

* dev: chg: bump deps

* internal/cli/server, rpc: lower down http readtimeout to 10s

* dev: chg: get p2p adapter

* dev: chg: lower down jsonrpc readtimeout to 10s

* cherry-pick txpool optimisation changes

* add check for empty lists in txpool (#704)

* add check

* linters

* core, miner: add empty instrumentation name for tracing

---------

Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* packaging,params: bump to v0.3.6 (#782)

* v0.3.6 fix (#787)

* Fix get validator set in header verifier

* chg : commit tx logs from info to debug (#673)

* chg : commit tx logs from info to debug

* fix : minor changes

* chg : miner : commitTransactions-stats moved from info to debug

* lint : fix linters

* refactor logging

* miner : chg : UnauthorizedSignerError to debug

* lint : fix lint

* fix : log.Logger interface compatibility

---------

Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Remove unnecessary sorting of valset from header in verification

* dev: chg: version bump

---------

Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>

* core: improve locks in txpool (#807)

* added a write lock to the txs.filter method and a read lock to the txs.reheap method - both of which are called by Filter during reorg adjustments to txpool

* txpool reorg locks

* more locks

* locks

* linters

* params, packaging: update version for v0.3.8-beta release

* core: add logs in reheap

---------

Co-authored-by: Alex <dalexwatts@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Merge qa to master (#808)

* Added checks to RPC requests and introduced new flags to customise the parameters (#657)

* added a check to reject rpc requests with batch size > the one set using a newly added flag (rpcbatchlimit)

* added a check to reject rpc requests whose result size > the one set using a newly added flag (rpcreturndatalimit)

* updated the config files and docs

* chg : trieTimeout from 60 to 10 mins (#692)

* chg : trieTimeout from 60 to 10 mins

* chg : cache.timout to 10m from 1h in configs

* internal/cli/server : fix : added triesInMemory in config (#691)

* changed version from 0.3.0 to 0.3.4-beta (#693)

* fix nil state-sync issue, increase grpc limit (#695)

* Increase grpc message size limit in pprof

* consensus/bor/bor.go : stateSyncs init fixed [Fix #686]

* eth/filters: handle nil state-sync before notify

* eth/filters: update check

Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>

* core, tests/bor: add more tests for state-sync validation (#710)

* core: add get state sync function for tests

* tests/bor: add validation for state sync events post consensus

* Arpit/temp bor sync (#701)

* Increase grpc message size limit in pprof

* ReadBorReceipts improvements

* use internal function

* fix tests

* fetch geth upstread for ReadBorReceiptRLP

* Only query bor receipt when the query index is equal to # tx in block body

This change reduces the frequency of calling ReadBorReceipt and
ReadBorTransaction, which are CPU and db intensive.

* Revert "fetch geth upstread for ReadBorReceiptRLP"

This reverts commit 2e838a6b1313d26674f3a8df4b044e35dcbf35a0.

* Restore ReadBorReceiptRLP

* fix bor receipts

* remove unused

* fix lints

---------

Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>

* Revert "chg : trieTimeout from 60 to 10 mins (#692)" (#720)

This reverts commit 241843c7e7.

* Arpit/add execution pool 2 (#719)

* initial

* linters

* linters

* remove timeout

* update pool

* change pool size function

* check nil

* check nil

* fix tests

* Use execution pool from server in all handlers

* simplify things

* test fix

* add support for cli, config

* add to cli and config

* merge base branch

* debug statements

* fix bug

* atomic pointer timeout

* add apis

* update workerpool

* fix issues

* change params

* fix issues

* fix ipc issue

* remove execution pool from IPC

* revert

* fix tests

* mutex

* refactor flag and value names

* ordering fix

* refactor flag and value names

* update default ep size to 40

* fix bor start issues

* revert file changes

* debug statements

* fix bug

* update workerpool

* atomic pointer timeout

* add apis

* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* fix issues

* change params

* fix issues

* fix ipc issue

* remove execution pool from IPC

* revert

* merge base branch

* Merge branch 'add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* mutex

* fix tests

* Merge branch 'arpit/add-execution-pool' of github.com:maticnetwork/bor into arpit/add-execution-pool

* Change default size of execution pool to 40

* refactor flag and value names

* fix merge conflicts

* ordering fix

* refactor flag and value names

* update default ep size to 40

* fix bor start issues

* revert file changes

* fix linters

* fix go.mod

* change sec to ms

* change default value for ep timeout

* fix node api calls

* comment setter for ep timeout

---------

Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* version change (#721)

* Event based pprof (#732)

* feature

* Save pprof to /tmp

---------

Co-authored-by: Jerry <jerrycgh@gmail.com>

* Cherry-pick changes from develop (#738)

* Check if block is nil to prevent panic (#736)

* miner: use env for tracing instead of block object (#728)

---------

Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>

* add max code init size check in txpool (#739)

* Revert "Event based pprof" and update version (#742)

* Revert "Event based pprof (#732)"

This reverts commit 22fa4033e8.

* params: update version to 0.3.4-beta3

* packaging/templates: update bor version

* internal/ethapi :: Fix : newRPCTransactionFromBlockIndex

* fix: remove assignment for bor receipt

---------

Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>

* Setting up bor to use hosted 18.04 runner as ubuntu provided 18.04 runner is end of life

---------

Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Alex <dalexwatts@gmail.com>
Co-authored-by: Daniel Jones <djones@polygon.technology>

* core: remove duplicate tests

* miner: use get validators by hash in tests

---------

Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: marcello33 <marcelloardizzone@hotmail.it>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Alex <dalexwatts@gmail.com>

---------

Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Raneet Debnath <raneetdebnath10@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: builder90210 <anonbuilder90210@gmail.com>
Co-authored-by: Krishna Upadhyaya <KRISHNAU1604@GMAIL.COM>
Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Raneet Debnath <35629432+Raneet10@users.noreply.github.com>
Co-authored-by: Didi <github@d10r.net>
Co-authored-by: ephess <e+github@ephess.nz>
Co-authored-by: Dmitry <46797839+dkeysil@users.noreply.github.com>
Co-authored-by: Daniil <daniil.melnik@chainstack.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Alex <dalexwatts@gmail.com>
2023-04-17 12:53:39 +02:00

333 lines
8.1 KiB
Go

package server
import (
"context"
"encoding/json"
"errors"
"fmt"
"math/big"
"reflect"
"strings"
"time"
grpc_net_conn "github.com/JekaMas/go-grpc-net-conn"
"github.com/ethereum/go-ethereum/core"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/eth/tracers"
"github.com/ethereum/go-ethereum/eth/tracers/logger"
"github.com/ethereum/go-ethereum/internal/cli/server/pprof"
"github.com/ethereum/go-ethereum/internal/cli/server/proto"
"github.com/ethereum/go-ethereum/p2p"
"github.com/ethereum/go-ethereum/p2p/enode"
)
const chunkSize = 1024 * 1024 * 1024
var ErrUnavailable = errors.New("bor service is currently unavailable, try again later")
var ErrUnavailable2 = errors.New("bor service unavailable even after waiting for 10 seconds, make sure bor is running")
func sendStreamDebugFile(stream proto.Bor_DebugPprofServer, headers map[string]string, data []byte) error {
// open the stream and send the headers
err := stream.Send(&proto.DebugFileResponse{
Event: &proto.DebugFileResponse_Open_{
Open: &proto.DebugFileResponse_Open{
Headers: headers,
},
},
})
if err != nil {
return err
}
// Wrap our conn around the response.
encoder := grpc_net_conn.SimpleEncoder(func(msg *proto.DebugFileResponse_Input) *[]byte {
return &msg.Data
})
conn := &grpc_net_conn.Conn[*proto.DebugFileResponse_Input, *proto.DebugFileResponse_Input]{
Stream: stream,
Request: &proto.DebugFileResponse_Input{},
Encode: grpc_net_conn.ChunkedEncoder(encoder, chunkSize),
}
if _, err := conn.Write(data); err != nil {
return err
}
// send the eof
err = stream.Send(&proto.DebugFileResponse{
Event: &proto.DebugFileResponse_Eof{},
})
if err != nil {
return err
}
return nil
}
func (s *Server) DebugPprof(req *proto.DebugPprofRequest, stream proto.Bor_DebugPprofServer) error {
var (
payload []byte
headers map[string]string
err error
)
ctx := context.Background()
switch req.Type {
case proto.DebugPprofRequest_CPU:
payload, headers, err = pprof.CPUProfile(ctx, int(req.Seconds))
case proto.DebugPprofRequest_TRACE:
payload, headers, err = pprof.Trace(ctx, int(req.Seconds))
case proto.DebugPprofRequest_LOOKUP:
payload, headers, err = pprof.Profile(req.Profile, 0, 0)
}
if err != nil {
return err
}
// send the file on a grpc stream
if err := sendStreamDebugFile(stream, headers, payload); err != nil {
return err
}
return nil
}
func (s *Server) PeersAdd(ctx context.Context, req *proto.PeersAddRequest) (*proto.PeersAddResponse, error) {
node, err := enode.Parse(enode.ValidSchemes, req.Enode)
if err != nil {
return nil, fmt.Errorf("invalid enode: %v", err)
}
srv := s.node.Server()
if req.Trusted {
srv.AddTrustedPeer(node)
} else {
srv.AddPeer(node)
}
return &proto.PeersAddResponse{}, nil
}
func (s *Server) PeersRemove(ctx context.Context, req *proto.PeersRemoveRequest) (*proto.PeersRemoveResponse, error) {
node, err := enode.Parse(enode.ValidSchemes, req.Enode)
if err != nil {
return nil, fmt.Errorf("invalid enode: %v", err)
}
srv := s.node.Server()
if req.Trusted {
srv.RemoveTrustedPeer(node)
} else {
srv.RemovePeer(node)
}
return &proto.PeersRemoveResponse{}, nil
}
func (s *Server) PeersList(ctx context.Context, req *proto.PeersListRequest) (*proto.PeersListResponse, error) {
resp := &proto.PeersListResponse{}
peers := s.node.Server().PeersInfo()
for _, p := range peers {
resp.Peers = append(resp.Peers, peerInfoToPeer(p))
}
return resp, nil
}
func (s *Server) PeersStatus(ctx context.Context, req *proto.PeersStatusRequest) (*proto.PeersStatusResponse, error) {
var peerInfo *p2p.PeerInfo
for _, p := range s.node.Server().PeersInfo() {
if strings.HasPrefix(p.ID, req.Enode) {
if peerInfo != nil {
return nil, fmt.Errorf("more than one peer with the same prefix")
}
peerInfo = p
}
}
resp := &proto.PeersStatusResponse{}
if peerInfo != nil {
resp.Peer = peerInfoToPeer(peerInfo)
}
return resp, nil
}
func peerInfoToPeer(info *p2p.PeerInfo) *proto.Peer {
return &proto.Peer{
Id: info.ID,
Enode: info.Enode,
Enr: info.ENR,
Caps: info.Caps,
Name: info.Name,
Trusted: info.Network.Trusted,
Static: info.Network.Static,
}
}
func (s *Server) ChainSetHead(ctx context.Context, req *proto.ChainSetHeadRequest) (*proto.ChainSetHeadResponse, error) {
s.backend.APIBackend.SetHead(req.Number)
return &proto.ChainSetHeadResponse{}, nil
}
func (s *Server) Status(ctx context.Context, in *proto.StatusRequest) (*proto.StatusResponse, error) {
if s.backend == nil && !in.Wait {
return nil, ErrUnavailable
}
// check for s.backend at an interval of 2 seconds
// wait for a maximum of 10 seconds (5 iterations)
if s.backend == nil && in.Wait {
i := 1
for {
time.Sleep(2 * time.Second)
if s.backend == nil {
if i == 5 {
return nil, ErrUnavailable2
}
} else {
break
}
i++
}
}
apiBackend := s.backend.APIBackend
syncProgress := apiBackend.SyncProgress()
resp := &proto.StatusResponse{
CurrentHeader: headerToProtoHeader(apiBackend.CurrentHeader()),
CurrentBlock: headerToProtoHeader(apiBackend.CurrentBlock().Header()),
NumPeers: int64(len(s.node.Server().PeersInfo())),
SyncMode: s.config.SyncMode,
Syncing: &proto.StatusResponse_Syncing{
StartingBlock: int64(syncProgress.StartingBlock),
HighestBlock: int64(syncProgress.HighestBlock),
CurrentBlock: int64(syncProgress.CurrentBlock),
},
Forks: gatherForks(s.config.chain.Genesis.Config, s.config.chain.Genesis.Config.Bor),
}
return resp, nil
}
func headerToProtoHeader(h *types.Header) *proto.Header {
return &proto.Header{
Hash: h.Hash().String(),
Number: h.Number.Uint64(),
}
}
func (s *Server) DebugBlock(req *proto.DebugBlockRequest, stream proto.Bor_DebugBlockServer) error {
traceReq := &tracers.TraceBlockRequest{
Number: req.Number,
Config: &tracers.TraceConfig{
Config: &logger.Config{
EnableMemory: true,
},
},
}
res, err := s.tracerAPI.TraceBorBlock(traceReq)
if err != nil {
return err
}
// this is memory heavy
data, err := json.Marshal(res)
if err != nil {
return err
}
if err := sendStreamDebugFile(stream, map[string]string{}, data); err != nil {
return err
}
return nil
}
var bigIntT = reflect.TypeOf(new(big.Int)).Kind()
// gatherForks gathers all the fork numbers via reflection
func gatherForks(configList ...interface{}) []*proto.StatusResponse_Fork {
var forks []*proto.StatusResponse_Fork
for _, config := range configList {
kind := reflect.TypeOf(config)
for kind.Kind() == reflect.Ptr {
kind = kind.Elem()
}
skip := "DAOForkBlock"
conf := reflect.ValueOf(config).Elem()
for i := 0; i < kind.NumField(); i++ {
// Fetch the next field and skip non-fork rules
field := kind.Field(i)
if strings.Contains(field.Name, skip) {
continue
}
if !strings.HasSuffix(field.Name, "Block") {
continue
}
fork := &proto.StatusResponse_Fork{
Name: strings.TrimSuffix(field.Name, "Block"),
}
val := conf.Field(i)
switch field.Type.Kind() {
case bigIntT:
rule := val.Interface().(*big.Int)
if rule != nil {
fork.Block = rule.Int64()
} else {
fork.Disabled = true
}
case reflect.Uint64:
fork.Block = int64(val.Uint())
default:
continue
}
forks = append(forks, fork)
}
}
return forks
}
func convertBlockToBlockStub(blocks []*types.Block) []*proto.BlockStub {
var blockStubs []*proto.BlockStub
for _, block := range blocks {
blockStub := &proto.BlockStub{
Hash: block.Hash().String(),
Number: block.NumberU64(),
}
blockStubs = append(blockStubs, blockStub)
}
return blockStubs
}
func (s *Server) ChainWatch(req *proto.ChainWatchRequest, reply proto.Bor_ChainWatchServer) error {
chain2HeadChanSize := 10
chain2HeadCh := make(chan core.Chain2HeadEvent, chain2HeadChanSize)
headSub := s.backend.APIBackend.SubscribeChain2HeadEvent(chain2HeadCh)
defer headSub.Unsubscribe()
for {
msg := <-chain2HeadCh
err := reply.Send(&proto.ChainWatchResponse{Type: msg.Type,
Newchain: convertBlockToBlockStub(msg.NewChain),
Oldchain: convertBlockToBlockStub(msg.OldChain),
})
if err != nil {
return err
}
}
}