Commit graph

118 commits

Author SHA1 Message Date
Pratik Patil
6f16d006fd Merge branch 'develop' of https://github.com/maticnetwork/bor into block-stm 2022-12-23 09:41:30 +05:30
Arpit Temani
f261c933d5 upgrade grpc version 2022-12-12 12:07:27 +05:30
marcello33
50a778207e
dev: add: pos-944: snyk and govuln integration (#578)
* dev: add: pos-944 security ci and readme

* dev: add: pos-944 remove linters as this is included already in build ci

* dev: chg: pos-947 dependencies upgrade to solve snyk security issues

* dev: chg: update security-ci

* dev: chg: remove linter to allow replacements for security issues

* dev: add: pos-944 verify path when updating metrics from config

* dev: add: pos-944 fix linter

* dev: add: pos-944 add .snyk policy file / fix snyk code vulnerabilities

* dev: fix: pos-944 import common package / gitignore snyk dccache file

* dev: fix: pos-944 verify canonical path for crashers

* dev: fix: pos-944 linter

* dev: add: pos-976 add govuln check

* dev: add: pos-976 test upload with permissions

* dev: add: pos-976 remove duplicated upload

* dev: add: pos-976 report upload

* dev: add: pos-976 remove upload

* dev: fix: pos-944 fix govuln action

* dev: fix: pos-944 move govulncheck to security-ci

* dev: fix: pos-944 bump golvun action and golang versions

* dev: fix: pos-944 remove persmissions and fix conflicts

* dev: chg: restore err msg

* dev: chg: remove duplicated function

* dev: chg: sort import

* dev: chg: fix linter

* dev: add: use common VerifyCrasher function to avoid duplications / replace deprecated ioutils.ReadFile

* dev: fix: typo
2022-12-06 10:53:55 +01:00
Pratik Patil
96e66e5256 Merge branch 'develop' of https://github.com/maticnetwork/bor into block-stm 2022-11-21 17:00:12 +05:30
Shivam Sharma
554712e954 chg : use https://github.com/maticnetwork/crand instead of JekaMas/crand 2022-10-17 15:44:11 +05:30
Pratik Patil
c59bb6ef16
Added unit tests for MV HashMap (#492)
* Create MVHashMap and use it StateDB

* Parallel state processor

* Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions

* Re-execute parallel tasks when there is a read in coinbase or burn address

* Block-stm optimization

Added tests for executor and two major improvements:

1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure.
2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings.

* Remove cache from executor test

* added mvhashmap unit tests (with  as key)

* Shard mvhashmap to reduce the time spent in global mutex

* Skip applying intermediate states

* Dependency improvement

* added test for status

* Create MVHashMap and use it StateDB

* Parallel state processor

* Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions

* Re-execute parallel tasks when there is a read in coinbase or burn address

* Txn prioritizer implemented using mutex map (#487)

* basic txn prioritizer implemented using mutex map

* Re-execute parallel tasks when there is a read in coinbase or burn address

* Re-execute parallel tasks when there is a read in coinbase or burn address

* using *sync.RWMutex{} in mutexMap

Co-authored-by: Jerry <jerrycgh@gmail.com>

* added getReadMap and getWriteMap (#473)

* Block-stm optimization

Added tests for executor and some improvements:

1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure.
2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings.
3. Use sync.Map to reduce the time spent in global mutex.
4. Skip applying intermediate states.
5. Estimate dependency when an execution fails without dependency information.
6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions.
7. Setting dependencies of Txs coming from the same sender before starting parallel execution.
8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue.

* Do not write entire objects directly when applying write set in blockstm

* fixed a small bug in the Report function (#530)

* linters

Co-authored-by: Jerry <jerrycgh@gmail.com>
2022-10-07 09:14:18 +05:30
Arpit Temani
2a6ea470a0
Mining Analysis (#429)
* base setup for miner opentel

* version change

* modify ctx passing

* add attributes

* update fill txs span attributes

* fix: use common attributes for remote and local txs

* pass context in seal

* fix

* fix

* add traces to finalize and assemble

* fix

* use task ctx in result loop

* only start parent span if no error

* send nil tracer from Finalize

* clean up

* add sub function timings in span attribute

* modify span attributes

* set time attribute to milliseconds

* linters fix

* fix linters for consensus

* add nolint to worker

* fix testcase

* Added fillTransactions subTraces

* add traces in intermediate root hash function

* add traces in WriteBlockAndSetHead function, fix linters

* fix: linting errors

* fix: test cases

* fix: go.mod

* fix: testcase

* extract tracing package

* linters fix

* debug

* Revert "debug"

This reverts commit 2d68b7c7b1105080563a4e1a6949dabc10acaff8.

* fix: panic in NewTransactionsByPriceAndNonce iteration

* change heimdall version to develop

* miner/worker: fix duplicate call to tx ordering

* miner/worker: refactor tracing

* consensus/bor: use tracing package

* tracing: add more abstraction for spans

* tracing: set all attributes at once

* remove nested tracing from blockchain.WriteBlockAndSetHead function

* remove nested tracing from statedb.IntermediateRoot function

* handle end span in bor.Seal function

* fix: typo

* minor fixes

* fix: linters

* fix: remove nolint

* go mod tidy

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
2022-09-30 13:40:18 +05:30
Jerry
c36ad88aec Block-stm optimization
Added tests for executor and some improvements:

1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure.
2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings.
3. Use sync.Map to reduce the time spent in global mutex.
4. Skip applying intermediate states.
5. Estimate dependency when an execution fails without dependency information.
6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions.
7. Setting dependencies of Txs coming from the same sender before starting parallel execution.
8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue.
2022-09-28 16:12:20 -07:00
Jerry
b1ba97c4f5 Create MVHashMap and use it StateDB 2022-09-28 16:12:20 -07:00
Pratik Patil
77db80cc91
Added script to generate config.toml fromstart.sh (#518)
* added go and bash script to get config out of start.sh and updated flagset.go

* changed 'requiredblocks' flag back to 'eth.requiredblocks'

* updated script

* changed 'requiredblocks' flag back to 'eth.requiredblocks'

* updated tests, and removed requiredblocks from json and hcl

* addressed comments
2022-09-23 13:37:04 +05:30
Evgeny Danilenko
a11952f858
Crypto-secure package with math/rand interface (#509)
* initial

* use crand

* fix

* fix
2022-09-14 14:54:43 +03:00
Evgeny Danilenko
714aef422b
Property based tests on TxPool: pending, locals (#466)
* initial

* measure

* initial test

* block size debug

* current progress

* debug

* progress

* debug

* it works

* todo

* linters

* clean up

* check for nonce gaps

* clean up

* cr

* cr

* cr

* check tx pool on an error

* stats

* params

* introduce config

* min-max accounts

* debug

* debug

* update

* debug

* skip
2022-09-03 11:20:24 +03:00
Evgeny Danilenko
e699254142
Use atomic pointer in go 1.19 (#446)
* use atomic pointer

* golang version

* golang version

* go1.19

* linters

* Bump golangci-lint

* linters

* linters

* linters after merge

* generic logger

* generic logger

* logger

* logger

* linters

* bump toml

* linters1

* linters

* linters

* linter

* linter

* linters

* linters

* linters
2022-08-09 22:11:09 +03:00
Pratik Patil
16408ba324
internal/cli, cmd/geth: replaced package naoina/toml with BurntSushi/toml and updated config name-tags (#486)
* removed package naoina/toml from dumpconfig and added BurntSushi/toml

* updated cmd/gethconfig.go
2022-08-05 14:56:41 +05:30
Krishna Upadhyaya
0c0fb63491 Implemented retry middleware in heimdall gRPC 2022-07-28 18:11:59 +05:30
Krishna Upadhyaya
75bb3bdeb1 Merge branch 'develop' into POS-648 2022-07-27 19:09:04 +05:30
Pratik Patil
71980bf5b1
internal/cli: use config file for flags in new-cli, builder/files: update defaults (#457)
* updated simple.json and simple.hcl

* added annotations for developer and grpc block

* added toml tags and simple.toml file

* added support for toml config files

* updated simple files toml, hcl, json

* added config.toml in builder/files and updated bor.service

* add dumpconfig command in cli for exporting configs

* update docs

* updated .goreleaser.yml (POS-651)

* changed --config to -config

* added test config for tests and fixed lint errors

* made fields of type big.int and time.Duration private, removed merge from dumpconfig, setting up default values to the Raw fields in dumpconfig, and fixed one lint error

* fixed lint errors (strange)

* lint fix

* private no-more, using '-' in name tags to ignore

* updated name tags, made c.configFile as a stringFlag (only one config file supported) and updated the merge logic in command.go -> Run()

* fix: set method for big.Int flags, added a TODO

* handeled bigInt and timeDuration type, bug fix in config_legacy, lint fix

* updated flags, consistent across flags.go and config.go

* fixed config test and updated test hcl, json config files

* updated config legacy test

* added test to check values of cmd flags, restructured Run in command.go, linter fix

* fix linters

* lint again

* changed 2 flags and assert -> require

* changed the 2 flags back

* updated correct congig.toml path and made mainnet default

* updated config.toml with new flags

* added sample config (toml) file

* removed sample-config.toml and added it in docs/config.md

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
2022-07-27 12:28:26 +05:30
Krishna Upadhyaya
dbbacb8b19 Using polyproto v0.0.2 and removed type utils 2022-07-26 11:16:11 +05:30
Krishna Upadhyaya
385aff85f0 Implemented heimdall gRPC client 2022-07-26 02:18:37 +05:30
Evgeny Danilenko
04e90e2426
internal/cli: update grpc/protobuf dependency (#449)
* update grpc_conn

* fix

* remove more deps from deprecated library

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
2022-07-11 19:09:49 +03:00
Evgeny Danilenko
342bf3089e
Merge develop branch into v0.3.0-dev (#430)
* merge geth v1.10.15

* fix: Removed FastSync from cli server

* fix: TestHeadersRLPStorage

* Added t.skip(ETH2 in bor)

* fix: flow in create consensus engine

* bumped version

* Fix typo

* increase block time

* remove file

* bumped version

* merge gethv1.10.17

* bumped version

* fix failing tests

* Bump Go version to v1.18 (#368)

* Bump Go version to v1.18.1

* Build using netgo tag

This will create a static build using Go native networking stack.

Checked and it works stable for all archs and distros.

* Fix meta

* initial implementation for common ancestor approach

* extract whitelist interface

* fix types

* fix tests and format

* add unit tests for IsValidChain function

* more tests

* wip

* test ErrCheckpointMismatch

* minor fixes

* fix test

* dont panic

* fmt

* Limit state sync by gas

* Added logging for state-sync total gas usage

* Added number of event-records in log

* Minor Changes

* Minor Fix

* Adding individual gasUsed

* Minor Fix

* fix: return value for no remote block

* handle all errors

* modularise fake chain validator in downloader

* add more tests

* fix tests

* Modifying miner.recommit flag and its adjustment function. (#370)

* changed min/max/current recommit values

* Remove Hardcoded min/max

* Code Sanitization

* Skipping tests for constant recommit interval

* Adding default miner.recommit value

* Minor Change

* Increased default value of rpc.txfeecap to 5

* add debug rpc endpoints for checkpoint whitelist service

* minor fixes and enhancements

* avoid capping warnings for gas set by internal system transactions

* use typed mocks

* fix

* fix

* fix

* fix close

* fix

* Create stale.yml

* Fix bor consensus checkpoint bug

Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Sandeep Sreenath <sandeep.sreenath@gmail.com>
Co-authored-by: Victor Castell <victor@victorcastell.com>
Co-authored-by: Ferran <ferranbt@protonmail.com>
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
Co-authored-by: Karlo <karlonovak@gmail.com>
Co-authored-by: Sandeep Sreenath <ssandeep@users.noreply.github.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
2022-06-15 09:02:01 +03:00
Evgeny Danilenko
9c8bf51f57
Prepare Bor package for testing (#416)
* Limit state sync by gas

* Added logging for state-sync total gas usage

* Added number of event-records in log

* Minor Changes

* Minor Fix

* Adding individual gasUsed

* Minor Fix

* it works

* fix tests

* log wiggle and delay with block number

* log delays as numbers

* linters

* fix tests

* restore linters for the project

* fix linters

* fix

* fix

* fix

* linters

* generation

* fix tests

* remove heimdall wrapper response

* linters

* remove possible collisions

* remove possible collisions

* remove possible collisions

* tests for unique address generation

* generalize set

* bor miner tests got restored

* fixes after CR

* final step and mining test

* fix

* fix e2e

* more tests for Heimdall requests

* fix linters

Co-authored-by: Ferran <ferranbt@protonmail.com>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
2022-06-08 16:39:30 +03:00
Evgeny Danienko
f0f113c0fc run versus origin/master 2022-05-12 15:25:59 +03:00
Evgeny Danienko
e36a54d600 goleak 2022-05-11 16:43:43 +03:00
Evgeny Danienko
15900477fb linters 2022-05-11 14:05:02 +03:00
Arpit Temani
2321e64ed7
V0.2.16 candidate (#373)
* merge geth v1.10.15

* fix: Removed FastSync from cli server

* fix: TestHeadersRLPStorage

* Added t.skip(ETH2 in bor)

* fix: flow in create consensus engine

* bumped version

* Fix typo

* increase block time

* remove file

* bumped version

* merge gethv1.10.17

* bumped version

* fix failing tests

* Bump Go version to v1.18 (#368)

* Bump Go version to v1.18.1

* Build using netgo tag

This will create a static build using Go native networking stack.

Checked and it works stable for all archs and distros.

* Fix meta

* Bump version

* Meta as stable

Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Sandeep Sreenath <sandeep.sreenath@gmail.com>
Co-authored-by: Victor Castell <victor@victorcastell.com>
2022-05-06 12:31:48 +02:00
Ferran Borreguero
4829114def
Use stream to send debug files (#279)
* Fix protobuf file

* Small fix

* Use grpc streams to send big debug files
2022-01-13 16:46:30 +01:00
Arpit Temani
543170235d merge conflicts 2021-11-23 11:55:41 +05:30
Sandeep Sreenath
983e32baf2 merged geth v1.10.10 2021-11-22 19:04:07 +05:30
Manav Darji
322e964233
added opentelemetry config, flags, and setup (#235)
* added opentelemetry config, flags, and setup

* modifications in opencollector flag and config

* bug fix

* modified error log
2021-11-17 09:47:04 +01:00
Shivam Sharma
fc58176e80 Added Prometheus flag on bor-cli 2021-11-11 21:11:54 +05:30
Ferran Borreguero
7282cf9c25 Resolve merge 2021-10-25 14:05:03 +02:00
Ferran Borreguero
c9cc1412a2 Add config file 2021-10-25 13:54:10 +02:00
ferranbt
7225eec0f6 Finish flags 2021-10-17 16:26:25 +00:00
Felix Lange
f9d683b07f
go.mod: upgrade goja (#23721)
This update adds support for some more ES6 functionality, notably
arrow functions.
2021-10-13 14:42:00 +02:00
Ferran Borreguero
e98379679f Add legacy config file 2021-10-10 15:06:46 +02:00
Felix Lange
57ff2dee06
go.mod: upgrade github.com/karalable/usb (#23684)
This update includes fixes for the *BSD build and go module vendoring.
2021-10-05 18:33:53 +02:00
Ferran Borreguero
74d384b6a4 Add account commands 2021-10-04 16:19:58 +02:00
Ferran Borreguero
a54cec4595 Initial spec for an alternative cli 2021-09-22 08:53:30 +02:00
Péter Szilágyi
5b246af54e
go.mod: fix module file to prevent go from updating it during build 2021-09-20 19:12:20 +03:00
Péter Szilágyi
e28f713ada
internal: support optional filter expression for debug.stacks (#23605)
* internal: support optional filter expression for debug.stacks

* internal/debug: fix string regexp

* internal/debug: support searching for line numbers too
2021-09-20 16:29:07 +03:00
Krishna Upadhyaya
adbd18f42c fix: genesis alloc issue 2021-09-01 15:04:25 +05:30
Krishna Upadhyaya
a49bc2172a Merge remote-tracking branch 'go-ethereum/master' into krishna/header-hash-fix 2021-08-28 12:50:40 +05:30
ucwong
fe2f153b55
go.mod: update goleveldb (#23417) 2021-08-24 22:02:46 +03:00
Arpit Temani
7e76e9f27c Merge tag 'v1.10.8' of github.com:ethereum/go-ethereum into arpit/v1.10.8-upgrade 2021-08-24 12:56:59 +05:30
ucwong
dfeb2f7e80
go.mod: upgrade golang.org/x/sys for go1.17 support (#23406) 2021-08-23 13:45:16 +02:00
Arpit Temani
b27dca6bbf merge v1.10.7 2021-08-19 15:56:13 +05:30
jwasinger
6902485767
cmd, metrics: add support for influxdb-v2 (cherry-picking from italoacasas' changes), leave existing support for v1 to maintain backwards-compatibility. (#23194)
This PR adds flag to enable InfluxDB v2 (--metrics.influxdbv2), flags for v2-specific features (--metrics.influxdb.token, --metrics.influxdb.bucket), also carries over addition of support for specifying organization (--metrics.influxdb.organization), but still retains backwards compatibility with InfluxDB v1.
2021-08-17 18:40:14 +02:00
Sina Mahmoodi
b157bae2c9
go.mod: bump golang.org/x/text to v0.3.6 (#23291) 2021-07-29 15:20:45 +02:00
ucwong
64a5e125c5
go.mod: upgrade to goupnp v1.0.2 (#23197) 2021-07-29 15:20:15 +02:00