Commit graph

538 commits

Author SHA1 Message Date
Péter Szilágyi
7b6ff527d5
cmd, eth: switch the dev synctarget to hash from block (#28209)
* cmd, eth: switch the dev synctarget to hash from block

* cmd/utils, eth/catalyst: terminate node wyen synctarget reached
2023-09-29 22:11:15 +03:00
rjl493456442
73f5bcb75b
core, accounts, eth, trie: handle genesis state missing (#28171)
* core, accounts, eth, trie: handle genesis state missing

* core, eth, trie: polish

* core: manage txpool subscription in mainpool

* eth/backend: fix test

* cmd, eth: fix test

* core/rawdb, trie/triedb/pathdb: address comments

* eth, trie: address comments

* eth: inline the function

* eth: use synced flag

* core/txpool: revert changes in txpool

* core, eth, trie: rename functions
2023-09-28 10:00:53 +03:00
rjl493456442
b85c183ea7
eth/downloader: remove header rollback mechanism (#28147)
* eth/downloader: remove rollback mechanism in downloader

* eth/downloader: remove the tests
2023-09-26 16:29:13 +03:00
phenix3443
40219109b0
eth/downloader: typo in comment (#28196) 2023-09-26 11:59:41 +03:00
Arpit Temani
08b4937cb6 fix lint 2023-09-23 16:26:23 +05:30
Arpit Temani
3626e5f7e4 fix testcase syntax 2023-09-20 15:57:37 +05:30
rjl493456442
c53b0fef2a
core, eth/downloader: fix genesis state missing due to state sync (#28124)
* core: fix chain repair corner case in path-based scheme

* eth/downloader: disable trie database whenever state sync is launched
2023-09-17 17:35:09 +03:00
Péter Szilágyi
4fa3db49a1
eth/downloader: prevent pivot moves after state commit (#28126) 2023-09-15 15:06:25 +03:00
Arpit Temani
986486bb37 merge develop 2023-09-04 14:14:19 +05:30
Arpit Temani
b48b89f852
all: implement milestones (#961)
* Milestone Implementation

* Merge branch 'POS-347' into reciept-e2e-test

* Changes for testing, will be removed after testing

* Debugged the error

* Just for testing purpose

* refactor debug api methods, rename whitelist -> checkpoint

* remove first iteration based vars

* fix linters

* Rewind Changes

* Error changes

* RewindBack function in bor_checkpoint_verifier

* Testcases

* Added the fetch test for milestone and checkpoint

* Debugged the lint changes

* Debugged the lint changes

* Debugged the lint changes

* Debugged the lint changes

* Improved the error in Miner test file

* Improved the error of pointing to the wrong function

* Locking the sprint after the vote has been made on it.

* Adding more logs for testing

* Adding more logs for testing

* Adding more logs for testing

* Implemented the NoAckMilestone fetching mechanism

* Testcases for milestone implementation

* Testing code  for fetchNoAckMilestone and fetchLastNoAckMilestone

* Testing changes

* refactor else-if

* Corrected the number of params in bor_ext.go

* Dummy API for testing

* Defined the GetVoteOnRootHash in interface

* Defined the GetVoteOnRootHash in interface

* Made changes in the web3ext file

* Added the GetVoteOnRootHash in PublicBlockChain API

* Added the GetVoteOnRootHash in filterBackend

* Added the log of Root and RooHash

* Removed the 0x from rootHash

* Just for testing purpose

* "GetVoteOnRootHash" mock implementation

* bor_test.go

* Added the test for milestone implementation

* Added service for fetching milestone by ID

* Improved the comments

* Removed the duplicate code

* use setter for borVerifier

* use setter for borVerifier

* refactor handleNoAckMilestone

* remove code repetition with retry function

* Converged the repetitive code

* after CR

* persistence

* persistence implementation

* feature flag

* Persistence Changes

* cr

* initial

* fix

* fix

* Whitelist Flag

* 1 Add:Included the milestone flag  2.Add:Hardlimit the rewind to maximum of 255 blocks

* Chg:Updated go.mod file

* Remove:Dubai Hardfork code

* Add:checked errors for call functions to the Db, Rmv: Remote Header variable from the IsValidPeer() function

* Fix:Linting issues'

* Add:MilestoneGRPC functions

* Fix:Lint issues

* Fix:Lint issues

* Fix: TestFetchMilestoneFromMockHeimdall

* Fix:Integrations tests

* Add:Test for sprint length and milestone changes

* Add:Functionality to fetch the finalized block

* Chg:Changed default val of TriesInmemory to 1024

* fix:Some functions of heimdallGRPC client

* Restored the GRPC functionality, was commented out for  developing purpose

* Fix:Bor_checkpoint_Verfier function

* Test:Added the chain Rewinding test

* Test:Added the Sprint Length + Milestone merge test

* Add:Implemented the future milestone

* Add:Future milestone changes

* Add:Future milestone changes

* Chg: Voting on endBlockHash rather than rootHash

* Chg: Changed the logic of future milestone from rootHash checking to endBlockHash checking

* Fix:Using endBockHash while verifying the incoming milestone

* Chg:Variable names for better readiblity

* Fix:Testing changes

* Add:metrics for milestone implementation

* Add:Metrics for milestone implementatian

* Fix:Order of statements in a function for better optimization

* Chg:Removed unrequired file

* Fix:new variable intialization

* Add:Comment to increase readiblity

* Fix:Logs

* Chg:Name of GetVoteOnRootHash to GetVoteOnHash

* Fix:Linting issues

* Fixed linting issues

* Rmv: Unnecessary logs and Add:Skip test for long tests

* Fix:Checking current chain with whitelisted milestone or checkpoint in Finalized block function

* Fix:Test

* Fix:Whitelisting of Milestone and Checkpoint process

* Fix: Milestone JSON structure

* Chg:Testcases changes

* Fix:Change from VoteOnRootHash to VoteOnHash

* Fix:Variable name fix

* Fix:Finalized API

* internal/jsre/deps: update web3.js bundle

* Fix:milestone verifier

* Chg:Handling the long future chain import issue

* Fix:Lint issues

* Fix:TestLowDiffLongChain and TestPrunedImportSide tests, used hardcoded value 128 instead of DefaultTriesInMemory value

* Chg:Testcode for producing metrics

* Chg:Milestong polling value to 32 secs

* Add:Testcases

* Add:Implemented the check to fetch the milestoneId from heimdall before locking the fork

* Added GRPC method for FetchMilestoneID

* Fix:lint issue

* Fix:lint issue

* Skiped out the tests which were mainly used to produce the supporting data

* remove vcs build when running snyk

* Add:Improved the logs and comments

* fix linters

* Skipped some test as they are panic due to timeout issue in github

* Chg:Variable name LockerSprintNumber to LockedMilestoneNumber for better readablity and clarity

* Chg:Conflicting variable names in milestone test file

* Chg:Conflicting function names in milestone test file

* fix : minor fix in TestInsertingSpanSizeBlocks

* Fix:Mocking issue in TestInsertingSpanSizeBlocks

* Fix:GRPC Polyproto Version

* eth/downloader: skip peer drop due to whitelisting err

* eth, tests/bor: bug fixes and minor refactor

* Add:Implemented the milestone related functions in the HeimdallApp

* Fix:Lint Errors & Remove:Redundant Code

* Fix:Testing Errors

* Fix:Bor integeration tests

* Fix:Test errors

* update heimdall client mock files

* remove unused arguments

* remove redundant code

* Chg:Changed the milestone polling intervals

* Add: added block finality from whitelisted checkpoint

* skip future chain validation

* Add:confirmation check of 16 blocks over the end block while voting for the milestone in GetVoteHash() function

* Chg:Included endBlockNum in UnlockMutex function

* Add:Property based test for milestone

* Fix:Opening the lock while processing future milestone

* Add:Property based test for futureMilestone

* Defined the value of TempTriesInMemory

* Fixed the finalized api

* Fixed lint issues

* eth: add logs while fetching and rewinding

* fix linters: use default returns instead of recursive calls

* Fix:Milestone intergration test

* Add:GetVoteHash fn in mock backend

* tests/bor: fix mock span

* tests/bor: remove t.Parallel()

* use bor namespace in ethclient, fix mock function

---------

Co-authored-by: Vaibhav Jindal <vaibhavjindal29@gmail.com>
Co-authored-by: VaibhavJindal <74560896+VAIBHAVJINDAL3012@users.noreply.github.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
2023-08-28 18:42:21 +05:30
Arpit Temani
d3c5bc9b45 fix build 2023-08-21 17:48:09 +05:30
Arpit Temani
b343cffdb4 merge geth v1.12.2 into develop 2023-08-18 14:05:34 +05:30
Delweng
649deb69f3
eth/downloader: fix rare crash when parent header missing in db (#27945)
ReadSkeletonHeader can return nil if the header is missing, so we should
not access fields on it. Note that calling .Hash() on a nil header is fine, so there 
is no need to actually check for nil.

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-08-17 23:15:29 +02:00
Felix Lange
2a6beb6a39
core/types: support for optional blob sidecar in BlobTx (#27841)
This PR removes the newly added txpool.Transaction wrapper type, and instead adds a way
of keeping the blob sidecar within types.Transaction. It's better this way because most
code in go-ethereum does not care about blob transactions, and probably never will. This
will start mattering especially on the client side of RPC, where all APIs are based on
types.Transaction. Users need to be able to use the same signing flows they already
have.

However, since blobs are only allowed in some places but not others, we will now need to
add checks to avoid creating invalid blocks. I'm still trying to figure out the best place
to do some of these. The way I have it currently is as follows:

- In block validation (import), txs are verified not to have a blob sidecar.
- In miner, we strip off the sidecar when committing the transaction into the block.
- In TxPool validation, txs must have a sidecar to be added into the blobpool.
  - Note there is a special case here: when transactions are re-added because of a chain
    reorg, we cannot use the transactions gathered from the old chain blocks as-is,
    because they will be missing their blobs. This was previously handled by storing the
    blobs into the 'blobpool limbo'. The code has now changed to store the full
    transaction in the limbo instead, but it might be confusing for code readers why we're
    not simply adding the types.Transaction we already have.

Code changes summary:

- txpool.Transaction removed and all uses replaced by types.Transaction again
- blobpool now stores types.Transaction instead of defining its own blobTx format for storage
- the blobpool limbo now stores types.Transaction instead of storing only the blobs
- checks to validate the presence/absence of the blob sidecar added in certain critical places
2023-08-14 10:13:34 +02:00
rjl493456442
503f1f7ada
all: activate pbss as experimental feature (#26274)
* all: activate pbss

* core/rawdb: fix compilation error

* cma, core, eth, les, trie: address comments

* cmd, core, eth, trie: polish code

* core, cmd, eth: address comments

* cmd, core, eth, les, light, tests: address comment

* cmd/utils: shorten log message

* trie/triedb/pathdb: limit node buffer size to 1gb

* cmd/utils: fix opening non-existing db

* cmd/utils: rename flag name

* cmd, core: group chain history flags and fix tests

* core, eth, trie: fix memory leak in snapshot generation

* cmd, eth, internal: deprecate flags

* all: enable state tests for pathdb, fixes

* cmd, core: polish code

* trie/triedb/pathdb: limit the node buffer size to 256mb

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2023-08-10 22:21:36 +03:00
Péter Szilágyi
57268f7e6c
all: rename dataGas to blobGas (#27789) 2023-07-27 16:53:28 +03:00
Shivam Sharma
6528e4500b fix : lint 2023-07-24 14:44:50 +05:30
Shivam Sharma
616e33f290 fix : TestBeaconSync66 2023-07-04 16:50:05 +05:30
Shivam Sharma
2ad1e0f7e9 fix : TestTransactionIndices, testBeaconSync 2023-07-04 13:41:04 +05:30
Raneet Debnath
2b06029fb8
core,eth,miner: fix initial test cases (#922) 2023-07-04 10:43:28 +05:30
marcello33
99693ef56c dev: fix: all remaining lint issues 2023-06-19 11:39:59 +02:00
marcello33
f3ffacf2d7 dev: fix: most of wsl lint issues 2023-06-15 15:48:26 +02:00
marcello33
3af5e435c7 dev: fix: some wsl lint issues 2023-06-15 12:27:20 +02:00
marcello33
631efbe675 dev: fix: thelper and tparallel lint 2023-06-14 16:29:26 +02:00
marcello33
1c106dc251 dev: add: merge develop in upstream-merge 2023-06-13 12:37:20 +02:00
marcello33
a043ec6d62 dev: add: merge upstream 2023-06-09 14:34:30 +02:00
Péter Szilágyi
a7b2106edf
eth/downloader: drop beacon head updates if the syncer is restarting (#27397)
* eth/downloader: drop beacon head updates if the syncer is restarting

* eth/donwloader: v2 of the goroutine spike preventer
2023-06-05 09:59:38 +03:00
Péter Szilágyi
008086f935
core, eth/downloader: validate blobtx.To at serialization time (#27393) 2023-05-31 13:08:15 +03:00
Péter Szilágyi
495692c9db
core, eth/downloader, params: validate blob tx bodies (#27392) 2023-05-31 11:12:26 +03:00
Péter Szilágyi
1f9b69b36d
consensus, core, eth/downloader, params: 4844 chain validation (#27382) 2023-05-31 10:21:13 +03:00
Pratik Patil
b6de7aaf17
fixed checkoint misspell to checkpoint (#854) 2023-05-09 12:37:25 +05:30
rjl493456442
79a57d49cb
eth/downloader: fix error aggregator (#27217) 2023-05-05 08:55:32 +03:00
Péter Szilágyi
dde2da0efb
all: remove ethash pow, only retain shims needed for consensus and tests (#27178)
* all: remove ethash pow, only retain shims needed for consensus and tests

* all: thank you linter

* all: disallow launching Geth in legacy PoW mode

* cmd/env/internal/t8ntool: remove dangling ethash flag
2023-05-03 12:58:39 +03:00
Péter Szilágyi
1e556d220c
all: remove notion of trusted checkpoints in the post-merge world (#27147)
* all: remove notion of trusted checkpoints in the post-merge world

* light: remove unused function

* eth/ethconfig, les: remove unused config option

* les: make linter happy

---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-04-24 09:37:10 +03:00
Péter Szilágyi
bbc565ab05
core/types, params: add blob transaction type, RLP encoded for now (#27049)
* core/types, params: add blob transaction type, RLP encoded for now

* all: integrate Cancun (and timestamp based forks) into MakeSigner

* core/types: fix 2 back-and-forth type refactors

* core: fix review comment

* core/types: swap blob tx type id to 0x03
2023-04-21 12:52:02 +03:00
s7v7nislands
db18293c32
eth/downloader: use atomic types (#27030)
* eth/downloader: use atomic type

* Update eth/downloader/downloader_test.go

Co-authored-by: Martin Holst Swende <martin@swende.se>

* Update eth/downloader/downloader_test.go

Co-authored-by: Martin Holst Swende <martin@swende.se>

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-04-03 22:48:10 +03:00
Manav Darji
b39e64c335
eth/downloader/whitelist: skip future chain validation (#796)
* eth/downloader/whitelist: skip future chain validation

* eth/downloader/whitelist: fix tests for future chain import
2023-03-31 12:03:19 +05:30
Péter Szilágyi
cd31f2dee2
all: change chain head markers from block to header (#26777) 2023-03-02 08:29:15 +02:00
Péter Szilágyi
b3ae073488
eth: use the last announced finalized block as the sync ancient limit (#26685) 2023-02-23 13:22:41 +02:00
Péter Szilágyi
90d25514af
core, eth: merge snap-sync chain download progress logs (#26676) 2023-02-21 12:17:34 +02:00
ucwong
e9d42499bb
eth/downloader: fix typo (#26716) 2023-02-16 19:40:16 +02:00
Martin Holst Swende
13d7de77f4
eth/downloader: fix empty-body case in queue fetchresult (#26707) 2023-02-16 12:28:01 +02:00
Martin Holst Swende
645e3e86c4
core, eth/downloader: make body validation more strict (#26704) 2023-02-16 12:10:16 +02:00
Marius van der Wijden
7fb42e6db2
eth/downloader: handle missing withdrawals if empty list is expected (#26675)
This PR relaxes the block body ingress handling a bit: if block body withdrawals are missing (but expected to be empty), the body withdrawals are set to 'empty list' before being passed to upper layers. 

This fixes an issue where a block passed from EthereumJS to geth was deemed invalid.
2023-02-15 07:23:07 -05:00
Péter Szilágyi
3086c256c9
eth/downloader: fix timeout resurrection panic (#26652)
* common/prque, eth/downloader: fix timeout resurrection panic

* common/prque: revert -1 hack for les, temporaryly!
2023-02-09 14:56:15 +02:00
Péter Szilágyi
bf1798e04e
common/prque: generic priority queue (#26290)
* common, core, eth, les, trie: make prque generic

* les/vflux/server: fixed issues in priorityPool

* common, core, eth, les, trie: make priority also generic in prque

* les/flowcontrol: add test case for priority accumulator overflow

* les/flowcontrol: avoid priority value overflow

* common/prque: use int priority in some tests

No need to convert to int64 when we can just change the type used by the
queue.

* common/prque: remove comment about int64 range

---------

Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
2023-02-09 13:03:54 +02:00
Marius van der Wijden
2a2b0419fb
all: implement withdrawals (EIP-4895) (#26484)
This change implements withdrawals as specified in EIP-4895.

Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: marioevz <marioevz@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2023-01-25 15:32:25 +01:00
SHIVAM SHARMA
9aeaf03f05
Merge branch 'qa' and 'master' into develop (#663)
* Adding in Mumbai/Mainnet precursor deb packaging for tests to use during upgrade(iterations to come)

* Added changes per discussion in PR, more changes may be necessary

* Adding prerelease true

* Disabling goreleaser

* Removing README swap file

* change bor_dir and add bor user for v0.3.0 release

* rollback bor user and use root

* metrics: handle equal to separated config flag (#596)

* metrics: handle  based config path

* internal/cli/server: add more context to logs

* use space separated flag and value in bor.service

* fixed static-nodes related buf (os independent) (#598)

* fixed static-nodes related buf (os independent)

* taking static-nodes as input if default not present

* Update default flags (#600)

* internal/cli/server: use geth's default for txpool.pricelimit and add comments

* builder/files: update config.toml for mainnet

* packaging/templates: update defaults for mainnet and mumbai

* internal/cli/server: skip overriding cache

* packaging/templates: update cache value for mainnet

* packaging/templates: update gcmode for archive mumbai node

* metrics: handle nil telemetry config (#601)

* resolve merge conflicts

* update go version in release.yml

* update goversion in makefile

* update Docker login for goreleaser-cross v1.19

* Cleanup for the packager to use git tag in the package profile naming. Added conditional check for directory structure, this is in prep for v0.3.1, as this will create a failure on upgrade path in package due to file exist

* added a toml configuration file with comments describing each flag (#607)

* added a toml configuration file with comments describing each flag

* internal/cli/server: update flag description

* docs/cli: update example config and description of flags

* docs: update new-cli docs

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* Adding of 0.3.0 package changes, control file updates, postinst changes, and packager update

* added ancient datadir flag and toml field, need to decide on default value and update the conversion script

* updated toml files with ancient field

* Add support for new flags in new config.toml, which were present in old config.toml (#612)

* added HTTPTimeouts, and TrieTimeout flag in new tol, from old toml

* added RAW fields for these time.Duration flags

* updated the conversion script to support these extra 4 flags

* removed hcl and json config tests as we are only supporting toml config files

* updated toml files with cache.timeout field

* updated toml files with jsonrpc.timeouts field

* tests/bor: expect a call for latest checkpoint

* tests/bor: expect a call for latest checkpoint

* packaging/templates: update cache values for archive nodes

Co-authored-by: Manav Darji <manavdarji.india@gmail.com>

* remove unwanted code

* Fix docker publish authentication issue

In gorelease-cross 1.19+, dockerhub authentication will require docker
logion action followed by mounting docker config file. See
https://github.com/goreleaser/goreleaser-cross#github-actions.

* Revert "update Docker login for goreleaser-cross v1.19"

This reverts commit 4d19cf5342.

* Bump version to stable

* Revert "Merge pull request #435 from maticnetwork/POS-553"

This reverts commit 657d262def, reversing
changes made to 88dbfa1c13.

* revert change for release for go1.19

* Add default values to CLI helper and docs

This commit adds default values to CLI helper and docs. When the default value of a string flag, slice string flag, or map string flag is empty, its helper message won't show any default value.

* Add a summary of new CLI in docs

* Updating packager as binutils changed version so that apt-get installs current versions

* Add state pruning to new CLI

* Minor wording fix in prune state description

* Bumping control file versions

* Mainnet Delhi fork

* Set version to stable

* change delhi hardfork block number

* handle future chain import and skip peer drop (#650)

* handle future chain import and skip peer drop

* add block import metric

* params: bump version to v0.3.3-stable

* Bump bor version in control files for v0.3.3 mainnet release

Co-authored-by: Daniel Jones <djones@polygon.technology>
Co-authored-by: Will Button <wbutton@polygon.technology>
Co-authored-by: Will Button <will@willbutton.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Daniel Jones <105369507+djpolygon@users.noreply.github.com>
Co-authored-by: Pratik Patil <pratikspatil024@gmail.com>
Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
2023-01-13 14:47:14 +05:30
rjl493456442
b748709a11
eth/downloader: fix cornercase when clean stale beacon headers (#26441) 2023-01-12 11:19:16 +01:00
Péter Szilágyi
452a12aa79
eth/downloader: move the pivot in beacon sync mode too (#26453)
In legacy (pre-merge) sync mode, headers were contiguously downloaded from the network and when no more headers were available, we checked every few seconds whether there are 64 new blocks to move the pivot.

In beacon (post-merge) sync mode, we don't need to check for new skeleton headers non stop, since those re delivered one by one by the engine API. The missing code snippet from the header fetcher was to actually look at the latest head and move the pivot if it was more than 2*64-8 away. This PR adds the missing movement logic.
2023-01-10 12:16:54 -05:00
Péter Szilágyi
71f7988b0f
eth/downloader: create repro testcase for beacon header loss 2023-01-09 12:12:25 +02:00
Gary Rong
686f7438d3 eth/downloader: fix unexpected skeleton header deletion 2023-01-09 15:07:08 +08:00
Ha DANG
9d38466437
eth/downloader: fix some typos (#26396) 2023-01-03 11:03:44 +02:00
rjl493456442
743e404906
core, eth, les, tests, trie: abstract node scheme (#25532)
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.

Apart from that, a few changes are also included which is worth mentioning:

-  port the changes in the stacktrie, tracking the path prefix of nodes during commit
-  use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
2022-11-28 14:31:28 +01:00
Péter Szilágyi
c4a662176e
core, eth: for types with accurate size calcs, return uint64, not float (#26046)
* core, eth: for types with accurate size calcs, return uint64, not float

* core/types: proper tx size tests

* core/types: extend tx size test with decoded sizes, fix error

* core/txpool: fix linter

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-10-26 15:23:07 +03:00
Martin Holst Swende
5a02b2d6d0
all: fix spelling mistakes (#25961) 2022-10-11 09:37:00 +02:00
Martin Holst Swende
ee301c750b
all: fix docstrings 2022-10-04 09:18:02 +02:00
Martin Holst Swende
80cc34ac3c
eth/downloader: make flakey test less flakey (#25879) 2022-09-27 15:49:52 +02:00
zhiqiangxu
b32d20324e
eth/downloader, les/downloader: fix subtle flaw in queue delivery (#25861)
* fix queue.deliver

* les/downloader: fix queue.deliver

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-26 12:33:21 +02:00
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
rjl493456442
dea1fb3cfc
all: cleanup tests (#25641)
Follow-up to PR #25523 to cleanup all relevant tests.
2022-09-07 20:21:59 +02:00
rjl493456442
d10c280309
all: move genesis initialization to blockchain (#25523)
* all: move genesis initialization to blockchain

* core: add one more check

* core: fix tests
2022-08-30 18:22:28 +02:00
Péter Szilágyi
81bd998353
core, eth/downloader: handle spurious junk bodies from racey rollbacks (#25578)
* eth/downloader: handle junkbodies/receipts in the beacon sync

* core: check for header presence when checking for blocks
2022-08-23 14:02:51 +03:00
Justin Traglia
656dc8cc00
eth, les: unlock downloader peerSet if there's an error (#25546)
Unlock peerSet if there's an error in the downloader
2022-08-19 09:02:47 +03:00
Justin Traglia
2c5648d891
all: fix some typos (#25551)
* Fix some typos

* Fix some mistakes

* Revert 4byte.json

* Fix an incorrect fix

* Change files to fails
2022-08-19 09:00:21 +03:00
Evgeny Danilenko
e699254142
Use atomic pointer in go 1.19 (#446)
* use atomic pointer

* golang version

* golang version

* go1.19

* linters

* Bump golangci-lint

* linters

* linters

* linters after merge

* generic logger

* generic logger

* logger

* logger

* linters

* bump toml

* linters1

* linters

* linters

* linter

* linter

* linters

* linters

* linters
2022-08-09 22:11:09 +03:00
rjl493456442
a41ea8a97c
all: cleanup the APIs for initializing genesis (#25473)
* all: polish tests

* core: apply feedback from Guillaume

* core: fix comment
2022-08-09 12:44:39 +03:00
int88
e93442c6cf
eth/downloader: fix log errors of queue_test.go (#25494) 2022-08-08 16:07:54 +03:00
Péter Szilágyi
f3af3fd8df
eth: support bubbling up bad blocks from sync to the engine API (#25190)
* eth: support bubbling up bad blocks from sync to the engine API

* eth/catalyst: fix typo

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>

* eth/catalyst: fix typo

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>

* Update eth/catalyst/api.go

* eth/catalyst: when forgetting bad hashes, also forget descendants

* eth/catalyst: minor bad block tweaks for resilience

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-07-25 16:51:04 +03:00
Delweng
b196ad1c16
all: add whitespace linter (#25312)
* golangci: typo

Signed-off-by: Delweng <delweng@gmail.com>

* golangci: add whietspace

Signed-off-by: Delweng <delweng@gmail.com>

* *: rm whitesapce using golangci-lint

Signed-off-by: Delweng <delweng@gmail.com>

* cmd/puppeth: revert accidental resurrection

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2022-07-25 13:14:03 +03:00
rjl493456442
1657e43931
core, les, eth: port snap sync changes (#24898)
core, eth, les, trie: rework snap sync
2022-07-15 14:55:51 +03:00
Manav Darji
c44693706a
RFC35/Common Ancestor: Modifying the forkchoice rule (#425)
* initial

* update

* fix: add validator to NewBlockchain function calls

* handle past chain reorg

* fix: only check with last checkpoint

* rm logs

* add: handle future chain import case

* fix: handle single block case

* add unit tests for past and future chain

* modularise forker tests

* minor fixes

* add: overlapping chain test case, minor fixes

* minor fixes

* add: isolated unit test for IsValidChain

* add more test case for IsValidChain

* fix: use index for header time

* add: fetch last N checkpoints in first run

* fix: change checkpoint count to int64

* fix: handle edge case

* fix: handle no checkpoint case separately

* fix: consider offset for future chain calculation

* re-write test case for split chain

* fix: typo

* add: split chain properties test

* separate reorg checks, validate chain before inserting

* fix: handle err incase of invalid chain

* fix: use error from whitelist service

* split chain property tests

* remove duplicate test cases

* cleanup

* clean up

* fix linters

* fix: fetch checkpoint count bug, add tests

* fix more linters

* fix: handle nil chain validator in downloader

* fix: mock bor tests

Co-authored-by: Evgeny Danienko <6655321@bk.ru>
2022-07-15 10:37:35 +05:30
Arpit Temani
7b25997830
Reciept e2e test (#431)
* initial

* fixed tests

* eip155 tests

* progress

* fix

* fix TestFetchStateSyncEvents*

* debug

* fixed with updated list of validators

* fix

* a bit more stable

* optimize allocations

* linters

* fix vals copy

* a bit of refactoring

* update TestGetTransactionReceiptsByBlock

* remove logs from TestGetTransactionReceiptsByBlock in favour of checks

* comments

* Linters

* linters

* fixes after merge

* fixes after merge

* fixes after merge

* fixes after merge

* fail fast

Co-authored-by: Evgeny Danienko <6655321@bk.ru>
2022-07-14 19:28:56 +03:00
lightclient
10dc5dce08
all: remove concept of public/private API definitions (#25053)
* internal/ethapi: rename PublicEthereumAPI to EthereumAPI

* eth: rename PublicEthereumAPI to EthereumAPI

* internal/ethapi: rename PublicTxPoolAPI to TxPoolAPI

* internal/ethapi: rename PublicAccountAPI to EthereumAccountAPI

* internal/ethapi: rename PrivateAccountAPI to PersonalAccountAPI

* internal/ethapi: rename PublicBlockChainAPI to BlockChainAPI

* internal/ethapi: rename PublicTransactionPoolAPI to TransactionAPI

* internal/ethapi: rename PublicDebugAPI to DebugAPI

* internal/ethapi: move PrivateDebugAPI methods to DebugAPI

* internal/ethapi: rename PublicNetAPI to NetAPI

* les: rename PrivateLightServerAPI to LightServerAPI

* les: rename PrivateLightAPI to LightAPI

* les: rename PrivateDebugAPI to DebugAPI

* les: rename PublicDownloaderAPI to DownloaderAPI

* eth,les: rename PublicFilterAPI to FilterAPI

* eth: rename PublicMinerAPI to MinerAPI

* eth: rename PublicDownloaderAPI to DownloaderAPI

* eth: move PrivateMinerAPI methods to MinerAPI

* eth: rename PrivateAdminAPI to AdminAPI

* eth: rename PublicDebugAPI to DebugAPI

* eth: move PrivateDebugAPI methods to DebugAPI

* node: rename publicAdminAPI to adminAPI

* node: move privateAdminAPI methods to adminAPI

* node: rename publicWeb3API to web3API

* eth,internal/ethapi: sync comments with previous renamings
2022-06-21 12:05:43 +03:00
rjl493456442
30602163d5
eth: introduce eth67 protocol (#24093)
The new protocol version removes support for GetNodeData.
See https://eips.ethereum.org/EIPS/eip-4938 for more information.

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-06-15 12:56:47 +02:00
lightclient
3273ad1a58
eth: add missing period at end of sentences (#25058)
eth: add missing periods on end of comments
2022-06-15 09:29:23 +02:00
Evgeny Danilenko
342bf3089e
Merge develop branch into v0.3.0-dev (#430)
* merge geth v1.10.15

* fix: Removed FastSync from cli server

* fix: TestHeadersRLPStorage

* Added t.skip(ETH2 in bor)

* fix: flow in create consensus engine

* bumped version

* Fix typo

* increase block time

* remove file

* bumped version

* merge gethv1.10.17

* bumped version

* fix failing tests

* Bump Go version to v1.18 (#368)

* Bump Go version to v1.18.1

* Build using netgo tag

This will create a static build using Go native networking stack.

Checked and it works stable for all archs and distros.

* Fix meta

* initial implementation for common ancestor approach

* extract whitelist interface

* fix types

* fix tests and format

* add unit tests for IsValidChain function

* more tests

* wip

* test ErrCheckpointMismatch

* minor fixes

* fix test

* dont panic

* fmt

* Limit state sync by gas

* Added logging for state-sync total gas usage

* Added number of event-records in log

* Minor Changes

* Minor Fix

* Adding individual gasUsed

* Minor Fix

* fix: return value for no remote block

* handle all errors

* modularise fake chain validator in downloader

* add more tests

* fix tests

* Modifying miner.recommit flag and its adjustment function. (#370)

* changed min/max/current recommit values

* Remove Hardcoded min/max

* Code Sanitization

* Skipping tests for constant recommit interval

* Adding default miner.recommit value

* Minor Change

* Increased default value of rpc.txfeecap to 5

* add debug rpc endpoints for checkpoint whitelist service

* minor fixes and enhancements

* avoid capping warnings for gas set by internal system transactions

* use typed mocks

* fix

* fix

* fix

* fix close

* fix

* Create stale.yml

* Fix bor consensus checkpoint bug

Co-authored-by: Arpit Temani <temaniarpit27@gmail.com>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Sandeep Sreenath <sandeep.sreenath@gmail.com>
Co-authored-by: Victor Castell <victor@victorcastell.com>
Co-authored-by: Ferran <ferranbt@protonmail.com>
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
Co-authored-by: Karlo <karlonovak@gmail.com>
Co-authored-by: Sandeep Sreenath <ssandeep@users.noreply.github.com>
Co-authored-by: Jerry <jerrycgh@gmail.com>
2022-06-15 09:02:01 +03:00
Martin Holst Swende
a907d7e81a
all: more linters (#24783)
This enables the following linters

- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec

WIth a few exceptions.

- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
2022-06-13 16:24:45 +02:00
Felix Lange
9244d5cd61
all: update license headers and AUTHORS from git history (#24947) 2022-05-24 20:39:40 +02:00
s7v7nislands
7caa2d8163
all: replace strings.Replace with string.ReplaceAll (#24835) 2022-05-09 13:13:23 +03:00
Arpit Temani
2321e64ed7
V0.2.16 candidate (#373)
* merge geth v1.10.15

* fix: Removed FastSync from cli server

* fix: TestHeadersRLPStorage

* Added t.skip(ETH2 in bor)

* fix: flow in create consensus engine

* bumped version

* Fix typo

* increase block time

* remove file

* bumped version

* merge gethv1.10.17

* bumped version

* fix failing tests

* Bump Go version to v1.18 (#368)

* Bump Go version to v1.18.1

* Build using netgo tag

This will create a static build using Go native networking stack.

Checked and it works stable for all archs and distros.

* Fix meta

* Bump version

* Meta as stable

Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Sandeep Sreenath <sandeep.sreenath@gmail.com>
Co-authored-by: Victor Castell <victor@victorcastell.com>
2022-05-06 12:31:48 +02:00
Ikko Ashimine
40cfe71002
eth/downloader: fix typo in downloader.go (#24704)
synchornization -> synchronization
2022-04-20 16:17:29 +02:00
rjl493456442
f0328f241b
eth/downloader: resolve local header by hash for beacon sync (#24691)
* eth/downlaoder: resolve local header by hash for beacon sync

* eth/downloader: fix error message

* eth/downloader: cap the reverse header resolving

* eth/downloader: re-enable tests

* eth/downloader: add warning logs
2022-04-14 09:49:23 +03:00
Péter Szilágyi
86216189a5
eth/downloader: remove stale beacon headers as backfilling progresses (#24670)
* eth/downloader: remove stale beacon headers as backfilling progresses

* eth/downloader: remove leftover from a previous design

* eth/downloader: do partial beacon cleanups if chain is large

* eth/downloader: linter != heart
2022-04-13 20:31:08 +03:00
Eng Zer Jun
8d066f1f42
all: use T.TempDir to create temporary test directories (#24633)
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.

	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}

is also tedious, but t.TempDir handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-08 15:44:55 +02:00
rjl493456442
28ec26094b
eth/downloader: retrieve pivot header from local chain if necessary (#24610)
* eth/downloader: retrieve pivot header from local chain if necessary

* eth/downloader: improve readability

* eth/downloader: update fix

* eth/downloader: add beacon sync tests

* eth/downloader: remove duplicated code
2022-04-04 10:10:16 +03:00
Martin Holst Swende
f252154599
eth/downloader: fix flakey test (#24576) 2022-03-23 20:57:53 +01:00
rjl493456442
e0e8bf31c5
eth/downloader: ignore zero size header batch for importing (#24569)
* eth/downloader: ignore zero size header batch for importing

* core, light: reject empty header batch for importing
2022-03-22 18:58:05 +01:00
Péter Szilágyi
51de2bc9dc
eth/downloader: terminate beacon sync early when linked to local chain (#24550)
* eth/downloader: terminate beacon sync early when linked to local chain

* eth/downloader: fix backfiller resume on early beacon termination
2022-03-21 10:43:54 +02:00
Péter Szilágyi
6b8718c374
eth/downloader: fix off-by-one error in test causing 50% fails 2022-03-16 08:57:00 +02:00
Péter Szilágyi
8f66ea3786
eth/downloader: implement beacon sync (#23982)
* eth/downloader: implement beacon sync

* eth/downloader: fix a crash if the beacon chain is reduced in length

* eth/downloader: fix beacon sync start/stop thrashing data race

* eth/downloader: use a non-nil pivot even in degenerate sync requests

* eth/downloader: don't touch internal state on beacon Head retrieval

* eth/downloader: fix spelling mistakes

* eth/downloader: fix some typos

* eth: integrate legacy/beacon sync switchover and UX

* eth: handle UX wise being stuck on post-merge TTD

* core, eth: integrate the beacon client with the beacon sync

* eth/catalyst: make some warning messages nicer

* eth/downloader: remove Ethereum 1&2 notions in favor of merge

* core/beacon, eth: clean up engine API returns a bit

* eth/downloader: add skeleton extension tests

* eth/catalyst: keep non-kiln spec, handle mining on ttd

* eth/downloader: add beacon header retrieval tests

* eth: fixed spelling, commented failing tests out

* eth/downloader: review fixes

* eth/downloader: drop peers failing to deliver beacon headers

* core/rawdb: track beacon sync data in db inspect

* eth: fix review concerns

* internal/web3ext: nit

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2022-03-11 14:14:45 +02:00
Martin Holst Swende
db03faa10d
core, eth: improve delivery speed on header requests (#23105)
This PR reduces the amount of work we do when answering header queries, e.g. when a peer
is syncing from us.

For some items, e.g block bodies, when we read the rlp-data from database, we plug it
directly into the response package. We didn't do that for headers, but instead read
headers-rlp, decode to types.Header, and re-encode to rlp. This PR changes that to keep it
in RLP-form as much as possible. When a node is syncing from us, it typically requests 192
contiguous headers. On master it has the following effect:

- For headers not in ancient: 2 db lookups. One for translating hash->number (even though
  the request is by number), and another for reading by hash (this latter one is sometimes
  cached).
  
- For headers in ancient: 1 file lookup/syscall for translating hash->number (even though
  the request is by number), and another for reading the header itself. After this, it
  also performes a hashing of the header, to ensure that the hash is what it expected. In
  this PR, I instead move the logic for "give me a sequence of blocks" into the lower
  layers, where the database can determine how and what to read from leveldb and/or
  ancients.

There are basically four types of requests; three of them are improved this way. The
fourth, by hash going backwards, is more tricky to optimize. However, since we know that
the gap is 0, we can look up by the parentHash, and stlil shave off all the number->hash
lookups.

The gapped collection can be optimized similarly, as a follow-up, at least in three out of
four cases.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-12-07 17:50:58 +01:00
Péter Szilágyi
58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync 2021-12-03 12:32:41 +02:00
Péter Szilágyi
c893488349
eth: pre-process downloader responses on the peer reader thread 2021-12-01 20:18:12 +02:00
Péter Szilágyi
c10a0a62c3
eth: request id dispatcher and direct req/reply APIs (#23576)
* eth: request ID based message dispatcher

* eth: fix dispatcher cancellation, rework fetchers idleness tracker

* eth/downloader: drop peers who refuse to serve advertised chains
2021-11-26 13:26:03 +02:00
Marius van der Wijden
3038e480f5
all: core rework for the merge transition (#23761)
* all: work for eth1/2 transtition

* consensus/beacon, eth: change beacon difficulty to 0

* eth: updates

* all: add terminalBlockDifficulty config, fix rebasing issues

* eth: implemented merge interop spec

* internal/ethapi: update to v1.0.0.alpha.2

                                                                 This commit updates the code to the new spec, moving payloadId into
                                                                 it's own object. It also fixes an issue with finalizing an empty blockhash.
                                                                 It also properly sets the basefee

* all: sync polishes, other fixes + refactors

* core, eth: correct semantics for LeavePoW, EnterPoS

* core: fixed rebasing artifacts

* core: light: performance improvements

* core: use keyed field (f)

* core: eth: fix compilation issues + tests

* eth/catalyst: dbetter error codes

* all: move Merger to consensus/, remove reliance on it in bc

* all: renamed EnterPoS and LeavePoW to ReachTDD and FinalizePoS

* core: make mergelogs a function

* core: use InsertChain instead of InsertBlock

* les: drop merger from lightchain object

* consensus: add merger

* core: recoverAncestors in catalyst mode

* core: fix nitpick

* all: removed merger from beacon, use TTD, nitpicks

* consensus: eth: add docstring, removed unnecessary code duplication

* consensus/beacon: better comment

* all: easy to fix nitpicks by karalabe

* consensus/beacon: verify known headers to be sure

* core: comments

* core: eth: don't drop peers who advertise blocks, nitpicks

* core: never add beacon blocks to the future queue

* core: fixed nitpicks

* consensus/beacon: simplify IsTTDReached check

* consensus/beacon: correct IsTTDReached check

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-11-26 13:23:02 +02:00
Péter Szilágyi
d9c13d407f
core, eth/downloader: fix resetting below freezer threshold 2021-11-22 11:12:51 +02:00
Ferran Borreguero
e4eceefbb0 Merge conflicts 2021-10-14 12:50:41 +02:00
Ferran Borreguero
54d3cf40e5 Limit binary search to local height 2021-09-20 10:30:10 +02:00
Péter Szilágyi
0db0b27754
Revert "Revert "eth: drop eth/65, the last non-reqid protocol version" (#23426)" (#23456)
This reverts commit c368f728c1.
2021-08-24 21:52:58 +03:00