This changes how the downloader works, a little bit. Previously, when block sync started,
we immediately started filling up to 8192 blocks. Usually this is fine, blocks are small
in the early numbers. The threshold then is lowered as we measure the size of the blocks
that are filled.
However, if the node is shut down and restarts syncing while we're in a heavy segment,
that might be bad. This PR introduces a more conservative initial threshold of 2K blocks
instead.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Some tests take quite some time during exit, which I think causes
some appveyor fails like this:
https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3
One of the things that seem to take time during exit is waiting
(up to 100ms) for the syncbloom to close. This PR changes it to use
a channel, instead of looping with a 100ms wait.
This also includes some unrelated changes improving the reliability of
eth/fetcher tests, which fail a lot because they are time-dependent.
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/downloader: refactor downloader + queue
downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache
downloader: more accurate deliverytime calculation, less mem overhead in state requests
downloader/queue: increase underlying buffer of results, new throttle mechanism
eth/downloader: updates to tests
eth/downloader: fix up some review concerns
eth/downloader/queue: minor fixes
eth/downloader: minor fixes after review call
eth/downloader: testcases for queue.go
eth/downloader: minor change, don't set progress unless progress...
eth/downloader: fix flaw which prevented useless peers from being dropped
eth/downloader: try to fix tests
eth/downloader: verify non-deliveries against advertised remote head
eth/downloader: fix flaw with checking closed-status causing hang
eth/downloader: hashing avoidance
eth/downloader: review concerns + simplify resultcache and queue
eth/downloader: add back some locks, address review concerns
downloader/queue: fix remaining lock flaw
* eth/downloader: nitpick fixes
* eth/downloader: remove the *2*3/4 throttling threshold dance
* eth/downloader: print correct throttle threshold in stats
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
The test did not synchronize with per-case goroutines, and thus didn't notice
that some tests were just hanging. This change adds missing synchronization
and fixes the broken tests.
This PR modifies how the metrics library handles `Enabled`: previously,
the package `init` decided whether to serve real metrics or just
dummy-types.
This has several drawbacks:
- During pkg init, we need to determine whether metrics are enabled or
not. So we first hacked in a check if certain geth-specific
commandline-flags were enabled. Then we added a similar check for
geth-env-vars. Then we almost added a very elaborate check for
toml-config-file, plus toml parsing.
- Using "real" types and dummy types interchangeably means that
everything is hidden behind interfaces. This has a performance penalty,
and also it just adds a lot of code.
This PR removes the interface stuff, uses concrete types, and allows for
the setting of Enabled to happen later. It is still assumed that
`metrics.Enable()` is invoked early on.
The somewhat 'heavy' operations, such as ticking meters and exp-decay,
now checks the enable-flag to prevent resource leak.
The change may be large, but it's mostly pretty trivial, and from the
last time I gutted the metrics, I ensured that we have fairly good test
coverage.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
The name of a method’s receiver should be a reflection of its identity;
often a one or two letter abbreviation of its type suffices (such as
“c” or “cl” for “Client”). Don’t use generic names such as “me”, “this”
or “self”, identifiers typical of object-oriented languages that place
more emphasis on methods as opposed to functions. The name need not be
as descriptive as that of a method argument, as its role is obvious and
serves no documentary purpose. It can be very short as it will appear
on almost every line of every method of the type; familiarity admits
brevity. Be consistent, too: if you call the receiver “c” in one method,
don’t call it “cl” in another.
* intro new timeout (#651)
* intro new timeout
* correct comment
* disable ProcessForensics
* disable ProcessForensics
* change version
* enable periodicProfilingFlag
* fix: ignore old timeout msg
* fix: ignore old timeout msg including equal to the current round
* udpate version file
* feat: rename Tracer interface to EVMLogger;
minor changes in API
refine api_tracer.go
refine Tracer interface
* fix: broken tracer tests
* feat: add BenchmarkTransactionTrace
* feat: tracer CaptureEnter CaptureExit in evm
* feat: upgrade js tracers with geth upstream
* chore: clean test
* feat: eth/tracers: support for golang tracers + add golang callTracer
cf. https://github.com/ethereum/go-ethereum/pull/23708
* chore: clean testdata json
* fix: change test due to IntrinsicGas is not upgraded
* feat: make native Tracer the default Tracer
* fix: update tracers.New in api
* fix: addr prefix in callTracer
* fix: remove `native` in BenchmarkTracers
* fix: return consensus error of InsufficientBalance for tx, instead of vmerr
* chore: drop js tracers: call and noop
This PR adds some hardening in the lower levels of the protocol stack, to bail early on invalid data. Primarily, attacks that this PR protects against are on the "annoyance"-level, which would otherwise write a couple of megabytes of data into the log output, which is a bit resource intensive.
* eth: chain config (genesis + fork) ENR entry
* core/forkid, eth: protocol independent fork ID, update to CRC32 spec
* core/forkid, eth: make forkid a struct, next uint64, enr struct, RLP
* core/forkid: change forkhash rlp encoding from int to [4]byte
* eth: fixup eth entry a bit and update it every block
* eth: fix lint
* eth: fix crash in ethclient tests
Package p2p/enode provides a generalized representation of p2p nodes
which can contain arbitrary information in key/value pairs. It is also
the new home for the node database. The "v4" identity scheme is also
moved here from p2p/enr to remove the dependency on Ethereum crypto from
that package.
Record signature handling is changed significantly. The identity scheme
registry is removed and acceptable schemes must be passed to any method
that needs identity. This means records must now be validated explicitly
after decoding.
The enode API is designed to make signature handling easy and safe: most
APIs around the codebase work with enode.Node, which is a wrapper around
a valid record. Going from enr.Record to enode.Node requires a valid
signature.
* p2p/discover: port to p2p/enode
This ports the discovery code to the new node representation in
p2p/enode. The wire protocol is unchanged, this can be considered a
refactoring change. The Kademlia table can now deal with nodes using an
arbitrary identity scheme. This requires a few incompatible API changes:
- Table.Lookup is not available anymore. It used to take a public key
as argument because v4 protocol requires one. Its replacement is
LookupRandom.
- Table.Resolve takes *enode.Node instead of NodeID. This is also for
v4 protocol compatibility because nodes cannot be looked up by ID
alone.
- Types Node and NodeID are gone. Further commits in the series will be
fixes all over the the codebase to deal with those removals.
* p2p: port to p2p/enode and discovery changes
This adapts package p2p to the changes in p2p/discover. All uses of
discover.Node and discover.NodeID are replaced by their equivalents from
p2p/enode.
New API is added to retrieve the enode.Node instance of a peer. The
behavior of Server.Self with discovery disabled is improved. It now
tries much harder to report a working IP address, falling back to
127.0.0.1 if no suitable address can be determined through other means.
These changes were needed for tests of other packages later in the
series.
* p2p/simulations, p2p/testing: port to p2p/enode
No surprises here, mostly replacements of discover.Node, discover.NodeID
with their new equivalents. The 'interesting' API changes are:
- testing.ProtocolSession tracks complete nodes, not just their IDs.
- adapters.NodeConfig has a new method to create a complete node.
These changes were needed to make swarm tests work.
Note that the NodeID change makes the code incompatible with old
simulation snapshots.
* whisper/whisperv5, whisper/whisperv6: port to p2p/enode
This port was easy because whisper uses []byte for node IDs and
URL strings in the API.
* eth: port to p2p/enode
Again, easy to port because eth uses strings for node IDs and doesn't
care about node information in any way.
* les: port to p2p/enode
Apart from replacing discover.NodeID with enode.ID, most changes are in
the server pool code. It now deals with complete nodes instead
of (Pubkey, IP, Port) triples. The database format is unchanged for now,
but we should probably change it to use the node database later.
* node: port to p2p/enode
This change simply replaces discover.Node and discover.NodeID with their
new equivalents.
* swarm/network: port to p2p/enode
Swarm has its own node address representation, BzzAddr, containing both
an overlay address (the hash of a secp256k1 public key) and an underlay
address (enode:// URL).
There are no changes to the BzzAddr format in this commit, but certain
operations such as creating a BzzAddr from a node ID are now impossible
because node IDs aren't public keys anymore.
Most swarm-related changes in the series remove uses of
NewAddrFromNodeID, replacing it with NewAddr which takes a complete node
as argument. ToOverlayAddr is removed because we can just use the node
ID directly.
* core/vm: use fixed uint256 library instead of big
* core/vm: remove intpools
* core/vm: upgrade uint256, fixes uint256.NewFromBig
* core/vm: use uint256.Int by value in Stack
* core/vm: upgrade uint256 to v1.0.0
* core/vm: don't preallocate space for 1024 stack items (only 16)
Co-authored-by: Martin Holst Swende <martin@swende.se>
* V2 truncate MaxMasternodes from candidates after penalty,
V1 same as before
TestUpdateMultipleMasterNodes: test V2, in snapshot we have all candidates, but at epoch switch, we pick MaxMasternodes
* code looks better
* v2 Hook Reward, need test
* test reward
* fix RewardHook due to modifying params config directly (#56)
* more test
* finish test
Co-authored-by: Jerome <wjrjerome@gmail.com>
* add isEpochSwitch function and refactor utils
* fix broken first v2 epoch switch block
* use adaptor epoch switch function to determine v1 v2 epoch swtich block
* add test for the GetMasternodesByNumber and GetCurrentEpochSwitchBlock function
* add v2 test for isAuthroisedAddress
* Use GetCurrentEpochSwitchBlock in findNearestSignedBlock api
* New struct in consensus/XDPoS/utils/types.go, util functions, and test. (#14)
* define vote, timeout, sync info, qc, tc, extra fields in types.go, add test in types_test.go
* add json tag in types.go, refine encoder decoder of extra fields
* refactor types.go utils.go
* re-write types, comments
* add Hash SigHash for types, and tests
* define Round type
* remove unnecessary logs
* add v2 engine functions placeholder
* typo fix on the consensus v2 function placeholders
* add countdown timer
* make initilised private to countdown
* add v2 specific config struct
* rename some config variables
* Implement BFT Message receiver (#13)
* fix or skip tests due to PR-136 changes
* add bft receiver functions
* add bft receiver functions
* rename tc to TimeoutCert
* implement more functions
* New struct in consensus/XDPoS/utils/types.go, util functions, and test. (#14)
* define vote, timeout, sync info, qc, tc, extra fields in types.go, add test in types_test.go
* add json tag in types.go, refine encoder decoder of extra fields
* refactor types.go utils.go
* re-write types, comments
* add Hash SigHash for types, and tests
* define Round type
* remove unnecessary logs
* add temp functions
* add v2 engine functions placeholder
* typo fix on the consensus v2 function placeholders
* add countdown timer
* make initilised private to countdown
* push verify function
* add test on receiving vote
* revert type change
* add async on broadcast function
* add quit initial
* fix test
Co-authored-by: Jianrong <wjrjerome@gmail.com>
Co-authored-by: wgr523 <wgr523@gmail.com>
* generate and verify timeout message
* Consensus V2 variable, timeout pool (#19)
* fill in XDPoS_v2 variables and processQC/TC
* add timeout pool, refine engine variables
* refactor type functions
* solve a small pointer bug
* create general pool and its test, refine engine
* refine pool, add xdpos v2 config cert threshold
* refine config
* vote and timeout handlers
* fix pool test
* bft miner preparation
* review comment improvement
* update
* relocate tests
* add and remove comment
* fix the syntax error
* update network layer and add handler functions (#23)
* update network layer and add handler functions
* fix test syntax error
* add ProcessQC implementation
* add ProcessQC tests
* add snapshot test
* add wait qc process
* remove testing files
* add route snapshot
* fix merge issue
* add default v2 behaviour (#24)
* add v2 ecrecover functions and refactor test
* fix all the tests
* put minimun lock variable
* debugging prepare and seal v2 blocks
* Trigger proposeBlockHandler after v2 block received and verified in fetcher
* skip snapshot apply related tests
* update test check
* rename bfter to bft handler and ignore normal behviour
* fix bugs during local 4 node run
* fix test
* fix sync info test
* fix bugs during local 4 node run
* rebase and fix bug
* remove hook validators function"
Co-authored-by: wgr523 <wgr523@gmail.com>
Co-authored-by: Jianrong <wjrjerome@gmail.com>
* fix or skip tests due to PR-136 changes
* add bft receiver functions
* add bft receiver functions
* rename tc to TimeoutCert
* implement more functions
* New struct in consensus/XDPoS/utils/types.go, util functions, and test. (#14)
* define vote, timeout, sync info, qc, tc, extra fields in types.go, add test in types_test.go
* add json tag in types.go, refine encoder decoder of extra fields
* refactor types.go utils.go
* re-write types, comments
* add Hash SigHash for types, and tests
* define Round type
* remove unnecessary logs
* add temp functions
* add v2 engine functions placeholder
* typo fix on the consensus v2 function placeholders
* add countdown timer
* make initilised private to countdown
* push verify function
* add test on receiving vote
* revert type change
* add async on broadcast function
* add quit initial
* fix test
Co-authored-by: Jianrong <wjrjerome@gmail.com>
Co-authored-by: wgr523 <wgr523@gmail.com>
