mirror of
https://github.com/ethereum/go-ethereum.git
synced 2026-08-01 09:33:46 +00:00
37 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Jaynti Kanani
|
5da86c0df4
|
Bor update 1.9.16 (#77)
* cmd, consensus: add option to disable mmap for DAG caches/datasets (#20484) * cmd, consensus: add option to disable mmap for DAG caches/datasets * consensus: add benchmarks for mmap with/with lock * cmd/clef: add newaccount command (#20782) * cmd/clef: add newaccount command * cmd/clef: document clef_New, update API versioning * Update cmd/clef/intapi_changelog.md Co-Authored-By: ligi <ligi@ligi.de> * Update signer/core/uiapi.go Co-Authored-By: ligi <ligi@ligi.de> Co-authored-by: ligi <ligi@ligi.de> * eth: add debug_accountRange API (#19645) This new API allows reading accounts and their content by address range. Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Felix Lange <fjl@twurst.com> * travis: allow cocoapods deploy to fail (#20833) * whisper/whisperv6: decrease pow requirement in tests (#20815) * metrics: improve TestTimerFunc (#20818) The test failed due to what appears to be fluctuations in time.Sleep, which is not the actual method under test. This change modifies it so we compare the metered Max to the actual time instead of the desired time. * les: create utilities as common package (#20509) * les: move execqueue into utilities package execqueue is a util for executing queued functions in a serial order which is used by both les server and les client. Move it to common package. * les: move randselect to utilities package weighted_random_selector is a helpful tool for randomly select items maintained in a set but based on the item weight. It's used anywhere is LES package, mainly by les client but will be used in les server with very high chance. So move it into a common package as the second step for les separation. * les: rename to utils * README: update private network genesis spec with istanbul (#20841) * add istanbul and muirGlacier to genesis states in README * remove muirGlacier, relocate istanbul * whisper: fix whisper go routine leak with sync wait group (#20844) * cmd/evm: Rework execution stats (#20792) - Dump stats also for --bench flag. - From memory stats only show number and size of allocations. This is what `test -bench` shows. I doubt others like number of GC runs are any useful, but can be added if requested. - Now the mem stats are for single execution in case of --bench. * cmd/devp2p, cmd/wnode, whisper: add missing calls to Timer.Stop (#20843) * p2p/server: add UDP port mapping goroutine to wait group (#20846) * accounts/abi faster unpacking of int256 (#20850) * p2p/discv5: add missing Timer.Stop calls (#20853) * miner/worker: add missing timer.Stop call (#20857) * cmd/geth: fix bad genesis test (#20860) * eth/filters: add missing Ticker.Stop call (#20862) * eth/fetcher: add missing timer.Stop calls (#20861) * les: add missing Ticker.Stop call (#20864) * event: add missing timer.Stop call in TestFeed (#20868) * metrics: add missing calls to Ticker.Stop in tests (#20866) * ethstats: add missing Ticker.Stop call (#20867) * p2p/discv5, p2p/testing: add missing Timer.Stop calls in tests (#20869) * core: add missing Timer.Stop call in TestLogReorgs (#20870) * rpc: add missing timer.Stop calls in websocket tests (#20863) * crypto/ecies: improve concatKDF (#20836) This removes a bunch of weird code around the counter overflow check in concatKDF and makes it actually work for different hash output sizes. The overflow check worked as follows: concatKDF applies the hash function N times, where N is roundup(kdLen, hashsize) / hashsize. N should not overflow 32 bits because that would lead to a repetition in the KDF output. A couple issues with the overflow check: - It used the hash.BlockSize, which is wrong because the block size is about the input of the hash function. Luckily, all standard hash functions have a block size that's greater than the output size, so concatKDF didn't crash, it just generated too much key material. - The check used big.Int to compare against 2^32-1. - The calculation could still overflow before reaching the check. The new code in concatKDF doesn't check for overflow. Instead, there is a new check on ECIESParams which ensures that params.KeyLen is < 512. This removes any possibility of overflow. There are a couple of miscellaneous improvements bundled in with this change: - The key buffer is pre-allocated instead of appending the hash output to an initially empty slice. - The code that uses concatKDF to derive keys is now shared between Encrypt and Decrypt. - There was a redundant invocation of IsOnCurve in Decrypt. This is now removed because elliptic.Unmarshal already checks whether the input is a valid curve point since Go 1.5. Co-authored-by: Felix Lange <fjl@twurst.com> * rpc: metrics for JSON-RPC method calls (#20847) This adds a couple of metrics for tracking the timing and frequency of method calls: - rpc/requests gauge counts all requests - rpc/success gauge counts requests which return err == nil - rpc/failure gauge counts requests which return err != nil - rpc/duration/all timer tracks timing of all requests - rpc/duration/<method>/<success/failure> tracks per-method timing * mobile: use bind.NewKeyedTransactor instead of duplicating (#20888) It's better to reuse the existing code to create a keyed transactor than to rewrite the logic again. * all: fix a bunch of inconsequential goroutine leaks (#20667) The leaks were mostly in unit tests, and could all be resolved by adding suitably-sized channel buffers or by restructuring the test to not send on a channel after an error has occurred. There is an unavoidable goroutine leak in Console.Interactive: when we receive a signal, the line reader cannot be unblocked and will get stuck. This leak is now documented and I've tried to make it slightly less bad by adding a one-element buffer to the output channels of the line-reading loop. Should the reader eventually awake from its blocked state (i.e. when stdin is closed), at least it won't get stuck trying to send to the interpreter loop which has quit long ago. Co-authored-by: Felix Lange <fjl@twurst.com> * internal/ethapi: add CallArgs.ToMessage method (#20854) ToMessage is used to convert between ethapi.CallArgs and types.Message. It reduces the length of the DoCall method by about half by abstracting out the conversion between the CallArgs and the Message. This should improve the code's maintainability and reusability. * eth, les: fix flaky tests (#20897) * les: fix flaky test * eth: fix flaky test * cmd/geth: enable metrics for geth import command (#20738) * cmd/geth: enable metrics for geth import command * cmd/geth: enable metrics-flags for import command * core/vm: use a callcontext struct (#20761) * core/vm: use a callcontext struct * core/vm: fix tests * core/vm/runtime: benchmark * core/vm: make intpool push inlineable, unexpose callcontext * docs/audits: add discv5 protocol audits from LA and C53 (#20898) * .github: change gitter reference to discord link in issue template (#20896) * couple of fixes to docs in clef (#20900) * p2p/discover: add initial discovery v5 implementation (#20750) This adds an implementation of the current discovery v5 spec. There is full integration with cmd/devp2p and enode.Iterator in this version. In theory we could enable the new protocol as a replacement of discovery v4 at any time. In practice, there will likely be a few more changes to the spec and implementation before this can happen. * build: upgrade to golangci-lint 1.24.0 (#20901) * accounts/scwallet: remove unnecessary uses of fmt.Sprintf * cmd/puppeth: remove unnecessary uses of fmt.Sprintf * p2p/discv5: remove unnecessary use of fmt.Sprintf * whisper/mailserver: remove unnecessary uses of fmt.Sprintf * core: goimports -w tx_pool_test.go * eth/downloader: goimports -w downloader_test.go * build: upgrade to golangci-lint 1.24.0 * whisper/mailserver : recover corrupt db files before opening (#20891) * whisper/mailserver : recover db file when openfile corrupted * whisper/mailserver : fix db -> s.db * whisper/mailserver : common/errors for dbfile * accounts/abi/bind: Refactored topics (#20851) * accounts/abi/bind: refactored topics * accounts/abi/bind: use store function to remove code duplication * accounts/abi/bind: removed unused type defs * accounts/abi/bind: error on tuples in topics * Cosmetic changes to restart travis build Co-authored-by: Guillaume Ballet <gballet@gmail.com> * node: allow websocket and HTTP on the same port (#20810) This change makes it possible to run geth with JSON-RPC over HTTP and WebSocket on the same TCP port. The default port for WebSocket is still 8546. geth --rpc --rpcport 8545 --ws --wsport 8545 This also removes a lot of deprecated API surface from package rpc. The rpc package is now purely about serving JSON-RPC and no longer provides a way to start an HTTP server. * crypto: improve error messages in LoadECDSA (#20718) This improves error messages when the file is too short or too long. Also rewrite the test for SaveECDSA because LoadECDSA has its own test now. Co-authored-by: Felix Lange <fjl@twurst.com> * changed date of rpcstack.go since new file (#20904) * accounts/abi/bind: fixed erroneous filtering of negative ints (#20865) * accounts/abi/bind: fixed erroneous packing of negative ints * accounts/abi/bind: added test cases for negative ints in topics * accounts/abi/bind: fixed genIntType for go 1.12 * accounts/abi: minor nitpick * cmd: deprecate --testnet, use named networks instead (#20852) * cmd/utils: make goerli the default testnet * cmd/geth: explicitly rename testnet to ropsten * core: explicitly rename testnet to ropsten * params: explicitly rename testnet to ropsten * cmd: explicitly rename testnet to ropsten * miner: explicitly rename testnet to ropsten * mobile: allow for returning the goerli spec * tests: explicitly rename testnet to ropsten * docs: update readme to reflect changes to the default testnet * mobile: allow for configuring goerli and rinkeby nodes * cmd/geth: revert --testnet back to ropsten and mark as legacy * cmd/util: mark --testnet flag as deprecated * docs: update readme to properly reflect the 3 testnets * cmd/utils: add an explicit deprecation warning on startup * cmd/utils: swap goerli and ropsten in usage * cmd/geth: swap goerli and ropsten in usage * cmd/geth: if running a known preset, log it for convenience * docs: improve readme on usage of ropsten's testnet datadir * cmd/utils: check if legacy `testnet` datadir exists for ropsten * cmd/geth: check for legacy testnet path in console command * cmd/geth: use switch statement for complex conditions in main * cmd/geth: move known preset log statement to the very top * cmd/utils: create new ropsten configurations in the ropsten datadir * cmd/utils: makedatadir should check for existing testnet dir * cmd/geth: add legacy testnet flag to the copy db command * cmd/geth: add legacy testnet flag to the inspect command * les, les/lespay/client: add service value statistics and API (#20837) This PR adds service value measurement statistics to the light client. It also adds a private API that makes these statistics accessible. A follow-up PR will add the new server pool which uses these statistics to select servers with good performance. This document describes the function of the new components: https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4 Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> * README: update min go version to 1.13 (#20911) * travis, appveyor, build, Dockerfile: bump Go to 1.14.2 (#20913) * travis, appveyor, build, Dockerfile: bump Go to 1.14.2 * travis, appveyor: force GO111MODULE=on for every build * core/rawdb: fix data race between Retrieve and Close (#20919) * core/rawdb: fixed data race between retrieve and close closes https://github.com/ethereum/go-ethereum/issues/20420 * core/rawdb: use non-atomic load while holding mutex * accounts/abi: implement new fallback functions (#20764) * accounts/abi: implement new fackball functions In Solidity v0.6.0, the original fallback is separated into two different sub types: fallback and receive. This PR addes the support for parsing new format abi and the relevant abigen functionalities. * accounts/abi: fix unit tests * accounts/abi: minor fixes * accounts/abi, mobile: support jave binding * accounts/abi: address marius's comment * accounts/abi: Work around the uin64 conversion issue Co-authored-by: Guillaume Ballet <gballet@gmail.com> * trie: fix concurrent usage of secKeyBuf, ref #20920 * all: simplify and fix database iteration with prefix/start (#20808) * core/state/snapshot: start fixing disk iterator seek * ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start * les, core/state/snapshot: iterator fixes * all: remove two iterator methods * all: rename Iteratee.NewIteratorWith -> NewIterator * ethdb: fix review concerns * params: update CHTs for the 1.9.13 release * params: release Geth v1.9.13 * params: begin v1.9.14 release cycle * p2p/discover: add helper methods to UDPv5 (#20918) This adds two new methods to UDPv5, AllNodes and LocalNode. AllNodes returns all the nodes stored in the local table; this is useful for the purposes of metrics collection and also debugging any potential issues with other discovery v5 implementations. LocalNode returns the local node object. The reason for exposing this is so that users can modify and set/delete new key-value entries in the local record. * event: fix inconsistency in Lock and Unlock (#20933) Co-authored-by: Felix Lange <fjl@twurst.com> * accounts/abi: Prevent recalculation of internal fields (#20895) * accounts/abi: prevent recalculation of ID, Sig and String * accounts/abi: fixed unpacking of no values * accounts/abi: multiple fixes to arguments * accounts/abi: refactored methodName and eventName This commit moves the complicated logic of how we assign method names and event names if they already exist into their own functions for better readability. * accounts/abi: prevent recalculation of internal In this commit, I changed the way we calculate the string representations, sig representations and the id's of methods. Before that these fields would be recalculated everytime someone called .Sig() .String() or .ID() on a method or an event. Additionally this commit fixes issue #20856 as we assign names to inputs with no name (input with name "" becomes "arg0") * accounts/abi: added unnamed event params test * accounts/abi: fixed rebasing errors in method sig * accounts/abi: fixed rebasing errors in method sig * accounts/abi: addressed comments * accounts/abi: added FunctionType enumeration * accounts/abi/bind: added test for unnamed arguments * accounts/abi: improved readability in NewMethod, nitpicks * accounts/abi: method/eventName -> overloadedMethodName * cmd, core: remove override muir glacier and override istanbul (#20942) * snapshot: add Unlock before return (#20948) * Forget Unlock in snapshot * Remove Unlock before panic * core/state/snapshot: make difflayer account iterator seek operation inclusive * p2p: defer wait group done in protocol start (#20951) * go.mod : update fastcache to 1.5.7 (#20936) * all: seperate consensus error and evm internal error (#20830) * all: seperate consensus error and evm internal error There are actually two types of error will be returned when a tranaction/message call is executed: (a) consensus error (b) evm internal error. The former should be converted to a consensus issue, e.g. The sender doesn't enough asset to purchase the gas it specifies. The latter is allowed since evm itself is a blackbox and internal error is allowed to happen. This PR emphasizes the difference by introducing a executionResult structure. The evm error is embedded inside. So if any error returned, it indicates consensus issue happens. And also this PR improve the `EstimateGas` API to return the concrete revert reason if the transaction always fails * all: polish * accounts/abi/bind/backends: add tests * accounts/abi/bind/backends, internal: cleanup error message * all: address comments * core: fix lint * accounts, core, eth, internal: address comments * accounts, internal: resolve revert reason if possible * accounts, internal: address comments * core: mirror full node reorg logic in light client too (#20931) * core: fix the condition of reorg * core: fix nitpick to only retrieve head once * core: don't reorg if received chain is longer at same diff Co-authored-by: Péter Szilágyi <peterke@gmail.com> * accounts/keystore: fix double import race (#20915) * accounts/keystore: fix race in Import/ImportECDSA * accounts/keystore: added import/export tests * cmd/geth: improved TestAccountImport test * accounts/keystore: added import/export tests * accounts/keystore: fixed naming * accounts/keystore: fixed typo * accounts/keystore: use mutex instead of rwmutex * accounts: use errors instead of fmt * trie: initial implementation for range proof (#20908) * trie: initial implementation for range proof * trie: add benchmark * trie: fix lint * trie: fix minor issue * trie: unset the edge valuenode as well * trie: unset the edge valuenode as nilValuenode * core/state/snapshot: fix binary iterator (#20970) * p2p/enode: update code comment (#20972) It is possible to specify enode URLs using domain name since commit |
||
|
Arpit Agarwal
|
c2ca66adad
|
Upgrade geth to v1.9.14 (#75)
* fix: resolve conflict * chg: use bor url for import * Fix regressions from merge Co-authored-by: Jaynti Kanani <jdkanani@gmail.com> |
||
|
atvanguard
|
122a2304db | minor: Run gofmt -w . | ||
|
Jaynti Kanani
|
0f172fd0eb
|
change package name | ||
|
Jaynti Kanani
|
cc9b230fc0
|
merge release/1.9 | ||
|
lash
|
2f5b6cb442 |
swarm/network: Use different privatekey for bzz overlay in sim (#19313)
* cmd/swarm, p2p, swarm: Enable ENR in binary/execadapter * cmd/p2p/swarm: Remove comments + config.Enode nomarshal * p2p/simulations: Remove superfluous error check * p2p/simulation: Move init enode comment * swarm, p2p/simulations, cmd/swarm: Use nodekey in binary record sign * swarm/network, swarm/pss: Dervice bzzkey * swarm/pss: Remove unused function * swarm/network: Store swarm private key in simulation bucket * swarm/pss: Shorten TextProxNetwork shortrunning test timeout * swarm/pss: Increase prox test timeout * swarm/pss: Increase timeout slightly on shortrunning proxtest * swarm/network: Simplify bucket instantiation in servicectx func * p2p/simulations: Tcpport -> udpport * swarm/network, swarm/pss: Simplify + correct lock in servicefunc sim * swarm/network: Cleanup after rebase on extract swarm enode new * p2p/simulations, swarm/network: Make exec disc test pass * swarm/network: Prune ye olde comment * swarm/pss: Correct revised bzzkey method call * swarm/network: Clarify comment about privatekey generation data * swarm/pss: Fix syntax errors after rebase * swarm/network: Rename misleadingly named method (amend commit to trigger ci - attempt 5) |
||
|
lash
|
09924cbcaa |
cmd/swarm, p2p, swarm: Enable ENR in binary/execadapter (#19309)
* cmd/swarm, p2p, swarm: Enable ENR in binary/execadapter * cmd/p2p/swarm: Remove comments + config.Enode nomarshal * p2p/simulations: Remove superfluous error check * p2p/simulation: Move init enode comment * swarm/api: Check error in config test * swarm, p2p/simulations, cmd/swarm: Use nodekey in binary record sign * cmd/swarm: Make nodekey available for swarm api config |
||
|
lash
|
4b4f03ca37 |
swarm, p2p: Prerequities for ENR replacing handshake (#19275)
* swarm/api, swarm/network, p2p/simulations: Prerequisites for handshake remove * swarm, p2p: Add full sim node configs for protocoltester * swarm/network: Make stream package pass tests * swarm/network: Extract peer and addr types out of protocol file * p2p, swarm: Make p2p/protocols tests pass + rename types.go * swarm/network: Deactivate ExecAdapter test until binary ENR prep * swarm/api: Remove comments * swarm/network: Uncomment bootnode record load |
||
|
holisticode
|
81ed700157 |
Enable longrunning tests to run (#19208)
* p2p/simulations: increased snapshot load timeout for debugging * swarm/network/stream: less nodes for snapshot longrunning tests * swarm/network: fixed longrunning tests * swarm/network/stream: store kademlia in bucket * swarm/network/stream: disabled healthy check in delivery tests * swarm/network/stream: longer SyncUpdateDelay for longrunning tests * swarm/network/stream: more debug output * swarm/network/stream: reduced longrunning snapshot tests to 64 nodes * swarm/network/stream: don't WaitTillHealthy in SyncerSimulation * swarm/network/stream: cleanup for PR |
||
|
Felföldi Zsolt
|
c2003ed63b |
les, les/flowcontrol: improved request serving and flow control (#18230)
This change - implements concurrent LES request serving even for a single peer. - replaces the request cost estimation method with a cost table based on benchmarks which gives much more consistent results. Until now the allowed number of light peers was just a guess which probably contributed a lot to the fluctuating quality of available service. Everything related to request cost is implemented in a single object, the 'cost tracker'. It uses a fixed cost table with a global 'correction factor'. Benchmark code is included and can be run at any time to adapt costs to low-level implementation changes. - reimplements flowcontrol.ClientManager in a cleaner and more efficient way, with added capabilities: There is now control over bandwidth, which allows using the flow control parameters for client prioritization. Target utilization over 100 percent is now supported to model concurrent request processing. Total serving bandwidth is reduced during block processing to prevent database contention. - implements an RPC API for the LES servers allowing server operators to assign priority bandwidth to certain clients and change prioritized status even while the client is connected. The new API is meant for cases where server operators charge for LES using an off-protocol mechanism. - adds a unit test for the new client manager. - adds an end-to-end test using the network simulator that tests bandwidth control functions through the new API. |
||
|
Matthew Halpern
|
d3ccedc767 | p2p/simulations: enforce camel case variable names (#19053) | ||
|
Ferenc Szabo
|
50b872bf05 |
p2p, swarm: fix node up races by granular locking (#18976)
* swarm/network: DRY out repeated giga comment I not necessarily agree with the way we wait for event propagation. But I truly disagree with having duplicated giga comments. * p2p/simulations: encapsulate Node.Up field so we avoid data races The Node.Up field was accessed concurrently without "proper" locking. There was a lock on Network and that was used sometimes to access the field. Other times the locking was missed and we had a data race. For example: https://github.com/ethereum/go-ethereum/pull/18464 The case above was solved, but there were still intermittent/hard to reproduce races. So let's solve the issue permanently. resolves: ethersphere/go-ethereum#1146 * p2p/simulations: fix unmarshal of simulations.Node Making Node.Up field private in |
||
|
Janoš Guljaš
|
3fd6db2bf6 |
swarm: fix network/stream data races (#19051)
* swarm/network/stream: newStreamerTester cleanup only if err is nil * swarm/network/stream: raise newStreamerTester waitForPeers timeout * swarm/network/stream: fix data races in GetPeerSubscriptions * swarm/storage: prevent data race on LDBStore.batchesC https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461775049 * swarm/network/stream: fix TestGetSubscriptionsRPC data race https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461768477 * swarm/network/stream: correctly use Simulation.Run callback https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461783804 * swarm/network: protect addrCountC in Kademlia.AddrCountC function https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462273444 * p2p/simulations: fix a deadlock calling getRandomNode with lock https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462317407 * swarm/network/stream: terminate disconnect goruotines in tests * swarm/network/stream: reduce memory consumption when testing data races * swarm/network/stream: add watchDisconnections helper function * swarm/network/stream: add concurrent counter for tests * swarm/network/stream: rename race/norace test files and use const * swarm/network/stream: remove watchSim and its panic * swarm/network/stream: pass context in watchDisconnections * swarm/network/stream: add concurrent safe bool for watchDisconnections * swarm/storage: fix LDBStore.batchesC data race by not closing it |
||
|
Janoš Guljaš
|
26aea73673 |
cmd, node, p2p/simulations: fix node account manager leak (#19004)
* node: close AccountsManager in new Close method * p2p/simulations, p2p/simulations/adapters: handle node close on shutdown * node: move node ephemeralKeystore cleanup to stop method * node: call Stop in Node.Close method * cmd/geth: close node.Node created with makeFullNode in cli commands * node: close Node instances in tests * cmd/geth, node: minor code style fixes * cmd, console, miner, mobile: proper node Close() termination |
||
|
Elad
|
85a79b3ad3 | p2p/simulations: fix data race on swarm/network/simulations (#18464) | ||
|
Ferenc Szabo
|
2eb838ed97 | p2p/simulations: eliminate concept of pivot (#18426) | ||
|
Ferenc Szabo
|
fe03b76ffe |
A few minor code inspection fixes (#18393)
* swarm/network: fix code inspection problems - typos - redundant import alias * p2p/simulations: fix code inspection problems - typos - unused function parameters - redundant import alias - code style issue: snake case * swarm/network: fix unused method parameters inspections |
||
|
lash
|
e1edfe0689 |
p2p/simulation: Test snapshot correctness and minimal benchmark (#18287)
* p2p/simulation: WIP minimal snapshot test * p2p/simulation: Add snapshot create, load and verify to snapshot test * build: add test tag for tests * p2p/simulations, build: Revert travis change, build test sym always * p2p/simulations: Add comments, timeout check on additional events * p2p/simulation: Add benchmark template for minimal peer protocol init * p2p/simulations: Remove unused code * p2p/simulation: Correct timer reset * p2p/simulations: Put snapshot check events in buffer and call blocking * p2p/simulations: TestSnapshot fail if Load function returns early * p2p/simulations: TestSnapshot wait for all connections before returning * p2p/simulation: Revert to before wait for snap load (5e75594) * p2p/simulations: add "conns after load" subtest to TestSnapshot and nudge |
||
|
Elad
|
472c23a801 | p2p/simulation: move connection methods from swarm/network/simulation (#18323) | ||
|
Janoš Guljaš
|
661809714e |
swarm: snapshot load improvement (#18220)
* swarm/network: Hive - do not notify peer if discovery is disabled * p2p/simulations: validate all connections on loading a snapshot * p2p/simulations: track all connections in on snapshot loading * p2p/simulations: add snapshotLoadTimeout variable * p2p/simulations: ignore control events in snapshot load * p2p/simulations: simplify event loop synchronization * p2p/simulations: return already connected error from Load function * p2p/simulations: log warning on snapshot loading disconnection |
||
|
lash
|
201a0bf181 |
p2p/simulations, swarm/network: Custom services in snapshot (#17991)
* p2p/simulations: Add custom services to simnodes + remove sim down conn objs * p2p/simulation, swarm/network: Add selective services to discovery sim * p2p/simulations, swarm/network: Remove useless comments * p2p/simulations, swarm/network: Clean up mess from rebase * p2p/simulation: Add sleep to prevent connect flakiness in http test * p2p/simulations: added concurrent goroutines to prevent sleeps on simulation connect/disconnect * p2p/simulations, swarm/network/simulations: address pr comments * reinstated dummy service * fixed http snapshot test |
||
|
Felix Lange
|
dcae0d348b
|
p2p/simulations: fix a deadlock and clean up adapters (#17891)
This fixes a rare deadlock with the inproc adapter:
- A node is stopped, which acquires Network.lock.
- The protocol code being simulated (swarm/network in my case)
waits for its goroutines to shut down.
- One of those goroutines calls into the simulation to add a peer,
which waits for Network.lock.
The fix for the deadlock is really simple, just release the lock
before stopping the simulation node.
Other changes in this PR clean up the exec adapter so it reports
node startup errors better and remove the docker adapter because
it just adds overhead.
In the exec adapter, node information is now posted to a one-shot
server. This avoids log parsing and allows reporting startup
errors to the simulation host.
A small change in package node was needed because simulation
nodes use port zero. Node.{HTTP,WS}Endpoint now return the live
endpoints after startup by checking the TCP listener.
|
||
|
holisticode
|
11d0ff6578 |
Fix retrieval tests and simulation backends (#17723)
* swarm/network/stream: introduced visualized snapshot sync test * swarm/network/stream: non-existing hash visualization sim * swarm/network/stream: fixed retrieval tests; new backend for visualization * swarm/network/stream: cleanup of visualized_snapshot_sync_sim_test.go * swarm/network/stream: rebased PR on master * swarm/network/stream: fixed loop logic in retrieval tests * swarm/network/stream: fixed iterations for snapshot tests * swarm/network/stream: address PR comments * swarm/network/stream: addressed PR comments |
||
|
Felix Lange
|
30cd5c1854
|
all: new p2p node representation (#17643)
Package p2p/enode provides a generalized representation of p2p nodes
which can contain arbitrary information in key/value pairs. It is also
the new home for the node database. The "v4" identity scheme is also
moved here from p2p/enr to remove the dependency on Ethereum crypto from
that package.
Record signature handling is changed significantly. The identity scheme
registry is removed and acceptable schemes must be passed to any method
that needs identity. This means records must now be validated explicitly
after decoding.
The enode API is designed to make signature handling easy and safe: most
APIs around the codebase work with enode.Node, which is a wrapper around
a valid record. Going from enr.Record to enode.Node requires a valid
signature.
* p2p/discover: port to p2p/enode
This ports the discovery code to the new node representation in
p2p/enode. The wire protocol is unchanged, this can be considered a
refactoring change. The Kademlia table can now deal with nodes using an
arbitrary identity scheme. This requires a few incompatible API changes:
- Table.Lookup is not available anymore. It used to take a public key
as argument because v4 protocol requires one. Its replacement is
LookupRandom.
- Table.Resolve takes *enode.Node instead of NodeID. This is also for
v4 protocol compatibility because nodes cannot be looked up by ID
alone.
- Types Node and NodeID are gone. Further commits in the series will be
fixes all over the the codebase to deal with those removals.
* p2p: port to p2p/enode and discovery changes
This adapts package p2p to the changes in p2p/discover. All uses of
discover.Node and discover.NodeID are replaced by their equivalents from
p2p/enode.
New API is added to retrieve the enode.Node instance of a peer. The
behavior of Server.Self with discovery disabled is improved. It now
tries much harder to report a working IP address, falling back to
127.0.0.1 if no suitable address can be determined through other means.
These changes were needed for tests of other packages later in the
series.
* p2p/simulations, p2p/testing: port to p2p/enode
No surprises here, mostly replacements of discover.Node, discover.NodeID
with their new equivalents. The 'interesting' API changes are:
- testing.ProtocolSession tracks complete nodes, not just their IDs.
- adapters.NodeConfig has a new method to create a complete node.
These changes were needed to make swarm tests work.
Note that the NodeID change makes the code incompatible with old
simulation snapshots.
* whisper/whisperv5, whisper/whisperv6: port to p2p/enode
This port was easy because whisper uses []byte for node IDs and
URL strings in the API.
* eth: port to p2p/enode
Again, easy to port because eth uses strings for node IDs and doesn't
care about node information in any way.
* les: port to p2p/enode
Apart from replacing discover.NodeID with enode.ID, most changes are in
the server pool code. It now deals with complete nodes instead
of (Pubkey, IP, Port) triples. The database format is unchanged for now,
but we should probably change it to use the node database later.
* node: port to p2p/enode
This change simply replaces discover.Node and discover.NodeID with their
new equivalents.
* swarm/network: port to p2p/enode
Swarm has its own node address representation, BzzAddr, containing both
an overlay address (the hash of a secp256k1 public key) and an underlay
address (enode:// URL).
There are no changes to the BzzAddr format in this commit, but certain
operations such as creating a BzzAddr from a node ID are now impossible
because node IDs aren't public keys anymore.
Most swarm-related changes in the series remove uses of
NewAddrFromNodeID, replacing it with NewAddr which takes a complete node
as argument. ToOverlayAddr is removed because we can just use the node
ID directly.
|
||
|
Oleg Kovalov
|
d42ce0f2c1 |
all: simplify switches (#17267)
* all: simplify switches * silly mistake |
||
|
Janoš Guljaš
|
dcaaa3c804 |
swarm: network simulation for swarm tests (#769)
* cmd/swarm: minor cli flag text adjustments * cmd/swarm, swarm/storage, swarm: fix mingw on windows test issues * cmd/swarm: support for smoke tests on the production swarm cluster * cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion * changed colour of landing page * landing page reacts to enter keypress * swarm/api/http: sticky footer for swarm landing page using flex * swarm/api/http: sticky footer for error pages and fix for multiple choices * swarm: propagate ctx to internal apis (#754) * swarm/simnet: add basic node/service functions * swarm/netsim: add buckets for global state and kademlia health check * swarm/netsim: Use sync.Map as bucket and provide cleanup function for... * swarm, swarm/netsim: adjust SwarmNetworkTest * swarm/netsim: fix tests * swarm: added visualization option to sim net redesign * swarm/netsim: support multiple services per node * swarm/netsim: remove redundant return statement * swarm/netsim: add comments * swarm: shutdown HTTP in Simulation.Close * swarm: sim HTTP server timeout * swarm/netsim: add more simulation methods and peer events examples * swarm/netsim: add WaitKademlia example * swarm/netsim: fix comments * swarm/netsim: terminate peer events goroutines on simulation done * swarm, swarm/netsim: naming updates * swarm/netsim: return not healthy kademlias on WaitTillHealthy * swarm: fix WaitTillHealthy call in testSwarmNetwork * swarm/netsim: allow bucket to have any type for a key * swarm: Added snapshots to new netsim * swarm/netsim: add more tests for bucket * swarm/netsim: move http related things into separate files * swarm/netsim: add AddNodeWithService option * swarm/netsim: add more tests and Start* methods * swarm/netsim: add peer events and kademlia tests * swarm/netsim: fix some tests flakiness * swarm/netsim: improve random nodes selection, fix TestStartStop* tests * swarm/netsim: remove time measurement from TestClose to avoid flakiness * swarm/netsim: builder pattern for netsim HTTP server (#773) * swarm/netsim: add connect related tests * swarm/netsim: add comment for TestPeerEvents * swarm: rename netsim package to network/simulation |
||
|
ethersphere
|
e187711c65 | swarm: network rewrite merge | ||
|
Elad
|
1836366ac1 |
all: library changes for swarm-network-rewrite (#16898)
This commit adds all changes needed for the merge of swarm-network-rewrite. The changes: - build: increase linter timeout - contracts/ens: export ensNode - log: add Output method and enable fractional seconds in format - metrics: relax test timeout - p2p: reduced some log levels, updates to simulation packages - rpc: increased maxClientSubscriptionBuffer to 20000 |
||
|
Ivan Daniluk
|
c4a4613d95 | p2p/simulations/adapters: fix websocket log line parsing in exec adapter (#16667) | ||
|
kiel barry
|
864e80a48f | p2p: fix some golint warnings (#16577) | ||
|
GagziW
|
f2447bd4c3 | p2p: changed if-else blocks to conform with golint (#16660) | ||
|
Felix Lange
|
a5c0bbb4f4
|
all: update license information (#16089) | ||
|
Felix Lange
|
3654aeaa4f
|
p2p/simulations: fix gosimple nit (#15661) | ||
|
holisticode
|
fd777bb210 |
p2p/simulations: add mocker functionality (#15207)
This commit adds mocker functionality to p2p/simulations. A mocker allows to starting/stopping of nodes via the HTTP API. |
||
|
Zach
|
3da1bf8ca1 | all: use gometalinter.v2, fix new gosimple issues (#15650) | ||
|
Lewis Marshall
|
54aeb8e4c0 |
p2p/simulations: various stability fixes (#15198)
p2p/simulations: introduce dialBan - Refactor simulations/network connection getters to support avoiding simultaneous dials between two peers If two peers dial simultaneously, the connection will be dropped to help avoid that, we essentially lock the connection object with a timestamp which serves as a ban on dialing for a period of time (dialBanTimeout). - The connection getter InitConn can be wrapped and passed to the nodes via adapters.NodeConfig#Reachable field and then used by the respective services when they initiate connections. This massively stablise the emerging connectivity when running with hundreds of nodes bootstrapping a network. p2p: add Inbound public method to p2p.Peer p2p/simulations: Add server id to logs to support debugging in-memory network simulations when multiple peers are logging. p2p: SetupConn now returns error. The dialer checks the error and only calls resolve if the actual TCP dial fails. |
||
|
Lewis Marshall
|
9feec51e2d |
p2p: add network simulation framework (#14982)
This commit introduces a network simulation framework which can be used to run simulated networks of devp2p nodes. The intention is to use this for testing protocols, performing benchmarks and visualising emergent network behaviour. |