Merge branch 'master' into eip191/712

README.md

@@ -1,6 +1,6 @@
 ## Go Ethereum
 
-Official golang implementation of the Ethereum protocol.
+Official Golang implementation of the Ethereum protocol.
 
 [![API Reference](
 https://camo.githubusercontent.com/915b7be44ada53c290eb157634330494ebe3e30a/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f676f6c616e672f6764646f3f7374617475732e737667
@@ -35,11 +35,11 @@ The go-ethereum project comes with several wrappers/executables found in the `cm
 | Command    | Description |
 |:----------:|-------------|
 | **`geth`** | Our main Ethereum CLI client. It is the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default), archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. `geth --help` and the [CLI Wiki page](https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options) for command line options. |
-| `abigen` | Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI) with expanded functionality if the contract bytecode is also available. However it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://github.com/ethereum/go-ethereum/wiki/Native-DApps:-Go-bindings-to-Ethereum-contracts) wiki page for details. |
+| `abigen` | Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI) with expanded functionality if the contract bytecode is also available. However, it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://github.com/ethereum/go-ethereum/wiki/Native-DApps:-Go-bindings-to-Ethereum-contracts) wiki page for details. |
 | `bootnode` | Stripped down version of our Ethereum client implementation that only takes part in the network node discovery protocol, but does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks. |
 | `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow isolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug`). |
 | `gethrpctest` | Developer utility tool to support our [ethereum/rpc-test](https://github.com/ethereum/rpc-tests) test suite which validates baseline conformity to the [Ethereum JSON RPC](https://github.com/ethereum/wiki/wiki/JSON-RPC) specs. Please see the [test suite's readme](https://github.com/ethereum/rpc-tests/blob/master/README.md) for details. |
-| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://github.com/ethereum/wiki/wiki/RLP)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
+| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://github.com/ethereum/wiki/wiki/RLP)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user-friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
 | `swarm`    | Swarm daemon and tools. This is the entry point for the Swarm network. `swarm --help` for command line options and subcommands. See [Swarm README](https://github.com/ethereum/go-ethereum/tree/master/swarm) for more information. |
 | `puppeth`    | a CLI wizard that aids in creating a new Ethereum network. |
 
@@ -72,7 +72,7 @@ This command will:
    This tool is optional and if you leave it out you can always attach to an already running Geth instance
    with `geth attach`.
 
-### Full node on the Ethereum test network
+### A Full node on the Ethereum test network
 
 Transitioning towards developers, if you'd like to play around with creating Ethereum contracts, you
 almost certainly would like to do that without any real money involved until you get the hang of the
@@ -83,10 +83,10 @@ network with your node, which is fully equivalent to the main network, but with
 $ geth --testnet console
 ```
 
-The `console` subcommand have the exact same meaning as above and they are equally useful on the
+The `console` subcommand has the exact same meaning as above and they are equally useful on the
-testnet too. Please see above for their explanations if you've skipped to here.
+testnet too. Please see above for their explanations if you've skipped here.
 
-Specifying the `--testnet` flag however will reconfigure your Geth instance a bit:
+Specifying the `--testnet` flag, however, will reconfigure your Geth instance a bit:
 
  * Instead of using the default data directory (`~/.ethereum` on Linux for example), Geth will nest
    itself one level deeper into a `testnet` subfolder (`~/.ethereum/testnet` on Linux). Note, on OSX
@@ -103,7 +103,7 @@ separate the two networks and will not make any accounts available between them.
 
 ### Full node on the Rinkeby test network
 
-The above test network is a cross client one based on the ethash proof-of-work consensus algorithm. As such, it has certain extra overhead and is more susceptible to reorganization attacks due to the network's low difficulty / security. Go Ethereum also supports connecting to a proof-of-authority based test network called [*Rinkeby*](https://www.rinkeby.io) (operated by members of the community). This network is lighter, more secure, but is only supported by go-ethereum.
+The above test network is a cross-client one based on the ethash proof-of-work consensus algorithm. As such, it has certain extra overhead and is more susceptible to reorganization attacks due to the network's low difficulty/security. Go Ethereum also supports connecting to a proof-of-authority based test network called [*Rinkeby*](https://www.rinkeby.io) (operated by members of the community). This network is lighter, more secure, but is only supported by go-ethereum.
 
 ```
 $ geth --rinkeby console
@@ -139,13 +139,13 @@ This will start geth in fast-sync mode with a DB memory allowance of 1GB just as
 
 Do not forget `--rpcaddr 0.0.0.0`, if you want to access RPC from other containers and/or hosts. By default, `geth` binds to the local interface and RPC endpoints is not accessible from the outside.
 
-### Programatically interfacing Geth nodes
+### Programmatically interfacing Geth nodes
 
 As a developer, sooner rather than later you'll want to start interacting with Geth and the Ethereum
 network via your own programs and not manually through the console. To aid this, Geth has built-in
 support for a JSON-RPC based APIs ([standard APIs](https://github.com/ethereum/wiki/wiki/JSON-RPC) and
 [Geth specific APIs](https://github.com/ethereum/go-ethereum/wiki/Management-APIs)). These can be
-exposed via HTTP, WebSockets and IPC (unix sockets on unix based platforms, and named pipes on Windows).
+exposed via HTTP, WebSockets and IPC (UNIX sockets on UNIX based platforms, and named pipes on Windows).
 
 The IPC interface is enabled by default and exposes all the APIs supported by Geth, whereas the HTTP
 and WS interfaces need to manually be enabled and only expose a subset of APIs due to security reasons.
@@ -241,7 +241,7 @@ that other nodes can use to connect to it and exchange peer information. Make su
 displayed IP address information (most probably `[::]`) with your externally accessible IP to get the
 actual `enode` URL.
 
-*Note: You could also use a full fledged Geth node as a bootnode, but it's the less recommended way.*
+*Note: You could also use a full-fledged Geth node as a bootnode, but it's the less recommended way.*
 
 #### Starting up your member nodes
 
@@ -264,7 +264,7 @@ an OpenCL or CUDA enabled `ethminer` instance. For information on such a setup,
 [EtherMining subreddit](https://www.reddit.com/r/EtherMining/) and the [Genoil miner](https://github.com/Genoil/cpp-ethereum)
 repository.
 
-In a private network setting however, a single CPU miner instance is more than enough for practical
+In a private network setting, however a single CPU miner instance is more than enough for practical
 purposes as it can produce a stable stream of blocks at the correct intervals without needing heavy
 resources (consider running on a single thread, no need for multiple ones either). To start a Geth
 instance for mining, run it with all your usual flags, extended by:
@@ -298,7 +298,7 @@ Please make sure your contributions adhere to our coding guidelines:
    * E.g. "eth, rpc: make trace configs optional"
 
 Please see the [Developers' Guide](https://github.com/ethereum/go-ethereum/wiki/Developers'-Guide)
-for more details on configuring your environment, managing project dependencies and testing procedures.
+for more details on configuring your environment, managing project dependencies, and testing procedures.
 
 ## License
 

accounts/accounts.go

@@ -18,12 +18,14 @@
 package accounts
 
 import (
+	"fmt"
 	"math/big"
 
 	ethereum "github.com/ethereum/go-ethereum"
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/core/types"
 	"github.com/ethereum/go-ethereum/event"
+	"golang.org/x/crypto/sha3"
 )
 
 // Account represents an Ethereum account located at a specific location defined
@@ -87,8 +89,7 @@ type Wallet interface {
 	// chain state reader.
 	SelfDerive(base DerivationPath, chain ethereum.ChainStateReader)
 
-	// SignHash requests the wallet to sign the given hash.
+	// SignData requests the wallet to sign the hash of the given data
-	//
 	// It looks up the account specified either solely via its address contained within,
 	// or optionally with the aid of any location metadata from the embedded URL field.
 	//
@@ -98,7 +99,20 @@ type Wallet interface {
 	// about which fields or actions are needed. The user may retry by providing
 	// the needed details via SignHashWithPassphrase, or by other means (e.g. unlock
 	// the account in a keystore).
-	SignHash(account Account, hash []byte) ([]byte, error)
+	SignData(account Account, mimeType string, data []byte) ([]byte, error)
+
+	// Signtext requests the wallet to sign the hash of a given piece of data, prefixed
+	// by the Ethereum prefix scheme
+	// It looks up the account specified either solely via its address contained within,
+	// or optionally with the aid of any location metadata from the embedded URL field.
+	//
+	// If the wallet requires additional authentication to sign the request (e.g.
+	// a password to decrypt the account, or a PIN code o verify the transaction),
+	// an AuthNeededError instance will be returned, containing infos for the user
+	// about which fields or actions are needed. The user may retry by providing
+	// the needed details via SignHashWithPassphrase, or by other means (e.g. unlock
+	// the account in a keystore).
+	SignText(account Account, text []byte) ([]byte, error)
 
 	// SignTx requests the wallet to sign the given transaction.
 	//
@@ -113,12 +127,12 @@ type Wallet interface {
 	// the account in a keystore).
 	SignTx(account Account, tx *types.Transaction, chainID *big.Int) (*types.Transaction, error)
 
-	// SignHashWithPassphrase requests the wallet to sign the given hash with the
+	// SignTextWithPassphrase requests the wallet to sign the given text with the
 	// given passphrase as extra authentication information.
 	//
 	// It looks up the account specified either solely via its address contained within,
 	// or optionally with the aid of any location metadata from the embedded URL field.
-	SignHashWithPassphrase(account Account, passphrase string, hash []byte) ([]byte, error)
+	SignTextWithPassphrase(account Account, passphrase string, hash []byte) ([]byte, error)
 
 	// SignTxWithPassphrase requests the wallet to sign the given transaction, with the
 	// given passphrase as extra authentication information.
@@ -148,6 +162,19 @@ type Backend interface {
 	Subscribe(sink chan<- WalletEvent) event.Subscription
 }
 
+// TextHash is a helper function that calculates a hash for the given message that can be
+// safely used to calculate a signature from.
+//
+// The hash is calulcated as
+//   keccak256("\x19Ethereum Signed Message:\n"${message length}${message}).
+//
+// This gives context to the signed message and prevents signing of transactions.
+func TextHash(data []byte) []byte {
+	hash := sha3.NewLegacyKeccak256()
+	fmt.Fprintf(hash, "\x19Ethereum Signed Message:\n%d%s", len(data), data)
+	return hash.Sum(nil)
+}
+
 // WalletEventType represents the different event types that can be fired by
 // the wallet subscription subsystem.
 type WalletEventType int

accounts/accounts_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 The go-ethereum Authors
+// Copyright 2019 The go-ethereum Authors
 // This file is part of the go-ethereum library.
 //
 // The go-ethereum library is free software: you can redistribute it and/or modify
@@ -14,30 +14,19 @@
 // You should have received a copy of the GNU Lesser General Public License
 // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
 
-package rpc
+package accounts
 
 import (
-	"strings"
+	"bytes"
 	"testing"
+
+	"github.com/ethereum/go-ethereum/common/hexutil"
 )
 
-func TestNewID(t *testing.T) {
+func TestTextHash(t *testing.T) {
-	hexchars := "0123456789ABCDEFabcdef"
+	hash := TextHash([]byte("Hello Joe"))
-	for i := 0; i < 100; i++ {
+	want := hexutil.MustDecode("0xa080337ae51c4e064c189e113edd0ba391df9206e2f49db658bb32cf2911730b")
-		id := string(NewID())
+	if !bytes.Equal(hash, want) {
-		if !strings.HasPrefix(id, "0x") {
+		t.Fatalf("wrong hash: %x", hash)
-			t.Fatalf("invalid ID prefix, want '0x...', got %s", id)
-		}
-
-		id = id[2:]
-		if len(id) == 0 || len(id) > 32 {
-			t.Fatalf("invalid ID length, want len(id) > 0 && len(id) <= 32), got %d", len(id))
-		}
-
-		for i := 0; i < len(id); i++ {
-			if strings.IndexByte(hexchars, id[i]) == -1 {
-				t.Fatalf("unexpected byte, want any valid hex char, got %c", id[i])
-			}
-		}
 	}
 }

accounts/external/backend.go

@@ -0,0 +1,220 @@
+// Copyright 2018 The go-ethereum Authors
+// This file is part of go-ethereum.
+//
+// go-ethereum is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// go-ethereum is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with go-ethereum. If not, see <http://www.gnu.org/licenses/>.
+
+package external
+
+import (
+	"fmt"
+	"math/big"
+	"sync"
+
+	"github.com/ethereum/go-ethereum"
+	"github.com/ethereum/go-ethereum/accounts"
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/common/hexutil"
+	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/crypto"
+	"github.com/ethereum/go-ethereum/event"
+	"github.com/ethereum/go-ethereum/internal/ethapi"
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/rpc"
+	"github.com/ethereum/go-ethereum/signer/core"
+)
+
+type ExternalBackend struct {
+	signers []accounts.Wallet
+}
+
+func (eb *ExternalBackend) Wallets() []accounts.Wallet {
+	return eb.signers
+}
+
+func NewExternalBackend(endpoint string) (*ExternalBackend, error) {
+	signer, err := NewExternalSigner(endpoint)
+	if err != nil {
+		return nil, err
+	}
+	return &ExternalBackend{
+		signers: []accounts.Wallet{signer},
+	}, nil
+}
+
+func (eb *ExternalBackend) Subscribe(sink chan<- accounts.WalletEvent) event.Subscription {
+	return event.NewSubscription(func(quit <-chan struct{}) error {
+		<-quit
+		return nil
+	})
+}
+
+// ExternalSigner provides an API to interact with an external signer (clef)
+// It proxies request to the external signer while forwarding relevant
+// request headers
+type ExternalSigner struct {
+	client   *rpc.Client
+	endpoint string
+	status   string
+	cacheMu  sync.RWMutex
+	cache    []accounts.Account
+}
+
+func NewExternalSigner(endpoint string) (*ExternalSigner, error) {
+	client, err := rpc.Dial(endpoint)
+	if err != nil {
+		return nil, err
+	}
+	extsigner := &ExternalSigner{
+		client:   client,
+		endpoint: endpoint,
+	}
+	// Check if reachable
+	version, err := extsigner.pingVersion()
+	if err != nil {
+		return nil, err
+	}
+	extsigner.status = fmt.Sprintf("ok [version=%v]", version)
+	return extsigner, nil
+}
+
+func (api *ExternalSigner) URL() accounts.URL {
+	return accounts.URL{
+		Scheme: "extapi",
+		Path:   api.endpoint,
+	}
+}
+
+func (api *ExternalSigner) Status() (string, error) {
+	return api.status, nil
+}
+
+func (api *ExternalSigner) Open(passphrase string) error {
+	return fmt.Errorf("operation not supported on external signers")
+}
+
+func (api *ExternalSigner) Close() error {
+	return fmt.Errorf("operation not supported on external signers")
+}
+
+func (api *ExternalSigner) Accounts() []accounts.Account {
+	var accnts []accounts.Account
+	res, err := api.listAccounts()
+	if err != nil {
+		log.Error("account listing failed", "error", err)
+		return accnts
+	}
+	for _, addr := range res {
+		accnts = append(accnts, accounts.Account{
+			URL: accounts.URL{
+				Scheme: "extapi",
+				Path:   api.endpoint,
+			},
+			Address: addr,
+		})
+	}
+	api.cacheMu.Lock()
+	api.cache = accnts
+	api.cacheMu.Unlock()
+	return accnts
+}
+
+func (api *ExternalSigner) Contains(account accounts.Account) bool {
+	api.cacheMu.RLock()
+	defer api.cacheMu.RUnlock()
+	for _, a := range api.cache {
+		if a.Address == account.Address && (account.URL == (accounts.URL{}) || account.URL == api.URL()) {
+			return true
+		}
+	}
+	return false
+}
+
+func (api *ExternalSigner) Derive(path accounts.DerivationPath, pin bool) (accounts.Account, error) {
+	return accounts.Account{}, fmt.Errorf("operation not supported on external signers")
+}
+
+func (api *ExternalSigner) SelfDerive(base accounts.DerivationPath, chain ethereum.ChainStateReader) {
+	log.Error("operation SelfDerive not supported on external signers")
+}
+
+func (api *ExternalSigner) signHash(account accounts.Account, hash []byte) ([]byte, error) {
+	return []byte{}, fmt.Errorf("operation not supported on external signers")
+}
+
+// SignData signs keccak256(data). The mimetype parameter describes the type of data being signed
+func (api *ExternalSigner) SignData(account accounts.Account, mimeType string, data []byte) ([]byte, error) {
+	// TODO! Replace this with a call to clef SignData with correct mime-type for Clique, once we
+	// have that in place
+	return api.signHash(account, crypto.Keccak256(data))
+}
+
+func (api *ExternalSigner) SignText(account accounts.Account, text []byte) ([]byte, error) {
+	return api.signHash(account, accounts.TextHash(text))
+}
+
+func (api *ExternalSigner) SignTx(account accounts.Account, tx *types.Transaction, chainID *big.Int) (*types.Transaction, error) {
+	res := ethapi.SignTransactionResult{}
+	to := common.NewMixedcaseAddress(*tx.To())
+	data := hexutil.Bytes(tx.Data())
+	args := &core.SendTxArgs{
+		Data:     &data,
+		Nonce:    hexutil.Uint64(tx.Nonce()),
+		Value:    hexutil.Big(*tx.Value()),
+		Gas:      hexutil.Uint64(tx.Gas()),
+		GasPrice: hexutil.Big(*tx.GasPrice()),
+		To:       &to,
+		From:     common.NewMixedcaseAddress(account.Address),
+	}
+
+	if err := api.client.Call(&res, "account_signTransaction", args); err != nil {
+		return nil, err
+	}
+	return res.Tx, nil
+}
+
+func (api *ExternalSigner) SignTextWithPassphrase(account accounts.Account, passphrase string, text []byte) ([]byte, error) {
+	return []byte{}, fmt.Errorf("operation not supported on external signers")
+}
+
+func (api *ExternalSigner) SignTxWithPassphrase(account accounts.Account, passphrase string, tx *types.Transaction, chainID *big.Int) (*types.Transaction, error) {
+	return nil, fmt.Errorf("operation not supported on external signers")
+}
+
+func (api *ExternalSigner) listAccounts() ([]common.Address, error) {
+	var res []common.Address
+	if err := api.client.Call(&res, "account_list"); err != nil {
+		return nil, err
+	}
+	return res, nil
+}
+
+func (api *ExternalSigner) signCliqueBlock(a common.Address, rlpBlock hexutil.Bytes) (hexutil.Bytes, error) {
+	var sig hexutil.Bytes
+	if err := api.client.Call(&sig, "account_signData", "application/clique", a, rlpBlock); err != nil {
+		return nil, err
+	}
+	if sig[64] != 27 && sig[64] != 28 {
+		return nil, fmt.Errorf("invalid Ethereum signature (V is not 27 or 28)")
+	}
+	sig[64] -= 27 // Transform V from 27/28 to 0/1 for Clique use
+	return sig, nil
+}
+
+func (api *ExternalSigner) pingVersion() (string, error) {
+	var v string
+	if err := api.client.Call(&v, "account_version"); err != nil {
+		return "", err
+	}
+	return v, nil
+}

accounts/keystore/wallet.go

@@ -22,6 +22,7 @@ import (
 	ethereum "github.com/ethereum/go-ethereum"
 	"github.com/ethereum/go-ethereum/accounts"
 	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/crypto"
 )
 
 // keystoreWallet implements the accounts.Wallet interface for the original
@@ -78,11 +79,11 @@ func (w *keystoreWallet) Derive(path accounts.DerivationPath, pin bool) (account
 // there is no notion of hierarchical account derivation for plain keystore accounts.
 func (w *keystoreWallet) SelfDerive(base accounts.DerivationPath, chain ethereum.ChainStateReader) {}
 
-// SignHash implements accounts.Wallet, attempting to sign the given hash with
+// signHash attempts to sign the given hash with
 // the given account. If the wallet does not wrap this particular account, an
 // error is returned to avoid account leakage (even though in theory we may be
 // able to sign via our shared keystore backend).
-func (w *keystoreWallet) SignHash(account accounts.Account, hash []byte) ([]byte, error) {
+func (w *keystoreWallet) signHash(account accounts.Account, hash []byte) ([]byte, error) {
 	// Make sure the requested account is contained within
 	if !w.Contains(account) {
 		return nil, accounts.ErrUnknownAccount
@@ -91,6 +92,15 @@ func (w *keystoreWallet) SignHash(account accounts.Account, hash []byte) ([]byte
 	return w.keystore.SignHash(account, hash)
 }
 
+// SignData signs keccak256(data). The mimetype parameter describes the type of data being signed
+func (w *keystoreWallet) SignData(account accounts.Account, mimeType string, data []byte) ([]byte, error) {
+	return w.signHash(account, crypto.Keccak256(data))
+}
+
+func (w *keystoreWallet) SignText(account accounts.Account, text []byte) ([]byte, error) {
+	return w.signHash(account, accounts.TextHash(text))
+}
+
 // SignTx implements accounts.Wallet, attempting to sign the given transaction
 // with the given account. If the wallet does not wrap this particular account,
 // an error is returned to avoid account leakage (even though in theory we may
@@ -106,13 +116,13 @@ func (w *keystoreWallet) SignTx(account accounts.Account, tx *types.Transaction,
 
 // SignHashWithPassphrase implements accounts.Wallet, attempting to sign the
 // given hash with the given account using passphrase as extra authentication.
-func (w *keystoreWallet) SignHashWithPassphrase(account accounts.Account, passphrase string, hash []byte) ([]byte, error) {
+func (w *keystoreWallet) SignTextWithPassphrase(account accounts.Account, passphrase string, text []byte) ([]byte, error) {
 	// Make sure the requested account is contained within
 	if !w.Contains(account) {
 		return nil, accounts.ErrUnknownAccount
 	}
 	// Account seems valid, request the keystore to sign
-	return w.keystore.SignHashWithPassphrase(account, passphrase, hash)
+	return w.keystore.SignHashWithPassphrase(account, passphrase, accounts.TextHash(text))
 }
 
 // SignTxWithPassphrase implements accounts.Wallet, attempting to sign the given

accounts/usbwallet/wallet.go

@@ -29,6 +29,7 @@ import (
 	"github.com/ethereum/go-ethereum/accounts"
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/core/types"
+	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/karalabe/hid"
 )
@@ -495,12 +496,21 @@ func (w *wallet) SelfDerive(base accounts.DerivationPath, chain ethereum.ChainSt
 	w.deriveChain = chain
 }
 
-// SignHash implements accounts.Wallet, however signing arbitrary data is not
+// signHash implements accounts.Wallet, however signing arbitrary data is not
 // supported for hardware wallets, so this method will always return an error.
-func (w *wallet) SignHash(account accounts.Account, hash []byte) ([]byte, error) {
+func (w *wallet) signHash(account accounts.Account, hash []byte) ([]byte, error) {
 	return nil, accounts.ErrNotSupported
 }
 
+// SignData signs keccak256(data). The mimetype parameter describes the type of data being signed
+func (w *wallet) SignData(account accounts.Account, mimeType string, data []byte) ([]byte, error) {
+	return w.signHash(account, crypto.Keccak256(data))
+}
+
+func (w *wallet) SignText(account accounts.Account, text []byte) ([]byte, error) {
+	return w.signHash(account, accounts.TextHash(text))
+}
+
 // SignTx implements accounts.Wallet. It sends the transaction over to the Ledger
 // wallet to request a confirmation from the user. It returns either the signed
 // transaction or a failure if the user denied the transaction.
@@ -550,8 +560,8 @@ func (w *wallet) SignTx(account accounts.Account, tx *types.Transaction, chainID
 // SignHashWithPassphrase implements accounts.Wallet, however signing arbitrary
 // data is not supported for Ledger wallets, so this method will always return
 // an error.
-func (w *wallet) SignHashWithPassphrase(account accounts.Account, passphrase string, hash []byte) ([]byte, error) {
+func (w *wallet) SignTextWithPassphrase(account accounts.Account, passphrase string, text []byte) ([]byte, error) {
-	return w.SignHash(account, hash)
+	return w.SignText(account, accounts.TextHash(text))
 }
 
 // SignTxWithPassphrase implements accounts.Wallet, attempting to sign the given

cmd/clef/main.go

@@ -52,12 +52,6 @@ import (
 	"gopkg.in/urfave/cli.v1"
 )
 
-// ExternalAPIVersion -- see extapi_changelog.md
-const ExternalAPIVersion = "5.0.0"
-
-// InternalAPIVersion -- see intapi_changelog.md
-const InternalAPIVersion = "3.1.0"
-
 const legalWarning = `
 WARNING! 
 
@@ -482,8 +476,8 @@ func signer(c *cli.Context) error {
 	}
 	ui.OnSignerStartup(core.StartupInfo{
 		Info: map[string]interface{}{
-			"extapi_version": ExternalAPIVersion,
+			"extapi_version": core.ExternalAPIVersion,
-			"intapi_version": InternalAPIVersion,
+			"intapi_version": core.InternalAPIVersion,
 			"extapi_http":    extapiURL,
 			"extapi_ipc":     ipcapiURL,
 		},

cmd/geth/main.go

@@ -31,8 +31,10 @@ import (
 	"github.com/ethereum/go-ethereum/accounts"
 	"github.com/ethereum/go-ethereum/accounts/keystore"
 	"github.com/ethereum/go-ethereum/cmd/utils"
+	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/console"
 	"github.com/ethereum/go-ethereum/eth"
+	"github.com/ethereum/go-ethereum/eth/downloader"
 	"github.com/ethereum/go-ethereum/ethclient"
 	"github.com/ethereum/go-ethereum/internal/debug"
 	"github.com/ethereum/go-ethereum/log"
@@ -60,6 +62,7 @@ var (
 		utils.BootnodesV5Flag,
 		utils.DataDirFlag,
 		utils.KeyStoreDirFlag,
+		utils.ExternalSignerFlag,
 		utils.NoUSBFlag,
 		utils.DashboardEnabledFlag,
 		utils.DashboardAddrFlag,
@@ -87,6 +90,7 @@ var (
 		utils.ULCTrustedNodesFlag,
 		utils.ULCMinTrustedFractionFlag,
 		utils.SyncModeFlag,
+		utils.ExitWhenSyncedFlag,
 		utils.GCModeFlag,
 		utils.LightServFlag,
 		utils.LightPeersFlag,
@@ -125,6 +129,7 @@ var (
 		utils.DeveloperPeriodFlag,
 		utils.TestnetFlag,
 		utils.RinkebyFlag,
+		utils.GoerliFlag,
 		utils.VMEnableDebugFlag,
 		utils.NetworkIdFlag,
 		utils.ConstantinopleOverrideFlag,
@@ -289,8 +294,8 @@ func startNode(ctx *cli.Context, stack *node.Node) {
 	utils.StartNode(stack)
 
 	// Unlock any account specifically requested
-	ks := stack.AccountManager().Backends(keystore.KeyStoreType)[0].(*keystore.KeyStore)
+	if keystores := stack.AccountManager().Backends(keystore.KeyStoreType); len(keystores) > 0 {
-
+		ks := keystores[0].(*keystore.KeyStore)
 		passwords := utils.MakePasswordList(ctx)
 		unlocks := strings.Split(ctx.GlobalString(utils.UnlockedAccountFlag.Name), ",")
 		for i, account := range unlocks {
@@ -298,6 +303,7 @@ func startNode(ctx *cli.Context, stack *node.Node) {
 				unlockAccount(ctx, ks, trimmed, i, passwords)
 			}
 		}
+	}
 	// Register wallet event handlers to open and auto-derive wallets
 	events := make(chan accounts.WalletEvent, 16)
 	stack.AccountManager().Subscribe(events)
@@ -339,6 +345,32 @@ func startNode(ctx *cli.Context, stack *node.Node) {
 			}
 		}
 	}()
+
+	// Spawn a standalone goroutine for status synchronization monitoring,
+	// close the node when synchronization is complete if user required.
+	if ctx.GlobalBool(utils.ExitWhenSyncedFlag.Name) {
+		go func() {
+			sub := stack.EventMux().Subscribe(downloader.DoneEvent{})
+			defer sub.Unsubscribe()
+			for {
+				event := <-sub.Chan()
+				if event == nil {
+					continue
+				}
+				done, ok := event.Data.(downloader.DoneEvent)
+				if !ok {
+					continue
+				}
+				if timestamp := time.Unix(done.Latest.Time.Int64(), 0); time.Since(timestamp) < 10*time.Minute {
+					log.Info("Synchronisation completed", "latestnum", done.Latest.Number, "latesthash", done.Latest.Hash(),
+						"age", common.PrettyAge(timestamp))
+					stack.Stop()
+				}
+
+			}
+		}()
+	}
+
 	// Start auxiliary services if enabled
 	if ctx.GlobalBool(utils.MiningEnabledFlag.Name) || ctx.GlobalBool(utils.DeveloperFlag.Name) {
 		// Mining only makes sense if a full Ethereum node is running

cmd/geth/usage.go

@@ -74,7 +74,9 @@ var AppHelpFlagGroups = []flagGroup{
 			utils.NetworkIdFlag,
 			utils.TestnetFlag,
 			utils.RinkebyFlag,
+			utils.GoerliFlag,
 			utils.SyncModeFlag,
+			utils.ExitWhenSyncedFlag,
 			utils.GCModeFlag,
 			utils.EthStatsURLFlag,
 			utils.IdentityFlag,
@@ -143,6 +145,7 @@ var AppHelpFlagGroups = []flagGroup{
 		Flags: []cli.Flag{
 			utils.UnlockedAccountFlag,
 			utils.PasswordFileFlag,
+			utils.ExternalSignerFlag,
 		},
 	},
 	{

cmd/swarm/swarm-smoke/feed_upload_and_sync.go

@@ -2,13 +2,11 @@ package main
 
 import (
 	"bytes"
-	"context"
 	"crypto/md5"
+	crand "crypto/rand"
 	"fmt"
 	"io"
 	"io/ioutil"
-	"net/http"
-	"net/http/httptrace"
 	"os"
 	"os/exec"
 	"strings"
@@ -18,13 +16,7 @@ import (
 	"github.com/ethereum/go-ethereum/common/hexutil"
 	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/log"
-	"github.com/ethereum/go-ethereum/metrics"
-	"github.com/ethereum/go-ethereum/swarm/api/client"
-	"github.com/ethereum/go-ethereum/swarm/spancontext"
 	"github.com/ethereum/go-ethereum/swarm/storage/feed"
-	"github.com/ethereum/go-ethereum/swarm/testutil"
-	colorable "github.com/mattn/go-colorable"
-	opentracing "github.com/opentracing/opentracing-go"
 	"github.com/pborman/uuid"
 	cli "gopkg.in/urfave/cli.v1"
 )
@@ -33,27 +25,6 @@ const (
 	feedRandomDataLength = 8
 )
 
-func cliFeedUploadAndSync(c *cli.Context) error {
-	metrics.GetOrRegisterCounter("feed-and-sync", nil).Inc(1)
-	log.Root().SetHandler(log.CallerFileHandler(log.LvlFilterHandler(log.Lvl(verbosity), log.StreamHandler(colorable.NewColorableStderr(), log.TerminalFormat(true)))))
-
-	errc := make(chan error)
-	go func() {
-		errc <- feedUploadAndSync(c)
-	}()
-
-	select {
-	case err := <-errc:
-		if err != nil {
-			metrics.GetOrRegisterCounter("feed-and-sync.fail", nil).Inc(1)
-		}
-		return err
-	case <-time.After(time.Duration(timeout) * time.Second):
-		metrics.GetOrRegisterCounter("feed-and-sync.timeout", nil).Inc(1)
-		return fmt.Errorf("timeout after %v sec", timeout)
-	}
-}
-
 // TODO: retrieve with manifest + extract repeating code
 func feedUploadAndSync(c *cli.Context) error {
 	defer func(now time.Time) { log.Info("total time", "time", time.Since(now), "size (kb)", filesize) }(time.Now())
@@ -232,9 +203,10 @@ func feedUploadAndSync(c *cli.Context) error {
 	seed := int(time.Now().UnixNano() / 1e6)
 	log.Info("feed uploading to "+endpoints[0]+" and syncing", "seed", seed)
 
-	randomBytes := testutil.RandomBytes(seed, filesize*1000)
+	h = md5.New()
+	r := io.TeeReader(io.LimitReader(crand.Reader, int64(filesize*1000)), h)
 
-	hash, err := upload(&randomBytes, endpoints[0])
+	hash, err := upload(r, filesize*1000, endpoints[0])
 	if err != nil {
 		return err
 	}
@@ -243,10 +215,7 @@ func feedUploadAndSync(c *cli.Context) error {
 		return err
 	}
 	multihashHex := hexutil.Encode(hashBytes)
-	fileHash, err := digest(bytes.NewReader(randomBytes))
+	fileHash := h.Sum(nil)
-	if err != nil {
-		return err
-	}
 
 	log.Info("uploaded successfully", "hash", hash, "digest", fmt.Sprintf("%x", fileHash))
 
@@ -307,60 +276,3 @@ func feedUploadAndSync(c *cli.Context) error {
 
 	return nil
 }
-
-func fetchFeed(topic string, user string, endpoint string, original []byte, ruid string) error {
-	ctx, sp := spancontext.StartSpan(context.Background(), "feed-and-sync.fetch")
-	defer sp.Finish()
-
-	log.Trace("sleeping", "ruid", ruid)
-	time.Sleep(3 * time.Second)
-
-	log.Trace("http get request (feed)", "ruid", ruid, "api", endpoint, "topic", topic, "user", user)
-
-	var tn time.Time
-	reqUri := endpoint + "/bzz-feed:/?topic=" + topic + "&user=" + user
-	req, _ := http.NewRequest("GET", reqUri, nil)
-
-	opentracing.GlobalTracer().Inject(
-		sp.Context(),
-		opentracing.HTTPHeaders,
-		opentracing.HTTPHeadersCarrier(req.Header))
-
-	trace := client.GetClientTrace("feed-and-sync - http get", "feed-and-sync", ruid, &tn)
-
-	req = req.WithContext(httptrace.WithClientTrace(ctx, trace))
-	transport := http.DefaultTransport
-
-	//transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
-
-	tn = time.Now()
-	res, err := transport.RoundTrip(req)
-	if err != nil {
-		log.Error(err.Error(), "ruid", ruid)
-		return err
-	}
-
-	log.Trace("http get response (feed)", "ruid", ruid, "api", endpoint, "topic", topic, "user", user, "code", res.StatusCode, "len", res.ContentLength)
-
-	if res.StatusCode != 200 {
-		return fmt.Errorf("expected status code %d, got %v (ruid %v)", 200, res.StatusCode, ruid)
-	}
-
-	defer res.Body.Close()
-
-	rdigest, err := digest(res.Body)
-	if err != nil {
-		log.Warn(err.Error(), "ruid", ruid)
-		return err
-	}
-
-	if !bytes.Equal(rdigest, original) {
-		err := fmt.Errorf("downloaded imported file md5=%x is not the same as the generated one=%x", rdigest, original)
-		log.Warn(err.Error(), "ruid", ruid)
-		return err
-	}
-
-	log.Trace("downloaded file matches random file", "ruid", ruid, "len", res.ContentLength)
-
-	return nil
-}

cmd/swarm/swarm-smoke/main.go

@@ -140,19 +140,25 @@ func main() {
 			Name:    "upload_and_sync",
 			Aliases: []string{"c"},
 			Usage:   "upload and sync",
-			Action:  cliUploadAndSync,
+			Action:  wrapCliCommand("upload-and-sync", true, uploadAndSync),
 		},
 		{
 			Name:    "feed_sync",
 			Aliases: []string{"f"},
 			Usage:   "feed update generate, upload and sync",
-			Action:  cliFeedUploadAndSync,
+			Action:  wrapCliCommand("feed-and-sync", true, feedUploadAndSync),
 		},
 		{
 			Name:    "upload_speed",
 			Aliases: []string{"u"},
 			Usage:   "measure upload speed",
-			Action:  cliUploadSpeed,
+			Action:  wrapCliCommand("upload-speed", true, uploadSpeed),
+		},
+		{
+			Name:    "sliding_window",
+			Aliases: []string{"s"},
+			Usage:   "measure network aggregate capacity",
+			Action:  wrapCliCommand("sliding-window", false, slidingWindow),
 		},
 	}
 

cmd/swarm/swarm-smoke/sliding_window.go

@@ -0,0 +1,115 @@
+// Copyright 2018 The go-ethereum Authors
+// This file is part of go-ethereum.
+//
+// go-ethereum is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// go-ethereum is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with go-ethereum. If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"crypto/md5"
+	crand "crypto/rand"
+	"fmt"
+	"io"
+	"math/rand"
+	"time"
+
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/metrics"
+	"github.com/pborman/uuid"
+
+	cli "gopkg.in/urfave/cli.v1"
+)
+
+var seed = time.Now().UTC().UnixNano()
+
+func init() {
+	rand.Seed(seed)
+}
+
+type uploadResult struct {
+	hash   string
+	digest []byte
+}
+
+func slidingWindow(c *cli.Context) error {
+	generateEndpoints(scheme, cluster, appName, from, to)
+	hashes := []uploadResult{} //swarm hashes of the uploads
+	nodes := to - from
+	const iterationTimeout = 30 * time.Second
+	log.Info("sliding window test started", "nodes", nodes, "filesize(kb)", filesize, "timeout", timeout)
+	uploadedBytes := 0
+	networkDepth := 0
+	errored := false
+
+outer:
+	for {
+		log.Info("uploading to "+endpoints[0]+" and syncing", "seed", seed)
+
+		h := md5.New()
+		r := io.TeeReader(io.LimitReader(crand.Reader, int64(filesize*1000)), h)
+		t1 := time.Now()
+
+		hash, err := upload(r, filesize*1000, endpoints[0])
+		if err != nil {
+			log.Error(err.Error())
+			return err
+		}
+
+		metrics.GetOrRegisterResettingTimer("sliding-window.upload-time", nil).UpdateSince(t1)
+
+		fhash := h.Sum(nil)
+
+		log.Info("uploaded successfully", "hash", hash, "digest", fmt.Sprintf("%x", fhash), "sleeping", syncDelay)
+		hashes = append(hashes, uploadResult{hash: hash, digest: fhash})
+		time.Sleep(time.Duration(syncDelay) * time.Second)
+		uploadedBytes += filesize * 1000
+
+		for i, v := range hashes {
+			timeout := time.After(time.Duration(timeout) * time.Second)
+			errored = false
+
+		inner:
+			for {
+				select {
+				case <-timeout:
+					errored = true
+					log.Error("error retrieving hash. timeout", "hash idx", i, "err", err)
+					metrics.GetOrRegisterCounter("sliding-window.single.error", nil).Inc(1)
+					break inner
+				default:
+					randIndex := 1 + rand.Intn(len(endpoints)-1)
+					ruid := uuid.New()[:8]
+					start := time.Now()
+					err := fetch(v.hash, endpoints[randIndex], v.digest, ruid)
+					if err != nil {
+						continue inner
+					}
+					metrics.GetOrRegisterResettingTimer("sliding-window.single.fetch-time", nil).UpdateSince(start)
+					break inner
+				}
+			}
+
+			if errored {
+				break outer
+			}
+			networkDepth = i
+			metrics.GetOrRegisterGauge("sliding-window.network-depth", nil).Update(int64(networkDepth))
+		}
+	}
+
+	log.Info("sliding window test finished", "errored?", errored, "networkDepth", networkDepth, "networkDepth(kb)", networkDepth*filesize)
+	log.Info("stats", "uploadedFiles", len(hashes), "uploadedKb", uploadedBytes/1000, "filesizeKb", filesize)
+
+	return nil
+}

cmd/swarm/swarm-smoke/upload_and_sync.go

@@ -17,102 +17,39 @@
 package main
 
 import (
-	"bytes"
-	"context"
 	"crypto/md5"
 	crand "crypto/rand"
-	"errors"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"math/rand"
-	"net/http"
-	"net/http/httptrace"
-	"os"
 	"sync"
 	"time"
 
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/metrics"
-	"github.com/ethereum/go-ethereum/swarm/api"
-	"github.com/ethereum/go-ethereum/swarm/api/client"
-	"github.com/ethereum/go-ethereum/swarm/spancontext"
-	"github.com/ethereum/go-ethereum/swarm/testutil"
-	opentracing "github.com/opentracing/opentracing-go"
 	"github.com/pborman/uuid"
 
 	cli "gopkg.in/urfave/cli.v1"
 )
 
-func generateEndpoints(scheme string, cluster string, app string, from int, to int) {
-	if cluster == "prod" {
-		for port := from; port < to; port++ {
-			endpoints = append(endpoints, fmt.Sprintf("%s://%v.swarm-gateways.net", scheme, port))
-		}
-	} else if cluster == "private-internal" {
-		for port := from; port < to; port++ {
-			endpoints = append(endpoints, fmt.Sprintf("%s://swarm-private-internal-%v:8500", scheme, port))
-		}
-	} else {
-		for port := from; port < to; port++ {
-			endpoints = append(endpoints, fmt.Sprintf("%s://%s-%v-%s.stg.swarm-gateways.net", scheme, app, port, cluster))
-		}
-	}
-
-	if includeLocalhost {
-		endpoints = append(endpoints, "http://localhost:8500")
-	}
-}
-
-func cliUploadAndSync(c *cli.Context) error {
-	log.PrintOrigins(true)
-	log.Root().SetHandler(log.LvlFilterHandler(log.Lvl(verbosity), log.StreamHandler(os.Stdout, log.TerminalFormat(true))))
-
-	metrics.GetOrRegisterCounter("upload-and-sync", nil).Inc(1)
-
-	errc := make(chan error)
-	go func() {
-		errc <- uploadAndSync(c)
-	}()
-
-	select {
-	case err := <-errc:
-		if err != nil {
-			metrics.GetOrRegisterCounter("upload-and-sync.fail", nil).Inc(1)
-		}
-		return err
-	case <-time.After(time.Duration(timeout) * time.Second):
-		metrics.GetOrRegisterCounter("upload-and-sync.timeout", nil).Inc(1)
-		return fmt.Errorf("timeout after %v sec", timeout)
-	}
-}
-
 func uploadAndSync(c *cli.Context) error {
-	defer func(now time.Time) {
-		totalTime := time.Since(now)
-		log.Info("total time", "time", totalTime, "kb", filesize)
-		metrics.GetOrRegisterResettingTimer("upload-and-sync.total-time", nil).Update(totalTime)
-	}(time.Now())
-
 	generateEndpoints(scheme, cluster, appName, from, to)
 	seed := int(time.Now().UnixNano() / 1e6)
+
 	log.Info("uploading to "+endpoints[0]+" and syncing", "seed", seed)
 
-	randomBytes := testutil.RandomBytes(seed, filesize*1000)
+	h := md5.New()
+	r := io.TeeReader(io.LimitReader(crand.Reader, int64(filesize*1000)), h)
 
 	t1 := time.Now()
-	hash, err := upload(&randomBytes, endpoints[0])
+	hash, err := upload(r, filesize*1000, endpoints[0])
 	if err != nil {
 		log.Error(err.Error())
 		return err
 	}
 	metrics.GetOrRegisterResettingTimer("upload-and-sync.upload-time", nil).UpdateSince(t1)
 
-	fhash, err := digest(bytes.NewReader(randomBytes))
+	fhash := h.Sum(nil)
-	if err != nil {
-		log.Error(err.Error())
-		return err
-	}
 
 	log.Info("uploaded successfully", "hash", hash, "digest", fmt.Sprintf("%x", fhash))
 
@@ -161,96 +98,3 @@ func uploadAndSync(c *cli.Context) error {
 
 	return nil
 }
-
-// fetch is getting the requested `hash` from the `endpoint` and compares it with the `original` file
-func fetch(hash string, endpoint string, original []byte, ruid string) error {
-	ctx, sp := spancontext.StartSpan(context.Background(), "upload-and-sync.fetch")
-	defer sp.Finish()
-
-	log.Trace("http get request", "ruid", ruid, "api", endpoint, "hash", hash)
-
-	var tn time.Time
-	reqUri := endpoint + "/bzz:/" + hash + "/"
-	req, _ := http.NewRequest("GET", reqUri, nil)
-
-	opentracing.GlobalTracer().Inject(
-		sp.Context(),
-		opentracing.HTTPHeaders,
-		opentracing.HTTPHeadersCarrier(req.Header))
-
-	trace := client.GetClientTrace("upload-and-sync - http get", "upload-and-sync", ruid, &tn)
-
-	req = req.WithContext(httptrace.WithClientTrace(ctx, trace))
-	transport := http.DefaultTransport
-
-	//transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
-
-	tn = time.Now()
-	res, err := transport.RoundTrip(req)
-	if err != nil {
-		log.Error(err.Error(), "ruid", ruid)
-		return err
-	}
-	log.Trace("http get response", "ruid", ruid, "api", endpoint, "hash", hash, "code", res.StatusCode, "len", res.ContentLength)
-
-	if res.StatusCode != 200 {
-		err := fmt.Errorf("expected status code %d, got %v", 200, res.StatusCode)
-		log.Warn(err.Error(), "ruid", ruid)
-		return err
-	}
-
-	defer res.Body.Close()
-
-	rdigest, err := digest(res.Body)
-	if err != nil {
-		log.Warn(err.Error(), "ruid", ruid)
-		return err
-	}
-
-	if !bytes.Equal(rdigest, original) {
-		err := fmt.Errorf("downloaded imported file md5=%x is not the same as the generated one=%x", rdigest, original)
-		log.Warn(err.Error(), "ruid", ruid)
-		return err
-	}
-
-	log.Trace("downloaded file matches random file", "ruid", ruid, "len", res.ContentLength)
-
-	return nil
-}
-
-// upload is uploading a file `f` to `endpoint` via the `swarm up` cmd
-func upload(dataBytes *[]byte, endpoint string) (string, error) {
-	swarm := client.NewClient(endpoint)
-	f := &client.File{
-		ReadCloser: ioutil.NopCloser(bytes.NewReader(*dataBytes)),
-		ManifestEntry: api.ManifestEntry{
-			ContentType: "text/plain",
-			Mode:        0660,
-			Size:        int64(len(*dataBytes)),
-		},
-	}
-
-	// upload data to bzz:// and retrieve the content-addressed manifest hash, hex-encoded.
-	return swarm.Upload(f, "", false)
-}
-
-func digest(r io.Reader) ([]byte, error) {
-	h := md5.New()
-	_, err := io.Copy(h, r)
-	if err != nil {
-		return nil, err
-	}
-	return h.Sum(nil), nil
-}
-
-// generates random data in heap buffer
-func generateRandomData(datasize int) ([]byte, error) {
-	b := make([]byte, datasize)
-	c, err := crand.Read(b)
-	if err != nil {
-		return nil, err
-	} else if c != datasize {
-		return nil, errors.New("short read")
-	}
-	return b, nil
-}

cmd/swarm/swarm-smoke/upload_speed.go

@@ -17,81 +17,35 @@
 package main
 
 import (
-	"bytes"
+	"crypto/md5"
+	crand "crypto/rand"
 	"fmt"
-	"os"
+	"io"
 	"time"
 
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/metrics"
-	"github.com/ethereum/go-ethereum/swarm/testutil"
 
 	cli "gopkg.in/urfave/cli.v1"
 )
 
-var endpoint string
-
-//just use the first endpoint
-func generateEndpoint(scheme string, cluster string, app string, from int) {
-	if cluster == "prod" {
-		endpoint = fmt.Sprintf("%s://%v.swarm-gateways.net", scheme, from)
-	} else if cluster == "private-internal" {
-		endpoint = fmt.Sprintf("%s://swarm-private-internal-%v:8500", scheme, from)
-	} else {
-		endpoint = fmt.Sprintf("%s://%s-%v-%s.stg.swarm-gateways.net", scheme, app, from, cluster)
-	}
-}
-
-func cliUploadSpeed(c *cli.Context) error {
-	log.PrintOrigins(true)
-	log.Root().SetHandler(log.LvlFilterHandler(log.Lvl(verbosity), log.StreamHandler(os.Stdout, log.TerminalFormat(true))))
-
-	metrics.GetOrRegisterCounter("upload-speed", nil).Inc(1)
-
-	errc := make(chan error)
-	go func() {
-		errc <- uploadSpeed(c)
-	}()
-
-	select {
-	case err := <-errc:
-		if err != nil {
-			metrics.GetOrRegisterCounter("upload-speed.fail", nil).Inc(1)
-		}
-		return err
-	case <-time.After(time.Duration(timeout) * time.Second):
-		metrics.GetOrRegisterCounter("upload-speed.timeout", nil).Inc(1)
-		return fmt.Errorf("timeout after %v sec", timeout)
-	}
-}
-
 func uploadSpeed(c *cli.Context) error {
-	defer func(now time.Time) {
+	endpoint := generateEndpoint(scheme, cluster, appName, from)
-		totalTime := time.Since(now)
-
-		log.Info("total time", "time", totalTime, "kb", filesize)
-		metrics.GetOrRegisterCounter("upload-speed.total-time", nil).Inc(int64(totalTime))
-	}(time.Now())
-
-	generateEndpoint(scheme, cluster, appName, from)
 	seed := int(time.Now().UnixNano() / 1e6)
 	log.Info("uploading to "+endpoint, "seed", seed)
 
-	randomBytes := testutil.RandomBytes(seed, filesize*1000)
+	h := md5.New()
+	r := io.TeeReader(io.LimitReader(crand.Reader, int64(filesize*1000)), h)
 
 	t1 := time.Now()
-	hash, err := upload(&randomBytes, endpoint)
+	hash, err := upload(r, filesize*1000, endpoint)
 	if err != nil {
 		log.Error(err.Error())
 		return err
 	}
 	metrics.GetOrRegisterCounter("upload-speed.upload-time", nil).Inc(int64(time.Since(t1)))
 
-	fhash, err := digest(bytes.NewReader(randomBytes))
+	fhash := h.Sum(nil)
-	if err != nil {
-		log.Error(err.Error())
-		return err
-	}
 
 	log.Info("uploaded successfully", "hash", hash, "digest", fmt.Sprintf("%x", fhash))
 	return nil

cmd/swarm/swarm-smoke/util.go

@@ -0,0 +1,267 @@
+// Copyright 2018 The go-ethereum Authors
+// This file is part of go-ethereum.
+//
+// go-ethereum is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// go-ethereum is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with go-ethereum. If not, see <http://www.gnu.org/licenses/>.
+
+package main
+
+import (
+	"bytes"
+	"context"
+	"crypto/md5"
+	crand "crypto/rand"
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"net/http/httptrace"
+	"os"
+	"time"
+
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/metrics"
+	"github.com/ethereum/go-ethereum/swarm/api"
+	"github.com/ethereum/go-ethereum/swarm/api/client"
+	"github.com/ethereum/go-ethereum/swarm/spancontext"
+	opentracing "github.com/opentracing/opentracing-go"
+	cli "gopkg.in/urfave/cli.v1"
+)
+
+var (
+	commandName = ""
+)
+
+func wrapCliCommand(name string, killOnTimeout bool, command func(*cli.Context) error) func(*cli.Context) error {
+	return func(ctx *cli.Context) error {
+		log.PrintOrigins(true)
+		log.Root().SetHandler(log.LvlFilterHandler(log.Lvl(verbosity), log.StreamHandler(os.Stdout, log.TerminalFormat(false))))
+
+		defer func(now time.Time) {
+			totalTime := time.Since(now)
+			log.Info("total time", "time", totalTime, "kb", filesize)
+			metrics.GetOrRegisterResettingTimer(name+".total-time", nil).Update(totalTime)
+		}(time.Now())
+
+		log.Info("smoke test starting", "task", name, "timeout", timeout)
+		commandName = name
+		metrics.GetOrRegisterCounter(name, nil).Inc(1)
+
+		errc := make(chan error)
+		done := make(chan struct{})
+
+		if killOnTimeout {
+			go func() {
+				<-time.After(time.Duration(timeout) * time.Second)
+				close(done)
+			}()
+		}
+
+		go func() {
+			errc <- command(ctx)
+		}()
+
+		select {
+		case err := <-errc:
+			if err != nil {
+				metrics.GetOrRegisterCounter(fmt.Sprintf("%s.fail", name), nil).Inc(1)
+			}
+			return err
+		case <-done:
+			metrics.GetOrRegisterCounter(fmt.Sprintf("%s.timeout", name), nil).Inc(1)
+			return fmt.Errorf("timeout after %v sec", timeout)
+		}
+	}
+}
+
+func generateEndpoints(scheme string, cluster string, app string, from int, to int) {
+	if cluster == "prod" {
+		for port := from; port < to; port++ {
+			endpoints = append(endpoints, fmt.Sprintf("%s://%v.swarm-gateways.net", scheme, port))
+		}
+	} else if cluster == "private-internal" {
+		for port := from; port < to; port++ {
+			endpoints = append(endpoints, fmt.Sprintf("%s://swarm-private-internal-%v:8500", scheme, port))
+		}
+	} else {
+		for port := from; port < to; port++ {
+			endpoints = append(endpoints, fmt.Sprintf("%s://%s-%v-%s.stg.swarm-gateways.net", scheme, app, port, cluster))
+		}
+	}
+
+	if includeLocalhost {
+		endpoints = append(endpoints, "http://localhost:8500")
+	}
+}
+
+//just use the first endpoint
+func generateEndpoint(scheme string, cluster string, app string, from int) string {
+	if cluster == "prod" {
+		return fmt.Sprintf("%s://%v.swarm-gateways.net", scheme, from)
+	} else if cluster == "private-internal" {
+		return fmt.Sprintf("%s://swarm-private-internal-%v:8500", scheme, from)
+	} else {
+		return fmt.Sprintf("%s://%s-%v-%s.stg.swarm-gateways.net", scheme, app, from, cluster)
+	}
+}
+
+func fetchFeed(topic string, user string, endpoint string, original []byte, ruid string) error {
+	ctx, sp := spancontext.StartSpan(context.Background(), "feed-and-sync.fetch")
+	defer sp.Finish()
+
+	log.Trace("sleeping", "ruid", ruid)
+	time.Sleep(3 * time.Second)
+
+	log.Trace("http get request (feed)", "ruid", ruid, "api", endpoint, "topic", topic, "user", user)
+
+	var tn time.Time
+	reqUri := endpoint + "/bzz-feed:/?topic=" + topic + "&user=" + user
+	req, _ := http.NewRequest("GET", reqUri, nil)
+
+	opentracing.GlobalTracer().Inject(
+		sp.Context(),
+		opentracing.HTTPHeaders,
+		opentracing.HTTPHeadersCarrier(req.Header))
+
+	trace := client.GetClientTrace("feed-and-sync - http get", "feed-and-sync", ruid, &tn)
+
+	req = req.WithContext(httptrace.WithClientTrace(ctx, trace))
+	transport := http.DefaultTransport
+
+	//transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+
+	tn = time.Now()
+	res, err := transport.RoundTrip(req)
+	if err != nil {
+		log.Error(err.Error(), "ruid", ruid)
+		return err
+	}
+
+	log.Trace("http get response (feed)", "ruid", ruid, "api", endpoint, "topic", topic, "user", user, "code", res.StatusCode, "len", res.ContentLength)
+
+	if res.StatusCode != 200 {
+		return fmt.Errorf("expected status code %d, got %v (ruid %v)", 200, res.StatusCode, ruid)
+	}
+
+	defer res.Body.Close()
+
+	rdigest, err := digest(res.Body)
+	if err != nil {
+		log.Warn(err.Error(), "ruid", ruid)
+		return err
+	}
+
+	if !bytes.Equal(rdigest, original) {
+		err := fmt.Errorf("downloaded imported file md5=%x is not the same as the generated one=%x", rdigest, original)
+		log.Warn(err.Error(), "ruid", ruid)
+		return err
+	}
+
+	log.Trace("downloaded file matches random file", "ruid", ruid, "len", res.ContentLength)
+
+	return nil
+}
+
+// fetch is getting the requested `hash` from the `endpoint` and compares it with the `original` file
+func fetch(hash string, endpoint string, original []byte, ruid string) error {
+	ctx, sp := spancontext.StartSpan(context.Background(), "upload-and-sync.fetch")
+	defer sp.Finish()
+
+	log.Trace("http get request", "ruid", ruid, "api", endpoint, "hash", hash)
+
+	var tn time.Time
+	reqUri := endpoint + "/bzz:/" + hash + "/"
+	req, _ := http.NewRequest("GET", reqUri, nil)
+
+	opentracing.GlobalTracer().Inject(
+		sp.Context(),
+		opentracing.HTTPHeaders,
+		opentracing.HTTPHeadersCarrier(req.Header))
+
+	trace := client.GetClientTrace(commandName+" - http get", commandName, ruid, &tn)
+
+	req = req.WithContext(httptrace.WithClientTrace(ctx, trace))
+	transport := http.DefaultTransport
+
+	//transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+
+	tn = time.Now()
+	res, err := transport.RoundTrip(req)
+	if err != nil {
+		log.Error(err.Error(), "ruid", ruid)
+		return err
+	}
+	log.Trace("http get response", "ruid", ruid, "api", endpoint, "hash", hash, "code", res.StatusCode, "len", res.ContentLength)
+
+	if res.StatusCode != 200 {
+		err := fmt.Errorf("expected status code %d, got %v", 200, res.StatusCode)
+		log.Warn(err.Error(), "ruid", ruid)
+		return err
+	}
+
+	defer res.Body.Close()
+
+	rdigest, err := digest(res.Body)
+	if err != nil {
+		log.Warn(err.Error(), "ruid", ruid)
+		return err
+	}
+
+	if !bytes.Equal(rdigest, original) {
+		err := fmt.Errorf("downloaded imported file md5=%x is not the same as the generated one=%x", rdigest, original)
+		log.Warn(err.Error(), "ruid", ruid)
+		return err
+	}
+
+	log.Trace("downloaded file matches random file", "ruid", ruid, "len", res.ContentLength)
+
+	return nil
+}
+
+// upload an arbitrary byte as a plaintext file  to `endpoint` using the api client
+func upload(r io.Reader, size int, endpoint string) (string, error) {
+	swarm := client.NewClient(endpoint)
+	f := &client.File{
+		ReadCloser: ioutil.NopCloser(r),
+		ManifestEntry: api.ManifestEntry{
+			ContentType: "text/plain",
+			Mode:        0660,
+			Size:        int64(size),
+		},
+	}
+
+	// upload data to bzz:// and retrieve the content-addressed manifest hash, hex-encoded.
+	return swarm.Upload(f, "", false)
+}
+
+func digest(r io.Reader) ([]byte, error) {
+	h := md5.New()
+	_, err := io.Copy(h, r)
+	if err != nil {
+		return nil, err
+	}
+	return h.Sum(nil), nil
+}
+
+// generates random data in heap buffer
+func generateRandomData(datasize int) ([]byte, error) {
+	b := make([]byte, datasize)
+	c, err := crand.Read(b)
+	if err != nil {
+		return nil, err
+	} else if c != datasize {
+		return nil, errors.New("short read")
+	}
+	return b, nil
+}

cmd/utils/flags.go

@@ -141,6 +141,10 @@ var (
 		Name:  "rinkeby",
 		Usage: "Rinkeby network: pre-configured proof-of-authority test network",
 	}
+	GoerliFlag = cli.BoolFlag{
+		Name:  "goerli",
+		Usage: "Görli network: pre-configured proof-of-authority test network",
+	}
 	ConstantinopleOverrideFlag = cli.Uint64Flag{
 		Name:  "override.constantinople",
 		Usage: "Manually specify constantinople fork-block, overriding the bundled setting",
@@ -162,6 +166,10 @@ var (
 		Usage: "Document Root for HTTPClient file scheme",
 		Value: DirectoryString{homeDir()},
 	}
+	ExitWhenSyncedFlag = cli.BoolFlag{
+		Name:  "exitwhensynced",
+		Usage: "Exists syncing after block synchronisation",
+	}
 	ULCModeConfigFlag = cli.StringFlag{
 		Name:  "ulc.config",
 		Usage: "Config file to use for ultra light client mode",
@@ -178,7 +186,6 @@ var (
 		Name:  "ulc.trusted",
 		Usage: "List of trusted ULC servers",
 	}
-
 	defaultSyncMode = eth.DefaultConfig.SyncMode
 	SyncModeFlag    = TextMarshalerFlag{
 		Name:  "syncmode",
@@ -420,7 +427,11 @@ var (
 		Usage: "Password file to use for non-interactive password input",
 		Value: "",
 	}
-
+	ExternalSignerFlag = cli.StringFlag{
+		Name:  "signer",
+		Usage: "External signer (url or path to ipc file)",
+		Value: "",
+	}
 	VMEnableDebugFlag = cli.BoolFlag{
 		Name:  "vmdebug",
 		Usage: "Record information useful for VM and contract debugging",
@@ -689,6 +700,9 @@ func MakeDataDir(ctx *cli.Context) string {
 		if ctx.GlobalBool(RinkebyFlag.Name) {
 			return filepath.Join(path, "rinkeby")
 		}
+		if ctx.GlobalBool(GoerliFlag.Name) {
+			return filepath.Join(path, "goerli")
+		}
 		return path
 	}
 	Fatalf("Cannot determine default data directory, please set manually (--datadir)")
@@ -743,6 +757,8 @@ func setBootstrapNodes(ctx *cli.Context, cfg *p2p.Config) {
 		urls = params.TestnetBootnodes
 	case ctx.GlobalBool(RinkebyFlag.Name):
 		urls = params.RinkebyBootnodes
+	case ctx.GlobalBool(GoerliFlag.Name):
+		urls = params.GoerliBootnodes
 	case cfg.BootstrapNodes != nil:
 		return // already set, don't apply defaults.
 	}
@@ -772,6 +788,8 @@ func setBootstrapNodesV5(ctx *cli.Context, cfg *p2p.Config) {
 		}
 	case ctx.GlobalBool(RinkebyFlag.Name):
 		urls = params.RinkebyBootnodes
+	case ctx.GlobalBool(GoerliFlag.Name):
+		urls = params.GoerliBootnodes
 	case cfg.BootstrapNodesV5 != nil:
 		return // already set, don't apply defaults.
 	}
@@ -976,11 +994,15 @@ func setEtherbase(ctx *cli.Context, ks *keystore.KeyStore, cfg *eth.Config) {
 	}
 	// Convert the etherbase into an address and configure it
 	if etherbase != "" {
+		if ks != nil {
 			account, err := MakeAddress(ks, etherbase)
 			if err != nil {
 				Fatalf("Invalid miner etherbase: %v", err)
 			}
 			cfg.Etherbase = account.Address
+		} else {
+			Fatalf("No etherbase configured")
+		}
 	}
 }
 
@@ -1077,9 +1099,12 @@ func SetNodeConfig(ctx *cli.Context, cfg *node.Config) {
 	setGraphQL(ctx, cfg)
 	setWS(ctx, cfg)
 	setNodeUserIdent(ctx, cfg)
-
 	setDataDir(ctx, cfg)
 
+	if ctx.GlobalIsSet(ExternalSignerFlag.Name) {
+		cfg.ExternalSigner = ctx.GlobalString(ExternalSignerFlag.Name)
+	}
+
 	if ctx.GlobalIsSet(KeyStoreDirFlag.Name) {
 		cfg.KeyStoreDir = ctx.GlobalString(KeyStoreDirFlag.Name)
 	}
@@ -1101,6 +1126,8 @@ func setDataDir(ctx *cli.Context, cfg *node.Config) {
 		cfg.DataDir = filepath.Join(node.DefaultDataDir(), "testnet")
 	case ctx.GlobalBool(RinkebyFlag.Name):
 		cfg.DataDir = filepath.Join(node.DefaultDataDir(), "rinkeby")
+	case ctx.GlobalBool(GoerliFlag.Name):
+		cfg.DataDir = filepath.Join(node.DefaultDataDir(), "goerli")
 	}
 }
 
@@ -1257,10 +1284,14 @@ func SetShhConfig(ctx *cli.Context, stack *node.Node, cfg *whisper.Config) {
 // SetEthConfig applies eth-related command line flags to the config.
 func SetEthConfig(ctx *cli.Context, stack *node.Node, cfg *eth.Config) {
 	// Avoid conflicting network flags
-	checkExclusive(ctx, DeveloperFlag, TestnetFlag, RinkebyFlag)
+	checkExclusive(ctx, DeveloperFlag, TestnetFlag, RinkebyFlag, GoerliFlag)
 	checkExclusive(ctx, LightServFlag, SyncModeFlag, "light")
-
+	// Can't use both ephemeral unlocked and external signer
-	ks := stack.AccountManager().Backends(keystore.KeyStoreType)[0].(*keystore.KeyStore)
+	checkExclusive(ctx, DeveloperFlag, ExternalSignerFlag)
+	var ks *keystore.KeyStore
+	if keystores := stack.AccountManager().Backends(keystore.KeyStoreType); len(keystores) > 0 {
+		ks = keystores[0].(*keystore.KeyStore)
+	}
 	setEtherbase(ctx, ks, cfg)
 	setGPO(ctx, &cfg.GPO)
 	setTxPool(ctx, &cfg.TxPool)
@@ -1356,6 +1387,11 @@ func SetEthConfig(ctx *cli.Context, stack *node.Node, cfg *eth.Config) {
 			cfg.NetworkId = 4
 		}
 		cfg.Genesis = core.DefaultRinkebyGenesisBlock()
+	case ctx.GlobalBool(GoerliFlag.Name):
+		if !ctx.GlobalIsSet(NetworkIdFlag.Name) {
+			cfg.NetworkId = 5
+		}
+		cfg.Genesis = core.DefaultGoerliGenesisBlock()
 	case ctx.GlobalBool(DeveloperFlag.Name):
 		if !ctx.GlobalIsSet(NetworkIdFlag.Name) {
 			cfg.NetworkId = 1337
@@ -1513,6 +1549,8 @@ func MakeGenesis(ctx *cli.Context) *core.Genesis {
 		genesis = core.DefaultTestnetGenesisBlock()
 	case ctx.GlobalBool(RinkebyFlag.Name):
 		genesis = core.DefaultRinkebyGenesisBlock()
+	case ctx.GlobalBool(GoerliFlag.Name):
+		genesis = core.DefaultGoerliGenesisBlock()
 	case ctx.GlobalBool(DeveloperFlag.Name):
 		Fatalf("Developer chains are ephemeral")
 	}

common/compiler/solidity_test.go

@@ -23,9 +23,10 @@ import (
 
 const (
 	testSource = `
+pragma solidity >0.0.0;
 contract test {
    /// @notice Will multiply ` + "`a`" + ` by 7.
-   function multiply(uint a) returns(uint d) {
+   function multiply(uint a) public returns(uint d) {
        return a * 7;
    }
 }

consensus/clique/clique.go

@@ -20,6 +20,7 @@ package clique
 import (
 	"bytes"
 	"errors"
+	"io"
 	"math/big"
 	"math/rand"
 	"sync"
@@ -136,40 +137,9 @@ var (
 	errRecentlySigned = errors.New("recently signed")
 )
 
-// SignerFn is a signer callback function to request a hash to be signed by a
+// SignerFn is a signer callback function to request a header to be signed by a
 // backing account.
-type SignerFn func(accounts.Account, []byte) ([]byte, error)
+type SignerFn func(accounts.Account, string, []byte) ([]byte, error)
-
-// sigHash returns the hash which is used as input for the proof-of-authority
-// signing. It is the hash of the entire header apart from the 65 byte signature
-// contained at the end of the extra data.
-//
-// Note, the method requires the extra data to be at least 65 bytes, otherwise it
-// panics. This is done to avoid accidentally using both forms (signature present
-// or not), which could be abused to produce different hashes for the same header.
-func sigHash(header *types.Header) (hash common.Hash) {
-	hasher := sha3.NewLegacyKeccak256()
-
-	rlp.Encode(hasher, []interface{}{
-		header.ParentHash,
-		header.UncleHash,
-		header.Coinbase,
-		header.Root,
-		header.TxHash,
-		header.ReceiptHash,
-		header.Bloom,
-		header.Difficulty,
-		header.Number,
-		header.GasLimit,
-		header.GasUsed,
-		header.Time,
-		header.Extra[:len(header.Extra)-65], // Yes, this will panic if extra is too short
-		header.MixDigest,
-		header.Nonce,
-	})
-	hasher.Sum(hash[:0])
-	return hash
-}
 
 // ecrecover extracts the Ethereum account address from a signed header.
 func ecrecover(header *types.Header, sigcache *lru.ARCCache) (common.Address, error) {
@@ -185,7 +155,7 @@ func ecrecover(header *types.Header, sigcache *lru.ARCCache) (common.Address, er
 	signature := header.Extra[len(header.Extra)-extraSeal:]
 
 	// Recover the public key and the Ethereum address
-	pubkey, err := crypto.Ecrecover(sigHash(header).Bytes(), signature)
+	pubkey, err := crypto.Ecrecover(SealHash(header).Bytes(), signature)
 	if err != nil {
 		return common.Address{}, err
 	}
@@ -646,7 +616,7 @@ func (c *Clique) Seal(chain consensus.ChainReader, block *types.Block, results c
 		log.Trace("Out-of-turn signing requested", "wiggle", common.PrettyDuration(wiggle))
 	}
 	// Sign all the things!
-	sighash, err := signFn(accounts.Account{Address: signer}, sigHash(header).Bytes())
+	sighash, err := signFn(accounts.Account{Address: signer}, "application/x-clique-header", CliqueRLP(header))
 	if err != nil {
 		return err
 	}
@@ -663,7 +633,7 @@ func (c *Clique) Seal(chain consensus.ChainReader, block *types.Block, results c
 		select {
 		case results <- block.WithSeal(header):
 		default:
-			log.Warn("Sealing result is not read by miner", "sealhash", c.SealHash(header))
+			log.Warn("Sealing result is not read by miner", "sealhash", SealHash(header))
 		}
 	}()
 
@@ -693,7 +663,7 @@ func CalcDifficulty(snap *Snapshot, signer common.Address) *big.Int {
 
 // SealHash returns the hash of a block prior to it being sealed.
 func (c *Clique) SealHash(header *types.Header) common.Hash {
-	return sigHash(header)
+	return SealHash(header)
 }
 
 // Close implements consensus.Engine. It's a noop for clique as there are no background threads.
@@ -711,3 +681,47 @@ func (c *Clique) APIs(chain consensus.ChainReader) []rpc.API {
 		Public:    false,
 	}}
 }
+
+// SealHash returns the hash of a block prior to it being sealed.
+func SealHash(header *types.Header) (hash common.Hash) {
+	hasher := sha3.NewLegacyKeccak256()
+	encodeSigHeader(hasher, header)
+	hasher.Sum(hash[:0])
+	return hash
+}
+
+// CliqueRLP returns the rlp bytes which needs to be signed for the proof-of-authority
+// sealing. The RLP to sign consists of the entire header apart from the 65 byte signature
+// contained at the end of the extra data.
+//
+// Note, the method requires the extra data to be at least 65 bytes, otherwise it
+// panics. This is done to avoid accidentally using both forms (signature present
+// or not), which could be abused to produce different hashes for the same header.
+func CliqueRLP(header *types.Header) []byte {
+	b := new(bytes.Buffer)
+	encodeSigHeader(b, header)
+	return b.Bytes()
+}
+
+func encodeSigHeader(w io.Writer, header *types.Header) {
+	err := rlp.Encode(w, []interface{}{
+		header.ParentHash,
+		header.UncleHash,
+		header.Coinbase,
+		header.Root,
+		header.TxHash,
+		header.ReceiptHash,
+		header.Bloom,
+		header.Difficulty,
+		header.Number,
+		header.GasLimit,
+		header.GasUsed,
+		header.Time,
+		header.Extra[:len(header.Extra)-65], // Yes, this will panic if extra is too short
+		header.MixDigest,
+		header.Nonce,
+	})
+	if err != nil {
+		panic("can't encode: " + err.Error())
+	}
+}

consensus/clique/snapshot_test.go

@@ -80,7 +80,7 @@ func (ap *testerAccountPool) sign(header *types.Header, signer string) {
 		ap.accounts[signer], _ = crypto.GenerateKey()
 	}
 	// Sign the header and embed the signature in extra data
-	sig, _ := crypto.Sign(sigHash(header).Bytes(), ap.accounts[signer])
+	sig, _ := crypto.Sign(SealHash(header).Bytes(), ap.accounts[signer])
 	copy(header.Extra[len(header.Extra)-extraSeal:], sig)
 }
 

core/blockchain.go

@@ -1143,7 +1143,7 @@ func (bc *BlockChain) insertChain(chain types.Blocks, verifySeals bool) (int, []
 	switch {
 	// First block is pruned, insert as sidechain and reorg only if TD grows enough
 	case err == consensus.ErrPrunedAncestor:
-		return bc.insertSidechain(it)
+		return bc.insertSidechain(block, it)
 
 	// First block is future, shove it (and all children) to the future queue (unknown ancestor)
 	case err == consensus.ErrFutureBlock || (err == consensus.ErrUnknownAncestor && bc.futureBlocks.Contains(it.first().ParentHash())):
@@ -1285,7 +1285,7 @@ func (bc *BlockChain) insertChain(chain types.Blocks, verifySeals bool) (int, []
 //
 // The method writes all (header-and-body-valid) blocks to disk, then tries to
 // switch over to the new chain if the TD exceeded the current chain.
-func (bc *BlockChain) insertSidechain(it *insertIterator) (int, []interface{}, []*types.Log, error) {
+func (bc *BlockChain) insertSidechain(block *types.Block, it *insertIterator) (int, []interface{}, []*types.Log, error) {
 	var (
 		externTd *big.Int
 		current  = bc.CurrentBlock()
@@ -1294,7 +1294,7 @@ func (bc *BlockChain) insertSidechain(it *insertIterator) (int, []interface{}, [
 	// Since we don't import them here, we expect ErrUnknownAncestor for the remaining
 	// ones. Any other errors means that the block is invalid, and should not be written
 	// to disk.
-	block, err := it.current(), consensus.ErrPrunedAncestor
+	err := consensus.ErrPrunedAncestor
 	for ; block != nil && (err == consensus.ErrPrunedAncestor); block, err = it.next() {
 		// Check the canonical state root for that number
 		if number := block.NumberU64(); current.NumberU64() >= number {

core/blockchain_insert.go

@@ -111,14 +111,6 @@ func (it *insertIterator) next() (*types.Block, error) {
 	return it.chain[it.index], it.validator.ValidateBody(it.chain[it.index])
 }
 
-// current returns the current block that's being processed.
-func (it *insertIterator) current() *types.Block {
-	if it.index < 0 || it.index+1 >= len(it.chain) {
-		return nil
-	}
-	return it.chain[it.index]
-}
-
 // previous returns the previous block was being processed, or nil
 func (it *insertIterator) previous() *types.Block {
 	if it.index < 1 {

core/genesis.go

@@ -157,7 +157,6 @@ func SetupGenesisBlockWithOverride(db ethdb.Database, genesis *Genesis, constant
 	if genesis != nil && genesis.Config == nil {
 		return params.AllEthashProtocolChanges, common.Hash{}, errGenesisNoConfig
 	}
-
 	// Just commit the new block if there is no stored genesis block.
 	stored := rawdb.ReadCanonicalHash(db, 0)
 	if (stored == common.Hash{}) {
@@ -340,6 +339,18 @@ func DefaultRinkebyGenesisBlock() *Genesis {
 	}
 }
 
+// DefaultGoerliGenesisBlock returns the Görli network genesis block.
+func DefaultGoerliGenesisBlock() *Genesis {
+	return &Genesis{
+		Config:     params.GoerliChainConfig,
+		Timestamp:  1548854791,
+		ExtraData:  hexutil.MustDecode("0x22466c6578692069732061207468696e6722202d204166726900000000000000e0a2bd4258d2768837baa26a28fe71dc079f84c70000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"),
+		GasLimit:   10485760,
+		Difficulty: big.NewInt(1),
+		Alloc:      decodePrealloc(goerliAllocData),
+	}
+}
+
 // DeveloperGenesisBlock returns the 'geth --dev' genesis block. Note, this must
 // be seeded with the
 func DeveloperGenesisBlock(period uint64, faucet common.Address) *Genesis {

core/vm/interpreter.go

@@ -182,6 +182,7 @@ func (in *EVMInterpreter) Run(contract *Contract, input []byte, readOnly bool) (
 		pcCopy  uint64 // needed for the deferred Tracer
 		gasCopy uint64 // for Tracer to log gas remaining before execution
 		logged  bool   // deferred Tracer should ignore already logged steps
+		res     []byte // result of the opcode execution function
 	)
 	contract.Input = input
 
@@ -216,11 +217,11 @@ func (in *EVMInterpreter) Run(contract *Contract, input []byte, readOnly bool) (
 		if !operation.valid {
 			return nil, fmt.Errorf("invalid opcode 0x%x", int(op))
 		}
-		if err := operation.validateStack(stack); err != nil {
+		if err = operation.validateStack(stack); err != nil {
 			return nil, err
 		}
 		// If the operation is valid, enforce and write restrictions
-		if err := in.enforceRestrictions(op, operation, stack); err != nil {
+		if err = in.enforceRestrictions(op, operation, stack); err != nil {
 			return nil, err
 		}
 
@@ -254,7 +255,7 @@ func (in *EVMInterpreter) Run(contract *Contract, input []byte, readOnly bool) (
 		}
 
 		// execute the operation
-		res, err := operation.execute(&pc, in, contract, mem, stack)
+		res, err = operation.execute(&pc, in, contract, mem, stack)
 		// verifyPool is a build flag. Pool verification makes sure the integrity
 		// of the integer pool by comparing values to a default value.
 		if verifyPool {

eth/api_tracer.go

@@ -214,7 +214,8 @@ func (api *PrivateDebugAPI) traceChain(ctx context.Context, start, end *types.Bl
 						log.Warn("Tracing failed", "hash", tx.Hash(), "block", task.block.NumberU64(), "err", err)
 						break
 					}
-					task.statedb.Finalise(true)
+					// Only delete empty objects if EIP158/161 (a.k.a Spurious Dragon) is in effect
+					task.statedb.Finalise(api.eth.blockchain.Config().IsEIP158(task.block.Number()))
 					task.results[i] = &txTraceResult{Result: res}
 				}
 				// Stream the result back to the user or abort on teardown
@@ -506,7 +507,8 @@ func (api *PrivateDebugAPI) traceBlock(ctx context.Context, block *types.Block,
 			break
 		}
 		// Finalize the state so any modifications are written to the trie
-		statedb.Finalise(true)
+		// Only delete empty objects if EIP158/161 (a.k.a Spurious Dragon) is in effect
+		statedb.Finalise(vmenv.ChainConfig().IsEIP158(block.Number()))
 	}
 	close(jobs)
 	pend.Wait()
@@ -612,7 +614,8 @@ func (api *PrivateDebugAPI) standardTraceBlockToFile(ctx context.Context, block
 			return dumps, err
 		}
 		// Finalize the state so any modifications are written to the trie
-		statedb.Finalise(true)
+		// Only delete empty objects if EIP158/161 (a.k.a Spurious Dragon) is in effect
+		statedb.Finalise(vmenv.ChainConfig().IsEIP158(block.Number()))
 
 		// If we've traced the transaction we were looking for, abort
 		if tx.Hash() == txHash {
@@ -803,7 +806,8 @@ func (api *PrivateDebugAPI) computeTxEnv(blockHash common.Hash, txIndex int, ree
 			return nil, vm.Context{}, nil, fmt.Errorf("transaction %#x failed: %v", tx.Hash(), err)
 		}
 		// Ensure any modifications are committed to the state
-		statedb.Finalise(true)
+		// Only delete empty objects if EIP158/161 (a.k.a Spurious Dragon) is in effect
+		statedb.Finalise(vmenv.ChainConfig().IsEIP158(block.Number()))
 	}
 	return nil, vm.Context{}, nil, fmt.Errorf("transaction index %d out of range for block %#x", txIndex, blockHash)
 }

eth/backend.go

@@ -434,7 +434,7 @@ func (s *Ethereum) StartMining(threads int) error {
 				log.Error("Etherbase account unavailable locally", "err", err)
 				return fmt.Errorf("signer missing: %v", err)
 			}
-			clique.Authorize(eb, wallet.SignHash)
+			clique.Authorize(eb, wallet.SignData)
 		}
 		// If mining is started, we can disable the transaction rejection mechanism
 		// introduced to speed sync times.

eth/downloader/downloader.go

@@ -415,7 +415,8 @@ func (d *Downloader) syncWithPeer(p *peerConnection, hash common.Hash, td *big.I
 		if err != nil {
 			d.mux.Post(FailedEvent{err})
 		} else {
-			d.mux.Post(DoneEvent{})
+			latest := d.lightchain.CurrentHeader()
+			d.mux.Post(DoneEvent{latest})
 		}
 	}()
 	if p.version < 62 {
@@ -1369,7 +1370,6 @@ func (d *Downloader) processHeaders(origin uint64, pivot uint64, td *big.Int) er
 			}
 			// Otherwise split the chunk of headers into batches and process them
 			gotHeaders = true
-
 			for len(headers) > 0 {
 				// Terminate if something failed in between processing chunks
 				select {
@@ -1432,7 +1432,6 @@ func (d *Downloader) processHeaders(origin uint64, pivot uint64, td *big.Int) er
 				headers = headers[limit:]
 				origin += uint64(limit)
 			}
-
 			// Update the highest block number we know if a higher one is found.
 			d.syncStatsLock.Lock()
 			if d.syncStatsChainHeight < origin {

eth/downloader/events.go

@@ -16,6 +16,10 @@
 
 package downloader
 
-type DoneEvent struct{}
+import "github.com/ethereum/go-ethereum/core/types"
+
+type DoneEvent struct {
+	Latest *types.Header
+}
 type StartEvent struct{}
 type FailedEvent struct{ Err error }

internal/ethapi/api.go

@@ -217,7 +217,7 @@ func NewPrivateAccountAPI(b Backend, nonceLock *AddrLocker) *PrivateAccountAPI {
 	}
 }
 
-// ListAccounts will return a list of addresses for accounts this node manages.
+// listAccounts will return a list of addresses for accounts this node manages.
 func (s *PrivateAccountAPI) ListAccounts() []common.Address {
 	addresses := make([]common.Address, 0) // return [] instead of nil if empty
 	for _, wallet := range s.am.Wallets() {
@@ -409,18 +409,6 @@ func (s *PrivateAccountAPI) SignTransaction(ctx context.Context, args SendTxArgs
 	return &SignTransactionResult{data, signed}, nil
 }
 
-// signHash is a helper function that calculates a hash for the given message that can be
-// safely used to calculate a signature from.
-//
-// The hash is calulcated as
-//   keccak256("\x19Ethereum Signed Message:\n"${message length}${message}).
-//
-// This gives context to the signed message and prevents signing of transactions.
-func signHash(data []byte) []byte {
-	msg := fmt.Sprintf("\x19Ethereum Signed Message:\n%d%s", len(data), data)
-	return crypto.Keccak256([]byte(msg))
-}
-
 // Sign calculates an Ethereum ECDSA signature for:
 // keccack256("\x19Ethereum Signed Message:\n" + len(message) + message))
 //
@@ -439,7 +427,7 @@ func (s *PrivateAccountAPI) Sign(ctx context.Context, data hexutil.Bytes, addr c
 		return nil, err
 	}
 	// Assemble sign the data with the wallet
-	signature, err := wallet.SignHashWithPassphrase(account, passwd, signHash(data))
+	signature, err := wallet.SignTextWithPassphrase(account, passwd, data)
 	if err != nil {
 		log.Warn("Failed data sign attempt", "address", addr, "err", err)
 		return nil, err
@@ -467,7 +455,7 @@ func (s *PrivateAccountAPI) EcRecover(ctx context.Context, data, sig hexutil.Byt
 	}
 	sig[64] -= 27 // Transform yellow paper V from 27/28 to 0/1
 
-	rpk, err := crypto.SigToPub(signHash(data), sig)
+	rpk, err := crypto.SigToPub(accounts.TextHash(data), sig)
 	if err != nil {
 		return common.Address{}, err
 	}
@@ -1357,7 +1345,7 @@ func (s *PublicTransactionPoolAPI) Sign(addr common.Address, data hexutil.Bytes)
 		return nil, err
 	}
 	// Sign the requested hash with the wallet
-	signature, err := wallet.SignHash(account, signHash(data))
+	signature, err := wallet.SignText(account, data)
 	if err == nil {
 		signature[64] += 27 // Transform V from 0/1 to 27/28 according to the yellow paper
 	}

node/config.go

@@ -27,6 +27,7 @@ import (
 	"sync"
 
 	"github.com/ethereum/go-ethereum/accounts"
+	"github.com/ethereum/go-ethereum/accounts/external"
 	"github.com/ethereum/go-ethereum/accounts/keystore"
 	"github.com/ethereum/go-ethereum/accounts/usbwallet"
 	"github.com/ethereum/go-ethereum/common"
@@ -80,6 +81,9 @@ type Config struct {
 	// is created by New and destroyed when the node is stopped.
 	KeyStoreDir string `toml:",omitempty"`
 
+	// ExternalSigner specifies an external URI for a clef-type signer
+	ExternalSigner string `toml:"omitempty"`
+
 	// UseLightweightKDF lowers the memory and CPU requirements of the key store
 	// scrypt KDF at the expense of security.
 	UseLightweightKDF bool `toml:",omitempty"`
@@ -462,9 +466,21 @@ func makeAccountManager(conf *Config) (*accounts.Manager, string, error) {
 		return nil, "", err
 	}
 	// Assemble the account manager and supported backends
-	backends := []accounts.Backend{
+	backends := []accounts.Backend{}
-		keystore.NewKeyStore(keydir, scryptN, scryptP),
+	if len(conf.ExternalSigner) > 0 {
+		log.Info("Using external signer", "url", conf.ExternalSigner)
+		if extapi, err := external.NewExternalBackend(conf.ExternalSigner); err == nil {
+			backends = append(backends, extapi)
+		} else {
+			log.Info("Error configuring external signer", "error", err)
 		}
+	}
+	if len(backends) == 0 {
+		// For now, we're using EITHER external signer OR local signers.
+		// If/when we implement some form of lockfile for USB and keystore wallets,
+		// we can have both, but it's very confusing for the user to see the same
+		// accounts in both externally and locally, plus very racey.
+		backends = append(backends, keystore.NewKeyStore(keydir, scryptN, scryptP))
 		if !conf.NoUSB {
 			// Start a USB hub for Ledger hardware wallets
 			if ledgerhub, err := usbwallet.NewLedgerHub(); err != nil {
@@ -479,6 +495,8 @@ func makeAccountManager(conf *Config) (*accounts.Manager, string, error) {
 				backends = append(backends, trezorhub)
 			}
 		}
+	}
+
 	return accounts.NewManager(backends...), ephemeral, nil
 }
 

p2p/discover/table.go

@@ -328,7 +328,7 @@ func (tab *Table) findnode(n *node, targetKey encPubkey, reply chan<- []*node) {
 	// Grab as many nodes as possible. Some of them might not be alive anymore, but we'll
 	// just remove those again during revalidation.
 	for _, n := range r {
-		tab.add(n)
+		tab.addSeenNode(n)
 	}
 	reply <- r
 }
@@ -443,7 +443,7 @@ func (tab *Table) loadSeedNodes() {
 		seed := seeds[i]
 		age := log.Lazy{Fn: func() interface{} { return time.Since(tab.db.LastPongReceived(seed.ID(), seed.IP())) }}
 		log.Trace("Found seed node in database", "id", seed.ID(), "addr", seed.addr(), "age", age)
-		tab.add(seed)
+		tab.addSeenNode(seed)
 	}
 }
 
@@ -468,7 +468,7 @@ func (tab *Table) doRevalidate(done chan<- struct{}) {
 		// The node responded, move it to the front.
 		last.livenessChecks++
 		log.Debug("Revalidated node", "b", bi, "id", last.ID(), "checks", last.livenessChecks)
-		b.bump(last)
+		tab.bumpInBucket(b, last)
 		return
 	}
 	// No reply received, pick a replacement or delete the node if there aren't
@@ -551,12 +551,12 @@ func (tab *Table) bucket(id enode.ID) *bucket {
 	return tab.buckets[d-bucketMinDistance-1]
 }
 
-// add attempts to add the given node to its corresponding bucket. If the bucket has space
+// addSeenNode adds a node which may or may not be live to the end of a bucket. If the
-// available, adding the node succeeds immediately. Otherwise, the node is added if the
+// bucket has space available, adding the node succeeds immediately. Otherwise, the node is
-// least recently active node in the bucket does not respond to a ping packet.
+// added to the replacements list.
 //
 // The caller must not hold tab.mutex.
-func (tab *Table) add(n *node) {
+func (tab *Table) addSeenNode(n *node) {
 	if n.ID() == tab.self().ID() {
 		return
 	}
@@ -564,23 +564,68 @@ func (tab *Table) add(n *node) {
 	tab.mutex.Lock()
 	defer tab.mutex.Unlock()
 	b := tab.bucket(n.ID())
-	if !tab.bumpOrAdd(b, n) {
+	if contains(b.entries, n.ID()) {
-		// Node is not in table. Add it to the replacement list.
+		// Already in bucket, don't add.
+		return
+	}
+	if len(b.entries) >= bucketSize {
+		// Bucket full, maybe add as replacement.
 		tab.addReplacement(b, n)
+		return
+	}
+	if !tab.addIP(b, n.IP()) {
+		// Can't add: IP limit reached.
+		return
+	}
+	// Add to end of bucket:
+	b.entries = append(b.entries, n)
+	b.replacements = deleteNode(b.replacements, n)
+	n.addedAt = time.Now()
+	if tab.nodeAddedHook != nil {
+		tab.nodeAddedHook(n)
 	}
 }
 
-// addThroughPing adds the given node to the table. Compared to plain
+// addVerifiedNode adds a node whose existence has been verified recently to the front of a
-// 'add' there is an additional safety measure: if the table is still
+// bucket. If the node is already in the bucket, it is moved to the front. If the bucket
-// initializing the node is not added. This prevents an attack where the
+// has no space, the node is added to the replacements list.
-// table could be filled by just sending ping repeatedly.
+//
+// There is an additional safety measure: if the table is still initializing the node
+// is not added. This prevents an attack where the table could be filled by just sending
+// ping repeatedly.
 //
 // The caller must not hold tab.mutex.
-func (tab *Table) addThroughPing(n *node) {
+func (tab *Table) addVerifiedNode(n *node) {
 	if !tab.isInitDone() {
 		return
 	}
-	tab.add(n)
+	if n.ID() == tab.self().ID() {
+		return
+	}
+
+	tab.mutex.Lock()
+	defer tab.mutex.Unlock()
+	b := tab.bucket(n.ID())
+	if tab.bumpInBucket(b, n) {
+		// Already in bucket, moved to front.
+		return
+	}
+	if len(b.entries) >= bucketSize {
+		// Bucket full, maybe add as replacement.
+		tab.addReplacement(b, n)
+		return
+	}
+	if !tab.addIP(b, n.IP()) {
+		// Can't add: IP limit reached.
+		return
+	}
+	// Add to front of bucket.
+	b.entries, _ = pushNode(b.entries, n, bucketSize)
+	b.replacements = deleteNode(b.replacements, n)
+	n.addedAt = time.Now()
+	if tab.nodeAddedHook != nil {
+		tab.nodeAddedHook(n)
+	}
 }
 
 // delete removes an entry from the node table. It is used to evacuate dead nodes.
@@ -651,12 +696,21 @@ func (tab *Table) replace(b *bucket, last *node) *node {
 	return r
 }
 
-// bump moves the given node to the front of the bucket entry list
+// bumpInBucket moves the given node to the front of the bucket entry list
 // if it is contained in that list.
-func (b *bucket) bump(n *node) bool {
+func (tab *Table) bumpInBucket(b *bucket, n *node) bool {
 	for i := range b.entries {
 		if b.entries[i].ID() == n.ID() {
-			// move it to the front
+			if !n.IP().Equal(b.entries[i].IP()) {
+				// Endpoint has changed, ensure that the new IP fits into table limits.
+				tab.removeIP(b, b.entries[i].IP())
+				if !tab.addIP(b, n.IP()) {
+					// It doesn't, put the previous one back.
+					tab.addIP(b, b.entries[i].IP())
+					return false
+				}
+			}
+			// Move it to the front.
 			copy(b.entries[1:], b.entries[:i])
 			b.entries[0] = n
 			return true
@@ -665,29 +719,20 @@ func (b *bucket) bump(n *node) bool {
 	return false
 }
 
-// bumpOrAdd moves n to the front of the bucket entry list or adds it if the list isn't
-// full. The return value is true if n is in the bucket.
-func (tab *Table) bumpOrAdd(b *bucket, n *node) bool {
-	if b.bump(n) {
-		return true
-	}
-	if len(b.entries) >= bucketSize || !tab.addIP(b, n.IP()) {
-		return false
-	}
-	b.entries, _ = pushNode(b.entries, n, bucketSize)
-	b.replacements = deleteNode(b.replacements, n)
-	n.addedAt = time.Now()
-	if tab.nodeAddedHook != nil {
-		tab.nodeAddedHook(n)
-	}
-	return true
-}
-
 func (tab *Table) deleteInBucket(b *bucket, n *node) {
 	b.entries = deleteNode(b.entries, n)
 	tab.removeIP(b, n.IP())
 }
 
+func contains(ns []*node, id enode.ID) bool {
+	for _, n := range ns {
+		if n.ID() == id {
+			return true
+		}
+	}
+	return false
+}
+
 // pushNode adds n to the front of list, keeping at most max items.
 func pushNode(list []*node, n *node, max int) ([]*node, *node) {
 	if len(list) < max {

p2p/discover/table_test.go

@@ -30,6 +30,7 @@ import (
 	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/p2p/enode"
 	"github.com/ethereum/go-ethereum/p2p/enr"
+	"github.com/ethereum/go-ethereum/p2p/netutil"
 )
 
 func TestTable_pingReplace(t *testing.T) {
@@ -64,7 +65,7 @@ func testPingReplace(t *testing.T, newNodeIsResponding, lastInBucketIsResponding
 	// its bucket if it is unresponsive. Revalidate again to ensure that
 	transport.dead[last.ID()] = !lastInBucketIsResponding
 	transport.dead[pingSender.ID()] = !newNodeIsResponding
-	tab.add(pingSender)
+	tab.addSeenNode(pingSender)
 	tab.doRevalidate(make(chan struct{}, 1))
 	tab.doRevalidate(make(chan struct{}, 1))
 
@@ -114,10 +115,14 @@ func TestBucket_bumpNoDuplicates(t *testing.T) {
 	}
 
 	prop := func(nodes []*node, bumps []int) (ok bool) {
+		tab, db := newTestTable(newPingRecorder())
+		defer db.Close()
+		defer tab.Close()
+
 		b := &bucket{entries: make([]*node, len(nodes))}
 		copy(b.entries, nodes)
 		for i, pos := range bumps {
-			b.bump(b.entries[pos])
+			tab.bumpInBucket(b, b.entries[pos])
 			if hasDuplicates(b.entries) {
 				t.Logf("bucket has duplicates after %d/%d bumps:", i+1, len(bumps))
 				for _, n := range b.entries {
@@ -126,6 +131,7 @@ func TestBucket_bumpNoDuplicates(t *testing.T) {
 				return false
 			}
 		}
+		checkIPLimitInvariant(t, tab)
 		return true
 	}
 	if err := quick.Check(prop, cfg); err != nil {
@@ -142,11 +148,12 @@ func TestTable_IPLimit(t *testing.T) {
 
 	for i := 0; i < tableIPLimit+1; i++ {
 		n := nodeAtDistance(tab.self().ID(), i, net.IP{172, 0, 1, byte(i)})
-		tab.add(n)
+		tab.addSeenNode(n)
 	}
 	if tab.len() > tableIPLimit {
 		t.Errorf("too many nodes in table")
 	}
+	checkIPLimitInvariant(t, tab)
 }
 
 // This checks that the per-bucket IP limit is applied correctly.
@@ -159,11 +166,28 @@ func TestTable_BucketIPLimit(t *testing.T) {
 	d := 3
 	for i := 0; i < bucketIPLimit+1; i++ {
 		n := nodeAtDistance(tab.self().ID(), d, net.IP{172, 0, 1, byte(i)})
-		tab.add(n)
+		tab.addSeenNode(n)
 	}
 	if tab.len() > bucketIPLimit {
 		t.Errorf("too many nodes in table")
 	}
+	checkIPLimitInvariant(t, tab)
+}
+
+// checkIPLimitInvariant checks that ip limit sets contain an entry for every
+// node in the table and no extra entries.
+func checkIPLimitInvariant(t *testing.T, tab *Table) {
+	t.Helper()
+
+	tabset := netutil.DistinctNetSet{Subnet: tableSubnet, Limit: tableIPLimit}
+	for _, b := range tab.buckets {
+		for _, n := range b.entries {
+			tabset.Add(n.IP())
+		}
+	}
+	if tabset.String() != tab.ips.String() {
+		t.Errorf("table IP set is incorrect:\nhave: %v\nwant: %v", tab.ips, tabset)
+	}
 }
 
 func TestTable_closest(t *testing.T) {
@@ -281,6 +305,69 @@ func (*closeTest) Generate(rand *rand.Rand, size int) reflect.Value {
 	return reflect.ValueOf(t)
 }
 
+func TestTable_addVerifiedNode(t *testing.T) {
+	tab, db := newTestTable(newPingRecorder())
+	<-tab.initDone
+	defer db.Close()
+	defer tab.Close()
+
+	// Insert two nodes.
+	n1 := nodeAtDistance(tab.self().ID(), 256, net.IP{88, 77, 66, 1})
+	n2 := nodeAtDistance(tab.self().ID(), 256, net.IP{88, 77, 66, 2})
+	tab.addSeenNode(n1)
+	tab.addSeenNode(n2)
+
+	// Verify bucket content:
+	bcontent := []*node{n1, n2}
+	if !reflect.DeepEqual(tab.bucket(n1.ID()).entries, bcontent) {
+		t.Fatalf("wrong bucket content: %v", tab.bucket(n1.ID()).entries)
+	}
+
+	// Add a changed version of n2.
+	newrec := n2.Record()
+	newrec.Set(enr.IP{99, 99, 99, 99})
+	newn2 := wrapNode(enode.SignNull(newrec, n2.ID()))
+	tab.addVerifiedNode(newn2)
+
+	// Check that bucket is updated correctly.
+	newBcontent := []*node{newn2, n1}
+	if !reflect.DeepEqual(tab.bucket(n1.ID()).entries, newBcontent) {
+		t.Fatalf("wrong bucket content after update: %v", tab.bucket(n1.ID()).entries)
+	}
+	checkIPLimitInvariant(t, tab)
+}
+
+func TestTable_addSeenNode(t *testing.T) {
+	tab, db := newTestTable(newPingRecorder())
+	<-tab.initDone
+	defer db.Close()
+	defer tab.Close()
+
+	// Insert two nodes.
+	n1 := nodeAtDistance(tab.self().ID(), 256, net.IP{88, 77, 66, 1})
+	n2 := nodeAtDistance(tab.self().ID(), 256, net.IP{88, 77, 66, 2})
+	tab.addSeenNode(n1)
+	tab.addSeenNode(n2)
+
+	// Verify bucket content:
+	bcontent := []*node{n1, n2}
+	if !reflect.DeepEqual(tab.bucket(n1.ID()).entries, bcontent) {
+		t.Fatalf("wrong bucket content: %v", tab.bucket(n1.ID()).entries)
+	}
+
+	// Add a changed version of n2.
+	newrec := n2.Record()
+	newrec.Set(enr.IP{99, 99, 99, 99})
+	newn2 := wrapNode(enode.SignNull(newrec, n2.ID()))
+	tab.addSeenNode(newn2)
+
+	// Check that bucket content is unchanged.
+	if !reflect.DeepEqual(tab.bucket(n1.ID()).entries, bcontent) {
+		t.Fatalf("wrong bucket content after update: %v", tab.bucket(n1.ID()).entries)
+	}
+	checkIPLimitInvariant(t, tab)
+}
+
 func TestTable_Lookup(t *testing.T) {
 	tab, db := newTestTable(lookupTestnet)
 	defer db.Close()
@@ -535,7 +622,6 @@ func (tn *preminedTestnet) findnode(toid enode.ID, toaddr *net.UDPAddr, target e
 }
 
 func (*preminedTestnet) close()                                        {}
-func (*preminedTestnet) waitping(from enode.ID) error                  { return nil }
 func (*preminedTestnet) ping(toid enode.ID, toaddr *net.UDPAddr) error { return nil }
 
 // mine generates a testnet struct literal with nodes at

p2p/discover/table_util_test.go

@@ -86,17 +86,8 @@ func fillBucket(tab *Table, n *node) (last *node) {
 // fillTable adds nodes the table to the end of their corresponding bucket
 // if the bucket is not full. The caller must not hold tab.mutex.
 func fillTable(tab *Table, nodes []*node) {
-	tab.mutex.Lock()
-	defer tab.mutex.Unlock()
-
 	for _, n := range nodes {
-		if n.ID() == tab.self().ID() {
+		tab.addSeenNode(n)
-			continue // don't add self
-		}
-		b := tab.bucket(n.ID())
-		if len(b.entries) < bucketSize {
-			tab.bumpOrAdd(b, n)
-		}
 	}
 }
 
@@ -154,15 +145,6 @@ func hasDuplicates(slice []*node) bool {
 	return false
 }
 
-func contains(ns []*node, id enode.ID) bool {
-	for _, n := range ns {
-		if n.ID() == id {
-			return true
-		}
-	}
-	return false
-}
-
 func sortedByDistanceTo(distbase enode.ID, slice []*node) bool {
 	var last enode.ID
 	for i, e := range slice {

p2p/discover/udp.go

@@ -661,10 +661,10 @@ func (req *ping) handle(t *udp, from *net.UDPAddr, fromID enode.ID, mac []byte)
 	n := wrapNode(enode.NewV4(req.senderKey, from.IP, int(req.From.TCP), from.Port))
 	if time.Since(t.db.LastPongReceived(n.ID(), from.IP)) > bondExpiration {
 		t.sendPing(fromID, from, func() {
-			t.tab.addThroughPing(n)
+			t.tab.addVerifiedNode(n)
 		})
 	} else {
-		t.tab.addThroughPing(n)
+		t.tab.addVerifiedNode(n)
 	}
 
 	// Update node database and endpoint predictor.

params/bootnodes.go

@@ -47,6 +47,20 @@ var RinkebyBootnodes = []string{
 	"enode://b6b28890b006743680c52e64e0d16db57f28124885595fa03a562be1d2bf0f3a1da297d56b13da25fb992888fd556d4c1a27b1f39d531bde7de1921c90061cc6@159.89.28.211:30303", // AKASHA
 }
 
+// GoerliBootnodes are the enode URLs of the P2P bootstrap nodes running on the
+// Görli test network.
+var GoerliBootnodes = []string{
+	// Upstrem bootnodes
+	"enode://011f758e6552d105183b1761c5e2dea0111bc20fd5f6422bc7f91e0fabbec9a6595caf6239b37feb773dddd3f87240d99d859431891e4a642cf2a0a9e6cbb98a@51.141.78.53:30303",
+	"enode://176b9417f511d05b6b2cf3e34b756cf0a7096b3094572a8f6ef4cdcb9d1f9d00683bf0f83347eebdf3b81c3521c2332086d9592802230bf528eaf606a1d9677b@13.93.54.137:30303",
+	"enode://46add44b9f13965f7b9875ac6b85f016f341012d84f975377573800a863526f4da19ae2c620ec73d11591fa9510e992ecc03ad0751f53cc02f7c7ed6d55c7291@94.237.54.114:30313",
+	"enode://c1f8b7c2ac4453271fa07d8e9ecf9a2e8285aa0bd0c07df0131f47153306b0736fd3db8924e7a9bf0bed6b1d8d4f87362a71b033dc7c64547728d953e43e59b2@52.64.155.147:30303",
+	"enode://f4a9c6ee28586009fb5a96c8af13a58ed6d8315a9eee4772212c1d4d9cebe5a8b8a78ea4434f318726317d04a3f531a1ef0420cf9752605a562cfe858c46e263@213.186.16.82:30303",
+
+	// Ethereum Foundation bootnode
+	"enode://573b6607cd59f241e30e4c4943fd50e99e2b6f42f9bd5ca111659d309c06741247f4f1e93843ad3e8c8c18b6e2d94c161b7ef67479b3938780a97134b618b5ce@52.56.136.200:30303",
+}
+
 // DiscoveryV5Bootnodes are the enode URLs of the P2P bootstrap nodes for the
 // experimental RLPx v5 topic-discovery network.
 var DiscoveryV5Bootnodes = []string{

params/config.go

@@ -109,6 +109,33 @@ var (
 		BloomRoot:    common.HexToHash("0xcf74ca2c14e843b366561dab4fc64237bf6bb335119cbc97d723f3b501863470"),
 	}
 
+	// GoerliChainConfig contains the chain parameters to run a node on the Görli test network.
+	GoerliChainConfig = &ChainConfig{
+		ChainID:             big.NewInt(5),
+		HomesteadBlock:      big.NewInt(0),
+		DAOForkBlock:        nil,
+		DAOForkSupport:      true,
+		EIP150Block:         big.NewInt(0),
+		EIP155Block:         big.NewInt(0),
+		EIP158Block:         big.NewInt(0),
+		ByzantiumBlock:      big.NewInt(0),
+		ConstantinopleBlock: big.NewInt(0),
+		PetersburgBlock:     big.NewInt(0),
+		Clique: &CliqueConfig{
+			Period: 15,
+			Epoch:  30000,
+		},
+	}
+
+	// GoerliTrustedCheckpoint contains the light client trusted checkpoint for the Görli test network.
+	GoerliTrustedCheckpoint = &TrustedCheckpoint{
+		Name:         "goerli",
+		SectionIndex: 0,
+		SectionHead:  common.HexToHash("0x0000000000000000000000000000000000000000000000000000000000000000"),
+		CHTRoot:      common.HexToHash("0x0000000000000000000000000000000000000000000000000000000000000000"),
+		BloomRoot:    common.HexToHash("0x0000000000000000000000000000000000000000000000000000000000000000"),
+	}
+
 	// AllEthashProtocolChanges contains every protocol change (EIPs) introduced
 	// and accepted by the Ethereum core developers into the Ethash consensus.
 	//

rpc/client.go

@@ -18,17 +18,13 @@ package rpc
 
 import (
 	"bytes"
-	"container/list"
 	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
-	"net"
 	"net/url"
 	"reflect"
 	"strconv"
-	"strings"
-	"sync"
 	"sync/atomic"
 	"time"
 
@@ -39,13 +35,14 @@ var (
 	ErrClientQuit                = errors.New("client is closed")
 	ErrNoResult                  = errors.New("no result in JSON-RPC response")
 	ErrSubscriptionQueueOverflow = errors.New("subscription queue overflow")
+	errClientReconnected         = errors.New("client reconnected")
+	errDead                      = errors.New("connection lost")
 )
 
 const (
 	// Timeouts
 	tcpKeepAliveInterval = 30 * time.Second
-	defaultDialTimeout   = 10 * time.Second // used when dialing if the context has no deadline
+	defaultDialTimeout   = 10 * time.Second // used if context has no deadline
-	defaultWriteTimeout  = 10 * time.Second // used for calls if the context has no deadline
 	subscribeTimeout     = 5 * time.Second  // overall timeout eth_subscribe, rpc_modules calls
 )
 
@@ -76,56 +73,57 @@ type BatchElem struct {
 	Error error
 }
 
-// A value of this type can a JSON-RPC request, notification, successful response or
-// error response. Which one it is depends on the fields.
-type jsonrpcMessage struct {
-	Version string          `json:"jsonrpc"`
-	ID      json.RawMessage `json:"id,omitempty"`
-	Method  string          `json:"method,omitempty"`
-	Params  json.RawMessage `json:"params,omitempty"`
-	Error   *jsonError      `json:"error,omitempty"`
-	Result  json.RawMessage `json:"result,omitempty"`
-}
-
-func (msg *jsonrpcMessage) isNotification() bool {
-	return msg.ID == nil && msg.Method != ""
-}
-
-func (msg *jsonrpcMessage) isResponse() bool {
-	return msg.hasValidID() && msg.Method == "" && len(msg.Params) == 0
-}
-
-func (msg *jsonrpcMessage) hasValidID() bool {
-	return len(msg.ID) > 0 && msg.ID[0] != '{' && msg.ID[0] != '['
-}
-
-func (msg *jsonrpcMessage) String() string {
-	b, _ := json.Marshal(msg)
-	return string(b)
-}
-
 // Client represents a connection to an RPC server.
 type Client struct {
-	idCounter   uint32
+	idgen    func() ID // for subscriptions
-	connectFunc func(ctx context.Context) (net.Conn, error)
 	isHTTP   bool
+	services *serviceRegistry
 
-	// writeConn is only safe to access outside dispatch, with the
+	idCounter uint32
-	// write lock held. The write lock is taken by sending on
+
-	// requestOp and released by sending on sendDone.
+	// This function, if non-nil, is called when the connection is lost.
-	writeConn net.Conn
+	reconnectFunc reconnectFunc
+
+	// writeConn is used for writing to the connection on the caller's goroutine. It should
+	// only be accessed outside of dispatch, with the write lock held. The write lock is
+	// taken by sending on requestOp and released by sending on sendDone.
+	writeConn jsonWriter
 
 	// for dispatch
 	close       chan struct{}
 	closing     chan struct{}    // closed when client is quitting
 	didClose    chan struct{}    // closed when client quits
-	reconnected chan net.Conn                  // where write/reconnect sends the new connection
+	reconnected chan ServerCodec // where write/reconnect sends the new connection
+	readOp      chan readOp      // read messages
 	readErr     chan error       // errors from read
-	readResp    chan []*jsonrpcMessage         // valid messages from read
+	reqInit     chan *requestOp  // register response IDs, takes write lock
-	requestOp   chan *requestOp                // for registering response IDs
+	reqSent     chan error       // signals write completion, releases write lock
-	sendDone    chan error                     // signals write completion, releases write lock
+	reqTimeout  chan *requestOp  // removes response IDs when call timeout expires
-	respWait    map[string]*requestOp          // active requests
+}
-	subs        map[string]*ClientSubscription // active subscriptions
+
+type reconnectFunc func(ctx context.Context) (ServerCodec, error)
+
+type clientContextKey struct{}
+
+type clientConn struct {
+	codec   ServerCodec
+	handler *handler
+}
+
+func (c *Client) newClientConn(conn ServerCodec) *clientConn {
+	ctx := context.WithValue(context.Background(), clientContextKey{}, c)
+	handler := newHandler(ctx, conn, c.idgen, c.services)
+	return &clientConn{conn, handler}
+}
+
+func (cc *clientConn) close(err error, inflightReq *requestOp) {
+	cc.handler.close(err, inflightReq)
+	cc.codec.Close()
+}
+
+type readOp struct {
+	msgs  []*jsonrpcMessage
+	batch bool
 }
 
 type requestOp struct {
@@ -135,9 +133,14 @@ type requestOp struct {
 	sub  *ClientSubscription  // only set for EthSubscribe requests
 }
 
-func (op *requestOp) wait(ctx context.Context) (*jsonrpcMessage, error) {
+func (op *requestOp) wait(ctx context.Context, c *Client) (*jsonrpcMessage, error) {
 	select {
 	case <-ctx.Done():
+		// Send the timeout to dispatch so it can remove the request IDs.
+		select {
+		case c.reqTimeout <- op:
+		case <-c.closing:
+		}
 		return nil, ctx.Err()
 	case resp := <-op.resp:
 		return resp, op.err
@@ -181,36 +184,57 @@ func DialContext(ctx context.Context, rawurl string) (*Client, error) {
 	}
 }
 
-func newClient(initctx context.Context, connectFunc func(context.Context) (net.Conn, error)) (*Client, error) {
+// Client retrieves the client from the context, if any. This can be used to perform
-	conn, err := connectFunc(initctx)
+// 'reverse calls' in a handler method.
+func ClientFromContext(ctx context.Context) (*Client, bool) {
+	client, ok := ctx.Value(clientContextKey{}).(*Client)
+	return client, ok
+}
+
+func newClient(initctx context.Context, connect reconnectFunc) (*Client, error) {
+	conn, err := connect(initctx)
 	if err != nil {
 		return nil, err
 	}
+	c := initClient(conn, randomIDGenerator(), new(serviceRegistry))
+	c.reconnectFunc = connect
+	return c, nil
+}
+
+func initClient(conn ServerCodec, idgen func() ID, services *serviceRegistry) *Client {
 	_, isHTTP := conn.(*httpConn)
 	c := &Client{
-		writeConn:   conn,
+		idgen:       idgen,
 		isHTTP:      isHTTP,
-		connectFunc: connectFunc,
+		services:    services,
+		writeConn:   conn,
 		close:       make(chan struct{}),
 		closing:     make(chan struct{}),
 		didClose:    make(chan struct{}),
-		reconnected: make(chan net.Conn),
+		reconnected: make(chan ServerCodec),
+		readOp:      make(chan readOp),
 		readErr:     make(chan error),
-		readResp:    make(chan []*jsonrpcMessage),
+		reqInit:     make(chan *requestOp),
-		requestOp:   make(chan *requestOp),
+		reqSent:     make(chan error, 1),
-		sendDone:    make(chan error, 1),
+		reqTimeout:  make(chan *requestOp),
-		respWait:    make(map[string]*requestOp),
-		subs:        make(map[string]*ClientSubscription),
 	}
 	if !isHTTP {
 		go c.dispatch(conn)
 	}
-	return c, nil
+	return c
+}
+
+// RegisterName creates a service for the given receiver type under the given name. When no
+// methods on the given receiver match the criteria to be either a RPC method or a
+// subscription an error is returned. Otherwise a new service is created and added to the
+// service collection this client provides to the server.
+func (c *Client) RegisterName(name string, receiver interface{}) error {
+	return c.services.registerName(name, receiver)
 }
 
 func (c *Client) nextID() json.RawMessage {
 	id := atomic.AddUint32(&c.idCounter, 1)
-	return []byte(strconv.FormatUint(uint64(id), 10))
+	return strconv.AppendUint(nil, uint64(id), 10)
 }
 
 // SupportedModules calls the rpc_modules method, retrieving the list of
@@ -267,7 +291,7 @@ func (c *Client) CallContext(ctx context.Context, result interface{}, method str
 	}
 
 	// dispatch has accepted the request and will close the channel when it quits.
-	switch resp, err := op.wait(ctx); {
+	switch resp, err := op.wait(ctx, c); {
 	case err != nil:
 		return err
 	case resp.Error != nil:
@@ -325,7 +349,7 @@ func (c *Client) BatchCallContext(ctx context.Context, b []BatchElem) error {
 	// Wait for all responses to come back.
 	for n := 0; n < len(b) && err == nil; n++ {
 		var resp *jsonrpcMessage
-		resp, err = op.wait(ctx)
+		resp, err = op.wait(ctx, c)
 		if err != nil {
 			break
 		}
@@ -352,6 +376,22 @@ func (c *Client) BatchCallContext(ctx context.Context, b []BatchElem) error {
 	return err
 }
 
+// Notify sends a notification, i.e. a method call that doesn't expect a response.
+func (c *Client) Notify(ctx context.Context, method string, args ...interface{}) error {
+	op := new(requestOp)
+	msg, err := c.newMessage(method, args...)
+	if err != nil {
+		return err
+	}
+	msg.ID = nil
+
+	if c.isHTTP {
+		return c.sendHTTP(ctx, op, msg)
+	} else {
+		return c.send(ctx, op, msg)
+	}
+}
+
 // EthSubscribe registers a subscripion under the "eth" namespace.
 func (c *Client) EthSubscribe(ctx context.Context, channel interface{}, args ...interface{}) (*ClientSubscription, error) {
 	return c.Subscribe(ctx, "eth", channel, args...)
@@ -402,30 +442,30 @@ func (c *Client) Subscribe(ctx context.Context, namespace string, channel interf
 	if err := c.send(ctx, op, msg); err != nil {
 		return nil, err
 	}
-	if _, err := op.wait(ctx); err != nil {
+	if _, err := op.wait(ctx, c); err != nil {
 		return nil, err
 	}
 	return op.sub, nil
 }
 
 func (c *Client) newMessage(method string, paramsIn ...interface{}) (*jsonrpcMessage, error) {
-	params, err := json.Marshal(paramsIn)
+	msg := &jsonrpcMessage{Version: vsn, ID: c.nextID(), Method: method}
-	if err != nil {
+	if paramsIn != nil { // prevent sending "params":null
+		var err error
+		if msg.Params, err = json.Marshal(paramsIn); err != nil {
 			return nil, err
 		}
-	return &jsonrpcMessage{Version: "2.0", ID: c.nextID(), Method: method, Params: params}, nil
+	}
+	return msg, nil
 }
 
 // send registers op with the dispatch loop, then sends msg on the connection.
 // if sending fails, op is deregistered.
 func (c *Client) send(ctx context.Context, op *requestOp, msg interface{}) error {
 	select {
-	case c.requestOp <- op:
+	case c.reqInit <- op:
-		log.Trace("", "msg", log.Lazy{Fn: func() string {
-			return fmt.Sprint("sending ", msg)
-		}})
 		err := c.write(ctx, msg)
-		c.sendDone <- err
+		c.reqSent <- err
 		return err
 	case <-ctx.Done():
 		// This can happen if the client is overloaded or unable to keep up with
@@ -433,25 +473,17 @@ func (c *Client) send(ctx context.Context, op *requestOp, msg interface{}) error
 		return ctx.Err()
 	case <-c.closing:
 		return ErrClientQuit
-	case <-c.didClose:
-		return ErrClientQuit
 	}
 }
 
 func (c *Client) write(ctx context.Context, msg interface{}) error {
-	deadline, ok := ctx.Deadline()
-	if !ok {
-		deadline = time.Now().Add(defaultWriteTimeout)
-	}
 	// The previous write failed. Try to establish a new connection.
 	if c.writeConn == nil {
 		if err := c.reconnect(ctx); err != nil {
 			return err
 		}
 	}
-	c.writeConn.SetWriteDeadline(deadline)
+	err := c.writeConn.Write(ctx, msg)
-	err := json.NewEncoder(c.writeConn).Encode(msg)
-	c.writeConn.SetWriteDeadline(time.Time{})
 	if err != nil {
 		c.writeConn = nil
 	}
@@ -459,9 +491,18 @@ func (c *Client) write(ctx context.Context, msg interface{}) error {
 }
 
 func (c *Client) reconnect(ctx context.Context) error {
-	newconn, err := c.connectFunc(ctx)
+	if c.reconnectFunc == nil {
+		return errDead
+	}
+
+	if _, ok := ctx.Deadline(); !ok {
+		var cancel func()
+		ctx, cancel = context.WithTimeout(ctx, defaultDialTimeout)
+		defer cancel()
+	}
+	newconn, err := c.reconnectFunc(ctx)
 	if err != nil {
-		log.Trace(fmt.Sprintf("reconnect failed: %v", err))
+		log.Trace("RPC client reconnect failed", "err", err)
 		return err
 	}
 	select {
@@ -477,322 +518,107 @@ func (c *Client) reconnect(ctx context.Context) error {
 // dispatch is the main loop of the client.
 // It sends read messages to waiting calls to Call and BatchCall
 // and subscription notifications to registered subscriptions.
-func (c *Client) dispatch(conn net.Conn) {
+func (c *Client) dispatch(codec ServerCodec) {
-	// Spawn the initial read loop.
-	go c.read(conn)
-
 	var (
 		lastOp      *requestOp  // tracks last send operation
-		requestOpLock = c.requestOp // nil while the send lock is held
+		reqInitLock = c.reqInit // nil while the send lock is held
-		reading       = true        // if true, a read loop is running
+		conn        = c.newClientConn(codec)
+		reading     = true
 	)
-	defer close(c.didClose)
 	defer func() {
 		close(c.closing)
-		c.closeRequestOps(ErrClientQuit)
-		conn.Close()
 		if reading {
-			// Empty read channels until read is dead.
+			conn.close(ErrClientQuit, nil)
-			for {
+			c.drainRead()
-				select {
-				case <-c.readResp:
-				case <-c.readErr:
-					return
-				}
-			}
 		}
+		close(c.didClose)
 	}()
 
+	// Spawn the initial read loop.
+	go c.read(codec)
+
 	for {
 		select {
 		case <-c.close:
 			return
 
-		// Read path.
+		// Read path:
-		case batch := <-c.readResp:
+		case op := <-c.readOp:
-			for _, msg := range batch {
+			if op.batch {
-				switch {
+				conn.handler.handleBatch(op.msgs)
-				case msg.isNotification():
+			} else {
-					log.Trace("", "msg", log.Lazy{Fn: func() string {
+				conn.handler.handleMsg(op.msgs[0])
-						return fmt.Sprint("<-readResp: notification ", msg)
-					}})
-					c.handleNotification(msg)
-				case msg.isResponse():
-					log.Trace("", "msg", log.Lazy{Fn: func() string {
-						return fmt.Sprint("<-readResp: response ", msg)
-					}})
-					c.handleResponse(msg)
-				default:
-					log.Debug("", "msg", log.Lazy{Fn: func() string {
-						return fmt.Sprint("<-readResp: dropping weird message", msg)
-					}})
-					// TODO: maybe close
-				}
 			}
 
 		case err := <-c.readErr:
-			log.Debug("<-readErr", "err", err)
+			conn.handler.log.Debug("RPC connection read error", "err", err)
-			c.closeRequestOps(err)
+			conn.close(err, lastOp)
-			conn.Close()
 			reading = false
 
-		case newconn := <-c.reconnected:
+		// Reconnect:
-			log.Debug("<-reconnected", "reading", reading, "remote", conn.RemoteAddr())
+		case newcodec := <-c.reconnected:
+			log.Debug("RPC client reconnected", "reading", reading, "conn", newcodec.RemoteAddr())
 			if reading {
-				// Wait for the previous read loop to exit. This is a rare case.
+				// Wait for the previous read loop to exit. This is a rare case which
-				conn.Close()
+				// happens if this loop isn't notified in time after the connection breaks.
-				<-c.readErr
+				// In those cases the caller will notice first and reconnect. Closing the
+				// handler terminates all waiting requests (closing op.resp) except for
+				// lastOp, which will be transferred to the new handler.
+				conn.close(errClientReconnected, lastOp)
+				c.drainRead()
 			}
-			go c.read(newconn)
+			go c.read(newcodec)
 			reading = true
-			conn = newconn
+			conn = c.newClientConn(newcodec)
+			// Re-register the in-flight request on the new handler
+			// because that's where it will be sent.
+			conn.handler.addRequestOp(lastOp)
 
-		// Send path.
+		// Send path:
-		case op := <-requestOpLock:
+		case op := <-reqInitLock:
-			// Stop listening for further send ops until the current one is done.
+			// Stop listening for further requests until the current one has been sent.
-			requestOpLock = nil
+			reqInitLock = nil
 			lastOp = op
-			for _, id := range op.ids {
+			conn.handler.addRequestOp(op)
-				c.respWait[string(id)] = op
-			}
 
-		case err := <-c.sendDone:
+		case err := <-c.reqSent:
 			if err != nil {
-				// Remove response handlers for the last send. We remove those here
+				// Remove response handlers for the last send. When the read loop
-				// because the error is already handled in Call or BatchCall. When the
+				// goes down, it will signal all other current operations.
-				// read loop goes down, it will signal all other current operations.
+				conn.handler.removeRequestOp(lastOp)
-				for _, id := range lastOp.ids {
-					delete(c.respWait, string(id))
 			}
-			}
+			// Let the next request in.
-			// Listen for send ops again.
+			reqInitLock = c.reqInit
-			requestOpLock = c.requestOp
 			lastOp = nil
+
+		case op := <-c.reqTimeout:
+			conn.handler.removeRequestOp(op)
 		}
 	}
 }
 
-// closeRequestOps unblocks pending send ops and active subscriptions.
+// drainRead drops read messages until an error occurs.
-func (c *Client) closeRequestOps(err error) {
+func (c *Client) drainRead() {
-	didClose := make(map[*requestOp]bool)
-
-	for id, op := range c.respWait {
-		// Remove the op so that later calls will not close op.resp again.
-		delete(c.respWait, id)
-
-		if !didClose[op] {
-			op.err = err
-			close(op.resp)
-			didClose[op] = true
-		}
-	}
-	for id, sub := range c.subs {
-		delete(c.subs, id)
-		sub.quitWithError(err, false)
-	}
-}
-
-func (c *Client) handleNotification(msg *jsonrpcMessage) {
-	if !strings.HasSuffix(msg.Method, notificationMethodSuffix) {
-		log.Debug("dropping non-subscription message", "msg", msg)
-		return
-	}
-	var subResult struct {
-		ID     string          `json:"subscription"`
-		Result json.RawMessage `json:"result"`
-	}
-	if err := json.Unmarshal(msg.Params, &subResult); err != nil {
-		log.Debug("dropping invalid subscription message", "msg", msg)
-		return
-	}
-	if c.subs[subResult.ID] != nil {
-		c.subs[subResult.ID].deliver(subResult.Result)
-	}
-}
-
-func (c *Client) handleResponse(msg *jsonrpcMessage) {
-	op := c.respWait[string(msg.ID)]
-	if op == nil {
-		log.Debug("unsolicited response", "msg", msg)
-		return
-	}
-	delete(c.respWait, string(msg.ID))
-	// For normal responses, just forward the reply to Call/BatchCall.
-	if op.sub == nil {
-		op.resp <- msg
-		return
-	}
-	// For subscription responses, start the subscription if the server
-	// indicates success. EthSubscribe gets unblocked in either case through
-	// the op.resp channel.
-	defer close(op.resp)
-	if msg.Error != nil {
-		op.err = msg.Error
-		return
-	}
-	if op.err = json.Unmarshal(msg.Result, &op.sub.subid); op.err == nil {
-		go op.sub.start()
-		c.subs[op.sub.subid] = op.sub
-	}
-}
-
-// Reading happens on a dedicated goroutine.
-
-func (c *Client) read(conn net.Conn) error {
-	var (
-		buf json.RawMessage
-		dec = json.NewDecoder(conn)
-	)
-	readMessage := func() (rs []*jsonrpcMessage, err error) {
-		buf = buf[:0]
-		if err = dec.Decode(&buf); err != nil {
-			return nil, err
-		}
-		if isBatch(buf) {
-			err = json.Unmarshal(buf, &rs)
-		} else {
-			rs = make([]*jsonrpcMessage, 1)
-			err = json.Unmarshal(buf, &rs[0])
-		}
-		return rs, err
-	}
-
 	for {
-		resp, err := readMessage()
+		select {
+		case <-c.readOp:
+		case <-c.readErr:
+			return
+		}
+	}
+}
+
+// read decodes RPC messages from a codec, feeding them into dispatch.
+func (c *Client) read(codec ServerCodec) {
+	for {
+		msgs, batch, err := codec.Read()
+		if _, ok := err.(*json.SyntaxError); ok {
+			codec.Write(context.Background(), errorMessage(&parseError{err.Error()}))
+		}
 		if err != nil {
 			c.readErr <- err
-			return err
+			return
 		}
-		c.readResp <- resp
+		c.readOp <- readOp{msgs, batch}
 	}
 }
-
-// Subscriptions.
-
-// A ClientSubscription represents a subscription established through EthSubscribe.
-type ClientSubscription struct {
-	client    *Client
-	etype     reflect.Type
-	channel   reflect.Value
-	namespace string
-	subid     string
-	in        chan json.RawMessage
-
-	quitOnce sync.Once     // ensures quit is closed once
-	quit     chan struct{} // quit is closed when the subscription exits
-	errOnce  sync.Once     // ensures err is closed once
-	err      chan error
-}
-
-func newClientSubscription(c *Client, namespace string, channel reflect.Value) *ClientSubscription {
-	sub := &ClientSubscription{
-		client:    c,
-		namespace: namespace,
-		etype:     channel.Type().Elem(),
-		channel:   channel,
-		quit:      make(chan struct{}),
-		err:       make(chan error, 1),
-		in:        make(chan json.RawMessage),
-	}
-	return sub
-}
-
-// Err returns the subscription error channel. The intended use of Err is to schedule
-// resubscription when the client connection is closed unexpectedly.
-//
-// The error channel receives a value when the subscription has ended due
-// to an error. The received error is nil if Close has been called
-// on the underlying client and no other error has occurred.
-//
-// The error channel is closed when Unsubscribe is called on the subscription.
-func (sub *ClientSubscription) Err() <-chan error {
-	return sub.err
-}
-
-// Unsubscribe unsubscribes the notification and closes the error channel.
-// It can safely be called more than once.
-func (sub *ClientSubscription) Unsubscribe() {
-	sub.quitWithError(nil, true)
-	sub.errOnce.Do(func() { close(sub.err) })
-}
-
-func (sub *ClientSubscription) quitWithError(err error, unsubscribeServer bool) {
-	sub.quitOnce.Do(func() {
-		// The dispatch loop won't be able to execute the unsubscribe call
-		// if it is blocked on deliver. Close sub.quit first because it
-		// unblocks deliver.
-		close(sub.quit)
-		if unsubscribeServer {
-			sub.requestUnsubscribe()
-		}
-		if err != nil {
-			if err == ErrClientQuit {
-				err = nil // Adhere to subscription semantics.
-			}
-			sub.err <- err
-		}
-	})
-}
-
-func (sub *ClientSubscription) deliver(result json.RawMessage) (ok bool) {
-	select {
-	case sub.in <- result:
-		return true
-	case <-sub.quit:
-		return false
-	}
-}
-
-func (sub *ClientSubscription) start() {
-	sub.quitWithError(sub.forward())
-}
-
-func (sub *ClientSubscription) forward() (err error, unsubscribeServer bool) {
-	cases := []reflect.SelectCase{
-		{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(sub.quit)},
-		{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(sub.in)},
-		{Dir: reflect.SelectSend, Chan: sub.channel},
-	}
-	buffer := list.New()
-	defer buffer.Init()
-	for {
-		var chosen int
-		var recv reflect.Value
-		if buffer.Len() == 0 {
-			// Idle, omit send case.
-			chosen, recv, _ = reflect.Select(cases[:2])
-		} else {
-			// Non-empty buffer, send the first queued item.
-			cases[2].Send = reflect.ValueOf(buffer.Front().Value)
-			chosen, recv, _ = reflect.Select(cases)
-		}
-
-		switch chosen {
-		case 0: // <-sub.quit
-			return nil, false
-		case 1: // <-sub.in
-			val, err := sub.unmarshal(recv.Interface().(json.RawMessage))
-			if err != nil {
-				return err, true
-			}
-			if buffer.Len() == maxClientSubscriptionBuffer {
-				return ErrSubscriptionQueueOverflow, true
-			}
-			buffer.PushBack(val)
-		case 2: // sub.channel<-
-			cases[2].Send = reflect.Value{} // Don't hold onto the value.
-			buffer.Remove(buffer.Front())
-		}
-	}
-}
-
-func (sub *ClientSubscription) unmarshal(result json.RawMessage) (interface{}, error) {
-	val := reflect.New(sub.etype)
-	err := json.Unmarshal(result, val.Interface())
-	return val.Elem().Interface(), err
-}
-
-func (sub *ClientSubscription) requestUnsubscribe() error {
-	var result interface{}
-	return sub.client.Call(&result, sub.namespace+unsubscribeMethodSuffix, sub.subid)
-}

rpc/client_test.go

@@ -35,13 +35,13 @@ import (
 )
 
 func TestClientRequest(t *testing.T) {
-	server := newTestServer("service", new(Service))
+	server := newTestServer()
 	defer server.Stop()
 	client := DialInProc(server)
 	defer client.Close()
 
 	var resp Result
-	if err := client.Call(&resp, "service_echo", "hello", 10, &Args{"world"}); err != nil {
+	if err := client.Call(&resp, "test_echo", "hello", 10, &Args{"world"}); err != nil {
 		t.Fatal(err)
 	}
 	if !reflect.DeepEqual(resp, Result{"hello", 10, &Args{"world"}}) {
@@ -50,19 +50,19 @@ func TestClientRequest(t *testing.T) {
 }
 
 func TestClientBatchRequest(t *testing.T) {
-	server := newTestServer("service", new(Service))
+	server := newTestServer()
 	defer server.Stop()
 	client := DialInProc(server)
 	defer client.Close()
 
 	batch := []BatchElem{
 		{
-			Method: "service_echo",
+			Method: "test_echo",
 			Args:   []interface{}{"hello", 10, &Args{"world"}},
 			Result: new(Result),
 		},
 		{
-			Method: "service_echo",
+			Method: "test_echo",
 			Args:   []interface{}{"hello2", 11, &Args{"world"}},
 			Result: new(Result),
 		},
@@ -77,12 +77,12 @@ func TestClientBatchRequest(t *testing.T) {
 	}
 	wantResult := []BatchElem{
 		{
-			Method: "service_echo",
+			Method: "test_echo",
 			Args:   []interface{}{"hello", 10, &Args{"world"}},
 			Result: &Result{"hello", 10, &Args{"world"}},
 		},
 		{
-			Method: "service_echo",
+			Method: "test_echo",
 			Args:   []interface{}{"hello2", 11, &Args{"world"}},
 			Result: &Result{"hello2", 11, &Args{"world"}},
 		},
@@ -90,7 +90,7 @@ func TestClientBatchRequest(t *testing.T) {
 			Method: "no_such_method",
 			Args:   []interface{}{1, 2, 3},
 			Result: new(int),
-			Error:  &jsonError{Code: -32601, Message: "The method no_such_method_ does not exist/is not available"},
+			Error:  &jsonError{Code: -32601, Message: "the method no_such_method does not exist/is not available"},
 		},
 	}
 	if !reflect.DeepEqual(batch, wantResult) {
@@ -98,6 +98,17 @@ func TestClientBatchRequest(t *testing.T) {
 	}
 }
 
+func TestClientNotify(t *testing.T) {
+	server := newTestServer()
+	defer server.Stop()
+	client := DialInProc(server)
+	defer client.Close()
+
+	if err := client.Notify(context.Background(), "test_echo", "hello", 10, &Args{"world"}); err != nil {
+		t.Fatal(err)
+	}
+}
+
 // func TestClientCancelInproc(t *testing.T) { testClientCancel("inproc", t) }
 func TestClientCancelWebsocket(t *testing.T) { testClientCancel("ws", t) }
 func TestClientCancelHTTP(t *testing.T)      { testClientCancel("http", t) }
@@ -106,7 +117,12 @@ func TestClientCancelIPC(t *testing.T)       { testClientCancel("ipc", t) }
 // This test checks that requests made through CallContext can be canceled by canceling
 // the context.
 func testClientCancel(transport string, t *testing.T) {
-	server := newTestServer("service", new(Service))
+	// These tests take a lot of time, run them all at once.
+	// You probably want to run with -parallel 1 or comment out
+	// the call to t.Parallel if you enable the logging.
+	t.Parallel()
+
+	server := newTestServer()
 	defer server.Stop()
 
 	// What we want to achieve is that the context gets canceled
@@ -142,11 +158,6 @@ func testClientCancel(transport string, t *testing.T) {
 		panic("unknown transport: " + transport)
 	}
 
-	// These tests take a lot of time, run them all at once.
-	// You probably want to run with -parallel 1 or comment out
-	// the call to t.Parallel if you enable the logging.
-	t.Parallel()
-
 	// The actual test starts here.
 	var (
 		wg       sync.WaitGroup
@@ -174,7 +185,8 @@ func testClientCancel(transport string, t *testing.T) {
 			}
 			// Now perform a call with the context.
 			// The key thing here is that no call will ever complete successfully.
-			err := client.CallContext(ctx, nil, "service_sleep", 2*maxContextCancelTimeout)
+			sleepTime := maxContextCancelTimeout + 20*time.Millisecond
+			err := client.CallContext(ctx, nil, "test_sleep", sleepTime)
 			if err != nil {
 				log.Debug(fmt.Sprint("got expected error:", err))
 			} else {
@@ -191,7 +203,7 @@ func testClientCancel(transport string, t *testing.T) {
 }
 
 func TestClientSubscribeInvalidArg(t *testing.T) {
-	server := newTestServer("service", new(Service))
+	server := newTestServer()
 	defer server.Stop()
 	client := DialInProc(server)
 	defer client.Close()
@@ -221,14 +233,14 @@ func TestClientSubscribeInvalidArg(t *testing.T) {
 }
 
 func TestClientSubscribe(t *testing.T) {
-	server := newTestServer("eth", new(NotificationTestService))
+	server := newTestServer()
 	defer server.Stop()
 	client := DialInProc(server)
 	defer client.Close()
 
 	nc := make(chan int)
 	count := 10
-	sub, err := client.EthSubscribe(context.Background(), nc, "someSubscription", count, 0)
+	sub, err := client.Subscribe(context.Background(), "nftest", nc, "someSubscription", count, 0)
 	if err != nil {
 		t.Fatal("can't subscribe:", err)
 	}
@@ -251,46 +263,17 @@ func TestClientSubscribe(t *testing.T) {
 	}
 }
 
-func TestClientSubscribeCustomNamespace(t *testing.T) {
+// In this test, the connection drops while Subscribe is waiting for a response.
-	namespace := "custom"
-	server := newTestServer(namespace, new(NotificationTestService))
-	defer server.Stop()
-	client := DialInProc(server)
-	defer client.Close()
-
-	nc := make(chan int)
-	count := 10
-	sub, err := client.Subscribe(context.Background(), namespace, nc, "someSubscription", count, 0)
-	if err != nil {
-		t.Fatal("can't subscribe:", err)
-	}
-	for i := 0; i < count; i++ {
-		if val := <-nc; val != i {
-			t.Fatalf("value mismatch: got %d, want %d", val, i)
-		}
-	}
-
-	sub.Unsubscribe()
-	select {
-	case v := <-nc:
-		t.Fatal("received value after unsubscribe:", v)
-	case err := <-sub.Err():
-		if err != nil {
-			t.Fatalf("Err returned a non-nil error after explicit unsubscribe: %q", err)
-		}
-	case <-time.After(1 * time.Second):
-		t.Fatalf("subscription not closed within 1s after unsubscribe")
-	}
-}
-
-// In this test, the connection drops while EthSubscribe is
-// waiting for a response.
 func TestClientSubscribeClose(t *testing.T) {
-	service := &NotificationTestService{
+	server := newTestServer()
+	service := &notificationTestService{
 		gotHangSubscriptionReq:  make(chan struct{}),
 		unblockHangSubscription: make(chan struct{}),
 	}
-	server := newTestServer("eth", service)
+	if err := server.RegisterName("nftest2", service); err != nil {
+		t.Fatal(err)
+	}
+
 	defer server.Stop()
 	client := DialInProc(server)
 	defer client.Close()
@@ -302,7 +285,7 @@ func TestClientSubscribeClose(t *testing.T) {
 		err  error
 	)
 	go func() {
-		sub, err = client.EthSubscribe(context.Background(), nc, "hangSubscription", 999)
+		sub, err = client.Subscribe(context.Background(), "nftest2", nc, "hangSubscription", 999)
 		errc <- err
 	}()
 
@@ -313,27 +296,26 @@ func TestClientSubscribeClose(t *testing.T) {
 	select {
 	case err := <-errc:
 		if err == nil {
-			t.Errorf("EthSubscribe returned nil error after Close")
+			t.Errorf("Subscribe returned nil error after Close")
 		}
 		if sub != nil {
-			t.Error("EthSubscribe returned non-nil subscription after Close")
+			t.Error("Subscribe returned non-nil subscription after Close")
 		}
 	case <-time.After(1 * time.Second):
-		t.Fatalf("EthSubscribe did not return within 1s after Close")
+		t.Fatalf("Subscribe did not return within 1s after Close")
 	}
 }
 
 // This test reproduces https://github.com/ethereum/go-ethereum/issues/17837 where the
 // client hangs during shutdown when Unsubscribe races with Client.Close.
 func TestClientCloseUnsubscribeRace(t *testing.T) {
-	service := &NotificationTestService{}
+	server := newTestServer()
-	server := newTestServer("eth", service)
 	defer server.Stop()
 
 	for i := 0; i < 20; i++ {
 		client := DialInProc(server)
 		nc := make(chan int)
-		sub, err := client.EthSubscribe(context.Background(), nc, "someSubscription", 3, 1)
+		sub, err := client.Subscribe(context.Background(), "nftest", nc, "someSubscription", 3, 1)
 		if err != nil {
 			t.Fatal(err)
 		}
@@ -350,7 +332,7 @@ func TestClientCloseUnsubscribeRace(t *testing.T) {
 // This test checks that Client doesn't lock up when a single subscriber
 // doesn't read subscription events.
 func TestClientNotificationStorm(t *testing.T) {
-	server := newTestServer("eth", new(NotificationTestService))
+	server := newTestServer()
 	defer server.Stop()
 
 	doTest := func(count int, wantError bool) {
@@ -362,7 +344,7 @@ func TestClientNotificationStorm(t *testing.T) {
 		// Subscribe on the server. It will start sending many notifications
 		// very quickly.
 		nc := make(chan int)
-		sub, err := client.EthSubscribe(ctx, nc, "someSubscription", count, 0)
+		sub, err := client.Subscribe(ctx, "nftest", nc, "someSubscription", count, 0)
 		if err != nil {
 			t.Fatal("can't subscribe:", err)
 		}
@@ -384,7 +366,7 @@ func TestClientNotificationStorm(t *testing.T) {
 				return
 			}
 			var r int
-			err := client.CallContext(ctx, &r, "eth_echo", i)
+			err := client.CallContext(ctx, &r, "nftest_echo", i)
 			if err != nil {
 				if !wantError {
 					t.Fatalf("(%d/%d) call error: %v", i, count, err)
@@ -399,7 +381,7 @@ func TestClientNotificationStorm(t *testing.T) {
 }
 
 func TestClientHTTP(t *testing.T) {
-	server := newTestServer("service", new(Service))
+	server := newTestServer()
 	defer server.Stop()
 
 	client, hs := httpTestClient(server, "http", nil)
@@ -416,7 +398,7 @@ func TestClientHTTP(t *testing.T) {
 	for i := range results {
 		i := i
 		go func() {
-			errc <- client.Call(&results[i], "service_echo",
+			errc <- client.Call(&results[i], "test_echo",
 				wantResult.String, wantResult.Int, wantResult.Args)
 		}()
 	}
@@ -445,16 +427,16 @@ func TestClientHTTP(t *testing.T) {
 
 func TestClientReconnect(t *testing.T) {
 	startServer := func(addr string) (*Server, net.Listener) {
-		srv := newTestServer("service", new(Service))
+		srv := newTestServer()
 		l, err := net.Listen("tcp", addr)
 		if err != nil {
-			t.Fatal(err)
+			t.Fatal("can't listen:", err)
 		}
 		go http.Serve(l, srv.WebsocketHandler([]string{"*"}))
 		return srv, l
 	}
 
-	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
 	defer cancel()
 
 	// Start a server and corresponding client.
@@ -466,21 +448,22 @@ func TestClientReconnect(t *testing.T) {
 
 	// Perform a call. This should work because the server is up.
 	var resp Result
-	if err := client.CallContext(ctx, &resp, "service_echo", "", 1, nil); err != nil {
+	if err := client.CallContext(ctx, &resp, "test_echo", "", 1, nil); err != nil {
 		t.Fatal(err)
 	}
 
-	// Shut down the server and try calling again. It shouldn't work.
+	// Shut down the server and allow for some cool down time so we can listen on the same
+	// address again.
 	l1.Close()
 	s1.Stop()
-	if err := client.CallContext(ctx, &resp, "service_echo", "", 2, nil); err == nil {
+	time.Sleep(2 * time.Second)
+
+	// Try calling again. It shouldn't work.
+	if err := client.CallContext(ctx, &resp, "test_echo", "", 2, nil); err == nil {
 		t.Error("successful call while the server is down")
 		t.Logf("resp: %#v", resp)
 	}
 
-	// Allow for some cool down time so we can listen on the same address again.
-	time.Sleep(2 * time.Second)
-
 	// Start it up again and call again. The connection should be reestablished.
 	// We spawn multiple calls here to check whether this hangs somehow.
 	s2, l2 := startServer(l1.Addr().String())
@@ -493,7 +476,7 @@ func TestClientReconnect(t *testing.T) {
 		go func() {
 			<-start
 			var resp Result
-			errors <- client.CallContext(ctx, &resp, "service_echo", "", 3, nil)
+			errors <- client.CallContext(ctx, &resp, "test_echo", "", 3, nil)
 		}()
 	}
 	close(start)
@@ -503,20 +486,12 @@ func TestClientReconnect(t *testing.T) {
 			errcount++
 		}
 	}
-	t.Log("err:", err)
+	t.Logf("%d errors, last error: %v", errcount, err)
 	if errcount > 1 {
 		t.Errorf("expected one error after disconnect, got %d", errcount)
 	}
 }
 
-func newTestServer(serviceName string, service interface{}) *Server {
-	server := NewServer()
-	if err := server.RegisterName(serviceName, service); err != nil {
-		panic(err)
-	}
-	return server
-}
-
 func httpTestClient(srv *Server, transport string, fl *flakeyListener) (*Client, *httptest.Server) {
 	// Create the HTTP server.
 	var hs *httptest.Server

rpc/doc.go

@@ -15,43 +15,49 @@
 // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
 
 /*
-Package rpc provides access to the exported methods of an object across a network
+
-or other I/O connection. After creating a server instance objects can be registered,
+Package rpc implements bi-directional JSON-RPC 2.0 on multiple transports.
-making it visible from the outside. Exported methods that follow specific
+
-conventions can be called remotely. It also has support for the publish/subscribe
+It provides access to the exported methods of an object across a network or other I/O
-pattern.
+connection. After creating a server or client instance, objects can be registered to make
+them visible as 'services'. Exported methods that follow specific conventions can be
+called remotely. It also has support for the publish/subscribe pattern.
+
+RPC Methods
 
 Methods that satisfy the following criteria are made available for remote access:
- - object must be exported
+
  - method must be exported
  - method returns 0, 1 (response or error) or 2 (response and error) values
  - method argument(s) must be exported or builtin types
  - method returned value(s) must be exported or builtin types
 
 An example method:
+
  func (s *CalcService) Add(a, b int) (int, error)
 
-When the returned error isn't nil the returned integer is ignored and the error is
+When the returned error isn't nil the returned integer is ignored and the error is sent
-sent back to the client. Otherwise the returned integer is sent back to the client.
+back to the client. Otherwise the returned integer is sent back to the client.
 
-Optional arguments are supported by accepting pointer values as arguments. E.g.
+Optional arguments are supported by accepting pointer values as arguments. E.g. if we want
-if we want to do the addition in an optional finite field we can accept a mod
+to do the addition in an optional finite field we can accept a mod argument as pointer
-argument as pointer value.
+value.
 
- func (s *CalService) Add(a, b int, mod *int) (int, error)
+ func (s *CalcService) Add(a, b int, mod *int) (int, error)
 
-This RPC method can be called with 2 integers and a null value as third argument.
+This RPC method can be called with 2 integers and a null value as third argument. In that
-In that case the mod argument will be nil. Or it can be called with 3 integers,
+case the mod argument will be nil. Or it can be called with 3 integers, in that case mod
-in that case mod will be pointing to the given third argument. Since the optional
+will be pointing to the given third argument. Since the optional argument is the last
-argument is the last argument the RPC package will also accept 2 integers as
+argument the RPC package will also accept 2 integers as arguments. It will pass the mod
-arguments. It will pass the mod argument as nil to the RPC method.
+argument as nil to the RPC method.
 
-The server offers the ServeCodec method which accepts a ServerCodec instance. It will
+The server offers the ServeCodec method which accepts a ServerCodec instance. It will read
-read requests from the codec, process the request and sends the response back to the
+requests from the codec, process the request and sends the response back to the client
-client using the codec. The server can execute requests concurrently. Responses
+using the codec. The server can execute requests concurrently. Responses can be sent back
-can be sent back to the client out of order.
+to the client out of order.
 
 An example server which uses the JSON codec:
+
  type CalculatorService struct {}
 
  func (s *CalculatorService) Add(a, b int) int {
@@ -73,26 +79,40 @@ An example server which uses the JSON codec:
  for {
 	c, _ := l.AcceptUnix()
 	codec := v2.NewJSONCodec(c)
-	go server.ServeCodec(codec)
+	go server.ServeCodec(codec, 0)
  }
 
+Subscriptions
+
 The package also supports the publish subscribe pattern through the use of subscriptions.
-A method that is considered eligible for notifications must satisfy the following criteria:
+A method that is considered eligible for notifications must satisfy the following
- - object must be exported
+criteria:
+
  - method must be exported
  - first method argument type must be context.Context
  - method argument(s) must be exported or builtin types
- - method must return the tuple Subscription, error
+ - method must have return types (rpc.Subscription, error)
 
 An example method:
- func (s *BlockChainService) NewBlocks(ctx context.Context) (Subscription, error) {
+
+ func (s *BlockChainService) NewBlocks(ctx context.Context) (rpc.Subscription, error) {
  	...
  }
 
-Subscriptions are deleted when:
+When the service containing the subscription method is registered to the server, for
- - the user sends an unsubscribe request
+example under the "blockchain" namespace, a subscription is created by calling the
- - the connection which was used to create the subscription is closed. This can be initiated
+"blockchain_subscribe" method.
-   by the client and server. The server will close the connection on a write error or when
+
-   the queue of buffered notifications gets too big.
+Subscriptions are deleted when the user sends an unsubscribe request or when the
+connection which was used to create the subscription is closed. This can be initiated by
+the client and server. The server will close the connection for any write error.
+
+For more information about subscriptions, see https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB.
+
+Reverse Calls
+
+In any method handler, an instance of rpc.Client can be accessed through the
+ClientFromContext method. Using this client instance, server-to-client method calls can be
+performed on the RPC connection.
 */
 package rpc

rpc/errors.go

@@ -18,18 +18,31 @@ package rpc
 
 import "fmt"
 
-// request is for an unknown service
+const defaultErrorCode = -32000
-type methodNotFoundError struct {
+
-	service string
+type methodNotFoundError struct{ method string }
-	method  string
-}
 
 func (e *methodNotFoundError) ErrorCode() int { return -32601 }
 
 func (e *methodNotFoundError) Error() string {
-	return fmt.Sprintf("The method %s%s%s does not exist/is not available", e.service, serviceMethodSeparator, e.method)
+	return fmt.Sprintf("the method %s does not exist/is not available", e.method)
 }
 
+type subscriptionNotFoundError struct{ namespace, subscription string }
+
+func (e *subscriptionNotFoundError) ErrorCode() int { return -32601 }
+
+func (e *subscriptionNotFoundError) Error() string {
+	return fmt.Sprintf("no %q subscription in %s namespace", e.subscription, e.namespace)
+}
+
+// Invalid JSON was received by the server.
+type parseError struct{ message string }
+
+func (e *parseError) ErrorCode() int { return -32700 }
+
+func (e *parseError) Error() string { return e.message }
+
 // received message isn't a valid request
 type invalidRequestError struct{ message string }
 
@@ -50,17 +63,3 @@ type invalidParamsError struct{ message string }
 func (e *invalidParamsError) ErrorCode() int { return -32602 }
 
 func (e *invalidParamsError) Error() string { return e.message }
-
-// logic error, callback returned an error
-type callbackError struct{ message string }
-
-func (e *callbackError) ErrorCode() int { return -32000 }
-
-func (e *callbackError) Error() string { return e.message }
-
-// issued when a request is received after the server is issued to stop.
-type shutdownError struct{}
-
-func (e *shutdownError) ErrorCode() int { return -32000 }
-
-func (e *shutdownError) Error() string { return "server is shutting down" }

rpc/handler.go

@@ -0,0 +1,397 @@
+// Copyright 2018 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package rpc
+
+import (
+	"context"
+	"encoding/json"
+	"reflect"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/ethereum/go-ethereum/log"
+)
+
+// handler handles JSON-RPC messages. There is one handler per connection. Note that
+// handler is not safe for concurrent use. Message handling never blocks indefinitely
+// because RPCs are processed on background goroutines launched by handler.
+//
+// The entry points for incoming messages are:
+//
+//    h.handleMsg(message)
+//    h.handleBatch(message)
+//
+// Outgoing calls use the requestOp struct. Register the request before sending it
+// on the connection:
+//
+//    op := &requestOp{ids: ...}
+//    h.addRequestOp(op)
+//
+// Now send the request, then wait for the reply to be delivered through handleMsg:
+//
+//    if err := op.wait(...); err != nil {
+//        h.removeRequestOp(op) // timeout, etc.
+//    }
+//
+type handler struct {
+	reg            *serviceRegistry
+	unsubscribeCb  *callback
+	idgen          func() ID                      // subscription ID generator
+	respWait       map[string]*requestOp          // active client requests
+	clientSubs     map[string]*ClientSubscription // active client subscriptions
+	callWG         sync.WaitGroup                 // pending call goroutines
+	rootCtx        context.Context                // canceled by close()
+	cancelRoot     func()                         // cancel function for rootCtx
+	conn           jsonWriter                     // where responses will be sent
+	log            log.Logger
+	allowSubscribe bool
+
+	subLock    sync.Mutex
+	serverSubs map[ID]*Subscription
+}
+
+type callProc struct {
+	ctx       context.Context
+	notifiers []*Notifier
+}
+
+func newHandler(connCtx context.Context, conn jsonWriter, idgen func() ID, reg *serviceRegistry) *handler {
+	rootCtx, cancelRoot := context.WithCancel(connCtx)
+	h := &handler{
+		reg:            reg,
+		idgen:          idgen,
+		conn:           conn,
+		respWait:       make(map[string]*requestOp),
+		clientSubs:     make(map[string]*ClientSubscription),
+		rootCtx:        rootCtx,
+		cancelRoot:     cancelRoot,
+		allowSubscribe: true,
+		serverSubs:     make(map[ID]*Subscription),
+		log:            log.Root(),
+	}
+	if conn.RemoteAddr() != "" {
+		h.log = h.log.New("conn", conn.RemoteAddr())
+	}
+	h.unsubscribeCb = newCallback(reflect.Value{}, reflect.ValueOf(h.unsubscribe))
+	return h
+}
+
+// handleBatch executes all messages in a batch and returns the responses.
+func (h *handler) handleBatch(msgs []*jsonrpcMessage) {
+	// Emit error response for empty batches:
+	if len(msgs) == 0 {
+		h.startCallProc(func(cp *callProc) {
+			h.conn.Write(cp.ctx, errorMessage(&invalidRequestError{"empty batch"}))
+		})
+		return
+	}
+
+	// Handle non-call messages first:
+	calls := make([]*jsonrpcMessage, 0, len(msgs))
+	for _, msg := range msgs {
+		if handled := h.handleImmediate(msg); !handled {
+			calls = append(calls, msg)
+		}
+	}
+	if len(calls) == 0 {
+		return
+	}
+	// Process calls on a goroutine because they may block indefinitely:
+	h.startCallProc(func(cp *callProc) {
+		answers := make([]*jsonrpcMessage, 0, len(msgs))
+		for _, msg := range calls {
+			if answer := h.handleCallMsg(cp, msg); answer != nil {
+				answers = append(answers, answer)
+			}
+		}
+		h.addSubscriptions(cp.notifiers)
+		if len(answers) > 0 {
+			h.conn.Write(cp.ctx, answers)
+		}
+		for _, n := range cp.notifiers {
+			n.activate()
+		}
+	})
+}
+
+// handleMsg handles a single message.
+func (h *handler) handleMsg(msg *jsonrpcMessage) {
+	if ok := h.handleImmediate(msg); ok {
+		return
+	}
+	h.startCallProc(func(cp *callProc) {
+		answer := h.handleCallMsg(cp, msg)
+		h.addSubscriptions(cp.notifiers)
+		if answer != nil {
+			h.conn.Write(cp.ctx, answer)
+		}
+		for _, n := range cp.notifiers {
+			n.activate()
+		}
+	})
+}
+
+// close cancels all requests except for inflightReq and waits for
+// call goroutines to shut down.
+func (h *handler) close(err error, inflightReq *requestOp) {
+	h.cancelAllRequests(err, inflightReq)
+	h.cancelRoot()
+	h.callWG.Wait()
+	h.cancelServerSubscriptions(err)
+}
+
+// addRequestOp registers a request operation.
+func (h *handler) addRequestOp(op *requestOp) {
+	for _, id := range op.ids {
+		h.respWait[string(id)] = op
+	}
+}
+
+// removeRequestOps stops waiting for the given request IDs.
+func (h *handler) removeRequestOp(op *requestOp) {
+	for _, id := range op.ids {
+		delete(h.respWait, string(id))
+	}
+}
+
+// cancelAllRequests unblocks and removes pending requests and active subscriptions.
+func (h *handler) cancelAllRequests(err error, inflightReq *requestOp) {
+	didClose := make(map[*requestOp]bool)
+	if inflightReq != nil {
+		didClose[inflightReq] = true
+	}
+
+	for id, op := range h.respWait {
+		// Remove the op so that later calls will not close op.resp again.
+		delete(h.respWait, id)
+
+		if !didClose[op] {
+			op.err = err
+			close(op.resp)
+			didClose[op] = true
+		}
+	}
+	for id, sub := range h.clientSubs {
+		delete(h.clientSubs, id)
+		sub.quitWithError(err, false)
+	}
+}
+
+func (h *handler) addSubscriptions(nn []*Notifier) {
+	h.subLock.Lock()
+	defer h.subLock.Unlock()
+
+	for _, n := range nn {
+		if sub := n.takeSubscription(); sub != nil {
+			h.serverSubs[sub.ID] = sub
+		}
+	}
+}
+
+// cancelServerSubscriptions removes all subscriptions and closes their error channels.
+func (h *handler) cancelServerSubscriptions(err error) {
+	h.subLock.Lock()
+	defer h.subLock.Unlock()
+
+	for id, s := range h.serverSubs {
+		s.err <- err
+		close(s.err)
+		delete(h.serverSubs, id)
+	}
+}
+
+// startCallProc runs fn in a new goroutine and starts tracking it in the h.calls wait group.
+func (h *handler) startCallProc(fn func(*callProc)) {
+	h.callWG.Add(1)
+	go func() {
+		ctx, cancel := context.WithCancel(h.rootCtx)
+		defer h.callWG.Done()
+		defer cancel()
+		fn(&callProc{ctx: ctx})
+	}()
+}
+
+// handleImmediate executes non-call messages. It returns false if the message is a
+// call or requires a reply.
+func (h *handler) handleImmediate(msg *jsonrpcMessage) bool {
+	start := time.Now()
+	switch {
+	case msg.isNotification():
+		if strings.HasSuffix(msg.Method, notificationMethodSuffix) {
+			h.handleSubscriptionResult(msg)
+			return true
+		}
+		return false
+	case msg.isResponse():
+		h.handleResponse(msg)
+		h.log.Trace("Handled RPC response", "reqid", idForLog{msg.ID}, "t", time.Since(start))
+		return true
+	default:
+		return false
+	}
+}
+
+// handleSubscriptionResult processes subscription notifications.
+func (h *handler) handleSubscriptionResult(msg *jsonrpcMessage) {
+	var result subscriptionResult
+	if err := json.Unmarshal(msg.Params, &result); err != nil {
+		h.log.Debug("Dropping invalid subscription message")
+		return
+	}
+	if h.clientSubs[result.ID] != nil {
+		h.clientSubs[result.ID].deliver(result.Result)
+	}
+}
+
+// handleResponse processes method call responses.
+func (h *handler) handleResponse(msg *jsonrpcMessage) {
+	op := h.respWait[string(msg.ID)]
+	if op == nil {
+		h.log.Debug("Unsolicited RPC response", "reqid", idForLog{msg.ID})
+		return
+	}
+	delete(h.respWait, string(msg.ID))
+	// For normal responses, just forward the reply to Call/BatchCall.
+	if op.sub == nil {
+		op.resp <- msg
+		return
+	}
+	// For subscription responses, start the subscription if the server
+	// indicates success. EthSubscribe gets unblocked in either case through
+	// the op.resp channel.
+	defer close(op.resp)
+	if msg.Error != nil {
+		op.err = msg.Error
+		return
+	}
+	if op.err = json.Unmarshal(msg.Result, &op.sub.subid); op.err == nil {
+		go op.sub.start()
+		h.clientSubs[op.sub.subid] = op.sub
+	}
+}
+
+// handleCallMsg executes a call message and returns the answer.
+func (h *handler) handleCallMsg(ctx *callProc, msg *jsonrpcMessage) *jsonrpcMessage {
+	start := time.Now()
+	switch {
+	case msg.isNotification():
+		h.handleCall(ctx, msg)
+		h.log.Debug("Served "+msg.Method, "t", time.Since(start))
+		return nil
+	case msg.isCall():
+		resp := h.handleCall(ctx, msg)
+		if resp.Error != nil {
+			h.log.Info("Served "+msg.Method, "reqid", idForLog{msg.ID}, "t", time.Since(start), "err", resp.Error.Message)
+		} else {
+			h.log.Debug("Served "+msg.Method, "reqid", idForLog{msg.ID}, "t", time.Since(start))
+		}
+		return resp
+	case msg.hasValidID():
+		return msg.errorResponse(&invalidRequestError{"invalid request"})
+	default:
+		return errorMessage(&invalidRequestError{"invalid request"})
+	}
+}
+
+// handleCall processes method calls.
+func (h *handler) handleCall(cp *callProc, msg *jsonrpcMessage) *jsonrpcMessage {
+	if msg.isSubscribe() {
+		return h.handleSubscribe(cp, msg)
+	}
+	var callb *callback
+	if msg.isUnsubscribe() {
+		callb = h.unsubscribeCb
+	} else {
+		callb = h.reg.callback(msg.Method)
+	}
+	if callb == nil {
+		return msg.errorResponse(&methodNotFoundError{method: msg.Method})
+	}
+	args, err := parsePositionalArguments(msg.Params, callb.argTypes)
+	if err != nil {
+		return msg.errorResponse(&invalidParamsError{err.Error()})
+	}
+
+	return h.runMethod(cp.ctx, msg, callb, args)
+}
+
+// handleSubscribe processes *_subscribe method calls.
+func (h *handler) handleSubscribe(cp *callProc, msg *jsonrpcMessage) *jsonrpcMessage {
+	if !h.allowSubscribe {
+		return msg.errorResponse(ErrNotificationsUnsupported)
+	}
+
+	// Subscription method name is first argument.
+	name, err := parseSubscriptionName(msg.Params)
+	if err != nil {
+		return msg.errorResponse(&invalidParamsError{err.Error()})
+	}
+	namespace := msg.namespace()
+	callb := h.reg.subscription(namespace, name)
+	if callb == nil {
+		return msg.errorResponse(&subscriptionNotFoundError{namespace, name})
+	}
+
+	// Parse subscription name arg too, but remove it before calling the callback.
+	argTypes := append([]reflect.Type{stringType}, callb.argTypes...)
+	args, err := parsePositionalArguments(msg.Params, argTypes)
+	if err != nil {
+		return msg.errorResponse(&invalidParamsError{err.Error()})
+	}
+	args = args[1:]
+
+	// Install notifier in context so the subscription handler can find it.
+	n := &Notifier{h: h, namespace: namespace}
+	cp.notifiers = append(cp.notifiers, n)
+	ctx := context.WithValue(cp.ctx, notifierKey{}, n)
+
+	return h.runMethod(ctx, msg, callb, args)
+}
+
+// runMethod runs the Go callback for an RPC method.
+func (h *handler) runMethod(ctx context.Context, msg *jsonrpcMessage, callb *callback, args []reflect.Value) *jsonrpcMessage {
+	result, err := callb.call(ctx, msg.Method, args)
+	if err != nil {
+		return msg.errorResponse(err)
+	}
+	return msg.response(result)
+}
+
+// unsubscribe is the callback function for all *_unsubscribe calls.
+func (h *handler) unsubscribe(ctx context.Context, id ID) (bool, error) {
+	h.subLock.Lock()
+	defer h.subLock.Unlock()
+
+	s := h.serverSubs[id]
+	if s == nil {
+		return false, ErrSubscriptionNotFound
+	}
+	close(s.err)
+	delete(h.serverSubs, id)
+	return true, nil
+}
+
+type idForLog struct{ json.RawMessage }
+
+func (id idForLog) String() string {
+	if s, err := strconv.Unquote(string(id.RawMessage)); err == nil {
+		return s
+	}
+	return string(id.RawMessage)
+}

rpc/http.go

@@ -37,38 +37,39 @@ import (
 
 const (
 	maxRequestContentLength = 1024 * 512
+	contentType             = "application/json"
 )
 
-var (
 // https://www.jsonrpc.org/historical/json-rpc-over-http.html#id13
-	acceptedContentTypes = []string{"application/json", "application/json-rpc", "application/jsonrequest"}
+var acceptedContentTypes = []string{contentType, "application/json-rpc", "application/jsonrequest"}
-	contentType          = acceptedContentTypes[0]
-	nullAddr, _          = net.ResolveTCPAddr("tcp", "127.0.0.1:0")
-)
 
 type httpConn struct {
 	client    *http.Client
 	req       *http.Request
 	closeOnce sync.Once
-	closed    chan struct{}
+	closed    chan interface{}
 }
 
 // httpConn is treated specially by Client.
-func (hc *httpConn) LocalAddr() net.Addr              { return nullAddr }
+func (hc *httpConn) Write(context.Context, interface{}) error {
-func (hc *httpConn) RemoteAddr() net.Addr             { return nullAddr }
+	panic("Write called on httpConn")
-func (hc *httpConn) SetReadDeadline(time.Time) error  { return nil }
-func (hc *httpConn) SetWriteDeadline(time.Time) error { return nil }
-func (hc *httpConn) SetDeadline(time.Time) error      { return nil }
-func (hc *httpConn) Write([]byte) (int, error)        { panic("Write called") }
-
-func (hc *httpConn) Read(b []byte) (int, error) {
-	<-hc.closed
-	return 0, io.EOF
 }
 
-func (hc *httpConn) Close() error {
+func (hc *httpConn) RemoteAddr() string {
+	return hc.req.URL.String()
+}
+
+func (hc *httpConn) Read() ([]*jsonrpcMessage, bool, error) {
+	<-hc.closed
+	return nil, false, io.EOF
+}
+
+func (hc *httpConn) Close() {
 	hc.closeOnce.Do(func() { close(hc.closed) })
-	return nil
+}
+
+func (hc *httpConn) Closed() <-chan interface{} {
+	return hc.closed
 }
 
 // HTTPTimeouts represents the configuration params for the HTTP RPC server.
@@ -114,8 +115,8 @@ func DialHTTPWithClient(endpoint string, client *http.Client) (*Client, error) {
 	req.Header.Set("Accept", contentType)
 
 	initctx := context.Background()
-	return newClient(initctx, func(context.Context) (net.Conn, error) {
+	return newClient(initctx, func(context.Context) (ServerCodec, error) {
-		return &httpConn{client: client, req: req, closed: make(chan struct{})}, nil
+		return &httpConn{client: client, req: req, closed: make(chan interface{})}, nil
 	})
 }
 
@@ -184,17 +185,30 @@ func (hc *httpConn) doRequest(ctx context.Context, msg interface{}) (io.ReadClos
 	return resp.Body, nil
 }
 
-// httpReadWriteNopCloser wraps a io.Reader and io.Writer with a NOP Close method.
+// httpServerConn turns a HTTP connection into a Conn.
-type httpReadWriteNopCloser struct {
+type httpServerConn struct {
 	io.Reader
 	io.Writer
+	r *http.Request
 }
 
-// Close does nothing and returns always nil
+func newHTTPServerConn(r *http.Request, w http.ResponseWriter) ServerCodec {
-func (t *httpReadWriteNopCloser) Close() error {
+	body := io.LimitReader(r.Body, maxRequestContentLength)
-	return nil
+	conn := &httpServerConn{Reader: body, Writer: w, r: r}
+	return NewJSONCodec(conn)
 }
 
+// Close does nothing and always returns nil.
+func (t *httpServerConn) Close() error { return nil }
+
+// RemoteAddr returns the peer address of the underlying connection.
+func (t *httpServerConn) RemoteAddr() string {
+	return t.r.RemoteAddr
+}
+
+// SetWriteDeadline does nothing and always returns nil.
+func (t *httpServerConn) SetWriteDeadline(time.Time) error { return nil }
+
 // NewHTTPServer creates a new HTTP RPC server around an API provider.
 //
 // Deprecated: Server implements http.Handler
@@ -226,7 +240,7 @@ func NewHTTPServer(cors []string, vhosts []string, timeouts HTTPTimeouts, srv ht
 }
 
 // ServeHTTP serves JSON-RPC requests over HTTP.
-func (srv *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	// Permit dumb empty requests for remote health-checks (AWS)
 	if r.Method == http.MethodGet && r.ContentLength == 0 && r.URL.RawQuery == "" {
 		return
@@ -249,12 +263,10 @@ func (srv *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		ctx = context.WithValue(ctx, "Origin", origin)
 	}
 
-	body := io.LimitReader(r.Body, maxRequestContentLength)
-	codec := NewJSONCodec(&httpReadWriteNopCloser{body, w})
-	defer codec.Close()
-
 	w.Header().Set("content-type", contentType)
-	srv.ServeSingleRequest(ctx, codec, OptionMethodInvocation)
+	codec := newHTTPServerConn(r, w)
+	defer codec.Close()
+	s.serveSingleRequest(ctx, codec)
 }
 
 // validateRequest returns a non-zero response code and error message if the

rpc/inproc.go

@@ -24,10 +24,10 @@ import (
 // DialInProc attaches an in-process connection to the given RPC server.
 func DialInProc(handler *Server) *Client {
 	initctx := context.Background()
-	c, _ := newClient(initctx, func(context.Context) (net.Conn, error) {
+	c, _ := newClient(initctx, func(context.Context) (ServerCodec, error) {
 		p1, p2 := net.Pipe()
 		go handler.ServeCodec(NewJSONCodec(p1), OptionMethodInvocation|OptionSubscriptions)
-		return p2, nil
+		return NewJSONCodec(p2), nil
 	})
 	return c
 }

rpc/ipc.go

@@ -25,17 +25,17 @@ import (
 )
 
 // ServeListener accepts connections on l, serving JSON-RPC on them.
-func (srv *Server) ServeListener(l net.Listener) error {
+func (s *Server) ServeListener(l net.Listener) error {
 	for {
 		conn, err := l.Accept()
 		if netutil.IsTemporaryError(err) {
-			log.Warn("IPC accept error", "err", err)
+			log.Warn("RPC accept error", "err", err)
 			continue
 		} else if err != nil {
 			return err
 		}
-		log.Trace("IPC accepted connection")
+		log.Trace("Accepted RPC connection", "conn", conn.RemoteAddr())
-		go srv.ServeCodec(NewJSONCodec(conn), OptionMethodInvocation|OptionSubscriptions)
+		go s.ServeCodec(NewJSONCodec(conn), OptionMethodInvocation|OptionSubscriptions)
 	}
 }
 
@@ -46,7 +46,11 @@ func (srv *Server) ServeListener(l net.Listener) error {
 // The context is used for the initial connection establishment. It does not
 // affect subsequent interactions with the client.
 func DialIPC(ctx context.Context, endpoint string) (*Client, error) {
-	return newClient(ctx, func(ctx context.Context) (net.Conn, error) {
+	return newClient(ctx, func(ctx context.Context) (ServerCodec, error) {
-		return newIPCConnection(ctx, endpoint)
+		conn, err := newIPCConnection(ctx, endpoint)
+		if err != nil {
+			return nil, err
+		}
+		return NewJSONCodec(conn), err
 	})
 }

rpc/json.go

@@ -18,36 +18,104 @@ package rpc
 
 import (
 	"bytes"
+	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"reflect"
-	"strconv"
 	"strings"
 	"sync"
-
+	"time"
-	"github.com/ethereum/go-ethereum/log"
 )
 
 const (
-	jsonrpcVersion           = "2.0"
+	vsn                      = "2.0"
 	serviceMethodSeparator   = "_"
 	subscribeMethodSuffix    = "_subscribe"
 	unsubscribeMethodSuffix  = "_unsubscribe"
 	notificationMethodSuffix = "_subscription"
+
+	defaultWriteTimeout = 10 * time.Second // used if context has no deadline
 )
 
-type jsonRequest struct {
+var null = json.RawMessage("null")
-	Method  string          `json:"method"`
+
-	Version string          `json:"jsonrpc"`
+type subscriptionResult struct {
-	Id      json.RawMessage `json:"id,omitempty"`
+	ID     string          `json:"subscription"`
-	Payload json.RawMessage `json:"params,omitempty"`
+	Result json.RawMessage `json:"result,omitempty"`
 }
 
-type jsonSuccessResponse struct {
+// A value of this type can a JSON-RPC request, notification, successful response or
-	Version string      `json:"jsonrpc"`
+// error response. Which one it is depends on the fields.
-	Id      interface{} `json:"id,omitempty"`
+type jsonrpcMessage struct {
-	Result  interface{} `json:"result"`
+	Version string          `json:"jsonrpc,omitempty"`
+	ID      json.RawMessage `json:"id,omitempty"`
+	Method  string          `json:"method,omitempty"`
+	Params  json.RawMessage `json:"params,omitempty"`
+	Error   *jsonError      `json:"error,omitempty"`
+	Result  json.RawMessage `json:"result,omitempty"`
+}
+
+func (msg *jsonrpcMessage) isNotification() bool {
+	return msg.ID == nil && msg.Method != ""
+}
+
+func (msg *jsonrpcMessage) isCall() bool {
+	return msg.hasValidID() && msg.Method != ""
+}
+
+func (msg *jsonrpcMessage) isResponse() bool {
+	return msg.hasValidID() && msg.Method == "" && msg.Params == nil && (msg.Result != nil || msg.Error != nil)
+}
+
+func (msg *jsonrpcMessage) hasValidID() bool {
+	return len(msg.ID) > 0 && msg.ID[0] != '{' && msg.ID[0] != '['
+}
+
+func (msg *jsonrpcMessage) isSubscribe() bool {
+	return strings.HasSuffix(msg.Method, subscribeMethodSuffix)
+}
+
+func (msg *jsonrpcMessage) isUnsubscribe() bool {
+	return strings.HasSuffix(msg.Method, unsubscribeMethodSuffix)
+}
+
+func (msg *jsonrpcMessage) namespace() string {
+	elem := strings.SplitN(msg.Method, serviceMethodSeparator, 2)
+	return elem[0]
+}
+
+func (msg *jsonrpcMessage) String() string {
+	b, _ := json.Marshal(msg)
+	return string(b)
+}
+
+func (msg *jsonrpcMessage) errorResponse(err error) *jsonrpcMessage {
+	resp := errorMessage(err)
+	resp.ID = msg.ID
+	return resp
+}
+
+func (msg *jsonrpcMessage) response(result interface{}) *jsonrpcMessage {
+	enc, err := json.Marshal(result)
+	if err != nil {
+		// TODO: wrap with 'internal server error'
+		return msg.errorResponse(err)
+	}
+	return &jsonrpcMessage{Version: vsn, ID: msg.ID, Result: enc}
+}
+
+func errorMessage(err error) *jsonrpcMessage {
+	msg := &jsonrpcMessage{Version: vsn, ID: null, Error: &jsonError{
+		Code:    defaultErrorCode,
+		Message: err.Error(),
+	}}
+	ec, ok := err.(Error)
+	if ok {
+		msg.Error.Code = ec.ErrorCode()
+	}
+	return msg
 }
 
 type jsonError struct {
@@ -56,35 +124,6 @@ type jsonError struct {
 	Data    interface{} `json:"data,omitempty"`
 }
 
-type jsonErrResponse struct {
-	Version string      `json:"jsonrpc"`
-	Id      interface{} `json:"id,omitempty"`
-	Error   jsonError   `json:"error"`
-}
-
-type jsonSubscription struct {
-	Subscription string      `json:"subscription"`
-	Result       interface{} `json:"result,omitempty"`
-}
-
-type jsonNotification struct {
-	Version string           `json:"jsonrpc"`
-	Method  string           `json:"method"`
-	Params  jsonSubscription `json:"params"`
-}
-
-// jsonCodec reads and writes JSON-RPC messages to the underlying connection. It
-// also has support for parsing arguments and serializing (result) objects.
-type jsonCodec struct {
-	closer sync.Once                 // close closed channel once
-	closed chan interface{}          // closed on Close
-	decMu  sync.Mutex                // guards the decoder
-	decode func(v interface{}) error // decoder to allow multiple transports
-	encMu  sync.Mutex                // guards the encoder
-	encode func(v interface{}) error // encoder to allow multiple transports
-	rw     io.ReadWriteCloser        // connection
-}
-
 func (err *jsonError) Error() string {
 	if err.Message == "" {
 		return fmt.Sprintf("json-rpc error %d", err.Code)
@@ -96,34 +135,126 @@ func (err *jsonError) ErrorCode() int {
 	return err.Code
 }
 
+// Conn is a subset of the methods of net.Conn which are sufficient for ServerCodec.
+type Conn interface {
+	io.ReadWriteCloser
+	SetWriteDeadline(time.Time) error
+}
+
+// ConnRemoteAddr wraps the RemoteAddr operation, which returns a description
+// of the peer address of a connection. If a Conn also implements ConnRemoteAddr, this
+// description is used in log messages.
+type ConnRemoteAddr interface {
+	RemoteAddr() string
+}
+
+// connWithRemoteAddr overrides the remote address of a connection.
+type connWithRemoteAddr struct {
+	Conn
+	addr string
+}
+
+func (c connWithRemoteAddr) RemoteAddr() string { return c.addr }
+
+// jsonCodec reads and writes JSON-RPC messages to the underlying connection. It also has
+// support for parsing arguments and serializing (result) objects.
+type jsonCodec struct {
+	remoteAddr string
+	closer     sync.Once                 // close closed channel once
+	closed     chan interface{}          // closed on Close
+	decode     func(v interface{}) error // decoder to allow multiple transports
+	encMu      sync.Mutex                // guards the encoder
+	encode     func(v interface{}) error // encoder to allow multiple transports
+	conn       Conn
+}
+
 // NewCodec creates a new RPC server codec with support for JSON-RPC 2.0 based
 // on explicitly given encoding and decoding methods.
-func NewCodec(rwc io.ReadWriteCloser, encode, decode func(v interface{}) error) ServerCodec {
+func NewCodec(conn Conn, encode, decode func(v interface{}) error) ServerCodec {
-	return &jsonCodec{
+	codec := &jsonCodec{
 		closed: make(chan interface{}),
 		encode: encode,
 		decode: decode,
-		rw:     rwc,
+		conn:   conn,
 	}
+	if ra, ok := conn.(ConnRemoteAddr); ok {
+		codec.remoteAddr = ra.RemoteAddr()
+	}
+	return codec
 }
 
 // NewJSONCodec creates a new RPC server codec with support for JSON-RPC 2.0.
-func NewJSONCodec(rwc io.ReadWriteCloser) ServerCodec {
+func NewJSONCodec(conn Conn) ServerCodec {
-	enc := json.NewEncoder(rwc)
+	enc := json.NewEncoder(conn)
-	dec := json.NewDecoder(rwc)
+	dec := json.NewDecoder(conn)
 	dec.UseNumber()
-
+	return NewCodec(conn, enc.Encode, dec.Decode)
-	return &jsonCodec{
-		closed: make(chan interface{}),
-		encode: enc.Encode,
-		decode: dec.Decode,
-		rw:     rwc,
 }
+
+func (c *jsonCodec) RemoteAddr() string {
+	return c.remoteAddr
+}
+
+func (c *jsonCodec) Read() (msg []*jsonrpcMessage, batch bool, err error) {
+	// Decode the next JSON object in the input stream.
+	// This verifies basic syntax, etc.
+	var rawmsg json.RawMessage
+	if err := c.decode(&rawmsg); err != nil {
+		return nil, false, err
+	}
+	msg, batch = parseMessage(rawmsg)
+	return msg, batch, nil
+}
+
+// Write sends a message to client.
+func (c *jsonCodec) Write(ctx context.Context, v interface{}) error {
+	c.encMu.Lock()
+	defer c.encMu.Unlock()
+
+	deadline, ok := ctx.Deadline()
+	if !ok {
+		deadline = time.Now().Add(defaultWriteTimeout)
+	}
+	c.conn.SetWriteDeadline(deadline)
+	return c.encode(v)
+}
+
+// Close the underlying connection
+func (c *jsonCodec) Close() {
+	c.closer.Do(func() {
+		close(c.closed)
+		c.conn.Close()
+	})
+}
+
+// Closed returns a channel which will be closed when Close is called
+func (c *jsonCodec) Closed() <-chan interface{} {
+	return c.closed
+}
+
+// parseMessage parses raw bytes as a (batch of) JSON-RPC message(s). There are no error
+// checks in this function because the raw message has already been syntax-checked when it
+// is called. Any non-JSON-RPC messages in the input return the zero value of
+// jsonrpcMessage.
+func parseMessage(raw json.RawMessage) ([]*jsonrpcMessage, bool) {
+	if !isBatch(raw) {
+		msgs := []*jsonrpcMessage{{}}
+		json.Unmarshal(raw, &msgs[0])
+		return msgs, false
+	}
+	dec := json.NewDecoder(bytes.NewReader(raw))
+	dec.Token() // skip '['
+	var msgs []*jsonrpcMessage
+	for dec.More() {
+		msgs = append(msgs, new(jsonrpcMessage))
+		dec.Decode(&msgs[len(msgs)-1])
+	}
+	return msgs, true
 }
 
 // isBatch returns true when the first non-whitespace characters is '['
-func isBatch(msg json.RawMessage) bool {
+func isBatch(raw json.RawMessage) bool {
-	for _, c := range msg {
+	for _, c := range raw {
 		// skip insignificant whitespace (http://www.ietf.org/rfc/rfc4627.txt)
 		if c == 0x20 || c == 0x09 || c == 0x0a || c == 0x0d {
 			continue
@@ -133,231 +264,67 @@ func isBatch(msg json.RawMessage) bool {
 	return false
 }
 
-// ReadRequestHeaders will read new requests without parsing the arguments. It will
-// return a collection of requests, an indication if these requests are in batch
-// form or an error when the incoming message could not be read/parsed.
-func (c *jsonCodec) ReadRequestHeaders() ([]rpcRequest, bool, Error) {
-	c.decMu.Lock()
-	defer c.decMu.Unlock()
-
-	var incomingMsg json.RawMessage
-	if err := c.decode(&incomingMsg); err != nil {
-		return nil, false, &invalidRequestError{err.Error()}
-	}
-	if isBatch(incomingMsg) {
-		return parseBatchRequest(incomingMsg)
-	}
-	return parseRequest(incomingMsg)
-}
-
-// checkReqId returns an error when the given reqId isn't valid for RPC method calls.
-// valid id's are strings, numbers or null
-func checkReqId(reqId json.RawMessage) error {
-	if len(reqId) == 0 {
-		return fmt.Errorf("missing request id")
-	}
-	if _, err := strconv.ParseFloat(string(reqId), 64); err == nil {
-		return nil
-	}
-	var str string
-	if err := json.Unmarshal(reqId, &str); err == nil {
-		return nil
-	}
-	return fmt.Errorf("invalid request id")
-}
-
-// parseRequest will parse a single request from the given RawMessage. It will return
-// the parsed request, an indication if the request was a batch or an error when
-// the request could not be parsed.
-func parseRequest(incomingMsg json.RawMessage) ([]rpcRequest, bool, Error) {
-	var in jsonRequest
-	if err := json.Unmarshal(incomingMsg, &in); err != nil {
-		return nil, false, &invalidMessageError{err.Error()}
-	}
-
-	if err := checkReqId(in.Id); err != nil {
-		return nil, false, &invalidMessageError{err.Error()}
-	}
-
-	// subscribe are special, they will always use `subscribeMethod` as first param in the payload
-	if strings.HasSuffix(in.Method, subscribeMethodSuffix) {
-		reqs := []rpcRequest{{id: &in.Id, isPubSub: true}}
-		if len(in.Payload) > 0 {
-			// first param must be subscription name
-			var subscribeMethod [1]string
-			if err := json.Unmarshal(in.Payload, &subscribeMethod); err != nil {
-				log.Debug(fmt.Sprintf("Unable to parse subscription method: %v\n", err))
-				return nil, false, &invalidRequestError{"Unable to parse subscription request"}
-			}
-
-			reqs[0].service, reqs[0].method = strings.TrimSuffix(in.Method, subscribeMethodSuffix), subscribeMethod[0]
-			reqs[0].params = in.Payload
-			return reqs, false, nil
-		}
-		return nil, false, &invalidRequestError{"Unable to parse subscription request"}
-	}
-
-	if strings.HasSuffix(in.Method, unsubscribeMethodSuffix) {
-		return []rpcRequest{{id: &in.Id, isPubSub: true,
-			method: in.Method, params: in.Payload}}, false, nil
-	}
-
-	elems := strings.Split(in.Method, serviceMethodSeparator)
-	if len(elems) != 2 {
-		return nil, false, &methodNotFoundError{in.Method, ""}
-	}
-
-	// regular RPC call
-	if len(in.Payload) == 0 {
-		return []rpcRequest{{service: elems[0], method: elems[1], id: &in.Id}}, false, nil
-	}
-
-	return []rpcRequest{{service: elems[0], method: elems[1], id: &in.Id, params: in.Payload}}, false, nil
-}
-
-// parseBatchRequest will parse a batch request into a collection of requests from the given RawMessage, an indication
-// if the request was a batch or an error when the request could not be read.
-func parseBatchRequest(incomingMsg json.RawMessage) ([]rpcRequest, bool, Error) {
-	var in []jsonRequest
-	if err := json.Unmarshal(incomingMsg, &in); err != nil {
-		return nil, false, &invalidMessageError{err.Error()}
-	}
-
-	requests := make([]rpcRequest, len(in))
-	for i, r := range in {
-		if err := checkReqId(r.Id); err != nil {
-			return nil, false, &invalidMessageError{err.Error()}
-		}
-
-		id := &in[i].Id
-
-		// subscribe are special, they will always use `subscriptionMethod` as first param in the payload
-		if strings.HasSuffix(r.Method, subscribeMethodSuffix) {
-			requests[i] = rpcRequest{id: id, isPubSub: true}
-			if len(r.Payload) > 0 {
-				// first param must be subscription name
-				var subscribeMethod [1]string
-				if err := json.Unmarshal(r.Payload, &subscribeMethod); err != nil {
-					log.Debug(fmt.Sprintf("Unable to parse subscription method: %v\n", err))
-					return nil, false, &invalidRequestError{"Unable to parse subscription request"}
-				}
-
-				requests[i].service, requests[i].method = strings.TrimSuffix(r.Method, subscribeMethodSuffix), subscribeMethod[0]
-				requests[i].params = r.Payload
-				continue
-			}
-
-			return nil, true, &invalidRequestError{"Unable to parse (un)subscribe request arguments"}
-		}
-
-		if strings.HasSuffix(r.Method, unsubscribeMethodSuffix) {
-			requests[i] = rpcRequest{id: id, isPubSub: true, method: r.Method, params: r.Payload}
-			continue
-		}
-
-		if len(r.Payload) == 0 {
-			requests[i] = rpcRequest{id: id, params: nil}
-		} else {
-			requests[i] = rpcRequest{id: id, params: r.Payload}
-		}
-		if elem := strings.Split(r.Method, serviceMethodSeparator); len(elem) == 2 {
-			requests[i].service, requests[i].method = elem[0], elem[1]
-		} else {
-			requests[i].err = &methodNotFoundError{r.Method, ""}
-		}
-	}
-
-	return requests, true, nil
-}
-
-// ParseRequestArguments tries to parse the given params (json.RawMessage) with the given
-// types. It returns the parsed values or an error when the parsing failed.
-func (c *jsonCodec) ParseRequestArguments(argTypes []reflect.Type, params interface{}) ([]reflect.Value, Error) {
-	if args, ok := params.(json.RawMessage); !ok {
-		return nil, &invalidParamsError{"Invalid params supplied"}
-	} else {
-		return parsePositionalArguments(args, argTypes)
-	}
-}
-
 // parsePositionalArguments tries to parse the given args to an array of values with the
 // given types. It returns the parsed values or an error when the args could not be
 // parsed. Missing optional arguments are returned as reflect.Zero values.
-func parsePositionalArguments(rawArgs json.RawMessage, types []reflect.Type) ([]reflect.Value, Error) {
+func parsePositionalArguments(rawArgs json.RawMessage, types []reflect.Type) ([]reflect.Value, error) {
-	// Read beginning of the args array.
 	dec := json.NewDecoder(bytes.NewReader(rawArgs))
-	if tok, _ := dec.Token(); tok != json.Delim('[') {
+	var args []reflect.Value
-		return nil, &invalidParamsError{"non-array args"}
+	tok, err := dec.Token()
+	switch {
+	case err == io.EOF || tok == nil && err == nil:
+		// "params" is optional and may be empty. Also allow "params":null even though it's
+		// not in the spec because our own client used to send it.
+	case err != nil:
+		return nil, err
+	case tok == json.Delim('['):
+		// Read argument array.
+		if args, err = parseArgumentArray(dec, types); err != nil {
+			return nil, err
 		}
-	// Read args.
+	default:
-	args := make([]reflect.Value, 0, len(types))
+		return nil, errors.New("non-array args")
-	for i := 0; dec.More(); i++ {
-		if i >= len(types) {
-			return nil, &invalidParamsError{fmt.Sprintf("too many arguments, want at most %d", len(types))}
-		}
-		argval := reflect.New(types[i])
-		if err := dec.Decode(argval.Interface()); err != nil {
-			return nil, &invalidParamsError{fmt.Sprintf("invalid argument %d: %v", i, err)}
-		}
-		if argval.IsNil() && types[i].Kind() != reflect.Ptr {
-			return nil, &invalidParamsError{fmt.Sprintf("missing value for required argument %d", i)}
-		}
-		args = append(args, argval.Elem())
-	}
-	// Read end of args array.
-	if _, err := dec.Token(); err != nil {
-		return nil, &invalidParamsError{err.Error()}
 	}
 	// Set any missing args to nil.
 	for i := len(args); i < len(types); i++ {
 		if types[i].Kind() != reflect.Ptr {
-			return nil, &invalidParamsError{fmt.Sprintf("missing value for required argument %d", i)}
+			return nil, fmt.Errorf("missing value for required argument %d", i)
 		}
 		args = append(args, reflect.Zero(types[i]))
 	}
 	return args, nil
 }
 
-// CreateResponse will create a JSON-RPC success response with the given id and reply as result.
+func parseArgumentArray(dec *json.Decoder, types []reflect.Type) ([]reflect.Value, error) {
-func (c *jsonCodec) CreateResponse(id interface{}, reply interface{}) interface{} {
+	args := make([]reflect.Value, 0, len(types))
-	return &jsonSuccessResponse{Version: jsonrpcVersion, Id: id, Result: reply}
+	for i := 0; dec.More(); i++ {
+		if i >= len(types) {
+			return args, fmt.Errorf("too many arguments, want at most %d", len(types))
+		}
+		argval := reflect.New(types[i])
+		if err := dec.Decode(argval.Interface()); err != nil {
+			return args, fmt.Errorf("invalid argument %d: %v", i, err)
+		}
+		if argval.IsNil() && types[i].Kind() != reflect.Ptr {
+			return args, fmt.Errorf("missing value for required argument %d", i)
+		}
+		args = append(args, argval.Elem())
+	}
+	// Read end of args array.
+	_, err := dec.Token()
+	return args, err
 }
 
-// CreateErrorResponse will create a JSON-RPC error response with the given id and error.
+// parseSubscriptionName extracts the subscription name from an encoded argument array.
-func (c *jsonCodec) CreateErrorResponse(id interface{}, err Error) interface{} {
+func parseSubscriptionName(rawArgs json.RawMessage) (string, error) {
-	return &jsonErrResponse{Version: jsonrpcVersion, Id: id, Error: jsonError{Code: err.ErrorCode(), Message: err.Error()}}
+	dec := json.NewDecoder(bytes.NewReader(rawArgs))
+	if tok, _ := dec.Token(); tok != json.Delim('[') {
+		return "", errors.New("non-array args")
 	}
-
+	v, _ := dec.Token()
-// CreateErrorResponseWithInfo will create a JSON-RPC error response with the given id and error.
+	method, ok := v.(string)
-// info is optional and contains additional information about the error. When an empty string is passed it is ignored.
+	if !ok {
-func (c *jsonCodec) CreateErrorResponseWithInfo(id interface{}, err Error, info interface{}) interface{} {
+		return "", errors.New("expected subscription name as first argument")
-	return &jsonErrResponse{Version: jsonrpcVersion, Id: id,
-		Error: jsonError{Code: err.ErrorCode(), Message: err.Error(), Data: info}}
 	}
-
+	return method, nil
-// CreateNotification will create a JSON-RPC notification with the given subscription id and event as params.
-func (c *jsonCodec) CreateNotification(subid, namespace string, event interface{}) interface{} {
-	return &jsonNotification{Version: jsonrpcVersion, Method: namespace + notificationMethodSuffix,
-		Params: jsonSubscription{Subscription: subid, Result: event}}
-}
-
-// Write message to client
-func (c *jsonCodec) Write(res interface{}) error {
-	c.encMu.Lock()
-	defer c.encMu.Unlock()
-
-	return c.encode(res)
-}
-
-// Close the underlying connection
-func (c *jsonCodec) Close() {
-	c.closer.Do(func() {
-		close(c.closed)
-		c.rw.Close()
-	})
-}
-
-// Closed returns a channel which will be closed when Close is called
-func (c *jsonCodec) Closed() <-chan interface{} {
-	return c.closed
 }

rpc/json_test.go

@@ -1,178 +0,0 @@
-// Copyright 2015 The go-ethereum Authors
-// This file is part of the go-ethereum library.
-//
-// The go-ethereum library is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Lesser General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// The go-ethereum library is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-// GNU Lesser General Public License for more details.
-//
-// You should have received a copy of the GNU Lesser General Public License
-// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
-
-package rpc
-
-import (
-	"bufio"
-	"bytes"
-	"encoding/json"
-	"reflect"
-	"strconv"
-	"testing"
-)
-
-type RWC struct {
-	*bufio.ReadWriter
-}
-
-func (rwc *RWC) Close() error {
-	return nil
-}
-
-func TestJSONRequestParsing(t *testing.T) {
-	server := NewServer()
-	service := new(Service)
-
-	if err := server.RegisterName("calc", service); err != nil {
-		t.Fatalf("%v", err)
-	}
-
-	req := bytes.NewBufferString(`{"id": 1234, "jsonrpc": "2.0", "method": "calc_add", "params": [11, 22]}`)
-	var str string
-	reply := bytes.NewBufferString(str)
-	rw := &RWC{bufio.NewReadWriter(bufio.NewReader(req), bufio.NewWriter(reply))}
-
-	codec := NewJSONCodec(rw)
-
-	requests, batch, err := codec.ReadRequestHeaders()
-	if err != nil {
-		t.Fatalf("%v", err)
-	}
-
-	if batch {
-		t.Fatalf("Request isn't a batch")
-	}
-
-	if len(requests) != 1 {
-		t.Fatalf("Expected 1 request but got %d requests - %v", len(requests), requests)
-	}
-
-	if requests[0].service != "calc" {
-		t.Fatalf("Expected service 'calc' but got '%s'", requests[0].service)
-	}
-
-	if requests[0].method != "add" {
-		t.Fatalf("Expected method 'Add' but got '%s'", requests[0].method)
-	}
-
-	if rawId, ok := requests[0].id.(*json.RawMessage); ok {
-		id, e := strconv.ParseInt(string(*rawId), 0, 64)
-		if e != nil {
-			t.Fatalf("%v", e)
-		}
-		if id != 1234 {
-			t.Fatalf("Expected id 1234 but got %d", id)
-		}
-	} else {
-		t.Fatalf("invalid request, expected *json.RawMesage got %T", requests[0].id)
-	}
-
-	var arg int
-	args := []reflect.Type{reflect.TypeOf(arg), reflect.TypeOf(arg)}
-
-	v, err := codec.ParseRequestArguments(args, requests[0].params)
-	if err != nil {
-		t.Fatalf("%v", err)
-	}
-
-	if len(v) != 2 {
-		t.Fatalf("Expected 2 argument values, got %d", len(v))
-	}
-
-	if v[0].Int() != 11 || v[1].Int() != 22 {
-		t.Fatalf("expected %d == 11 && %d == 22", v[0].Int(), v[1].Int())
-	}
-}
-
-func TestJSONRequestParamsParsing(t *testing.T) {
-
-	var (
-		stringT = reflect.TypeOf("")
-		intT    = reflect.TypeOf(0)
-		intPtrT = reflect.TypeOf(new(int))
-
-		stringV = reflect.ValueOf("abc")
-		i       = 1
-		intV    = reflect.ValueOf(i)
-		intPtrV = reflect.ValueOf(&i)
-	)
-
-	var validTests = []struct {
-		input    string
-		argTypes []reflect.Type
-		expected []reflect.Value
-	}{
-		{`[]`, []reflect.Type{}, []reflect.Value{}},
-		{`[]`, []reflect.Type{intPtrT}, []reflect.Value{intPtrV}},
-		{`[1]`, []reflect.Type{intT}, []reflect.Value{intV}},
-		{`[1,"abc"]`, []reflect.Type{intT, stringT}, []reflect.Value{intV, stringV}},
-		{`[null]`, []reflect.Type{intPtrT}, []reflect.Value{intPtrV}},
-		{`[null,"abc"]`, []reflect.Type{intPtrT, stringT, intPtrT}, []reflect.Value{intPtrV, stringV, intPtrV}},
-		{`[null,"abc",null]`, []reflect.Type{intPtrT, stringT, intPtrT}, []reflect.Value{intPtrV, stringV, intPtrV}},
-	}
-
-	codec := jsonCodec{}
-
-	for _, test := range validTests {
-		params := (json.RawMessage)([]byte(test.input))
-		args, err := codec.ParseRequestArguments(test.argTypes, params)
-
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		var match []interface{}
-		json.Unmarshal([]byte(test.input), &match)
-
-		if len(args) != len(test.argTypes) {
-			t.Fatalf("expected %d parsed args, got %d", len(test.argTypes), len(args))
-		}
-
-		for i, arg := range args {
-			expected := test.expected[i]
-
-			if arg.Kind() != expected.Kind() {
-				t.Errorf("expected type for param %d in %s", i, test.input)
-			}
-
-			if arg.Kind() == reflect.Int && arg.Int() != expected.Int() {
-				t.Errorf("expected int(%d), got int(%d) in %s", expected.Int(), arg.Int(), test.input)
-			}
-
-			if arg.Kind() == reflect.String && arg.String() != expected.String() {
-				t.Errorf("expected string(%s), got string(%s) in %s", expected.String(), arg.String(), test.input)
-			}
-		}
-	}
-
-	var invalidTests = []struct {
-		input    string
-		argTypes []reflect.Type
-	}{
-		{`[]`, []reflect.Type{intT}},
-		{`[null]`, []reflect.Type{intT}},
-		{`[1]`, []reflect.Type{stringT}},
-		{`[1,2]`, []reflect.Type{stringT}},
-		{`["abc", null]`, []reflect.Type{stringT, intT}},
-	}
-
-	for i, test := range invalidTests {
-		if _, err := codec.ParseRequestArguments(test.argTypes, test.input); err == nil {
-			t.Errorf("expected test %d - %s to fail", i, test.input)
-		}
-	}
-}

rpc/server.go

@@ -18,11 +18,7 @@ package rpc
 
 import (
 	"context"
-	"fmt"
+	"io"
-	"reflect"
-	"runtime"
-	"strings"
-	"sync"
 	"sync/atomic"
 
 	mapset "github.com/deckarep/golang-set"
@@ -31,7 +27,9 @@ import (
 
 const MetadataApi = "rpc"
 
-// CodecOption specifies which type of messages this codec supports
+// CodecOption specifies which type of messages a codec supports.
+//
+// Deprecated: this option is no longer honored by Server.
 type CodecOption int
 
 const (
@@ -42,22 +40,94 @@ const (
 	OptionSubscriptions = 1 << iota // support pub sub
 )
 
-// NewServer will create a new server instance with no registered handlers.
+// Server is an RPC server.
-func NewServer() *Server {
+type Server struct {
-	server := &Server{
+	services serviceRegistry
-		services: make(serviceRegistry),
+	idgen    func() ID
-		codecs:   mapset.NewSet(),
+	run      int32
-		run:      1,
+	codecs   mapset.Set
 }
 
-	// register a default service which will provide meta information about the RPC service such as the services and
+// NewServer creates a new server instance with no registered handlers.
-	// methods it offers.
+func NewServer() *Server {
+	server := &Server{idgen: randomIDGenerator(), codecs: mapset.NewSet(), run: 1}
+	// Register the default service providing meta information about the RPC service such
+	// as the services and methods it offers.
 	rpcService := &RPCService{server}
 	server.RegisterName(MetadataApi, rpcService)
-
 	return server
 }
 
+// RegisterName creates a service for the given receiver type under the given name. When no
+// methods on the given receiver match the criteria to be either a RPC method or a
+// subscription an error is returned. Otherwise a new service is created and added to the
+// service collection this server provides to clients.
+func (s *Server) RegisterName(name string, receiver interface{}) error {
+	return s.services.registerName(name, receiver)
+}
+
+// ServeCodec reads incoming requests from codec, calls the appropriate callback and writes
+// the response back using the given codec. It will block until the codec is closed or the
+// server is stopped. In either case the codec is closed.
+//
+// Note that codec options are no longer supported.
+func (s *Server) ServeCodec(codec ServerCodec, options CodecOption) {
+	defer codec.Close()
+
+	// Don't serve if server is stopped.
+	if atomic.LoadInt32(&s.run) == 0 {
+		return
+	}
+
+	// Add the codec to the set so it can be closed by Stop.
+	s.codecs.Add(codec)
+	defer s.codecs.Remove(codec)
+
+	c := initClient(codec, s.idgen, &s.services)
+	<-codec.Closed()
+	c.Close()
+}
+
+// serveSingleRequest reads and processes a single RPC request from the given codec. This
+// is used to serve HTTP connections. Subscriptions and reverse calls are not allowed in
+// this mode.
+func (s *Server) serveSingleRequest(ctx context.Context, codec ServerCodec) {
+	// Don't serve if server is stopped.
+	if atomic.LoadInt32(&s.run) == 0 {
+		return
+	}
+
+	h := newHandler(ctx, codec, s.idgen, &s.services)
+	h.allowSubscribe = false
+	defer h.close(io.EOF, nil)
+
+	reqs, batch, err := codec.Read()
+	if err != nil {
+		if err != io.EOF {
+			codec.Write(ctx, errorMessage(&invalidMessageError{"parse error"}))
+		}
+		return
+	}
+	if batch {
+		h.handleBatch(reqs)
+	} else {
+		h.handleMsg(reqs[0])
+	}
+}
+
+// Stop stops reading new requests, waits for stopPendingRequestTimeout to allow pending
+// requests to finish, then closes all codecs which will cancel pending requests and
+// subscriptions.
+func (s *Server) Stop() {
+	if atomic.CompareAndSwapInt32(&s.run, 1, 0) {
+		log.Debug("RPC server shutting down")
+		s.codecs.Each(func(c interface{}) bool {
+			c.(ServerCodec).Close()
+			return true
+		})
+	}
+}
+
 // RPCService gives meta information about the server.
 // e.g. gives information about the loaded modules.
 type RPCService struct {
@@ -66,377 +136,12 @@ type RPCService struct {
 
 // Modules returns the list of RPC services with their version number
 func (s *RPCService) Modules() map[string]string {
+	s.server.services.mu.Lock()
+	defer s.server.services.mu.Unlock()
+
 	modules := make(map[string]string)
-	for name := range s.server.services {
+	for name := range s.server.services.services {
 		modules[name] = "1.0"
 	}
 	return modules
 }
-
-// RegisterName will create a service for the given rcvr type under the given name. When no methods on the given rcvr
-// match the criteria to be either a RPC method or a subscription an error is returned. Otherwise a new service is
-// created and added to the service collection this server instance serves.
-func (s *Server) RegisterName(name string, rcvr interface{}) error {
-	if s.services == nil {
-		s.services = make(serviceRegistry)
-	}
-
-	svc := new(service)
-	svc.typ = reflect.TypeOf(rcvr)
-	rcvrVal := reflect.ValueOf(rcvr)
-
-	if name == "" {
-		return fmt.Errorf("no service name for type %s", svc.typ.String())
-	}
-	if !isExported(reflect.Indirect(rcvrVal).Type().Name()) {
-		return fmt.Errorf("%s is not exported", reflect.Indirect(rcvrVal).Type().Name())
-	}
-
-	methods, subscriptions := suitableCallbacks(rcvrVal, svc.typ)
-
-	if len(methods) == 0 && len(subscriptions) == 0 {
-		return fmt.Errorf("Service %T doesn't have any suitable methods/subscriptions to expose", rcvr)
-	}
-
-	// already a previous service register under given name, merge methods/subscriptions
-	if regsvc, present := s.services[name]; present {
-		for _, m := range methods {
-			regsvc.callbacks[formatName(m.method.Name)] = m
-		}
-		for _, s := range subscriptions {
-			regsvc.subscriptions[formatName(s.method.Name)] = s
-		}
-		return nil
-	}
-
-	svc.name = name
-	svc.callbacks, svc.subscriptions = methods, subscriptions
-
-	s.services[svc.name] = svc
-	return nil
-}
-
-// serveRequest will reads requests from the codec, calls the RPC callback and
-// writes the response to the given codec.
-//
-// If singleShot is true it will process a single request, otherwise it will handle
-// requests until the codec returns an error when reading a request (in most cases
-// an EOF). It executes requests in parallel when singleShot is false.
-func (s *Server) serveRequest(ctx context.Context, codec ServerCodec, singleShot bool, options CodecOption) error {
-	var pend sync.WaitGroup
-
-	defer func() {
-		if err := recover(); err != nil {
-			const size = 64 << 10
-			buf := make([]byte, size)
-			buf = buf[:runtime.Stack(buf, false)]
-			log.Error(string(buf))
-		}
-		s.codecsMu.Lock()
-		s.codecs.Remove(codec)
-		s.codecsMu.Unlock()
-	}()
-
-	//	ctx, cancel := context.WithCancel(context.Background())
-	ctx, cancel := context.WithCancel(ctx)
-	defer cancel()
-
-	// if the codec supports notification include a notifier that callbacks can use
-	// to send notification to clients. It is tied to the codec/connection. If the
-	// connection is closed the notifier will stop and cancels all active subscriptions.
-	if options&OptionSubscriptions == OptionSubscriptions {
-		ctx = context.WithValue(ctx, notifierKey{}, newNotifier(codec))
-	}
-	s.codecsMu.Lock()
-	if atomic.LoadInt32(&s.run) != 1 { // server stopped
-		s.codecsMu.Unlock()
-		return &shutdownError{}
-	}
-	s.codecs.Add(codec)
-	s.codecsMu.Unlock()
-
-	// test if the server is ordered to stop
-	for atomic.LoadInt32(&s.run) == 1 {
-		reqs, batch, err := s.readRequest(codec)
-		if err != nil {
-			// If a parsing error occurred, send an error
-			if err.Error() != "EOF" {
-				log.Debug(fmt.Sprintf("read error %v\n", err))
-				codec.Write(codec.CreateErrorResponse(nil, err))
-			}
-			// Error or end of stream, wait for requests and tear down
-			pend.Wait()
-			return nil
-		}
-
-		// check if server is ordered to shutdown and return an error
-		// telling the client that his request failed.
-		if atomic.LoadInt32(&s.run) != 1 {
-			err = &shutdownError{}
-			if batch {
-				resps := make([]interface{}, len(reqs))
-				for i, r := range reqs {
-					resps[i] = codec.CreateErrorResponse(&r.id, err)
-				}
-				codec.Write(resps)
-			} else {
-				codec.Write(codec.CreateErrorResponse(&reqs[0].id, err))
-			}
-			return nil
-		}
-		// If a single shot request is executing, run and return immediately
-		if singleShot {
-			if batch {
-				s.execBatch(ctx, codec, reqs)
-			} else {
-				s.exec(ctx, codec, reqs[0])
-			}
-			return nil
-		}
-		// For multi-shot connections, start a goroutine to serve and loop back
-		pend.Add(1)
-
-		go func(reqs []*serverRequest, batch bool) {
-			defer pend.Done()
-			if batch {
-				s.execBatch(ctx, codec, reqs)
-			} else {
-				s.exec(ctx, codec, reqs[0])
-			}
-		}(reqs, batch)
-	}
-	return nil
-}
-
-// ServeCodec reads incoming requests from codec, calls the appropriate callback and writes the
-// response back using the given codec. It will block until the codec is closed or the server is
-// stopped. In either case the codec is closed.
-func (s *Server) ServeCodec(codec ServerCodec, options CodecOption) {
-	defer codec.Close()
-	s.serveRequest(context.Background(), codec, false, options)
-}
-
-// ServeSingleRequest reads and processes a single RPC request from the given codec. It will not
-// close the codec unless a non-recoverable error has occurred. Note, this method will return after
-// a single request has been processed!
-func (s *Server) ServeSingleRequest(ctx context.Context, codec ServerCodec, options CodecOption) {
-	s.serveRequest(ctx, codec, true, options)
-}
-
-// Stop will stop reading new requests, wait for stopPendingRequestTimeout to allow pending requests to finish,
-// close all codecs which will cancel pending requests/subscriptions.
-func (s *Server) Stop() {
-	if atomic.CompareAndSwapInt32(&s.run, 1, 0) {
-		log.Debug("RPC Server shutdown initiatied")
-		s.codecsMu.Lock()
-		defer s.codecsMu.Unlock()
-		s.codecs.Each(func(c interface{}) bool {
-			c.(ServerCodec).Close()
-			return true
-		})
-	}
-}
-
-// createSubscription will call the subscription callback and returns the subscription id or error.
-func (s *Server) createSubscription(ctx context.Context, c ServerCodec, req *serverRequest) (ID, error) {
-	// subscription have as first argument the context following optional arguments
-	args := []reflect.Value{req.callb.rcvr, reflect.ValueOf(ctx)}
-	args = append(args, req.args...)
-	reply := req.callb.method.Func.Call(args)
-
-	if !reply[1].IsNil() { // subscription creation failed
-		return "", reply[1].Interface().(error)
-	}
-
-	return reply[0].Interface().(*Subscription).ID, nil
-}
-
-// handle executes a request and returns the response from the callback.
-func (s *Server) handle(ctx context.Context, codec ServerCodec, req *serverRequest) (interface{}, func()) {
-	if req.err != nil {
-		return codec.CreateErrorResponse(&req.id, req.err), nil
-	}
-
-	if req.isUnsubscribe { // cancel subscription, first param must be the subscription id
-		if len(req.args) >= 1 && req.args[0].Kind() == reflect.String {
-			notifier, supported := NotifierFromContext(ctx)
-			if !supported { // interface doesn't support subscriptions (e.g. http)
-				return codec.CreateErrorResponse(&req.id, &callbackError{ErrNotificationsUnsupported.Error()}), nil
-			}
-
-			subid := ID(req.args[0].String())
-			if err := notifier.unsubscribe(subid); err != nil {
-				return codec.CreateErrorResponse(&req.id, &callbackError{err.Error()}), nil
-			}
-
-			return codec.CreateResponse(req.id, true), nil
-		}
-		return codec.CreateErrorResponse(&req.id, &invalidParamsError{"Expected subscription id as first argument"}), nil
-	}
-
-	if req.callb.isSubscribe {
-		subid, err := s.createSubscription(ctx, codec, req)
-		if err != nil {
-			return codec.CreateErrorResponse(&req.id, &callbackError{err.Error()}), nil
-		}
-
-		// active the subscription after the sub id was successfully sent to the client
-		activateSub := func() {
-			notifier, _ := NotifierFromContext(ctx)
-			notifier.activate(subid, req.svcname)
-		}
-
-		return codec.CreateResponse(req.id, subid), activateSub
-	}
-
-	// regular RPC call, prepare arguments
-	if len(req.args) != len(req.callb.argTypes) {
-		rpcErr := &invalidParamsError{fmt.Sprintf("%s%s%s expects %d parameters, got %d",
-			req.svcname, serviceMethodSeparator, req.callb.method.Name,
-			len(req.callb.argTypes), len(req.args))}
-		return codec.CreateErrorResponse(&req.id, rpcErr), nil
-	}
-
-	arguments := []reflect.Value{req.callb.rcvr}
-	if req.callb.hasCtx {
-		arguments = append(arguments, reflect.ValueOf(ctx))
-	}
-	if len(req.args) > 0 {
-		arguments = append(arguments, req.args...)
-	}
-
-	// execute RPC method and return result
-	reply := req.callb.method.Func.Call(arguments)
-	if len(reply) == 0 {
-		return codec.CreateResponse(req.id, nil), nil
-	}
-	if req.callb.errPos >= 0 { // test if method returned an error
-		if !reply[req.callb.errPos].IsNil() {
-			e := reply[req.callb.errPos].Interface().(error)
-			res := codec.CreateErrorResponse(&req.id, &callbackError{e.Error()})
-			return res, nil
-		}
-	}
-	return codec.CreateResponse(req.id, reply[0].Interface()), nil
-}
-
-// exec executes the given request and writes the result back using the codec.
-func (s *Server) exec(ctx context.Context, codec ServerCodec, req *serverRequest) {
-	var response interface{}
-	var callback func()
-	if req.err != nil {
-		response = codec.CreateErrorResponse(&req.id, req.err)
-	} else {
-		response, callback = s.handle(ctx, codec, req)
-	}
-
-	if err := codec.Write(response); err != nil {
-		log.Error(fmt.Sprintf("%v\n", err))
-		codec.Close()
-	}
-
-	// when request was a subscribe request this allows these subscriptions to be actived
-	if callback != nil {
-		callback()
-	}
-}
-
-// execBatch executes the given requests and writes the result back using the codec.
-// It will only write the response back when the last request is processed.
-func (s *Server) execBatch(ctx context.Context, codec ServerCodec, requests []*serverRequest) {
-	responses := make([]interface{}, len(requests))
-	var callbacks []func()
-	for i, req := range requests {
-		if req.err != nil {
-			responses[i] = codec.CreateErrorResponse(&req.id, req.err)
-		} else {
-			var callback func()
-			if responses[i], callback = s.handle(ctx, codec, req); callback != nil {
-				callbacks = append(callbacks, callback)
-			}
-		}
-	}
-
-	if err := codec.Write(responses); err != nil {
-		log.Error(fmt.Sprintf("%v\n", err))
-		codec.Close()
-	}
-
-	// when request holds one of more subscribe requests this allows these subscriptions to be activated
-	for _, c := range callbacks {
-		c()
-	}
-}
-
-// readRequest requests the next (batch) request from the codec. It will return the collection
-// of requests, an indication if the request was a batch, the invalid request identifier and an
-// error when the request could not be read/parsed.
-func (s *Server) readRequest(codec ServerCodec) ([]*serverRequest, bool, Error) {
-	reqs, batch, err := codec.ReadRequestHeaders()
-	if err != nil {
-		return nil, batch, err
-	}
-
-	requests := make([]*serverRequest, len(reqs))
-
-	// verify requests
-	for i, r := range reqs {
-		var ok bool
-		var svc *service
-
-		if r.err != nil {
-			requests[i] = &serverRequest{id: r.id, err: r.err}
-			continue
-		}
-
-		if r.isPubSub && strings.HasSuffix(r.method, unsubscribeMethodSuffix) {
-			requests[i] = &serverRequest{id: r.id, isUnsubscribe: true}
-			argTypes := []reflect.Type{reflect.TypeOf("")} // expect subscription id as first arg
-			if args, err := codec.ParseRequestArguments(argTypes, r.params); err == nil {
-				requests[i].args = args
-			} else {
-				requests[i].err = &invalidParamsError{err.Error()}
-			}
-			continue
-		}
-
-		if svc, ok = s.services[r.service]; !ok { // rpc method isn't available
-			requests[i] = &serverRequest{id: r.id, err: &methodNotFoundError{r.service, r.method}}
-			continue
-		}
-
-		if r.isPubSub { // eth_subscribe, r.method contains the subscription method name
-			if callb, ok := svc.subscriptions[r.method]; ok {
-				requests[i] = &serverRequest{id: r.id, svcname: svc.name, callb: callb}
-				if r.params != nil && len(callb.argTypes) > 0 {
-					argTypes := []reflect.Type{reflect.TypeOf("")}
-					argTypes = append(argTypes, callb.argTypes...)
-					if args, err := codec.ParseRequestArguments(argTypes, r.params); err == nil {
-						requests[i].args = args[1:] // first one is service.method name which isn't an actual argument
-					} else {
-						requests[i].err = &invalidParamsError{err.Error()}
-					}
-				}
-			} else {
-				requests[i] = &serverRequest{id: r.id, err: &methodNotFoundError{r.service, r.method}}
-			}
-			continue
-		}
-
-		if callb, ok := svc.callbacks[r.method]; ok { // lookup RPC method
-			requests[i] = &serverRequest{id: r.id, svcname: svc.name, callb: callb}
-			if r.params != nil && len(callb.argTypes) > 0 {
-				if args, err := codec.ParseRequestArguments(callb.argTypes, r.params); err == nil {
-					requests[i].args = args
-				} else {
-					requests[i].err = &invalidParamsError{err.Error()}
-				}
-			}
-			continue
-		}
-
-		requests[i] = &serverRequest{id: r.id, err: &methodNotFoundError{r.service, r.method}}
-	}
-
-	return requests, batch, nil
-}

rpc/server_test.go

@@ -17,146 +17,136 @@
 package rpc
 
 import (
-	"context"
+	"bufio"
-	"encoding/json"
+	"bytes"
+	"io"
+	"io/ioutil"
 	"net"
-	"reflect"
+	"path/filepath"
+	"strings"
 	"testing"
 	"time"
 )
 
-type Service struct{}
-
-type Args struct {
-	S string
-}
-
-func (s *Service) NoArgsRets() {
-}
-
-type Result struct {
-	String string
-	Int    int
-	Args   *Args
-}
-
-func (s *Service) Echo(str string, i int, args *Args) Result {
-	return Result{str, i, args}
-}
-
-func (s *Service) EchoWithCtx(ctx context.Context, str string, i int, args *Args) Result {
-	return Result{str, i, args}
-}
-
-func (s *Service) Sleep(ctx context.Context, duration time.Duration) {
-	select {
-	case <-time.After(duration):
-	case <-ctx.Done():
-	}
-}
-
-func (s *Service) Rets() (string, error) {
-	return "", nil
-}
-
-func (s *Service) InvalidRets1() (error, string) {
-	return nil, ""
-}
-
-func (s *Service) InvalidRets2() (string, string) {
-	return "", ""
-}
-
-func (s *Service) InvalidRets3() (string, string, error) {
-	return "", "", nil
-}
-
-func (s *Service) Subscription(ctx context.Context) (*Subscription, error) {
-	return nil, nil
-}
-
 func TestServerRegisterName(t *testing.T) {
 	server := NewServer()
-	service := new(Service)
+	service := new(testService)
-
-	if err := server.RegisterName("calc", service); err != nil {
-		t.Fatalf("%v", err)
-	}
-
-	if len(server.services) != 2 {
-		t.Fatalf("Expected 2 service entries, got %d", len(server.services))
-	}
-
-	svc, ok := server.services["calc"]
-	if !ok {
-		t.Fatalf("Expected service calc to be registered")
-	}
-
-	if len(svc.callbacks) != 5 {
-		t.Errorf("Expected 5 callbacks for service 'calc', got %d", len(svc.callbacks))
-	}
-
-	if len(svc.subscriptions) != 1 {
-		t.Errorf("Expected 1 subscription for service 'calc', got %d", len(svc.subscriptions))
-	}
-}
-
-func testServerMethodExecution(t *testing.T, method string) {
-	server := NewServer()
-	service := new(Service)
 
 	if err := server.RegisterName("test", service); err != nil {
 		t.Fatalf("%v", err)
 	}
 
-	stringArg := "string arg"
+	if len(server.services.services) != 2 {
-	intArg := 1122
+		t.Fatalf("Expected 2 service entries, got %d", len(server.services.services))
-	argsArg := &Args{"abcde"}
+	}
-	params := []interface{}{stringArg, intArg, argsArg}
 
-	request := map[string]interface{}{
+	svc, ok := server.services.services["test"]
-		"id":      12345,
+	if !ok {
-		"method":  "test_" + method,
+		t.Fatalf("Expected service calc to be registered")
-		"version": "2.0",
+	}
-		"params":  params,
+
+	wantCallbacks := 7
+	if len(svc.callbacks) != wantCallbacks {
+		t.Errorf("Expected %d callbacks for service 'service', got %d", wantCallbacks, len(svc.callbacks))
+	}
+}
+
+func TestServer(t *testing.T) {
+	files, err := ioutil.ReadDir("testdata")
+	if err != nil {
+		t.Fatal("where'd my testdata go?")
+	}
+	for _, f := range files {
+		if f.IsDir() || strings.HasPrefix(f.Name(), ".") {
+			continue
+		}
+		path := filepath.Join("testdata", f.Name())
+		name := strings.TrimSuffix(f.Name(), filepath.Ext(f.Name()))
+		t.Run(name, func(t *testing.T) {
+			runTestScript(t, path)
+		})
+	}
+}
+
+func runTestScript(t *testing.T, file string) {
+	server := newTestServer()
+	content, err := ioutil.ReadFile(file)
+	if err != nil {
+		t.Fatal(err)
 	}
 
 	clientConn, serverConn := net.Pipe()
 	defer clientConn.Close()
-
+	go server.ServeCodec(NewJSONCodec(serverConn), OptionMethodInvocation|OptionSubscriptions)
-	go server.ServeCodec(NewJSONCodec(serverConn), OptionMethodInvocation)
+	readbuf := bufio.NewReader(clientConn)
-
+	for _, line := range strings.Split(string(content), "\n") {
-	out := json.NewEncoder(clientConn)
+		line = strings.TrimSpace(line)
-	in := json.NewDecoder(clientConn)
+		switch {
-
+		case len(line) == 0 || strings.HasPrefix(line, "//"):
-	if err := out.Encode(request); err != nil {
+			// skip comments, blank lines
-		t.Fatal(err)
+			continue
+		case strings.HasPrefix(line, "--> "):
+			t.Log(line)
+			// write to connection
+			clientConn.SetWriteDeadline(time.Now().Add(5 * time.Second))
+			if _, err := io.WriteString(clientConn, line[4:]+"\n"); err != nil {
+				t.Fatalf("write error: %v", err)
 			}
-
+		case strings.HasPrefix(line, "<-- "):
-	response := jsonSuccessResponse{Result: &Result{}}
+			t.Log(line)
-	if err := in.Decode(&response); err != nil {
+			want := line[4:]
-		t.Fatal(err)
+			// read line from connection and compare text
+			clientConn.SetReadDeadline(time.Now().Add(5 * time.Second))
+			sent, err := readbuf.ReadString('\n')
+			if err != nil {
+				t.Fatalf("read error: %v", err)
 			}
-
+			sent = strings.TrimRight(sent, "\r\n")
-	if result, ok := response.Result.(*Result); ok {
+			if sent != want {
-		if result.String != stringArg {
+				t.Errorf("wrong line from server\ngot:  %s\nwant: %s", sent, want)
-			t.Errorf("expected %s, got : %s\n", stringArg, result.String)
 			}
-		if result.Int != intArg {
+		default:
-			t.Errorf("expected %d, got %d\n", intArg, result.Int)
+			panic("invalid line in test script: " + line)
 		}
-		if !reflect.DeepEqual(result.Args, argsArg) {
-			t.Errorf("expected %v, got %v\n", argsArg, result)
-		}
-	} else {
-		t.Fatalf("invalid response: expected *Result - got: %T", response.Result)
 	}
 }
 
-func TestServerMethodExecution(t *testing.T) {
+// This test checks that responses are delivered for very short-lived connections that
-	testServerMethodExecution(t, "echo")
+// only carry a single request.
-}
+func TestServerShortLivedConn(t *testing.T) {
+	server := newTestServer()
+	defer server.Stop()
 
-func TestServerMethodWithCtx(t *testing.T) {
+	listener, err := net.Listen("tcp", "127.0.0.1:0")
-	testServerMethodExecution(t, "echoWithCtx")
+	if err != nil {
+		t.Fatal("can't listen:", err)
+	}
+	defer listener.Close()
+	go server.ServeListener(listener)
+
+	var (
+		request  = `{"jsonrpc":"2.0","id":1,"method":"rpc_modules"}` + "\n"
+		wantResp = `{"jsonrpc":"2.0","id":1,"result":{"nftest":"1.0","rpc":"1.0","test":"1.0"}}` + "\n"
+		deadline = time.Now().Add(10 * time.Second)
+	)
+	for i := 0; i < 20; i++ {
+		conn, err := net.Dial("tcp", listener.Addr().String())
+		if err != nil {
+			t.Fatal("can't dial:", err)
+		}
+		defer conn.Close()
+		conn.SetDeadline(deadline)
+		// Write the request, then half-close the connection so the server stops reading.
+		conn.Write([]byte(request))
+		conn.(*net.TCPConn).CloseWrite()
+		// Now try to get the response.
+		buf := make([]byte, 2000)
+		n, err := conn.Read(buf)
+		if err != nil {
+			t.Fatal("read error:", err)
+		}
+		if !bytes.Equal(buf[:n], []byte(wantResp)) {
+			t.Fatalf("wrong response: %s", buf[:n])
+		}
+	}
 }

rpc/service.go

@@ -0,0 +1,285 @@
+// Copyright 2015 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package rpc
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"reflect"
+	"runtime"
+	"strings"
+	"sync"
+	"unicode"
+	"unicode/utf8"
+
+	"github.com/ethereum/go-ethereum/log"
+)
+
+var (
+	contextType      = reflect.TypeOf((*context.Context)(nil)).Elem()
+	errorType        = reflect.TypeOf((*error)(nil)).Elem()
+	subscriptionType = reflect.TypeOf(Subscription{})
+	stringType       = reflect.TypeOf("")
+)
+
+type serviceRegistry struct {
+	mu       sync.Mutex
+	services map[string]service
+}
+
+// service represents a registered object.
+type service struct {
+	name          string               // name for service
+	callbacks     map[string]*callback // registered handlers
+	subscriptions map[string]*callback // available subscriptions/notifications
+}
+
+// callback is a method callback which was registered in the server
+type callback struct {
+	fn          reflect.Value  // the function
+	rcvr        reflect.Value  // receiver object of method, set if fn is method
+	argTypes    []reflect.Type // input argument types
+	hasCtx      bool           // method's first argument is a context (not included in argTypes)
+	errPos      int            // err return idx, of -1 when method cannot return error
+	isSubscribe bool           // true if this is a subscription callback
+}
+
+func (r *serviceRegistry) registerName(name string, rcvr interface{}) error {
+	rcvrVal := reflect.ValueOf(rcvr)
+	if name == "" {
+		return fmt.Errorf("no service name for type %s", rcvrVal.Type().String())
+	}
+	callbacks := suitableCallbacks(rcvrVal)
+	if len(callbacks) == 0 {
+		return fmt.Errorf("service %T doesn't have any suitable methods/subscriptions to expose", rcvr)
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if r.services == nil {
+		r.services = make(map[string]service)
+	}
+	svc, ok := r.services[name]
+	if !ok {
+		svc = service{
+			name:          name,
+			callbacks:     make(map[string]*callback),
+			subscriptions: make(map[string]*callback),
+		}
+		r.services[name] = svc
+	}
+	for name, cb := range callbacks {
+		if cb.isSubscribe {
+			svc.subscriptions[name] = cb
+		} else {
+			svc.callbacks[name] = cb
+		}
+	}
+	return nil
+}
+
+// callback returns the callback corresponding to the given RPC method name.
+func (r *serviceRegistry) callback(method string) *callback {
+	elem := strings.SplitN(method, serviceMethodSeparator, 2)
+	if len(elem) != 2 {
+		return nil
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return r.services[elem[0]].callbacks[elem[1]]
+}
+
+// subscription returns a subscription callback in the given service.
+func (r *serviceRegistry) subscription(service, name string) *callback {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return r.services[service].subscriptions[name]
+}
+
+// suitableCallbacks iterates over the methods of the given type. It determines if a method
+// satisfies the criteria for a RPC callback or a subscription callback and adds it to the
+// collection of callbacks. See server documentation for a summary of these criteria.
+func suitableCallbacks(receiver reflect.Value) map[string]*callback {
+	typ := receiver.Type()
+	callbacks := make(map[string]*callback)
+	for m := 0; m < typ.NumMethod(); m++ {
+		method := typ.Method(m)
+		if method.PkgPath != "" {
+			continue // method not exported
+		}
+		cb := newCallback(receiver, method.Func)
+		if cb == nil {
+			continue // function invalid
+		}
+		name := formatName(method.Name)
+		callbacks[name] = cb
+	}
+	return callbacks
+}
+
+// newCallback turns fn (a function) into a callback object. It returns nil if the function
+// is unsuitable as an RPC callback.
+func newCallback(receiver, fn reflect.Value) *callback {
+	fntype := fn.Type()
+	c := &callback{fn: fn, rcvr: receiver, errPos: -1, isSubscribe: isPubSub(fntype)}
+	// Determine parameter types. They must all be exported or builtin types.
+	c.makeArgTypes()
+	if !allExportedOrBuiltin(c.argTypes) {
+		return nil
+	}
+	// Verify return types. The function must return at most one error
+	// and/or one other non-error value.
+	outs := make([]reflect.Type, fntype.NumOut())
+	for i := 0; i < fntype.NumOut(); i++ {
+		outs[i] = fntype.Out(i)
+	}
+	if len(outs) > 2 || !allExportedOrBuiltin(outs) {
+		return nil
+	}
+	// If an error is returned, it must be the last returned value.
+	switch {
+	case len(outs) == 1 && isErrorType(outs[0]):
+		c.errPos = 0
+	case len(outs) == 2:
+		if isErrorType(outs[0]) || !isErrorType(outs[1]) {
+			return nil
+		}
+		c.errPos = 1
+	}
+	return c
+}
+
+// makeArgTypes composes the argTypes list.
+func (c *callback) makeArgTypes() {
+	fntype := c.fn.Type()
+	// Skip receiver and context.Context parameter (if present).
+	firstArg := 0
+	if c.rcvr.IsValid() {
+		firstArg++
+	}
+	if fntype.NumIn() > firstArg && fntype.In(firstArg) == contextType {
+		c.hasCtx = true
+		firstArg++
+	}
+	// Add all remaining parameters.
+	c.argTypes = make([]reflect.Type, fntype.NumIn()-firstArg)
+	for i := firstArg; i < fntype.NumIn(); i++ {
+		c.argTypes[i-firstArg] = fntype.In(i)
+	}
+}
+
+// call invokes the callback.
+func (c *callback) call(ctx context.Context, method string, args []reflect.Value) (res interface{}, errRes error) {
+	// Create the argument slice.
+	fullargs := make([]reflect.Value, 0, 2+len(args))
+	if c.rcvr.IsValid() {
+		fullargs = append(fullargs, c.rcvr)
+	}
+	if c.hasCtx {
+		fullargs = append(fullargs, reflect.ValueOf(ctx))
+	}
+	fullargs = append(fullargs, args...)
+
+	// Catch panic while running the callback.
+	defer func() {
+		if err := recover(); err != nil {
+			const size = 64 << 10
+			buf := make([]byte, size)
+			buf = buf[:runtime.Stack(buf, false)]
+			log.Error("RPC method " + method + " crashed: " + fmt.Sprintf("%v\n%s", err, buf))
+			errRes = errors.New("method handler crashed")
+		}
+	}()
+	// Run the callback.
+	results := c.fn.Call(fullargs)
+	if len(results) == 0 {
+		return nil, nil
+	}
+	if c.errPos >= 0 && !results[c.errPos].IsNil() {
+		// Method has returned non-nil error value.
+		err := results[c.errPos].Interface().(error)
+		return reflect.Value{}, err
+	}
+	return results[0].Interface(), nil
+}
+
+// Is this an exported - upper case - name?
+func isExported(name string) bool {
+	rune, _ := utf8.DecodeRuneInString(name)
+	return unicode.IsUpper(rune)
+}
+
+// Are all those types exported or built-in?
+func allExportedOrBuiltin(types []reflect.Type) bool {
+	for _, typ := range types {
+		for typ.Kind() == reflect.Ptr {
+			typ = typ.Elem()
+		}
+		// PkgPath will be non-empty even for an exported type,
+		// so we need to check the type name as well.
+		if !isExported(typ.Name()) && typ.PkgPath() != "" {
+			return false
+		}
+	}
+	return true
+}
+
+// Is t context.Context or *context.Context?
+func isContextType(t reflect.Type) bool {
+	for t.Kind() == reflect.Ptr {
+		t = t.Elem()
+	}
+	return t == contextType
+}
+
+// Does t satisfy the error interface?
+func isErrorType(t reflect.Type) bool {
+	for t.Kind() == reflect.Ptr {
+		t = t.Elem()
+	}
+	return t.Implements(errorType)
+}
+
+// Is t Subscription or *Subscription?
+func isSubscriptionType(t reflect.Type) bool {
+	for t.Kind() == reflect.Ptr {
+		t = t.Elem()
+	}
+	return t == subscriptionType
+}
+
+// isPubSub tests whether the given method has as as first argument a context.Context and
+// returns the pair (Subscription, error).
+func isPubSub(methodType reflect.Type) bool {
+	// numIn(0) is the receiver type
+	if methodType.NumIn() < 2 || methodType.NumOut() != 2 {
+		return false
+	}
+	return isContextType(methodType.In(1)) &&
+		isSubscriptionType(methodType.Out(0)) &&
+		isErrorType(methodType.Out(1))
+}
+
+// formatName converts to first character of name to lowercase.
+func formatName(name string) string {
+	ret := []rune(name)
+	if len(ret) > 0 {
+		ret[0] = unicode.ToLower(ret[0])
+	}
+	return string(ret)
+}

rpc/stdio.go

@@ -26,8 +26,8 @@ import (
 
 // DialStdIO creates a client on stdin/stdout.
 func DialStdIO(ctx context.Context) (*Client, error) {
-	return newClient(ctx, func(_ context.Context) (net.Conn, error) {
+	return newClient(ctx, func(_ context.Context) (ServerCodec, error) {
-		return stdioConn{}, nil
+		return NewJSONCodec(stdioConn{}), nil
 	})
 }
 
@@ -45,20 +45,8 @@ func (io stdioConn) Close() error {
 	return nil
 }
 
-func (io stdioConn) LocalAddr() net.Addr {
+func (io stdioConn) RemoteAddr() string {
-	return &net.UnixAddr{Name: "stdio", Net: "stdio"}
+	return "/dev/stdin"
-}
-
-func (io stdioConn) RemoteAddr() net.Addr {
-	return &net.UnixAddr{Name: "stdio", Net: "stdio"}
-}
-
-func (io stdioConn) SetDeadline(t time.Time) error {
-	return &net.OpError{Op: "set", Net: "stdio", Source: nil, Addr: nil, Err: errors.New("deadline not supported")}
-}
-
-func (io stdioConn) SetReadDeadline(t time.Time) error {
-	return &net.OpError{Op: "set", Net: "stdio", Source: nil, Addr: nil, Err: errors.New("deadline not supported")}
 }
 
 func (io stdioConn) SetWriteDeadline(t time.Time) error {

rpc/subscription.go

@@ -17,9 +17,19 @@
 package rpc
 
 import (
+	"bufio"
+	"container/list"
 	"context"
+	crand "crypto/rand"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
 	"errors"
+	"math/rand"
+	"reflect"
+	"strings"
 	"sync"
+	"time"
 )
 
 var (
@@ -29,10 +39,147 @@ var (
 	ErrSubscriptionNotFound = errors.New("subscription not found")
 )
 
+var globalGen = randomIDGenerator()
+
 // ID defines a pseudo random number that is used to identify RPC subscriptions.
 type ID string
 
-// a Subscription is created by a notifier and tight to that notifier. The client can use
+// NewID returns a new, random ID.
+func NewID() ID {
+	return globalGen()
+}
+
+// randomIDGenerator returns a function generates a random IDs.
+func randomIDGenerator() func() ID {
+	seed, err := binary.ReadVarint(bufio.NewReader(crand.Reader))
+	if err != nil {
+		seed = int64(time.Now().Nanosecond())
+	}
+	var (
+		mu  sync.Mutex
+		rng = rand.New(rand.NewSource(seed))
+	)
+	return func() ID {
+		mu.Lock()
+		defer mu.Unlock()
+		id := make([]byte, 16)
+		rng.Read(id)
+		return encodeID(id)
+	}
+}
+
+func encodeID(b []byte) ID {
+	id := hex.EncodeToString(b)
+	id = strings.TrimLeft(id, "0")
+	if id == "" {
+		id = "0" // ID's are RPC quantities, no leading zero's and 0 is 0x0.
+	}
+	return ID("0x" + id)
+}
+
+type notifierKey struct{}
+
+// NotifierFromContext returns the Notifier value stored in ctx, if any.
+func NotifierFromContext(ctx context.Context) (*Notifier, bool) {
+	n, ok := ctx.Value(notifierKey{}).(*Notifier)
+	return n, ok
+}
+
+// Notifier is tied to a RPC connection that supports subscriptions.
+// Server callbacks use the notifier to send notifications.
+type Notifier struct {
+	h         *handler
+	namespace string
+
+	mu           sync.Mutex
+	sub          *Subscription
+	buffer       []json.RawMessage
+	callReturned bool
+	activated    bool
+}
+
+// CreateSubscription returns a new subscription that is coupled to the
+// RPC connection. By default subscriptions are inactive and notifications
+// are dropped until the subscription is marked as active. This is done
+// by the RPC server after the subscription ID is send to the client.
+func (n *Notifier) CreateSubscription() *Subscription {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+
+	if n.sub != nil {
+		panic("can't create multiple subscriptions with Notifier")
+	} else if n.callReturned {
+		panic("can't create subscription after subscribe call has returned")
+	}
+	n.sub = &Subscription{ID: n.h.idgen(), namespace: n.namespace, err: make(chan error, 1)}
+	return n.sub
+}
+
+// Notify sends a notification to the client with the given data as payload.
+// If an error occurs the RPC connection is closed and the error is returned.
+func (n *Notifier) Notify(id ID, data interface{}) error {
+	enc, err := json.Marshal(data)
+	if err != nil {
+		return err
+	}
+
+	n.mu.Lock()
+	defer n.mu.Unlock()
+
+	if n.sub == nil {
+		panic("can't Notify before subscription is created")
+	} else if n.sub.ID != id {
+		panic("Notify with wrong ID")
+	}
+	if n.activated {
+		return n.send(n.sub, enc)
+	}
+	n.buffer = append(n.buffer, enc)
+	return nil
+}
+
+// Closed returns a channel that is closed when the RPC connection is closed.
+// Deprecated: use subscription error channel
+func (n *Notifier) Closed() <-chan interface{} {
+	return n.h.conn.Closed()
+}
+
+// takeSubscription returns the subscription (if one has been created). No subscription can
+// be created after this call.
+func (n *Notifier) takeSubscription() *Subscription {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	n.callReturned = true
+	return n.sub
+}
+
+// acticate is called after the subscription ID was sent to client. Notifications are
+// buffered before activation. This prevents notifications being sent to the client before
+// the subscription ID is sent to the client.
+func (n *Notifier) activate() error {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+
+	for _, data := range n.buffer {
+		if err := n.send(n.sub, data); err != nil {
+			return err
+		}
+	}
+	n.activated = true
+	return nil
+}
+
+func (n *Notifier) send(sub *Subscription, data json.RawMessage) error {
+	params, _ := json.Marshal(&subscriptionResult{ID: string(sub.ID), Result: data})
+	ctx := context.Background()
+	return n.h.conn.Write(ctx, &jsonrpcMessage{
+		Version: vsn,
+		Method:  n.namespace + notificationMethodSuffix,
+		Params:  params,
+	})
+}
+
+// A Subscription is created by a notifier and tight to that notifier. The client can use
 // this subscription to wait for an unsubscribe request for the client, see Err().
 type Subscription struct {
 	ID        ID
@@ -45,105 +192,136 @@ func (s *Subscription) Err() <-chan error {
 	return s.err
 }
 
-// notifierKey is used to store a notifier within the connection context.
+// MarshalJSON marshals a subscription as its ID.
-type notifierKey struct{}
+func (s *Subscription) MarshalJSON() ([]byte, error) {
-
+	return json.Marshal(s.ID)
-// Notifier is tight to a RPC connection that supports subscriptions.
-// Server callbacks use the notifier to send notifications.
-type Notifier struct {
-	codec    ServerCodec
-	subMu    sync.Mutex
-	active   map[ID]*Subscription
-	inactive map[ID]*Subscription
-	buffer   map[ID][]interface{} // unsent notifications of inactive subscriptions
 }
 
-// newNotifier creates a new notifier that can be used to send subscription
+// ClientSubscription is a subscription established through the Client's Subscribe or
-// notifications to the client.
+// EthSubscribe methods.
-func newNotifier(codec ServerCodec) *Notifier {
+type ClientSubscription struct {
-	return &Notifier{
+	client    *Client
-		codec:    codec,
+	etype     reflect.Type
-		active:   make(map[ID]*Subscription),
+	channel   reflect.Value
-		inactive: make(map[ID]*Subscription),
+	namespace string
-		buffer:   make(map[ID][]interface{}),
+	subid     string
-	}
+	in        chan json.RawMessage
+
+	quitOnce sync.Once     // ensures quit is closed once
+	quit     chan struct{} // quit is closed when the subscription exits
+	errOnce  sync.Once     // ensures err is closed once
+	err      chan error
 }
 
-// NotifierFromContext returns the Notifier value stored in ctx, if any.
+func newClientSubscription(c *Client, namespace string, channel reflect.Value) *ClientSubscription {
-func NotifierFromContext(ctx context.Context) (*Notifier, bool) {
+	sub := &ClientSubscription{
-	n, ok := ctx.Value(notifierKey{}).(*Notifier)
+		client:    c,
-	return n, ok
+		namespace: namespace,
+		etype:     channel.Type().Elem(),
+		channel:   channel,
+		quit:      make(chan struct{}),
+		err:       make(chan error, 1),
+		in:        make(chan json.RawMessage),
+	}
+	return sub
 }
 
-// CreateSubscription returns a new subscription that is coupled to the
+// Err returns the subscription error channel. The intended use of Err is to schedule
-// RPC connection. By default subscriptions are inactive and notifications
+// resubscription when the client connection is closed unexpectedly.
-// are dropped until the subscription is marked as active. This is done
+//
-// by the RPC server after the subscription ID is send to the client.
+// The error channel receives a value when the subscription has ended due
-func (n *Notifier) CreateSubscription() *Subscription {
+// to an error. The received error is nil if Close has been called
-	s := &Subscription{ID: NewID(), err: make(chan error)}
+// on the underlying client and no other error has occurred.
-	n.subMu.Lock()
+//
-	n.inactive[s.ID] = s
+// The error channel is closed when Unsubscribe is called on the subscription.
-	n.subMu.Unlock()
+func (sub *ClientSubscription) Err() <-chan error {
-	return s
+	return sub.err
 }
 
-// Notify sends a notification to the client with the given data as payload.
+// Unsubscribe unsubscribes the notification and closes the error channel.
-// If an error occurs the RPC connection is closed and the error is returned.
+// It can safely be called more than once.
-func (n *Notifier) Notify(id ID, data interface{}) error {
+func (sub *ClientSubscription) Unsubscribe() {
-	n.subMu.Lock()
+	sub.quitWithError(nil, true)
-	defer n.subMu.Unlock()
+	sub.errOnce.Do(func() { close(sub.err) })
-
-	if sub, active := n.active[id]; active {
-		n.send(sub, data)
-	} else {
-		n.buffer[id] = append(n.buffer[id], data)
-	}
-	return nil
 }
 
-func (n *Notifier) send(sub *Subscription, data interface{}) error {
+func (sub *ClientSubscription) quitWithError(err error, unsubscribeServer bool) {
-	notification := n.codec.CreateNotification(string(sub.ID), sub.namespace, data)
+	sub.quitOnce.Do(func() {
-	err := n.codec.Write(notification)
+		// The dispatch loop won't be able to execute the unsubscribe call
+		// if it is blocked on deliver. Close sub.quit first because it
+		// unblocks deliver.
+		close(sub.quit)
+		if unsubscribeServer {
+			sub.requestUnsubscribe()
+		}
 		if err != nil {
-		n.codec.Close()
+			if err == ErrClientQuit {
+				err = nil // Adhere to subscription semantics.
 			}
-	return err
+			sub.err <- err
+		}
+	})
 }
 
-// Closed returns a channel that is closed when the RPC connection is closed.
+func (sub *ClientSubscription) deliver(result json.RawMessage) (ok bool) {
-func (n *Notifier) Closed() <-chan interface{} {
+	select {
-	return n.codec.Closed()
+	case sub.in <- result:
+		return true
+	case <-sub.quit:
+		return false
+	}
 }
 
-// unsubscribe a subscription.
+func (sub *ClientSubscription) start() {
-// If the subscription could not be found ErrSubscriptionNotFound is returned.
+	sub.quitWithError(sub.forward())
-func (n *Notifier) unsubscribe(id ID) error {
-	n.subMu.Lock()
-	defer n.subMu.Unlock()
-	if s, found := n.active[id]; found {
-		close(s.err)
-		delete(n.active, id)
-		return nil
-	}
-	return ErrSubscriptionNotFound
 }
 
-// activate enables a subscription. Until a subscription is enabled all
+func (sub *ClientSubscription) forward() (err error, unsubscribeServer bool) {
-// notifications are dropped. This method is called by the RPC server after
+	cases := []reflect.SelectCase{
-// the subscription ID was sent to client. This prevents notifications being
+		{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(sub.quit)},
-// send to the client before the subscription ID is send to the client.
+		{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(sub.in)},
-func (n *Notifier) activate(id ID, namespace string) {
+		{Dir: reflect.SelectSend, Chan: sub.channel},
-	n.subMu.Lock()
+	}
-	defer n.subMu.Unlock()
+	buffer := list.New()
+	defer buffer.Init()
+	for {
+		var chosen int
+		var recv reflect.Value
+		if buffer.Len() == 0 {
+			// Idle, omit send case.
+			chosen, recv, _ = reflect.Select(cases[:2])
+		} else {
+			// Non-empty buffer, send the first queued item.
+			cases[2].Send = reflect.ValueOf(buffer.Front().Value)
+			chosen, recv, _ = reflect.Select(cases)
+		}
 
-	if sub, found := n.inactive[id]; found {
+		switch chosen {
-		sub.namespace = namespace
+		case 0: // <-sub.quit
-		n.active[id] = sub
+			return nil, false
-		delete(n.inactive, id)
+		case 1: // <-sub.in
-		// Send buffered notifications.
+			val, err := sub.unmarshal(recv.Interface().(json.RawMessage))
-		for _, data := range n.buffer[id] {
+			if err != nil {
-			n.send(sub, data)
+				return err, true
 			}
-		delete(n.buffer, id)
+			if buffer.Len() == maxClientSubscriptionBuffer {
+				return ErrSubscriptionQueueOverflow, true
+			}
+			buffer.PushBack(val)
+		case 2: // sub.channel<-
+			cases[2].Send = reflect.Value{} // Don't hold onto the value.
+			buffer.Remove(buffer.Front())
 		}
 	}
+}
+
+func (sub *ClientSubscription) unmarshal(result json.RawMessage) (interface{}, error) {
+	val := reflect.New(sub.etype)
+	err := json.Unmarshal(result, val.Interface())
+	return val.Elem().Interface(), err
+}
+
+func (sub *ClientSubscription) requestUnsubscribe() error {
+	var result interface{}
+	return sub.client.Call(&result, sub.namespace+unsubscribeMethodSuffix, sub.subid)
+}

rpc/subscription_test.go

@@ -17,232 +17,62 @@
 package rpc
 
 import (
-	"context"
 	"encoding/json"
 	"fmt"
 	"net"
-	"sync"
+	"strings"
 	"testing"
 	"time"
 )
 
-type NotificationTestService struct {
+func TestNewID(t *testing.T) {
-	mu                      sync.Mutex
+	hexchars := "0123456789ABCDEFabcdef"
-	unsubscribed            chan string
+	for i := 0; i < 100; i++ {
-	gotHangSubscriptionReq  chan struct{}
+		id := string(NewID())
-	unblockHangSubscription chan struct{}
+		if !strings.HasPrefix(id, "0x") {
+			t.Fatalf("invalid ID prefix, want '0x...', got %s", id)
 		}
 
-func (s *NotificationTestService) Echo(i int) int {
+		id = id[2:]
-	return i
+		if len(id) == 0 || len(id) > 32 {
+			t.Fatalf("invalid ID length, want len(id) > 0 && len(id) <= 32), got %d", len(id))
 		}
 
-func (s *NotificationTestService) Unsubscribe(subid string) {
+		for i := 0; i < len(id); i++ {
-	if s.unsubscribed != nil {
+			if strings.IndexByte(hexchars, id[i]) == -1 {
-		s.unsubscribed <- subid
+				t.Fatalf("unexpected byte, want any valid hex char, got %c", id[i])
 			}
 		}
-
-func (s *NotificationTestService) SomeSubscription(ctx context.Context, n, val int) (*Subscription, error) {
-	notifier, supported := NotifierFromContext(ctx)
-	if !supported {
-		return nil, ErrNotificationsUnsupported
-	}
-
-	// by explicitly creating an subscription we make sure that the subscription id is send back to the client
-	// before the first subscription.Notify is called. Otherwise the events might be send before the response
-	// for the eth_subscribe method.
-	subscription := notifier.CreateSubscription()
-
-	go func() {
-		// test expects n events, if we begin sending event immediately some events
-		// will probably be dropped since the subscription ID might not be send to
-		// the client.
-		for i := 0; i < n; i++ {
-			if err := notifier.Notify(subscription.ID, val+i); err != nil {
-				return
-			}
-		}
-
-		select {
-		case <-notifier.Closed():
-		case <-subscription.Err():
-		}
-		if s.unsubscribed != nil {
-			s.unsubscribed <- string(subscription.ID)
-		}
-	}()
-
-	return subscription, nil
-}
-
-// HangSubscription blocks on s.unblockHangSubscription before
-// sending anything.
-func (s *NotificationTestService) HangSubscription(ctx context.Context, val int) (*Subscription, error) {
-	notifier, supported := NotifierFromContext(ctx)
-	if !supported {
-		return nil, ErrNotificationsUnsupported
-	}
-
-	s.gotHangSubscriptionReq <- struct{}{}
-	<-s.unblockHangSubscription
-	subscription := notifier.CreateSubscription()
-
-	go func() {
-		notifier.Notify(subscription.ID, val)
-	}()
-	return subscription, nil
-}
-
-func TestNotifications(t *testing.T) {
-	server := NewServer()
-	service := &NotificationTestService{unsubscribed: make(chan string)}
-
-	if err := server.RegisterName("eth", service); err != nil {
-		t.Fatalf("unable to register test service %v", err)
-	}
-
-	clientConn, serverConn := net.Pipe()
-
-	go server.ServeCodec(NewJSONCodec(serverConn), OptionMethodInvocation|OptionSubscriptions)
-
-	out := json.NewEncoder(clientConn)
-	in := json.NewDecoder(clientConn)
-
-	n := 5
-	val := 12345
-	request := map[string]interface{}{
-		"id":      1,
-		"method":  "eth_subscribe",
-		"version": "2.0",
-		"params":  []interface{}{"someSubscription", n, val},
-	}
-
-	// create subscription
-	if err := out.Encode(request); err != nil {
-		t.Fatal(err)
-	}
-
-	var subid string
-	response := jsonSuccessResponse{Result: subid}
-	if err := in.Decode(&response); err != nil {
-		t.Fatal(err)
-	}
-
-	var ok bool
-	if _, ok = response.Result.(string); !ok {
-		t.Fatalf("expected subscription id, got %T", response.Result)
-	}
-
-	for i := 0; i < n; i++ {
-		var notification jsonNotification
-		if err := in.Decode(&notification); err != nil {
-			t.Fatalf("%v", err)
-		}
-
-		if int(notification.Params.Result.(float64)) != val+i {
-			t.Fatalf("expected %d, got %d", val+i, notification.Params.Result)
-		}
-	}
-
-	clientConn.Close() // causes notification unsubscribe callback to be called
-	select {
-	case <-service.unsubscribed:
-	case <-time.After(1 * time.Second):
-		t.Fatal("Unsubscribe not called after one second")
-	}
-}
-
-func waitForMessages(t *testing.T, in *json.Decoder, successes chan<- jsonSuccessResponse,
-	failures chan<- jsonErrResponse, notifications chan<- jsonNotification, errors chan<- error) {
-
-	// read and parse server messages
-	for {
-		var rmsg json.RawMessage
-		if err := in.Decode(&rmsg); err != nil {
-			return
-		}
-
-		var responses []map[string]interface{}
-		if rmsg[0] == '[' {
-			if err := json.Unmarshal(rmsg, &responses); err != nil {
-				errors <- fmt.Errorf("Received invalid message: %s", rmsg)
-				return
-			}
-		} else {
-			var msg map[string]interface{}
-			if err := json.Unmarshal(rmsg, &msg); err != nil {
-				errors <- fmt.Errorf("Received invalid message: %s", rmsg)
-				return
-			}
-			responses = append(responses, msg)
-		}
-
-		for _, msg := range responses {
-			// determine what kind of msg was received and broadcast
-			// it to over the corresponding channel
-			if _, found := msg["result"]; found {
-				successes <- jsonSuccessResponse{
-					Version: msg["jsonrpc"].(string),
-					Id:      msg["id"],
-					Result:  msg["result"],
-				}
-				continue
-			}
-			if _, found := msg["error"]; found {
-				params := msg["params"].(map[string]interface{})
-				failures <- jsonErrResponse{
-					Version: msg["jsonrpc"].(string),
-					Id:      msg["id"],
-					Error:   jsonError{int(params["subscription"].(float64)), params["message"].(string), params["data"]},
-				}
-				continue
-			}
-			if _, found := msg["params"]; found {
-				params := msg["params"].(map[string]interface{})
-				notifications <- jsonNotification{
-					Version: msg["jsonrpc"].(string),
-					Method:  msg["method"].(string),
-					Params:  jsonSubscription{params["subscription"].(string), params["result"]},
-				}
-				continue
-			}
-			errors <- fmt.Errorf("Received invalid message: %s", msg)
-		}
 	}
 }
 
-// TestSubscriptionMultipleNamespaces ensures that subscriptions can exists
+func TestSubscriptions(t *testing.T) {
-// for multiple different namespaces.
-func TestSubscriptionMultipleNamespaces(t *testing.T) {
 	var (
 		namespaces        = []string{"eth", "shh", "bzz"}
-		service           = NotificationTestService{}
+		service           = &notificationTestService{}
-		subCount          = len(namespaces) * 2
+		subCount          = len(namespaces)
 		notificationCount = 3
 
 		server                 = NewServer()
 		clientConn, serverConn = net.Pipe()
 		out                    = json.NewEncoder(clientConn)
 		in                     = json.NewDecoder(clientConn)
-		successes              = make(chan jsonSuccessResponse)
+		successes              = make(chan subConfirmation)
-		failures               = make(chan jsonErrResponse)
+		notifications          = make(chan subscriptionResult)
-		notifications          = make(chan jsonNotification)
+		errors                 = make(chan error, subCount*notificationCount+1)
-		errors                 = make(chan error, 10)
 	)
 
 	// setup and start server
 	for _, namespace := range namespaces {
-		if err := server.RegisterName(namespace, &service); err != nil {
+		if err := server.RegisterName(namespace, service); err != nil {
 			t.Fatalf("unable to register test service %v", err)
 		}
 	}
-
 	go server.ServeCodec(NewJSONCodec(serverConn), OptionMethodInvocation|OptionSubscriptions)
 	defer server.Stop()
 
 	// wait for message and write them to the given channels
-	go waitForMessages(t, in, successes, failures, notifications, errors)
+	go waitForMessages(in, successes, notifications, errors)
 
 	// create subscriptions one by one
 	for i, namespace := range namespaces {
@@ -252,27 +82,11 @@ func TestSubscriptionMultipleNamespaces(t *testing.T) {
 			"version": "2.0",
 			"params":  []interface{}{"someSubscription", notificationCount, i},
 		}
-
 		if err := out.Encode(&request); err != nil {
 			t.Fatalf("Could not create subscription: %v", err)
 		}
 	}
 
-	// create all subscriptions in 1 batch
-	var requests []interface{}
-	for i, namespace := range namespaces {
-		requests = append(requests, map[string]interface{}{
-			"id":      i,
-			"method":  fmt.Sprintf("%s_subscribe", namespace),
-			"version": "2.0",
-			"params":  []interface{}{"someSubscription", notificationCount, i},
-		})
-	}
-
-	if err := out.Encode(&requests); err != nil {
-		t.Fatalf("Could not create subscription in batch form: %v", err)
-	}
-
 	timeout := time.After(30 * time.Second)
 	subids := make(map[string]string, subCount)
 	count := make(map[string]int, subCount)
@@ -285,17 +99,14 @@ func TestSubscriptionMultipleNamespaces(t *testing.T) {
 		}
 		return done
 	}
-
 	for !allReceived() {
 		select {
-		case suc := <-successes: // subscription created
+		case confirmation := <-successes: // subscription created
-			subids[namespaces[int(suc.Id.(float64))]] = suc.Result.(string)
+			subids[namespaces[confirmation.reqid]] = string(confirmation.subid)
 		case notification := <-notifications:
-			count[notification.Params.Subscription]++
+			count[notification.ID]++
 		case err := <-errors:
 			t.Fatal(err)
-		case failure := <-failures:
-			t.Errorf("received error: %v", failure.Error)
 		case <-timeout:
 			for _, namespace := range namespaces {
 				subid, found := subids[namespace]
@@ -311,3 +122,85 @@ func TestSubscriptionMultipleNamespaces(t *testing.T) {
 		}
 	}
 }
+
+// This test checks that unsubscribing works.
+func TestServerUnsubscribe(t *testing.T) {
+	// Start the server.
+	server := newTestServer()
+	service := &notificationTestService{unsubscribed: make(chan string)}
+	server.RegisterName("nftest2", service)
+	p1, p2 := net.Pipe()
+	go server.ServeCodec(NewJSONCodec(p1), OptionMethodInvocation|OptionSubscriptions)
+
+	p2.SetDeadline(time.Now().Add(10 * time.Second))
+
+	// Subscribe.
+	p2.Write([]byte(`{"jsonrpc":"2.0","id":1,"method":"nftest2_subscribe","params":["someSubscription",0,10]}`))
+
+	// Handle received messages.
+	resps := make(chan subConfirmation)
+	notifications := make(chan subscriptionResult)
+	errors := make(chan error)
+	go waitForMessages(json.NewDecoder(p2), resps, notifications, errors)
+
+	// Receive the subscription ID.
+	var sub subConfirmation
+	select {
+	case sub = <-resps:
+	case err := <-errors:
+		t.Fatal(err)
+	}
+
+	// Unsubscribe and check that it is handled on the server side.
+	p2.Write([]byte(`{"jsonrpc":"2.0","method":"nftest2_unsubscribe","params":["` + sub.subid + `"]}`))
+	for {
+		select {
+		case id := <-service.unsubscribed:
+			if id != string(sub.subid) {
+				t.Errorf("wrong subscription ID unsubscribed")
+			}
+			return
+		case err := <-errors:
+			t.Fatal(err)
+		case <-notifications:
+			// drop notifications
+		}
+	}
+}
+
+type subConfirmation struct {
+	reqid int
+	subid ID
+}
+
+func waitForMessages(in *json.Decoder, successes chan subConfirmation, notifications chan subscriptionResult, errors chan error) {
+	for {
+		var msg jsonrpcMessage
+		if err := in.Decode(&msg); err != nil {
+			errors <- fmt.Errorf("decode error: %v", err)
+			return
+		}
+		switch {
+		case msg.isNotification():
+			var res subscriptionResult
+			if err := json.Unmarshal(msg.Params, &res); err != nil {
+				errors <- fmt.Errorf("invalid subscription result: %v", err)
+			} else {
+				notifications <- res
+			}
+		case msg.isResponse():
+			var c subConfirmation
+			if msg.Error != nil {
+				errors <- msg.Error
+			} else if err := json.Unmarshal(msg.Result, &c.subid); err != nil {
+				errors <- fmt.Errorf("invalid response: %v", err)
+			} else {
+				json.Unmarshal(msg.ID, &c.reqid)
+				successes <- c
+			}
+		default:
+			errors <- fmt.Errorf("unrecognized message: %v", msg)
+			return
+		}
+	}
+}

rpc/testdata/invalid-badid.js

@@ -0,0 +1,7 @@
+// This test checks processing of messages with invalid ID.
+
+--> {"id":[],"method":"test_foo"}
+<-- {"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}}
+
+--> {"id":{},"method":"test_foo"}
+<-- {"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}}

rpc/testdata/invalid-batch.js

@@ -0,0 +1,14 @@
+// This test checks the behavior of batches with invalid elements.
+// Empty batches are not allowed. Batches may contain junk.
+
+--> []
+<-- {"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"empty batch"}}
+
+--> [1]
+<-- [{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}}]
+
+--> [1,2,3]
+<-- [{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}},{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}},{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}}]
+
+--> [{"jsonrpc":"2.0","id":1,"method":"test_echo","params":["foo",1]},55,{"jsonrpc":"2.0","id":2,"method":"unknown_method"},{"foo":"bar"}]
+<-- [{"jsonrpc":"2.0","id":1,"result":{"String":"foo","Int":1,"Args":null}},{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}},{"jsonrpc":"2.0","id":2,"error":{"code":-32601,"message":"the method unknown_method does not exist/is not available"}},{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}}]

rpc/testdata/invalid-idonly.js

@@ -0,0 +1,7 @@
+// This test checks processing of messages that contain just the ID and nothing else.
+
+--> {"id":1}
+<-- {"jsonrpc":"2.0","id":1,"error":{"code":-32600,"message":"invalid request"}}
+
+--> {"jsonrpc":"2.0","id":1}
+<-- {"jsonrpc":"2.0","id":1,"error":{"code":-32600,"message":"invalid request"}}

rpc/testdata/invalid-nonobj.js

@@ -0,0 +1,4 @@
+// This test checks behavior for invalid requests.
+
+--> 1
+<-- {"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"invalid request"}}

rpc/testdata/invalid-syntax.json

@@ -0,0 +1,5 @@
+// This test checks that an error is written for invalid JSON requests.
+
+--> 'f
+<-- {"jsonrpc":"2.0","id":null,"error":{"code":-32700,"message":"invalid character '\\'' looking for beginning of value"}}
+

rpc/testdata/reqresp-batch.js

@@ -0,0 +1,8 @@
+// There is no response for all-notification batches.
+
+--> [{"jsonrpc":"2.0","method":"test_echo","params":["x",99]}]
+
+// This test checks regular batch calls.
+
+--> [{"jsonrpc":"2.0","id":2,"method":"test_echo","params":[]}, {"jsonrpc":"2.0","id": 3,"method":"test_echo","params":["x",3]}]
+<-- [{"jsonrpc":"2.0","id":2,"error":{"code":-32602,"message":"missing value for required argument 0"}},{"jsonrpc":"2.0","id":3,"result":{"String":"x","Int":3,"Args":null}}]

rpc/testdata/reqresp-echo.js

@@ -0,0 +1,16 @@
+// This test calls the test_echo method.
+
+--> {"jsonrpc": "2.0", "id": 2, "method": "test_echo", "params": []}
+<-- {"jsonrpc":"2.0","id":2,"error":{"code":-32602,"message":"missing value for required argument 0"}}
+
+--> {"jsonrpc": "2.0", "id": 2, "method": "test_echo", "params": ["x"]}
+<-- {"jsonrpc":"2.0","id":2,"error":{"code":-32602,"message":"missing value for required argument 1"}}
+
+--> {"jsonrpc": "2.0", "id": 2, "method": "test_echo", "params": ["x", 3]}
+<-- {"jsonrpc":"2.0","id":2,"result":{"String":"x","Int":3,"Args":null}}
+
+--> {"jsonrpc": "2.0", "id": 2, "method": "test_echo", "params": ["x", 3, {"S": "foo"}]}
+<-- {"jsonrpc":"2.0","id":2,"result":{"String":"x","Int":3,"Args":{"S":"foo"}}}
+
+--> {"jsonrpc": "2.0", "id": 2, "method": "test_echoWithCtx", "params": ["x", 3, {"S": "foo"}]}
+<-- {"jsonrpc":"2.0","id":2,"result":{"String":"x","Int":3,"Args":{"S":"foo"}}}

rpc/testdata/reqresp-namedparam.js

@@ -0,0 +1,5 @@
+// This test checks that an error response is sent for calls
+// with named parameters. 
+
+--> {"jsonrpc":"2.0","method":"test_echo","params":{"int":23},"id":3}
+<-- {"jsonrpc":"2.0","id":3,"error":{"code":-32602,"message":"non-array args"}}

rpc/testdata/reqresp-noargsrets.js

@@ -0,0 +1,4 @@
+// This test calls the test_noArgsRets method.
+
+--> {"jsonrpc": "2.0", "id": "foo", "method": "test_noArgsRets", "params": []}
+<-- {"jsonrpc":"2.0","id":"foo","result":null}

rpc/testdata/reqresp-nomethod.js

@@ -0,0 +1,4 @@
+// This test calls a method that doesn't exist.
+
+--> {"jsonrpc": "2.0", "id": 2, "method": "invalid_method", "params": [2, 3]}
+<-- {"jsonrpc":"2.0","id":2,"error":{"code":-32601,"message":"the method invalid_method does not exist/is not available"}}

rpc/testdata/reqresp-noparam.js

@@ -0,0 +1,4 @@
+// This test checks that calls with no parameters work.
+
+--> {"jsonrpc":"2.0","method":"test_noArgsRets","id":3}
+<-- {"jsonrpc":"2.0","id":3,"result":null}

rpc/testdata/reqresp-paramsnull.js

@@ -0,0 +1,4 @@
+// This test checks that calls with "params":null work.
+
+--> {"jsonrpc":"2.0","method":"test_noArgsRets","params":null,"id":3}
+<-- {"jsonrpc":"2.0","id":3,"result":null}

rpc/testdata/revcall.js

@@ -0,0 +1,6 @@
+// This test checks reverse calls.
+
+--> {"jsonrpc":"2.0","id":2,"method":"test_callMeBack","params":["foo",[1]]}
+<-- {"jsonrpc":"2.0","id":1,"method":"foo","params":[1]}
+--> {"jsonrpc":"2.0","id":1,"result":"my result"}
+<-- {"jsonrpc":"2.0","id":2,"result":"my result"}

rpc/testdata/revcall2.js

@@ -0,0 +1,7 @@
+// This test checks reverse calls.
+
+--> {"jsonrpc":"2.0","id":2,"method":"test_callMeBackLater","params":["foo",[1]]}
+<-- {"jsonrpc":"2.0","id":2,"result":null}
+<-- {"jsonrpc":"2.0","id":1,"method":"foo","params":[1]}
+--> {"jsonrpc":"2.0","id":1,"result":"my result"}
+

rpc/testdata/subscription.js

@@ -0,0 +1,12 @@
+// This test checks basic subscription support.
+
+--> {"jsonrpc":"2.0","id":1,"method":"nftest_subscribe","params":["someSubscription",5,1]}
+<-- {"jsonrpc":"2.0","id":1,"result":"0x1"}
+<-- {"jsonrpc":"2.0","method":"nftest_subscription","params":{"subscription":"0x1","result":1}}
+<-- {"jsonrpc":"2.0","method":"nftest_subscription","params":{"subscription":"0x1","result":2}}
+<-- {"jsonrpc":"2.0","method":"nftest_subscription","params":{"subscription":"0x1","result":3}}
+<-- {"jsonrpc":"2.0","method":"nftest_subscription","params":{"subscription":"0x1","result":4}}
+<-- {"jsonrpc":"2.0","method":"nftest_subscription","params":{"subscription":"0x1","result":5}}
+
+--> {"jsonrpc":"2.0","id":2,"method":"nftest_echo","params":[11]}
+<-- {"jsonrpc":"2.0","id":2,"result":11}

rpc/testservice_test.go

@@ -0,0 +1,180 @@
+// Copyright 2018 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package rpc
+
+import (
+	"context"
+	"encoding/binary"
+	"errors"
+	"sync"
+	"time"
+)
+
+func newTestServer() *Server {
+	server := NewServer()
+	server.idgen = sequentialIDGenerator()
+	if err := server.RegisterName("test", new(testService)); err != nil {
+		panic(err)
+	}
+	if err := server.RegisterName("nftest", new(notificationTestService)); err != nil {
+		panic(err)
+	}
+	return server
+}
+
+func sequentialIDGenerator() func() ID {
+	var (
+		mu      sync.Mutex
+		counter uint64
+	)
+	return func() ID {
+		mu.Lock()
+		defer mu.Unlock()
+		counter++
+		id := make([]byte, 8)
+		binary.BigEndian.PutUint64(id, counter)
+		return encodeID(id)
+	}
+}
+
+type testService struct{}
+
+type Args struct {
+	S string
+}
+
+type Result struct {
+	String string
+	Int    int
+	Args   *Args
+}
+
+func (s *testService) NoArgsRets() {}
+
+func (s *testService) Echo(str string, i int, args *Args) Result {
+	return Result{str, i, args}
+}
+
+func (s *testService) EchoWithCtx(ctx context.Context, str string, i int, args *Args) Result {
+	return Result{str, i, args}
+}
+
+func (s *testService) Sleep(ctx context.Context, duration time.Duration) {
+	time.Sleep(duration)
+}
+
+func (s *testService) Rets() (string, error) {
+	return "", nil
+}
+
+func (s *testService) InvalidRets1() (error, string) {
+	return nil, ""
+}
+
+func (s *testService) InvalidRets2() (string, string) {
+	return "", ""
+}
+
+func (s *testService) InvalidRets3() (string, string, error) {
+	return "", "", nil
+}
+
+func (s *testService) CallMeBack(ctx context.Context, method string, args []interface{}) (interface{}, error) {
+	c, ok := ClientFromContext(ctx)
+	if !ok {
+		return nil, errors.New("no client")
+	}
+	var result interface{}
+	err := c.Call(&result, method, args...)
+	return result, err
+}
+
+func (s *testService) CallMeBackLater(ctx context.Context, method string, args []interface{}) error {
+	c, ok := ClientFromContext(ctx)
+	if !ok {
+		return errors.New("no client")
+	}
+	go func() {
+		<-ctx.Done()
+		var result interface{}
+		c.Call(&result, method, args...)
+	}()
+	return nil
+}
+
+func (s *testService) Subscription(ctx context.Context) (*Subscription, error) {
+	return nil, nil
+}
+
+type notificationTestService struct {
+	unsubscribed            chan string
+	gotHangSubscriptionReq  chan struct{}
+	unblockHangSubscription chan struct{}
+}
+
+func (s *notificationTestService) Echo(i int) int {
+	return i
+}
+
+func (s *notificationTestService) Unsubscribe(subid string) {
+	if s.unsubscribed != nil {
+		s.unsubscribed <- subid
+	}
+}
+
+func (s *notificationTestService) SomeSubscription(ctx context.Context, n, val int) (*Subscription, error) {
+	notifier, supported := NotifierFromContext(ctx)
+	if !supported {
+		return nil, ErrNotificationsUnsupported
+	}
+
+	// By explicitly creating an subscription we make sure that the subscription id is send
+	// back to the client before the first subscription.Notify is called. Otherwise the
+	// events might be send before the response for the *_subscribe method.
+	subscription := notifier.CreateSubscription()
+	go func() {
+		for i := 0; i < n; i++ {
+			if err := notifier.Notify(subscription.ID, val+i); err != nil {
+				return
+			}
+		}
+		select {
+		case <-notifier.Closed():
+		case <-subscription.Err():
+		}
+		if s.unsubscribed != nil {
+			s.unsubscribed <- string(subscription.ID)
+		}
+	}()
+	return subscription, nil
+}
+
+// HangSubscription blocks on s.unblockHangSubscription before sending anything.
+func (s *notificationTestService) HangSubscription(ctx context.Context, val int) (*Subscription, error) {
+	notifier, supported := NotifierFromContext(ctx)
+	if !supported {
+		return nil, ErrNotificationsUnsupported
+	}
+	s.gotHangSubscriptionReq <- struct{}{}
+	<-s.unblockHangSubscription
+	subscription := notifier.CreateSubscription()
+
+	go func() {
+		notifier.Notify(subscription.ID, val)
+	}()
+	return subscription, nil
+}

rpc/types.go

@@ -17,13 +17,11 @@
 package rpc
 
 import (
+	"context"
 	"fmt"
 	"math"
-	"reflect"
 	"strings"
-	"sync"
 
-	mapset "github.com/deckarep/golang-set"
 	"github.com/ethereum/go-ethereum/common/hexutil"
 )
 
@@ -35,57 +33,6 @@ type API struct {
 	Public    bool        // indication if the methods must be considered safe for public use
 }
 
-// callback is a method callback which was registered in the server
-type callback struct {
-	rcvr        reflect.Value  // receiver of method
-	method      reflect.Method // callback
-	argTypes    []reflect.Type // input argument types
-	hasCtx      bool           // method's first argument is a context (not included in argTypes)
-	errPos      int            // err return idx, of -1 when method cannot return error
-	isSubscribe bool           // indication if the callback is a subscription
-}
-
-// service represents a registered object
-type service struct {
-	name          string        // name for service
-	typ           reflect.Type  // receiver type
-	callbacks     callbacks     // registered handlers
-	subscriptions subscriptions // available subscriptions/notifications
-}
-
-// serverRequest is an incoming request
-type serverRequest struct {
-	id            interface{}
-	svcname       string
-	callb         *callback
-	args          []reflect.Value
-	isUnsubscribe bool
-	err           Error
-}
-
-type serviceRegistry map[string]*service // collection of services
-type callbacks map[string]*callback      // collection of RPC callbacks
-type subscriptions map[string]*callback  // collection of subscription callbacks
-
-// Server represents a RPC server
-type Server struct {
-	services serviceRegistry
-
-	run      int32
-	codecsMu sync.Mutex
-	codecs   mapset.Set
-}
-
-// rpcRequest represents a raw incoming RPC request
-type rpcRequest struct {
-	service  string
-	method   string
-	id       interface{}
-	isPubSub bool
-	params   interface{}
-	err      Error // invalid batch element
-}
-
 // Error wraps RPC errors, which contain an error code in addition to the message.
 type Error interface {
 	Error() string  // returns the message
@@ -96,24 +43,19 @@ type Error interface {
 // a RPC session. Implementations must be go-routine safe since the codec can be called in
 // multiple go-routines concurrently.
 type ServerCodec interface {
-	// Read next request
+	Read() (msgs []*jsonrpcMessage, isBatch bool, err error)
-	ReadRequestHeaders() ([]rpcRequest, bool, Error)
-	// Parse request argument to the given types
-	ParseRequestArguments(argTypes []reflect.Type, params interface{}) ([]reflect.Value, Error)
-	// Assemble success response, expects response id and payload
-	CreateResponse(id interface{}, reply interface{}) interface{}
-	// Assemble error response, expects response id and error
-	CreateErrorResponse(id interface{}, err Error) interface{}
-	// Assemble error response with extra information about the error through info
-	CreateErrorResponseWithInfo(id interface{}, err Error, info interface{}) interface{}
-	// Create notification response
-	CreateNotification(id, namespace string, event interface{}) interface{}
-	// Write msg to client.
-	Write(msg interface{}) error
-	// Close underlying data stream
 	Close()
-	// Closed when underlying connection is closed
+	jsonWriter
+}
+
+// jsonWriter can write JSON messages to its underlying connection.
+// Implementations must be safe for concurrent use.
+type jsonWriter interface {
+	Write(context.Context, interface{}) error
+	// Closed returns a channel which is closed when the connection is closed.
 	Closed() <-chan interface{}
+	// RemoteAddr returns the peer address of the connection.
+	RemoteAddr() string
 }
 
 type BlockNumber int64

rpc/utils.go

@@ -1,226 +0,0 @@
-// Copyright 2015 The go-ethereum Authors
-// This file is part of the go-ethereum library.
-//
-// The go-ethereum library is free software: you can redistribute it and/or modify
-// it under the terms of the GNU Lesser General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// The go-ethereum library is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-// GNU Lesser General Public License for more details.
-//
-// You should have received a copy of the GNU Lesser General Public License
-// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
-
-package rpc
-
-import (
-	"bufio"
-	"context"
-	crand "crypto/rand"
-	"encoding/binary"
-	"encoding/hex"
-	"math/rand"
-	"reflect"
-	"strings"
-	"sync"
-	"time"
-	"unicode"
-	"unicode/utf8"
-)
-
-var (
-	subscriptionIDGenMu sync.Mutex
-	subscriptionIDGen   = idGenerator()
-)
-
-// Is this an exported - upper case - name?
-func isExported(name string) bool {
-	rune, _ := utf8.DecodeRuneInString(name)
-	return unicode.IsUpper(rune)
-}
-
-// Is this type exported or a builtin?
-func isExportedOrBuiltinType(t reflect.Type) bool {
-	for t.Kind() == reflect.Ptr {
-		t = t.Elem()
-	}
-	// PkgPath will be non-empty even for an exported type,
-	// so we need to check the type name as well.
-	return isExported(t.Name()) || t.PkgPath() == ""
-}
-
-var contextType = reflect.TypeOf((*context.Context)(nil)).Elem()
-
-// isContextType returns an indication if the given t is of context.Context or *context.Context type
-func isContextType(t reflect.Type) bool {
-	for t.Kind() == reflect.Ptr {
-		t = t.Elem()
-	}
-	return t == contextType
-}
-
-var errorType = reflect.TypeOf((*error)(nil)).Elem()
-
-// Implements this type the error interface
-func isErrorType(t reflect.Type) bool {
-	for t.Kind() == reflect.Ptr {
-		t = t.Elem()
-	}
-	return t.Implements(errorType)
-}
-
-var subscriptionType = reflect.TypeOf((*Subscription)(nil)).Elem()
-
-// isSubscriptionType returns an indication if the given t is of Subscription or *Subscription type
-func isSubscriptionType(t reflect.Type) bool {
-	for t.Kind() == reflect.Ptr {
-		t = t.Elem()
-	}
-	return t == subscriptionType
-}
-
-// isPubSub tests whether the given method has as as first argument a context.Context
-// and returns the pair (Subscription, error)
-func isPubSub(methodType reflect.Type) bool {
-	// numIn(0) is the receiver type
-	if methodType.NumIn() < 2 || methodType.NumOut() != 2 {
-		return false
-	}
-
-	return isContextType(methodType.In(1)) &&
-		isSubscriptionType(methodType.Out(0)) &&
-		isErrorType(methodType.Out(1))
-}
-
-// formatName will convert to first character to lower case
-func formatName(name string) string {
-	ret := []rune(name)
-	if len(ret) > 0 {
-		ret[0] = unicode.ToLower(ret[0])
-	}
-	return string(ret)
-}
-
-// suitableCallbacks iterates over the methods of the given type. It will determine if a method satisfies the criteria
-// for a RPC callback or a subscription callback and adds it to the collection of callbacks or subscriptions. See server
-// documentation for a summary of these criteria.
-func suitableCallbacks(rcvr reflect.Value, typ reflect.Type) (callbacks, subscriptions) {
-	callbacks := make(callbacks)
-	subscriptions := make(subscriptions)
-
-METHODS:
-	for m := 0; m < typ.NumMethod(); m++ {
-		method := typ.Method(m)
-		mtype := method.Type
-		mname := formatName(method.Name)
-		if method.PkgPath != "" { // method must be exported
-			continue
-		}
-
-		var h callback
-		h.isSubscribe = isPubSub(mtype)
-		h.rcvr = rcvr
-		h.method = method
-		h.errPos = -1
-
-		firstArg := 1
-		numIn := mtype.NumIn()
-		if numIn >= 2 && mtype.In(1) == contextType {
-			h.hasCtx = true
-			firstArg = 2
-		}
-
-		if h.isSubscribe {
-			h.argTypes = make([]reflect.Type, numIn-firstArg) // skip rcvr type
-			for i := firstArg; i < numIn; i++ {
-				argType := mtype.In(i)
-				if isExportedOrBuiltinType(argType) {
-					h.argTypes[i-firstArg] = argType
-				} else {
-					continue METHODS
-				}
-			}
-
-			subscriptions[mname] = &h
-			continue METHODS
-		}
-
-		// determine method arguments, ignore first arg since it's the receiver type
-		// Arguments must be exported or builtin types
-		h.argTypes = make([]reflect.Type, numIn-firstArg)
-		for i := firstArg; i < numIn; i++ {
-			argType := mtype.In(i)
-			if !isExportedOrBuiltinType(argType) {
-				continue METHODS
-			}
-			h.argTypes[i-firstArg] = argType
-		}
-
-		// check that all returned values are exported or builtin types
-		for i := 0; i < mtype.NumOut(); i++ {
-			if !isExportedOrBuiltinType(mtype.Out(i)) {
-				continue METHODS
-			}
-		}
-
-		// when a method returns an error it must be the last returned value
-		h.errPos = -1
-		for i := 0; i < mtype.NumOut(); i++ {
-			if isErrorType(mtype.Out(i)) {
-				h.errPos = i
-				break
-			}
-		}
-
-		if h.errPos >= 0 && h.errPos != mtype.NumOut()-1 {
-			continue METHODS
-		}
-
-		switch mtype.NumOut() {
-		case 0, 1, 2:
-			if mtype.NumOut() == 2 && h.errPos == -1 { // method must one return value and 1 error
-				continue METHODS
-			}
-			callbacks[mname] = &h
-		}
-	}
-
-	return callbacks, subscriptions
-}
-
-// idGenerator helper utility that generates a (pseudo) random sequence of
-// bytes that are used to generate identifiers.
-func idGenerator() *rand.Rand {
-	if seed, err := binary.ReadVarint(bufio.NewReader(crand.Reader)); err == nil {
-		return rand.New(rand.NewSource(seed))
-	}
-	return rand.New(rand.NewSource(int64(time.Now().Nanosecond())))
-}
-
-// NewID generates a identifier that can be used as an identifier in the RPC interface.
-// e.g. filter and subscription identifier.
-func NewID() ID {
-	subscriptionIDGenMu.Lock()
-	defer subscriptionIDGenMu.Unlock()
-
-	id := make([]byte, 16)
-	for i := 0; i < len(id); i += 7 {
-		val := subscriptionIDGen.Int63()
-		for j := 0; i+j < len(id) && j < 7; j++ {
-			id[i+j] = byte(val)
-			val >>= 8
-		}
-	}
-
-	rpcId := hex.EncodeToString(id)
-	// rpc ID's are RPC quantities, no leading zero's and 0 is 0x0
-	rpcId = strings.TrimLeft(rpcId, "0")
-	if rpcId == "" {
-		rpcId = "0"
-	}
-
-	return ID("0x" + rpcId)
-}

rpc/websocket.go

@@ -22,6 +22,7 @@ import (
 	"crypto/tls"
 	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"net"
 	"net/http"
@@ -56,22 +57,37 @@ var websocketJSONCodec = websocket.Codec{
 //
 // allowedOrigins should be a comma-separated list of allowed origin URLs.
 // To allow connections with any origin, pass "*".
-func (srv *Server) WebsocketHandler(allowedOrigins []string) http.Handler {
+func (s *Server) WebsocketHandler(allowedOrigins []string) http.Handler {
 	return websocket.Server{
 		Handshake: wsHandshakeValidator(allowedOrigins),
 		Handler: func(conn *websocket.Conn) {
+			codec := newWebsocketCodec(conn)
+			s.ServeCodec(codec, OptionMethodInvocation|OptionSubscriptions)
+		},
+	}
+}
+
+func newWebsocketCodec(conn *websocket.Conn) ServerCodec {
 	// Create a custom encode/decode pair to enforce payload size and number encoding
 	conn.MaxPayloadBytes = maxRequestContentLength
-
 	encoder := func(v interface{}) error {
 		return websocketJSONCodec.Send(conn, v)
 	}
 	decoder := func(v interface{}) error {
 		return websocketJSONCodec.Receive(conn, v)
 	}
-			srv.ServeCodec(NewCodec(conn, encoder, decoder), OptionMethodInvocation|OptionSubscriptions)
+	rpcconn := Conn(conn)
-		},
+	if conn.IsServerConn() {
+		// Override remote address with the actual socket address because
+		// package websocket crashes if there is no request origin.
+		addr := conn.Request().RemoteAddr
+		if wsaddr := conn.RemoteAddr().(*websocket.Addr); wsaddr.URL != nil {
+			// Add origin if present.
+			addr += "(" + wsaddr.URL.String() + ")"
 		}
+		rpcconn = connWithRemoteAddr{conn, addr}
+	}
+	return NewCodec(rpcconn, encoder, decoder)
 }
 
 // NewWSServer creates a new websocket RPC server around an API provider.
@@ -105,15 +121,16 @@ func wsHandshakeValidator(allowedOrigins []string) func(*websocket.Config, *http
 		}
 	}
 
-	log.Debug(fmt.Sprintf("Allowed origin(s) for WS RPC interface %v\n", origins.ToSlice()))
+	log.Debug(fmt.Sprintf("Allowed origin(s) for WS RPC interface %v", origins.ToSlice()))
 
 	f := func(cfg *websocket.Config, req *http.Request) error {
+		// Verify origin against whitelist.
 		origin := strings.ToLower(req.Header.Get("Origin"))
 		if allowAllOrigins || origins.Contains(origin) {
 			return nil
 		}
-		log.Warn(fmt.Sprintf("origin '%s' not allowed on WS-RPC interface\n", origin))
+		log.Warn("Rejected WebSocket connection", "origin", origin)
-		return fmt.Errorf("origin %s not allowed", origin)
+		return errors.New("origin not allowed")
 	}
 
 	return f
@@ -155,8 +172,12 @@ func DialWebsocket(ctx context.Context, endpoint, origin string) (*Client, error
 		return nil, err
 	}
 
-	return newClient(ctx, func(ctx context.Context) (net.Conn, error) {
+	return newClient(ctx, func(ctx context.Context) (ServerCodec, error) {
-		return wsDialContext(ctx, config)
+		conn, err := wsDialContext(ctx, config)
+		if err != nil {
+			return nil, err
+		}
+		return newWebsocketCodec(conn), nil
 	})
 }
 

signer/core/api.go

@@ -35,8 +35,14 @@ import (
 	"github.com/ethereum/go-ethereum/rlp"
 )
 
+const (
 	// numberOfAccountsToDerive For hardware wallets, the number of accounts to derive
-const numberOfAccountsToDerive = 10
+	numberOfAccountsToDerive = 10
+	// ExternalAPIVersion -- see extapi_changelog.md
+	ExternalAPIVersion = "5.0.0"
+	// InternalAPIVersion -- see intapi_changelog.md
+	InternalAPIVersion = "3.1.0"
+)
 
 // ExternalAPI defines the external API through which signing requests are made.
 type ExternalAPI interface {
@@ -58,6 +64,7 @@ type ExternalAPI interface {
 	// Should be moved to Internal API, in next phase when we have
 	// bi-directional communication
 	//Import(ctx context.Context, keyJSON json.RawMessage) (Account, error)
+	Version(ctx context.Context) (string, error)
 }
 
 // SignerUI specifies what method a UI needs to implement to be able to be used as a UI for the signer
@@ -514,6 +521,56 @@ func (api *SignerAPI) SignTransaction(ctx context.Context, args SendTxArgs, meth
 
 }
 
+// Sign calculates an Ethereum ECDSA signature for:
+// keccack256("\x19Ethereum Signed Message:\n" + len(message) + message))
+//
+// Note, the produced signature conforms to the secp256k1 curve R, S and V values,
+// where the V value will be 27 or 28 for legacy reasons.
+//
+// The key used to calculate the signature is decrypted with the given password.
+//
+// https://github.com/ethereum/go-ethereum/wiki/Management-APIs#personal_sign
+func (api *SignerAPI) Sign(ctx context.Context, addr common.MixedcaseAddress, data hexutil.Bytes) (hexutil.Bytes, error) {
+	sighash, msg := SignHash(data)
+	// We make the request prior to looking up if we actually have the account, to prevent
+	// account-enumeration via the API
+	req := &SignDataRequest{Address: addr, Rawdata: data, Message: msg, Hash: sighash, Meta: MetadataFromContext(ctx)}
+	res, err := api.UI.ApproveSignData(req)
+
+	if err != nil {
+		return nil, err
+	}
+	if !res.Approved {
+		return nil, ErrRequestDenied
+	}
+	// Look up the wallet containing the requested signer
+	account := accounts.Account{Address: addr.Address()}
+	wallet, err := api.am.Find(account)
+	if err != nil {
+		return nil, err
+	}
+	// Assemble sign the data with the wallet
+	signature, err := wallet.SignTextWithPassphrase(account, res.Password, data)
+	if err != nil {
+		api.UI.ShowError(err.Error())
+		return nil, err
+	}
+	signature[64] += 27 // Transform V from 0/1 to 27/28 according to the yellow paper
+	return signature, nil
+}
+
+// SignHash is a helper function that calculates a hash for the given message that can be
+// safely used to calculate a signature from.
+//
+// The hash is calculated as
+//   keccak256("\x19Ethereum Signed Message:\n"${message length}${message}).
+//
+// This gives context to the signed message and prevents signing of transactions.
+func SignHash(data []byte) ([]byte, string) {
+	msg := fmt.Sprintf("\x19Ethereum Signed Message:\n%d%s", len(data), data)
+	return crypto.Keccak256([]byte(msg)), msg
+}
+
 // Export returns encrypted private key associated with the given address in web3 keystore format.
 func (api *SignerAPI) Export(ctx context.Context, addr common.Address) (json.RawMessage, error) {
 	res, err := api.UI.ApproveExport(&ExportRequest{Address: addr, Meta: MetadataFromContext(ctx)})
@@ -564,3 +621,9 @@ func (api *SignerAPI) Import(ctx context.Context, keyJSON json.RawMessage) (Acco
 	}
 	return Account{Typ: "Account", URL: acc.URL, Address: acc.Address}, nil
 }
+
+// Returns the external api version. This method does not require user acceptance. Available methods are
+// available via enumeration anyway, and this info does not contain user-specific data
+func (api *SignerAPI) Version(ctx context.Context) (string, error) {
+	return ExternalAPIVersion, nil
+}

signer/core/api_test.go

@@ -29,7 +29,6 @@ import (
 	"time"
 
 	"github.com/ethereum/go-ethereum/accounts/keystore"
-	"github.com/ethereum/go-ethereum/cmd/utils"
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/common/hexutil"
 	"github.com/ethereum/go-ethereum/core/types"
@@ -135,7 +134,7 @@ func setup(t *testing.T) (*SignerAPI, chan string) {
 
 	db, err := NewAbiDBFromFile("../../cmd/clef/4byte.json")
 	if err != nil {
-		utils.Fatalf(err.Error())
+		t.Fatal(err.Error())
 	}
 	var (
 		ui  = &HeadlessUI{controller}

signer/core/auditlog.go

@@ -95,14 +95,13 @@ func (l *AuditLogger) Export(ctx context.Context, addr common.Address) (json.Raw
 	return j, e
 }
 
-//func (l *AuditLogger) Import(ctx context.Context, keyJSON json.RawMessage) (Account, error) {
+func (l *AuditLogger) Version(ctx context.Context) (string, error) {
-//	// Don't actually log the json contents
+	l.log.Info("Version", "type", "request", "metadata", MetadataFromContext(ctx).String())
-//	l.log.Info("Import", "type", "request", "metadata", MetadataFromContext(ctx).String(),
+	data, err := l.api.Version(ctx)
-//		"keyJSON size", len(keyJSON))
+	l.log.Info("Version", "type", "response", "data", data, "error", err)
-//	a, e := l.api.Import(ctx, keyJSON)
+	return data, err
-//	l.log.Info("Import", "type", "response", "addr", a.String(), "error", e)
+
-//	return a, e
+}
-//}
 
 func NewAuditLogger(path string, api ExternalAPI) (*AuditLogger, error) {
 	l := log.New("api", "signer")

swarm/network/hive_test.go

@@ -18,9 +18,9 @@ package network
 
 import (
 	"io/ioutil"
-	"log"
 	"os"
 	"testing"
+	"time"
 
 	p2ptest "github.com/ethereum/go-ethereum/p2p/testing"
 	"github.com/ethereum/go-ethereum/swarm/state"
@@ -35,6 +35,8 @@ func newHiveTester(t *testing.T, params *HiveParams, n int, store state.Store) (
 	return newBzzBaseTester(t, n, addr, DiscoverySpec, pp.Run), pp
 }
 
+// TestRegisterAndConnect verifies that the protocol runs successfully
+// and that the peer connection exists afterwards
 func TestRegisterAndConnect(t *testing.T) {
 	params := NewHiveParams()
 	s, pp := newHiveTester(t, params, 1, nil)
@@ -43,25 +45,57 @@ func TestRegisterAndConnect(t *testing.T) {
 	raddr := NewAddr(node)
 	pp.Register(raddr)
 
-	// start the hive and wait for the connection
+	// start the hive
 	err := pp.Start(s.Server)
 	if err != nil {
 		t.Fatal(err)
 	}
 	defer pp.Stop()
-	// retrieve and broadcast
+
+	// both hive connect and disconect check have time delays
+	// therefore we need to verify that peer is connected
+	// so that we are sure that the disconnect timeout doesn't complete
+	// before the hive connect method is run at least once
+	timeout := time.After(time.Second)
+	for {
+		select {
+		case <-timeout:
+			t.Fatalf("expected connection")
+		default:
+		}
+		i := 0
+		pp.Kademlia.EachConn(nil, 256, func(addr *Peer, po int) bool {
+			i++
+			return true
+		})
+		if i > 0 {
+			break
+		}
+		time.Sleep(time.Millisecond)
+	}
+
+	// check that the connection actually exists
+	// the timeout error means no disconnection events
+	// were received within the a certain timeout
 	err = s.TestDisconnected(&p2ptest.Disconnect{
 		Peer:  s.Nodes[0].ID(),
 		Error: nil,
 	})
 
 	if err == nil || err.Error() != "timed out waiting for peers to disconnect" {
-		t.Fatalf("expected peer to connect")
+		t.Fatalf("expected no disconnection event")
 	}
 }
 
+// TestHiveStatePersistance creates a protocol simulation with n peers for a node
+// After protocols complete, the node is shut down and the state is stored.
+// Another simulation is created, where 0 nodes are created, but where the stored state is passed
+// The test succeeds if all the peers from the stored state are known after the protocols of the
+// second simulation have completed
+//
+// Actual connectivity is not in scope for this test, as the peers loaded from state are not known to
+// the simulation; the test only verifies that the peers are known to the node
 func TestHiveStatePersistance(t *testing.T) {
-	log.SetOutput(os.Stdout)
 
 	dir, err := ioutil.TempDir("", "hive_test_store")
 	if err != nil {
@@ -84,7 +118,8 @@ func TestHiveStatePersistance(t *testing.T) {
 		peers[raddr.String()] = true
 	}
 
-	// start the hive and wait for the connection
+	// start and stop the hive
+	// the known peers should be saved upon stopping
 	err = pp.Start(s.Server)
 	if err != nil {
 		t.Fatal(err)
@@ -92,15 +127,15 @@ func TestHiveStatePersistance(t *testing.T) {
 	pp.Stop()
 	store.Close()
 
-	persistedStore, err := state.NewDBStore(dir) //start the hive with an empty dbstore
+	// start the hive with an empty dbstore
+	persistedStore, err := state.NewDBStore(dir)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	s1, pp := newHiveTester(t, params, 1, persistedStore)
+	s1, pp := newHiveTester(t, params, 0, persistedStore)
-
-	//start the hive and wait for the connection
 
+	// start the hive and check that we know of all expected peers
 	pp.Start(s1.Server)
 	i := 0
 	pp.Kademlia.EachAddr(nil, 256, func(addr *BzzAddr, po int) bool {
@@ -108,10 +143,13 @@ func TestHiveStatePersistance(t *testing.T) {
 		i++
 		return true
 	})
+	// TODO remove this line when verified that test passes
+	time.Sleep(time.Second)
 	if i != 5 {
-		t.Errorf("invalid number of entries: got %v, want %v", i, 5)
+		t.Fatalf("invalid number of entries: got %v, want %v", i, 5)
 	}
 	if len(peers) != 0 {
 		t.Fatalf("%d peers left over: %v", len(peers), peers)
 	}
+
 }

swarm/network/stream/common_test.go

@@ -26,16 +26,19 @@ import (
 	"math/rand"
 	"os"
 	"strings"
+	"sync"
 	"sync/atomic"
 	"time"
 
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/p2p/enode"
+	"github.com/ethereum/go-ethereum/p2p/simulations/adapters"
 	p2ptest "github.com/ethereum/go-ethereum/p2p/testing"
 	"github.com/ethereum/go-ethereum/swarm/network"
 	"github.com/ethereum/go-ethereum/swarm/network/simulation"
 	"github.com/ethereum/go-ethereum/swarm/state"
 	"github.com/ethereum/go-ethereum/swarm/storage"
+	mockmem "github.com/ethereum/go-ethereum/swarm/storage/mock/mem"
 	"github.com/ethereum/go-ethereum/swarm/testutil"
 	colorable "github.com/mattn/go-colorable"
 )
@@ -66,6 +69,80 @@ func init() {
 	log.Root().SetHandler(log.LvlFilterHandler(log.Lvl(*loglevel), log.StreamHandler(colorable.NewColorableStderr(), log.TerminalFormat(true))))
 }
 
+// newNetStoreAndDelivery is a default constructor for BzzAddr, NetStore and Delivery, used in Simulations
+func newNetStoreAndDelivery(ctx *adapters.ServiceContext, bucket *sync.Map) (*network.BzzAddr, *storage.NetStore, *Delivery, func(), error) {
+	addr := network.NewAddr(ctx.Config.Node())
+
+	netStore, delivery, cleanup, err := netStoreAndDeliveryWithAddr(ctx, bucket, addr)
+	if err != nil {
+		return nil, nil, nil, nil, err
+	}
+
+	netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
+
+	return addr, netStore, delivery, cleanup, nil
+}
+
+// newNetStoreAndDeliveryWithBzzAddr is a constructor for NetStore and Delivery, used in Simulations, accepting any BzzAddr
+func newNetStoreAndDeliveryWithBzzAddr(ctx *adapters.ServiceContext, bucket *sync.Map, addr *network.BzzAddr) (*storage.NetStore, *Delivery, func(), error) {
+	netStore, delivery, cleanup, err := netStoreAndDeliveryWithAddr(ctx, bucket, addr)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
+
+	return netStore, delivery, cleanup, nil
+}
+
+// newNetStoreAndDeliveryWithRequestFunc is a constructor for NetStore and Delivery, used in Simulations, accepting any NetStore.RequestFunc
+func newNetStoreAndDeliveryWithRequestFunc(ctx *adapters.ServiceContext, bucket *sync.Map, rf network.RequestFunc) (*network.BzzAddr, *storage.NetStore, *Delivery, func(), error) {
+	addr := network.NewAddr(ctx.Config.Node())
+
+	netStore, delivery, cleanup, err := netStoreAndDeliveryWithAddr(ctx, bucket, addr)
+	if err != nil {
+		return nil, nil, nil, nil, err
+	}
+
+	netStore.NewNetFetcherFunc = network.NewFetcherFactory(rf, true).New
+
+	return addr, netStore, delivery, cleanup, nil
+}
+
+func netStoreAndDeliveryWithAddr(ctx *adapters.ServiceContext, bucket *sync.Map, addr *network.BzzAddr) (*storage.NetStore, *Delivery, func(), error) {
+	n := ctx.Config.Node()
+
+	store, datadir, err := createTestLocalStorageForID(n.ID(), addr)
+	if *useMockStore {
+		store, datadir, err = createMockStore(mockmem.NewGlobalStore(), n.ID(), addr)
+	}
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	localStore := store.(*storage.LocalStore)
+	netStore, err := storage.NewNetStore(localStore, nil)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	fileStore := storage.NewFileStore(netStore, storage.NewFileStoreParams())
+
+	kad := network.NewKademlia(addr.Over(), network.NewKadParams())
+	delivery := NewDelivery(kad, netStore)
+
+	bucket.Store(bucketKeyStore, store)
+	bucket.Store(bucketKeyDB, netStore)
+	bucket.Store(bucketKeyDelivery, delivery)
+	bucket.Store(bucketKeyFileStore, fileStore)
+
+	cleanup := func() {
+		netStore.Close()
+		os.RemoveAll(datadir)
+	}
+
+	return netStore, delivery, cleanup, nil
+}
+
 func newStreamerTester(registryOptions *RegistryOptions) (*p2ptest.ProtocolTester, *Registry, *storage.LocalStore, func(), error) {
 	// setup
 	addr := network.RandomAddr() // tested peers peer address

swarm/network/stream/delivery.go

@@ -259,13 +259,6 @@ func (d *Delivery) RequestFromPeers(ctx context.Context, req *network.Request) (
 			if sp == nil {
 				return true
 			}
-			// nodes that do not provide stream protocol
-			// should not be requested, e.g. bootnodes
-			if !p.HasCap("stream") {
-				// TODO: if we have no errors, delete this if
-				log.Error("Delivery.RequestFromPeers: peer doesn't have stream cap. we should have returned at sp == nil")
-				return true
-			}
 			spID = &id
 			return false
 		})

swarm/network/stream/delivery_test.go

@@ -21,7 +21,6 @@ import (
 	"context"
 	"errors"
 	"fmt"
-	"os"
 	"sync"
 	"sync/atomic"
 	"testing"
@@ -285,7 +284,7 @@ func TestRequestFromPeers(t *testing.T) {
 	addr := network.RandomAddr()
 	to := network.NewKademlia(addr.OAddr, network.NewKadParams())
 	delivery := NewDelivery(to, nil)
-	protocolsPeer := protocols.NewPeer(p2p.NewPeer(dummyPeerID, "dummy", []p2p.Cap{{Name: "stream"}}), nil, nil)
+	protocolsPeer := protocols.NewPeer(p2p.NewPeer(dummyPeerID, "dummy", nil), nil, nil)
 	peer := network.NewPeer(&network.BzzPeer{
 		BzzAddr:   network.RandomAddr(),
 		LightNode: false,
@@ -457,26 +456,10 @@ func TestDeliveryFromNodes(t *testing.T) {
 func testDeliveryFromNodes(t *testing.T, nodes, chunkCount int, skipCheck bool) {
 	sim := simulation.New(map[string]simulation.ServiceFunc{
 		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-			node := ctx.Config.Node()
+			addr, netStore, delivery, clean, err := newNetStoreAndDelivery(ctx, bucket)
-			addr := network.NewAddr(node)
-			store, datadir, err := createTestLocalStorageForID(node.ID(), addr)
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			cleanup = func() {
-				os.RemoveAll(datadir)
-				store.Close()
-			}
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				SkipCheck: skipCheck,
@@ -485,11 +468,12 @@ func testDeliveryFromNodes(t *testing.T, nodes, chunkCount int, skipCheck bool)
 			}, nil)
 			bucket.Store(bucketKeyRegistry, r)
 
-			fileStore := storage.NewFileStore(netStore, storage.NewFileStoreParams())
+			cleanup = func() {
-			bucket.Store(bucketKeyFileStore, fileStore)
+				r.Close()
+				clean()
+			}
 
 			return r, cleanup, nil
-
 		},
 	})
 	defer sim.Close()
@@ -644,25 +628,10 @@ func BenchmarkDeliveryFromNodesWithCheck(b *testing.B) {
 func benchmarkDeliveryFromNodes(b *testing.B, nodes, chunkCount int, skipCheck bool) {
 	sim := simulation.New(map[string]simulation.ServiceFunc{
 		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-			node := ctx.Config.Node()
+			addr, netStore, delivery, clean, err := newNetStoreAndDelivery(ctx, bucket)
-			addr := network.NewAddr(node)
-			store, datadir, err := createTestLocalStorageForID(node.ID(), addr)
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			cleanup = func() {
-				os.RemoveAll(datadir)
-				store.Close()
-			}
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				SkipCheck:       skipCheck,
@@ -670,12 +639,14 @@ func benchmarkDeliveryFromNodes(b *testing.B, nodes, chunkCount int, skipCheck b
 				Retrieval:       RetrievalDisabled,
 				SyncUpdateDelay: 0,
 			}, nil)
+			bucket.Store(bucketKeyRegistry, r)
 
-			fileStore := storage.NewFileStore(netStore, storage.NewFileStoreParams())
+			cleanup = func() {
-			bucket.Store(bucketKeyFileStore, fileStore)
+				r.Close()
+				clean()
+			}
 
 			return r, cleanup, nil
-
 		},
 	})
 	defer sim.Close()

swarm/network/stream/intervals_test.go

@@ -21,7 +21,6 @@ import (
 	"encoding/binary"
 	"errors"
 	"fmt"
-	"os"
 	"sync"
 	"sync/atomic"
 	"testing"
@@ -31,7 +30,6 @@ import (
 	"github.com/ethereum/go-ethereum/node"
 	"github.com/ethereum/go-ethereum/p2p/enode"
 	"github.com/ethereum/go-ethereum/p2p/simulations/adapters"
-	"github.com/ethereum/go-ethereum/swarm/network"
 	"github.com/ethereum/go-ethereum/swarm/network/simulation"
 	"github.com/ethereum/go-ethereum/swarm/state"
 	"github.com/ethereum/go-ethereum/swarm/storage"
@@ -62,26 +60,11 @@ func testIntervals(t *testing.T, live bool, history *Range, skipCheck bool) {
 	externalStreamMaxKeys := uint64(100)
 
 	sim := simulation.New(map[string]simulation.ServiceFunc{
-		"intervalsStreamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
+		"intervalsStreamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (node.Service, func(), error) {
-			n := ctx.Config.Node()
+			addr, netStore, delivery, clean, err := newNetStoreAndDelivery(ctx, bucket)
-			addr := network.NewAddr(n)
-			store, datadir, err := createTestLocalStorageForID(n.ID(), addr)
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			cleanup = func() {
-				store.Close()
-				os.RemoveAll(datadir)
-			}
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				Retrieval: RetrievalDisabled,
@@ -97,11 +80,12 @@ func testIntervals(t *testing.T, live bool, history *Range, skipCheck bool) {
 				return newTestExternalServer(t, externalStreamSessionAt, externalStreamMaxKeys, nil), nil
 			})
 
-			fileStore := storage.NewFileStore(localStore, storage.NewFileStoreParams())
+			cleanup := func() {
-			bucket.Store(bucketKeyFileStore, fileStore)
+				r.Close()
+				clean()
+			}
 
 			return r, cleanup, nil
-
 		},
 	})
 	defer sim.Close()

swarm/network/stream/snapshot_retrieval_test.go

@@ -18,7 +18,6 @@ package stream
 import (
 	"context"
 	"fmt"
-	"os"
 	"sync"
 	"testing"
 	"time"
@@ -27,7 +26,6 @@ import (
 	"github.com/ethereum/go-ethereum/p2p/enode"
 	"github.com/ethereum/go-ethereum/p2p/simulations/adapters"
 	"github.com/ethereum/go-ethereum/swarm/log"
-	"github.com/ethereum/go-ethereum/swarm/network"
 	"github.com/ethereum/go-ethereum/swarm/network/simulation"
 	"github.com/ethereum/go-ethereum/swarm/state"
 	"github.com/ethereum/go-ethereum/swarm/storage"
@@ -105,26 +103,11 @@ func TestRetrieval(t *testing.T) {
 }
 
 var retrievalSimServiceMap = map[string]simulation.ServiceFunc{
-	"streamer": retrievalStreamerFunc,
+	"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-}
+		addr, netStore, delivery, clean, err := newNetStoreAndDelivery(ctx, bucket)
-
-func retrievalStreamerFunc(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-	n := ctx.Config.Node()
-	addr := network.NewAddr(n)
-	store, datadir, err := createTestLocalStorageForID(n.ID(), addr)
 		if err != nil {
 			return nil, nil, err
 		}
-	bucket.Store(bucketKeyStore, store)
-
-	localStore := store.(*storage.LocalStore)
-	netStore, err := storage.NewNetStore(localStore, nil)
-	if err != nil {
-		return nil, nil, err
-	}
-	kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-	delivery := NewDelivery(kad, netStore)
-	netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
 
 		r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 			Retrieval:       RetrievalEnabled,
@@ -132,16 +115,13 @@ func retrievalStreamerFunc(ctx *adapters.ServiceContext, bucket *sync.Map) (s no
 			SyncUpdateDelay: 3 * time.Second,
 		}, nil)
 
-	fileStore := storage.NewFileStore(netStore, storage.NewFileStoreParams())
-	bucket.Store(bucketKeyFileStore, fileStore)
-
 		cleanup = func() {
-		os.RemoveAll(datadir)
-		netStore.Close()
 			r.Close()
+			clean()
 		}
 
 		return r, cleanup, nil
+	},
 }
 
 /*

swarm/network/stream/snapshot_sync_test.go

@@ -107,25 +107,11 @@ func TestSyncingViaGlobalSync(t *testing.T) {
 }
 
 var simServiceMap = map[string]simulation.ServiceFunc{
-	"streamer": streamerFunc,
+	"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-}
+		addr, netStore, delivery, clean, err := newNetStoreAndDeliveryWithRequestFunc(ctx, bucket, dummyRequestFromPeers)
-
-func streamerFunc(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-	n := ctx.Config.Node()
-	addr := network.NewAddr(n)
-	store, datadir, err := createTestLocalStorageForID(n.ID(), addr)
 		if err != nil {
 			return nil, nil, err
 		}
-	bucket.Store(bucketKeyStore, store)
-	localStore := store.(*storage.LocalStore)
-	netStore, err := storage.NewNetStore(localStore, nil)
-	if err != nil {
-		return nil, nil, err
-	}
-	kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-	delivery := NewDelivery(kad, netStore)
-	netStore.NewNetFetcherFunc = network.NewFetcherFactory(dummyRequestFromPeers, true).New
 
 		r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 			Retrieval:       RetrievalDisabled,
@@ -136,13 +122,12 @@ func streamerFunc(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Servic
 		bucket.Store(bucketKeyRegistry, r)
 
 		cleanup = func() {
-		os.RemoveAll(datadir)
-		netStore.Close()
 			r.Close()
+			clean()
 		}
 
 		return r, cleanup, nil
-
+	},
 }
 
 func testSyncingViaGlobalSync(t *testing.T, chunkCount int, nodeCount int) {

swarm/network/stream/stream.go

@@ -929,3 +929,27 @@ func (api *API) SubscribeStream(peerId enode.ID, s Stream, history *Range, prior
 func (api *API) UnsubscribeStream(peerId enode.ID, s Stream) error {
 	return api.streamer.Unsubscribe(peerId, s)
 }
+
+/*
+GetPeerSubscriptions is a API function which allows to query a peer for stream subscriptions it has.
+It can be called via RPC.
+It returns a map of node IDs with an array of string representations of Stream objects.
+*/
+func (api *API) GetPeerSubscriptions() map[string][]string {
+	//create the empty map
+	pstreams := make(map[string][]string)
+	//iterate all streamer peers
+	for id, p := range api.streamer.peers {
+		var streams []string
+		//every peer has a map of stream servers
+		//every stream server represents a subscription
+		for s := range p.servers {
+			//append the string representation of the stream
+			//to the list for this peer
+			streams = append(streams, s.String())
+		}
+		//set the array of stream servers to the map
+		pstreams[id.String()] = streams
+	}
+	return pstreams
+}

swarm/network/stream/streamer_test.go

@@ -22,14 +22,20 @@ import (
 	"errors"
 	"fmt"
 	"strconv"
+	"strings"
+	"sync"
 	"testing"
 	"time"
 
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/node"
 	"github.com/ethereum/go-ethereum/p2p/enode"
+	"github.com/ethereum/go-ethereum/p2p/simulations/adapters"
 	p2ptest "github.com/ethereum/go-ethereum/p2p/testing"
 	"github.com/ethereum/go-ethereum/swarm/network"
+	"github.com/ethereum/go-ethereum/swarm/network/simulation"
+	"github.com/ethereum/go-ethereum/swarm/state"
 	"golang.org/x/crypto/sha3"
 )
 
@@ -1105,7 +1111,6 @@ func TestRequestPeerSubscriptions(t *testing.T) {
 			}
 		}
 	}
-
 	// print some output
 	for p, subs := range fakeSubscriptions {
 		log.Debug(fmt.Sprintf("Peer %s has the following fake subscriptions: ", p))
@@ -1114,3 +1119,217 @@ func TestRequestPeerSubscriptions(t *testing.T) {
 		}
 	}
 }
+
+// TestGetSubscriptions is a unit test for the api.GetPeerSubscriptions() function
+func TestGetSubscriptions(t *testing.T) {
+	// create an amount of dummy peers
+	testPeerCount := 8
+	// every peer will have this amount of dummy servers
+	testServerCount := 4
+	// the peerMap which will store this data for the registry
+	peerMap := make(map[enode.ID]*Peer)
+	// create the registry
+	r := &Registry{}
+	api := NewAPI(r)
+	// call once, at this point should be empty
+	regs := api.GetPeerSubscriptions()
+	if len(regs) != 0 {
+		t.Fatal("Expected subscription count to be 0, but it is not")
+	}
+
+	// now create a number of dummy servers for each node
+	for i := 0; i < testPeerCount; i++ {
+		addr := network.RandomAddr()
+		id := addr.ID()
+		p := &Peer{}
+		p.servers = make(map[Stream]*server)
+		for k := 0; k < testServerCount; k++ {
+			s := Stream{
+				Name: strconv.Itoa(k),
+				Key:  "",
+				Live: false,
+			}
+			p.servers[s] = &server{}
+		}
+		peerMap[id] = p
+	}
+	r.peers = peerMap
+
+	// call the subscriptions again
+	regs = api.GetPeerSubscriptions()
+	// count how many (fake) subscriptions there are
+	cnt := 0
+	for _, reg := range regs {
+		for range reg {
+			cnt++
+		}
+	}
+	// check expected value
+	expectedCount := testPeerCount * testServerCount
+	if cnt != expectedCount {
+		t.Fatalf("Expected %d subscriptions, but got %d", expectedCount, cnt)
+	}
+}
+
+/*
+TestGetSubscriptionsRPC sets up a simulation network of `nodeCount` nodes,
+starts the simulation, waits for SyncUpdateDelay in order to kick off
+stream registration, then tests that there are subscriptions.
+*/
+func TestGetSubscriptionsRPC(t *testing.T) {
+	// arbitrarily set to 4
+	nodeCount := 4
+	// run with more nodes if `longrunning` flag is set
+	if *longrunning {
+		nodeCount = 64
+	}
+	// set the syncUpdateDelay for sync registrations to start
+	syncUpdateDelay := 200 * time.Millisecond
+	// holds the msg code for SubscribeMsg
+	var subscribeMsgCode uint64
+	var ok bool
+	var expectedMsgCount = 0
+
+	// this channel signalizes that the expected amount of subscriptiosn is done
+	allSubscriptionsDone := make(chan struct{})
+	lock := sync.RWMutex{}
+	// after the test, we need to reset the subscriptionFunc to the default
+	defer func() { subscriptionFunc = doRequestSubscription }()
+
+	// we use this subscriptionFunc for this test: just increases count and calls the actual subscription
+	subscriptionFunc = func(r *Registry, p *network.Peer, bin uint8, subs map[enode.ID]map[Stream]struct{}) bool {
+		lock.Lock()
+		expectedMsgCount++
+		lock.Unlock()
+		doRequestSubscription(r, p, bin, subs)
+		return true
+	}
+	// create a standard sim
+	sim := simulation.New(map[string]simulation.ServiceFunc{
+		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
+			addr, netStore, delivery, clean, err := newNetStoreAndDeliveryWithRequestFunc(ctx, bucket, dummyRequestFromPeers)
+			if err != nil {
+				return nil, nil, err
+			}
+
+			// configure so that sync registrations actually happen
+			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
+				Retrieval:       RetrievalEnabled,
+				Syncing:         SyncingAutoSubscribe, //enable sync registrations
+				SyncUpdateDelay: syncUpdateDelay,
+			}, nil)
+
+			// get the SubscribeMsg code
+			subscribeMsgCode, ok = r.GetSpec().GetCode(SubscribeMsg{})
+			if !ok {
+				t.Fatal("Message code for SubscribeMsg not found")
+			}
+
+			cleanup = func() {
+				r.Close()
+				clean()
+			}
+
+			return r, cleanup, nil
+		},
+	})
+	defer sim.Close()
+
+	ctx, cancelSimRun := context.WithTimeout(context.Background(), 1*time.Minute)
+	defer cancelSimRun()
+
+	// upload a snapshot
+	err := sim.UploadSnapshot(fmt.Sprintf("testing/snapshot_%d.json", nodeCount))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// setup the filter for SubscribeMsg
+	msgs := sim.PeerEvents(
+		context.Background(),
+		sim.NodeIDs(),
+		simulation.NewPeerEventsFilter().ReceivedMessages().Protocol("stream").MsgCode(subscribeMsgCode),
+	)
+
+	// strategy: listen to all SubscribeMsg events; after every event we wait
+	// if after `waitDuration` no more messages are being received, we assume the
+	// subscription phase has terminated!
+
+	// the loop in this go routine will either wait for new message events
+	// or times out after 1 second, which signals that we are not receiving
+	// any new subscriptions any more
+	go func() {
+		//for long running sims, waiting 1 sec will not be enough
+		waitDuration := time.Duration(nodeCount/16) * time.Second
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case m := <-msgs: // just reset the loop
+				if m.Error != nil {
+					log.Error("stream message", "err", m.Error)
+					continue
+				}
+				log.Trace("stream message", "node", m.NodeID, "peer", m.PeerID)
+			case <-time.After(waitDuration):
+				// one second passed, don't assume more subscriptions
+				allSubscriptionsDone <- struct{}{}
+				log.Info("All subscriptions received")
+				return
+
+			}
+		}
+	}()
+
+	//run the simulation
+	result := sim.Run(ctx, func(ctx context.Context, sim *simulation.Simulation) error {
+		log.Info("Simulation running")
+		nodes := sim.Net.Nodes
+
+		//wait until all subscriptions are done
+		select {
+		case <-allSubscriptionsDone:
+		case <-ctx.Done():
+			t.Fatal("Context timed out")
+		}
+
+		log.Debug("Expected message count: ", "expectedMsgCount", expectedMsgCount)
+		//now iterate again, this time we call each node via RPC to get its subscriptions
+		realCount := 0
+		for _, node := range nodes {
+			//create rpc client
+			client, err := node.Client()
+			if err != nil {
+				t.Fatalf("create node 1 rpc client fail: %v", err)
+			}
+
+			//ask it for subscriptions
+			pstreams := make(map[string][]string)
+			err = client.Call(&pstreams, "stream_getPeerSubscriptions")
+			if err != nil {
+				t.Fatal(err)
+			}
+			//length of the subscriptions can not be smaller than number of peers
+			log.Debug("node subscriptions", "node", node.String())
+			for p, ps := range pstreams {
+				log.Debug("... with", "peer", p)
+				for _, s := range ps {
+					log.Debug(".......", "stream", s)
+					// each node also has subscriptions to RETRIEVE_REQUEST streams,
+					// we need to ignore those, we are only counting SYNC streams
+					if !strings.HasPrefix(s, "RETRIEVE_REQUEST") {
+						realCount++
+					}
+				}
+			}
+		}
+		// every node is mutually subscribed to each other, so the actual count is half of it
+		if realCount/2 != expectedMsgCount {
+			return fmt.Errorf("Real subscriptions and expected amount don't match; real: %d, expected: %d", realCount/2, expectedMsgCount)
+		}
+		return nil
+	})
+	if result.Error != nil {
+		t.Fatal(result.Error)
+	}
+}

swarm/network/stream/syncer_test.go

@@ -22,7 +22,6 @@ import (
 	"fmt"
 	"io/ioutil"
 	"math"
-	"os"
 	"sync"
 	"sync/atomic"
 	"testing"
@@ -38,7 +37,6 @@ import (
 	"github.com/ethereum/go-ethereum/swarm/state"
 	"github.com/ethereum/go-ethereum/swarm/storage"
 	"github.com/ethereum/go-ethereum/swarm/storage/mock"
-	mockmem "github.com/ethereum/go-ethereum/swarm/storage/mock/mem"
 	"github.com/ethereum/go-ethereum/swarm/testutil"
 )
 
@@ -73,38 +71,14 @@ func testSyncBetweenNodes(t *testing.T, nodes, chunkCount int, skipCheck bool, p
 
 	sim := simulation.New(map[string]simulation.ServiceFunc{
 		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-			var store storage.ChunkStore
+			addr := network.NewAddr(ctx.Config.Node())
-			var datadir string
-
-			node := ctx.Config.Node()
-			addr := network.NewAddr(node)
 			//hack to put addresses in same space
 			addr.OAddr[0] = byte(0)
 
-			if *useMockStore {
+			netStore, delivery, clean, err := newNetStoreAndDeliveryWithBzzAddr(ctx, bucket, addr)
-				store, datadir, err = createMockStore(mockmem.NewGlobalStore(), node.ID(), addr)
-			} else {
-				store, datadir, err = createTestLocalStorageForID(node.ID(), addr)
-			}
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			cleanup = func() {
-				store.Close()
-				os.RemoveAll(datadir)
-			}
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-			bucket.Store(bucketKeyDB, netStore)
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
-
-			bucket.Store(bucketKeyDelivery, delivery)
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				Retrieval: RetrievalDisabled,
@@ -112,11 +86,12 @@ func testSyncBetweenNodes(t *testing.T, nodes, chunkCount int, skipCheck bool, p
 				SkipCheck: skipCheck,
 			}, nil)
 
-			fileStore := storage.NewFileStore(netStore, storage.NewFileStoreParams())
+			cleanup = func() {
-			bucket.Store(bucketKeyFileStore, fileStore)
+				r.Close()
+				clean()
+			}
 
 			return r, cleanup, nil
-
 		},
 	})
 	defer sim.Close()
@@ -251,44 +226,26 @@ func TestSameVersionID(t *testing.T) {
 	v := uint(1)
 	sim := simulation.New(map[string]simulation.ServiceFunc{
 		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-			var store storage.ChunkStore
+			addr, netStore, delivery, clean, err := newNetStoreAndDelivery(ctx, bucket)
-			var datadir string
-
-			node := ctx.Config.Node()
-			addr := network.NewAddr(node)
-
-			store, datadir, err = createTestLocalStorageForID(node.ID(), addr)
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			cleanup = func() {
-				store.Close()
-				os.RemoveAll(datadir)
-			}
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-			bucket.Store(bucketKeyDB, netStore)
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
-
-			bucket.Store(bucketKeyDelivery, delivery)
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				Retrieval: RetrievalDisabled,
 				Syncing:   SyncingAutoSubscribe,
 			}, nil)
+			bucket.Store(bucketKeyRegistry, r)
+
 			//assign to each node the same version ID
 			r.spec.Version = v
 
-			bucket.Store(bucketKeyRegistry, r)
+			cleanup = func() {
+				r.Close()
+				clean()
+			}
 
 			return r, cleanup, nil
-
 		},
 	})
 	defer sim.Close()
@@ -333,46 +290,27 @@ func TestDifferentVersionID(t *testing.T) {
 	v := uint(0)
 	sim := simulation.New(map[string]simulation.ServiceFunc{
 		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-			var store storage.ChunkStore
+			addr, netStore, delivery, clean, err := newNetStoreAndDelivery(ctx, bucket)
-			var datadir string
-
-			node := ctx.Config.Node()
-			addr := network.NewAddr(node)
-
-			store, datadir, err = createTestLocalStorageForID(node.ID(), addr)
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			cleanup = func() {
-				store.Close()
-				os.RemoveAll(datadir)
-			}
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-			bucket.Store(bucketKeyDB, netStore)
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(delivery.RequestFromPeers, true).New
-
-			bucket.Store(bucketKeyDelivery, delivery)
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				Retrieval: RetrievalDisabled,
 				Syncing:   SyncingAutoSubscribe,
 			}, nil)
+			bucket.Store(bucketKeyRegistry, r)
 
 			//increase the version ID for each node
 			v++
 			r.spec.Version = v
 
-			bucket.Store(bucketKeyRegistry, r)
+			cleanup = func() {
+				r.Close()
+				clean()
+			}
 
 			return r, cleanup, nil
-
 		},
 	})
 	defer sim.Close()

swarm/network/stream/testing/snapshot_4.json

@@ -0,0 +1 @@
+{"nodes":[{"node":{"config":{"id":"73d6ad4a75069dced660fa4cb98143ee5573df7cb15d9a295acf1655e9683384","private_key":"e567b7d9c554e5102cdc99b6523bace02dbb8951415c8816d82ba2d2e97fa23b","name":"node01","services":["bzz","pss"],"enable_msg_events":false,"port":0},"up":true}},{"node":{"config":{"id":"6e8da86abb894ab35044c8c455147225df96cab498da067a118f1fb9a417f9e3","private_key":"c7526db70acd02f36d3b201ef3e1d85e38c52bee6931453213dbc5edec4d0976","name":"node02","services":["bzz","pss"],"enable_msg_events":false,"port":0},"up":true}},{"node":{"config":{"id":"8a1eb78ff13df318e7f8116dffee98cd7d9905650fa53f16766b754a63f387ac","private_key":"61b5728f59bc43080c3b8eb0458fb30d7723e2747355b6dc980f35f3ed431199","name":"node03","services":["bzz","pss"],"enable_msg_events":false,"port":0},"up":true}},{"node":{"config":{"id":"d7768334f79d626adb433f44b703a818555e3331056036ef3f8d1282586bf044","private_key":"075b07c29ceac4ffa2a114afd67b21dfc438126bc169bf7c154be6d81d86ed38","name":"node04","services":["bzz","pss"],"enable_msg_events":false,"port":0},"up":true}}],"conns":[{"one":"6e8da86abb894ab35044c8c455147225df96cab498da067a118f1fb9a417f9e3","other":"8a1eb78ff13df318e7f8116dffee98cd7d9905650fa53f16766b754a63f387ac","up":true},{"one":"73d6ad4a75069dced660fa4cb98143ee5573df7cb15d9a295acf1655e9683384","other":"6e8da86abb894ab35044c8c455147225df96cab498da067a118f1fb9a417f9e3","up":true},{"one":"8a1eb78ff13df318e7f8116dffee98cd7d9905650fa53f16766b754a63f387ac","other":"d7768334f79d626adb433f44b703a818555e3331056036ef3f8d1282586bf044","up":true}]}

swarm/network/stream/visualized_snapshot_sync_sim_test.go

@@ -24,7 +24,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"os"
 	"sync"
 	"testing"
 	"time"
@@ -37,7 +36,6 @@ import (
 	"github.com/ethereum/go-ethereum/p2p/simulations/adapters"
 	"github.com/ethereum/go-ethereum/rlp"
 	"github.com/ethereum/go-ethereum/swarm/log"
-	"github.com/ethereum/go-ethereum/swarm/network"
 	"github.com/ethereum/go-ethereum/swarm/network/simulation"
 	"github.com/ethereum/go-ethereum/swarm/state"
 	"github.com/ethereum/go-ethereum/swarm/storage"
@@ -169,21 +167,10 @@ func TestSnapshotSyncWithServer(t *testing.T) {
 
 	sim := simulation.New(map[string]simulation.ServiceFunc{
 		"streamer": func(ctx *adapters.ServiceContext, bucket *sync.Map) (s node.Service, cleanup func(), err error) {
-			n := ctx.Config.Node()
+			addr, netStore, delivery, clean, err := newNetStoreAndDeliveryWithRequestFunc(ctx, bucket, dummyRequestFromPeers)
-			addr := network.NewAddr(n)
-			store, datadir, err := createTestLocalStorageForID(n.ID(), addr)
 			if err != nil {
 				return nil, nil, err
 			}
-			bucket.Store(bucketKeyStore, store)
-			localStore := store.(*storage.LocalStore)
-			netStore, err := storage.NewNetStore(localStore, nil)
-			if err != nil {
-				return nil, nil, err
-			}
-			kad := network.NewKademlia(addr.Over(), network.NewKadParams())
-			delivery := NewDelivery(kad, netStore)
-			netStore.NewNetFetcherFunc = network.NewFetcherFactory(dummyRequestFromPeers, true).New
 
 			r := NewRegistry(addr.ID(), delivery, netStore, state.NewInmemoryStore(), &RegistryOptions{
 				Retrieval:       RetrievalDisabled,
@@ -199,9 +186,8 @@ func TestSnapshotSyncWithServer(t *testing.T) {
 			bucket.Store(bucketKeyRegistry, tr)
 
 			cleanup = func() {
-				netStore.Close()
 				tr.Close()
-				os.RemoveAll(datadir)
+				clean()
 			}
 
 			return tr, cleanup, nil

trie/database.go

@@ -679,6 +679,7 @@ func (db *Database) Commit(node common.Hash, report bool) error {
 		}
 		if batch.ValueSize() > ethdb.IdealBatchSize {
 			if err := batch.Write(); err != nil {
+				db.lock.RUnlock()
 				return err
 			}
 			batch.Reset()