forks/go-ethereum
1
0
Fork 1
mirror of https://github.com/ethereum/go-ethereum.git synced 2026-07-25 06:06:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

SWORM - swarm poc 0.2 stability RC3

Browse source 
rebase on current develop branch 2016-07-09

swarm/services/ens:
* Rework ENS functionality to use the current spec.
* Refactor ens.go to set up sessions as needed
* Refactoring to support independent registrar and personal resolver contracts
* fixed tests

swarm/examples/album: improve upload UX
* No more unhandled errors and loose ends in the UI.
* Forward/Back buttons work as undo/redo
* all long operations display an informative modal dialog
* minor problems with cursor movements fixed
* reorder labels
* throbber as default image
* change browser history after delete or upload photo

swarm/storage:
* optimise splitter
* simplify reader
* benchmarks, tests improved
* simplify IO, remove chunkreader.go and io.SectionReader clone
* simplify error/timeout handling
* port pyramid splitter by karalabe, adapt to Splitter/DPA interface, rework params etc
* make NetStore.Put -> cloud.Store synchronous
* chunker join fix process leak and keep parallelisation limited to depth 1
* reset the base hash to SHA3. !!hard fork = no backward compatibility :)
* SectionReader -> LazySectionReader: Size method signature change
* introduce abort channel to joiner to fix process leak
* add abort channel   context to all manifest retrieval methods
* global waitgroup fixes intermittent upload test failures due to unfinished storage of chunks
* streamline joiner logic, contexts now unique to each readAt call
* LazyChunkReader seeker complains about missing size only if whence=2
* chunker join fix process leak and keep parallelisation limited to depth 1

swarm/network/kademlia:
* simplify code
* now really fix prox limit adjustment + tests and comments
* simplify and make readable findclosest algo code + comments
* reset initial time interval settings for kaddb findbest
* fix bucket replace scheme to optimise stability and availability
* absolute idle peers are disconnected after maxIdleInterval
* fix index out of range when deleting last idle peer from kaddb
* unwanted peers (due to full kad bucket) are now properly dropped with ErrUnwanted
* improve logging and reorg in kademlia
* simplify and fix code that deletes expired/unconnectable nodes
* kaddb.findBest does not get stuck on empty row but finds an actual missing node
* must replace node if bucket is full otherwise nodes will get stuck on empty rows
* kademlia: timer fix
* remove logging the state when syncing (resulting in deadlock)
* improved logs, fix potential deadlock in String()
* native go time marshalling
* tests fixed

swarm/network:
* log node address consistently in syncdb
* syncer logs queue cardinalities properly in syncUnSyncedKeys loop
* fix hive stop issue leading to send on closed chan + minor logging fixes
* better logging
* hive interface change
* fix process leak by adding select to state.synced <- false

swarm/cmd: swarm control cli improvements
* no more alias, swarm  is executable so it can be called via ssh
* environment vars now set to default, no need to preconfigure
* local and remote hive monitoring
* gethup.sh script now merged into swarm script and nice modularised
* simplify bash code and fix e2e tests in swarm/test
* stop method falls back to kill -9 after 10s
* enode method now supplies ip addr via ipecho request
* execute, options, rawoptions, setup, create-account addpeers and hive subcommands
* update-src -> update
* remote-update-scripts, remote-update-bin and remote-run
* local and remote monitoring of kademlia
* extensive documentation in  swarm/cmd/README.md
* add cleanlog, cleanbzz, update-src, remote-update-scripts, remote-update-bin, remote-run
* for remote binary update, take scripts from swarm/cmd/swarm
* raise default maxpeers to 40
* add latency loggingo to upload and download
* add checkdownload and checkaccess subcommands
* add mem/cpu/disk-info commands
* include randomfile cmd from test
* add enode, connect subcommands, improve startup (only one round needed)
* TODO: simplify tests using new checks

swarm/api/http: server handler: check protocol substring length to fix slice out of bounds crash

swarm/api:
* fix filesystem API - upload/Modify tests
* simplify downloader code and fix process leak
* change KAD defaults bucketsize 4, minproxbinsize 2
* add back final slash to paths in manifest matching
* fix filesystem api tests
This commit is contained in:
zelig 2016-05-30 11:15:48 +02:00
parent 81111e112c
commit 075eb6ad99
69 changed files with 3298 additions and 3476 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.gitignore vendored
View file

@ -31,3 +31,10 @@ Godeps/_workspace/bin
# travis # travis
profile.tmp profile.tmp
profile.cov profile.cov
# vagrant
.vagrant
abigen
geth

15
accounts/abi/bind/ext.go
View file

@ -31,9 +31,9 @@ func DefaultDeployOptions() *DeployOptions {
// implemented by eth.APIBackend // implemented by eth.APIBackend
type Backend interface { type Backend interface {
GetTxReceipt(txhash common.Hash) *types.Receipt GetTxReceipt(txhash common.Hash) (map[string]interface{}, error)
CodeAt(address common.Address) string BalanceAt(address common.Address) (*big.Int, error)
BalanceAt(address common.Address) *big.Int CodeAt(address common.Address) (string, error)
ContractBackend ContractBackend
} }
@ -64,7 +64,7 @@ DEPLOY:
receiptQueryTimer = time.NewTimer(0).C receiptQueryTimer = time.NewTimer(0).C
case <-receiptQueryTimer: case <-receiptQueryTimer:
receipt := backend.GetTxReceipt(txhash) receipt, _ := backend.GetTxReceipt(txhash)
receiptQueries++ receiptQueries++
if receipt == nil { if receipt == nil {
if receiptQueries == opt.MaxReceiptQueryAttempts { if receiptQueries == opt.MaxReceiptQueryAttempts {
@ -80,7 +80,7 @@ DEPLOY:
continue DEPLOY continue DEPLOY
} }
contractAddr = receipt.ContractAddress contractAddr = receipt["contractAddress"].(common.Address)
glog.V(logger.Detail).Infof("new chequebook contract mined at %v (owner: %v)", contractAddr.Hex(), deployTransactor.From.Hex()) glog.V(logger.Detail).Infof("new chequebook contract mined at %v (owner: %v)", contractAddr.Hex(), deployTransactor.From.Hex())
<-time.NewTimer(opt.ConfirmationInterval).C <-time.NewTimer(opt.ConfirmationInterval).C
err = Validate(contractAddr, contractCode, backend) err = Validate(contractAddr, contractCode, backend)
@ -101,7 +101,10 @@ func Validate(contractAddr common.Address, expCode string, backend Backend) (err
if (contractAddr == common.Address{}) { if (contractAddr == common.Address{}) {
return fmt.Errorf("zero address") return fmt.Errorf("zero address")
} }
code := backend.CodeAt(contractAddr) code, err := backend.CodeAt(contractAddr)
if err != nil {
return err
}
if len(expCode) > 0 && code != expCode { if len(expCode) > 0 && code != expCode {
return fmt.Errorf("incorrect code %v:\n%v\n%v", contractAddr.Hex(), code, expCode) return fmt.Errorf("incorrect code %v:\n%v\n%v", contractAddr.Hex(), code, expCode)
} }

18
cmd/geth/accountcmd.go
View file

@ -63,7 +63,7 @@ import a private key into a new account.
It supports interactive mode, when you are prompted for password as well as It supports interactive mode, when you are prompted for password as well as
non-interactive mode where passwords are supplied via a given password file. non-interactive mode where passwords are supplied via a given password file.
Non-interactive mode is only meant for scripted use on test networks or known Non-interactive mode is only meant for scripted use on test networks or known
safe environ>>>ments. safe environments.
Make sure you remember the password you gave when creating a new account (with Make sure you remember the password you gave when creating a new account (with
either new or import). Without it you are not able to unlock your account. either new or import). Without it you are not able to unlock your account.
@ -119,7 +119,7 @@ password to file or expose in any other way.
Description: ` Description: `
ethereum account update <address> ethereum account update <address>
>>>>>>>>>>>>
Update an existing account. Update an existing account.
The account is saved in the newest version in encrypted format, you are prompted The account is saved in the newest version in encrypted format, you are prompted
@ -175,7 +175,7 @@ func accountList(ctx *cli.Context) error {
return nil return nil
} }
// tries unlocking the specifiedqqa few times. // tries unlocking the specified account a few times.
func unlockAccount(ctx *cli.Context, accman *accounts.Manager, address string, i int, passwords []string) (accounts.Account, string) { func unlockAccount(ctx *cli.Context, accman *accounts.Manager, address string, i int, passwords []string) (accounts.Account, string) {
account, err := utils.MakeAddress(accman, address) account, err := utils.MakeAddress(accman, address)
if err != nil { if err != nil {
@ -203,7 +203,7 @@ func unlockAccount(ctx *cli.Context, accman *accounts.Manager, address string, i
return accounts.Account{}, "" return accounts.Account{}, ""
} }
// getPassPhrase retrieves the password associated with an account, either fetched // getPassPhrase retrieves the passwor associated with an account, either fetched
// from a list of preloaded passphrases, or requested interactively from the user. // from a list of preloaded passphrases, or requested interactively from the user.
func getPassPhrase(prompt string, confirmation bool, i int, passwords []string) string { func getPassPhrase(prompt string, confirmation bool, i int, passwords []string) string {
// If a list of passwords was supplied, retrieve from them // If a list of passwords was supplied, retrieve from them
@ -262,7 +262,7 @@ func ambiguousAddrRecovery(am *accounts.Manager, err *accounts.AmbiguousAddrErro
// accountCreate creates a new account into the keystore defined by the CLI flags. // accountCreate creates a new account into the keystore defined by the CLI flags.
func accountCreate(ctx *cli.Context) error { func accountCreate(ctx *cli.Context) error {
accman := utils.MakeAccountManager(ctx) accman := utils.MakeAccountManager(ctx)
password := utils.GetPassPhrase("Your new account is locked with a password. Please give a password. Do not forget this password.", true, 0, utils.MakePasswordList(ctx)) password := getPassPhrase("Your new account is locked with a password. Please give a password. Do not forget this password.", true, 0, utils.MakePasswordList(ctx))
account, err := accman.NewAccount(password) account, err := accman.NewAccount(password)
if err != nil { if err != nil {
@ -280,8 +280,8 @@ func accountUpdate(ctx *cli.Context) error {
} }
accman := utils.MakeAccountManager(ctx) accman := utils.MakeAccountManager(ctx)
account, oldPassword := utils.UnlockAccount(accman, ctx.Args().First(), 0, nil) account, oldPassword := unlockAccount(ctx, accman, ctx.Args().First(), 0, nil)
newPassword := utils.GetPassPhrase("Please give a new password. Do not forget this password.", true, 0, nil) newPassword := getPassPhrase("Please give a new password. Do not forget this password.", true, 0, nil)
if err := accman.Update(account, oldPassword, newPassword); err != nil { if err := accman.Update(account, oldPassword, newPassword); err != nil {
utils.Fatalf("Could not update the account: %v", err) utils.Fatalf("Could not update the account: %v", err)
} }
@ -299,7 +299,7 @@ func importWallet(ctx *cli.Context) error {
} }
accman := utils.MakeAccountManager(ctx) accman := utils.MakeAccountManager(ctx)
passphrase := utils.GetPassPhrase("", false, 0, utils.MakePasswordList(ctx)) passphrase := getPassPhrase("", false, 0, utils.MakePasswordList(ctx))
acct, err := accman.ImportPreSaleKey(keyJson, passphrase) acct, err := accman.ImportPreSaleKey(keyJson, passphrase)
if err != nil { if err != nil {
@ -319,7 +319,7 @@ func accountImport(ctx *cli.Context) error {
utils.Fatalf("Failed to load the private key: %v", err) utils.Fatalf("Failed to load the private key: %v", err)
} }
accman := utils.MakeAccountManager(ctx) accman := utils.MakeAccountManager(ctx)
passphrase := utils.GetPassPhrase("Your new account is locked with a password. Please give a password. Do not forget this password.", true, 0, utils.MakePasswordList(ctx)) passphrase := getPassPhrase("Your new account is locked with a password. Please give a password. Do not forget this password.", true, 0, utils.MakePasswordList(ctx))
acct, err := accman.ImportECDSA(key, passphrase) acct, err := accman.ImportECDSA(key, passphrase)
if err != nil { if err != nil {
utils.Fatalf("Could not create the account: %v", err) utils.Fatalf("Could not create the account: %v", err)

424
cmd/geth/js.go
View file

@ -1,424 +0,0 @@
// Copyright 2015 The go-ethereum Authors
// This file is part of go-ethereum.
//
// go-ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// go-ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with go-ethereum. If not, see <http://www.gnu.org/licenses/>.
package main
import (
"fmt"
"math/big"
"os"
"os/signal"
"path/filepath"
"regexp"
"sort"
"strings"
"github.com/codegangsta/cli"
"github.com/ethereum/go-ethereum/accounts"
"github.com/ethereum/go-ethereum/cmd/utils"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/eth"
"github.com/ethereum/go-ethereum/internal/web3ext"
re "github.com/ethereum/go-ethereum/jsre"
"github.com/ethereum/go-ethereum/node"
"github.com/ethereum/go-ethereum/rpc"
"github.com/peterh/liner"
"github.com/robertkrimen/otto"
)
var (
passwordRegexp = regexp.MustCompile("personal.[nu]")
leadingSpace = regexp.MustCompile("^ ")
onlyws = regexp.MustCompile("^\\s*$")
exit = regexp.MustCompile("^\\s*exit\\s*;*\\s*$")
)
type jsre struct {
re *re.JSRE
stack *node.Node
wait chan *big.Int
ps1 string
atexit func()
corsDomain string
client rpc.Client
}
func makeCompleter(re *jsre) liner.WordCompleter {
return func(line string, pos int) (head string, completions []string, tail string) {
if len(line) == 0 || pos == 0 {
return "", nil, ""
}
// chuck data to relevant part for autocompletion, e.g. in case of nested lines eth.getBalance(eth.coinb<tab><tab>
i := 0
for i = pos - 1; i > 0; i-- {
if line[i] == '.' || (line[i] >= 'a' && line[i] <= 'z') || (line[i] >= 'A' && line[i] <= 'Z') {
continue
}
if i >= 3 && line[i] == '3' && line[i-3] == 'w' && line[i-2] == 'e' && line[i-1] == 'b' {
continue
}
i += 1
break
}
return line[:i], re.re.CompleteKeywords(line[i:pos]), line[pos:]
}
}
func newLightweightJSRE(docRoot string, client rpc.Client, datadir string, interactive bool) *jsre {
js := &jsre{ps1: "> "}
js.wait = make(chan *big.Int)
js.client = client
js.re = re.New(docRoot)
if err := js.apiBindings(); err != nil {
utils.Fatalf("Unable to initialize console - %v", err)
}
js.setupInput(datadir)
return js
}
func newJSRE(stack *node.Node, docRoot, corsDomain string, client rpc.Client, interactive bool) *jsre {
js := &jsre{stack: stack, ps1: "> "}
// set default cors domain used by startRpc from CLI flag
js.corsDomain = corsDomain
js.wait = make(chan *big.Int)
js.client = client
js.re = re.New(docRoot)
if err := js.apiBindings(); err != nil {
utils.Fatalf("Unable to connect - %v", err)
}
js.setupInput(stack.DataDir())
return js
}
func (self *jsre) setupInput(datadir string) {
self.withHistory(datadir, func(hist *os.File) { utils.Stdin.ReadHistory(hist) })
utils.Stdin.SetCtrlCAborts(true)
utils.Stdin.SetWordCompleter(makeCompleter(self))
utils.Stdin.SetTabCompletionStyle(liner.TabPrints)
self.atexit = func() {
self.withHistory(datadir, func(hist *os.File) {
hist.Truncate(0)
utils.Stdin.WriteHistory(hist)
})
utils.Stdin.Close()
close(self.wait)
}
}
func (self *jsre) batch(statement string) {
err := self.re.EvalAndPrettyPrint(statement)
if err != nil {
fmt.Printf("%v", jsErrorString(err))
}
if self.atexit != nil {
self.atexit()
}
self.re.Stop(false)
}
// show summary of current geth instance
func (self *jsre) welcome() {
self.re.Run(`
(function () {
console.log('instance: ' + web3.version.node);
console.log("coinbase: " + eth.coinbase);
var ts = 1000 * eth.getBlock(eth.blockNumber).timestamp;
console.log("at block: " + eth.blockNumber + " (" + new Date(ts) + ")");
console.log(' datadir: ' + admin.datadir);
})();
`)
if modules, err := self.supportedApis(); err == nil {
loadedModules := make([]string, 0)
for api, version := range modules {
loadedModules = append(loadedModules, fmt.Sprintf("%s:%s", api, version))
}
sort.Strings(loadedModules)
}
}
func (self *jsre) supportedApis() (map[string]string, error) {
return self.client.SupportedModules()
}
func (js *jsre) apiBindings() error {
apis, err := js.supportedApis()
if err != nil {
return err
}
apiNames := make([]string, 0, len(apis))
for a, _ := range apis {
apiNames = append(apiNames, a)
}
jeth := utils.NewJeth(js.re, js.client)
js.re.Set("jeth", struct{}{})
t, _ := js.re.Get("jeth")
jethObj := t.Object()
jethObj.Set("send", jeth.Send)
jethObj.Set("sendAsync", jeth.Send)
err = js.re.Compile("bignumber.js", re.BigNumber_JS)
if err != nil {
utils.Fatalf("Error loading bignumber.js: %v", err)
}
err = js.re.Compile("web3.js", re.Web3_JS)
if err != nil {
utils.Fatalf("Error loading web3.js: %v", err)
}
_, err = js.re.Run("var Web3 = require('web3');")
if err != nil {
utils.Fatalf("Error requiring web3: %v", err)
}
_, err = js.re.Run("var web3 = new Web3(jeth);")
if err != nil {
utils.Fatalf("Error setting web3 provider: %v", err)
}
// load only supported API's in javascript runtime
shortcuts := "var eth = web3.eth; var personal = web3.personal; "
for _, apiName := range apiNames {
if apiName == "web3" || apiName == "rpc" {
continue // manually mapped or ignore
}
if jsFile, ok := web3ext.Modules[apiName]; ok {
if err = js.re.Compile(fmt.Sprintf("%s.js", apiName), jsFile); err == nil {
shortcuts += fmt.Sprintf("var %s = web3.%s; ", apiName, apiName)
} else {
utils.Fatalf("Error loading %s.js: %v", apiName, err)
}
}
}
_, err = js.re.Run(shortcuts)
if err != nil {
utils.Fatalf("Error setting namespaces: %v", err)
}
// overrule some of the methods that require password as input and ask for it interactively
p, err := js.re.Get("personal")
if err != nil {
fmt.Println("Unable to overrule sensitive methods in personal module")
return nil
}
// Override the unlockAccount and newAccount methods on the personal object since these require user interaction.
// Assign the jeth.unlockAccount and jeth.newAccount in the jsre the original web3 callbacks. These will be called
// by the jeth.* methods after they got the password from the user and send the original web3 request to the backend.
if persObj := p.Object(); persObj != nil { // make sure the personal api is enabled over the interface
js.re.Run(`jeth.unlockAccount = personal.unlockAccount;`)
persObj.Set("unlockAccount", jeth.UnlockAccount)
js.re.Run(`jeth.newAccount = personal.newAccount;`)
persObj.Set("newAccount", jeth.NewAccount)
}
// The admin.sleep and admin.sleepBlocks are offered by the console and not by the RPC layer.
// Bind these if the admin module is available.
if a, err := js.re.Get("admin"); err == nil {
if adminObj := a.Object(); adminObj != nil {
adminObj.Set("sleepBlocks", jeth.SleepBlocks)
adminObj.Set("sleep", jeth.Sleep)
}
}
return nil
}
func (self *jsre) AskPassword() (string, bool) {
pass, err := utils.Stdin.PasswordPrompt("Passphrase: ")
if err != nil {
return "", false
}
return pass, true
}
func (self *jsre) ConfirmTransaction(tx string) bool {
// Retrieve the Ethereum instance from the node
var ethereum *eth.Ethereum
if err := self.stack.Service(&ethereum); err != nil {
return false
}
// If natspec is enabled, ask for permission
if ethereum.NatSpec && false /* disabled for now */ {
// notice := natspec.GetNotice(self.xeth, tx, ethereum.HTTPClient())
// fmt.Println(notice)
// answer, _ := self.Prompt("Confirm Transaction [y/n]")
// return strings.HasPrefix(strings.Trim(answer, " "), "y")
}
return true
}
func (self *jsre) UnlockAccount(addr []byte) bool {
fmt.Printf("Please unlock account %x.\n", addr)
pass, err := utils.Stdin.PasswordPrompt("Passphrase: ")
if err != nil {
return false
}
// TODO: allow retry
var ethereum *eth.Ethereum
if err := self.stack.Service(&ethereum); err != nil {
return false
}
a := accounts.Account{Address: common.BytesToAddress(addr)}
if err := ethereum.AccountManager().Unlock(a, pass); err != nil {
return false
} else {
fmt.Println("Account is now unlocked for this session.")
return true
}
}
// preloadJSFiles loads JS files that the user has specified with ctx.PreLoadJSFlag into
// the JSRE. If not all files could be loaded it will return an error describing the error.
func (self *jsre) preloadJSFiles(ctx *cli.Context) error {
if ctx.GlobalString(utils.PreLoadJSFlag.Name) != "" {
assetPath := ctx.GlobalString(utils.JSpathFlag.Name)
jsFiles := strings.Split(ctx.GlobalString(utils.PreLoadJSFlag.Name), ",")
for _, file := range jsFiles {
filename := common.AbsolutePath(assetPath, strings.TrimSpace(file))
if err := self.re.Exec(filename); err != nil {
return fmt.Errorf("%s: %v", file, jsErrorString(err))
}
}
}
return nil
}
// jsErrorString adds a backtrace to errors generated by otto.
func jsErrorString(err error) string {
if ottoErr, ok := err.(*otto.Error); ok {
return ottoErr.String()
}
return err.Error()
}
func (self *jsre) interactive() {
// Read input lines.
prompt := make(chan string)
inputln := make(chan string)
go func() {
defer close(inputln)
for {
line, err := utils.Stdin.Prompt(<-prompt)
if err != nil {
if err == liner.ErrPromptAborted { // ctrl-C
self.resetPrompt()
inputln <- ""
continue
}
return
}
inputln <- line
}
}()
// Wait for Ctrl-C, too.
sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt)
defer func() {
if self.atexit != nil {
self.atexit()
}
self.re.Stop(false)
}()
for {
prompt <- self.ps1
select {
case <-sig:
fmt.Println("caught interrupt, exiting")
return
case input, ok := <-inputln:
if !ok || indentCount <= 0 && exit.MatchString(input) {
return
}
if onlyws.MatchString(input) {
continue
}
str += input + "\n"
self.setIndent()
if indentCount <= 0 {
if mustLogInHistory(str) {
utils.Stdin.AppendHistory(str[:len(str)-1])
}
self.parseInput(str)
str = ""
}
}
}
}
func mustLogInHistory(input string) bool {
return len(input) == 0 ||
passwordRegexp.MatchString(input) ||
!leadingSpace.MatchString(input)
}
func (self *jsre) withHistory(datadir string, op func(*os.File)) {
hist, err := os.OpenFile(filepath.Join(datadir, "history"), os.O_RDWR|os.O_CREATE, os.ModePerm)
if err != nil {
fmt.Printf("unable to open history file: %v\n", err)
return
}
op(hist)
hist.Close()
}
func (self *jsre) parseInput(code string) {
defer func() {
if r := recover(); r != nil {
fmt.Println("[native] error", r)
}
}()
if err := self.re.EvalAndPrettyPrint(code); err != nil {
if ottoErr, ok := err.(*otto.Error); ok {
fmt.Println(ottoErr.String())
} else {
fmt.Println(err)
}
return
}
}
var indentCount = 0
var str = ""
func (self *jsre) resetPrompt() {
indentCount = 0
str = ""
self.ps1 = "> "
}
func (self *jsre) setIndent() {
open := strings.Count(str, "{")
open += strings.Count(str, "(")
closed := strings.Count(str, "}")
closed += strings.Count(str, ")")
indentCount = open - closed
if indentCount <= 0 {
self.ps1 = "> "
} else {
self.ps1 = strings.Join(make([]string, indentCount*2), "..")
self.ps1 += " "
}
}

502
cmd/geth/js_test.go
View file

@ -1,502 +0,0 @@
// Copyright 2015 The go-ethereum Authors
// This file is part of go-ethereum.
//
// go-ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// go-ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with go-ethereum. If not, see <http://www.gnu.org/licenses/>.
package main
import (
"fmt"
"io/ioutil"
"math/big"
"os"
"path/filepath"
"regexp"
"runtime"
"strconv"
"testing"
"time"
"github.com/ethereum/go-ethereum/accounts"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/compiler"
"github.com/ethereum/go-ethereum/common/httpclient"
"github.com/ethereum/go-ethereum/core"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/eth"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/node"
)
const (
testSolcPath = ""
solcVersion = "0.9.23"
testAddress = "0x8605cdbbdb6d264aa742e77020dcbc58fcdce182"
testBalance = "10000000000000000000"
// of empty string
testHash = "0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470"
)
var (
versionRE = regexp.MustCompile(strconv.Quote(`"compilerVersion":"` + solcVersion + `"`))
testNodeKey, _ = crypto.HexToECDSA("4b50fa71f5c3eeb8fdc452224b2395af2fcc3d125e06c32c82e048c0559db03f")
testAccount, _ = crypto.HexToECDSA("e6fab74a43941f82d89cb7faa408e227cdad3153c4720e540e855c19b15e6674")
testGenesis = `{"` + testAddress[2:] + `": {"balance": "` + testBalance + `"}}`
)
type testjethre struct {
*jsre
lastConfirm string
client *httpclient.HTTPClient
}
// Temporary disabled while natspec hasn't been migrated
//func (self *testjethre) ConfirmTransaction(tx string) bool {
// var ethereum *eth.Ethereum
// self.stack.Service(&ethereum)
//
// if ethereum.NatSpec {
// self.lastConfirm = natspec.GetNotice(self.xeth, tx, self.client)
// }
// return true
//}
func testJEthRE(t *testing.T) (string, *testjethre, *node.Node) {
return testREPL(t, nil)
}
func testREPL(t *testing.T, config func(*eth.Config)) (string, *testjethre, *node.Node) {
tmp, err := ioutil.TempDir("", "geth-test")
if err != nil {
t.Fatal(err)
}
// Create a networkless protocol stack
stack, err := node.New(&node.Config{DataDir: tmp, PrivateKey: testNodeKey, Name: "test", NoDiscovery: true})
if err != nil {
t.Fatalf("failed to create node: %v", err)
}
// Initialize and register the Ethereum protocol
accman := accounts.NewPlaintextManager(filepath.Join(tmp, "keystore"))
db, _ := ethdb.NewMemDatabase()
core.WriteGenesisBlockForTesting(db, core.GenesisAccount{
Address: common.HexToAddress(testAddress),
Balance: common.String2Big(testBalance),
})
ethConf := &eth.Config{
ChainConfig: &core.ChainConfig{HomesteadBlock: new(big.Int)},
TestGenesisState: db,
AccountManager: accman,
DocRoot: "/",
SolcPath: testSolcPath,
PowTest: true,
}
if config != nil {
config(ethConf)
}
if err := stack.Register(func(ctx *node.ServiceContext) (node.Service, error) {
return eth.New(ctx, ethConf)
}); err != nil {
t.Fatalf("failed to register ethereum protocol: %v", err)
}
// Initialize all the keys for testing
a, err := accman.ImportECDSA(testAccount, "")
if err != nil {
t.Fatal(err)
}
if err := accman.Unlock(a, ""); err != nil {
t.Fatal(err)
}
// Start the node and assemble the REPL tester
if err := stack.Start(); err != nil {
t.Fatalf("failed to start test stack: %v", err)
}
var ethereum *eth.Ethereum
stack.Service(&ethereum)
assetPath := filepath.Join(os.Getenv("GOPATH"), "src", "github.com", "ethereum", "go-ethereum", "cmd", "mist", "assets", "ext")
client, err := stack.Attach()
if err != nil {
t.Fatalf("failed to attach to node: %v", err)
}
tf := &testjethre{client: ethereum.HTTPClient()}
repl := newJSRE(stack, assetPath, "", client, false)
tf.jsre = repl
return tmp, tf, stack
}
func TestNodeInfo(t *testing.T) {
t.Skip("broken after p2p update")
tmp, repl, ethereum := testJEthRE(t)
defer ethereum.Stop()
defer os.RemoveAll(tmp)
want := `{"DiscPort":0,"IP":"0.0.0.0","ListenAddr":"","Name":"test","NodeID":"4cb2fc32924e94277bf94b5e4c983beedb2eabd5a0bc941db32202735c6625d020ca14a5963d1738af43b6ac0a711d61b1a06de931a499fe2aa0b1a132a902b5","NodeUrl":"enode://4cb2fc32924e94277bf94b5e4c983beedb2eabd5a0bc941db32202735c6625d020ca14a5963d1738af43b6ac0a711d61b1a06de931a499fe2aa0b1a132a902b5@0.0.0.0:0","TCPPort":0,"Td":"131072"}`
checkEvalJSON(t, repl, `admin.nodeInfo`, want)
}
func TestAccounts(t *testing.T) {
tmp, repl, node := testJEthRE(t)
defer node.Stop()
defer os.RemoveAll(tmp)
checkEvalJSON(t, repl, `eth.accounts`, `["`+testAddress+`"]`)
checkEvalJSON(t, repl, `eth.coinbase`, `"`+testAddress+`"`)
val, err := repl.re.Run(`jeth.newAccount("password")`)
if err != nil {
t.Errorf("expected no error, got %v", err)
}
addr := val.String()
if !regexp.MustCompile(`0x[0-9a-f]{40}`).MatchString(addr) {
t.Errorf("address not hex: %q", addr)
}
checkEvalJSON(t, repl, `eth.accounts`, `["`+testAddress+`","`+addr+`"]`)
}
func TestBlockChain(t *testing.T) {
tmp, repl, node := testJEthRE(t)
defer node.Stop()
defer os.RemoveAll(tmp)
// get current block dump before export/import.
val, err := repl.re.Run("JSON.stringify(debug.dumpBlock(eth.blockNumber))")
if err != nil {
t.Errorf("expected no error, got %v", err)
}
beforeExport := val.String()
// do the export
extmp, err := ioutil.TempDir("", "geth-test-export")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(extmp)
tmpfile := filepath.Join(extmp, "export.chain")
tmpfileq := strconv.Quote(tmpfile)
var ethereum *eth.Ethereum
node.Service(&ethereum)
ethereum.BlockChain().Reset()
checkEvalJSON(t, repl, `admin.exportChain(`+tmpfileq+`)`, `true`)
if _, err := os.Stat(tmpfile); err != nil {
t.Fatal(err)
}
// check import, verify that dumpBlock gives the same result.
checkEvalJSON(t, repl, `admin.importChain(`+tmpfileq+`)`, `true`)
checkEvalJSON(t, repl, `debug.dumpBlock(eth.blockNumber)`, beforeExport)
}
func TestMining(t *testing.T) {
tmp, repl, node := testJEthRE(t)
defer node.Stop()
defer os.RemoveAll(tmp)
checkEvalJSON(t, repl, `eth.mining`, `false`)
}
func TestRPC(t *testing.T) {
tmp, repl, node := testJEthRE(t)
defer node.Stop()
defer os.RemoveAll(tmp)
checkEvalJSON(t, repl, `admin.startRPC("127.0.0.1", 5004, "*", "web3,eth,net")`, `true`)
}
func TestCheckTestAccountBalance(t *testing.T) {
t.Skip() // i don't think it tests the correct behaviour here. it's actually testing
// internals which shouldn't be tested. This now fails because of a change in the core
// and i have no means to fix this, sorry - @obscuren
tmp, repl, node := testJEthRE(t)
defer node.Stop()
defer os.RemoveAll(tmp)
repl.re.Run(`primary = "` + testAddress + `"`)
checkEvalJSON(t, repl, `eth.getBalance(primary)`, `"`+testBalance+`"`)
}
func TestSignature(t *testing.T) {
tmp, repl, node := testJEthRE(t)
defer node.Stop()
defer os.RemoveAll(tmp)
val, err := repl.re.Run(`eth.sign("` + testAddress + `", "` + testHash + `")`)
// This is a very preliminary test, lacking actual signature verification
if err != nil {
t.Errorf("Error running js: %v", err)
return
}
output := val.String()
t.Logf("Output: %v", output)
regex := regexp.MustCompile(`^0x[0-9a-f]{130}$`)
if !regex.MatchString(output) {
t.Errorf("Signature is not 65 bytes represented in hexadecimal.")
return
}
}
func TestContract(t *testing.T) {
t.Skip("contract testing is implemented with mining in ethash test mode. This takes about 7seconds to run. Unskip and run on demand")
coinbase := common.HexToAddress(testAddress)
tmp, repl, ethereum := testREPL(t, func(conf *eth.Config) {
conf.Etherbase = coinbase
conf.PowTest = true
})
if err := ethereum.Start(); err != nil {
t.Errorf("error starting ethereum: %v", err)
return
}
defer ethereum.Stop()
defer os.RemoveAll(tmp)
// Temporary disabled while registrar isn't migrated
//reg := registrar.New(repl.xeth)
//_, err := reg.SetGlobalRegistrar("", coinbase)
//if err != nil {
// t.Errorf("error setting HashReg: %v", err)
//}
//_, err = reg.SetHashReg("", coinbase)
//if err != nil {
// t.Errorf("error setting HashReg: %v", err)
//}
//_, err = reg.SetUrlHint("", coinbase)
//if err != nil {
// t.Errorf("error setting HashReg: %v", err)
//}
/* TODO:
* lookup receipt and contract addresses by tx hash
* name registration for HashReg and UrlHint addresses
* mine those transactions
* then set once more SetHashReg SetUrlHint
*/
source := `contract test {\n` +
" /// @notice Will multiply `a` by 7." + `\n` +
` function multiply(uint a) returns(uint d) {\n` +
` return a * 7;\n` +
` }\n` +
`}\n`
if checkEvalJSON(t, repl, `admin.stopNatSpec()`, `true`) != nil {
return
}
contractInfo, err := ioutil.ReadFile("info_test.json")
if err != nil {
t.Fatalf("%v", err)
}
if checkEvalJSON(t, repl, `primary = eth.accounts[0]`, `"`+testAddress+`"`) != nil {
return
}
if checkEvalJSON(t, repl, `source = "`+source+`"`, `"`+source+`"`) != nil {
return
}
// if solc is found with right version, test it, otherwise read from file
sol, err := compiler.New("")
if err != nil {
t.Logf("solc not found: mocking contract compilation step")
} else if sol.Version() != solcVersion {
t.Logf("WARNING: solc different version found (%v, test written for %v, may need to update)", sol.Version(), solcVersion)
}
if err != nil {
info, err := ioutil.ReadFile("info_test.json")
if err != nil {
t.Fatalf("%v", err)
}
_, err = repl.re.Run(`contract = JSON.parse(` + strconv.Quote(string(info)) + `)`)
if err != nil {
t.Errorf("%v", err)
}
} else {
if checkEvalJSON(t, repl, `contract = eth.compile.solidity(source).test`, string(contractInfo)) != nil {
return
}
}
if checkEvalJSON(t, repl, `contract.code`, `"0x605880600c6000396000f3006000357c010000000000000000000000000000000000000000000000000000000090048063c6888fa114602e57005b603d6004803590602001506047565b8060005260206000f35b60006007820290506053565b91905056"`) != nil {
return
}
if checkEvalJSON(
t, repl,
`contractaddress = eth.sendTransaction({from: primary, data: contract.code})`,
`"0x46d69d55c3c4b86a924a92c9fc4720bb7bce1d74"`,
) != nil {
return
}
if !processTxs(repl, t, 8) {
return
}
callSetup := `abiDef = JSON.parse('[{"constant":false,"inputs":[{"name":"a","type":"uint256"}],"name":"multiply","outputs":[{"name":"d","type":"uint256"}],"type":"function"}]');
Multiply7 = eth.contract(abiDef);
multiply7 = Multiply7.at(contractaddress);
`
_, err = repl.re.Run(callSetup)
if err != nil {
t.Errorf("unexpected error setting up contract, got %v", err)
return
}
expNotice := ""
if repl.lastConfirm != expNotice {
t.Errorf("incorrect confirmation message: expected %v, got %v", expNotice, repl.lastConfirm)
return
}
if checkEvalJSON(t, repl, `admin.startNatSpec()`, `true`) != nil {
return
}
if checkEvalJSON(t, repl, `multiply7.multiply.sendTransaction(6, { from: primary })`, `"0x4ef9088431a8033e4580d00e4eb2487275e031ff4163c7529df0ef45af17857b"`) != nil {
return
}
if !processTxs(repl, t, 1) {
return
}
expNotice = `About to submit transaction (no NatSpec info found for contract: content hash not found for '0x87e2802265838c7f14bb69eecd2112911af6767907a702eeaa445239fb20711b'): {"params":[{"to":"0x46d69d55c3c4b86a924a92c9fc4720bb7bce1d74","data": "0xc6888fa10000000000000000000000000000000000000000000000000000000000000006"}]}`
if repl.lastConfirm != expNotice {
t.Errorf("incorrect confirmation message: expected\n%v, got\n%v", expNotice, repl.lastConfirm)
return
}
var contentHash = `"0x86d2b7cf1e72e9a7a3f8d96601f0151742a2f780f1526414304fbe413dc7f9bd"`
if sol != nil && solcVersion != sol.Version() {
modContractInfo := versionRE.ReplaceAll(contractInfo, []byte(`"compilerVersion":"`+sol.Version()+`"`))
fmt.Printf("modified contractinfo:\n%s\n", modContractInfo)
contentHash = `"` + common.ToHex(crypto.Keccak256([]byte(modContractInfo))) + `"`
}
if checkEvalJSON(t, repl, `filename = "/tmp/info.json"`, `"/tmp/info.json"`) != nil {
return
}
if checkEvalJSON(t, repl, `contentHash = admin.saveInfo(contract.info, filename)`, contentHash) != nil {
return
}
if checkEvalJSON(t, repl, `admin.register(primary, contractaddress, contentHash)`, `true`) != nil {
return
}
if checkEvalJSON(t, repl, `admin.registerUrl(primary, contentHash, "file://"+filename)`, `true`) != nil {
return
}
if checkEvalJSON(t, repl, `admin.startNatSpec()`, `true`) != nil {
return
}
if !processTxs(repl, t, 3) {
return
}
if checkEvalJSON(t, repl, `multiply7.multiply.sendTransaction(6, { from: primary })`, `"0x60 {
t.Errorf("%d trasactions were not mined", txc)
return f6d7635c12ad0b231e66da2f987ca3dfdca58ffe49c6442aa55960858103fd0c"`) != nil {
return
}
if !processTxs(repl, t, 1) {
return
}
expNotice = "Will multiply 6 by 7."
if repl.lastConfirm != expNotice {
t.Errorf("incorrect confirmation message: expected\n%v, got\n%v", expNotice, repl.lastConfirm)
return
}
}
func pendingTransactions(repl *testjethre, t *testing.T) (txc int64, err error) {
var ethereum *eth.Ethereum
repl.stack.Service(&ethereum)
txs := ethereum.TxPool().GetTransactions()
return int64(len(txs)), nil
}
func processTxs(repl *testjethre, t *testing.T, expTxc int) bool {
var txc int64
var err error
for i := 0; i < 50; i++ {
txc, err = pendingTransactions(repl, t)
if err != nil {
t.Errorf("unexpected error checking pending transactions: %v", err)
return false
}
if expTxc < int(txc) {
t.Errorf("too many pending transactions: expected %v, got %v", expTxc, txc)
return false
} else if expTxc == int(txc) {
break
}
time.Sleep(100 * time.Millisecond)
}
if int(txc) != expTxc {
t.Errorf("incorrect number of pending transactions, expected %v, got %v", expTxc, txc)
return false
}
var ethereum *eth.Ethereum
repl.stack.Service(&ethereum)
err = ethereum.StartMining(runtime.NumCPU(), "")
if err != nil {
t.Errorf("unexpected error mining: %v", err)
return false
}
defer ethereum.StopMining()
timer := time.NewTimer(100 * time.Second)
blockNr := ethereum.BlockChain().CurrentBlock().Number()
height := new(big.Int).Add(blockNr, big.NewInt(1))
repl.wait <- height
select {
case <-timer.C:
// if times out make sure the xeth loop does not block
go func() {
select {
case repl.wait <- nil:
case <-repl.wait:
}
}()
case <-repl.wait:
}
txc, err = pendingTransactions(repl, t)
if err != nil {
t.Errorf("unexpected error checking pending transactions: %v", err)
return false
}
if txc != 0 {
t.Errorf("%d trasactions were not mined", txc)
return false
}
return true
}
func checkEvalJSON(t *testing.T, re *testjethre, expr, want string) error {
val, err := re.re.Run("JSON.stringify(" + expr + ")")
if err == nil && val.String() != want {
err = fmt.Errorf("Output mismatch for `%s`:\ngot: %s\nwant: %s", expr, val.String(), want)
}
if err != nil {
_, file, line, _ := runtime.Caller(1)
file = filepath.Base(file)
fmt.Printf("\t%s:%d: %v\n", file, line, err)
t.Fail()
}
return err
}

24
cmd/geth/main.go
View file

@ -29,7 +29,7 @@ import (
"time" "time"
"github.com/ethereum/ethash" "github.com/ethereum/ethash"
"github.com/ethereum/go-ethereum/accounts" // "github.com/ethereum/go-ethereum/accounts"
"github.com/ethereum/go-ethereum/cmd/utils" "github.com/ethereum/go-ethereum/cmd/utils"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/console" "github.com/ethereum/go-ethereum/console"
@ -322,18 +322,18 @@ func startNode(ctx *cli.Context, stack *node.Node) {
utils.StartNode(stack) utils.StartNode(stack)
// Unlock any account specifically requested // Unlock any account specifically requested
var accman *accounts.Manager // var accman *accounts.Manager
if err := stack.Service(&accman); err != nil { // if err := stack.Service(&accman); err != nil {
utils.Fatalf("ethereum service not running: %v", err) // utils.Fatalf("ethereum service not running: %v", err)
} // }
passwords := utils.MakePasswordList(ctx) // passwords := utils.MakePasswordList(ctx)
accounts := strings.Split(ctx.GlobalString(utils.UnlockedAccountFlag.Name), ",") // accounts := strings.Split(ctx.GlobalString(utils.UnlockedAccountFlag.Name), ",")
for i, account := range accounts { // for i, account := range accounts {
if trimmed := strings.TrimSpace(account); trimmed != "" { // if trimmed := strings.TrimSpace(account); trimmed != "" {
unlockAccount(ctx, accman, trimmed, i, passwords) // unlockAccount(ctx, accman, trimmed, i, passwords)
} // }
} // }
// Start auxiliary services if enabled // Start auxiliary services if enabled
if ctx.GlobalBool(utils.MiningEnabledFlag.Name) { if ctx.GlobalBool(utils.MiningEnabledFlag.Name) {
var ethereum *eth.Ethereum var ethereum *eth.Ethereum

55
cmd/utils/cmd.go
View file

@ -24,7 +24,6 @@ import (
"os/signal" "os/signal"
"regexp" "regexp"
"github.com/ethereum/go-ethereum/accounts"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core" "github.com/ethereum/go-ethereum/core"
"github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/core/types"
@ -210,57 +209,3 @@ func ExportAppendChain(blockchain *core.BlockChain, fn string, first uint64, las
glog.Infoln("Exported blockchain to ", fn) glog.Infoln("Exported blockchain to ", fn)
return nil return nil
} }
// tries unlocking the specified account a few times.
func UnlockAccount(accman *accounts.Manager, address string, i int, passwords []string) (accounts.Account, string) {
account, err := MakeAddress(accman, address)
if err != nil {
Fatalf("Could not list accounts: %v", err)
}
for trials := 0; trials < 3; trials++ {
prompt := fmt.Sprintf("Unlocking account %s | Attempt %d/%d", address, trials+1, 3)
password := GetPassPhrase(prompt, false, i, passwords)
err = accman.Unlock(account, password)
if err == nil {
glog.V(logger.Info).Infof("Unlocked account %x", account.Address)
return account, password
}
if err, ok := err.(*accounts.AmbiguousAddrError); ok {
glog.V(logger.Info).Infof("Unlocked account %x", account.Address)
return ambiguousAddrRecovery(accman, err, password), password
}
if err != accounts.ErrDecrypt {
// No need to prompt again if the error is not decryption-related.
break
}
}
// All trials expended to unlock account, bail out
Fatalf("Failed to unlock account %s (%v)", address, err)
return accounts.Account{}, ""
}
func ambiguousAddrRecovery(am *accounts.Manager, err *accounts.AmbiguousAddrError, auth string) accounts.Account {
fmt.Printf("Multiple key files exist for address %x:\n", err.Addr)
for _, a := range err.Matches {
fmt.Println(" ", a.File)
}
fmt.Println("Testing your passphrase against all of them...")
var match *accounts.Account
for _, a := range err.Matches {
if err := am.Unlock(a, auth); err == nil {
match = &a
break
}
}
if match == nil {
Fatalf("None of the listed files could be unlocked.")
}
fmt.Printf("Your passphrase unlocked %s\n", match.File)
fmt.Println("In order to avoid this warning, you need to remove the following duplicate key files:")
for _, a := range err.Matches {
if a != *match {
fmt.Println(" ", a.File)
}
}
return *match
}

4
cmd/utils/customflags.go
View file

@ -39,7 +39,7 @@ func (self *DirectoryString) String() string {
} }
func (self *DirectoryString) Set(value string) error { func (self *DirectoryString) Set(value string) error {
self.Value = ExpandPath(value) self.Value = expandPath(value)
return nil return nil
} }
@ -135,7 +135,7 @@ func (self *DirectoryFlag) Set(value string) {
// 2. expands embedded environment variables // 2. expands embedded environment variables
// 3. cleans the path, e.g. /a/b/../c -> /a/c // 3. cleans the path, e.g. /a/b/../c -> /a/c
// Note, it has limitations, e.g. ~someuser/tmp will not be expanded // Note, it has limitations, e.g. ~someuser/tmp will not be expanded
func ExpandPath(p string) string { func expandPath(p string) string {
if strings.HasPrefix(p, "~/") || strings.HasPrefix(p, "~\\") { if strings.HasPrefix(p, "~/") || strings.HasPrefix(p, "~\\") {
if user, err := user.Current(); err == nil { if user, err := user.Current(); err == nil {
p = user.HomeDir + p[1:] p = user.HomeDir + p[1:]

2
cmd/utils/customflags_test.go
View file

@ -33,7 +33,7 @@ func TestPathExpansion(t *testing.T) {
} }
os.Setenv("DDDXXX", "/tmp") os.Setenv("DDDXXX", "/tmp")
for test, expected := range tests { for test, expected := range tests {
got := ExpandPath(test) got := expandPath(test)
if got != expected { if got != expected {
t.Errorf("test %s, got %s, expected %s\n", test, got, expected) t.Errorf("test %s, got %s, expected %s\n", test, got, expected)
} }

2
cmd/utils/flags.go
View file

@ -712,7 +712,7 @@ func MakeSystemNode(name, version string, relconf release.Config, extra []byte,
accounts := strings.Split(ctx.GlobalString(UnlockedAccountFlag.Name), ",") accounts := strings.Split(ctx.GlobalString(UnlockedAccountFlag.Name), ",")
for i, account := range accounts { for i, account := range accounts {
if trimmed := strings.TrimSpace(account); trimmed != "" { if trimmed := strings.TrimSpace(account); trimmed != "" {
UnlockAccount(accman, trimmed, i, passwords) UnlockAccount(ctx, accman, trimmed, i, passwords)
} }
} }

128
cmd/utils/input.go
View file

@ -1,128 +0,0 @@
// Copyright 2016 The go-ethereum Authors
// This file is part of go-ethereum.
//
// go-ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// go-ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with go-ethereum. If not, see <http://www.gnu.org/licenses/>.
package utils
import (
"fmt"
"strings"
"github.com/peterh/liner"
)
// Holds the stdin line reader.
// Only this reader may be used for input because it keeps
// an internal buffer.
var Stdin = newUserInputReader()
type userInputReader struct {
*liner.State
warned bool
supported bool
normalMode liner.ModeApplier
rawMode liner.ModeApplier
}
func newUserInputReader() *userInputReader {
r := new(userInputReader)
// Get the original mode before calling NewLiner.
// This is usually regular "cooked" mode where characters echo.
normalMode, _ := liner.TerminalMode()
// Turn on liner. It switches to raw mode.
r.State = liner.NewLiner()
rawMode, err := liner.TerminalMode()
if err != nil || !liner.TerminalSupported() {
r.supported = false
} else {
r.supported = true
r.normalMode = normalMode
r.rawMode = rawMode
// Switch back to normal mode while we're not prompting.
normalMode.ApplyMode()
}
return r
}
func (r *userInputReader) Prompt(prompt string) (string, error) {
if r.supported {
r.rawMode.ApplyMode()
defer r.normalMode.ApplyMode()
} else {
// liner tries to be smart about printing the prompt
// and doesn't print anything if input is redirected.
// Un-smart it by printing the prompt always.
fmt.Print(prompt)
prompt = ""
defer fmt.Println()
}
return r.State.Prompt(prompt)
}
func (r *userInputReader) PasswordPrompt(prompt string) (passwd string, err error) {
if r.supported {
r.rawMode.ApplyMode()
defer r.normalMode.ApplyMode()
return r.State.PasswordPrompt(prompt)
}
if !r.warned {
fmt.Println("!! Unsupported terminal, password will be echoed.")
r.warned = true
}
// Just as in Prompt, handle printing the prompt here instead of relying on liner.
fmt.Print(prompt)
passwd, err = r.State.Prompt("")
fmt.Println()
return passwd, err
}
func (r *userInputReader) ConfirmPrompt(prompt string) (bool, error) {
prompt = prompt + " [y/N] "
input, err := r.Prompt(prompt)
if len(input) > 0 && strings.ToUpper(input[:1]) == "Y" {
return true, nil
}
return false, err
}
// getPassPhrase retrieves the password associated with an account, either fetched
// from a list of preloaded passphrases, or requested interactively from the user.
func GetPassPhrase(prompt string, confirmation bool, i int, passwords []string) string {
// If a list of passwords was supplied, retrieve from them
if len(passwords) > 0 {
if i < len(passwords) {
return passwords[i]
}
return passwords[len(passwords)-1]
}
// Otherwise prompt the user for the password
if prompt != "" {
fmt.Println(prompt)
}
password, err := Stdin.PasswordPrompt("Passphrase: ")
if err != nil {
Fatalf("Failed to read passphrase: %v", err)
}
if confirmation {
confirm, err := Stdin.PasswordPrompt("Repeat passphrase: ")
if err != nil {
Fatalf("Failed to read passphrase confirmation: %v", err)
}
if password != confirm {
Fatalf("Passphrases do not match")
}
}
return password
}

4
eth/backend.go
View file

@ -341,7 +341,7 @@ func (s *Ethereum) APIs() []rpc.API {
}, { }, {
Namespace: "debug", Namespace: "debug",
Version: "1.0", Version: "1.0",
Service: NewPrivateDebugAPI(s.ChainConfig(), s), Service: NewPrivateDebugAPI(s.chainConfig, s),
}, { }, {
Namespace: "net", Namespace: "net",
Version: "1.0", Version: "1.0",
@ -379,8 +379,6 @@ func (s *Ethereum) Miner() *miner.Miner { return s.miner }
func (s *Ethereum) AccountManager() *accounts.Manager { return s.accountManager } func (s *Ethereum) AccountManager() *accounts.Manager { return s.accountManager }
func (s *Ethereum) BlockChain() *core.BlockChain { return s.blockchain } func (s *Ethereum) BlockChain() *core.BlockChain { return s.blockchain }
func (s *Ethereum) GPO() *GasPriceOracle { return s.gpo }
func (s *Ethereum) ChainConfig() *core.ChainConfig { return s.chainConfig }
func (s *Ethereum) TxPool() *core.TxPool { return s.txPool } func (s *Ethereum) TxPool() *core.TxPool { return s.txPool }
func (s *Ethereum) EventMux() *event.TypeMux { return s.eventMux } func (s *Ethereum) EventMux() *event.TypeMux { return s.eventMux }
func (s *Ethereum) Pow() *ethash.Ethash { return s.pow } func (s *Ethereum) Pow() *ethash.Ethash { return s.pow }

23
eth/bind.go
View file

@ -20,8 +20,6 @@ import (
"math/big" "math/big"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core"
"github.com/ethereum/go-ethereum/core/state"
"github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/internal/ethapi" "github.com/ethereum/go-ethereum/internal/ethapi"
"github.com/ethereum/go-ethereum/rlp" "github.com/ethereum/go-ethereum/rlp"
@ -70,7 +68,7 @@ func (b *ContractBackend) HasCode(ctx context.Context, contract common.Address,
// call with the specified data as the input. The pending flag requests execution // call with the specified data as the input. The pending flag requests execution
// against the pending block, not the stable head of the chain. // against the pending block, not the stable head of the chain.
func (b *ContractBackend) ContractCall(ctx context.Context, contract common.Address, data []byte, pending bool) ([]byte, error) { func (b *ContractBackend) ContractCall(ctx context.Context, contract common.Address, data []byte, pending bool) ([]byte, error) {
if ctx ==-- nil { if ctx == nil {
ctx = context.Background() ctx = context.Background()
} }
// Convert the input args to the API spec // Convert the input args to the API spec
@ -82,7 +80,7 @@ func (b *ContractBackend) ContractCall(ctx context.Context, contract common.Addr
if pending { if pending {
block = rpc.PendingBlockNumber block = rpc.PendingBlockNumber
} }
// Execute the call and convert the output ba--ck to Go types // Execute the call and convert the output back to Go types
out, err := b.bcapi.Call(ctx, args, block) out, err := b.bcapi.Call(ctx, args, block)
return common.FromHex(out), err return common.FromHex(out), err
} }
@ -135,19 +133,14 @@ func (b *ContractBackend) SendTransaction(ctx context.Context, tx *types.Transac
return err return err
} }
func (b *ContractBackend) GetTxReceipt(txhash common.Hash) *types.Receipt { func (b *ContractBackend) GetTxReceipt(txhash common.Hash) (map[string]interface{}, error) {
return core.GetReceipt(b.eth.ChainDb(), txhash) return b.txapi.GetTransactionReceipt(txhash)
} }
func (b *ContractBackend) BalanceAt(address common.Address) *big.Int { func (b *ContractBackend) BalanceAt(address common.Address) (*big.Int, error) {
currentState, err := state.New(b.eth.BlockChain().CurrentBlock().Root(), b.eth.ChainDb()) return b.bcapi.GetBalance(context.Background(), address, rpc.LatestBlockNumber)
if err != nil {
return new(big.Int)
}
return currentState.GetBalance(address)
} }
func (b *ContractBackend) CodeAt(address common.Address) string { func (b *ContractBackend) CodeAt(address common.Address) (string, error) {
currentState, _ := state.New(b.eth.BlockChain().CurrentBlock().Root(), b.eth.ChainDb()) return b.bcapi.GetCode(context.Background(), address, rpc.LatestBlockNumber)
return common.ToHex(currentState.GetCode(address))
} }

82
internal/web3ext/web3ext.go
View file

@ -27,7 +27,6 @@ var Modules = map[string]string{
"rpc": RPC_JS, "rpc": RPC_JS,
"shh": Shh_JS, "shh": Shh_JS,
"txpool": TxPool_JS, "txpool": TxPool_JS,
"net": Net_JS,
"bzz": Bzz_JS, "bzz": Bzz_JS,
"ens": ENS_JS, "ens": ENS_JS,
"chequebook": Chequebook_JS, "chequebook": Chequebook_JS,
@ -123,8 +122,13 @@ web3._extend({
params: 2, params: 2,
inputFormatter: [null, null] inputFormatter: [null, null]
}), }),
new web3._extend.Method( new web3._extend.Method({
{ name: 'setContentHash',
call: 'ens_setContentHash',
params: 2,
inputFormatter: [null, null]
}),
new web3._extend.Method({
name: 'resolve', name: 'resolve',
call: 'ens_resolve', call: 'ens_resolve',
params: 1, params: 1,
@ -139,23 +143,13 @@ web3._extend({
property: 'chequebook', property: 'chequebook',
methods: methods:
[ [
new web3._extend.Method( new web3._extend.Method({
{
name: 'deposit', name: 'deposit',
call: 'chequebook_deposit', call: 'chequebook_deposit',
}
out, err := b.txapi.GetTransactionCount(ctx, account, rpc.PendingBlockNumber)
return out.Uint64(), err
}
// SuggestGasPrice implements bind.ContractTransactor retrieving the currently
// suggested gas price to allow a timely execution of a transaction.
func (b *ContractBackend
params: 1, params: 1,
inputFormatter: [null] inputFormatter: [null]
}), }),
new web3._extend.Propert new web3._extend.Property({
y({
name: 'balance', name: 'balance',
getter: 'chequebook_balance', getter: 'chequebook_balance',
outputFormatter: web3._extend.utils.toDecimal outputFormatter: web3._extend.utils.toDecimal
@ -168,7 +162,6 @@ y({
}), }),
new web3._extend.Method({ new web3._extend.Method({
name: 'issue', name: 'issue',
call: 'chequebook_issue', call: 'chequebook_issue',
params: 2, params: 2,
inputFormatter: [null, null] inputFormatter: [null, null]
@ -177,50 +170,6 @@ y({
}); });
` `
const Personal_JS = `
web3._extend({
property: 'personal',
methods:
[
new web3._extend.Method({
name: 'importRawKey',
call: 'personal_importRawKey',
params: 2
})
]
});
`
const TxPool_JS = `web3._extend({
property: 'txpool',
methods:
[
],
properties:
[
new web3._extend.Property({
name: 'content',
getter: 'txpool_content'
}),
new web3._extend.Property({
name: 'inspect',
getter: 'txpool_inspect'
}),
new web3._extend.Property({
name: 'status',
getter: 'txpool_status',
outputFormatter: function(status) {
status.pending = web3._extend.utils.toDecimal(status.pending);
status.queued = web3._extend.utils.toDecimal(status.queued);
return status;
}
})
]
});
`s
const Admin_JS = ` const Admin_JS = `
web3._extend({ web3._extend({
property: 'admin', property: 'admin',
@ -326,19 +275,6 @@ web3._extend({
}); });
` `
const Net_JS = `
web3._extend({
property: 'net',
methods: [],
properties:
[
new web3._extend.Property({
name: 'version',
getter: 'net_version'
})
]
});
`
const Debug_JS = ` const Debug_JS = `
web3._extend({ web3._extend({
property: 'debug', property: 'debug',

27
swarm/api/api.go
View file

@ -43,12 +43,12 @@ func NewApi(dpa *storage.DPA, dns Resolver) (self *Api) {
} }
// DPA reader API // DPA reader API
func (self *Api) Retrieve(key storage.Key) storage.SectionReader { func (self *Api) Retrieve(key storage.Key) storage.LazySectionReader {
return self.dpa.Retrieve(key) return self.dpa.Retrieve(key)
} }
func (self *Api) Store(data storage.SectionReader, wg *sync.WaitGroup) (key storage.Key, err error) { func (self *Api) Store(data io.Reader, size int64, wg *sync.WaitGroup) (key storage.Key, err error) {
return self.dpa.Store(data, wg) return self.dpa.Store(data, size, wg)
} }
type ErrResolve error type ErrResolve error
@ -105,15 +105,15 @@ func (self *Api) parseAndResolve(uri string, nameresolver bool) (contentHash sto
// Put provides singleton manifest creation on top of dpa store // Put provides singleton manifest creation on top of dpa store
func (self *Api) Put(content, contentType string) (string, error) { func (self *Api) Put(content, contentType string) (string, error) {
sr := io.NewSectionReader(strings.NewReader(content), 0, int64(len(content))) r := strings.NewReader(content)
wg := &sync.WaitGroup{} wg := &sync.WaitGroup{}
key, err := self.dpa.Store(sr, wg) key, err := self.dpa.Store(r, int64(len(content)), wg)
if err != nil { if err != nil {
return "", err return "", err
} }
manifest := fmt.Sprintf(`{"entries":[{"hash":"%v","contentType":"%s"}]}`, key, contentType) manifest := fmt.Sprintf(`{"entries":[{"hash":"%v","contentType":"%s"}]}`, key, contentType)
sr = io.NewSectionReader(strings.NewReader(manifest), 0, int64(len(manifest))) r = strings.NewReader(manifest)
key, err = self.dpa.Store(sr, wg) key, err = self.dpa.Store(r, int64(len(manifest)), wg)
if err != nil { if err != nil {
return "", err return "", err
} }
@ -124,11 +124,11 @@ func (self *Api) Put(content, contentType string) (string, error) {
// Get uses iterative manifest retrieval and prefix matching // Get uses iterative manifest retrieval and prefix matching
// to resolve path to content using dpa retrieve // to resolve path to content using dpa retrieve
// it returns a section reader, mimeType, status and an error // it returns a section reader, mimeType, status and an error
func (self *Api) Get(uri string, nameresolver bool) (reader storage.SectionReader, mimeType string, status int, err error) { func (self *Api) Get(uri string, nameresolver bool) (reader storage.LazySectionReader, mimeType string, status int, err error) {
key, _, path, err := self.parseAndResolve(uri, nameresolver) key, _, path, err := self.parseAndResolve(uri, nameresolver)
quitC := make(chan bool)
trie, err := loadManifest(self.dpa, key) trie, err := loadManifest(self.dpa, key, quitC)
if err != nil { if err != nil {
glog.V(logger.Warn).Infof("[BZZ] Swarm: loadManifestTrie error: %v", err) glog.V(logger.Warn).Infof("[BZZ] Swarm: loadManifestTrie error: %v", err)
return return
@ -151,7 +151,8 @@ func (self *Api) Get(uri string, nameresolver bool) (reader storage.SectionReade
func (self *Api) Modify(uri, contentHash, contentType string, nameresolver bool) (newRootHash string, err error) { func (self *Api) Modify(uri, contentHash, contentType string, nameresolver bool) (newRootHash string, err error) {
root, _, path, err := self.parseAndResolve(uri, nameresolver) root, _, path, err := self.parseAndResolve(uri, nameresolver)
trie, err := loadManifest(self.dpa, root) quitC := make(chan bool)
trie, err := loadManifest(self.dpa, root, quitC)
if err != nil { if err != nil {
return return
} }
@ -162,9 +163,9 @@ func (self *Api) Modify(uri, contentHash, contentType string, nameresolver bool)
Hash: contentHash, Hash: contentHash,
ContentType: contentType, ContentType: contentType,
} }
trie.addEntry(entry) trie.addEntry(entry, quitC)
} else { } else {
trie.deleteEntry(path) trie.deleteEntry(path, quitC)
} }
err = trie.recalcAndStore() err = trie.recalcAndStore()

23
swarm/api/api_test.go
View file

@ -1,11 +1,13 @@
package api package api
import ( import (
// "bytes" "io"
"io/ioutil" "io/ioutil"
"os" "os"
"testing" "testing"
"github.com/ethereum/go-ethereum/logger"
"github.com/ethereum/go-ethereum/logger/glog"
"github.com/ethereum/go-ethereum/swarm/storage" "github.com/ethereum/go-ethereum/swarm/storage"
) )
@ -27,7 +29,7 @@ func testApi(t *testing.T, f func(*Api)) {
} }
type testResponse struct { type testResponse struct {
reader storage.SectionReader reader storage.LazySectionReader
*Response *Response
} }
@ -51,13 +53,14 @@ func checkResponse(t *testing.T, resp *testResponse, exp *Response) {
resp.Content = string(content) resp.Content = string(content)
} }
if resp.Content != exp.Content { if resp.Content != exp.Content {
// if !bytes.Equal(resp.Content, exp.Content) { // if !bytes.Equal(resp.Content, exp.Content)
t.Errorf("incorrect content. expected '%s...', got '%s...'", string(exp.Content), string(resp.Content)) t.Errorf("incorrect content. expected '%s...', got '%s...'", string(exp.Content), string(resp.Content))
} }
} }
// func expResponse(content []byte, mimeType string, status int) *Response { // func expResponse(content []byte, mimeType string, status int) *Response {
func expResponse(content string, mimeType string, status int) *Response { func expResponse(content string, mimeType string, status int) *Response {
glog.V(logger.Detail).Infof("expected content (%v): %v ", len(content), content)
return &Response{mimeType, status, int64(len(content)), content} return &Response{mimeType, status, int64(len(content)), content}
} }
@ -67,7 +70,19 @@ func testGet(t *testing.T, api *Api, bzzhash string) *testResponse {
if err != nil { if err != nil {
t.Fatalf("unexpected error: %v", err) t.Fatalf("unexpected error: %v", err)
} }
return &testResponse{reader, &Response{mimeType, status, reader.Size(), ""}} quitC := make(chan bool)
size, err := reader.Size(quitC)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
glog.V(logger.Detail).Infof("reader size: %v ", size)
s := make([]byte, size)
_, err = reader.Read(s)
if err != io.EOF {
t.Fatalf("unexpected error: %v", err)
}
reader.Seek(0, 0)
return &testResponse{reader, &Response{mimeType, status, size, string(s)}}
// return &testResponse{reader, &Response{mimeType, status, reader.Size(), nil}} // return &testResponse{reader, &Response{mimeType, status, reader.Size(), nil}}
} }

1
swarm/api/config.go
View file

@ -32,6 +32,7 @@ type Config struct {
Port string Port string
PublicKey string PublicKey string
BzzKey string BzzKey string
EnsRoot common.Address
} }
// config is agnostic to where private key is coming from // config is agnostic to where private key is coming from

18
swarm/api/config_test.go
View file

@ -19,16 +19,15 @@ var (
"CacheCapacity": 5000, "CacheCapacity": 5000,
"Radius": 0, "Radius": 0,
"Branches": 128, "Branches": 128,
"Hash": "SHA256", "Hash": "SHA3",
"JoinTimeout": 120, "CallInterval": 3000000000,
"SplitTimeout": 120,
"CallInterval": 10000000000,
"KadDbPath": "` + filepath.Join("TMPDIR", "0d2f62485607cf38d9d795d93682a517661e513e", "bzz-peers.json") + `", "KadDbPath": "` + filepath.Join("TMPDIR", "0d2f62485607cf38d9d795d93682a517661e513e", "bzz-peers.json") + `",
"MaxProx": 10, "MaxProx": 8,
"ProxBinSize": 8, "ProxBinSize": 2,
"BucketSize": 3, "BucketSize": 4,
"PurgeInterval": 151200000000000, "PurgeInterval": 151200000000000,
"InitialRetryInterval": 4200000000, "InitialRetryInterval": 42000000,
"MaxIdleInterval": 4200000000,
"ConnRetryExp": 2, "ConnRetryExp": 2,
"Swap": { "Swap": {
"BuyAt": 20000000000, "BuyAt": 20000000000,
@ -67,7 +66,8 @@ var (
"Path": "` + filepath.Join("TMPDIR", "0d2f62485607cf38d9d795d93682a517661e513e") + `", "Path": "` + filepath.Join("TMPDIR", "0d2f62485607cf38d9d795d93682a517661e513e") + `",
"Port": "8500", "Port": "8500",
"PublicKey": "0x045f5cfd26692e48d0017d380349bcf50982488bc11b5145f3ddf88b24924299048450542d43527fbe29a5cb32f38d62755393ac002e6bfdd71b8d7ba725ecd7a3", "PublicKey": "0x045f5cfd26692e48d0017d380349bcf50982488bc11b5145f3ddf88b24924299048450542d43527fbe29a5cb32f38d62755393ac002e6bfdd71b8d7ba725ecd7a3",
"BzzKey": "0xe861964402c0b78e2d44098329b8545726f215afa737d803714a4338552fcb81" "BzzKey": "0xe861964402c0b78e2d44098329b8545726f215afa737d803714a4338552fcb81",
"EnsRoot": "0x0000000000000000000000000000000000000000"
}` }`
) )

128
swarm/api/filesystem.go
View file

@ -86,27 +86,29 @@ func (self *FileSystem) Upload(lpath, index string) (string, error) {
errors := make([]error, cnt) errors := make([]error, cnt)
done := make(chan bool, maxParallelFiles) done := make(chan bool, maxParallelFiles)
dcnt := 0 dcnt := 0
awg := &sync.WaitGroup{}
for i, entry := range list { for i, entry := range list {
if i >= dcnt+maxParallelFiles { if i >= dcnt+maxParallelFiles {
<-done <-done
dcnt++ dcnt++
} }
awg.Add(1)
go func(i int, entry *manifestTrieEntry, done chan bool) { go func(i int, entry *manifestTrieEntry, done chan bool) {
f, err := os.Open(entry.Path) f, err := os.Open(entry.Path)
if err == nil { if err == nil {
stat, _ := f.Stat() stat, _ := f.Stat()
sr := io.NewSectionReader(f, 0, stat.Size())
wg := &sync.WaitGroup{}
var hash storage.Key var hash storage.Key
hash, err = self.api.dpa.Store(sr, wg) wg := &sync.WaitGroup{}
hash, err = self.api.dpa.Store(f, stat.Size(), wg)
if hash != nil { if hash != nil {
list[i].Hash = hash.String() list[i].Hash = hash.String()
} }
wg.Wait() wg.Wait()
awg.Done()
if err == nil { if err == nil {
first512 := make([]byte, 512) first512 := make([]byte, 512)
fread, _ := sr.ReadAt(first512, 0) fread, _ := f.ReadAt(first512, 0)
if fread > 0 { if fread > 0 {
mimeType := http.DetectContentType(first512[:fread]) mimeType := http.DetectContentType(first512[:fread])
if filepath.Ext(entry.Path) == ".css" { if filepath.Ext(entry.Path) == ".css" {
@ -129,6 +131,7 @@ func (self *FileSystem) Upload(lpath, index string) (string, error) {
trie := &manifestTrie{ trie := &manifestTrie{
dpa: self.api.dpa, dpa: self.api.dpa,
} }
quitC := make(chan bool)
for i, entry := range list { for i, entry := range list {
if errors[i] != nil { if errors[i] != nil {
return "", errors[i] return "", errors[i]
@ -140,9 +143,9 @@ func (self *FileSystem) Upload(lpath, index string) (string, error) {
Hash: entry.Hash, Hash: entry.Hash,
ContentType: entry.ContentType, ContentType: entry.ContentType,
} }
trie.addEntry(ientry) trie.addEntry(ientry, quitC)
} }
trie.addEntry(entry) trie.addEntry(entry, quitC)
} }
err2 := trie.recalcAndStore() err2 := trie.recalcAndStore()
@ -150,6 +153,7 @@ func (self *FileSystem) Upload(lpath, index string) (string, error) {
if err2 == nil { if err2 == nil {
hs = trie.hash.String() hs = trie.hash.String()
} }
awg.Wait()
return hs, err2 return hs, err2
} }
@ -170,11 +174,13 @@ func (self *FileSystem) Download(bzzpath, localpath string) error {
if err != nil { if err != nil {
return err return err
} }
// if len(path) > 0 {
// path += "/"
// }
trie, err := loadManifest(self.api.dpa, key) if len(path) > 0 {
path += "/"
}
quitC := make(chan bool)
trie, err := loadManifest(self.api.dpa, key, quitC)
if err != nil { if err != nil {
glog.V(logger.Warn).Infof("[BZZ] fs.Download: loadManifestTrie error: %v", err) glog.V(logger.Warn).Infof("[BZZ] fs.Download: loadManifestTrie error: %v", err)
return err return err
@ -186,72 +192,76 @@ func (self *FileSystem) Download(bzzpath, localpath string) error {
} }
var list []*downloadListEntry var list []*downloadListEntry
var mde, mderr error var mde error
prevPath := lpath prevPath := lpath
err = trie.listWithPrefix(path, func(entry *manifestTrieEntry, suffix string) { // TODO: paralellize err = trie.listWithPrefix(path, quitC, func(entry *manifestTrieEntry, suffix string) {
glog.V(logger.Detail).Infof("[BZZ] fs.Download: %#v", entry) glog.V(logger.Detail).Infof("[BZZ] fs.Download: %#v", entry)
key := common.Hex2Bytes(entry.Hash) key = common.Hex2Bytes(entry.Hash)
path := lpath + "/" + suffix path := lpath + "/" + suffix
dir := filepath.Dir(path) dir := filepath.Dir(path)
if dir != prevPath { if dir != prevPath {
mde = os.MkdirAll(dir, os.ModePerm) mde = os.MkdirAll(dir, os.ModePerm)
if mde != nil {
mderr = mde
}
prevPath = dir prevPath = dir
} }
if (mde == nil) && (path != dir+"/") { if (mde == nil) && (path != dir+"/") {
list = append(list, &downloadListEntry{key: key, path: path}) list = append(list, &downloadListEntry{key: key, path: path})
} }
}) })
if err == nil {
err = mderr
}
cnt := len(list)
errors := make([]error, cnt)
done := make(chan bool, maxParallelFiles)
dcnt := 0
for i, entry := range list {
if i >= dcnt+maxParallelFiles {
<-done
dcnt++
}
go func(i int, entry *downloadListEntry, done chan bool) {
f, err := os.Create(entry.path) // TODO: path separators
if err == nil {
reader := self.api.dpa.Retrieve(entry.key)
writer := bufio.NewWriter(f)
_, err = io.CopyN(writer, reader, reader.Size()) // TODO: handle errors
err2 := writer.Flush()
if err == nil {
err = err2
}
err2 = f.Close()
if err == nil {
err = err2
}
}
errors[i] = err
done <- true
}(i, entry, done)
}
for dcnt < cnt {
<-done
dcnt++
}
if err != nil { if err != nil {
return err return err
} }
for i, _ := range list {
if errors[i] != nil { wg := sync.WaitGroup{}
return errors[i] errC := make(chan error)
done := make(chan bool, maxParallelFiles)
for i, entry := range list {
select {
case done <- true:
wg.Add(1)
case <-quitC:
return fmt.Errorf("aborted")
} }
go func(i int, entry *downloadListEntry) {
defer wg.Done()
f, err := os.Create(entry.path) // TODO: path separators
if err == nil {
reader := self.api.dpa.Retrieve(entry.key)
writer := bufio.NewWriter(f)
size, err := reader.Size(quitC)
if err == nil {
_, err = io.CopyN(writer, reader, size) // TODO: handle errors
err2 := writer.Flush()
if err == nil {
err = err2
}
err2 = f.Close()
if err == nil {
err = err2
}
}
}
if err != nil {
select {
case errC <- err:
case <-quitC:
}
return
}
<-done
}(i, entry)
} }
return err go func() {
wg.Wait()
close(errC)
}()
select {
case err = <-errC:
return err
case <-quitC:
return fmt.Errorf("aborted")
}
} }

29
swarm/api/filesystem_test.go
View file

@ -1,10 +1,12 @@
package api package api
import ( import (
"bytes"
"io/ioutil" "io/ioutil"
"os" "os"
"path" "path"
"runtime" "runtime"
"sync"
"testing" "testing"
) )
@ -26,9 +28,9 @@ func testFileSystem(t *testing.T, f func(*FileSystem)) {
} }
func readPath(t *testing.T, parts ...string) string { func readPath(t *testing.T, parts ...string) string {
// func readPath(t *testing.T, parts ...string) []byte {
file := path.Join(parts...) file := path.Join(parts...)
content, err := ioutil.ReadFile(file) content, err := ioutil.ReadFile(file)
if err != nil { if err != nil {
t.Fatalf("unexpected error reading '%v': %v", file, err) t.Fatalf("unexpected error reading '%v': %v", file, err)
} }
@ -36,14 +38,12 @@ func readPath(t *testing.T, parts ...string) string {
} }
func TestApiDirUpload0(t *testing.T) { func TestApiDirUpload0(t *testing.T) {
// t.Skip("FIXME")
testFileSystem(t, func(fs *FileSystem) { testFileSystem(t, func(fs *FileSystem) {
api := fs.api api := fs.api
bzzhash, err := fs.Upload(path.Join(testDir, "test0"), "") bzzhash, err := fs.Upload(path.Join(testDir, "test0"), "")
if err != nil { if err != nil {
t.Fatalf("unexpected error: %v", err) t.Fatalf("unexpected error: %v", err)
} }
content := readPath(t, testDir, "test0", "index.html") content := readPath(t, testDir, "test0", "index.html")
resp := testGet(t, api, bzzhash+"/index.html") resp := testGet(t, api, bzzhash+"/index.html")
exp := expResponse(content, "text/html; charset=utf-8", 0) exp := expResponse(content, "text/html; charset=utf-8", 0)
@ -54,17 +54,12 @@ func TestApiDirUpload0(t *testing.T) {
exp = expResponse(content, "text/css", 0) exp = expResponse(content, "text/css", 0)
checkResponse(t, resp, exp) checkResponse(t, resp, exp)
content = readPath(t, testDir, "test0", "img", "logo.png")
resp = testGet(t, api, bzzhash+"/img/logo.png")
exp = expResponse(content, "image/png", 0)
_, _, _, err = api.Get(bzzhash, true) _, _, _, err = api.Get(bzzhash, true)
if err == nil { if err == nil {
t.Fatalf("expected error: %v", err) t.Fatalf("expected error: %v", err)
} }
downloadDir := path.Join(testDownloadDir, "test0") downloadDir := path.Join(testDownloadDir, "test0")
os.RemoveAll(downloadDir)
defer os.RemoveAll(downloadDir) defer os.RemoveAll(downloadDir)
err = fs.Download(bzzhash, downloadDir) err = fs.Download(bzzhash, downloadDir)
if err != nil { if err != nil {
@ -77,12 +72,10 @@ func TestApiDirUpload0(t *testing.T) {
if bzzhash != newbzzhash { if bzzhash != newbzzhash {
t.Fatalf("download %v reuploaded has incorrect hash, expected %v, got %v", downloadDir, bzzhash, newbzzhash) t.Fatalf("download %v reuploaded has incorrect hash, expected %v, got %v", downloadDir, bzzhash, newbzzhash)
} }
}) })
} }
func TestApiDirUploadModify(t *testing.T) { func TestApiDirUploadModify(t *testing.T) {
// t.Skip("FIXME")
testFileSystem(t, func(fs *FileSystem) { testFileSystem(t, func(fs *FileSystem) {
api := fs.api api := fs.api
bzzhash, err := fs.Upload(path.Join(testDir, "test0"), "") bzzhash, err := fs.Upload(path.Join(testDir, "test0"), "")
@ -96,12 +89,24 @@ func TestApiDirUploadModify(t *testing.T) {
t.Errorf("unexpected error: %v", err) t.Errorf("unexpected error: %v", err)
return return
} }
bzzhash, err = api.Modify(bzzhash+"/index2.html", "9ea1f60ebd80786d6005f6b256376bdb494a82496cd86fe8c307cdfb23c99e71", "text/html; charset=utf-8", true) index, err := ioutil.ReadFile(path.Join(testDir, "test0", "index.html"))
if err != nil { if err != nil {
t.Errorf("unexpected error: %v", err) t.Errorf("unexpected error: %v", err)
return return
} }
bzzhash, err = api.Modify(bzzhash+"/img/logo.png", "9ea1f60ebd80786d6005f6b256376bdb494a82496cd86fe8c307cdfb23c99e71", "text/html; charset=utf-8", true) wg := &sync.WaitGroup{}
hash, err := api.Store(bytes.NewReader(index), int64(len(index)), wg)
wg.Wait()
if err != nil {
t.Errorf("unexpected error: %v", err)
return
}
bzzhash, err = api.Modify(bzzhash+"/index2.html", hash.Hex(), "text/html; charset=utf-8", true)
if err != nil {
t.Errorf("unexpected error: %v", err)
return
}
bzzhash, err = api.Modify(bzzhash+"/img/logo.png", hash.Hex(), "text/html; charset=utf-8", true)
if err != nil { if err != nil {
t.Errorf("unexpected error: %v", err) t.Errorf("unexpected error: %v", err)
return return

6
swarm/api/http/roundtripper_test.go
View file

@ -10,6 +10,8 @@ import (
"github.com/ethereum/go-ethereum/common/httpclient" "github.com/ethereum/go-ethereum/common/httpclient"
) )
const port = "3222"
func TestRoundTripper(t *testing.T) { func TestRoundTripper(t *testing.T) {
serveMux := http.NewServeMux() serveMux := http.NewServeMux()
serveMux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { serveMux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
@ -20,9 +22,9 @@ func TestRoundTripper(t *testing.T) {
http.Error(w, "Method "+r.Method+" is not supported.", http.StatusMethodNotAllowed) http.Error(w, "Method "+r.Method+" is not supported.", http.StatusMethodNotAllowed)
} }
}) })
go http.ListenAndServe(":8600", serveMux) go http.ListenAndServe(":"+port, serveMux)
rt := &RoundTripper{Port: "8600"} rt := &RoundTripper{Port: port}
client := httpclient.New("/") client := httpclient.New("/")
client.RegisterProtocol("bzz", rt) client.RegisterProtocol("bzz", rt)

21
swarm/api/http/server.go
View file

@ -86,8 +86,10 @@ func handler(w http.ResponseWriter, r *http.Request, a *api.Api) {
return return
} }
} }
raw = proto[1:5] == "bzzr" if len(proto) > 4 {
nameresolver = proto[1:5] != "bzzi" raw = proto[1:5] == "bzzr"
nameresolver = proto[1:5] != "bzzi"
}
glog.V(logger.Debug).Infof( glog.V(logger.Debug).Infof(
"[BZZ] Swarm: %s request over protocol %s '%s' received.", "[BZZ] Swarm: %s request over protocol %s '%s' received.",
@ -96,10 +98,7 @@ func handler(w http.ResponseWriter, r *http.Request, a *api.Api) {
switch { switch {
case r.Method == "POST" || r.Method == "PUT": case r.Method == "POST" || r.Method == "PUT":
key, err := a.Store(io.NewSectionReader(&sequentialReader{ key, err := a.Store(r.Body, r.ContentLength, nil)
reader: r.Body,
ahead: make(map[int64]chan bool),
}, 0, r.ContentLength), nil)
if err == nil { if err == nil {
glog.V(logger.Debug).Infof("[BZZ] Swarm: Content for %v stored", key.Log()) glog.V(logger.Debug).Infof("[BZZ] Swarm: Content for %v stored", key.Log())
} else { } else {
@ -165,7 +164,9 @@ func handler(w http.ResponseWriter, r *http.Request, a *api.Api) {
// retrieving content // retrieving content
reader := a.Retrieve(key) reader := a.Retrieve(key)
glog.V(logger.Debug).Infof("[BZZ] Swarm: Reading %d bytes.", reader.Size()) quitC := make(chan bool)
size, err := reader.Size(quitC)
glog.V(logger.Debug).Infof("[BZZ] Swarm: Reading %d bytes.", size)
// setting mime type // setting mime type
qv := requestURL.Query() qv := requestURL.Query()
@ -176,7 +177,7 @@ func handler(w http.ResponseWriter, r *http.Request, a *api.Api) {
w.Header().Set("Content-Type", mimeType) w.Header().Set("Content-Type", mimeType)
http.ServeContent(w, r, uri, forever(), reader) http.ServeContent(w, r, uri, forever(), reader)
glog.V(logger.Debug).Infof("[BZZ] Swarm: Serve raw content '%s' (%d bytes) as '%s'", uri, reader.Size(), mimeType) glog.V(logger.Debug).Infof("[BZZ] Swarm: Serve raw content '%s' (%d bytes) as '%s'", uri, size, mimeType)
// retrieve path via manifest // retrieve path via manifest
} else { } else {
@ -203,7 +204,9 @@ func handler(w http.ResponseWriter, r *http.Request, a *api.Api) {
} else { } else {
status = 200 status = 200
} }
glog.V(logger.Debug).Infof("[BZZ] Swarm: Served '%s' (%d bytes) as '%s' (status code: %v)", uri, reader.Size(), mimeType, status) quitC := make(chan bool)
size, err := reader.Size(quitC)
glog.V(logger.Debug).Infof("[BZZ] Swarm: Served '%s' (%d bytes) as '%s' (status code: %v)", uri, size, mimeType, status)
http.ServeContent(w, r, path, forever(), reader) http.ServeContent(w, r, path, forever(), reader)

77
swarm/api/manifest.go
View file

@ -4,7 +4,6 @@ import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"fmt" "fmt"
"io"
"sync" "sync"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
@ -35,24 +34,24 @@ type manifestTrieEntry struct {
subtrie *manifestTrie subtrie *manifestTrie
} }
func loadManifest(dpa *storage.DPA, hash storage.Key) (trie *manifestTrie, err error) { // non-recursive, subtrees are downloaded on-demand func loadManifest(dpa *storage.DPA, hash storage.Key, quitC chan bool) (trie *manifestTrie, err error) { // non-recursive, subtrees are downloaded on-demand
glog.V(logger.Detail).Infof("[BZZ] manifest lookup key: '%v'.", hash.Log()) glog.V(logger.Detail).Infof("[BZZ] manifest lookup key: '%v'.", hash.Log())
// retrieve manifest via DPA // retrieve manifest via DPA
manifestReader := dpa.Retrieve(hash) manifestReader := dpa.Retrieve(hash)
return readManifest(manifestReader, hash, dpa) return readManifest(manifestReader, hash, dpa, quitC)
} }
func readManifest(manifestReader storage.SectionReader, hash storage.Key, dpa *storage.DPA) (trie *manifestTrie, err error) { // non-recursive, subtrees are downloaded on-demand func readManifest(manifestReader storage.LazySectionReader, hash storage.Key, dpa *storage.DPA, quitC chan bool) (trie *manifestTrie, err error) { // non-recursive, subtrees are downloaded on-demand
// TODO check size for oversized manifests // TODO check size for oversized manifests
manifestData := make([]byte, manifestReader.Size()) size, err := manifestReader.Size(quitC)
var size int manifestData := make([]byte, size)
size, err = manifestReader.Read(manifestData) read, err := manifestReader.Read(manifestData)
if int64(size) < manifestReader.Size() { if int64(read) < size {
glog.V(logger.Detail).Infof("[BZZ] Manifest %v not found.", hash.Log()) glog.V(logger.Detail).Infof("[BZZ] Manifest %v not found.", hash.Log())
if err == nil { if err == nil {
err = fmt.Errorf("Manifest retrieval cut short: read %v, expect %v", size, manifestReader.Size()) err = fmt.Errorf("Manifest retrieval cut short: read %v, expect %v", read, size)
} }
return return
} }
@ -72,12 +71,12 @@ func readManifest(manifestReader storage.SectionReader, hash storage.Key, dpa *s
dpa: dpa, dpa: dpa,
} }
for _, entry := range man.Entries { for _, entry := range man.Entries {
trie.addEntry(entry) trie.addEntry(entry, quitC)
} }
return return
} }
func (self *manifestTrie) addEntry(entry *manifestTrieEntry) { func (self *manifestTrie) addEntry(entry *manifestTrieEntry, quitC chan bool) {
self.hash = nil // trie modified, hash needs to be re-calculated on demand self.hash = nil // trie modified, hash needs to be re-calculated on demand
if len(entry.Path) == 0 { if len(entry.Path) == 0 {
@ -98,11 +97,11 @@ func (self *manifestTrie) addEntry(entry *manifestTrieEntry) {
} }
if (oldentry.ContentType == manifestType) && (cpl == len(oldentry.Path)) { if (oldentry.ContentType == manifestType) && (cpl == len(oldentry.Path)) {
if self.loadSubTrie(oldentry) != nil { if self.loadSubTrie(oldentry, quitC) != nil {
return return
} }
entry.Path = entry.Path[cpl:] entry.Path = entry.Path[cpl:]
oldentry.subtrie.addEntry(entry) oldentry.subtrie.addEntry(entry, quitC)
oldentry.Hash = "" oldentry.Hash = ""
return return
} }
@ -114,8 +113,8 @@ func (self *manifestTrie) addEntry(entry *manifestTrieEntry) {
} }
entry.Path = entry.Path[cpl:] entry.Path = entry.Path[cpl:]
oldentry.Path = oldentry.Path[cpl:] oldentry.Path = oldentry.Path[cpl:]
subtrie.addEntry(entry) subtrie.addEntry(entry, quitC)
subtrie.addEntry(oldentry) subtrie.addEntry(oldentry, quitC)
self.entries[b] = &manifestTrieEntry{ self.entries[b] = &manifestTrieEntry{
Path: commonPrefix, Path: commonPrefix,
@ -135,7 +134,7 @@ func (self *manifestTrie) getCountLast() (cnt int, entry *manifestTrieEntry) {
return return
} }
func (self *manifestTrie) deleteEntry(path string) { func (self *manifestTrie) deleteEntry(path string, quitC chan bool) {
self.hash = nil // trie modified, hash needs to be re-calculated on demand self.hash = nil // trie modified, hash needs to be re-calculated on demand
if len(path) == 0 { if len(path) == 0 {
@ -155,10 +154,10 @@ func (self *manifestTrie) deleteEntry(path string) {
epl := len(entry.Path) epl := len(entry.Path)
if (entry.ContentType == manifestType) && (len(path) >= epl) && (path[:epl] == entry.Path) { if (entry.ContentType == manifestType) && (len(path) >= epl) && (path[:epl] == entry.Path) {
if self.loadSubTrie(entry) != nil { if self.loadSubTrie(entry, quitC) != nil {
return return
} }
entry.subtrie.deleteEntry(path[epl:]) entry.subtrie.deleteEntry(path[epl:], quitC)
entry.Hash = "" entry.Hash = ""
// remove subtree if it has less than 2 elements // remove subtree if it has less than 2 elements
cnt, lastentry := entry.subtrie.getCountLast() cnt, lastentry := entry.subtrie.getCountLast()
@ -198,24 +197,24 @@ func (self *manifestTrie) recalcAndStore() error {
return err return err
} }
sr := io.NewSectionReader(bytes.NewReader(manifest), 0, int64(len(manifest))) sr := bytes.NewReader(manifest)
wg := &sync.WaitGroup{} wg := &sync.WaitGroup{}
key, err2 := self.dpa.Store(sr, wg) key, err2 := self.dpa.Store(sr, int64(len(manifest)), wg)
wg.Wait() wg.Wait()
self.hash = key self.hash = key
return err2 return err2
} }
func (self *manifestTrie) loadSubTrie(entry *manifestTrieEntry) (err error) { func (self *manifestTrie) loadSubTrie(entry *manifestTrieEntry, quitC chan bool) (err error) {
if entry.subtrie == nil { if entry.subtrie == nil {
hash := common.Hex2Bytes(entry.Hash) hash := common.Hex2Bytes(entry.Hash)
entry.subtrie, err = loadManifest(self.dpa, hash) entry.subtrie, err = loadManifest(self.dpa, hash, quitC)
entry.Hash = "" // might not match, should be recalculated entry.Hash = "" // might not match, should be recalculated
} }
return return
} }
func (self *manifestTrie) listWithPrefixInt(prefix, rp string, cb func(entry *manifestTrieEntry, suffix string)) (err error) { func (self *manifestTrie) listWithPrefixInt(prefix, rp string, quitC chan bool, cb func(entry *manifestTrieEntry, suffix string)) error {
plen := len(prefix) plen := len(prefix)
var start, stop int var start, stop int
if plen == 0 { if plen == 0 {
@ -227,6 +226,11 @@ func (self *manifestTrie) listWithPrefixInt(prefix, rp string, cb func(entry *ma
} }
for i := start; i <= stop; i++ { for i := start; i <= stop; i++ {
select {
case <-quitC:
return fmt.Errorf("aborted")
default:
}
entry := self.entries[i] entry := self.entries[i]
if entry != nil { if entry != nil {
epl := len(entry.Path) epl := len(entry.Path)
@ -236,11 +240,13 @@ func (self *manifestTrie) listWithPrefixInt(prefix, rp string, cb func(entry *ma
l = epl l = epl
} }
if prefix[:l] == entry.Path[:l] { if prefix[:l] == entry.Path[:l] {
sterr := self.loadSubTrie(entry) err := self.loadSubTrie(entry, quitC)
if sterr == nil { if err != nil {
entry.subtrie.listWithPrefixInt(prefix[l:], rp+entry.Path[l:], cb) return err
} else { }
err = sterr err = entry.subtrie.listWithPrefixInt(prefix[l:], rp+entry.Path[l:], quitC, cb)
if err != nil {
return err
} }
} }
} else { } else {
@ -250,14 +256,14 @@ func (self *manifestTrie) listWithPrefixInt(prefix, rp string, cb func(entry *ma
} }
} }
} }
return return nil
} }
func (self *manifestTrie) listWithPrefix(prefix string, cb func(entry *manifestTrieEntry, suffix string)) (err error) { func (self *manifestTrie) listWithPrefix(prefix string, quitC chan bool, cb func(entry *manifestTrieEntry, suffix string)) (err error) {
return self.listWithPrefixInt(prefix, "", cb) return self.listWithPrefixInt(prefix, "", quitC, cb)
} }
func (self *manifestTrie) findPrefixOf(path string) (entry *manifestTrieEntry, pos int) { func (self *manifestTrie) findPrefixOf(path string, quitC chan bool) (entry *manifestTrieEntry, pos int) {
glog.V(logger.Detail).Infof("[BZZ] findPrefixOf(%s)", path) glog.V(logger.Detail).Infof("[BZZ] findPrefixOf(%s)", path)
@ -275,10 +281,10 @@ func (self *manifestTrie) findPrefixOf(path string) (entry *manifestTrieEntry, p
if (len(path) >= epl) && (path[:epl] == entry.Path) { if (len(path) >= epl) && (path[:epl] == entry.Path) {
glog.V(logger.Detail).Infof("[BZZ] entry.ContentType = %v", entry.ContentType) glog.V(logger.Detail).Infof("[BZZ] entry.ContentType = %v", entry.ContentType)
if entry.ContentType == manifestType { if entry.ContentType == manifestType {
if self.loadSubTrie(entry) != nil { if self.loadSubTrie(entry, quitC) != nil {
return nil, 0 return nil, 0
} }
entry, pos = entry.subtrie.findPrefixOf(path[epl:]) entry, pos = entry.subtrie.findPrefixOf(path[epl:], quitC)
if entry != nil { if entry != nil {
pos += epl pos += epl
} }
@ -308,6 +314,7 @@ func RegularSlashes(path string) (res string) {
func (self *manifestTrie) getEntry(spath string) (entry *manifestTrieEntry, fullpath string) { func (self *manifestTrie) getEntry(spath string) (entry *manifestTrieEntry, fullpath string) {
path := RegularSlashes(spath) path := RegularSlashes(spath)
var pos int var pos int
entry, pos = self.findPrefixOf(path) quitC := make(chan bool)
entry, pos = self.findPrefixOf(path, quitC)
return entry, path[:pos] return entry, path[:pos]
} }

7
swarm/api/manifest_test.go
View file

@ -10,18 +10,19 @@ import (
"github.com/ethereum/go-ethereum/swarm/storage" "github.com/ethereum/go-ethereum/swarm/storage"
) )
func manifest(paths ...string) (manifestReader storage.SectionReader) { func manifest(paths ...string) (manifestReader storage.LazySectionReader) {
var entries []string var entries []string
for _, path := range paths { for _, path := range paths {
entry := fmt.Sprintf(`{"path":"%s"}`, path) entry := fmt.Sprintf(`{"path":"%s"}`, path)
entries = append(entries, entry) entries = append(entries, entry)
} }
manifest := fmt.Sprintf(`{"entries":[%s]}`, strings.Join(entries, ",")) manifest := fmt.Sprintf(`{"entries":[%s]}`, strings.Join(entries, ","))
return io.NewSectionReader(strings.NewReader(manifest), 0, int64(len(manifest))) return &storage.LazyTestSectionReader{io.NewSectionReader(strings.NewReader(manifest), 0, int64(len(manifest)))}
} }
func testGetEntry(t *testing.T, path, match string, paths ...string) *manifestTrie { func testGetEntry(t *testing.T, path, match string, paths ...string) *manifestTrie {
trie, err := readManifest(manifest(paths...), nil, nil) quitC := make(chan bool)
trie, err := readManifest(manifest(paths...), nil, nil, quitC)
if err != nil { if err != nil {
t.Errorf("unexpected error making manifest: %v", err) t.Errorf("unexpected error making manifest: %v", err)
} }

8
swarm/api/storage.go
View file

@ -34,7 +34,11 @@ func (self *Storage) Get(bzzpath string) (*Response, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
expsize := reader.Size() quitC := make(chan bool)
expsize, err := reader.Size(quitC)
if err != nil {
return nil, err
}
body := make([]byte, expsize) body := make([]byte, expsize)
size, err := reader.Read(body) size, err := reader.Read(body)
if int64(size) == expsize { if int64(size) == expsize {
@ -43,6 +47,8 @@ func (self *Storage) Get(bzzpath string) (*Response, error) {
return &Response{mimeType, status, expsize, string(body[:size])}, err return &Response{mimeType, status, expsize, string(body[:size])}, err
} }
// Modify(rootHash, path, contentHash, contentType) takes th e manifest trie rooted in rootHash,
// and merge on to it. creating an entry w conentType (mime)
func (self *Storage) Modify(rootHash, path, contentHash, contentType string) (newRootHash string, err error) { func (self *Storage) Modify(rootHash, path, contentHash, contentType string) (newRootHash string, err error) {
return self.api.Modify(rootHash+"/"+path, contentHash, contentType, true) return self.api.Modify(rootHash+"/"+path, contentHash, contentType, true)
} }

229
swarm/cmd/README.md Normal file
View file

@ -0,0 +1,229 @@
# install and setup swarm
swarm is developed on a branch of the ethereum/go-ethereum repo
at this stage of the project there is no packages or binary distro, you need to a dev environment and compile from source.
[This document spells out a complete server setup on ubuntu linux](https://gist.github.com/zelig/74eb365752ceaacf15e860fb80eacb3e) including git/ssh/screen config, golang and compilation, node/npm and network monitoring (might contain a few bits that are tangential to swarm).
Assuming you got your setup working, you will use the `swarm` command line tool to control your swarm of instances.
This command line tool is at the moment geared towards developers and testing.
It is likely that it will be replaced by two different tools, one for devel/testing and one for end users
The command can be used to update the code
```shell
swarm update upstream/swarm
```
Then compile with
```shell
godep go build -v ./cmd/geth
```
Make sure you have `GOPATH` variable set and also that the `swarm` executable is in your PATH.
These environment variables are relevant and set to the following defaults.
Make sure you are happy with them, otherwise change them, in which case best to put these lines in your `~/.profile`.
```
export GETH_DIR=$GOPATH/src/github.com/ethereum/go-ethereum
export GETH=$GETH_DIR/geth
export SWARM_DIR=~/bzz
export SWARM_NETWORK_ID=322
```
* `GETH_DIR` points to your git working copy (given `GOPATH` its standardly under `$GOPATH/src/github.com/ethereum/go-ethereum`)
* `GETH` points to the `geth` executable compiled from the swarm branch. If you have systemwide install or use multiple geths you may need to change this, otherwise it is assumed you compile to the working copy of the repo.
* `SWARM_DIR` is the root directory for all swarm related stuff: logs, configs, as well as geth datadirs, make sure this dir is on a device with sufficient disk space
* `SWARM_NETWORK_ID`: this is by default the network id of the swarm testnet. If you run your own swarm, you need to change it, choose a number that is not likely chosen by others to avoid others joining you.
# Deploying and remote control
the swarm command supports remote update and remote control of your instances.
In our setting we assume you want to run a cluster of potentially remote swarm nodes each running a local cluster of instances
The only assumption is that you have (passwordless) ssh access set up to your swarm servers.
Assume `nodes.lst` is a list of nodes in the format of `username@ip` one per line. blank lines and lines commented out with `#` are ignored.
This copies the scripts found in `swarm/cmd/swarm` on all remote nodes listed in `nodes.lst`
```
swarm remote-update-scripts nodes.lst
```
If you just want to deploy a locally compiled binaries to all your remote nodes, this will fail if the remote instances are running, so make sure you stop them beforehand
```shell
swarm remote-run nodes.lst swarm stop all
swarm remote-update-bin nodes.lst
```
Once you deployed the executables to the nodes, you can control them all with one command. For instance the following line initialises a cluster of two test swarm instances on each remote node.
Watch out, this will wipe your storage and all swarm related data
```shell
swarm remote-run nodes.lst swarm init 2
```
To (re) start a particular instance on a specific remote node with alternative options (for instance mining and different logging verbosity), you can just:
```shell
swarm remote-run cicada@3.3.0.1 'swarm restart 01 --mine --verbosity=0 --vmodule=swarm/*=5'
```
# Logging
To check logs
```shell
swarm log 00 # taillog flow
swarm remote-run cicada@3.3.0.1 swarm log 00
```
You can view the log with a pager for an instance with
```
swarm viewlog 00
```
Logs are preserved and viewable with the above commands even when nodes are offline
Each new run logs to a different file
To purge logs
```
swarm cleanlog 01
```
To remove all logs on all nodes:
```
swarm remote-run nodes.lst swarm cleanlog all
```
# upload and dowload
upload and download via a running local instance
```shell
swarm up 00 /path/to/file/or/directory
swarm down 01 hash /path/to/destination
```
upload via remote swarm proxy or public gateway
```shell
swarm remote-up gateway-url /path/to/file/or/directory
wget -O- gateway-url/bzz:/swarm-url
```
# Further examples
```shell
# start with updaing
swarm update chambers
# display CLI options given to geth used to launch swarm instance 02
swarm options 02
# restart swarm instance 00 with alternatiev options
swarm restart 00 --mine --bzznosync --verbosity=0 --vmodule=swarm/*=6
# attach console to a running swarm instance
swarm attach 00
# execute a command; e.g., start mining on a running instance
swarm execute 00 'miner.stop(1)'
# display static info about a instance (even if its offline)
swarm info 00
# displays the enode url of a running instance
swarm enode 01
# add peers to a running swarm instance
swarm addpeers 00 "enode://1033c1cada...@3.3.0.1:30301"
# to compile a list of enodes from all instances on all remote nodes:
swarm remote-run nodes.lst 'swarm enode all' > enodes.lst
# to add all peers to all instances on each node
for node in `cat nodes.lst|grep -v '^#'`; do scp enodes.lst $node:; done
swarm remote-run nodes.lst 'swarm addpeers enodes.lst'
# if you run a local network and your nodes do not listen to external IPs
swarm remote-run pivot.lst 'swarm restart all'
# to add just one or a few guardians and let the network bootstrap
# swarm remote-run 'swarm enode all'
swarm addpeers pivot.lst
# or directly
swarm addpeers all <(IP_ADDR='[::]' swarm enode 01|tr -d '"')
# stop all running instances on the node
swarm stop all
# stop all running instances on all remote nodes
swarm remote-run nodes.lst swarm stop all
# display peer connection table of running instance 00
swarm hive 00
# display peer connection table for a running instance and continually refresh every 4 seconds
swarm monitor 00 4
# display peer connection table for all running instance on a remote node and continually refresh every 10 seconds
swarm monitor cicada@3.3.0.1 all 10
# configure eth-net-intelligence-api network monitoring client API for a node (the name argument appears as a prefix for all instances in your cluster)
swarm netstatconf cicada-sworm
# restart the net monitor client API
swarm netstatun
# configure eth-net-intelligence-api network monitoring client API and (re)start the monitor tool on all remote nodes
swarm remote-run nodes.lst 'swarm netstatconf cicada-sworm; swarm netstatrun'
swarm remote nodes.lst 'swarm netstatconf cicada-sworm; swarm netstatrun'
```
see also:
* https://github.com/ethereum/go-ethereum/tree/swarm/swarm/test
* https://github.com/ethereum/go-ethereum/tree/swarm/swarm/cmd
# ethereum netstats client setup
## install
nodejs and npm are prerequisites
```shell
# MAC
brew install node npm
# ubuntu
sudo apt-get install npm nodejs
```
clone the git repo and install:
```
git clone git@github.com:cubedro/eth-net-intelligence-api.git
cd eth-net-intelligence-api
npm install
npm install -g pm2
```
## configure and run netstats client for each node
```shell
swarm remote-run nodes.lst 'swarm netstatconf cicada-sworm; swarm netstatrun'
```

11
swarm/cmd/bzzhash/bzzhash.go
View file

@ -3,7 +3,6 @@ package main
import ( import (
"fmt" "fmt"
"io"
"os" "os"
"runtime" "runtime"
@ -24,15 +23,11 @@ func main() {
} }
stat, _ := f.Stat() stat, _ := f.Stat()
sr := io.NewSectionReader(f, 0, stat.Size())
chunker := storage.NewTreeChunker(storage.NewChunkerParams()) chunker := storage.NewTreeChunker(storage.NewChunkerParams())
hash := make([]byte, chunker.KeySize()) key, err := chunker.Split(f, stat.Size(), nil, nil, nil)
errC := chunker.Split(hash, sr, nil, nil)
err, ok := <-errC
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err) fmt.Fprintf(os.Stderr, "%v\n", err)
} } else {
if !ok { fmt.Printf("%v\n", key)
fmt.Printf("%064x\n", hash)
} }
} }

6
swarm/cmd/swarm/env.sh Normal file
View file

@ -0,0 +1,6 @@
export PATH=$HOME/bin:$PATH
export GETH_DIR=$HOME/bin
export SWARM_DIR=
export NVM_DIR=$HOME/.nvm
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm

149
swarm/cmd/swarm/gethup.sh
View file

@ -1,149 +0,0 @@
#!/bin/bash
# Usage:
# bash /path/to/eth-utils/gethup.sh <datadir> <instance_name> <ip_addr>
root=$1 # base directory to use for datadir and logs
shift
id=$1 # double digit instance id like 00 01 02
shift
ip_addr=$1 # ip address to substitute
shift
# logs are output to a date-tagged file for each run , while a link is
# created to the latest, so that monitoring be easier with the same filename
# TODO: use this if GETH not set
# GETH=geth
# echo "ls -l $GETH"
# ls -l $GETH
# geth CLI params e.g., (dd=04, run=09)
datetag=`date "+%c%y%m%d-%H%M%S"|cut -d ' ' -f 5`
datadir=$root/data/$id # /tmp/eth/04
log=$root/log/$id.$datetag.log # /tmp/eth/04.09.log
linklog=$root/log/$id.current.log # /tmp/eth/04.09.log
stablelog=$root/log/$id.log # /tmp/eth/04.09.log
password=$id # 04
port=303$id # 34504
bzzport=322$id # 32204
rpcport=302$id # 3204
mkdir -p $root/data
mkdir -p $root/enodes
mkdir -p $root/pids
mkdir -p $root/log
ln -sf "$log" "$linklog"
# if we do not have an account, create one
# will not prompt for password, we use the double digit instance id as passwd
# NEVER EVER USE THESE ACCOUNTS FOR INTERACTING WITH A LIVE CHAIN
keystoredir="$datadir/keystore/"
# echo "KeyStore dir: $keystoredir"
if [ ! -d "$keystoredir" ]; then
# echo "create an account with password $id [DO NOT EVER USE THIS ON LIVE]"
# mkdir -p $datadir/keystore
$GETH --datadir $datadir --password <(echo -n $id) account new >/dev/null 2>&1
# create account with password 00, 01, ...
# note that the account key will be stored also separately outside
# datadir
# this way you can safely clear the data directory and still keep your key
# under `<rootdir>/keystore/dd
# LS=`ls $datadir/keystore`
# echo $LS
while [ ! -d "$keystoredir" ]; do
echo "."
((i++))
if ((i>10)); then break; fi
sleep 1
done
# echo "copying keys $datadir/keystore $root/keystore/$id"
mkdir -p $root/keystore/$id
cp -R "$datadir/keystore/" $root/keystore/$id
fi
# # mkdir -p $datadir/keystore
# if [ ! -d "$datadir/keystore" ]; then
# echo "copying keys $root/keystore/$id $datadir/keystore"
# cp -R $root/keystore/$id/keystore/ $datadir/keystore/
# fi
# query node's enode url
if [ $ip_addr="" ]; then
pattern='\d+\.\d+\.\d+\.\d+'
ip_addr="[::]"
else
pattern='\[\:\:\]'
fi
geth="$GETH --datadir $datadir --port $port"
# echo -n "enode for instance $id... "
if [ ! "$GETH" = "" ] && [ ! -f $root/enodes/$id.enode ]; then
cmd="$geth js <(echo 'console.log(admin.nodeInfo.enode); exit();') "
# echo $cmd '2>/dev/null |grep enode | perl -pe "s/'$pattern'/'$ip_addr'/g" | perl -pe "s/^/\"/; s/\s*$/\"/;" > '$root/enodes/$id.enode
eval $cmd 2>/dev/null |grep enode | perl -pe "s/$pattern/$ip_addr/g" | perl -pe "s/^/\"/; s/\s*\$/\"/;" > $root/enodes/$id.enode
fi
# cat $root/enodes/$id.enode
echo
# copy cluster enodes list to node's static node list
# echo "copy cluster enodes list to node's static node list"
if [ -f $root/enodes.all ]; then
cp $root/enodes.all $datadir/static-nodes.json
fi
if [ ! -f $root/pids/$id.pid ]; then
# bring up node `dd` (double digit)
# - using <rootdir>/<dd>
# - listening on port 303dd, (like 30300, 30301, ...)
# - with the account unlocked
# - launching json-rpc server on port 81dd (like 8100, 8101, 8102, ...)
# echo "BZZKEY=$geth account list|head -n1|perl -ne '/([a-f0-9]{40})/ && print \$1'"
BZZKEY=`$geth account list|head -n1|perl -ne '/([a-f0-9]{40})/ && print \$1'`
echo -n "starting instance $id ($BZZKEY @ $datadir )..."
# echo "$geth \
# --identity=$id \
# --bzzaccount=$BZZKEY --bzzport=$bzzport \
# --unlock=$BZZKEY \
# --password=<(echo -n $id) \
# --rpc --rpcport=$rpcport --rpccorsdomain='*' $* \
# 2>&1 | tee "$stablelog" > "$log" & # comment out if you pipe it to a tty etc.
# " >&2
if [ -f $log ] && [ -f $root/stablelog ]; then
cp $stablelog `cat $root/prevlog`
fi
echo $log > $root/prevlog
$GETH --datadir=$datadir \
--identity=$id \
--bzzaccount=$BZZKEY --bzzport=$bzzport \
--port=$port \
--unlock=$BZZKEY \
--password=<(echo -n $id) \
--rpc --rpcport=$rpcport --rpccorsdomain='*' $* \
> "$stablelog" 2>&1 & # comment out if you pipe it to a tty etc.
# wait until ready
# pid=`ps auxwww|grep geth|grep "ty=$id"|grep -v grep|awk '{print $2}'`
# echo "pid: $pid"
# ps auxwww|grep geth|grep "ty=$id"|grep -v grep
# echo $pid > $root/pids/$id.pid
#echo $! > $root/pids/$id.pid
while true; do
$GETH --exec="net" attach ipc:$datadir/geth.ipc > /dev/null 2>&1 && break
sleep 1
echo -n "."
if ((i++>10)); then
echo "instance $id failed to start"
exit 1
fi
done
echo -n "started - "
pid=`ps auxwww|grep geth|grep "ty=$id"|grep -v grep|awk '{print $2}'`
echo "pid: $pid"
# ps auxwww|grep geth|grep "ty=$id"|grep -v grep
echo $pid > $root/pids/$id.pid
fi
# to bring up logs, uncomment
# tail -f $log

759
swarm/cmd/swarm/swarm Executable file
View file

@ -0,0 +1,759 @@
#!/bin/bash
if [ "$GETH_DIR" = "" ]; then
if [ "$GOPATH" = "" ]; then echo "either GETH_DIR or GOPATH environment variable must be set"; exit 1; fi
export GETH_DIR=$GOPATH/src/github.com/ethereum/go-ethereum
fi
if [ "$GETH" = "" ]; then
export GETH=$GETH_DIR/geth
fi
if [ "$SWARM_NETWORK_ID" = "" ]; then export SWARM_NETWORK_ID=322; fi
if [ "$SWARM_DIR" = "" ]; then export SWARM_DIR=$HOME/bzz; fi
if [ "$IP_ADDR" = "" ]; then
# export IP_ADDR=`curl ipecho.net/plain 2>/dev/null;echo f`
export IP_ADDR=
fi
root=$SWARM_DIR
network_id=$SWARM_NETWORK_ID
cmd=$1
shift
dir="$root/$network_id"
tmpdir=/tmp
function randomfile {
dd if=/dev/urandom of=/dev/stdout bs=1024 count=$1 2>/dev/null
}
# swarm attach 00 brings up a console attached to a running instance
function attach {
id=$1
shift
echo "attaching console to instance $id"
cmd="$GETH --datadir=$dir/data/$id $* attach ipc:$dir/data/$id/geth.ipc"
# echo $cmd
eval $cmd
}
# swarm attach 00 brings up a console attached to a running instance
function execute {
id=$1
shift
# attach $id --exec "'$*' "
cmd="$GETH --datadir=$dir/data/$id --exec '$*' attach ipc:$dir/data/$id/geth.ipc"
# echo $cmd
eval $cmd
}
# swarm hive 00 displays the kademlia table of the given running instance
function hive {
if [ "$1" = "all" ]; then
N=`ls -1 -d $dir/data/* |wc -l`
for ((i=0;i<N;++i)); do
instance=`printf "%02d" $i`
hive $instance
done
else
# echo "kademlia table of instance $id"
execute $1 'console.log(bzz.hive)'|grep -v undefined
fi
}
# swarm log 00 shows the running tail logs of an instance
function log {
id=$1
shift
echo "streaming logs for instance $id"
cmd="tail -f $dir/log/$id.log"
echo $cmd
eval $cmd
}
# swarm cleanlog 00 removes the old log files for a given instance (all for every instance)
function cleanlog {
id=$1
shift
if [ $id = "all" ]; then
echo "remove logs for all instances"
rm -rf "$dir/log/"
else
echo "remove logs for instance $id"
rm -rf $dir/log/$id*
fi
}
# display kademlia tables of istances
function monitor {
if [ "$3" = "" ]; then
id=$1
period=$2
while true; do
hive $id
sleep $period
done
else
node=$1
id=$2
period=$3
while true; do
remote-run $node swarm hive $id
sleep $period
done
fi
}
# swarm cleanbzz 00 removes the bzz subdirectory for a given instance (all for every instance)
function cleanbzz {
id=$1
shift
if [ $id = "all" ]; then
echo "remove bzz data for all instances"
rm -rf $dir/data/*/bzz
else
echo "remove bzz data for instance $id"
rm -rf "$dir/data/$id"
fi
}
# swarm less/viewlogd 00 displays the last (current) log for the given instance (in a pager)
function viewlog {
id=$1
shift
echo "viewing logs for instance $id"
cmd="/usr/bin/less $dir/log/$id.log"
echo $cmd
eval $cmd
}
# display the swawrm base account for an instance
function key {
id=$1
shift
mkdir -p $dir/data/$id/
$GETH --datadir=$dir/data/$id account list|head -n1|perl -ne '/([a-f0-9]{40})/ && print $1'
}
# swarm options 00 displays the geth command line options used to start the swarm
function rawoptions {
id=$1
shift
globaloptions="
--dev
--maxpeers=40
--shh=false
--nodiscover
--networkid=$network_id
--bzznoswap
--verbosity=0
--vmodule=swarm/*=5"
datadir=$dir/data/$id
password=$id
port=303$id
bzzport=322$id
rpcport=302$id
key=`swarm key $id`
instanceoptions="
--datadir=$datadir
--identity=$id
--bzzaccount=$key
--unlock=$key
--bzzport=$bzzport
--port=$port
--rpc
--rpcport=$rpcport
--rpccorsdomain='*'"
echo "$globaloptions"
echo "$instanceoptions"
echo "$*"
}
function options {
echo "The command line options passed to geth are the following:"
echo "use 'geth help' to see further options"
rawoptions $*
}
function start {
id=$1
shift
if [ -f $dir/pids/$id.pid ]; then
echo "instance $id already running"
return
fi
datetag=`date "+%Y-%m-%d-%H:%M:%S"`
log=$dir/log/$id.$datetag.log
linklog=$dir/log/$id.log
opts=`swarm rawoptions $id $*|tr '\r' ' '`
# echo; echo "$GETH $opts > $log 2>&1 &"
$GETH $opts --password=<(echo -n $id) > "$log" 2>&1 & # comment out if you pipe it to a tty etc.
ln -sf "$log" "$linklog"
# wait until ready
((j=0))
while true; do
execute $id "net" > /dev/null 2>&1 && break
sleep 1
echo -n "."
if ((j++>10)); then
echo "instance $id failed to start"
exit 1
fi
done
echo -n "started - "
pid=`ps auxwww|grep geth|grep "ty=$id"|grep -v grep|awk '{print $2}'`
echo "pid: $pid"
echo $pid > $dir/pids/$id.pid
}
# setup 00 creates the direcories for instance
function setup {
id=$1
shift
mkdir -p $dir/data/$id
mkdir -p $dir/enodes
mkdir -p $dir/pids
mkdir -p $dir/log
}
# creates the swarm base account for an instance
function create-account {
id=$1
datadir=$dir/data/$id
# if we do not have an account, create one
# will not prompt for password, we use the double digit instance id as passwd
# NEVER EVER USE THESE ACCOUNTS FOR INTERACTING WITH A LIVE CHAIN
keystoredir="$datadir/keystore/"
# echo "KeyStore dir: $keystoredir"
if [ ! -d "$keystoredir" ]; then
# echo "create an account with password $id [DO NOT EVER USE THIS ON LIVE]"
# mkdir -p $datadir/keystore
$GETH --datadir=$datadir --password=<(echo -n $id) account new >/dev/null 2>&1
# create account with password 00, 01, ...
# note that the account key will be stored also separately outside
# datadir
# this way you can safely clear the data directory and still keep your key
# under <rootdir>/keystore/dd
# LS=$(ls $datadir/keystore)
# echo $LS
while [ ! -d "$keystoredir" ]; do
echo "."
((i++))
if ((i>10)); then break; fi
sleep 1
done
# echo "copying keys $datadir/keystore $root/keystore/$id"
mkdir -p $dir/keystore/$id
cp -R "$keystoredir" $dir/keystore/$id
fi
}
# shuts down a running instance, cleans the pid
function stop {
id=$1
shift
if [ $id = "all" ]; then
procs=`cat $dir/pids/*.pid 2>/dev/null |perl -pe 's/^\s+//;s/\s+\\$//;s/\s+/\n/g'`
# echo "stopping processes $procs"
for p in $procs; do
shutdown $p
done
rm -rf $dir/pids/*
else
pid=$dir/pids/$id.pid
if [ -f $pid ]; then
echo "stopping instance $id, pid="`cat $pid`
shutdown `cat $pid`
rm $pid
fi
fi
}
# shutdown kills the node with interrupt 2 - if it resits falls back to -9 after 10s
function shutdown {
echo -n "stopping $1..."
kill -2 $1
while true; do
ps auxwww|grep geth|grep -v grep|awk '{print $2}'|grep -ql $1 || break
if ((i++>5)); then
echo "not stopping. killing it"
kill -QUIT $1
break
fi
echo '.'
sleep 1
done
echo "stopped"
}
# swarm restart 00 calls stop and start
function restart {
id=$1
shift
if [ $id = "all" ]; then
stop all
N=`ls -d1 $dir/data/*|wc -l`
cluster $N $*
else
stop $id
start $id $*
fi
}
# swarm init X sets up and starts a new client instance
##########################################################
#
# IT WIPES THE DATABASE
#
##########################################################
function init {
killall geth
reset all
cluster $*
enode all
connect all
}
# reset wipes the datadirs of the instance
function reset {
id=$1
shift
if [ $id = "all" ]; then
rm -rf $dir
else
rm -rf$dir/*/$id*
fi
}
# enode displays the instance's enode address
# swarm enode all writes all instances' enodes in a file
function enode {
id=$1
shift
if [ $id = "all" ]; then
json=$dir/static-nodes.json
enodes=$dir/enodes.lst
cmd=$dir/connect.js
rm -f $enodes $json $cmd
# build a static nodes(-like) list of all enodes of the local cluster
echo "[" >> $json
N=`ls -1 -d $dir/data/* |wc -l`
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
f=$dir/enodes/$id.enode
enode $id > $f
echo -n "admin.addPeer(" >> $cmd
cat "$f" | perl -pe 's/\s*$//' >> $cmd
echo ");" >> $cmd
cat $f |perl -pe 's/"//g'>> $enodes
cat $f >> $json
echo "," >> $json
done
echo "\"\"]" >> $json
cat $enodes
else
# echo "local IP: $ip_addr "
execute $id 'admin.nodeInfo.enode' |perl -pe "s/\[\:\:\]/$IP_ADDR/ "
fi
}
# connect sources the local or remote set of peers and connects the node to the peers
function connect {
id=$1
shift
if [ $id = "all" ]; then
N=`ls -1 -d $dir/data/* |wc -l`
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
connect $id
done
else
echo -n 'peer count: '
attach $id --preload $dir/connect.js --exec net.peerCount
fi
}
# swarm cluster N launches N nodes; 00 01 02 ...
function cluster {
N=$1
shift
echo "launching cluster of $N instances"
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
setup $id $*
create-account $id
echo "launching node $i/$N.."
start $id $*
# info $id
enode $id
done
}
# swarm needs instance keyfile destination
# tests if the content for the key is available for the intance
#
function needs {
id=$1
keyfile=$2
target=$3
dest=$tmpdir/down
mkdir -p $dest
file=$dest/`basename $target`
rm -f $file
echo -n "waiting for root hash in '$keyfile'..."
while true; do
if [ ! -z $keyfile ]; then
break
fi
sleep 1
echo -n "."
done
key=`cat $keyfile|tr -d \"`
echo " => $key"
download $id $key $dest && cmp --silent $file $target && echo "PASS" || echo "FAIL"
}
# swarm up 00 file uploads file via instances CLI
function up { #port, file
echo "Upload file '$2' to node $1... " 1>&2
file=`basename $2`
/usr/bin/time -f "latency: %e" swarm execute $1 "bzz.upload(\"$2\", \"$file\")"|tail -n1> /tmp/key
cat /tmp/key
}
# swarm download 00 file download file via instances CLI
function download {
echo "download '$2' from node $1 to '$3'"
execute $1 "bzz.download(\"$2\", \"$3\")" >/dev/null
}
# swarm down issues bzz.get to download the content 10 attempts
function down {
echo -n "Download hash '$2' from node $1... "
while true; do
execute $1 "bzz.get(\"$2\")" 2> /dev/null |grep -qil "status" && break
sleep 1
echo -n "."
if ((i++>10)); then
echo "not found"
return
fi
done
echo "found OK"
}
# static info about an instance (available even if node is off)
function info {
echo "swarm node information"
echo "ROOTDIR: $root"
echo "DATADIR: $dir/data/$1"
echo "LOGFILE: $dir/log/$1.log"
echo "HTTPAPI: http://localhost:322$1"
echo "ETHPORT: 303$1"
echo "RPCPORT: 302$1"
echo "ACCOUNT:" 0x`ls -1 $dir/data/$1/bzz`
echo "CHEQUEB:" `cat $dir/data/$1/bzz/*/config.json|grep Contract|awk -F\" '{print $4}'`
echo "ROOTDIR: $root"
echo "DATADIR: $dir/data/$1"
echo "LOGFILE: $dir/log/$1.log"
}
# live into about an instance
function status {
echo -n "account balance: "
execute $1 'eth.getBalance(eth.accounts[0])'
echo -n "swap contract balance: "
execute $1 "eth.getBalance(bzz.info.Swap.Contract)"
echo -n "chequebook balance: "
execute $1 "chequebook.balance"
echo -n "peer count: "
execute $1 'net.peerCount'
echo -n "latest block number: "
execute $1 "eth.blockNumber"
}
# display peers for an instance
function peers {
execute $1 'admin.peers'
}
# add peers into an instance (connection not guaranteed)
function addpeers {
id=$1
peers=$2
if [ $id = "all" ]; then
N=`ls -1 -d $dir/data/* |wc -l`
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
addpeers $id $peers
done
else
echo "addpeer to instance $id"
for peer in `cat $peers|grep -v '^#'`; do
execute $id "admin.addPeer(\"$peer\")"
done
fi
}
# configures the eth-net-intelligence-api network monitor
function netstatconf {
group=$1
N=`ls -1 -d $dir/data/* |wc -l`
ip=`curl ipecho.net/plain 2>/dev/null;echo `
ws_server="ws://146.185.130.117:3000"
ws_secret=BZZ322
conf="$dir/$group-$ip.netstat.json"
echo "writing netstat conf for cluster $group-$ip ($N instances) -> $conf"
echo -e "[" > $conf
for ((i=0;i<$N;++i)); do
id=`printf "%02d" $i`
single_template=" {\n \"name\" : \"$group-$ip-$i\",\n \"cwd\" : \".\",\n \"script\" : \"app.js\",\n \"log_date_format\" : \"YYYY-MM-DD HH:mm Z\",\n \"merge_logs\" : false,\n \"watch\" : false,\n \"exec_interpreter\" : \"node\",\n \"exec_mode\" : \"fork_mode\",\n \"env\":\n {\n \"NODE_ENV\" : \"production\",\n \"RPC_HOST\" : \"localhost\",\n \"RPC_PORT\" : \"302$id\",\n \"INSTANCE_NAME\" : \"$group-$ip-$i\",\n \"WS_SERVER\" : \"$ws_server\",\n \"WS_SECRET\" : \"$ws_secret\",\n }\n }"
endline=""
if ((i<$N-1)); then
# if [ "$i" -ne "$N" ]; then
endline=","
fi
echo -e "$single_template$endline" >> $conf
done
echo "]" >> $conf
}
# (re)starts the eth-net-intelligence-api network monitor
function netstatrun {
cd $GETH_DIR/../eth-net-intelligence-api
pm2 kill
pm2 start $dir/*.netstat.json
}
# kills the eth-net-intelligence-api network monitor
function netstatkill {
cd $GETH_DIR/../eth-net-intelligence-api
pm2 kill
}
# copies the swarm control script to the remote node(s)
function remote-update-scripts {
scriptdir=$GETH_DIR/swarm/cmd/swarm/
remotes=$1
for remote in `cat $remotes|grep -v '^#'`; do echo "updating scripts on $remote..."; ssh $remote mkdir -p bin && scp -r $scriptdir/* $remote:bin/; done
}
# copies the geth executable to the remote nodes
function remote-update-bin {
remotes=$1
# remote-update-scripts $remotes
for remote in `cat $remotes|grep -v '^#'`; do echo "updating binary on $remote..."; scp -r $GETH_DIR/geth $remote:bin/; done
}
# runs a command on remote node or nodes from a file
function remote-run {
remotes=$1
shift
if `echo "$remotes" | grep -qil @`; then
ip=`echo "$remotes"|cut -d@ -f2`
ssh $remotes "export IP_ADDR=$ip;" '. $HOME/bin/env.sh;' "$*"
else
for remote in `cat $remotes|grep -v '^#'`; do echo "running on $remote..."; remote-run $remote "$*"; done
fi
}
# updates the code from the given branch
function update {
branch=$1
echo "cd $GETH_DIR && git remote update && git reset --hard $branch"
(cd $GETH_DIR && git remote update && git reset --hard $branch)
}
function checksum {
tar -cf - $1 | md5sum|awk '{print $1}'
}
function checkaccess {
nodes=$1
target=`basename $2`
chsum=`md5sum $2|cut -f1 -d' '`
master=`head -1 $nodes`
echo "uploading target on $master (md5sum $chsum, size: `du -b -d0 $2|cut -f1`)"
scp $2 $master:$target
hash=`swarm remote-run $master "swarm up 00 $target $file"|tr -d '"'`
remote-run $nodes swarm checkdownload all $hash $chsum
}
function checkdownload {
id=$1
if [ "$id" = "all" ]; then
shift
N=`ls -1 -d $dir/data/* |wc -l`
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
checkdownload $id $*
done
else
hash=$2
target=$3
datetag=`date "+%Y-%m-%d-%H:%M:%S"`
file="swarm-$datetag"
rm -rf $tmpdir/$file
/usr/bin/time -o $tmpdir/$file.log -f "%e" swarm download $id $hash $tmpdir/$file > /dev/null
echo
if [ -f $target ]; then
cmp --silent $target $tmpdir/$file/* && echo PASS || echo FAIL
elif [ -r $target ]; then
diff -r $target $tmpdir/$file/ >/dev/null && echo PASS || echo FAIL
else
exp=`md5sum $tmpdir/$file/*|cut -f1 -d' '`
if [ "$exp" = "$target" ]; then
echo -n PASS
else
echo FAIL "$exp = $target"
fi
fi
echo " latency: " `cat $tmpdir/$file.log`
fi
}
function meminfo {
pid="$dir/pids/$1.pid"
if [ -f "$pid" ]; then
# cd /proc/`cat "$pid"` && cat status
ps aux|awk -v PID=`cat $pid` '$2 == PID {print $5 "Kb (" $4 "%)" }'
fi
}
function cpuinfo {
pid="$dir/pids/$1.pid"
if [ -f "$pid" ]; then
# cd /proc/`cat "$pid"` && cat status
ps aux|awk -v PID=`cat $pid` '$2 == PID {print $3 "%" }'
fi
}
function diskusage {
if [ "$1" == "" ]; then
echo "DISK USAGE:" `df -m |grep '/$'|awk '{print $(NF-2) "Mb (" $(NF-1) ")"} '`
else
du -m -d0 $*|cut -f1
fi
}
function diskinfo {
echo "DISK USAGE $1:"
echo "blockchain:" `diskusage $dir/data/$1/chaindata`
echo "chunkstore:" `diskusage $dir/data/$1/bzz/*/chunks`
echo "overall: /" `diskusage`
}
case $cmd in
"info" )
info $*;;
"enode" )
enode $*;;
"status" )
status $*;;
"peers" )
peers $*;;
"addpeers" )
addpeers $*;;
"clean" )
clean $*;;
"needs" )
needs $*;;
"up" )
up $*;;
"key" )
key $*;;
"down" )
down $*;;
"download" )
download $*;;
"init" )
init $*;;
"exec" )
execute $*;;
"hive" )
hive $*;;
"start" )
start $*;;
"stop" )
stop $* ;;
"restart" )
restart $*;;
"reset" )
reset $*;;
"cluster" )
cluster $*;;
"attach" )
attach $*;;
"execute" )
execute $*;;
"exec" )
execute $*;;
"cleanbzz" )
cleanbzz $*;;
"cleanlog" )
cleanlog $*;;
"log" )
log $*;;
"viewlog" )
viewlog $*;;
"less" )
viewlog $*;;
"connect" )
connect $*;;
"monitor" )
monitor $*;;
"remote-update-scripts" )
remote-update-scripts $*;;
"remote-update-bin" )
remote-update-bin $*;;
"update-src" )
update-src $*;;
"remote-run" )
remote-run $*;;
"netstatconf" )
netstatconf $*;;
"netstatkill" )
netstatkill $*;;
"netstatrun" )
netstatrun $*;;
"options" )
options $*;;
"rawoptions" )
rawoptions $*;;
"randomfile" )
randomfile $*;;
"diskinfo" )
diskinfo $*;;
"meminfo" )
meminfo $*;;
"cpuinfo" )
cpuinfo $*;;
"setup" )
setup $* ;;
"create-account" )
create-account $*;;
"checkaccess" )
checkaccess $* ;;
"checkdownload" )
checkdownload $* ;;
esac

316
swarm/cmd/swarm/swarm.sh
View file

@ -1,316 +0,0 @@
# !/bin/bash
# bash cluster <root> <network_id> <number_of_nodes> <runid> <local_IP> [[params]...]
# https://github.com/ethereum/go-ethereum/wiki/Setting-up-monitoring-on-local-cluster
# sets up a local ethereum network cluster of nodes
# - <number_of_nodes> is the number of nodes in cluster
# - <root> is the root directory for the cluster, the nodes are set up
# with datadir `<root>/<network_id>/00`, `<root>/ <network_id>/01`, ...
# - new accounts are created for each node
# - they launch on port 30300, 30301, ...
# - they star rpc on port 8100, 8101, ...
# - by collecting the nodes nodeUrl, they get connected to each other
# - if enode has no IP, `<local_IP>` is substituted
# - if `<network_id>` is not 0, they will not connect to a default client,
# resulting in a private isolated network
# - the nodes log into `<root>/<network_id>/00.<runid>.log`, `<root>/<network_id>/01.<runid>.log`, ...
# - The nodes launch in mining mode
# - the cluster can be killed with `killall geth` (FIXME: should record PIDs)
# and restarted from the same state
# - if you want to interact with the nodes, use rpc
# - you can supply additional params on the command line which will be passed
# to each node, for instance `-mine`
if [ "$GETH" = "" ]; then
echo "env var GETH not set "
exit 1
fi
srcdir=`dirname $0`
root=$1
shift
network_id=$1
shift
cmd=$1
shift
# ip_addr=`curl ipecho.net/plain 2>/dev/null;echo `
# echo "external IP: $ip_addr"
swarmoptions='--dev --maxpeers=20 --shh=false --nodiscover'
tmpdir=/tmp
function attach {
id=$1
shift
echo "attaching console to instance $id"
cmd="$GETH $* attach ipc:$root/$network_id/data/$id/geth.ipc"
# echo $cmd
eval $cmd
}
function log {
id=$1
shift
echo "streaming logs for instance $id"
cmd="tail -f $root/$network_id/log/$id.log"
echo $cmd
eval $cmd
}
function less {
id=$1
shift
echo "viewing logs for instance $id"
cmd="/usr/bin/less $root/$network_id/log/$id.log"
echo $cmd
eval $cmd
}
function start {
id=$1
shift
# echo -n "starting instance $id - "
cmd="bash $srcdir/gethup.sh $root/$network_id/ $id '$ip_addr' --networkid=$network_id $swarmoptions $*"
# echo "pid="`cat $root/$network_id/pids/$id.pid`
# echo $cmd
eval $cmd
}
function stop {
id=$1
shift
if [ $id = "all" ]; then
procs=`cat $root/$network_id/pids/*.pid 2>/dev/null |perl -pe 's/^\s+//;s/\s+\\$//;s/\s+/\n/g'`
# echo "stopping processes $procs"
for p in $procs; do
shutdown $p
done
rm -rf $root/$network_id/pids/*
else
pid=$root/$network_id/pids/$id.pid
if [ -f $pid ]; then
echo "stopping instance $id, pid="`cat $pid`
shutdown `cat $pid`
rm $pid
fi
fi
# ps auxwww|grep geth|grep bzz|grep -v grep
}
function shutdown {
echo -n "stopping $1..."
kill -2 $1
while true ;do
ps auxwww|grep geth|grep -v grep|awk '{print $2}'|grep -ql $1 || break
sleep 1
# ps auxwww|grep geth|grep -v grep |grep $1 #|awk '{print $2}'
done
echo "stopped"
}
function restart {
id=$1
shift
stop $id
start $id $*
}
function init {
stop all
killall geth
reset all
cluster $*
stop all
cluster $*
}
function reset {
id=$1
shift
if [ $id = "all" ]; then
rm -rf $root/$network_id
else
rm -rf$root/$network_id/*/$id*
fi
}
function cluster {
N=$1
shift
echo "launching cluster of $N instances"
# cmd="bash $srcdir/gethcluster.sh $root $network_id $N '' $swarmoptions $*"
# echo $cmd
# eval $cmd
dir=$root/$network_id
mkdir -p $dir/data
mkdir -p $dir/enodes
mkdir -p $dir/pids
mkdir -p $dir/log
enodes=$dir/enodes.all
rm -f $enodes
# build a static nodes(-like) list of all enodes of the local cluster
echo "[" >> $enodes
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
enode=$dir/enodes/$id.enode
if [ -f "$enode" ]; then
cat "$enode" >> $enodes
echo "," >> $enodes
fi
done
echo "\"\"]" >> $enodes
for ((i=0;i<N;++i)); do
id=`printf "%02d" $i`
mkdir -p $dir/data/$id
echo "launching node $i/$N ---> tail -f $dir/log/$id.log"
start $id $vmodule $*
done
}
function needs {
id=$1
keyfile=$2
target=$3
dir=`dirname $3`
dest=$tmpdir/down
mkdir -p $dest
file=$dest/`basename $target`
rm -f $file
echo -n "waiting for root hash in '$keyfile'..."
while true; do
if [ -f $keyfile ] && [ ! -z $keyfile ]; then
break
fi
sleep 1
echo -n "."
done
key=`cat $keyfile|tr -d \"`
echo " => $key"
download $id $key $dest && cmp --silent $file $target && echo "PASS" || echo "FAIL"
# && ls -l $keyfile $file $target
}
function up { #port, file
echo "Upload file '$2' to node $1... " 1>&2
file=`basename $2`
attach $1 "--exec 'bzz.upload(\"$2\", \"$file\")'"|tail -n1> /tmp/key
# key=`bash swarm/cmd/bzzup.sh $2 86$1`
cat /tmp/key
}
function download {
echo "download '$2' from node $1 to '$3'"
# echo attach $1 "--exec 'bzz.download(\"$2\", \"$3\")'"
attach $1 "--exec 'bzz.download(\"$2\", \"$3\")'" > /dev/null
}
function down {
echo -n "Download hash '$2' from node $1... "
# echo "wget -O- http://localhost:86$1/$2 > /dev/null 2>&1 && echo 'got it' || echo 'not found'"
# wget -O- http://localhost:86$1/$2 > /dev/null 2>&1 && echo "got it" || echo "not found"
while true; do
attach $1 "--exec 'bzz.get(\"$2\")'" 2> /dev/null |grep -qil "status" && break
sleep 1
echo -n "."
if ((i++>10)); then
echo "not found"
return
fi
done
echo "found OK"
}
function clean { #index
echo "Clean up for $1"
rm -rf $root/$network_id/data/$1/{bzz/*/chunks,bzz/*/requests/,bzz/*/bzz-peers.json,chaindata,nodes}
}
function info {
echo "swarm node information"
echo "ROOTDIR: $root"
echo "DATADIR: $root/$network_id/data/$1"
echo "LOGFILE: $root/$network_id/log/$1.log"
echo "HTTPAPI: http://localhost:322$1"
echo "ETHPORT: 303$1"
echo "RPCPORT: 302$1"
echo "ACCOUNT:" 0x`ls -1 $root/$network_id/data/$1/bzz`
echo "CHEQUEB:" `cat $root/$network_id/data/$1/bzz/*/config.json|grep Contract|awk -F\" '{print $4}'`
echo "ROOTDIR: $root"
echo "DATADIR: $root/$network_id/data/$1"
echo "LOGFILE: $root/$network_id/log/$1.log"
}
function status {
attach 00 -exec "'console.log(eth.getBalance(eth.accounts[0])); console.log(eth.getBalance(bzz.info().Swap.Contract)); console.log(chequebook.balance)'"
}
function netstatconf {
begin=$1
N=$2
name_prefix=$3
ws_server=$4
ws_secret=$5
conf="$root/$network_id/$name_prefix.netstat.json"
echo "writing netstat conf for cluster $name_prefix to $conf"
echo -e "[" > $conf
for ((i=$begin;i<$start+$N;++i)); do
id=`printf "%02d" $i`
single_template=" {\n \"name\" : \"$name_prefix-$i\",\n \"cwd\" : \".\",\n \"script\" : \"app.js\",\n \"log_date_format\" : \"YYYY-MM-DD HH:mm Z\",\n \"merge_logs\" : false,\n \"watch\" : false,\n \"exec_interpreter\" : \"node\",\n \"exec_mode\" : \"fork_mode\",\n \"env\":\n {\n \"NODE_ENV\" : \"production\",\n \"RPC_HOST\" : \"localhost\",\n \"RPC_PORT\" : \"302$id\",\n \"INSTANCE_NAME\" : \"$name_prefix-$i\",\n \"WS_SERVER\" : \"$ws_server\",\n \"WS_SECRET\" : \"$ws_secret\",\n }\n }"
endline=""
if (($i<$N-1)); then
# if [ "$i" -ne "$N" ]; then
endline=","
fi
echo -e "$single_template$endline" >> $conf
done
echo "]" >> $conf
}
case $cmd in
"info" )
info $*;;
"status" )
status $*;;
"clean" )
clean $*;;
"needs" )
needs $*;;
"up" )
up $*;;
"down" )
down $*;;
"init" )
init $*;;
"start" )
start $*;;
"stop" )
stop $* ;;
"restart" )
restart $*;;
"reset" )
reset $*;;
"cluster" )
cluster $*;;
"attach" )
attach $*;;
"log" )
log $*;;
"less" )
less $*;;
"netstatconf" )
netstatconf $*;;
esac

28
swarm/cmd/swarm/test.sh
View file

@ -1,28 +0,0 @@
#!/bin/bash
TEST_DIR=`dirname $0`
TEST_NAME=`basename $0 .sh`
TEST_TYPE=`basename $TEST_DIR`
export SWARM_BIN=$TEST_DIR/../../cmd/swarm
export GETH=$SWARM_BIN/../../../geth
export NETWORKID=322$TEST_NAME
export TMPDIR=~/BZZ/test/$TEST_TYPE
export DATA_ROOT=$TMPDIR/$NETWORKID
# alias swarm='bash $SWARM_BIN/swarm.sh $DATA_ROOT $NETWORKID'
EXTRA_ARGS=$*
rm -rf $DATA_ROOT
wait=1
function swarm {
# echo bash $SWARM_BIN/swarm.sh $TMPDIR $NETWORKID $* $EXTRA_ARGS
bash $SWARM_BIN/swarm.sh $TMPDIR $NETWORKID $* $EXTRA_ARGS
}
function randomfile {
dd if=/dev/urandom of=/dev/stdout bs=1024 count=$1 2>/dev/null
}

3
swarm/examples/album/file-manager.js
View file

@ -103,13 +103,14 @@ function onUploadingComplete(uri) {
xhr.onreadystatechange = function () { xhr.onreadystatechange = function () {
if (xhr.readyState === 4) { if (xhr.readyState === 4) {
var i = xhr.responseText; var i = xhr.responseText;
window.location.replace("/bzz:/" + i + "/"); window.location = "/bzz:/" + i + "/" + window.location.hash;
} }
}; };
sendImages(xhr, uri); sendImages(xhr, uri);
} }
function handleFiles(files) { function handleFiles(files) {
showModal('Uploading photos..');
uploadFile(files, 0, ""); uploadFile(files, 0, "");
} }

8
swarm/examples/album/index.html
View file

@ -5,6 +5,7 @@
<meta name="viewport" content="width=device-width,initial-scale=1"/> <meta name="viewport" content="width=device-width,initial-scale=1"/>
<link rel="shortcut icon" href="images/favicon.ico"/> <link rel="shortcut icon" href="images/favicon.ico"/>
<script src="jquery.js"></script> <script src="jquery.js"></script>
<script src="jquery.modal.js" type="text/javascript" charset="utf-8"></script>
<script> <script>
$.noConflict(); $.noConflict();
</script> </script>
@ -16,8 +17,15 @@
<script src="index.js" type="text/javascript"></script> <script src="index.js" type="text/javascript"></script>
<link href="index.css" rel="stylesheet" type="text/css"/> <link href="index.css" rel="stylesheet" type="text/css"/>
<link rel="stylesheet" href="jquery.modal.css" type="text/css" media="screen"/>
</head> </head>
<body> <body>
<div id="preview-modal" style="display:none; text-align: center">
<p id="action-text"></p>
<img style="text-align: center; padding-top: 20px" src="throbber.gif" id="currentPreview">
</div>
<noscript> <noscript>
<h2>Frak! JavaScript required :'(</h2> <h2>Frak! JavaScript required :'(</h2>
</noscript> </noscript>

96
swarm/examples/album/index.js
View file

@ -2,8 +2,7 @@
// Copyright(c) 2003-2014 by wave++ "Yuri D'Elia" <wavexx@thregr.org> // Copyright(c) 2003-2014 by wave++ "Yuri D'Elia" <wavexx@thregr.org>
// Distributed under GPL2 (see COPYING) WITHOUT ANY WARRANTY. // Distributed under GPL2 (see COPYING) WITHOUT ANY WARRANTY.
var datafile = 'data.json'; var datafile = 'data.json';
var padding = 100; var padding = 20;
var marginTop = 50;
var duration = 500; var duration = 500;
var thrdelay = 1500; var thrdelay = 1500;
var hidedelay = 3000; var hidedelay = 3000;
@ -42,7 +41,6 @@ var eback; // background
var enoise; // additive noise var enoise; // additive noise
var eflash; // flashing object var eflash; // flashing object
var ehdr; // header var ehdr; // header
var progress; // progress
var elist; // thumbnail list var elist; // thumbnail list
var fscr; // thumbnail list scroll fx var fscr; // thumbnail list scroll fx
var econt; // picture container var econt; // picture container
@ -61,6 +59,18 @@ var idle; // idle timer
var clayout; // current layout var clayout; // current layout
var csr; // current scaling ratio var csr; // current scaling ratio
function showModal(text, previewImageUrl) {
if (!previewImageUrl) {
previewImageUrl = 'throbber.gif';
}
jQuery('#currentPreview').attr('src', previewImageUrl);
jQuery('#action-text').text(text);
jQuery('#preview-modal').modal({
fadeDuration: 250
});
}
function resize() function resize()
{ {
// best layout // best layout
@ -233,7 +243,7 @@ function resizeMainImg(img)
img.setStyles( img.setStyles(
{ {
'position': 'absolute', 'position': 'absolute',
'top': (contSize.y / 2 - img.height / 2) + marginTop, 'top': contSize.y / 2 - img.height / 2,
'left': contSize.x / 2 - img.width / 2 'left': contSize.x / 2 - img.width / 2
}); });
} }
@ -275,41 +285,53 @@ function imageToUrl(img, w, h) {
return can.toDataURL(); return can.toDataURL();
} }
function deleteImg() function deleteImg() {
{ if (imgs.data.length < 2) return; // empty albums not allowed
if(imgs.data.length < 2) return; // empty albums not allowed var fname = imgs.data[eidx].img[0];
var fname = imgs.data[eidx].img[0]; imgs.data.splice(eidx, 1);
imgs.data.splice(eidx,1);
// construct an HTTP request showModal('Deleting photo..', fname);
var xhr = new XMLHttpRequest(); // construct an HTTP request
var xhr = new XMLHttpRequest();
// set response handler // set response handler
xhr.onreadystatechange = function () { if (xhr.readyState === 4) { xhr.onreadystatechange = function () {
var i = xhr.responseText; if (xhr.readyState === 4) {
var xhrd = new XMLHttpRequest(); var i = xhr.responseText;
xhrd.onreadystatechange = function () { if (xhrd.readyState === 4) { var xhrd = new XMLHttpRequest();
var j = xhrd.responseText; xhrd.onreadystatechange = function () {
window.location.replace("/bzz:/" + j + "/"); if (xhrd.readyState === 4) {
}}; var j = xhrd.responseText;
xhrd.open("DELETE", "/bzz%3A/" + i + "/" + fname, true); window.location = "/bzz:/" + j + "/" + window.location.hash;
xhrd.send(); }
}}; };
xhrd.open("DELETE", "/bzz%3A/" + i + "/" + fname, true);
xhrd.send();
}
};
sendImages(xhr, ""); sendImages(xhr, "");
} }
function moveUpDown(off) function moveUpDown(off) {
{ var me = imgs.data[eidx];
var me = imgs.data[eidx]; console.log(me.thumb[0]);
imgs.data[eidx] = imgs.data[eidx + off]; var moveText = 'Moving up..';
imgs.data[eidx + off] = me; if (off > 0) {
var xhr = new XMLHttpRequest(); moveText = 'Moving down..';
xhr.onreadystatechange = function () { if (xhr.readyState === 4) { }
var i = xhr.responseText;
window.location.replace("/bzz:/" + i + "/#" + (eidx + off)); showModal(moveText, me.thumb[0]);
}}; imgs.data[eidx] = imgs.data[eidx + off];
sendImages(xhr, ""); imgs.data[eidx + off] = me;
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function () {
if (xhr.readyState === 4) {
var i = xhr.responseText;
window.location.replace("/bzz:/" + i + "/#" + (eidx + off));
}
};
sendImages(xhr, "");
} }
function moveUp() function moveUp()
@ -364,9 +386,6 @@ function onMainReady()
ehdr.set('html', dsc.join(' ')); ehdr.set('html', dsc.join(' '));
ehdr.setStyle('display', (dsc.length? 'block': 'none')); ehdr.setStyle('display', (dsc.length? 'block': 'none'));
progress.set('html', '<img id="currentPreview">');
progress.set('style', 'text-align: center; padding-top: 20px');
// complete thumbnails // complete thumbnails
var d = duration; var d = duration;
if(first !== false) if(first !== false)
@ -601,9 +620,6 @@ function initGallery(data)
ehdr = new Element('div', { id: 'header' }); ehdr = new Element('div', { id: 'header' });
ehdr.inject(econt); ehdr.inject(econt);
progress = new Element('div', { id: 'progress' });
progress.inject(econt);
elist = new Element('div', { id: 'list' }); elist = new Element('div', { id: 'list' });
elist.inject(emain); elist.inject(emain);

70
swarm/examples/album/jquery.modal.css Normal file
View file

@ -0,0 +1,70 @@
.blocker {
position: fixed;
top: 0; right: 0; bottom: 0; left: 0;
width: 100%; height: 100%;
overflow: auto;
z-index: 1;
padding: 20px;
box-sizing: border-box;
background-color: rgb(0,0,0);
background-color: rgba(0,0,0,0.75);
text-align: center;
}
.blocker:before{
content: "";
display: inline-block;
height: 100%;
vertical-align: middle;
margin-right: -0.05em;
}
.blocker.behind {
background-color: transparent;
}
.modal {
display: inline-block;
vertical-align: middle;
position: relative;
z-index: 2;
width: 400px;
background: #fff;
padding: 15px 30px;
-webkit-border-radius: 8px;
-moz-border-radius: 8px;
-o-border-radius: 8px;
-ms-border-radius: 8px;
border-radius: 8px;
-webkit-box-shadow: 0 0 10px #000;
-moz-box-shadow: 0 0 10px #000;
-o-box-shadow: 0 0 10px #000;
-ms-box-shadow: 0 0 10px #000;
box-shadow: 0 0 10px #000;
text-align: left;
}
.modal a.close-modal {
position: absolute;
top: -12.5px;
right: -12.5px;
display: block;
width: 30px;
height: 30px;
text-indent: -9999px;
background: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAB4AAAAeCAYAAAA7MK6iAAAACXBIWXMAAAsTAAALEwEAmpwYAAAABGdBTUEAANjr9RwUqgAAACBjSFJNAABtmAAAc44AAPJxAACDbAAAg7sAANTIAAAx7AAAGbyeiMU/AAAG7ElEQVR42mJkwA8YoZjBwcGB6fPnz4w/fvxg/PnzJ2N6ejoLFxcX47Rp036B5Dk4OP7z8vL+P3DgwD+o3v9QjBUABBALHguZoJhZXV2dVUNDgxNIcwEtZnn27Nl/ZmZmQRYWFmag5c90dHQY5OXl/z98+PDn1atXv79+/foPUN9fIP4HxRgOAAggRhyWMoOwqKgoq6GhIZe3t7eYrq6uHBDb8/Pz27Gysloga/jz588FYGicPn/+/OapU6deOnXq1GdgqPwCOuA31AF/0S0HCCB0xAQNBU4FBQWB0NBQublz59oADV37Hw28ePHi74MHD/6ii3/8+HEFMGQUgQ6WEhQU5AeZBTWTCdkigABC9ylIAZeMjIxQTEyMysaNG/3+/v37AGTgr1+//s2cOfOXm5vbN6Caz8jY1NT0a29v76/v37//g6q9sHfv3khjY2M5YAgJgsyEmg0PYYAAQreUk4+PT8jd3V1l1apVgUAzfoIM2rlz5x9gHH5BtxAdA9PB1zNnzvyB+R6oLxoopgC1nBPZcoAAgiFQnLIDMb+enp5iV1eXBzDeHoI0z58//xcwIX0mZCkMg9S2trb+hFk+ffr0QCkpKVmQ2VA7QHYxAgQQzLesQMwjIiIilZWVZfPu3bstMJ+SYikyBmUzkBnA9HEMyNcCYgmQHVC7mAACCJagOEBBbGdnp7lgwYJEkIavX7/+BcY1SvAaGRl9tba2xohjMTGxL8nJyT+AWQsuxsbG9vnp06e/QWYdPHiwHmiWKlBcCGQXyNcAAQSzmBuoSQqYim3u37+/EKR48uTJv5ANB+bVr7Dga2xs/AkTV1JS+gq0AJyoQIkPWU9aWtoPkPibN2/2A/l6QCwJ9TULQADB4hcY//xKXl5eHt++fbsAUmxhYYHiM1DiAsr9R7ZcVVUVbikIdHd3/0TWIyws/AWYVsByAgICdkAxRSAWAGI2gACClV7C4uLiOv7+/lEgRZ8+ffqLLd6ABck3ZMuB6uCWrlu37je29HDx4kVwQisvL88FFqkaQDERUHADBBAomBl5eHiYgQmLE1hSgQQZgIUD1lJm69atf4HR8R1YKoH5QIPAWWP9+vV/gOI/gHkeQw+wGAXTwAJJ5t+/f/BUDRBA4NIEKMDMyMjICtQIiniG379/4yza7t69+//Lly8oDrty5co/bJaCAEwcZCkwwTJDLWYCCCCwxcDgY3z16hXDnTt3voP4EhISWA0BFgZMwNqHExh3jMiG1tbWsgHjnA2bHmAeBtdWwOL1MycnJ7wAAQggBmi+kgIW/OaKiorJwOLuFShO0LMSMPF9AUYBSpz6+vqixHlOTs4P9MIEWHaDsxSwYMoE2mEGFJcG5SKAAGJCqjv/AbPUn8ePH98ACQQHB6NUmZqamkzABIgSp5s3bwbHORCA1QDLAWZkPc7OzszA8oHl5cuXVy5duvQBGIXwWgoggGA+FgO6xkBNTS28r69vDrT2+Y1cIMDyJchX6KkXVEmAshd6KB06dAic94EO3AzkBwGxPhCLg8ptgACCZyeQp9jZ2b2AmsuAefM8tnxJCk5ISPgOLTKfAdNEOVDMA2QHLDsBBBC8AAFlbmCLwlZISCg5JSVlJizeQAaQaimoWAUFK0g/sGGwHiiWCMS2yAUIQAAxI7c4gEmeFZi4OJ48ecLMzc39CRiEmgEBASxA/QzA8vYvAxEgNjaWZc2aNezAsprp2LFjp4FpZRdQ+AkQvwLij0AMSoC/AQIIXklAC3AVUBoBxmE8sPXQAiyvN8J8fuPGjR/h4eHf0eMdhkENhOPHj8OT+NGjR88BxZuBOA5kJtRseCUBEECMSI0AdmgBDooDaaDl8sASTSkyMlKzpqZGU1paGlS7MABLrX83b978A6zwwakTmE0YgIkSnHpBfGCV+gxYh98qKSk5CeTeAxVeQPwUiN8AMSjxgdLNX4AAYkRqCLBAXcMHtVwSaLkMMMHJAvOq9IQJE9R8fHxElJWV1bEF8aNHj+7t27fvLTDlXwXGLyhoH0OD+DnU0k/QYAa1QP8BBBAjWsuSFWo5LzRYxKFYAljqiAHzqxCwIBEwMTERBdZeoOYMA7Bl+RFYEbwB5oS3IA9D4/IFEL+E4nfQ6IDFLTgvAwQQI5ZmLRtSsINSuyA0uwlBUyQPMPWD20/AKo8ByP4DTJTfgRgUjB+gFoEc8R6amGDB+wu5mQsQQIxYmrdMUJ+zQTM6NzQEeKGO4UJqOzFADQMZ/A1qCSzBfQXi71ALfyM17sEAIIAY8fQiWKAYFgIwzIbWTv4HjbdfUAf8RPLhH1icojfoAQKIEU8bG9kRyF0aRiz6YP0k5C4LsmUY9TtAADEyEA+IVfufGEUAAQYABejinPr4dLEAAAAASUVORK5CYII=") no-repeat 0 0;
}
.modal-spinner {
display: none;
width: 64px;
height: 64px;
position: fixed;
top: 50%;
left: 50%;
margin-right: -32px;
margin-top: -32px;
background: url("data:image/gif;base64,R0lGODlhIAAgAPMAABEREf///0VFRYKCglRUVG5ubsvLy62trTQ0NCkpKU5OTuLi4vr6+gAAAAAAAAAAACH+GkNyZWF0ZWQgd2l0aCBhamF4bG9hZC5pbmZvACH5BAAKAAAAIf8LTkVUU0NBUEUyLjADAQAAACwAAAAAIAAgAAAE5xDISWlhperN52JLhSSdRgwVo1ICQZRUsiwHpTJT4iowNS8vyW2icCF6k8HMMBkCEDskxTBDAZwuAkkqIfxIQyhBQBFvAQSDITM5VDW6XNE4KagNh6Bgwe60smQUB3d4Rz1ZBApnFASDd0hihh12BkE9kjAJVlycXIg7CQIFA6SlnJ87paqbSKiKoqusnbMdmDC2tXQlkUhziYtyWTxIfy6BE8WJt5YJvpJivxNaGmLHT0VnOgSYf0dZXS7APdpB309RnHOG5gDqXGLDaC457D1zZ/V/nmOM82XiHRLYKhKP1oZmADdEAAAh+QQACgABACwAAAAAIAAgAAAE6hDISWlZpOrNp1lGNRSdRpDUolIGw5RUYhhHukqFu8DsrEyqnWThGvAmhVlteBvojpTDDBUEIFwMFBRAmBkSgOrBFZogCASwBDEY/CZSg7GSE0gSCjQBMVG023xWBhklAnoEdhQEfyNqMIcKjhRsjEdnezB+A4k8gTwJhFuiW4dokXiloUepBAp5qaKpp6+Ho7aWW54wl7obvEe0kRuoplCGepwSx2jJvqHEmGt6whJpGpfJCHmOoNHKaHx61WiSR92E4lbFoq+B6QDtuetcaBPnW6+O7wDHpIiK9SaVK5GgV543tzjgGcghAgAh+QQACgACACwAAAAAIAAgAAAE7hDISSkxpOrN5zFHNWRdhSiVoVLHspRUMoyUakyEe8PTPCATW9A14E0UvuAKMNAZKYUZCiBMuBakSQKG8G2FzUWox2AUtAQFcBKlVQoLgQReZhQlCIJesQXI5B0CBnUMOxMCenoCfTCEWBsJColTMANldx15BGs8B5wlCZ9Po6OJkwmRpnqkqnuSrayqfKmqpLajoiW5HJq7FL1Gr2mMMcKUMIiJgIemy7xZtJsTmsM4xHiKv5KMCXqfyUCJEonXPN2rAOIAmsfB3uPoAK++G+w48edZPK+M6hLJpQg484enXIdQFSS1u6UhksENEQAAIfkEAAoAAwAsAAAAACAAIAAABOcQyEmpGKLqzWcZRVUQnZYg1aBSh2GUVEIQ2aQOE+G+cD4ntpWkZQj1JIiZIogDFFyHI0UxQwFugMSOFIPJftfVAEoZLBbcLEFhlQiqGp1Vd140AUklUN3eCA51C1EWMzMCezCBBmkxVIVHBWd3HHl9JQOIJSdSnJ0TDKChCwUJjoWMPaGqDKannasMo6WnM562R5YluZRwur0wpgqZE7NKUm+FNRPIhjBJxKZteWuIBMN4zRMIVIhffcgojwCF117i4nlLnY5ztRLsnOk+aV+oJY7V7m76PdkS4trKcdg0Zc0tTcKkRAAAIfkEAAoABAAsAAAAACAAIAAABO4QyEkpKqjqzScpRaVkXZWQEximw1BSCUEIlDohrft6cpKCk5xid5MNJTaAIkekKGQkWyKHkvhKsR7ARmitkAYDYRIbUQRQjWBwJRzChi9CRlBcY1UN4g0/VNB0AlcvcAYHRyZPdEQFYV8ccwR5HWxEJ02YmRMLnJ1xCYp0Y5idpQuhopmmC2KgojKasUQDk5BNAwwMOh2RtRq5uQuPZKGIJQIGwAwGf6I0JXMpC8C7kXWDBINFMxS4DKMAWVWAGYsAdNqW5uaRxkSKJOZKaU3tPOBZ4DuK2LATgJhkPJMgTwKCdFjyPHEnKxFCDhEAACH5BAAKAAUALAAAAAAgACAAAATzEMhJaVKp6s2nIkolIJ2WkBShpkVRWqqQrhLSEu9MZJKK9y1ZrqYK9WiClmvoUaF8gIQSNeF1Er4MNFn4SRSDARWroAIETg1iVwuHjYB1kYc1mwruwXKC9gmsJXliGxc+XiUCby9ydh1sOSdMkpMTBpaXBzsfhoc5l58Gm5yToAaZhaOUqjkDgCWNHAULCwOLaTmzswadEqggQwgHuQsHIoZCHQMMQgQGubVEcxOPFAcMDAYUA85eWARmfSRQCdcMe0zeP1AAygwLlJtPNAAL19DARdPzBOWSm1brJBi45soRAWQAAkrQIykShQ9wVhHCwCQCACH5BAAKAAYALAAAAAAgACAAAATrEMhJaVKp6s2nIkqFZF2VIBWhUsJaTokqUCoBq+E71SRQeyqUToLA7VxF0JDyIQh/MVVPMt1ECZlfcjZJ9mIKoaTl1MRIl5o4CUKXOwmyrCInCKqcWtvadL2SYhyASyNDJ0uIiRMDjI0Fd30/iI2UA5GSS5UDj2l6NoqgOgN4gksEBgYFf0FDqKgHnyZ9OX8HrgYHdHpcHQULXAS2qKpENRg7eAMLC7kTBaixUYFkKAzWAAnLC7FLVxLWDBLKCwaKTULgEwbLA4hJtOkSBNqITT3xEgfLpBtzE/jiuL04RGEBgwWhShRgQExHBAAh+QQACgAHACwAAAAAIAAgAAAE7xDISWlSqerNpyJKhWRdlSAVoVLCWk6JKlAqAavhO9UkUHsqlE6CwO1cRdCQ8iEIfzFVTzLdRAmZX3I2SfZiCqGk5dTESJeaOAlClzsJsqwiJwiqnFrb2nS9kmIcgEsjQydLiIlHehhpejaIjzh9eomSjZR+ipslWIRLAgMDOR2DOqKogTB9pCUJBagDBXR6XB0EBkIIsaRsGGMMAxoDBgYHTKJiUYEGDAzHC9EACcUGkIgFzgwZ0QsSBcXHiQvOwgDdEwfFs0sDzt4S6BK4xYjkDOzn0unFeBzOBijIm1Dgmg5YFQwsCMjp1oJ8LyIAACH5BAAKAAgALAAAAAAgACAAAATwEMhJaVKp6s2nIkqFZF2VIBWhUsJaTokqUCoBq+E71SRQeyqUToLA7VxF0JDyIQh/MVVPMt1ECZlfcjZJ9mIKoaTl1MRIl5o4CUKXOwmyrCInCKqcWtvadL2SYhyASyNDJ0uIiUd6GGl6NoiPOH16iZKNlH6KmyWFOggHhEEvAwwMA0N9GBsEC6amhnVcEwavDAazGwIDaH1ipaYLBUTCGgQDA8NdHz0FpqgTBwsLqAbWAAnIA4FWKdMLGdYGEgraigbT0OITBcg5QwPT4xLrROZL6AuQAPUS7bxLpoWidY0JtxLHKhwwMJBTHgPKdEQAACH5BAAKAAkALAAAAAAgACAAAATrEMhJaVKp6s2nIkqFZF2VIBWhUsJaTokqUCoBq+E71SRQeyqUToLA7VxF0JDyIQh/MVVPMt1ECZlfcjZJ9mIKoaTl1MRIl5o4CUKXOwmyrCInCKqcWtvadL2SYhyASyNDJ0uIiUd6GAULDJCRiXo1CpGXDJOUjY+Yip9DhToJA4RBLwMLCwVDfRgbBAaqqoZ1XBMHswsHtxtFaH1iqaoGNgAIxRpbFAgfPQSqpbgGBqUD1wBXeCYp1AYZ19JJOYgH1KwA4UBvQwXUBxPqVD9L3sbp2BNk2xvvFPJd+MFCN6HAAIKgNggY0KtEBAAh+QQACgAKACwAAAAAIAAgAAAE6BDISWlSqerNpyJKhWRdlSAVoVLCWk6JKlAqAavhO9UkUHsqlE6CwO1cRdCQ8iEIfzFVTzLdRAmZX3I2SfYIDMaAFdTESJeaEDAIMxYFqrOUaNW4E4ObYcCXaiBVEgULe0NJaxxtYksjh2NLkZISgDgJhHthkpU4mW6blRiYmZOlh4JWkDqILwUGBnE6TYEbCgevr0N1gH4At7gHiRpFaLNrrq8HNgAJA70AWxQIH1+vsYMDAzZQPC9VCNkDWUhGkuE5PxJNwiUK4UfLzOlD4WvzAHaoG9nxPi5d+jYUqfAhhykOFwJWiAAAIfkEAAoACwAsAAAAACAAIAAABPAQyElpUqnqzaciSoVkXVUMFaFSwlpOCcMYlErAavhOMnNLNo8KsZsMZItJEIDIFSkLGQoQTNhIsFehRww2CQLKF0tYGKYSg+ygsZIuNqJksKgbfgIGepNo2cIUB3V1B3IvNiBYNQaDSTtfhhx0CwVPI0UJe0+bm4g5VgcGoqOcnjmjqDSdnhgEoamcsZuXO1aWQy8KAwOAuTYYGwi7w5h+Kr0SJ8MFihpNbx+4Erq7BYBuzsdiH1jCAzoSfl0rVirNbRXlBBlLX+BP0XJLAPGzTkAuAOqb0WT5AH7OcdCm5B8TgRwSRKIHQtaLCwg1RAAAOwAAAAAAAAAAAA==") #111 no-repeat center center;
-webkit-border-radius: 8px;
-moz-border-radius: 8px;
-o-border-radius: 8px;
-ms-border-radius: 8px;
border-radius: 8px;
}

228
swarm/examples/album/jquery.modal.js Normal file
View file

@ -0,0 +1,228 @@
/*
A simple jQuery modal (http://github.com/kylefox/jquery-modal)
Version 0.7.0
*/
(function($) {
var modals = [],
getCurrent = function() {
return modals.length ? modals[modals.length - 1] : null;
},
selectCurrent = function() {
var i,
selected = false;
for (i=modals.length-1; i>=0; i--) {
if (modals[i].$blocker) {
modals[i].$blocker.toggleClass('current',!selected).toggleClass('behind',selected);
selected = true;
}
}
};
$.modal = function(el, options) {
var remove, target;
this.$body = $('body');
this.options = $.extend({}, $.modal.defaults, options);
this.options.doFade = !isNaN(parseInt(this.options.fadeDuration, 10));
this.$blocker = null;
if (this.options.closeExisting)
while ($.modal.isActive())
$.modal.close(); // Close any open modals.
modals.push(this);
if (el.is('a')) {
target = el.attr('href');
//Select element by id from href
if (/^#/.test(target)) {
this.$elm = $(target);
if (this.$elm.length !== 1) return null;
this.$body.append(this.$elm);
this.open();
//AJAX
} else {
this.$elm = $('<div>');
this.$body.append(this.$elm);
remove = function(event, modal) { modal.elm.remove(); };
this.showSpinner();
el.trigger($.modal.AJAX_SEND);
$.get(target).done(function(html) {
if (!$.modal.isActive()) return;
el.trigger($.modal.AJAX_SUCCESS);
var current = getCurrent();
current.$elm.empty().append(html).on($.modal.CLOSE, remove);
current.hideSpinner();
current.open();
el.trigger($.modal.AJAX_COMPLETE);
}).fail(function() {
el.trigger($.modal.AJAX_FAIL);
var current = getCurrent();
current.hideSpinner();
modals.pop(); // remove expected modal from the list
el.trigger($.modal.AJAX_COMPLETE);
});
}
} else {
this.$elm = el;
this.$body.append(this.$elm);
this.open();
}
};
$.modal.prototype = {
constructor: $.modal,
open: function() {
var m = this;
this.block();
if(this.options.doFade) {
setTimeout(function() {
m.show();
}, this.options.fadeDuration * this.options.fadeDelay);
} else {
this.show();
}
$(document).off('keydown.modal').on('keydown.modal', function(event) {
var current = getCurrent();
if (event.which == 27 && current.options.escapeClose) current.close();
});
if (this.options.clickClose)
this.$blocker.click(function(e) {
if (e.target==this)
$.modal.close();
});
},
close: function() {
modals.pop();
this.unblock();
this.hide();
if (!$.modal.isActive())
$(document).off('keydown.modal');
},
block: function() {
this.$elm.trigger($.modal.BEFORE_BLOCK, [this._ctx()]);
this.$body.css('overflow','hidden');
this.$blocker = $('<div class="jquery-modal blocker current"></div>').appendTo(this.$body);
selectCurrent();
if(this.options.doFade) {
this.$blocker.css('opacity',0).animate({opacity: 1}, this.options.fadeDuration);
}
this.$elm.trigger($.modal.BLOCK, [this._ctx()]);
},
unblock: function(now) {
if (!now && this.options.doFade)
this.$blocker.fadeOut(this.options.fadeDuration, this.unblock.bind(this,true));
else {
this.$blocker.children().appendTo(this.$body);
this.$blocker.remove();
this.$blocker = null;
selectCurrent();
if (!$.modal.isActive())
this.$body.css('overflow','');
}
},
show: function() {
this.$elm.trigger($.modal.BEFORE_OPEN, [this._ctx()]);
if (this.options.showClose) {
this.closeButton = $('<a href="#close-modal" rel="modal:close" class="close-modal ' + this.options.closeClass + '">' + this.options.closeText + '</a>');
this.$elm.append(this.closeButton);
}
this.$elm.addClass(this.options.modalClass).appendTo(this.$blocker);
if(this.options.doFade) {
this.$elm.css('opacity',0).show().animate({opacity: 1}, this.options.fadeDuration);
} else {
this.$elm.show();
}
this.$elm.trigger($.modal.OPEN, [this._ctx()]);
},
hide: function() {
this.$elm.trigger($.modal.BEFORE_CLOSE, [this._ctx()]);
if (this.closeButton) this.closeButton.remove();
var _this = this;
if(this.options.doFade) {
this.$elm.fadeOut(this.options.fadeDuration, function () {
_this.$elm.trigger($.modal.AFTER_CLOSE, [_this._ctx()]);
});
} else {
this.$elm.hide(0, function () {
_this.$elm.trigger($.modal.AFTER_CLOSE, [_this._ctx()]);
});
}
this.$elm.trigger($.modal.CLOSE, [this._ctx()]);
},
showSpinner: function() {
if (!this.options.showSpinner) return;
this.spinner = this.spinner || $('<div class="' + this.options.modalClass + '-spinner"></div>')
.append(this.options.spinnerHtml);
this.$body.append(this.spinner);
this.spinner.show();
},
hideSpinner: function() {
if (this.spinner) this.spinner.remove();
},
//Return context for custom events
_ctx: function() {
return { elm: this.$elm, $blocker: this.$blocker, options: this.options };
}
};
$.modal.close = function(event) {
if (!$.modal.isActive()) return;
if (event) event.preventDefault();
var current = getCurrent();
current.close();
return current.$elm;
};
// Returns if there currently is an active modal
$.modal.isActive = function () {
return modals.length > 0;
}
$.modal.defaults = {
closeExisting: true,
escapeClose: true,
clickClose: true,
closeText: 'Close',
closeClass: '',
modalClass: "modal",
spinnerHtml: null,
showSpinner: true,
showClose: true,
fadeDuration: null, // Number of milliseconds the fade animation takes.
fadeDelay: 1.0 // Point during the overlay's fade-in that the modal begins to fade in (.5 = 50%, 1.5 = 150%, etc.)
};
// Event constants
$.modal.BEFORE_BLOCK = 'modal:before-block';
$.modal.BLOCK = 'modal:block';
$.modal.BEFORE_OPEN = 'modal:before-open';
$.modal.OPEN = 'modal:open';
$.modal.BEFORE_CLOSE = 'modal:before-close';
$.modal.CLOSE = 'modal:close';
$.modal.AFTER_CLOSE = 'modal:after-close';
$.modal.AJAX_SEND = 'modal:ajax:send';
$.modal.AJAX_SUCCESS = 'modal:ajax:success';
$.modal.AJAX_FAIL = 'modal:ajax:fail';
$.modal.AJAX_COMPLETE = 'modal:ajax:complete';
$.fn.modal = function(options){
if (this.length === 1) {
new $.modal(this, options);
}
return this;
};
// Automatically bind links with rel="modal:close" to, well, close the modal.
$(document).on('click.modal', 'a[rel="modal:close"]', $.modal.close);
$(document).on('click.modal', 'a[rel="modal:open"]', function(event) {
event.preventDefault();
$(this).modal();
});
})(jQuery);

58
swarm/network/hive.go
View file

@ -30,6 +30,7 @@ type Hive struct {
addr kademlia.Address addr kademlia.Address
kad *kademlia.Kademlia kad *kademlia.Kademlia
path string path string
quit chan bool
toggle chan bool toggle chan bool
more chan bool more chan bool
@ -106,6 +107,7 @@ func (self *Hive) Addr() kademlia.Address {
func (self *Hive) Start(id discover.NodeID, listenAddr func() string, connectPeer func(string) error) (err error) { func (self *Hive) Start(id discover.NodeID, listenAddr func() string, connectPeer func(string) error) (err error) {
self.toggle = make(chan bool) self.toggle = make(chan bool)
self.more = make(chan bool) self.more = make(chan bool)
self.quit = make(chan bool)
self.id = id self.id = id
self.listenAddr = listenAddr self.listenAddr = listenAddr
err = self.kad.Load(self.path, nil) err = self.kad.Load(self.path, nil)
@ -123,13 +125,15 @@ func (self *Hive) Start(id discover.NodeID, listenAddr func() string, connectPee
// to attempt to write to more (remove Peer when shutting down) // to attempt to write to more (remove Peer when shutting down)
return return
} }
node, proxLimit := self.kad.FindBest() node, need, proxLimit := self.kad.Suggest()
if node != nil && len(node.Url) > 0 { if node != nil && len(node.Url) > 0 {
glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: call for bee %v", node.Url) glog.V(logger.Debug).Infof("[BZZ] KΛÐΞMLIΛ hive: call known bee %v", node.Url)
// enode or any lower level connection address is unnecessary in future // enode or any lower level connection address is unnecessary in future
// discovery table is used to look it up. // discovery table is used to look it up.
connectPeer(node.Url) connectPeer(node.Url)
} else if proxLimit > -1 { }
if need {
// a random peer is taken from the table // a random peer is taken from the table
peers := self.kad.FindClosest(kademlia.RandomAddressAt(self.addr, rand.Intn(self.kad.MaxProx)), 1) peers := self.kad.FindClosest(kademlia.RandomAddressAt(self.addr, rand.Intn(self.kad.MaxProx)), 1)
if len(peers) > 0 { if len(peers) > 0 {
@ -138,15 +142,21 @@ func (self *Hive) Start(id discover.NodeID, listenAddr func() string, connectPee
req := &retrieveRequestMsgData{ req := &retrieveRequestMsgData{
Key: storage.Key(randAddr[:]), Key: storage.Key(randAddr[:]),
} }
glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: call any bee in area %v messenger bee %v", randAddr, peers[0]) glog.V(logger.Debug).Infof("[BZZ] KΛÐΞMLIΛ hive: call any bee near %v (PO%03d) - messenger bee: %v", randAddr, proxLimit, peers[0])
peers[0].(*peer).retrieve(req) peers[0].(*peer).retrieve(req)
} else {
glog.V(logger.Warn).Infof("[BZZ] KΛÐΞMLIΛ hive: no peer")
} }
self.toggle <- true
glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: buzz kept alive") glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: buzz kept alive")
} else { } else {
self.toggle <- false glog.V(logger.Info).Infof("[BZZ] KΛÐΞMLIΛ hive: no need for more bees")
} }
glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: queen's address: %v, population: %d (%d)", self.addr, self.kad.Count(), self.kad.DBCount()) select {
case self.toggle <- need:
case <-self.quit:
return
}
glog.V(logger.Debug).Infof("[BZZ] KΛÐΞMLIΛ hive: queen's address: %v, population: %d (%d)", self.addr, self.kad.Count(), self.kad.DBCount())
} }
}() }()
return return
@ -165,14 +175,11 @@ func (self *Hive) keepAlive() {
if self.kad.DBCount() > 0 { if self.kad.DBCount() > 0 {
select { select {
case self.more <- true: case self.more <- true:
glog.V(logger.Debug).Infof("[BZZ] KΛÐΞMLIΛ hive: buzz wakeup")
default: default:
} }
} }
case need, alive := <-self.toggle: case need := <-self.toggle:
if !alive {
self.more <- false
return
}
if alarm == nil && need { if alarm == nil && need {
alarm = time.NewTicker(time.Duration(self.callInterval)).C alarm = time.NewTicker(time.Duration(self.callInterval)).C
} }
@ -180,20 +187,31 @@ func (self *Hive) keepAlive() {
alarm = nil alarm = nil
} }
case <-self.quit:
return
} }
} }
} }
func (self *Hive) Stop() error { func (self *Hive) Stop() error {
// closing toggle channel quits the updateloop // closing toggle channel quits the updateloop
close(self.toggle) close(self.quit)
return self.kad.Save(self.path, saveSync) return self.kad.Save(self.path, saveSync)
} }
// called at the end of a successful protocol handshake // called at the end of a successful protocol handshake
func (self *Hive) addPeer(p *peer) { func (self *Hive) addPeer(p *peer) error {
defer func() {
select {
case self.more <- true:
default:
}
}()
glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: hi new bee %v", p) glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: hi new bee %v", p)
self.kad.On(p, loadSync) err := self.kad.On(p, loadSync)
if err != nil {
return err
}
// self lookup (can be encoded as nil/zero key since peers addr known) + no id () // self lookup (can be encoded as nil/zero key since peers addr known) + no id ()
// the most common way of saying hi in bzz is initiation of gossip // the most common way of saying hi in bzz is initiation of gossip
// let me know about anyone new from my hood , here is the storageradius // let me know about anyone new from my hood , here is the storageradius
@ -201,10 +219,8 @@ func (self *Hive) addPeer(p *peer) {
// we do not record as request or forward it, just reply with peers // we do not record as request or forward it, just reply with peers
p.retrieve(&retrieveRequestMsgData{}) p.retrieve(&retrieveRequestMsgData{})
glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: 'whatsup wheresdaparty' sent to %v", p) glog.V(logger.Detail).Infof("[BZZ] KΛÐΞMLIΛ hive: 'whatsup wheresdaparty' sent to %v", p)
select {
case self.more <- true: return nil
default:
}
} }
// called after peer disconnected // called after peer disconnected
@ -241,7 +257,7 @@ func (self *Hive) DropAll() {
// contructor for kademlia.NodeRecord based on peer address alone // contructor for kademlia.NodeRecord based on peer address alone
// TODO: should go away and only addr passed to kademlia // TODO: should go away and only addr passed to kademlia
func newNodeRecord(addr *peerAddr) *kademlia.NodeRecord { func newNodeRecord(addr *peerAddr) *kademlia.NodeRecord {
now := kademlia.Time(time.Now()) now := time.Now()
return &kademlia.NodeRecord{ return &kademlia.NodeRecord{
Addr: addr.Addr, Addr: addr.Addr,
Url: addr.String(), Url: addr.String(),
@ -336,7 +352,7 @@ func (self *Hive) peers(req *retrieveRequestMsgData) {
for _, peer := range self.getPeers(key, int(req.MaxPeers)) { for _, peer := range self.getPeers(key, int(req.MaxPeers)) {
addrs = append(addrs, peer.remoteAddr) addrs = append(addrs, peer.remoteAddr)
} }
glog.V(logger.Detail).Infof("[BZZ] Hive sending %d peer addresses to %v. req.Id: %v, req.Key: %x", len(addrs), req.from, req.Id, req.Key.Log()) glog.V(logger.Debug).Infof("[BZZ] Hive sending %d peer addresses to %v. req.Id: %v, req.Key: %v", len(addrs), req.from, req.Id, req.Key.Log())
peersData := &peersMsgData{ peersData := &peersMsgData{
Peers: addrs, Peers: addrs,

230
swarm/network/kademlia/kaddb.go
View file

@ -12,26 +12,6 @@ import (
"github.com/ethereum/go-ethereum/logger/glog" "github.com/ethereum/go-ethereum/logger/glog"
) )
type Time time.Time
func (t *Time) MarshalJSON() (out []byte, err error) {
return []byte(fmt.Sprintf("%d", t.Unix())), nil
}
func (t *Time) UnmarshalJSON(value []byte) error {
var i int64
_, err := fmt.Sscanf(string(value), "%d", &i)
if err != nil {
return err
}
*t = Time(time.Unix(i, 0))
return nil
}
func (t Time) Unix() int64 {
return time.Time(t).Unix()
}
type NodeData interface { type NodeData interface {
json.Marshaler json.Marshaler
json.Unmarshaler json.Unmarshaler
@ -41,17 +21,17 @@ type NodeData interface {
type NodeRecord struct { type NodeRecord struct {
Addr Address // address of node Addr Address // address of node
Url string // Url, used to connect to node Url string // Url, used to connect to node
After Time // next call after time After time.Time // next call after time
Seen Time // last connected at time Seen time.Time // last connected at time
Meta *json.RawMessage // arbitrary metadata saved for a peer Meta *json.RawMessage // arbitrary metadata saved for a peer
node Node node Node
connected bool
} }
// set checked to current time,
func (self *NodeRecord) setSeen() { func (self *NodeRecord) setSeen() {
self.Seen = Time(time.Now()) t := time.Now()
self.Seen = t
self.After = t
} }
func (self *NodeRecord) String() string { func (self *NodeRecord) String() string {
@ -64,7 +44,7 @@ type KadDb struct {
Nodes [][]*NodeRecord Nodes [][]*NodeRecord
index map[Address]*NodeRecord index map[Address]*NodeRecord
cursors []int cursors []int
lock sync.Mutex lock sync.RWMutex
purgeInterval time.Duration purgeInterval time.Duration
initialRetryInterval time.Duration initialRetryInterval time.Duration
connRetryExp int connRetryExp int
@ -142,16 +122,19 @@ This is used to pick candidates for live nodes that are most wanted for
a higly connected low centrality network structure for Swarm which best suits a higly connected low centrality network structure for Swarm which best suits
for a Kademlia-style routing. for a Kademlia-style routing.
The candidate is chosen using the following strategy. * Starting as naive node with empty db, this implements Kademlia bootstrapping
* As a mature node, it fills short lines. All on demand.
The candidate is chosen using the following strategy:
We check for missing online nodes in the buckets for 1 upto Max BucketSize rounds. We check for missing online nodes in the buckets for 1 upto Max BucketSize rounds.
On each round we proceed from the low to high proximity order buckets. On each round we proceed from the low to high proximity order buckets.
If the number of active nodes (=connected peers) is < rounds, then start looking If the number of active nodes (=connected peers) is < rounds, then start looking
for a known candidate. To determine if there is a candidate to recommend the for a known candidate. To determine if there is a candidate to recommend the
node record database row corresponding to the bucket is checked. kaddb node record database row corresponding to the bucket is checked.
If the row cursor is on position i, the ith element in the row is chosen. If the row cursor is on position i, the ith element in the row is chosen.
If the record is scheduled not to be retried before NOW, the next element is taken. If the record is scheduled not to be retried before NOW, the next element is taken.
If the record is scheduled can be retried, it is set as checked, scheduled for If the record is scheduled to be retried, it is set as checked, scheduled for
checking and is returned. The time of the next check is in X (duration) such that checking and is returned. The time of the next check is in X (duration) such that
X = ConnRetryExp * delta where delta is the time past since the last check and X = ConnRetryExp * delta where delta is the time past since the last check and
ConnRetryExp is constant obsoletion factor. (Note that when node records are added ConnRetryExp is constant obsoletion factor. (Note that when node records are added
@ -167,121 +150,109 @@ offline past peer)
|| (proxBin(a) < proxBin(b) && |proxBin(a)| == |proxBin(b)|) || (proxBin(a) < proxBin(b) && |proxBin(a)| == |proxBin(b)|)
|| (proxBin(a) == proxBin(b) && lastChecked(a) < lastChecked(b)) || (proxBin(a) == proxBin(b) && lastChecked(a) < lastChecked(b))
This has double role. Starting as naive node with empty db, this implements
Kademlia bootstrapping
As a mature node, it fills short lines. All on demand.
The second argument returned names the first missing slot found The second argument returned names the first missing slot found
*/ */
func (self *KadDb) findBest(bucketSize int, binsize func(int) int) (node *NodeRecord, proxLimit int) { func (self *KadDb) findBest(maxBinSize int, binSize func(int) int) (node *NodeRecord, need bool, proxLimit int) {
// return value -1 indicates that buckets are filled in all // return nil, proxLimit indicates that all buckets are filled
proxLimit = -1
defer self.lock.Unlock() defer self.lock.Unlock()
self.lock.Lock() self.lock.Lock()
var interval int64 var interval time.Duration
var found bool var found bool
for rounds := 1; rounds <= bucketSize; rounds++ { var purge []bool
var delta time.Duration
var cursor int
var count int
var after time.Time
// iterate over columns maximum bucketsize times
for rounds := 1; rounds <= maxBinSize; rounds++ {
ROUND: ROUND:
// iterate over rows from PO 0 upto MaxProx
for po, dbrow := range self.Nodes { for po, dbrow := range self.Nodes {
if po > len(self.Nodes) { // if row has rounds connected peers, then take the next
break ROUND if binSize(po) >= rounds {
continue ROUND
} }
size := binsize(po) if !need {
if size < rounds { // set proxlimit to the PO where the first missing slot is found
if proxLimit < 0 { proxLimit = po
// set the first missing slot found need = true
proxLimit = po }
purge = make([]bool, len(dbrow))
// there is a missing slot - finding a node to connect to
// select a node record from the relavant kaddb row (of identical prox order)
ROW:
for cursor = self.cursors[po]; !found && count < len(dbrow); cursor = (cursor + 1) % len(dbrow) {
count++
node = dbrow[cursor]
// skip already connected nodes
if node.node != nil {
glog.V(logger.Debug).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d/%d) already connected", node.Addr, po, cursor, len(dbrow))
continue ROW
} }
var count int
var purge []int
n := self.cursors[po]
// try node records in the relavant kaddb row (of identical prox order) // if node is scheduled to connect
// if they are ripe for checking if time.Time(node.After).After(time.Now()) {
ROW: glog.V(logger.Debug).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d) skipped. seen at %v (%v ago), scheduled at %v", node.Addr, po, cursor, node.Seen, delta, node.After)
for count < len(dbrow) { continue ROW
node = dbrow[n]
// skip already connected nodes
if !node.connected {
glog.V(logger.Detail).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d) not to be retried before %v", node.Addr, po, n, node.After)
// time since last known connection attempt
delta := node.After.Unix() - node.Seen.Unix()
// if delta < 4 {
// node.After = Time(time.Time{})
// }
// if node is scheduled to connect
if time.Time(node.After).Before(time.Now()) {
// if checked longer than purge interval
if time.Time(node.Seen).Add(self.purgeInterval).Before(time.Now()) {
// delete node
purge = append(purge, n)
glog.V(logger.Detail).Infof("[KΛÐ]: inactive node record %v (PO%03d:%d) last check: %v, next check: %v", node.Addr, po, n, node.Seen, node.After)
} else {
// scheduling next check
if (node.After == Time(time.Time{})) {
node.After = Time(time.Now().Add(self.initialRetryInterval))
} else {
interval = delta * int64(self.connRetryExp)
node.After = Time(time.Unix(time.Now().Unix()+interval, 0))
}
glog.V(logger.Detail).Infof("[KΛÐ]: serve node record %v (PO%03d:%d), last check: %v, next check: %v", node.Addr, po, n, node.Seen, node.After)
}
found = true
break ROW
}
glog.V(logger.Detail).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d) not ready. skipped. not to be retried before: %v", node.Addr, po, n, node.After)
} // if node.node == nil
n++
count++
// cycle: n = n % len(dbrow)
if n >= len(dbrow) {
n = 0
}
} }
self.cursors[po] = n
self.delete(po, purge...)
if found {
glog.V(logger.Detail).Infof("[KΛÐ]: rounds %d: prox limit: PO%03d\n%v", rounds, proxLimit, node)
node.setSeen()
return
}
} // if len < rounds
} // for po-s
glog.V(logger.Detail).Infof("[KΛÐ]: rounds %d: proxlimit: PO%03d", rounds, proxLimit)
if proxLimit == 0 || proxLimit < 0 && bucketSize == rounds {
return
}
} // for round
return delta = time.Since(time.Time(node.Seen))
if delta < self.initialRetryInterval {
delta = self.initialRetryInterval
}
if delta > self.purgeInterval {
// remove node
purge[cursor] = true
glog.V(logger.Debug).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d) unreachable since %v. Removed", node.Addr, po, cursor, node.Seen)
continue ROW
}
glog.V(logger.Debug).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d) ready to be tried. seen at %v (%v ago), scheduled at %v", node.Addr, po, cursor, node.Seen, delta, node.After)
// scheduling next check
interval = time.Duration(delta * time.Duration(self.connRetryExp))
after = time.Now().Add(interval)
glog.V(logger.Debug).Infof("[KΛÐ]: kaddb record %v (PO%03d:%d) selected as candidate connection %v. seen at %v (%v ago), selectable since %v, retry after %v (in %v)", node.Addr, po, cursor, rounds, node.Seen, delta, node.After, after, interval)
node.After = after
found = true
} // ROW
self.cursors[po] = cursor
self.delete(po, purge)
if found {
return node, need, proxLimit
}
} // ROUND
} // ROUNDS
return nil, need, proxLimit
} }
// deletes the noderecords of a kaddb row corresponding to the indexes // deletes the noderecords of a kaddb row corresponding to the indexes
// caller must hold the dblock // caller must hold the dblock
// the call is unsafe, no index checks // the call is unsafe, no index checks
func (self *KadDb) delete(row int, indexes ...int) { func (self *KadDb) delete(row int, purge []bool) {
var prev int
var nodes []*NodeRecord var nodes []*NodeRecord
dbrow := self.Nodes[row] dbrow := self.Nodes[row]
for _, next := range indexes { for i, del := range purge {
// need to adjust dbcursor if i == self.cursors[row] {
if next > 0 { //reset cursor
if next <= self.cursors[row] { self.cursors[row] = len(nodes)
self.cursors[row]--
}
nodes = append(nodes, dbrow[prev:next]...)
} }
prev = next + 1 // delete the entry to be purged
delete(self.index, dbrow[next].Addr) if del {
delete(self.index, dbrow[i].Addr)
continue
}
// otherwise append to new list
nodes = append(nodes, dbrow[i])
} }
self.Nodes[row] = append(nodes, dbrow[prev:]...) self.Nodes[row] = nodes
} }
// save persists kaddb on disk (written to file on path in json format. // save persists kaddb on disk (written to file on path in json format.
@ -294,8 +265,8 @@ func (self *KadDb) save(path string, cb func(*NodeRecord, Node)) error {
for _, b := range self.Nodes { for _, b := range self.Nodes {
for _, node := range b { for _, node := range b {
n++ n++
node.After = Time(time.Now()) node.After = time.Now()
node.Seen = Time(time.Now()) node.Seen = time.Now()
if cb != nil { if cb != nil {
cb(node, node.node) cb(node, node.node)
} }
@ -331,24 +302,25 @@ func (self *KadDb) load(path string, cb func(*NodeRecord, Node) error) (err erro
return return
} }
var n int var n int
var purge []int var purge []bool
for po, b := range self.Nodes { for po, b := range self.Nodes {
purge = make([]bool, len(b))
ROW: ROW:
for i, node := range b { for i, node := range b {
if cb != nil { if cb != nil {
err = cb(node, node.node) err = cb(node, node.node)
if err != nil { if err != nil {
purge = append(purge, i) purge[i] = true
continue ROW continue ROW
} }
} }
n++ n++
if (node.After == Time(time.Time{})) { if (node.After == time.Time{}) {
node.After = Time(time.Now()) node.After = time.Now()
} }
self.index[node.Addr] = node self.index[node.Addr] = node
} }
self.delete(po, purge...) self.delete(po, purge)
} }
glog.V(logger.Info).Infof("[KΛÐ] loaded kaddb with %v nodes from %v", n, path) glog.V(logger.Info).Infof("[KΛÐ] loaded kaddb with %v nodes from %v", n, path)

291
swarm/network/kademlia/kademlia.go
View file

@ -12,16 +12,17 @@ import (
) )
const ( const (
bucketSize = 3 bucketSize = 4
proxBinSize = 4 proxBinSize = 2
maxProx = 8 maxProx = 8
connRetryExp = 2 connRetryExp = 2
maxPeers = 100
) )
var ( var (
purgeInterval = 42 * time.Hour purgeInterval = 42 * time.Hour
initialRetryInterval = 42 * 100 * time.Millisecond initialRetryInterval = 42 * time.Millisecond
maxIdleInterval = 42 * 10 * time.Second maxIdleInterval = 42 * 100 * time.Millisecond
) )
type KadParams struct { type KadParams struct {
@ -31,6 +32,7 @@ type KadParams struct {
BucketSize int BucketSize int
PurgeInterval time.Duration PurgeInterval time.Duration
InitialRetryInterval time.Duration InitialRetryInterval time.Duration
MaxIdleInterval time.Duration
ConnRetryExp int ConnRetryExp int
} }
@ -41,6 +43,7 @@ func NewKadParams() *KadParams {
BucketSize: bucketSize, BucketSize: bucketSize,
PurgeInterval: purgeInterval, PurgeInterval: purgeInterval,
InitialRetryInterval: initialRetryInterval, InitialRetryInterval: initialRetryInterval,
MaxIdleInterval: maxIdleInterval,
ConnRetryExp: connRetryExp, ConnRetryExp: connRetryExp,
} }
} }
@ -52,7 +55,7 @@ type Kademlia struct {
proxLimit int // state, the PO of the first row of the most proximate bin proxLimit int // state, the PO of the first row of the most proximate bin
proxSize int // state, the number of peers in the most proximate bin proxSize int // state, the number of peers in the most proximate bin
count int // number of active peers (w live connection) count int // number of active peers (w live connection)
buckets []*bucket // the actual bins buckets [][]Node // the actual bins
db *KadDb // kaddb, node record database db *KadDb // kaddb, node record database
lock sync.RWMutex // mutex to access buckets lock sync.RWMutex // mutex to access buckets
} }
@ -68,14 +71,7 @@ type Node interface {
// add is the base address of the table // add is the base address of the table
// params is KadParams configuration // params is KadParams configuration
func New(addr Address, params *KadParams) *Kademlia { func New(addr Address, params *KadParams) *Kademlia {
buckets := make([]*bucket, params.MaxProx+1) buckets := make([][]Node, params.MaxProx+1)
for i, _ := range buckets {
buckets[i] = &bucket{size: params.BucketSize} // will initialise bucket{int(0),[]Node(nil),sync.Mutex}
}
glog.V(logger.Info).Infof("[KΛÐ] base address %v", addr)
// ! temporary hack fixme:
params.ProxBinSize = 4
return &Kademlia{ return &Kademlia{
addr: addr, addr: addr,
KadParams: params, KadParams: params,
@ -104,14 +100,12 @@ func (self *Kademlia) DBCount() int {
// On is the entry point called when a new nodes is added // On is the entry point called when a new nodes is added
// unsafe in that node is not checked to be already active node (to be called once) // unsafe in that node is not checked to be already active node (to be called once)
func (self *Kademlia) On(node Node, cb func(*NodeRecord, Node) error) (err error) { func (self *Kademlia) On(node Node, cb func(*NodeRecord, Node) error) (err error) {
glog.V(logger.Warn).Infof("[KΛÐ]: %v", self)
defer self.lock.Unlock() defer self.lock.Unlock()
self.lock.Lock() self.lock.Lock()
index := self.proximityBin(node.Addr()) index := self.proximityBin(node.Addr())
record := self.db.findOrCreate(index, node.Addr(), node.Url()) record := self.db.findOrCreate(index, node.Addr(), node.Url())
// callback on add node
// setting the node on the record, set it checked (for connectivity)
record.node = node
if cb != nil { if cb != nil {
err = cb(record, node) err = cb(record, node)
@ -119,33 +113,46 @@ func (self *Kademlia) On(node Node, cb func(*NodeRecord, Node) error) (err error
if err != nil { if err != nil {
return fmt.Errorf("unable to add node %v, callback error: %v", node.Addr(), err) return fmt.Errorf("unable to add node %v, callback error: %v", node.Addr(), err)
} }
glog.V(logger.Info).Infof("[KΛÐ]: add node record %v with node %v", record, node) glog.V(logger.Debug).Infof("[KΛÐ]: add node record %v with node %v", record, node)
} }
record.connected = true
// insert in kademlia table of active nodes // insert in kademlia table of active nodes
bucket := self.buckets[index] bucket := self.buckets[index]
// if bucket is full insertion replaces the worst node // if bucket is full insertion replaces the worst node
// TODO: give priority to peers with active traffic // TODO: give priority to peers with active traffic
replaced, err := bucket.insert(node) if len(bucket) >= self.BucketSize { // >= allows us to add peers beyond the bucketsize limitation
if err != nil { // always rotate peers
glog.V(logger.Debug).Infof("[KΛÐ]: node %v not needed: %v", node, err) idle := self.MaxIdleInterval
return err var pos int
// no prox adjustment needed var replaced Node
// do not change count for i, p := range bucket {
idleInt := time.Since(p.LastActive())
if idleInt > idle {
idle = idleInt
pos = i
replaced = p
}
}
if replaced == nil {
glog.V(logger.Debug).Infof("[KΛÐ]: all peers wanted, PO%03d bucket full", index)
return fmt.Errorf("bucket full")
}
glog.V(logger.Debug).Infof("[KΛÐ]: node %v replaced by %v (idle for %v > %v)", replaced, node, idle, self.MaxIdleInterval)
replaced.Drop()
self.buckets[index] = append(bucket[:pos], bucket[(pos+1):]...)
// there is no change in bucket cardinalities so no prox limit adjustment is needed
return nil
} else {
self.buckets[index] = append(bucket, node)
glog.V(logger.Debug).Infof("[KΛÐ]: add node %v to table", node)
self.count++
self.setProxLimit(index, true)
} }
if replaced != nil { record.node = node
glog.V(logger.Debug).Infof("[KΛÐ]: node %v replaced by %v ", replaced, node) return nil
return
}
// new node added
glog.V(logger.Info).Infof("[KΛÐ]: add node %v to table", node)
self.count++
self.setProxLimit(index, false)
return
} }
// is the entrypoint called when a node is taken offline // Off is the called when a node is taken offline (from the protocol main loop exit)
func (self *Kademlia) Off(node Node, cb func(*NodeRecord, Node)) (err error) { func (self *Kademlia) Off(node Node, cb func(*NodeRecord, Node)) (err error) {
self.lock.Lock() self.lock.Lock()
defer self.lock.Unlock() defer self.lock.Unlock()
@ -153,70 +160,73 @@ func (self *Kademlia) Off(node Node, cb func(*NodeRecord, Node)) (err error) {
var found bool var found bool
index := self.proximityBin(node.Addr()) index := self.proximityBin(node.Addr())
bucket := self.buckets[index] bucket := self.buckets[index]
for i := 0; i < len(bucket.nodes); i++ { for i := 0; i < len(bucket); i++ {
if node.Addr() == bucket.nodes[i].Addr() { if node.Addr() == bucket[i].Addr() {
found = true found = true
bucket.nodes = append(bucket.nodes[:i], bucket.nodes[(i+1):]...) self.buckets[index] = append(bucket[:i], bucket[(i+1):]...)
break
} }
} }
if !found { if !found {
return // gracefully return without error if peer already unregistered
glog.V(logger.Warn).Infof("[KΛÐ]: remove node %v not in table, population now is %v", node, self.count)
return nil
} }
glog.V(logger.Info).Infof("[KΛÐ]: remove node %v from table", node)
self.count-- self.count--
if len(bucket.nodes) < bucket.size { glog.V(logger.Debug).Infof("[KΛÐ]: remove node %v from table, population now is %v", node, self.count)
err = fmt.Errorf("insufficient nodes (%v) in bucket %v", len(bucket.nodes), index) self.setProxLimit(index, false)
}
self.setProxLimit(index, true) record := self.db.index[node.Addr()]
r := self.db.index[node.Addr()]
// callback on remove // callback on remove
if cb != nil { if cb != nil {
cb(r, r.node) cb(record, record.node)
} }
r.node = nil record.node = nil
r.connected = false
return return
} }
// proxLimit is dynamically adjusted so that // proxLimit is dynamically adjusted so that
// 1) there is no empty buckets in bin < proxLimit and // 1) there is no empty buckets in bin < proxLimit and
// 2) the sum of all items sare the maximpossible but lower than ProxBinSize // 2) the sum of all items are the minimum possible but higher than ProxBinSize
// adjust Prox (proxLimit and proxSize after an insertion/removal of nodes) // adjust Prox (proxLimit and proxSize after an insertion/removal of nodes)
// caller holds the lock // caller holds the lock
func (self *Kademlia) setProxLimit(r int, off bool) { func (self *Kademlia) setProxLimit(r int, on bool) {
// glog.V(logger.Info).Infof("[KΛÐ]: adjust proxbin for (bin: %v, off: %v)", r, off) // if the change is outside the core (PO lower)
if r < self.proxLimit && len(self.buckets[r].nodes) > 0 { // and the change does not leave a bucket empty then
// no adjustment needed
if r < self.proxLimit && len(self.buckets[r]) > 0 {
return return
} }
glog.V(logger.Detail).Infof("[KΛÐ]: set proxbin (size: %v, limit: %v, bin: %v, off: %v)", self.proxSize, self.proxLimit, r, off) // if on=a node was added, then r must be within prox limit so increment cardinality
if off { if on {
self.proxSize-- self.proxSize++
for (self.proxSize < self.ProxBinSize || r < self.proxLimit) && curr := len(self.buckets[self.proxLimit])
self.proxLimit > 0 { // if now core is big enough without the furthest bucket, then contract
// // this can result in more than one bucket change
self.proxLimit-- for self.proxSize >= self.ProxBinSize+curr && curr > 0 {
self.proxSize += len(self.buckets[self.proxLimit].nodes) self.proxSize -= curr
glog.V(logger.Detail).Infof("[KΛÐ]: proxbin expansion (size: %v, limit: %v, bin: %v, off: %v)", self.proxSize, self.proxLimit, r, off) self.proxLimit++
} curr = len(self.buckets[self.proxLimit])
glog.V(logger.Detail).Infof("%v", self)
return
}
self.proxSize++
for self.proxLimit < self.MaxProx &&
len(self.buckets[self.proxLimit].nodes) > 0 &&
self.proxSize-len(self.buckets[self.proxLimit].nodes) >= self.ProxBinSize {
//
self.proxSize -= len(self.buckets[self.proxLimit].nodes)
self.proxLimit++
glog.V(logger.Detail).Infof("[KΛÐ]: proxbin contraction (size: %v, limit: %v, bin: %v, off: %v)", self.proxSize, self.proxLimit, r, off)
}
glog.V(logger.Detail).Infof("%v", self)
glog.V(logger.Detail).Infof("[KΛÐ]: proxbin contraction (size: %v, limit: %v, bin: %v)", self.proxSize, self.proxLimit, r)
}
return
}
// otherwise
if r >= self.proxLimit {
self.proxSize--
}
// expand core by lowering prox limit until hit zero or cover the empty bucket or reached target cardinality
for (self.proxSize < self.ProxBinSize || r < self.proxLimit) &&
self.proxLimit > 0 {
//
self.proxLimit--
self.proxSize += len(self.buckets[self.proxLimit])
glog.V(logger.Detail).Infof("[KΛÐ]: proxbin expansion (size: %v, limit: %v, bin: %v)", self.proxSize, self.proxLimit, r)
}
} }
/* /*
@ -225,62 +235,54 @@ as the target. The most proximate bin will be the union of the bins between
proxLimit and MaxProx. proxLimit and MaxProx.
*/ */
func (self *Kademlia) FindClosest(target Address, max int) []Node { func (self *Kademlia) FindClosest(target Address, max int) []Node {
defer self.lock.RUnlock() self.lock.Lock()
self.lock.RLock() defer self.lock.Unlock()
r := nodesByDistance{ r := nodesByDistance{
target: target, target: target,
} }
index := self.proximityBin(target)
start := index po := self.proximityBin(target)
var down bool index := po
if index >= self.proxLimit { step := 1
index = self.proxLimit glog.V(logger.Detail).Infof("[KΛÐ]: serving %v nodes at %v (PO%02d)", max, index, po)
start = self.MaxProx
down = true // if max is set to 0, just want a full bucket, dynamic number
} min := max
var n int // set limit to max
limit := max limit := max
if max == 0 { if max == 0 {
limit = 1000 min = 1
limit = maxPeers
} }
for {
bucket := self.buckets[start].nodes var n int
for i := 0; i < len(bucket); i++ { for index >= 0 {
r.push(bucket[i], limit) // add entire bucket
for _, p := range self.buckets[index] {
r.push(p, limit)
n++ n++
} }
if max == 0 && start <= index && (n > 0 || start == 0) || max > 0 && down && start <= index && (n >= limit || n == self.count || start == 0) { // terminate if index reached the bottom or enough peers > min
glog.V(logger.Detail).Infof("[KΛÐ]: add %v -> %v (PO%02d, PO%03d)", len(self.buckets[index]), n, index, po)
if n >= min && (step < 0 || max == 0) {
break break
} }
if down { // reach top most non-empty PO bucket, turn around
start-- if index == self.MaxProx {
} else { index = po
if start == self.MaxProx { step = -1
if index == 0 {
break
}
start = index - 1
down = true
} else {
start++
}
} }
index += step
} }
glog.V(logger.Detail).Infof("[KΛÐ]: serve %d (=<%d) nodes for target lookup %v (PO%d)", n, self.MaxProx, target, index) glog.V(logger.Detail).Infof("[KΛÐ]: serve %d (<=%d) nodes for target lookup %v (PO%03d)", n, max, target, po)
return r.nodes return r.nodes
} }
func (self *Kademlia) binsize(p int) int { func (self *Kademlia) Suggest() (*NodeRecord, bool, int) {
b := self.buckets[p] defer self.lock.RUnlock()
defer b.lock.RUnlock() self.lock.RLock()
b.lock.RLock() return self.db.findBest(self.BucketSize, func(i int) int { return len(self.buckets[i]) })
return len(b.nodes)
}
func (self *Kademlia) FindBest() (node *NodeRecord, proxLimit int) {
return self.db.findBest(self.BucketSize, self.binsize)
} }
// adds node records to kaddb (persisted node record db) // adds node records to kaddb (persisted node record db)
@ -288,13 +290,6 @@ func (self *Kademlia) Add(nrs []*NodeRecord) {
self.db.add(nrs, self.proximityBin) self.db.add(nrs, self.proximityBin)
} }
// in situ mutable bucket
type bucket struct {
size int
nodes []Node
lock sync.RWMutex
}
// nodesByDistance is a list of nodes, ordered by distance to target. // nodesByDistance is a list of nodes, ordered by distance to target.
type nodesByDistance struct { type nodesByDistance struct {
nodes []Node nodes []Node
@ -331,27 +326,6 @@ func (h *nodesByDistance) push(node Node, max int) {
} }
} }
// insert adds a peer to a bucket either by appending to existing items if
// bucket length does not exceed bucketSize, or by replacing the worst
// Node in the bucket
func (self *bucket) insert(node Node) (replaced Node, err error) {
self.lock.Lock()
defer self.lock.Unlock()
if len(self.nodes) >= self.size { // >= allows us to add peers beyond the bucketsize limitation
// dev p2p kicks out nodes idle for > 30 s, so here we never replace nodes if
// bucket is full
return nil, fmt.Errorf("bucket full")
}
self.nodes = append(self.nodes, node)
return
}
func (self *bucket) length(node Node) int {
self.lock.Lock()
defer self.lock.Unlock()
return len(self.nodes)
}
/* /*
Taking the proximity order relative to a fix point x classifies the points in Taking the proximity order relative to a fix point x classifies the points in
the space (n byte long byte sequences) into bins. Items in each are at the space (n byte long byte sequences) into bins. Items in each are at
@ -396,43 +370,46 @@ func (self *Kademlia) Load(path string, cb func(*NodeRecord, Node) error) (err e
} }
// kademlia table + kaddb table displayed with ascii // kademlia table + kaddb table displayed with ascii
// callerholds the lock
func (self *Kademlia) String() string { func (self *Kademlia) String() string {
defer self.lock.RUnlock()
self.lock.RLock()
defer self.db.lock.RUnlock()
self.db.lock.RLock()
var rows []string var rows []string
rows = append(rows, "=========================================================================") rows = append(rows, "=========================================================================")
rows = append(rows, fmt.Sprintf("KΛÐΞMLIΛ hive: queen's address: %v, population: %d (%d)", self.addr, self.Count(), self.DBCount())) rows = append(rows, fmt.Sprintf("%v KΛÐΞMLIΛ hive: queen's address: %v", time.Now().UTC().Format(time.UnixDate), self.addr.String()[:6]))
rows = append(rows, fmt.Sprintf("%v : MaxProx: %d, ProxBinSize: %d, BucketSize: %d, proxLimit: %d, proxSize: %d", time.Now(), self.MaxProx, self.ProxBinSize, self.BucketSize, self.proxLimit, self.proxSize)) rows = append(rows, fmt.Sprintf("population: %d (%d), proxLimit: %d, proxSize: %d", self.count, len(self.db.index), self.proxLimit, self.proxSize))
rows = append(rows, fmt.Sprintf("MaxProx: %d, ProxBinSize: %d, BucketSize: %d", self.MaxProx, self.ProxBinSize, self.BucketSize))
for i, b := range self.buckets { for i, bucket := range self.buckets {
if i == self.proxLimit { if i == self.proxLimit {
rows = append(rows, fmt.Sprintf("===================== PROX LIMIT: %d =================================", i)) rows = append(rows, fmt.Sprintf("============ PROX LIMIT: %d ==========================================", i))
} }
row := []string{fmt.Sprintf("%03d", i), fmt.Sprintf("%2d", len(b.nodes))} row := []string{fmt.Sprintf("%03d", i), fmt.Sprintf("%2d", len(bucket))}
var k int var k int
c := self.db.cursors[i] c := self.db.cursors[i]
for ; k < len(b.nodes); k++ { for ; k < len(bucket); k++ {
p := b.nodes[(c+k)%len(b.nodes)] p := bucket[(c+k)%len(bucket)]
row = append(row, fmt.Sprintf("%s", p.Addr().String()[:8])) row = append(row, p.Addr().String()[:6])
if k == 3 { if k == 4 {
break break
} }
} }
for ; k < 3; k++ { for ; k < 4; k++ {
row = append(row, " ") row = append(row, " ")
} }
row = append(row, fmt.Sprintf("| %2d %2d", len(self.db.Nodes[i]), self.db.cursors[i])) row = append(row, fmt.Sprintf("| %2d %2d", len(self.db.Nodes[i]), self.db.cursors[i]))
for j, p := range self.db.Nodes[i] { for j, p := range self.db.Nodes[i] {
row = append(row, fmt.Sprintf("%08x", p.Addr[:4])) row = append(row, p.Addr.String()[:6])
if j == 2 { if j == 3 {
break break
} }
} }
rows = append(rows, strings.Join(row, " ")) rows = append(rows, strings.Join(row, " "))
if i == self.MaxProx { if i == self.MaxProx {
break
} }
} }
rows = append(rows, "=========================================================================") rows = append(rows, "=========================================================================")

100
swarm/network/kademlia/kademlia_test.go
View file

@ -2,6 +2,7 @@ package kademlia
import ( import (
"fmt" "fmt"
"math"
"math/rand" "math/rand"
"reflect" "reflect"
"testing" "testing"
@ -11,8 +12,8 @@ import (
var ( var (
quickrand = rand.New(rand.NewSource(time.Now().Unix())) quickrand = rand.New(rand.NewSource(time.Now().Unix()))
quickcfgFindClosest = &quick.Config{MaxCount: 5000, Rand: quickrand} quickcfgFindClosest = &quick.Config{MaxCount: 50, Rand: quickrand}
quickcfgBootStrap = &quick.Config{MaxCount: 1000, Rand: quickrand} quickcfgBootStrap = &quick.Config{MaxCount: 100, Rand: quickrand}
) )
type testNode struct { type testNode struct {
@ -60,43 +61,36 @@ func TestBootstrap(t *testing.T) {
kad := New(test.Self, params) kad := New(test.Self, params)
var err error var err error
addr := RandomAddress() for p := 0; p < 9; p++ {
prox := proximity(addr, test.Self)
for p := 0; p <= prox; p++ {
var nrs []*NodeRecord var nrs []*NodeRecord
for i := 0; i < 3; i++ { n := math.Pow(float64(2), float64(7-p))
for i := 0; i < int(n); i++ {
addr := RandomAddressAt(test.Self, p)
nrs = append(nrs, &NodeRecord{ nrs = append(nrs, &NodeRecord{
Addr: RandomAddressAt(test.Self, p), Addr: addr,
}) })
} }
kad.Add(nrs) kad.Add(nrs)
} }
node := &testNode{addr} node := &testNode{test.Self}
n := 0 n := 0
for n < 100 { for n < 100 {
err = kad.On(node, nil) err = kad.On(node, nil)
if err != nil { if err != nil {
t.Errorf("backend not accepting node") t.Fatalf("backend not accepting node: %v", err)
return false
} }
var nrs []*NodeRecord
prox := proximity(test.Self, node.addr)
for i := 0; i < 13; i++ {
nrs = append(nrs, &NodeRecord{
Addr: RandomAddressAt(test.Self, prox+1),
})
}
kad.Add(nrs)
record, _ := kad.FindBest() record, need, _ := kad.Suggest()
if record == nil { if !need {
break break
} }
node = &testNode{record.Addr}
n++ n++
if record == nil {
continue
}
node = &testNode{record.Addr}
} }
exp := test.BucketSize * (test.MaxProx + 1) exp := test.BucketSize * (test.MaxProx + 1)
if kad.Count() != exp { if kad.Count() != exp {
@ -116,15 +110,13 @@ func TestFindClosest(t *testing.T) {
test := func(test *FindClosestTest) bool { test := func(test *FindClosestTest) bool {
// for any node kad.le, Target and N // for any node kad.le, Target and N
params := NewKadParams() params := NewKadParams()
params.MaxProx = 10 params.MaxProx = 7
kad := New(test.Self, params) kad := New(test.Self, params)
var err error var err error
// t.Logf("FindClosestTest %v: %v\n", len(test.All), test)
for _, node := range test.All { for _, node := range test.All {
err = kad.On(node, nil) err = kad.On(node, nil)
if err != nil { if err != nil && err.Error() != "bucket full" {
t.Errorf("backend not accepting node") t.Fatalf("backend not accepting node: %v", err)
return false
} }
} }
@ -157,17 +149,13 @@ func TestFindClosest(t *testing.T) {
// check that the result nodes have minimum distance to target. // check that the result nodes have minimum distance to target.
farthestResult := nodes[len(nodes)-1].Addr() farthestResult := nodes[len(nodes)-1].Addr()
for i, b := range kad.buckets { for i, b := range kad.buckets {
for j, n := range b.nodes { for j, n := range b {
if contains(nodes, n.Addr()) { if contains(nodes, n.Addr()) {
continue // don't run the check below for nodes in result continue // don't run the check below for nodes in result
} }
if test.Target.ProxCmp(n.Addr(), farthestResult) < 0 { if test.Target.ProxCmp(n.Addr(), farthestResult) < 0 {
_ = i * j _ = i * j
t.Errorf("kad.le contains node that is closer to target but it's not in result") t.Errorf("kad.le contains node that is closer to target but it's not in result")
// t.Logf("bucket %v, item %v\n", i, j)
// t.Logf(" Target: %x", test.Target)
// t.Logf(" Farthest Result: %x", farthestResult)
// t.Logf(" ID: %x (%d)", n.Addr(), kad.proximityBin(n.Addr()))
return false return false
} }
} }
@ -193,7 +181,7 @@ func TestProxAdjust(t *testing.T) {
r := rand.New(rand.NewSource(time.Now().UnixNano())) r := rand.New(rand.NewSource(time.Now().UnixNano()))
self := gen(Address{}, r).(Address) self := gen(Address{}, r).(Address)
params := NewKadParams() params := NewKadParams()
params.MaxProx = 10 params.MaxProx = 7
kad := New(self, params) kad := New(self, params)
var err error var err error
@ -201,15 +189,13 @@ func TestProxAdjust(t *testing.T) {
a := gen(Address{}, r).(Address) a := gen(Address{}, r).(Address)
addresses = append(addresses, a) addresses = append(addresses, a)
err = kad.On(&testNode{addr: a}, nil) err = kad.On(&testNode{addr: a}, nil)
if err != nil { if err != nil && err.Error() != "bucket full" {
t.Errorf("backend not accepting node") t.Fatalf("backend not accepting node: %v", err)
return
} }
if !kad.proxCheck(t) { if !kad.proxCheck(t) {
return return
} }
} }
test := func(test *proxTest) bool { test := func(test *proxTest) bool {
node := &testNode{test.addr} node := &testNode{test.addr}
if test.add { if test.add {
@ -229,35 +215,33 @@ func TestSaveLoad(t *testing.T) {
addresses := gen([]Address{}, r).([]Address) addresses := gen([]Address{}, r).([]Address)
self := RandomAddress() self := RandomAddress()
params := NewKadParams() params := NewKadParams()
params.MaxProx = 10 params.MaxProx = 7
kad := New(self, params) kad := New(self, params)
var err error var err error
for _, a := range addresses { for _, a := range addresses {
err = kad.On(&testNode{addr: a}, nil) err = kad.On(&testNode{addr: a}, nil)
if err != nil { if err != nil && err.Error() != "bucket full" {
t.Errorf("backend not accepting node") t.Fatalf("backend not accepting node: %v", err)
return
} }
} }
nodes := kad.FindClosest(self, 100) nodes := kad.FindClosest(self, 100)
path := "/tmp/bzz.peers" path := "/tmp/bzz.peers"
err = kad.Save(path, nil) err = kad.Save(path, nil)
if err != nil { if err != nil && err.Error() != "bucket full" {
t.Fatalf("unepected error saving kaddb: %v", err) t.Fatalf("unepected error saving kaddb: %v", err)
} }
kad = New(self, params) kad = New(self, params)
err = kad.Load(path, nil) err = kad.Load(path, nil)
if err != nil { if err != nil && err.Error() != "bucket full" {
t.Fatalf("unepected error loading kaddb: %v", err) t.Fatalf("unepected error loading kaddb: %v", err)
} }
for _, b := range kad.db.Nodes { for _, b := range kad.db.Nodes {
for _, node := range b { for _, node := range b {
err = kad.On(&testNode{node.Addr}, nil) err = kad.On(&testNode{node.Addr}, nil)
if err != nil { if err != nil && err.Error() != "bucket full" {
t.Errorf("backend not accepting node") t.Fatalf("backend not accepting node: %v", err)
return
} }
} }
} }
@ -270,30 +254,30 @@ func TestSaveLoad(t *testing.T) {
} }
func (self *Kademlia) proxCheck(t *testing.T) bool { func (self *Kademlia) proxCheck(t *testing.T) bool {
var sum, i int var sum int
var b *bucket for i, b := range self.buckets {
for i, b = range self.buckets { l := len(b)
l := len(b.nodes)
// if we are in the high prox multibucket // if we are in the high prox multibucket
if i >= self.proxLimit { if i >= self.proxLimit {
sum += l sum += l
} else if l == 0 { } else if l == 0 {
t.Errorf("bucket %d empty, yet proxLimit is %d\n%v", len(b.nodes), self.proxLimit, self) t.Errorf("bucket %d empty, yet proxLimit is %d\n%v", len(b), self.proxLimit, self)
return false return false
} }
} }
// check if merged high prox bucket does not exceed size // check if merged high prox bucket does not exceed size
if sum > 0 { if sum > 0 {
// if sum > self.ProxBinSize {
// t.Errorf("bucket %d is empty, yet proxSize is %d\n%v", i, self.proxSize, self)
// return false
// }
if sum != self.proxSize { if sum != self.proxSize {
t.Errorf("proxSize incorrect, expected %v, got %v", sum, self.proxSize) t.Errorf("proxSize incorrect, expected %v, got %v", sum, self.proxSize)
return false return false
} }
if self.proxLimit > 0 && sum+len(self.buckets[self.proxLimit-1].nodes) < self.ProxBinSize { last := len(self.buckets[self.proxLimit])
t.Errorf("proxBinSize incorrect, expected %v got %v", sum, self.proxSize) if last > 0 && sum >= self.ProxBinSize+last {
t.Errorf("proxLimit %v incorrect, redundant non-empty bucket %d added to proxBin with %v (target %v)\n%v", self.proxLimit, last, sum-last, self.ProxBinSize, self)
return false
}
if self.proxLimit > 0 && sum < self.ProxBinSize {
t.Errorf("proxLimit %v incorrect. proxSize %v is less than target %v, yet there is more peers", self.proxLimit, sum, self.ProxBinSize)
return false return false
} }
} }
@ -309,7 +293,7 @@ type bootstrapTest struct {
func (*bootstrapTest) Generate(rand *rand.Rand, size int) reflect.Value { func (*bootstrapTest) Generate(rand *rand.Rand, size int) reflect.Value {
t := &bootstrapTest{ t := &bootstrapTest{
Self: gen(Address{}, rand).(Address), Self: gen(Address{}, rand).(Address),
MaxProx: 10 + rand.Intn(3), MaxProx: 5 + rand.Intn(2),
BucketSize: rand.Intn(3) + 1, BucketSize: rand.Intn(3) + 1,
} }
return reflect.ValueOf(t) return reflect.ValueOf(t)

15
swarm/network/protocol.go
View file

@ -49,6 +49,7 @@ const (
ErrExtraStatusMsg ErrExtraStatusMsg
ErrSwap ErrSwap
ErrSync ErrSync
ErrUnwanted
) )
var errorToString = map[int]string{ var errorToString = map[int]string{
@ -61,6 +62,7 @@ var errorToString = map[int]string{
ErrExtraStatusMsg: "Extra status message", ErrExtraStatusMsg: "Extra status message",
ErrSwap: "SWAP error", ErrSwap: "SWAP error",
ErrSync: "Sync error", ErrSync: "Sync error",
ErrUnwanted: "Unwanted peer",
} }
// bzz represents the swarm wire protocol // bzz represents the swarm wire protocol
@ -179,7 +181,8 @@ func run(requestDb *storage.LDBDatabase, depo StorageHandler, backend bind.Backe
// the main forever loop that handles incoming requests // the main forever loop that handles incoming requests
for { for {
if self.hive.blockRead { if self.hive.blockRead {
time.Sleep(1 * time.Second) glog.V(logger.Warn).Infof("[BZZ] Cannot read network")
time.Sleep(100 * time.Millisecond)
continue continue
} }
err = self.handle() err = self.handle()
@ -223,7 +226,7 @@ func (self *bzz) handle() error {
if err := msg.Decode(&req); err != nil { if err := msg.Decode(&req); err != nil {
return self.protoError(ErrDecode, "<- %v: %v", msg, err) return self.protoError(ErrDecode, "<- %v: %v", msg, err)
} }
glog.V(logger.Debug).Infof("[BZZ] incoming store request: %s", req.String()) glog.V(logger.Detail).Infof("[BZZ] incoming store request: %s", req.String())
// swap accounting is done within forwarding // swap accounting is done within forwarding
self.storage.HandleStoreRequestMsg(&req, &peer{bzz: self}) self.storage.HandleStoreRequestMsg(&req, &peer{bzz: self})
@ -254,7 +257,7 @@ func (self *bzz) handle() error {
return self.protoError(ErrDecode, "<- %v: %v", msg, err) return self.protoError(ErrDecode, "<- %v: %v", msg, err)
} }
req.from = &peer{bzz: self} req.from = &peer{bzz: self}
glog.V(logger.Debug).Infof("[BZZ] <- peer addresses: %v", req) glog.V(logger.Detail).Infof("[BZZ] <- peer addresses: %v", req)
self.hive.HandlePeersMsg(&req, &peer{bzz: self}) self.hive.HandlePeersMsg(&req, &peer{bzz: self})
case syncRequestMsg: case syncRequestMsg:
@ -366,7 +369,10 @@ func (self *bzz) handleStatus() (err error) {
} }
glog.V(logger.Info).Infof("[BZZ] Peer %08x is [bzz] capable (%d/%d)", self.remoteAddr.Addr[:4], status.Version, status.NetworkId) glog.V(logger.Info).Infof("[BZZ] Peer %08x is [bzz] capable (%d/%d)", self.remoteAddr.Addr[:4], status.Version, status.NetworkId)
self.hive.addPeer(&peer{bzz: self}) err = self.hive.addPeer(&peer{bzz: self})
if err != nil {
return self.protoError(ErrUnwanted, "%v", err)
}
// hive sets syncstate so sync should start after node added // hive sets syncstate so sync should start after node added
glog.V(logger.Info).Infof("[BZZ] syncronisation request sent with %v", self.syncState) glog.V(logger.Info).Infof("[BZZ] syncronisation request sent with %v", self.syncState)
@ -516,7 +522,6 @@ func (self *bzz) send(msg uint64, data interface{}) error {
if self.hive.blockWrite { if self.hive.blockWrite {
return fmt.Errorf("network write blocked") return fmt.Errorf("network write blocked")
} }
// self.messages = append(self.messages, "")
glog.V(logger.Detail).Infof("[BZZ] -> %v: %v (%T) to %v", msg, data, data, self) glog.V(logger.Detail).Infof("[BZZ] -> %v: %v (%T) to %v", msg, data, data, self)
err := p2p.Send(self.rw, msg, data) err := p2p.Send(self.rw, msg, data)
if err != nil { if err != nil {

30
swarm/network/syncdb.go
View file

@ -126,7 +126,7 @@ LOOP:
// if syncdb is stopped. In this case we need to save the item to the db // if syncdb is stopped. In this case we need to save the item to the db
more = deliver(req, self.quit) more = deliver(req, self.quit)
if !more { if !more {
glog.V(logger.Debug).Infof("[BZZ] syncDb[%v] quit: switching to db. session tally (db/total): %v/%v", self.priority, self.dbTotal, self.total) glog.V(logger.Debug).Infof("[BZZ] syncDb[%v/%v] quit: switching to db. session tally (db/total): %v/%v", self.key.Log(), self.priority, self.dbTotal, self.total)
// received quit signal, save request currently waiting delivery // received quit signal, save request currently waiting delivery
// by switching to db mode and closing the buffer // by switching to db mode and closing the buffer
buffer = nil buffer = nil
@ -136,12 +136,12 @@ LOOP:
break // break from select, this item will be written to the db break // break from select, this item will be written to the db
} }
self.total++ self.total++
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] deliver (db/total): %v/%v", self.priority, self.dbTotal, self.total) glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] deliver (db/total): %v/%v", self.key.Log(), self.priority, self.dbTotal, self.total)
// by the time deliver returns, there were new writes to the buffer // by the time deliver returns, there were new writes to the buffer
// if buffer contention is detected, switch to db mode which drains // if buffer contention is detected, switch to db mode which drains
// the buffer so no process will block on pushing store requests // the buffer so no process will block on pushing store requests
if len(buffer) == cap(buffer) { if len(buffer) == cap(buffer) {
glog.V(logger.Debug).Infof("[BZZ] syncDb[%v] buffer full %v: switching to db. session tally (db/total): %v/%v", self.priority, cap(buffer), self.dbTotal, self.total) glog.V(logger.Debug).Infof("[BZZ] syncDb[%v/%v] buffer full %v: switching to db. session tally (db/total): %v/%v", self.key.Log(), self.priority, cap(buffer), self.dbTotal, self.total)
buffer = nil buffer = nil
db = self.buffer db = self.buffer
} }
@ -154,18 +154,18 @@ LOOP:
binary.BigEndian.PutUint64(counterValue, counter) binary.BigEndian.PutUint64(counterValue, counter)
batch.Put(self.counterKey, counterValue) // persist counter in batch batch.Put(self.counterKey, counterValue) // persist counter in batch
self.writeSyncBatch(batch) // save batch self.writeSyncBatch(batch) // save batch
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] quitting: save current batch to db", self.priority) glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] quitting: save current batch to db", self.key.Log(), self.priority)
break LOOP break LOOP
} }
self.dbTotal++ self.dbTotal++
self.total++ self.total++
// otherwise break after selec // otherwise break after select
case dbSize = <-self.batch: case dbSize = <-self.batch:
// explicit request for batch // explicit request for batch
if inBatch == 0 && quit != nil { if inBatch == 0 && quit != nil {
// there was no writes since the last batch so db depleted // there was no writes since the last batch so db depleted
// switch to buffer mode // switch to buffer mode
glog.V(logger.Debug).Infof("[BZZ] syncDb[%v] empty db: switching to buffer", self.priority) glog.V(logger.Debug).Infof("[BZZ] syncDb[%v/%v] empty db: switching to buffer", self.key.Log(), self.priority)
db = nil db = nil
buffer = self.buffer buffer = self.buffer
dbSize <- 0 // indicates to 'caller' that batch has been written dbSize <- 0 // indicates to 'caller' that batch has been written
@ -174,7 +174,7 @@ LOOP:
} }
binary.BigEndian.PutUint64(counterValue, counter) binary.BigEndian.PutUint64(counterValue, counter)
batch.Put(self.counterKey, counterValue) batch.Put(self.counterKey, counterValue)
glog.V(logger.Debug).Infof("[BZZ] syncDb[%v] write batch %v/%v - %x - %x", self.priority, inBatch, counter, self.counterKey, counterValue) glog.V(logger.Debug).Infof("[BZZ] syncDb[%v/%v] write batch %v/%v - %x - %x", self.key.Log(), self.priority, inBatch, counter, self.counterKey, counterValue)
batch = self.writeSyncBatch(batch) batch = self.writeSyncBatch(batch)
dbSize <- inBatch // indicates to 'caller' that batch has been written dbSize <- inBatch // indicates to 'caller' that batch has been written
inBatch = 0 inBatch = 0
@ -186,7 +186,7 @@ LOOP:
db = self.buffer db = self.buffer
buffer = nil buffer = nil
quit = nil quit = nil
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] quitting: save buffer to db", self.priority) glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] quitting: save buffer to db", self.key.Log(), self.priority)
close(db) close(db)
continue LOOP continue LOOP
} }
@ -194,15 +194,15 @@ LOOP:
// only get here if we put req into db // only get here if we put req into db
entry, err = self.newSyncDbEntry(req, counter) entry, err = self.newSyncDbEntry(req, counter)
if err != nil { if err != nil {
glog.V(logger.Warn).Infof("[BZZ] syncDb[%v] saving request %v (#%v/%v) failed: %v", self.priority, req, inBatch, inDb, err) glog.V(logger.Warn).Infof("[BZZ] syncDb[%v/%v] saving request %v (#%v/%v) failed: %v", self.key.Log(), self.priority, req, inBatch, inDb, err)
continue LOOP continue LOOP
} }
batch.Put(entry.key, entry.val) batch.Put(entry.key, entry.val)
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] to batch %v '%v' (#%v/%v/%v)", self.priority, req, entry, inBatch, inDb, counter) glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] to batch %v '%v' (#%v/%v/%v)", self.key.Log(), self.priority, req, entry, inBatch, inDb, counter)
// if just switched to db mode and not quitting, then launch dbRead // if just switched to db mode and not quitting, then launch dbRead
// in a parallel go routine to send deliveries from db // in a parallel go routine to send deliveries from db
if inDb == 0 && quit != nil { if inDb == 0 && quit != nil {
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] start dbRead") glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] start dbRead")
go self.dbRead(true, counter, deliver) go self.dbRead(true, counter, deliver)
} }
inDb++ inDb++
@ -221,7 +221,7 @@ LOOP:
func (self *syncDb) writeSyncBatch(batch *leveldb.Batch) *leveldb.Batch { func (self *syncDb) writeSyncBatch(batch *leveldb.Batch) *leveldb.Batch {
err := self.db.Write(batch) err := self.db.Write(batch)
if err != nil { if err != nil {
glog.V(logger.Warn).Infof("[BZZ] syncDb[%v] saving batch to db failed: %v", self.priority, err) glog.V(logger.Warn).Infof("[BZZ] syncDb[%v/%v] saving batch to db failed: %v", self.key.Log(), self.priority, err)
return batch return batch
} }
return new(leveldb.Batch) return new(leveldb.Batch)
@ -295,7 +295,7 @@ func (self *syncDb) dbRead(useBatches bool, counter uint64, fun func(interface{}
continue continue
} }
del = new(leveldb.Batch) del = new(leveldb.Batch)
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v]: new iterator: %x (batch %v, count %v)", self.priority, key, batches, cnt) glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v]: new iterator: %x (batch %v, count %v)", self.key.Log(), self.priority, key, batches, cnt)
for n = 0; !useBatches || n < cnt; it.Next() { for n = 0; !useBatches || n < cnt; it.Next() {
copy(key, it.Key()) copy(key, it.Key())
@ -307,11 +307,11 @@ func (self *syncDb) dbRead(useBatches bool, counter uint64, fun func(interface{}
val := make([]byte, 40) val := make([]byte, 40)
copy(val, it.Value()) copy(val, it.Value())
entry = &syncDbEntry{key, val} entry = &syncDbEntry{key, val}
// glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] - %v, batches: %v, total: %v, session total from db: %v/%v", self.priority, self.key.Log(), batches, total, self.dbTotal, self.total) // glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] - %v, batches: %v, total: %v, session total from db: %v/%v", self.key.Log(), self.priority, self.key.Log(), batches, total, self.dbTotal, self.total)
more = fun(entry, self.quit) more = fun(entry, self.quit)
if !more { if !more {
// quit received when waiting to deliver entry, the entry will not be deleted // quit received when waiting to deliver entry, the entry will not be deleted
glog.V(logger.Detail).Infof("[BZZ] syncDb[%v] batch %v quit after %v/%v items", self.priority, batches, n, cnt) glog.V(logger.Detail).Infof("[BZZ] syncDb[%v/%v] batch %v quit after %v/%v items", self.key.Log(), self.priority, batches, n, cnt)
break break
} }
// since subsequent batches of the same db session are indexed incrementally // since subsequent batches of the same db session are indexed incrementally

60
swarm/network/syncer.go
View file

@ -298,6 +298,7 @@ func (self *syncer) sync() {
if state.LastSeenAt < state.SessionAt { if state.LastSeenAt < state.SessionAt {
state.Last = state.SessionAt state.Last = state.SessionAt
glog.V(logger.Debug).Infof("[BZZ] syncer[%v]: start syncronising history since last disconnect at %v up until session start at %v: %v", self.key.Log(), state.LastSeenAt, state.SessionAt, state) glog.V(logger.Debug).Infof("[BZZ] syncer[%v]: start syncronising history since last disconnect at %v up until session start at %v: %v", self.key.Log(), state.LastSeenAt, state.SessionAt, state)
// blocks until state syncing is finished
self.syncState(state) self.syncState(state)
} }
glog.V(logger.Info).Infof("[BZZ] syncer[%v]: syncing all history complete", self.key.Log()) glog.V(logger.Info).Infof("[BZZ] syncer[%v]: syncing all history complete", self.key.Log())
@ -316,7 +317,7 @@ func (self *syncer) syncState(state *syncState) {
// stop quits both request processor and saves the request cache to disk // stop quits both request processor and saves the request cache to disk
func (self *syncer) stop() { func (self *syncer) stop() {
close(self.quit) close(self.quit)
glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: stopand save sync request db backlog", self.key.Log()) glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: stop and save sync request db backlog", self.key.Log())
for _, db := range self.queues { for _, db := range self.queues {
db.stop() db.stop()
} }
@ -358,19 +359,18 @@ func (self *syncer) syncHistory(state *syncState) chan interface{} {
IT: IT:
for { for {
key := it.Next() key := it.Next()
if key != nil { if key == nil {
select {
// blocking until history channel is read from
case history <- storage.Key(key):
n++
glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: history: %v (%v keys)", self.key.Log(), key.Log(), n)
state.Latest = key
case <-self.quit:
return
}
} else {
break IT break IT
} }
select {
// blocking until history channel is read from
case history <- storage.Key(key):
n++
glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: history: %v (%v keys)", self.key.Log(), key.Log(), n)
state.Latest = key
case <-self.quit:
return
}
} }
glog.V(logger.Debug).Infof("[BZZ] syncer[%v]: finished syncing history between %v - %v for chunk addresses %v - %v (at %v) (chunks = %v)", self.key.Log(), state.First, state.Last, state.Start, state.Stop, state.Latest, n) glog.V(logger.Debug).Infof("[BZZ] syncer[%v]: finished syncing history between %v - %v for chunk addresses %v - %v (at %v) (chunks = %v)", self.key.Log(), state.First, state.Last, state.Start, state.Stop, state.Latest, n)
}() }()
@ -416,25 +416,31 @@ LOOP:
// are checked first - integrity can only be guaranteed if writing // are checked first - integrity can only be guaranteed if writing
// is locked while selecting // is locked while selecting
if priority != High || len(keys) == 0 { if priority != High || len(keys) == 0 {
// selection is not needed if the High priority queue has items
keys = nil keys = nil
PRIORITIES:
for priority = High; priority >= 0; priority-- { for priority = High; priority >= 0; priority-- {
// the first priority channel that is non-empty will be assigned to keys
if len(self.keys[priority]) > 0 { if len(self.keys[priority]) > 0 {
glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: reading request with priority %v", self.key.Log(), priority) glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: reading request with priority %v", self.key.Log(), priority)
keys = self.keys[priority] keys = self.keys[priority]
break break PRIORITIES
} }
glog.V(logger.Debug).Infof("[BZZ] syncer[%v/%v]: queue: [%v, %v, %v]", self.key.Log(), priority, len(self.keys[High]), len(self.keys[Medium]), len(self.keys[Low]))
// if the input queue is empty on this level, resort to history if there is any
if uint(priority) == histPrior && history != nil { if uint(priority) == histPrior && history != nil {
glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: reading history for %v", self.key.Log(), self.key) glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: reading history for %v", self.key.Log(), self.key)
keys = history keys = history
break break PRIORITIES
} }
} }
// if peer ready to receive but nothing to send }
if keys == nil && deliveryRequest == nil {
// if no items left and switch to waiting mode // if peer ready to receive but nothing to send
glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: buffers consumed. Waiting", self.key.Log()) if keys == nil && deliveryRequest == nil {
newUnsyncedKeys = self.newUnsyncedKeys // if no items left and switch to waiting mode
} glog.V(logger.Detail).Infof("[BZZ] syncer[%v]: buffers consumed. Waiting", self.key.Log())
newUnsyncedKeys = self.newUnsyncedKeys
} }
// send msg iff // send msg iff
@ -447,7 +453,7 @@ LOOP:
len(unsynced) > 0 && keys == nil || len(unsynced) > 0 && keys == nil ||
len(unsynced) == int(self.SyncBatchSize)) { len(unsynced) == int(self.SyncBatchSize)) {
justSynced = false justSynced = false
// listen to requests again // listen to requests
deliveryRequest = self.deliveryRequest deliveryRequest = self.deliveryRequest
newUnsyncedKeys = nil // not care about data until next req comes in newUnsyncedKeys = nil // not care about data until next req comes in
// set sync to current counter // set sync to current counter
@ -458,11 +464,11 @@ LOOP:
// send the unsynced keys // send the unsynced keys
stateCopy := *state stateCopy := *state
err := self.unsyncedKeys(unsynced, &stateCopy) err := self.unsyncedKeys(unsynced, &stateCopy)
self.state = state
glog.V(logger.Debug).Infof("[BZZ] syncer[%v]: --> %v keys sent: (total: %v (%v), history: %v), sent sync state: %v", self.key.Log(), len(unsynced), keyCounts, keyCount, historyCnt, stateCopy)
if err != nil { if err != nil {
glog.V(logger.Warn).Infof("[BZZ] syncer[%v]: unable to send unsynced keys: %v", err) glog.V(logger.Warn).Infof("[BZZ] syncer[%v]: unable to send unsynced keys: %v", err)
} }
self.state = state
glog.V(logger.Debug).Infof("[BZZ] syncer[%v]: --> %v keys sent: (total: %v (%v), history: %v), sent sync state: %v", self.key.Log(), len(unsynced), keyCounts, keyCount, historyCnt, stateCopy)
unsynced = nil unsynced = nil
keys = nil keys = nil
} }
@ -477,7 +483,11 @@ LOOP:
// history channel is closed, waiting for new state (called from sync()) // history channel is closed, waiting for new state (called from sync())
syncStates = self.syncStates syncStates = self.syncStates
state.Synced = true // this signals that the current segment is complete state.Synced = true // this signals that the current segment is complete
state.synced <- false select {
case state.synced <- false:
case <-self.quit:
break LOOP
}
justSynced = true justSynced = true
history = nil history = nil
} }
@ -575,7 +585,7 @@ func (self *syncer) syncDeliveries() {
total++ total++
msg, err = self.newStoreRequestMsgData(req) msg, err = self.newStoreRequestMsgData(req)
if err != nil { if err != nil {
glog.V(logger.Warn).Infof("[BZZ] syncer[%v]: failed to deliver %v: %v", self.key.Log(), req, err) glog.V(logger.Warn).Infof("[BZZ] syncer[%v]: failed to create store request for %v: %v", self.key.Log(), req, err)
} else { } else {
err = self.store(msg) err = self.store(msg)
if err != nil { if err != nil {

6
swarm/services/chequebook/cheque.go
View file

@ -131,8 +131,10 @@ func NewChequebook(path string, contractAddr common.Address, prvKey *ecdsa.Priva
} }
func (self *Chequebook) setBalanceFromBlockChain() { func (self *Chequebook) setBalanceFromBlockChain() {
balance := self.backend.BalanceAt(self.contractAddr) balance, err := self.backend.BalanceAt(self.contractAddr)
self.balance.Set(balance) if err != nil {
self.balance.Set(balance)
}
} }
// LoadChequebook(path, prvKey, backend) loads a chequebook from disk (file path) // LoadChequebook(path, prvKey, backend) loads a chequebook from disk (file path)

13
swarm/services/chequebook/cheque_test.go
View file

@ -10,7 +10,6 @@ import (
"github.com/ethereum/go-ethereum/accounts/abi/bind/backends" "github.com/ethereum/go-ethereum/accounts/abi/bind/backends"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core" "github.com/ethereum/go-ethereum/core"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/crypto" "github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/swarm/services/chequebook/contract" "github.com/ethereum/go-ethereum/swarm/services/chequebook/contract"
) )
@ -42,16 +41,16 @@ func newTestBackend() *testBackend {
return &testBackend{SimulatedBackend: backends.NewSimulatedBackend(accs...)} return &testBackend{SimulatedBackend: backends.NewSimulatedBackend(accs...)}
} }
func (b *testBackend) GetTxReceipt(txhash common.Hash) *types.Receipt { func (b *testBackend) GetTxReceipt(txhash common.Hash) (map[string]interface{}, error) {
return nil return nil, nil
} }
func (b *testBackend) CodeAt(address common.Address) string { func (b *testBackend) CodeAt(address common.Address) (string, error) {
return "" return "", nil
} }
func (b *testBackend) BalanceAt(address common.Address) *big.Int { func (b *testBackend) BalanceAt(address common.Address) (*big.Int, error) {
return big.NewInt(0) return big.NewInt(0), nil
} }
func deploy(prvKey *ecdsa.PrivateKey, amount *big.Int, backend *backends.SimulatedBackend) (common.Address, error) { func deploy(prvKey *ecdsa.PrivateKey, amount *big.Int, backend *backends.SimulatedBackend) (common.Address, error) {

684
swarm/services/ens/contract/ens.go
View file

@ -12,108 +12,108 @@ import (
"github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/core/types"
) )
// ENSABI is the input ABI used to generate the binding from. // ResolverABI is the input ABI used to generate the binding from.
const ENSABI = `[{"constant":true,"inputs":[{"name":"","type":"bytes32"}],"name":"Owners","outputs":[{"name":"","type":"address"}],"type":"function"},{"constant":false,"inputs":[],"name":"kill","outputs":[],"type":"function"},{"constant":true,"inputs":[{"name":"","type":"bytes32"}],"name":"Registry","outputs":[{"name":"","type":"bytes32"}],"type":"function"},{"constant":false,"inputs":[{"name":"host","type":"bytes32"},{"name":"content","type":"bytes32"}],"name":"Set","outputs":[],"type":"function"}]` const ResolverABI = `[{"constant":false,"inputs":[{"name":"rootNodeId","type":"bytes12"},{"name":"name","type":"bytes32[]"}],"name":"deletePrivateRR","outputs":[],"type":"function"},{"constant":true,"inputs":[],"name":"isPersonalResolver","outputs":[{"name":"","type":"bool"}],"type":"function"},{"constant":false,"inputs":[{"name":"label","type":"bytes32"},{"name":"newOwner","type":"address"}],"name":"setOwner","outputs":[],"type":"function"},{"constant":true,"inputs":[{"name":"id","type":"bytes32"}],"name":"getExtended","outputs":[{"name":"data","type":"bytes"}],"type":"function"},{"constant":false,"inputs":[{"name":"rootNodeId","type":"bytes12"},{"name":"name","type":"string"},{"name":"rtype","type":"bytes16"},{"name":"ttl","type":"uint32"},{"name":"len","type":"uint16"},{"name":"data","type":"bytes32"}],"name":"setRR","outputs":[],"type":"function"},{"constant":false,"inputs":[{"name":"rootNodeId","type":"bytes12"},{"name":"name","type":"bytes32[]"},{"name":"rtype","type":"bytes16"},{"name":"ttl","type":"uint32"},{"name":"len","type":"uint16"},{"name":"data","type":"bytes32"}],"name":"setPrivateRR","outputs":[],"type":"function"},{"constant":true,"inputs":[{"name":"nodeId","type":"bytes12"},{"name":"qtype","type":"bytes32"},{"name":"index","type":"uint16"}],"name":"resolve","outputs":[{"name":"rcode","type":"uint16"},{"name":"rtype","type":"bytes16"},{"name":"ttl","type":"uint32"},{"name":"len","type":"uint16"},{"name":"data","type":"bytes32"}],"type":"function"},{"constant":false,"inputs":[{"name":"label","type":"bytes32"},{"name":"resolver","type":"address"},{"name":"nodeId","type":"bytes12"}],"name":"register","outputs":[],"type":"function"},{"constant":false,"inputs":[{"name":"label","type":"bytes32"},{"name":"resolver","type":"address"},{"name":"nodeId","type":"bytes12"}],"name":"setResolver","outputs":[],"type":"function"},{"constant":false,"inputs":[{"name":"rootNodeId","type":"bytes12"},{"name":"name","type":"string"}],"name":"deleteRR","outputs":[],"type":"function"},{"constant":true,"inputs":[{"name":"label","type":"bytes32"}],"name":"getOwner","outputs":[{"name":"","type":"address"}],"type":"function"},{"constant":true,"inputs":[{"name":"nodeId","type":"bytes12"},{"name":"label","type":"bytes32"}],"name":"findResolver","outputs":[{"name":"rcode","type":"uint16"},{"name":"ttl","type":"uint32"},{"name":"rnode","type":"bytes12"},{"name":"raddress","type":"address"}],"type":"function"}]`
// ENSBin is the compiled bytecode used for deploying new contracts. // ResolverBin is the compiled bytecode used for deploying new contracts.
const ENSBin = `0x606060405260008054600160a060020a03191633179055610148806100246000396000f3606060405260e060020a60003504633d14b257811461003c57806341c0e1b51461005d578063a0d03d3614610085578063be36e6761461009d575b005b61013c600435600260205260009081526040902054600160a060020a031681565b61003a60005433600160a060020a039081169116141561014657600054600160a060020a0316ff5b61013c60043560016020526000908152604090205481565b61003a600435602435600082815260026020526040812054600160a060020a031614156100e7576040600020805473ffffffffffffffffffffffffffffffffffffffff1916321790555b32600160a060020a03166002600050600084815260200190815260200160002060009054906101000a9004600160a060020a0316600160a060020a0316141561013857600160205260406000208190555b5050565b6060908152602090f35b56` const ResolverBin = `0x`
// DeployENS deploys a new Ethereum contract, binding an instance of ENS to it. // DeployResolver deploys a new Ethereum contract, binding an instance of Resolver to it.
func DeployENS(auth *bind.TransactOpts, backend bind.ContractBackend) (common.Address, *types.Transaction, *ENS, error) { func DeployResolver(auth *bind.TransactOpts, backend bind.ContractBackend) (common.Address, *types.Transaction, *Resolver, error) {
parsed, err := abi.JSON(strings.NewReader(ENSABI)) parsed, err := abi.JSON(strings.NewReader(ResolverABI))
if err != nil { if err != nil {
return common.Address{}, nil, nil, err return common.Address{}, nil, nil, err
} }
address, tx, contract, err := bind.DeployContract(auth, parsed, common.FromHex(ENSBin), backend) address, tx, contract, err := bind.DeployContract(auth, parsed, common.FromHex(ResolverBin), backend)
if err != nil { if err != nil {
return common.Address{}, nil, nil, err return common.Address{}, nil, nil, err
} }
return address, tx, &ENS{ENSCaller: ENSCaller{contract: contract}, ENSTransactor: ENSTransactor{contract: contract}}, nil return address, tx, &Resolver{ResolverCaller: ResolverCaller{contract: contract}, ResolverTransactor: ResolverTransactor{contract: contract}}, nil
} }
// ENS is an auto generated Go binding around an Ethereum contract. // Resolver is an auto generated Go binding around an Ethereum contract.
type ENS struct { type Resolver struct {
ENSCaller // Read-only binding to the contract ResolverCaller // Read-only binding to the contract
ENSTransactor // Write-only binding to the contract ResolverTransactor // Write-only binding to the contract
} }
// ENSCaller is an auto generated read-only Go binding around an Ethereum contract. // ResolverCaller is an auto generated read-only Go binding around an Ethereum contract.
type ENSCaller struct { type ResolverCaller struct {
contract *bind.BoundContract // Generic contract wrapper for the low level calls contract *bind.BoundContract // Generic contract wrapper for the low level calls
} }
// ENSTransactor is an auto generated write-only Go binding around an Ethereum contract. // ResolverTransactor is an auto generated write-only Go binding around an Ethereum contract.
type ENSTransactor struct { type ResolverTransactor struct {
contract *bind.BoundContract // Generic contract wrapper for the low level calls contract *bind.BoundContract // Generic contract wrapper for the low level calls
} }
// ENSSession is an auto generated Go binding around an Ethereum contract, // ResolverSession is an auto generated Go binding around an Ethereum contract,
// with pre-set call and transact options. // with pre-set call and transact options.
type ENSSession struct { type ResolverSession struct {
Contract *ENS // Generic contract binding to set the session for Contract *Resolver // Generic contract binding to set the session for
CallOpts bind.CallOpts // Call options to use throughout this session CallOpts bind.CallOpts // Call options to use throughout this session
TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session
} }
// ENSCallerSession is an auto generated read-only Go binding around an Ethereum contract, // ResolverCallerSession is an auto generated read-only Go binding around an Ethereum contract,
// with pre-set call options. // with pre-set call options.
type ENSCallerSession struct { type ResolverCallerSession struct {
Contract *ENSCaller // Generic contract caller binding to set the session for Contract *ResolverCaller // Generic contract caller binding to set the session for
CallOpts bind.CallOpts // Call options to use throughout this session CallOpts bind.CallOpts // Call options to use throughout this session
} }
// ENSTransactorSession is an auto generated write-only Go binding around an Ethereum contract, // ResolverTransactorSession is an auto generated write-only Go binding around an Ethereum contract,
// with pre-set transact options. // with pre-set transact options.
type ENSTransactorSession struct { type ResolverTransactorSession struct {
Contract *ENSTransactor // Generic contract transactor binding to set the session for Contract *ResolverTransactor // Generic contract transactor binding to set the session for
TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session
} }
// ENSRaw is an auto generated low-level Go binding around an Ethereum contract. // ResolverRaw is an auto generated low-level Go binding around an Ethereum contract.
type ENSRaw struct { type ResolverRaw struct {
Contract *ENS // Generic contract binding to access the raw methods on Contract *Resolver // Generic contract binding to access the raw methods on
} }
// ENSCallerRaw is an auto generated low-level read-only Go binding around an Ethereum contract. // ResolverCallerRaw is an auto generated low-level read-only Go binding around an Ethereum contract.
type ENSCallerRaw struct { type ResolverCallerRaw struct {
Contract *ENSCaller // Generic read-only contract binding to access the raw methods on Contract *ResolverCaller // Generic read-only contract binding to access the raw methods on
} }
// ENSTransactorRaw is an auto generated low-level write-only Go binding around an Ethereum contract. // ResolverTransactorRaw is an auto generated low-level write-only Go binding around an Ethereum contract.
type ENSTransactorRaw struct { type ResolverTransactorRaw struct {
Contract *ENSTransactor // Generic write-only contract binding to access the raw methods on Contract *ResolverTransactor // Generic write-only contract binding to access the raw methods on
} }
// NewENS creates a new instance of ENS, bound to a specific deployed contract. // NewResolver creates a new instance of Resolver, bound to a specific deployed contract.
func NewENS(address common.Address, backend bind.ContractBackend) (*ENS, error) { func NewResolver(address common.Address, backend bind.ContractBackend) (*Resolver, error) {
contract, err := bindENS(address, backend.(bind.ContractCaller), backend.(bind.ContractTransactor)) contract, err := bindResolver(address, backend.(bind.ContractCaller), backend.(bind.ContractTransactor))
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &ENS{ENSCaller: ENSCaller{contract: contract}, ENSTransactor: ENSTransactor{contract: contract}}, nil return &Resolver{ResolverCaller: ResolverCaller{contract: contract}, ResolverTransactor: ResolverTransactor{contract: contract}}, nil
} }
// NewENSCaller creates a new read-only instance of ENS, bound to a specific deployed contract. // NewResolverCaller creates a new read-only instance of Resolver, bound to a specific deployed contract.
func NewENSCaller(address common.Address, caller bind.ContractCaller) (*ENSCaller, error) { func NewResolverCaller(address common.Address, caller bind.ContractCaller) (*ResolverCaller, error) {
contract, err := bindENS(address, caller, nil) contract, err := bindResolver(address, caller, nil)
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &ENSCaller{contract: contract}, nil return &ResolverCaller{contract: contract}, nil
} }
// NewENSTransactor creates a new write-only instance of ENS, bound to a specific deployed contract. // NewResolverTransactor creates a new write-only instance of Resolver, bound to a specific deployed contract.
func NewENSTransactor(address common.Address, transactor bind.ContractTransactor) (*ENSTransactor, error) { func NewResolverTransactor(address common.Address, transactor bind.ContractTransactor) (*ResolverTransactor, error) {
contract, err := bindENS(address, nil, transactor) contract, err := bindResolver(address, nil, transactor)
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &ENSTransactor{contract: contract}, nil return &ResolverTransactor{contract: contract}, nil
} }
// bindENS binds a generic wrapper to an already deployed contract. // bindResolver binds a generic wrapper to an already deployed contract.
func bindENS(address common.Address, caller bind.ContractCaller, transactor bind.ContractTransactor) (*bind.BoundContract, error) { func bindResolver(address common.Address, caller bind.ContractCaller, transactor bind.ContractTransactor) (*bind.BoundContract, error) {
parsed, err := abi.JSON(strings.NewReader(ENSABI)) parsed, err := abi.JSON(strings.NewReader(ResolverABI))
if err != nil { if err != nil {
return nil, err return nil, err
} }
@ -124,443 +124,353 @@ func bindENS(address common.Address, caller bind.ContractCaller, transactor bind
// sets the output to result. The result type might be a single field for simple // sets the output to result. The result type might be a single field for simple
// returns, a slice of interfaces for anonymous returns and a struct for named // returns, a slice of interfaces for anonymous returns and a struct for named
// returns. // returns.
func (_ENS *ENSRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error { func (_Resolver *ResolverRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error {
return _ENS.Contract.ENSCaller.contract.Call(opts, result, method, params...) return _Resolver.Contract.ResolverCaller.contract.Call(opts, result, method, params...)
} }
// Transfer initiates a plain transaction to move funds to the contract, calling // Transfer initiates a plain transaction to move funds to the contract, calling
// its default method if one is available. // its default method if one is available.
func (_ENS *ENSRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) { func (_Resolver *ResolverRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) {
return _ENS.Contract.ENSTransactor.contract.Transfer(opts) return _Resolver.Contract.ResolverTransactor.contract.Transfer(opts)
} }
// Transact invokes the (paid) contract method with params as input values. // Transact invokes the (paid) contract method with params as input values.
func (_ENS *ENSRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) { func (_Resolver *ResolverRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) {
return _ENS.Contract.ENSTransactor.contract.Transact(opts, method, params...) return _Resolver.Contract.ResolverTransactor.contract.Transact(opts, method, params...)
} }
// Call invokes the (constant) contract method with params as input values and // Call invokes the (constant) contract method with params as input values and
// sets the output to result. The result type might be a single field for simple // sets the output to result. The result type might be a single field for simple
// returns, a slice of interfaces for anonymous returns and a struct for named // returns, a slice of interfaces for anonymous returns and a struct for named
// returns. // returns.
func (_ENS *ENSCallerRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error { func (_Resolver *ResolverCallerRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error {
return _ENS.Contract.contract.Call(opts, result, method, params...) return _Resolver.Contract.contract.Call(opts, result, method, params...)
} }
// Transfer initiates a plain transaction to move funds to the contract, calling // Transfer initiates a plain transaction to move funds to the contract, calling
// its default method if one is available. // its default method if one is available.
func (_ENS *ENSTransactorRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) { func (_Resolver *ResolverTransactorRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) {
return _ENS.Contract.contract.Transfer(opts) return _Resolver.Contract.contract.Transfer(opts)
} }
// Transact invokes the (paid) contract method with params as input values. // Transact invokes the (paid) contract method with params as input values.
func (_ENS *ENSTransactorRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) { func (_Resolver *ResolverTransactorRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) {
return _ENS.Contract.contract.Transact(opts, method, params...) return _Resolver.Contract.contract.Transact(opts, method, params...)
} }
// Owners is a free data retrieval call binding the contract method 0x3d14b257. // FindResolver is a free data retrieval call binding the contract method 0xedc0277c.
// //
// Solidity: function Owners( bytes32) constant returns(address) // Solidity: function findResolver(nodeId bytes12, label bytes32) constant returns(rcode uint16, ttl uint32, rnode bytes12, raddress address)
func (_ENS *ENSCaller) Owners(opts *bind.CallOpts, arg0 [32]byte) (common.Address, error) { func (_Resolver *ResolverCaller) FindResolver(opts *bind.CallOpts, nodeId [12]byte, label [32]byte) (struct {
Rcode uint16
Ttl uint32
Rnode [12]byte
Raddress common.Address
}, error) {
ret := new(struct {
Rcode uint16
Ttl uint32
Rnode [12]byte
Raddress common.Address
})
out := ret
err := _Resolver.contract.Call(opts, out, "findResolver", nodeId, label)
return *ret, err
}
// FindResolver is a free data retrieval call binding the contract method 0xedc0277c.
//
// Solidity: function findResolver(nodeId bytes12, label bytes32) constant returns(rcode uint16, ttl uint32, rnode bytes12, raddress address)
func (_Resolver *ResolverSession) FindResolver(nodeId [12]byte, label [32]byte) (struct {
Rcode uint16
Ttl uint32
Rnode [12]byte
Raddress common.Address
}, error) {
return _Resolver.Contract.FindResolver(&_Resolver.CallOpts, nodeId, label)
}
// FindResolver is a free data retrieval call binding the contract method 0xedc0277c.
//
// Solidity: function findResolver(nodeId bytes12, label bytes32) constant returns(rcode uint16, ttl uint32, rnode bytes12, raddress address)
func (_Resolver *ResolverCallerSession) FindResolver(nodeId [12]byte, label [32]byte) (struct {
Rcode uint16
Ttl uint32
Rnode [12]byte
Raddress common.Address
}, error) {
return _Resolver.Contract.FindResolver(&_Resolver.CallOpts, nodeId, label)
}
// GetExtended is a free data retrieval call binding the contract method 0x8021061c.
//
// Solidity: function getExtended(id bytes32) constant returns(data bytes)
func (_Resolver *ResolverCaller) GetExtended(opts *bind.CallOpts, id [32]byte) ([]byte, error) {
var (
ret0 = new([]byte)
)
out := ret0
err := _Resolver.contract.Call(opts, out, "getExtended", id)
return *ret0, err
}
// GetExtended is a free data retrieval call binding the contract method 0x8021061c.
//
// Solidity: function getExtended(id bytes32) constant returns(data bytes)
func (_Resolver *ResolverSession) GetExtended(id [32]byte) ([]byte, error) {
return _Resolver.Contract.GetExtended(&_Resolver.CallOpts, id)
}
// GetExtended is a free data retrieval call binding the contract method 0x8021061c.
//
// Solidity: function getExtended(id bytes32) constant returns(data bytes)
func (_Resolver *ResolverCallerSession) GetExtended(id [32]byte) ([]byte, error) {
return _Resolver.Contract.GetExtended(&_Resolver.CallOpts, id)
}
// GetOwner is a free data retrieval call binding the contract method 0xdeb931a2.
//
// Solidity: function getOwner(label bytes32) constant returns(address)
func (_Resolver *ResolverCaller) GetOwner(opts *bind.CallOpts, label [32]byte) (common.Address, error) {
var ( var (
ret0 = new(common.Address) ret0 = new(common.Address)
) )
out := ret0 out := ret0
err := _ENS.contract.Call(opts, out, "Owners", arg0) err := _Resolver.contract.Call(opts, out, "getOwner", label)
return *ret0, err return *ret0, err
} }
// Owners is a free data retrieval call binding the contract method 0x3d14b257. // GetOwner is a free data retrieval call binding the contract method 0xdeb931a2.
// //
// Solidity: function Owners( bytes32) constant returns(address) // Solidity: function getOwner(label bytes32) constant returns(address)
func (_ENS *ENSSession) Owners(arg0 [32]byte) (common.Address, error) { func (_Resolver *ResolverSession) GetOwner(label [32]byte) (common.Address, error) {
return _ENS.Contract.Owners(&_ENS.CallOpts, arg0) return _Resolver.Contract.GetOwner(&_Resolver.CallOpts, label)
} }
// Owners is a free data retrieval call binding the contract method 0x3d14b257. // GetOwner is a free data retrieval call binding the contract method 0xdeb931a2.
// //
// Solidity: function Owners( bytes32) constant returns(address) // Solidity: function getOwner(label bytes32) constant returns(address)
func (_ENS *ENSCallerSession) Owners(arg0 [32]byte) (common.Address, error) { func (_Resolver *ResolverCallerSession) GetOwner(label [32]byte) (common.Address, error) {
return _ENS.Contract.Owners(&_ENS.CallOpts, arg0) return _Resolver.Contract.GetOwner(&_Resolver.CallOpts, label)
} }
// Registry is a free data retrieval call binding the contract method 0xa0d03d36. // IsPersonalResolver is a free data retrieval call binding the contract method 0x3f5665e7.
// //
// Solidity: function Registry( bytes32) constant returns(bytes32) // Solidity: function isPersonalResolver() constant returns(bool)
func (_ENS *ENSCaller) Registry(opts *bind.CallOpts, arg0 [32]byte) ([32]byte, error) { func (_Resolver *ResolverCaller) IsPersonalResolver(opts *bind.CallOpts) (bool, error) {
var ( var (
ret0 = new([32]byte) ret0 = new(bool)
) )
out := ret0 out := ret0
err := _ENS.contract.Call(opts, out, "Registry", arg0) err := _Resolver.contract.Call(opts, out, "isPersonalResolver")
return *ret0, err return *ret0, err
} }
// Registry is a free data retrieval call binding the contract method 0xa0d03d36. // IsPersonalResolver is a free data retrieval call binding the contract method 0x3f5665e7.
// //
// Solidity: function Registry( bytes32) constant returns(bytes32) // Solidity: function isPersonalResolver() constant returns(bool)
func (_ENS *ENSSession) Registry(arg0 [32]byte) ([32]byte, error) { func (_Resolver *ResolverSession) IsPersonalResolver() (bool, error) {
return _ENS.Contract.Registry(&_ENS.CallOpts, arg0) return _Resolver.Contract.IsPersonalResolver(&_Resolver.CallOpts)
} }
// Registry is a free data retrieval call binding the contract method 0xa0d03d36. // IsPersonalResolver is a free data retrieval call binding the contract method 0x3f5665e7.
// //
// Solidity: function Registry( bytes32) constant returns(bytes32) // Solidity: function isPersonalResolver() constant returns(bool)
func (_ENS *ENSCallerSession) Registry(arg0 [32]byte) ([32]byte, error) { func (_Resolver *ResolverCallerSession) IsPersonalResolver() (bool, error) {
return _ENS.Contract.Registry(&_ENS.CallOpts, arg0) return _Resolver.Contract.IsPersonalResolver(&_Resolver.CallOpts)
} }
// Set is a paid mutator transaction binding the contract method 0xbe36e676. // Resolve is a free data retrieval call binding the contract method 0xa16fdafa.
// //
// Solidity: function Set(host bytes32, content bytes32) returns() // Solidity: function resolve(nodeId bytes12, qtype bytes32, index uint16) constant returns(rcode uint16, rtype bytes16, ttl uint32, len uint16, data bytes32)
func (_ENS *ENSTransactor) Set(opts *bind.TransactOpts, host [32]byte, content [32]byte) (*types.Transaction, error) { func (_Resolver *ResolverCaller) Resolve(opts *bind.CallOpts, nodeId [12]byte, qtype [32]byte, index uint16) (struct {
return _ENS.contract.Transact(opts, "Set", host, content) Rcode uint16
Rtype [16]byte
Ttl uint32
Len uint16
Data [32]byte
}, error) {
ret := new(struct {
Rcode uint16
Rtype [16]byte
Ttl uint32
Len uint16
Data [32]byte
})
out := ret
err := _Resolver.contract.Call(opts, out, "resolve", nodeId, qtype, index)
return *ret, err
} }
// Set is a paid mutator transaction binding the contract method 0xbe36e676. // Resolve is a free data retrieval call binding the contract method 0xa16fdafa.
// //
// Solidity: function Set(host bytes32, content bytes32) returns() // Solidity: function resolve(nodeId bytes12, qtype bytes32, index uint16) constant returns(rcode uint16, rtype bytes16, ttl uint32, len uint16, data bytes32)
func (_ENS *ENSSession) Set(host [32]byte, content [32]byte) (*types.Transaction, error) { func (_Resolver *ResolverSession) Resolve(nodeId [12]byte, qtype [32]byte, index uint16) (struct {
return _ENS.Contract.Set(&_ENS.TransactOpts, host, content) Rcode uint16
Rtype [16]byte
Ttl uint32
Len uint16
Data [32]byte
}, error) {
return _Resolver.Contract.Resolve(&_Resolver.CallOpts, nodeId, qtype, index)
} }
// Set is a paid mutator transaction binding the contract method 0xbe36e676. // Resolve is a free data retrieval call binding the contract method 0xa16fdafa.
// //
// Solidity: function Set(host bytes32, content bytes32) returns() // Solidity: function resolve(nodeId bytes12, qtype bytes32, index uint16) constant returns(rcode uint16, rtype bytes16, ttl uint32, len uint16, data bytes32)
func (_ENS *ENSTransactorSession) Set(host [32]byte, content [32]byte) (*types.Transaction, error) { func (_Resolver *ResolverCallerSession) Resolve(nodeId [12]byte, qtype [32]byte, index uint16) (struct {
return _ENS.Contract.Set(&_ENS.TransactOpts, host, content) Rcode uint16
Rtype [16]byte
Ttl uint32
Len uint16
Data [32]byte
}, error) {
return _Resolver.Contract.Resolve(&_Resolver.CallOpts, nodeId, qtype, index)
} }
// Kill is a paid mutator transaction binding the contract method 0x41c0e1b5. // DeletePrivateRR is a paid mutator transaction binding the contract method 0x1b370194.
// //
// Solidity: function kill() returns() // Solidity: function deletePrivateRR(rootNodeId bytes12, name bytes32[]) returns()
func (_ENS *ENSTransactor) Kill(opts *bind.TransactOpts) (*types.Transaction, error) { func (_Resolver *ResolverTransactor) DeletePrivateRR(opts *bind.TransactOpts, rootNodeId [12]byte, name [][32]byte) (*types.Transaction, error) {
return _ENS.contract.Transact(opts, "kill") return _Resolver.contract.Transact(opts, "deletePrivateRR", rootNodeId, name)
} }
// Kill is a paid mutator transaction binding the contract method 0x41c0e1b5. // DeletePrivateRR is a paid mutator transaction binding the contract method 0x1b370194.
// //
// Solidity: function kill() returns() // Solidity: function deletePrivateRR(rootNodeId bytes12, name bytes32[]) returns()
func (_ENS *ENSSession) Kill() (*types.Transaction, error) { func (_Resolver *ResolverSession) DeletePrivateRR(rootNodeId [12]byte, name [][32]byte) (*types.Transaction, error) {
return _ENS.Contract.Kill(&_ENS.TransactOpts) return _Resolver.Contract.DeletePrivateRR(&_Resolver.TransactOpts, rootNodeId, name)
} }
// Kill is a paid mutator transaction binding the contract method 0x41c0e1b5. // DeletePrivateRR is a paid mutator transaction binding the contract method 0x1b370194.
// //
// Solidity: function kill() returns() // Solidity: function deletePrivateRR(rootNodeId bytes12, name bytes32[]) returns()
func (_ENS *ENSTransactorSession) Kill() (*types.Transaction, error) { func (_Resolver *ResolverTransactorSession) DeletePrivateRR(rootNodeId [12]byte, name [][32]byte) (*types.Transaction, error) {
return _ENS.Contract.Kill(&_ENS.TransactOpts) return _Resolver.Contract.DeletePrivateRR(&_Resolver.TransactOpts, rootNodeId, name)
} }
// MortalABI is the input ABI used to generate the binding from. // DeleteRR is a paid mutator transaction binding the contract method 0xbc06183d.
const MortalABI = `[{"constant":false,"inputs":[],"name":"kill","outputs":[],"type":"function"}]`
// MortalBin is the compiled bytecode used for deploying new contracts.
const MortalBin = `0x606060405260008054600160a060020a03191633179055605c8060226000396000f3606060405260e060020a600035046341c0e1b58114601a575b005b60186000543373ffffffffffffffffffffffffffffffffffffffff90811691161415605a5760005473ffffffffffffffffffffffffffffffffffffffff16ff5b56`
// DeployMortal deploys a new Ethereum contract, binding an instance of Mortal to it.
func DeployMortal(auth *bind.TransactOpts, backend bind.ContractBackend) (common.Address, *types.Transaction, *Mortal, error) {
parsed, err := abi.JSON(strings.NewReader(MortalABI))
if err != nil {
return common.Address{}, nil, nil, err
}
address, tx, contract, err := bind.DeployContract(auth, parsed, common.FromHex(MortalBin), backend)
if err != nil {
return common.Address{}, nil, nil, err
}
return address, tx, &Mortal{MortalCaller: MortalCaller{contract: contract}, MortalTransactor: MortalTransactor{contract: contract}}, nil
}
// Mortal is an auto generated Go binding around an Ethereum contract.
type Mortal struct {
MortalCaller // Read-only binding to the contract
MortalTransactor // Write-only binding to the contract
}
// MortalCaller is an auto generated read-only Go binding around an Ethereum contract.
type MortalCaller struct {
contract *bind.BoundContract // Generic contract wrapper for the low level calls
}
// MortalTransactor is an auto generated write-only Go binding around an Ethereum contract.
type MortalTransactor struct {
contract *bind.BoundContract // Generic contract wrapper for the low level calls
}
// MortalSession is an auto generated Go binding around an Ethereum contract,
// with pre-set call and transact options.
type MortalSession struct {
Contract *Mortal // Generic contract binding to set the session for
CallOpts bind.CallOpts // Call options to use throughout this session
TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session
}
// MortalCallerSession is an auto generated read-only Go binding around an Ethereum contract,
// with pre-set call options.
type MortalCallerSession struct {
Contract *MortalCaller // Generic contract caller binding to set the session for
CallOpts bind.CallOpts // Call options to use throughout this session
}
// MortalTransactorSession is an auto generated write-only Go binding around an Ethereum contract,
// with pre-set transact options.
type MortalTransactorSession struct {
Contract *MortalTransactor // Generic contract transactor binding to set the session for
TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session
}
// MortalRaw is an auto generated low-level Go binding around an Ethereum contract.
type MortalRaw struct {
Contract *Mortal // Generic contract binding to access the raw methods on
}
// MortalCallerRaw is an auto generated low-level read-only Go binding around an Ethereum contract.
type MortalCallerRaw struct {
Contract *MortalCaller // Generic read-only contract binding to access the raw methods on
}
// MortalTransactorRaw is an auto generated low-level write-only Go binding around an Ethereum contract.
type MortalTransactorRaw struct {
Contract *MortalTransactor // Generic write-only contract binding to access the raw methods on
}
// NewMortal creates a new instance of Mortal, bound to a specific deployed contract.
func NewMortal(address common.Address, backend bind.ContractBackend) (*Mortal, error) {
contract, err := bindMortal(address, backend.(bind.ContractCaller), backend.(bind.ContractTransactor))
if err != nil {
return nil, err
}
return &Mortal{MortalCaller: MortalCaller{contract: contract}, MortalTransactor: MortalTransactor{contract: contract}}, nil
}
// NewMortalCaller creates a new read-only instance of Mortal, bound to a specific deployed contract.
func NewMortalCaller(address common.Address, caller bind.ContractCaller) (*MortalCaller, error) {
contract, err := bindMortal(address, caller, nil)
if err != nil {
return nil, err
}
return &MortalCaller{contract: contract}, nil
}
// NewMortalTransactor creates a new write-only instance of Mortal, bound to a specific deployed contract.
func NewMortalTransactor(address common.Address, transactor bind.ContractTransactor) (*MortalTransactor, error) {
contract, err := bindMortal(address, nil, transactor)
if err != nil {
return nil, err
}
return &MortalTransactor{contract: contract}, nil
}
// bindMortal binds a generic wrapper to an already deployed contract.
func bindMortal(address common.Address, caller bind.ContractCaller, transactor bind.ContractTransactor) (*bind.BoundContract, error) {
parsed, err := abi.JSON(strings.NewReader(MortalABI))
if err != nil {
return nil, err
}
return bind.NewBoundContract(address, parsed, caller, transactor), nil
}
// Call invokes the (constant) contract method with params as input values and
// sets the output to result. The result type might be a single field for simple
// returns, a slice of interfaces for anonymous returns and a struct for named
// returns.
func (_Mortal *MortalRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error {
return _Mortal.Contract.MortalCaller.contract.Call(opts, result, method, params...)
}
// Transfer initiates a plain transaction to move funds to the contract, calling
// its default method if one is available.
func (_Mortal *MortalRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) {
return _Mortal.Contract.MortalTransactor.contract.Transfer(opts)
}
// Transact invokes the (paid) contract method with params as input values.
func (_Mortal *MortalRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) {
return _Mortal.Contract.MortalTransactor.contract.Transact(opts, method, params...)
}
// Call invokes the (constant) contract method with params as input values and
// sets the output to result. The result type might be a single field for simple
// returns, a slice of interfaces for anonymous returns and a struct for named
// returns.
func (_Mortal *MortalCallerRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error {
return _Mortal.Contract.contract.Call(opts, result, method, params...)
}
// Transfer initiates a plain transaction to move funds to the contract, calling
// its default method if one is available.
func (_Mortal *MortalTransactorRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) {
return _Mortal.Contract.contract.Transfer(opts)
}
// Transact invokes the (paid) contract method with params as input values.
func (_Mortal *MortalTransactorRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) {
return _Mortal.Contract.contract.Transact(opts, method, params...)
}
// Kill is a paid mutator transaction binding the contract method 0x41c0e1b5.
// //
// Solidity: function kill() returns() // Solidity: function deleteRR(rootNodeId bytes12, name string) returns()
func (_Mortal *MortalTransactor) Kill(opts *bind.TransactOpts) (*types.Transaction, error) { func (_Resolver *ResolverTransactor) DeleteRR(opts *bind.TransactOpts, rootNodeId [12]byte, name string) (*types.Transaction, error) {
return _Mortal.contract.Transact(opts, "kill") return _Resolver.contract.Transact(opts, "deleteRR", rootNodeId, name)
} }
// Kill is a paid mutator transaction binding the contract method 0x41c0e1b5. // DeleteRR is a paid mutator transaction binding the contract method 0xbc06183d.
// //
// Solidity: function kill() returns() // Solidity: function deleteRR(rootNodeId bytes12, name string) returns()
func (_Mortal *MortalSession) Kill() (*types.Transaction, error) { func (_Resolver *ResolverSession) DeleteRR(rootNodeId [12]byte, name string) (*types.Transaction, error) {
return _Mortal.Contract.Kill(&_Mortal.TransactOpts) return _Resolver.Contract.DeleteRR(&_Resolver.TransactOpts, rootNodeId, name)
} }
// Kill is a paid mutator transaction binding the contract method 0x41c0e1b5. // DeleteRR is a paid mutator transaction binding the contract method 0xbc06183d.
// //
// Solidity: function kill() returns() // Solidity: function deleteRR(rootNodeId bytes12, name string) returns()
func (_Mortal *MortalTransactorSession) Kill() (*types.Transaction, error) { func (_Resolver *ResolverTransactorSession) DeleteRR(rootNodeId [12]byte, name string) (*types.Transaction, error) {
return _Mortal.Contract.Kill(&_Mortal.TransactOpts) return _Resolver.Contract.DeleteRR(&_Resolver.TransactOpts, rootNodeId, name)
} }
// OwnedABI is the input ABI used to generate the binding from. // Register is a paid mutator transaction binding the contract method 0xa1f8f8f0.
const OwnedABI = `[{"inputs":[],"type":"constructor"}]` //
// Solidity: function register(label bytes32, resolver address, nodeId bytes12) returns()
// OwnedBin is the compiled bytecode used for deploying new contracts. func (_Resolver *ResolverTransactor) Register(opts *bind.TransactOpts, label [32]byte, resolver common.Address, nodeId [12]byte) (*types.Transaction, error) {
const OwnedBin = `0x606060405260008054600160a060020a0319163317905560068060226000396000f3606060405200` return _Resolver.contract.Transact(opts, "register", label, resolver, nodeId)
// DeployOwned deploys a new Ethereum contract, binding an instance of Owned to it.
func DeployOwned(auth *bind.TransactOpts, backend bind.ContractBackend) (common.Address, *types.Transaction, *Owned, error) {
parsed, err := abi.JSON(strings.NewReader(OwnedABI))
if err != nil {
return common.Address{}, nil, nil, err
}
address, tx, contract, err := bind.DeployContract(auth, parsed, common.FromHex(OwnedBin), backend)
if err != nil {
return common.Address{}, nil, nil, err
}
return address, tx, &Owned{OwnedCaller: OwnedCaller{contract: contract}, OwnedTransactor: OwnedTransactor{contract: contract}}, nil
} }
// Owned is an auto generated Go binding around an Ethereum contract. // Register is a paid mutator transaction binding the contract method 0xa1f8f8f0.
type Owned struct { //
OwnedCaller // Read-only binding to the contract // Solidity: function register(label bytes32, resolver address, nodeId bytes12) returns()
OwnedTransactor // Write-only binding to the contract func (_Resolver *ResolverSession) Register(label [32]byte, resolver common.Address, nodeId [12]byte) (*types.Transaction, error) {
return _Resolver.Contract.Register(&_Resolver.TransactOpts, label, resolver, nodeId)
} }
// OwnedCaller is an auto generated read-only Go binding around an Ethereum contract. // Register is a paid mutator transaction binding the contract method 0xa1f8f8f0.
type OwnedCaller struct { //
contract *bind.BoundContract // Generic contract wrapper for the low level calls // Solidity: function register(label bytes32, resolver address, nodeId bytes12) returns()
func (_Resolver *ResolverTransactorSession) Register(label [32]byte, resolver common.Address, nodeId [12]byte) (*types.Transaction, error) {
return _Resolver.Contract.Register(&_Resolver.TransactOpts, label, resolver, nodeId)
} }
// OwnedTransactor is an auto generated write-only Go binding around an Ethereum contract. // SetOwner is a paid mutator transaction binding the contract method 0x5b0fc9c3.
type OwnedTransactor struct { //
contract *bind.BoundContract // Generic contract wrapper for the low level calls // Solidity: function setOwner(label bytes32, newOwner address) returns()
func (_Resolver *ResolverTransactor) SetOwner(opts *bind.TransactOpts, label [32]byte, newOwner common.Address) (*types.Transaction, error) {
return _Resolver.contract.Transact(opts, "setOwner", label, newOwner)
} }
// OwnedSession is an auto generated Go binding around an Ethereum contract, // SetOwner is a paid mutator transaction binding the contract method 0x5b0fc9c3.
// with pre-set call and transact options. //
type OwnedSession struct { // Solidity: function setOwner(label bytes32, newOwner address) returns()
Contract *Owned // Generic contract binding to set the session for func (_Resolver *ResolverSession) SetOwner(label [32]byte, newOwner common.Address) (*types.Transaction, error) {
CallOpts bind.CallOpts // Call options to use throughout this session return _Resolver.Contract.SetOwner(&_Resolver.TransactOpts, label, newOwner)
TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session
} }
// OwnedCallerSession is an auto generated read-only Go binding around an Ethereum contract, // SetOwner is a paid mutator transaction binding the contract method 0x5b0fc9c3.
// with pre-set call options. //
type OwnedCallerSession struct { // Solidity: function setOwner(label bytes32, newOwner address) returns()
Contract *OwnedCaller // Generic contract caller binding to set the session for func (_Resolver *ResolverTransactorSession) SetOwner(label [32]byte, newOwner common.Address) (*types.Transaction, error) {
CallOpts bind.CallOpts // Call options to use throughout this session return _Resolver.Contract.SetOwner(&_Resolver.TransactOpts, label, newOwner)
} }
// OwnedTransactorSession is an auto generated write-only Go binding around an Ethereum contract, // SetPrivateRR is a paid mutator transaction binding the contract method 0x91c8e7b9.
// with pre-set transact options. //
type OwnedTransactorSession struct { // Solidity: function setPrivateRR(rootNodeId bytes12, name bytes32[], rtype bytes16, ttl uint32, len uint16, data bytes32) returns()
Contract *OwnedTransactor // Generic contract transactor binding to set the session for func (_Resolver *ResolverTransactor) SetPrivateRR(opts *bind.TransactOpts, rootNodeId [12]byte, name [][32]byte, rtype [16]byte, ttl uint32, len uint16, data [32]byte) (*types.Transaction, error) {
TransactOpts bind.TransactOpts // Transaction auth options to use throughout this session return _Resolver.contract.Transact(opts, "setPrivateRR", rootNodeId, name, rtype, ttl, len, data)
} }
// OwnedRaw is an auto generated low-level Go binding around an Ethereum contract. // SetPrivateRR is a paid mutator transaction binding the contract method 0x91c8e7b9.
type OwnedRaw struct { //
Contract *Owned // Generic contract binding to access the raw methods on // Solidity: function setPrivateRR(rootNodeId bytes12, name bytes32[], rtype bytes16, ttl uint32, len uint16, data bytes32) returns()
func (_Resolver *ResolverSession) SetPrivateRR(rootNodeId [12]byte, name [][32]byte, rtype [16]byte, ttl uint32, len uint16, data [32]byte) (*types.Transaction, error) {
return _Resolver.Contract.SetPrivateRR(&_Resolver.TransactOpts, rootNodeId, name, rtype, ttl, len, data)
} }
// OwnedCallerRaw is an auto generated low-level read-only Go binding around an Ethereum contract. // SetPrivateRR is a paid mutator transaction binding the contract method 0x91c8e7b9.
type OwnedCallerRaw struct { //
Contract *OwnedCaller // Generic read-only contract binding to access the raw methods on // Solidity: function setPrivateRR(rootNodeId bytes12, name bytes32[], rtype bytes16, ttl uint32, len uint16, data bytes32) returns()
func (_Resolver *ResolverTransactorSession) SetPrivateRR(rootNodeId [12]byte, name [][32]byte, rtype [16]byte, ttl uint32, len uint16, data [32]byte) (*types.Transaction, error) {
return _Resolver.Contract.SetPrivateRR(&_Resolver.TransactOpts, rootNodeId, name, rtype, ttl, len, data)
} }
// OwnedTransactorRaw is an auto generated low-level write-only Go binding around an Ethereum contract. // SetRR is a paid mutator transaction binding the contract method 0x8bba944d.
type OwnedTransactorRaw struct { //
Contract *OwnedTransactor // Generic write-only contract binding to access the raw methods on // Solidity: function setRR(rootNodeId bytes12, name string, rtype bytes16, ttl uint32, len uint16, data bytes32) returns()
func (_Resolver *ResolverTransactor) SetRR(opts *bind.TransactOpts, rootNodeId [12]byte, name string, rtype [16]byte, ttl uint32, len uint16, data [32]byte) (*types.Transaction, error) {
return _Resolver.contract.Transact(opts, "setRR", rootNodeId, name, rtype, ttl, len, data)
} }
// NewOwned creates a new instance of Owned, bound to a specific deployed contract. // SetRR is a paid mutator transaction binding the contract method 0x8bba944d.
func NewOwned(address common.Address, backend bind.ContractBackend) (*Owned, error) { //
contract, err := bindOwned(address, backend.(bind.ContractCaller), backend.(bind.ContractTransactor)) // Solidity: function setRR(rootNodeId bytes12, name string, rtype bytes16, ttl uint32, len uint16, data bytes32) returns()
if err != nil { func (_Resolver *ResolverSession) SetRR(rootNodeId [12]byte, name string, rtype [16]byte, ttl uint32, len uint16, data [32]byte) (*types.Transaction, error) {
return nil, err return _Resolver.Contract.SetRR(&_Resolver.TransactOpts, rootNodeId, name, rtype, ttl, len, data)
}
return &Owned{OwnedCaller: OwnedCaller{contract: contract}, OwnedTransactor: OwnedTransactor{contract: contract}}, nil
} }
// NewOwnedCaller creates a new read-only instance of Owned, bound to a specific deployed contract. // SetRR is a paid mutator transaction binding the contract method 0x8bba944d.
func NewOwnedCaller(address common.Address, caller bind.ContractCaller) (*OwnedCaller, error) { //
contract, err := bindOwned(address, caller, nil) // Solidity: function setRR(rootNodeId bytes12, name string, rtype bytes16, ttl uint32, len uint16, data bytes32) returns()
if err != nil { func (_Resolver *ResolverTransactorSession) SetRR(rootNodeId [12]byte, name string, rtype [16]byte, ttl uint32, len uint16, data [32]byte) (*types.Transaction, error) {
return nil, err return _Resolver.Contract.SetRR(&_Resolver.TransactOpts, rootNodeId, name, rtype, ttl, len, data)
}
return &OwnedCaller{contract: contract}, nil
} }
// NewOwnedTransactor creates a new write-only instance of Owned, bound to a specific deployed contract. // SetResolver is a paid mutator transaction binding the contract method 0xa9f2a1b2.
func NewOwnedTransactor(address common.Address, transactor bind.ContractTransactor) (*OwnedTransactor, error) { //
contract, err := bindOwned(address, nil, transactor) // Solidity: function setResolver(label bytes32, resolver address, nodeId bytes12) returns()
if err != nil { func (_Resolver *ResolverTransactor) SetResolver(opts *bind.TransactOpts, label [32]byte, resolver common.Address, nodeId [12]byte) (*types.Transaction, error) {
return nil, err return _Resolver.contract.Transact(opts, "setResolver", label, resolver, nodeId)
}
return &OwnedTransactor{contract: contract}, nil
} }
// bindOwned binds a generic wrapper to an already deployed contract. // SetResolver is a paid mutator transaction binding the contract method 0xa9f2a1b2.
func bindOwned(address common.Address, caller bind.ContractCaller, transactor bind.ContractTransactor) (*bind.BoundContract, error) { //
parsed, err := abi.JSON(strings.NewReader(OwnedABI)) // Solidity: function setResolver(label bytes32, resolver address, nodeId bytes12) returns()
if err != nil { func (_Resolver *ResolverSession) SetResolver(label [32]byte, resolver common.Address, nodeId [12]byte) (*types.Transaction, error) {
return nil, err return _Resolver.Contract.SetResolver(&_Resolver.TransactOpts, label, resolver, nodeId)
}
return bind.NewBoundContract(address, parsed, caller, transactor), nil
} }
// Call invokes the (constant) contract method with params as input values and // SetResolver is a paid mutator transaction binding the contract method 0xa9f2a1b2.
// sets the output to result. The result type might be a single field for simple //
// returns, a slice of interfaces for anonymous returns and a struct for named // Solidity: function setResolver(label bytes32, resolver address, nodeId bytes12) returns()
// returns. func (_Resolver *ResolverTransactorSession) SetResolver(label [32]byte, resolver common.Address, nodeId [12]byte) (*types.Transaction, error) {
func (_Owned *OwnedRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error { return _Resolver.Contract.SetResolver(&_Resolver.TransactOpts, label, resolver, nodeId)
return _Owned.Contract.OwnedCaller.contract.Call(opts, result, method, params...)
}
// Transfer initiates a plain transaction to move funds to the contract, calling
// its default method if one is available.
func (_Owned *OwnedRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) {
return _Owned.Contract.OwnedTransactor.contract.Transfer(opts)
}
// Transact invokes the (paid) contract method with params as input values.
func (_Owned *OwnedRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) {
return _Owned.Contract.OwnedTransactor.contract.Transact(opts, method, params...)
}
// Call invokes the (constant) contract method with params as input values and
// sets the output to result. The result type might be a single field for simple
// returns, a slice of interfaces for anonymous returns and a struct for named
// returns.
func (_Owned *OwnedCallerRaw) Call(opts *bind.CallOpts, result interface{}, method string, params ...interface{}) error {
return _Owned.Contract.contract.Call(opts, result, method, params...)
}
// Transfer initiates a plain transaction to move funds to the contract, calling
// its default method if one is available.
func (_Owned *OwnedTransactorRaw) Transfer(opts *bind.TransactOpts) (*types.Transaction, error) {
return _Owned.Contract.contract.Transfer(opts)
}
// Transact invokes the (paid) contract method with params as input values.
func (_Owned *OwnedTransactorRaw) Transact(opts *bind.TransactOpts, method string, params ...interface{}) (*types.Transaction, error) {
return _Owned.Contract.contract.Transact(opts, method, params...)
} }

42
swarm/services/ens/contract/ens.sol
View file

@ -1,19 +1,31 @@
import "mortal"; /**
/// @title Swarm Distributed Preimage Archive * ENS resolver interface.
/// @author Viktor Tron <viktor@ethdev.com> */
contract ENS is mortal contract Resolver {
{ bytes32 constant TYPE_STAR = "*";
mapping (bytes32 => bytes32) public Registry; // Response codes.
mapping (bytes32 => address) public Owners; uint16 constant RCODE_OK = 0;
uint16 constant RCODE_NXDOMAIN = 3;
function Set(bytes32 host, bytes32 content) { // These methods are shared by all resolvers
if (Owners[host] == 0x0) { function findResolver(bytes12 nodeId, bytes32 label) constant
Owners[host] = tx.origin; returns (uint16 rcode, uint32 ttl, bytes12 rnode, address raddress);
} function resolve(bytes12 nodeId, bytes32 qtype, uint16 index) constant
if (Owners[host] == tx.origin) { returns (uint16 rcode, bytes16 rtype, uint32 ttl, uint16 len,
Registry[host] = content; bytes32 data);
} function getExtended(bytes32 id) constant returns (bytes data);
}
// These methods are implemented by personal resolvers
function isPersonalResolver() constant returns (bool);
function setRR(bytes12 rootNodeId, string name, bytes16 rtype, uint32 ttl, uint16 len, bytes32 data);
function setPrivateRR(bytes12 rootNodeId, bytes32[] name, bytes16 rtype, uint32 ttl, uint16 len, bytes32 data);
function deleteRR(bytes12 rootNodeId, string name);
function deletePrivateRR(bytes12 rootNodeId, bytes32[] name);
// These methods are implemented by open registrar implementations.
function register(bytes32 label, address resolver, bytes12 nodeId);
function setOwner(bytes32 label, address newOwner);
function setResolver(bytes32 label, address resolver, bytes12 nodeId);
function getOwner(bytes32 label) constant returns (address);
} }

191
swarm/services/ens/ens.go
View file

@ -3,82 +3,183 @@ package ens
import ( import (
"fmt" "fmt"
"math/big"
"regexp" "regexp"
"strings"
"github.com/ethereum/go-ethereum/accounts/abi/bind" "github.com/ethereum/go-ethereum/accounts/abi/bind"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/crypto" "github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/logger"
"github.com/ethereum/go-ethereum/logger/glog"
"github.com/ethereum/go-ethereum/swarm/services/ens/contract" "github.com/ethereum/go-ethereum/swarm/services/ens/contract"
"github.com/ethereum/go-ethereum/swarm/storage" "github.com/ethereum/go-ethereum/swarm/storage"
) )
var domainAndVersion = regexp.MustCompile("[@:;,]+") var domainAndVersion = regexp.MustCompile("[@:;,]+")
var qtypeChash = [32]byte{ 0x43, 0x48, 0x41, 0x53, 0x48}
var rtypeChash = [16]byte{ 0x43, 0x48, 0x41, 0x53, 0x48}
// swarm domain name registry and resolver // swarm domain name registry and resolver
// the ENS instance can be directly wrapped in rpc.Api // the ENS instance can be directly wrapped in rpc.Api
type ENS struct { type ENS struct {
*contract.ENSSession transactOpts *bind.TransactOpts;
contractBackend bind.ContractBackend;
rootAddress common.Address;
} }
// NewENS creates a proxy instance wrapping the abigen interface to the ENS contract
// using the transaction options passed as first argument, it sets up a session
func NewENS(transactOpts *bind.TransactOpts, contractAddr common.Address, contractBackend bind.ContractBackend) *ENS { func NewENS(transactOpts *bind.TransactOpts, contractAddr common.Address, contractBackend bind.ContractBackend) *ENS {
ens, err := contract.NewENS(contractAddr, contractBackend)
if err != nil {
glog.V(logger.Debug).Infof("error setting up name server on %v, skipping: %v", contractAddr.Hex(), err)
}
return &ENS{ return &ENS{
&contract.ENSSession{ transactOpts: transactOpts,
Contract: ens, contractBackend: contractBackend,
TransactOpts: *transactOpts, rootAddress: contractAddr,
},
} }
} }
// Register(name, hash ) func (self *ENS) newResolver(contractAddr common.Address) (*contract.ResolverSession, error) {
//involves sending a transaction (sent by sender specified as From of Transact) resolver, err := contract.NewResolver(contractAddr, self.contractBackend)
func (self *ENS) Register(name string, hash common.Hash) (*types.Transaction, error) {
namehash := crypto.Sha3Hash([]byte(name))
owner, err := self.Owners(namehash)
if err != nil { if err != nil {
return nil, fmt.Errorf("error registering '%s': %v", name, err) return nil, err
} }
if (owner != common.Address{} && owner != self.TransactOpts.From) { return &contract.ResolverSession{
return nil, fmt.Errorf("error registering '%s': already set as %", name) Contract: resolver,
} TransactOpts: *self.transactOpts,
glog.V(logger.Debug).Infof("[ENS]: host '%s' (hash: '%v') to be registered as '%v'", name, namehash.Hex(), hash.Hex()) }, nil
return self.Set(namehash, hash)
}
func (self *ENS) WhoseIs(name string) (common.Address, error) {
namehash := crypto.Sha3Hash([]byte(name))
return self.Owners(namehash)
} }
// resolve is a non-tranasctional call, returns hash as storage.Key // resolve is a non-tranasctional call, returns hash as storage.Key
func (self *ENS) Resolve(hostPort string) (storage.Key, error) { func (self *ENS) Resolve(hostPort string) (storage.Key, error) {
host := hostPort host := hostPort
var version *big.Int
parts := domainAndVersion.Split(host, 3) parts := domainAndVersion.Split(host, 3)
if len(parts) > 1 && parts[1] != "" { if len(parts) > 1 && parts[1] != "" {
host = parts[0] host = parts[0]
version = common.Big(parts[1])
} }
hash := crypto.Sha3Hash([]byte(host)) return self.resolveName(self.rootAddress, host)
_ = version }
// hash, err = self.registrar.Resolver(version).HashToHash(hostHash)
hash, err := self.Registry(hash) func (self *ENS) nextResolver(resolver *contract.ResolverSession, nodeId [12]byte, label string) (*contract.ResolverSession, [12]byte, error) {
if err != nil { hash := crypto.Sha3Hash([]byte(label))
return nil, fmt.Errorf("error resolving '%v': %v", hash.Hex(), err) ret, err := resolver.FindResolver(nodeId, hash)
} if err != nil {
if (hash == common.Hash{}) { err = fmt.Errorf("error resolving label '%v': %v", label, err)
return nil, fmt.Errorf("unable to resolve '%v': not found", hash) return nil, [12]byte{}, err
} }
contentHash := storage.Key(hash.Bytes()) if ret.Rcode != 0 {
glog.V(logger.Debug).Infof("[ENS] resolve host '%v' to contentHash: '%v'", hash, contentHash) err = fmt.Errorf("error resolving label '%v': got response code %v", label, ret.Rcode)
return contentHash, nil return nil, [12]byte{}, err
}
nodeId = ret.Rnode;
resolver, err = self.newResolver(ret.Raddress)
if err != nil {
return nil, [12]byte{}, err
}
return resolver, nodeId, nil
}
func (self *ENS) findResolver(rootAddress common.Address, host string) (*contract.ResolverSession, [12]byte, error) {
resolver, err := self.newResolver(self.rootAddress)
if err != nil {
return nil, [12]byte{}, err
}
if len(host) == 0 {
return resolver, [12]byte{}, nil
}
labels := strings.Split(host, ".")
var nodeId [12]byte
for i := len(labels) - 1; i >= 0; i-- {
var err error
resolver, nodeId, err = self.nextResolver(resolver, nodeId, labels[i])
if err != nil {
return nil, [12]byte{}, err
}
}
return resolver, nodeId, nil
}
func (self *ENS) resolveName(rootAddress common.Address, host string) (storage.Key, error) {
resolver, nodeId, err := self.findResolver(rootAddress, host)
if err != nil {
return nil, err
}
ret, err := resolver.Resolve(nodeId, qtypeChash, 0)
if err != nil {
return nil, fmt.Errorf("error looking up RR on '%v': %v", host, err)
}
if ret.Rcode != 0 {
return nil, fmt.Errorf("error looking up RR on '%v': got response code %v", host, ret.Rcode)
}
return storage.Key(ret.Data[:]), nil
}
/**
* Registers a new domain name for the caller, making them the owner of the new name.
*/
func (self *ENS) Register(name string, resolverAddress common.Address) (*types.Transaction, error) {
// Find the resolver that we should register with (the one that controls the parent domain)
parts := strings.SplitN(name, ".", 2)
baseName := ""
if len(parts) > 1 {
baseName = parts[1]
}
resolver, nodeId, err := self.findResolver(self.rootAddress, baseName)
if err != nil {
return nil, err
}
if nodeId != [12]byte{} {
return nil, fmt.Errorf("cannot register domains on %v: not a root node", baseName)
}
// Send it a register transaction
hash := crypto.Sha3Hash([]byte(parts[0]))
return resolver.Register(hash, resolverAddress, [12]byte{})
}
/**
* Steps through name components until it finds a PersonalResolver contract.
* Returns the resolver, the node ID, and the remaining name components.
*/
func (self *ENS) findPersonalResolver(name string) (*contract.ResolverSession, [12]byte, string, error) {
var nodeId [12]byte
resolver, err := self.newResolver(self.rootAddress)
if err != nil {
return nil, [12]byte{}, "", err
}
labels := strings.Split(name, ".")
for i := len(labels) - 1; i >= 0; i-- {
if personal, _ := resolver.IsPersonalResolver(); personal {
return resolver, nodeId, strings.Join(labels[0:i + 1], "."), nil
}
resolver, nodeId, err = self.nextResolver(resolver, nodeId, labels[i])
if err != nil {
return nil, [12]byte{}, "", err
}
}
if personal, _ := resolver.IsPersonalResolver(); !personal {
return nil, [12]byte{}, "", fmt.Errorf("Personal resolver not found in any name component")
} else {
return resolver, nodeId, "", nil
}
}
/**
* Sets the content hash associated with a name.
*/
func (self *ENS) SetContentHash(name string, hash common.Hash) (*types.Transaction, error) {
resolver, nodeId, name, err := self.findPersonalResolver(name)
if err != nil {
return nil, err
}
return resolver.SetRR(nodeId, name, rtypeChash, 3600, 20, [32]byte(hash))
} }

23
swarm/services/ens/ens_test.go
View file

@ -10,9 +10,15 @@ import (
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core" "github.com/ethereum/go-ethereum/core"
"github.com/ethereum/go-ethereum/crypto" "github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/logger/glog"
"github.com/ethereum/go-ethereum/swarm/services/ens/contract" "github.com/ethereum/go-ethereum/swarm/services/ens/contract"
) )
func init() {
glog.SetV(6)
glog.SetToStderr(true)
}
var ( var (
key, _ = crypto.HexToECDSA("b71c71a67e1177ad4e901695e1b4b9ee17ae16c6668d313eac2f96dbcda3f291") key, _ = crypto.HexToECDSA("b71c71a67e1177ad4e901695e1b4b9ee17ae16c6668d313eac2f96dbcda3f291")
name = "my name on ENS" name = "my name on ENS"
@ -23,7 +29,7 @@ var (
func deploy(prvKey *ecdsa.PrivateKey, amount *big.Int, backend *backends.SimulatedBackend) (common.Address, error) { func deploy(prvKey *ecdsa.PrivateKey, amount *big.Int, backend *backends.SimulatedBackend) (common.Address, error) {
deployTransactor := bind.NewKeyedTransactor(prvKey) deployTransactor := bind.NewKeyedTransactor(prvKey)
deployTransactor.Value = amount deployTransactor.Value = amount
addr, _, _, err := contract.DeployENS(deployTransactor, backend) addr, _, _, err := contract.DeployResolver(deployTransactor, backend)
if err != nil { if err != nil {
return common.Address{}, err return common.Address{}, err
} }
@ -38,12 +44,25 @@ func TestENS(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
resolverAddr, err := deploy(key, big.NewInt(0), contractBackend)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
ens := NewENS(transactOpts, contractAddr, contractBackend) ens := NewENS(transactOpts, contractAddr, contractBackend)
_, err = ens.Register(name, hash) _, err = ens.Register(name, resolverAddr)
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
contractBackend.Commit() contractBackend.Commit()
_, err = ens.SetContentHash(name, hash)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
contractBackend.Commit()
vhost, err := ens.Resolve(name) vhost, err := ens.Resolve(name)
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)

531
swarm/storage/chunker.go
View file

@ -2,10 +2,11 @@ package storage
import ( import (
"encoding/binary" "encoding/binary"
"errors"
"fmt" "fmt"
"hash"
"io" "io"
"sync" "sync"
"time"
"github.com/ethereum/go-ethereum/logger" "github.com/ethereum/go-ethereum/logger"
"github.com/ethereum/go-ethereum/logger/glog" "github.com/ethereum/go-ethereum/logger/glog"
@ -37,11 +38,9 @@ data_{i} := size(subtree_{i}) || key_{j} || key_{j+1} .... || key_{j+n-1}
*/ */
const ( const (
// defaultHash = "SHA3" // http://golang.org/pkg/hash/#Hash defaultHash = "SHA3" // http://golang.org/pkg/hash/#Hash
defaultHash = "SHA256" // http://golang.org/pkg/hash/#Hash // defaultHash = "SHA256" // http://golang.org/pkg/hash/#Hash
defaultBranches int64 = 128 defaultBranches int64 = 128
joinTimeout = 120 // second
splitTimeout = 120 // second
// hashSize int64 = hasherfunc.New().Size() // hasher knows about its own length in bytes // hashSize int64 = hasherfunc.New().Size() // hasher knows about its own length in bytes
// chunksize int64 = branches * hashSize // chunk is defined as this // chunksize int64 = branches * hashSize // chunk is defined as this
) )
@ -55,130 +54,107 @@ The hashing itself does use extra copies and allocation though, since it does ne
*/ */
type ChunkerParams struct { type ChunkerParams struct {
Branches int64 Branches int64
Hash string Hash string
JoinTimeout time.Duration
SplitTimeout time.Duration
} }
func NewChunkerParams() *ChunkerParams { func NewChunkerParams() *ChunkerParams {
return &ChunkerParams{ return &ChunkerParams{
Branches: defaultBranches, Branches: defaultBranches,
Hash: defaultHash, Hash: defaultHash,
JoinTimeout: joinTimeout,
SplitTimeout: splitTimeout,
} }
} }
type TreeChunker struct { type TreeChunker struct {
branches int64 branches int64
hashFunc Hasher hashFunc Hasher
joinTimeout time.Duration
splitTimeout time.Duration
// calculated // calculated
hashSize int64 // self.hashFunc.New().Size() hashSize int64 // self.hashFunc.New().Size()
chunkSize int64 // hashSize* branches chunkSize int64 // hashSize* branches
workerCount int
} }
func NewTreeChunker(params *ChunkerParams) (self *TreeChunker) { func NewTreeChunker(params *ChunkerParams) (self *TreeChunker) {
self = &TreeChunker{} self = &TreeChunker{}
self.hashFunc = MakeHashFunc(params.Hash) self.hashFunc = MakeHashFunc(params.Hash)
self.branches = params.Branches self.branches = params.Branches
self.joinTimeout = params.JoinTimeout * time.Second
self.splitTimeout = params.SplitTimeout * time.Second
self.hashSize = int64(self.hashFunc().Size()) self.hashSize = int64(self.hashFunc().Size())
self.chunkSize = self.hashSize * self.branches self.chunkSize = self.hashSize * self.branches
self.workerCount = 1
return return
} }
func (self *TreeChunker) KeySize() int64 { // func (self *TreeChunker) KeySize() int64 {
return self.hashSize // return self.hashSize
} // }
// String() for pretty printing // String() for pretty printing
func (self *Chunk) String() string { func (self *Chunk) String() string {
return fmt.Sprintf("Key: %v TreeSize: %v Chunksize: %v", self.Key.Log(), self.Size, len(self.SData)) return fmt.Sprintf("Key: %v TreeSize: %v Chunksize: %v", self.Key.Log(), self.Size, len(self.SData))
} }
// The treeChunkers own Hash hashes together type hashJob struct {
// - the size (of the subtree encoded in the Chunk) key Key
// - the Chunk, ie. the contents read from the input reader chunk []byte
func (self *TreeChunker) Hash(input []byte) []byte { size int64
hasher := self.hashFunc() parentWg *sync.WaitGroup
hasher.Write(input)
return hasher.Sum(nil)
} }
func (self *TreeChunker) Split(key Key, data SectionReader, chunkC chan *Chunk, swg *sync.WaitGroup) (errC chan error) { func (self *TreeChunker) Split(data io.Reader, size int64, chunkC chan *Chunk, swg, wwg *sync.WaitGroup) (Key, error) {
if swg != nil {
swg.Add(1)
defer swg.Done()
}
if self.chunkSize <= 0 { if self.chunkSize <= 0 {
panic("chunker must be initialised") panic("chunker must be initialised")
} }
if int64(len(key)) != self.hashSize { jobC := make(chan *hashJob, 2*processors)
panic(fmt.Sprintf("root key buffer must be allocated byte slice of length %d", self.hashSize)) wg := &sync.WaitGroup{}
errC := make(chan error)
// wwg = workers waitgroup keeps track of hashworkers spawned by this split call
if wwg != nil {
wwg.Add(1)
}
go self.hashWorker(jobC, chunkC, errC, swg, wwg)
depth := 0
treeSize := self.chunkSize
// takes lowest depth such that chunksize*HashCount^(depth+1) > size
// power series, will find the order of magnitude of the data size in base hashCount or numbers of levels of branching in the resulting tree.
for ; treeSize < size; treeSize *= self.branches {
depth++
} }
wg := &sync.WaitGroup{} key := make([]byte, self.hashFunc().Size())
errC = make(chan error) // glog.V(logger.Detail).Infof("[BZZ] split request received for data (%v bytes, depth: %v)", size, depth)
rerrC := make(chan error) // this waitgroup member is released after the root hash is calculated
timeout := time.After(self.splitTimeout)
wg.Add(1) wg.Add(1)
go func() { //launch actual recursive function passing the waitgroups
go self.split(depth, treeSize/self.branches, key, data, size, jobC, chunkC, errC, wg, swg, wwg)
depth := 0
treeSize := self.chunkSize
size := data.Size()
// takes lowest depth such that chunksize*HashCount^(depth+1) > size
// power series, will find the order of magnitude of the data size in base hashCount or numbers of levels of branching in the resulting tree.
for ; treeSize < size; treeSize *= self.branches {
depth++
}
// glog.V(logger.Detail).Infof("[BZZ] split request received for data (%v bytes, depth: %v)", size, depth)
//launch actual recursive function passing the workgroup
self.split(depth, treeSize/self.branches, key, data, chunkC, rerrC, wg, swg)
}()
// closes internal error channel if all subprocesses in the workgroup finished // closes internal error channel if all subprocesses in the workgroup finished
go func() { go func() {
// waiting for all threads to finish
wg.Wait() wg.Wait()
close(rerrC) // if storage waitgroup is non-nil, we wait for storage to finish too
if swg != nil {
}() // glog.V(logger.Detail).Infof("Waiting for storage to finish")
swg.Wait()
// waiting for request to end with wg finishing, error, or timeout
go func() {
select {
case err := <-rerrC:
if err != nil {
errC <- err
} // otherwise splitting is complete
case <-timeout:
errC <- fmt.Errorf("split time out")
} }
close(errC) close(errC)
}() }()
return select {
case err := <-errC:
if err != nil {
return nil, err
}
//
}
return key, nil
} }
func (self *TreeChunker) split(depth int, treeSize int64, key Key, data SectionReader, chunkC chan *Chunk, errc chan error, parentWg *sync.WaitGroup, swg *sync.WaitGroup) { func (self *TreeChunker) split(depth int, treeSize int64, key Key, data io.Reader, size int64, jobC chan *hashJob, chunkC chan *Chunk, errC chan error, parentWg, swg, wwg *sync.WaitGroup) {
defer parentWg.Done()
size := data.Size()
var newChunk *Chunk
var hash Key
// glog.V(logger.Detail).Infof("[BZZ] depth: %v, max subtree size: %v, data size: %v", depth, treeSize, size)
for depth > 0 && size < treeSize { for depth > 0 && size < treeSize {
treeSize /= self.branches treeSize /= self.branches
@ -187,185 +163,233 @@ func (self *TreeChunker) split(depth int, treeSize int64, key Key, data SectionR
if depth == 0 { if depth == 0 {
// leaf nodes -> content chunks // leaf nodes -> content chunks
chunkData := make([]byte, data.Size()+8) chunkData := make([]byte, size+8)
binary.LittleEndian.PutUint64(chunkData[0:8], uint64(size)) binary.LittleEndian.PutUint64(chunkData[0:8], uint64(size))
data.ReadAt(chunkData[8:], 0) data.Read(chunkData[8:])
hash = self.Hash(chunkData) select {
// glog.V(logger.Detail).Infof("[BZZ] content chunk: max subtree size: %v, data size: %v", treeSize, size) case jobC <- &hashJob{key, chunkData, size, parentWg}:
newChunk = &Chunk{ case <-errC:
Key: hash,
SData: chunkData,
Size: size,
} }
} else { // glog.V(logger.Detail).Infof("[BZZ] read %v", size)
// intermediate chunk containing child nodes hashes return
branchCnt := int64((size + treeSize - 1) / treeSize) }
// glog.V(logger.Detail).Infof("[BZZ] intermediate node: setting branches: %v, depth: %v, max subtree size: %v, data size: %v", branches, depth, treeSize, size) // intermediate chunk containing child nodes hashes
branchCnt := int64((size + treeSize - 1) / treeSize)
// glog.V(logger.Detail).Infof("[BZZ] intermediate node: setting branches: %v, depth: %v, max subtree size: %v, data size: %v", branches, depth, treeSize, size)
var chunk []byte = make([]byte, branchCnt*self.hashSize+8) var chunk []byte = make([]byte, branchCnt*self.hashSize+8)
var pos, i int64 var pos, i int64
binary.LittleEndian.PutUint64(chunk[0:8], uint64(size)) binary.LittleEndian.PutUint64(chunk[0:8], uint64(size))
childrenWg := &sync.WaitGroup{} childrenWg := &sync.WaitGroup{}
var secSize int64 var secSize int64
for i < branchCnt { for i < branchCnt {
// the last item can have shorter data // the last item can have shorter data
if size-pos < treeSize { if size-pos < treeSize {
secSize = size - pos secSize = size - pos
} else { } else {
secSize = treeSize secSize = treeSize
}
// the hash of that data
subTreeKey := chunk[8+i*self.hashSize : 8+(i+1)*self.hashSize]
childrenWg.Add(1)
self.split(depth-1, treeSize/self.branches, subTreeKey, data, secSize, jobC, chunkC, errC, childrenWg, swg, wwg)
i++
pos += treeSize
}
// wait for all the children to complete calculating their hashes and copying them onto sections of the chunk
// parentWg.Add(1)
// go func() {
childrenWg.Wait()
if len(jobC) > self.workerCount && self.workerCount < processors {
if wwg != nil {
wwg.Add(1)
}
self.workerCount++
go self.hashWorker(jobC, chunkC, errC, swg, wwg)
}
select {
case jobC <- &hashJob{key, chunk, size, parentWg}:
case <-errC:
}
}
func (self *TreeChunker) hashWorker(jobC chan *hashJob, chunkC chan *Chunk, errC chan error, swg, wwg *sync.WaitGroup) {
hasher := self.hashFunc()
if wwg != nil {
defer wwg.Done()
}
for {
select {
case job, ok := <-jobC:
if !ok {
return
} }
// take the section of the data encoded in the subTree // now we got the hashes in the chunk, then hash the chunks
subTreeData := NewChunkReader(data, pos, secSize) hasher.Reset()
// the hash of that data self.hashChunk(hasher, job, chunkC, swg)
subTreeKey := chunk[8+i*self.hashSize : 8+(i+1)*self.hashSize] // glog.V(logger.Detail).Infof("[BZZ] hash chunk (%v)", job.size)
case <-errC:
childrenWg.Add(1) return
go self.split(depth-1, treeSize/self.branches, subTreeKey, subTreeData, chunkC, errc, childrenWg, swg)
i++
pos += treeSize
}
// wait for all the children to complete calculating their hashes and copying them onto sections of the chunk
childrenWg.Wait()
// now we got the hashes in the chunk, then hash the chunks
hash = self.Hash(chunk)
newChunk = &Chunk{
Key: hash,
SData: chunk,
Size: size,
wg: swg,
} }
}
}
// The treeChunkers own Hash hashes together
// - the size (of the subtree encoded in the Chunk)
// - the Chunk, ie. the contents read from the input reader
func (self *TreeChunker) hashChunk(hasher hash.Hash, job *hashJob, chunkC chan *Chunk, swg *sync.WaitGroup) {
hasher.Write(job.chunk)
h := hasher.Sum(nil)
newChunk := &Chunk{
Key: h,
SData: job.chunk,
Size: job.size,
wg: swg,
}
// report hash of this chunk one level up (keys corresponds to the proper subslice of the parent chunk)
copy(job.key, h)
// send off new chunk to storage
if chunkC != nil {
if swg != nil { if swg != nil {
swg.Add(1) swg.Add(1)
} }
} }
job.parentWg.Done()
// send off new chunk to storage
if chunkC != nil { if chunkC != nil {
chunkC <- newChunk chunkC <- newChunk
} }
// report hash of this chunk one level up (keys corresponds to the proper subslice of the parent chunk)x
copy(key, hash)
}
func (self *TreeChunker) Join(key Key, chunkC chan *Chunk) SectionReader {
return &LazyChunkReader{
key: key,
chunkC: chunkC,
quitC: make(chan bool),
errC: make(chan error),
chunker: self,
}
} }
// LazyChunkReader implements LazySectionReader // LazyChunkReader implements LazySectionReader
type LazyChunkReader struct { type LazyChunkReader struct {
key Key // root key key Key // root key
chunkC chan *Chunk // chunk channel to send retrieve requests on chunkC chan *Chunk // chunk channel to send retrieve requests on
size int64 // size of the entire subtree chunk *Chunk // size of the entire subtree
off int64 // offset off int64 // offset
quitC chan bool // channel to abort retrieval chunkSize int64 // inherit from chunker
errC chan error // error channel to monitor retrieve errors branches int64 // inherit from chunker
chunker *TreeChunker // needs TreeChunker params TODO: should just take hashSize int64 // inherit from chunker
// the chunkSize, branches etc as params
} }
func (self *LazyChunkReader) ReadAt(b []byte, off int64) (read int, err error) { // implements the Joiner interface
self.errC = make(chan error) func (self *TreeChunker) Join(key Key, chunkC chan *Chunk) LazySectionReader {
chunk := &Chunk{
Key: self.key,
C: make(chan bool), // close channel to signal data delivery
}
self.chunkC <- chunk // submit retrieval request, someone should be listening on the other side (or we will time out globally)
glog.V(logger.Detail).Infof("[BZZ] readAt: reading %v into %d bytes at offset %d.", chunk.Key.Log(), len(b), off)
// waiting for the chunk retrieval return &LazyChunkReader{
select { key: key,
case <-self.quitC: chunkC: chunkC,
// this is how we control process leakage (quitC is closed once join is finished (after timeout)) chunkSize: self.chunkSize,
// glog.V(logger.Detail).Infof("[BZZ] quit") branches: self.branches,
return hashSize: self.hashSize,
case <-chunk.C: // bells are ringing, data have been delivered
// glog.V(logger.Detail).Infof("[BZZ] chunk data received for %v", chunk.Key.Log())
} }
if len(chunk.SData) == 0 { }
// glog.V(logger.Detail).Infof("[BZZ] No payload in %v", chunk.Key.Log())
return 0, notFound // Size is meant to be called on the LazySectionReader
func (self *LazyChunkReader) Size(quitC chan bool) (n int64, err error) {
if self.chunk != nil {
return self.chunk.Size, nil
} }
chunk.Size = int64(binary.LittleEndian.Uint64(chunk.SData[0:8])) chunk := retrieve(self.key, self.chunkC, quitC)
self.size = chunk.Size if chunk == nil {
if b == nil { select {
case <-quitC:
return 0, errors.New("aborted")
default:
return 0, fmt.Errorf("root chunk not found for %v", self.key.Hex())
}
}
self.chunk = chunk
return chunk.Size, nil
}
// read at can be called numerous times
// concurrent reads are allowed
// Size() needs to be called synchronously on the LazyChunkReader first
func (self *LazyChunkReader) ReadAt(b []byte, off int64) (read int, err error) {
// this is correct, a swarm doc cannot be zero length, so no EOF is expected
if len(b) == 0 {
// glog.V(logger.Detail).Infof("[BZZ] Size query for %v", chunk.Key.Log()) // glog.V(logger.Detail).Infof("[BZZ] Size query for %v", chunk.Key.Log())
return return 0, nil
} }
want := int64(len(b)) quitC := make(chan bool)
if off+want > self.size { size, err := self.Size(quitC)
want = self.size - off if err != nil {
return 0, err
} }
glog.V(logger.Detail).Infof("readAt: len(b): %v, off: %v, size: %v ", len(b), off, size)
errC := make(chan error)
// glog.V(logger.Detail).Infof("[BZZ] readAt: reading %v into %d bytes at offset %d.", self.chunk.Key.Log(), len(b), off)
// }
// glog.V(logger.Detail).Infof("-> want: %v, off: %v size: %v ", want, off, self.size)
var treeSize int64 var treeSize int64
var depth int var depth int
// calculate depth and max treeSize // calculate depth and max treeSize
treeSize = self.chunker.chunkSize treeSize = self.chunkSize
for ; treeSize < chunk.Size; treeSize *= self.chunker.branches { for ; treeSize < size; treeSize *= self.branches {
depth++ depth++
} }
wg := sync.WaitGroup{} wg := sync.WaitGroup{}
wg.Add(1) wg.Add(1)
go self.join(b, off, off+want, depth, treeSize/self.chunker.branches, chunk, &wg) go self.join(b, off, off+int64(len(b)), depth, treeSize/self.branches, self.chunk, &wg, errC, quitC)
go func() { go func() {
wg.Wait() wg.Wait()
close(self.errC) close(errC)
}() }()
select {
case err = <-self.errC: err = <-errC
// glog.V(logger.Detail).Infof("[BZZ] ReadAt received %v", err) if err != nil {
read = len(b) close(quitC)
if off+int64(read) == self.size {
err = io.EOF return 0, err
}
// glog.V(logger.Detail).Infof("[BZZ] ReadAt returning at %d: %v", read, err)
case <-self.quitC:
// glog.V(logger.Detail).Infof("[BZZ] ReadAt aborted at %d: %v", read, err)
} }
return // glog.V(logger.Detail).Infof("[BZZ] ReadAt received %v", err)
glog.V(logger.Detail).Infof("end: len(b): %v, off: %v, size: %v ", len(b), off, size)
if off+int64(len(b)) >= size {
glog.V(logger.Detail).Infof(" len(b): %v EOF", len(b))
return len(b), io.EOF
}
// glog.V(logger.Detail).Infof("[BZZ] ReadAt returning at %d: %v", read, err)
return len(b), nil
} }
func (self *LazyChunkReader) join(b []byte, off int64, eoff int64, depth int, treeSize int64, chunk *Chunk, parentWg *sync.WaitGroup) { func (self *LazyChunkReader) join(b []byte, off int64, eoff int64, depth int, treeSize int64, chunk *Chunk, parentWg *sync.WaitGroup, errC chan error, quitC chan bool) {
defer parentWg.Done() defer parentWg.Done()
// return NewDPA(&LocalStore{})
glog.V(logger.Detail).Infof("inh len(b): %v, off: %v eoff: %v ", len(b), off, eoff)
// glog.V(logger.Detail).Infof("[BZZ] depth: %v, loff: %v, eoff: %v, chunk.Size: %v, treeSize: %v", depth, off, eoff, chunk.Size, treeSize) // glog.V(logger.Detail).Infof("[BZZ] depth: %v, loff: %v, eoff: %v, chunk.Size: %v, treeSize: %v", depth, off, eoff, chunk.Size, treeSize)
chunk.Size = int64(binary.LittleEndian.Uint64(chunk.SData[0:8])) // chunk.Size = int64(binary.LittleEndian.Uint64(chunk.SData[0:8]))
// find appropriate block level // find appropriate block level
for chunk.Size < treeSize && depth > 0 { for chunk.Size < treeSize && depth > 0 {
treeSize /= self.chunker.branches treeSize /= self.branches
depth-- depth--
} }
// leaf chunk found
if depth == 0 { if depth == 0 {
// glog.V(logger.Detail).Infof("[BZZ] depth: %v, len(b): %v, off: %v, eoff: %v, chunk.Size: %v, treeSize: %v", depth, len(b), off, eoff, chunk.Size, treeSize) glog.V(logger.Detail).Infof("[BZZ] depth: %v, len(b): %v, off: %v, eoff: %v, chunk.Size: %v, treeSize: %v", depth, len(b), off, eoff, chunk.Size, treeSize)
if int64(len(b)) != eoff-off {
//fmt.Printf("len(b) = %v off = %v eoff = %v", len(b), off, eoff)
panic("len(b) does not match")
}
copy(b, chunk.SData[8+off:8+eoff]) copy(b, chunk.SData[8+off:8+eoff])
return // simply give back the chunks reader for content chunks return // simply give back the chunks reader for content chunks
} }
// subtree index // subtree
start := off / treeSize start := off / treeSize
end := (eoff + treeSize - 1) / treeSize end := (eoff + treeSize - 1) / treeSize
wg := sync.WaitGroup{}
wg := &sync.WaitGroup{}
defer wg.Wait()
glog.V(logger.Detail).Infof("[BZZ] start %v,end %v", start, end)
for i := start; i < end; i++ { for i := start; i < end; i++ {
soff := i * treeSize soff := i * treeSize
roff := soff roff := soff
seoff := soff + treeSize seoff := soff + treeSize
@ -376,36 +400,91 @@ func (self *LazyChunkReader) join(b []byte, off int64, eoff int64, depth int, tr
if seoff > eoff { if seoff > eoff {
seoff = eoff seoff = eoff
} }
if depth > 1 {
wg.Wait()
}
wg.Add(1) wg.Add(1)
go func(j int64) { go func(j int64) {
childKey := chunk.SData[8+j*self.chunker.hashSize : 8+(j+1)*self.chunker.hashSize] childKey := chunk.SData[8+j*self.hashSize : 8+(j+1)*self.hashSize]
// glog.V(logger.Detail).Infof("[BZZ] subtree index: %v -> %v", j, childKey.Log()) // glog.V(logger.Detail).Infof("[BZZ] subtree ind.ex: %v -> %v", j, childKey.Log())
chunk := retrieve(childKey, self.chunkC, quitC)
ch := &Chunk{ if chunk == nil {
Key: childKey, select {
C: make(chan bool), // close channel to signal data delivery case errC <- fmt.Errorf("chunk %v-%v not found", off, off+treeSize):
} case <-quitC:
// glog.V(logger.Detail).Infof("[BZZ] chunk data sent for %v (key interval in chunk %v-%v)", ch.Key.Log(), j*self.chunker.hashSize, (j+1)*self.chunker.hashSize) }
self.chunkC <- ch // submit retrieval request, someone should be listening on the other side (or we will time out globally)
// waiting for the chunk retrieval
select {
case <-self.quitC:
// this is how we control process leakage (quitC is closed once join is finished (after timeout))
return return
case <-ch.C: // bells are ringing, data have been delivered
// glog.V(logger.Detail).Infof("[BZZ] chunk data received")
} }
if soff < off { if soff < off {
soff = off soff = off
} }
if len(ch.SData) == 0 { self.join(b[soff-off:seoff-off], soff-roff, seoff-roff, depth-1, treeSize/self.branches, chunk, wg, errC, quitC)
self.errC <- fmt.Errorf("chunk %v-%v not found", off, off+treeSize)
return
}
self.join(b[soff-off:seoff-off], soff-roff, seoff-roff, depth-1, treeSize/self.chunker.branches, ch, &wg)
}(i) }(i)
} //for } //for
wg.Wait() }
// the helper method submits chunks for a key to a oueue (DPA) and
// block until they time out or arrive
// abort if quitC is readable
func retrieve(key Key, chunkC chan *Chunk, quitC chan bool) *Chunk {
chunk := &Chunk{
Key: key,
C: make(chan bool), // close channel to signal data delivery
}
// glog.V(logger.Detail).Infof("[BZZ] chunk data sent for %v (key interval in chunk %v-%v)", ch.Key.Log(), j*self.chunker.hashSize, (j+1)*self.chunker.hashSize)
// submit chunk for retrieval
select {
case chunkC <- chunk: // submit retrieval request, someone should be listening on the other side (or we will time out globally)
case <-quitC:
return nil
}
// waiting for the chunk retrieval
select { // chunk.Size = int64(binary.LittleEndian.Uint64(chunk.SData[0:8]))
case <-quitC:
// this is how we control process leakage (quitC is closed once join is finished (after timeout))
return nil
case <-chunk.C: // bells are ringing, data have been delivered
// glog.V(logger.Detail).Infof("[BZZ] chunk data received")
}
if len(chunk.SData) == 0 {
return nil // chunk.Size = int64(binary.LittleEndian.Uint64(chunk.SData[0:8]))
}
return chunk
}
// Read keeps a cursor so cannot be called simulateously, see ReadAt
func (self *LazyChunkReader) Read(b []byte) (read int, err error) {
read, err = self.ReadAt(b, self.off)
glog.V(logger.Detail).Infof("[BZZ] read: %v, off: %v, error: %v", read, self.off, err)
self.off += int64(read)
return
}
// completely analogous to standard SectionReader implementation
var errWhence = errors.New("Seek: invalid whence")
var errOffset = errors.New("Seek: invalid offset")
func (s *LazyChunkReader) Seek(offset int64, whence int) (int64, error) {
switch whence {
default:
return 0, errWhence
case 0:
offset += 0
case 1:
offset += s.off
case 2:
if s.chunk == nil {
return 0, fmt.Errorf("seek from the end requires rootchunk for size. call Size first")
}
offset += s.chunk.Size
}
if offset < 0 {
return 0, errOffset
}
s.off = offset
return offset, nil
} }

270
swarm/storage/chunker_test.go
View file

@ -2,20 +2,33 @@ package storage
import ( import (
"bytes" "bytes"
// "fmt" "fmt"
"io" "io"
"runtime"
"sync"
"testing" "testing"
"time" "time"
"github.com/ethereum/go-ethereum/logger"
"github.com/ethereum/go-ethereum/logger/glog"
) )
func init() {
glog.SetV(logger.Info)
glog.SetToStderr(true)
}
/* /*
Tests TreeChunker by splitting and joining a random byte slice Tests TreeChunker by splitting and joining a random byte slice
*/ */
type test interface {
Fatalf(string, ...interface{})
}
type chunkerTester struct { type chunkerTester struct {
errors []error chunks []*Chunk
chunks []*Chunk t test
timeout bool
} }
func (self *chunkerTester) checkChunks(t *testing.T, want int) { func (self *chunkerTester) checkChunks(t *testing.T, want int) {
@ -25,77 +38,70 @@ func (self *chunkerTester) checkChunks(t *testing.T, want int) {
} }
} }
func (self *chunkerTester) Split(chunker *TreeChunker, l int) (key Key, input []byte) { func (self *chunkerTester) Split(chunker Splitter, data io.Reader, size int64, chunkC chan *Chunk, swg *sync.WaitGroup) (key Key) {
// reset // reset
self.errors = nil
self.chunks = nil self.chunks = nil
self.timeout = false
data, slice := testDataReader(l)
input = slice
key = make([]byte, 32)
chunkC := make(chan *Chunk, 1000)
errC := chunker.Split(key, data, chunkC, nil)
quitC := make(chan bool) quitC := make(chan bool)
timeout := time.After(600 * time.Second) timeout := time.After(600 * time.Second)
if chunkC != nil {
go func() {
for {
select {
case <-timeout:
self.t.Fatalf("Join timeout error")
go func() { case chunk, ok := <-chunkC:
LOOP: if !ok {
for { // glog.V(logger.Info).Infof("chunkC closed quitting")
select { close(quitC)
case <-timeout: return
self.timeout = true }
break LOOP // glog.V(logger.Info).Infof("chunk %v received", len(self.chunks))
case chunk := <-chunkC:
if chunk != nil {
self.chunks = append(self.chunks, chunk) self.chunks = append(self.chunks, chunk)
} else { if chunk.wg != nil {
break LOOP chunk.wg.Done()
} }
case err, ok := <-errC:
if err != nil {
self.errors = append(self.errors, err)
}
// fmt.Printf("err %v", err)
if !ok {
close(chunkC)
errC = nil
} }
} }
}()
}
key, err := chunker.Split(data, size, chunkC, swg, nil)
if err != nil {
self.t.Fatalf("Split error: %v", err)
}
if chunkC != nil {
if swg != nil {
// glog.V(logger.Info).Infof("Waiting for storage to finish")
swg.Wait()
// glog.V(logger.Info).Infof("St orage finished")
} }
close(quitC) close(chunkC)
}() }
<-quitC // waiting for it to finish if chunkC != nil {
<-quitC
}
return return
} }
func (self *chunkerTester) Join(chunker *TreeChunker, key Key, c int) SectionReader { func (self *chunkerTester) Join(chunker *TreeChunker, key Key, c int, chunkC chan *Chunk, quitC chan bool) LazySectionReader {
// reset but not the chunks // reset but not the chunks
self.errors = nil
self.timeout = false
chunkC := make(chan *Chunk, 1000)
reader := chunker.Join(key, chunkC) reader := chunker.Join(key, chunkC)
quitC := make(chan bool)
timeout := time.After(600 * time.Second) timeout := time.After(600 * time.Second)
i := 0 i := 0
go func() { go func() {
LOOP:
for { for {
select { select {
case <-quitC:
break LOOP
case <-timeout: case <-timeout:
self.timeout = true self.t.Fatalf("Join timeout error")
break LOOP
case chunk := <-chunkC: case chunk, ok := <-chunkC:
if !ok {
close(quitC)
return
}
i++ i++
// dpaLogger.DebugDetailf("TESTER: chunk request %x", chunk.Key[:4])
// this just mocks the behaviour of a chunk store retrieval // this just mocks the behaviour of a chunk store retrieval
var found bool var found bool
for _, ch := range self.chunks { for _, ch := range self.chunks {
@ -106,56 +112,58 @@ func (self *chunkerTester) Join(chunker *TreeChunker, key Key, c int) SectionRea
} }
} }
if !found { if !found {
// fmt.Printf("TESTER: chunk unknown for %x", chunk.Key[:4]) self.t.Fatalf("not found ")
} }
close(chunk.C) close(chunk.C)
// dpaLogger.DebugDetailf("TESTER: chunk request served %x", chunk.Key[:4])
} }
} }
}() }()
return reader return reader
} }
func testRandomData(chunker *TreeChunker, tester *chunkerTester, n int, chunks int, t *testing.T) { func testRandomData(n int, chunks int, t *testing.T) {
key, input := tester.Split(chunker, n) chunker := NewTreeChunker(&ChunkerParams{
Branches: 128,
Hash: "SHA3",
})
tester := &chunkerTester{t: t}
data, input := testDataReaderAndSlice(n)
t.Logf(" Key = %x\n", key) chunkC := make(chan *Chunk, 1000)
swg := &sync.WaitGroup{}
tester.checkChunks(t, chunks) splitter := chunker
time.Sleep(100 * time.Millisecond) key := tester.Split(splitter, data, int64(n), chunkC, swg)
reader := tester.Join(chunker, key, 0) // t.Logf(" Key = %v\n", key)
// tester.checkChunks(t, chunks)
chunkC = make(chan *Chunk, 1000)
quitC := make(chan bool)
reader := tester.Join(chunker, key, 0, chunkC, quitC)
output := make([]byte, n) output := make([]byte, n)
r, err := reader.Read(output) r, err := reader.Read(output)
if r != n || err != io.EOF { if r != n || err != io.EOF {
t.Errorf("read error read: %v n = %v err = %v\n", r, n, err) t.Fatalf("read error read: %v n = %v err = %v\n", r, n, err)
} }
// t.Logf(" IN: %x\nOUT: %x\n", input, output) if input != nil {
if !bytes.Equal(output, input) { if !bytes.Equal(output, input) {
t.Errorf("input and output mismatch\n IN: %x\nOUT: %x\n", input, output) t.Fatalf("input and output mismatch\n IN: %v\nOUT: %v\n", input, output)
}
} }
close(chunkC)
<-quitC
} }
func TestRandomData(t *testing.T) { func TestRandomData(t *testing.T) {
chunker, tester := chunkerAndTester() testRandomData(60, 1, t)
testRandomData(chunker, tester, 60, 1, t) testRandomData(83, 3, t)
testRandomData(chunker, tester, 179, 5, t) testRandomData(179, 5, t)
testRandomData(chunker, tester, 253, 7, t) testRandomData(253, 7, t)
// t.Logf("chunks %v", tester.chunks)
} }
func chunkerAndTester() (chunker *TreeChunker, tester *chunkerTester) { func readAll(reader LazySectionReader, result []byte) {
chunker = NewTreeChunker(&ChunkerParams{
Branches: 2,
Hash: "SHA256",
SplitTimeout: 10,
JoinTimeout: 10,
})
tester = &chunkerTester{}
return
}
func readAll(reader SectionReader, result []byte) {
size := int64(len(result)) size := int64(len(result))
var end int64 var end int64
@ -169,46 +177,98 @@ func readAll(reader SectionReader, result []byte) {
} }
} }
func benchReadAll(reader SectionReader) { func benchReadAll(reader LazySectionReader) {
size := reader.Size() size, _ := reader.Size(nil)
output := make([]byte, 1000) output := make([]byte, 1000)
for pos := int64(0); pos < size; pos += 1000 { for pos := int64(0); pos < size; pos += 1000 {
reader.ReadAt(output, pos) reader.ReadAt(output, pos)
} }
} }
func benchmarkJoinRandomData(n int, chunks int, t *testing.B) { func benchmarkJoin(n int, t *testing.B) {
t.StopTimer()
for i := 0; i < t.N; i++ { for i := 0; i < t.N; i++ {
// fmt.Printf("round %v\n", i) chunker := NewTreeChunker(&ChunkerParams{
chunker, tester := chunkerAndTester() Branches: 128,
key, _ := tester.Split(chunker, n) Hash: "SHA3",
// fmt.Printf("split done %v, joining...\n", i) })
tester := &chunkerTester{t: t}
data := testDataReader(n)
chunkC := make(chan *Chunk, 1000)
swg := &sync.WaitGroup{}
key := tester.Split(chunker, data, int64(n), chunkC, swg)
t.StartTimer() t.StartTimer()
reader := tester.Join(chunker, key, i) chunkC = make(chan *Chunk, 1000)
// fmt.Printf("join done %v, reading...\n", i) quitC := make(chan bool)
reader := tester.Join(chunker, key, i, chunkC, quitC)
t.StopTimer()
benchReadAll(reader) benchReadAll(reader)
close(chunkC)
<-quitC
} }
} }
func benchmarkSplitRandomData(n int, chunks int, t *testing.B) { func benchmarkSplitTree(n int, t *testing.B) {
t.ReportAllocs()
for i := 0; i < t.N; i++ { for i := 0; i < t.N; i++ {
chunker, tester := chunkerAndTester() chunker := NewTreeChunker(&ChunkerParams{
tester.Split(chunker, n) Branches: 128,
Hash: "SHA3",
})
tester := &chunkerTester{t: t}
data := testDataReader(n)
// glog.V(logger.Info).Infof("splitting data of length %v", n)
tester.Split(chunker, data, int64(n), nil, nil)
} }
stats := new(runtime.MemStats)
runtime.ReadMemStats(stats)
fmt.Println(stats.Sys)
} }
func BenchmarkJoinRandomData_100_2(t *testing.B) { benchmarkJoinRandomData(100, 3, t) } func benchmarkSplitPyramid(n int, t *testing.B) {
func BenchmarkJoinRandomData_1000_2(t *testing.B) { benchmarkJoinRandomData(1000, 3, t) } t.ReportAllocs()
func BenchmarkJoinRandomData_10000_2(t *testing.B) { benchmarkJoinRandomData(10000, 3, t) } for i := 0; i < t.N; i++ {
func BenchmarkJoinRandomData_100000_2(t *testing.B) { benchmarkJoinRandomData(100000, 3, t) } splitter := NewPyramidChunker(&ChunkerParams{
func BenchmarkJoinRandomData_1000000_2(t *testing.B) { benchmarkJoinRandomData(1000000, 3, t) } Branches: 128,
Hash: "SHA3",
})
tester := &chunkerTester{t: t}
data := testDataReader(n)
// glog.V(logger.Info).Infof("splitting data of length %v", n)
tester.Split(splitter, data, int64(n), nil, nil)
}
stats := new(runtime.MemStats)
runtime.ReadMemStats(stats)
fmt.Println(stats.Sys)
}
func BenchmarkSplitRandomData_100_2(t *testing.B) { benchmarkSplitRandomData(100, 3, t) } func BenchmarkJoin_100_2(t *testing.B) { benchmarkJoin(100, t) }
func BenchmarkSplitRandomData_1000_2(t *testing.B) { benchmarkSplitRandomData(1000, 3, t) } func BenchmarkJoin_1000_2(t *testing.B) { benchmarkJoin(1000, t) }
func BenchmarkSplitRandomData_10000_2(t *testing.B) { benchmarkSplitRandomData(10000, 3, t) } func BenchmarkJoin_10000_2(t *testing.B) { benchmarkJoin(10000, t) }
func BenchmarkSplitRandomData_100000_2(t *testing.B) { benchmarkSplitRandomData(100000, 3, t) } func BenchmarkJoin_100000_2(t *testing.B) { benchmarkJoin(100000, t) }
func BenchmarkSplitRandomData_1000000_2(t *testing.B) { benchmarkSplitRandomData(1000000, 3, t) } func BenchmarkJoin_1000000_2(t *testing.B) { benchmarkJoin(1000000, t) }
func BenchmarkSplitRandomData_10000000_2(t *testing.B) { benchmarkSplitRandomData(10000000, 3, t) }
// go test -bench ./bzz -cpuprofile cpu.out -memprofile mem.out func BenchmarkSplitTree_2(t *testing.B) { benchmarkSplitTree(100, t) }
func BenchmarkSplitTree_2h(t *testing.B) { benchmarkSplitTree(500, t) }
func BenchmarkSplitTree_3(t *testing.B) { benchmarkSplitTree(1000, t) }
func BenchmarkSplitTree_3h(t *testing.B) { benchmarkSplitTree(5000, t) }
func BenchmarkSplitTree_4(t *testing.B) { benchmarkSplitTree(10000, t) }
func BenchmarkSplitTree_4h(t *testing.B) { benchmarkSplitTree(50000, t) }
func BenchmarkSplitTree_5(t *testing.B) { benchmarkSplitTree(100000, t) }
func BenchmarkSplitTree_6(t *testing.B) { benchmarkSplitTree(1000000, t) }
func BenchmarkSplitTree_7(t *testing.B) { benchmarkSplitTree(10000000, t) }
func BenchmarkSplitTree_8(t *testing.B) { benchmarkSplitTree(100000000, t) }
func BenchmarkSplitPyramid_2(t *testing.B) { benchmarkSplitPyramid(100, t) }
func BenchmarkSplitPyramid_2h(t *testing.B) { benchmarkSplitPyramid(500, t) }
func BenchmarkSplitPyramid_3(t *testing.B) { benchmarkSplitPyramid(1000, t) }
func BenchmarkSplitPyramid_3h(t *testing.B) { benchmarkSplitPyramid(5000, t) }
func BenchmarkSplitPyramid_4(t *testing.B) { benchmarkSplitPyramid(10000, t) }
func BenchmarkSplitPyramid_4h(t *testing.B) { benchmarkSplitPyramid(50000, t) }
func BenchmarkSplitPyramid_5(t *testing.B) { benchmarkSplitPyramid(100000, t) }
func BenchmarkSplitPyramid_6(t *testing.B) { benchmarkSplitPyramid(1000000, t) }
func BenchmarkSplitPyramid_7(t *testing.B) { benchmarkSplitPyramid(10000000, t) }
func BenchmarkSplitPyramid_8(t *testing.B) { benchmarkSplitPyramid(100000000, t) }
// godep go test -bench ./swarm/storage -cpuprofile cpu.out -memprofile mem.out

194
swarm/storage/chunkreader.go
View file

@ -1,194 +0,0 @@
package storage
import (
"bytes"
"errors"
"io"
)
type Bounded interface {
Size() int64
}
type Sliced interface {
Slice(int64, int64) (b []byte, err error)
}
// Size, Seek, Read, ReadAt
type SectionReader interface {
Bounded
io.Seeker
io.Reader
io.ReaderAt
}
// ChunkReader implements SectionReader on a section
// of an underlying ReaderAt.
type ChunkReader struct {
r io.ReaderAt
base int64
off int64
limit int64
}
// NewChunkReader returns a ChunkReader that reads from r
// starting at offset off and stops with EOF after n bytes.
func NewChunkReader(r io.ReaderAt, off int64, n int64) *ChunkReader {
return &ChunkReader{r: r, base: off, off: off, limit: off + n}
}
// ByteSliceReader just extends byte.Reader to make base slice accessible
type ByteSliceReader struct {
*bytes.Reader
base []byte
}
func NewByteSliceReader(b []byte) *ByteSliceReader {
return &ByteSliceReader{
base: b,
Reader: bytes.NewReader(b),
}
}
// ByteSliceReader implements the Sliced interface
func (self *ByteSliceReader) Slice(from, to int64) (b []byte, err error) {
if from < 0 || to >= int64(self.Len()) {
err = io.EOF
} else {
b = self.base[from:to]
}
return
}
// NewChunkReaderFromBytes is a convenience shortcut to get a SectionReader over a byte slice
func NewChunkReaderFromBytes(b []byte) *ChunkReader {
return NewChunkReader(NewByteSliceReader(b), 0, int64(len(b)))
}
/*
The following is adapted from io.SectionReader
*/
func (s *ChunkReader) Size() int64 {
return s.limit - s.base
}
var errWhence = errors.New("Seek: invalid whence")
var errOffset = errors.New("Seek: invalid offset")
func (s *ChunkReader) Seek(offset int64, whence int) (int64, error) {
switch whence {
default:
return 0, errWhence
case 0:
offset += s.base
case 1:
offset += s.off
case 2:
offset += s.limit
}
if offset < s.base {
return 0, errOffset
}
s.off = offset
return offset - s.base, nil
}
func (s *ChunkReader) Read(p []byte) (n int, err error) {
if s.off >= s.limit {
return 0, io.EOF
}
if max := s.limit - s.off; int64(len(p)) > max {
p = p[0:max]
}
n, err = s.r.ReadAt(p, s.off)
s.off += int64(n)
return
}
func (s *ChunkReader) ReadAt(p []byte, off int64) (n int, err error) {
if off < 0 || off >= s.limit-s.base {
return 0, io.EOF
}
off += s.base
if max := s.limit - off; int64(len(p)) > max {
p = p[0:max]
n, err = s.r.ReadAt(p, off)
if err == nil {
err = io.EOF
}
return n, err
}
n, err = s.r.ReadAt(p, off)
return
}
// added methods to that ChunkReader implements the Sliced interface
func (s *ChunkReader) Slice(from, to int64) (b []byte, err error) {
if from < 0 || to >= s.Size() {
err = io.EOF
} else {
if sl, ok := s.r.(Sliced); ok {
b, err = sl.Slice(s.base+from, s.base+to)
} else {
err = errors.New("not sliceable base")
}
}
return
}
// added method so that ChunkReader implements the io.WriterTo interface
// WriteTo method is used by io.Copy
// This is so that we avoid one extra step of allocation (if the underlying initial Reader implements Sliced
func (r *ChunkReader) WriteTo(w io.Writer) (n int64, err error) {
var b []byte
var m int
// if b, _ := r.Slice(r.off-r.base, r.limit-r.base); b == nil {
// if slices not available we do it with extra allocation
b = make([]byte, r.limit-r.off)
m, err = r.Read(b)
if err != nil {
return
}
// }
m, err = w.Write(b)
if m > len(b) {
panic("bytes.Reader.WriteTo: invalid Write count")
}
r.off = r.base + int64(m)
n = int64(m)
if m != len(b) && err == nil {
err = io.ErrShortWrite
}
// w
return
}
func (self *LazyChunkReader) Size() (n int64) {
self.ReadAt(nil, 0)
return self.size
}
func (self *LazyChunkReader) Read(b []byte) (read int, err error) {
read, err = self.ReadAt(b, self.off)
self.off += int64(read)
return
}
func (s *LazyChunkReader) Seek(offset int64, whence int) (int64, error) {
switch whence {
default:
return 0, errWhence
case 0:
offset += 0
case 1:
offset += s.off
case 2:
offset += s.size
}
if offset < 0 {
return 0, errOffset
}
s.off = offset
return offset, nil
}

71
swarm/storage/common_test.go
View file

@ -1,6 +1,7 @@
package storage package storage
import ( import (
"bytes"
"crypto/rand" "crypto/rand"
"io" "io"
"sync" "sync"
@ -10,61 +11,39 @@ import (
"github.com/ethereum/go-ethereum/logger/glog" "github.com/ethereum/go-ethereum/logger/glog"
) )
func testDataReader(l int) (r *ChunkReader, slice []byte) { func testDataReader(l int) (r io.Reader) {
return io.LimitReader(rand.Reader, int64(l))
}
func testDataReaderAndSlice(l int) (r io.Reader, slice []byte) {
slice = make([]byte, l) slice = make([]byte, l)
if _, err := rand.Read(slice); err != nil { if _, err := rand.Read(slice); err != nil {
panic("rand error") panic("rand error")
} }
r = NewChunkReaderFromBytes(slice) r = bytes.NewReader(slice)
return
}
func randomChunks(l int64, branches int64, chunkC chan *Chunk) (key Key, errC chan error) {
chunker := NewTreeChunker(&ChunkerParams{
Branches: branches,
Hash: defaultHash,
SplitTimeout: splitTimeout,
})
key = make([]byte, 32)
b := make([]byte, l)
_, err := rand.Read(b)
if err != nil {
panic("no rand")
}
wg := &sync.WaitGroup{}
errC = chunker.Split(key, NewChunkReaderFromBytes(b), chunkC, wg)
wg.Wait()
return return
} }
func testStore(m ChunkStore, l int64, branches int64, t *testing.T) { func testStore(m ChunkStore, l int64, branches int64, t *testing.T) {
chunkC := make(chan *Chunk) chunkC := make(chan *Chunk)
key, errC := randomChunks(l, branches, chunkC) go func() {
for chunk := range chunkC {
SPLIT:
for {
select {
case chunk := <-chunkC:
m.Put(chunk) m.Put(chunk)
case err, ok := <-errC: if chunk.wg != nil {
if err != nil { chunk.wg.Done()
t.Errorf("Chunker error: %v", err)
return
}
if !ok {
break SPLIT
} }
} }
} }()
chunker := NewTreeChunker(&ChunkerParams{ chunker := NewTreeChunker(&ChunkerParams{
Branches: branches, Branches: branches,
Hash: defaultHash, Hash: defaultHash,
SplitTimeout: splitTimeout,
}) })
swg := &sync.WaitGroup{}
key, err := chunker.Split(rand.Reader, l, chunkC, swg, nil)
swg.Wait()
close(chunkC)
chunkC = make(chan *Chunk) chunkC = make(chan *Chunk)
var r SectionReader
r = chunker.Join(key, chunkC)
quit := make(chan bool) quit := make(chan bool)
@ -73,22 +52,26 @@ SPLIT:
go func(chunk *Chunk) { go func(chunk *Chunk) {
storedChunk, err := m.Get(chunk.Key) storedChunk, err := m.Get(chunk.Key)
if err == notFound { if err == notFound {
glog.V(logger.Detail).Infof("[BZZ] chunk '%x' not found", chunk.Key) glog.V(logger.Detail).Infof("[BZZ] chunk '%v' not found", chunk.Key.Log())
} else if err != nil { } else if err != nil {
glog.V(logger.Detail).Infof("[BZZ] error retrieving chunk %x: %v", chunk.Key, err) glog.V(logger.Detail).Infof("[BZZ] error retrieving chunk %v: %v", chunk.Key.Log(), err)
} else { } else {
chunk.SData = storedChunk.SData chunk.SData = storedChunk.SData
chunk.Size = storedChunk.Size
} }
glog.V(logger.Detail).Infof("[BZZ] chunk '%x' not found", chunk.Key[:4]) glog.V(logger.Detail).Infof("[BZZ] chunk '%v' not found", chunk.Key.Log())
close(chunk.C) close(chunk.C)
}(ch) }(ch)
} }
close(quit)
}() }()
r := chunker.Join(key, chunkC)
b := make([]byte, l) b := make([]byte, l)
n, err := r.ReadAt(b, 0) n, err := r.ReadAt(b, 0)
if err != io.EOF { if err != io.EOF {
t.Errorf("read error (%v/%v) %v", n, l, err) t.Fatalf("read error (%v/%v) %v", n, l, err)
close(quit)
} }
close(chunkC)
<-quit
} }

28
swarm/storage/dpa.go
View file

@ -2,6 +2,7 @@ package storage
import ( import (
"errors" "errors"
"io"
"sync" "sync"
"time" "time"
@ -72,33 +73,14 @@ func NewDPA(store ChunkStore, params *ChunkerParams) *DPA {
// FS-aware API and httpaccess // FS-aware API and httpaccess
// Chunk retrieval blocks on netStore requests with a timeout so reader will // Chunk retrieval blocks on netStore requests with a timeout so reader will
// report error if retrieval of chunks within requested range time out. // report error if retrieval of chunks within requested range time out.
func (self *DPA) Retrieve(key Key) SectionReader { func (self *DPA) Retrieve(key Key) LazySectionReader {
return self.Chunker.Join(key, self.retrieveC) return self.Chunker.Join(key, self.retrieveC)
} }
// Public API. Main entry point for document storage directly. Used by the // Public API. Main entry point for document storage directly. Used by the
// FS-aware API and httpaccess // FS-aware API and httpaccess
func (self *DPA) Store(data SectionReader, wg *sync.WaitGroup) (key Key, err error) { func (self *DPA) Store(data io.Reader, size int64, wg *sync.WaitGroup) (key Key, err error) {
key = make([]byte, self.Chunker.KeySize()) return self.Chunker.Split(data, size, self.storeC, nil, wg)
errC := self.Chunker.Split(key, data, self.storeC, wg)
SPLIT:
for {
select {
case err, ok := <-errC:
if err != nil {
glog.V(logger.Error).Infof("[BZZ] chunker split error: %v", err)
}
if !ok {
break SPLIT
}
case <-self.quitC:
break SPLIT
}
}
return
} }
func (self *DPA) Start() { func (self *DPA) Start() {
@ -164,7 +146,7 @@ func (self *DPA) storeLoop() {
go func(chunk *Chunk) { go func(chunk *Chunk) {
self.Put(chunk) self.Put(chunk)
if chunk.wg != nil { if chunk.wg != nil {
glog.V(logger.Detail).Infof("[BZZ] DPA.storeLoop %v", chunk.Key.Log()) glog.V(logger.Detail).Infof("[BZZ] dpa: store loop %v", chunk.Key.Log())
chunk.wg.Done() chunk.wg.Done()
} }
}(ch) }(ch)

8
swarm/storage/dpa_test.go
View file

@ -29,9 +29,9 @@ func TestDPArandom(t *testing.T) {
ChunkStore: localStore, ChunkStore: localStore,
} }
dpa.Start() dpa.Start()
reader, slice := testDataReader(testDataSize) reader, slice := testDataReaderAndSlice(testDataSize)
wg := &sync.WaitGroup{} wg := &sync.WaitGroup{}
key, err := dpa.Store(reader, wg) key, err := dpa.Store(reader, testDataSize, wg)
if err != nil { if err != nil {
t.Errorf("Store error: %v", err) t.Errorf("Store error: %v", err)
} }
@ -85,9 +85,9 @@ func TestDPA_capacity(t *testing.T) {
ChunkStore: localStore, ChunkStore: localStore,
} }
dpa.Start() dpa.Start()
reader, slice := testDataReader(testDataSize) reader, slice := testDataReaderAndSlice(testDataSize)
wg := &sync.WaitGroup{} wg := &sync.WaitGroup{}
key, err := dpa.Store(reader, wg) key, err := dpa.Store(reader, testDataSize, wg)
if err != nil { if err != nil {
t.Errorf("Store error: %v", err) t.Errorf("Store error: %v", err)
} }

5
swarm/storage/localstore.go
View file

@ -1,5 +1,9 @@
package storage package storage
import (
"encoding/binary"
)
// LocalStore is a combination of inmemory db over a disk persisted db // LocalStore is a combination of inmemory db over a disk persisted db
// implements a Get/Put with fallback (caching) logic using any 2 ChunkStores // implements a Get/Put with fallback (caching) logic using any 2 ChunkStores
type LocalStore struct { type LocalStore struct {
@ -48,6 +52,7 @@ func (self *LocalStore) Get(key Key) (chunk *Chunk, err error) {
if err != nil { if err != nil {
return return
} }
chunk.Size = int64(binary.LittleEndian.Uint64(chunk.SData[0:8]))
self.memStore.Put(chunk) self.memStore.Put(chunk)
return return
} }

36
swarm/storage/memstore.go
View file

@ -3,7 +3,6 @@
package storage package storage
import ( import (
"bytes"
"sync" "sync"
) )
@ -44,41 +43,6 @@ func NewMemStore(d *DbStore, capacity uint) (m *MemStore) {
return return
} }
func (x Key) Size() uint {
return uint(len(x))
}
func (x Key) isEqual(y Key) bool {
return bytes.Compare(x, y) == 0
}
func (h Key) bits(i, j uint) uint {
ii := i >> 3
jj := i & 7
if ii >= h.Size() {
return 0
}
if jj+j <= 8 {
return uint((h[ii] >> jj) & ((1 << j) - 1))
}
res := uint(h[ii] >> jj)
jj = 8 - jj
j -= jj
for j != 0 {
ii++
if j < 8 {
res += uint(h[ii]&((1<<j)-1)) << jj
return res
}
res += uint(h[ii]) << jj
jj += 8
j -= 8
}
return res
}
type memTree struct { type memTree struct {
subtree []*memTree subtree []*memTree
parent *memTree parent *memTree

3
swarm/storage/netstore.go
View file

@ -91,7 +91,8 @@ func (self *NetStore) Put(entry *Chunk) {
} else { } else {
glog.V(logger.Detail).Infof("[BZZ] NetStore.Put: localStore.Put %v stored locally", entry.Key.Log()) glog.V(logger.Detail).Infof("[BZZ] NetStore.Put: localStore.Put %v stored locally", entry.Key.Log())
// handle propagating store requests // handle propagating store requests
go self.cloud.Store(entry) // go self.cloud.Store(entry)
self.cloud.Store(entry)
} }
} }

165
swarm/storage/pyramid.go Normal file
View file

@ -0,0 +1,165 @@
package storage
import (
"io"
"math"
"sync"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/logger"
"github.com/ethereum/go-ethereum/logger/glog"
)
const (
processors = 8
)
type Tree struct {
Chunks int64
Levels []map[int64]*Node
Lock sync.RWMutex
}
type Node struct {
Pending int64
Children []common.Hash
Last bool
}
type Task struct {
Index int64 // Index of the chunk being processed
Data []byte // Binary blob of the chunk
Last bool
}
type PyramidChunker struct {
hashFunc Hasher
chunkSize int64
hashSize int64
branches int64
workerCount int
}
func NewPyramidChunker(params *ChunkerParams) (self *PyramidChunker) {
self = &PyramidChunker{}
self.hashFunc = MakeHashFunc(params.Hash)
self.branches = params.Branches
self.hashSize = int64(self.hashFunc().Size())
self.chunkSize = self.hashSize * self.branches
self.workerCount = 1
return
}
func (self *PyramidChunker) Split(data io.Reader, size int64, chunkC chan *Chunk, swg, wwg *sync.WaitGroup) (Key, error) {
chunks := (size + self.chunkSize - 1) / self.chunkSize
depth := int(math.Ceil(math.Log(float64(chunks))/math.Log(float64(self.branches)))) + 1
glog.V(logger.Detail).Infof("chunks: %v, depth: %v", chunks, depth)
results := Tree{
Chunks: chunks,
Levels: make([]map[int64]*Node, depth),
}
for i := 0; i < depth; i++ {
results.Levels[i] = make(map[int64]*Node)
}
// Create a pool of workers to crunch through the file
tasks := make(chan *Task, 2*processors)
pend := new(sync.WaitGroup)
abortC := make(chan bool)
for i := 0; i < processors; i++ {
pend.Add(1)
go self.processor(pend, tasks, &results)
}
// Feed the chunks into the task pool
for index := 0; ; index++ {
buffer := make([]byte, self.chunkSize+8)
n, err := io.ReadFull(data, buffer)
last := err == io.ErrUnexpectedEOF
if err != nil && !last {
glog.V(logger.Info).Infof("error: %v", err)
close(abortC)
}
pend.Add(1)
// glog.V(logger.Info).Infof("-> task %v (%v)", index, n)
select {
case tasks <- &Task{Index: int64(index), Data: buffer[:n+8], Last: last}:
case <-abortC:
return nil, err
}
if last {
// glog.V(logger.Info).Infof("last task %v (%v)", index, n)
break
}
}
// Wait for the workers and return
close(tasks)
pend.Wait()
// glog.V(logger.Info).Infof("len: %v", results.Levels[0][0])
key := results.Levels[0][0].Children[0][:]
return key, nil
}
func (self *PyramidChunker) processor(pend *sync.WaitGroup, tasks chan *Task, results *Tree) {
defer pend.Done()
// glog.V(logger.Info).Infof("processor started")
// Start processing leaf chunks ad infinitum
hasher := self.hashFunc()
for task := range tasks {
depth, pow := len(results.Levels)-1, self.branches
// glog.V(logger.Info).Infof("task: %v, last: %v", task.Index, task.Last)
var node *Node
for depth >= 0 {
// New chunk received, reset the hasher and start processing
hasher.Reset()
if node == nil { // Leaf node, hash the data chunk
hasher.Write(task.Data)
} else { // Internal node, hash the children
for _, hash := range node.Children {
hasher.Write(hash[:])
}
}
hash := hasher.Sum(nil)
last := task.Last || (node != nil) && node.Last
// Insert the subresult into the memoization tree
results.Lock.Lock()
if node = results.Levels[depth][task.Index/pow]; node == nil {
// Figure out the pending tasks
pending := self.branches
if task.Index/pow == results.Chunks/pow {
pending = (results.Chunks + pow/self.branches - 1) / (pow / self.branches) % self.branches
}
node = &Node{pending, make([]common.Hash, pending), last}
results.Levels[depth][task.Index/pow] = node
}
node.Pending--
i := task.Index / (pow / self.branches) % self.branches
if last {
node.Pending -= self.branches - i
node.Children = node.Children[:i+1]
node.Last = true
}
copy(node.Children[i][:], hash)
left := node.Pending
if depth+1 < len(results.Levels) {
delete(results.Levels[depth+1], task.Index/(pow/self.branches))
}
results.Lock.Unlock()
// If there's more work to be done, leave for others
// glog.V(logger.Info).Infof("left %v", left)
if left > 0 {
break
}
// We're the last ones in this batch, merge the children together
depth--
pow *= self.branches
}
pend.Done()
}
}

73
swarm/storage/types.go
View file

@ -5,6 +5,7 @@ import (
"crypto" "crypto"
"fmt" "fmt"
"hash" "hash"
"io"
"sync" "sync"
"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common"
@ -13,10 +14,47 @@ import (
type Hasher func() hash.Hash type Hasher func() hash.Hash
// Peer is the recorded as Source on the chunk
// should probably not be here? but network should wrap chunk object
type Peer interface{} type Peer interface{}
type Key []byte type Key []byte
func (x Key) Size() uint {
return uint(len(x))
}
func (x Key) isEqual(y Key) bool {
return bytes.Compare(x, y) == 0
}
func (h Key) bits(i, j uint) uint {
ii := i >> 3
jj := i & 7
if ii >= h.Size() {
return 0
}
if jj+j <= 8 {
return uint((h[ii] >> jj) & ((1 << j) - 1))
}
res := uint(h[ii] >> jj)
jj = 8 - jj
j -= jj
for j != 0 {
ii++
if j < 8 {
res += uint(h[ii]&((1<<j)-1)) << jj
return res
}
res += uint(h[ii]) << jj
jj += 8
j -= 8
}
return res
}
func IsZeroKey(key Key) bool { func IsZeroKey(key Key) bool {
return len(key) == 0 || bytes.Equal(key, ZeroKey) return len(key) == 0 || bytes.Equal(key, ZeroKey)
} }
@ -127,7 +165,7 @@ After getting notified that all the data has been split (the error channel is cl
When calling Join with a root key, the caller gets returned a seekable lazy reader. The caller again provides a channel on which the caller receives placeholder chunks with missing data. The DPA is supposed to forward this to the chunk stores and notify the chunker if the data has been delivered (i.e. retrieved from memory cache, disk-persisted db or cloud based swarm delivery). As the seekable reader is used, the chunker then puts these together the relevant parts on demand. When calling Join with a root key, the caller gets returned a seekable lazy reader. The caller again provides a channel on which the caller receives placeholder chunks with missing data. The DPA is supposed to forward this to the chunk stores and notify the chunker if the data has been delivered (i.e. retrieved from memory cache, disk-persisted db or cloud based swarm delivery). As the seekable reader is used, the chunker then puts these together the relevant parts on demand.
*/ */
type Chunker interface { type Splitter interface {
/* /*
When splitting, data is given as a SectionReader, and the key is a hashSize long byte slice (Key), the root hash of the entire content will fill this once processing finishes. When splitting, data is given as a SectionReader, and the key is a hashSize long byte slice (Key), the root hash of the entire content will fill this once processing finishes.
New chunks to store are coming to caller via the chunk storage channel, which the caller provides. New chunks to store are coming to caller via the chunk storage channel, which the caller provides.
@ -135,7 +173,10 @@ type Chunker interface {
The caller gets returned an error channel, if an error is encountered during splitting, it is fed to errC error channel. The caller gets returned an error channel, if an error is encountered during splitting, it is fed to errC error channel.
A closed error signals process completion at which point the key can be considered final if there were no errors. A closed error signals process completion at which point the key can be considered final if there were no errors.
*/ */
Split(key Key, data SectionReader, chunkC chan *Chunk, wg *sync.WaitGroup) chan error Split(io.Reader, int64, chan *Chunk, *sync.WaitGroup, *sync.WaitGroup) (Key, error)
}
type Joiner interface {
/* /*
Join reconstructs original content based on a root key. Join reconstructs original content based on a root key.
When joining, the caller gets returned a Lazy SectionReader, which is When joining, the caller gets returned a Lazy SectionReader, which is
@ -148,8 +189,28 @@ type Chunker interface {
The chunks are not meant to be validated by the chunker when joining. This The chunks are not meant to be validated by the chunker when joining. This
is because it is left to the DPA to decide which sources are trusted. is because it is left to the DPA to decide which sources are trusted.
*/ */
Join(key Key, chunkC chan *Chunk) SectionReader Join(key Key, chunkC chan *Chunk) LazySectionReader
}
// returns the key length
KeySize() int64 type Chunker interface {
Joiner
Splitter
// returns the key length
// KeySize() int64
}
// Size, Seek, Read, ReadAt
type LazySectionReader interface {
Size(chan bool) (int64, error)
io.Seeker
io.Reader
io.ReaderAt
}
type LazyTestSectionReader struct {
*io.SectionReader
}
func (self *LazyTestSectionReader) Size(chan bool) (int64, error) {
return self.SectionReader.Size(), nil
} }

6
swarm/swarm.go
View file

@ -31,8 +31,6 @@ const (
Version = "0.1" // versioning reflect POC and release versions Version = "0.1" // versioning reflect POC and release versions
) )
var ENSContractAddr = common.HexToAddress("0x504cdf3992d8f81a4182bd7b24e270d3a28711e3")
// the swarm stack // the swarm stack
type Swarm struct { type Swarm struct {
ethereum *eth.Ethereum ethereum *eth.Ethereum
@ -141,8 +139,8 @@ func NewSwarm(ctx *node.ServiceContext, config *api.Config, swapEnabled, syncEna
// set up high level api // set up high level api
transactOpts := bind.NewKeyedTransactor(self.privateKey) transactOpts := bind.NewKeyedTransactor(self.privateKey)
// backend := ethereum.ContractBackend() // backend := ethereum.ContractBackend()
self.dns = ens.NewENS(transactOpts, ENSContractAddr, self.backend) self.dns = ens.NewENS(transactOpts, config.EnsRoot, self.backend)
glog.V(logger.Debug).Infof("[BZZ] -> Swarm Domain Name Registrar") glog.V(logger.Debug).Infof("[BZZ] -> Swarm Domain Name Registrar @ address %v", config.EnsRoot)
self.api = api.NewApi(self.dpa, self.dns) self.api = api.NewApi(self.dpa, self.dns)
// Manifests for Smart Hosting // Manifests for Smart Hosting

30
swarm/test/connections/00.sh
View file

@ -1,33 +1,25 @@
#!/bin/bash #!/bin/bash
dir=`dirname $0`
source $dir/../../cmd/swarm/test.sh
swarm init 4 swarm init 4
echo "expect each node to have 3 peers" echo "expect each node to have 3 peers"
cmd="'net.peerCount'" cmd="net.peerCount"
sleep 5 sleep 5
swarm attach 00 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 00 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 01 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 01 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 02 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 02 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 03 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 03 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm stop all swarm stop all
echo "after static nodes is deleted, connections are recovered from kaddb in bzz-peers.json" echo "connections are recovered from kaddb in bzz-peers.json"
# echo rm -rf $DATA_ROOT/enodes\*
# echo rm -rf $DATA_ROOT/data/\*/static-nodes.json
rm -rf $DATA_ROOT/enodes*
rm -rf $DATA_ROOT/data/*/static-nodes.json
swarm cluster 4 swarm cluster 4
echo "expect each node to have 3 peers" echo "expect each node to have 3 peers"
cmd="'net.peerCount'" sleep 5
sleep 10 swarm execute 00 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 00 --exec "$cmd" |tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 01 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 01 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 02 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 02 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL" swarm execute 03 "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm attach 03 --exec "$cmd"|tail -n1|grep -ql 3&& echo "PASS"||echo "FAIL"
swarm stop all swarm stop all

2
swarm/test/swap/00.sh
View file

@ -4,7 +4,7 @@ echo " two nodes that do not sync and do not have any funds"
echo " cannot retrieve content from each other" echo " cannot retrieve content from each other"
dir=`dirname $0` dir=`dirname $0`
source $dir/../../cmd/swarm/test.sh source $dir/../test.sh
FILE_00=/tmp/1K.0 FILE_00=/tmp/1K.0
randomfile 1 > $FILE_00 randomfile 1 > $FILE_00

13
swarm/test/swap/01.sh
View file

@ -3,19 +3,20 @@ echo " two nodes that do not sync but have enough funds"
echo " can retrieve content from each other" echo " can retrieve content from each other"
dir=`dirname $0` dir=`dirname $0`
source $dir/../../cmd/swarm/test.sh source $dir/..s/test.sh
file=/tmp/test.file file=/tmp/test.file
mininginterval=120 mininginterval=120
key=/tmp/key key=/tmp/key
logargs="--verbosity=0 --vmodule='swarm/*=6'" # logargs="--verbosity=0 --vmodule='swarm/*=6'"
# logargs='--verbosity=6' logargs='--verbosity=6'
# swarm init 2 --mine --bzznosync --bzznoswap=false $logargsc
# echo "Mining some ether..."
# sleep $mininginterval
swarm init 2 --mine --bzznosync $logargs swarm cluster 2 --mine --bzznosync --bzznoswap=false $logargsc
echo "Mining some ether..."
sleep $mininginterval
randomfile 10 > $file randomfile 10 > $file
swarm up 00 $file|tail -n1 > $key swarm up 00 $file|tail -n1 > $key

6
swarm/test/syncing/00.sh
View file

@ -4,7 +4,7 @@ echo " two nodes that sync (no swap and do not have any funds)"
echo " can be in sync content with each other" echo " can be in sync content with each other"
dir=`dirname $0` dir=`dirname $0`
source $dir/../../cmd/swarm/test.sh source $dir/../test.sh
mkdir -p /tmp/swarm-test-files mkdir -p /tmp/swarm-test-files
FILE_00=/tmp/swarm-test-files/00 FILE_00=/tmp/swarm-test-files/00
@ -28,7 +28,6 @@ swarm needs 00 $key $FILE_00
swarm needs 01 $key $FILE_00 swarm needs 01 $key $FILE_00
swarm stop 01 swarm stop 01
# exit 1;
swarm up 00 $FILE_01|tail -n1 > $key swarm up 00 $FILE_01|tail -n1 > $key
swarm needs 00 $key $FILE_01 swarm needs 00 $key $FILE_01
@ -46,7 +45,8 @@ swarm needs 00 $key $FILE_03
swarm stop 00 swarm stop 00
swarm up 01 $FILE_04|tail -n1 > $key swarm up 01 $FILE_04|tail -n1 > $key
swarm needs 01 $key $FILE_04 swarm needs 01 $key $FILE_04
swarm start 00 #--bzznoswap swarm start 00
sleep $wait
swarm needs 00 $key $FILE_04 swarm needs 00 $key $FILE_04
swarm stop all swarm stop all

2
swarm/test/syncing/01.sh
View file

@ -4,7 +4,7 @@ echo " two nodes that do not have any funds"
echo " can still sync content with each other" echo " can still sync content with each other"
dir=`dirname $0` dir=`dirname $0`
source $dir/../../cmd/swarm/test.sh source $dir/../test.sh
key=/tmp/key key=/tmp/key
long=/tmp/10M long=/tmp/10M

28
swarm/test/syncing/02.sh
View file

@ -5,26 +5,26 @@ echo " two nodes that sync (no swap and do not have any funds)"
echo " can sync content with each other even with intermittent network connection" echo " can sync content with each other even with intermittent network connection"
dir=`dirname $0` dir=`dirname $0`
source $dir/../../cmd/swarm/test.sh source $dir/../test.sh
long=/tmp/10M long=/tmp/10M
key=/tmp/key key=/tmp/key
randomfile 10000 > $long randomfile 100000 > $long
ls -l $long ls -l $long
swarm init 2 swarm init 2 --vmodule='swarm/*=5'
sleep $wait
swarm up 00 $long |tail -n1 > $key & swarm up 00 $long |tail -n1 > $key &
sleep $wait sleep 1
swarm attach 01 -exec "'bzz.blockNetworkRead(true)'" swarm execute 01 'bzz.blockNetworkRead(true)'
sleep $wait sleep 3
swarm attach 01 -exec "'bzz.blockNetworkRead(false)'" swarm execute 01 'bzz.blockNetworkRead(false)'
sleep $wait # sleep $wait
swarm attach 01 -exec "'bzz.blockNetworkRead(true)'" # swarm attach 01 -exec "'bzz.blockNetworkRead(true)'"
sleep $wait # sleep $wait
swarm stop 01 swarm stop 01
swarm start 01 # swarm start 01
swarm needs 01 $key $long # sleep $wait
# swarm needs 01 $key $long
# sleep 3
swarm stop all swarm stop all

20
swarm/test/test.sh Normal file
View file

@ -0,0 +1,20 @@
#!/bin/bash
TEST_DIR=`dirname $0`
TEST_NAME=`basename $0 .sh`
TEST_TYPE=`basename $TEST_DIR`
export IP_ADDR="[::]"
export SWARM_NETWORK_ID=322$TEST_NAME
export SWARM_DIR=~/bzz/test/$TEST_TYPE
rm -rf $SWARM_DIR/$SWARM_NETWORK_ID
wait=1
function randomfile {
dd if=/dev/urandom of=/dev/stdout bs=1024 count=$1 2>/dev/null
}