go-ethereum/swarm/api/api.go
zelig 0460fbad11 SWORM - swarm poc 0.2 stability RC3
rebase on current develop branch 2016-07-09

swarm/services/ens:
* Rework ENS functionality to use the current spec.
* Refactor ens.go to set up sessions as needed
* Refactoring to support independent registrar and personal resolver contracts
* fixed tests

swarm/examples/album: improve upload UX
* No more unhandled errors and loose ends in the UI.
* Forward/Back buttons work as undo/redo
* all long operations display an informative modal dialog
* minor problems with cursor movements fixed
* reorder labels
* throbber as default image
* change browser history after delete or upload photo

swarm/storage:
* optimise splitter
* simplify reader
* benchmarks, tests improved
* simplify IO, remove chunkreader.go and io.SectionReader clone
* simplify error/timeout handling
* port pyramid splitter by karalabe, adapt to Splitter/DPA interface, rework params etc
* make NetStore.Put -> cloud.Store synchronous
* chunker join fix process leak and keep parallelisation limited to depth 1
* reset the base hash to SHA3. !!hard fork = no backward compatibility :)
* SectionReader -> LazySectionReader: Size method signature change
* introduce abort channel to joiner to fix process leak
* add abort channel   context to all manifest retrieval methods
* global waitgroup fixes intermittent upload test failures due to unfinished storage of chunks
* streamline joiner logic, contexts now unique to each readAt call
* LazyChunkReader seeker complains about missing size only if whence=2
* chunker join fix process leak and keep parallelisation limited to depth 1

swarm/network/kademlia:
* simplify code
* now really fix prox limit adjustment + tests and comments
* simplify and make readable findclosest algo code + comments
* reset initial time interval settings for kaddb findbest
* fix bucket replace scheme to optimise stability and availability
* absolute idle peers are disconnected after maxIdleInterval
* fix index out of range when deleting last idle peer from kaddb
* unwanted peers (due to full kad bucket) are now properly dropped with ErrUnwanted
* improve logging and reorg in kademlia
* simplify and fix code that deletes expired/unconnectable nodes
* kaddb.findBest does not get stuck on empty row but finds an actual missing node
* must replace node if bucket is full otherwise nodes will get stuck on empty rows
* kademlia: timer fix
* remove logging the state when syncing (resulting in deadlock)
* improved logs, fix potential deadlock in String()
* native go time marshalling
* tests fixed

swarm/network:
* log node address consistently in syncdb
* syncer logs queue cardinalities properly in syncUnSyncedKeys loop
* fix hive stop issue leading to send on closed chan + minor logging fixes
* better logging
* hive interface change
* fix process leak by adding select to state.synced <- false

swarm/cmd: swarm control cli improvements
* no more alias, swarm  is executable so it can be called via ssh
* environment vars now set to default, no need to preconfigure
* local and remote hive monitoring
* gethup.sh script now merged into swarm script and nice modularised
* simplify bash code and fix e2e tests in swarm/test
* stop method falls back to kill -9 after 10s
* enode method now supplies ip addr via ipecho request
* execute, options, rawoptions, setup, create-account addpeers and hive subcommands
* update-src -> update
* remote-update-scripts, remote-update-bin and remote-run
* local and remote monitoring of kademlia
* extensive documentation in  swarm/cmd/README.md
* add cleanlog, cleanbzz, update-src, remote-update-scripts, remote-update-bin, remote-run
* for remote binary update, take scripts from swarm/cmd/swarm
* raise default maxpeers to 40
* add latency loggingo to upload and download
* add checkdownload and checkaccess subcommands
* add mem/cpu/disk-info commands
* include randomfile cmd from test
* add enode, connect subcommands, improve startup (only one round needed)
* TODO: simplify tests using new checks

swarm/api/http: server handler: check protocol substring length to fix slice out of bounds crash

swarm/api:
* fix filesystem API - upload/Modify tests
* simplify downloader code and fix process leak
* change KAD defaults bucketsize 4, minproxbinsize 2
* add back final slash to paths in manifest matching
* fix filesystem api tests
2016-07-27 12:18:26 +02:00

176 lines
4.6 KiB
Go

package api
import (
"fmt"
"io"
"regexp"
"strings"
"sync"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/logger"
"github.com/ethereum/go-ethereum/logger/glog"
"github.com/ethereum/go-ethereum/swarm/storage"
)
var (
hashMatcher = regexp.MustCompile("^[0-9A-Fa-f]{64}")
slashes = regexp.MustCompile("/+")
domainAndVersion = regexp.MustCompile("[@:;,]+")
)
type Resolver interface {
Resolve(string) (storage.Key, error)
}
/*
Api implements webserver/file system related content storage and retrieval
on top of the dpa
it is the public interface of the dpa which is included in the ethereum stack
*/
type Api struct {
dpa *storage.DPA
dns Resolver
}
//the api constructor initialises
func NewApi(dpa *storage.DPA, dns Resolver) (self *Api) {
self = &Api{
dpa: dpa,
dns: dns,
}
return
}
// DPA reader API
func (self *Api) Retrieve(key storage.Key) storage.LazySectionReader {
return self.dpa.Retrieve(key)
}
func (self *Api) Store(data io.Reader, size int64, wg *sync.WaitGroup) (key storage.Key, err error) {
return self.dpa.Store(data, size, wg)
}
type ErrResolve error
// DNS Resolver
func (self *Api) Resolve(hostPort string, nameresolver bool) (contentHash storage.Key, err error) {
if hashMatcher.MatchString(hostPort) || self.dns == nil {
glog.V(logger.Detail).Infof("[BZZ] host is a contentHash: '%v'", hostPort)
return storage.Key(common.Hex2Bytes(hostPort)), nil
}
if !nameresolver {
err = fmt.Errorf("'%s' is not a content hash value.", hostPort)
return
}
contentHash, err = self.dns.Resolve(hostPort)
if err != nil {
err = ErrResolve(err)
glog.V(logger.Warn).Infof("[BZZ] DNS error : %v", err)
}
glog.V(logger.Detail).Infof("[BZZ] host lookup: %v -> %v", err)
return
}
func parse(uri string) (hostPort, path string) {
parts := slashes.Split(uri, 3)
var i int
if len(parts) == 0 {
return
}
// beginning with slash is now optional
for len(parts[i]) == 0 {
i++
}
hostPort = parts[i]
for i < len(parts)-1 {
i++
if len(path) > 0 {
path = path + "/" + parts[i]
} else {
path = parts[i]
}
}
glog.V(logger.Debug).Infof("[BZZ] Swarm: host: '%s', path '%s' requested.", hostPort, path)
return
}
func (self *Api) parseAndResolve(uri string, nameresolver bool) (contentHash storage.Key, hostPort, path string, err error) {
hostPort, path = parse(uri)
//resolving host and port
contentHash, err = self.Resolve(hostPort, nameresolver)
glog.V(logger.Debug).Infof("[BZZ] Resolved '%s' to contentHash: '%s', path: '%s'", uri, contentHash, path)
return
}
// Put provides singleton manifest creation on top of dpa store
func (self *Api) Put(content, contentType string) (string, error) {
r := strings.NewReader(content)
wg := &sync.WaitGroup{}
key, err := self.dpa.Store(r, int64(len(content)), wg)
if err != nil {
return "", err
}
manifest := fmt.Sprintf(`{"entries":[{"hash":"%v","contentType":"%s"}]}`, key, contentType)
r = strings.NewReader(manifest)
key, err = self.dpa.Store(r, int64(len(manifest)), wg)
if err != nil {
return "", err
}
wg.Wait()
return key.String(), nil
}
// Get uses iterative manifest retrieval and prefix matching
// to resolve path to content using dpa retrieve
// it returns a section reader, mimeType, status and an error
func (self *Api) Get(uri string, nameresolver bool) (reader storage.LazySectionReader, mimeType string, status int, err error) {
key, _, path, err := self.parseAndResolve(uri, nameresolver)
quitC := make(chan bool)
trie, err := loadManifest(self.dpa, key, quitC)
if err != nil {
glog.V(logger.Warn).Infof("[BZZ] Swarm: loadManifestTrie error: %v", err)
return
}
glog.V(logger.Detail).Infof("[BZZ] Swarm: getEntry(%s)", path)
entry, _ := trie.getEntry(path)
if entry != nil {
key = common.Hex2Bytes(entry.Hash)
status = entry.Status
mimeType = entry.ContentType
glog.V(logger.Detail).Infof("[BZZ] Swarm: content lookup key: '%v' (%v)", key, mimeType)
reader = self.dpa.Retrieve(key)
} else {
err = fmt.Errorf("manifest entry for '%s' not found", path)
glog.V(logger.Warn).Infof("[BZZ] Swarm: %v", err)
}
return
}
func (self *Api) Modify(uri, contentHash, contentType string, nameresolver bool) (newRootHash string, err error) {
root, _, path, err := self.parseAndResolve(uri, nameresolver)
quitC := make(chan bool)
trie, err := loadManifest(self.dpa, root, quitC)
if err != nil {
return
}
if contentHash != "" {
entry := &manifestTrieEntry{
Path: path,
Hash: contentHash,
ContentType: contentType,
}
trie.addEntry(entry, quitC)
} else {
trie.deleteEntry(path, quitC)
}
err = trie.recalcAndStore()
if err != nil {
return
}
return trie.hash.String(), nil
}