cmd/clef: polishes + Geth integration in the tutorial

cmd/clef/README.md

@@ -21,6 +21,7 @@ COMMANDS:
    init    Initialize the signer, generate secret storage
    attest  Attest that a js-file is to be used
    setpw   Store a credential for a keystore file
+   delpw   Remove a credential for a keystore file
    gendoc  Generate documentation about json-rpc format
    help    Shows a list of commands or help for one command
 
@@ -28,7 +29,7 @@ GLOBAL OPTIONS:
    --loglevel value        log level to emit to the screen (default: 4)
    --keystore value        Directory for the keystore (default: "$HOME/.ethereum/keystore")
    --configdir value       Directory for Clef configuration (default: "$HOME/.clef")
-   --chainid value         Chain id to use for signing (1=mainnet, 3=ropsten, 4=rinkeby, 5=Goerli) (default: 1)
+   --chainid value         Chain id to use for signing (1=mainnet, 3=Ropsten, 4=Rinkeby, 5=Goerli) (default: 1)
    --lightkdf              Reduce key-derivation RAM & CPU usage at some expense of KDF strength
    --nousb                 Disables monitoring for and managing USB hardware wallets
    --pcscdpath value       Path to the smartcard daemon (pcscd) socket file (default: "/run/pcscd/pcscd.comm")
@@ -41,7 +42,7 @@ GLOBAL OPTIONS:
    --signersecret value    A file containing the (encrypted) master seed to encrypt Clef data, e.g. keystore credentials and ruleset hash
    --4bytedb-custom value  File used for writing new 4byte-identifiers submitted via API (default: "./4byte-custom.json")
    --auditlog value        File used to emit audit logs. Set to "" to disable (default: "audit.log")
-   --rules value           Enable rule-engine
+   --rules value           Path to the rule file to auto-authorize requests with
    --stdio-ui              Use STDIN/STDOUT as a channel for an external UI. This means that an STDIN/STDOUT is used for RPC-communication with a e.g. a graphical user interface, and can be used when Clef is started by an external process.
    --stdio-ui-test         Mechanism to test interface between Clef and UI. Requires 'stdio-ui'.
    --advanced              If enabled, issues warnings instead of rejections for suspicious requests. Default off
@@ -52,44 +53,35 @@ GLOBAL OPTIONS:
 Example:
 
 ```
-clef -keystore /my/keystore -chainid 4
+$ clef -keystore /my/keystore -chainid 4
 ```
 
 ## Security model
 
-The security model of the signer is as follows:
+The security model of Clef is as follows:
 
-* One critical component (the signer binary / daemon) is responsible for handling cryptographic operations: signing, private keys, encryption/decryption of keystore files.
+* One critical component (the Clef binary / daemon) is responsible for handling cryptographic operations: signing, private keys, encryption/decryption of keystore files.
-* The signer binary has a well-defined 'external' API.
+* Clef has a well-defined 'external' API.
 * The 'external' API is considered UNTRUSTED.
-* The signer binary also communicates with whatever process that invoked the binary, via stdin/stdout.
+* Clef also communicates with whatever process that invoked the binary, via stdin/stdout.
   * This channel is considered 'trusted'. Over this channel, approvals and passwords are communicated.
 
-The general flow for signing a transaction using e.g. geth is as follows:
+The general flow for signing a transaction using e.g. Geth is as follows:
 ![image](sign_flow.png)
 
-In this case, `geth` would be started with `--externalsigner=http://localhost:8550` and would relay requests to `eth.sendTransaction`.
+In this case, `geth` would be started with `--signer http://localhost:8550` and would relay requests to `eth.sendTransaction`.
 
 ## TODOs
 
 Some snags and todos
 
-* [ ] The signer should take a startup param "--no-change", for UIs that do not contain the capability
+* [ ] Clef should take a startup param "--no-change", for UIs that do not contain the capability to perform changes to things, only approve/deny. Such a UI should be able to start the signer in a more secure mode by telling it that it only wants approve/deny capabilities.
-   to perform changes to things, only approve/deny. Such a UI should be able to start the signer in
+* [x] It would be nice if Clef could collect new 4byte-id:s/method selectors, and have a secondary database for those (`4byte_custom.json`). Users could then (optionally) submit their collections for inclusion upstream.
-   a more secure mode by telling it that it only wants approve/deny capabilities.
+* [ ] It should be possible to configure Clef to check if an account is indeed known to it, before passing on to the UI. The reason it currently does not, is that it would make it possible to enumerate accounts if it immediately returned "unknown account" (side channel attack).
-
+* [x] It should be possible to configure Clef to auto-allow listing (certain) accounts, instead of asking every time.
-* [x] It would be nice if the signer could collect new 4byte-id:s/method selectors, and have a
+* [x] Done Upon startup, Clef should spit out some info to the caller (particularly important when executed in `stdio-ui`-mode), invoking methods with the following info:
-secondary database for those (`4byte_custom.json`). Users could then (optionally) submit their collections for
-inclusion upstream.
-
-* It should be possible to configure the signer to check if an account is indeed known to it, before
-passing on to the UI. The reason it currently does not, is that it would make it possible to enumerate
-accounts if it immediately returned "unknown account".
-* [x] It should be possible to configure the signer to auto-allow listing (certain) accounts, instead of asking every time.
-* [x] Done Upon startup, the signer should spit out some info to the caller (particularly important when executed in `stdio-ui`-mode),
-invoking methods with the following info:
   * [x] Version info about the signer
-  * [x] Address of API (http/ipc)
+  * [x] Address of API (HTTP/IPC)
   * [ ] List of known accounts
 * [ ] Have a default timeout on signing operations, so that if the user has not answered within e.g. 60 seconds, the request is rejected.
 * [ ] `account_signRawTransaction`
@@ -102,21 +94,16 @@ invoking methods with the following info:
       * the number of unique recipients
 
 * Geth todos
-    - The signer should pass the `Origin` header as call-info to the UI. As of right now, the way that info about the request is
+    - The signer should pass the `Origin` header as call-info to the UI. As of right now, the way that info about the request is put together is a bit of a hack into the HTTP server. This could probably be greatly improved.
-put together is a bit of a hack into the http server. This could probably be greatly improved
+    - Relay: Geth should be started in `geth --signer localhost:8550`.
-    - Relay: Geth should be started in `geth --external_signer localhost:8550`.
+    - Currently, the Geth APIs use `common.Address` in the arguments to transaction submission (e.g `to` field). This type is 20 `bytes`, and is incapable of carrying checksum information. The signer uses `common.MixedcaseAddress`, which retains the original input.
-    - Currently, the Geth APIs use `common.Address` in the arguments to transaction submission (e.g `to` field). This
+    - The Geth API should switch to use the same type, and relay `to`-account verbatim to the external API.
-  type is 20 `bytes`, and is incapable of carrying checksum information. The signer uses `common.MixedcaseAddress`, which
-  retains the original input.
-    - The Geth api should switch to use the same type, and relay `to`-account verbatim to the external api.
-
 * [x] Storage
-    * [x] An encrypted key-value storage should be implemented
+    * [x] An encrypted key-value storage should be implemented.
     * See [rules.md](rules.md) for more info about this.
-
 * Another potential thing to introduce is pairing.
   * To prevent spurious requests which users just accept, implement a way to "pair" the caller with the signer (external API).
-  * Thus geth/mist/cpp would cryptographically handshake and afterwards the caller would be allowed to make signing requests.
+  * Thus Geth/cpp would cryptographically handshake and afterwards the caller would be allowed to make signing requests.
   * This feature would make the addition of rules less dangerous.
 
 * Wallets / accounts. Add API methods for wallets.
@@ -125,37 +112,31 @@ put together is a bit of a hack into the http server. This could probably be gre
 
 ### External API
 
-The signer listens to HTTP requests on `rpcaddr`:`rpcport`, with the same JSONRPC standard as Geth. The messages are
+Clef listens to HTTP requests on `rpcaddr`:`rpcport` (or to IPC on `ipcpath`), with the same JSON-RPC standard as Geth. The messages are expected to be [JSON-RPC 2.0 standard](https://www.jsonrpc.org/specification).
-expected to be JSON [jsonrpc 2.0 standard](http://www.jsonrpc.org/specification).
 
-Some of these call can require user interaction. Clients must be aware that responses
+Some of these call can require user interaction. Clients must be aware that responses may be delayed significantly or may never be received if a users decides to ignore the confirmation request.
-may be delayed significantly or may never be received if a users decides to ignore the confirmation request.
 
-The External API is **untrusted** : it does not accept credentials over this api, nor does it expect
+The External API is **untrusted**: it does not accept credentials over this API, nor does it expect that requests have any authority.
-that requests have any authority.
 
-### UI API
+### Internal UI API
 
-The signer has one native console-based UI, for operation without any standalone tools.
+Clef has one native console-based UI, for operation without any standalone tools. However, there is also an API to communicate with an external UI. To enable that UI, the signer needs to be executed with the `--stdio-ui` option, which allocates `stdin` / `stdout` for the UI API.
-However, there is also an API to communicate with an external UI. To enable that UI,
-the signer needs to be executed with the `--stdio-ui` option, which allocates the
-`stdin`/`stdout` for the UI-api.
 
 An example (insecure) proof-of-concept of has been implemented in `pythonsigner.py`.
 
 The model is as follows:
 
 * The user starts the UI app (`pythonsigner.py`).
-* The UI app starts the `signer` with `--stdio-ui`, and listens to the
+* The UI app starts `clef` with `--stdio-ui`, and listens to the
 process output for confirmation-requests.
-* The `signer` opens the external http api.
+* `clef` opens the external HTTP API.
-* When the `signer` receives requests, it sends a `jsonrpc` request via `stdout`.
+* When the `signer` receives requests, it sends a JSON-RPC request via `stdout`.
-* The UI app prompts the user accordingly, and responds to the `signer`
+* The UI app prompts the user accordingly, and responds to `clef`.
-* The `signer` signs (or not), and responds to the original request.
+* `clef` signs (or not), and responds to the original request.
 
 ## External API
 
-See the [external api changelog](extapi_changelog.md) for information about changes to this API.
+See the [external API changelog](extapi_changelog.md) for information about changes to this API.
 
 ### Encoding
 - number: positive integers that are hex encoded
@@ -636,8 +617,6 @@ Response
 }
 ```
 
-
-
 ## UI API
 
 These methods needs to be implemented by a UI listener.
@@ -648,7 +627,7 @@ See `pythonsigner`, which can be invoked via `python3 pythonsigner.py test` to p
 
 All methods in this API uses object-based parameters, so that there can be no mixups of parameters: each piece of data is accessed by key.
 
-See the [ui api changelog](intapi_changelog.md) for information about changes to this API.
+See the [ui API changelog](intapi_changelog.md) for information about changes to this API.
 
 OBS! A slight deviation from `json` standard is in place: every request and response should be confined to a single line.
 Whereas the `json` specification allows for linebreaks, linebreaks __should not__ be used in this communication channel, to make
@@ -902,7 +881,7 @@ TLDR; Use this method to keep track of signed transactions, instead of using the
 
 ### OnSignerStartup / `ui_onSignerStartup`
 
-This method provide the UI with information about what API version the signer uses (both internal and external) aswell as build-info and external api,
+This method provide the UI with information about what API version the signer uses (both internal and external) aswell as build-info and external API,
 in k/v-form.
 
 Example call:

cmd/clef/intapi_changelog.md

@@ -13,6 +13,7 @@ Additional labels for pre-release and build metadata are available as extensions
 ### 7.0.0
 
 - The `message` field was renamed to `messages` in all data signing request methods to better reflect that it's a list, not a value.
+- The `storage.Put` and `storage.Get` methods in the rule execution engine were lower-cased to `storage.put` and `storage.get` to be consistent with JavaScript call conventions.
 
 ### 6.0.0
 

cmd/clef/pythonsigner.py

@@ -42,7 +42,6 @@ class PipeTransport(ServerTransport):
         self.output.write("\n")
 
 class StdIOHandler():
-
     def __init__(self):
         pass
 
@@ -158,8 +157,7 @@ class StdIOHandler():
         return
 
 def main(args):
-
+    cmd = ["clef", "--stdio-ui"]
-    cmd = ["./clef", "--stdio-ui"]
     if len(args) > 0 and args[0] == "test":
         cmd.extend(["--stdio-ui-test"])
     print("cmd: {}".format(" ".join(cmd)))

cmd/clef/tutorial.md

@@ -289,3 +289,65 @@ t=2019-07-01T15:52:23+0300 lvl=info msg=SignData   api=signer type=response data
 ```
 
 For more details on writing automatic rules, please see the [rules spec](https://github.com/ethereum/go-ethereum/blob/master/cmd/clef/rules.md).
+
+## Geth integration
+
+Of course, as awesome as Clef is, it's not feasible to interact with it via JSON RPC by hand. Long term, we're hoping to convince the general Ethereum community to support Clef as a general signer (it's only 3-5 methods), thus allowing your favorite DApp, Metamask, MyCrypto, etc to request signatures directly.
+
+Until then however, we're trying to pave the way via Geth. Geth v1.9.0 has built in support via `--signer <API endpoint>` for using a local or remote Clef instance as an account backend!
+
+We can try this by running Clef with our previous rules on Rinkeby (for now it's a good idea to allow auto-listing accounts, since Geth likes to retrieve them once in a while).
+
+```text
+$ clef --keystore ~/.ethereum/rinkeby/keystore --chainid 4 --rules rules.js
+```
+
+In a different window we can start Geth, list our accounts, even list our wallets to see where the accounts originate from:
+
+```text
+$ geth --rinkeby --signer=~/.clef/clef.ipc console
+
+> eth.accounts
+["0xd9c9cd5f6779558b6e0ed4e6acf6b1947e7fa1f3", "0x086278a6c067775f71d6b2bb1856db6e28c30418"]
+
+> personal.listWallets
+[{
+    accounts: [{
+        address: "0xd9c9cd5f6779558b6e0ed4e6acf6b1947e7fa1f3",
+        url: "extapi://$HOME/.clef/clef.ipc"
+    }, {
+        address: "0x086278a6c067775f71d6b2bb1856db6e28c30418",
+        url: "extapi://$HOME/.clef/clef.ipc"
+    }],
+    status: "ok [version=6.0.0]",
+    url: "extapi://$HOME/.clef/clef.ipc"
+}]
+
+> eth.sendTransaction({from: eth.accounts[0], to: eth.accounts[0]})
+```
+
+Lastly, when we requested a transaction to be sent, Clef prompted us in the original window to approve it:
+
+```text
+--------- Transaction request-------------
+to:       0xD9C9Cd5f6779558b6e0eD4e6Acf6b1947E7fA1F3
+from:     0xD9C9Cd5f6779558b6e0eD4e6Acf6b1947E7fA1F3 [chksum ok]
+value:    0 wei
+gas:      0x5208 (21000)
+gasprice: 1000000000 wei
+nonce:    0x2366 (9062)
+
+Request context:
+	NA -> NA -> NA
+
+Additional HTTP header data, provided by the external caller:
+	User-Agent:
+	Origin:
+-------------------------------------------
+Approve? [y/N]:
+> y
+```
+
+:boom:
+
+*Note, if you enable the external signer backend in Geth, all other account management is disabled. This is because long term we want to remove account management from Geth.*