Squash the main verkle PR ahead of rebase
don't call Bytes() in GetTreeKey (#137)
trie: avoid endianness conversion in GetTreeKey (#140)
* trie/utils: add concrete expected value in trie key generation test
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* mod: update to latest go-verkle
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* trie/utils: avoid endianness conversions
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* apply review changes & update to official go-verkle version
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138)
updating ci to use self-hosted machine (#143)
fix: storage offset in non-header group + reuse of value buffer (#145)
dedup call to ChunkifyCode, same as replay branch (#156)
* dedup call to ChunkifyCode, same as replay branch
* fix some linter issues
fix code offset in tree update (#157)
fix REVERT in state processor test execution (#158)
* fix code offset in tree update
* fix REVERT in test execution
save on key hashing: lump code size update with first code chunk group (#159)
fix code chunk key calculation and storage key calculation (#161)
* fix codeKey calculation
* Remove
* fix storageOffset
* fix the fix to the fix to the offset fix
* Remove copy/pasted, unused code in test
* fix linter
---------
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
fix: infinite loop when calling extcodecopy on empty code (#151)
upgrade to latest go-verkle
fix: only update code in the tree if it's dirty (#174)
fix: read-touch the code size and Keccak of the origin (#175)
List of changes for converting a sepolia database (#182)
* naive conversion rebased on top of beverly hills
* changes for the sepolia shadow fork conversion
* fixes to please the linter
* fixes to please the linter
Unified point cache (#180)
* Unified point cache
* Use cache for Try*Account
* alter Trie interface to use caching for slots (#181)
* alter Trie interface to use caching for slots
* fix: use a lock to protect the point cache (#185)
* use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184)
* mod: update to fastest go-verkle version today
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* trie/verkle: use new batch serialization api
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
* fix: TryDelete signature in unit tests
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
trie/utils: fix potential overflow (#191)
* trie/utils: fix potential overflow
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* trie/utils: receive storage key as a byte slice
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* revert formatter changes
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* trie/utils: fix mod 256
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
trie/utils: fix incorrect bigint assignment (#193)
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
upgrade precomp link to fix CI
fix: add missing code size&keccak leaves in empty accounts (#192)
fixes to use the latest go-verkle@master (#197)
* fixes to use the latest go-verkle@master
* linter fixes
* linter fixes for tests
* fix: use jsign's go-verkle fix
refactor: remove unused (*StateDB).GetXLittleEndian methods (#204)
fix gas accounting issue in state_processor_test.go (#207)
update go-verkle not to use StatelessNode anymore (#206)
* update go-verkle not to use StatelessNode anymore
* update go-verkle to latest
refactor: move verkle gas accounting to its own block in TransitionDB (#208)
fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements
use the cachingDB instead of a custom VerkleDB (#209)
* use the cachingDB instead of a custom VerkleDB
* fix stack trace in LES
remove holiman from CODEOWNERS as he gets too many emails
read from tree in state object if the snapshot is nil (#205)
add missing error checks for the root node type (#214)
implement OpenStorageTrie for verkle trees (#210)
* implement OpenStorageTrie for verkle trees
* add a few comments for future maintenance
* fix linter issue
fix: copy balance leaf to new buffer in TryGetAccount (#217)
implement some heretofore unimplemented iterator methods (#219)
params: move verkle params to their own file (#228)
fix: proper number of chunk evals (#215)
overlay transition (#244)
* overlay transition
Fix some bugs identified in the code review
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
Include base -> overlay key-values migration logic (#199)
* mod: add go-verkle version with key-value migration new apis
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* core/stateprocessor: use constant for max number of migrated key-values
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* core: add base->overlay key-values migration logic
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* core: fix some compiler errors
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* trie: consider removing transition trie api in the future
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* mod: use latest go-verkle
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
fix some unit tests errors
get convresion block from file
fix compilation issues
fix initialization issue in migrator
fix: changes needed to run the first 28 blocks
important sutff: fix the banner
fix: use nonce instead of balance in nonce leaf (#202)
fixes for performing the overlay transition (#203)
* fixes for performing the overlay transition
* fixes for the full replay
* fix: deletion-and-recreation of EoA
* fixes to replay 2M+ blocks
* upgrade to go-verkle@master
* fix: proper number of chunk evals
* rewrite conversion loop to fix known issues
changes to make replay work with the overlay method (#216)
* fixes for performing the overlay transition
fixes for the full replay
fix: deletion-and-recreation of EoA
fixes to replay 2M+ blocks
upgrade to go-verkle@master
fix: proper number of chunk evals
rewrite conversion loop to fix known issues
changes to make replay work with the overlay method
fixes to replay 2M+ blocks
update to latest go-verkle@master
* use a PBSS-like scheme for internal nodes (#221)
* use a PBSS-like scheme for internal nodes
* a couple of fixes coming from debugging replay
* fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222)
* fixes for pbss replay (#227)
* fixes for pbss replay
* trie/verkle: use capped batch size (#229)
* trie/verkle: use capped batch size
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* trie/verkle: avoid path variable allocation per db.Put
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* don't keep more than 32 state root conversions in RAM (#230)
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
* cleanup some code
* mod: update go-verkle
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* re-enable snapshot (#231)
* re-enable cancun block / snapshot (#226)
* clear storage conversion key upon translating account (#234)
* clear storage conversion key upon translating account
* mod: use latest go-verkle
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
* fix: self-deadlock with translated root map mutex (#236)
* return compressed commitment as root commitment (#237)
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
fix first panic in *TransitionTrie.Copy()
upgrade go-verkle to latest master
mod: update go-verkle (#239)
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
core: print state root every 100 blocks (#240)
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
fix: only Commit the account trie (#242)
fixes to get TestProcessVerkle to work with the overlay branch (#238)
* fixes to get TestProcessVerkle to work with the overlay branch
* fix all panics in verkle state processor test
* fix proof verification
move transition management to cachingDB
* fix: mark the verkle transition as started if it's ended without being started
* fix the verkle state processing test
* fix linter errors
* Add a function to clear verkle params for replay
* fix: handle TransitionTrie in OpenStorageTrie
* fix linter issue
* fix the deleted account error (#247)
* code cleanup (#248)
* fix: don't error on a missing conversion.txt (#249)
* Overlay Tree preimages exporting and usage (#246)
* export overlay preimages tool
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* use preimages flat file in overlay tree migration logic
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* cmd/geth: add --roothash to overlay tree preimage exporting command
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* cleanup
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* review feedback
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
* fix: reduce the PR footprint (#250)
* fix: don't fail when preimages.bin is missing (#251)
* fix: don't fail when preimages.bin is missing
* fix: don't open the preimages file when outside of transition
---------
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
review changes
remove replay-specific code
* core/state, light, les: make signature of ContractCode hash-independent
* push current state for feedback
* les: fix unit test
* core, les, light: fix les unittests
* core/state, trie, les, light: fix state iterator
* core, les: address comments
* les: fix lint
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
This commit fixes a flaw in two testcases, and brings down the exec-time from ~40s to ~8s for trie/TestIncompleteSync.
The checkConsistency was performed over and over again on the complete set of nodes, not just the recently added, turning it into a quadratic runtime.
When we flush a batch of trie nodes into database during the state
sync, we should guarantee that all children should be flushed before
parent.
Actually the trie nodes commit order is strict by: children -> parent.
But when we flush all ready nodes into db, we don't need the order
anymore since
(1) they are all ready nodes (no more dependency)
(2) underlying database provides write atomicity
* core, eth, trie: bloom filter for trie node dedup during fast sync
* eth/downloader, trie: address review comments
* core, ethdb, trie: restart fast-sync bloom construction now and again
* eth/downloader: initialize fast sync bloom on startup
* eth: reenable eth/62 until we properly remove it
With this commit, core/state's access to the underlying key/value database is
mediated through an interface. Database errors are tracked in StateDB and
returned by CommitTo or the new Error method.
Motivation for this change: We can remove the light client's duplicated copy of
core/state. The light client now supports node iteration, so tracing and storage
enumeration can work with the light client (not implemented in this commit).
* eth/downloader: separate state sync from queue
Scheduling of state node downloads hogged the downloader queue lock when
new requests were scheduled. This caused timeouts for other requests.
With this change, state sync is fully independent of all other downloads
and doesn't involve the queue at all.
State sync is started and checked on in processContent. This is slightly
awkward because processContent doesn't have a select loop. Instead, the
queue is closed by an auxiliary goroutine when state sync fails. We
tried several alternatives to this but settled on the current approach
because it's the least amount of change overall.
Handling of the pivot block has changed slightly: the queue previously
prevented import of pivot block receipts before the state of the pivot
block was available. In this commit, the receipt will be imported before
the state. This causes an annoyance where the pivot block is committed
as fast block head even when state downloads fail. Stay tuned for more
updates in this area ;)
* eth/downloader: remove cancelTimeout channel
* eth/downloader: retry state requests on timeout
* eth/downloader: improve comment
* eth/downloader: mark peers idle when state sync is done
* eth/downloader: move pivot block splitting to processContent
This change also ensures that pivot block receipts aren't imported
before the pivot block itself.
* eth/downloader: limit state node retries
* eth/downloader: improve state node error handling and retry check
* eth/downloader: remove maxStateNodeRetries
It fails the sync too much.
* eth/downloader: remove last use of cancelCh in statesync.go
Fixes TestDeliverHeadersHang*Fast and (hopefully)
the weird cancellation behaviour at the end of fast sync.
* eth/downloader: fix leak in runStateSync
* eth/downloader: don't run processFullSyncContent in LightSync mode
* eth/downloader: improve comments
* eth/downloader: fix vet, megacheck
* eth/downloader: remove unrequested tasks anyway
* eth/downloader, trie: various polishes around duplicate items
This commit explicitly tracks duplicate and unexpected state
delieveries done against a trie Sync structure, also adding
there to import info logs.
The commit moves the db batch used to commit trie changes one
level deeper so its flushed after every node insertion. This
is needed to avoid a lot of duplicate retrievals caused by
inconsistencies between Sync internals and database. A better
approach is to track not-yet-written states in trie.Sync and
flush on commit, but I'm focuing on correctness first now.
The commit fixes a regression around pivot block fail count.
The counter previously was reset to 1 if and only if a sync
cycle progressed (inserted at least 1 entry to the database).
The current code reset it already if a node was delivered,
which is not stong enough, because unless it ends up written
to disk, an attacker can just loop and attack ad infinitum.
The commit also fixes a regression around state deliveries
and timeouts. The old downloader tracked if a delivery is
stale (none of the deliveries were requestedt), in which
case it didn't mark the node idle and did not send further
requests, since it signals a past timeout. The current code
did mark it idle even on stale deliveries, which eventually
caused two requests to be in flight at the same time, making
the deliveries always stale and mass duplicating retrievals
between multiple peers.
* eth/downloader: fix state request leak
This commit fixes the hang seen sometimes while doing the state
sync. The cause of the hang was a rare combination of events:
request state data from peer, peer drops and reconnects almost
immediately. This caused a new download task to be assigned to
the peer, overwriting the old one still waiting for a timeout,
which in turned leaked the requests out, never to be retried.
The fix is to ensure that a task assignment moves any pending
one back into the retry queue.
The commit also fixes a regression with peer dropping due to
stalls. The current code considered a peer stalling if they
timed out delivering 1 item. However, the downloader never
requests only one, the minimum is 2 (attempt to fine tune
estimated latency/bandwidth). The fix is simply to drop if
a timeout is detected at 2 items.
Apart from the above bugfixes, the commit contains some code
polishes I made while debugging the hang.
* core, eth, trie: support batched trie sync db writes
* trie: rename SyncMemCache to syncMemBatch
This commit implements EIP158 part 1, 2, 3 & 4
1. If an account is empty it's no longer written to the trie. An empty
account is defined as (balance=0, nonce=0, storage=0, code=0).
2. Delete an empty account if it's touched
3. An empty account is redefined as either non-existent or empty.
4. Zero value calls and zero value suicides no longer consume the 25k
reation costs.
params: moved core/config to params
Signed-off-by: Jeffrey Wilcke <jeffrey@ethereum.org>
This commit replaces the deep-copy based state revert mechanism with a
linear complexity journal. This commit also hides several internal
StateDB methods to limit the number of ways in which calling code can
use the journal incorrectly.
As usual consultation and bug fixes to the initial implementation were
provided by @karalabe, @obscuren and @Arachnid. Thank you!
