Péter Garamvölgyi
983d630244
feat: Sync and relay L1 messages ( #350 )
...
* add l1 config in genesis config (#249 )
* add l1 config in genesis config
* fix lint
* Update params/config.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* extend node configuration (#251 )
* extend node configuration
* use block number instead of hash
* accept safe, finalized and numbers for L1Confirmations
* fix typos
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* Fix/improve node config parsing (#260 )
* raise error on failed parsing
* default value
* add l1-message-type, transaction methods (#252 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* txpool l1 check, pointer change, marhsal test
* draft: start implementing l1message gas behavior
* draft: start implementing l1message gas behavior
* change to gas usage
* error comment typo
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* goimports
* update nonce, add hash test (fails), marshal test
* goimports
* target addr cant be nil
* change call msg
* comment out test
* lint
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Add L1 message database (#255 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* add L1 message store to rawdb
* remove comments
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* add range check
* fix tests
* update comments
* nit
* support blocks with 0 l1 messages
---------
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
* Fix L1Message Deep Copy, Complete Bridge Tx Hash test (#269 )
* deep copy value field, add tx hash test
comment
* typo
* Rename nonce to queueindex, increment sender nonce on L1 message execution (#271 )
* change nonce to queueindex, increment nonce on L1 message
* fix db acccessors
* Update core/types/transaction_marshalling.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* Fix db inspect command (#276 )
fix db inspect command
* Add l1 sync service (#256 )
* extend node configuration
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* use block number instead of hash
* accept safe, finalized and numbers for L1Confirmations
* add L1 message store to rawdb
* remove comments
* fix typos
* add L1 message sync service
* use l1 contract address and chain ID
* use L1DeploymentBlock
* add confirmation config
* move bridge client to separate file
* use uint64 block number
* fix bigint comparison
* rename constants
* add more logs
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* check if config is provided
* improve sync service DB batched writes
* add range check
* fix tests
* update comments
* nit
* fix flush range and improve comments
* solve circular dependency
* update stress tests
* initialize l1 client for geth
* start sync service
* add more comments
* check nil correctly
* address comments
* fix merge
* fix genesis l1config deserialization
* add sync progress logs
* initial sync
* handle leveldb not found error
* use errors.Is
* address comments
* update DefaultPollInterval
---------
Co-authored-by: Nazarii Denha <dengaaa2002@gmail.com>
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
* Add L1 message validation (#272 )
* add L1 message validation
* add comments and better error handling
* handle leveldb not found error
* update incorrect condition for genesis block
* typo
* change inclusion index logic
* disable L1 message check for legacy tests
* set NumL1MessagesPerBlock to 0 in tests
* update default genesis config
* Add L1 msg validation tests (#303 )
add L1 msg validation tests
* Update miner include l1 messages (#265 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* add L1 message store to rawdb
* add L1 message sync service
* remove comments
* use l1 contract address and chain ID
* extend node configuration
* use block number instead of hash
* accept safe, finalized and numbers for L1Confirmations
* fix typos
* use L1DeploymentBlock
* add confirmation config
* move bridge client to separate file
* use uint64 block number
* fix bigint comparison
* rename constants
* add more logs
* Fix/improve node config parsing (#260 )
* raise error on failed parsing
* default value
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* check if config is provided
* improve sync service DB batched writes
* include l1 messages in blocks: part 1
* add l1-message-type, transaction methods (#252 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* txpool l1 check, pointer change, marhsal test
* draft: start implementing l1message gas behavior
* draft: start implementing l1message gas behavior
* change to gas usage
* error comment typo
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* goimports
* update nonce, add hash test (fails), marshal test
* goimports
* target addr cant be nil
* change call msg
* comment out test
* lint
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Add L1 message database (#255 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* add L1 message store to rawdb
* remove comments
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* add range check
* fix tests
* update comments
* nit
* support blocks with 0 l1 messages
---------
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
* build(docker): auto docker push when pushing git tags (#258 )
* build(docker): update docker trigger tag prefix (#259 )
* Fix L1Message Deep Copy, Complete Bridge Tx Hash test (#269 )
* deep copy value field, add tx hash test
comment
* typo
* commitl1messages
* lint
* Revert "add L1 message sync service"
This reverts commit 5305e8a5de14766ed249e1a7d64042c7a72cf5c2.
* Revert "move bridge client to separate file"
This reverts commit 0b220bee37de93c3250545e23430db2c401a2f90.
* update branch
* use commitMessages for l1Txs
* little fix
* fix config
* fix test
* comment fixes
* fix
* fix config check
---------
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
* Add ErrUnknownAncestor tests (#305 )
add ErrUnknownAncestor tests
* worker test include l1 msgs (#306 )
* worker test include l1 msgs
* move L1 message index update next to block insertion
---------
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* exclude l1 messages from transaction count limit in block (#307 )
* exclude l1 messages from transaction count limit in block
* fix comments
* trigger ci
* nit
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* Expose queueIndex on Transaction (#316 )
expose queueIndex on Transaction
* test that l1msg doesn't count in maxTxPerBlock limit (#312 )
* test that l1msg doesn't count in maxTxPerBlock limit
* fix, comment
* retrigger ci
* change order inside test
---------
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* reuse trace nonce field for queueIndex
* expose scroll APIs on the geth console
* add L1 message query APIs
* Trigger new block on new l1 messages (#343 )
* trigger new block on new l1 messages
* typo
* initialize l1MsgCh
* fix worker l1msg tests (#345 )
---------
Co-authored-by: Nazarii Denha <dengaaa2002@gmail.com>
* test(worker): ensure that l1 messages are included in the correct order (#346 )
test that l1msgs added in correct order
* rename enqueueIndex --> queueIndex
* move QueueIndex into transaction
* improve l1 db interface
* formatting
* bump version
* print l1config
* add API to query latest included message queue index
* clean up tx limit logic
* add clarifying comments and todos to ValidateL1Messages
* improve db comments and logs
* clean up L1MessageTx type handling
* format
* format
* improve L1 message block check
* fix missing L1 event handling
* fix TestL1MessageValidationFailure
* simplify sync height resume logic
* make l1Config.l1MessageQueueAddress non-pointer
* improve command line flags
* remove todo
* use abigen tools for log filtering
* cache block L1 message count
* nit: fix variable name case
* improve logs
* flush pending writes to DB before shutdown
---------
Co-authored-by: Nazarii Denha <dengaaa2002@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
Co-authored-by: HAOYUatHZ <haoyu@protonmail.com>
2023-06-07 07:24:19 -07:00
Steven
b6cc89fabe
Add goimports lint ( #54 )
...
* Add `goimports` lint.
* Fix `goimports` issue.
* Fix group to `github.com/scroll-tech/go-thereum`.
2022-03-24 16:53:59 +08:00
maskpp
b4d60884d2
Fix import fmt and run tests for given architecture ( #18 )
...
* fix gofmt
* Run tests for given architecture
* Speed up CI execution.
2022-01-12 19:45:06 -08:00
Jules
de7ed6af40
Update mod file and all imports to use scroll-tech URL ( #15 )
...
* Update mod file and all imports to use scroll-tech URL
Fixes #14
* Update Discord link
2021-12-24 10:59:22 +01:00
rjl493456442
2e8b58f076
cmd/geth: implement data import and export ( #22931 )
...
This PR offers two more database sub commands for exporting and importing data.
Two exporters are implemented: preimage and snapshot data respectively.
The import command is generic, it can take any data export and import into leveldb.
The data format has a 'magic' for disambiguation, and a version field for future compatibility.
2021-11-02 11:31:45 +01:00
Martin Holst Swende
0e7efd696b
core/rawdb, ethdb: introduce batched/atomic reads from ancients ( #23566 )
...
This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.
2021-10-25 16:24:27 +02:00
Martin Holst Swende
794c6133ef
core/rawdb: freezer batch write ( #23462 )
...
This change is a rewrite of the freezer code.
When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.
In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.
To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-07 12:31:17 +02:00
Martin Holst Swende
5f98020a21
core/rawdb: implement sequential reads in freezer_table ( #23117 )
...
* core/rawdb: implement sequential reads in freezer_table
* core/rawdb, ethdb: add sequential reader to db interface
* core/rawdb: lint nitpicks
* core/rawdb: fix some nitpicks
* core/rawdb: fix flaw with deferred reads not being performed
* core/rawdb: better documentation
2021-08-13 11:51:01 +03:00
Evolution404
248572ee54
core/rawdb: db inspect move 'config' and 'shutdown' into 'meta data' ( #22978 )
...
* core/rawdb: db inspect move 'config' and 'shutdown' into 'meta data'
* gofmt
2021-06-08 11:39:24 +03:00
Martin Holst Swende
bb9f9ccf4f
core/rawdb: wait for background freezing to exit when closing freezer ( #22878 )
2021-05-18 01:30:01 +02:00
Péter Szilágyi
745757ac6b
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:03:10 +03:00
gary rong
0c70b83e00
all: add read-only option to database ( #22407 )
...
* all: add read-only option to database
* all: fixes tests
* cmd/geth: migrate flags
* cmd/geth: fix the compact
* cmd/geth: fix the format
* cmd/geth: fix log
* cmd: add chain-readonly
* core: add readonly notion to freezer
* core/rawdb: add log
* core/rawdb: fix freezer close
* cmd: fix
* cmd, core: construct db
* core: update tests
2021-03-22 19:06:30 +01:00
Martin Holst Swende
c4a2b682ff
cmd/geth: add db commands stats, compact, put, get, delete ( #22014 )
...
This PR introduces:
- db.put to put a value into the database
- db.get to read a value from the database
- db.delete to delete a value from the database
- db.stats to check compaction info from the database
- db.compact to trigger a db compaction
It also moves inspectdb to db.inspect.
2021-02-23 11:27:32 +01:00
gary rong
f566dd305e
all: bloom-filter based pruning mechanism ( #21724 )
...
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 13:16:30 +02:00
gary rong
5a1b384352
core: persist bad blocks ( #21827 )
...
* core: persist bad blocks
* core, eth, internal: address comments
* core/rawdb: add badblocks to inspector
* core, eth: update
* internal: revert
* core, eth: only save 10 bad blocks
* core/rawdb: address comments
* core/rawdb: fix
* core: address comments
2021-01-10 12:54:15 +01:00
Martin Holst Swende
4d48980e74
core, eth, les: implement unclean-shutdown marker ( #21893 )
...
This PR implements unclean shutdown marker. Every time geth boots, it adds a timestamp to a list of timestamps in the database. This list is capped at 10. At a clean shutdown, the timestamp is removed again.
Thus, when geth exits unclean, the marker remains, and at boot up we show the most recent unclean shutdowns to the user, which makes it easier to diagnose root-causes to certain problems.
Co-authored-by: Nagy Salem <me@muhnagy.com>
2020-12-11 15:56:00 +01:00
Giuseppe Bertone
07751c3d26
cmd/geth: added counters to the geth inspect report ( #21495 )
...
* database: added counters
* Improved stats for ancient db
* Small improvement
* Better message and added percentage while counting receipts
* Fast counting for receipts
* added info message
* Show both receips itemscount from ancient db and counted receipts
* Fixed default case
* Removed counter for receipts in ancient store
* Removed counting of receipts present in leveldb
2020-09-17 10:23:56 +02:00
gary rong
87c0ba9213
core, eth, les, trie: add a prefix to contract code ( #21080 )
2020-08-21 15:10:40 +03:00
Péter Szilágyi
8cbdc8638f
core: define and test chain rewind corner cases ( #21409 )
...
* core: define and test chain reparation cornercases
* core: write up a variety of set-head tests
* core, eth: unify chain rollbacks, handle all the cases
* core: make linter smile
* core: remove commented out legacy code
* core, eth/downloader: fix review comments
* core: revert a removed recovery mechanism
2020-08-20 13:01:24 +03:00
gary rong
79addac698
core/rawdb: better log messages for ancient failure ( #21327 )
2020-07-13 20:43:30 +02:00
AusIV
069a7e1f8a
core/rawdb: stop freezer process as part of freezer.Close() ( #21010 )
...
* core/rawdb: Stop freezer process as part of freezer.Close()
When you call db.Close(), it was closing the leveldb database first,
then closing the freezer, but never stopping the freezer process.
This could cause the freezer to attempt to write to leveldb after
leveldb had been closed, leading to a crash with a non-zero exit code.
This change adds a quit channel to the freezer, and freezer.Close()
will not return until the freezer process has stopped.
Additionally, when you call freezerdb.Close(), it will close the
AncientStore before closing leveldb, to ensure that the freezer goroutine
will be stopped before leveldb is closed.
* core/rawdb: Fix formatting for golint
* core/rawdb: Use backoff flag to avoid repeating select
* core/rawdb: Include accidentally omitted backoff
2020-05-11 15:11:17 +03:00
Martin Holst Swende
6402c42b67
all: simplify and fix database iteration with prefix/start ( #20808 )
...
* core/state/snapshot: start fixing disk iterator seek
* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start
* les, core/state/snapshot: iterator fixes
* all: remove two iterator methods
* all: rename Iteratee.NewIteratorWith -> NewIterator
* ethdb: fix review concerns
2020-04-15 14:08:53 +03:00
Péter Szilágyi
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation
2020-02-25 12:51:08 +02:00
Péter Szilágyi
542df8898e
core: initial version of state snapshots
2020-02-25 12:51:04 +02:00
Guillaume Ballet
7b189d6f1f
core: fix staticcheck warnings ( #20384 )
...
* core: fix staticcheck warnings
* fix goimports
2019-11-27 09:50:30 +01:00
Péter Szilágyi
9eba3a9fff
cmd/geth, core/rawdb: seamless freezer consistency, friendly removedb
2019-05-16 17:01:56 +03:00
Péter Szilágyi
536b3b416c
cosensus, core, eth, params, trie: fixes + clique history cap
2019-05-16 10:39:35 +03:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool ( #15 )
...
* core, eth: some fixes for freezer
* vendor, core/rawdb, cmd/geth: add db inspector
* core, cmd/utils: check ancient store path forceily
* cmd/geth, common, core/rawdb: a few fixes
* cmd/geth: support windows file rename and fix rename error
* core: support ancient plugin
* core, cmd: streaming file copy
* cmd, consensus, core, tests: keep genesis in leveldb
* core: write txlookup during ancient init
* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong
80469bea0c
all: integrate the freezer with fast sync
...
* all: freezer style syncing
core, eth, les, light: clean up freezer relative APIs
core, eth, les, trie, ethdb, light: clean a bit
core, eth, les, light: add unit tests
core, light: rewrite setHead function
core, eth: fix downloader unit tests
core: add receipt chain insertion test
core: use constant instead of hardcoding table name
core: fix rollback
core: fix setHead
core/rawdb: remove canonical block first and then iterate side chain
core/rawdb, ethdb: add hasAncient interface
eth/downloader: calculate ancient limit via cht first
core, eth, ethdb: lots of fixes
* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
rjl493456442
b6cac42e9f
core/rawdb: add file lock for freezer
2019-05-16 10:39:31 +03:00
Péter Szilágyi
006c21efc7
cmd, core, eth, les, node: chain freezer on top of db rework
2019-05-16 10:39:29 +03:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access
2019-03-06 13:35:03 +02:00
