Péter Garamvölgyi
68697b0fee
fix: use non-overlapping DB prefix for L1 messages ( #532 )
...
* fix: use non-overlapping db prefix for L1 messages
* bump version
2023-10-04 19:28:26 +02:00
Péter Garamvölgyi
751dbda234
feat: add metrics in L1 message iterator ( #527 )
...
* feat: add metrics in L1 message iterator
* add L1 message size metric
2023-10-01 14:48:59 +02:00
Péter Garamvölgyi
fe8232a40f
fix(worker): short-circuit l1 message iteration ( #525 )
...
* fix(worker): short-circuit l1 message iteration
* Update core/rawdb/accessors_l1_message_test.go
Co-authored-by: colin <102356659+colinlyguo@users.noreply.github.com>
---------
Co-authored-by: colin <102356659+colinlyguo@users.noreply.github.com>
2023-09-29 21:16:37 +02:00
Péter Garamvölgyi
9c1433b62a
fix(trace): use correct roots in trace of block with one transaction ( #508 )
...
fix: use correct roots in trace of block with one transaction
2023-09-11 12:13:50 +02:00
HAOYUatHZ
dade96d416
fix(rawdb, api): fix rawdb.writeSkippedTransaction & api.GetSkippedTransaction ( #503 )
...
* fix(rawsdb): fix `writeSkippedTransaction`
* Update version.go
* minor
* clean up
2023-09-07 15:35:04 +08:00
HAOYUatHZ
0b6c3be40d
feat(api/rawdb): allow storing traces for skipped txs ( #500 )
2023-09-05 11:03:15 +08:00
Péter Garamvölgyi
7de261b1be
feat: store skipped transactions in local db ( #467 )
...
* feat: store skipped txs in local db
* bump version
* bump version
* fix test
* include L2 txs in skipped index
* goimports
* rename more
* bump version
* fix missed renames
* fix the bug when calculating l2TxCount. (#479 )
* fix bug when calculate l2 tx count
* Update version
* fix: exclude L1 message from block payload size validation (#476 )
* fix: exclude L1 message from block payload size validation
* fix the bug when calculating l2TxCount. (#479 )
* fix bug when calculate l2 tx count
* Update version
* bump version
---------
Co-authored-by: maskpp <maskpp266@gmail.com>
* fix: update row estimation with scroll-prover `v0.7.2` (#475 )
* Fix row estimation.
* Update libzkp.
* more
* prepare
* finish
* upgrade
* bump version
* fix tests
* Update to scroll-prover `v0.7.2`.
* fix tests
* Update miner/worker.go
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* Update miner/worker.go
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* Reset ccc when skips first tx.
* do not unnecessarily skip L1 message
* fix ccc reset and improve code readability
* seal block on circuitcapacitychecker.ErrUnknown
---------
Co-authored-by: HAOYUatHZ <haoyu@protonmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* refactor: simplify ccc revert to snapshot (#480 )
* simplify ccc revert to snapshot
* Update version.go
---------
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
* feat: use --gcmode=archive and --cache.noprefetch=true by default (#482 )
* feat: use --gcmode=archive and --cache.noprefetch=true by default
* refuse to start with invalid config
* lint
* lint
* feat(sdk): support compressed response (#469 )
* enable use compression algorithm
* fix ci
* Just enable decode compressed content at ethclient
* fix comments
---------
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* fix: disable pruning and prefetch if not flags are provided (#483 )
Co-authored-by: colin <102356659+colinlyguo@users.noreply.github.com>
* fix: update libzkp to use scroll-prover `v0.7.5` (#484 )
* Update libzkp to use scroll-prover `v0.7.5`.
* Update version.
* address comments
* bump version
* nit
---------
Co-authored-by: maskpp <maskpp266@gmail.com>
Co-authored-by: Steven <asongala@163.com>
Co-authored-by: HAOYUatHZ <haoyu@protonmail.com>
Co-authored-by: iczc <iczcalan@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: colin <102356659+colinlyguo@users.noreply.github.com>
2023-08-25 15:41:05 +02:00
HAOYUatHZ
2f9edf73ca
feat(rollup): add circuits capacity checker ( #321 )
...
* add proof for predeployed storages
* reverse inneeded code
* update for mainbranch merging
* add pertx storage trace
* dummy tx proof
* add txstorage trace
* add coinbase storage as trace
* enable (sync) tracing by default
* basic
* init cgo framework
* fix break loop
* integrate the right zkevm version (#323 )
* finish rust codes
* use dylib (#325 )
* flip
* ?
* use cdylib
* revert
* fix
* apply_tx
* rename
* fixing types
fixing types
fixing types
* clean up
* ExecutionResults (#328 )
* filling
* filling
* more
* clean up
* filling
* coinbase
* add
* MPTWitness
* ExecutionResults WIP
* L1fee
L1fee
* sender
* to
* Failed & ReturnValue
* createdAcc & after
* remove MPTWitness
* txStorageTrace
* add FeeRecipient
* add StorageTrace
* fix FFI types
* better logger
* cargo fmt
* fix
* add build tags
* update Makefile
* fix library
* improve ld path
* correctly deal with circuit_capacity_checker returned result
* fix return value
* update cargo (#333 )
* update cargo
* update
* update go
* refactor
* raname `circuits capacity checker` to `circuit capacity checker`
* some refactorings
* [Fix] storage proof generation in capacity checker (#348 )
* make per-tx storage and deletion proof work
* format
* fix misplaced markdeletion
---------
Co-authored-by: HAOYUatHZ <haoyu@protonmail.com>
* docker (#363 )
* update Dockerfile
* build: update go version to 1.19
* update
* fix
* fix
* try
* simplify
* revert go version
update l2geth Dockerfiles
* fix
* fix coinbase
* fix (#369 )
* format
* Update version.go
* address comments
* Capacity refactor (#374 )
* init
* id
* support multiple instances
* fix id
* fix conflicts
* refactor to use same codes (#379 )
* re-init
* WIP
* WIP
* refactor
* go
* minor
* fix storage proof of l1gas price oracle
* move 1
* move 2
* move 3
* move 4
* move 5
move 5
* move 6
move 6
* move 7
* move 8
* move 9
* move 10
* clean up
clean up
---------
Co-authored-by: Ho Vei <noelwei@gmail.com>
* finish basic
* minor
* config capacity check in block_validator (#380 )
* init
* done
ref
* fix tests
fix tests
fix tests
fix tests
* add more comments
* apply_block
* improve logs
* cargo fmt
* Capacity big refactor (#383 )
* CreateTraceEnv
* WIP
* draft
more
fix
* for test
* fortet
* clean up
* add more comments
* goimports -local github.com/scroll-tech/go-ethereum -w .
* fix typos
* attempt 1
* attempt 2
* attempt 3
* gogogo
* clean up
* fix
* fix
* rename
* minor
* fix
* minor
* minor
* improve doc
* use dedicated `checkCircuitCapacity` flag (#394 )
* refactor
* fix
* add lock
* [feat] capacity checking: upgrade libzkp (#395 )
* upgrade
* upgrade libzkp
* write RowConsumption (#396 )
* write RowConsumption
* name alignments
* revert some formatting
* add lock to CircuitCapacityChecker in BlockValidator
* remove mutex pointer
* improve github workflow
* improve
* store row consumption in mining (#397 )
* prepare
* finish
* add more logs
* mark `ApplyBlock` as ready
* update libzkp (#401 )
* fix
* Capacity detail (#402 )
* fix(block-validation): consider skipping in ValidateL1Messages (#405 )
* fix(block-validation): consider skipping in ValidateL1Messages
* fix(block): consider skipping in L1MessageCount
* fix l1 validation tests
* fix NumL1Messages
* fix impl.go return types
fix
* better error handling (#407 )
* add
add
* add
* add
* add
* add
* cargo fmt
* add
* update
* add
* WIP
* minor
* gogogo
* gogogo
* fix
* fix
* fix
* cargo clippy
* improve
* improve
* creation lock (#408 )
* creation lock
* update
* Debug log (#409 )
* add more logs
* more
* more
* fix
* improve
* Update cmd/utils/flags.go
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* refactor worker.commit()
* avoid re-calculate
* txpool ccc err handling (#411 )
* more explicit error comments
* add more logs
* fix unnecessary commit
* add more logs
* fix `ineffassign`
* add more comments
* log id for `NewCircuitCapacityChecker` (#414 )
add log to `NewCircuitCapacityChecker`
* Persist skip info for block where all L1 msgs are skipped (#415 )
persist skip info for block where all L1 msgs are skipped
* Update version.go
---------
Co-authored-by: Ho Vei <noelwei@gmail.com>
Co-authored-by: Zhang Zhuo <mycinbrin@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
2023-08-01 13:52:31 +08:00
Nazarii Denha
a677ee3ccf
fix(ethapi): change scroll_getBlockByHash ethapi behaviour ( #398 )
...
* fix api behaviour
* update version
* not retrun error in ethclient
* add comment
* modify rowconsumption struct
* fix lint
* change naming
2023-07-20 00:10:34 +08:00
Nazarii Denha
3943547960
feat(rawdb&rpc): add block row consumption rpc ( #388 )
...
* add db api for rowconsumption
* add rpc api for block row consumption
* return nil if rc is is not known
* add method to sdk
* bump version
* small fix
* update version
---------
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
2023-07-11 22:07:09 +08:00
Péter Garamvölgyi
983d630244
feat: Sync and relay L1 messages ( #350 )
...
* add l1 config in genesis config (#249 )
* add l1 config in genesis config
* fix lint
* Update params/config.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* extend node configuration (#251 )
* extend node configuration
* use block number instead of hash
* accept safe, finalized and numbers for L1Confirmations
* fix typos
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* Fix/improve node config parsing (#260 )
* raise error on failed parsing
* default value
* add l1-message-type, transaction methods (#252 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* txpool l1 check, pointer change, marhsal test
* draft: start implementing l1message gas behavior
* draft: start implementing l1message gas behavior
* change to gas usage
* error comment typo
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* goimports
* update nonce, add hash test (fails), marshal test
* goimports
* target addr cant be nil
* change call msg
* comment out test
* lint
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Add L1 message database (#255 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* add L1 message store to rawdb
* remove comments
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* add range check
* fix tests
* update comments
* nit
* support blocks with 0 l1 messages
---------
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
* Fix L1Message Deep Copy, Complete Bridge Tx Hash test (#269 )
* deep copy value field, add tx hash test
comment
* typo
* Rename nonce to queueindex, increment sender nonce on L1 message execution (#271 )
* change nonce to queueindex, increment nonce on L1 message
* fix db acccessors
* Update core/types/transaction_marshalling.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* Fix db inspect command (#276 )
fix db inspect command
* Add l1 sync service (#256 )
* extend node configuration
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* use block number instead of hash
* accept safe, finalized and numbers for L1Confirmations
* add L1 message store to rawdb
* remove comments
* fix typos
* add L1 message sync service
* use l1 contract address and chain ID
* use L1DeploymentBlock
* add confirmation config
* move bridge client to separate file
* use uint64 block number
* fix bigint comparison
* rename constants
* add more logs
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* check if config is provided
* improve sync service DB batched writes
* add range check
* fix tests
* update comments
* nit
* fix flush range and improve comments
* solve circular dependency
* update stress tests
* initialize l1 client for geth
* start sync service
* add more comments
* check nil correctly
* address comments
* fix merge
* fix genesis l1config deserialization
* add sync progress logs
* initial sync
* handle leveldb not found error
* use errors.Is
* address comments
* update DefaultPollInterval
---------
Co-authored-by: Nazarii Denha <dengaaa2002@gmail.com>
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
* Add L1 message validation (#272 )
* add L1 message validation
* add comments and better error handling
* handle leveldb not found error
* update incorrect condition for genesis block
* typo
* change inclusion index logic
* disable L1 message check for legacy tests
* set NumL1MessagesPerBlock to 0 in tests
* update default genesis config
* Add L1 msg validation tests (#303 )
add L1 msg validation tests
* Update miner include l1 messages (#265 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* add L1 message store to rawdb
* add L1 message sync service
* remove comments
* use l1 contract address and chain ID
* extend node configuration
* use block number instead of hash
* accept safe, finalized and numbers for L1Confirmations
* fix typos
* use L1DeploymentBlock
* add confirmation config
* move bridge client to separate file
* use uint64 block number
* fix bigint comparison
* rename constants
* add more logs
* Fix/improve node config parsing (#260 )
* raise error on failed parsing
* default value
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* check if config is provided
* improve sync service DB batched writes
* include l1 messages in blocks: part 1
* add l1-message-type, transaction methods (#252 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* txpool l1 check, pointer change, marhsal test
* draft: start implementing l1message gas behavior
* draft: start implementing l1message gas behavior
* change to gas usage
* error comment typo
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* goimports
* update nonce, add hash test (fails), marshal test
* goimports
* target addr cant be nil
* change call msg
* comment out test
* lint
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Add L1 message database (#255 )
* add l1-message-type, transaction methods
* goimports
* Update core/types/transaction.go
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* add L1 message store to rawdb
* remove comments
* rename to l1_message
* rename variables and add comments
* write l1 msgs in a batch
* add more comments
* update tests
* allow batched and non-batched writes
* rename to accessors_l1_message
* handle error
* add range check
* fix tests
* update comments
* nit
* support blocks with 0 l1 messages
---------
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
* build(docker): auto docker push when pushing git tags (#258 )
* build(docker): update docker trigger tag prefix (#259 )
* Fix L1Message Deep Copy, Complete Bridge Tx Hash test (#269 )
* deep copy value field, add tx hash test
comment
* typo
* commitl1messages
* lint
* Revert "add L1 message sync service"
This reverts commit 5305e8a5de14766ed249e1a7d64042c7a72cf5c2.
* Revert "move bridge client to separate file"
This reverts commit 0b220bee37de93c3250545e23430db2c401a2f90.
* update branch
* use commitMessages for l1Txs
* little fix
* fix config
* fix test
* comment fixes
* fix
* fix config check
---------
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
* Add ErrUnknownAncestor tests (#305 )
add ErrUnknownAncestor tests
* worker test include l1 msgs (#306 )
* worker test include l1 msgs
* move L1 message index update next to block insertion
---------
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* exclude l1 messages from transaction count limit in block (#307 )
* exclude l1 messages from transaction count limit in block
* fix comments
* trigger ci
* nit
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
* Expose queueIndex on Transaction (#316 )
expose queueIndex on Transaction
* test that l1msg doesn't count in maxTxPerBlock limit (#312 )
* test that l1msg doesn't count in maxTxPerBlock limit
* fix, comment
* retrigger ci
* change order inside test
---------
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
* reuse trace nonce field for queueIndex
* expose scroll APIs on the geth console
* add L1 message query APIs
* Trigger new block on new l1 messages (#343 )
* trigger new block on new l1 messages
* typo
* initialize l1MsgCh
* fix worker l1msg tests (#345 )
---------
Co-authored-by: Nazarii Denha <dengaaa2002@gmail.com>
* test(worker): ensure that l1 messages are included in the correct order (#346 )
test that l1msgs added in correct order
* rename enqueueIndex --> queueIndex
* move QueueIndex into transaction
* improve l1 db interface
* formatting
* bump version
* print l1config
* add API to query latest included message queue index
* clean up tx limit logic
* add clarifying comments and todos to ValidateL1Messages
* improve db comments and logs
* clean up L1MessageTx type handling
* format
* format
* improve L1 message block check
* fix missing L1 event handling
* fix TestL1MessageValidationFailure
* simplify sync height resume logic
* make l1Config.l1MessageQueueAddress non-pointer
* improve command line flags
* remove todo
* use abigen tools for log filtering
* cache block L1 message count
* nit: fix variable name case
* improve logs
* flush pending writes to DB before shutdown
---------
Co-authored-by: Nazarii Denha <dengaaa2002@gmail.com>
Co-authored-by: Max Wolff <max@scroll.io>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: Max Wolff <maxcwolff@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
Co-authored-by: HAOYUatHZ <haoyu@protonmail.com>
2023-06-07 07:24:19 -07:00
HAOYUatHZ
32aacc3fd6
style: revert some formatting ( #294 )
...
Revert "style: remove blank line & white space formatting"
This reverts commit 349def8d5d78f83c19e899d5d80b1d9f6a5d18c6.
2023-04-20 17:49:35 +08:00
Max Wolff
a74b35e86f
feat(api): add L1 fee to eth_getTransactionReceipt ( #244 )
...
* Update api.go
* fmt
* revert changes
* add l1fee to receipt in transaction processor
* decoder
* fmt
* fmt
* add encoding, test
* fmt
* removed duplicate l1 fee calc, added gen receipt
* remove dup l1feecalc
* lint
* add omitempty
* bump version
---------
Co-authored-by: Péter Garamvölgyi <th307q@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
Co-authored-by: Péter Garamvölgyi <peter@scroll.io>
2023-03-28 09:08:57 +08:00
maskpp
d1f9523b08
feat(build): Upgrade go version to 1.18 ( #241 )
...
* update go version
* update go version
* format code by goimports
* format code by goimports
* go mod tidy
* update go version
* update go version
* upgrade linter
* add exclude rule
* fix golangci exclude
---------
Co-authored-by: vincent <419436363@qq.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
2023-03-06 13:32:39 +08:00
Steven
b6cc89fabe
Add goimports lint ( #54 )
...
* Add `goimports` lint.
* Fix `goimports` issue.
* Fix group to `github.com/scroll-tech/go-thereum`.
2022-03-24 16:53:59 +08:00
maskpp
69c291cf7a
feat: add more data into the execution trace ( #20 )
2022-02-21 10:15:57 +08:00
maskpp
7745fd5840
Extract execution trace from EVM execution and provide subscription API ( #19 )
...
* Create go.yml
* Merge from zkrollup and fix conflict
* go mod tidy
* fix worker_test test case
* fix worker_test test case
* Delete go.yml
* add log content, enable memory trace
* add tracer pool handler
* Add comments and format code
* add evmTrace subscribe api
* Move the evmTrace struct.
* Fix miner bug.
* upgrade evmTrace api
* fix bug about evmTracesByHash api
* Fix the bug about block.timestamp and remove unnecessary copy.
* Update eth/filters/api.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Upgrade comments.
* Delete useless code in test file
* Update miner/worker.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Change the return result to BlockResult.
* Change return type.
* Change blockResult to blockResults.
* Add ReturnValue.
* Update core/rawdb/l2trace.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Update core/rawdb/l2trace.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Update core/types/l2trace.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Add indent to the comment and rm json encoding flag.
* Rm json encoding flag.
* Update core/rawdb/l2trace.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Rm json encoding flag.
* Update ethclient/ethclient.go
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
* Update eth/filters/api.go
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
* Use as the blockResult prefix flag.
* Update eth/filters/filter_system.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Update eth/filters/filter_system.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Update ethclient/ethclient.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
* Update eth/filters/api.go
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: Haichen Shen <shenhaichen@gmail.com>
Co-authored-by: HAOYUatHZ <37070449+HAOYUatHZ@users.noreply.github.com>
2022-01-23 20:41:15 +08:00
maskpp
b4d60884d2
Fix import fmt and run tests for given architecture ( #18 )
...
* fix gofmt
* Run tests for given architecture
* Speed up CI execution.
2022-01-12 19:45:06 -08:00
Jules
de7ed6af40
Update mod file and all imports to use scroll-tech URL ( #15 )
...
* Update mod file and all imports to use scroll-tech URL
Fixes #14
* Update Discord link
2021-12-24 10:59:22 +01:00
Martin Holst Swende
347c37b362
core/rawdb: use AncientRange when initializing leveldb from freezer ( #23612 )
...
* core/rawdb: utilize AncientRange when initiating from freezer
* core/rawdb: remove debug sanity check
2021-11-23 12:37:26 +01:00
Martin Holst Swende
b7a6409cc1
core/rawdb: better error message in freezer ( #23901 )
...
* core/rawdb: better error message in freezer
* Apply suggestions from code review
2021-11-16 11:33:56 +02:00
Sina Mahmoodi
c57df9ca28
core/rawdb: add slow path for getting legacy logs ( #23879 )
...
* eth/tracers: add slow path for getting legacy logs
* core/rawdb: fix test
2021-11-11 15:04:06 +01:00
rjl493456442
2e8b58f076
cmd/geth: implement data import and export ( #22931 )
...
This PR offers two more database sub commands for exporting and importing data.
Two exporters are implemented: preimage and snapshot data respectively.
The import command is generic, it can take any data export and import into leveldb.
The data format has a 'magic' for disambiguation, and a version field for future compatibility.
2021-11-02 11:31:45 +01:00
Martin Holst Swende
0e7efd696b
core/rawdb, ethdb: introduce batched/atomic reads from ancients ( #23566 )
...
This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.
2021-10-25 16:24:27 +02:00
Sina Mahmoodi
783e97ef1f
core/rawdb: avoid unnecessary receipt processing for log filtering ( #23147 )
...
* core/types: rm extranous check in test
* core/rawdb: add lightweight types for block logs
* core/rawdb,eth: use lightweight accessor for log filtering
* core/rawdb: add bench for decoding into rlpLogs
2021-09-28 12:54:49 +02:00
Martin Holst Swende
794c6133ef
core/rawdb: freezer batch write ( #23462 )
...
This change is a rewrite of the freezer code.
When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.
In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.
To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-07 12:31:17 +02:00
chuwt
5566e5d152
eth/downloader: fix typo in comment ( #23413 )
2021-08-18 13:03:41 +03:00
Martin Holst Swende
5f98020a21
core/rawdb: implement sequential reads in freezer_table ( #23117 )
...
* core/rawdb: implement sequential reads in freezer_table
* core/rawdb, ethdb: add sequential reader to db interface
* core/rawdb: lint nitpicks
* core/rawdb: fix some nitpicks
* core/rawdb: fix flaw with deferred reads not being performed
* core/rawdb: better documentation
2021-08-13 11:51:01 +03:00
Shihao Xia
9e59474e46
core/rawdb: close database in test to avoid goroutine leak ( #23287 )
...
* add db close to avoid goroutine leak
* core/rawdb: move close to defer
Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-08-08 15:44:42 +02:00
Patrick O'Grady
d3e3a460ec
core/rawdb: fix logs to print block number, not address ( #23328 )
2021-08-04 11:10:37 +03:00
Evolution404
248572ee54
core/rawdb: db inspect move 'config' and 'shutdown' into 'meta data' ( #22978 )
...
* core/rawdb: db inspect move 'config' and 'shutdown' into 'meta data'
* gofmt
2021-06-08 11:39:24 +03:00
Evolution404
81662fe827
core/rawdb: handle prefix in table.compact method ( #22911 )
2021-05-21 10:33:59 +02:00
Martin Holst Swende
bb9f9ccf4f
core/rawdb: wait for background freezing to exit when closing freezer ( #22878 )
2021-05-18 01:30:01 +02:00
Péter Szilágyi
8681a2536c
Merge pull request #22777 from karalabe/snapshots-abort-resume-on-sync
...
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:04:05 +03:00
Péter Szilágyi
745757ac6b
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:03:10 +03:00
aaronbuchwald
56f533d00c
docs: fix docstring on read head block ( #22776 )
2021-04-29 19:23:07 +02:00
Péter Szilágyi
fae165a5de
core, eth, ethdb, trie: simplify range proofs
2021-04-29 10:59:08 +03:00
Péter Szilágyi
caea6c4661
eth/protocols/snap: generate storage trie from full dirty snap data ( #22668 )
...
* eth/protocols/snap: generate storage trie from full dirty snap data
* eth/protocols/snap: get rid of some more dead code
* eth/protocols/snap: less frequent logs, also log during trie generation
* eth/protocols/snap: implement dirty account range stack-hashing
* eth/protocols/snap: don't loop on account trie generation
* eth/protocols/snap: fix account format in trie
* core, eth, ethdb: glue snap packets together, but not chunks
* eth/protocols/snap: print completion log for snap phase
* eth/protocols/snap: extended tests
* eth/protocols/snap: make testcase pass
* eth/protocols/snap: fix account stacktrie commit without defer
* ethdb: fix key counts on reset
* eth/protocols: fix typos
* eth/protocols/snap: make better use of delivered data (#44 )
* eth/protocols/snap: make better use of delivered data
* squashme
* eth/protocols/snap: reduce chunking
* squashme
* eth/protocols/snap: reduce chunking further
* eth/protocols/snap: break out hash range calculations
* eth/protocols/snap: use sort.Search instead of looping
* eth/protocols/snap: prevent crash on storage response with no keys
* eth/protocols/snap: nitpicks all around
* eth/protocols/snap: clear heal need on 1-chunk storage completion
* eth/protocols/snap: fix range chunker, add tests
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* trie: fix test API error
* eth/protocols/snap: fix some further liter issues
* eth/protocols/snap: fix accidental batch reuse
Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-04-27 17:19:59 +03:00
Martin Holst Swende
9b99e3dfe0
core/rawdb: fix datarace in freezer ( #22728 )
...
The Append / truncate operations were racy. When a datafile reaches 2Gb, a new file is needed. For this operation, we require a writelock, which is not needed in the 99.99% of all cases where the data does fit in the current head-file.
This transition from readlock to writelock was incorrect, and as the readlock was released, a truncate operation could slip in between, and truncate the data. This would have been fine, however, the Append operation continued writing as if no truncation had occurred, e.g writing item 5 where item 0 should reside.
This PR changes the behaviour, so that if when we run into the situation that a new file is needed, it aborts, and retries, this time with a writelock.
The outcome of the situation described above, running on this PR, would instead be that the Append operation exits with a failure.
2021-04-26 18:19:07 +02:00
Martin Holst Swende
271e5b7fc9
cmd/geth: add db-command to inspect freezer index ( #22633 )
...
This PR makes it easier to inspect the freezer index, which could be useful to investigate things like #22111
2021-04-13 15:45:30 +02:00
gary rong
0c70b83e00
all: add read-only option to database ( #22407 )
...
* all: add read-only option to database
* all: fixes tests
* cmd/geth: migrate flags
* cmd/geth: fix the compact
* cmd/geth: fix the format
* cmd/geth: fix log
* cmd: add chain-readonly
* core: add readonly notion to freezer
* core/rawdb: add log
* core/rawdb: fix freezer close
* cmd: fix
* cmd, core: construct db
* core: update tests
2021-03-22 19:06:30 +01:00
Martin Holst Swende
94ab4ea341
core/rawdb: fix transaction indexing/unindexing hashing error ( #22457 )
...
* core/rawdb: more verbose error logs + better hashing
* core/rawdb: add failing testcase
* core/rawdb: properly hash transactions while indexing/unindexing
* core/rawdb: exit on error + better log msg
2021-03-16 12:15:14 +02:00
gary rong
19d7a37abb
core/rawdb: fix the transaction indexer ( #22395 )
2021-03-01 11:26:10 +02:00
Martin Holst Swende
c4a2b682ff
cmd/geth: add db commands stats, compact, put, get, delete ( #22014 )
...
This PR introduces:
- db.put to put a value into the database
- db.get to read a value from the database
- db.delete to delete a value from the database
- db.stats to check compaction info from the database
- db.compact to trigger a db compaction
It also moves inspectdb to db.inspect.
2021-02-23 11:27:32 +01:00
gary rong
f566dd305e
all: bloom-filter based pruning mechanism ( #21724 )
...
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 13:16:30 +02:00
Martin Holst Swende
681618275c
core: speed up header import ( #21967 )
...
This PR implements the following modifications
- Don't shortcut check if block is present, thus avoid disk lookup
- Don't check hash ancestry in early-check (it's still done in parallel checker)
- Don't check time.Now for every single header
Charts and background info can be found here: https://github.com/holiman/headerimport/blob/main/README.md
With these changes, writing 1M headers goes down to from 80s to 62s.
2021-01-26 12:17:11 +01:00
gary rong
5a1b384352
core: persist bad blocks ( #21827 )
...
* core: persist bad blocks
* core, eth, internal: address comments
* core/rawdb: add badblocks to inspector
* core, eth: update
* internal: revert
* core, eth: only save 10 bad blocks
* core/rawdb: address comments
* core/rawdb: fix
* core: address comments
2021-01-10 12:54:15 +01:00
ucwong
a425a47ddc
core/rawdb, eth/protocols : Method name typo fix ( #22026 )
2020-12-27 22:38:16 +01:00
Péter Szilágyi
017831dd5b
core, eth: split eth package, implement snap protocol ( #21482 )
...
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md ) , but does not enable it by default.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Martin Holst Swende
4d48980e74
core, eth, les: implement unclean-shutdown marker ( #21893 )
...
This PR implements unclean shutdown marker. Every time geth boots, it adds a timestamp to a list of timestamps in the database. This list is capped at 10. At a clean shutdown, the timestamp is removed again.
Thus, when geth exits unclean, the marker remains, and at boot up we show the most recent unclean shutdowns to the user, which makes it easier to diagnose root-causes to certain problems.
Co-authored-by: Nagy Salem <me@muhnagy.com>
2020-12-11 15:56:00 +01:00
