This adds warning logs when the read does not match the expected count.
We can also remove the size limit since the function documentation explicitly states
that callers should limit the count.
* cmd, core: resolve scheme from a read-write database
* cmd, core, eth: move the scheme check in the ethereum constructor
* cmd/geth: dump should in ro mode
* cmd: reverts
* Add CLI flags to config LevelDB table/total sizes
I wired up CLI flags to allow configuring LevelDB table and total sizes:
- `--leveldb.compaction.table.size`, LevelDB SSTable file size factor in MiB (default: 2)
- `--leveldb.compaction.table.multiplier`, multiplier on LevelDB SSTable file size (default: 1)
- `--leveldb.compaction.total.size`, total size factor in MiB of LevelDB levels (default: 10)
- `--leveldb.compaction.total.multiplier`, multiplier on LevelDB total level size (default: 10)
N.B. that the default values for these configs are exactly the same as
before this changset and so Bor behavior should not change unless these
flags are deliberately overridden. Bor/Geth inherited the default values
from [the `goleveldb`
defaults](126854af5e/leveldb/opt/options.go).
We (Alchemy) found it necessary to override these configs as follows to
keep Bor archive nodes tracking the canonical chain:
- `--leveldb.compaction.table.size=4`
- `--leveldb.compaction.total.size=20`
These overrides double the size of LevelDB SSTable files (2 MiB -> 4
MiB) and also the total amount of data in each level (100 MiB -> 200
MiB, 1,000 MiB -> 2,000 MiB, etc.). The idea is to have LevelDB read and
write data in larger chunks while keeping the proportional frequency of
compaction operations the same as in the original defaults defined by
Dean and Ghemawat.
Without these overrides we found that our archive nodes would tend to
fall into a "LevelDB compaction loop of death" where the incoming stream
of blockchain data could not be flowed into LevelDB's structure quickly
enough, resulting in the node blocking writes for long periods of time
while LevelDB's single-threaded compaction organized the data. Over
time the nodes would fall farther and farther behind the canonical chain
head, metaphorically dying a slow node's death.
These configs can be changed on existing node databases (resyncing is
not necessary). LevelDB appears to work correctly with SSTable files of
different sizes. Note that the database does not undergo any sort of
migration when changing these configs. Only newly-written files (due to
new data or compaction) are affected by these configs.
* Update docs
* Adjust line spacing for linter
* Replace map with `ExtraDBConfig`
* Rename `LevelDbConfig` to `ExtraDBConfig`
* Regenerate docs
* Milestone Implementation
* Merge branch 'POS-347' into reciept-e2e-test
* Changes for testing, will be removed after testing
* Debugged the error
* Just for testing purpose
* refactor debug api methods, rename whitelist -> checkpoint
* remove first iteration based vars
* fix linters
* Rewind Changes
* Error changes
* RewindBack function in bor_checkpoint_verifier
* Testcases
* Added the fetch test for milestone and checkpoint
* Debugged the lint changes
* Debugged the lint changes
* Debugged the lint changes
* Debugged the lint changes
* Improved the error in Miner test file
* Improved the error of pointing to the wrong function
* Locking the sprint after the vote has been made on it.
* Adding more logs for testing
* Adding more logs for testing
* Adding more logs for testing
* Implemented the NoAckMilestone fetching mechanism
* Testcases for milestone implementation
* Testing code for fetchNoAckMilestone and fetchLastNoAckMilestone
* Testing changes
* refactor else-if
* Corrected the number of params in bor_ext.go
* Dummy API for testing
* Defined the GetVoteOnRootHash in interface
* Defined the GetVoteOnRootHash in interface
* Made changes in the web3ext file
* Added the GetVoteOnRootHash in PublicBlockChain API
* Added the GetVoteOnRootHash in filterBackend
* Added the log of Root and RooHash
* Removed the 0x from rootHash
* Just for testing purpose
* "GetVoteOnRootHash" mock implementation
* bor_test.go
* Added the test for milestone implementation
* Added service for fetching milestone by ID
* Improved the comments
* Removed the duplicate code
* use setter for borVerifier
* use setter for borVerifier
* refactor handleNoAckMilestone
* remove code repetition with retry function
* Converged the repetitive code
* after CR
* persistence
* persistence implementation
* feature flag
* Persistence Changes
* cr
* initial
* fix
* fix
* Whitelist Flag
* 1 Add:Included the milestone flag 2.Add:Hardlimit the rewind to maximum of 255 blocks
* Chg:Updated go.mod file
* Remove:Dubai Hardfork code
* Add:checked errors for call functions to the Db, Rmv: Remote Header variable from the IsValidPeer() function
* Fix:Linting issues'
* Add:MilestoneGRPC functions
* Fix:Lint issues
* Fix:Lint issues
* Fix: TestFetchMilestoneFromMockHeimdall
* Fix:Integrations tests
* Add:Test for sprint length and milestone changes
* Add:Functionality to fetch the finalized block
* Chg:Changed default val of TriesInmemory to 1024
* fix:Some functions of heimdallGRPC client
* Restored the GRPC functionality, was commented out for developing purpose
* Fix:Bor_checkpoint_Verfier function
* Test:Added the chain Rewinding test
* Test:Added the Sprint Length + Milestone merge test
* Add:Implemented the future milestone
* Add:Future milestone changes
* Add:Future milestone changes
* Chg: Voting on endBlockHash rather than rootHash
* Chg: Changed the logic of future milestone from rootHash checking to endBlockHash checking
* Fix:Using endBockHash while verifying the incoming milestone
* Chg:Variable names for better readiblity
* Fix:Testing changes
* Add:metrics for milestone implementation
* Add:Metrics for milestone implementatian
* Fix:Order of statements in a function for better optimization
* Chg:Removed unrequired file
* Fix:new variable intialization
* Add:Comment to increase readiblity
* Fix:Logs
* Chg:Name of GetVoteOnRootHash to GetVoteOnHash
* Fix:Linting issues
* Fixed linting issues
* Rmv: Unnecessary logs and Add:Skip test for long tests
* Fix:Checking current chain with whitelisted milestone or checkpoint in Finalized block function
* Fix:Test
* Fix:Whitelisting of Milestone and Checkpoint process
* Fix: Milestone JSON structure
* Chg:Testcases changes
* Fix:Change from VoteOnRootHash to VoteOnHash
* Fix:Variable name fix
* Fix:Finalized API
* internal/jsre/deps: update web3.js bundle
* Fix:milestone verifier
* Chg:Handling the long future chain import issue
* Fix:Lint issues
* Fix:TestLowDiffLongChain and TestPrunedImportSide tests, used hardcoded value 128 instead of DefaultTriesInMemory value
* Chg:Testcode for producing metrics
* Chg:Milestong polling value to 32 secs
* Add:Testcases
* Add:Implemented the check to fetch the milestoneId from heimdall before locking the fork
* Added GRPC method for FetchMilestoneID
* Fix:lint issue
* Fix:lint issue
* Skiped out the tests which were mainly used to produce the supporting data
* remove vcs build when running snyk
* Add:Improved the logs and comments
* fix linters
* Skipped some test as they are panic due to timeout issue in github
* Chg:Variable name LockerSprintNumber to LockedMilestoneNumber for better readablity and clarity
* Chg:Conflicting variable names in milestone test file
* Chg:Conflicting function names in milestone test file
* fix : minor fix in TestInsertingSpanSizeBlocks
* Fix:Mocking issue in TestInsertingSpanSizeBlocks
* Fix:GRPC Polyproto Version
* eth/downloader: skip peer drop due to whitelisting err
* eth, tests/bor: bug fixes and minor refactor
* Add:Implemented the milestone related functions in the HeimdallApp
* Fix:Lint Errors & Remove:Redundant Code
* Fix:Testing Errors
* Fix:Bor integeration tests
* Fix:Test errors
* update heimdall client mock files
* remove unused arguments
* remove redundant code
* Chg:Changed the milestone polling intervals
* Add: added block finality from whitelisted checkpoint
* skip future chain validation
* Add:confirmation check of 16 blocks over the end block while voting for the milestone in GetVoteHash() function
* Chg:Included endBlockNum in UnlockMutex function
* Add:Property based test for milestone
* Fix:Opening the lock while processing future milestone
* Add:Property based test for futureMilestone
* Defined the value of TempTriesInMemory
* Fixed the finalized api
* Fixed lint issues
* eth: add logs while fetching and rewinding
* fix linters: use default returns instead of recursive calls
* Fix:Milestone intergration test
* Add:GetVoteHash fn in mock backend
* tests/bor: fix mock span
* tests/bor: remove t.Parallel()
* use bor namespace in ethclient, fix mock function
---------
Co-authored-by: Vaibhav Jindal <vaibhavjindal29@gmail.com>
Co-authored-by: VaibhavJindal <74560896+VAIBHAVJINDAL3012@users.noreply.github.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: Evgeny Danienko <6655321@bk.ru>
Co-authored-by: Shivam Sharma <shivam691999@gmail.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
This change removes a chainconfig parameter passed into rawdb.ReadLogs, which is not used nor needed.
It also modifies the filter loop slightly, avoiding a labeled break and instead using a method.
This change does not modify any behaviour.
* all: implement path-based state scheme
* all: edits from review
* core/rawdb, trie/triedb/pathdb: review changes
* core, light, trie, eth, tests: reimplement pbss history
* core, trie/triedb/pathdb: track block number in state history
* trie/triedb/pathdb: add history documentation
* core, trie/triedb/pathdb: address comments from Peter's review
Important changes to list:
- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated
* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache
* core/rawdb: fix tests
* trie/triedb/pathdb: rename metrics, change clean cache key
* trie/triedb: manage the clean cache inside of disk layer
* trie/triedb/pathdb: move journal function
* trie/triedb/path: fix tests
* trie/triedb/pathdb: fix journal
* trie/triedb/pathdb: fix history
* trie/triedb/pathdb: try to fix tests on windows
* core, trie: address comments
* trie/triedb/pathdb: fix test issues
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/types: add data gas fields in Receipt
* core/types: use BlobGas method of tx
* core: fix test
* core/types: fix receipt tests, add data gas used field test
---------
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change adds the ability to perform reads from freezer without size limitation. This can be useful in cases where callers are certain that out-of-memory will not happen (e.g. reading only a few elements).
The previous API was designed to behave both optimally and secure while servicing a request from a peer, whereas this change should _not_ be used when an untrusted peer can influence the query size.
