mirror of
https://github.com/ethereum/go-ethereum.git
synced 2026-03-27 05:12:58 +00:00
d5efd34010
It's a PR based on #33303 and introduces an approach for trienode history indexing. --- In the current archive node design, resolving a historical trie node at a specific block involves the following steps: - Look up the corresponding trie node index and locate the first entry whose state ID is greater than the target state ID. - Resolve the trie node from the associated trienode history object. A naive approach would be to store mutation records for every trie node, similar to how flat state mutations are recorded. However, the total number of trie nodes is extremely large (approximately 2.4 billion), and the vast majority of them are rarely modified. Creating an index entry for each individual trie node would be very wasteful in both storage and indexing overhead. To address this, we aggregate multiple trie nodes into chunks and index mutations at the chunk level instead. --- For a storage trie, the trie is vertically partitioned into multiple sub tries, each spanning three consecutive levels. The top three levels (1 + 16 + 256 nodes) form the first chunk, and every subsequent three-level segment forms another chunk. ``` Original trie structure Level 0 [ ROOT ] 1 node Level 1 [0] [1] [2] ... [f] 16 nodes Level 2 [00] [01] ... [0f] [10] ... [ff] 256 nodes Level 3 [000] [001] ... [00f] [010] ... [fff] 4096 nodes Level 4 [0000] ... [000f] [0010] ... [001f] ... [ffff] 65536 nodes Vertical split into chunks (3 levels per chunk) Level0 [ ROOT ] 1 chunk Level3 [000] ... [fff] 4096 chunks Level6 [000000] ... [fffffff] 16777216 chunks ``` Within each chunk, there are 273 nodes in total, regardless of the chunk's depth in the trie. ``` Level 0 [ 0 ] 1 node Level 1 [ 1 ] … [ 16 ] 16 nodes Level 2 [ 17 ] … … [ 272 ] 256 nodes ``` Each chunk is uniquely identified by the path prefix of the root node of its corresponding sub-trie. Within a chunk, nodes are identified by a numeric index ranging from 0 to 272. For example, suppose that at block 100, the nodes with paths `[]`, `[0]`, `[f]`, `[00]`, and `[ff]` are modified. The mutation record for chunk 0 is then appended with the following entry: `[100 → [0, 1, 16, 17, 272]]`, `272` is the numeric ID of path `[ff]`. Furthermore, due to the structural properties of the Merkle Patricia Trie, if a child node is modified, all of its ancestors along the same path must also be updated. As a result, in the above example, recording mutations for nodes `00` and `ff` alone is sufficient, as this implicitly indicates that their ancestor nodes `[]`, `[0]` and `[f]` were also modified at block 100. --- Query processing is slightly more complicated. Since trie nodes are indexed at the chunk level, each individual trie node lookup requires an additional filtering step to ensure that a given mutation record actually corresponds to the target trie node. As mentioned earlier, mutation records store only the numeric identifiers of leaf nodes, while ancestor nodes are omitted for storage efficiency. Consequently, when querying an ancestor node, additional checks are required to determine whether the mutation record implicitly represents a modification to that ancestor. Moreover, since trie nodes are indexed at the chunk level, some trie nodes may be updated frequently, causing their mutation records to dominate the index. Queries targeting rarely modified trie nodes would then scan a large amount of irrelevant index data, significantly degrading performance. To address this issue, a bitmap is introduced for each index block and stored in the chunk's metadata. Before loading a specific index block, the bitmap is checked to determine whether the block contains mutation records relevant to the target trie node. If the bitmap indicates that the block does not contain such records, the block is skipped entirely.
83 lines
2.3 KiB
Go
83 lines
2.3 KiB
Go
// Copyright 2025 The go-ethereum Authors
|
|
// This file is part of the go-ethereum library.
|
|
//
|
|
// The go-ethereum library is free software: you can redistribute it and/or modify
|
|
// it under the terms of the GNU Lesser General Public License as published by
|
|
// the Free Software Foundation, either version 3 of the License, or
|
|
// (at your option) any later version.
|
|
//
|
|
// The go-ethereum library is distributed in the hope that it will be useful,
|
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
// GNU Lesser General Public License for more details.
|
|
//
|
|
// You should have received a copy of the GNU Lesser General Public License
|
|
// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
package pathdb
|
|
|
|
import (
|
|
"encoding/binary"
|
|
"fmt"
|
|
"slices"
|
|
)
|
|
|
|
// commonPrefixLen returns the length of the common prefix shared by a and b.
|
|
func commonPrefixLen(a, b []byte) int {
|
|
n := min(len(a), len(b))
|
|
for i := range n {
|
|
if a[i] != b[i] {
|
|
return i
|
|
}
|
|
}
|
|
return n
|
|
}
|
|
|
|
// encodeIDs sorts the given list of uint16 IDs and encodes them into a
|
|
// compact byte slice using variable-length unsigned integer encoding.
|
|
func encodeIDs(ids []uint16) []byte {
|
|
slices.Sort(ids)
|
|
buf := make([]byte, 0, len(ids))
|
|
for _, id := range ids {
|
|
buf = binary.AppendUvarint(buf, uint64(id))
|
|
}
|
|
return buf
|
|
}
|
|
|
|
// decodeIDs decodes a sequence of variable-length encoded uint16 IDs from the
|
|
// given byte slice and returns them as a set.
|
|
//
|
|
// Returns an error if the input buffer does not contain a complete Uvarint value.
|
|
func decodeIDs(buf []byte) ([]uint16, error) {
|
|
var res []uint16
|
|
for len(buf) > 0 {
|
|
id, n := binary.Uvarint(buf)
|
|
if n <= 0 {
|
|
return nil, fmt.Errorf("too short for decoding node id, %v", buf)
|
|
}
|
|
buf = buf[n:]
|
|
res = append(res, uint16(id))
|
|
}
|
|
return res, nil
|
|
}
|
|
|
|
// isAncestor reports whether node x is the ancestor of node y.
|
|
func isAncestor(x, y uint16) bool {
|
|
for y > x {
|
|
y = (y - 1) / 16 // parentID(y) = (y - 1) / 16
|
|
if y == x {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// isBitSet reports whether the bit at `index` in the byte slice `b` is set.
|
|
func isBitSet(b []byte, index int) bool {
|
|
return b[index/8]&(1<<(7-index%8)) != 0
|
|
}
|
|
|
|
// setBit sets the bit at `index` in the byte slice `b` to 1.
|
|
func setBit(b []byte, index int) {
|
|
b[index/8] |= 1 << (7 - index%8)
|
|
}
|