Commit graph

4 commits

Author SHA1 Message Date
Emualliug Tellab
2ee9408be5 trie/archiver: streaming archival + pathdb journal consistency (#567)
* trie/archiver: streaming subtree archival to fix OOM

Replace the recursive approach that loaded the entire trie into memory
with a streaming NodeIterator-based approach:

- processTrie now uses NodeIterator to walk the trie node-by-node
- probeHeight reads nodes from raw DB, computes height bounded at 3,
  and discards decoded nodes immediately (no in-memory trie buildup)
- collectSubtree only materializes the bounded height-3 subtree being
  archived (at most ~4096 nodes)
- Memory usage: O(iterator_stack) + O(current_subtree) instead of
  O(entire_trie)

This fixes OOM kills on large storage tries (e.g. contracts with
millions of storage slots) where the previous approach would load
all nodes and subtreeInfo into memory before archiving any of them.

* cmd/geth: flush diff layers before archiving, re-journal after

Instead of deleting the pathdb journal after archive generation (which
breaks the chain head and prevents block imports), properly integrate
with pathdb:

1. Open triedb with pathdb support (not just raw KV)
2. Disable state history freezer (avoid append gaps)
3. Flush all diff layers to disk via Commit() before archiving
4. After archiving, re-journal the pathdb state (disk layer only)

This ensures geth can restart cleanly after archiving and continue
importing blocks without 'unknown ancestor' errors.

* trie: add resurrection timing and depth metrics to expired node resolution

* Update trie/archiver.go

---------

Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
2026-06-11 13:02:04 +02:00
Guillaume Ballet
5119945e25 trie, cmd/geth: add archive verify command, Walk(), and archiver improvements
Add Trie.Walk() for exhaustive traversal that resolves expired nodes
with hash verification. Add `archive verify` subcommand that walks
the full state (account + storage tries) to validate all archived
data can be correctly resurrected.

Delete both the journal KV entry and file after archiving to force
geth to restart with a bare disk layer, rewinding the chain head to
the persistent disk state and re-executing blocks.

Also adds markSubtreeDirty() to resolveExpiredNodeData() so that all
nodes in a resolved expired subtree are captured in the NodeSet during
commit — preventing them from being lost between diff layers and the
raw DB.
2026-06-11 13:02:02 +02:00
Guillaume Ballet
68621ae05a a few fixes after hoodi testing 2026-06-11 13:01:00 +02:00
Guillaume Ballet
d82e83d2d9 trie, cmd/geth: add archiver command 2026-06-11 13:01:00 +02:00