go-ethereum/cmd
Jonathan Oppenheimer bc1967f088
core/state/snapshot: snapshot generation shutdown race condition (#33540)
## Overview

This PR fixes a race condition during blockchain shutdown where snapshot
generation could continue accessing the trie database after it has been
closed, leading to iterator errors. We noticed this in one of our nodes
on https://github.com/ava-labs/avalanchego, which relies on an older
version of geth with the same issue (so this behavior does happen!).

During node shutdown, the following sequence occurs:

1. `BlockChain.Stop()` calls `snaps.Release()` to clean up snapshot
resources
2. `Release()` only resets the cache but doesn't stop the generator
goroutine
3. The trie database is then closed via `triedb.Close()`
4. The still-running generator attempts to iterate storage tries
5. Iterator fails because the database is closed (`"Generator failed to
iterate storage trie"`)

## Problem

There are three related bugs:

1. `Release()` doesn't stop generation: The `diskLayer.Release()` method
only resets the cache without stopping ongoing snapshot generation,
leaving the generator goroutine running after database closure.
2. `stopGeneration()` has an incorrect completion check: The
`stopGeneration()` method checks `genMarker != nil` to determine if
generation is running. However, `genMarker` is set to nil when
generation completes successfully, even though the generator goroutine
is still waiting for the abort signal at the end of `generate()`. See
line 705 in `generate.go`:
eaaa5b716d/core/state/snapshot/generate.go (L699-L707)
This means `stopGeneration()` returns early without sending the abort
signal.
3. Node shutdown doesn't stop generation: During shutdown, no code path
calls `stopGeneration()` or sends the abort signal to the generator,
causing the generator to access a closed database and error.

## Fix

- Modified `diskLayer.Release()` to call `stopGeneration()` before
releasing resources
- Added cancelation architecture, removing reliance on someone having to
wait
- Fixed `stopGeneration()` to properly and safely stop snapshot
generation
- Added `TestGenerateGoroutineLeak` to verify the fix and prevent
regression. The test fails without the fix and passes with it.
- The test creates a snapshot with active generation, waits for
completion, then calls `Release()`, and uses `go.uber.org/goleak` to
assert no generator goroutine survives.
- Without the fix, the test fails: `Release()` returns without stopping
the generator, which stays parked at `generate.go:705` waiting for an
abort signal that never comes:

    ```
    --- FAIL: TestGenerateGoroutineLeak (0.88s)
        generate_test.go: found unexpected goroutines:
        [Goroutine 6 in state chan receive, with
         core/state/snapshot.(*diskLayer).generate on top of the stack:
         core/state/snapshot.(*diskLayer).generate(...)
            core/state/snapshot/generate.go:705
         created by core/state/snapshot.generateSnapshot
            core/state/snapshot/generate.go:79 ]
    ```
- With the fix, the test passes: `Release()` -> `stopGeneration()`
blocks until the generator goroutine has fully exited, so nothing leaks

Note that this fix follows the same pattern used in `Tree.Disable()` in
https://github.com/ethereum/go-ethereum/pull/30040, which introduced
`stopGeneration()` for use in `Disable()` and `Rebuild()` but didn't
address the shutdown path.

The test follows the same pattern used in
`TestCheckSimBackendGoroutineLeak`
2026-06-04 21:22:58 -05:00
..
abidump all: update license headers and AUTHORS from git history (#24947) 2022-05-24 20:39:40 +02:00
abigen cmd/abigen: respect --v2=false (#34943) 2026-05-12 09:02:40 -04:00
blsync beacon/blsync: add checkpoint import/export file feature (#31469) 2025-04-03 16:04:11 +02:00
devp2p cmd/devp2p: fix typo in ENR IP printing code (#34909) 2026-05-09 15:59:03 +02:00
era internal/era: update to latest ere spec (#34896) 2026-06-03 11:52:10 +02:00
ethkey cmd: fix some typos in readmes (#29405) 2024-04-11 14:06:49 +03:00
evm trie/bintrie: record inserted leaves for t8n (#34843) 2026-05-28 17:06:47 +02:00
fetchpayload cmd/fetchpayload: add payload-building utility (#33919) 2026-03-11 16:18:42 +01:00
geth internal/era: update to latest ere spec (#34896) 2026-06-03 11:52:10 +02:00
keeper core/state/snapshot: snapshot generation shutdown race condition (#33540) 2026-06-04 21:22:58 -05:00
rlpdump build: update to golangci-lint 1.61.0 (#30587) 2024-10-14 19:25:22 +02:00
utils cmd/utils: avoid extra newlines when reading era checksums (#35104) 2026-06-03 11:48:45 -06:00
workload core/history: refactor pruning configuration (#34036) 2026-03-18 13:54:29 +01:00