go-ethereum/cmd/keeper
Jonathan Oppenheimer bc1967f088
core/state/snapshot: snapshot generation shutdown race condition (#33540)
## Overview

This PR fixes a race condition during blockchain shutdown where snapshot
generation could continue accessing the trie database after it has been
closed, leading to iterator errors. We noticed this in one of our nodes
on https://github.com/ava-labs/avalanchego, which relies on an older
version of geth with the same issue (so this behavior does happen!).

During node shutdown, the following sequence occurs:

1. `BlockChain.Stop()` calls `snaps.Release()` to clean up snapshot
resources
2. `Release()` only resets the cache but doesn't stop the generator
goroutine
3. The trie database is then closed via `triedb.Close()`
4. The still-running generator attempts to iterate storage tries
5. Iterator fails because the database is closed (`"Generator failed to
iterate storage trie"`)

## Problem

There are three related bugs:

1. `Release()` doesn't stop generation: The `diskLayer.Release()` method
only resets the cache without stopping ongoing snapshot generation,
leaving the generator goroutine running after database closure.
2. `stopGeneration()` has an incorrect completion check: The
`stopGeneration()` method checks `genMarker != nil` to determine if
generation is running. However, `genMarker` is set to nil when
generation completes successfully, even though the generator goroutine
is still waiting for the abort signal at the end of `generate()`. See
line 705 in `generate.go`:
eaaa5b716d/core/state/snapshot/generate.go (L699-L707)
This means `stopGeneration()` returns early without sending the abort
signal.
3. Node shutdown doesn't stop generation: During shutdown, no code path
calls `stopGeneration()` or sends the abort signal to the generator,
causing the generator to access a closed database and error.

## Fix

- Modified `diskLayer.Release()` to call `stopGeneration()` before
releasing resources
- Added cancelation architecture, removing reliance on someone having to
wait
- Fixed `stopGeneration()` to properly and safely stop snapshot
generation
- Added `TestGenerateGoroutineLeak` to verify the fix and prevent
regression. The test fails without the fix and passes with it.
- The test creates a snapshot with active generation, waits for
completion, then calls `Release()`, and uses `go.uber.org/goleak` to
assert no generator goroutine survives.
- Without the fix, the test fails: `Release()` returns without stopping
the generator, which stays parked at `generate.go:705` waiting for an
abort signal that never comes:

    ```
    --- FAIL: TestGenerateGoroutineLeak (0.88s)
        generate_test.go: found unexpected goroutines:
        [Goroutine 6 in state chan receive, with
         core/state/snapshot.(*diskLayer).generate on top of the stack:
         core/state/snapshot.(*diskLayer).generate(...)
            core/state/snapshot/generate.go:705
         created by core/state/snapshot.generateSnapshot
            core/state/snapshot/generate.go:79 ]
    ```
- With the fix, the test passes: `Release()` -> `stopGeneration()`
blocks until the generator goroutine has fully exited, so nothing leaks

Note that this fix follows the same pattern used in `Tree.Disable()` in
https://github.com/ethereum/go-ethereum/pull/30040, which introduced
`stopGeneration()` for use in `Disable()` and `Rebuild()` but didn't
address the shutdown path.

The test follows the same pattern used in
`TestCheckSimBackendGoroutineLeak`
2026-06-04 21:22:58 -05:00
..
1192c3_block.rlp cmd/keeper: add the keeper zkvm guest program (#32543) 2025-09-15 19:47:41 +02:00
1192c3_witness.rlp cmd/keeper: add the keeper zkvm guest program (#32543) 2025-09-15 19:47:41 +02:00
chainconfig.go cmd/keeper: add the keeper zkvm guest program (#32543) 2025-09-15 19:47:41 +02:00
getpayload_example.go build, cmd/keeper: add "womir" target (#34079) 2026-03-28 11:39:44 +01:00
getpayload_wasm.go build, cmd/keeper: add "womir" target (#34079) 2026-03-28 11:39:44 +01:00
getpayload_womir.go build, cmd/keeper: add "womir" target (#34079) 2026-03-28 11:39:44 +01:00
getpayload_ziren.go cmd/keeper: use the ziren keccak precompile (#32816) 2025-10-20 11:52:02 +02:00
go.mod go.mod: bump go.opentelemetry.io from 1.40.0 to 1.41.0 (#35073) 2026-05-29 22:51:26 +02:00
go.sum core/state/snapshot: snapshot generation shutdown race condition (#33540) 2026-06-04 21:22:58 -05:00
main.go eth/catalyst: add initial OpenTelemetry tracing for newPayload (#33521) 2026-02-17 17:08:57 +01:00
README.md cmd/keeper: add the keeper zkvm guest program (#32543) 2025-09-15 19:47:41 +02:00
stubs.go build, cmd/keeper: add "womir" target (#34079) 2026-03-28 11:39:44 +01:00

Keeper - geth as a zkvm guest

Keeper command is a specialized tool for validating stateless execution of Ethereum blocks. It's designed to run as a zkvm guest.

Overview

The keeper reads an RLP-encoded payload containing:

  • A block to execute
  • A witness with the necessary state data
  • A chainID

It then executes the block statelessly and validates that the computed state root and receipt root match the values in the block header.

Building Keeper

The keeper uses build tags to compile platform-specific input methods and chain configurations:

Example Implementation

See getpayload_example.go for a complete example with embedded Hoodi block data:

# Build example with different chain configurations
go build -tags "example" ./cmd/keeper

Ziren zkVM Implementation

Build for the Ziren zkVM platform, which is a MIPS ISA-based zkvm:

GOOS=linux GOARCH=mipsle GOMIPS=softfloat go build -tags "ziren" ./cmd/keeper

As an example runner, refer to https://gist.github.com/gballet/7b669a99eb3ab2b593324e3a76abd23d

Creating a Custom Platform Implementation

To add support for a new platform (e.g., "myplatform"), create a new file with the appropriate build tag:

1. Create getinput_myplatform.go

//go:build myplatform

package main

import (
    "github.com/ethereum/go-ethereum/params"
    // ... other imports as needed
)

// getInput returns the RLP-encoded payload
func getInput() []byte {
    // Your platform-specific code to retrieve the RLP-encoded payload
    // This might read from:
    // - Memory-mapped I/O
    // - Hardware registers
    // - Serial port
    // - Network interface
    // - File system

    // The payload must be RLP-encoded and contain:
    // - Block with transactions
    // - Witness with parent headers and state data

    return encodedPayload
}