go-ethereum/eth/tracers/firehose_concurrency_2.md

Report 2: Concurrent Block Flushing in Tracer Processing System

Section 1: Introduction

Currently, the tracer processing system operates in a linear fashion, requiring the complete processing of a block's code before proceeding to the next one. As a result, computationally expensive operations—such as proto.Marshal and base64 encoding, which are essential for flushing a block to the firehose—can introduce significant delays. To mitigate this bottleneck, a potential solution is to leverage concurrency by introducing goroutines. These heavy operations can be offloaded to a separate channel, enabling asynchronous processing. This approach allows the main execution flow to begin processing subsequent blocks while previous ones are being flushed, thereby improving overall throughput and reducing latency. The goal of this report is to outline the solution and provide benchmarking results.

Section 2: Background

proto.Marshal, part of Protocol Buffers (protobuf), serializes structured data into a compact binary format. While efficient in output size, the process of traversing and encoding complex data structures—especially those with deeply nested or repeated fields—can be CPU-intensive. Additionally, memory allocations during marshaling can introduce further overhead.

Similarly, base64 encoding, which transforms binary data into an ASCII string format for transmission or storage, involves non-trivial byte-wise transformations and increases the data size. This added computational cost becomes significant when processing large blocks or high-throughput workloads.

Together, these operations introduce latency in a linear processing pipeline.

Section 3: Method

3.1 Proposed Solution

A critical point in the tracer processing system is the OnBlockEnd hook, which invokes the printBlockToFirehose method. This method includes computationally expensive operations such as proto.Marshal and base64 encoding, which are necessary to serialize and flush the block data to the firehose.

To address this, the proposed solution introduces a concurrency mechanism. Specifically, the user specifies a number of worker goroutines that will be concurrently working through the FirehoseConfig.ConcurrencyBlockFlushing configuration. There are three layers to this process:

1. A worker queue channel with a buffer of 100 is created. All printBlockToFirehose tasks are enqueued while waiting for a worker to take and process it.
2. A number of workers specified by the configuration will be taking and processing these tasks asynchronously until the data is in a byte format and ready to be sent to stdout. The goroutine will then send that data, along with the block number, to a second channel.
3. A final channel is created to store pairs of (block number, [byte]). This channel allows linear flushing by only allowing the current expected block number to be flushed, while storing the remaining blocks until it is their turn. To achieve this, a block number is stored globally the first time OnBlockEnd is called. Therefore, the channel will know the first block number expected to be flushed and will increment the expected number by one each time.

3.2 Validation and Performance Metrics

The implementation was validated at multiple levels to ensure correctness, configurability, and performance improvements of the concurrent block flushing mechanism.

1. Unit-Level Validation

   To confirm the functional correctness of the concurrent flushing implementation, a unit test was written that creates and processes 1,000 blocks, flushing each to an InternalTestingBuffer. The results were then compared against expected outputs to verify equivalence. The test confirms that the output produced by the concurrent mechanism matches that of the original linear method, thereby validating correctness at the unit level.

2. Integration Testing on Battlefield-Ethereum

   To verify integration within the battlefield-ethereum environment, the feature was exposed via a new configuration flag: CONCURRENT_BLOCK_FLUSHING. This flag determines the number of workers that will be processing tasks concurrently. \

   Original (linear flushing):

   ./scripts/run_firehose_geth_dev.sh 3.0 prague
   

   Concurrent flushing enabled:

   CONCURRENT_BLOCK_FLUSHING=1 ./scripts/run_firehose_geth_dev.sh 3.0 prague
   

   Behavioral differences were observed through log output. In the concurrent mode, log lines such as:

    "Firehose closing, flushing queued blocks to standard output"
   

   appear when the program is interrupted (e.g., via Ctrl + C), indicating that the concurrent flushing logic and cleanup path are active. These lines are absent in the linear configuration, confirming that the switch is functioning as intended.

   Furthermore, when running the integration test suite using:

   pnpm test:fh3.0:geth-dev
   

   all tests passed successfully (64 passing), indicating that the concurrent implementation does not introduce regressions in battlefield compatibility.

3. Performance Benchmarking

   To quantify performance differences, benchmarking was conducted using firehose-ethereum. The following command was used for both the baseline and concurrent configurations:

   time geth --vmtrace=firehose \
   --vmtrace.jsonconfig='{"concurrentBlockFlushing":<number_of_workers>}' \
   --synctarget=<last_block_hash> \
   --syncmode=full --holesky --datadir=./geth --db.engine=pebble \
   --state.scheme=path --port=30305 --authrpc.jwtsecret=jwt.txt \
   --authrpc.addr=0.0.0.0 --authrpc.port=9551 --authrpc.vhosts="*" \
   --http --http.addr=0.0.0.0 --http.api=eth,net,web3 --http.port=9545 \
   --http.vhosts="*" --port=40303 --ws.port=9546 --ipcpath=/tmp/geth.ipc > /dev/null
   

   The benchmark was conducted in four phases:

   • With concurrentBlockFlushing: 0, the node was synced to block 10,000, the data directory (./geth) was removed, and then resynced up to block 100,000.
   • The same steps were repeated with concurrentBlockFlushing: 10.
   • The same steps were repeated with concurrentBlockFlushing: 100.
   • The same steps were repeated with concurrentBlockFlushing: 1000.

   The time command outputs wall-clock time and system/user CPU usage upon completion, providing a baseline for comparing performance between the linear and concurrent implementations. This methodology enables a controlled, reproducible environment for evaluating the effectiveness of the concurrent block flushing feature.

Section 4: Analysis

The specifications of the operating system used for testing are as follows:

Model: Macbook Air
Processor: Apple M1 chip
Memory: 8 GB

4.1 Results

The following outlines the results for metric three:

Until Block 10,000

No Concurrency

• Run 1: 84.43s user, 15.27s system, 61% CPU, 2:41.39 total
• Run 2: 86.76s user, 15.93s system, 64% CPU, 2:40.36 total
• Run 3: 78.30s user, 14.62s system, 55% CPU, 2:47.51 total
• Run 4: 78.29s user, 13.72s system, 55% CPU, 2:44.45 total
• Run 5: 83.29s user, 13.37s system, 59% CPU, 2:43.47 total

Concurrency (10 Goroutines)

• Run 1: 68.14s user, 13.85s system, 51% CPU, 2:39.27 total
• Run 2: 70.81s user, 14.10s system, 52% CPU, 2:40.37 total
• Run 3: 64.79s user, 13.65s system, 49% CPU, 2:39.22 total
• Run 4: 64.34s user, 15.18s system, 48% CPU, 2:43.25 total
• Run 5: 70.21s user, 15.58s system, 51% CPU, 2:47.34 total

Concurrency (100 Goroutines)

• Run 1: 71.19s user, 16.06s system, 53% CPU, 2:42.77 total
• Run 2: 66.74s user, 12.12s system, 51% CPU, 2:34.47 total
• Run 3: 72.53s user, 16.11s system, 56% CPU, 2:37.50 total
• Run 4: 70.99s user, 14.02s system, 56% CPU, 2:29.46 total
• Run 5: 65.86s user, 13.66s system, 54% CPU, 2:24.97 total

Concurrency (1000 Goroutines)

• Run 1: 68.44s user, 14.65s system, 54% CPU, 2:33.83 total
• Run 2: 72.79s user, 15.06s system, 55% CPU, 2:38.76 total
• Run 3: 65.60s user, 13.19s system, 53% CPU, 2:28.36 total
• Run 4: 71.82s user, 15.05s system, 54% CPU, 2:38.10 total
• Run 5: 71.36s user, 14.87s system, 56% CPU, 2:33.32 total

---

Until Block 100,000

No Concurrency

• 331.03s user, 165.76s system, 32% CPU, 25:28.66 total

Concurrency (10 Goroutines)

• 383.44s user, 183.39s system, 37% CPU, 25:22.84 total

Concurrency (100 Goroutines)

• 399.57s user, 175.51s system, 37% CPU, 25:28.03 total

Concurrency (1000 Goroutines)

• 346.83s user, 165.96s system, 34% CPU, 25:02.78 total

---

Table 1: Result Comparison with and Without Concurrency

Block 10,000 Summary

Goroutines	User Time	System Time	CPU	Total Time
0	82.214s	14.582s	58.8%	2:43.44
10	67.666s	14.270s	50.2%	2:41.89
100	69.462s	14.394s	54.0%	2:33.83
1000	69.990s	14.564s	54.4%	2:34.47

Block 100,000 Summary

Goroutines	User Time	System Time	CPU	Total Time
0	331.03s	165.76s	32.0%	25:28.66
10	383.44s	183.39s	37.0%	25:22.84
100	399.57s	175.51s	37.0%	25:28.03
1000	346.83s	165.96s	34.0%	25:02.78

---

4.2 Discussion

The four parameters analyzed are user, system, CPU, and total.

• System and CPU times do not vary significantly with the implementation of goroutines.
• User and Total time are generally lower with the use of concurrency for 10,000 blocks.
• However, performance improvement for 100,000 blocks is less conclusive.

This could be due to:

• Single-run variance
• Varying system conditions during execution
• Larger processing overhead over a long duration

More runs and broader statistical analysis may be required to solidify findings.

Section 5: Conclusion

The introduction of concurrency to the block flushing process demonstrates performance improvements for smaller workloads (up to 10,000 blocks), particularly in reducing user and total processing time. While results for larger workloads (100,000 blocks) show less consistent gains.