Report: Concurrent Block Flushing in Tracer Processing System

Section 1: Introduction

Currently, the tracer processing system operates in a linear fashion, requiring the complete processing of a block's code before proceeding to the next one. As a result, computationally expensive operations—such as proto.Marshal and base64 encoding, which are essential for flushing a block to the firehose—can introduce significant delays. To mitigate this bottleneck, a potential solution is to leverage concurrency by introducing goroutines. These heavy operations can be offloaded to a separate channel, enabling asynchronous processing. This approach allows the main execution flow to begin processing subsequent blocks while previous ones are being flushed, thereby improving overall throughput and reducing latency. The goal of this report is to outline the solution and provide benchmarking results.

Section 2: Background

proto.Marshal, part of Protocol Buffers (protobuf), serializes structured data into a compact binary format. While efficient in output size, the process of traversing and encoding complex data structures—especially those with deeply nested or repeated fields—can be CPU-intensive. Additionally, memory allocations during marshaling can introduce further overhead.

Similarly, base64 encoding, which transforms binary data into an ASCII string format for transmission or storage, involves non-trivial byte-wise transformations and increases the data size. This added computational cost becomes significant when processing large blocks or high-throughput workloads.

Together, these operations introduce latency in a linear processing pipeline.

Section 3: Method

3.1 Proposed Solution

A critical point in the tracer processing system is the OnBlockEnd hook, which invokes the printBlockToFirehose method. This method includes computationally expensive operations such as proto.Marshal and base64 encoding, which are necessary to serialize and flush the block data to the firehose.

To address this, the current solution introduces a worker queue mechanism. Specifically, a single goroutine backed by a channel is used to enqueue and process printBlockToFirehose tasks asynchronously. This decouples the expensive flush operations from the main block processing path, allowing the tracer to begin handling the next block immediately after OnBlockEnd is invoked. This behavior is controlled by the FirehoseConfig.ConcurrencyBlockFlushing flag: when set to true, the asynchronous flushing mode is enabled; when set to false, the system falls back to the default linear execution of printBlockToFirehose.

As part of future work, this model can be extended from a single worker goroutine to multiple concurrent workers. This could further improve throughput by increasing parallelism. However, such an enhancement must address critical challenges, including proper synchronization of shared resources like output.Buffer and maintaining the strict block ordering requirement—i.e., block N must be flushed before block N+1 to preserve data consistency.

3.2 Validation and Performance Metrics

The implementation was validated at multiple levels to ensure correctness, configurability, and performance improvements of the concurrent block flushing mechanism.

Unit-Level Validation

To confirm the functional correctness of the concurrent flushing implementation, a unit test was written that creates and processes 1,000 blocks, flushing each to an InternalTestingBuffer. The results were then compared against expected outputs to verify equivalence. The test confirms that the output produced by the concurrent mechanism matches that of the original linear method, thereby validating correctness at the unit level.
Integration Testing on Battlefield-Ethereum

To verify integration within the battlefield-ethereum environment, the feature was exposed via a new configuration flag: CONCURRENT_BLOCK_FLUSHING. This flag determines whether the system uses the default sequential method or the new concurrent implementation. The system can be toggled between these modes with the following commands:

Original (linear flushing):
```
./scripts/run_firehose_geth_dev.sh 3.0 prague
```
Concurrent flushing enabled:
```
CONCURRENT_BLOCK_FLUSHING=true ./scripts/run_firehose_geth_dev.sh 3.0 prague
```
Behavioral differences were observed through log output. In the concurrent mode, log lines such as:
```
 "Closing channel: flushing the remaining blocks to firehose"
```
appear when the program is interrupted (e.g., via Ctrl + C), indicating that the concurrent flushing logic and cleanup path are active. These lines are absent in the linear configuration, confirming that the switch is functioning as intended.

Furthermore, when running the integration test suite using:
```
pnpm test:fh3.0:geth-dev
```
all tests passed successfully (64 passing), indicating that the concurrent implementation does not introduce regressions in battlefield compatibility.
Performance Benchmarking

To quantify performance differences, benchmarking was conducted using firehose-ethereum. The following command was used for both the baseline and concurrent configurations:
```
time geth --vmtrace=firehose \
--vmtrace.jsonconfig='{"concurrentBlockFlushing":<true|false>}' \
--synctarget=0x7ae82cb3e60f13272a59319a4b617022228227258e18e0c5e7404236d773d2a3 \
--syncmode=full --holesky --datadir=./geth --db.engine=pebble \
--state.scheme=path --port=30305 --authrpc.jwtsecret=jwt.txt \
--authrpc.addr=0.0.0.0 --authrpc.port=9551 --authrpc.vhosts="*" \
--http --http.addr=0.0.0.0 --http.api=eth,net,web3 --http.port=9545 \
--http.vhosts="*" --port=40303 --ws.port=9546 --ipcpath=/tmp/geth.ipc > /dev/null
```
The benchmark was conducted in two phases:
- With concurrentBlockFlushing: false, the node was synced to block 10,000, the data directory (./geth) was removed, and then resynced up to block 100,000.
- The same steps were repeated with concurrentBlockFlushing: true.
Note: A channel with a buffer of 100 was created to allow the tasks to queue without blocking the producer.
The time command outputs wall-clock time and system/user CPU usage upon completion, providing a baseline for comparing performance between the linear and concurrent implementations. This methodology enables a controlled, reproducible environment for evaluating the effectiveness of the concurrent block flushing feature.

Section 4: Analysis

The specifications of the operating system used for testing are as follows:

Model: Macbook Air
Processor: Apple M1 chip
Memory: 8 GB

4.1 Results

The following outlines the results for metric three:

Until Block 10000

No concurrency

Run 1: 71.49s user 15.56s system 56% cpu 2:34.28 total
Run 2: 69.77s user 15.60s system 54% cpu 2:37.73 total
Run 3: 68.81s user 15.12s system 53% cpu 2:38.18 total

Concurrency

Run 1: 69.61s user 14.97s system 55% cpu 2:32.85 total
Run 2: 67.94s user 15.13s system 54% cpu 2:33.59 total
Run 3: 68.60s user 16.37s system 48% cpu 2:54.22 total (Not sure what happened here)

Until Block 100000

No concurrency

364.19s user 172.14s system 34% cpu 26:13.33 total

Concurrency

358.42s user 171.21s system 35% cpu 25:11.97 total

Table 1: Result comparison with and without concurrency

	No Concurrency	Concurrency
Block 10 000
user	70.02s	68.72s
system	15.43s	15.49s
cpu	54.33%	52.33%
total	2:36.73	2:40.22 (because of last run)
Block 100 000
user	364.19s	358.42s
system	172.14s	171.21s
cpu	34%	35%
total	26:13.33	25:11.97

4.2 Discussion

Run 3 with concurrency seems to be an outlier. Without it, the general trend would be that every block saves around 0.0006 second, or 0.6 millisecond.
User seems slightly lower, whereas system and cpu are relatively the same.

Section 5: Conclusion

The implementation of a single goroutine seems to lead to a decrease in the total time by a factor of 0.6 millisecond per block.

8.5 KiB Raw Blame History