Providing an update regarding the issue

Work so far

Few weeks ago I created tooling for simulating calls, txs and blocks via native, JIT and AOT execution. The motivation was to a) interact with revmc in minimalistic environment and b) investigate the performance benefits of AOT/JIT compilation.

The initial findings were not encouraging. They showed significant latency reduction (~100x) for simple execution without external calls and storage fetches - like the Fibonacci example from revmc. However, this was not the case for entire block simulation; in fact, there was no significant improvement at all.

These findings could be explained by the fact that going out of context (storage fetch & external call) incurs enough latency to minimize the benefits of JIT/AOT compilation, or that the majority of transactions don't spend as much gas/resources as calls like Fibonacci do. Alternatively, there could simply be a bug in the simulation. Further investigation was required.

When benching the block transactions individually, I found signs of significant latency reduction. After tweaking things and further investigation, I found:
a) Simulating solely the top of the block (10%) shows 7% reduction in latency: notebook
b) Simulating random transactions from a block range shows 5% reduction in latency: notebook
c) Simulating the entire block still shows no significant improvement: notebook

These findings suggest a potential bug in the block simulation or other factors affecting performance (e.g., reduction in available memory due to caching all compiled contracts for the block).

Next steps

Further simulation discrepancy investigation

Further investigation is needed to determine what caused the discrepancy between whole-block vs. top-of-block and whole-block vs. individual-txs results.

Latency reduction analysis for limited set of compiled contracts

In the current measurements, all contracts interacted with during execution were compiled AOT. This, however, might not be a realistic scenario as having every single contract AOT compiled could be very difficult and costly - compilation takes time, and additional storage requirements could be significant. Instead, only the top contracts regarding gas consumption or some other metric for the past could be considered.
An analysis would be conducted to investigate the performance of such a strategy.

RBuilder integration

The revmc-sim code is modular, meaning that to integrate the code into the rbuilder, only the loader and builder crates need to be used. The crate where I run simulations and where I suspect a bug is hidden is not required, as it is only a tool to compare performance between different modes of execution.

Priority

The primary goal of this issue is to integrate revmc into the rbuilder; therefore, I will prioritize integration before any further investigative work.

That said, I believe that investigative work is important. Even with revmc integrated, it is beneficial to know how much performance improvement can be expected and what compilation strategy (e.g., how many top gas contracts for a given timeframe to compile) makes sense.

Note that in my tooling and primary measurements, I included JIT compilation. I did so only for the sake of completion and not because JIT is needed for rbuilder integration. In the following work, JIT will be skipped/deprioritized.

I am not married to the approach and would be happy to discuss alternatives.

feat: integrate revmc #24

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions