Testing ABCI as a blackbox without Tendermint

[p-offtermatt]

Hi,

I'm working on a utility for testing ABCI, and I'd be interested in discussing whether it seems like a right way to approach this, or if similar things already exist; and I'd like to discuss a feature that would be useful to have in the context of this utility.

Problem

We would like to test applications by feeding traces of transactions, e.g. this is what Atomkraft does.
Currently, Atomkraft runs a small testnet and submits transactions to Tendermint. So Atomkraft needs to wait until Tendermint comes to consensus and submits the transactions to the application - instead we'd like to just submit transactions to the app directly, without having to go through the real consensus engine.
We also would like to have more control over blocks:

• Timestamps: Right now, if we want the application to receive a timestamp 1 hour in the future, we'd have to manipulate the system time, or simply wait an hour and then submit the transaction. If we submit the transactions to the app directly, we can arbitrarily manipulate timestamps.
• Signers/Proposers: In Tendermint, it's cumbersome to choose who signs a block and who doesn't, but if we submit the transactions to the app without going through Tendermint, we have full control over which addresses we enter as signers in the block, which address to set as the proposer, etc
• Simulate networks with many validators: If we use the real tendermint, then to have 200 validators, we'd have to run the testnet with 200 validators. If we mock Tendermint, then we'd not need to run 200 nodes, but we could simply submit blocks that look like there are 200 validators in the net, while only running a single instance of the app.

Proposed solution

I implemented a utility that acts like a mock in place of Tendermint: It supplies the RPC endpoints that Tendermint supplies, and itself calls ABCI (via grpc). When Atomkraft wants to submit a transaction to the app, it calls the mock, and the mock supplies the transaction right to the application, with no need to do consensus. A demo of the mock lives here: https://github.com/informalsystems/tendermock/tree/main/pipeline_poc

Small issue with the solution

To start simapp without Tendermint, but with the ABCI server, there exists a flag --with-tendermint=false
However, this doesn't give me access to the cosmos sdk grpc endpoints , it seems the grpc server is not started with this option.
On the other hand, there exists a flag that starts only the grpc server, namely --grpc-only. However, this doesn't start the ABCI server, so it's not an option, since the ABCI server is what we want to test.

There's a workaround, since I can call simapp via the CLI to e.g. submit transactions, but for compatibility reasons, it would be preferable to interact with the mock-Tendermint via grpc, since this is how Atomkraft communicated with Tendermint previously.
Would it make sense to have the simapp support an option which runs the ABCI server and the grpc server, but without Tendermint? As mentioned above, both options separately already exist, so it seems reasonable they could be run together, but maybe there is something prohibiting this?

Alternative solutions?

I'd also be interested in finding out whether there already are similar approaches to testing ABCI - I found a few mocks of Tendermint, but none seemed to allow submitting transactions to a "black-box" application via ABCI.