Move or remove the mutex on the application when applicable

[evan-forbes]

Currently, comet holds a mutex for each ABCI call

• celestia-core/abci/client/local_client.go

  Lines 95 to 100 in 0f6a0e1

  	func (app *localClient) CheckTx(ctx context.Context, req *types.RequestCheckTx) (*types.ResponseCheckTx, error) {
  	app.mtx.Lock()
  	defer app.mtx.Unlock()
  	return app.Application.CheckTx(ctx, req)
  	}

For obvious reasons, this is terrible. We cannot process txs in the mempool while PrepareProposal, ProcessProposal, or FinalizeBlock is being called. Even RPC queries can holdup consensus as we utilize the Query method to generate inclusion proofs.

In theory, most of these we should be able to remove. The extent of work that is required to make this doable is TBD. afaik, this lock has always been included in comet. Some projects such as SEI have removed it

[rootulp]

Prerequisite

• Determine if multiplexer uses the local ABCI client or GRPC ABCI client.
  • If uses local ABCI client then delete the GRPC ABCI client and remove mutexes from local ABCI client
  • If it uses GRPC ABCI client then the removal of mutexes may be impossible so need to re-think approach.

After this code change, we need to verify performance by running a talis test to see if performance improved. It may be possible to get signal on it before a talis test though.

[rootulp]

Some projects such as SEI have removed it

Sei's ABCI local client doesn't appear to have CheckTx: https://github.com/sei-protocol/sei-tendermint/blob/81bf836ca6981f48d6811a724d91789d43cc8791/abci/client/local_client.go

Sei's ABCI GRPC client doesn't have a mutex for CheckTx: https://github.com/sei-protocol/sei-tendermint/blob/81bf836ca6981f48d6811a724d91789d43cc8791/abci/server/grpc_server.go#L85-L91 but it also doesn't have all the other ABCI methods.

[rootulp]

On a single-node devnet locally, only local client is used. Confirmed with

./scripts/single-node.sh
./scripts/single-node-upgrades.sh

so it's true when running embedded apps and the native app. Also confirmed in this diagram

[rootulp]

Upstream still has the mutex for all methods: https://github.com/cometbft/cometbft/blob/main/abci/client/local_client.go