Support gRPC/protobuf for RESTful APIs and node-to-node communication

[anasalkouz]

Replace REST/JSON-based API with gRPC/protobuf to improve serialization/deserialization performance.
Ref: https://grpc.io/docs/what-is-grpc/introduction/

[AmiStrn]

Would this also help with streaming rather than doc by doc or bulk indexing?

[penghuo]

Would gRPC also benefit node-to-node communication?

[anasalkouz]

Would gRPC also benefit node-to-node communication?

Sure, this should help in both node to node communication (Internal) and client to server communication (External).

[dblock]

@saratvemulapalli This becomes theoretically swappable via the extensions work, doesn't it?

[anasalkouz]

Based on my understanding of how the internal communication between the node works now, it's using a single transport layer and it should be easy to change how to works. @mch2 please keep me honest.

[saratvemulapalli]

@dblock Extensions are being built over transport (layer-4), GPRC/Protobuf operate atleast at layer 5.
Extensions are adaptable to new communication mechanisms.

@anasalkouz that is correct. Netty4 transport is the module which implements transport in OpenSearch via NIO. This is swappable with new communication mechanisms. Netty4 also includes Layer 7 (HTTP) support.

[anasalkouz]

Shall we swap Netty4 with gPRC vs upgrading to Netty 5 ?

[dblock]

Shall we swap Netty4 with gPRC vs upgrading to Netty 5 ?

Yes! I would enable that as an option/experiment and benchmark it.

[saratvemulapalli]

Opened up a RFC for Protobuf in OpenSearch: #6844

[navneet1v]

@dblock is there a separate issue to track the gRPC protocol integration in OpenSearch? or it is already done?

[saratvemulapalli]

@navneet1v there isn't one. We could use this issue to track.
cc: @VachaShah

[navneet1v]

@saratvemulapalli if we are using this issue to track can we add more details in the description around what we are doing and all the tasks which are tracking for this.

[Bukhtawar]

Just so as to clarify we are only evaluating the data serialisation protocol(protobuf/avro/ion) etc and not really the transport layers like Netty/gRPC etc

[saratvemulapalli]

@navneet1v as @Bukhtawar said there is no body working on transport protocol yet (like GPRC). @VachaShah is exploring different serialization/de-serialization mechanisms we can adopt for OpenSearch.

Do you see a use case for gRPC? What are you looking for ?

[navneet1v]

What I am looking for is a light weight protocol than HTTP. While doing a search we spend a significant amount to time at this HTTP layer so, my thinking was how we can replace this so that we can get better query latencies.

[Bukhtawar]

@navneet1v to clarify, we use TCP for node to node communication, did you mean HTTP at the REST layer? Might be good to see some profiling results nonetheless based on your observation

[anirudha]

Ideally i would like to see support for Apache Arrow 0-copy via any http/2 transport

[anirudha]

@anasalkouz this way you remove the need for serDeSer but; this might need more extensive changes.
https://arrow.apache.org/docs/format/Flight.html

[Bukhtawar]

@anirudha Netty(default HTTP/transport) already supports zero-copy. Lets not complicate this issue further

[dblock]

I've renamed this issue to "Support gRPC/protobuf for RESTful APIs and node-to-node communication". I think we want it all, including _bulk support.