Access Actor store directly from SDKs

[dgrove-oss]

With the advent of rpclib and in particular tail calls, KAR is arriving at a semantics where the central primitives are all based on updates to the messaging system. As a result, access to the actor store is not a part of the fault tolerance and recovery scheme provided by the kar sidecars. Therefore, we could allow direct access to the actor store from the application processes (ie access not mediated by the sidecar) without weakening the system's guarantees.

Direct access to the actor store from the SDKs, has several potential benefits:

• It would remove load from the sidecars
• It would avoid the extra interprocess communication and resulting serialization/deserialization
• In the long run, we could decide to remove the actor state APIs from the sidecar's REST API and their implementation from the sidecar which would simplify things quite a bit
• It could simplify supporting multiple backend implementations of the actor state stores since the store's client libraries could be added to the SDK (or even to the user program) without changing the sidecar

There are some drawbacks:

• We would lose the benefit of only implementing the lowest level of the store functions in one place.
• We might need an additional hook to allow the sidecar to request that an SDK remove an actor's state
• We would probably need an extra hook to support the kar get operation that displays an actor instance's state since the sidecar would potentially not know how to access the actor store to get it directly.