Consistency guarantees related to the PDO Transaction Processor (documentation and questions)

[bvavala]

(the following is meant as a starting point to gather documentation and for a discussion on the PDO TP consistency guarantees and requirements at the client-side and app-side; ideally, this will result in readme document that clarifies these aspects)

The PDO Transaction Processor (TP) is a decentralized application which maintains registries for enclaves, contracts and state updates.
The TP (and similarly the client-side app submitting transactions to it) exposes a set of API which can be conveniently implemented with different distributed ledger technologies. Right now, it is based on CCF.

At the client-side, a successful TP transaction (tx) does not mean that the tx was committed. It actually means successful execution by the TP. If the verification of the commit is necessary, then the client should contact CCF to check the tx's status (see https://microsoft.github.io/CCF/main/use_apps/verify_tx.html).

Questions for the client-side:
QC1. Can it happen, and under what circumstances, that a successful tx eventually does not commit?
QC2. Is it always necessary to verify the commit of a tx? When is it unnecessary, and what are the repercussions?
QC3. What's PDO's policy/guideline?

At the CCF app side, CCF exposes some APIs to interact with the ledger key-value store, namely put, get and get_globally_committed. The PDO TP uses such APIs.
From some comments on CCF (see comments and references below):
P1. these gets may return 2 different versions of a key
P2. the globally committed version may be arbitrarily stale
P3. any globally committed read will not raise conflicts (as a local read would instead do)

Questions:
QA1. what's the role of multi-threading in this context? Also, how is it defined (e.g., one thread per client; one thread per client session, but each sessions can have multiple worker threads; etc.)? Does CCF multi-threading have to be taken into consideration at the client-side to reason about consistency? Does it have to be taken into consideration at the CCF-app-side (by the TP)?
QA2. what about concurrency in a multi-party setting (where multiple parties run a CCF replica)? Is multi-party concurrency similar to, or different from, the multi-threading setting? Can clients issue concurrent (maybe conflicting) tx's to different replicas? Do tx's get all executed at the primary replica, and should this information be taken into account (or should it be assumed the CCF abstracts the underlying replicated system)?
QA3. With respect to the properties (1.,2.,3.) listed above,
a. when and how do a global read followed by a write create an issue?
b. what's the guideline for using these operations (e.g., when only regular put/get should be used; when it is ok to mix global reads with local reads/writes; when it is appropriate or necessary to verify a tx commit, and what the consequences are if this is not done).
c. should a global read only be used (or is this only useful) when the CCF app (running inside an enclave, and thus protecting secrets) has to decide on whether to perform a sensitive action (e.g., release a secret, use its private key to produce a signature, or decrypt some data, etc.), thereby binding the sensitive action to stable/persistent information that cannot be reverted?

---

Excerpts from github comments:

"Note that using get_globally_committed() in the context of a write transaction is dangerous and will potentially lead to a serializability break. If mixed with get(), it will also break snapshot isolation, since you may end up reading from two different versions. It is only safe to use get_globally_committed() in very restricted contexts, such as a read-only handler, doing all its reads through get_globally_committed()." -- microsoft/CCF#3926 (comment)

"If you use get() and put() in the same transaction, then CCF will detect any conflicts with concurrently executing transactions (ie - those that have completed in the gap between this transaction selecting a read version and it completing), and automatically rerun the transaction (reading newer state). get_globally_committed() can be used to deliberately circumvent this flow to report only committed state, but reads of that committed state will not be treated as conflicts, so if they influence writes you may get surprising results.
When a transaction begins, it is assigned 2 distinct versions to read from: the first is the most recent state, that all get()s will operate over, and the second is the current committed state, which may be arbitrarily stale, but is guaranteed to be persisted." -- microsoft/CCF#3962 (comment)

[bvavala]

Further discussion is available here https://github.com/hyperledger-labs/private-data-objects/pull/467/files#r1498352047