Define and implement a framework for evaluating async IO traits

[yoshuawuyts]

Introduction

I was requested on Zulip to file an issue about any misrepresentation or items that I think should be addressed further in the IO traits section of this repository.

My main concern is that the document currently starts with a conclusion ("proposal"), and shares some of the rationale afterwards. As I mentioned before: I don't know which metrics have been used to evaluate designs by, I don't know how the current proposal stacks up against those metrics (including tradeoffs). Right now I cannot independently agree or disagree with the conclusion ("proposal"), because I don't know what it's based on. And I believe it's crucial that all of this is documented before we make any recommendations, so other members of the working group can individually evaluate the requirements, evaluations, and conclusions.

Path forward.

The way I'd like to see this document restructured is:

1. Define the set of requirements for IO traits. These are what proposed designs will be evaluated against.
2. List each possible design, and evaluate it against the requirements.
3. Summarize the findings. This would include making a recommendation from the possible designs.

I don't expect us to immediately agree on which requirements to list, or how important they are. But those disagreements should be documented as well. We're still in an active research phase, and documenting different viewpoints is essential (in an RFC this would be summarized in an "alternatives" section). That way if requirements are added, or if our understanding of designs change, we can have this be reflected in our shared understanding.

This approach would make it possible to modify and clarify requirements, gradually polishing the evaluation of proposals, and building towards a better understanding, collaboratively. This work is essential to gaining a shared understanding of the problem space by the working group, and I believe that this repository would be the right place to do that work.

Conclusion

@nrc I understand that what I'm requesting is non-trivial. I was a bit hesitant to open an issue because I believe what I'm saying could be understood as: "please change everything". And even though I do think this is essential work that needs to happen before we can start making any recommendations, I don't expect you to do this work alone. We already meet regularly, and I'd be super happy to pair on this, and making sure that we can get this work done together!

[yoshuawuyts]

Another reason to define a framework for evaluation ahead of time is that it will allow us to use that as we test, evaluate, and iterate on designs. As we start testing, we should be able to check our assumptions of the design against reality. If we find our assumptions deviate from what we're experiencing, we should be able to come back and re-evaluate.

That all becomes a lot harder if we don't clearly establish our requirements and document our assumptions ahead of time.

[nrc]

So, I think the document is not the document you want, but I'm not sure that it is the wrong document. I think what you are asking for is a process, rather than a document (i.e., you want the document to be structured chronologically with respect to the process). And I think that the structure of the process you are proposing is a good one (plus some iteration as we discover new requirements whilst elaborating designs, etc.). However, I think that this document should present the results of the process. This is somewhat a question of audience, I have in mind an 'outsider' audience who wants to know our recommendation and why we came to that conclusion. I think you have in mind an 'insider' audience (such as yourself) who actually want to participate in the process to some extent, rather than just consume the results.

I think that we do want to have a document for 'outsiders' as our final output, and that is something I am iterating towards, but I also think we need to satisfy the 'insider' audience who want to participate or re-evaluate the results. I'm not exactly sure how to do this.

I will start by better organising the requirements and be more explicit about where the alternatives don't meet them.

I want to dig into your concerns a little bit:

My main concern is that the document currently starts with a conclusion ("proposal"), and shares some of the rationale afterwards.

I think this is very normal when reporting work. As I said above, I think you want to be more involved in the process than just reading a report about the work

As I mentioned before: I don't know which metrics have been used to evaluate designs by,

The document starts by listing the requirements. Could you elaborate on what would be more useful?

I don't know how the current proposal stacks up against those metrics (including tradeoffs).

I am iterating towards better evaluation against the requirements, but the Read part already answers each requirement directly

And I believe it's crucial that all of this is documented before we make any recommendations, so other members of the working group can individually evaluate the requirements, evaluations, and conclusions.

I'm kinda unsure about your usage of 'before' here. I think this comes back to the difference between the process and the document. I think maybe you want to get WG agreement on the requirements and how the designs meet or do not meet them before evaluating the alternatives against each other? And by before I means before in time rather than before in the document. In which case it is probably best I organise some kind of WG meeting to discuss things, rather than working further on the document?