Nrunner plugin: per test sysinfo support

[richtja]

In the legacy runner we have sysinfo plugin which can save sysinfo before and after execution of each test. We would like to bring this feature to the nrunner architecture. With nrunner comes new problems to be dealt with.

• The test can be run in parallel so the results of sysinfo can be affected by the other tests.

• The test is run isolated from avocado core via runners and can by run in the container or even on the remote machine. This means that the sysinfo have to be gathered on the runner side.

This problem is tracked in #3877

A possible solution will be to create specific runner for gathering sysinfo @willianrampazzo doing for the Requirements resolver. Such a runner will be run before and after each test execution in the same environment as the test itself, and it will send sysinfo to the avocado core. And because the tests are run in parallel the sysinfo has to have information about running tests, so the user can decide if the sysinfo was affected by other runs.

This is just an outline of the solution and I would like to know your opinions on it. From the result of this discussion I can create a blueprint if it will be necessary.

[beraldoleal]

Hi @richtja , my only concern with that approach is that we are replacing a "plugin" with a "requirement". This might be work for this particular case, but this could lead us to always trying to use the requirements.

I guess, that what I'm trying to say is: today we have the "plugin" concept, and they can be executed at the test level (pre/post). With the next runner architecture are we going to remove that functionality? IIUC our options so far are:

1. We are limiting our plugins to not run on runner side and stick with the "requirements" solution;
2. We could introduce the "Runner-plugin" concept;
3. We could deploy our current plugins (and possible Avocado) to the runner machine as well (related to Improve how we distribute the runners #4336)

Since we are always talking about how much "standalone" should be a runner and I have to confess that item 3 feels like a good choice.

[willianrampazzo]

The current design of Requirements Resolver proposed on #4532 uses a simple dependency graph to set a dependency of a test and requirements.

As we evolve the idea of runners and tasks on nrunner, I see a dependency graph as a good fit for it. I can think of a mechanism that generates a dependency graph for requirements, plugins, and any other piece of code that need to run before and after a test. If the plugins are adapted to run as tasks, this solution allows using plugins already available.

In this architecture of dependency graph, the Requirements Resolver is just a small piece.

[beraldoleal]

I understand how nice and powerful is this new architecture. But if we go on that road, we are changing how plugins are designed and this should be taken into consideration.

[willianrampazzo]

I agree, but we already changed how the runner is designed, so I don't see a problem if we change other parts to fit the new architecture. I see it as an expected outcome.

[beraldoleal]

My understanding is that "runners" are internal and "more or less" new, which is getting more mature. So changes are expected there. But plugins are something very used by our outside contributors and I'm seeing this as a "limitation" since plugins could not behave like before.

Again, I'm not strongly against this approach, I'm just trying to point out that we are "adapting" a plugin to be executed as a requirement.

[willianrampazzo]

To clarify, my proposal is not to adapt the current plugins to work as requirements, in the sense of the requirements resolver architecture, but to use a dependency graph. The result of the execution of a plugin will be the same as we have today.

[pevogam]

I have to agree with @beraldoleal here and the third option and just want to share the opinion of an outsider.

From an external point of view, Avocado constantly changes its concept making it very hard to somebody that just wants a functioning library for their own tests to follow. The concept of plugins is still fresh in the heads of most of us users and we have just caught up with some of it and now we start talking about runner-plugins (considering runner currently has 3-4 different meanings across the code base) and pre- post- requirements. My point here is simply that the time scale users are catching up is much slower than the one maintainers generate.

So if we could reuse already developed, well-tested, and known code to implement this new behavior this would be optimal for everyone.

[richtja]

I guess, that what I'm trying to say is: today we have the "plugin" concept, and they can be executed at the test level (pre/post). With the next runner architecture are we going to remove that functionality? IIUC our options so far are:

@beraldoleal I understand your concerns, but right now in the legacy runner we don't have any pre/post plugins on test level the sysinfo plugin is hard-coded inside Test class. What we have is ResultEvents plugins with start_test, test_progress and end_test methods. I agree with you that with the nrunner we have to create a new plugin type for pre/post test events.

I like the @willianrampazzo idea that we adapt the plugin system to be run as Task. We can add a new layer to it so the creating of new plugins will be unchanged, but internally the avocado will be crating Tasks and for users the functionality stay the same.

[pevogam]

Yes, I would recommend that new and immature concept start merely as internal ones and the interface is left as the one that is known to everyone, at least until such a new concept actually shows promise.