[epic] Integrations can be tested on windows both locally and on CI

[marc-gr]

This issue is to discuss and track the work required to implement the ability to define and run system tests on windows hosts for integrations.

Prior context:

• Add ability to deploy custom elastic-agents on different OS or runtimes #787
• [system tests] Add support to deploy custom elastic-agents in VMs #829

Problem

Some integrations can run, or run exclusively on windows systems. Currently we lack the ability to run automated tests on windows platforms, neither locally or in CI. This leaves us blind under any possible problems that we may introduce, and we would only find out either performing tedious manual testing or once they are released.

Some of the integrations affected by this are (but maybe not limited to):

• Windows
• System (some data streams)
• network traffic
• IIS
• custom winlog input

Even if the number of affected integrations seems low compared to our total, this effort is still needed for both completeness and the fact that some of these are very widely used integrations.

Requirements

• elastic-package "officially" supports windows
  • Currently some things do not work as expected when running on windows ie Elastic Stack Up not working on Windows or Ubuntu WSL #1282, also some partners pointed out weird behaviour when using it under a native windows host. We might need to add testing to ensure it works properly on windows hosts.
• elastic-package supports running windows images for system tests on windows hosts (related to [system tests] Add support to deploy custom elastic-agents in VMs #829)
  • windows docker images should be able to run on windows hosts if docker is available
• A windows docker image needs to be built for elastic-agent so it can be used by the system tests
• Buildkite migration needs to be completed in integrations
  • Having this done will ease a lot the setup of a windows host and will prevent doubling the work for both Jenkins and Buildkite
• Buildkite windows agents need to be able to run docker
• We need a way to limit the amount of windows hosts required on CI, it might be something as easy as an explicit opt in list of packages or something elastic-package can detect dynamically.

This is a general overview and might be incomplete, would be interesting to gather feedback and ideas to be able to define a specific course of actions towards this.

[marc-gr]

Some additional observations after some testing:

• We have readily available Windows agents for buildkite with docker windows containers available. (see [do not merge] Testing windows on CI integrations#8461).
• On windows, linux and windows containers can't run side by side out of the box (https://forums.docker.com/t/was-support-for-mixed-mode-windows-linux-containers-removed/136484) and needs to be worked around by manually setting it up with WSL and docker contexts but maintainability of this is unclear without some more investigation.

I can think of several paths that can be taken to achieve what we want here, in no particular order:

• Following with the initial suggestion, add support for custom agents in windows containers/hosts, and work around the issue of running linux containers on windows hosts on CI.
• Have a windows profile in ep that has dedicated windows images of the stack to run tests as they do now, but in windows platforms.
• Enhance the current tf service deployer so it can be used to deploy a custom agent on a windows VM. For this we would want to have the VM image pre-created for ease of use.

Is there any preferred path towards this from ecosystem standpoint? Do you have additional ideas about how this could be faced?

cc @jsoriano @mrodm

[jsoriano]

Hey @marc-gr,

If we only want to run the agent in Windows, some way to run custom agents would be a good option, yes. These custom agents could be run natively. We would need to check if we can run multiple agents in the same host, with custom data dirs and so on.

If we want to run the whole stack in Windows, we could implement a new "native" stack provider, that would use the native tar.gz/zip artifacts instead of docker. This would allow to use native artifacts on each platform. We would also need some way to run multiple agents in the same host.

If we want to be able to test windows on linux or the other way around we would need some kind of virtualization, yes. Maybe we can do it based on terraform as you mention. This may be also useful in other use cases, as when testing the system package. I think we had some issues about this.

In general for these cases it would also help to move agent management from the stack providers to the test runners, this way each test could manage its own agent, and we could offer different mechanisms to do that. The kubernetes service deployer already does something like this. Apart of helping on providing agents with different configurations, runnings on different platforms and so on, this would also help on running tests on isolation, avoiding issues like #405.