[8.x] functionaltests: restart Integrations Server after ingestion (backport #15356)

[mergify]

Motivation/summary

This is an alternative #15317.

Instead of expanding the logic to wait for ingestion completion by monitoring data streams and document count, rely on a side effect of shutting down the APM Server. On shutdown APM Server flushes all the data, included the aggregation for the data already ingested.

We use this behaviour triggering a restart of the Integration Server after the test data ingestion is done to ensure ingestion in completed before moving to checking document counts and data streams details.

I run 5 tests and in most cases the restart took ~1 minutes. In 1 case it took 6 and in 1 case it took more than 10 minutes (but the code has a timeout at 10 minutes).

In 1 case the test failed due to race condition when testing that the lazy rollover happened: it did not at the moment of testing but I confirmed manually that it indeed happened. This behaviour suggests lazy rollover may happen later than expected (at ingestion after an upgrade) and is not impacted by this change, but further investigation is required to prevent flakyness.

Checklist

How to test these changes

Functional tests pipeline for this branch should be green: https://github.com/elastic/apm-server/actions/workflows/functional-tests.yml?query=branch%3Afunctionaltests-restartapmserver

Related issues

#14100

---

This is an automatic backport of pull request #15356 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of c230240 has failed:

On branch mergify/bp/8.x/pr-15356
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit c230240a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   functionaltests/8_15_test.go
	modified:   functionaltests/TestUpgrade_8_15_4_to_8_16_0/outputs.tf
	modified:   functionaltests/go.mod
	new file:   functionaltests/internal/ecclient/client.go
	modified:   functionaltests/internal/gen/generator.go
	modified:   functionaltests/main_test.go
	modified:   testing/infra/terraform/modules/ec_deployment/outputs.tf

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   functionaltests/go.sum

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

This pull request has not been merged yet. Could you please review and merge it @endorama? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @endorama? 🙏

[endorama]

Tests are run from main, no need to backport them.