Migrate performance tests to test packages.

[alopezari]

This PR migrates the performance test infrastructure to use the new test packages architecture, bringing consistency with E2E tests. The changes extract common environment functionality into shared base classes (QITEnvironment and QITEnvInfo) and implement performance-specific extensions. This enables performance tests to leverage the same test package system, environment setup, and orchestration as E2E tests.

Changes added in this request

• Created shared base classes for all environment types

  • Added QITEnvInfo as base class for common environment properties
  • Added QITEnvironment as base class with shared environment lifecycle methods
  • Extracted common functionality: Docker setup, plugin/theme activation, test package orchestration, and environment monitoring

• Refactored E2E environment to use shared base classes

  • Removed duplicated Docker, plugin activation, and setup orchestration code in src/src/Environment/Environments/E2E/E2EEnvironment.php
  • Made E2EEnvInfo extend QITEnvInfo in src/src/Environment/Environments/E2E/E2EEnvInfo.php
  • Maintained E2E-specific features like Playwright support and database backups

• Created performance environment using shared architecture

  • Added PerformanceEnvInfo extending QITEnvInfo in src/src/Environment/Environments/Performance/PerformanceEnvInfo.php
  • Added PerformanceEnvironment extending QITEnvironment in src/src/Environment/Environments/Performance/PerformanceEnvironment.php
  • Moved performance environment from local tests directory to shared environments directory
  • Deleted old performance environment implementation

• Updated performance test runner (src/src/LocalTests/Performance/Commands/RunPerformanceTestCommand.php)

  • Refactored to use new PerformanceEnvironment class
  • Added support for test packages with k6 scenarios
  • Integrated with global setup orchestrator for test package phases
  • Improved result handling and test execution flow

• Enhanced K6 runner for test packages (src/src/LocalTests/Performance/Runner/K6Runner.php)

  • Added support for running k6 tests from test packages
  • Implemented test package result processing and CTRF validation
  • Enhanced Docker configuration with proper volume mounting for test packages
  • Improved error handling and output parsing

• Updated environment management

  • Modified UpEnvironmentCommand in src/src/Commands/Environment/UpEnvironmentCommand.php to support both E2E and performance environments
  • Added dynamic environment type detection and appropriate class instantiation
  • Made WooCommerce default to 'latest' for performance environments
  • Updated EnvironmentVars in src/src/Environment/EnvironmentVars.php to work with QITEnvInfo base class

• Expanded test package manifest schema (src/src/PreCommand/Schemas/test-package-manifest-schema.json)

  • Added performance test package support with k6-specific configuration
  • Added optional options section for k6 test configuration
  • Added required results section with summary, json, and optional dashboard outputs
  • Enhanced validation rules for performance test packages

• Updated performance test result handling (src/src/LocalTests/Performance/Result/PerformanceTestResult.php)

  • Enhanced result processing for test package outputs
  • Added support for CTRF format results
  • Improved metric extraction and averaging
  • Better handling of test package result paths

• Updated command registration (src/src/Commands/CreateRunCommands.php)

  • Added performance to ignored test types (handled both locally and remotely)

• Minor improvements

  • Updated MetricAverager with better type handling
  • Fixed minor issues in PrepareDebugLog, ThemeActivation, and K6DockerConfig
  • Updated test snapshots to reflect new environment structure

• Test updates

  • Updated test snapshots to reflect new QITEnvInfo structure
  • Updated MetricAveragerTest for compatibility

Testing instructions

These testing instructions focus on local execution. While though the code was updated to apply these changes to remote performance tests execution, it requires special treatment for deploying the changes so they can be tested, so I plan to test it further after this PR is merged.

• Set up the local environment with the QIT-CLI changes from this branch and CD changes from https://github.com/Automattic/compatibility-dashboard/pull/1520.
• Ensure the default activation tests package is published (you can publish it with php src/qit-cli.php package:publish ../compatibility-dashboard/ci/tests/activation/test-package/ latest, just make sure you're using the correct path to the cd folder that contains the activation test package).
• Ensure the default performance tests package is published (you can publish it with php src/qit-cli.php package:publish ../compatibility-dashboard/ci/tests/performance latest).
• Run the activation tests with php src/qit-cli.php run:activation woocommerce, ensure they pass and no issues were found.
• Run the performance tests with php src/qit-cli.php run:performance automatewoo -vvv --local, ensure they pass and no issues were found.
• Create a local performance tests environment with php src/qit-cli.php env:up --environment_type=performance, check the environment is created successfully with 1k products and 100k orders.
• Create a local e2e tests environment with php src/qit-cli.php env:up, check the environment is created successfully without WooCommerce.

[MrJnrman]

I think we can move this to the top level src/Commands/CustomTests folder since it would fit that description. Being here would make it seem as if its the only local test.

[alopezari]

Precisely I did that before pushing the initial changes, but decided to revert it so you could review the changes in this class, since moving git would consider this file as a new one after moving it to a new path.

That said, after reorganizing all files from LocalTests, I wonder if it would be better to remove src/Commands/CustomTests and move those commands to src/Commands, as there only 3 of them within this folder:

• RunE2ECommand
• ShowReportCommand
• RunPerformanceTestsCommand

Even RunActivationTestsCommand, which might fit within the CustomTests folder too, is there placed in the root Commands folder. So, in my opinion, I'd remove the CustomTests folder and move these files to src/Commands. Please let me know if this sounds good to you before applying the changes.

[alopezari]

Hi @MrJnrman! As we agreed yesterday in the team meeting, I've removed the CustomTests folder and moved the files inside to the base Commands folder. That was covered in 4427515.

With this we should be ready to merge this PR. Please let me know if there is any objection! Cc @zhongruige , in case I'm missing anything (the more eyes, the better!)

[MrJnrman]

Great work here @alopezari and it tests great! Just one nitpick. Could we get rid of the LocalTests directory and merge its contents into the existing folders under the src/src directory? All test packages have a local capability so having only performance related logic in the directory is misleading.

[alopezari]

Thanks for raising this @MrJnrman!

The LocalTests still has some more classes apart from Performance-related ones:

• E2E/Result/TestResult.php
• EnvironmentRunner.php
• LocalTestRunNotifier.php
• PlaywrightToPuppeteerConverter.php
• PrepareDebugLog.php
• PrepareQMLog.php

I'll reorganize them within the src/src path. If you think it could be handled differently, please let me know, we can always change it or revert the commit.

[alopezari]

Updated in 01d72e9, @MrJnrman 🙌

[MrJnrman]

Thanks for these changes @alopezari! Retested and everything looks good! 🚢