Add Jetpack performance testing CI infrastructure

[LiamSarsfield]

Addresses HOG-438: Create Jetpack Performance Tooling for LCP

Proposed changes:

• Add performance testing infrastructure under tools/performance/ to measure wp-admin dashboard LCP (Largest
  Contentful Paint) with Jetpack connected
• Uses Docker for isolated WordPress environment with simulated WordPress.com connection (fake tokens + mocked API
  with 200ms latency)
• Includes CPU throttling calibration for consistent results across different machines
• Posts metrics to CodeVitals for tracking over time

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?
• Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with
  a script to run)?

Jetpack product discussion

pc9hqz-3Rb-p2

Does this pull request change what data or activity we track or use?

No

Testing instructions:

Prerequisites: Docker running, Node 18+

cd tools/performance
pnpm install
pnpm exec playwright install chromium
pnpm calibrate
pnpm test -- --skip-codevitals

The test suite will automatically clone the pre-built plugin from jetpack-production on first run.

Expected output: LCP measurement for wp-admin dashboard with Jetpack connected (simulated)

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the add/perf-testing-ci-mvp branch.
• To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack add/perf-testing-ci-mvp

bin/jetpack-downloader test jetpack-mu-wpcom-plugin add/perf-testing-ci-mvp

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[jp-launch-control]

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

[LiamSarsfield]

Hey @anomiex👋 would you mind taking a look at this when you get a chance?

This is the performance testing infrastructure I built during HACK week (more details here pc9hqz-3Rb-p2) it measures wp-admin dashboard LCP for Jetpack and posts results to CodeVitals.
It's a big PR, sorry about that. Most of it breaks down into:

• scripts/ - Node.js orchestration and Playwright measurement code
• docker/ - Docker Compose setup for WordPress + Jetpack environment
• docker/mu-plugins/simulate-wpcom-connection.php - mu-plugin that fakes Jetpack connection and mocks WP.com API
  responses

The mu-plugin is probably the most relevant bit for review from a Jetpack perspective as it intercepts pre_http_request to return mock responses for various WP.com endpoints.

Happy to walk through any of it if that's easier.

[anomiex]

Seems ok from a monorepo perspective. I didn't look too closely at the code.

If you're wanting a review of the faked-connection stuff in the mu-plugin, @Automattic/jetpack-vulcan would be the team to ask.

[anomiex]

I wonder if it'd make more sense to run it in Actions rather than TeamCity, on each commit to trunk instead of backfilling weekly.

If you're running it in TeamCity, are you looking at the monorepo, or at https://github.com/Automattic/jetpack-production which has the already-built plugin?

[kraftbj]

One of the exploration intents is to see about reusing work. With our GHE instance, TeamCity is the runner we have to use, so seeing the pros/cons of TC here.

[LiamSarsfield]

I wonder if it'd make more sense to run it in Actions rather than TeamCity, on each commit to trunk instead of backfilling weekly.

GitHub Actions runners can have unpredictable performance variability due to shared infrastructure. For detecting small regressions (5-10%), this noise can drown out minor regressions over time. The TeamCity build runs on a dedicated machine with no other agents interfering.

If you're running it in TeamCity, are you looking at the monorepo, or at Automattic/jetpack-production which has the already-built plugin?

The monorep is used instead of the already-built plugin mainly so we can leverage commit level tracking, so we can have the ability to bisect when issues appear (where needed). It may not be needed for now though, but you can see within CodeVitals that a point can be clicked and we can see the commit details when clicking on the commit SHA.

[anomiex]

Note that every commit that changes anything in Jetpack gets mirrored to Automattic/jetpack-production, absent rare cases where something goes wrong with the mirroring. Each commit there also includes a footer like Upstream-Ref: Automattic/jetpack@d5b54134bb471f3a54d04b12e128e6e0e2d77bde to make it easy to find the corresponding monorepo commit.

It's up to you, but it may save having to build each monorepo commit before you can test it.

It'll also either save you testing commits that don't affect Jetpack-the-plugin at all or save you having to maintain code to try to identify which ones do. I see your P2 post mentions

that touched PHP files in Jetpack or packages

That would unnecessarily test changes to packages like packages/jetpack-mu-wpcom, unless you have a list of packages to ignore. It could also miss JS-only changes that might still affect the LCP timing.

[LiamSarsfield]

Note that every commit that changes anything in Jetpack gets mirrored to Automattic/jetpack-production, absent rare cases where something goes wrong with the mirroring. Each commit there also includes a footer like Upstream-Ref: d5b5413 to make it easy to find the corresponding monorepo commit.

Actually yes! I should have checked this before hand, I didn't realise Automattic/jetpack-production has bissectable commits like this. It'll also help regarding the issue you mentioned for testing commits that don't affect Jetpack-the-plugin also. Thanks, I'll update the TeamCity build to use that instead.

[LiamSarsfield]

Updated to use jetpack-production, the implementation now:

1. Clones jetpack-production instead of building from the monorepo
2. Parses Upstream-Ref from each mirror commit to track the original monorepo SHA in CodeVitals
3. Removes file filtering from the TC scheduler

The monorepo VCS root is still used for the tools/performance/ scripts, but the plugin itself comes directly from the pre-built mirror.

[anomiex]

That doesn't matter, eslint will look in tools/performance/node_modules for files under here.

Also, when I remove this and run eslint, it doesn't seem to complain about anything.

[LiamSarsfield]

Nice catch! Removed the import/no-unresolved rule and its comment. Also simplified the config to match tools/cli/eslint.config.mjs

[kraftbj]

I'm not sure where it came from; I was given a noticed that MacOS's rsync doesn't handle symlinks well and stating I should brew install rsync. It let me proceed, but it failed.

Installing rsync via brew install rsync and trying again worked. I don't think it's a blocker but wanted to note it.

[anomiex]

I'm not sure where it came from; I was given a noticed that MacOS's rsync doesn't handle symlinks well and stating I should brew install rsync. It let me proceed, but it failed.

Probably from

jetpack/tools/cli/commands/rsync.js

Lines 45 to 93 in 55e48ba

	// Apple ships a special fork of openrsync, which has various quirks.
	// In particular, it doesn't handle symlink recursion well, which breaks
	// in macOS 15.4 and copies child node_modules dirs in 15.5.
	//
	// See also:
	// * p1742486518169009-slack-CDLH4C1UZ
	// * https://github.com/apple-oss-distributions/rsync/tree/main/openrsync
	if ( os.platform() === 'darwin' ) {
	const { stdout: rsyncVersion } = await execa( 'rsync', [ '--version' ] );
	isOpenrsync = rsyncVersion.indexOf( 'openrsync' ) >= 0;
	if ( isOpenrsync ) {
	const { stdout: macOS_version } = await execa( 'sw_vers', [ '--productVersion' ] );
	if ( macOS_version === '15.4' ) {
	console.error(
	chalk.red(
	'The implementation of rsync in macOS 15.4 is unable to properly sync symlinks.'
	)
	);
	console.error( chalk.red( 'Please install standard rsync (e.g. `brew install rsync`).' ) );
	process.exit( 1 );
	} else {
	console.error(
	chalk.yellow(
	'The implementation of rsync in macOS is unable to properly sync symlinks.'
	)
	);
	console.error(
	chalk.yellow( 'Installing standard rsync (e.g. `brew install rsync`) is recommended.' )
	);
	if ( argv.nonInteractive ) {
	process.exit( 1 );
	}
	console.error();
	await enquirer
	.prompt( {
	type: 'confirm',
	name: 'proceedWithOpenrsync',
	message:
	'Continuing will not break anything, but will copy many unneeded files.\nProceed to sync files?',
	initial: false,
	} )
	.then( answer => {
	if ( ! answer.proceedWithOpenrsync ) {
	process.exit( 0 );
	}
	} );
	}
	}
	}

[kraftbj]

Probably from

Yup! Thanks! I scanned the PR but didn't check the existing commands. I wouldn't consider it a blocker for the PR, but it is required to use the fully-leaded version of rsync, not the Mac variant.

[LiamSarsfield]

Yup! Thanks! I scanned the PR but didn't check the existing commands. I wouldn't consider it a blocker for the PR, but it is required to use the fully-leaded version of rsync, not the Mac variant.

Ah nice catch, I must have included already installed rsync previously hence why I missed it, I've updated the testing instructions accordingly.

Actually, this is no longer relevant - I've switched to using jetpack-production (the pre-built mirror) instead of
building from the monorepo, so rsync is no longer needed at all. Updated the testing instructions to reflect the
simpler prerequisites (just Docker and Node 18+).

[LiamSarsfield]

Hey @Automattic/jetpack-vulcan 👋

I've been working on performance testing infrastructure for Jetpack (measuring wp-admin LCP with Jetpack connected).
Part of this involves an mu-plugin that simulates a WordPress.com connection without actually connecting to WP.com.
Could someone from the team review the mock implementation at tools/performance/docker/mu-plugins/simulate-wpcom-connection.php?

It currently:

• Sets up fake blog/user tokens via Jetpack_Options
• Intercepts HTTP requests to *.wordpress.com and *.wp.com
• Returns mock responses for common endpoints (token health, site info, stats, sync, etc.)
• Adds configurable latency (default 200ms) to simulate real-world conditions

Specifically looking for feedback on:

1. Are the fake connection tokens set up correctly?
2. Are we missing any critical API endpoints that get called on wp-admin load?
3. Any concerns with this approach for performance testing?

[fgiannar]

Hi @LiamSarsfield ,

Thanks for the ping and working on this!

Are the fake connection tokens set up correctly?

Yes 👍

Are we missing any critical API endpoints that get called on wp-admin load?

This would depend on the Jetpack plugin module configuration. I can confirm the endpoints that are called by Connection and Sync packages, but we can't know how each consumer of the Jetpack Connection behaves.
I noticed that you only enable modules that don't require a JP Connection but I wonder if we should enable all of them to get a realistic worst case scenario.
One way to check the remote calls to WPCOM on every page load, would be to sandbox your environment and monitor debug.log as we do log all sandboxed requests to WPCOM there.

That said, it might make sense to add some logging that would answer this question within the performance testing logic itself? This could be helpful in case more endpoints are added in the future, that we don't handle within the testing infrastructure.

Any concerns with this approach for performance testing?

This is not a blocker, but you could consider refactoring get_mock_response to avoid the if else logic and use eg a factory for setting up the fake endpoints.
One additional idea I had, is around the latency. Atm my understanding is that we assume a generic latency for each endpoint. If we extracted each fake endpoint definition to each own class we could potentially set the latency per endpoint using actual real world data we have on WPCOM.
As an example, the jetpack-sync-actions endpoint has a ~470ms median response time and ~3s for the 95th percentile.
We could take it even one step further and define a median and p95 latency per endpoint and repeat our tests for both cases to simulate a normal scenario and a site under stress.

[LiamSarsfield]

@fgiannar Thanks for the thorough review!

I noticed that you only enable modules that don't require a JP Connection but I wonder if we should enable all of them to get a realistic worst case scenario.

Good point. I initially took the conservative approach to avoid errors from modules expecting real WP.com responses, but enabling all modules would give us a more realistic measurement. I'll look into expanding the module list and adding mock responses for any additional endpoints they require.

It might make sense to add some logging that would answer this question within the performance testing logic itself?

Great idea, I added logging for any intercepted requests that hit the fallback response. That way it'll catch any unhandled endpoints as they appear instead of discovering them later.

You could consider refactoring get_mock_response to avoid the if else logic and use eg a factory for setting up the fake endpoints.

Agreed that the current implementation is a bit unwieldy. I'll refactor to a registry/factory pattern which will also make it easier to add per endpoint configuration.

We could take it even one step further and define a median and p95 latency per endpoint

Love this idea. Using actual latency data would make the measurements much more representative. I created a follow-up issue to:

1. Extract endpoint definitions to a registry with configurable latency
2. Add real-world latency values from WP.com metrics
3. Consider test modes for median vs p95 scenarios

For now I'll focus on the logging improvement as a quick win, thanks again for the detailed feedback! 🙏