Skip to content

Commit 8417a97

Browse files
joemcgilljoemcgill
committed
Build/Test Tools: Add a performance measurement workflow.
This adds a new GitHub Action workflow that measures a set of performance metrics on every commit, so we can track changes in the performance of WordPress over time and more easily identify changes that are responsible for significant performance improvements or regressions during development cycles. The workflow measures the homepage of a classic theme (Twenty Twenty-One) and a block theme (Twenty Twenty-Three) set up with demo content from the Theme Test Data project. Using the e2e testing framework, it makes 20 requests and records the median value of the following Server Timing metrics, generated by an mu-plugin installed as part of this workflow: - Total server response time - Server time before templates are loaded - Server time during template rendering In addition to measuring the performance metrics of the current commit, it also records performance metrics of a consistent version of WordPress (6.1.1) to be used as a baseline measurement in order to remove variance caused by the GitHub workers themselves from our reporting. The measurements are collected and displayed at https://www.codevitals.run/project/wordpress. Props adamsilverstein, mukesh27, flixos90, youknowriad, oandregal, desrosj, costdev, swissspidy. Fixes #57687. git-svn-id: https://develop.svn.wordpress.org/trunk@55459 602fd350-edb4-49c9-b593-d223f7449a82
1 parent 90f3acd commit 8417a97

File tree

12 files changed

+628
-0
lines changed

12 files changed

+628
-0
lines changed

Diff for: ‎.github/workflows/performance.yml

+231
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
name: Performance Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- trunk
7+
- '6.[2-9]'
8+
- '[7-9].[0-9]'
9+
tags:
10+
- '[0-9]+.[0-9]'
11+
- '[0-9]+.[0-9].[0-9]+'
12+
- '![45].[0-9].[0-9]+'
13+
- '!6.[01].[0-9]+'
14+
pull_request:
15+
branches:
16+
- trunk
17+
- '6.[2-9]'
18+
- '[7-9].[0-9]'
19+
20+
# Cancels all previous workflow runs for pull requests that have not completed.
21+
concurrency:
22+
# The concurrency group contains the workflow name and the branch name for pull requests
23+
# or the commit hash for any other events.
24+
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.head_ref || github.sha }}
25+
cancel-in-progress: true
26+
27+
env:
28+
# This workflow takes two sets of measurements — one for the current commit,
29+
# and another against a consistent version that is used as a baseline measurement.
30+
# This is done to isolate variance in measurements caused by the GitHub runners
31+
# from differences caused by code changes between commits. The BASE_TAG value here
32+
# represents the version being used for baseline measurements. It should only be
33+
# changed if we want to normalize results against a different baseline.
34+
BASE_TAG: '6.1.1'
35+
LOCAL_DIR: build
36+
37+
jobs:
38+
# Runs the performance test suite.
39+
#
40+
# Performs the following steps:
41+
# - Configure environment variables.
42+
# - Checkout repository.
43+
# - Set up Node.js.
44+
# - Log debug information.
45+
# - Install npm dependencies.
46+
# - Build WordPress.
47+
# - Start Docker environment.
48+
# - Log running Docker containers.
49+
# - Docker debug information.
50+
# - Install WordPress.
51+
# - Install WordPress Importer plugin.
52+
# - Import mock data.
53+
# - Update permalink structure.
54+
# - Install MU plugin.
55+
# - Run performance tests (current commit).
56+
# - Print performance tests results.
57+
# - Set the environment to the baseline version.
58+
# - Run baseline performance tests.
59+
# - Print base line performance tests results.
60+
# - Set the base sha.
61+
# - Set commit details.
62+
# - Publish performance results.
63+
# - Ensure version-controlled files are not modified or deleted.
64+
# - Dispatch workflow run.
65+
performance:
66+
name: Run performance tests
67+
runs-on: ubuntu-latest
68+
if: ${{ github.repository == 'WordPress/wordpress-develop' || github.event_name == 'pull_request' }}
69+
70+
steps:
71+
- name: Configure environment variables
72+
run: |
73+
echo "PHP_FPM_UID=$(id -u)" >> $GITHUB_ENV
74+
echo "PHP_FPM_GID=$(id -g)" >> $GITHUB_ENV
75+
76+
- name: Checkout repository
77+
uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # v3.3.0
78+
79+
- name: Set up Node.js
80+
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
81+
with:
82+
node-version-file: '.nvmrc'
83+
cache: npm
84+
85+
- name: Log debug information
86+
run: |
87+
npm --version
88+
node --version
89+
curl --version
90+
git --version
91+
svn --version
92+
locale -a
93+
94+
- name: Install npm dependencies
95+
run: npm ci
96+
97+
- name: Build WordPress
98+
run: npm run build
99+
100+
- name: Start Docker environment
101+
run: |
102+
npm run env:start
103+
104+
- name: Log running Docker containers
105+
run: docker ps -a
106+
107+
- name: Docker debug information
108+
run: |
109+
docker -v
110+
docker-compose -v
111+
docker-compose run --rm mysql mysql --version
112+
docker-compose run --rm php php --version
113+
docker-compose run --rm php php -m
114+
docker-compose run --rm php php -i
115+
docker-compose run --rm php locale -a
116+
117+
- name: Install WordPress
118+
run: npm run env:install
119+
120+
- name: Install WordPress Importer plugin
121+
run: npm run env:cli -- plugin install wordpress-importer --activate --path=/var/www/${{ env.LOCAL_DIR }}
122+
123+
- name: Import mock data
124+
run: |
125+
curl -O https://raw.githubusercontent.com/WPTT/theme-test-data/b9752e0533a5acbb876951a8cbb5bcc69a56474c/themeunittestdata.wordpress.xml
126+
npm run env:cli -- import themeunittestdata.wordpress.xml --authors=create --path=/var/www/${{ env.LOCAL_DIR }}
127+
rm themeunittestdata.wordpress.xml
128+
129+
- name: Update permalink structure
130+
run: |
131+
npm run env:cli -- rewrite structure '/%year%/%monthnum%/%postname%/' --path=/var/www/${{ env.LOCAL_DIR }}
132+
133+
- name: Install MU plugin
134+
run: |
135+
mkdir ./${{ env.LOCAL_DIR }}/wp-content/mu-plugins
136+
cp ./tests/performance/wp-content/mu-plugins/server-timing.php ./${{ env.LOCAL_DIR }}/wp-content/mu-plugins/server-timing.php
137+
138+
- name: Run performance tests (current commit)
139+
run: npm run test:performance
140+
141+
- name: Print performance tests results
142+
run: "node ./tests/performance/results.js"
143+
144+
- name: Set the environment to the baseline version
145+
run: |
146+
npm run env:cli -- core update --version=${{ env.BASE_TAG }} --force --path=/var/www/${{ env.LOCAL_DIR }}
147+
npm run env:cli -- core version --path=/var/www/${{ env.LOCAL_DIR }}
148+
149+
- name: Run baseline performance tests
150+
run: npm run test:performance -- --prefix=base
151+
152+
- name: Print base line performance tests results
153+
run: "node ./tests/performance/results.js --prefix=base"
154+
155+
- name: Set the base sha
156+
# Only needed when publishing results.
157+
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/trunk' }}
158+
uses: actions/github-script@98814c53be79b1d30f795b907e553d8679345975 # v6.4.0
159+
id: base-sha
160+
with:
161+
github-token: ${{ secrets.GITHUB_TOKEN }}
162+
script: |
163+
const baseRef = await github.rest.git.getRef({ owner: context.repo.owner, repo: context.repo.repo, ref: 'tags/${{ env.BASE_TAG }}' });
164+
return baseRef.data.object.sha;
165+
166+
- name: Set commit details
167+
# Only needed when publishing results.
168+
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/trunk' }}
169+
uses: actions/github-script@98814c53be79b1d30f795b907e553d8679345975 # v6.4.0
170+
id: commit-timestamp
171+
with:
172+
github-token: ${{ secrets.GITHUB_TOKEN }}
173+
script: |
174+
const commit_details = await github.rest.git.getCommit({ owner: context.repo.owner, repo: context.repo.repo, commit_sha: context.sha });
175+
return parseInt((new Date( commit_details.data.author.date ).getTime() / 1000).toFixed(0))
176+
177+
- name: Publish performance results
178+
# Only publish results on pushes to trunk.
179+
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/trunk' }}
180+
env:
181+
BASE_SHA: ${{ steps.base-sha.outputs.result }}
182+
COMMITTED_AT: ${{ steps.commit-timestamp.outputs.result }}
183+
CODEVITALS_PROJECT_TOKEN: ${{ secrets.CODEVITALS_PROJECT_TOKEN }}
184+
HOST_NAME: "codevitals.run"
185+
run: node ./tests/performance/log-results.js $CODEVITALS_PROJECT_TOKEN trunk $GITHUB_SHA $BASE_SHA $COMMITTED_AT $HOST_NAME
186+
187+
- name: Ensure version-controlled files are not modified or deleted
188+
run: git diff --exit-code
189+
190+
slack-notifications:
191+
name: Slack Notifications
192+
uses: WordPress/wordpress-develop/.github/workflows/slack-notifications.yml@trunk
193+
needs: [ performance ]
194+
if: ${{ github.repository == 'WordPress/wordpress-develop' && github.event_name != 'pull_request' && always() }}
195+
with:
196+
calling_status: ${{ needs.performance.result == 'success' && 'success' || needs.performance.result == 'cancelled' && 'cancelled' || 'failure' }}
197+
secrets:
198+
SLACK_GHA_SUCCESS_WEBHOOK: ${{ secrets.SLACK_GHA_SUCCESS_WEBHOOK }}
199+
SLACK_GHA_CANCELLED_WEBHOOK: ${{ secrets.SLACK_GHA_CANCELLED_WEBHOOK }}
200+
SLACK_GHA_FIXED_WEBHOOK: ${{ secrets.SLACK_GHA_FIXED_WEBHOOK }}
201+
SLACK_GHA_FAILURE_WEBHOOK: ${{ secrets.SLACK_GHA_FAILURE_WEBHOOK }}
202+
203+
failed-workflow:
204+
name: Failed workflow tasks
205+
runs-on: ubuntu-latest
206+
needs: [ performance, slack-notifications ]
207+
if: |
208+
always() &&
209+
github.repository == 'WordPress/wordpress-develop' &&
210+
github.event_name != 'pull_request' &&
211+
github.run_attempt < 2 &&
212+
(
213+
needs.performance.result == 'cancelled' || needs.performance.result == 'failure'
214+
)
215+
216+
steps:
217+
- name: Dispatch workflow run
218+
uses: actions/github-script@98814c53be79b1d30f795b907e553d8679345975 # v6.4.0
219+
with:
220+
retries: 2
221+
retry-exempt-status-codes: 418
222+
script: |
223+
github.rest.actions.createWorkflowDispatch({
224+
owner: context.repo.owner,
225+
repo: context.repo.repo,
226+
workflow_id: 'failed-workflow.yml',
227+
ref: 'trunk',
228+
inputs: {
229+
run_id: '${{ github.run_id }}'
230+
}
231+
});

Diff for: ‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ wp-tests-config.php
1414
/tests/phpunit/data/plugins/wordpress-importer
1515
/tests/phpunit/data/.trac-ticket-cache*
1616
/tests/qunit/compiled.html
17+
/tests/performance/**/*.test.results.json
1718
/src/.wp-tests-version
1819
/node_modules
1920
/npm-debug.log

Diff for: ‎package.json

+1
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,7 @@
176176
"env:cli": "node ./tools/local-env/scripts/docker.js run cli",
177177
"env:logs": "node ./tools/local-env/scripts/docker.js logs",
178178
"env:pull": "node ./tools/local-env/scripts/docker.js pull",
179+
"test:performance": "node ./tests/performance/run-tests.js",
179180
"test:php": "node ./tools/local-env/scripts/docker.js run -T php composer update -W && node ./tools/local-env/scripts/docker.js run php ./vendor/bin/phpunit",
180181
"test:e2e": "node ./tests/e2e/run-tests.js",
181182
"test:visual": "node ./tests/visual-regression/run-tests.js",

Diff for: ‎tests/performance/config/bootstrap.js

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
/**
2+
* WordPress dependencies.
3+
*/
4+
import {
5+
clearLocalStorage,
6+
enablePageDialogAccept,
7+
setBrowserViewport,
8+
} from '@wordpress/e2e-test-utils';
9+
10+
/**
11+
* Timeout, in seconds, that the test should be allowed to run.
12+
*
13+
* @type {string|undefined}
14+
*/
15+
const PUPPETEER_TIMEOUT = process.env.PUPPETEER_TIMEOUT;
16+
17+
// The Jest timeout is increased because these tests are a bit slow.
18+
jest.setTimeout( PUPPETEER_TIMEOUT || 100000 );
19+
20+
async function setupBrowser() {
21+
await clearLocalStorage();
22+
await setBrowserViewport( 'large' );
23+
}
24+
25+
/*
26+
* Before every test suite run, delete all content created by the test. This ensures
27+
* other posts/comments/etc. aren't dirtying tests and tests don't depend on
28+
* each other's side-effects.
29+
*/
30+
beforeAll( async () => {
31+
enablePageDialogAccept();
32+
33+
await setBrowserViewport( 'large' );
34+
await page.emulateMediaFeatures( [
35+
{ name: 'prefers-reduced-motion', value: 'reduce' },
36+
] );
37+
} );
38+
39+
afterEach( async () => {
40+
await setupBrowser();
41+
} );

Diff for: ‎tests/performance/jest.config.js

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
const config = require( '@wordpress/scripts/config/jest-e2e.config' );
2+
3+
const jestE2EConfig = {
4+
...config,
5+
setupFilesAfterEnv: [
6+
'<rootDir>/config/bootstrap.js',
7+
],
8+
globals: {
9+
// Number of requests to run per test.
10+
TEST_RUNS: 20,
11+
}
12+
};
13+
14+
module.exports = jestE2EConfig;

0 commit comments

Comments
 (0)