Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.98%. Comparing base (ff428a3) to head (7aff9b2).

@@           Coverage Diff           @@
##            trunk    #1838   +/-   ##
=======================================
  Coverage   65.98%   65.98%           
=======================================
  Files          88       88           
  Lines        6896     6896           
=======================================
  Hits         4550     4550           
  Misses       2346     2346           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: ShyamGadde <[email protected]>
Co-authored-by: westonruter <[email protected]>
Co-authored-by: felixarntz <[email protected]>
Co-authored-by: swissspidy <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

How about we implement this change so that it only applies to pull requests, but for merges to our main branches it would still run all test suites? IMO this would be the best of both worlds:

This should already be the case with this logic:

If any of the config files are changed or if the changes are not part of a PR (i.e. a push to trunk) then all plugins are tested.

@westonruter Thanks, I missed that. LGTM in that regard then 👍

In that case, this leaves the outstanding point @swissspidy raised about confusing codecov reports.

In that case, this leaves the outstanding point @swissspidy raised about confusing Codecov reports.

I think a good solution to this would be the carryforward flag. Codecov provides this flag to allow previously uploaded coverage reports to persist across runs, meaning untested files retain their last known coverage instead of being omitted from the report, which can lead to misleading drops in coverage. However, this won't work in the current setup. Right now, we are using two flags, single and multisite, and partial reports are being uploaded under these flags, which results in the PR comments showing a drop in reported code coverage (as untested files appear as uncovered). For carryforward to work, I believe no reports should be uploaded using the flag being carried forward; otherwise, the newly uploaded report will override the carried-forward coverage.

Another possible approach is to only show patch coverage in the PR comment, as that isn’t affected by partial testing. This would be the simpler solution, ensuring that PR reports remain clear and focused on the actual changes. Since full test runs still occur on merge commits and commits to trunk, the overall project coverage will still be recalculated periodically.

That said, the first approach (using carryforward) might still work with some adjustments. Instead of using just single and multisite, we could assign unique flags per plugin so that their coverage data is carried forward independently. That way, when a report for a particular plugin isn’t uploaded, its previous report would get carried forward automatically. (I'm still reviewing Codecov's documentation to confirm if this setup would work as expected.)

Screenshots:

1. With partial testing:
   
2. PR comment with only patch coverage:
   

cc @felixarntz @westonruter

So, after experimenting with Codecov's coverage reporting and reviewing its documentation, I’ve concluded that if we want to continue with partial testing, there are two possible ways to handle Codecov’s potentially confusing reports—depending on whether we want to display the project's overall coverage in the PR comment or not.

Option 1: Show Project Coverage in PR Comment

If we want to display project coverage in the PR comment, we need to use the carryforward flag to handle missing reports. But that would mean that we will not be able to use the flags such as single and multisite as we will be uploading partial reports with these flags when we don't test all the plugins which will lead to the confusing reports.

To address this, we can:

• Remove the single and multisite flags (also delete them from the Codecov dashboard).
• Instead, introduce plugin-specific flags and group single-site and multisite reports under those flags.

That way when we don't test a particular plugin the result will look something like in the below screenshot and even the project coverage which is displayed won't be affected when partial testing.

I personally find the reports (for both single and multisite) grouped by plugin flags more intuitive to read (because of how Codecov merges these reports) than the old reporting format, where we see 38.45% for single and 65.86% for multisite in PR comments as seen below:

For this to work, we'll have to change the workflow and Codecov's config file somewhat as follows:

• Changes to php-test-plugins.yml

diff --git a/.github/workflows/php-test-plugins.yml b/.github/workflows/php-test-plugins.yml
index 15e2101d..9211255f 100644
--- a/.github/workflows/php-test-plugins.yml
+++ b/.github/workflows/php-test-plugins.yml
@@ -128,6 +128,19 @@ jobs:
         run: npm run wp-env run tests-cli -- --env-cwd="wp-content/plugins/$(basename $(pwd))" composer install --no-interaction --no-progress
       - name: Update Composer Dependencies
         run: composer update --with-all-dependencies --no-interaction --no-progress
+      - name: Download Codecov CLI
+        if: ${{ matrix.coverage == true }}
+        run: |
+          # Create a temporary directory for GPG operations
+          mkdir -p .gpg
+
+          curl https://keybase.io/codecovsecurity/pgp_keys.asc | gpg --homedir .gpg --no-default-keyring --keyring trustedkeys.gpg --import
+          curl -Os https://cli.codecov.io/latest/linux/codecov
+          curl -Os https://cli.codecov.io/latest/linux/codecov.SHA256SUM
+          curl -Os https://cli.codecov.io/latest/linux/codecov.SHA256SUM.sig
+          gpg --homedir .gpg --no-default-keyring --keyring trustedkeys.gpg --verify codecov.SHA256SUM.sig codecov.SHA256SUM
+          shasum -a 256 -c codecov.SHA256SUM
+          chmod +x codecov
       - name: Install PHPUnit
         run: |
           if [ "${{ matrix.php }}" == "8.2" ]; then
@@ -141,6 +154,13 @@ jobs:
           if [ "${{ matrix.coverage }}" == "true" ]; then
             for PLUGIN in ${{ steps.changed-plugins.outputs.all_changed_plugins }}; do
               npm run test-php:$PLUGIN -- -- -- --coverage-clover=./single-site-reports/coverage-$PLUGIN.xml
+
+              # Upload coverage report to Codecov.
+              ./codecov --verbose upload-process --disable-search --fail-on-error \
+              -t ${{ secrets.CODECOV_TOKEN }} \
+              -n ${{ matrix.php }}-$PLUGIN-single-site-coverage \
+              -F $PLUGIN \
+              -f ./single-site-reports/coverage-$PLUGIN.xml
             done
           else
             for PLUGIN in ${{ steps.changed-plugins.outputs.all_changed_plugins }}; do
@@ -152,27 +172,16 @@ jobs:
           if [ "${{ matrix.coverage }}" == "true" ]; then
             for PLUGIN in ${{ steps.changed-plugins.outputs.all_changed_plugins }}; do
               npm run test-php-multisite:$PLUGIN -- -- -- --coverage-clover=./multisite-reports/coverage-multisite-$PLUGIN.xml
+
+              # Upload coverage report to Codecov.
+              ./codecov --verbose upload-process --disable-search --fail-on-error \
+              -t ${{ secrets.CODECOV_TOKEN }} \
+              -n ${{ matrix.php }}-$PLUGIN-multisite-coverage \
+              -F $PLUGIN \
+              -f ./multisite-reports/coverage-multisite-$PLUGIN.xml
             done
           else
             for PLUGIN in ${{ steps.changed-plugins.outputs.all_changed_plugins }}; do
               npm run test-php-multisite:$PLUGIN
             done
           fi
-      - name: Upload single site coverage reports to Codecov
-        if: ${{ matrix.coverage == true }}
-        uses: codecov/codecov-action@v5
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          directory: ./single-site-reports
-          flags: single
-          name: ${{ matrix.php }}-single-site-coverage
-          fail_ci_if_error: true
-      - name: Upload multisite coverage reports to Codecov
-        if: ${{ matrix.coverage == true }}
-        uses: codecov/codecov-action@v5
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          directory: ./multisite-reports
-          flags: multisite
-          name: ${{ matrix.php }}-multisite-coverage
-          fail_ci_if_error: true

• Changes to codecov.yml

diff --git a/codecov.yml b/codecov.yml
index f91d5654..83035cea 100644
--- a/codecov.yml
+++ b/codecov.yml
@@ -12,3 +12,38 @@ coverage:
       default:
         threshold: 80%
         informational: true
+comment:
+  hide_comment_details: true
+  hide_project_coverage: false
+  show_carryforward_flags: true
+flag_management:
+  default_rules:
+    carryforward: true
+  individual_flags:
+    - name: auto-sizes
+      paths:
+      - plugins/auto-sizes/
+    - name: dominant-color-images
+      paths:
+        - plugins/dominant-color-images/
+    - name: embed-optimizer
+      paths:
+        - plugins/embed-optimizer/
+    - name: image-prioritizer
+      paths:
+        - plugins/image-prioritizer/
+    - name: optimization-detective
+      paths:
+        - plugins/optimization-detective/
+    - name: performance-lab
+      paths:
+        - plugins/performance-lab/
+    - name: speculation-rules
+      paths:
+        - plugins/speculation-rules/
+    - name: web-worker-offloading
+      paths:
+        - plugins/web-worker-offloading/
+    - name: webp-uploads
+      paths:
+        - plugins/webp-uploads/

Option 2: Show Only Patch Coverage in PR Comment

A simpler alternative is to only show patch coverage in PR comments. This won't show project-wide coverage fluctuations when running partial tests and keeps the report focused on the actual changes.

Additionally, if we were to only show "Patch coverage", we could:

• Limit Codecov's "Project Coverage" status check to only commits to the trunk branch, preventing confusing reports in Codecov's dashboard and GitHub's Actions runs. Since we run all tests on commits to trunk, everything should function normally there.
• Consider uploading reports without the single and multisite flags, since they are optional. This would prevent temporary drops in coverage reporting under these flags.

With this approach, the PR comment will look like this:

Given these options, assuming we still want to do partial testing, which approach would be preferable?

cc @felixarntz @westonruter

@ShyamGadde Option 2 sounds like the better approach to me! Thank you for doing that research.