Build/Test Tools: Add plugin compatibility testing workflow. - #13198
Build/Test Tools: Add plugin compatibility testing workflow.#13198adamsilverstein wants to merge 11 commits into
Conversation
Core CI covers core itself, but nothing checks that a new version of WordPress can still boot with popular plugins active. When a plugin's assumptions about core stop holding the result is a fatal error on every request, which is a white screen for real sites and is only discovered after release. Add a workflow that fetches the most popular plugins from the WordPress.org API at run time, then installs and activates each one on its own against the version of WordPress under test. A fatal is caught whether it happens on activation, while WP-CLI loads WordPress, on a front end or login request, or in the debug log, so a white screen with error display turned off is still detected. Failures are reported per plugin in the workflow summary and one broken plugin never stops the rest of the shard from being tested. Plugins that cannot be downloaded are reported as skipped rather than failed so that a network flake does not turn the run red. The run is manual or weekly rather than part of every commit, since a third party plugin breaking should be a signal to release leads, not a red check on unrelated work.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The first run of this workflow reported four failures that were not fatals. Core refuses to activate a plugin whose `Requires Plugins` dependency is missing, which every WooCommerce extension hits when plugins are tested one at a time, and WP-CLI exits non-zero when a plugin redirects while loading. Both are correct behaviour, so record them as skipped and reserve a failure for an actual fatal. A front end request also reported the nonsense status "200000", because the curl fallback appended to output curl had already written. Capture the exit code separately so a stalled transfer is reported as what it is. The stall itself came from WordPress spawning WP-Cron as a loopback request that the single threaded built-in server could not answer while still serving the request that spawned it. Disable WP-Cron and give the server workers so plugin loopback requests cannot deadlock it. Ignore the zizmor unpinned image finding on the database service, which cannot be pinned to a digest while the version is an input.
The WordPress.org API caps `per_page` at 250 and quietly returns 250 for anything larger, so asking for more than that silently tested fewer plugins than requested. Page through the API instead and trim to the requested count, de-duplicating across pages because popularity ordering can shift between two requests. Reject a count above 1000 with a clear message rather than truncating without saying so. Size the shards to the plugin count rather than always splitting into five, so a run of 10 does not spin up five near empty jobs and a run of 250 is not squeezed into the same five. Point pull request and push runs at the latest stable release with a small count. Those runs exist to check that this workflow still works, and a genuine ecosystem fatal against nightly should not sit as a red check on every later change to these files.
A run that builds a single shard logged "across 1 shards".
While Tide is still running, its been generally unsupported for years and I've long considered trying to get it shut down as I'm unsure how many entities are even making use of its available audit data set (with the exception of the PHP Compat Checker plugin). If there was interest in having this functionality more formally supported within Tide and then in consuming the resulting data, then that seems great to me but we'd likely want to get someone / a team / a sponsor to help ensure Tide continues to be supported after this implementation completes. |
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
This seems worth a follow-up to me given they make the popular list. So in general, if a plugin has |
|
Claude ran the workflow at full scale and pulled this from the logs: Proof the check does its job - a full 200 plugin run on a fork caught a real fatal.
Full run results: 186 passed, 1 failed, 13 skipped (WooCommerce add-ons with an unmet Worth noting how the triage played out: the failure reproduces identically on latest and nightly, so it is a pre-existing plugin bug rather than a core regression. WP-CLI loads WordPress inside a function scope, so the plugin's file-level |
Good suggestion, thanks! |
…sting. Testing each plugin on its own meant core refused to activate anything with a `Requires Plugins` header, so every WooCommerce extension was recorded as skipped without ever being loaded. Those extensions are a large slice of the most popular plugins, which left a gap in exactly the part of the ecosystem the workflow exists to watch. Read the `Requires Plugins` header with WordPress' own parser, install and activate what it names, and only then activate the plugin under test. Chains are followed a level at a time so a dependency declaring its own is installed too, and anything already present is left alone, which terminates a circular declaration. Dependencies mean the plugin under test is no longer alone on the site, so the front page and login screen are checked with only the dependencies active first. A site already broken at that point is the dependency's doing, and the plugin is skipped rather than blamed for it. The debug log is cleared at the same point for the same reason. The results table gains a column naming what else was active so a failure can be read in context. Installing and activating a dependency the size of WooCommerce is not quick, so the job timeout goes from 30 to 45 minutes.
|
Claude took a swing at this one, here is where it landed:
|
Trying this workflow out meant taking whatever the popularity query returned, so anyone wanting to check one plugin against a release candidate, or to reproduce a failure from an earlier run, had to sit through a full run and read past everything else in the shard. Reviewers could not run it at all: the repository guard skips both jobs everywhere but this repository, and dispatching it here needs write access. Add an optional `plugin-slugs` input that replaces the directory query with a list written by hand. Commas and whitespace both separate slugs so a list can be pasted in however it was written down, and a value outside the character set the directory uses stops the run immediately rather than being reported as an impossible download several minutes later. The list goes through the same sharding as a fetched one, so naming forty plugins still splits across two jobs. Let a dispatched run through the repository guard wherever it is triggered, which is what makes a fork useful for trying this out. Two runs dispatched against the same version of WordPress used to land in the same concurrency group and cancel each other, which is wrong once the plugins being tested can differ between them. Dispatched runs are now grouped by run id.
…dd/plugin-compatibility-workflow
`slack-notifications.yml` no longer accepts `SLACK_GHA_TIMEOUT_WEBHOOK`, and every other caller stopped passing it, so actionlint failed on this workflow once trunk was merged in.
…a terminal. Move the per-plugin checks out of the reusable workflow and into tools/plugin-compatibility/test-plugins.sh, so that reproducing a result from a run no longer means dispatching a workflow and waiting for it. The script provisions a database and a PHP with WP-CLI in throwaway containers by default, so nothing but Docker is needed locally. The workflow passes --no-docker and runs the same code against the runner's own PHP and database service, which keeps one copy of the logic that decides whether a plugin passed.
WP-CLI requires wp-settings.php from inside a method, so a plugin that assigns a variable at file scope and reads it back with global later finds nothing there. eps-301-redirects does exactly that and fatals under WP-CLI on a fresh install, while the front page, the login screen and the debug log are all clean. Failing it reported breakage that no visitor would ever see. Move the WP-CLI boot check after the HTTP and debug log checks and stop letting it fail a plugin on its own. A plugin that is healthy over HTTP now passes with the WP-CLI fatal recorded as a note. Anything that genuinely fatals on load still fails, because it fails the HTTP checks. The reorder also matters because the WP-CLI fatal is written to the debug log, which the log check would otherwise pick up as a failure.
|
Testing the local runner turned up something that changes the story on Claude chased the discrepancy down, here is what came back:
Sorry for the noise on the earlier claim - the check works, but that particular plugin was the wrong example for it. |
4 participants
Claude Code built this workflow, requirements and steering from me:
Adds a workflow that installs the top plugins from the WordPress.org directory one at a time against a given version of WordPress and checks that nothing fatals. The idea is to catch the WP Rocket class of breakage - a popular plugin fataling on every request against a new core version, taking real sites offline - while there is still time to fix core or reach the plugin author. See adamsilverstein#64 and https://x.com/austinginder/status/2090199834787541074
Trac ticket: https://core.trac.wordpress.org/ticket/65920
How it works
plugin-compatibility.ymlquerieshttps://api.wordpress.org/plugins/info/1.2/for the most popular plugins at run time, so there is no list to go stale. The API capsper_pageat 250 and returns 250 without complaint for anything larger, so the fetch pages through the API and trims to the requested count, de-duplicating across pages because popularity ordering can shift between two requests. Counts above 1000 are rejected with a message rather than quietly truncated.Shards are sized rather than counted: it aims for 25 plugins per shard up to 10 shards, so a run of 10 does not spin up five near empty jobs and a run of 250 is not squeezed into the same five. Past 250 the shards get longer instead of more numerous.
Each shard calls
reusable-plugin-compatibility.yml, which installs WordPress withWP_DEBUGandWP_DEBUG_LOGon, startsphp -S, and then for each plugin installs it, activates it, runswp eval, requests/and/wp-login.php, checkswp-content/debug.log, and removes the plugin before moving to the next one.The checks themselves live in
tools/plugin-compatibility/test-plugins.shrather than inside the YAML, so the same code that decides whether a plugin passed can be run from a terminal. Reproducing a red shard is then a one line command instead of a dispatched run and a wait. See below.A plugin that declares
Requires Pluginsgets those dependencies installed and activated first, otherwise core refuses to activate it and it goes untested. The header is read withget_plugin_data()rather than a grep, so the same rules apply here as when core decides whether a plugin's requirements are met. Chains are followed a level at a time in case a dependency declares one of its own, and anything already installed is left alone, which is what terminates a circular declaration.Dependencies mean the plugin under test is no longer alone on the site, so the front page and login screen are requested with only the dependencies active before it is activated. A site already broken at that point is the dependency's doing, and the plugin under test is skipped rather than blamed for it; the debug log is cleared at the same point for the same reason. The results table carries an "Also active" column so a failure can be read in context.
What a real request does is what decides whether a plugin passed. The WP-CLI boot check runs last and only ever adds a note, because WP-CLI requires
wp-settings.phpfrom inside a method: a plugin that assigns a variable at file scope and reads it back withgloballater finds nothing there.eps-301-redirectsis one of those, and it fatals under WP-CLI on a fresh install while the front page, the login screen and the debug log stay clean. Failing it would report breakage no visitor could ever hit, which is the opposite of what this is for.WP_DEBUG_DISPLAYis left off on purpose so the site behaves the way a production site does - a fatal is an empty page and an HTTP 500 rather than a printed stack trace. The fatal error handler is disabled too, otherwise recovery mode swallows the fatal and deactivates the plugin mid-test.Runs are
workflow_dispatch(so a release lead can point it at a beta or RC as part of the pre-release checklist, and pick how many plugins to cover) and weekly against nightly. It is signal-only, not a check on every commit - a third party plugin breaking shouldn't turn core CI red on unrelated work.A dispatched run can also name the plugins to test rather than taking them from the directory, which is what makes it usable for a single plugin: checking one against an RC takes about a minute and a half instead of a full run. See below.
What the first CI run turned up
The workflow ran on this PR through its own
pull_requestpaths trigger, which was the point of including that trigger. Three of five shards passed and two failed, and the failures were worth having:Four WooCommerce extensions (
woocommerce-payments,google-listings-and-ads,woocommerce-paypal-payments,woocommerce-gateway-stripe) were reported as failed when core had simply refused to activate them -Requires Plugins: woocommerce, which is unmet by design when plugins are tested one at a time. Those now havewoocommerceinstalled and activated alongside them and are tested properly.wp-reviews-plugin-for-googlewas reported as failed because it callswp_safe_redirect()while loading, which makes WP-CLI exit non-zero without anything being broken. Thewp evalcheck now only fails on an actual fatal.instagram-feedwas reported with the nonsense statusHTTP 200000, which was a real bug - the curl fallback appended to output curl had already written. That is fixed by capturing the exit code separately. Digging into why the transfer stalled found something more interesting: WordPress spawned WP-Cron as a loopback request, and the single threaded built-in server could not answer it while still serving the request that spawned it, so the two deadlocked for the full 60 second curl timeout. WP-Cron is now disabled and the server getsPHP_CLI_SERVER_WORKERS.Underneath that deadlock was a genuine fatal on nightly, which is exactly the class of thing this workflow is for:
It fires from a cron callback, so with WP-Cron disabled the current checks no longer reach it. Running due cron events through WP-CLI would catch it deterministically and is worth doing as a follow up. It looks like it deserves its own ticket either way.
pull_requestandpushruns now test the latest stable release with 10 plugins instead of nightly with 100. Those runs exist to check that the workflow itself still works, and a real ecosystem fatal against nightly would otherwise sit as a red check on every later change to these two files. Scheduled and manually dispatched runs keep nightly and the full count, which is where the ecosystem signal belongs.How to run it yourself
From a terminal
tools/plugin-compatibility/test-plugins.shruns exactly what a shard in the workflow runs:With Docker running that is all that is needed. A database and a PHP with WP-CLI come up in throwaway containers, WordPress is installed inside them, and both are removed on the way out, so nothing lands on the machine and nothing is left behind. Those two plugins against nightly take about half a minute once the images are pulled.
It ends with the same rows the workflow summary carries:
It exits 0 when nothing fataled and 1 when something did, so it can be dropped into a script of your own.
--wp-versionacceptslatest,nightly, or a version number such as a beta or RC, which is the case this is meant for.--helpcovers the rest:--php-imageand--db-imagefor testing on a different PHP or against MariaDB, and--no-dockerto run the checks against a PHP, WP-CLI and database you already have.--no-dockeris the path the workflow takes, since a runner already has all three.Worth pointing it at:
hello-dolly classic-editorfor a plain pass.woocommerce-paymentson its own, to watchwoocommerceget installed and activated ahead of it because ofRequires Plugins, and to see the "Also active" column fill in.eps-301-redirects, which passes with a note. It serves fine over HTTP and fatals only when WP-CLI loads WordPress with it active, and the Details column says so."Woo Commerce!", which is refused before anything is downloaded, naming both values it could not read as slugs.From Actions
Fill in
plugin-slugson a dispatched run and the directory is not queried at all - those plugins are tested instead.plugin-countis ignored when it is set, and commas, spaces and newlines all separate slugs so a list can be pasted in however it was written down.From the Actions tab: Plugin Compatibility Tests, Run workflow. Or with the CLI:
Dispatching a workflow here needs write access, so the repository guard now lets a dispatched run through wherever it is triggered. That makes a fork enough to try this out:
Note that
--refhas to name a branch that already exists on the fork, and pushing it is what creates it. Dispatching before that comes back asHTTP 422: No ref found. Also note that GitHub only offers a workflow for dispatch once it has seen it run. If a fork does not list it yet, opening a pull request on the fork from that branch against the fork's own trunk runs it once, which is enough to register it.Every run writes a table to the workflow summary: a row per plugin with the version tested, what else was active, and the reason for anything that is not a pass.
How has this been tested
actionlint1.7.12 withshellcheck0.10.0 on PATH reports 0 errors for both new files and 0 across the whole.github/workflowsdirectory. The repo's own actionlint and Zizmor jobs pass. A separatezizmorcode scanning check failed on a newunpinned-imagesalert for the database service image; since the image comes fromdb-typeanddb-versioninputs it cannot be pinned to a digest, so it carries an inline# zizmor: ignore[unpinned-images]with a comment explaining why.zizmor1.24.1 locally reports no findings after that.The matrix builder was run locally against the live API. 10 gave 1 shard of 10, 100 gave 4 shards of 25, 250 gave 10 shards of 25, 300 gave 10 shards of 30 across 2 API pages, and 600 gave 10 shards of 60 across 3 API pages with 600 unique slugs. A count of 1001,
abc, and 0 each exit 1 with a message.The per-plugin loop was extracted from the YAML and run unchanged in a container with WP-CLI, PHP 8.3 and MySQL 8.4, which is as close to the runner as could be managed locally:
instagram-feedandwp-reviews-plugin-for-googlenow PASS,classic-editorstill PASS, job exits 0.template_redirectcaught as HTTP 500 on/, fatal onshutdowncaught at activation, and a fatal inwp_footer(which still returns HTTP 200 because output already started) caught indebug.log. A healthy plugin in the same list still passed and cleanup leftwp-content/pluginsandactive_pluginsclean.latest, matching what apull_requestrun does - all 10 passed.Dependency handling was tested the same way, on PHP 8.3 with MySQL 8.4 and WP-CLI 2.12:
woocommerceand pass, where the script from the previous commit reports them as SKIPPED on the same setup.wp-content/pluginsandactive_pluginswere clean after every run.Worth noting that
instagram-feed6.12.0 now fatals onwp_loadedagainst 7.1 on a fresh database, with the samewp_get_image_editor( WP_Error )trace as above rather than through cron. The script from the previous commit reports it identically, so it is a real plugin bug and not something the dependency handling introduced.The checks now live in a script rather than in the YAML, so this round was run through the script itself, on a MySQL 8.4 container with the PHP 8.3 and WP-CLI the
wordpress:cli-php8.3image carries:hello-dolly classic-editoragainst latest: both pass, exit 0.woocommerce-paymentsagainst latest:woocommerceis installed and activated ahead of it, the plugin passes, and "Also active" names the dependency.eps-301-redirectsagainst nightly: passes with the WP-CLI fatal recorded as a note, exit 0. Front page 200, login screen 200,/wp-admin/302, and no debug log written, so nothing a visitor would see is broken.template_redirectand not under WP-CLI:hello-dollyon latest fails with "The request to / returned HTTP 500", exit 1. A genuine fatal still fails.hello-dollyplus a slug that is not in the directory: the missing slug is recorded as skipped and the run still exits 0."Woo Commerce!": refused before anything is downloaded, naming both values, exit 1.GITHUB_ACTIONSandGITHUB_STEP_SUMMARYset: a log group per plugin, and the markdown table written to the summary file rather than only to stdout.shellcheck0.11.0 reports no findings on the script.The
plugin-slugsinput was tested by extracting the matrix step and running it against the live API: a mixed comma and space separated list, a list carrying duplicates and stray whitespace, forty names splitting into two shards, and the popularity path unchanged at a count of 10. A list of 1001 names, a list of nothing but separators, andWoo Commerce!each exit 1 with a message naming the problem.Four runs were then dispatched on a fork, which is the same path a reviewer without write access would take:
classic-editor, woocommerce-payments, hello-dollyagainst nightly: all three pass in 1m19s, withwoocommerceinstalled and activated first forwoocommerce-payments. https://github.com/adamsilverstein/wordpress-develop/actions/runs/33450424680instagram-feedplus a slug that does not exist: the plugin passes, the missing slug is recorded as skipped, and the job stays green. https://github.com/adamsilverstein/wordpress-develop/actions/runs/33451373337eps-301-redirectsagainst nightly: red at the time. That run predates the change above, and it failed on the WP-CLI check that now only adds a note. https://github.com/adamsilverstein/wordpress-develop/actions/runs/33451622309Woo Commerce!: the matrix job stops after 11 seconds, listing both invalid values. https://github.com/adamsilverstein/wordpress-develop/actions/runs/33451382261actionlint1.7.12 reports 0 errors for both workflow files after these changes.Not tested: multisite, PHP versions other than 8.3, MariaDB, and the Slack notification and failed-workflow jobs, which are copied from
install-testing.ymlunchanged.instagram-feedpassed on nightly in the run above, so the cron-time fatal noted earlier is not currently reachable from the checks this workflow makes.Types of changes
.github/workflows/plugin-compatibility.yml, the caller, which pages the WordPress.org API and fans out to a matrix sized to the plugin count..github/workflows/reusable-plugin-compatibility.yml, which installs WordPress and tests one shard of plugins in isolation.Requires Pluginsdependencies before testing it, and check the site with only those active first so a dependency's breakage is not reported against the plugin under test.plugin-slugsdispatch input naming the plugins to test in place of the popularity query.SLACK_GHA_TIMEOUT_WEBHOOK, whichslack-notifications.ymlno longer accepts.tools/plugin-compatibility/test-plugins.sh, so the code that decides whether a plugin passed is the same one whether it runs in a shard or in a terminal.Open questions
.version-support-*.jsonfor the PHP version instead of pinning8.3, or is one current PHP version the right scope for a smoke test?AI Use
Code and description written with 🤖 Claude Code, working from acceptance criteria in the linked issue. I will review and test.