Skip to content

Conversation

@Garbee
Copy link
Member

@Garbee Garbee commented Dec 23, 2025

This patch removes the dependency on chromedriver entirely from the package. Now we check for the environment variable and if it isn't present we fail with a meaningful error message. Since this is only used in tests, it only impacts developers working in the project and not consumers of it.

The contributing documentation has been updated to include information on how to get the needed binaries installed and configured.

Upon opening this PR, the Prettier check was failing. The reason for this was the dependencies were installed before node was setup on the version we run on locally. So Prettier was then expecting a newer node version then what it should be operating on. To fix this, I swapped the order of node-setup and Install Dependencies in the formatter workflow.

No QA Required

@Garbee Garbee self-assigned this Dec 23, 2025
Copy link
Contributor

@straker straker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the motivation of removing chromedriver from the dev workflow? I'm not sure I like the idea of completely removing chromedriver from the dev workflow. Having latest chromedriver in dev was never a problem and makes development work out of the box. This step requires devs to manually and constantly update their chrome and chromdriver versions. I'd rather CI used the binary and dev used installed chrome + chromedriver.

@Garbee
Copy link
Member Author

Garbee commented Jan 5, 2026

The main driving reason, it adds another point of failure in CI that isn't necessary. If the chromedriver binary fails to download from this for whatever reason, it fails the run. When we already install the chromedriver separately. So it's also just wasting resources in CI by double-downloading every time a box comes online.

Secondarily, chromedriver isn't a direct dependency to run the software. It's only needed for a portion of the testing. Which, the contribution guide is updated to help developers understand how to do that.

Removing this dependency helps the system become more idempotent. Having chromedriver@latest isn't actually a useful dependency since it is changing all the time. We aren't really tracking anything with this.

This is a convenience for developers yes, but it has a cost that no one needs to be paying. Especially in CI where things are downloaded a lot. And caching the downloaded binary from this as well isn't a huge help, since it does change regularly enough and is just wasting cache resources on something that isn't even used in CI. Caching it would alleviate any failures from external network/host issues. But, why apply that effort when we don't need the dependency to begin with?

@Garbee Garbee force-pushed the garbee/remove/chromedriver branch from 6a2f01d to ddffa90 Compare January 5, 2026 17:39
@Garbee Garbee marked this pull request as ready for review January 5, 2026 17:40
@Garbee Garbee requested a review from a team as a code owner January 5, 2026 17:40
Copilot AI review requested due to automatic review settings January 5, 2026 17:40
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes the chromedriver npm package dependency and replaces it with a requirement for developers to manually install and configure ChromeDriver via the CHROMEDRIVER_BIN environment variable. The change only affects developers working on the project, not consumers of the package.

  • Replaced automatic chromedriver installation with environment variable requirement
  • Added validation to throw clear error messages when CHROMEDRIVER_BIN is not set
  • Updated contributing documentation with instructions for installing ChromeDriver using @puppeteer/browsers

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
test/integration/full/test-webdriver.js Added validation to check CHROMEDRIVER_BIN environment variable and throw error if not set
test/get-webdriver.js Added validation to check CHROMEDRIVER_BIN environment variable at module load time
package.json Removed chromedriver from devDependencies
package-lock.json Removed chromedriver and all its transitive dependencies from the lock file
CONTRIBUTING.md Added new "Chrome Driver Configuration" section with installation and setup instructions
.github/workflows/format.yml Reordered steps to setup Node.js before installing dependencies to fix Prettier check

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.

To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command has an unnecessary semicolon at the end. Shell commands typically don't need semicolons unless chaining multiple commands. Remove the semicolon for consistency with line 179.

Suggested change
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`

Copilot uses AI. Check for mistakes.
Comment on lines +172 to +179
### Chrome Driver Configuration

Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.

To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;

To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent capitalization. "Chrome Driver" should be "ChromeDriver" to match the standard naming convention used elsewhere in the documentation and in the environment variable name CHROMEDRIVER_BIN.

Suggested change
### Chrome Driver Configuration
Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`
### ChromeDriver Configuration
Some tests require the ChromeDriver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the ChromeDriver run: `npx @puppeteer/browsers install chromedriver`

Copilot uses AI. Check for mistakes.
Comment on lines +172 to +179
### Chrome Driver Configuration

Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.

To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;

To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent capitalization. "Chrome Driver" should be "ChromeDriver" to match the standard naming convention used elsewhere in the documentation and in the environment variable name CHROMEDRIVER_BIN.

Suggested change
### Chrome Driver Configuration
Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`
### ChromeDriver Configuration
Some tests require the ChromeDriver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the ChromeDriver run: `npx @puppeteer/browsers install chromedriver`

Copilot uses AI. Check for mistakes.
Comment on lines +172 to +179
### Chrome Driver Configuration

Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.

To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;

To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent capitalization. "Chrome Driver" should be "ChromeDriver" to match the standard naming convention used elsewhere in the documentation and in the environment variable name CHROMEDRIVER_BIN.

Suggested change
### Chrome Driver Configuration
Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`
### ChromeDriver Configuration
Some tests require the ChromeDriver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
To get the Chrome For Testing binary run: `npx @puppeteer/browsers install chrome`;
To get the ChromeDriver run: `npx @puppeteer/browsers install chromedriver`

Copilot uses AI. Check for mistakes.
To get the Chrome Driver run: `npx @puppeteer/browsers install chromedriver`

> [!NOTE]
> Do check the commands arguments, you can set a path to install to so they
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grammatical issue. "command's arguments" should be "command's arguments" or better yet "the command's arguments" for clarity.

Suggested change
> Do check the commands arguments, you can set a path to install to so they
> Do check the command's arguments, you can set a path to install to so they

Copilot uses AI. Check for mistakes.
Comment on lines +126 to +130
if (!chromedriverPath || chromedriverPath.trim() === '') {
throw new Error(
'CHROMEDRIVER_BIN environment variable is not set. Please set it to the path of the chromedriver executable.'
);
}
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation logic for chromedriverPath is duplicated between this file and test/get-webdriver.js. Consider extracting this validation into a shared utility function to maintain consistency and reduce duplication.

Copilot uses AI. Check for mistakes.
### Chrome Driver Configuration

Some tests require the Chrome Driver binary be installed with a matching Chrome version. This can be achieved with the
`@puppeteer/browsers` command from the npm registry.
Copy link

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing period at the end of the sentence. It should be "...from the npm registry." to maintain consistency with the rest of the documentation.

Copilot uses AI. Check for mistakes.
@straker
Copy link
Contributor

straker commented Jan 5, 2026

As a one-time contributor to axe-core, the pr is mostly fine (see point 4 below). However from a maintainer standpoint I'm not sure I like this approach.

All bug fixes and testing happen by running npm start and using the test/playground.html file in a live browser to test and debug use cases and HTML snippets. All test/integration/full tests can also be opened in the browser and used to debug problems there. Furthermore unit tests can be debugged using npm test:debug which then allows opening the browser to the debug page. All of these use the current version of Chrome installed on my machine. If the tests then use a different version of Chrome and chromedriver that is not the one I'm using for debugging, this is going to cause problems as the tests will show different results than my debugging. I'm not going to remember that the tests are using a different Chrome version and will waste time trying to debug the problem rather than remember to update my out-of-date Chrome version from puppeteer.

With that in mind, here are the things that would have to be solved in order to accept the changes proposed:

  1. The testing version of Chrome will need to match the debugging version of Chrome and remain version matched as time goes on. At the very least some sort of message should inform me if the testing version of chrome is out-of-date so I'm made aware of the problem
  2. Chrome + driver should be installed by default (an npm post:install should suffice). There's no reason to not set up everything automatically for the user and make them do it manually
  3. Updating the version of Chrome + driver should be a 1 line command that also updates the export file, especially since the directory that Chrome + driver are installed to will change with every version and it'll be a hassle to manually update that every time in my ENV vars.
  4. Importantly, we should not be installing Chrome + driver into the repo directory, unless you also npm ignore the directories that are installed (chromedriver and chrome). Currently the instructions says to do this and that will cause problems. If you want the user to install them globally at the root directory of the user, that is a bigger issue and not something I think we want to make users do that are just wanting to run tests for a simple pr

@Garbee
Copy link
Member Author

Garbee commented Jan 5, 2026

Chrome + driver should be installed by default (an npm post:install should suffice). There's no reason to not set up everything automatically for the user and make them do it manually

Installing binaries without explicit user consent is an anti-pattern in package management. These types of automated installations can introduce significant security risks. In fact, similar practices contributed to the npm supply chain attacks we saw last year, which ultimately led to npm deprecating legacy authentication tokens in favor of more secure OIDC-based authentication.

Instead, I'd propose we focus on improving the developer experience through:

  • Clear documentation on setup requirements
  • Helpful error messages that guide users when dependencies are missing
  • Optional setup scripts that users can run explicitly when they're ready
  • We can provide those optional scripts in a way that prompts users when they try to do things to help them in-context as well.

This approach respects the principle of least surprise and gives users control over what gets installed in their environment, while still making the setup process straightforward.

Happy to discuss alternative approaches that balance convenience with security best practices. However, the approach to make this all smoother should be detached from removing the problematic dependency that we have. As to require a full and complete implementation to maintain the ease-of-use in one go means the change set easily becomes a thousand line delta pull request.

we should not be installing Chrome + driver into the repo directory

Valid point on the docs being unclear on where to store things. I'll get that clarified and updated for sure. It might be we introduce a tmp folder that is ignored and give instructions to install in there. The original thinking was people install wherever they do since these binaries are useful across multiple things. But, very good point we should make isolated instructions to the project and then give guidance that they can be installed elsewhere if someone does get that de-duplication benefit.

@Garbee Garbee marked this pull request as draft January 9, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants