[🐛 Bug]: [py] add_extra_header() hangs driver.get() and drops the Accept header in Chrome #18116

Description

driver.network.add_extra_header() pauses every request with a match-everything intercept and continues it with a header list rebuilt from the beforeRequestSent event. In Chrome this causes two problems.

1. Classic navigation hangs. With an extra header set, driver.get() blocks until the page load timeout: ChromeDriver doesn't process BiDi commands such as network.continueRequest while a classic command is running, so the paused request is never continued. That ChromeDriver behaviour is already reported for the Node binding in [🐛 Bug]: network.continueRequest() hangs navigation in Chrome #17373 and upstream in https://issues.chromium.org/issues/425906330. What is specific here is that the Python helper puts everyone who navigates with driver.get() into that situation, without asking for any interception. It is a race rather than a hard block: in 1 of 10 runs here driver.get() returned normally, the other 9 hung until the page load timeout (headed and --headless=new behave the same).

2. Headers get lost. Chrome's beforeRequestSent event lists only some request headers (in my run: Upgrade-Insecure-Requests, User-Agent and the sec-ch-ua ones). Continuing with that list drops Accept from navigation requests, so adding one header removes another.

Firefox lists all headers in the event and handles the classic/BiDi mix, so both work there.

The docstring and #17632 say BiDi has no dedicated command for extra headers, but network.setExtraHeaders is in the spec, driver.network.set_extra_headers() already wraps it, and the .NET and Ruby bindings expose it too. I checked in Chrome 154 and Firefox 154 that it covers what add_extra_header promises: it replaces an existing header of the same name, an empty list removes the headers, and it composes with request handlers (they see the extra header in beforeRequestSent, and it is still sent when a handler rewrites headers with set_headers). It also does not pause requests at all.

I would be happy to send a PR that switches add_extra_header, remove_extra_header and clear_extra_headers to network.setExtraHeaders, with tests that navigate with driver.get() and check that Accept is kept.

Reproducible Code

import http.server
import json
import threading

from selenium import webdriver


class Echo(http.server.BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def do_GET(self):
        body = json.dumps({k.lower(): v for k, v in self.headers.items()}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body)


server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Echo)
threading.Thread(target=server.serve_forever, daemon=True).start()
url = f"http://127.0.0.1:{server.server_port}/"


def received(driver):
    return json.loads(driver.find_element("tag name", "body").text)


options = webdriver.ChromeOptions()
options.enable_bidi = True
driver = webdriver.Chrome(options=options)
driver.set_page_load_timeout(10)
try:
    driver.network.add_extra_header("x-extra", "1")

    # 1. Classic navigation hangs until the page load timeout.
    try:
        driver.get(url)
        print("driver.get: ok")
    except Exception as e:
        print("driver.get:", type(e).__name__)

    # 2. BiDi navigation works, but the Accept header is gone.
    driver.browsing_context.navigate(context=driver.current_window_handle, url=url, wait="complete")
    headers = received(driver)
    print("BiDi navigate: x-extra =", headers.get("x-extra"), "| accept =", headers.get("accept"))

    # 3. The native network.setExtraHeaders command has neither problem.
    driver.network.clear_extra_headers()
    driver.network.set_extra_headers(headers=[{"name": "x-extra", "value": {"type": "string", "value": "1"}}])
    driver.get(url)
    headers = received(driver)
    print("native + driver.get: x-extra =", headers.get("x-extra"), "| accept =", headers.get("accept"))
finally:
    driver.quit()
    server.shutdown()

Output with Chrome 154.0.8037.93 / ChromeDriver 154.0.8037.92:

driver.get: TimeoutException
BiDi navigate: x-extra = 1 | accept = None
native + driver.get: x-extra = 1 | accept = text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7

Debugging Logs

After the page load timeout the background handler still tries to continue the request and fails:

selenium.common.exceptions.WebDriverException: Message: unknown error: Invalid InterceptionId.

What version of Selenium are you currently using?

4.50.0

The following statements are true

• This applies to the most recent version of Selenium (we can't fix old versions)
• This hasn't already been reported (I searched and didn't find it)
• All information necessary to reproduce the issue has been provided above

Did this work for you before?

Not sure, this is my first time trying it

Operating System

macOS

Selenium Language Binding

Python

Which browsers are you experiencing the issue with?

Chrome

Are you using Selenium Grid?

No