Website Detecting Bot Only in Docker

[jdholtz]

Over the past month, many people (including myself) have been getting a 429 status code error when trying to do specific actions on a website (going to pages, logging in, etc.) when using Undetected Chrome over SeleniumBase. I've narrowed down the issue to the website generating invalid request headers, which I only can assume that is a result of detecting the browser is a bot as it never happens when I navigate to the website in my browser.

The strange thing about the issue is that it only occurs when I run within Docker. Outside of Docker, everything works perfectly fine. While I understand this may be totally out of scope of the work you are doing on SeleniumBase, would there be any specific reasons a website can detect a containerized browser being controlled by a bot, but not when it is run outside of the container?

To note, I am using Chromium mostly for Undetected mode (which has worked perfectly for me), I also reproduce the exact same detection issue on the latest Google Chrome version and SeleniumBase version (Python 3.12). Let me know if I can provide any more information. This issue may not relate a whole lot to SB, but I'm just at a loss of what to do to fix it, so any insights would be helpful. Thanks for this amazing framework!

[mdmintz]

@jdholtz you mentioned:

"when using Undetected Chrome over SeleniumBase"

Was your issue only happening with the original Undetected Chromedriver and not SeleniumBase UC Mode? Last time I checked, UC Mode was working in a Docker container. Is that no longer the case?

Make sure you're using Chrome and not Chromium. They are different enough that the modified Chromedriver doesn't fully mask the traces of bot activity with Chromium, but it does with Chrome. (Lots of Docker containers come with Chromium instead of Chrome.)

You're also going to want to use Xvfb (xvfb=True) instead of using headless mode. (Set headless=False to make sure Linux runs don't default to using headless mode.) It worked here: #2390 (comment)

[jdholtz]

Was your issue only happening with the original Undetected Chromedriver and not SeleniumBase UC Mode? Last time I checked, UC Mode was working in a Docker container. Is that no longer the case?

Sorry, I mean UC Mode within SeleniumBase. I have the same exact issue with Undetected Chromedriver, but my project uses SeleniumBase now.

You're also going to want to use Xvfb (xvfb=True) instead of using headless mode. (Set headless=False to make sure Linux runs don't default to using headless mode.)

I tried to run the driver with headed=True and got a VNC server running to see what is going on inside the Docker container, but unfortunately nothing seems visibly wrong (and inspecting the requests/responses don't hint to any issues either).

Strangely, I am not able to pass the nowsecure.nl test in either Docker or my regular machine (script is below). Since everything within the browser appears to work (although invalid request headers are still generated), I don't really know if this is particularly with SeleniumBase or not, so I'll keep trying to see what the issue is. I really appreciate the advice!

Here is the script that is currently failing for me on the latest SB and Python 3.11. It probably isn't failing for other people, so again, I don't know what issue I am having.

from seleniumbase import SB

with SB(uc=True, headed=True, binary_location="/usr/bin/google-chrome-stable") as sb:
    sb.driver.get("https://nowsecure.nl/#relax")
    sb.sleep(1)
    if not sb.is_text_visible("OH YEAH, you passed!", "h1"):
        sb.get_new_driver(undetectable=True)
        sb.driver.get("https://nowsecure.nl/#relax")
        sb.sleep(1)
    if not sb.is_text_visible("OH YEAH, you passed!", "h1"):
        if sb.is_element_visible('iframe[src*="challenge"]'):
            with sb.frame_switch('iframe[src*="challenge"]'):
                sb.click("span.mark")
                sb.sleep(2)
    sb.activate_demo_mode()
    sb.assert_text("OH YEAH, you passed!", "h1", timeout=3)

[mdmintz]

There are a few things happening. Chrome 121 has a memory leak that makes Selenium easily detectable in the browser. Upgrading to Chrome 122 (currently in Beta) is one way to fix that. Another thing is that the nowsecure.nl website is now limiting how many times a user can visit their site before getting blocked no matter what. To verify that UC Mode still works, try switching to a different site protected by Cloudflare, such as https://www.g2.com/categories/video

[jdholtz]

I downloaded Chrome 122 and it is passing the nowsecure test now. Strangely, Chrome 121 is working too. However, I'm still getting the original issue I am having, but I'll keep working on seeing what the exact issue is/how to fix it. I really appreciate your help working through this!

[VovkoO]

Hi @jdholtz, did you succeed bypass detection running in docker? I have same trouble

[jdholtz]

Unfortunately I’ve still been having trouble occasionally with it. Some contributors have changed the methodology of how clicks are done and what pages are loaded, which has helped the success rate tremendously though. More of a fix specifically for my script, not a general solution though.