JavascriptException Stale element not found after (multiple) redirects

[victorbmlabs]

I'm building a SeleniumBase scraper that takes a set of tasks and performs them on a website. After performing some actions on the site the script calls driver.execute_script("return window.variable;"). This works fine on all domains I've scraped even with any redirects after clicks.

Today I scraped a new website which includes pressing two buttons that both redirect you to a page (2 redirects total). After the redirects the script runs execute_script and throws a selenium.common.exceptions.JavascriptException.

Traceback (most recent call last):
  File "/home/victor/.pyenv/versions/3.12.3/envs/tagprotection-v3/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/victor/Webvitals/tagprotection-v3/tagprotection/worker.py", line 28, in __collect_datalayer
    datalayer = sb.execute_script("return window.variable;")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/victor/.pyenv/versions/3.12.3/envs/tagprotection-v3/lib/python3.12/site-packages/seleniumbase/fixtures/base_case.py", line 3372, in execute_script
    return self.driver.execute_script(script, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/victor/.pyenv/versions/3.12.3/envs/tagprotection-v3/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 414, in execute_script
    return self.execute(command, {"script": script, "args": converted_args})["value"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/victor/.pyenv/versions/3.12.3/envs/tagprotection-v3/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 354, in execute
    self.error_handler.check_response(response)
  File "/home/victor/.pyenv/versions/3.12.3/envs/tagprotection-v3/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.JavascriptException: Message: javascript error: {"status":10,"value":"stale element not found in the current frame"}

I'm not quite sure why this javascript variable is a stale element. Data is being pushed to it on every page (similar to Google's dataLayer)

[mdmintz]

The element is "stale" because it no longer exists in the state that you found it in. (A page reload or removing the element would cause that.) Each time the page refreshes, you must relocate the element before using the variable that you saved it in.

[mdmintz]

If the page gets redirected, then the element goes stale, and you have to find the element again (and save it back into the variable).

[victorbmlabs]

I updated my previous reply, I don't quite get how I have to find anything again when I'm just directly executing a return script.

[mdmintz]

If you save an element into a variable, then you are actually only saving a pointer to the element, rather than saving a copy of the original element itself. If the page reloads, then the pointer you saved no longer points to anything. (This is not a SeleniumBase-specific thing, and so I cannot change how the element pointer system works.)

[victorbmlabs]

Okay I get that, however I don't make any calls to execute_script until both clicks have happened. Then I assign value = driver.execute_script("return window.foo");. So I don't get how it's stale at that point, perhaps I have to wait for the dom to fully load?

[mdmintz]

Possible. Try waiting for the DOM to fully load and then save the variables.