chrome driver started to die off??

[yarikoptic]

not sure what has changed in the past two days, we started to get emails with crashes like

+ xvfb-run python tools/make_webshots.py -i dandi                                                                                                                                                                                                                                                             
2022-06-17T21:30:17-0400 [WARNING ] MainProcess[1724596]: dandi: A newer version (0.40.1) of dandi/dandi-cli is available. You are using 0.38.0                                                                                                                                                               
2022-06-17T21:30:23-0400 [INFO    ] Process-1[1725490]: make_webshots: Logging in ...                                                                                                                                                                                                                         
tools/make_webshots.py:131: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead                                                                                                                                                                                  
  username_field = self.driver.find_element_by_id("login_field")                                                                                                                                                                                                                                              
tools/make_webshots.py:132: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead                                                                                                                                                                                  
  password_field = self.driver.find_element_by_id("password")                                                                                                                                                                                                                                                 
tools/make_webshots.py:136: DeprecationWarning: find_elements_by_tag_name is deprecated. Please use find_elements(by=By.TAG_NAME, value=name) instead                                                                                                                                                         
  self.driver.find_elements_by_tag_name("form")[0].submit()                                                                                                                                                                                                                                                   
tools/make_webshots.py:143: DeprecationWarning: find_element_by_xpath is deprecated. Please use find_element(by=By.XPATH, value=xpath) instead                                                                                                                                                                
  self.driver.find_element_by_xpath(                                                                                                                                                                                                                                                                          
tools/make_webshots.py:154: DeprecationWarning: find_elements_by_class_name is deprecated. Please use find_elements(by=By.CLASS_NAME, value=name) instead                                                                                                                                                     
  or self.driver.find_elements_by_class_name("v-avatar")                                                                                                                                                                                                                                                      
2022-06-17T21:30:29-0400 [INFO    ] Process-1[1725490]: make_webshots: 000003 landing                                                                                                                                                                                                                         
2022-06-17T21:30:35-0400 [INFO    ] Process-1[1725490]: make_webshots: 000003 edit-metadata                                                                                                                                                                                                                   
Process Process-1:                                                                                                                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                                                                                            
  File "/home/dandi/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap                                                                                                                                                                                                             
    self.run()                                                                                                                                                                                                                                                                                                
  File "/home/dandi/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run                                                                                                                                                                                                                    
    self._target(*self._args, **self._kwargs)                                                                                                                                                                                                                                                                 
  File "tools/make_webshots.py", line 410, in snapshot_pipe                                                                                                                                                                                                                                                   
    t = ws.process_dandiset_page(ds, urlsuf, page, wait_cls, pbar_cls, act)                                                                                                                                                                                                                                   
  File "tools/make_webshots.py", line 241, in process_dandiset_page                                                                                                                                                                                                                                           
    self.driver.get(f"{self.gui_url}/dandiset/{ds}")                                                                                                                                                                                                                                                          
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 437, in get                                                                                                                                                                       
    self.execute(Command.GET, {'url': url})                                                                                                                                                                                                                                                                   
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 425, in execute                                                                                                                                                                   
    self.error_handler.check_response(response)                                                                                                                                                                                                                                                               
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response                                                                                                                                                         
    raise exception_class(message, screen, stacktrace)                                                                                                                                                                                                                                                        
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_SSL_PROTOCOL_ERROR                                                                                                                                                                                                            
  (Session info: headless chrome=100.0.4896.60)                                                                                                                                                                                                                                                               
Stacktrace:                                                                                                                                                                                                                                                                                                   
#0 0x55d01a0c7af3 <unknown>                                                                                                                                                                                                                                                                                   
#1 0x55d019dd88b6 <unknown>                                                                                                                                                                                                                                                                                   
#2 0x55d019dd676a <unknown>                                                                                                                                                                                                                                                                                   
#3 0x55d019dc2ae7 <unknown>                                                                                                                                                                                                                                                                                   
#4 0x55d019dc39be <unknown>                                                                                                                                                                                                                                                                                   
#5 0x55d019dc2de4 <unknown>                                                                                                                                                                                                                                                                                   
#6 0x55d019dc213b <unknown>                                                                                                                                                                                                                                                                                   
#7 0x55d019dc1042 <unknown>                                                                                                                                                                                                                                                                                   
#8 0x55d019dc14ac <unknown>                                                                                                                                                                                                                                                                                   
#9 0x55d019ddaba4 <unknown>                                                                                                                                                                                                                                                                                   
#10 0x55d019e47374 <unknown>                                                                                                                                                                                                                                                                                  
#11 0x55d019e34762 <unknown>                                                                                                                                                                                                                                                                                  
#12 0x55d019e4702b <unknown>                                                                                                                                                                                                                                                                                  
#13 0x55d019e34623 <unknown>                                                                                                                                                                                                                                                                                  
#14 0x55d019e0427b <unknown>                                                                                                                                                                                                                                                                                  
#15 0x55d019e05aae <unknown>                                                                                                                                                                                                                                                                                  
#16 0x55d01a0f391d <unknown>                                                                                                                                                                                                                                                                                  
#17 0x55d01a105ea3 <unknown>                                                                                                                                                                                                                                                                                  
#18 0x55d01a105b99 <unknown>                                                                                                                                                                                                                                                                                  
#19 0x55d01a106462 <unknown>                                                                                                                                                                                                                                                                                  
#20 0x55d01a0f4e3b <unknown>                                                                                                                                                                                                                                                                                  
#21 0x55d01a1066e1 <unknown>                                                                                                                                                                                                                                                                                  
#22 0x55d01a0e95c9 <unknown>                                                                                                                                                                                                                                                                                  
#23 0x55d01a11d1d5 <unknown>                                                                                                                                                                                                                                                                                  
#24 0x55d01a11d373 <unknown>                                                                                                                                                                                                                                                                                  
#25 0x55d01a136f41 <unknown>                                                                                                                                                                                                                                                                                  
#26 0x7fab75120ea7 <unknown>                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                              
2022-06-17T21:31:07-0400 [WARNING ] MainProcess[1724596]: make_webshots: Subprocess exited while processing ('000003', 'edit-metadata'); restarting                                                                                                                                                           
2022-06-17T21:31:07-0400 [INFO    ] Process-2[1726124]: make_webshots: Logging in ...                                                                                                                                                                                                                         
tools/make_webshots.py:131: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead                              

and then finally die entirely with

2022-06-17T21:43:09-0400 [WARNING ] MainProcess[1724596]: make_webshots: Subprocess exited while processing ('000008', 'view-data'); restarting                                                                                                                                                               
2022-06-17T21:43:10-0400 [INFO    ] Process-18[1730267]: make_webshots: Logging in ...                                                                                                                                                                                                                        
tools/make_webshots.py:131: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead                                                                                                                                                                                  
  username_field = self.driver.find_element_by_id("login_field")                                                                                                                                                                                                                                              
tools/make_webshots.py:132: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead                                                                                                                                                                                  
  password_field = self.driver.find_element_by_id("password")                                                                                                                                                                                                                                                 
tools/make_webshots.py:136: DeprecationWarning: find_elements_by_tag_name is deprecated. Please use find_elements(by=By.TAG_NAME, value=name) instead                                                                                                                                                         
  self.driver.find_elements_by_tag_name("form")[0].submit()                                                                                                                                                                                                                                                   
tools/make_webshots.py:143: DeprecationWarning: find_element_by_xpath is deprecated. Please use find_element(by=By.XPATH, value=xpath) instead                                                                                                                                                                
  self.driver.find_element_by_xpath(                                                                                                                                                                                                                                                                          
2022-06-17T21:43:13-0400 [INFO    ] Process-18[1730267]: make_webshots: Cleaning up 7 child processes                                                                                                                                                                                                         
Process Process-18:                                                                                                                                                                                                                                                                                           
Traceback (most recent call last):                                                                                                                                                                                                                                                                            
  File "/home/dandi/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap                                                                                                                                                                                                             
    self.run()                                                                                                                                                                                                                                                                                                
  File "/home/dandi/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run                                                                                                                                                                                                                    
    self._target(*self._args, **self._kwargs)                                                                                                                                                                                                                                                                 
  File "tools/make_webshots.py", line 401, in snapshot_pipe                                                                                                                                                                                                                                                   
    with Webshotter(gui_url, headless, login) as ws:                                                                                                                                                                                                                                                          
  File "tools/make_webshots.py", line 89, in __init__                                                                                                                                                                                                                                                         
    self.set_driver()                                                                                                                                                                                                                                                                                         
  File "tools/make_webshots.py", line 111, in set_driver                                                                                                                                                                                                                                                      
    self.login(os.environ["DANDI_USERNAME"], os.environ["DANDI_PASSWORD"])                                                                                                                                                                                                                                    
  File "tools/make_webshots.py", line 149, in login                                                                                                                                                                                                                                                           
    raise RateLimitError("GitHub secondary rate limit exceeded")                                                                                                                                                                                                                                              
RateLimitError: GitHub secondary rate limit exceeded                                                                                                                                                                                                                                                          
Traceback (most recent call last):                                                                                                                                                                                                                                                                            
  File "tools/make_webshots.py", line 552, in <module>                                                                                                                                                                                                                                                        
    main()                                                                                                                                                                                                                                                                                                    
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__                                                                                                                                                                                          
    return self.main(*args, **kwargs)                                                                                                                                                                                                                                                                         
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main                                                                                                                                                                                              
    rv = self.invoke(ctx)                                                                                                                                                                                                                                                                                     
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke                                                                                                                                                                                            
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                                            
  File "/mnt/backup/dandi/dandi-api-webshots/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke                                                                                                                                                                                             
    return __callback(*args, **kwargs)                                                                                                                                                                                                                                                                        
  File "tools/make_webshots.py", line 477, in main                                                                                                                                                                                                                                                            
    stats.append(ff(ds, page))                                                                                                                                                                                                                                                                                
  File "tools/make_webshots.py", line 331, in __call__                                                                                                                                                                                                                                                        
    raise RuntimeError(                                                                                                                                                                                                                                                                                       
RuntimeError: Subprocess encountered unrecoverable error: GitHub secondary rate limit exceeded 

that is why we don't have webshots updated here for two days now. Happened on June 16 and 17.

Any ideas @jwodder ?

[jwodder]

@yarikoptic Well, the webshots have started updating again, so it seems to have been a temporary server issue.