You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, i use scrapy (2.8.0), scrapoxy (with docker image fabienvauchelles/scrapoxy:latest) and splash (3.5) to scrape data but i got a 500 Internal Server Error when splash is running. To illustrate the error I use the website https://quotes.toscrape.com/login
Scrapy is running on macos on host 192.168.0.12.
Scrapoxy is running with docker image on debian 11.9 on host 192.168.0.103.
Splash is running with docker image on debian 11.9 on host 192.168.0.102.
Scrapy settings.py configuration :
# Scrapoxy setupCONCURRENT_REQUESTS_PER_DOMAIN=1RETRY_TIMES=0SCRAPOXY_MASTER="http://192.168.0.103:8888"SCRAPOXY_API="http://192.168.0.103:8890/api"SCRAPOXY_USERNAME="username"SCRAPOXY_PASSWORD="password"SCRAPOXY_BLACKLIST_HTTP_STATUS_CODES= [400, 429, 503]
SCRAPOXY_SLEEP_MIN=60SCRAPOXY_SLEEP_MAX=180# End Scrapoxy setup# Splash setupSPLASH_URL='http://192.168.0.102:8050'# End Splash setupUSER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"ROBOTSTXT_OBEY=FalseSPIDER_MIDDLEWARES= {
"scrapoxy.StickySpiderMiddleware": 101,
}
DOWNLOADER_MIDDLEWARES= {
# scrapoxy middleware'scrapoxy.ProxyDownloaderMiddleware': 100,
'scrapoxy.BlacklistDownloaderMiddleware': 101,
###################'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 300,
#################### splash middleware'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
###################
}
REQUEST_FINGERPRINTER_IMPLEMENTATION="2.7"TWISTED_REACTOR="twisted.internet.asyncioreactor.AsyncioSelectorReactor"FEED_EXPORT_ENCODING="utf-8"
Current Behavior
Hi, i use scrapy (2.8.0), scrapoxy (with docker image fabienvauchelles/scrapoxy:latest) and splash (3.5) to scrape data but i got a 500 Internal Server Error when splash is running. To illustrate the error I use the website https://quotes.toscrape.com/login
Scrapy is running on macos on host 192.168.0.12.
Scrapoxy is running with docker image on debian 11.9 on host 192.168.0.103.
Splash is running with docker image on debian 11.9 on host 192.168.0.102.
Scrapy settings.py configuration :
Scrapy spider :
Expected Behavior
Everything works with scrapy and scrapoxy.
Everything works with scrapy and splash.
But the aim is to be able to use scrapy, scrapoxy and splash in the same scrapy project.
Steps to Reproduce
I use OVH Public Cloud with 6 proxies.
Failure Logs
Scrapoxy Version
docker version
Custom Version
Deployment
Operating System
Storage
Additional Information
No response
The text was updated successfully, but these errors were encountered: