NetworkX errors not retried

[lopuhin]

I think this errors are not retried

2016-04-12 21:30:49 [scrapy_splash.middleware] WARNING: Bad request to Splash: {'error': 400, 'description': 'Error happened while executing Lua script', 'info': {'message': 'Lua
 error: [string "function get_arg(arg, default)..."]:52: network5', 'error': 'network5', 'type': 'LUA_ERROR', 'source': '[string "function get_arg(arg, default)..."]', 'line_numb
er': 52}, 'type': 'ScriptError'}
2016-04-12 21:29:31 [scrapy] DEBUG: Crawled (400) <POST http://XXX:8050/execute> (referer: None)
2016-04-12 21:29:31 [scrapy] DEBUG: Ignoring response <400 XXX/>: HTTP status code is not handled or not allowed

[kmike]

It makes sense to add this feature to scrapy-splash (handle network.. error codes in addition to http.. codes when http_status_from_error_code is True). But I'm not sure what should we set response status code to in these cases - there was no response in first place.

[kmike]

network... status codes: http://doc.qt.io/qt-5/qnetworkreply.html#NetworkError-enum

[lopuhin]

I think at least for network5 this could mean that our splash script timed out?

[kmike]

It means resource_timeout was applied for the first request, and it timed out; this is a bit different from regular Splash timeouts. But yeah, it makes sense to handle network5 errors as 504 HTTP errors.

[kmike]

It also could make sense to apply a larger timeout for the first request (see Example 6 here: http://splash.readthedocs.org/en/stable/scripting-ref.html#splash-on-request) - what do you think?