Undercrawler concurrency and Splash slots

[nehakansal]

I would like to make sure I understand the Undercrawler CONCURRENT_REQUESTS setting and the number of Splash slots interaction. If I have 3 splash instances and each instance has a slot setting of 5, does that mean that the effective concurrency would be 15 even if the concurrent_requests in Undercrawler is set to 32? Thanks.

[lopuhin]

@nehakansal that's right. I think it's better to have CONCURRENT_REQUESTS approximately equal to the total available slots, but that's just from reading http://splash.readthedocs.io/en/stable/faq.html?highlight=slot#splash-instance-is-overloaded and not from experience. Also I'm not sure if using https://github.com/TeamHG-Memex/aquarium changes this.

[nehakansal]

Okay, thanks. In case you find out any more info about the effective concurrency with Aquarium, please post here, thank you.