fragment is removed from pagination links

[kmike]

Here:

undercrawler/undercrawler/spiders/base_spider.py

Line 203 in 87551a6

def _pagination_urls(self, response):

canonicalize_url is called; by default it removes fragment. I think we should check what does fragment usually mean in pagination URLs and decide whether to keep it or drop. If that's too much to check that I think we should keep the fragment by default.