Question about Downloader Middlewares

[nehakansal]

There is a list of Downloader Middlewares in the settings file. Does that list mean that those are the only middlewares that are activated for any crawl? I am asking this because I noticed that the RetryMiddleware is working but the MetaRefreshMiddleware doesn't seem to be working - and none of them are specified in the middlewares list. Could someone explain that please? Thanks.

[lopuhin]

@nehakansal sure, middlewares active by default are listed here https://doc.scrapy.org/en/latest/topics/settings.html#downloader-middlewares-base

[nehakansal]

Got it, thanks. So, the MetaRefreshMiddleware is active and the METAREFRESH_ENABLED setting is true by default - but I saw that it didnt work on a site crawl. Any thoughts on that?

[lopuhin]

@nehakansal indeed it should be enabled according to the docs, and it's not disabled in the project explicitly. I never used it myself, so no ideas. But if you can build a small reproducible example, or provide more details, you can try submitting this to the scrapy issues: https://github.com/scrapy/scrapy/issues/

[nehakansal]

Okay, thanks. I will submit it to the scrapy issues.