Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't refetch sitemaps that were already fetched #13

Open
pypt opened this issue Jul 19, 2019 · 0 comments
Open

Don't refetch sitemaps that were already fetched #13

pypt opened this issue Jul 19, 2019 · 0 comments
Assignees
@pypt
Labels
bug Something isn't working

Comments

@pypt
Copy link
Contributor

pypt commented Jul 19, 2019 

2019-07-19 14:26:46,279 INFO mediawords.util.sitemap.media [95859/MainThread]: Fetching sitemap pages for media ID 10 (https://globalvoices.org/)...
2019-07-19 14:26:46,282 INFO usp.fetch_parse [95859/MainThread]: Fetching level 0 sitemap from https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,282 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,800 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,801 INFO usp.fetch_parse [95859/MainThread]: Fetching level 0 sitemap from https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,802 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,963 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,979 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:46,979 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:47,407 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:47,413 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:26:47,414 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:26:48,998 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
<...>
2019-07-19 14:54:06,695 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,696 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,869 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,871 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:54:06,872 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:54:07,033 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
<...>
@pypt pypt added the bug Something isn't working label Jul 19, 2019
@pypt pypt self-assigned this Jul 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pypt pypt

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pypt