Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed Trimming Links for 6-1-2024 wiki dump #149

Open
lennonc1atwit opened this issue Jun 19, 2024 · 1 comment
Open

Failed Trimming Links for 6-1-2024 wiki dump #149

lennonc1atwit opened this issue Jun 19, 2024 · 1 comment
Assignees

Comments

@lennonc1atwit
Copy link

[INFO] Download date: 20240601
[INFO] Download URL: https://dumps.wikimedia.org/enwiki/20240601
[INFO] Output directory: dump

[WARN] Already downloaded sha1sums file
[WARN] Already downloaded redirects file
[WARN] Already downloaded pages file
[WARN] Already downloaded links file
[WARN] Already trimmed redirects file
[WARN] Already trimmed pages file

[INFO] Trimming links file

real    13m37.643s
user    13m43.964s
sys     3m58.226s

real    13m37.648s
user    13m43.967s
sys     3m58.229s

Tried using n1-highmem-16 (16 vCPU, 8 core, 104 GB memory) due to larger file sizes then when procedure was written (fails using recommended too), produces an empty gz temp file.

@jwngr
Copy link
Owner

jwngr commented Jun 19, 2024

Possibly related to #100.

@jwngr jwngr self-assigned this Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants