use gallery-dl download links in text

[shawn8888]

Gallery-dl is a fantastic tool, and "r:URL" proves to be very powerful when the URL link is not officially supported. However, there is an issue when the links in the web page are in text format instead of being in hyperlink form, as gallery-dl cannot process them.
For instance, the following command won't work with the screens images:
gallery-dl -D c:\temp\test --filesize-min 50k "r:https://thepiratebay.org/description.php?id=70685244"
If you have any solutions or suggestions, please help!

[Hrxn]

Frankly, not sure what to think of this..

I get what you mean, but, to be honest, I don't think that gallery-dl is the right tool for this.
Or rather, that something like this is totally out of scope for gallery-dl.

I mean, what are you asking for here? To download and parse some random HTML document from the web?

If you want an honest suggestion, use your shell to do this and match your desired links with some regex, or use the Python route with something like BeautifulSoup

[shawn8888]

Thanks for your suggestion!
I am not a Python guru. What is the easiest way to scrape the URL and get the image links and feed them to gallery-dl?

[mikf]

there is an issue when the links in the web page are in text format instead of being in hyperlink form, as gallery-dl cannot process them.

The r: thingy matches everything starting with http:/https: regardless of it being in a href="…" or not.

gallery-dl/gallery_dl/extractor/recursive.py

Line 29 in d50c312

for match in re.finditer(r"https?://[^\s\"']+", page):

(You can check all detected URLs with -g)

It does, however, only download from recognized URLs and https://everest.picturedent.org/image/* is not among them since picturedent is not supported and they don't end with a filename extension like .jpg.

[shawn8888]

@mikf

1. I tried the -g option and it didn't detect the urls in text format.
   c:\>gallery-dl -g -D c:\temp\test --filesize-min 50k "r:https://thepiratebay.org/description.php?id=70685244"
   
   As you see above, none of the picturedent.org links in text were detected.

2. Although https://everest.picturedent.org/image/* is not supported but I can download it with r:

gallery-dl -D c:\temp\test --filesize-min 50k "r:https://everest.picturedent.org/image/7jR0"
My problem is how to do this without writing Python code.

[mikf]

As it turns out, /description.php sites on thepiratebay.org are mostly empty and get filled in with data from https://apibay.org/t.php?id=70685244 which r: doesn't see.

What you could do for this one page is copy-paste the links into a text file, add r: in front of each one, and then run gallery-dl with that file as --input-file

$ cat urls.txt 
r:https://everest.picturedent.org/image/7jR0
r:https://everest.picturedent.org/image/7jRD
r:https://everest.picturedent.org/image/7jRl
r:https://everest.picturedent.org/image/7jRo
r:https://everest.picturedent.org/image/7jRZ

$ gallery-dl --input-file urls.txt 
[1/5] r:https://everest.picturedent.org/image/7jR0
/tmp/_/directlink/everest.picturedent.org_images_2023_07_24_cEpHg3n.jpg
[2/5] r:https://everest.picturedent.org/image/7jRD
/tmp/_/directlink/everest.picturedent.org_images_2023_07_24_Au9Bbh.jpg
[3/5] r:https://everest.picturedent.org/image/7jRl
/tmp/_/directlink/everest.picturedent.org_images_2023_07_24_UKDu5GMY.jpg
[4/5] r:https://everest.picturedent.org/image/7jRo
/tmp/_/directlink/everest.picturedent.org_images_2023_07_24_2dnaL3.jpg
[5/5] r:https://everest.picturedent.org/image/7jRZ
/tmp/_/directlink/everest.picturedent.org_images_2023_07_24_nDkYhAmNt.jpg

[shawn8888]

Thank you @mikf for help!
Copying the URLs and using r: option to download is also one thing I want to avoid. I guess I have to take @Hrxn advise and use Python and BeautifulSoup to make it work.