How can I skip already checked links?

[Benjamin-Berger]

When I run
"gallery-dl -i saved_users.txt --download-archive archive-DA.sqlite3"

Where saved_users.txt is a list of reddit usernames, I want it to skip previously checked links.

The following is an implementation I wrote to download link by link, but I figure there is a way i should be able to handle this in the command line.

def download_users(reddit, saved_users):
	downloaded_links_json = "downloaded_links.json"
	with open(downloaded_links_json) as f:
		downloaded_links = json.load(f)

	for username in saved_users:
		submissions = reddit.redditor(username).submissions.new()
		old_urls = set(downloaded_links.get(username, []))
		new_urls = submitted_urls.difference(old_urls)
		actually_download(username, new_urls)
		downloaded_links[username] = list(old_urls.union(submitted_urls))

def actually_download(username, submitted_urls):
	output_dir = f"gallery-dl\\reddit\{username}"
	for gallery_url in submitted_urls:
		command = f"gallery-dl -d {output_dir} {gallery_url}"
		subprocess.run(command, shell=True)

The idea behind the above is just to skip any previously checked urls. This runs way faster, and I dont mind saving a urls file. Is there anyway I can do this automatically? Thank you!

[Benjamin-Berger]

my issue with my implementation is that i cant get it to format the files the way i want them to.

even with this config file

    "extractor": {
        "directory": [],
        "reddit": {
            "directory": ["{category}", "{author}"],
            "parent-directory": true
        }
    },

sometimes i get the desired output
gallery-dl\reddit\[USERNAME]\[POST_TITLE]

but other times i get
gallery-dl\reddit\[USERNAME]\reddit\[USERNAME]\[POST_TITLE]
or even
gallery-dl\reddit\[USERNAME]\reddit\None\[URL]

[mikf]

I think this happens when a reddit post links to another site which then links to another reddit post. (links from reddit directly to another reddit URL are ignored by default, see extractor.blacklist)

Try adding "blacklist": ["reddit"] besides "directory": []
and maybe "blacklist": null to your reddit settings.

[Benjamin-Berger]

So I think something isn't working quite right in the archive function, or I am misunderstanding how it works.

I wrote my own implementation and the TL;DR: is that it takes half the storage space of the archive file, and runs ~25x faster

My use case was for reddit accounts.

my config looks like this

{
  "extractor": {
    "directory": [],
    "reddit": {
      "directory": [],
      "parent-directory": false
    },
    "imgur": {
      "directory": [],
      "parent-directory": false
    },
    "redgifs": {
      "directory": [],
      "parent-directory": false
    }
  },
  "downloader": {},
  "output": {
    "directory": "reddit",
    "filename": "{ifreddit|{title}|{name}}"
  },
  "postprocessor": {}
}

and the code looks something like this
reddit is just the reddit api

def downloader(reddit, saved_users):
	downloaded_links_json = "downloaded_links.json"
	with open(downloaded_links_json) as f:
		downloaded_links = json.load(f)
	for username in saved_users:
		submissions = reddit.redditor(username).submissions.new()
		submitted_urls = {}
		visited_urls = set()
		try:
			for submission in submissions:
				title = submission.title
				url = submission.url
				id = submission.id
			
				if url not in visited_urls:
					submitted_urls[url] = f"{id} {title.replace(chr(34), '_')}{get_file_extension(url)}"
					visited_urls.add(url)
		except Exception:
			pass

		old_urls = set(downloaded_links.get(username, []))
		new_urls = visited_urls.difference(old_urls)
		run_download(username, new_urls, submitted_urls)
		downloaded_links[username] = list(old_urls.union(visited_urls))
		
		with open(downloaded_links_json, 'w') as f:
			json.dump(downloaded_links, f, indent=4)
	

def run_download(username, new_urls, submitted_urls):
	output_dir = f"gallery-dl\\reddit\{username}"
	for gallery_url in new_urls:
		if "i.redd.it" in gallery_url: #because im getting the direct links, they dont have titles
			command = f'gallery-dl -d {output_dir} --filename "{submitted_urls[gallery_url]}" {gallery_url}'
		elif "/r/" in gallery_url and not f'/r/u_{username}' in gallery_url:
			continue #i dont want to have to worry about downloading subreddits that are linked
		else:
			command = f"gallery-dl -d {output_dir} {gallery_url}"
		subprocess.run(command, shell=True)

basically i go through a list of users, get their links. I check if i have already searched their links and if not, download them.
over 450 users rerunning with a completed archive took ~30000 seconds. running the same on this method took ~1100 seconds

[mikf]

gallery-dl uses an SQLite database and queries it for each and every file, while your implementation does most its work in memory and stores its data as simple JSON.

Using a database has the benefits of allowing multiple processes to access it at the same time and data persistence/consistency in case something goes wrong, but it is a lot slower than other methods as you found out.

[chapmanjacobd]

I would be interested in integrating gallery-dl to my project, library, in a similar way that I've integrated yt-dlp: https://github.com/chapmanjacobd/library/blob/main/xklb/tube_backend.py#L242

It looks like gallery-dl supports a similar postprocessor hooks mechanism. It would be nice if there was an example of how to use that similar to this: https://github.com/yt-dlp/yt-dlp#add-a-custom-postprocessor

The main idea is that I split up "playlist" parsing and downloading. This doesn't work with all extractors in yt-dlp and I imagine it won't work for all extractors in gallery-dl. For the yt-dlp extractors that it does work with, like YouTube, it allows for selective incremental downloading and multi-processing.

Do you think it would it work for any gallery-dl extractors? Or is the whole idea fruitless ?