Question about file skipping behavior

[zakman4466]

I am trying to download all image files from a specific Twitter account. The first time I got it working, it downloaded a bunch of files, hit the rate limit, so I cancelled it and waited a bit. Next time I tried, it proceeded to zoom through all the filenames it had downloaded before (not downloading anything this time) and hit the rate limit again - even though it hadn't downloaded anything. I do have an archive file configured and haven't moved the files from their download directory in any case.

Is there a way I can tell it to completely skip the checking part and avoid all those extra calls to the API, and thus getting rate-limited? Or is this expected behavior? Relevant configs below:

    "extractor":
    {
        "base-directory": "F:/Downloads/gallery-dl-downloads",
        "parent-directory": false,
        "postprocessors": null,
        "proxy": null,
 
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0",
        "retries": 3,
        "timeout": 10.0,
        "verify": true,
        "fallback": true,

        "sleep": 0,
        "sleep-request": 0,
        "sleep-extractor": 0,
        },
<...>
        "twitter":
        {
	    "archive": "f:/gallery-dl/twit-archive.sqlite3",
            "cookies": ["firefox"],
            "cards": false,
            "conversations": false,
            "pinned": false,
            "quoted": false,
            "replies": false,
            "retweets": false,
            "strategy": null,
            "text-tweets": false,
            "twitpic": false,
            "unique": true,
            "users": "timeline",
	    "skip": true,
            "videos": false
        },

[Hrxn]

You're using cookies from Firefox, so far so good.

But the question is: Do you actually use this version of Firefox Firefox/102.0?

Because this can be checked by a site, and some sites actually do..

[zakman4466]

No - using the most recent version. I thought that was more or less a placeholder, but have updated it anyhow. Thanks for pointing that out.

[Hrxn]

Not sure if this is still an issue?

From what I've heard, rate limits aren't as restricted anymore..

But, fundamentally, you can't avoid making API calls at all, no.

[nisehime]

The first time I got it working, it downloaded a bunch of files, hit the rate limit, so I cancelled it and waited a bit

Why? gallery-dl will continue downloading after waiting for the rate limit reset automatically.

Is there a way I can tell it to completely skip the checking part and avoid all those extra calls to the API, and thus getting rate-limited? Or is this expected behavior?

This is expected behavior, gallery-dl has to scroll through all posts from the begging every time you start it. In theory, it should be possible to continue from certain point if you save the cursor or "page", but I think it'd be a hassle to implement for gallery-dl since every site would require it's own approach.