Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various folders/config files - Priority: Getting the X.com files to download with date of being published in the filename along with the ID number string #6852

Open
bloodconfetti opened this issue Jan 18, 2025 · 0 comments

Comments

@bloodconfetti
Copy link

Sorry to be a bother, but please note that I'm very uneducated when it comes to technology, so anything you think is simple, probably has to be dumbed down a bit for me to be able to implement it.

I have three gallery-dl folders on my computer.

  1. C:\Users\redacted\AppData\Roaming
  2. C:\Users\redacted\gallery-dl
  3. One in my documents folder that's for a specific subject matter.

In 1. I have:

  • cach.squlite3
  • config.json

My config.json looks like:
{ "extractor":{ "cookies": [""], "x":{ "users": "https://x.com/{legacy[screen_name]}", "text-tweets":true, "quoted":true, "retweets":true, "logout":true, "replies":true, "filename": "twitter_{author[name]}_{tweet_id}_{num}.{extension}", "directory":{ "quote_id != 0": ["x", "{quote_by}" , "quote-retweets"], "retweet_id != 0": ["x", "{user[name]}", "retweets" ], "" : ["x", "{user[name]}" ] }, "postprocessors":[ {"name": "metadata", "event": "post", "filename": "twitter_{author[name]}_{tweet_id}_main.json"} ] } } }

In 2. I have:

  • gallery-dl.conf
  • x.com_cookies (file type: Text Document)

My gallery-dl.conf looks like:
`{
"extractor":
{
"base-directory": "./gallery-dl/",
"parent-directory": false,
"postprocessors": null,
"archive": null,
"cookies": null,
"cookies-update": true,
"proxy": null,
"skip": true,

    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0",
    "retries": 4,
    "timeout": 30.0,
    "verify": true,
    "fallback": true,

    "sleep": 0,
    "sleep-request": 0,
    "sleep-extractor": 0,

    "path-restrict": "auto",
    "path-replace": "_",
    "path-remove": "\\u0000-\\u001f\\u007f",
    "path-strip": "auto",
    "path-extended": true,

    "extension-map": {
        "jpeg": "jpg",
        "jpe" : "jpg",
        "jfif": "jpg",
        "jif" : "jpg",
        "jfi" : "jpg"
    },


    "artstation":
    {
        "external": false,
        "pro-first": true
    },
    "aryion":
    {
        "username": null,
        "password": null,
        "recursive": true
    },
    "bbc": {
        "width": 1920
    },
    "blogger":
    {
        "videos": true
    },
    "cyberdrop":
    {
        "domain": null
    },
    "danbooru":
    {
        "username": null,
        "password": null,
        "external": false,
        "metadata": false,
        "ugoira": false
    },
    "derpibooru":
    {
        "api-key": null,
        "filter": 56027
    },
    "deviantart":
    {
        "client-id": null,
        "client-secret": null,
        "refresh-token": null,

        "auto-watch": false,
        "auto-unwatch": false,
        "comments": false,
        "extra": false,
        "flat": true,
        "folders": false,
        "group": true,
        "include": "gallery",
        "journals": "html",
        "jwt": false,
        "mature": true,
        "metadata": false,
        "original": true,
        "pagination": "api",
        "public": true,
        "quality": 100,
        "wait-min": 0
    },
    "e621":
    {
        "username": null,
        "password": null
    },
    "exhentai":
    {
        "username": null,
        "password": null,
        "domain": "auto",
        "limits": true,
        "metadata": false,
        "original": true,
        "sleep-request": 5.0
    },
    "flickr":
    {
        "exif": false,
        "metadata": false,
        "size-max": null,
        "videos": true
    },
    "furaffinity":
    {
        "descriptions": "text",
        "external": false,
        "include": "gallery",
        "layout": "auto"
    },
    "gelbooru":
    {
        "api-key": null,
        "user-id": null
    },
    "gofile": {
        "api-token": null,
        "website-token": null
    },
    "hentaifoundry":
    {
        "include": "pictures"
    },
    "hitomi":
    {
        "format": "webp",
        "metadata": false
    },
    "idolcomplex":
    {
        "username": null,
        "password": null,
        "sleep-request": 5.0
    },
    "imagechest": {
        "access-token": null
    },
    "imgbb":
    {
        "username": null,
        "password": null
    },
    "imgur":
    {
        "mp4": true
    },
    "inkbunny":
    {
        "username": null,
        "password": null,
        "orderby": "create_datetime"
    },
    "instagram":
    {
        "api": "rest",
        "cookies": null,
        "include": "posts",
        "order-files": "asc",
        "order-posts": "asc",
        "previews": false,
        "sleep-request": [6.0, 12.0],
        "videos": true
    },
    "khinsider":
    {
        "format": "mp3"
    },
    "luscious":
    {
        "gif": false
    },
    "mangadex":
    {
        "api-server": "https://api.mangadex.org",
        "api-parameters": null,
        "lang": null,
        "ratings": ["safe", "suggestive", "erotica", "pornographic"]
    },
    "mangoxo":
    {
        "username": null,
        "password": null
    },
    "misskey": {
        "access-token": null,
        "renotes": false,
        "replies": true
    },
    "newgrounds":
    {
        "username": null,
        "password": null,
        "flash": true,
        "format": "original",
        "include": "art"
    },
    "nijie":
    {
        "username": null,
        "password": null,
        "include": "illustration,doujin"
    },
    "nitter": {
        "quoted": false,
        "retweets": false,
        "videos": true
    },
    "oauth":
    {
        "browser": true,
        "cache": true,
        "host": "localhost",
        "port": 6414
    },
    "paheal":
    {
        "metadata": false
    },
    "pillowfort":
    {
        "external": false,
        "inline": true,
        "reblogs": false
    },
    "pinterest":
    {
        "domain": "auto",
        "sections": true,
        "videos": true
    },
    "pixiv":
    {
        "refresh-token": null,
        "include": "artworks",
        "embeds": false,
        "metadata": false,
        "metadata-bookmark": false,
        "tags": "japanese",
        "ugoira": true
    },
    "reactor":
    {
        "gif": false,
        "sleep-request": 5.0
    },
    "reddit":
    {
        "client-id": null,
        "user-agent": null,
        "refresh-token": null,
        "comments": 0,
        "morecomments": false,
        "date-min": 0,
        "date-max": 253402210800,
        "date-format": "%Y-%m-%dT%H:%M:%S",
        "id-min": null,
        "id-max": null,
        "recursion": 0,
        "videos": true
    },
    "redgifs":
    {
        "format": ["hd", "sd", "gif"]
    },
    "sankaku":
    {
        "username": null,
        "password": null,
        "refresh": false
    },
    "sankakucomplex":
    {
        "embeds": false,
        "videos": true
    },
    "skeb":
    {
        "article": false,
        "filters": null,
        "sent-requests": false,
        "thumbnails": false
    },
    "smugmug":
    {
        "videos": true
    },
    "seiga":
    {
        "username": null,
        "password": null
    },
    "subscribestar":
    {
        "username": null,
        "password": null
    },
    "tsumino":
    {
        "username": null,
        "password": null
    },
    "tumblr":
    {
        "avatar": false,
        "external": false,
        "inline": true,
        "posts": "all",
        "offset": 0,
        "original": true,
        "reblogs": true
    },
    "twitter":
    {
        "username": null,
        "password": null,
        "cards": false,
        "conversations": false,
        "pinned": false,
        "quoted": false,
        "replies": true,
        "retweets": false,
        "strategy": null,
        "text-tweets": false,
        "twitpic": false,
        "unique": true,
        "users": "user",
        "videos": true
    },
    "unsplash":
    {
        "format": "raw"
    },
    "vsco":
    {
        "videos": true
    },
    "wallhaven":
    {
        "api-key": null,
        "metadata": false,
        "include": "uploads"
    },
    "weasyl":
    {
        "api-key": null,
        "metadata": false
    },
    "weibo":
    {
        "livephoto": true,
        "retweets": true,
        "videos": true
    },
    "ytdl":
    {
        "enabled": false,
        "format": null,
        "generic": true,
        "logging": true,
        "module": null,
        "raw-options": null
    },
    "zerochan":
    {
        "username": null,
        "password": null,
        "metadata": false
    },
    "booru":
    {
        "tags": false,
        "notes": false
    }
},

"downloader":
{
    "filesize-min": null,
    "filesize-max": null,
    "mtime": true,
    "part": true,
    "part-directory": null,
    "progress": 3.0,
    "rate": null,
    "retries": 4,
    "timeout": 30.0,
    "verify": true,

    "http":
    {
        "adjust-extensions": true,
        "chunk-size": 32768,
        "headers": null,
        "validate": true
    },

    "ytdl":
    {
        "format": null,
        "forward-cookies": false,
        "logging": true,
        "module": null,
        "outtmpl": null,
        "raw-options": null
    }
},

"output":
{
    "mode": "auto",
    "progress": true,
    "shorten": true,
    "ansi": false,
    "colors": {
        "success": "1;32",
        "skip"   : "2"
    },
    "skip": true,
    "log": "[{name}][{levelname}] {message}",
    "logfile": null,
    "unsupportedfile": null
},

"netrc": false

}`

Downloading an X.com account into 2. does not yield a publish date within the filename.

Do I just need to move the config.json into folder 2.? And likewise, would I have this line as such:
"cookies": ["C:\Users\redacted\gallery-dl\x.com_cookies.txt"],

For the record, originally I had my cookies file in location 1. but I got an error, so corrected it by moving that into the folder. Likewise, I was running into errors when I had my Cookies text file path in the location below, AND when I tried to say 'vivaldi' for my browser cookies to be used instead. So...

Right now, I scrape an X.com account with the command:
C:\Users\redacted\gallery-dl>gallery-dl --cookies x.com_cookies.txt URL

But I'd like not to have to include the --cookies part.

**Without moving the config.json file to folder 2. I receive the error:
C:\Users\redacted\gallery-dl>gallery-dl --cookies x.com_cookies.txt https://x.com/redacted [config][error] JSONDecodeError when loading 'C:\Users\redacted\AppData\Roaming\gallery-dl\config.json': Invalid \escape: line 3 column 24 (char 43)

When I move the config.json file to folder 2., having installed the above line in the cookies section, and try it out, I get the following error:
C:\Users\redaced\gallery-dl>gallery-dl https://x.com/redacted [config][error] JSONDecodeError when loading 'C:\Users\redacted\gallery-dl\config.json': Invalid \escape: line 3 column 24 (char 43) [twitter][error] AuthorizationError: Login required**

So I will be undoing those changes...

Thank you for any assistance! And please remember: if you think something is common sense. For me, it probably isn't! Lol, so sorry!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant