.py script via python.function

[masha9000]

I planned to run a simple script that adjast info.json from ytdl (yt-dlp).
If I run this script in the folder directly py -3 %APPDATA%/gallery-dl/ytdl/json-regexp-yt.py - it works without problems. But I can’t add it to the gallery-dl configuration python.function. It would be optimal for it to work only after downloader.ytdl, but I tried adding it to the extractors postprocessor: without result also. No error occurs, it is not even visible in the logs. There were many silly vars, the last one:

"python":					
	{
	"event":	"finalize",
	"function":	"%APPDATA%/gallery-dl/ytdl/json-regexp-yt.py"
	}

Clearly I'm making some stupid mistake. Could you help me figure it out?

[mikf]

You need to give it a name/type and you didn't specify the function name in json-regexp-yt.py:

{
    "name":     "python",
    "event":	"finalize",
    "function":	"%APPDATA%/gallery-dl/ytdl/json-regexp-yt.py:function_name"
}

[masha9000]

I still can’t get it to start from under:

"downloader":
	"ytdl":
		{
		"enabled":	true,
		"module":	"yt_dlp",
		"archive":	"%APPDATA%/gallery-dl/ytdl/archive-ytdl.sqlite3",
		"config-file":	"%APPDATA%/gallery-dl/config-ytdlp.txt",
		"format":	"bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b",
		"outtmpl":	"(%(upload_date>%Y-%m-%d)s) %(title)s.%(ext)s",
		"cmdline-args":	"--write-info-json",
		"generic":	true,
		"raw-options": 	null,
		"forward-cookies": false,
		"logging": 	true,
		"postprocessors": [
				{
				"name":     "python",
				"event":	"finalize",
				"function":	"%APPDATA%/gallery-dl/ytdl/json-regexp-yt.py:json_regexp"
				}
			]
		}

Could there be a problem with the script itself. If I transfer a section from downloader to extractor (e.g. Artstation), the script will run, but with an error - it will try to process json files in windows/system32. This is understandable, if there was a simple opportunity to transfer the working folder to it, it could generally be attached to yt-dlp --exec
I apologize for my stupidity, so as not to be unfounded, here is the script itself:
```
import os
import re

def json_regexp():
    # Regexp pattern
    pattern = r'(.+)"description": "(.*?)"(.+)"view_count": (\d*?),(.+)"webpage_url": "(.+?)"(.*)"tags": \[(.+?)\],(.+)"uploader": "(.+?)"(.+)"uploader_url": "(.+)", "upload_date": "(\d{4})(\d{2})(\d{2})"(.+)"duration_string": "(.+?)"(.+)"resolution": "(\d+)x(\d+)"(.+)'
    replacement = r'\6\n\13-\14-\15\n\19 x \20\nTime: \17\r\nView: \4\n\n\10\n\12\n\n\2\n\n[\8]'

    # File list of current directory
    current_dir = os.getcwd()
    file_list = [f for f in os.listdir(current_dir) if os.path.isfile(os.path.join(current_dir, f))]

    # Every .json file
    for file_name in file_list:
        if file_name.endswith('.json'):
            file_path = os.path.join(current_dir, file_name)
            with open(file_path, 'r') as file:
                content = file.read()

            # Regexp work
            replaced_content = re.sub(pattern, replacement, content)

            # Rewrite file
            with open(file_path, 'w') as file:
                file.write(replaced_content)

json_regexp()

[mikf]

Downloaders do not support a postprocessors option. This only works for extractors.

You'd need to use a global post processor for all extractors (and maybe filter by URLs starting with ytdl:) to run this function.

{
    "extractor": {
        "postprocessors": [
            {
                "name":     "python",
                "event":    "finalize",
                "function": "%APPDATA%/gallery-dl/ytdl/json-regexp-yt.py:json_regexp"
            }
        ]
    }
}

---

def json_regexp():

The called function needs to accept 1 argument: The metadata dict.

def json_regexp(metadata):

You should also add a simple print("foobar") to start with, just to see if it actually runs.

[masha9000]

ok, postprocessors don't work with downloaders. Now it’s clear why I was banging my head against the wall.

But on the second point there are more questions than answers. (I repeat once again, I am never a coder.)
It is clear that the script needs to be given a path each time, but for some reason I thought that it was more logical and safer to execute scripts locally in the actual download directory, and therefore the script would receive this path from the gallery-dl postprocess functions. But if I send the script to the metadata argument (they themselves also do not contain a custom path), does this mean I need to create an argument for my script from these metadata fields?
Isn't there a way to just tell it to run in a folder that I've already created in the extractor section (I don't know, using the directory value as a normal path-argument to the script)?
That is, of course, I’ll try to create a directory_path from metadata fields, but firstly different extractors have different designs for folder names, and secondly then the script will no longer be universal, but “tailored” exclusively for gallery-dl.
Let me remind that this script should be launched only after yt-dlp downloads the videos found using gallery-dl extractors. In this case, yt-dlp creates info-json, which obviosly cannot be edited independently on the fly with yt-dlp options, as, unlike it, content-format can do with gallery-dl metadata. And yes, in other cases the gallery itself does a great job.

[mikf]

To run this script after every yt-dlp download, you'd have to enable metadata-url and then check if a URL starts with ytdl:, which means it got downloaded by yt-dlp from another extractor.

{
    "extractor": {
        "metadata-url": "_url",

        "postprocessors": [
            {
                "name"    : "python",
                "event"   : "after",
                "filter"  : "_url.startswith('ytdl:')",
                "function": "%APPDATA%/gallery-dl/ytdl/json-regexp-yt.py:json_regexp"
            }
        ]
    }
}

(I'm using event: after to make sure yt-dlp is definitely done and has written all of its output files)

Regarding paths and download locations, you can enable metadata-path and then use this object's attributes in your script.

    "extractor": {
        "metadata-url" : "_url",
        "metadata-path": "_path",

       "...": "..."
    }

import os
import re

def json_regexp(metadata):
    # Regexp pattern
    pattern = r'(.+)"description": "(.*?)"(.+)"view_count": (\d*?),(.+)"webpage_url": "(.+?)"(.*)"tags": \[(.+?)\],(.+)"uploader": "(.+?)"(.+)"uploader_url": "(.+)", "upload_date": "(\d{4})(\d{2})(\d{2})"(.+)"duration_string": "(.+?)"(.+)"resolution": "(\d+)x(\d+)"(.+)'
    replacement = r'\6\n\13-\14-\15\n\19 x \20\nTime: \17\r\nView: \4\n\n\10\n\12\n\n\2\n\n[\8]'

    # take the full path and replace its extension with 'info.json'
    file_path = metadata["_path"].realpath.rpartition(".")[0] + ".info.json"

    with open(file_path, 'r') as file:
        content = file.read()

    # Regexp work
    replaced_content = re.sub(pattern, replacement, content)

    # Rewrite file
    with open(file_path, 'w') as file:
        file.write(replaced_content)

[masha9000]

Work like a charm. A little later I’ll tune regexps for vimeo and others - it will be absolutely great.
Thank you again. I couldn't do this without your help.

[Hrxn]

Short question here, if you use something like
"config-file": "%APPDATA%/gallery-dl/config-ytdlp.txt",
(or anywhere else gallery-dl's Path type gets used in the end, like in python.function etc.)

Which makes use of Python's own os.path.expanduser() and os.path.expandvars(), is there a potential issue with paths on Windows?

I've only tested this here locally in my Python interpreter, I don't want to touch my gallery-dl config yet with this, because this is basically productive data for me.

Because the problem, at least on Windows, is that something like %APPDATA% is a bit ambiguously defined.
It's always a path, but it both works with a trailing backslash/path-separator as well as without one, because this is handled internally by Windows' own APIs.

Which is a) good, but also means that b) Windows does not specify the pathnames consistently, even in its own default settings, e.g.

C:\Windows\System32
C:\Windows\System32\OpenSSH\

(Which should be the defaults, even on a fresh Windows installation, AFAIK)

And because Python's own os.path functions apparently do the expansion always by a simple string replacing, you can end up with something like this here:

>>> os.path.expandvars('$APPDATA')
'C:\\Users\\Hrxn\\AppData\\Roaming'
>>>
>>> os.path.expandvars('${Example_Path_Env_Var}')
'C:\\Sources\\gallery-dl\\'

So if a path coming from the environment ends with a trailing path separator, you'd ultimately have either a // or a \\\\ in your path that gets then used further by the Python interpreter.

If, for example, the environment variable does not exist, Python does not bail immediately with an error

>>> os.path.expandvars('${Does_not_exist}')
'${Does_not_exist}'
>>>

But in such a case, the path would simply be invalid, and gallery-dl could stop operation here as soon as such an error occurs.

[mikf]

is there a potential issue with paths on Windows?

Probably, but I'd think that at least APPDATA is always defined and does not end with a trailing \. At least by default.

So if a path coming from the environment ends with a trailing path separator, you'd ultimately have either a // or a \\\\ in your path that gets then used further by the Python interpreter.

At least on Linux (and probably any OS that isn't Windows) duplicate / don't matter.

$ mkdir -p a///b//c
$ cd a///b//c
$ pwd
~/a/b/c

I don't want to touch my gallery-dl config yet with this, because this is basically productive data for me.

Make a backup copy.

[Hrxn]

Probably, but I'd think that at least APPDATA is always defined and does not end with a trailing \. At least by default.

Well, yes and no. It does not have a trailing path-sep by default, that is certainly true, but it's not something that's defined, or a definite requirement.

Make a backup copy.

I did in the meantime, and from what I can tell, it seems to work.
But not entirely sure if this is happenstance, or actually correct operation of Python on Windows..

At least on Linux (and probably any OS that isn't Windows) duplicate / don't matter.

$ mkdir -p a///b//c
$ cd a///b//c
$ pwd
~/a/b/c

It doesn't really matter on Windows either, but I wasn't really sure about Python (on Windows), hence my asking this question.

PS itself is completely agnostic to / and \ - command syntax is also the same, although the -p flag is technically not the same.

PS C:\Apps\Temp> mkdir -p a///b//c

    Directory: C:\Apps\Temp\a\b

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          2023-12-09    15:20                c

PS C:\Apps\Temp> cd a///b//c
PS C:\Apps\Temp\a\b\c> pwd

Path
----
C:\Apps\Temp\a\b\c

PS C:\Apps\Temp\a\b\c>

(The only real difference here is that PS returns output by default for mkdir. Which could be turned off, but not without modifying the used input command-line)

This even works in CMD if you use multiple backslashes, and remove -p because that would be wrong option syntax there.

I did this test here locally with the Python interpreter, so, again, it seems to work?
Or is gallery-dl doing something different, and I maybe shouldn't make any deduction here?

PS C:\> py
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> open('C:////Apps\\\\Temp\\//\\test.ext')
<_io.TextIOWrapper name='C:////Apps\\\\Temp\\//\\test.ext' mode='r' encoding='utf-8'>
>>> open('C:\\\\Apps////Temp//\\//test.ext').read()
'This is simply a test!\n'
>>>

Not sure if that \n is expected, though.
The file is UTF-8 (no BOM) and ends with a single LF. May still be correct..

[mikf]

It doesn't really matter on Windows either

Then paths like "%APPDATA%/gallery-dl/..." shouldn't be a problem, regardless of the value of %APPDATA%.

Where it does matter, I think, are raw/extended paths, but those do only get used for downloaded files.

Not sure if that \n is expected, though.

It is. Python does its own, platform-independent text file processing and converts all line separators to just \n. See open() for details.

[rautamiekka]

>>> open('C:////Apps\\\\Temp\\//\\test.ext')
>>> open('C:\\\\Apps////Temp//\\//test.ext').read()

Need to remember Python uses \ as the escape character, so in pure Py making it a raw string is recommended for simplicity.