kmod: eliminate duplication in calls to modinfo

[BroadlyWhitaker]

The original guard when building the new todo is:

todo += [m for m in depinfo.modules if m not in mods and m in nametofile]

mods only accumulates modules from previous iterations of the outer while loop. Within a single iteration, modules in the current batch are added to mods one at a time as moddep.items() is iterated. This means two distinct failure modes:

1. Cross-dependency within a batch: if modules A and B are both in the current todo and both depend on X, X passes the m not in mods check when processing A's deps and again when processing B's (since X hasn't been added to mods yet). X ends up in todo twice.

2. Intra-batch back-edge: if A depends on B and both are already in the current todo/moddep, B still passes m not in mods when processing A's dep list, queuing B for a redundant second modinfo call in the next iteration.

Both boil down to the same root cause: the check doesn't account for modules currently being processed.

The fix addresses both issues:

1. m not in moddep.keys() prevents modules from the current batch being re-queued into the next iteration's todo.

2. Turningtodo into a set prevents a module from appearing multiple times in the same next todo, when two modules in the current batch share a dependency

Because these duplicates frequently occur in practice, eliminating them provides an incremental speedup on top of the original optimization made in GH4092/GH4017.

[behrmann]

Thanks for the contribution! I will only have time to look at this later this week, but I started the CI run to see whether it works.

[behrmann]

Thank you! Sorry for the lag on my end.

[behrmann]

I've sat down an ran a tiny bit of benchmarking on this and running this against my host system, with empty inputs as

required_kernel_modules(kver, modules_include=[], modules_exclude=[], firmware_include=[], firmware_exclude=[], resolve_func=resolve_module_dependencies)

after making the function to resolve module dependencies pluggable to easily compare this, I get roughly the same run time for both functions, with the version from this PR actually a tiny bit, but consistently, slower.

At first I thought it was the chunking and I changed that to

        ittodo = iter(todo)
        while chunk := tuple(itertools.islice(ittodo, 8500)):
            moddep |= modinfo(kver, chunk)

which is roughly itertools.batch, which we don't yet have because it was only added in 3.12.

That helps a bit, but doesn't get one all the way, but the real trick is in lifting the todo update

        todo = set()
        mods |= moddep.keys()

        for name, depinfo in moddep.items():
            for d in depinfo.modules:
                if d not in nametofile and d not in builtin:
                    logging.warning(f"{d} is a dependency of {name} but is not installed, ignoring ")

            firmware.update(depinfo.firmware)
            todo.update(m for m in depinfo.modules if m not in mods and m in nametofile)

Since mods just gets the names from moddep added we can just add all keys from moddep to mods in one go. This then also makes the mods | moddep.keys() unnecessary. This shaves of 30% of the runtime off of required_kernel_modules for me.

Could you check that this makes sense and works in your usecase?

[BroadlyWhitaker]

Hi and thanks for looking at my PR.

How much benefit this does in practice depends both on the specifics of your KernelModules and on how fast your SSD is and It's difficult to give a meaningful general number. It's most impact comes when you specify a subset of modules, that have many common deps, and modinfo is called multiple times to resolve children-of-children. This isn't a common case, and the benefits are generally modest.

With your change I see smaller 0-10% faster here, and your version is also cleaner so I'm happy to include your suggestion. I also added a Co-Authored-By.

[behrmann]

Is this annotation necessary? I don't see mypy (or pyright or ty) complaning about this. My mypy version is 2.1. We do avoid unneeded annotations.

If it were necessary, there would be this small nit

Suggested change

	mods : set[str] = set()
	mods: set[str] = set()

[BroadlyWhitaker]

I removed the space but yes I get mypy error.

mypy --version
mypy 2.1.0 (compiled: yes)

mypy mkosi
pyproject.toml: [mypy]: python_version: Python 3.9 is not supported (must be 3.10 or higher). You may need to put quotes around your Python version
mkosi/kmod.py:296:5: error: Need type annotation for "mods" (hint: "mods: set[<type>] = ...")  [var-annotated]
        mods = set()
        ^~~~
Found 1 error in 1 file (checked 52 source files)

[behrmann]

Could you also incorporate this?

        # We could run modinfo once for each module but that's slow. Luckily we can pass multiple modules
        # to modinfo and it'll process them all in a single go. We get the modinfo for all modules to
        # build a map that maps the module name to both its module dependencies and its firmware
        # dependencies. Because there's more kernel modules than the max number of accepted CLI
        # arguments, we split the modules list up into chunks if needed.
        # TODO: use itertools.batched starting with 3.12
        ittodo = iter(todo)
        while chunk := tuple(itertools.islice(ittodo, 8500)):
            moddep |= modinfo(kver, chunk)

[BroadlyWhitaker]

you probably meant

moddep |= modinfo(context, kver, chunk)