treewide: replace custom color scheme generator with Matugen

[make-42]

This PR is meant to further the discussion in (#694).

I believe matugen could be a good candidate for replacing the builtin color scheme generation engine stylix currently uses and this PR serves as a demo for it.

The code is not final:

• the old engine is still in the codebase
• polarity doesn't work like it's supposed to (it selects the color scheme and "either" defaults to black)
• a lightnessScale parameter could also be implemented as in: stylix: add darker and even-darker polarities #694
• the choice of colors used in the dark and light themes might need to be changed.

However this is what it looks like:

Closes: #269
Closes: #1862

[make-42]

primaryScale parameters have been implemented

[make-42]

primaryScale.dark = -0.05; gives you:

and primaryScale.dark = -0.1; gives you:

[danth]

This is definitely an improvement over the current generator, at least when I tried it on my own wallpaper :)

As this is still a WIP I've just left a couple of comments.

[make-42]

Okay, I'll work on cleaning this up, do I remove the code for the old palette-generator or do I make an option to have it be selectable for legacy purposes?

[make-42]

I fixed all of the issues that you brought up.

[make-42]

Do you want me to create a gallery of different color scheme options?

[make-42]

although for some reason, i can't get the removed option warning to show up

[make-42]

that was an easy fix...

[Flameopathic]

very cool! will this be able to extend to generating a base24 or vim highlight group scheme?

[make-42]

The base16 palette colors are exposed through config.lib.stylix.colors.baseXX, same as before, so you could definitely make any custom color scheme you want for vim if you don't like the default mappings.

As for base24 support that'd just involve exposing an extra 8 colors (matugen generates more colors than we use so that is definitely possible) and could probably be done if the project owners were okay with doing so. Although that would then force people to add the extra 8 colors if they want to import their own color palette and as of now no theme would take advantage of the extra colors.

[Flameopathic]

As for base24 support that'd just involve exposing an extra 8 colors (matugen generates more colors than we use so that is definitely possible)

awesome! that's good futureproofing and tackling #252

[danth]

do I remove the code for the old palette-generator or do I make an option to have it be selectable for legacy purposes?

The current palette generator was designed essentially through trial and error, so I'm not surprised that matugen is a lot better at generating nice looking themes. I would just remove it.

(Some more history - the original idea for Stylix was to take just a single option, that being the wallpaper, and do everything automatically from there. This is why the palette generator exists, and why #200 was an issue for so long.)

Anyone who wants to continue using a color scheme from the old palette generator may read /etc/stylix/palette.json or $XDG_CONFIG_HOME/stylix/palette.json before updating, and copy the values into their configuration as a manually defined scheme.

[make-42]

I'm also not sure about the colors I have assigned to the Base16 colors based off what matugen returns. I went with what looked good to me but they might need reassigning...

[make-42]

(also that last commit seems to have broken some things...)

[make-42]

If not for different base16 color assignments, it should be good to go.

[trueNAHO]

CI is failing. Maybe a git rebase master resolves the issue: 65c4263.

[make-42]

Let's see

[make-42]

error: 'checks.aarch64-linux' is not an attribute set... I'll look into it right now.

[make-42]

Might be easier to create a new branch and apply my changes from there instead of rebasing... Will do that instead...

[make-42]

still fails...

[make-42]

maybe it's because the palette-generator module is removed?

[trueNAHO]

The problem might come from:

error: attribute 'themeGeneration' missing
at /nix/store/227sl460khj2ld3y39a4qkg353vz7s21-source/stylix/testbed.nix:142:13:
   141|             testbed.name
   142|             stylix.themeGeneration.polarity
      |             ^
   143|             "image${lib.optionalString (stylix.image or null == null) "less"}"

Also, consider removing the hlint and stylish-haskell pre-commit hooks in /flake.nix.

[make-42]

Why should I put instead? stylix.themeGeneration.polarity is supposed to exist. I'll set it back to stylix.polarity to see if it works...

[make-42]

Oh but stylix is redefined just before so that's normal I suppose.

[make-42]

okay all tests pass except deadnix telling me that i need to remove the pkgs argument in flake.nix at lines 212, 226 and 239 but when i do all the other tests fail since they seem to rely on it...

[trueNAHO]

okay all tests pass except deadnix telling me that i need to remove the pkgs argument in flake.nix at lines 212, 226 and 239 but when i do all the other tests fail since they seem to rely on it...

I think you are supposed to replace the

Warning: Unused declarations were found.
     ╭─[flake.nix:212:11]
 212 │        { pkgs, ... }@args:
     │            ╰── Unused lambda pattern: pkgs
 227 │        { pkgs, ... }@args:
     │            ╰── Unused lambda pattern: pkgs
 241 │        { pkgs, ... }@args:
     │            ╰── Unused lambda pattern: pkgs

instances with just:

args:

[trueNAHO]

I will verify and experiment with the [extracted palette] values tomorrow.

After lots of investigation and experimentation, here are my relevant findings:

• Stylix custom palette generator:
• Old base16 mapping:
• Current base16 mapping:
• My first relevant mapping:
• My second relevant mapping:

Although the current mapping is the best one, it comes nowhere near the quality of the current custom base16 palette generator. While most of my mapping attempts output nonsense, even the currently best mapping results in illegible scenarios.

I also took the liberty to push some changes.

I think that the mappings should be the other way around. base24 > base16 > matugen
The reasoning being that base16 is style-wise a subset of base24, and base16 will have to support more "categories" distinguished by color than matugen. matugen only needs 3 colors that are distinct visually, and in most cases matugen themes will only really use the primary.

Either conversion direction is very lossy and suboptimal. I doubt an optimal mapping exists.

Maybe it would be possible to extend Matugen to also generate base-n color palettes, although this sounds like a lot of work. In the meantime, despite the Matugen to base16 mapping limitations, using Matugen seems like progress in the right direction.

I once had a discussion about this with maintainers of matugen, and it was deemed out of scope for them.

AFAICT, since Matugen is for our purposes "just" a thin https://crates.io/crates/material-colors CLI wrapper, extending it with base-n support seems very out of scope.

I guess that's true. I just worry that a bandage will reduce the motivation for an actual solution. So in the spirit of that, I think that having a proper base16 generator should be a high priority issue.

Considering Material You's inherently poor base-n conversion, I believe it would be better to entirely avoid it.

What about using a custom base-n palette generator and a Material You palette generator? In that case, this PR would merely add Matugen as a semantic representation. Before drastically shifting the scope of this PR, it would be better to first come to a general consensus.

Eventually replacing our Haskell implementation with a faster one is still relevant. Ideally, a true base-n palette generator exists to avoid upscaling base16 to base24 through interpolation.

[lucasew]

What if we go to a cromatic circle kind of approach where the image helps decide the color part but the brightness and saturation depends on user presets? Like if the user wants more colorful increase the saturation, wants darker reduce brightness. I wanted to test with oklch for that and then convert the colors to the RGB space so programs can use it. Maybe leaving the color chooser logic open or with a opinionated preset but allowing the user to override it, then the user can decide which colorscheme generator likes best or let them create their own.

[Mikilio]

What about using a custom base-n palette generator and a Material You palette generator? In that case, this PR would merely add Matugen as a semantic representation. Before drastically shifting the scope of this PR, it would be better to first come to a general consensus.

Eventually replacing our Haskell implementation with a faster one is still relevant. Ideally, a true base-n palette generator exists to avoid upscaling base16 to base24 through interpolation.

Full agreement. I would like to accept matugen as a generator for Material You themes as stylix already includes some applications that are better styled with that than base16. I am working on a base16 generator. Is there a style guide for base24?

What if we go to a cromatic circle kind of approach where the image helps decide the color part but the brightness and saturation depends on user presets? Like if the user wants more colorful increase the saturation, wants darker reduce brightness. I wanted to test with oklch for that and then convert the colors to the RGB space so programs can use it. Maybe leaving the color chooser logic open or with a opinionated preset but allowing the user to override it, then the user can decide which colorscheme generator likes best or let them create their own.

I used oklch and kclustering with selective dropout and early stopping to make sure we cover full colors even if the picture isn't very colorfull. It also can calculate primary/secondary colors. So far the results are pretty good. I just need to make it performant and also make some special adjustments for colorblind people that will be optional.

[trueNAHO]

What if we go to a cromatic circle kind of approach where the image helps decide the color part but the brightness and saturation depends on user presets? Like if the user wants more colorful increase the saturation, wants darker reduce brightness.

How would this specifically compare to matugen --type $SCHEME --resize-filter $FILTER?

Either way, adding advanced palette generator customizations should be considered at a later stage, as it is mostly speculative as this point. The interface and implementation might vastly differ depending on the number of palette generators. Very advanced palette generator customizations should be possible with #1956, although its implementation also vastly depends on the number of palette generators.

What about using a custom base-n palette generator and a Material You palette generator? In that case, this PR would merely add Matugen as a semantic representation. Before drastically shifting the scope of this PR, it would be better to first come to a general consensus.

Eventually replacing our Haskell implementation with a faster one is still relevant. Ideally, a true base-n palette generator exists to avoid upscaling base16 to base24 through interpolation.

Full agreement. I would like to accept matugen as a generator for Material You themes as stylix already includes some applications that are better styled with that than base16. I am working on a base16 generator.

I do not mean to discard your https://github.com/Mikilio/base16-generator efforts, but do base16 color scheme generators not already exist? For example, the results of the current Haskell implementation are already fairly good.

Is there a style guide for base24?

What if an application supports base32 or base64? Ideally, the palette generator algorithm is base-n generic.

Assuming some level of research goes into developing high-quality base16 color scheme generators, it would be better to extend an existing high-quality implementation instead of reinventing the wheel. Unless we are sure that the base16 to base-n generalization requires a complete redesign, we should extend an existing implementation.

It might be best to first answer the following questions:

1. As mentioned in treewide: replace custom color scheme generator with Matugen #892 (comment), should the scope of this PR be shifted to only adding a Material You semantic representation, instead of mapping Material You to base16?

2. How relevant is improving the performance of our Haskell implementation?

3. Is the base-n generalization a good idea?

Here are my answers: (1) Yes. (2) Very low priority. (3) Low priority: Since the only benefit is higher accuracy by avoiding upscale interpolation, this can be ignored for now.

---

CC: @0xda157, @Flameopathic, @danth

[Mikilio]

I do not mean to discard your https://github.com/Mikilio/base16-generator efforts, but do base16 color scheme generators not already exist? For example, the results of the current Haskell implementation are already fairly good

Not to disregard the efforts of the Haskell implementation. But I wouldn't put in the effort if I did agree with this statement. I also couldn't find a base theme generator from images that is genuinely good. The Haskell algorithm is probably a really nice piece of software, but the reason I believe it is lacking is that the fitness score is not calculated based on criteria that are significant for a "good" base16 theme.

What if an application supports base32 or base64? Ideally, the palette generator algorithm is base-n generic.

It is not correct to assume that there is somehow a agnostic way to generate these. That is also something that bothers me about the haskell generator, except for the background it treats all colors the same even though every color has a distinct purpose.

With respect to base24<->base16 conversion seems to be possible without issue using the base24builder by the creator of base24.

I have now added explanations to the experiment's notebook in https://github.com/Mikilio/base16-generator. Any further commits will only be for the actual performant implementation of this method.

That being said, if anyone likes the bold and monochrome themes of the Haskell generator, they are free to use those.
As for the base16 generator, I will create an issue to continue the discussion, as the conversation with respect to this PR has indeed reached a conclusion and I don't want to continue spamming. (sorry so far)