Unfree software is unintuitively hidden for new users

[grahamc]

Issue description

New users try to install unfree software, search for it with nix-env -qaP, and find it missing. Some more enterprising users search nixpkgs on GitHub and find it is definitely there but don't know how to find it.

Example chat where this happened: https://botbot.me/freenode/nixos/2016-07-20/?msg=69976623&page=9

Perhaps a way to improve this would be show the unfree software in the list, but with a note about it being unfree.

Steps to reproduce

1. Be a new user who is evaluating NixOS/Nixpkgs
2. Try to install hipchat
3. Find it isn't in nix-env's list
4. Confirm it is in nixpkgs via GitHub
5. ??? until they find { allowUnfree = true; } or ask in IRC

[grahamc]

re:

• make nix-env -qa include unfree packages #5628
• nix-env --query should show unfree packages by default nix#432

I think this should be revisited. I think we're doing users a disservice by making it difficult to find the software they're wanting to choose. Right now we're choosing to appease strict FOSS users over reducing the headache for newcomers.

I'd rather the onus on more advanced users to get rid of it, than new users to get started with the tools they need and are familiar with.

[copumpkin]

Agree. I think this is largely an incidental consequence of not wanting Hydra to do unfree stuff, but I think the mechanism should be a made a bit more fine-grained to be friendlier to users.

[vcunat]

I tried, and I think it's hard to find strong consensus on this point.

Personally I'm comfortable that official nixpkgs accepts significantly wider range of packages than it's usual for most other large distributions. BTW, we're slowly shifting to specifying packages directly by attribute names, and in that case the user gets a useful evaluation error.

[grahamc]

BTW, we're slowly shifting to specifying packages directly by attribute names, and in that case the user gets a useful evaluation error.

Yeah, this error is really helpful if they get far enough to try to install it that way. A bit tough, though, to get that far when a nix-env search tells you it isn't there.

[abbradar]

I think we really do need to show the user somehow that there are other packages. I've taken a liking to this suggestion, option II. Other solution is to show such packages but place big red label "UNFREE" next to them.

EDIT: after some thoughts I like my solution less, because this heavily depends on what do we consider "promotion". The other suggestion, however, is very unobtrusive IMHO and doesn't even name anything, just says that there are matches.

[PlasmaShift]

Put the allow unfree information where people go and look at how to install software

When they generate your Nixos configuration and have kde and wget " I think that is the one package they show as an example to install" commented out put allow unfree in a comment there as well

When they look at the manual on how to install packages put a note somewhere really close so they see it. Probably at two places, one for nix "nix-env -i ... or nix-env -qa ..." and one for Nixos "inside your configuration.nix

when they search on Nixos.org packages in the description make a note, ex nonfree -- rest of description
in package name or attribute name it would would get in the way of normal operations. I am not concerned about nonfree when I am greping around nix-env -qa

These are the places I would go/went to find out how to install applications
This way would also not interfere with the nix way as describing your os

As a side note it would also be nice if when you generated configuraiton.nix you would get a stateless user example.

[peti]

The real issue is that nix-env -qa searches are fundamentally broken and don't find literally thousands of useful packages that we have. This phenomenon is not limited to unfree software.

[domenkozar]

This came up again today on irc:

22:50:36           tosmi | hi, i'm a nixos newbie and unable to install the citrix-receiver package   
22:51:09           tosmi | nix-env -qa | grep doesn't find citrix-receiver               
22:51:34           tosmi | i suppose because it depends on an EULA protected blob...                                                                  

[domenkozar]

I think Nix UI redesign is a good opportunity to fix this.

[grahamc]

Today a user on IRC (ToxicFrog) spent four hours trying to package Sublime3 only to find out it was hidden behind unfree.

[grahamc]

Today another user on IRC (riptawr) spent quite some time trying to figure out discrepancies between nixos-rebuild (as root) and nix-env (as his user) due to his profile not having allowUnfree.

[vcunat]

It's very common that people are confused from nix-env not using nixpkgs config from nixos config. I put it into a separate *.nix file and import in both; I haven't seen a better approach yet – maybe it could be suggested in the docs. (But it seems a digression from the original topic.)

[Eilie]

Hello, I'm a new NixOs user and I've spent an hour trying to figure that out and then got help on IRC. Would be great to have this resolved in some intuitive way.

[alecmev]

Just spent 30~60 minutes trying to figure out why I can't find google-chrome via nix-env -qa, even though it's clearly in the repo, until I've been pointed to this issue by kind people in #nixos. Maybe add a quick note about allowUnfree somewhere high up in the manual, at least in Chapter 2 and Chapter 8, until a warning is implemented? Also, I don't see allowUnfree documented anywhere, it juts shows up in random issue descriptions and config excerpts (e.g. this wasn't obvious, spent extra 15 minutes on that).

[vcunat]

For reference, the documentation is in nixpkgs manual.

[vcunat]

Also note that restriction of unfree stuff is primarily a "political decision" IMHO. We've had lots of discussions, e.g. on #5628.

[7c6f434c]

I think the discussion right now is not as much a plea to reverse that decision as to declare it more visibly. Proudly declaring this decision is not against this decision, and those who want the override will know what to look for. For the record: I support requiring an explicit action for unfree packages, and I do have some unfree packages installed. I also think that mentioning this filter close to the beginning of NixPkgs manual is a good idea.

[ToxicFrog]

My preference would be for package search (both the website and nix-env -qa) to list nonfree packages by default, but display some kind of indication that they're nonfree. nixos-homepage PR 120 does this for the website, but is blocked on license information being included in the search page. (I noted on that PR that this is the approach taken by other free-software distros such as Debian.)

Failing that, there should at least be (by default) a prominent warning that non-free software isn't being included in the search; a large part of the problem here is that it's not at all evident to the user that their query isn't actually finding everything. An option to enable it would also be nice (manually searching the nixpkgs repo is pretty awful UX), but making sure the user knows that they need to do that at all is a necessary first step.

I wasted a lot of hours trying to repackage nonfree software like Steam, Chrome and Sublime3 before someone on IRC pointed out that the package search was lying to me and I needed to instead search the nixpkgs repo directly. Someone else repeats this experience on a regular basis, as evinced by this very thread.

[vcunat]

Suggestions on showing unfree packages in searches by default are consistently disliked by some core devs (Eelco, Rob, 7c6f434c above, maybe more), so I don't feel like pushing that point anymore.

I like the approach of mentioning the fact that they're hidden on some (more) prominent place. By itself that would seem more like propagating "free" SW than "unfree". However, I don't have a good idea about what parts of docs/website are best visible to newbies.

[ToxicFrog]

For my part I tended to lean heavily on https://nixos.org/nixos/packages.html

Even if they aren't shown by default there should be some way to enable them rather than just shrugging and going "use github search then".

[7c6f434c]

I think all the way that allow installation ($NIXPKGS_ALLOW_UNFREE and Nixpkgs config) enable search, too.

[ToxicFrog]

Good point. That works for people using the command line, and in that case I think the suggestion on #5628 of including allowUnfree = false in the autogenerated templates, with comments, will help a lot with making people aware that the option exists.

That said, the website is both a nicer UI and much faster than nix-env -qa even on a fast machine; it shouldn't be a second class citizen in what it allows you to search for.

Perhaps the answer is just to add "include unfree software" and "include broken packages" checkboxes next to the search box, and default both to off?

[7c6f434c]

I wonder if regardless of our current discussion it is a good idea to make it easy to generate a JavaScript-only local-HTML-file package list with search. Of course, it will follow the brokenness/insecurity/lack-of-freedom choices set by the user.

The problem with package list on the site is that it is a list of Hydra jobs. Of course, Hydra won't build things where it is unclear whether they are OK to distribute.

[jcrben]

Perhaps an interactive prompt when installing could be a compromise? Also spent some time trying to figure out what was going on before hopping on IRC. Not all beginners are up to ask on IRC. Note also that this is documented in the nixpkgs manual but not, as far as I can tell, in the nix manual, which is where I started...

[7c6f434c]

Technically, this is not a Nix-level decision, so it is not in the Nix manual — the license attribute lives in NixPkgs, the license check is performed by code in NixPkgs repository without any special support from Nix, etc.

I somewhat like being able to use NixPkgs for free software with no NixPkgs config at all, so I can understand unwillingness to put warnings unless configured.

Maybe enumerate the natural places like NixPkgs package search where adding word «free» and a footnote «unfree packages need to be explicitly enabled and are never built on Hydra» would make sense?

[samueldr]

The command-not-found handler, as it is, cannot have unfree software in it because it would imply the hydra build farm would need to build said software to list the binaries available. Same for nix-index or related friends; basically any introspection into the built package is probably out of the question with how things are for unfree.

As far as not introspecting too far, meaning only listing what is known into the Nixpkgs repository, it is possible to list unfree software, meaning it could be possible to list it on the website.

Maybe a middle ground could be reached, where a list of available binaries names (bin/*) is added into meta.bin, allowing command-not-found to be made aware of unfree software. It is technically possible to help make the situation better.

Though, this said, it probably wouldn't happen without the allowUnfree opt-in, if at all, unless @edolstra's opinion has changed, and other contributors have no similar moral objection.

As for my opinion, I'm definitely torn over the issue, the pragmatic in me says: "list the software already", the idealist in me says: "eww proprietary and unfree software", which is probably why I would implement it behind a checkbox or a boolean option, defaulting to unfree software being out of the system.

As this is a discussion about the discoverability of unfree software, let's not steer into "enableUnfree by default", and think of ways to help the UX while still holding on to our convictions.

^{N.B. This is all my personal opinion.}

[7c6f434c]

I wonder how acceptable or non-acceptable is having a NixOS option that simultaneously changes nixpkgs.config and adds NIXPKGS_ALLOW_UNFREE=1 in the default environment (obviously, a user's profile can unset the variable). Maybe also set allowUnfree=false; in generated profile, so that users can know that the flag exists.

[ToxicFrog]

@samueldr Also relevant is NixOS/nixos-homepage#120 (include all packages in the index). It was blocked on package licenses not being properly displayed, but that should have been fixed by NixOS/nixos-homepage@728f6fd

[Ekleog]

@samueldr Personal opinion: the technical solution you put forward for not introspecting too far sounds workable, and I think that unfree packages should be displayed with a big warning “This is unfree!” (or “This is broken!” or “This has security issues!”) sign next to it (not a checkbox, a checkbox will be missed and generate useless duplicate work). Something similar to F-Droid and its “anti-features” list, though F-Droid lists only free software because they have to redistribute binaries.

[ghost]

I would consider myself fairly close to being a FOSS hardliner, but there are situations – video calls – where even I have to run unfree software on my system and I find the best way to do so is to whitelist unfree packages rather than use allowUnfree=true; which would give me less control:

nixpkgs.config.allowUnfreePredicate = (pkg: builtins.elem (builtins.parseDrvName pkg.name).name [
    "unfree1"
    "unfree2"
    …
  ]);

But this makes unfree packages unsearchable by default using nix-search, which is a pain. I thus use the environment variable pointed out by @7c6f434c for a quick dirty workaround:

NIXPKGS_ALLOW_UNFREE=1 nix search ${pkg}

Just use a shell alias for increased convenience:

alias nix-search='NIXPKGS_ALLOW_UNFREE=1 nix search'

Hopefully this may be of use to some fellow user that does not feel like opening the unfree floodgates just to find all available software from the command line.

[samueldr]

@Ekleog right, I'm using "checkbox" here as a way to keep myself flexible because it depends on the direction the project wants to take as far as publicizing unfree software. One could go the idealist way and ensure it's hidden by default (current situation, minus ability to list in some search), or the pragmatic way, list everything and label accordingly.

I have no strong preference for one or the other, both generally have the same basic requirements (knowing the unfreeness), the rest is mainly an implementation detail according to the vision of the distro.

[jappeace]

Hi, I stumbled on this through google.
So I'm trying to use nix for a client, and it's a bit of a pain I have to opt in (not that I disagree with it in principle). For future reference, this is how you enable non-free in a pin: https://github.com/jappeace/haskell-template-project/blob/45fa410733376d588f7d29d7a9b642189fc8e402/pin.nix#L13

[gloaming]

For the record, I'm not in favour of information hiding in general. However:

Suggestions on showing unfree packages in searches by default are consistently disliked by some core devs

Nixpkgs is intended as a free software distribution, so it should not promote the availability of unfree packages.

The project's view about unfree software is known. We shouldn't promote unfree software.

I like my solution less, because this heavily depends on what do we consider "promotion"

I think it would be good to clarify this. If a user asks for a specific package, then they must already know about it; so does it count as "promotion" to acknowledge its existence? I think there are two use cases of package search that should be distinguished:

1. I want to learn about what packages are available to do a task:

$ nix search play music
...

2. I want to install a particular package, and I want to see its attribute name and version:

$ nix search spotify
...

Of course, we can't know what the user is trying to do. But if we choose to, we can make a best effort guess. I suggest the following compromise behaviour when search for unfree packages has not been explicitly enabled or disabled:

• Don't search unfree package descriptions or other meta
• Only search unfree names and attribute names
• Only return matches with reasonable word boundaries

With rules like these, a search for play music or spot would not return spotify, but spotify could. A search for lime or text editor would not return sublime3, but sublime could.

(The obvious UX disadvantage is that the search behaviour is unusual, potentially confusing, and inconsistent between free and unfree packages.)

These matching rules could also be combined with something like #5628 (comment), if we still don't want to display the results by default.

Also, I'm sure this must have been discussed before, but the current behaviour also has the problem that it actually encourages users to set allowUnfree so that they don't miss any packages. It might be better to have e.g. searchUnfree so that users don't have to do that.

[7c6f434c]

… and a problem of trying to go into maximum-purity evaluation mode is that NIXPKGS_ALLOW_UNFREE is very often the best way for dealing with one-off unfree package task.

[Dema]

Frankly speaking I, as a fellow NixOS user, don't really care if "some core devs don't like non-free software". If you don't want your users to use non-free stuff, just simply ban it completely from a distro. And if you allow it, make it first-class citizen.

[7c6f434c]

If you don't want your users to use non-free stuff, just simply ban it completely from a distro.

This is too black and white. The point is that some core devs want to interact with Nixpkgs _as if_ there were no non-free packages, while not banning such packages completely for those who wants them. Others have a need to install a specific nonfree package, but want to retain tight control over the list of installed nonfree packages. As there are many different ways to evaluate Nixpkgs on a typical system and they are controlled by slightly different settings, the experience code devs want for themselves requires default-off nonfree software.

And if you allow it, make it first-class citizen.

This can never happen. First-class whatever is supported by the binary cache, and nonfree packages are often illegal to redistribute in modified form.

[SRGOM]

Frankly speaking I, as a fellow NixOS user, don't really care if "some core devs don't like non-free software". If you don't want your users to use non-free stuff, just simply ban it completely from a distro. And if you allow it, make it first-class citizen.

I don't even think anybody on the Nix core team dislikes non-free software. I know one person did but he left to create his own GNU distribution which uses hurd ... GUIX.

[tbenst]

As a user of non-free software, it bothers me the number of times that an update to, say CUDA or MKL breaks a bunch of nix packages. I recently fixed scikitlearn to build with MKL, and less than a week later it’s broken again after MKL got updated. Even with the newly fixed tensorflow it seems that the default CUDA 10(.1) fails to build and user must manually override to 10.0.

First-class whatever is supported by the binary cache, and nonfree packages are often illegal to redistribute in modified form.

Anyone know how Anaconda gets away with distributing binaries of MKL and CUDA?

I wonder if there’s a middle ground where even if we don’t distribute binaries of unfree software, we might configure hydra to test builds against common unfree software? Would make packaging a lot more stable for common workflows.

[AMDphreak]

Frankly speaking I, as a fellow NixOS user, don't really care if "some core devs don't like non-free software". If you don't want your users to use non-free stuff, just simply ban it completely from a distro. And if you allow it, make it first-class citizen.

I don't even think anybody on the Nix core team dislikes non-free software. I know one person did but he left to create his own GNU distribution which uses hurd ... GUIX.

Interesting.

[vyp]

This is off topic because I think this issue is more of a discoverability/documentation/search issue that could be solved without defaulting to unfree = true, but I just wanted to share my opinion on issues regarding nonfree software to everyone arguing to allow it by default.

In my opinion, nonfree software goes against some of the core values of nix, such as reproducibility of packages and the source-based model of nix. (I know many would disagree with that. 🤷‍♀️ ) And really we should all be working towards removing our dependencies on any nonfree software we use, to achieve full reproducibility, thereby making our lives easier, instead of having it by default and probably introducing ourselves to more headaches over the long term. That's an idealistic view, and even if people agree with it, they can't just be expected to create their perfect free alternatives overnight to nonfree software that they need to use today, and it's definitely not easy to achieve long-term either (software's always changing and evolving).

But on the flip side I always see people coming to free software communities and almost shifting the blame onto them for not having good integration with nonfree software. We have to recognize that nonfree software is usually designed to be hard to work with or integrate into 'unsupported' systems such as linux distributions. I think in practice this usually comes down to having to modify a binary rather than having access to source code, but at the very least by definition there's always going to be the legal question of distributing modified nonfree software by default. (And yeah, there are lots of free software packages in nixpkgs where the packages are binary-based and not source-based but I see that as more of a manpower issue really.)

I see people invest years of effort into integration/support with nonfree software only for the vendor to remove something or make a breaking change. Forking a project can be difficult and time consuming, but at least the option is there with free software (especially if there's an active community around the project to help), meaning years of effort aren't in principle in danger of being wasted. So sometimes I wonder if they would have been better off investing their time into a free software project instead. This is all subjective and anecdotal, and also tangential to the original issue, and of course there are many cases where integration with nonfree software is desirable to the free software project/community, but just my observations.

I don't even think anybody on the Nix core team dislikes non-free software. I know one person did but he left to create his own GNU distribution which uses hurd ... GUIX.

The Guix System uses the linux-libre kernel by default. So primarily it's a linux system and its packages usually support linux the best.

[7c6f434c]

I don't even think anybody on the Nix core team dislikes non-free software.

Well, disliking is not a binary, it is a matter of degree. And as already linked in this discussion,
NixOS/nixos-homepage#120 (comment) (to link a statement by a person whose membership in the set of core developers cannot be reasonably disputed)

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[davidak]

there is still no good solution Am 1. Juni 2020 07:25:26 MESZ schrieb "stale[bot]" <[email protected]>:

…

Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly: 1. Search for maintainers and people that previously touched the related code and @ mention them in a comment. 2. Ask on the [NixOS Discourse](https://discourse.nixos.org/). 3. Ask on the [#nixos channel](irc://irc.freenode.net/#nixos) on [irc.freenode.net](https://freenode.net). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #17126 (comment)

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

[Enteee]

I think this is more of a documentation issue rather than a technical issue that can easily be resolved by changing the way how the nix tooling works. So why do we not just add a sub-section to the NixOs doc - Package Management that talks about this?

And maybe include a in the default config generated by nixos-generate-config a few lines like the following:

# If set to true, this option will allow you to
# install proprietary software.
# Please note: If set to false, searching for
# derivations using nix-env -qaP won't yield
# any proprietary results.
nixpkgs.config.allowUnfree = false;

... or similar. What do you think?

[7c6f434c]

I think this is more of a documentation issue rather than a technical issue that can easily be resolved by changing the way how the nix tooling works. So why do we not just add a sub-section to [the NixOs doc - Package Management](https://nixos.org/nixos/manual/index.html#sec-package-management) that talks about this?

… and a link to Nixpkgs manual, because there is already a subsection there.

[stale]

I marked this as stale due to inactivity. → More info

[Diti]

Still important to me.

[davidak]

Status:

I can find zoom-us in:

• nix-env -qa | grep zoom-us
• nix search zoom-us
• https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=zoom-us
• repology (packages.json export)

I think at this point it is clear that we want to show which packages we have (even unfree), but it's just a lack of consistency in our tools, which leads to bad UX and unhappy users.

I just noticed that steam is not in our packages.json, so we are not listed in ValveSoftware/steam-for-linux#7267 (comment). This undermines the marketing efforts i do in social media to advertise that Gaming on NixOS (with Steam) is a viable option.

Please show the same list of packages in all these places!

[fgaz]

It'd be nice to have them shown, but with an easy to notice warning.
For example:

• in search.nixos.org, "Unfree" could have a red background.
  Examples of unfree packages in repos that already do this:
  • https://web.archive.org/web/20221017103706/https://f-droid.org/en/packages/org.mozilla.fennec_fdroid/
  • https://web.archive.org/web/20221017103713/https://content.minetest.net/packages/jp/xdecor/
• In nix search, there could be a red "unfree" text next to the version, like this:

  * legacyPackages.x86_64-linux.zoom-us (5.11.10.4400) (unfree)