Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode support for neuron CLI #381

Open
cipharius opened this issue Sep 9, 2020 · 14 comments
Open

Unicode support for neuron CLI #381

cipharius opened this issue Sep 9, 2020 · 14 comments
Labels
awaiting-pr help wanted Extra attention is needed

Comments

@cipharius
Copy link

Neuron currently does not accept unicode characters in it's CLI.

For example, neuron new Piemērs results in a note with title Piem��rs. Same applies to other commands, such as neuron query -t piemērs.

This problem only exists when using neuron CLI. If notes have unicode characters, they get correctly rendered in the static website.

@srid
Copy link
Owner

srid commented Sep 9, 2020

Re-occurence of #273 ?

@srid
Copy link
Owner

srid commented Sep 9, 2020

Probably something to do with your terminal encoding? If so, perhaps we can document the environment fix in neuron guide.

@cipharius
Copy link
Author

cipharius commented Sep 9, 2020

Locale seems to be correct and other CLI tools seem to pick up on the unicode characters.
I'm using kitty as terminal emulator.

screenshot

Edit: Right, it seems that #273 describes the same issue.

@cipharius
Copy link
Author

Just discovered that I don't have this problem when running ./bin/run query -t piemērs in neuron source directory.
I updated my binary with nix-env -if https://github.com/srid/neuron/archive/master.tar.gz and even on newest version I still have the problem with unicode characters.

@cipharius
Copy link
Author

cipharius commented Sep 9, 2020

Alright, I figured it out, problem is with LOCALE_ARCHIVE environment variable.

Interactive sessions work because nix-shell environment has LOCALE_ARCHIVE pointing to locale archive in nix store. If I define this environment variable outside of nix-shell, neuron CLI works correctly.

This probably means that my locale-archive is not correctly generated, or that binary build with nix glibc locale archive is not compatible with my locale archive. I'm not sure, I don't understand the role of the locale-archive.

@cipharius
Copy link
Author

Just discovered that setting LANG to C.UTF-8 fixes this problem without changing archive variable.

Again, don't know enough about locale mechanism to tell if this is a viable solution, or just a temporary one.

Ideally I wouldn't want to change my system-wide locale just for one application.

@srid
Copy link
Owner

srid commented Sep 9, 2020

@maralorn Is this something that can be addressed in Nix?

@cipharius
Copy link
Author

Researching for some more shows that Arch Linux does not support C.UTF-8 locale.
Also, found out that other projects have faced this issue: electron-userland/electron-builder#2340

@srid
Copy link
Owner

srid commented Sep 9, 2020

Perhaps we need to add glibcLocales as a dependency? nh2/nixpkgs@9cfb480

@fiatjaf
Copy link
Contributor

fiatjaf commented Sep 9, 2020

I'm facing this issue. I'm on Archlinux. Setting LANG to C.UTF-8fixed it.
What should I do?

@srid
Copy link
Owner

srid commented Sep 13, 2020

Is this even a neuron issue?

@fiatjaf
Copy link
Contributor

fiatjaf commented Sep 13, 2020

I have no idea. It only occurs in Neuron. Maybe it is a nix issue?

@srid srid added help wanted Extra attention is needed hacktoberfest labels Oct 7, 2020
@TheKK
Copy link
Contributor

TheKK commented Oct 19, 2020

Here's my observations (on Ubuntu 16.04.7)

  • When running neuron query, it tried to read locale from glibc, /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/locale/locale-archive. This could be observed via strace
  • Running localedef --list-archive /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/locale/locale-archive tells only C.utf8

AFAIK, there's no way to make /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/locale/locale-archive to contain more locale data, that would violate the principle of Nix. So I believe the only way to fix this issue is directing neuron to desired locale-archive.

According to nix dev mailing list and setup hook of glibcLocales, we could set LOCALE_ARCHIVE environment variable to achieve what we want.

And it does work!

Running LOCALE_ARCHIVE=/usr/lib/locale/locale-archive neuron query reports the correct response (that "一些中文"):

{
  "query": [
    "ZettelQuery_ZettelsByTag",
    [
      [],
      "OrdinaryConnection",
      {
        "zettelsViewGroupByTag": false,
        "zettelsViewLinkView": "LinkView_Default"
      }
    ]
  ],
  "result": [
    {
      "zettelContent": [],
      "zettelDate": [
        "2020-10-07",
        "12:45:00"
      ],
      "zettelError": {
        "Right": []
      },
      "zettelFormat": "markdown",
      "zettelID": "71462dd7",
      "zettelPath": "71462dd7.md",
      "zettelQueries": [],
      "zettelTags": [],
      "zettelTitle": "一些中文",
      "zettelTitleInBody": true,
      "zettelUnlisted": false
    }
  ],
  "skipped": {}
}

So I think there's nothing could be improved on the neuron side, this kinds of setting could only happens in the user's running environment. Maybe there's more convenience way on NixOS though, consider there's a setupHook for glibcLocales.

@srid
Copy link
Owner

srid commented Jul 22, 2021

So I think there's nothing could be improved on the neuron side, this kinds of setting could only happens in the user's running environment.

I think we can automate this by creating a wrapper neuron script that uses this environment variable. It can be done in flake.nix (under defaultApp) so that users don't have to do anything special after installing neuron via nix.

For the corresponding docker change, see cbdaeee

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting-pr help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

4 participants