Always read metadata files as UTF-8 in setup.py

[EliahKagan]

Fixes #1747

This passes encoding="utf-8" to the open calls in setup.py, so the readme, version, and requirements files are always read as UTF-8, even on systems whose locale is not UTF-8. This fixes the bug described in #1747 where installation other than from a pre-built wheel would fail on many Windows systems using non-European languages.

The specific problem occurred with the README.md file. The requirements files are less likely to contain characters not in the ASCII subset, though maybe they could come to contain them, perhaps in comments. The VERSION file is even less likely to ever contain such characters. Nonetheless, for consistency, because it is a best practice, and because it appears to be the intent of the existing code, encoding="utf=8" is added for opening all of them.

This change is tested on a system whose locale uses Windows code page 936. Editable installation, as well as the other affected ways of installing (and building) described in #1747, are now working. (Installing from a pre-built wheel was never affected.)

I lean toward the position that it's not worth attempting a regression test for this, because although we do have test_installation, a truly reliable regression test for #1747 would require the presence of a locale in which the existing logic breaks, but in practice the systems where this happens are Windows, where a single locale (in the sense of "locale" relevant here) is systemwide. Furthermore, the fix is straightforward, following the current best practice of passing encoding="utf-8" explicitly where there is not reason to do otherwise. However, there may be a way to add a reasonably reliable test using mocking or other techniques, and I'd be pleased to pursue that if you consider it warranted.

[Byron]

Thanks so much for tackling this!

The handling of encodings always have been a weak spot for the project (and I think, are weak spots for python in general), but this small change is an important win for sure.

I wouldn't want to see your time wasted spent on trying to test this though, as I think it's reasonable to assume encoding related issues won't occur with that as the encoding of these files is under control of the repository.

Something I find interesting though if it would present an attack-vector. Could some installation-paths be made to fail by silently changing the encoding of README.md? I'd assume such an act shows up clearly in a diff, so I'd think the answer is "no" here. And with that, it should be fine to keep it as is without further regression-protection.