'charmap' codec can't encode character error from AWS CLI

[daivanov]

Describe the bug

I have aws cli:

$ aws --version
aws-cli/2.13.30 Python/3.11.6 Windows/10 exe/AMD64 prompt/off

And this python must be bundled with AWS CLI as my own Python is older:

$ python -V
Python 3.9.16

I defined all possible variables people talk on the internet:

$ env | grep -i UTF
LANGUAGE=en_US.UTF-8
LANG=en_US.UTF-8
PYTHONIOENCODING=UTF-8
LC_CTYPE=en_US.UTF-8
LC_ALL=en_US.UTF-8

When execute:

$ aws servicecatalog describe-product --id prod-6f5mpi6ncyfjk

Expected Behavior

No errors on non-ASCII characters

Current Behavior

I'm getting an error and aws cli halts

'charmap' codec can't encode character '\u0144' in position 25: character maps to

Reproduction Steps

Create products in AWS with non-ASCII characters in the description and try aws servicecatalog describe-product

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.13.30 Python/3.11.6 Windows/10 exe/AMD64 prompt/off

Environment details (OS name and version, etc.)

Windows 10, cygwin 3.4.9-1, bash 5.2.15-3

[daivanov]

Issues #7985 seems to be describing similar problem with character encoding.

Thanks, Daniil

[RyanFitzSimmonsAK]

Hi @daivanov, thanks for reaching out. I was able to reproduce this issue and bypass it by changing my system locale (see this Stack Overflow thread) on Windows 10. Could you try that and let me know if it works for you?

[daivanov]

No, I tried that as well. I'm still getting the error with charmap.

Thanks, Daniil

[daivanov]

It doesn't work even, if I do things like this:

$ cmd /c chcp 65001 && aws servicecatalog describe-product --id prod-6f5mpi6ncyfjk
Active code page: 65001
{
    "ProductViewSummary": {
[...skip...]
'charmap' codec can't encode character '\u0144' in position 25: character maps to <undefined>

Thanks, Daniil

[daivanov]

My local Python has UTF-8 as a default encoding:

$ python -c "import sys; print(sys.getdefaultencoding())"
utf-8

I don't have Python installed in Windows:

$ cmd /c "python -c 'import sys; print(sys.getdefaultencoding())'"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

So this Python 3.11.6

$ aws --version
aws-cli/2.13.30 Python/3.11.6 Windows/10 exe/AMD64 prompt/off

must be bundled with AWS CLI and it has non UTF-8 default encoding.

Thanks, Daniil

[daivanov]

Python in AWS CLI is clearly using CP1252 encoding

2023-11-03 00:56:45,042 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli\clidriver.py", line 460, in main
  File "awscli\clidriver.py", line 595, in __call__
  File "awscli\clidriver.py", line 798, in __call__
  File "awscli\clidriver.py", line 931, in invoke
  File "awscli\clidriver.py", line 953, in _display_response
  File "awscli\formatter.py", line 83, in __call__
  File "awscli\formatter.py", line 102, in _format_response
  File "json\__init__.py", line 180, in dump
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\u0144' in position 25: character maps to <undefined>

Thanks, Daniil

[RyanFitzSimmonsAK]

AWS CLI does use the character encoding provided by the system. After reverting my system locale to reproduce the issue again, the environment variable that did the trick for me was PYTHONUTF8=1. Could you give that a try?

[daivanov]

Hi,

Yes, that worked:

$ env | grep PYTHON
PYTHONIOENCODING=UTF-8
PYTHONUTF8=1
$ aws servicecatalog describe-product --id prod-6f5mpi6ncyfjk | grep kucz
            "Name": "mq-kuczyński-32575bf9c3",

Thank you very much for this hint!

Kind Regards, Daniil

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[ghost]

AWS CLI does use the character encoding provided by the system. After reverting my system locale to reproduce the issue again, the environment variable that did the trick for me was PYTHONUTF8=1. Could you give that a try?

This solved the issue for me