[WIP] Fix UTF-8 handling #76

pyhalov · 2020-03-07T12:16:39Z

No description provided.

This reverts commit d3187bb.

AndWac

The changes look good so far. What makes you sure that you didn't miss an open() call? There are lots of other places where open() is being called...

pyhalov · 2020-03-08T07:45:49Z

# Ran 1132 tests in 18490.569s - skipped 1 tests.
Skipped tests:
cli.t_pkg_help.py TestPkgHelp.test_help_character_encoding: Test Skipped: The test system must have the ja_JP.eucJP locale installed to run this test.


======================================================================
BASELINE MATCH
======================================================================

pyhalov · 2020-03-08T08:30:21Z

@citrus-it What do you think about utf-8 test? The only issue I see is that now you should run test suite with UTF-8 locale

pyhalov · 2020-03-10T17:46:17Z

@citrus-it It's not enough.
Got this exception with patched pkg:

$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/[email protected],5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) \
Traceback (most recent call last):
  File "/usr/bin/pkg", line 5968, in handle_errors
    __ret = func(*args, **kwargs)
  File "/usr/bin/pkg", line 5954, in main_func
    pargs=pargs, **opts)
  File "/usr/bin/pkg", line 2276, in install
    display_plan_cb=display_plan_cb, logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
    logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
    logger=logger, **kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
    display_plan_cb=display_plan_cb)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
    for pd in api_plan_func(**kwargs):
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
    log_op_end_all=True)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
    self._img.make_install_plan(**kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
    reject_list=reject_list)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
    self.__call_imageplan_evaluate(ip)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
    ip.evaluate()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3879, in evaluate
    self.__evaluate_pkg_plans()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4029, in __evaluate_pkg_plans
    ccancel=self.__check_cancel, progtrack=self.__progtrack)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 536, in wrapper
    return f(instance, *fargs, **f_kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1588, in prefetch_manifests
    mfstlist, excludes)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1681, in _prefetch_manifests_list
    dl_path)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1780, in _verify_manifest
    mcontent = mf.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 36861: ordinal not in range(128)

pyhalov · 2020-03-10T19:11:11Z

It's still not complete, now it dies without exception...

$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/[email protected],5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) |
pkg: 'ascii' codec can't encode character '\u0151' in position 36861: ordinal not in range(128)

pyhalov · 2020-03-10T19:56:36Z

The same issue - dies on writing to /var/pkg/cache/publisher/userland/pkg/crypto%2Fca-certificates/1.0%2C5.11-2019.0.0.7%3A20200310T174048Z/manifest.file.uosjwiqq

pyhalov · 2020-03-10T20:16:12Z

Now more issues:

Creating Plan (Checking for conflicting actions): -
Traceback (most recent call last):
  File "/usr/bin/pkg", line 5968, in handle_errors
    __ret = func(*args, **kwargs)
  File "/usr/bin/pkg", line 5954, in main_func
    pargs=pargs, **opts)
  File "/usr/bin/pkg", line 2276, in install
    display_plan_cb=display_plan_cb, logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
    logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
    logger=logger, **kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
    display_plan_cb=display_plan_cb)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
    for pd in api_plan_func(**kwargs):
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
    log_op_end_all=True)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
    self._img.make_install_plan(**kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
    reject_list=reject_list)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
    self.__call_imageplan_evaluate(ip)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
    ip.evaluate()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3880, in evaluate
    self.__merge_actions()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4747, in __merge_actions
    self.__find_all_conflicts()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3371, in __find_all_conflicts
    offset_dict = self.image._load_actdict(self.__progtrack)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 3881, in _load_actdict
    for line in of:
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5250: ordinal not in range(128)

pyhalov · 2020-03-10T20:28:12Z

And now we have to fix open() all over src/modules/client/image.py...

pyhalov · 2020-03-11T05:46:06Z

Now we can get another error:

Removing old actions                             1/1
Action install failed for 'etc/certs/CA/NetLock_Arany_(Class_Gold)_F\u0151tan\xfas\xedtv\xe1ny.pem' (pkg://userland/crypto/ca-certificates):
  UnicodeEncodeError: 'ascii' codec can't encode character '\u0151' in position 42: ordinal not in range(128)

Toasterson · 2021-11-03T14:52:52Z

@pyhalov With #93 could we merge this or are we still stuck on one case where it still calls open without encoding?

pyhalov · 2021-11-03T15:13:16Z

Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?

Toasterson · 2021-11-03T15:16:05Z

Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?

No we needed to remove C locale and switch to C.UTF-8 one should not call IPS with just the C locale. We could make a patch to the python source to fix the if case that checks if the string contains .UTF-8 in locale and simply tell it to assume UTF-8 aswell in those cases.

pyhalov and others added 2 commits March 7, 2020 15:14

Revert "Fix decoding file text data as ASCII"

f006b53

This reverts commit d3187bb.

Fix errors resulting from UTF-8 characters in manifest files

72fa78d

pyhalov added the testing needed label Mar 7, 2020

AndWac reviewed Mar 7, 2020

View reviewed changes

Test pkg_verify with utf-8 pkg data

900da3c

pyhalov changed the title ~~Fix UTF-8 handling~~ [WIP] Fix UTF-8 handling Mar 10, 2020

pyhalov added the needs_work label Mar 10, 2020

pyhalov force-pushed the utf-8 branch from d149651 to 0689757 Compare March 10, 2020 18:29

pyhalov force-pushed the utf-8 branch from 0689757 to 12dd412 Compare March 10, 2020 19:36

pyhalov force-pushed the utf-8 branch from 12dd412 to 94a7041 Compare March 10, 2020 20:03

More fixes for UTF-8 characters in manifests files

a2b4a1f

pyhalov force-pushed the utf-8 branch from 94a7041 to a2b4a1f Compare March 11, 2020 05:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Fix UTF-8 handling #76

[WIP] Fix UTF-8 handling #76

pyhalov commented Mar 7, 2020

AndWac left a comment

pyhalov commented Mar 8, 2020

pyhalov commented Mar 8, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 11, 2020

Toasterson commented Nov 3, 2021

pyhalov commented Nov 3, 2021

Toasterson commented Nov 3, 2021

[WIP] Fix UTF-8 handling #76

Are you sure you want to change the base?

[WIP] Fix UTF-8 handling #76

Conversation

pyhalov commented Mar 7, 2020

AndWac left a comment

Choose a reason for hiding this comment

pyhalov commented Mar 8, 2020

pyhalov commented Mar 8, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 10, 2020

pyhalov commented Mar 11, 2020

Toasterson commented Nov 3, 2021

pyhalov commented Nov 3, 2021

Toasterson commented Nov 3, 2021