Skip to content
This repository was archived by the owner on Dec 24, 2019. It is now read-only.

Bug: codevalidator -f crashes and leaves my file empty when using non-ASCII characters #37

Open
ePaul opened this issue Jul 15, 2015 · 2 comments · May be fixed by #39
Open

Bug: codevalidator -f crashes and leaves my file empty when using non-ASCII characters #37

ePaul opened this issue Jul 15, 2015 · 2 comments · May be fixed by #39

Comments

@ePaul
Copy link

ePaul commented Jul 15, 2015

Summary
When a YAML file contains a character outside of ASCII as well as trailing whitespaces,
codevalidator -f filename will crash with an (not that useful) error message and leave the file empty, at the same time not even creating a backup copy.

Fortunately I did a git add beforehand.

How to reproduce
I'm using codevalidator 0.8.2, judging from pip show codevalidator (codevalidator itself doesn't have a --version option).

Here is an example file (encoded as UTF-8):

definitions:
  purchase_order:
    type: object
    description: |
      An either sparse or complete representation of a purchase order. 
      (TODO: The definition here is not yet complete – e.g. positions
      are missing.)

This has a trailing space in line 5, and an – (en-dash) in line 6. I guess the latter causes codevalidator to crash, the former makes it try to correct at all.

Here is the output for this file:

$ codevalidator -v -f backend/src/main/resources/api/swagger-purchase-order.yaml 
backend/src/main/resources/api/swagger-purchase-order.yaml: contains lines with trailing whitespace
backend/src/main/resources/api/swagger-purchase-order.yaml: Trying to fix notrailingws..
Traceback (most recent call last):
  File "/usr/local/bin/codevalidator", line 9, in <module>
    load_entry_point('codevalidator==0.8.2', 'console_scripts', 'codevalidator')()
  File "/usr/local/lib/python2.7/dist-packages/codevalidator.py", line 953, in main
    fix_files()
  File "/usr/local/lib/python2.7/dist-packages/codevalidator.py", line 878, in fix_files
    fix_file(fname, rules)
  File "/usr/local/lib/python2.7/dist-packages/codevalidator.py", line 866, in fix_file
    fd.write(fixed.encode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 191: ordinal not in range(128)

No .pre-cvfix file is created in this case (but one is created if I replace the – by -).

It looks like it is using 'ascii' as the codec instead of an unicode one.

Expected behaviour
Codevalidator should be able to handle UTF-8 encoded files.
Even if it is not able to fix my file, it should certainly not destroy it (overwriting it with an empty file).
Even if it does that, it should create a backup copy (unless being told not to by --no-backup).

Workaround
Make sure only ASCII characters are used in files passed to codevalidator.
And if you are not sure, make a copy of the file beforehands.
@ePaul
Copy link
Author

ePaul commented Jul 15, 2015

Here is my ~/.codevalidatorrc. I think nothing special is in there.

{
    "exclude_dirs": [".svn", ".git", "xplan", "live-image", "calendar_connector", "archetype"],
    "rules": {
        "*.c": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.coffee": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "coffeelint"],
        "*.conf": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.cpp": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.css": ["utf8", "nobom", "notabs", "nocr"],
        "*.groovy": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.h": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.html": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.htm": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.java": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "jalopy"],
        "*.json": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "json"],
        "*.jsp": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.js": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.less": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.md": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.php": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.phtml": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*pom.xml": ["xml", "pomdesc"],
        "*.pp": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "puppet"],
        "*.properties": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.py": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "pyflakes", "pythontidy"],
        "*.rst": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.scala": ["invalidpath"],
        "* *": ["invalidpath"],
        "*.sh": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.sql_diff": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.sql": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.styl": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.txt": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.vm": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.wsdl": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.xml": ["utf8", "nobom", "notabs", "nocr", "notrailingws"],
        "*.yml": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "yaml"],
        "*.yaml": ["utf8", "nobom", "notabs", "nocr", "notrailingws", "yaml"]
    },
    "options": {"phpcs": {"standard": "PSR", "encoding": "UTF-8"},
                "database_dir": {"pgsql-parser-bin": "/bin/true"},
                "jalopy": {"classpath": "/home/paulo/tools/jalopy/lib/jalopy-1.9.4.jar://home/paulo/tools/jalopy/lib/jh.jar"}},
    "dir_rules": {"db_diffs": ["sql_diff_dir", "sql_diff_sql"], "database": ["database_dir"]}
}

@ePaul
Copy link
Author

ePaul commented Aug 2, 2015

I just tried this again on my home computer with a fresh clone of the repository, and it does create a backup copy in the .pre-cvfix file. I'll have to check again why this didn't happen at my work installation.

ePaul pushed a commit to ePaul/codevalidator that referenced this issue Aug 2, 2015
issue 37: add test file.
1c5b8fc 
This file can be checked with codevalidator (complains about
trailing white space, use of tabs, use of carriage return),
but with the current code it can't be fixed (will be overwritten
with an empty file).

hjacobs#37
@ePaul ePaul linked a pull request Aug 2, 2015 that will close this issue
Open
ePaul pushed a commit to ePaul/codevalidator that referenced this issue Aug 2, 2015
issue 37: add test file.
0138681 
This file can be checked with codevalidator (complains about
trailing white space, use of tabs, use of carriage return),
but with the current code it can't be fixed (will be overwritten
with an empty file).

hjacobs#37
ePaul pushed a commit to ePaul/codevalidator that referenced this issue Aug 5, 2015
issue 37: add test file.
48b8ffd 
This file can be checked with codevalidator (complains about
trailing white space, use of tabs, use of carriage return),
but with the current code it can't be fixed (will be overwritten
with an empty file).

hjacobs#37
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bugfix/issue 37 fix with non ascii ePaul/codevalidator
1 participant
@ePaul