feat: Added BOM capability for output files (#1267)

[alvaro-osvaldo-tm]

closes #1267

• Added the '--add-bom' parameter for almost utilities

[alvaro-osvaldo-tm]

Implementation

Implemented the feature to optionality add UTF-8 Byte Order Mark (BOM) into output content in all utilities,
except csvpy and sql2csv

Solution

• The UTF-8 BOM only will be added if the parameter '--add-bom' is specified, otherwise is ignored.
• The parameter configuration and execution was implemented in the file csvkit/features/AddBOM.py ,
  I used a 'feature' pattern to avoid 'spaghetti code', no problem if the code need to be put into CSVKitUtility class.
  • The advantage of this approach is the code is more clear.
  • But the CSVToolKit is not prepared for it as seen in 'argument' method. Also, a few more CPU cycles will be perceived, if the user process a HUGE amount of files.

Tests

• A attached a end-to-end test script

  • test-1267.txt

• No unit test was made because the tests use 'StringIO' as 'input file', but the BOM need to be added as bytes using 'TextIOWrapper'.

  • If you want, I can implement a conversion in 'CSVToolkit' and 'LazyFile' to enable the tests.

• All PyTests and end-to-end tests passed in the following versions:

  • Python 3.8.20
  • Python 3.9.21
  • Python 3.10.16
  • Python 3.11.11
  • Python 3.12.8
    • Except csvgrep and csvcut due CSVToolkitbug.

Checklist

• Unit Testing
• End-to-end Testing

Considerations

• I found some bugs related to CSVToolkit that affect this implementation, I reported each in csvgrep: Catastrophic failure in Python 3.12 #1275 , csvcut: Catastrophic failure in Python 3.12 #1276, and Ignoring PythonIOEnconing #1277

• The CSVToolkit architecture always output in UTF-8, even if PYTHONIOENCONDING environment variable is set,
  therefore, only UTF-8 BOM is supported.

References

• Feature Pattern.https://martinfowler.com/articles/feature-toggles.html

[alvaro-osvaldo-tm]

Done, all code refactored to current code organisation and passed in 'end-to-end' and 'pytest' tests , except for the Python 3.12 as specified in the opened issues.

Let me know if need something more.

[coveralls]

coverage: 90.435% (+0.03%) from 90.408%
when pulling 03bde26 on alvaro-osvaldo-tm:proposals/1267
into bd6095d on wireservice:master.