fix(tools): use consistent configurable file encodings 🤖🤖🤖 - #298
Draft
Sampoornnagpal wants to merge 1 commit into
Draft
fix(tools): use consistent configurable file encodings 🤖🤖🤖#298Sampoornnagpal wants to merge 1 commit into
Sampoornnagpal wants to merge 1 commit into
Conversation
Signed-off-by: Sampoorn Nagpal <nagpalsampoorn@gmail.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
ShellTools currently uses the platform default encoding for file reads, writes, replacements, and search anchors. Under a cp1252 default, writing emoji or Chinese text raises
UnicodeEncodeError; reading an existing UTF-8 file produces mojibake, including the text attached to editable search matches.Use a consistent, configurable file encoding across these operations. The proposed default is UTF-8. Callers handling legacy files can choose a codec such as
encoding="cp1252", orencoding=Noneto retain the previous platform-default behavior. Shell command-output decoding is unchanged.Draft / coordination: this changes the default file-encoding policy and overlaps the bounded file-I/O refactor in #94. That refactor explicitly preserves locale-default encoding. Please agree on the encoding policy and implementation ordering before marking this ready; I can adapt this change to the bounded helpers if #94 lands first. This is not a replacement for #94's bounded/atomic I/O work.
Validation
Three regressions fail against the original implementation with a simulated cp1252 default: Unicode writes, reading existing UTF-8 text, and harvesting Unicode search anchors. Two further cases verify explicit cp1252 and
Nonecompatibility. Coverage exercises both path-based and Match-based edits, including an edit from a harvested anchor. The search session is mocked; no Bash process is required.uv run pytest -q tests/tools/test_shell_file_encoding.py tests/tools/test_shell_tools_modern_behavior.py -k 'not run_persists and not run_reports and not cwd_change and not close_terminates and not timeout_flag_preserves'— 13 passed, 5 deselected.The deselections are four Bash-dependent cases and one existing POSIX-absolute-path fixture incompatible with Windows. Validation used Windows/Python 3.12 with external import-only helpers for #84/#85 (
fcntl/SIGUSR2); those helpers are not included and do not validate POSIX locking/signals. Real grep/Bash integration and a full Linux suite have not been run here.Ruff lint and formatting pass. Targeted Pyright reports one diagnostic in the unchanged shell-output callback (
strversusLiteral["stdout", "stderr"]); this PR does not modify that callback.Related issues
Related overlapping implementation: #94. The skill-body UTF-8 fix in #281 is separate and does not affect ShellTools.
Checklist