Skip to content

fix(core): pin PowerShell output encoding to UTF-8 on Windows - #58968

Open
fake-monkey wants to merge 2 commits into
anomalyco:devfrom
fake-monkey:powershell-utf8
Open

fake-monkey wants to merge 2 commits into
anomalyco:devfrom
fake-monkey:powershell-utf8

Conversation

@fake-monkey

@fake-monkey fake-monkey commented Oct 25, 2026 •

Copy link
Copy Markdown

Issue for this PR

Closes #56292

Type of change

  • Bug fix

What does this PR do?

On Windows, shell decodes command output as UTF-8, but PowerShell encodes stdout with the console code page. Commands are spawned with windowsHide, so there is no console and PowerShell falls back to the system ANSI code page (936 on zh-CN Windows). Every non-ASCII character comes back as U+FFFD.

This PR pins the encoding before the command runs:

[Console]::OutputEncoding = [Text.UTF8Encoding]::new($false); $OutputEncoding = [Console]::OutputEncoding;

Assigning OutputEncoding also sets the console code page, so native children (git, cmd /c ...) emit UTF-8 too. chcp alone is not enough here because PowerShell's own writer keeps the code page it started with.

using namespace / using module / using assembly must be the first statement in a script, so commands that start with one skip the prefix.

For the PTY path the same setup is passed as -NoExit -Command when the caller does not provide explicit args.

How did you verify your code works?

Windows 11 25H2 (build 26200), system ANSI/OEM code page 936. The harness has the same shape as the shell tool: ShellSelect.args(...) + spawnSync(..., { windowsHide: true }).

shell Write-Output 中文 / Write-Host 中文 / cmd /c echo 中文
Windows PowerShell 5.1.26100 UTF-8 ✅
pwsh 7.5.4 UTF-8 ✅

PTY: ConPTY through @lydell/node-pty with -NoExit -Command <prefix>; both versions render Write-Output PTY-中文 correctly.

Runs on the final version of the prefix (no chcp):

  • bun test test/tool-shell.test.ts — 46 pass, 63 skip, 0 fail
  • bun test test/shell.test.ts -t "builds command args" — pass
  • bun typecheck in packages/core — clean

Not verified on pwsh 7.6.x. #56292 reports that the in-session assignment is ignored there; if that is confirmed, the fallback is to set the code page before PowerShell starts, which needs a launcher. This PR keeps the smaller change until that is confirmed.

Screenshots / recordings

N/A

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Added by the opencode bot (AI-generated).

Related: #56292

@github-actions github-actions Bot added needs:compliance This means the issue will auto-close after 2 hours. needs:issue labels Oct 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't verify this. The bug only happens on Windows, and this sandbox is Linux with no PowerShell installed. I ran the nearby tests and type-checked packages/core with tsc. The public API is unchanged.

Main concerns:

  1. The approach is disputed. Two people on #56292 (pwsh 7.6.7 on es-* Windows, and Windows 10 zh-CN with CP936) say that assigning [Console]::OutputEncoding inside the session is ignored or comes too late when the process is spawned like this. Only setting the code page before PowerShell starts worked for them. Your notes say it works on 5.1 and 7.5. Could you post a capture from pwsh 7.6.x run through the shell tool (not an interactive console)? Please include Write-Output 中文 and a native child such as cmd /c echo 中文.
  2. The new shell.test.ts assertion fails on Linux and macOS (see inline comment).
  3. Commands that start with using break with the setup line in front of them (see inline comment).

The new tool-shell test can't catch this bug outside Windows, because pwsh already writes UTF-8 there. It passes with or without the fix, so it can be dropped unless it runs on Windows CI.

Related: #56292

Comment thread packages/core/test/shell.test.ts Outdated
"Write-Output hi",
process.platform === "win32" ? `${ShellSelect.PS_UTF8} Write-Output hi` : "Write-Output hi",
])
expect(ShellSelect.args("powershell.exe", "Write-Output hi")).toEqual([

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails on Linux and macOS. ShellSelect.name keeps the .exe off Windows, so args("powershell.exe", ...) returns ["-c", "Write-Output hi"] there. I ran bun test test/shell.test.ts and got shell > builds command args per shell family failing. Either drop this assertion or limit it to win32.

Comment thread packages/core/src/shell/select.ts Outdated
"-NoProfile",
"-NonInteractive",
"-Command",
process.platform === "win32" ? `${PS_UTF8} ${command}` : command,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting statements in front of the user's command breaks commands that must start the script. using namespace System.Text; [StringBuilder]::new() (and using module/using assembly) becomes a parse error once [Console]::OutputEncoding = ...; comes first. The same problem came up on #54511. Skipping the setup when the command starts with using would avoid it.

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Oct 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

@fake-monkey

Copy link
Copy Markdown
Author

Updated:

  • Dropped the powershell.exe assertion that failed outside Windows.
  • Commands starting with using namespace / using module / using assembly now skip the prefix, since using must be the first statement.
  • Dropped the extra tool-shell test. It only proves anything on a zh-CN Windows host, which CI does not have.

On pwsh 7.6.x: not verified locally. #56292 says the in-session assignment is ignored there; if that is confirmed, the fallback is to set the code page before PowerShell starts.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both earlier points are fixed. The assertion that failed on Linux is gone, and commands that start with using namespace|module|assembly (after blank or comment lines) now skip the setup line. The test that couldn't catch the bug outside Windows was removed. At this commit, test/shell.test.ts and test/tool-shell.test.ts pass on Linux (73 pass, 42 skip) and tsc is clean.

I still couldn't verify the fix, because it only applies on Windows. Whether the in-session assignment works on pwsh 7.6.x (the version in #56292) needs a check on a Windows machine before merging. The public API is unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shell: PowerShell output shows U+FFFD for every non-ASCII character on Windows

1 participant