feat(string): add UTF-8 string conversion and validation functions by bobtista · Pull Request #2528 · TheSuperHackers/GeneralsGameCode

bobtista · 2026-04-03T00:07:30Z

Relates to refactor(string): Add functions for handling UTF8 encoded strings #2045

Adds portable UTF-8/wide-character conversion helpers to WWLib and uses them for engine string translation and the existing GameSpy conversion paths.

This picks up the work from #2045 with API and behavior changes based on review feedback.

`WWLib/utf8.h` / `utf8.cpp`

Adds four conversion functions:

Wide_To_Utf8_Len
Utf8_To_Wide_Len
Wide_To_Utf8
Utf8_To_Wide

The _Len functions return the required destination size without counting a null terminator. The conversion functions return the number of units written and write a null terminator when the destination has spare capacity.

Malformed UTF-8 is reported as UTF8_INVALID, keeping it distinct from a successful empty conversion, which returns 0.

The decoder validates UTF-8 according to RFC 3629, rejecting:

malformed and truncated sequences;
overlong encodings;
surrogate codepoints;
codepoints above U+10FFFF.

The encoder substitutes U+FFFD for unpaired UTF-16 surrogates and wide values that have no valid UTF-8 representation. Its output therefore always decodes successfully through Utf8_To_Wide.

The implementation uses no platform text APIs. It adapts to the platform width of wchar_t:

16-bit wchar_t is handled as UTF-16, including surrogate pairs;
32-bit wchar_t is handled as UTF-32 codepoints.

`AsciiString::translate`

Replaces the original 7-bit-only wide-to-narrow translation with UTF-8 encoding.

`UnicodeString::translate`

Decodes valid UTF-8 into the platform wide-character representation.

When the input is not valid UTF-8, translation falls back to the previous unsigned-byte-to-wide-character behavior. This preserves legacy byte values instead of rejecting or replacing the input. It is a compatibility fallback, not a complete code-page-specific decoder.

Because AsciiString does not carry encoding metadata, valid UTF-8 is preferred over the legacy fallback.

`ThreadUtils.cpp`

Replaces the GameSpy-specific Win32 conversion calls with the shared WWLib functions.

MultiByteToWideCharSingleLine uses the same invalid-UTF-8 fallback as UnicodeString::translate. Newline and carriage-return replacement remains unchanged.

Non-goals

This change does not make narrow filesystem APIs Unicode-aware. Paths are still passed to the existing narrow platform APIs; updating those APIs belongs in a separate change.

Verification

Tested ASCII and 2-, 3-, and 4-byte UTF-8 round trips.
Tested both 16-bit and 32-bit wchar_t paths.
Tested rejection of malformed, truncated, overlong, surrogate, and out-of-range UTF-8 sequences.
Tested replacement of unpaired wide surrogates and out-of-range wide values.
All Generals and GeneralsMD VC6 and Win32 CI configurations pass.
GeneralsMD replay checks pass.

greptile-apps · 2026-04-03T00:12:52Z

Greptile Summary

Adds a portable UTF-8/wide-character transcoder and integrates it into engine string and GameSpy conversion paths.

Validates UTF-8 by rejecting malformed, overlong, surrogate, and out-of-range encodings.
Replaces the previous ASCII-only AsciiString and UnicodeString translations with UTF-8 conversion and a legacy byte-cast fallback.
Replaces GameSpy’s Win32 conversion calls with shared WWLib helpers.
Registers the new implementation in the WWLib build.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
Core/Libraries/Source/WWVegas/WWLib/utf8.cpp	Implements width-aware UTF-8 transcoding with checks for malformed, overlong, surrogate, and out-of-range sequences.
Core/Libraries/Source/WWVegas/WWLib/utf8.h	Defines the portable conversion API, invalid-input sentinel, sizing contracts, and destination-buffer behavior.
Core/GameEngine/Source/Common/System/AsciiString.cpp	Replaces lossy wide-to-byte casting with correctly sized UTF-8 encoding.
Core/GameEngine/Source/Common/System/UnicodeString.cpp	Adds validated UTF-8 decoding while preserving legacy non-UTF-8 data through byte-wise fallback.
Core/GameEngine/Source/GameNetwork/GameSpy/Thread/ThreadUtils.cpp	Migrates GameSpy string conversion from Win32 APIs to WWLib and handles invalid UTF-8 through the documented fallback.
Core/Libraries/Source/WWVegas/WWLib/CMakeLists.txt	Adds the portable UTF-8 source and header to the WWLib target.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Engine or GameSpy text] --> B{Conversion direction}
    B -->|UTF-8 to wide| C[Utf8_To_Wide_Len]
    C --> D{Valid UTF-8?}
    D -->|Yes| E[Utf8_To_Wide]
    D -->|No| F[Legacy one-to-one byte fallback]
    B -->|Wide to UTF-8| G[Wide_To_Utf8_Len]
    G --> H[Wide_To_Utf8]
    E --> I[UnicodeString or std::wstring]
    F --> I
    H --> J[AsciiString or std::string]

_{Reviews (17): Last reviewed commit: "build: Use qualified include path for WW..." | Re-trigger Greptile}

bobtista · 2026-04-03T02:32:50Z

Fixed the if formatting
added RFC 3629 overlong and out-of-range checks
RE the theoretical memory leak, can that even happen here? set() allocates via the engine's custom memory allocator which crashes on failure rather than throwing, so the leak path can't really be reached right?

xezon

Get_Utf8_Size should not include the null terminator in its size.

xezon · 2026-04-05T12:17:27Z

 	if (dest_size == 0)
-		return;
+		return false;
 	int result = MultiByteToWideChar(CP_UTF8, 0, src, -1, dest, (int)dest_size);


What happens if dest_size does not have enough room for a null terminator?

The doc says "Does not write a null terminator" - should we add more comments? Change the functions to always null-terminate? What do you want here?

Maybe make it behave like strncpy? Writes null if there is room, otherwise not.

The only issue with the current interface then is that we will not know if it wrote the null terminator. Maybe it should return size_t instead, returning the number of characters it writes or would like to write? MultiByteToWideChar also does that.

I suggest to think this through and design the function interface in a way that it can be conveniently be used for fixed size strings (std::string, AsciiString) and large throwaway buffers (char arr[512]).

The behavior definitely needs to be documented.

Done. It writes the terminator when there is room, otherwise not, and returns the units written. Bad input returns UTF8_INVALID, so it is distinct from an empty result.

xezon · 2026-04-06T19:57:25Z

The diff now shows unrelated changes.

bobtista · 2026-04-06T20:04:14Z

The diff now shows unrelated changes.

Try again cleaned up the commits and force pushed

Mauller · 2026-04-07T07:35:06Z

+	}
+	ensureUniqueBufferOfSize((Int)size + 1, false, nullptr, nullptr);
+	char* buf = peek();
+	if (!Unicode_To_Utf8(buf, src, srcLen, size))


So is this translating UTF16LE from windows into UTF8 that is then stored within AsciiString?

If so this may help with the paths issue with usernames and paths not using Latin characters, but file handling functions will need updating to use unicode variants instead of Ascii.

Yes. It does not fix path handling on its own though, the file APIs still take the narrow string. Should that be a separate change?

Mauller · 2026-04-07T07:47:21Z

I wonder if we should also add a flag to state that the Ascii string is holding a UTF8 string?

I guess all normal ascii characters will display properly, it's just extended character sets that will look garbled.

…ming, null-terminate when room

OmniBlade · 2026-04-08T08:56:40Z

Slight nitpick on function naming, shouldn't it be Utf16 rather than Wchar as Wchar is only Utf16 on windows yet we will likely want the conversion functions on other platforms as well for at least csf parsing if nothing else.

bobtista · 2026-04-08T14:20:32Z

Slight nitpick on function naming, shouldn't it be Utf16 rather than Wchar as Wchar is only Utf16 on windows yet we will likely want the conversion functions on other platforms as well for at least csf parsing if nothing else.

Yeah, but it's consistent with the other naming as it is for now. How about we keep the naming and using a uint16_t/char16_t type internally rather than wchar_t when we make non windows paths? Or would you rather we rename to something like Utf16_To_Utf8?

Mauller · 2026-04-13T09:33:52Z

Slight nitpick on function naming, shouldn't it be Utf16 rather than Wchar as Wchar is only Utf16 on windows yet we will likely want the conversion functions on other platforms as well for at least csf parsing if nothing else.

Yeah, but it's consistent with the other naming as it is for now. How about we keep the naming and using a uint16_t/char16_t type internally rather than wchar_t when we make non windows paths? Or would you rather we rename to something like Utf16_To_Utf8?

If anything it would be Utf16Le_To_Utf8, windows uses the little endian utf16 format. Not sure if Utf16Be is used much anywhere but worth being concise with it.

… truncation

… <= 0 checks

githubawn · 2026-04-21T23:27:36Z

Could the unconditional replacement of translate() silently break callers passing legacy CP1252 data, causing strings that are valid CP1252 but invalid UTF-8 to corrupt or clear?

Maybe something like:
Check if the source is valid UTF-8 via Utf8_Validate.
If valid: Proceed with UTF-8 conversion.
If invalid: Fall back to the legacy 1:1 byte-to-wide cast (treating it as CP1252).

xezon · 2026-04-23T20:54:00Z

Maybe put a breakpoint and check usage patterns.

xezon · 2026-05-30T10:46:43Z

What is the current state of this? Can it be finalized?

…ure handling

bobtista · 2026-06-29T19:06:18Z

What is the current state of this? Can it be finalized?

Yeah, I just pushed cleanups addressing the open threads, it should be ready for another pass

…ator change

…String::translate

bobtista · 2026-07-19T15:15:25Z

Pushed updates addressing the open comments:

Portable transcoder, dropped the Win32 WideCharToMultiByte/MultiByteToWideChar wrappers and the #else #error for a hand-rolled RFC 3629 encode/decode in utf8.cpp. It compiles on all platforms now, and the wide side adapts to wchar_t width at compile time (UTF-16 with surrogate pairs on Windows, UTF-32 elsewhere), so it's reusable for CSF parsing off-Windows. @OmniBlade — I kept the Utf16Le_* naming for now per the earlier thread; happy to rename if you'd prefer.
CP1252 fallback, UnicodeString::translate now validates first. Valid UTF-8 decodes as UTF-8; anything else (legacy CP1252) falls back to the old 1:1 byte cast instead of producing replacement characters. Bonus: data written by the old lossy path reads back byte-identical.
Scope, reverted the ensureUniqueBufferOfSize null-terminator change, so the string files now carry only the translate rewrite.

Tested the codec in isolation: ASCII / 2- / 3- / 4-byte round-trips on both wide widths, plus RFC 3629 rejection of overlong, surrogate, out-of-range, and truncated sequences.

Ready for another pass

…tion

bobtista · 2026-07-28T14:28:42Z

Reworked the conversion interface:

Utf8_To_Wide_Len / Utf8_To_Wide return UTF8_INVALID on bad input. 0 now means empty only, so callers can tell the two apart.
The encoder substitutes U+FFFD for unpaired surrogates and out of range wide values. Before this, a lone surrogate encoded to ED A0 80, which the decoder then rejected.
Renamed to Wide_To_Utf8 / Utf8_To_Wide. The wide side is wchar_t, which is UTF-32 off Windows, so the Utf16Le name was wrong.
MultiByteToWideCharSingleLine falls back to a byte cast on non-UTF-8, matching UnicodeString::translate. It returned an empty string before.
Dropped Utf8_Validate, Utf8_To_Wide_Len covers it and nothing else called it.

…oWideCharSingleLine

greptile-apps Bot reviewed Apr 3, 2026

View reviewed changes

Comment thread Core/Libraries/Source/WWVegas/WWLib/utf8.cpp Outdated

xezon reviewed Apr 3, 2026

View reviewed changes

xezon added Enhancement Is new feature or request Minor Severity: Minor < Major < Critical < Blocker labels Apr 3, 2026

greptile-apps Bot reviewed Apr 3, 2026

View reviewed changes

Comment thread Core/GameEngine/Source/GameNetwork/GameInfo.cpp Outdated

xezon reviewed Apr 4, 2026

View reviewed changes

xezon reviewed Apr 5, 2026

View reviewed changes

xezon reviewed Apr 6, 2026

View reviewed changes

feat(utf8): add UTF-8 string conversion and validation functions

40393b8

bobtista force-pushed the bobtista/feat/utf8-string-functions branch from 39d7229 to 40393b8 Compare April 6, 2026 20:02

Mauller reviewed Apr 7, 2026

View reviewed changes

bobtista added 2 commits April 7, 2026 14:37

refactor(utf8): Return size_t from conversions, use consistent len na…

abb71f0

…ming, null-terminate when room

refactor(utf8): Update callers to use new conversion API

0c9074d

greptile-apps Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread Core/Libraries/Source/WWVegas/WWLib/CMakeLists.txt

refactor(utf8): rename to Utf16Le_To_Utf8 and return required size on…

149a07f

… truncation

xezon reviewed Apr 17, 2026

View reviewed changes

refactor(utf8): add writeDirect mode, use _Len helpers, const locals,…

6097799

… <= 0 checks

greptile-apps Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread Core/Libraries/Source/WWVegas/WWLib/utf8.cpp Outdated

xezon reviewed Apr 18, 2026

View reviewed changes

Comment thread Core/Libraries/Source/WWVegas/WWLib/utf8.cpp Outdated

refactor(utf8): simplify conversion API and reject UTF-16 surrogates

9078de5

greptile-apps Bot reviewed Apr 18, 2026

View reviewed changes

Comment thread Core/GameEngine/Source/GameNetwork/GameSpy/Thread/ThreadUtils.cpp Outdated

xezon reviewed Apr 18, 2026

View reviewed changes

bobtista added 2 commits April 18, 2026 14:32

style(utf8): assert after write, add braces, const locals

327bb4b

style(utf8): Use >= 0 in length return ternaries

4d5d2dc

githubawn mentioned this pull request Apr 21, 2026

feat(input): Implement SDL3 input and window management #2639

Open

xezon mentioned this pull request Jun 15, 2026

refactor(string): Add functions for handling UTF8 encoded strings #2045

Closed

bobtista added 3 commits June 29, 2026 14:59

refactor(utf8): remove unused validators and simplify conversion fail…

f582ac8

…ure handling

style(string): add braces to translate conversion check

40683de

fix(string): return empty string when ThreadUtils conversion fails

44f8fca

bobtista mentioned this pull request Jul 14, 2026

Camera position starts at the wrong spot when player has a non ASCII character in their display name #2874

Open

4 tasks

bobtista added 3 commits July 19, 2026 18:50

refactor(string): drop unrelated ensureUniqueBufferOfSize null-termin…

fc3add5

…ator change

refactor(utf8): replace Win32 wrappers with portable RFC 3629 transcoder

ce8a2f4

fix(string): fall back to 1:1 byte cast for non-UTF-8 data in Unicode…

d4a7e73

…String::translate

bobtista added 2 commits July 28, 2026 15:27

fix(utf8): Substitute U+FFFD for wide values with no UTF-8 representa…

e0a425e

…tion

refactor(utf8): Return UTF8_INVALID instead of 0 for malformed UTF-8

159da01

bobtista added 4 commits July 28, 2026 15:28

fix(gamespy): Fall back to byte cast for non-UTF-8 text in MultiByteT…

51a0460

…oWideCharSingleLine

refactor(utf8): Rename conversions to Wide_To_Utf8 and Utf8_To_Wide

1a58d95

refactor(utf8): Remove unused Utf8_Validate

81a47bf

build: Use qualified include path for WWLib utf8.h

d98492c

Uh oh!

Conversation

bobtista commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WWLib/utf8.h / utf8.cpp

AsciiString::translate

UnicodeString::translate

ThreadUtils.cpp

Non-goals

Verification

Uh oh!

greptile-apps Bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

bobtista commented Apr 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

bobtista Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

xezon Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

bobtista Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon commented Apr 6, 2026

Uh oh!

bobtista commented Apr 6, 2026

Uh oh!

Mauller Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bobtista Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Mauller commented Apr 7, 2026

Uh oh!

Uh oh!

OmniBlade commented Apr 8, 2026

Uh oh!

bobtista commented Apr 3, 2026 •

edited

Loading

`WWLib/utf8.h` / `utf8.cpp`

`AsciiString::translate`

`UnicodeString::translate`

`ThreadUtils.cpp`

greptile-apps Bot commented Apr 3, 2026 •

edited

Loading

Mauller Apr 7, 2026 •

edited

Loading