Fails on a perfectly valid json

[gaouservbf]

---

EveryDiscord

Extra 0 at char position 7147

OK

There isn't even a "0" in that character. Though unsure if it means a NULL or a literal number 0.

[gaouservbf]

If its a NULL here. I can just adapt my code to remove it, I just need to be informed.

[gaouservbf]

Viewed character under a hex editor.

[cristianbuse]

I think it first converts from UTF8 to UTF16 in this case which means 7147 refers to the position in the converted string not the original.

[cristianbuse]

Try converting your file / json to UTF16 and then see what is the character at position 7147

[cristianbuse]

Maybe I should explain more clearly. If a conversion is needed from UTF8 to UTF16, the current implementation uses API calls to do that, before parsing even begins. The parser itself only works on UTF16. So, we cannot map the character position because in UTF8, a character can be composed of 1 up to 4 bytes each and we have no control of what the API does under the hood. The only way around this problem is to implement our own conversion - I have plans for that but not until autumn / winter.

In other words, try passing a UTF16 string directly so that the error message makes sense.

[gaouservbf]

Let me check!

[gaouservbf]

It points to a regular empty character/filler for UTF-16. Though I think I shouldn't use a hex editor here..

[cristianbuse]

Is it possible that you share a string around that position. E.g. 50 chars in front and 50 after?

[gaouservbf]

Is it possible that you share a string around that position. E.g. 50 chars in front and 50 after?

Around it is just regular text.

[cristianbuse]

The error message says 'extra 0'. I did not see a zero in your screenshot. It could be that the UTF8 to UTF16 API I use returns a different string than the converter you used. That could happen if invalid UTF8 is found and by default replaced with 0xFFFD.

Are you getting the error on Mac or Windows?

[gaouservbf]

The error message says 'extra 0'. I did not see a zero in your screenshot. It could be that the UTF8 to UTF16 API I use returns a different string than the converter you used. That could happen if invalid UTF8 is found and by default replaced with 0xFFFD.

Are you getting the error on Mac or Windows?

Legacy Windows (XP SP3 x86), normally works. It didn't in this case though?

[cristianbuse]

I was just trying to see which API is used for the conversion. Will provide some code tomorrow so you can run. I am away from computer this evening.

[gaouservbf]

Thanks!

[cristianbuse]

Drop this into a new module:

Option Explicit

Private Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" (ByVal codePage As Long, ByVal dwFlags As Long, ByVal lpMultiByteStr As LongPtr, ByVal cbMultiByte As Long, ByVal lpWideCharStr As LongPtr, ByVal cchWideChar As Long) As Long

Public Function UTF8ToUTF16LE(ByRef textToConvert As String) As String
    Dim charCount As Long
    Const pageCodeUTF8 = 65001
    '
    charCount = MultiByteToWideChar(pageCodeUTF8, 0, StrPtr(textToConvert) _
                                  , LenB(textToConvert), 0, 0)
    If charCount = 0 Then Exit Function
    '
    UTF8ToUTF16LE = Space$(charCount)
    MultiByteToWideChar pageCodeUTF8, 0, StrPtr(textToConvert) _
                      , LenB(textToConvert), StrPtr(UTF8ToUTF16LE), charCount
End Function

and then call UTF8ToUTF16LE(yourString) to get the same conversion as the parser gets.

Thanks!