Skip to content

Position.character is treated as a byte offset instead of UTF-16 code units #165

Description

@LuckySFT

Summary

pasls appears to interpret LSP Position.character as a byte offset into the
line, while the LSP specification defines it as a count of UTF-16 code units
(unless the server negotiates otherwise via positionEncoding).

pasls does not advertise positionEncoding in its initialize result, so per spec
the client is entitled to assume utf-16. The result is that textDocument/definition,
hover and textDocument/references silently return empty whenever any non-ASCII
character appears earlier on the same line as the identifier under the cursor.
There is no error — just no result — which makes it hard to notice.

Environment

  • pasls build 2026/03/14 (the prebuilt pasls-x86_64-win64 shipped in the
    coolchyni.fpctoolkit 1.2.7 VS Code extension)
  • Windows 11, FPC 3.2.2 + Lazarus 4.8 (PP / FPCDIR / LAZARUSDIR all set)
  • Driven directly over stdio by a minimal LSP client, so no editor is involved

Reproduction

Source line (a Delphi unit, CP950 on disk; mFileName is a unit-level var
declared on line 210). Six CJK characters precede the identifier:

WriteLog(FormatDateTime('yyyy/mm/dd hh:nn:ss', Now) + ' <6 CJK chars> ' + mFileName);

Two textDocument/definition requests at the same logical position:

character sent meaning result
70 UTF-16 code units (what the spec requires, and what VS Code sends) null
76 byte offset in the file's encoding correctly resolves to line 210

The same identifier resolves correctly on every pure-ASCII line in the same file
(5 of 5 occurrences), so parsing and symbol resolution themselves are fine.

Not an encoding issue

This is independent of the file's on-disk encoding. The same experiment on a
UTF-8 (with BOM) unit reproduces it: identifier xEXEC_RES, character 49
(UTF-16 units) returns null, while character 63 (UTF-8 byte offset) resolves
correctly. Reading non-UTF-8 sources itself works fine — documentSymbol returns
191 symbols with no mangled names on the CP950 file, presumably thanks to #116.

Impact

Any codebase with non-ASCII string literals or comments loses go-to-definition,
hover and find-references on a large share of its lines. In the file above, 1,276 of
8,560 lines contain non-ASCII characters. Clients cannot work around this, since
VS Code always sends UTF-16 offsets and no positionEncoding is offered.

Possible fixes

  1. Convert Position.character from UTF-16 code units to a byte offset when mapping
    an LSP position onto the source buffer (and convert back when emitting ranges), or
  2. advertise positionEncoding: 'utf-8' in the initialize result and honour the
    client's general.positionEncodings capability, so that clients supporting LSP
    3.17 negotiation send byte offsets.

Option 1 is what the spec expects by default; option 2 is cheaper but only helps
clients that implement the negotiation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions