Consider exposing the lexer

[dabrahams]

Description

I am able to use the parser to get the tokens, but the tokens are all I really need and building syntactic structure on top of it is wasteful. Not the end of the world for my application, but it would be cleaner if I could just get a token stream.

[ahoppen]

I don’t think you would want to use the tokens out of the lever straight away because the parser may modify the lexemes, eg.

• The parser performs state transitions in the lever during string interpolation parsing

  swift-syntax/Sources/SwiftParser/StringLiterals.swift

  Lines 566 to 576 in ef367f7

  	// The parser has more knowledge that we have reached the end of the
  	// string interpolation now, even if we haven't seen the closing ')'.
  	// For example, consider the following code
  	// "\(abc "
  	// Since the lexer doesn't know anything about the expression structure,
  	// it assumes that the `"` starts a new string literal. But since we
  	// know in the parser that an identifier cannot be followed by a string
  	// literal without a connecting binary operator and can thus consider
  	// it as the surrounding string literal end, which thus also terminates
  	// the string interpolation.
  	self.lexemes.perform(stateTransition: .pop, currentToken: &self.currentToken)

• The parser re-classifies trivia during string interpolation parsing
  

  swift-syntax/Sources/SwiftParser/StringLiterals.swift

  Lines 215 to 217 in ef367f7

  	// The newline at the end of the last line in the string literal is not part of the represented string.
  	// Mark it as trivia.
  	let content = self.reclassifyTrivia(in: lastMiddleSegment.content, trailing: newlineSuffix)

• The parser may decide to only consume the prefix of a lexeme, eg. during parsing of generics
  

  swift-syntax/Sources/SwiftParser/Nominals.swift

  Line 347 in ef367f7

  let langle = self.consumePrefix("<", as: .leftAngle)

[dabrahams]

Interesting. I just need to skip past balanced pairs of {...} to look for the next “}” in my input, but if the parser is needed to deduce where the bounds of a string literal are that would definitely require the parser. Hmm, how does SwiftSyntax express a string literal with interpolations in it? Is the literal still a token?