🍀 Features/minilexer

[hahwul]

I am adding a lightweight lexer for lexical analysis. It tokenizes the source code. Depending on the mode setting, the tokenize function will decide whether to store the tokenized results. The find function retrieves only the desired type of tokens when in a specific :persistent mode.

a = MiniLexer.new
#<MiniLexer:0x129000c20 @tokens=[], @mode=:normal>

a.mode = :persistent
# :persistent

a.tokenize "foo"
a.tokenize "foo"
a.tokenize "foo"
# <Token:0x129429b00 @type=:identifier, @value="foo", @position=3>
# ....

a.find :identifier
# [
# <Token:0x129429b00 @type=:identifier, @value="foo", @position=3>, 
# <Token:0x11c171fe0 @type=:identifier, @value="foo", @position=3>, 
# <Token:0x119d15ec0 @type=:identifier, @value="foo", @position=3>
# ]

# Token
a.type #=> :identifier
a.value  #=> foo
a.is? :abcd #=> false
a.is? :identifier #=> true

Since token processing varies by language, I think you can inherit MiniLexer and implement the main logic. If you have any good opinions, please share :D

require "../models/minilexer/*"

class SampleLexer < MiniLexer
  def initialize
    super
    # Add custom initial state
  end

  def tokenize_logic
    # Add logic for each state
  end
end

[ksg97031]

Great!!