Create README.md

Bale001 · web-flow · commit 7a8e0dd27f95 · 2021-12-04T15:31:17.000-08:00
diff --git a/tokenizer/README.md b/tokenizer/README.md
@@ -0,0 +1,3 @@
+Performs lexical analysis on source code. Will output a stream of tokens. These tokens should be fed to a parser.
+
+This is the first step of the compiler, see parser for the next step.

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+Performs lexical analysis on source code. Will output a stream of tokens. These tokens should be fed to a parser.`
	`2`	`+`
	`3`	`+This is the first step of the compiler, see parser for the next step.`