Join "optional" and "extra_info" rules in grammar

Since the "optional" statement isn't used in any way right now,
enforcing a specific layout for what comes after the "doctype" serves
no useful purpose. I don't intend docstub to be linter so I don't think
it's likely that we will ever enforce annotating that a parameter is
"optional" in the docstring – while it's nice to do so, it's reasonably
easy to see it from the signature.

Update the docs accordingly. Also try to be more precise about
differentiating "doctypes" from valid Python types.

Author: lagru

docs/typing_syntax.md

@@ -7,24 +7,24 @@
 > Several features are experimental and included to make adoption of docstub easier.
 > Long-term, some of these might be discouraged or removed as docstub matures.
 
-Docstub defines its own [grammar](../src/docstub/doctype.lark) to parse and transform type information in docstrings into valid type annotations.
+Docstub defines its own [grammar](../src/docstub/doctype.lark) to parse and transform type information in docstrings (doctypes) into valid Python type expressions.
 This grammar fully supports [Python's conventional typing syntax](https://typing.python.org/en/latest/index.html).
-So any type annotation that is valid in Python, can be used in a docstrings as is.
+So any type expression that is valid in Python, can be used in a docstrings as is.
 In addition, docstub extends this syntax with several "natural language" expressions that are commonly used in the scientific Python ecosystem.
 
-Docstrings are expected to follow the NumPyDoc style:
+Docstrings should follow a form that is inspired by the NumPyDoc style:
 ```
 Section name
 ------------
-name : annotation, optional, extra_info
+name : doctype, extra_info
   Description.
 ```
 
-- `name` might be the name of a parameter or attribute.
-  Other sections like "Returns" or "Yields" are supported.
-- `annotation` the actual type information that will be transformed into the type annotation.
-- `optional` and `extra_info` can be appended to provide additional information.
-  Their presence and content doesn't currently affect the resulting type annotation.
+- `name` might be the name of a parameter, attribute or similar.
+- `doctype` the actual type information that will be transformed into a Python type.
+- `extra_info` is optional and captures anything after the first comma (that is not inside a type expression).
+  It is useful to provide additional information for readers.
+  Its presence and content doesn't currently affect the resulting type annotation.
 
 
 ## Unions

docs/user_guide.md

@@ -93,21 +93,21 @@ def example_metric(
 There are several interesting things to note here:
 
 - Many existing conventions that the scientific Python ecosystem uses, will work out of the box.
-  In this case, docstub knew how to translate `array-like`, `array of dtype uint8` into a valid type annotation in the stub file.
-  In a similar manner, `or` can be used as a "natural language" alternative to `|`.
+  In this case, docstub knew how to translate `array-like`, `array of dtype uint8` into a valid Python type for the stub file.
+  In a similar manner, `or` can be used as a "natural language" alternative to `|` to form unions.
   You can find more details in [Typing syntax in docstrings](typing_syntax.md).
 
-- Optional arguments that default to `None` are recognized and a `| None` is appended automatically if the type doesn't include it already.
+- Optional arguments that default to `None` are recognized and a `| None` is appended automatically.
   The `optional` or `default = ...` part don't influence the annotation.
 
 - Referencing the `float` and `Iterable` types worked out of the box.
-  All builtin types as well as types from the standard libraries `typing` and `collections.abc` module can be used.
+  All builtin types as well as types from the standard libraries `typing` and `collections.abc` module can be used like this.
   Necessary imports will be added automatically to the stub file.
 
 
-## Using types & nicknames
+## Referencing types & nicknames
 
-To translate a type from a docstring into a valid type annotation, docstub needs to know where that type originates from and how to import it.
+To translate a type from a docstring into a valid type annotation, docstub needs to know where names in that type are defined from where to import them.
 Out of the box, docstub will know about builtin types such as `int` or `bool` that don't need an import, and types in `typing`, `collections.abc` from Python's standard library.
 It will source these from the Python environment it is installed in.
 In addition to that, docstub will collect all types in the package directory you are running it on.

src/docstub/doctype.lark

@@ -12,7 +12,7 @@
 // The basic structure of a full docstring annotation as it comes after the
 // `name : `. It includes additional meta information that is optional and
 // currently ignored.
-?annotation_with_meta: type ("," optional)? ("," extra_info)?
+?annotation_with_meta: type ("," extra_info)?
 
 
 // A type annotation. Can range from a simple qualified name to a complex
@@ -132,13 +132,8 @@ shape: "(" dim ",)"
 ?dim: INT | ELLIPSES | NAME ("=" INT)?
 
 
-// Optional information about a parameter has a default value, added after the
-// docstring annotation. Currently dropped during transformation.
-optional:  "optional" | "default" ("=" | ":")? literal_item
-
-
-// Extra meta information added after the docstring annotation.
-// Currently dropped during transformation.
+// Extra meta information added after the docstring annotation, e.g. "optional"
+// or "default: 3". Information is dropped and not used to generate stubs.
 extra_info: /[^\r\n]+/
 
 // A simple name. Can start with a number or character. Can be delimited by "_"