Gherkin: Comments after tags not present in GherkinDocument.getCommentsList()

[enikao]

Summary

Comments on the same line as a tag cannot be retrieved from the list of comments.
Applies to tags on both Features and Scenarios.

Example:

# I can find this comment
@myTag #but not this one
Feature: Make comments available through the parser

# comment to be found
@myOtherTag #inaccessible comment
Scenario: If a tag is followed by a comment, the comment should be accessible through the parser.

Expected Behavior

All comments, including the ones following a tag on the same line, should be present in GherkinDocument.getCommentsList().

Current Behavior

Comments after a tag on the same line are inaccessible via the parser.

Context & Motivation

I want to import the complete file contents.
We often use comments after tags to simulate "tags with spaces", so these comments contain valuable information.

Your Environment

• Version used: io.cucumber.gherkin 17.0.2
• Operating System and version: Windows 10, JDK 11.0.7 amd64
• Link to your project: private

[gasparnagy]

Normally Gherkin allows you to make full line comments only.
Commenting after tags on the same line was specially hacked in because many IDEs allowed this and it caused confusion. But since the current Gherkin parser uses the lines as tokens, it is not easy to handle these "special" comments as normal ones. So I'm afraid there is no easy fix for that. I encourage everyone to NOT use this kind of commenting... I recommend using the @tags_with_spaces style instead.

Maybe @mpkorstanje can add something to this.

[enikao]

I agree these comments don't adhere to the grammar. Unfortunately, I have limited influence how the files are written.

As a workaround, we could have a flag that disables the hack that allows such comments.
Then they would be reported as parsing errors, and we at least know we're losing some content.

[gasparnagy]

As this is deep in the grammar level, the usual cucumber settings cannot be applied there... so it is not easy to make a setting for that (the grammar/parser can be also used even without cucumber)

[mpkorstanje]

As trade-offs go I think we currently have the right one. The user experience should weight much higher then being able to build a a feature file from the AST.

Though I do see some room in the current AST to allow the comments on tag lines to be presented included. The location element uses both lines and columns. So perhaps the tag comments can be included as comments starting at a non-zero column?

  /**
   * A comment in a Gherkin document
   */
  message Comment {
    // The location of the comment
    Location location = 1;
    // The text of the comment
    string text = 2;
  }
  
/**
 * Points to a line and a column in a text file
 */
message Location {
  uint32 line = 1;
  uint32 column = 2;
}  

[gasparnagy]

@mpkorstanje I think the AST would be able to handle the comments (BTW the normal comments do not necessarily start at column 0 or 1, but the column value indicates the position of the # sign within the line), the more problem is how the token scanner can feed the comment token back to the parser... Anyway. I totally agree with you that we have the right trade-off.