Deprecate the MySQL Grammar from Positive Technologie

[mike-lischke]

It has been suggested in the past a few times already and it was discussed in issue #4412: keeping two different MySQL grammars leads to confusion and unnecessary work. There are:

• Oracle - A complete MySQL grammar including the latest language features of MySQL. This grammar is directly derived from the MySQL Yacc server grammar and is updated at least once a year to stay in sync with the MySQL server enhancements.
• Positive Technologies - A grammar created from the official documentation (5.6, 5.7, 8.0)

The first one is the preferred one as it includes all language features starting with MySQL 8.0, while the other one still supports version 5.6 and 5.7 (which are already EOL'd).

We can see occasional pull requests to add newer language features to the PT grammar, but they are always only selectively, not going the full way (and sometimes they are plain wrong).

Since the Oracle grammar supports the full MySQL language properly, is released under a very permissive license and is maintained by Oracle, it makes sense to keep that grammar as the only one for MySQL and deprecate the grammar from Positive Technologies.

I'm not a maintainer of this repository, so I can only suggest a deprecation process:

1. Add a prominent text about the deprecation to the readme in the sql/mysql folder and to the readme in the sql/mysql/Positive-Technologies.
2. Add a prominent note in the grammars (lexer/parser) that also inform about the deprecation.
3. Reject all future PRs for this grammar with a note that it is deprecated.
4. Give it a few months (maybe 3?) to let this sink in.
5. Remove the PT grammar and restructure the mysql folder and its associated tests.

It's important to stick to such a plan, once decided, to make downstream planning possible. Users of the PT grammar have to plan some time for migration.

An additional option could be to add a DEPRECATED folder in the root of this repository and move this and future deprecated grammars to that. Grammars in this folder are not tested and no PRs will be accepted for them.

[mike-lischke]

@teverett @KvanTTT @ani-sha any opinion about this proposal?

[Naros]

Hi @mike-lischke, thank you for starting this discussion. I think it's indeed good to have, despite the outcome.

First, I agree that deprecating the PT grammar makes the most long-term sense. Given that the Oracle grammar is driven by the upstream grammar used by the MySQL engine, this has the absolute benefit of implementing anything supported by the engine in its complete form in the grammar, which addresses all the shortcomings with the PT grammar led by the community.

Where I disagree is on a few subtle, yet essential details.

is updated at least once a year to stay in sync with the MySQL server enhancements

To avoid the shortcomings that have plagued the PT grammar with community contributions, Oracle should commit to synchronizing the grammar on each MySQL release that includes grammatical changes. This will minimize, and quite honestly should avoid, the need for any community-driven modifications to the grammar.

I'd also like to add that the Oracle MySQL grammar README is unclear about the expectation for grammar changes that pertain to MySQL variants like Percona. If those are to be excluded from the grammar, given explicitly that in the future you'd like to automate the port of the grammar from the MySQL yacc grammar proper, then it would likely make sense to be clear about that in the README so that forks understand those should be maintained only in forks.

An additional option could be to add a DEPRECATED folder in the root of this repository and move this and future deprecated grammars to that. Grammars in this folder are not tested and no PRs will be accepted for them.

The only concern I have about removing or rejecting PRs is that multiple projects could rely on the grammars and still need support for MySQL 5.6/5.7 features that may have either been removed or changed in 8.0. I like the idea of the community still being able to lead that effort in some way, but it should be in a way that imposes the least amount of overhead on the maintainers IMO.

What about a new grammars-v4-deprecated repository, and any grammars that are no longer officially supported are moved there. Community users would still be able to contribute PRs to allow collaboration on defects identified in the grammar. There would be no significant scrutiny about the correctness of the grammar, just the simple rule that if the tests pass, it's safe to merge.

This distinguishes between official, supported, and deprecated, unsupported grammars. It also continues to foster community-led collaboration for any community that may still rely on the 5.6/5.7-based grammars while not really imposing much overhead on the maintainers.

As for maintainer overhead, that could be addressed in a variety of ways:

1. The simplest would be if the tests pass, safe to merge.
2. Create folder-specific community-teams that are responsible for the review and merging of specific grammar changes. These teams would consist of contributors who have demonstrated consistent, valuable contributions to that grammar.

I'm sure there are others, but I wanted to jot down some quick thoughts.

[kaby76]

To avoid the shortcomings that have plagued the PT grammar with community contributions, Oracle should commit to synchronizing the grammar on each MySQL release that includes grammatical changes. This will minimize, and quite honestly should avoid, the need for any community-driven modifications to the grammar.

The Oracle grammar here is supposed to be scraped via script. It shouldn't be modified by hand because it's derived from the VSCode extension grammar, which is official Oracle code. I just haven't finished the scraping script yet.

There are differences between the grammar in this repo and that in the VSCode extension. The grammars in this repo are intended to work across all targets/ports. The Trash requires a C# port.

I'd also like to add that the Oracle MySQL grammar README is unclear about the expectation for grammar changes that pertain to MySQL variants like Percona.

I would recommend people try to use Antlr4's import to make variants. When the scraping script is added, I'll update the readme.md to let people know how to update the grammar, and how to make derivatives. Who knows if any of this will really fix the "grammar forking problem", but it's worth trying.

[mike-lischke]

To avoid the shortcomings that have plagued the PT grammar with community contributions, Oracle should commit to synchronizing the grammar on each MySQL release that includes grammatical changes. This will minimize, and quite honestly should avoid, the need for any community-driven modifications to the grammar.

The MySQL server has nothing to do with the ANLTR grammar. This must be maintained by a team that also uses it, so it's not related in any way to the MySQL server releases. The grammar itself however, is published with each release of the MySQL Shell, but since there's now a transformation involved (Ken explained it), there can't be an automatic PR for this repository.

I'd also like to add that the Oracle MySQL grammar README is unclear about the expectation for grammar changes that pertain to MySQL variants like Percona.

Well, from my standpoint it's clear. It doesn't mention any other DB "dialect" or "variant" and hence can only be expected to support the original MySQL language. If a project extends the MySQL language it also has to maintain an own grammar.

If those are to be excluded from the grammar, given explicitly that in the future you'd like to automate the port of the grammar from the MySQL yacc grammar proper, then it would likely make sense to be clear about that in the README so that forks understand those should be maintained only in forks.

I dare to disagree. Why should one project mention any actual or future forks to say it doesn't support them? The readme clearly says this is the MySQL grammar for version 8.0 and newer.

The only concern I have about removing or rejecting PRs is that multiple projects could rely on the grammars and still need support for MySQL 5.6/5.7 features that may have either been removed or changed in 8.0.

Yes, that's always a problem: backward compatibility. For those projects which cannot switch they have to live with the current grammar, which is available in a deprecated folder. But as the tag deprecated already implies, these are no longer in active development. Nobody should expect them get updated. If this is a too harsh requirement, we cannot deprecated the grammar.

What about a new grammars-v4-deprecated repository, and any grammars that are no longer officially supported are moved there.

Yes, that's what I proposed above.

Community users would still be able to contribute PRs to allow collaboration on defects identified in the grammar.

Not a good idea. Why would you deprecate the grammar at all, if you still support it (by accepting PRs)? Projects that need newer features should migrate to the Oracle grammar (within a comfortable migration period of course).

It also continues to foster community-led collaboration for any community that may still rely on the 5.6/5.7-based grammars while not really imposing much overhead on the maintainers.

Well, that's what should be decided here. My vote is not to add extra burden like that, and deprecating something means it is no longer maintained or changed. Deprecated OS or DB server versions don't get any updates anymore. Why should we handle that differently?

I'm sure there are others, but I wanted to jot down some quick thoughts.

Yes, excellent. That's the kind of feedback I hoped for! I also hope the repo maintainers join this discussion.

[jpechane]

Hi @mike-lischke, thanks for taking the initiative. I'd agree with most of your comments yet I'd like to throw few more

Well, from my standpoint it's clear. It doesn't mention any other DB "dialect" or "variant" and hence can only be expected to support the original MySQL language. If a project extends the MySQL language it also has to maintain an own grammar.

I don't see any harm in explicitly stating that. People can have (even unconcious) expecttion when they'll move for PT grammar so why not to highlight that fact?

I dare to disagree. Why should one project mention any actual or future forks to say it doesn't support them? The readme clearly says this is the MySQL grammar for version 8.0 and newer.

Here I'd argue it is differece between theory and pratice or de-iure and de-facto state and it relates to the point above. Over time people could build some (unsubstantiated) expectations. In both cases the argument goes for (over-)communication towards community. Nothing that would affect the project or grammar in any way.

Well, that's what should be decided here. My vote is not to add extra burden like that, and deprecating something means it is no longer maintained or changed. Deprecated OS or DB server versions don't get any updates anymore. Why should we handle that differently?

The problem is that we see people using dperecated database still used as the official lifecycle might differ from the user intended lifecycle or there can be a vendor/provider with a longer support period. The proposal effectivelly leads to creation of plaform where those that are still bound to using it can collaborate instead of having everybody to cope on its own.

[mike-lischke]

Well, from my standpoint it's clear. It doesn't mention any other DB "dialect" or "variant" and hence can only be expected to support the original MySQL language. If a project extends the MySQL language it also has to maintain an own grammar.

I don't see any harm in explicitly stating that.

Of course there's no harm, but also no reason. For example let's look at the Python 3 grammar readme. It states it's for Python 3, perfect. But it does not say it's not for Python 2 or Python 15 or any other Python variant. And that makes a whole lot of sense. To me asking for adding a note that a MySQL grammar is not for MariaDB or TokuDB or whatever is confusing.

People can have (even unconcious) expecttion when they'll move for PT grammar so why not to highlight that fact?

Migrating from one grammar to another is a pretty big task and that requires close attention to what changed. Usually projects owners will do a test and quickly see what works and what does not. People using ANTLR are smart. They figure out what works.

The problem is that we see people using dperecated database still used as the official lifecycle might differ from the user intended lifecycle or there can be a vendor/provider with a longer support period.

True, that's why there must be a useful deprecation period. Also, the old PT grammar does not secretly disappear. It's still in the world and nobody is forced to use any other grammar.

The proposal effectivelly leads to creation of plaform where those that are still bound to using it can collaborate instead of having everybody to cope on its own.

I'm sorry, I couldn't parse this sentence fully. What platform do you mean? My proposal does not lead to creation of another platform, it only suggests to keep only one MySQL grammar.

Thanks for your input @jpechane!

[Naros]

Hi @mike-lischke

To me asking for adding a note that a MySQL grammar is not for MariaDB or TokuDB or whatever is confusing.

It's no more confusing when Percona is advertised as a compliant MySQL drop-in replacement and then a user performs some operation, and the DDL statement executed cannot be parsed by the MySQL grammar.

I concur with Jiri here. There is no harm in overcommunicating this. It abundantly clarifies that no one should expect the grammar to work even when a system claims MySQL compliance.

I'm sorry, I couldn't parse this sentence fully. What platform do you mean? My proposal does not lead to creation of another platform, it only suggests to keep only one MySQL grammar.

By platform, a grammar-v4-deprecated repository is created, where the community can maintain the grammar for at least an extended period with minimal to little maintainer overhead.

This allows Debezium and other projects that still rely on the legacy grammar to continue to have a collaborative space where we can benefit from each project's collective work, rather than each project doing that on their own.

I simply suggested a separate repository as perhaps maintaining the write rights would be easier to control.

[mike-lischke]

OK, note taken, thanks. Will you or Jiri start such a repository and take over maintenance? Do you want to make the decision for this proposal dependent on whether such a repository is created?

Adding a note to the readme is easy. It's just amazing to see such a long discussion about such a little detail. I'd be more interested in additional input about the deprecation process itself, and that requires participation of the grammar repo maintainers here.

[jpechane]

In my opinion we have a reasonable consensus. Let's wait for maintainers to comment before taking additional steps. And WRT your question about the repo. No we don't want to make the final decision based on having the repo or not. We tried to popose what we see as the best course of action but it should not be interpreted as "conditio sine qua non".

[KvanTTT]

I have no objection against whatever decision you accept. I haven't been contributing to grammar for a long time, that's why I'm more concerned about active contributors and users.

[kaby76]

I'm not an official maintainer of this repo. But, I do have significant, general maintenance of most of the grammars in the repo over 4 years. I would recommend deprecating and moving the grammar to another repo, ASAP please.

The mysql/Positive-Technologies grammar has significant issues, in language recognition and performance. For an equivalent test suite (find sql/mysql/Oracle/examples -name '*.sql'), the PO grammar is generally an order of magnitude slower than the Oracle grammar.

For bitrix_queries_cut.sql, there are 7478 ambiguities. Of that, 3461 are for Decision state 938, which is the first Decision state immediately upon entering the fullColumnName rule. This is obvious just by looking at the two alts for the rule, whereby uid and . recognize the same string. #2945 (review)

[mike-lischke]

Putting the deprecated grammar into an own repo requires more work, however. Someone needs to take care for that repo. I can file a PR to remove the PT grammar, but I have no bandwidth to maintain a deprecation repo. Hence my suggestion to just move the grammar to a separate folder.

[kaby76]

We can move it to another directory, but testing needs to be changed. The Trash tester looks across the entire repo for desc.xml files. The desc.xml for PT needs to be removed, or the <targets> element commented out, to stop testing of the grammar. Otherwise, it's still being "maintained." In addition, various pom.xml, which is for the old tester, would have to be cleaned up. In other words, the meaning of "deprecated" needs to be defined. (I.e., Where does it go? Is it removed from testing? Who modifies it, if ever, after deprecation? Etc).