How to add the grammar of an entire programming language for better code generation?

[MoritzLaurer]

Use-case: I'm working on a code generation project and I need to make sure that the LLM generated code is syntactically correct / it compiles. A widely-used approach at the moment is to make an LLM generate N generations, then do some automatic tests to see which code compiles and discard the generations that do not compile. This is inefficient and I'm thinking about using grammars to increase the chance of getting working code.

Can I use the grammar of an entire programming language, say Python, with Outlines? What would be the main steps for using the grammar of a language like Python in Outlines? (Or are grammars for entire languages too complex / too inefficient?)

Based on the docs, I understand that the main steps are roughly:

1. Find the existing official grammar. E.g. a grammar for the Python language lives here in the official Python docs.
2. Based on your grammar docs this then needs to be translated to EBNF so that it can be parsed with Lark? Are there other pitfalls to consider?
3. How can this then be used in Outlines for valid code generation?

Here is what I could find in the docs so far: Outlines has a few grammars here; I see that you import an SQL grammar in this blog post; and you discuss in this issue how one could import YAML grammar.

[noah-nfm]

Hi @MoritzLaurer,

I'm working on a similar project and am facing the same issue.

Have you made any progress or found any additional information on this topic? Your insights would be greatly appreciated!

Thank you!

[MoritzLaurer]

Unfortunately no progress on my side