What is the best practice to extract JSON from the guidance output?

[aeft]

I want to extract the JSON from the guidance output. I can reconstruct the JSON using the stored variables in lm but it is not very elegant. What is the best practice? Thanks.

In general, the question pertains to extracting a fraction of data that contains both input and LLM-generated output.

import guidance

lm = phi2

# define a re-usable "guidance function" that we can use below
@guidance
def quoted_list(lm, name, n):
    for i in range(n):
        if i > 0:
            lm += ", "
        lm += '"' + gen(name, list_append=True, stop='"') + '"'
    return lm

lm = lm + f"""What are the most common commands used in the Linux operating system?

Here are the 5 most common commands in JSON format:
{{
    "commands": [{quoted_list('commands', 5)}],
    "my_favorite_command": "{gen('favorite_command', stop='"')}"
}}"""

print(str(lm))

output:

What are the most common commands used in the Linux operating system?

Here are the 5 most common commands in JSON format:
{
    "commands": ["ls", "cd", "mkdir", "rm", "pwd"],
    "my_favorite_command": "grep"
}
What are the most common commands used in the Linux operating system?

Here are the 5 most common commands in JSON format:
{
    "commands": ["ls", "cd", "mkdir", "rm", "pwd"],
    "my_favorite_command": "grep"
}

[slewie]

I used such methods to do it:

def fill_json(text: str, llm):
    prompt = '"... create it for such sentence:"
    output = llm + _fill_json(prompt, text)
    return json.loads(output['json'])

@guidance
def _fill_json(lm, prompt, text):
    lm += prompt + f" {text}\n```json" + gen('json', stop='```')
    return lm

[aeft]

In this case, you cannot define and control the JSON structure.

[hudson-ai]

You can enclose the entire bit that generates JSON in a guidance block :)
I.e.

import guidance
from guidance import block

lm = phi2

# define a re-usable "guidance function" that we can use below
@guidance
def quoted_list(lm, name, n):
    for i in range(n):
        if i > 0:
            lm += ", "
        lm += '"' + gen(name, list_append=True, stop='"') + '"'
    return lm

lm = lm + """What are the most common commands used in the Linux operating system?

Here are the 5 most common commands in JSON format:
"""
with block("my_json"):
  lm += f"""
{{
    "commands": [{quoted_list('commands', 5)}],
    "my_favorite_command": "{gen('favorite_command', stop='"')}"
}}"""

print(lm["my_json"])

Output:

{
    "commands": ["ls", "cd", "mkdir", "rm", "cp"],
    "my_favorite_command": "ls --color=auto"
}

[hudson-ai]

@aeft you may also be interested in a new feature (not yet packaged in an official release) -- json.

Here's some example usage:

from guidance import json

lm += """What are the most common commands used in the Linux operating system?
Here are the 5 most common commands in JSON format:
"""
lm += json(my_json_schema)

An easy way to build a json schema without having to "think in json" would be pydantic, e.g.

from pydantic import BaseModel
class MyJson(BaseModel):
    commands: list[str]
    my_favorite_command: str
my_json_schema = MyJson.model_json_schema()

Note that guidance.json doesn't yet support constraints such as "list with exactly 5 elements", but that would be a great issue if you'd like to get it on the developers' radars.