Skip to content

Reduce tokens in generated system prompt #1004

@jiahaog

Description

@jiahaog

Motivation: Better inference latency, accuracy

The following code is some minimal code that generates a system prompt:

schema_manager = manager.A2uiSchemaManager(
    version=constants.VERSION_0_9,
    catalogs=[
      # ...
    ],
)

system_prompt = schema_manager.generate_system_prompt(
    role_description="...",
    include_schema=True,
).strip()

print(system_prompt)

def generate_system_prompt(

The generated system prompt has this structure:

<role description>

## Workflow description

The generated response MUST...

---BEGIN A2UI JSON SCHEMA---

### Server to Client Schema:

<server to client schema>

### Common Types Schema:

<common types schema>

### Catalog Schema:

<catalog schema>

---END A2UI JSON SCHEMA---

This task relates to both the <server to client schema> and <common types schema>.

  1. <common types schema> can be automatically tree-shaken based on the <catalog schema> by traversing $refs. Recursively look up $refs, then drop unused types from <common types schemna>.
  2. <server to client schema> needs to be tree-shaken with intent from the user. Perhaps generate_system_prompt should be parameterized by a list of allowed messages (e.g. CreateSurfaceMessage)?

@nan-yu @jacobsimionato I'm still new to all this but let me know your thoughts!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions