Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow adding variant via prompttemplate files #14509

Merged
merged 4 commits into from
Nov 28, 2024
Merged

Allow adding variant via prompttemplate files #14509

merged 4 commits into from
Nov 28, 2024

Conversation

JonasHelming
Copy link
Contributor

fixed #14507

What it does

Allows users to add custom variants to existing main prompts via adding .prompttemplate files.
If the file name of a .prompttemplate file starts with the id of an existing main prompt, the custom prompt file is treated as a variant.

How to test

  • Create a new .prompttemplate file which starts with the id of an existing main prompt template.
  • Browse to the respective agent and select the variant
  • Edit the variant
  • No other prompt feature should be influenced

Follow-ups

we might allow users to create variants in the UI

Review checklist

Reminder for reviewers

@tsmaeder tsmaeder requested a review from sdirix November 25, 2024 16:07
Copy link
Member

@sdirix sdirix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me!

I tested editing the workspace agent template. Creating a new template and setting it in the AI configuration works fine.

There are some glitches when deleting this file:

  • Create this custom variant template
  • Set it in the UI for an agent
  • Send a request for the agent, observe it is correctly set as system prompt

If the file is now deleted, the following happens:

  • It's still shown in the AI configuration UI until the selection is switched back and forth
  • Although it's gone, clicking on "Edit" will still open an editor for the no longer existing file
  • When sending a request to the agent, it will use NO system prompt

What is likely going on is that, although deleted, Theia AI still tries to use the deleted prompt variant. In the UI this leads to the variant no longer appearing in the dropdown, but still being selected. At request time, the prompt can't be found and then no prompt is used.

This use case should be handled cleaner:

  • The UI should be updated immediately
  • The Edit button should always open the correct file
  • Requests should fallback to their regular prompt

@JonasHelming
Copy link
Contributor Author

Great catch, thank you!

New commit should fix the describe issue:

  • We update the selection box on file changes
  • We show to the user if a variant is no longer existing and disallow editing it
  • If a variant is not available, but configured, an empty system prompt is expected, though

@JonasHelming JonasHelming requested a review from sdirix November 27, 2024 20:59
Copy link
Member

@sdirix sdirix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one more problematic flow:

  • When there are multiple variants, the flow works as expected.
  • However when there is no variant left, then the UI disables all buttons, the dropdown vanishes, there is no error message and the user can't recover from the state without re-adding a variant

image

Steps to reproduce:

  • Open AI configuration view
  • Select workspace agent
  • Click "edit", edit the template and use "File > Save As" to save it as a variant
  • Select the newly added variant in the UI
  • Delete the prompt template again
  • Now the UI is in the broken state and does not recover without reloading or readding a variant

One more thing (could be a followup):

  • Deleting the variant can also be accomplished via the "Reset" button.
  • For these custom variants it would be nice if the button would show "Delete" instead as they are then gone in contrast to the "built-in" variants

Besides this edge case it seems to work great now 👍

@JonasHelming
Copy link
Contributor Author

Great catch, fixed the describe workflow!
I will create a follow up for "delete".

@JonasHelming JonasHelming requested a review from sdirix November 27, 2024 23:59
Copy link
Member

@sdirix sdirix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great for me. Thanks!

@JonasHelming JonasHelming merged commit 4e99bcd into master Nov 28, 2024
11 checks passed
@github-actions github-actions bot added this to the 1.56.0 milestone Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[Theia AI] Allow user to add custom prompt variants
2 participants