Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sequential panner stabilization #87

Merged
merged 16 commits into from
Aug 2, 2023

Conversation

teresaqhoang
Copy link
Contributor

@teresaqhoang teresaqhoang commented Aug 2, 2023

Motivation and Context

This PR sets SequentialPlanner as the default for Chat Copilot and introduces some changes to help stabilize the experience, including:

  • Giving the user the option to skip missing functions errors from the planner.
  • Allow the user to retry on invalid plan errors (i.e., invalid JSON, plans returned in non-XML or just in natural language)
  • Explicitly highlight points of failures in proposed plan to prompt user to fix before approving / denying

Description

webapi

Changes on behavior of below flags:

  • AllowRetriesOnInvalidPlan:
    • If true, ExternalInformationSkill will return plan creation if LLM returns an invalid plan. Fixed invalid plan retry logic when PlanningException.ErrorCodes.InvalidPlan is thrown.
  • SkipOnMissingFunctionsError: Whether to allow missing functions in the plan on creation then sanitize output. Functions are considered missing if they're not available in the planner's kernel's context.
    • If set to true, the plan will be created with missing functions as no-op steps that are filtered from the final proposed plan.
      - When plan returns, sanitize plan of functions not available in Planner's kernel. Merge context variables into plan parameters. Add flag $???. to indicate unknown variables in plan preview.
    • If this is set to false, the plan creation will fail if any functions are missing.

If plan contains function not available in planner's kernel's context, removed function is logged to console
image
image

webapp

  • On view plan:

    • Add validation for plan step inputs when edits are allowed (if user approval is required).
    • Display an error message when inputs contain interpolated variables or fields marked $???.
    • Fix PlanStepInput component to include validation for plan step inputs when edits are allowed (if user approval is required) and to update form value on submit.
    • Fix PlanViewer component to include updated plan in message property and to update context variables on plan action.
    • Fix an issue where updating a chat message's content was not properly updating the message in the UI.

    Invalid Inputs
    image

After valid edits
image

sequentialPlannerStabilization

Sequential Plan
image

Contribution Checklist

@github-actions github-actions bot added documentation Improvements or additions to documentation webapp Pull requests that update Typescript code webapi Pull requests that update .net code PR: ready for review labels Aug 2, 2023
@teresaqhoang teresaqhoang requested a review from a team August 2, 2023 05:50
@teresaqhoang teresaqhoang self-assigned this Aug 2, 2023
@github-actions github-actions bot added the github actions Pull requests that update GitHub Actions code label Aug 2, 2023
@github-actions github-actions bot removed the github actions Pull requests that update GitHub Actions code label Aug 2, 2023
@teresaqhoang teresaqhoang linked an issue Aug 2, 2023 that may be closed by this pull request
@github-actions github-actions bot removed the documentation Improvements or additions to documentation label Aug 2, 2023
@teresaqhoang teresaqhoang added PR: ready to merge PR has been approved by all reviewers, and is ready to merge. and removed PR: ready for review labels Aug 2, 2023
@alliscode alliscode added this pull request to the merge queue Aug 2, 2023
Merged via the queue into microsoft:main with commit cb37dee Aug 2, 2023
@teresaqhoang teresaqhoang deleted the planner-stabilization branch August 11, 2023 15:26
teamleader-dev pushed a commit to vlink-group/chat-copilot that referenced this pull request Oct 7, 2024
### Motivation and Context

<!-- Thank you for your contribution to the copilot-chat repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

This PR sets SequentialPlanner as the default for Chat Copilot and
introduces some changes to help stabilize the experience, including:
- Giving the user the option to skip missing functions errors from the
planner.
- Allow the user to retry on invalid plan errors (i.e., invalid JSON,
plans returned in non-XML or just in natural language)
- Explicitly highlight points of failures in proposed plan to prompt
user to fix before approving / denying


### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

#### webapi
Changes on behavior of below flags:
- AllowRetriesOnInvalidPlan: 
- If true, ExternalInformationSkill will return plan creation if LLM
returns an invalid plan. Fixed invalid plan retry logic when
PlanningException.ErrorCodes.InvalidPlan is thrown.
- SkipOnMissingFunctionsError: Whether to allow missing functions in the
plan on creation then sanitize output. Functions are considered missing
if they're not available in the planner's kernel's context.
- If set to true, the plan will be created with missing functions as
no-op steps that are filtered from the final proposed plan.
- When plan returns, sanitize plan of functions not available in
Planner's kernel. Merge context variables into plan parameters. Add flag
`$???`. to indicate unknown variables in plan preview.
- If this is set to false, the plan creation will fail if any functions
are missing.

If plan contains function not available in planner's kernel's context,
removed function is logged to console

![image](https://github.com/microsoft/chat-copilot/assets/125500434/1333c53f-b234-4b76-a469-3881a45e0817)

![image](https://github.com/microsoft/chat-copilot/assets/125500434/080c199d-a45e-4466-84fb-b7a4e8226587)

#### webapp
- On view plan:
- Add validation for plan step inputs when edits are allowed (if user
approval is required).
- Display an error message when inputs contain interpolated variables or
fields marked `$???`.
- Fix PlanStepInput component to include validation for plan step inputs
when edits are allowed (if user approval is required) and to update form
value on submit.
- Fix PlanViewer component to include updated plan in message property
and to update context variables on plan action.
- Fix an issue where updating a chat message's content was not properly
updating the message in the UI.
     
    Invalid Inputs

![image](https://github.com/microsoft/chat-copilot/assets/125500434/408cf387-bd7b-4f0e-969e-0f157011c80d)


After valid edits

![image](https://github.com/microsoft/chat-copilot/assets/125500434/7e6e6b52-09d5-4ad7-9566-cf8e1b343458)


![sequentialPlannerStabilization](https://github.com/microsoft/chat-copilot/assets/125500434/8b4410aa-a3be-49e3-a763-a70b402d94c0)


Sequential Plan

![image](https://github.com/microsoft/chat-copilot/assets/125500434/b63a63c7-c04f-4d3e-a688-6ff8a3e94adc)

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [Contribution
Guidelines](https://github.com/microsoft/copilot-chat/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/copilot-chat/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
~~- [ ] All unit tests pass, and I have added new tests where possible~~
- [x] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: ready to merge PR has been approved by all reviewers, and is ready to merge. sk team issue webapi Pull requests that update .net code webapp Pull requests that update Typescript code
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

ActionPlanner stabilization
4 participants