Rebase promptflow-evals from main #2595
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Description Currently call a flow in a flow node will fail due to wrongly handled operation context, in this PR, we fix it by copy the original context and recover. Issue: #2237 This pull request primarily focuses on refactoring the handling of operation contexts in the `promptflow` package. The most significant changes include the addition of a `copy` method to the `OperationContext` class, the introduction of a `set_instance` class method to the same class, and changes to the `exec_line_async` and `_update_operation_context` methods in the `FlowExecutor` class to utilize these new methods. Here is a breakdown of the key changes: Changes to `OperationContext` class in `src/promptflow/promptflow/_core/operation_context.py`: * A `copy` method has been added to the `OperationContext` class, which creates a copy of the current operation context and returns it. This method also ensures that a copy of the `_OTEL_ATTRIBUTES` is made. * A `set_instance` class method has been introduced. This method allows setting a new instance of the operation context. Changes to `FlowExecutor` class in `src/promptflow/promptflow/executor/flow_executor.py`: * In the `exec_line_async` method, the `OperationContext`'s new `copy` method is used to create a copy of the original operation context. * The `_update_operation_context` method has been significantly simplified. Instead of manually reverting changes to the operation context, it now simply restores the original context using the `OperationContext`'s `set_instance` method. These changes simplify the code and make the handling of operation contexts more robust and less error-prone. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. Co-authored-by: Heyi <[email protected]>
# Description As we will layer `promptflow` recently, and tracing should be the first to come out, this PR targets to separate `start_trace` to make it standalone from `promptflow`. All devkit (previous SDK and PFS) related functions are moved to a separate `_tracing.py` under `_sdk`. `start_trace` now works more like a tracer provider setting function without devkit; and session id now is an optional value, but with devkit, we will add it with default value `default` for easier management. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes.
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. This pull request primarily includes changes to the `flow-yaml-schema-reference.md` file, which is part of the documentation for the YAML schema. The changes mainly involve modifications to the formatting of the tables in the document. The most significant changes include the adjustment of the column width in several tables and the addition of a new field, `environment_variables`, in the `nodes` section. Formatting changes: * [`docs/reference/flow-yaml-schema-reference.md`](diffhunk://#diff-ee48baad7764b577cd9b539588af7e0d9856d7faedd0489dfcf0ec5e3e27bb39L12-R12): Adjusted the column width in the tables under the `YAML syntax`, `Flow input`, `Nodes`, and `Node variants` sections. These changes help to improve the readability of the tables. [[1]](diffhunk://#diff-ee48baad7764b577cd9b539588af7e0d9856d7faedd0489dfcf0ec5e3e27bb39L12-R12) [[2]](diffhunk://#diff-ee48baad7764b577cd9b539588af7e0d9856d7faedd0489dfcf0ec5e3e27bb39R21-R28) [[3]](diffhunk://#diff-ee48baad7764b577cd9b539588af7e0d9856d7faedd0489dfcf0ec5e3e27bb39L47-R48) [[4]](diffhunk://#diff-ee48baad7764b577cd9b539588af7e0d9856d7faedd0489dfcf0ec5e3e27bb39L64-R65) Addition of new field: * [`docs/reference/flow-yaml-schema-reference.md`](diffhunk://#diff-ee48baad7764b577cd9b539588af7e0d9856d7faedd0489dfcf0ec5e3e27bb39R21-R28): Added a new field, `environment_variables`, to the `nodes` section. This field allows users to set environment variables by specifying a property path and value. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description Currently, trace_collector use logger from flask to do log. We need to support pass in logger as parameter to let runtime can reuse this code. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Yangtong Xu <[email protected]>
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description This pull request includes a minor change to the `start_tracing` method in the `src/promptflow/promptflow/tracing/_tracer.py` file. The change removes a warning log message that was previously triggered when attempting to start tracing for a run while another run is already active. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. Co-authored-by: Heyi <[email protected]>
# Description This PR is focused on enabling the recording injection mode for CI/tests in the execution environment. The recording mode provides a mechanism to record and replay end-to-end tests, enhancing the reliability and efficiency of our testing process. For detailed information on the recording mode, please refer to our [documentation](https://github.com/microsoft/promptflow/blob/main/docs/dev/replay-e2e-test.md). In this PR, we still keep the connection configuration in CI to make sure backward compatibilities. Meanwhile, enable the recording injection fixture to switch test into recording mode. ## **Key Features of This PR:** ### **Generation of Test Records** The generation of records for execution tests when they are missing in the shelve database, to make it work under recording mode. ### **Multi-Process Compatibility** Resolves issues related to the recording injection mode in a multi-process environment, particularly with processes spawned/forkserver. **_What is the issue under multi/new Process(spawn/forkserver)?_** Spawn/forkserver mode will not replicate the resources/setups from main Process, including the recording setup. This would make the recording not working anymore.  **_How you resolved this issue?_** There are multiple ways to pass the recording setup in new Process, like environment variable, serializable object as argument etc. But these might incur interface change or too complex to squeeze into simple state object. We choose to re-import the state into new Process. 1) Create new target method for Process: this new target is used to re-import the state needed Example: For new Process target method _process_wrapper, for define another wrapper method outside and inject the recording state  2) Define a customized Process class with above new targets Enable this new target method whenever new Process spawned/forkservered  3) Override context.Process or multiprocess.Process class multiprocessing.get_context("spawn").Process = MockSpawnProcess or multiprocessing.Process = MockSpawnProcess We have implemented above logic in codes and integrated as part of recording injection fixture for testing. **_So all the tests under executor/sdk would intercept the third-party call like this as default in CI?_** Yes, all the CI is enable with "PROMPT_FLOW_TEST_MODE=replay". All the openai/aoai related calls would be mocked to do recording result retrieval rather than real api request. **_Sometimes we might have necessity to customize the openai/AOAI call in tests. How shall we do instead of using the default recording mock?_** Yes, 1) Create your own target with customized mocking about openai/aoai call . 2) Override the default recording target via context manager "override_process_target" Sample Tests: test_executor_openai_telemetry **_In which scope the recording mode is enabled?_** The recording mode is enabled per requiriment. If the test are involved with 3rd party service with connection, like openai/aoai etc, it is required to enable recording mode. If not, your PR will fail at the CI tests since the connections info are not configured. To enable recording mode, just make sure the fixture "recording_injection" is required in your test class level or method level. _**Why not make recording mode a session-level fixture or required fixture for all tests?**_ 1) recording setup is complicated. it might introduce expected behavior if abuse 2) Some tests like **test_executor_openai_telemetry** might server some special test purpose. If enable recording, the customized mocking might be not easy to be configured. **_Note:_** _Above logic resolved this issue for one layer of new Processing. If you have nested new Processing action , you need to repeat above logic at each layer._ ## **Todos and Ongoing Work[updated]:** ### **[Done]Metrics Mocks** Currently, this openai_injector is skipped as it does not involve actual openai/aoai requests. This omission leads to potential inaccuracies in metrics calculation, especially token usage, in recording mode. Future work will focus on integrating openai_injector.py into the recording mode. --- this fixed already by [[fundamental] Recording support metrics by crazygao · Pull Request #1762 · microsoft/promptflow (github.com)](#1762) ### **[Todo]Consolidation of Configuration** Efforts are underway to consolidate the configuration settings for recording injection in both the execution environment and the SDK, aiming for a more unified and streamlined setup. ### **[Done]Record error info** Record not only regular tool execution result, but also error response if exception ocurr ----- have fixed this in this PR, sample tests: test_executor_node_overrides ### **[Done]Test langchain** Support langchain test with agent ---- fixed by [[fundamental] Recording support metrics by crazygao · Pull Request #1762 · microsoft/promptflow (github.com)](#1762) ### **[Todo] Create pipeline CI without connection to monitor** We need to CI without connection to identify tests which needs the record setup. ### **[Todo] Make migration plan to make the non-connection CI as main CI** Need to make plan to safely switch to new record mode. Make sure there are fallback logic if record has flow design which might hinder the new feature delivery. ### **[Todo] Enable more apis with recording mode** like assistant apis & tests # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [x] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes. --------- Co-authored-by: Philip Gao <[email protected]> Co-authored-by: chjinche <[email protected]> Co-authored-by: Cheng Liu <[email protected]> Co-authored-by: Han Wang <[email protected]> Co-authored-by: chenslucky <[email protected]> Co-authored-by: cs_lucky <[email protected]> Co-authored-by: Ying Chen <[email protected]> Co-authored-by: Ying Chen <[email protected]> Co-authored-by: chenyang <[email protected]> Co-authored-by: Peiwen Gao <[email protected]>
# Description Remove variant_id concept in all executor related logic since it is used only for legacy logic. This pull request primarily involves the removal of `variant_id` from various methods across multiple files in the `src/promptflow/promptflow/` directory. The `variant_id` parameter was previously used in the initialization, copying, and execution of flow runs and node runs, but it has now been removed from these processes. Removal of `variant_id` from `FlowExecutionContext`: * [`src/promptflow/promptflow/_core/flow_execution_context.py`](diffhunk://#diff-8a45b6238b72974b62aa211aec63ef4cbeadfa8277f84525442c245a16ee4461L44-L52): Removed `variant_id` from the `__init__` method, `copy` method, `_prepare_node_run` method, and `bypass_node` method. [[1]](diffhunk://#diff-8a45b6238b72974b62aa211aec63ef4cbeadfa8277f84525442c245a16ee4461L44-L52) [[2]](diffhunk://#diff-8a45b6238b72974b62aa211aec63ef4cbeadfa8277f84525442c245a16ee4461L62) [[3]](diffhunk://#diff-8a45b6238b72974b62aa211aec63ef4cbeadfa8277f84525442c245a16ee4461L119) [[4]](diffhunk://#diff-8a45b6238b72974b62aa211aec63ef4cbeadfa8277f84525442c245a16ee4461L214) Removal of `variant_id` from `RunTracker`: * [`src/promptflow/promptflow/_core/run_tracker.py`](diffhunk://#diff-3ea41c85102f8318e1dc8568de0fb0de76ce92172e506df95e6f419808eabaf8L84): Removed `variant_id` from the `start_flow_run` method and `bypass_node_run` method. [[1]](diffhunk://#diff-3ea41c85102f8318e1dc8568de0fb0de76ce92172e506df95e6f419808eabaf8L84) [[2]](diffhunk://#diff-3ea41c85102f8318e1dc8568de0fb0de76ce92172e506df95e6f419808eabaf8L102) [[3]](diffhunk://#diff-3ea41c85102f8318e1dc8568de0fb0de76ce92172e506df95e6f419808eabaf8L143) [[4]](diffhunk://#diff-3ea41c85102f8318e1dc8568de0fb0de76ce92172e506df95e6f419808eabaf8L159) Removal of `variant_id` from `FlowExecutor`: * [`src/promptflow/promptflow/executor/flow_executor.py`](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L657-R662): Removed `variant_id` from the `exec` method, `exec_line` method, `exec_line_async` method, `_exec` method, and `_exec_async` method. [[1]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L657-R662) [[2]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L674) [[3]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L688-L689) [[4]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L716) [[5]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L730) [[6]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L743-L744) [[7]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L766) [[8]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L887) [[9]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L920) [[10]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L929) [[11]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L965) [[12]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L1000) [[13]](diffhunk://#diff-faa6c81d614b7e41b18a42a93139d961d92afa9aa9dd0b72cb6b7176d7541e69L1009) # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Heyi <[email protected]>
# Description Add tracing tests to tracing_test. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes. --------- Co-authored-by: Lina Tang <[email protected]>
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes.
# Description update tool package changelog to latest version # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [x] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description update runtime changelog for 20240228.v3 # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Meng Lan <[email protected]>
# Description Improve error message when user hit authentication error when using meid_token connection local:  cloud:  # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Meng Lan <[email protected]>
# Description Refine tracing e2e test pipeline. Currently the pipeline didn't do auth step, causing failure for 3p customers. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Lina Tang <[email protected]>
# Description Rename LLM and Embedding span attributes to align with the OpenTelemetry convention. See #2266 for details. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [x] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
…2211) # Description **Extension tests:** If don't config the workspace triad, throw internal error  **Portal tests:** tool internal package pip install --upgrade promptflow_tools[azure]==0.0.313 --extra-index-url https://azuremlsdktestpypi.azureedge.net/test-promptflow/ Runtime: https://ml.azure.com/prompts/runtime/llm-use-vision-model-test/details?wsid=/subscriptions/96aede12-2f73-41cb-b983-6d11a904839b/resourceGroups/chesi-eastus/providers/Microsoft.MachineLearningServices/workspaces/chesi-promptflow&tid=72f988bf-86f1-41af-91ab-2d7cd011db47 Experiment link: https://ml.azure.com/prompts/flow/cfa1c054-13cb-4cfc-b3a6-359bf0493da9/a65f5e8f-342c-43b0-8eb0-c391d59666da/details?wsid=/subscriptions/96aede12-2f73-41cb-b983-6d11a904839b/resourceGroups/chesi-eastus/providers/Microsoft.MachineLearningServices/workspaces/chesi-promptflow&tid=72f988bf-86f1-41af-91ab-2d7cd011db47 1. stop/response_format/logit_bias/max_tokens are none:  2. llm use vision with extra parameter:  3. only vision deployment name shows for vision tool  # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: cs_lucky <[email protected]>
…2298) # Description Existing docs give wrong command line at this url; user expects to get tabular output, but command shown gives json output: https://microsoft.github.io/promptflow/how-to-guides/manage-runs.html#show-run-metrics ISSUE: #2299 The TEST PLAN: updated md file renders in vscode, copy & paste of corrected command line works when a real run name is used. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [x] **CHANGELOG is updated for new features, bug fixes or other significant changes.** (N/A) - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [x] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x ] Pull request includes test coverage for the included changes. (N/A) Co-authored-by: BobbyO <bobbyo>
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [x] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
…e closely with a standard process pool (#2234) # Description ## Background After splitting the executor and runtime into two separate containers, the implementation of batch run in the executor server also requires a process pool to handle execution requests line by line. Therefore, we need the line process pool to have the capability to process individual lines. As a process pool, it should also have the ability to submit one or multiple tasks at a time, consistent with the python process pool interface. ## Usages - Use with context ``` python with LineExecutionProcessPool(...) as pool: line_results = await pool.run(zip(line_number, batch_inputs)) ``` - Use method ``` python pool = LineExecutionProcessPool(...) pool.start() line_results = await pool.run(zip(line_number, batch_inputs)) pool.close() ``` ## Public Functions - `start`: Create task queue, input/output queues, and start processes and monitor thread pool. - `close`: Send terminate signal to monitor threads, end processes and close thread pool. - `run`: Put all line inputs to task queue and get the line results list. - `submit`: Put one line input to task queue and get one line result. ## Implementation Process  ## Main Differences from Previous Implementation - Start the monitor threads at the beginning instead of doing it in the `run` method. - The monitor thread will not exit due to an empty task queue. It exits only when the batch run times out or when a termination signal is received from the task queue. - Update `run` to async function. ## Code Changes Sunmary This pull request includes changes to various parts of the `promptflow` system, with the main focus being the addition of new functionalities and the refactoring of existing code for better performance and readability. The most significant changes include the addition of a new function to convert multimedia data to string, the introduction of a method to determine the maximum number of workers that can be created, the conversion of some functions to asynchronous, and the implementation of new exception classes. New functionalities: * [`src/promptflow/promptflow/_utils/multimedia_utils.py`](diffhunk://#diff-46760b7ed265c4b29c9cc4111fb19da9b815717e4d7180f61e0d0da023d2a8f9R181-R185): Added a new function `convert_multimedia_data_to_string` to convert multimedia data to string. * [`src/promptflow/promptflow/_utils/process_utils.py`](diffhunk://#diff-25b180c7dfe70758ae52d59d4572a4bc24fe331b916575c8286ab5af05f112e4R5-R64): Introduced a new function `get_available_max_worker_count` to determine the maximum number of workers that can be created. * [`src/promptflow/promptflow/executor/_errors.py`](diffhunk://#diff-1f02407850413baccbb33e0078d9772e15ae1c46c020dc7457529c6f7a054315R192-R197): Implemented new exception classes `ThreadCrashError` and `ProcessCrashError` to handle thread and process crashes. Refactoring: * `src/promptflow/promptflow/batch/_batch_engine.py`, `src/promptflow/promptflow/batch/_python_executor_proxy.py`: Converted some functions to asynchronous for better performance. [[1]](diffhunk://#diff-ecf0905f2116abe08b5cf2931b856bf39aa8b38a30fadd9042538d076dbfde80L349-R349) [[2]](diffhunk://#diff-8c42522c7186705de9a6651343af6006be02b7342f72467c2785f23f93ea669eL47-R47) [[3]](diffhunk://#diff-8c42522c7186705de9a6651343af6006be02b7342f72467c2785f23f93ea669eL82-R82) * [`src/promptflow/promptflow/executor/_process_manager.py`](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258R4-R23): Refactored the `AbstractProcessManager` and `SpawnProcessManager` classes, added new methods for process management, and made changes to improve readability. [[1]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258R4-R23) [[2]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258R58-R60) [[3]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258L61-R72) [[4]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258R113-R141) [[5]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258L177-R217) [[6]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258L337-R374) [[7]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258L369-R406) [[8]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258L437-R471) [[9]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258L446-R480) [[10]](diffhunk://#diff-fb203fa9d75078068758d8934fd7711b63934b15faa6cee7c3c4d16ab3347258R529-R549) * `src/promptflow/tests/executor/e2etests/test_batch_timeout.py`, `src/promptflow/tests/executor/unittests/_utils/test_process_utils.py`, `src/promptflow/tests/executor/unittests/executor/test_line_execution_process_pool.py`: Updated test cases to reflect the changes made in the codebase. [[1]](diffhunk://#diff-e79471098e1447cae89a66ed2b5e2194ca7dfa2317d994f261de3eaf1be63c16R1-R24) [[2]](diffhunk://#diff-7888cd3b10ffbfe09c0035975a082a8e36481a9aa57818da5c5e7c2a6db8dea4R1-R38) [[3]](diffhunk://#diff-8b545092ba1c4e2536841f46825c30b22ba56195407de5b115ee5ca583c67fa6L22) [[4]](diffhunk://#diff-8b545092ba1c4e2536841f46825c30b22ba56195407de5b115ee5ca583c67fa6R186-R193) Additions: * [`src/promptflow/promptflow/storage/_queue_run_storage.py`](diffhunk://#diff-e79471098e1447cae89a66ed2b5e2194ca7dfa2317d994f261de3eaf1be63c16R1-R24): Added a new storage class `QueueRunStorage` for storing run information in a queue. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes.
# Description 1. Add name in created_by info (For UI show) 2. Call method to get created_by in each request instead of starting service As info required by local to cloud trace, don't depend on Configuration file is clearer.  # All Promptflow Contribution checklist: - [X] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [X] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [X] Title of the pull request is clear and informative. - [X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: robbenwang <[email protected]>
…stants (#2303) # Description Extract some constants used by both executor server and sdk/cli to `promptflow._constants`. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes.
# Description 1. Change experiment cli "--file" to "--template" 2. pf experiment start --template, create a new experiment Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description This pull request primarily involves 2 changes: 1. specifying user agent for telemetry from local PFS without touching OperationContext, which behaves as a global context and will be shared among requests; 2. support using the same port file as msi by set up environment variable. User Agent Handling: * [`src/promptflow/promptflow/_core/operation_context.py`](diffhunk://#diff-9178fcc0ca6dea0124a2a13bb8e734a77ffbc4baadfe29d18a366452ff8f865aL156-R162): The `append_user_agent` method has been refactored to handle multiple user agents. It now splits the user agent string and appends new user agents if they are not already present. Client Request Processing: * Multiple files: The `get_client_from_request` method has been replaced with `get_client_based_on_pfs_request` across several files to improve client request processing. [[1]](diffhunk://#diff-684bb9c38065c7dd14f39bceb6ed87efa1ac3a1e70329eeb617cc484297e3134L12-R12) [[2]](diffhunk://#diff-684bb9c38065c7dd14f39bceb6ed87efa1ac3a1e70329eeb617cc484297e3134L84-R84) [[3]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL17-R21) [[4]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL55-R59) [[5]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL91-R95) [[6]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL110-R128) [[7]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL133-R137) [[8]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL144-R148) [[9]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL156-R160) [[10]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL178-R182) [[11]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL189-R193) [[12]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL204-R208) [[13]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL218-R222) [[14]](diffhunk://#diff-0286484edb80bee8c05edc9c9625ac73b1504399198a518446f38406de78004fL227-R231) [[15]](diffhunk://#diff-591332c296439bb96d6a911a1d8ff539d742038b2af7fc20fb123ef74e1f4aa2L20-R20) [[16]](diffhunk://#diff-591332c296439bb96d6a911a1d8ff539d742038b2af7fc20fb123ef74e1f4aa2L111-R111) [[17]](diffhunk://#diff-8c6363e088ff6e019485525ed5392f33486bc537dc6389b19f19254d12c86a5dL27-R27) [[18]](diffhunk://#diff-8c6363e088ff6e019485525ed5392f33486bc537dc6389b19f19254d12c86a5dR234-R254) New Methods: * [`src/promptflow/promptflow/_sdk/_utils.py`](diffhunk://#diff-47208ac35b30920275fcd5e55d662647ef360129359bdc77fddd2a2157b6f47eR795-R801): A new method `pop_current_user_agent` has been introduced to handle changes in the user agent. Test Cases: * [`src/promptflow/tests/sdk_pfs_test/e2etests/test_connection_apis.py`](diffhunk://#diff-573a672b4ba71f30cd8cbc38f2b9c77640c9e276e31da5f725bb8482b793b358R35-R50): A new test case `test_list_connections_with_different_user_agent` has been added to verify the handling of different user agents. Other Changes: * [`src/promptflow/promptflow/_sdk/_service/entry.py`](diffhunk://#diff-9c01d3da61c43e2e419a02f16c2b60c345017a801b9b5e596971fb1806c44253L107-R107): The condition for using the SDK API to start pfs has been modified to also include the `args.debug` case. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
… tool func to avoid misleading errors (#2310) # Description Issue: #2238  Fix: - if no workspace triad available, return empty list. - set default values to workspace triad params of dynamic list tool func. After fix:  # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [x] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
…ForkProcessManager instead of specifying them explicitly (#2312)
# Description This pull request includes a change to the `write` method in the `log_manager.py` file. The change adds a check for `None` to avoid errors when the line execution timeout is reached and all running nodes are cancelled. This is necessary because for synchronous tools running in a worker thread, they can't be stopped and the context won't change in the worker thread as it's a thread-local variable. # All Promptflow Contribution checklist: - [X] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [X] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [X] Title of the pull request is clear and informative. - [X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. Co-authored-by: robbenwang <[email protected]>
…default http endpoint. (#2309) # Description Do not add otlp exporter when the environment is not set. Currently we will initialize an exporter with `endpoint=None`, then we will call a default http endpoint. However we need to avoid this. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. Co-authored-by: Heyi <[email protected]>
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Signed-off-by: Brynn Yin <[email protected]>
# Description For the flow with nested tools, we will set the span name of those nested tools to node name, which is wrong, only the direct children of flow level span should set the span name to node name. In this PR, we will check whether the node name is consumed, if not the span name will set to the node name, otherwise the span name will set to the function name. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes. --------- Co-authored-by: Lina Tang <[email protected]>
# Description
Please add an informative description that covers that changes made by
the pull request and link all relevant issues.
Sample usage:
```
def my_func():
pass
pf.run(flow=my_func, data="", code="../src/")
```
This pull request primarily introduces the ability to use callable
functions as flow entries in the PromptFlow SDK. The changes allow
functions, including asynchronous functions, to be used as flow entries,
and they add the necessary checks and conversions to support this new
feature. The changes also include updates to the test suite to verify
the correct behavior of these new features.
Here are the most significant changes:
Changes to support callable flow entries:
*
[`src/promptflow/promptflow/_sdk/_pf_client.py`](diffhunk://#diff-c100045cd304dbaa66dd61b7087f78b813ce01e8566d0b823b293bb35755eccfL68-R68):
The `flow` parameter in the `run` method can now accept `Callable`
objects in addition to `str` and `PathLike` objects. Checks were added
to handle callable flow entries.
[[1]](diffhunk://#diff-c100045cd304dbaa66dd61b7087f78b813ce01e8566d0b823b293bb35755eccfL68-R68)
[[2]](diffhunk://#diff-c100045cd304dbaa66dd61b7087f78b813ce01e8566d0b823b293bb35755eccfL160-R172)
*
[`src/promptflow/promptflow/_sdk/_utils.py`](diffhunk://#diff-47208ac35b30920275fcd5e55d662647ef360129359bdc77fddd2a2157b6f47eL1168-R1199):
The `generate_yaml_entry` and `create_temp_flex_flow_yaml` functions
were updated to handle callable flow entries. A new function,
`callable_to_entry_string`, was added to convert callable objects to
entry strings.
[[1]](diffhunk://#diff-47208ac35b30920275fcd5e55d662647ef360129359bdc77fddd2a2157b6f47eL1168-R1199)
[[2]](diffhunk://#diff-47208ac35b30920275fcd5e55d662647ef360129359bdc77fddd2a2157b6f47eR1221-R1244)
[[3]](diffhunk://#diff-47208ac35b30920275fcd5e55d662647ef360129359bdc77fddd2a2157b6f47eR1221-R1244)
Updates to the test suite:
*
[`src/promptflow/tests/sdk_cli_azure_test/unittests/test_run_operations.py`](diffhunk://#diff-e61f9744c35acda81ccf7220c70aad98d0a258eec5be24997c151732a6d1ac0eR84-R95):
A new test case was added to check for the correct error when a callable
flow entry is used.
*
[`src/promptflow/tests/sdk_cli_test/e2etests/test_flow_run.py`](diffhunk://#diff-94a59a05643476869fa3c6bc45466f1582944a935488075e2e63b6a6a196958fR1297-R1372):
Several new test cases were added to verify the correct behavior when
using callable flow entries, including asynchronous functions and
functions imported from local files and modules.
*
[`src/promptflow/tests/sdk_cli_test/unittests/test_run.py`](diffhunk://#diff-c3d1c4e4539af1a59525218043dad93dc866b761a70a16a21783e57a7d0adac5R240-R274):
A new test case was added to verify the correct conversion of callable
objects to entry strings.
Other changes:
* `src/promptflow/promptflow/_sdk/_pf_client.py` and
`src/promptflow/promptflow/_sdk/_utils.py`: The `Callable` type was
added to the import statements from the `typing` module.
[[1]](diffhunk://#diff-c100045cd304dbaa66dd61b7087f78b813ce01e8566d0b823b293bb35755eccfL7-R7)
[[2]](diffhunk://#diff-47208ac35b30920275fcd5e55d662647ef360129359bdc77fddd2a2157b6f47eR21-R24)
*
[`src/promptflow/promptflow/azure/_pf_client.py`](diffhunk://#diff-15b330dbb21116427aed3fa25ad8fd89f6b536d2818378bc56a9b36d7e5d9f3eL283-R283):
The `run` method was updated to raise an error if a callable flow entry
is used, as this feature is not supported in the Azure client.
# All Promptflow Contribution checklist:
- [ ] **The pull request does not introduce [breaking changes].**
- [ ] **CHANGELOG is updated for new features, bug fixes or other
significant changes.**
- [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).**
- [ ] **Create an issue and link to the pull request to get dedicated
review from promptflow team. Learn more: [suggested
workflow](../CONTRIBUTING.md#suggested-workflow).**
## General Guidelines and Best Practices
- [ ] Title of the pull request is clear and informative.
- [ ] There are a small number of commits, each of which have an
informative message. This means that previously merged commits do not
appear in the history of the PR. For more information on cleaning up the
commits in your PR, [see this
page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
### Testing Guidelines
- [ ] Pull request includes test coverage for the included changes.
---------
Co-authored-by: Zhengfei Wang <[email protected]>
# Description Support delete traces via SDK (`pf.traces.delete`) or CLI (`pf trace delete`); valid query parameters: - Specify `run` - Specify `collection` (non `"default"`) - Specify `collection` and `started_before` (ISO 8601) This PR does what #2389 does with currently new SQLite storage schema. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes.
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues.  # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Signed-off-by: Brynn Yin <[email protected]>
# Description Fix some bugs caused by this pr #2287 1. Fix MultimediaProcessor._process_multimedia_dict_recursively. 2. Because this pr 2287 fully refactored multimedia_utils, some changes made to the file by others during development were ignored. Specific changes: - #2122 - Remove `*,` for `get_file_reference_encoder` method. - Add parameter: `use_absolute_path` for `persist_multimedia_data` method. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description Add otel trace tests for batch run. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes. --------- Co-authored-by: Lina Tang <[email protected]>
# Description Several trace view UI enhancement # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Zhengfei Wang <[email protected]>
# Description Use batch run id as evaluation dict's key for batch run. https://int.ml.azure.com/prompts/trace/list?wsid=/subscriptions/96aede12-2f73-41cb-b983-6d11a904839b/resourceGroups/promptflow/providers/Microsoft.MachineLearningServices/workspaces/promptflow-canary-dev&searchText={%22batchRunId%22:%22test_main_variant_0_20240401_095544_611576%22}&tid=72f988bf-86f1-41af-91ab-2d7cd011db47 Current UX can't show eval output in LineSummary list after this change. Because UX will update to new UI soon, we accept this breaking change.   # All Promptflow Contribution checklist: - [X] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [X] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [X] Title of the pull request is clear and informative. - [X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [X] Pull request includes test coverage for the included changes. --------- Co-authored-by: robbenwang <[email protected]>
# Description For eval line run(s), use run name, instead of root span name. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes.
# Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. # Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Philip Gao <[email protected]>
github-actions
bot
added
documentation
promptflow-tracing test result 12 files 12 suites 3m 48s ⏱️ Results for commit 3af352f. ♻️ This comment has been updated with latest results. |
SDK CLI Global Config Test Result main1 tests 0 ✅ 3s ⏱️ For more details on these errors, see this check. Results for commit 3af352f. |
promptflow-core test result 12 files 12 suites 3m 4s ⏱️ Results for commit 3af352f. |
singankit
merged commit 27db35f
into
users/singankit/promptflow-eval
161 of 185 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
All Promptflow Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines