Merge branch 'chenlu/tutorial_prompt_eval' of https://github.com/micr…

…osoft/promptflow into chenlu/tutorial_prompt_eval

docs/how-to-guides/add-conditional-control-to-a-flow.md

@@ -55,11 +55,14 @@ activate:
 
     ![output_bypassed](../media/how-to-guides/conditional-flow-with-activate/output_bypassed.png)
 
-3. In a conditional flow, if a node is bypassed, it status will be marked as `Bypassed`, as shown in the figure below. There are three situations in which the node will be bypassed.
-  ![bypassed_nodes](../media/how-to-guides/conditional-flow-with-activate/bypassed_nodes.png)
+3. In a conditional flow, if a node has activate config, we will always use this config to determine whether the node should be bypassed. If a node is bypassed, its status will be marked as "Bypassed", as shown in the figure below Show. There are three situations in which a node is bypassed.
 
+    ![bypassed_nodes](../media/how-to-guides/conditional-flow-with-activate/bypassed_nodes.png)
 
-    (1) If a node has activate config and the value of `activate.when` is equals to `activate.is`, it will be bypassed.
+
+    (1) If a node has activate config and the value of `activate.when` is not equals to `activate.is`, it will be bypassed. If you want to fore a node to always be executed, you can set the activate config to `when dummy is dummy` which always meets the activate condition.
+
+    ![activate_condition_always_met](../media/how-to-guides/conditional-flow-with-activate/activate_condition_always_met.png)
 
     (2) If a node has activate config and the node pointed to by `activate.when` is bypassed, it will be bypassed.
 

docs/how-to-guides/develop-a-tool/create-and-use-tool-package.md

@@ -118,7 +118,7 @@ hello-world-proj/
 ![auto-list-tool-in-extension](../../media/how-to-guides/develop-a-tool/auto-list-tool-in-extension.png)
 
 
-## FAQ
+## FAQs
 ### Why is my custom tool not showing up in the UI?
 Confirm that the tool YAML files are included in your custom tool package. You can add the YAML files to [MANIFEST.in](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/MANIFEST.in) and include the package data in [setup.py](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/setup.py).
 Alternatively, you can test your tool package using the script below to ensure that you've packaged your tool YAML files and configured the package tool entry point correctly.
@@ -165,3 +165,6 @@ Alternatively, you can test your tool package using the script below to ensure t
 [Customize your tool icon](add-a-tool-icon.md)  
 [Add category and tags for tool](add-category-and-tags-for-tool.md)  
 [Use file path as tool input](use-file-path-as-tool-input.md)  
+[Customize an LLM Tool](customize_an_llm_tool.md)  
+[Creating Cascading Tool Inputs](create-cascading-tool-inputs.md)   
+[Create and Use Your Own Custom Strong Type Connection](create-your-own-custom-strong-type-connection.md) 

docs/how-to-guides/develop-a-tool/create-cascading-tool-inputs.md

@@ -0,0 +1,136 @@
+# Creating Cascading Tool Inputs
+
+Cascading input settings are useful when the value of one input field determines which subsequent inputs are shown. This makes the input process more streamlined, user-friendly, and error-free. This guide will walk through how to create cascading inputs for your tools.
+
+## Prerequisites
+Please make sure you have the latest version of [Prompt flow for VS Code](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) installed (v1.2.0+).
+
+
+## Create a tool with cascading inputs
+We'll build out an example tool to show how cascading inputs work. The `student_id` and `teacher_id` inputs will be controlled by the value selected for the `user_type` input. Here's how to configure this in the tool code and YAML.
+
+1. Develop the tool function, following the [cascading inputs example](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/my_tool_package/tools/tool_with_cascading_inputs.py). Key points:
+    * Use the `@tool` decorator to mark the function as a tool.
+    * Define `UserType` as an Enum class, as it accepts only a specific set of fixed values in this example.
+    * Conditionally use inputs in the tool logic based on `user_type`.
+
+```python
+from enum import Enum
+
+from promptflow import tool
+
+
+class UserType(str, Enum):
+    STUDENT = "student"
+    TEACHER = "teacher"
+
+
+@tool
+def my_tool(user_type: Enum, student_id: str = "", teacher_id: str = "") -> str:
+    """This is a dummy function to support cascading inputs.
+
+    :param user_type: user type, student or teacher.
+    :param student_id: student id.
+    :param teacher_id: teacher id.
+    :return: id of the user.
+    If user_type is student, return student_id.
+    If user_type is teacher, return teacher_id.
+    """
+    if user_type == UserType.STUDENT:
+        return student_id
+    elif user_type == UserType.TEACHER:
+        return teacher_id
+    else:
+        raise Exception("Invalid user.")
+```
+
+2. Generate a starting YAML for your tool per the [tool package guide](create-and-use-tool-package.md), then update it to enable cascading:
+
+    Add `enabled_by` and `enabled_by_value` to control visibility of dependent inputs. See the [example YAML](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/my_tool_package/yamls/tool_with_cascading_inputs.yaml) for reference.
+
+    * The `enabled_by` attribute specifies the input field, which must be an enum type, that controls the visibility of the dependent input field.
+
+    * The `enabled_by_value` attribute defines the accepted enum values from the `enabled_by` field that will make this dependent input field visible.
+    > Note: `enabled_by_value` takes a list, allowing multiple values to enable an input.
+
+```yaml
+my_tool_package.tools.tool_with_cascading_inputs.my_tool:
+  function: my_tool
+  inputs:
+    user_type:
+      type:
+      - string
+      enum:
+        - student
+        - teacher
+    student_id:
+      type:
+      - string
+      # This input is enabled by the input "user_type".
+      enabled_by: user_type
+      # This input is enabled when "user_type" is "student".
+      enabled_by_value: [student]
+    teacher_id:
+      type:
+        - string
+      enabled_by: user_type
+      enabled_by_value: [teacher]
+  module: my_tool_package.tools.tool_with_cascading_inputs
+  name: My Tool with Cascading Inputs
+  description: This is my tool with cascading inputs
+  type: python
+```
+
+## Use the tool in VS Code
+Once you package and share your tool, you can use it in VS Code per the [tool package guide](create-and-use-tool-package.md). We have a [demo flow](https://github.com/microsoft/promptflow/tree/main/examples/tools/use-cases/cascading-inputs-tool-showcase) you can try.
+
+Before selecting a `user_type`, the `student_id` and `teacher_id` inputs are hidden. Once you pick the `user_type`, the corresponding input appears.
+![before_user_type_selected.png](../../media/how-to-guides/develop-a-tool/before_user_type_selected.png)
+![after_user_type_selected_with_student.png](../../media/how-to-guides/develop-a-tool/after_user_type_selected_with_student.png)
+![after_user_type_selected_with_teacher.png](../../media/how-to-guides/develop-a-tool/after_user_type_selected_with_teacher.png)
+
+
+
+## FAQs
+### How do I create multi-layer cascading inputs?
+If you are dealing with multiple levels of cascading inputs, you can effectively manage the dependencies between them by using the `enabled_by` and `enabled_by_value` attributes. For example:
+```yaml
+my_tool_package.tools.tool_with_multi_layer_cascading_inputs.my_tool:
+  function: my_tool
+  inputs:
+    event_type:
+      type:
+      - string
+      enum:
+        - corporate
+        - private
+    corporate_theme:
+      type:
+      - string
+      # This input is enabled by the input "event_type".
+      enabled_by: event_type
+      # This input is enabled when "event_type" is "corporate".
+      enabled_by_value: [corporate]
+      enum:
+        - seminar
+        - team_building
+    seminar_location:
+      type:
+      - string
+      # This input is enabled by the input "corporate_theme".
+      enabled_by: corporate_theme
+      # This input is enabled when "corporate_theme" is "seminar".
+      enabled_by_value: [seminar]
+    private_theme:
+      type:
+        - string
+      # This input is enabled by the input "event_type".
+      enabled_by: event_type
+      # This input is enabled when "event_type" is "private".
+      enabled_by_value: [private]
+  module: my_tool_package.tools.tool_with_multi_layer_cascading_inputs
+  name: My Tool with Multi-Layer Cascading Inputs
+  description: This is my tool with multi-layer cascading inputs
+  type: python
+```
+Inputs will be enabled in a cascading way based on selections.

docs/how-to-guides/develop-a-tool/create-your-own-custom-strong-type-connection.md

@@ -0,0 +1,100 @@
+# Create and Use Your Own Custom Strong Type Connection
+Connections provide a secure method for managing credentials for external APIs and data sources in Prompt flow. This guide explains how to create and use a custom strong type connection.
+
+## What is a Custom Strong Type Connection?
+A custom strong type connection in Prompt flow allows you to define a custom connection class with strongly typed keys. This provides the following benefits:
+
+* Enhanced user experience - no need to manually enter connection keys.
+* Rich intellisense experience - defining key types enables real-time suggestions and auto-completion of available keys as you work in VS Code.
+* Central location to view available keys and data types.
+
+For other connections types, please refer to [Connections](https://microsoft.github.io/promptflow/concepts/concept-connections.html).
+
+## Prerequisites
+- Please ensure that your [Prompt flow for VS Code](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) is updated to at least version 1.2.1.
+- Please install promptflow package and ensure that its version is 0.1.0b8 or later.
+  ```
+  pip install promptflow>=0.1.0b8
+  ```
+
+## Create a custom strong type connection
+Follow these steps to create a custom strong type connection:
+
+1. Define a Python class inheriting from `CustomStrongTypeConnection`.
+  > [!Note] Please avoid using the `CustomStrongTypeConnection` class directly.
+
+2. Use the Secret type to indicate secure keys. This enhances security by scrubbing secret keys.
+
+3. Document with docstrings explaining each key.
+
+For example:
+
+```python
+from promptflow.connections import CustomStrongTypeConnection
+from promptflow.contracts.types import Secret
+
+
+class MyCustomConnection(CustomStrongTypeConnection):
+    """My custom strong type connection.
+
+    :param api_key: The api key.
+    :type api_key: Secret
+    :param api_base: The api base.
+    :type api_base: String
+    """
+    api_key: Secret
+    api_base: str = "This is a fake api base."
+
+```
+
+See [this example](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/my_tool_package/tools/tool_with_custom_strong_type_connection.py) for a complete implementation.
+
+## Use the connection in a flow
+Once you create a custom strong type connection, here are two ways to use it in your flows:
+
+### With Package Tools:
+
+1. Refer to the [Create and Use Tool Package](create-and-use-tool-package.md#create-custom-tool-package) to build and install your tool package containing the connection.
+
+2. Develop a flow with custom tools. Please take [this folder](https://github.com/microsoft/promptflow/tree/main/examples/tools/use-cases/custom-strong-type-connection-package-tool-showcase) as an example.
+
+3. Create a custom strong type connection using one of the following methods:
+    - If the connection type hasn't been created previously, click the 'Add connection' button to create the connection.
+    ![create_custom_strong_type_connection_in_node_interface](../../media/how-to-guides/develop-a-tool/create_custom_strong_type_connection_in_node_interface.png)
+    - Click the 'Create connection' plus sign in the CONNECTIONS section.
+    ![create_custom_strong_type_connection_add_sign](../../media/how-to-guides/develop-a-tool/create_custom_strong_type_connection_add_sign.png)
+    - Click 'Create connection' plus sign in the Custom category.
+    ![create_custom_strong_type_connection_in_custom_category](../../media/how-to-guides/develop-a-tool/create_custom_strong_type_connection_in_custom_category.png) 
+
+4. Fill in the `values` starting with `to-replace-with` in the connection template.
+![custom_strong_type_connection_template](../../media/how-to-guides/develop-a-tool/custom_strong_type_connection_template.png)
+
+5. Run the flow with the created custom strong type connection.
+![use_custom_strong_type_connection_in_flow](../../media/how-to-guides/develop-a-tool/use_custom_strong_type_connection_in_flow.png)
+
+### With Script Tools:
+
+1. Develop a flow with python script tools. Please take [this folder](https://github.com/microsoft/promptflow/tree/main/examples/tools/use-cases/custom-strong-type-connection-script-tool-showcase) as an example.
+
+2. Create a `CustomConnection`. Fill in the `keys` and `values` in the connection template.
+  ![custom](../../media/how-to-guides/develop-a-tool/custom_connection_template.png)
+
+3. Run the flow with the created custom connection.
+  ![use_custom_connection_in_flow](../../media/how-to-guides/develop-a-tool/use_custom_connection_in_flow.png)
+
+
+## Local to cloud
+When creating the necessary connections in Azure AI, you will need to create a `CustomConnection`. In the node interface of your flow, this connection will be displayed as the `CustomConnection` type.
+
+Please refer to [Run prompt flow in Azure AI](https://microsoft.github.io/promptflow/cloud/azureai/quick-start.html) for more details.
+
+Here is an example command:
+```
+pfazure run create --subscription 96aede12-2f73-41cb-b983-6d11a904839b -g promptflow -w my-pf-eus --flow D:\proj\github\ms\promptflow\examples\flows\standard\flow-with-package-tool-using-custom-strong-type-connection --data D:\proj\github\ms\promptflow\examples\flows\standard\flow-with-package-tool-using-custom-strong-type-connection\data.jsonl --runtime test-compute
+```
+
+## FAQs
+
+### I followed the steps to create a custom strong type connection, but it's not showing up. What could be the issue?
+
+Once the new tool package is installed in your local environment, a window reload is necessary. This action ensures that the new tools and custom strong type connections become visible and accessible.

docs/how-to-guides/develop-a-tool/customize_an_llm_tool.md

@@ -0,0 +1,60 @@
+# Customize an LLM Tool
+In this document, we will guide you through the process of customizing an LLM tool, allowing users to seamlessly connect to a large language model with prompt tuning experience using a `PromptTemplate`.
+
+## Prerequisites
+- Please ensure that your [Prompt flow for VS Code](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) is updated to version 1.2.0 or later.
+
+## How to customize an LLM tool
+Here we use [an existing tool package](https://github.com/microsoft/promptflow/tree/main/examples/tools/tool-package-quickstart/my_tool_package) as an example. If you want to create your own tool, please refer to [create and use tool package](create-and-use-tool-package.md).  
+
+1. Develop the tool code as in [this example](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/my_tool_package/tools/tool_with_custom_llm_type.py).
+- Add a `CustomConnection` input to the tool, which is used to authenticate and establish a connection to the large language model.
+- Add a `PromptTemplate` input to the tool, which serves as an argument to be passed into the large language model.
+
+    ```python
+    from jinja2 import Template
+    from promptflow import tool
+    from promptflow.connections import CustomConnection
+    from promptflow.contracts.types import PromptTemplate
+
+
+    @tool
+    def my_tool(connection: CustomConnection, prompt: PromptTemplate, **kwargs) -> str:
+        # Customize your own code to use the connection and prompt here.
+        rendered_prompt = Template(prompt, trim_blocks=True, keep_trailing_newline=True).render(**kwargs)
+        return rendered_prompt
+    ```
+
+2. Generate the custom LLM tool YAML.  
+    Run the command below in your tool project directory to automatically generate your tool YAML, use _-t "custom_llm"_ or _--tool-type "custom_llm"_ to indicate this is a custom LLM tool:  
+    ```
+    python <path-to-scripts>\tool\generate_package_tool_meta.py -m <tool_module> -o <tool_yaml_path> -t "custom_llm"
+    ```
+    Here we use [an existing tool](https://github.com/microsoft/promptflow/blob/main/examples/tools/tool-package-quickstart/my_tool_package/yamls/tool_with_custom_llm_type.yaml) as an example.
+    ```
+    cd D:\proj\github\promptflow\examples\tools\tool-package-quickstart
+
+    python D:\proj\github\promptflow\scripts\tool\generate_package_tool_meta.py -m my_tool_package.tools.tool_with_custom_llm_type -o my_tool_package\yamls\tool_with_custom_llm_type.yaml -n "My Custom LLM Tool" -d "This is a tool to demonstrate how to customize an LLM tool with a PromptTemplate." -t "custom_llm"
+    ```
+    This command will generate a YAML file as follows:
+
+    ```yaml
+    my_tool_package.tools.tool_with_custom_llm_type.my_tool:
+    name: My Custom LLM Tool
+    description: This is a tool to demonstrate how to customize an LLM tool with a PromptTemplate.
+    # The type is custom_llm.
+    type: custom_llm
+    module: my_tool_package.tools.tool_with_custom_llm_type
+    function: my_tool
+    inputs:
+        connection:
+        type:
+            - CustomConnection
+    ```
+
+## Use the tool in VS Code
+Follow the steps to [build and install your tool package](create-and-use-tool-package.md#build-and-share-the-tool-package) and [use your tool from VS Code extension](create-and-use-tool-package.md#use-your-tool-from-vscode-extension).  
+
+Here we use an existing flow to demonstrate the experience, open [this flow](https://github.com/microsoft/promptflow/blob/main/examples/tools/use-cases/custom_llm_tool_showcase/flow.dag.yaml) in VS Code extension.  
+- There is a node named "my_custom_llm_tool" with a prompt template file. You can either use an existing file or create a new one as the prompt template file.  
+![use_my_custom_llm_tool](../../media/how-to-guides/develop-a-tool/use_my_custom_llm_tool.png)

docs/how-to-guides/develop-a-tool/index.md

@@ -9,4 +9,7 @@ create-and-use-tool-package
 add-a-tool-icon
 add-category-and-tags-for-tool
 use-file-path-as-tool-input
-```
+customize_an_llm_tool
+create-cascading-tool-inputs
+create-your-own-custom-strong-type-connection
+```

docs/index.md

@@ -41,9 +41,9 @@ This documentation site contains guides for prompt flow [sdk, cli](https://pypi.
     Articles guide user to complete a specific task in prompt flow.<br/><br/>
     - [Develop a flow](how-to-guides/develop-a-flow/index.md)<br/>
     - [Initialize and test a flow](how-to-guides/init-and-test-a-flow.md)<br/>
-    - [Add conditional control to a flow](how-to-guides/add-conditional-control-to-a-flow.md)<br/>
     - [Run and evaluate a flow](how-to-guides/run-and-evaluate-a-flow/index.md)<br/>
     - [Tune prompts using variants](how-to-guides/tune-prompts-with-variants.md)<br/>
+    - [Develop custom tool](how-to-guides/develop-a-tool/create-and-use-tool-package.md)<br/>
     - [Deploy a flow](how-to-guides/deploy-a-flow/index.md)<br/>
   "
 ```