Merge branch 'main' into users/gewoods/open_source_llm/initial

microsoft · Sep 28, 2023 · a108fd9 · a108fd9
2 parents 5e7e114 + 037f627
commit a108fd9
Show file tree

Hide file tree

Showing 20 changed files with 189 additions and 15 deletions.
diff --git a/.cspell.json b/.cspell.json
@@ -127,6 +127,7 @@
     "pysqlite",
     "AADSTS700082",
     "levelno",
+    "LANCZOS",
     "Mobius"
   ],
   "allowCompoundWords": true

diff --git a/docs/README.md b/docs/README.md
@@ -10,7 +10,7 @@ Below is a table of important doc pages.
 |----------------|----------------|
 |Quick start|[Get started with prompt flow](./how-to-guides/quick-start.md)|
 |Concepts|[Flows](./concepts/concept-flows.md)<br> [Tools](./concepts/concept-tools.md)<br> [Connections](./concepts/concept-connections.md)<br> [Variants](./concepts/concept-variants.md)<br> |
-|How-to guides|[How to initialize and test a flow](./how-to-guides/init-and-test-a-flow.md) <br>[How to run and evaluate a flow](./how-to-guides/run-and-evaluate-a-flow.md)<br> [How to tune prompts using variants](./how-to-guides/tune-prompts-with-variants.md)<br>[How to deploy a flow](./how-to-guides/deploy-a-flow/index.md)<br>[How to create and use your own tool package](./how-to-guides/how-to-create-and-use-your-own-tool-package.md)|
+|How-to guides|[How to initialize and test a flow](./how-to-guides/init-and-test-a-flow.md) <br>[How to run and evaluate a flow](./how-to-guides/run-and-evaluate-a-flow.md)<br> [How to tune prompts using variants](./how-to-guides/tune-prompts-with-variants.md)<br>[How to deploy a flow](./how-to-guides/deploy-a-flow/index.md)<br>[How to create and use your own tool package](./how-to-guides/develop-a-tool/create-and-use-tool-package.md)|
 |Tools reference|[LLM tool](./tools-reference/llm-tool.md)<br> [Prompt tool](./tools-reference/prompt-tool.md)<br> [Python tool](./tools-reference/python-tool.md)<br> [SERP API tool](./tools-reference/serp-api-tool.md)<br> [Embedding tool](./tools-reference/embedding_tool.md)||
 
 

diff --git a/docs/concepts/concept-tools.md b/docs/concepts/concept-tools.md
@@ -22,7 +22,7 @@ Our partners also contributes other useful tools for advanced scenarios, here ar
 ## Custom tools
 
 You can create your own tools that can be shared with your team or anyone in the world. 
-Learn more on [Custom tool package creation and usage](../how-to-guides/how-to-create-and-use-your-own-tool-package.md)
+Learn more on [Create and Use Tool Package](../how-to-guides/develop-a-tool/create-and-use-tool-package.md)
 
 ## Next steps
 

diff --git a/docs/how-to-guides/develop-a-tool/add-a-tool-icon.md b/docs/how-to-guides/develop-a-tool/add-a-tool-icon.md
@@ -0,0 +1,58 @@
+# Add a Tool Icon
+A tool icon serves as a graphical representation of your tool in the user interface (UI). Follow this guidance to add a custom tool icon when developing your own tool package.  
+
+Adding a custom tool icon is optional. If you do not provide one, the system uses a default icon.
+
+## Prerequisites
+
+- Create a tool package as described in [Create and Use Tool Package](create-and-use-tool-package.md).
+- Prepare a custom tool icon image. The image should meet following requirements:
+  - The image should be in `PNG` or `JPG` format.
+  - 16x16 pixel image to avoid distortion from resizing.
+  - Avoid complex images with much detail or contrast as they may not resize well.   
+    You could see for example.
+- Install dependencies of generating icon data URI:
+
+  ```
+  pip install pillow
+  ```
+## Generate a tool icon data URI
+Run below command under the root folder to generate a data URI for your custom tool icon. Make sure the output file has an `.html` extension, this makes it easier to check the image's data URI: 
+```
+python <path-to-scripts>\tool\convert_image_to_data_url.py --image-path <image_input_path> -o <html_output_path>
+```     
+For example:
+```
+python D:\proj\github\promptflow\scripts\tool\convert_image_to_data_url.py --image-path D:\proj\github\promptflow\examples\tools\tool-package-quickstart\my_tool_package\icons\custom-tool-icon.png -o output.html
+```
+The content of `output.html` looks like the following, and you can open it in a web browser to see the result.
+```html
+<html>
+<body>
+<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAACq0lEQVR4nKWTPWxTVxiGn3N/bF/fGyd2YxsQ5LdSmoqGgKJMEQNZYGBFkVhY2JGYmFG3ioWZpWukqqpaqaVKq6pFIAECAkSxMA6RHZP4Jnbs+O/6xvdjcCLKT6ee7ZwjvfrOc55XiYjwP5bx740IZDf3+PVZicdrVeK2yfzJJBem02iaQv1XQCCCCNz+Lce91R1mRvtYd5uEDIfl9SqpWIjZLxOIgPooRQtE0JQiU6xx91mJfNkjZoX47vIkM2Nx6l6Xmz9kWHywgVIQBB++WImI1Nv7fP/XOqah0fKFdH8YQwXcf1Vh6WWZTrfLaDLK4rVZrJD+wSQGwJrbQtc0rs4PAXDr5xy/PHW5NJsmGQuhNI0/XrisFmtMjwxwOLVCYXTaTdq+kHagXq0iAo4phE2dn564XD8/zLlTRwn8gK1dQaHQtfcgDDMcwQo1Wc43mEp16YpibdsjEKHdEX5/8YZEpIhjCckBi9a+ibfvETEsIobdY1Bp+Pz4cAvP67C522IsbeN1A0zd5r77LWF7hebe1xxJrzKRmON56W/OHDnHwskbaCIQt03OTsbJljxeuz4rxSYXp2JcmYszedQhrNscj/ehJIKuQpiaBegHEFVPoOHBKOPpKMXdNtlSmzt/bpC0XTb9LxgcmGDq2CT5mpC0hhiO1UhGe8ANBYgCQ1dcnR9iJGnxT6ZMrtLmbV1H78/QrD0nagQ82ljCP+HzqLBEsB8wP7bQ+8ZDpoauuHA6xfnpFA3Px4mY3M2cJbeTZjTxFQYm44lv0MRkPDH1aRcOtdaUwon0rgrbBdbd10S1AXJbWRxzkLXNLDEz1VP54wDtQLHuQUl36xUKpTzl6jYFN89OdYeCm6eyV3mv8mdKxuFxueHS8PawTJuW3yAacmh26jiRfhL2IO8AhSUo7nmFnjUAAAAASUVORK5CYII=" alt="My Image">
+</body>
+</html>
+```
+
+## Use the tool icon data URI in the tool YAML
+In the auto-generated tool YAML file, customize the tool icon by adding the data URI:
+```yaml
+hello_world.tools.hello_world_tool.get_greeting_message
+  function: get_greeting_message
+  inputs:
+    connection:
+      type:
+      - CustomConnection
+    input_text:
+      type:
+      - string
+  module: hello_world.tools.hello_world_tool
+  name: Hello World Tool
+  description: This is hello world tool
+  type: python
+  icon: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAACq0lEQVR4nKWTPWxTVxiGn3N/bF/fGyd2YxsQ5LdSmoqGgKJMEQNZYGBFkVhY2JGYmFG3ioWZpWukqqpaqaVKq6pFIAECAkSxMA6RHZP4Jnbs+O/6xvdjcCLKT6ee7ZwjvfrOc55XiYjwP5bx740IZDf3+PVZicdrVeK2yfzJJBem02iaQv1XQCCCCNz+Lce91R1mRvtYd5uEDIfl9SqpWIjZLxOIgPooRQtE0JQiU6xx91mJfNkjZoX47vIkM2Nx6l6Xmz9kWHywgVIQBB++WImI1Nv7fP/XOqah0fKFdH8YQwXcf1Vh6WWZTrfLaDLK4rVZrJD+wSQGwJrbQtc0rs4PAXDr5xy/PHW5NJsmGQuhNI0/XrisFmtMjwxwOLVCYXTaTdq+kHagXq0iAo4phE2dn564XD8/zLlTRwn8gK1dQaHQtfcgDDMcwQo1Wc43mEp16YpibdsjEKHdEX5/8YZEpIhjCckBi9a+ibfvETEsIobdY1Bp+Pz4cAvP67C522IsbeN1A0zd5r77LWF7hebe1xxJrzKRmON56W/OHDnHwskbaCIQt03OTsbJljxeuz4rxSYXp2JcmYszedQhrNscj/ehJIKuQpiaBegHEFVPoOHBKOPpKMXdNtlSmzt/bpC0XTb9LxgcmGDq2CT5mpC0hhiO1UhGe8ANBYgCQ1dcnR9iJGnxT6ZMrtLmbV1H78/QrD0nagQ82ljCP+HzqLBEsB8wP7bQ+8ZDpoauuHA6xfnpFA3Px4mY3M2cJbeTZjTxFQYm44lv0MRkPDH1aRcOtdaUwon0rgrbBdbd10S1AXJbWRxzkLXNLDEz1VP54wDtQLHuQUl36xUKpTzl6jYFN89OdYeCm6eyV3mv8mdKxuFxueHS8PawTJuW3yAacmh26jiRfhL2IO8AhSUo7nmFnjUAAAAASUVORK5CYII=
+```
+
+## Verify the tool icon in VS Code extension
+Follow [steps](create-and-use-tool-package.md#use-your-tool-from-vscode-extension) to use your tool from VS Code extension. Your tool displays with the custom icon:  
+![custom-tool-with-icon-in-extension](../../media/contributing/custom-tool-with-icon-in-extension.png)
diff --git a/...o-create-and-use-your-own-tool-package.md → ...lop-a-tool/create-and-use-tool-package.md b/...o-create-and-use-your-own-tool-package.md → ...lop-a-tool/create-and-use-tool-package.md
@@ -1,10 +1,10 @@
-# Custom tool package creation and usage
+# Create and Use Tool Package
 In this document, we will guide you through the process of developing your own tool package, offering detailed steps and advice on how to utilize your creation.
 
 The custom tool is the prompt flow tool developed by yourself. If you find it useful, you can follow this guidance to make it a tool package. This will enable you to conveniently reuse it, share it with your team, or distribute it to anyone in the world.
 
 After successful installation of the package, your custom "tool" will show up in VSCode extension as below: 
-![custom-tool-list](../media/contributing/custom-tool-list-in-extension.png)
+![custom-tool-list](../../media/contributing/custom-tool-list-in-extension.png)
 
 ## Create your own tool package
 Your tool package should be a python package. To try it quickly, just use [my-tools-package 0.0.1](https://pypi.org/project/my-tools-package/) and skip this section.
@@ -115,7 +115,7 @@ hello-world-proj/
    ``` 
 
 * Step3: Go to the extension and open one flow folder. Click 'flow.dag.yaml' and preview the flow. Next, click `+` button and you will see your tools. You may need to reload the windows to clean previous cache if you don't see your tool in the list.
-![auto-list-tool-in-extension](../media/contributing/auto-list-tool-in-extension.png)
+![auto-list-tool-in-extension](../../media/contributing/auto-list-tool-in-extension.png)
 
 
 ## FAQ

diff --git a/docs/how-to-guides/develop-a-tool/index.md b/docs/how-to-guides/develop-a-tool/index.md
@@ -0,0 +1,10 @@
+# Develop a tool
+We provide guides on how to develop a tool and use it.
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+
+create-and-use-tool-package
+add-a-tool-icon
+```
diff --git a/docs/how-to-guides/index.md b/docs/how-to-guides/index.md
@@ -14,7 +14,7 @@ enable-streaming-mode
 manage-connections
 manage-runs
 column-mapping
-how-to-create-and-use-your-own-tool-package
+develop-a-tool/index
 faq
 
 ```

diff --git a/docs/media/contributing/custom-tool-with-icon-in-extension.png b/docs/media/contributing/custom-tool-with-icon-in-extension.png
diff --git a/examples/flows/standard/basic/README.md b/examples/flows/standard/basic/README.md
@@ -51,7 +51,7 @@ pf run create --flow . --data ./data.jsonl --stream
 pf run list
 
 # get a sample run name
-name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_default")) | .name'| head -n 1 | tr -d '"')
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_variant_0")) | .name'| head -n 1 | tr -d '"')
 
 # show specific run detail
 pf run show --name $name
@@ -92,7 +92,7 @@ pf run create --flow . --data ./data.jsonl --stream --environment-variables AZUR
 pf run create --file run.yml --stream
 
 # show outputs
-name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_default")) | .name'| head -n 1 | tr -d '"')
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_variant_0")) | .name'| head -n 1 | tr -d '"')
 pf run show-details --name $name
 ```
 
@@ -118,7 +118,7 @@ pfazure run create --file run.yml --stream --runtime demo-mir
 pfazure run list -r 3
 
 # get a sample run name
-name=$(pfazure run list -r 100 | jq '.[] | select(.name | contains("basic_default")) | .name'| head -n 1 | tr -d '"')
+name=$(pfazure run list -r 100 | jq '.[] | select(.name | contains("basic_variant_0")) | .name'| head -n 1 | tr -d '"')
 
 # show specific run detail
 pfazure run show --name $name

diff --git a/examples/tools/tool-package-quickstart/my_tool_package/icons/custom-tool-icon.png b/examples/tools/tool-package-quickstart/my_tool_package/icons/custom-tool-icon.png
diff --git a/scripts/tool/convert_image_to_data_url.py b/scripts/tool/convert_image_to_data_url.py
@@ -0,0 +1,64 @@
+import argparse
+import base64
+import os
+import io
+from PIL import Image
+
+
+def get_image_size(image_path):
+    with Image.open(image_path) as img:
+        width, height = img.size
+    return width, height
+
+
+def get_image_storage_size(image_path):
+    file_size_bytes = os.path.getsize(image_path)
+    file_size_mb = file_size_bytes / (1024 * 1024)
+    return file_size_mb
+
+
+def image_to_data_url(image_path):
+    with open(image_path, "rb") as image_file:
+        # Create a BytesIO object from the image file
+        image_data = io.BytesIO(image_file.read())
+
+    # Open the image and resize it
+    img = Image.open(image_data)
+    if img.size != (16, 16):
+        img = img.resize((16, 16), Image.Resampling.LANCZOS)
+
+    # Save the resized image to a data URL
+    buffered = io.BytesIO()
+    img.save(buffered, format="PNG")
+    img_str = base64.b64encode(buffered.getvalue())
+    data_url = 'data:image/png;base64,' + img_str.decode('utf-8')
+
+    return data_url
+
+
+def create_html_file(data_uri, output_path):
+    html_content = '<html>\n<body>\n<img src="{}" alt="My Image">\n</body>\n</html>'.format(data_uri)
+
+    with open(output_path, 'w') as file:
+        file.write(html_content)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--image-path",
+        type=str,
+        required=True,
+        help="Your image input path",
+    )
+    parser.add_argument(
+        "--output",
+        "-o",
+        type=str,
+        required=True,
+        help="Your image output path",
+    )
+    args = parser.parse_args()
+    data_url = image_to_data_url(args.image_path)
+    print("Your image data uri: \n{}".format(data_url))
+    create_html_file(data_url, args.output)
diff --git a/src/promptflow-tools/README.md b/src/promptflow-tools/README.md
@@ -7,4 +7,4 @@
 
 Tools are the fundamental building blocks of a flow in Azure Machine Learning prompt flow. Each tool is a simple, executable unit with a specific function, allowing users to perform various tasks. By combining different tools, users can create a flow that accomplishes a wide range of goals. One of the key benefit of prompt flow tools is their seamless integration with third-party APIs and python open source packages. This not only improves the functionality of large language models but also makes the development process more efficient.
 
-In this package, we provide a set of builtin tools of prompt flow, which are the most commonly used tools in the development of AI applications. We also provide a flexible way for users to create their own tools and share them with others. See [Custom tool package creation and usage](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/how-to-create-and-use-your-own-tool-package.md) for more details.
+In this package, we provide a set of builtin tools of prompt flow, which are the most commonly used tools in the development of AI applications. We also provide a flexible way for users to create their own tools and share them with others. See [Create and Use Tool Package](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/develop-a-tool/create-and-use-tool-package.md) for more details.
diff --git a/src/promptflow/CHANGELOG.md b/src/promptflow/CHANGELOG.md
@@ -1,6 +1,11 @@
 # Release History
 
-## 0.1.0b7 (Upcoming)
+## 0.1.0b7.post1 (2023.09.28)
+
+### Bug Fixed
+- Fix extra dependency bug when importing `promptflow` without `azure-ai-ml` installed.
+
+## 0.1.0b7 (2023.09.27)
 
 ### Features Added
 

diff --git a/src/promptflow/promptflow/_cli/_params.py b/src/promptflow/promptflow/_cli/_params.py
@@ -4,7 +4,8 @@
 
 import argparse
 
-from azure.ai.ml.constants._common import MAX_LIST_CLI_RESULTS
+# TODO: avoid azure dependency here
+MAX_LIST_CLI_RESULTS = 50
 
 
 class AppendToDictAction(argparse._AppendAction):  # pylint: disable=protected-access

diff --git a/src/promptflow/promptflow/_cli/_pf_azure/_run.py b/src/promptflow/promptflow/_cli/_pf_azure/_run.py
@@ -465,6 +465,7 @@ def show_run_details(
     """Show a run details from cloud."""
     pf = _get_azure_pf_client(subscription_id, resource_group, workspace_name, debug=debug)
     details = pf.runs.get_details(run=flow_run_id, max_results=max_results, all_results=all_results)
+    details.fillna(value="(Failed)", inplace=True)  # replace nan with explicit prompt
     pretty_print_dataframe_as_table(details)
 
 

diff --git a/src/promptflow/promptflow/contracts/tool.py b/src/promptflow/promptflow/contracts/tool.py
@@ -9,7 +9,7 @@
 
 from promptflow._constants import CONNECTION_NAME_PROPERTY
 
-from .types import PromptTemplate, Secret
+from .types import FilePath, PromptTemplate, Secret
 from .multimedia import Image
 
 T = TypeVar("T", bound="Enum")
@@ -35,6 +35,7 @@ class ValueType(str, Enum):
     PROMPT_TEMPLATE = "prompt_template"
     LIST = "list"
     OBJECT = "object"
+    FILE_PATH = "file_path"
     IMAGE = "image"
 
     @staticmethod
@@ -61,6 +62,8 @@ def from_value(t: Any) -> "ValueType":
             return ValueType.STRING
         if isinstance(t, list):
             return ValueType.LIST
+        if isinstance(t, FilePath):
+            return ValueType.FILE_PATH
         return ValueType.OBJECT
 
     @staticmethod
@@ -87,6 +90,8 @@ def from_type(t: type) -> "ValueType":
             return ValueType.SECRET
         if t == PromptTemplate:
             return ValueType.PROMPT_TEMPLATE
+        if t == FilePath:
+            return ValueType.FILE_PATH
         if t == Image:
             return ValueType.IMAGE
         return ValueType.OBJECT

diff --git a/src/promptflow/promptflow/contracts/types.py b/src/promptflow/promptflow/contracts/types.py
@@ -19,3 +19,9 @@ class PromptTemplate(str):
     """This class is used to hint a parameter is a prompt template."""
 
     pass
+
+
+class FilePath(str):
+    """This class is used to hint a parameter is a file path."""
+
+    pass
diff --git a/src/promptflow/promptflow/exceptions.py b/src/promptflow/promptflow/exceptions.py
@@ -75,8 +75,16 @@ def message_parameters(self):
         if not self._kwargs:
             return {}
 
-        arguments = self.get_arguments_from_message_format(self.message_format)
-        return {k: v for k, v in self._kwargs.items() if k in arguments}
+        required_arguments = self.get_arguments_from_message_format(self.message_format)
+        parameters = {}
+        for argument in required_arguments:
+            if argument not in self._kwargs:
+                # Set a default value for the missing argument to avoid KeyError.
+                # For long term solution, use CI to guarantee the message_format and message_parameters are in sync.
+                parameters[argument] = f"<{argument}>"
+            else:
+                parameters[argument] = self._kwargs[argument]
+        return parameters
 
     @cached_property
     def serializable_message_parameters(self):

diff --git a/src/promptflow/setup.py b/src/promptflow/setup.py
@@ -44,6 +44,7 @@
     "tiktoken>=0.4.0",
     "strictyaml>=1.5.0,<2.0.0",  # used to identify exact location of validation error
     "waitress>=2.1.2,<3.0.0",  # used to serve local service
+    "opencensus-ext-azure<2.0.0",  # configure opencensus to send telemetry to azure monitor
 ]
 
 setup(

diff --git a/src/promptflow/tests/executor/unittests/executor/test_exceptions.py b/src/promptflow/tests/executor/unittests/executor/test_exceptions.py
@@ -0,0 +1,14 @@
+import pytest
+
+from promptflow.exceptions import PromptflowException
+
+
+@pytest.mark.unittest
+class TestExceptions:
+    def test_exception_message(self):
+        ex = PromptflowException(
+            message_format="Test exception message with parameters: {param}, {param1}.",
+            param="test_param",
+        )
+
+        assert ex.message == "Test exception message with parameters: test_param, <param1>."
Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,4 +7,4 @@

		Tools are the fundamental building blocks of a flow in Azure Machine Learning prompt flow. Each tool is a simple, executable unit with a specific function, allowing users to perform various tasks. By combining different tools, users can create a flow that accomplishes a wide range of goals. One of the key benefit of prompt flow tools is their seamless integration with third-party APIs and python open source packages. This not only improves the functionality of large language models but also makes the development process more efficient.

		In this package, we provide a set of builtin tools of prompt flow, which are the most commonly used tools in the development of AI applications. We also provide a flexible way for users to create their own tools and share them with others. See [Custom tool package creation and usage](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/how-to-create-and-use-your-own-tool-package.md) for more details.
		In this package, we provide a set of builtin tools of prompt flow, which are the most commonly used tools in the development of AI applications. We also provide a flexible way for users to create their own tools and share them with others. See [Create and Use Tool Package](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/develop-a-tool/create-and-use-tool-package.md) for more details.