refactor: remove load_as_component and update doc

.github/workflows/samples_flowinpipeline_pipeline.yml

@@ -0,0 +1,47 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: samples_flowinpipeline_pipeline
+on:
+  schedule:
+    - cron: "28 19 * * *" # Every day starting at 3:28 BJT
+  pull_request:
+    branches: [ main ]
+    paths: [ examples/**, .github/workflows/samples_flowinpipeline_pipeline.yml ]
+  workflow_dispatch:
+
+jobs:
+  samples_notebook_ci:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo '${{ secrets.TEST_WORKSPACE_CONFIG_JSON_CANARY }}' > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY_TEST }}" api_base="${{ secrets.AOAI_API_ENDPOINT_TEST }}"
+      - name: Test Notebook
+        working-directory: examples/tutorials/flow-in-pipeline
+        run: |
+          papermill -k python pipeline.ipynb pipeline.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/tutorials/flow-in-pipeline

docs/cloud/azureai/use-flow-in-azure-ml-pipeline.md

@@ -0,0 +1,129 @@
+# Use flow in Azure ML pipeline job
+
+After you have developed and tested the flow in [init and test a flow](../../how-to-guides/init-and-test-a-flow.md), this guide will help you learn how to use a flow as a parallel component in a pipeline job on AzureML, so that you can integrate the created flow with existing pipelines and process a large amount of data.
+
+:::{admonition} Pre-requirements
+- Customer need to install the extension `ml>=2.21.0` to enable this feature in CLI and package `azure-ai-ml>=1.11.0` to enable this feature in SDK;
+- Customer need to put `$schema` in the target `flow.dag.yaml` to enable this feature;
+  - `flow.dag.yaml`: `$schema`: `https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json`
+  - `run.yaml`: `$schema`: `https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json`
+- Customer need to generate `flow.tools.json` for the target flow before below usage. The generation can be done by `pf flow validate`.
+:::
+
+For more information about AzureML and component:
+- [Install and set up the CLI(v2)](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-cli?view=azureml-api-2&tabs=public)
+- [Install and set up the SDK(v2)](https://learn.microsoft.com/en-us/python/api/overview/azure/ai-ml-readme?view=azure-python)
+- [What is a pipeline](https://learn.microsoft.com/en-us/azure/machine-learning/concept-ml-pipelines?view=azureml-api-2)
+- [What is a component](https://learn.microsoft.com/en-us/azure/machine-learning/concept-component?view=azureml-api-2)
+
+## Register a flow as a component
+
+Customer can register a flow as a component with either CLI or SDK. 
+
+::::{tab-set}
+:::{tab-item} CLI
+:sync: CLI
+
+```bash
+# Register flow as a component
+# Default component name will be the name of flow folder, which is not a valid component name, so we override it here; default version will be "1"
+az ml component create --file standard/web-classification/flow.dag.yaml --set name=web_classification
+
+# Register flow as a component with parameters override
+az ml component create --file standard/web-classification/flow.dag.yaml --version 2 --set name=web_classification_updated
+```
+
+:::
+
+:::{tab-item} SDK
+:sync: SDK
+
+```python
+from azure.ai.ml import MLClient, load_component
+
+ml_client = MLClient()
+
+# Register flow as a component
+flow_component = load_component("standard/web-classification/flow.dag.yaml")
+# Default component name will be the name of flow folder, which is not a valid component name, so we override it here; default version will be "1"
+flow_component.name = "web_classification"
+ml_client.components.create_or_update(flow_component)
+
+# Register flow as a component with parameters override
+ml_client.components.create_or_update(
+    "standard/web-classification/flow.dag.yaml",
+    version="2",
+    params_override=[
+        {"name": "web_classification_updated"}
+    ]
+)
+```
+
+:::
+
+::::
+
+After registered a flow as a component, they can be referred in a pipeline job like [regular registered components](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components/basics/1b_e2e_registered_components).
+
+## Directly use a flow in a pipeline job
+
+Besides explicitly registering a flow as a component, customer can also directly use flow in a pipeline job:
+- [CLI example](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components/pipeline_job_with_flow_as_component)
+- [SDK example](https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1l_flow_in_pipeline)
+
+All connections and flow inputs will be exposed as input parameters of the component. Default value can be provided in flow/run definition; they can also be set/overwrite on job submission:
+
+::::{tab-set}
+:::{tab-item} CLI
+:sync: CLI
+
+```yaml
+...
+jobs:
+  flow_node:
+    type: parallel
+    component: standard/web-classification/flow.dag.yaml
+    inputs:
+      data: ${{parent.inputs.web_classification_input}}
+      url: "${data.url}"
+      connections.summarize_text_content.connection: azure_open_ai_connection
+      connections.summarize_text_content.deployment_name: text-davinci-003
+...
+```
+
+:::
+
+:::{tab-item} SDK
+:sync: SDK
+
+```python
+from azure.ai.ml import dsl
+
+ml_client = MLClient()
+
+# Register flow as a component
+flow_component = load_component("standard/web-classification/flow.dag.yaml")
+data_input = Input(path="standard/web-classification/data.jsonl", type=AssetTypes.URI_FILE)
+
+@dsl.pipeline
+def pipeline_func_with_flow(data):
+    flow_node = flow_component(
+        data=data,
+        url="${data.url}",
+        connections={
+            "summarize_text_content": {
+                "connection": "azure_open_ai_connection",
+                "deployment_name": "text-davinci-003",
+            },
+        },
+    )
+    flow_node.compute = "cpu-cluster"
+
+pipeline_with_flow = pipeline_func_with_flow(data=data_input)
+
+ml_client.jobs.create_or_update(pipeline_with_flow)
+```
+
+:::
+
+::::

docs/cloud/index.md

@@ -7,5 +7,6 @@ You can develope your flow locally and seamlessly move the experience to azure c
 :maxdepth: 1
 azureai/quick-start
 azureai/deploy-to-azure-appservice
+azureai/use-flow-in-azure-ml-pipeline.md
 azureai/faq
 ```

docs/reference/pf-command-reference.md

@@ -20,8 +20,9 @@ Manage promptflow flow flows.
 | --- | --- |
 | [pf flow init](#pf-flow-init) | Initialize a prompt flow directory. |
 | [pf flow test](#pf-flow-test) | Test the prompt flow or flow node. |
+| [pf flow validate](#pf-flow-validate) | Validate a flow and generate `flow.tools.json` for it. |
 | [pf flow build](#pf-flow-build) | Build a flow for further sharing or deployment. |
-| [pf flow serve](#pf-flow-serve) | Serving a flow as an endpoint. |
+| [pf flow serve](#pf-flow-serve) | Serve a flow as an endpoint. |
 
 ### pf flow init
 
@@ -167,6 +168,30 @@ Start a interactive chat session for chat flow.
 
 Displays the output for each step in the chat flow.
 
+### pf flow validate
+
+Validate the prompt flow and generate a `flow.tools.json` under `.promptflow`. This file is required when using flow as a component in a Azure ML pipeline.
+
+```bash
+pf flow validate --source
+                 [--debug]
+                 [--verbose]
+```
+
+#### Examples
+
+Validate the flow.
+
+```bash
+pf flow validate --source <path-to-flow>
+```
+
+#### Required Parameter
+
+`--source`
+
+The flow source to validate.
+
 ### pf flow build
 
 Build a flow for further sharing or deployment.

examples/README.md

@@ -28,6 +28,7 @@
 ------|--------|-------------
 | [chat-with-pdf](tutorials/e2e-development/chat-with-pdf.md) | [![samples_tutorials_e2e_development_chat_with_pdf](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_e2e_development_chat_with_pdf.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_e2e_development_chat_with_pdf.yml) |  Retrieval Augmented Generation (or RAG) has become a prevalent pattern to build intelligent application with Large Language Models (or LLMs) since it can infuse external knowledge into the model, which is not trained with those up-to-date or proprietary information |
 | [azure-app-service](tutorials/flow-deploy/azure-app-service/README.md) | [![samples_tutorials_flow_deploy_azure_app_service](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_azure_app_service.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_azure_app_service.yml) |  This example demos how to deploy a flow using Azure App Service |
+| [distribute-flow-as-executable-app](tutorials/flow-deploy/distribute-flow-as-executable-app/README.md) | [![samples_tutorials_flow_deploy_distribute_flow_as_executable_app](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_distribute_flow_as_executable_app.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_distribute_flow_as_executable_app.yml) |  This example demos how to package flow as a executable app |
 | [docker](tutorials/flow-deploy/docker/README.md) | [![samples_tutorials_flow_deploy_docker](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_docker.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_docker.yml) |  This example demos how to deploy flow as a docker app |
 | [kubernetes](tutorials/flow-deploy/kubernetes/README.md) | [![samples_tutorials_flow_deploy_kubernetes](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_kubernetes.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_kubernetes.yml) |  This example demos how to deploy flow as a Kubernetes app |
 
@@ -87,6 +88,7 @@
 ------|--------|-------------
 | [quickstart.ipynb](tutorials/get-started/quickstart.ipynb) | [![samples_getstarted_quickstart](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstart.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstart.yml) | A quickstart tutorial to run a flow and evaluate it. |
 | [quickstart-azure.ipynb](tutorials/get-started/quickstart-azure.ipynb) | [![samples_getstarted_quickstartazure](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstartazure.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstartazure.yml) | A quickstart tutorial to run a flow in Azure AI and evaluate it. |
+| [pipeline.ipynb](tutorials/flow-in-pipeline/pipeline.ipynb) | [![samples_flowinpipeline_pipeline](https://github.com/microsoft/promptflow/actions/workflows/samples_flowinpipeline_pipeline.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flowinpipeline_pipeline.yml) | {'description': 'Create pipeline using components to run a distributed job with tensorflow'} |
 | [cloud-run-management.ipynb](tutorials/run-management/cloud-run-management.ipynb) | [![samples_runmanagement_cloudrunmanagement](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_cloudrunmanagement.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_cloudrunmanagement.yml) | Flow run management in Azure AI |
 | [connection.ipynb](connections/connection.ipynb) | [![samples_connections_connection](https://github.com/microsoft/promptflow/actions/workflows/samples_connections_connection.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_connections_connection.yml) | Manage various types of connections using sdk |
 | [chat-with-pdf-azure.ipynb](flows/chat/chat-with-pdf/chat-with-pdf-azure.ipynb) | [![samples_flows_chat_chatwithpdf_chatwithpdfazure](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chatwithpdf_chatwithpdfazure.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chatwithpdf_chatwithpdfazure.yml) | A tutorial of chat-with-pdf flow that executes in Azure AI |

examples/configuration.ipynb

@@ -121,6 +121,40 @@
     "print(pf)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Retrieve or create an Azure Machine Learning compute target\n",
+    "\n",
+    "To create a Azure Machine Learning job, you need a compute cluster as prerequisite. Below code ensures computes named `cpu-cluster` exists in your workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azure.ai.ml import MLClient\n",
+    "from azure.ai.ml.entities import AmlCompute\n",
+    "\n",
+    "# MLClient use the same configuration as PFClient\n",
+    "ml_client = MLClient.from_config(credential=credential)\n",
+    "\n",
+    "# specify aml compute name.\n",
+    "cpu_compute_target = \"cpu-cluster\"\n",
+    "\n",
+    "try:\n",
+    "    ml_client.compute.get(cpu_compute_target)\n",
+    "except Exception:\n",
+    "    print(\"Creating a new cpu compute target...\")\n",
+    "    compute = AmlCompute(\n",
+    "        name=cpu_compute_target, size=\"STANDARD_D2_V2\", min_instances=0, max_instances=4\n",
+    "    )\n",
+    "    ml_client.compute.begin_create_or_update(compute).result()"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,