[doc] Refine tracing doc by separating tracing and devkit

docs/how-to-guides/index.md

@@ -8,6 +8,9 @@ Simple and short articles grouped by topics, each introduces a core feature of p
 tracing/index
 ```
 
+```{toctree}
+```
+
 ```{toctree}
 :caption: Prompty
 :maxdepth: 1

docs/how-to-guides/tracing/index.md

@@ -1,14 +1,16 @@
-# Tracing
+# Trace with prompt flow
 
 :::{admonition} Experimental feature
 This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental).
 :::
 
-Prompt flow provides the trace feature to capture and visualize the internal execution details for all flows.
+Prompt flow provides the trace feature to enable user to trace LLM call or function, and LLM frameworks like `LangChain` and `AutoGen`, following [OpenTelemetry specification](https://opentelemetry.io/docs/specs/otel/).
 
-For `DAG flow`, user can track and visualize node level inputs/outputs of flow execution, it provides critical insights for developer to understand the internal details of execution. 
+## Installing the package
 
-For `Flex flow` developers, who might use different frameworks (langchain, semantic kernel, OpenAI, kinds of agents) to create LLM based applications, prompt flow allow user to instrument their code in a [OpenTelemetry](https://opentelemetry.io/) compatible way, and visualize using UI provided by promptflow devkit.
+```bash
+pip install promptflow-tracing
+```
 
 ## Instrumenting user's code
 
@@ -18,7 +20,7 @@ Let's start with the simplest example, add single line code **`start_trace()`**
 from openai import OpenAI
 from promptflow.tracing import start_trace
 
-# start_trace() will print a url for trace detail visualization 
+# instrument OpenAI
 start_trace()
 
 client = OpenAI()
@@ -34,18 +36,7 @@ completion = client.chat.completions.create(
 print(completion.choices[0].message)
 ```
 
-Running above python script will produce below example output:
-```
-Prompt flow service has started...
-You can view the traces from local: http://localhost:<port>/v1.0/ui/traces/?#collection=basic
-```
-
-Click the trace url, user will see a trace list that corresponding to each LLM calls:
-![LLM-trace-list](../../media/trace/LLM-trace-list.png)
-
-
-Click on one line record, the LLM detail will be displayed with chat window experience, together with other LLM call params:
-![LLM-trace-detail](../../media/trace/LLM-trace-detail.png)
+Then OpenAI is instrumented, and as prompt flow follows OpenTelemetry specification, user can fully leverage the OpenTelemetry knowledge to use these traces during the OpenAI calls.
 
 ### Trace on any function
 A more common scenario is the application has complicated code structure, and developer would like to add trace on critical path that they would like to debug and monitor. 
@@ -56,6 +47,7 @@ Execute below command will get an URL to display the trace records and trace det
 
 ```python
 from promptflow.tracing import trace
+
 # trace your function
 @trace
 def code_gen(client: AzureOpenAI, question: str) -> str:
@@ -83,80 +75,11 @@ def code_gen(client: AzureOpenAI, question: str) -> str:
 python math_to_code.py
 ```
 
-## Trace visualization in flow test and batch run
-
-### Flow test
-If your application is created with DAG flow, all flow test and batch run will be automatically enable trace function. Take the **[chat_with_pdf](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-with-pdf/)** as example. 
-
-Run `pf flow test --flow .`, each flow test will generate single line in the trace UI:
-![flow-trace-record](../../media/trace/flow-trace-records.png)
-
-Click a record, the trace details will be visualized as tree view.
-
-![flow-trace-detail](../../media/trace/flow-trace-detail.png)
-
-### Evaluate against batch data
-Keep using **[chat_with_pdf](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-with-pdf)** as example, to trigger a batch run, you can use below commands:
-
-```shell
-pf run create -f batch_run.yaml
-```
-Or
-```shell
-pf run create --flow . --data "./data/bert-paper-qna.jsonl" --column-mapping chat_history='${data.chat_history}' pdf_url='${data.pdf_url}' question='${data.question}'
-```
-Then you will get a run related trace URL, e.g. http://localhost:<port>/v1.0/ui/traces?run=chat_with_pdf_20240226_181222_219335
-
-![batch_run_record](../../media/trace/batch_run_record.png)
-
-### Search
-
-Trace UI supports simple Python expression for search experience, which is demonstrated in below GIF:
-
-![advanced_search](../../media/trace/advanced-search.gif)
-
-Currently it supports bool operator `and` and `or`, compare operator `==`, `!=`, `>`, `>=`, `<`, `<=`; and the fields that are searchable: `name`, `kind`, `status`, `start_time`, `cumulative_token_count.total`, `cumulative_token_count.prompt` and `cumulative_token_count.completion`. You can find the hints by clicking the button right to the search edit box.
-
-![search_hint](../../media/trace/trace-ui-search-hint.png)
-
-## Local trace management
-
-### Delete
-
-Prompt flow provides capability to delete traces in local storage, user can delete traces by collection, time range or prompt flow run with both CLI and SDK:
-
-::::{tab-set}
-:::{tab-item} CLI
-:sync: CLI
-
-```bash
-pf trace delete --collection <collection-name>  # delete specific collection
-pf trace delete --collection <collection-name> --started-before '2024-03-01T16:00:00.123456'  # delete traces started before the time in specific collection
-pf trace delete --run <run-name>  # delete traces originated from specific prompt flow run
-```
-:::
-
-:::{tab-item} SDK
-:sync: SDK
-
-```python
-from promptflow.client import PFClient
-
-pf = PFClient()
-pf.traces.delete(collection="<collection-name>")  # delete specific collection
-pf.traces.delete(collection="<collection-name>", started_before="2024-03-01T16:00:00.123456")  # delete traces started before the time in specific collection
-pf.traces.delete(run="<run-name>")  # delete traces originated from specific prompt flow run
-```
-
-:::
-
-::::
-
-## Trace with prompt flow
+## Trace LLM and frameworks
 
 Prompt flow tracing works not only for general LLM application, but also for more frameworks like `autogen` and `langchain`:
 
-1. Example: **[Add trace for LLM](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/tracing/llm)**
+1. Example: **[Add trace for LLM](https://microsoft.github.io/promptflow/tutorials/trace-llm.html)**
 
 ![llm-trace-detail](../../media/trace/llm-app-trace-detail.png)
 
@@ -167,3 +90,10 @@ Prompt flow tracing works not only for general LLM application, but also for mor
 3. Example: **[Add trace for Langchain](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/tracing/langchain)**
 
 ![langchain-trace-detail](../../media/trace/langchain-trace-detail.png)
+
+```{toctree}
+:maxdepth: 1
+
+manage
+trace-ui
+```

docs/how-to-guides/tracing/manage.md

@@ -0,0 +1,40 @@
+# Manage traces
+
+:::{admonition} Experimental feature
+This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental).
+:::
+
+Prompt flow provides several trace toolkits in `promptflow-devkit`. This page will introduce how to delete traces in local storage with CLI/SDK.
+
+## Local trace management
+
+### Delete
+
+Prompt flow provides capability to delete traces in local storage, user can delete traces by collection, time range or prompt flow run with both CLI and SDK:
+
+::::{tab-set}
+:::{tab-item} CLI
+:sync: CLI
+
+```bash
+pf trace delete --collection <collection-name>  # delete specific collection
+pf trace delete --collection <collection-name> --started-before '2024-03-01T16:00:00.123456'  # delete traces started before the time in specific collection
+pf trace delete --run <run-name>  # delete traces originated from specific prompt flow run
+```
+:::
+
+:::{tab-item} SDK
+:sync: SDK
+
+```python
+from promptflow.client import PFClient
+
+pf = PFClient()
+pf.traces.delete(collection="<collection-name>")  # delete specific collection
+pf.traces.delete(collection="<collection-name>", started_before="2024-03-01T16:00:00.123456")  # delete traces started before the time in specific collection
+pf.traces.delete(run="<run-name>")  # delete traces originated from specific prompt flow run
+```
+
+:::
+
+::::

docs/how-to-guides/tracing/trace-ui.md

@@ -0,0 +1,64 @@
+# Visualize traces with trace UI
+
+:::{admonition} Experimental feature
+This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental).
+:::
+
+Prompt flow provides several trace toolkits in `promptflow-devkit`. This page will introduce trace UI, where user can better capture and visualize the internal execution details for all flows:
+
+- For `DAG flow`, user can track and visualize node level inputs/outputs of flow execution, it provides critical insights for developer to understand the internal details of execution. 
+- For `Flex flow` developers, who might use different frameworks (langchain, semantic kernel, OpenAI, kinds of agents) to create LLM based applications, prompt flow allow user to instrument their code in a [OpenTelemetry](https://opentelemetry.io/) compatible way, and visualize using UI provided by promptflow devkit.
+
+## Overview
+
+With `promptflow-devkit` installed, running python script with `start_trace` will produce below example output:
+
+```bash
+Prompt flow service has started...
+You can view the traces from local: http://localhost:<port>/v1.0/ui/traces/?#collection=basic
+```
+
+Click the url, user will see a trace list that corresponding to each LLM calls:
+![LLM-trace-list](../../media/trace/LLM-trace-list.png)
+
+
+Click on one line record, the LLM detail will be displayed with chat window experience, together with other LLM call params:
+![LLM-trace-detail](../../media/trace/LLM-trace-detail.png)
+
+When combine trace and flow, trace UI provides a more comprehensive view of the flow execution, user can easily track the flow execution details, and debug the flow execution issues.
+
+### Flow test
+
+If your application is created with DAG flow, all flow test and batch run will be automatically enable trace function. Take the **[chat_with_pdf](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-with-pdf/)** as example. 
+
+Run `pf flow test --flow .`, each flow test will generate single line in the trace UI:
+![flow-trace-record](../../media/trace/flow-trace-records.png)
+
+Click a record, the trace details will be visualized as tree view.
+
+![flow-trace-detail](../../media/trace/flow-trace-detail.png)
+
+### Evaluate against batch data
+
+Keep using **[chat_with_pdf](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-with-pdf)** as example, to trigger a batch run, you can use below commands:
+
+```shell
+pf run create -f batch_run.yaml
+```
+Or
+```shell
+pf run create --flow . --data "./data/bert-paper-qna.jsonl" --column-mapping chat_history='${data.chat_history}' pdf_url='${data.pdf_url}' question='${data.question}'
+```
+Then you will get a run related trace URL, e.g. http://localhost:<port>/v1.0/ui/traces?run=chat_with_pdf_20240226_181222_219335
+
+![batch_run_record](../../media/trace/batch_run_record.png)
+
+### Search
+
+Trace UI supports simple Python expression for search experience, which is demonstrated in below GIF:
+
+![advanced_search](../../media/trace/advanced-search.gif)
+
+Currently it supports bool operator `and` and `or`, compare operator `==`, `!=`, `>`, `>=`, `<`, `<=`; and the fields that are searchable: `name`, `kind`, `status`, `start_time`, `cumulative_token_count.total`, `cumulative_token_count.prompt` and `cumulative_token_count.completion`. You can find the hints by clicking the button right to the search edit box.
+
+![search_hint](../../media/trace/trace-ui-search-hint.png)