diff --git a/docs/08-async-dynamic-routes.ipynb b/docs/08-async-dynamic-routes.ipynb new file mode 100644 index 00000000..0cfb4e0a --- /dev/null +++ b/docs/08-async-dynamic-routes.ipynb @@ -0,0 +1,1027 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "UxqB7_Ieur0s" + }, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/aurelio-labs/semantic-router/blob/main/docs/08-async-dynamic-routes.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/aurelio-labs/semantic-router/blob/main/docs/08-async-dynamic-routes.ipynb)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EduhQaNAur0u" + }, + "source": [ + "# Dynamic Routes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_4JgNeX4ur0v" + }, + "source": [ + "In semantic-router there are two types of routes that can be chosen. Both routes belong to the `Route` object, the only difference between them is that _static_ routes return a `Route.name` when chosen, whereas _dynamic_ routes use an LLM call to produce parameter input values.\n", + "\n", + "For example, a _static_ route will tell us if a query is talking about mathematics by returning the route name (which could be `\"math\"` for example). A _dynamic_ route does the same thing, but it also extracts key information from the input utterance to be used in a function associated with that route.\n", + "\n", + "For example we could provide a dynamic route with associated utterances:\n", + "\n", + "```\n", + "\"what is x to the power of y?\"\n", + "\"what is 9 to the power of 4?\"\n", + "\"calculate the result of base x and exponent y\"\n", + "\"calculate the result of base 10 and exponent 3\"\n", + "\"return x to the power of y\"\n", + "```\n", + "\n", + "and we could also provide the route with a schema outlining key features of the function:\n", + "\n", + "```\n", + "def power(base: float, exponent: float) -> float:\n", + " \"\"\"Raise base to the power of exponent.\n", + "\n", + " Args:\n", + " base (float): The base number.\n", + " exponent (float): The exponent to which the base is raised.\n", + "\n", + " Returns:\n", + " float: The result of base raised to the power of exponent.\n", + " \"\"\"\n", + " return base ** exponent\n", + "```\n", + "\n", + "Then, if the users input utterance is \"What is 2 to the power of 3?\", the route will be triggered, as the input utterance is semantically similar to the route utterances. Furthermore, the route utilizes an LLM to identify that `base=2` and `expoenent=3`. These values are returned in such a way that they can be used in the above `power` function. That is, the dynamic router automates the process of calling relevant functions from natural language inputs.\n", + "\n", + "***⚠️ Note: We have a fully local version of dynamic routes available at [docs/05-local-execution.ipynb](https://github.com/aurelio-labs/semantic-router/blob/main/docs/05-local-execution.ipynb). The local 05 version tends to outperform the OpenAI version we demo in this notebook, so we'd recommend trying [05](https://github.com/aurelio-labs/semantic-router/blob/main/docs/05-local-execution.ipynb)!***" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bbmw8CO4ur0v" + }, + "source": [ + "## Installing the Library" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dLElfRhgur0v", + "outputId": "da0e506e-24cf-43da-9243-894a7c4955db" + }, + "outputs": [], + "source": [ + "!pip install -qU \\\n", + " \"semantic-router==0.0.54\" \\\n", + " tzdata" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BixZd6Eour0w" + }, + "source": [ + "## Initializing Routes and RouteLayer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PxnW9qBvur0x" + }, + "source": [ + "Dynamic routes are treated in the same way as static routes, let's begin by initializing a `RouteLayer` consisting of static routes." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "kc9Ty6Lgur0x", + "outputId": "f32e3a25-c073-4802-ced3-d7a5663670c1" + }, + "outputs": [], + "source": [ + "from semantic_router import Route\n", + "\n", + "politics = Route(\n", + " name=\"politics\",\n", + " utterances=[\n", + " \"isn't politics the best thing ever\",\n", + " \"why don't you tell me about your political opinions\",\n", + " \"don't you just love the president\" \"don't you just hate the president\",\n", + " \"they're going to destroy this country!\",\n", + " \"they will save the country!\",\n", + " ],\n", + ")\n", + "chitchat = Route(\n", + " name=\"chitchat\",\n", + " utterances=[\n", + " \"how's the weather today?\",\n", + " \"how are things going?\",\n", + " \"lovely weather today\",\n", + " \"the weather is horrendous\",\n", + " \"let's go to the chippy\",\n", + " ],\n", + ")\n", + "\n", + "routes = [politics, chitchat]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "voWyqmffur0x" + }, + "source": [ + "We initialize our `RouteLayer` with our `encoder` and `routes`. We can use popular encoder APIs like `CohereEncoder` and `OpenAIEncoder`, or local alternatives like `FastEmbedEncoder`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "BI9AiDspur0y", + "outputId": "27329a54-3f16-44a5-ac20-13a6b26afb97" + }, + "outputs": [], + "source": [ + "import os\n", + "from getpass import getpass\n", + "from semantic_router import RouteLayer\n", + "from semantic_router.encoders import CohereEncoder, OpenAIEncoder\n", + "\n", + "\n", + "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\") or getpass(\n", + " \"Enter OpenAI API Key: \"\n", + ")\n", + "\n", + "# encoder = CohereEncoder()\n", + "encoder = OpenAIEncoder()\n", + "\n", + "rl = RouteLayer(encoder=encoder, routes=routes)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GuLCeIS5ur0y" + }, + "source": [ + "We run the solely static routes layer:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "RouteChoice(name='chitchat', function_call=None, similarity_score=None)" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "await rl.acall(\"how's the weather today?\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "McbLKO26ur0y" + }, + "source": [ + "## Creating a Dynamic Route" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ANAoEjxYur0y" + }, + "source": [ + "As with static routes, we must create a dynamic route before adding it to our route layer. To make a route dynamic, we need to provide the `function_schemas` as a list. Each function schema provides instructions on what a function is, so that an LLM can decide how to use it correctly." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "5jaF1Xa5ur0y" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "\n", + "def get_time(timezone: str) -> str:\n", + " \"\"\"Finds the current time in a specific timezone.\n", + "\n", + " :param timezone: The timezone to find the current time in, should\n", + " be a valid timezone from the IANA Time Zone Database like\n", + " \"America/New_York\" or \"Europe/London\". Do NOT put the place\n", + " name itself like \"rome\", or \"new york\", you must provide\n", + " the IANA format.\n", + " :type timezone: str\n", + " :return: The current time in the specified timezone.\"\"\"\n", + " now = datetime.now(ZoneInfo(timezone))\n", + " return now.strftime(\"%H:%M\")" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "YyFKV8jMur0z", + "outputId": "29cf80f4-552c-47bb-fbf9-019f5dfdf00a" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "'02:44'" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "get_time(\"America/New_York\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4qyaRuNXur0z" + }, + "source": [ + "To get the function schema we can use the `get_schema` function from the `function_call` module." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "tOjuhp5Xur0z", + "outputId": "ca88a3ea-d70a-4950-be9a-63fab699de3b" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'type': 'function',\n", + " 'function': {'name': 'get_time',\n", + " 'description': 'Finds the current time in a specific timezone.\\n\\n:param timezone: The timezone to find the current time in, should\\n be a valid timezone from the IANA Time Zone Database like\\n \"America/New_York\" or \"Europe/London\". Do NOT put the place\\n name itself like \"rome\", or \"new york\", you must provide\\n the IANA format.\\n:type timezone: str\\n:return: The current time in the specified timezone.',\n", + " 'parameters': {'type': 'object',\n", + " 'properties': {'timezone': {'type': 'string',\n", + " 'description': 'The timezone to find the current time in, should\\n be a valid timezone from the IANA Time Zone Database like\\n \"America/New_York\" or \"Europe/London\". Do NOT put the place\\n name itself like \"rome\", or \"new york\", you must provide\\n the IANA format.'}},\n", + " 'required': ['timezone']}}}]" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from semantic_router.llms.openai import get_schemas_openai\n", + "\n", + "schemas = get_schemas_openai([get_time])\n", + "schemas" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HcF7jGjAur0z" + }, + "source": [ + "We use this to define our dynamic route:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "iesBG9P3ur0z" + }, + "outputs": [], + "source": [ + "time_route = Route(\n", + " name=\"get_time\",\n", + " utterances=[\n", + " \"what is the time in new york city?\",\n", + " \"what is the time in london?\",\n", + " \"I live in Rome, what time is it?\",\n", + " ],\n", + " function_schemas=schemas,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZiUs3ovpur0z" + }, + "source": [ + "Add the new route to our `layer`:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "-0vY8PRXur0z", + "outputId": "db01e14c-eab3-4f93-f4c2-e30f508c8b5d" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2024-07-19 14:44:29 INFO semantic_router.utils.logger Adding `get_time` route\u001b[0m\n" + ] + } + ], + "source": [ + "rl.add(time_route)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7yoE0IrNur0z" + }, + "source": [ + "Now we can ask our layer a time related question to trigger our new dynamic route." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33m2024-07-19 14:44:32 WARNING semantic_router.utils.logger No LLM provided for dynamic route, will use OpenAI LLM default\u001b[0m\n", + "\u001b[32m2024-07-19 14:44:34 INFO semantic_router.utils.logger OpenAI => Function Inputs: [{'function_name': 'get_time', 'arguments': {'timezone': 'America/New_York'}}]\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "RouteChoice(name='get_time', function_call=[{'function_name': 'get_time', 'arguments': {'timezone': 'America/New_York'}}], similarity_score=None)" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = await rl.acall(\"what is the time in new york city?\")\n", + "response" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "xvdyUPKqg9hr", + "outputId": "4161e7e0-ab6d-4e76-f068-2d66728305ff" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "02:44\n" + ] + } + ], + "source": [ + "for call in response.function_call:\n", + " if call[\"function_name\"] == \"get_time\":\n", + " args = call[\"arguments\"]\n", + " result = get_time(**args)\n", + "print(result)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Qt0vkq2Xur00" + }, + "source": [ + "Our dynamic route provides both the route itself _and_ the input parameters required to use the route." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jToYBo8Ug9hr" + }, + "source": [ + "## Dynamic Routes with Multiple Functions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "J0oD1dxIur00" + }, + "source": [ + "---" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vEkTpoVAg9hr" + }, + "source": [ + "Routes can be assigned multiple functions. Then, when that particular Route is selected by the Route Layer, a number of those functions might be invoked due to the users utterance containing relevant information that fits their arguments." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BHUlB3org9hs" + }, + "source": [ + "Let's define a Route that has multiple functions." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "dtrksov0g9hs" + }, + "outputs": [], + "source": [ + "from datetime import datetime, timedelta\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "\n", + "# Function with one argument\n", + "def get_time(timezone: str) -> str:\n", + " \"\"\"Finds the current time in a specific timezone.\n", + "\n", + " :param timezone: The timezone to find the current time in, should\n", + " be a valid timezone from the IANA Time Zone Database like\n", + " \"America/New_York\" or \"Europe/London\". Do NOT put the place\n", + " name itself like \"rome\", or \"new york\", you must provide\n", + " the IANA format.\n", + " :type timezone: str\n", + " :return: The current time in the specified timezone.\"\"\"\n", + " now = datetime.now(ZoneInfo(timezone))\n", + " return now.strftime(\"%H:%M\")\n", + "\n", + "\n", + "def get_time_difference(timezone1: str, timezone2: str) -> str:\n", + " \"\"\"Calculates the time difference between two timezones.\n", + " :param timezone1: The first timezone, should be a valid timezone from the IANA Time Zone Database like \"America/New_York\" or \"Europe/London\".\n", + " :param timezone2: The second timezone, should be a valid timezone from the IANA Time Zone Database like \"America/New_York\" or \"Europe/London\".\n", + " :type timezone1: str\n", + " :type timezone2: str\n", + " :return: The time difference in hours between the two timezones.\"\"\"\n", + " # Get the current time in UTC\n", + " now_utc = datetime.utcnow().replace(tzinfo=ZoneInfo(\"UTC\"))\n", + "\n", + " # Convert the UTC time to the specified timezones\n", + " tz1_time = now_utc.astimezone(ZoneInfo(timezone1))\n", + " tz2_time = now_utc.astimezone(ZoneInfo(timezone2))\n", + "\n", + " # Calculate the difference in offsets from UTC\n", + " tz1_offset = tz1_time.utcoffset().total_seconds()\n", + " tz2_offset = tz2_time.utcoffset().total_seconds()\n", + "\n", + " # Calculate the difference in hours\n", + " hours_difference = (tz2_offset - tz1_offset) / 3600\n", + "\n", + " return f\"The time difference between {timezone1} and {timezone2} is {hours_difference} hours.\"\n", + "\n", + "\n", + "# Function with three arguments\n", + "def convert_time(time: str, from_timezone: str, to_timezone: str) -> str:\n", + " \"\"\"Converts a specific time from one timezone to another.\n", + " :param time: The time to convert in HH:MM format.\n", + " :param from_timezone: The original timezone of the time, should be a valid IANA timezone.\n", + " :param to_timezone: The target timezone for the time, should be a valid IANA timezone.\n", + " :type time: str\n", + " :type from_timezone: str\n", + " :type to_timezone: str\n", + " :return: The converted time in the target timezone.\n", + " :raises ValueError: If the time format or timezone strings are invalid.\n", + "\n", + " Example:\n", + " convert_time(\"12:30\", \"America/New_York\", \"Asia/Tokyo\") -> \"03:30\"\n", + " \"\"\"\n", + " try:\n", + " # Use today's date to avoid historical timezone issues\n", + " today = datetime.now().date()\n", + " datetime_string = f\"{today} {time}\"\n", + " time_obj = datetime.strptime(datetime_string, \"%Y-%m-%d %H:%M\").replace(\n", + " tzinfo=ZoneInfo(from_timezone)\n", + " )\n", + "\n", + " converted_time = time_obj.astimezone(ZoneInfo(to_timezone))\n", + "\n", + " formatted_time = converted_time.strftime(\"%H:%M\")\n", + " return formatted_time\n", + " except Exception as e:\n", + " raise ValueError(f\"Error converting time: {e}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "AjoYy7mFg9hs" + }, + "outputs": [], + "source": [ + "functions = [get_time, get_time_difference, convert_time]" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "DoOkXV2Tg9hs", + "outputId": "f1e0fe08-b6ed-4f50-d845-5c54832ca677" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'type': 'function',\n", + " 'function': {'name': 'get_time',\n", + " 'description': 'Finds the current time in a specific timezone.\\n\\n:param timezone: The timezone to find the current time in, should\\n be a valid timezone from the IANA Time Zone Database like\\n \"America/New_York\" or \"Europe/London\". Do NOT put the place\\n name itself like \"rome\", or \"new york\", you must provide\\n the IANA format.\\n:type timezone: str\\n:return: The current time in the specified timezone.',\n", + " 'parameters': {'type': 'object',\n", + " 'properties': {'timezone': {'type': 'string',\n", + " 'description': 'The timezone to find the current time in, should\\n be a valid timezone from the IANA Time Zone Database like\\n \"America/New_York\" or \"Europe/London\". Do NOT put the place\\n name itself like \"rome\", or \"new york\", you must provide\\n the IANA format.'}},\n", + " 'required': ['timezone']}}},\n", + " {'type': 'function',\n", + " 'function': {'name': 'get_time_difference',\n", + " 'description': 'Calculates the time difference between two timezones.\\n:param timezone1: The first timezone, should be a valid timezone from the IANA Time Zone Database like \"America/New_York\" or \"Europe/London\".\\n:param timezone2: The second timezone, should be a valid timezone from the IANA Time Zone Database like \"America/New_York\" or \"Europe/London\".\\n:type timezone1: str\\n:type timezone2: str\\n:return: The time difference in hours between the two timezones.',\n", + " 'parameters': {'type': 'object',\n", + " 'properties': {'timezone1': {'type': 'string',\n", + " 'description': 'The first timezone, should be a valid timezone from the IANA Time Zone Database like \"America/New_York\" or \"Europe/London\".'},\n", + " 'timezone2': {'type': 'string',\n", + " 'description': 'The second timezone, should be a valid timezone from the IANA Time Zone Database like \"America/New_York\" or \"Europe/London\".'}},\n", + " 'required': ['timezone1', 'timezone2']}}},\n", + " {'type': 'function',\n", + " 'function': {'name': 'convert_time',\n", + " 'description': 'Converts a specific time from one timezone to another.\\n:param time: The time to convert in HH:MM format.\\n:param from_timezone: The original timezone of the time, should be a valid IANA timezone.\\n:param to_timezone: The target timezone for the time, should be a valid IANA timezone.\\n:type time: str\\n:type from_timezone: str\\n:type to_timezone: str\\n:return: The converted time in the target timezone.\\n:raises ValueError: If the time format or timezone strings are invalid.\\n\\nExample:\\n convert_time(\"12:30\", \"America/New_York\", \"Asia/Tokyo\") -> \"03:30\"',\n", + " 'parameters': {'type': 'object',\n", + " 'properties': {'time': {'type': 'string',\n", + " 'description': 'The time to convert in HH:MM format.'},\n", + " 'from_timezone': {'type': 'string',\n", + " 'description': 'The original timezone of the time, should be a valid IANA timezone.'},\n", + " 'to_timezone': {'type': 'string',\n", + " 'description': 'The target timezone for the time, should be a valid IANA timezone.'}},\n", + " 'required': ['time', 'from_timezone', 'to_timezone']}}}]" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Generate schemas for all functions\n", + "from semantic_router.llms.openai import get_schemas_openai\n", + "\n", + "schemas = get_schemas_openai(functions)\n", + "schemas" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "YBRHxhnkg9hs" + }, + "outputs": [], + "source": [ + "# Define the dynamic route with multiple functions\n", + "multi_function_route = Route(\n", + " name=\"timezone_management\",\n", + " utterances=[\n", + " # Utterances for get_time function\n", + " \"what is the time in New York?\",\n", + " \"current time in Berlin?\",\n", + " \"tell me the time in Moscow right now\",\n", + " \"can you show me the current time in Tokyo?\",\n", + " \"please provide the current time in London\",\n", + " # Utterances for get_time_difference function\n", + " \"how many hours ahead is Tokyo from London?\",\n", + " \"time difference between Sydney and Cairo\",\n", + " \"what's the time gap between Los Angeles and New York?\",\n", + " \"how much time difference is there between Paris and Sydney?\",\n", + " \"calculate the time difference between Dubai and Toronto\",\n", + " # Utterances for convert_time function\n", + " \"convert 15:00 from New York time to Berlin time\",\n", + " \"change 09:00 from Paris time to Moscow time\",\n", + " \"adjust 20:00 from Rome time to London time\",\n", + " \"convert 12:00 from Madrid time to Chicago time\",\n", + " \"change 18:00 from Beijing time to Los Angeles time\"\n", + " # All three functions\n", + " \"What is the time in Seattle? What is the time difference between Mumbai and Tokyo? What is 5:53 Toronto time in Sydney time?\",\n", + " ],\n", + " function_schemas=schemas,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "yEbQadQbg9ht" + }, + "outputs": [], + "source": [ + "routes = [politics, chitchat, multi_function_route]" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "id": "C0aYIXaog9ht", + "outputId": "74114a86-4a6f-49c5-8e2e-600f577d63f5" + }, + "outputs": [], + "source": [ + "rl2 = RouteLayer(encoder=encoder, routes=routes)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cG98YLZ5g9ht" + }, + "source": [ + "### Function to Parse Route Layer Responses" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "PJR97klVg9ht" + }, + "outputs": [], + "source": [ + "def parse_response(response: str):\n", + "\n", + " for call in response.function_call:\n", + " args = call[\"arguments\"]\n", + " if call[\"function_name\"] == \"get_time\":\n", + " result = get_time(**args)\n", + " print(result)\n", + " if call[\"function_name\"] == \"get_time_difference\":\n", + " result = get_time_difference(**args)\n", + " print(result)\n", + " if call[\"function_name\"] == \"convert_time\":\n", + " result = convert_time(**args)\n", + " print(result)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OUbPbxZKg9ht" + }, + "source": [ + "### Checking that Politics Non-Dynamic Route Still Works" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "RouteChoice(name='politics', function_call=None, similarity_score=None)" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = await rl2.acall(\"What is your political leaning?\")\n", + "response" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZHgw8QoWg9ht" + }, + "source": [ + "### Checking that Chitchat Non-Dynamic Route Still Works" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "RouteChoice(name='chitchat', function_call=None, similarity_score=None)" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = await rl2.acall(\"Hello bot, how are you today?\")\n", + "response" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uZDiY787g9hu" + }, + "source": [ + "### Testing the `multi_function_route` - The `get_time` Function" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33m2024-07-19 14:45:02 WARNING semantic_router.utils.logger No LLM provided for dynamic route, will use OpenAI LLM default\u001b[0m\n", + "\u001b[32m2024-07-19 14:45:03 INFO semantic_router.utils.logger OpenAI => Function Inputs: [{'function_name': 'get_time', 'arguments': {'timezone': 'America/New_York'}}]\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "RouteChoice(name='timezone_management', function_call=[{'function_name': 'get_time', 'arguments': {'timezone': 'America/New_York'}}], similarity_score=None)" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = await rl2.acall(\"what is the time in New York?\")\n", + "response" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "02:45\n" + ] + } + ], + "source": [ + "parse_response(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wcjQ4Dbpg9hu" + }, + "source": [ + "### Testing the `multi_function_route` - The `get_time_difference` Function" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2024-07-19 14:45:07 INFO semantic_router.utils.logger OpenAI => Function Inputs: [{'function_name': 'get_time_difference', 'arguments': {'timezone1': 'America/Los_Angeles', 'timezone2': 'Europe/Istanbul'}}]\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "RouteChoice(name='timezone_management', function_call=[{'function_name': 'get_time_difference', 'arguments': {'timezone1': 'America/Los_Angeles', 'timezone2': 'Europe/Istanbul'}}], similarity_score=None)" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = await rl2.acall(\n", + " \"What is the time difference between Los Angeles and Istanbul?\"\n", + ")\n", + "response" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The time difference between America/Los_Angeles and Europe/Istanbul is 10.0 hours.\n" + ] + } + ], + "source": [ + "parse_response(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "14qz-ApLg9hv" + }, + "source": [ + "### Testing the `multi_function_route` - The `convert_time` Function" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2024-07-19 14:45:10 INFO semantic_router.utils.logger OpenAI => Function Inputs: [{'function_name': 'convert_time', 'arguments': {'time': '23:02', 'from_timezone': 'Asia/Dubai', 'to_timezone': 'Asia/Tokyo'}}]\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "RouteChoice(name='timezone_management', function_call=[{'function_name': 'convert_time', 'arguments': {'time': '23:02', 'from_timezone': 'Asia/Dubai', 'to_timezone': 'Asia/Tokyo'}}], similarity_score=None)" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = await rl2.acall(\n", + " \"What is 23:02 Dubai time in Tokyo time? Please and thank you.\"\n", + ")\n", + "response" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "04:02\n" + ] + } + ], + "source": [ + "parse_response(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TSRfC6JJg9hv" + }, + "source": [ + "### The Cool Bit - Testing `multi_function_route` - Multiple Functions at Once" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m2024-07-19 14:45:15 INFO semantic_router.utils.logger OpenAI => Function Inputs: [{'function_name': 'get_time', 'arguments': {'timezone': 'Europe/Prague'}}, {'function_name': 'get_time_difference', 'arguments': {'timezone1': 'Europe/Berlin', 'timezone2': 'Asia/Shanghai'}}, {'function_name': 'convert_time', 'arguments': {'time': '05:53', 'from_timezone': 'Europe/Lisbon', 'to_timezone': 'Asia/Bangkok'}}]\u001b[0m\n" + ] + } + ], + "source": [ + "response = await rl2.acall(\n", + " \"\"\"\n", + " What is the time in Prague?\n", + " What is the time difference between Frankfurt and Beijing?\n", + " What is 5:53 Lisbon time in Bangkok time?\n", + "\"\"\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "08:45\n", + "The time difference between Europe/Berlin and Asia/Shanghai is 6.0 hours.\n", + "11:53\n" + ] + } + ], + "source": [ + "parse_response(response)" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "decision-layer", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.5" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/pyproject.toml b/pyproject.toml index f78ed13f..1ecd207d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "semantic-router" -version = "0.0.53" +version = "0.0.54" description = "Super fast semantic router for AI decision making" authors = [ "James Briggs ", diff --git a/semantic_router/__init__.py b/semantic_router/__init__.py index 19d5381f..bd7f0932 100644 --- a/semantic_router/__init__.py +++ b/semantic_router/__init__.py @@ -4,4 +4,4 @@ __all__ = ["RouteLayer", "HybridRouteLayer", "Route", "LayerConfig"] -__version__ = "0.0.53" +__version__ = "0.0.54" diff --git a/semantic_router/layer.py b/semantic_router/layer.py index 5c2d7228..81f690b5 100644 --- a/semantic_router/layer.py +++ b/semantic_router/layer.py @@ -297,10 +297,15 @@ async def acall( "Route has a function schema, but no text was provided." ) if route.function_schemas and not isinstance(route.llm, BaseLLM): - raise NotImplementedError( - "Dynamic routes not yet supported for async calls." - ) - return route(text) + if not self.llm: + logger.warning( + "No LLM provided for dynamic route, will use OpenAI LLM default" + ) + self.llm = OpenAILLM() + route.llm = self.llm + else: + route.llm = self.llm + return await route.acall(text) elif passed and route is not None and simulate_static: return RouteChoice( name=route.name, diff --git a/semantic_router/llms/openai.py b/semantic_router/llms/openai.py index 2a531195..f22f409e 100644 --- a/semantic_router/llms/openai.py +++ b/semantic_router/llms/openai.py @@ -22,6 +22,7 @@ class OpenAILLM(BaseLLM): client: Optional[openai.OpenAI] + async_client: Optional[openai.AsyncOpenAI] temperature: Optional[float] max_tokens: Optional[int] @@ -39,6 +40,7 @@ def __init__( if api_key is None: raise ValueError("OpenAI API key cannot be 'None'.") try: + self.async_client = openai.AsyncOpenAI(api_key=api_key) self.client = openai.OpenAI(api_key=api_key) except Exception as e: raise ValueError( @@ -64,6 +66,23 @@ def _extract_tool_calls_info( ) return tool_calls_info + async def async_extract_tool_calls_info( + self, tool_calls: List[ChatCompletionMessageToolCall] + ) -> List[Dict[str, Any]]: + tool_calls_info = [] + for tool_call in tool_calls: + if tool_call.function.arguments is None: + raise ValueError( + "Invalid output, expected arguments to be specified for each tool call." + ) + tool_calls_info.append( + { + "function_name": tool_call.function.name, + "arguments": json.loads(tool_call.function.arguments), + } + ) + return tool_calls_info + def __call__( self, messages: List[Message], @@ -108,6 +127,50 @@ def __call__( logger.error(f"LLM error: {e}") raise Exception(f"LLM error: {e}") from e + async def acall( + self, + messages: List[Message], + function_schemas: Optional[List[Dict[str, Any]]] = None, + ) -> str: + if self.async_client is None: + raise ValueError("OpenAI async_client is not initialized.") + try: + tools: Union[List[Dict[str, Any]], NotGiven] = ( + function_schemas if function_schemas is not None else NOT_GIVEN + ) + + completion = await self.async_client.chat.completions.create( + model=self.name, + messages=[m.to_openai() for m in messages], + temperature=self.temperature, + max_tokens=self.max_tokens, + tools=tools, # type: ignore # We pass a list of dicts which get interpreted as Iterable[ChatCompletionToolParam]. + ) + + if function_schemas: + tool_calls = completion.choices[0].message.tool_calls + if tool_calls is None: + raise ValueError("Invalid output, expected a tool call.") + if len(tool_calls) < 1: + raise ValueError( + "Invalid output, expected at least one tool to be specified." + ) + + # Collecting multiple tool calls information + output = str( + await self.async_extract_tool_calls_info(tool_calls) + ) # str in keeping with base type. + else: + content = completion.choices[0].message.content + if content is None: + raise ValueError("Invalid output, expected content.") + output = content + return output + + except Exception as e: + logger.error(f"LLM error: {e}") + raise Exception(f"LLM error: {e}") from e + def extract_function_inputs( self, query: str, function_schemas: List[Dict[str, Any]] ) -> List[Dict[str, Any]]: @@ -122,6 +185,25 @@ def extract_function_inputs( output = output.replace("'", '"') function_inputs = json.loads(output) logger.info(f"Function inputs: {function_inputs}") + logger.info(f"function_schemas: {function_schemas}") + if not self._is_valid_inputs(function_inputs, function_schemas): + raise ValueError("Invalid inputs") + return function_inputs + + async def async_extract_function_inputs( + self, query: str, function_schemas: List[Dict[str, Any]] + ) -> List[Dict[str, Any]]: + system_prompt = "You are an intelligent AI. Given a command or request from the user, call the function to complete the request." + messages = [ + Message(role="system", content=system_prompt), + Message(role="user", content=query), + ] + output = await self.acall(messages=messages, function_schemas=function_schemas) + if not output: + raise Exception("No output generated for extract function input") + output = output.replace("'", '"') + function_inputs = json.loads(output) + logger.info(f"OpenAI => Function Inputs: {function_inputs}") if not self._is_valid_inputs(function_inputs, function_schemas): raise ValueError("Invalid inputs") return function_inputs diff --git a/semantic_router/route.py b/semantic_router/route.py index a32c778c..3fc3f040 100644 --- a/semantic_router/route.py +++ b/semantic_router/route.py @@ -76,6 +76,28 @@ def __call__(self, query: Optional[str] = None) -> RouteChoice: func_call = None return RouteChoice(name=self.name, function_call=func_call) + async def acall(self, query: Optional[str] = None) -> RouteChoice: + if self.function_schemas: + if not self.llm: + raise ValueError( + "LLM is required for dynamic routes. Please ensure the `llm` " + "attribute is set." + ) + elif query is None: + raise ValueError( + "Query is required for dynamic routes. Please ensure the `query` " + "argument is passed." + ) + # if a function schema is provided we generate the inputs + extracted_inputs = await self.llm.async_extract_function_inputs( # type: ignore # openai-llm + query=query, function_schemas=self.function_schemas + ) + func_call = extracted_inputs + else: + # otherwise we just pass None for the call + func_call = None + return RouteChoice(name=self.name, function_call=func_call) + # def to_dict(self) -> Dict[str, Any]: # return self.dict() diff --git a/semantic_router/utils/defaults.py b/semantic_router/utils/defaults.py index 75331c06..151a9935 100644 --- a/semantic_router/utils/defaults.py +++ b/semantic_router/utils/defaults.py @@ -8,8 +8,8 @@ class EncoderDefault(Enum): "language_model": "BAAI/bge-small-en-v1.5", } OPENAI = { - "embedding_model": os.getenv("OPENAI_MODEL_NAME", "text-embedding-ada-002"), - "language_model": os.getenv("OPENAI_CHAT_MODEL_NAME", "gpt-3.5-turbo"), + "embedding_model": os.getenv("OPENAI_MODEL_NAME", "text-embedding-3-small"), + "language_model": os.getenv("OPENAI_CHAT_MODEL_NAME", "gpt-4o"), } COHERE = { "embedding_model": os.getenv("COHERE_MODEL_NAME", "embed-english-v3.0"), @@ -20,10 +20,10 @@ class EncoderDefault(Enum): "language_model": os.getenv("MISTRALAI_CHAT_MODEL_NAME", "mistral-tiny"), } AZURE = { - "embedding_model": os.getenv("AZURE_OPENAI_MODEL", "text-embedding-ada-002"), - "language_model": os.getenv("OPENAI_CHAT_MODEL_NAME", "gpt-3.5-turbo"), + "embedding_model": os.getenv("AZURE_OPENAI_MODEL", "text-embedding-3-small"), + "language_model": os.getenv("OPENAI_CHAT_MODEL_NAME", "gpt-4o"), "deployment_name": os.getenv( - "AZURE_OPENAI_DEPLOYMENT_NAME", "text-embedding-ada-002" + "AZURE_OPENAI_DEPLOYMENT_NAME", "text-embedding-3-small" ), } GOOGLE = { diff --git a/tests/unit/llms/test_llm_azure_openai.py b/tests/unit/llms/test_llm_azure_openai.py index a50b08fb..79309195 100644 --- a/tests/unit/llms/test_llm_azure_openai.py +++ b/tests/unit/llms/test_llm_azure_openai.py @@ -13,9 +13,7 @@ def azure_openai_llm(mocker): class TestOpenAILLM: def test_azure_openai_llm_init_with_api_key(self, azure_openai_llm): assert azure_openai_llm.client is not None, "Client should be initialized" - assert ( - azure_openai_llm.name == "gpt-3.5-turbo" - ), "Default name not set correctly" + assert azure_openai_llm.name == "gpt-4o", "Default name not set correctly" def test_azure_openai_llm_init_success(self, mocker): mocker.patch("os.getenv", return_value="fake-api-key") diff --git a/tests/unit/llms/test_llm_openai.py b/tests/unit/llms/test_llm_openai.py index c5f98710..4287fc88 100644 --- a/tests/unit/llms/test_llm_openai.py +++ b/tests/unit/llms/test_llm_openai.py @@ -43,7 +43,7 @@ def openai_llm(mocker): class TestOpenAILLM: def test_openai_llm_init_with_api_key(self, openai_llm): assert openai_llm.client is not None, "Client should be initialized" - assert openai_llm.name == "gpt-3.5-turbo", "Default name not set correctly" + assert openai_llm.name == "gpt-4o", "Default name not set correctly" def test_openai_llm_init_success(self, mocker): mocker.patch("os.getenv", return_value="fake-api-key") diff --git a/tests/unit/test_hybrid_layer.py b/tests/unit/test_hybrid_layer.py index 416eb934..fbf14566 100644 --- a/tests/unit/test_hybrid_layer.py +++ b/tests/unit/test_hybrid_layer.py @@ -40,7 +40,7 @@ def cohere_encoder(mocker): @pytest.fixture def openai_encoder(mocker): mocker.patch.object(OpenAIEncoder, "__call__", side_effect=mock_encoder_call) - return OpenAIEncoder(name="text-embedding-ada-002", openai_api_key="test_api_key") + return OpenAIEncoder(name="text-embedding-3-small", openai_api_key="test_api_key") @pytest.fixture @@ -88,8 +88,8 @@ def test_initialization(self, openai_encoder, routes): alpha=0.8, ) assert route_layer.index is not None and route_layer.categories is not None - assert openai_encoder.score_threshold == 0.82 - assert route_layer.score_threshold == 0.82 + assert openai_encoder.score_threshold == 0.3 + assert route_layer.score_threshold == 0.3 assert route_layer.top_k == 10 assert route_layer.alpha == 0.8 assert len(route_layer.index) == 5 @@ -104,7 +104,7 @@ def test_initialization_different_encoders(self, cohere_encoder, openai_encoder) route_layer_openai = HybridRouteLayer( encoder=openai_encoder, sparse_encoder=sparse_encoder ) - assert route_layer_openai.score_threshold == 0.82 + assert route_layer_openai.score_threshold == 0.3 def test_add_route(self, openai_encoder): route_layer = HybridRouteLayer( diff --git a/tests/unit/test_layer.py b/tests/unit/test_layer.py index 4566401d..05979cd2 100644 --- a/tests/unit/test_layer.py +++ b/tests/unit/test_layer.py @@ -87,7 +87,7 @@ def cohere_encoder(mocker): @pytest.fixture def openai_encoder(mocker): mocker.patch.object(OpenAIEncoder, "__call__", side_effect=mock_encoder_call) - return OpenAIEncoder(name="text-embedding-ada-002", openai_api_key="test_api_key") + return OpenAIEncoder(name="text-embedding-3-small", openai_api_key="test_api_key") @pytest.fixture @@ -155,8 +155,8 @@ def test_initialization(self, openai_encoder, routes, index_cls): route_layer = RouteLayer( encoder=openai_encoder, routes=routes, top_k=10, index=index_cls() ) - assert openai_encoder.score_threshold == 0.82 - assert route_layer.score_threshold == 0.82 + assert openai_encoder.score_threshold == 0.3 + assert route_layer.score_threshold == 0.3 assert route_layer.top_k == 10 assert len(route_layer.index) if route_layer.index is not None else 0 == 5 assert ( @@ -172,7 +172,7 @@ def test_initialization_different_encoders( assert cohere_encoder.score_threshold == 0.3 assert route_layer_cohere.score_threshold == 0.3 route_layer_openai = RouteLayer(encoder=openai_encoder, index=index_cls()) - assert route_layer_openai.score_threshold == 0.82 + assert route_layer_openai.score_threshold == 0.3 def test_initialization_no_encoder(self, openai_encoder, index_cls): os.environ["OPENAI_API_KEY"] = "test_api_key" @@ -189,8 +189,8 @@ def test_initialization_dynamic_route( route_layer_openai = RouteLayer( encoder=openai_encoder, routes=dynamic_routes, index=index_cls() ) - assert openai_encoder.score_threshold == 0.82 - assert route_layer_openai.score_threshold == 0.82 + assert openai_encoder.score_threshold == 0.3 + assert route_layer_openai.score_threshold == 0.3 def test_add_route(self, openai_encoder, index_cls): route_layer = RouteLayer(encoder=openai_encoder, index=index_cls()) @@ -542,7 +542,7 @@ def test_get_thresholds(self, openai_encoder, routes, index_cls): route_layer = RouteLayer( encoder=openai_encoder, routes=routes, index=index_cls() ) - assert route_layer.get_thresholds() == {"Route 1": 0.82, "Route 2": 0.82} + assert route_layer.get_thresholds() == {"Route 1": 0.3, "Route 2": 0.3} def test_with_multiple_routes_passing_threshold( self, openai_encoder, routes, index_cls