From 7b95ff069a7c41f9c94084da6357c947ae0f2043 Mon Sep 17 00:00:00 2001 From: Aymeric Date: Wed, 3 Jul 2024 19:16:02 +0200 Subject: [PATCH 1/3] Add Agentic RAG --- notebooks/en/_toctree.yml | 4 +- notebooks/en/agent_rag.ipynb | 2800 +++++++++++++++++++++++++++++ notebooks/en/agents.ipynb | 6 +- notebooks/en/rag_evaluation.ipynb | 6 +- 4 files changed, 2809 insertions(+), 7 deletions(-) create mode 100644 notebooks/en/agent_rag.ipynb diff --git a/notebooks/en/_toctree.yml b/notebooks/en/_toctree.yml index 7e6386d9..31911acd 100644 --- a/notebooks/en/_toctree.yml +++ b/notebooks/en/_toctree.yml @@ -61,4 +61,6 @@ - title: Agents sections: - local: agents - title: Build an agent with tool-calling superpowers using Transformers Agents \ No newline at end of file + title: Build an agent with tool-calling superpowers using Transformers Agents + - local: agent_rag + title: Self-correcting RAG with an Agent diff --git a/notebooks/en/agent_rag.ipynb b/notebooks/en/agent_rag.ipynb new file mode 100644 index 00000000..7a0dc6ec --- /dev/null +++ b/notebooks/en/agent_rag.ipynb @@ -0,0 +1,2800 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Self-correcting RAG with an Agent\n", + "_Authored by: [Aymeric Roucher](https://huggingface.co/m-ric)_\n", + "\n", + "> This tutorial is advanced. You should have notions from [this other cookbook](advanced_rag) first!\n", + "\n", + "> Reminder: Retrieval-Augmented-Generation (RAG) is “using an LLM to answer a user query, but basing the answer on information retrieved from a knowledge base”. It has many advantages over using a vanilla or fine-tuned LLM: to name a few, it allows to ground the answer on true facts and reduce confabulations, it allows to provide the LLM with domain-specific knowledge, and it allows fine-grained control of access to information from the knowledge base.\n", + "\n", + "But vanilla RAG has limitations:\n", + "- It **performs only one retrieval step** of retrieval**: if the results are bad, the generation in turn will be bad.\n", + "- **Documents are retrieved or not based on semantic similarity with the user query**: but for instance, if the user query is a question and the document containing the true answer is in affirmative voice, its similarity score will be downgraded compared to other questions, so it might not be ranked high enough to be retrieved.\n", + "\n", + "Many frameworks have been proposed to alleviate these problems, with ideas in this vein:\n", + "- Instead of directly using the user query in semantic search, let the LLM formulate a sentence closer to the documents ([HyDE](https://huggingface.co/papers/2212.10496))\n", + "- Critique the generated snippets and re-retrieve if needed as in [Self-Query](https://docs.llamaindex.ai/en/stable/examples/evaluation/RetryQuery/)\n", + "\n", + "But we can recover both these by making a **RAG agent: very simply, an agent armed with a retriever tool!**\n", + "\n", + "This agent will: ✅ Formulate the query itself and ✅ Critique to re-retrieve if needed. So it should naively recover some advanced RAG techniques!\n", + "\n", + "Let's setup this system. \n", + "\n", + "Run the line below to install required dependancies:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install pandas langchain sentence-transformers faiss-cpu \"transformers[agents]\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We first load a knowledge base on which we want to perform RAG: this dataset is a compilation of the documentation pages for many `huggingface` packages, stored as markdown." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "import datasets\n", + "\n", + "knowledge_base = datasets.load_dataset(\"m-ric/huggingface_doc\", split=\"train\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we prepare the knowledge base by processing the dataset and storing it into a vector database to be used by the retriever. We are going to use LangChain, since it features excellent utilities for vector databases:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Splitting documents...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 2647/2647 [00:34<00:00, 75.77it/s] \n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Embedding documents... This should take a few minutes (5 minutes on my MacBook with M1 Pro, yes I'm GPU-quite-rich)\n" + ] + } + ], + "source": [ + "from transformers import AutoTokenizer\n", + "from langchain.docstore.document import Document\n", + "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", + "from langchain.vectorstores import FAISS\n", + "from langchain_community.embeddings import HuggingFaceEmbeddings\n", + "from langchain_community.vectorstores.utils import DistanceStrategy\n", + "from tqdm import tqdm\n", + "\n", + "source_docs = [\n", + " Document(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"].split(\"/\")[1]})\n", + " for doc in knowledge_base\n", + "]\n", + "\n", + "text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(\n", + " AutoTokenizer.from_pretrained(\"thenlper/gte-small\"),\n", + " chunk_size=200,\n", + " chunk_overlap=20,\n", + " add_start_index=True,\n", + " strip_whitespace=True,\n", + " separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n", + ")\n", + "\n", + "# Split docs and keep only unique ones\n", + "print(\"Splitting documents...\")\n", + "docs_processed = []\n", + "unique_texts = {}\n", + "for doc in tqdm(source_docs):\n", + " new_docs = text_splitter.split_documents([doc])\n", + " for new_doc in new_docs:\n", + " if new_doc.page_content not in unique_texts:\n", + " unique_texts[doc.page_content] = True\n", + " docs_processed.append(new_doc)\n", + "\n", + "print(\n", + " \"Embedding documents... This should take a few minutes (5 minutes on my MacBook with M1 Pro, yes I'm GPU-quite-rich)\"\n", + ")\n", + "embedding_model = HuggingFaceEmbeddings(model_name=\"thenlper/gte-small\")\n", + "vectordb = FAISS.from_documents(\n", + " documents=docs_processed,\n", + " embedding=embedding_model,\n", + " distance_strategy=DistanceStrategy.COSINE,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have the database ready, let’s build a RAG system that answers user queries based on it!\n", + "\n", + "We want our system to select only from the most relevant sources of information, depending on the query." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "👉 Let us build our RAG system as an agent that will be free to choose its query and can iterate on the retrieval results!" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from transformers.agents import Tool\n", + "from langchain_core.vectorstores import VectorStore\n", + "\n", + "\n", + "class RetrieverTool(Tool):\n", + " name = \"retriever\"\n", + " description = \"Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\"\n", + " inputs = {\n", + " \"query\": {\n", + " \"type\": \"text\",\n", + " \"description\": \"The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.\",\n", + " }\n", + " }\n", + " output_type = \"text\"\n", + "\n", + " def __init__(self, vectordb: VectorStore, **kwargs):\n", + " super().__init__(**kwargs)\n", + " self.vectordb = vectordb\n", + "\n", + " def forward(self, query: str, number_of_documents=7) -> str:\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + " number_of_documents = max(int(number_of_documents), 5)\n", + "\n", + " docs = self.vectordb.similarity_search(\n", + " query,\n", + " k=7,\n", + " )\n", + "\n", + " return \"\\nRetrieved documents:\\n\" + \"\".join(\n", + " [\n", + " f\"===== Document {str(i)} =====\\n\" + doc.page_content\n", + " for i, doc in enumerate(docs)\n", + " ]\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now it’s straightforward to create an agent that leverages this tool!\n", + "\n", + "The agent will need these arguments upon initialization:\n", + "- *`tools`*: a list of tools that the agent will be able to call.\n", + "- *`llm_engine`*: the LLM that powers the agent.\n", + "\n", + "Our `llm_engine` must be a callable that takes as input a list of [messages](https://huggingface.co/docs/transformers/main/chat_templating) and returns text. It also needs to accept a `stop_sequences` argument that indicates when to stop its generation. For convenience, we directly use the `HfEngine` class provided in the package to get a LLM engine that calls our [Inference API](https://huggingface.co/docs/api-inference/en/index)." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers.agents import HfEngine, ReactJsonAgent\n", + "\n", + "llm_engine = HfEngine(\"CohereForAI/c4ai-command-r-plus\")\n", + "\n", + "retriever_tool = RetrieverTool(vectordb)\n", + "agent = ReactJsonAgent(\n", + " tools=[retriever_tool], llm_engine=llm_engine, max_iterations=4, verbose=2\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since we initialized the agent as a `ReactJsonAgent`, it has been automatically given a default system prompt that tells the LLM engine to process step-by-step and generate tool calls as JSON blobs (you could replace this prompt template with your own as needed).\n", + "\n", + "Then when its `.run()` method is launched, the agent takes care of calling the LLM engine, parsing the tool call JSON blobs and executing these tool calls, all in a loop that ends only when the final answer is provided." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mWhat is the task performed by the `roberta-large-mnli` checkpoint?\u001b[0m\n", + "\u001b[38;20mSystem prompt is as follows:\u001b[0m\n", + "\u001b[38;20mYou are an expert assistant who can solve any task using JSON tool calls. You will be given a task to solve as best you can.\n", + "To do so, you have been given access to the following tools: 'retriever', 'final_answer'\n", + "The way you use the tools is by specifying a json blob, ending with ''.\n", + "Specifically, this json should have an `action` key (name of the tool to use) and an `action_input` key (input to the tool).\n", + "\n", + "The $ACTION_JSON_BLOB should only contain a SINGLE action, do NOT return a list of multiple actions. It should be formatted in json. Do not try to escape special characters. Here is the template of a valid $ACTION_JSON_BLOB:\n", + "{\n", + " \"action\": $TOOL_NAME,\n", + " \"action_input\": $INPUT\n", + "}\n", + "\n", + "Make sure to have the $INPUT as a dictionary in the right format for the tool you are using, and do not put variable names as input if you can find the right values.\n", + "\n", + "You should ALWAYS use the following format:\n", + "\n", + "Thought: you should always think about one action to take. Then use the action as follows:\n", + "Action:\n", + "$ACTION_JSON_BLOB\n", + "Observation: the result of the action\n", + "... (this Thought/Action/Observation can repeat N times, you should take several steps when needed. The $ACTION_JSON_BLOB must only use a SINGLE action at a time.)\n", + "\n", + "You can use the result of the previous action as input for the next action.\n", + "The observation will always be a string: it can represent a file, like \"image_1.jpg\".\n", + "Then you can use it as input for the next action. You can do it for instance as follows:\n", + "\n", + "Observation: \"image_1.jpg\"\n", + "\n", + "Thought: I need to transform the image that I received in the previous observation to make it green.\n", + "Action:\n", + "{\n", + " \"action\": \"image_transformer\",\n", + " \"action_input\": {\"image\": \"image_1.jpg\"}\n", + "}\n", + "\n", + "To provide the final answer to the task, use an action blob with \"action\": \"final_answer\" tool. It is the only way to complete the task, else you will be stuck on a loop. So your final output should look like this:\n", + "Action:\n", + "{\n", + " \"action\": \"final_answer\",\n", + " \"action_input\": {\"answer\": \"insert your final answer here\"}\n", + "}\n", + "\n", + "\n", + "Here are a few examples using notional tools:\n", + "---\n", + "Task: \"Generate an image of the oldest person in this document.\"\n", + "\n", + "Thought: I will proceed step by step and use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.\n", + "Action:\n", + "{\n", + " \"action\": \"document_qa\",\n", + " \"action_input\": {\"document\": \"document.pdf\", \"question\": \"Who is the oldest person mentioned?\"}\n", + "}\n", + "Observation: \"The oldest person in the document is John Doe, a 55 year old lumberjack living in Newfoundland.\"\n", + "\n", + "\n", + "Thought: I will now generate an image showcasing the oldest person.\n", + "Action:\n", + "{\n", + " \"action\": \"image_generator\",\n", + " \"action_input\": {\"text\": \"\"A portrait of John Doe, a 55-year-old man living in Canada.\"\"}\n", + "}\n", + "Observation: \"image.png\"\n", + "\n", + "Thought: I will now return the generated image.\n", + "Action:\n", + "{\n", + " \"action\": \"final_answer\",\n", + " \"action_input\": \"image.png\"\n", + "}\n", + "\n", + "---\n", + "Task: \"What is the result of the following operation: 5 + 3 + 1294.678?\"\n", + "\n", + "Thought: I will use python code evaluator to compute the result of the operation and then return the final answer using the `final_answer` tool\n", + "Action:\n", + "{\n", + " \"action\": \"python_interpreter\",\n", + " \"action_input\": {\"code\": \"5 + 3 + 1294.678\"}\n", + "}\n", + "Observation: 1302.678\n", + "\n", + "Thought: Now that I know the result, I will now return it.\n", + "Action:\n", + "{\n", + " \"action\": \"final_answer\",\n", + " \"action_input\": \"1302.678\"\n", + "}\n", + "\n", + "---\n", + "Task: \"Which city has the highest population , Guangzhou or Shanghai?\"\n", + "\n", + "Thought: I need to get the populations for both cities and compare them: I will use the tool `search` to get the population of both cities.\n", + "Action:\n", + "{\n", + " \"action\": \"search\",\n", + " \"action_input\": \"Population Guangzhou\"\n", + "}\n", + "Observation: ['Guangzhou has a population of 15 million inhabitants as of 2021.']\n", + "\n", + "\n", + "Thought: Now let's get the population of Shanghai using the tool 'search'.\n", + "Action:\n", + "{\n", + " \"action\": \"search\",\n", + " \"action_input\": \"Population Shanghai\"\n", + "}\n", + "Observation: '26 million (2019)'\n", + "\n", + "Thought: Now I know that Shanghai has a larger population. Let's return the result.\n", + "Action:\n", + "{\n", + " \"action\": \"final_answer\",\n", + " \"action_input\": \"Shanghai\"\n", + "}\n", + "\n", + "\n", + "Above example were using notional tools that might not exist for you. You only have acces to those tools:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\n", + "- final_answer: Provides a final answer to the given problem.\n", + " Takes inputs: {'answer': {'type': 'text', 'description': 'The final answer to the problem'}}\n", + "\n", + "Here are the rules you should always follow to solve your task:\n", + "1. ALWAYS provide a 'Thought:' sequence, and an 'Action:' sequence that ends with , else you will fail.\n", + "2. Always use the right arguments for the tools. Never use variable names in the 'action_input' field, use the value instead.\n", + "3. Call a tool only when needed: do not call the search agent if you do not need information, try to solve the task yourself.\n", + "4. Never re-do a tool call that you previously did with the exact same parameters.\n", + "\n", + "Now Begin! If you solve the task correctly, you will receive a reward of $1,000,000.\n", + "\u001b[0m\n", + "\u001b[38;20m===== New step =====\u001b[0m\n", + "===== Calling LLM with this last message: =====\n", + "{'role': , 'content': 'Task: What is the task performed by the `roberta-large-mnli` checkpoint?'}\n", + "\u001b[38;20m===== Output message of the LLM: =====\u001b[0m\n", + "\u001b[38;20mThought: To answer this question, I first need to retrieve some information about the \"roberta-large-mnli\" checkpoint. I can do this using the retriever tool with the query \"roberta-large-mnli checkpoint task\".\n", + "\n", + "Action: ```json\n", + "{\n", + " \"action\": \"retriever\",\n", + " \"action_input\": {\n", + " \"query\": \"roberta-large-mnli checkpoint task\"\n", + " }\n", + "}\u001b[0m\n", + "\u001b[38;20m===== Extracting action =====\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'roberta-large-mnli checkpoint task'}\u001b[0m\n", + "Retrieved documents:\n", + "===== Document 0 =====\n", + "!-- DISABLE-FRONTMATTER-SECTIONS -->\n", + "\n", + "# End-of-chapter quiz[[end-of-chapter-quiz]]\n", + "\n", + "\n", + "\n", + "This chapter covered a lot of ground! Don't worry if you didn't grasp all the details; the next chapters will help you understand how things work under the hood.\n", + "\n", + "First, though, let's test what you learned in this chapter!\n", + "\n", + "\n", + "### 1. Explore the Hub and look for the `roberta-large-mnli` checkpoint. What task does it perform?===== Document 1 =====\n", + "## Models\n", + "\n", + "### RoBERTa\n", + "\n", + "#### Load RoBERTa Checkpoints for the Classification Task\n", + "\n", + "We load the pre-trained RoBERTa model with a sequence classification head using the Hugging Face `AutoModelForSequenceClassification` class:\n", + "\n", + "```python\n", + "from transformers import AutoModelForSequenceClassification \n", + "roberta_model = AutoModelForSequenceClassification.from_pretrained(roberta_checkpoint, num_labels=2)\n", + "```\n", + "\n", + "\n", + "#### LoRA setup for RoBERTa classifier===== Document 2 =====\n", + "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", + "\n", + "\n", + "\n", + "## StableDiffusionInpaintPipeline\n", + "\n", + "[[autodoc]] StableDiffusionInpaintPipeline\n", + "\t- all\n", + "\t- __call__\n", + "\t- enable_attention_slicing\n", + "\t- disable_attention_slicing\n", + "\t- enable_xformers_memory_efficient_attention\n", + "\t- disable_xformers_memory_efficient_attention\n", + "\t- load_textual_inversion\n", + "\t- load_lora_weights\n", + "\t- save_lora_weights===== Document 3 =====\n", + "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", + "\n", + "\n", + "\n", + "## StableDiffusionDepth2ImgPipeline\n", + "\n", + "[[autodoc]] StableDiffusionDepth2ImgPipeline\n", + "\t- all\n", + "\t- __call__\n", + "\t- enable_attention_slicing\n", + "\t- disable_attention_slicing\n", + "\t- enable_xformers_memory_efficient_attention\n", + "\t- disable_xformers_memory_efficient_attention\n", + "\t- load_textual_inversion\n", + "\t- load_lora_weights\n", + "\t- save_lora_weights===== Document 4 =====\n", + "## Setup\n", + "\n", + "RoBERTa has a limitatiom of maximum sequence length of 512, so we set the `MAX_LEN=512` for all models to ensure a fair comparison.\n", + "\n", + "```python\n", + "MAX_LEN = 512 \n", + "roberta_checkpoint = \"roberta-large\"\n", + "mistral_checkpoint = \"mistralai/Mistral-7B-v0.1\"\n", + "llama_checkpoint = \"meta-llama/Llama-2-7b-hf\"\n", + "```\n", + "\n", + "## Data preparation\n", + "### Data loading\n", + "\n", + "We will load the dataset from Hugging Face:\n", + "```python\n", + "from datasets import load_dataset\n", + "dataset = load_dataset(\"mehdiiraqui/twitter_disaster\")\n", + "```\n", + " Now, let's split the dataset into training and validation datasets. Then add the test set:===== Document 5 =====\n", + "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", + "\n", + "\n", + "\n", + "## StableDiffusionLatentUpscalePipeline\n", + "\n", + "[[autodoc]] StableDiffusionLatentUpscalePipeline\n", + "\t- all\n", + "\t- __call__\n", + "\t- enable_sequential_cpu_offload\n", + "\t- enable_attention_slicing\n", + "\t- disable_attention_slicing\n", + "\t- enable_xformers_memory_efficient_attention\n", + "\t- disable_xformers_memory_efficient_attention\n", + "\n", + "## StableDiffusionPipelineOutput===== Document 6 =====\n", + "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", + "\n", + "\n", + "\n", + "## StableDiffusionUpscalePipeline\n", + "\n", + "[[autodoc]] StableDiffusionUpscalePipeline\n", + "\t- all\n", + "\t- __call__\n", + "\t- enable_attention_slicing\n", + "\t- disable_attention_slicing\n", + "\t- enable_xformers_memory_efficient_attention\n", + "\t- disable_xformers_memory_efficient_attention\n", + "\n", + "## StableDiffusionPipelineOutput\n", + "\u001b[38;20m===== New step =====\u001b[0m\n", + "===== Calling LLM with this last message: =====\n", + "{'role': , 'content': 'Observation: Retrieved documents:\\n===== Document 0 =====\\n!-- DISABLE-FRONTMATTER-SECTIONS -->\\n\\n# End-of-chapter quiz[[end-of-chapter-quiz]]\\n\\n\\n\\nThis chapter covered a lot of ground! Don\\'t worry if you didn\\'t grasp all the details; the next chapters will help you understand how things work under the hood.\\n\\nFirst, though, let\\'s test what you learned in this chapter!\\n\\n\\n### 1. Explore the Hub and look for the `roberta-large-mnli` checkpoint. What task does it perform?===== Document 1 =====\\n## Models\\n\\n### RoBERTa\\n\\n#### Load RoBERTa Checkpoints for the Classification Task\\n\\nWe load the pre-trained RoBERTa model with a sequence classification head using the Hugging Face `AutoModelForSequenceClassification` class:\\n\\n```python\\nfrom transformers import AutoModelForSequenceClassification \\nroberta_model = AutoModelForSequenceClassification.from_pretrained(roberta_checkpoint, num_labels=2)\\n```\\n\\n\\n#### LoRA setup for RoBERTa classifier===== Document 2 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionInpaintPipeline\\n\\n[[autodoc]] StableDiffusionInpaintPipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\t- load_textual_inversion\\n\\t- load_lora_weights\\n\\t- save_lora_weights===== Document 3 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionDepth2ImgPipeline\\n\\n[[autodoc]] StableDiffusionDepth2ImgPipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\t- load_textual_inversion\\n\\t- load_lora_weights\\n\\t- save_lora_weights===== Document 4 =====\\n## Setup\\n\\nRoBERTa has a limitatiom of maximum sequence length of 512, so we set the `MAX_LEN=512` for all models to ensure a fair comparison.\\n\\n```python\\nMAX_LEN = 512 \\nroberta_checkpoint = \"roberta-large\"\\nmistral_checkpoint = \"mistralai/Mistral-7B-v0.1\"\\nllama_checkpoint = \"meta-llama/Llama-2-7b-hf\"\\n```\\n\\n## Data preparation\\n### Data loading\\n\\nWe will load the dataset from Hugging Face:\\n```python\\nfrom datasets import load_dataset\\ndataset = load_dataset(\"mehdiiraqui/twitter_disaster\")\\n```\\n Now, let\\'s split the dataset into training and validation datasets. Then add the test set:===== Document 5 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionLatentUpscalePipeline\\n\\n[[autodoc]] StableDiffusionLatentUpscalePipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_sequential_cpu_offload\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\n## StableDiffusionPipelineOutput===== Document 6 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionUpscalePipeline\\n\\n[[autodoc]] StableDiffusionUpscalePipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\n## StableDiffusionPipelineOutput'}\n", + "\u001b[38;20m===== Output message of the LLM: =====\u001b[0m\n", + "\u001b[38;20mThought: In the retrieved documents, I found the answer: \"roberta-large-mnli performs a classification task\". Now I can provide the final answer.\n", + "\n", + "Action: ```json\n", + "{\n", + " \"action\": \"final_answer\",\n", + " \"action_input\": {\n", + " \"answer\": \"The roberta-large-mnli checkpoint performs a classification task.\"\n", + " }\n", + "}\u001b[0m\n", + "\u001b[38;20m===== Extracting action =====\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The roberta-large-mnli checkpoint performs a classification task.'}\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Final output:\n", + "The roberta-large-mnli checkpoint performs a classification task.\n" + ] + } + ], + "source": [ + "agent_output = agent.run(\n", + " \"What is the task performed by the `roberta-large-mnli` checkpoint?\"\n", + ")\n", + "\n", + "print(\"Final output:\")\n", + "print(agent_output)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Evaluate with LLM judge\n", + "\n", + "Does the agent setup improve the performance of our RAG system? Well, let's measure it.\n", + "\n", + "We'll compare the score of our agent system with the different RAG systems tested in [this other cookbook](advanced_rag).\n", + "\n", + "So we take care to use the same evaluation setup, with GPT-4-turbo as a judge for reproducibility. But nowadays I'd just be using [Llama-3](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "eval_dataset = datasets.load_dataset(\"m-ric/huggingface_doc_qa_eval\", split=\"train\")" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], + "source": [ + "EVALUATION_PROMPT = \"\"\"###Task Description:\n", + "An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n", + "1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n", + "2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n", + "3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n", + "4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n", + "\n", + "###The instruction to evaluate:\n", + "{instruction}\n", + "\n", + "###Response to evaluate:\n", + "{response}\n", + "\n", + "###Reference Answer (Score 5):\n", + "{reference_answer}\n", + "\n", + "###Score Rubrics:\n", + "[Is the response correct, accurate, and factual based on the reference answer?]\n", + "Score 1: The response is completely incorrect, inaccurate, and/or not factual.\n", + "Score 2: The response is mostly incorrect, inaccurate, and/or not factual.\n", + "Score 3: The response is somewhat correct, accurate, and/or factual.\n", + "Score 4: The response is mostly correct, accurate, and factual.\n", + "Score 5: The response is completely correct, accurate, and factual.\n", + "\n", + "###Feedback:\"\"\"\n", + "\n", + "from openai import OpenAI\n", + "import os\n", + "\n", + "client = OpenAI(\n", + " api_key=os.getenv(\"OPENAI_API_KEY\"),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before running the test let's make the agent less verbose." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import logging\n", + "\n", + "agent.logger.setLevel(logging.WARNING)" + ] + }, + { + "cell_type": "code", + "execution_count": 79, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 0%| | 0/65 [00:00 'Package Manager'.\\n3. Click '+' and select 'Add Package from git URL'.\\n4. Enter the URL: 'https://github.com/huggingface/unity-api.git'.\\n5. Once installed, the Unity API wizard should appear. If not, go to 'Window' -> 'Hugging Face API Wizard'.\\n6. Enter your API key, which can be created in your Hugging Face account settings.\\n7. Test the API key using the 'Test API key' option in the wizard.\\n8. Optionally, change the model endpoints to select a different model to use.\\n9. Configure any advanced settings as desired.\\n10. Install examples to see how to use the API.\"}\u001b[0m\n", + " 80%|████████ | 52/65 [12:36<03:42, 17.14s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the pretraining objective of the Wav2Vec2 context network?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How can you install the Hugging Face Unity API in your Unity project?\n", + "\n", + "Answer: To install the Hugging Face Unity API in your Unity project, follow these steps: \n", + "1. Open your Unity project.\n", + "2. Go to 'Window' -> 'Package Manager'.\n", + "3. Click '+' and select 'Add Package from git URL'.\n", + "4. Enter the URL: 'https://github.com/huggingface/unity-api.git'.\n", + "5. Once installed, the Unity API wizard should appear. If not, go to 'Window' -> 'Hugging Face API Wizard'.\n", + "6. Enter your API key, which can be created in your Hugging Face account settings.\n", + "7. Test the API key using the 'Test API key' option in the wizard.\n", + "8. Optionally, change the model endpoints to select a different model to use.\n", + "9. Configure any advanced settings as desired.\n", + "10. Install examples to see how to use the API.\n", + "True answer: To install the Hugging Face Unity API in your Unity project, go to `Window` -> `Package Manager`, click `+` and select `Add Package from git URL`, then enter `https://github.com/huggingface/unity-api.git`.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the pretraining objective of the Wav2Vec2 context network?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The pretraining objective of the Wav2Vec2 context network is a contrastive task. The model predicts a true quantized speech representation, from a set of false ones, using masked prediction. By doing so, the model finds the most similar context vector, and the quantized speech unit is the target label.'}\u001b[0m\n", + " 82%|████████▏ | 53/65 [12:48<03:06, 15.51s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the pretraining objective of the Wav2Vec2 context network?\n", + "\n", + "Answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task. The model predicts a true quantized speech representation, from a set of false ones, using masked prediction. By doing so, the model finds the most similar context vector, and the quantized speech unit is the target label.\n", + "True answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task where the model has to predict the true quantized speech representation of the masked prediction from a set of false ones.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Default checkpoint for sentiment analysis pipeline in Transformers library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english' (source document: 0).\"}\u001b[0m\n", + " 83%|████████▎ | 54/65 [13:02<02:45, 15.08s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", + "\n", + "Answer: The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english' (source document: 0).\n", + "True answer: distilbert base uncased finetuned sst2 english\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': \"What is the purpose of the notebook 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'?\"}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The purpose of the notebook is to show how to use DeepSpeed to pre-train and fine-tune the 1.6 billion parameter GPT2-XL model for causal language modeling on Habana Gaudi. This is demonstrated using a specific notebook titled 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'.\"}\u001b[0m\n", + " 85%|████████▍ | 55/65 [13:18<02:34, 15.48s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What command line module does PyTorch provide to run a script on multiple GPUs?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", + "\n", + "Answer: The purpose of the notebook is to show how to use DeepSpeed to pre-train and fine-tune the 1.6 billion parameter GPT2-XL model for causal language modeling on Habana Gaudi. This is demonstrated using a specific notebook titled 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'.\n", + "True answer: To show how to use DeepSpeed to pre-train/fine-tune the 1.6B-parameter GPT2-XL for causal language modeling on Habana Gaudi.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'PyTorch command line module to run a script on multiple GPUs'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How many nodes can you use with torchrun?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'PyTorch provides the command-line module `torchrun` to run a script on multiple GPUs. You can specify the number of GPUs using the argument `--nproc_per_node=integer`.'}\u001b[0m\n", + " 86%|████████▌ | 56/65 [13:35<02:21, 15.74s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What command line module does PyTorch provide to run a script on multiple GPUs?\n", + "\n", + "Answer: PyTorch provides the command-line module `torchrun` to run a script on multiple GPUs. You can specify the number of GPUs using the argument `--nproc_per_node=integer`.\n", + "True answer: torchrun\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Most popular vision transformer models on Hugging Face model hub'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'vision transformer model image classification'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The most popular vision transformer model on the Hugging Face Model Hub for image classification is `google/vit-base-patch16-224`.'}\u001b[0m\n", + " 88%|████████▊ | 57/65 [13:52<02:10, 16.30s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the command to upload an ESPnet model to a Hugging Face repository?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", + "\n", + "Answer: The most popular vision transformer model on the Hugging Face Model Hub for image classification is `google/vit-base-patch16-224`.\n", + "True answer: google/vit-base-patch16-224\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How to upload an ESPnet model to a Hugging Face repository?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To upload an ESPnet model to a Hugging Face repository, ensure you have an account and are logged in. Then, use the `run.sh` script with the appropriate arguments, specifying the stage, whether to skip the upload, and the name of your repository. Here is an example command: \\nbash\\n./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\\n \\nAlternatively, you can use the Hugging Face CLI with the `push_to_hub` argument, specifying the model directory and repository name.'}\u001b[0m\n", + " 89%|████████▉ | 58/65 [14:12<02:00, 17.20s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the command to upload an ESPnet model to a Hugging Face repository?\n", + "\n", + "Answer: To upload an ESPnet model to a Hugging Face repository, ensure you have an account and are logged in. Then, use the `run.sh` script with the appropriate arguments, specifying the stage, whether to skip the upload, and the name of your repository. Here is an example command: \n", + "bash\n", + "./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n", + " \n", + "Alternatively, you can use the Hugging Face CLI with the `push_to_hub` argument, specifying the model directory and repository name.\n", + "True answer: ./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'A `requirements.txt` file should be added to a model repository to install custom Python dependencies for Inference Endpoints.'}\u001b[0m\n", + " 91%|█████████ | 59/65 [14:25<01:36, 16.11s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", + "\n", + "Answer: A `requirements.txt` file should be added to a model repository to install custom Python dependencies for Inference Endpoints.\n", + "True answer: requirements.txt\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Between 3 and 5 images are needed to teach new concepts to Stable Diffusion using Textual Inversion.'}\u001b[0m\n", + " 92%|█████████▏| 60/65 [14:41<01:19, 15.99s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", + "\n", + "Answer: Between 3 and 5 images are needed to teach new concepts to Stable Diffusion using Textual Inversion.\n", + "True answer: 3-5 images\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Transformers version 4.18.0 model checkpoint automatic sharding'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'In Transformers version 4.18.0, model checkpoints larger than 10GB are automatically sharded. [Source Document 0]'}\u001b[0m\n", + " 94%|█████████▍| 61/65 [14:51<00:56, 14.07s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", + "\n", + "Answer: In Transformers version 4.18.0, model checkpoints larger than 10GB are automatically sharded. [Source Document 0]\n", + "True answer: 10GB\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Weights and Biases (W&B) is a tool that allows data scientists and machine learning scientists to track their experiments at every stage, from training to production. It provides a customisable dashboard to visualise metrics and samples, helping users to optimise their machine learning workflows.'}\u001b[0m\n", + " 95%|█████████▌| 62/65 [15:04<00:41, 13.97s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", + "\n", + "Answer: Weights and Biases (W&B) is a tool that allows data scientists and machine learning scientists to track their experiments at every stage, from training to production. It provides a customisable dashboard to visualise metrics and samples, helping users to optimise their machine learning workflows.\n", + "True answer: To track their machine learning experiments at every stage, from training to production.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face open-source library Transformer acceleration'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum (also referred to as '🤗 Optimum' and 'Optimum Intel'). This information is derived from multiple documents in the knowledge base.\"}\u001b[0m\n", + " 97%|█████████▋| 63/65 [15:12<00:24, 12.18s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What parameter is used to ensure that elements in a row have the same height in Gradio?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", + "\n", + "Answer: The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum (also referred to as '🤗 Optimum' and 'Optimum Intel'). This information is derived from multiple documents in the knowledge base.\n", + "True answer: Optimum\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What parameter is used to ensure that elements in a row have the same height in Gradio?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `equal_height` parameter is used to ensure that elements in a row have the same height in Gradio.'}\u001b[0m\n", + " 98%|█████████▊| 64/65 [15:24<00:12, 12.17s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "What is the command to install the latest version of Optimum with OpenVINO support?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What parameter is used to ensure that elements in a row have the same height in Gradio?\n", + "\n", + "Answer: The `equal_height` parameter is used to ensure that elements in a row have the same height in Gradio.\n", + "True answer: equal_height\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Optimum installation command with OpenVINO support'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The command to install the latest version of Optimum with OpenVINO support is:\\n\\nbash\\npip install --upgrade-strategy eager optimum['openvino']\\n\\n\\nThis command ensures that the latest version of Optimum is installed with support for OpenVINO, enabling inference on a variety of Intel processors.\"}\u001b[0m\n", + "100%|██████████| 65/65 [15:49<00:00, 14.60s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the command to install the latest version of Optimum with OpenVINO support?\n", + "\n", + "Answer: The command to install the latest version of Optimum with OpenVINO support is:\n", + "\n", + "bash\n", + "pip install --upgrade-strategy eager optimum['openvino']\n", + "\n", + "\n", + "This command ensures that the latest version of Optimum is installed with support for OpenVINO, enabling inference on a variety of Intel processors.\n", + "True answer: pip install --upgrade-strategy eager optimum[\"openvino\"]\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "from transformers.agents.agents import AgentError\n", + "\n", + "outputs = []\n", + "\n", + "\n", + "def serialize_agent_error(obj):\n", + " if isinstance(obj, AgentError):\n", + " return {\"error_type\": obj.__class__.__name__, \"message\": obj.message}\n", + " else:\n", + " return str(obj)\n", + "\n", + "\n", + "for example in tqdm(eval_dataset):\n", + " question = example[\"question\"]\n", + " if question in [el[\"question\"] for el in outputs]:\n", + " continue\n", + "\n", + " enhanced_question = f\"\"\"Using the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "Provide the number of the source document when relevant.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\n", + "Question:\n", + "{question}\"\"\"\n", + " answer = agent.run(enhanced_question)\n", + " print(\"=======================================================\")\n", + " print(f\"Question: {question}\")\n", + " print(f\"Answer: {answer}\")\n", + " print(f'True answer: {example[\"answer\"]}')\n", + "\n", + " result = {\n", + " \"question\": question,\n", + " \"true_answer\": example[\"answer\"],\n", + " \"source_doc\": example[\"source_doc\"],\n", + " \"generated_answer\": answer,\n", + " }\n", + " outputs.append(result)\n", + "\n", + " with open(\"outputs.jsonl\", \"w\") as f:\n", + " for d in outputs:\n", + " json.dump(d, f, default=serialize_agent_error)\n", + " f.write(\"\\n\") # add a newline for JSONL format" + ] + }, + { + "cell_type": "code", + "execution_count": 81, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 65 answers.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 65/65 [05:02<00:00, 4.65s/it]\n" + ] + } + ], + "source": [ + "for experiment in tqdm(outputs):\n", + " eval_prompt = EVALUATION_PROMPT.format(\n", + " instruction=experiment[\"question\"],\n", + " response=experiment[\"generated_answer\"],\n", + " reference_answer=experiment[\"true_answer\"],\n", + " )\n", + " messages = [\n", + " {\"role\": \"system\", \"content\": \"You are a fair evaluator language model.\"},\n", + " {\"role\": \"user\", \"content\": eval_prompt},\n", + " ]\n", + "\n", + " eval_result = (\n", + " client.chat.completions.create(\n", + " model=\"gpt-4-1106-preview\", messages=messages, temperature=0.5\n", + " )\n", + " .choices[0]\n", + " .message.content\n", + " )\n", + "\n", + " feedback, score = [item.strip() for item in eval_result.split(\"[RESULT]\")]\n", + " experiment[f\"eval_score_LLM_judge\"] = score\n", + " experiment[f\"eval_feedback_LLM_judge\"] = feedback" + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Average score: 0.850\n" + ] + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "result = pd.DataFrame.from_dict(outputs)\n", + "result = result.loc[~result[\"generated_answer\"].str.contains(\"Error\")]\n", + "result[\"eval_score_LLM_judge_int\"] = result[\"eval_score_LLM_judge\"].apply(\n", + " lambda x: int(x)\n", + ")\n", + "result[\"eval_score_LLM_judge_int\"] = (result[\"eval_score_LLM_judge_int\"] - 1) / 4\n", + "\n", + "print(f\"Average score: {result['eval_score_LLM_judge_int'].mean():.3f}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So we reach 85% score with this simple setup!" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "disposable", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.14" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/en/agents.ipynb b/notebooks/en/agents.ipynb index 92ba0ede..f156c8c4 100644 --- a/notebooks/en/agents.ipynb +++ b/notebooks/en/agents.ipynb @@ -703,9 +703,9 @@ ], "metadata": { "kernelspec": { - "display_name": "test2", + "display_name": "disposable", "language": "python", - "name": "test2" + "name": "python3" }, "language_info": { "codemirror_mode": { @@ -717,7 +717,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.0" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/notebooks/en/rag_evaluation.ipynb b/notebooks/en/rag_evaluation.ipynb index 16a109a9..5f144314 100644 --- a/notebooks/en/rag_evaluation.ipynb +++ b/notebooks/en/rag_evaluation.ipynb @@ -1144,7 +1144,7 @@ "source": [ "def run_rag_tests(\n", " eval_dataset: datasets.Dataset,\n", - " llm: BaseChatModel,\n", + " llm,\n", " knowledge_index: VectorStore,\n", " output_file: str,\n", " reranker: Optional[RAGPretrainedModel] = None,\n", @@ -1251,7 +1251,7 @@ "\n", "def evaluate_answers(\n", " answer_path: str,\n", - " eval_chat_model: BaseChatModel,\n", + " eval_chat_model,\n", " evaluator_name: str,\n", " evaluation_prompt_template: ChatPromptTemplate,\n", ") -> None:\n", @@ -1459,7 +1459,7 @@ " },\n", " color_continuous_scale=\"bluered\",\n", ")\n", - "fig.update_layout(w\n", + "fig.update_layout(\n", " width=1000,\n", " height=600,\n", " barmode=\"group\",\n", From a04027dd51535855b15112d7153e656cf9b1c0a6 Mon Sep 17 00:00:00 2001 From: Aymeric Date: Fri, 5 Jul 2024 18:14:56 +0200 Subject: [PATCH 2/3] Answer comments --- notebooks/en/_toctree.yml | 2 +- notebooks/en/agent_rag.ipynb | 5134 ++++++++++++++++++++++++++-------- 2 files changed, 4001 insertions(+), 1135 deletions(-) diff --git a/notebooks/en/_toctree.yml b/notebooks/en/_toctree.yml index 31911acd..dd5fa9d5 100644 --- a/notebooks/en/_toctree.yml +++ b/notebooks/en/_toctree.yml @@ -63,4 +63,4 @@ - local: agents title: Build an agent with tool-calling superpowers using Transformers Agents - local: agent_rag - title: Self-correcting RAG with an Agent + title: Agentic RAG - turbocharge your RAG with query reformulation and self-query diff --git a/notebooks/en/agent_rag.ipynb b/notebooks/en/agent_rag.ipynb index 7a0dc6ec..86c1fabc 100644 --- a/notebooks/en/agent_rag.ipynb +++ b/notebooks/en/agent_rag.ipynb @@ -4,28 +4,28 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Self-correcting RAG with an Agent\n", + "# Agentic RAG: turbocharge your RAG with query reformulation and self-query! 🚀\n", "_Authored by: [Aymeric Roucher](https://huggingface.co/m-ric)_\n", "\n", "> This tutorial is advanced. You should have notions from [this other cookbook](advanced_rag) first!\n", "\n", "> Reminder: Retrieval-Augmented-Generation (RAG) is “using an LLM to answer a user query, but basing the answer on information retrieved from a knowledge base”. It has many advantages over using a vanilla or fine-tuned LLM: to name a few, it allows to ground the answer on true facts and reduce confabulations, it allows to provide the LLM with domain-specific knowledge, and it allows fine-grained control of access to information from the knowledge base.\n", "\n", - "But vanilla RAG has limitations:\n", - "- It **performs only one retrieval step** of retrieval**: if the results are bad, the generation in turn will be bad.\n", - "- **Documents are retrieved or not based on semantic similarity with the user query**: but for instance, if the user query is a question and the document containing the true answer is in affirmative voice, its similarity score will be downgraded compared to other questions, so it might not be ranked high enough to be retrieved.\n", + "But vanilla RAG has limitations, most importantly these two:\n", + "- It **performs only one retrieval step**: if the results are bad, the generation in turn will be bad.\n", + "- __Semantic similarity is computed with the *user query* as a reference__, which might be suboptimal: for instance, the user query will often be a question and the document containing the true answer will be in affirmative voice, so its similarity score will be downgraded compared to other source documents in the interrogative form, leading to a risk of missing the relevant information.\n", "\n", - "Many frameworks have been proposed to alleviate these problems, with ideas in this vein:\n", - "- Instead of directly using the user query in semantic search, let the LLM formulate a sentence closer to the documents ([HyDE](https://huggingface.co/papers/2212.10496))\n", - "- Critique the generated snippets and re-retrieve if needed as in [Self-Query](https://docs.llamaindex.ai/en/stable/examples/evaluation/RetryQuery/)\n", + "But we can alleviate these problems by making a **RAG agent: very simply, an agent armed with a retriever tool!**\n", "\n", - "But we can recover both these by making a **RAG agent: very simply, an agent armed with a retriever tool!**\n", + "This agent will: ✅ Formulate the query itself and ✅ Critique to re-retrieve if needed.\n", "\n", - "This agent will: ✅ Formulate the query itself and ✅ Critique to re-retrieve if needed. So it should naively recover some advanced RAG techniques!\n", + "So it should naively recover some advanced RAG techniques!\n", + "- Instead of directly using the user query as the reference in semantic search, the agent formulates itself a reference sentence that can be closer to the targeted documents, as in [HyDE](https://huggingface.co/papers/2212.10496)\n", + "- The agent can the generated snippets and re-retrieve if needed, as in [Self-Query](https://docs.llamaindex.ai/en/stable/examples/evaluation/RetryQuery/)\n", "\n", - "Let's setup this system. \n", + "Let's build this system. 🛠️\n", "\n", - "Run the line below to install required dependancies:" + "Run the line below to install required dependencies:" ] }, { @@ -59,7 +59,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now we prepare the knowledge base by processing the dataset and storing it into a vector database to be used by the retriever. We are going to use LangChain, since it features excellent utilities for vector databases:" + "Now we prepare the knowledge base by processing the dataset and storing it into a vector database to be used by the retriever.\n", + "\n", + "We use [LangChain](https://python.langchain.com/) for its excellent vector database utilities.\n", + "For the embedding model, we use [thenlper/gte-small](https://huggingface.co/thenlper/gte-small) since it performed well in our `RAG_evaluation` cookbook." ] }, { @@ -78,14 +81,14 @@ "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 2647/2647 [00:34<00:00, 75.77it/s] \n" + "100%|██████████| 2647/2647 [00:34<00:00, 75.69it/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Embedding documents... This should take a few minutes (5 minutes on my MacBook with M1 Pro, yes I'm GPU-quite-rich)\n" + "Embedding documents... This should take a few minutes (5 minutes on MacBook with M1 Pro)\n" ] } ], @@ -124,7 +127,7 @@ " docs_processed.append(new_doc)\n", "\n", "print(\n", - " \"Embedding documents... This should take a few minutes (5 minutes on my MacBook with M1 Pro, yes I'm GPU-quite-rich)\"\n", + " \"Embedding documents... This should take a few minutes (5 minutes on MacBook with M1 Pro)\"\n", ")\n", "embedding_model = HuggingFaceEmbeddings(model_name=\"thenlper/gte-small\")\n", "vectordb = FAISS.from_documents(\n", @@ -152,7 +155,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "metadata": {}, "outputs": [], "source": [ @@ -176,9 +179,8 @@ " super().__init__(**kwargs)\n", " self.vectordb = vectordb\n", "\n", - " def forward(self, query: str, number_of_documents=7) -> str:\n", + " def forward(self, query: str) -> str:\n", " assert isinstance(query, str), \"Your search query must be a string\"\n", - " number_of_documents = max(int(number_of_documents), 5)\n", "\n", " docs = self.vectordb.similarity_search(\n", " query,\n", @@ -203,12 +205,16 @@ "- *`tools`*: a list of tools that the agent will be able to call.\n", "- *`llm_engine`*: the LLM that powers the agent.\n", "\n", - "Our `llm_engine` must be a callable that takes as input a list of [messages](https://huggingface.co/docs/transformers/main/chat_templating) and returns text. It also needs to accept a `stop_sequences` argument that indicates when to stop its generation. For convenience, we directly use the `HfEngine` class provided in the package to get a LLM engine that calls our [Inference API](https://huggingface.co/docs/api-inference/en/index)." + "Our `llm_engine` must be a callable that takes as input a list of [messages](https://huggingface.co/docs/transformers/main/chat_templating) and returns text. It also needs to accept a `stop_sequences` argument that indicates when to stop its generation. For convenience, we directly use the `HfEngine` class provided in the package to get a LLM engine that calls our [Inference API](https://huggingface.co/docs/api-inference/en/index).\n", + "\n", + "And we use [CohereForAI/c4ai-command-r-plus](https://huggingface.co/CohereForAI/c4ai-command-r-plus) as the llm engine because:\n", + "- It has a long 128k context, which is helpful for processing long source documents\n", + "- It is served for free at all times on HF's Inference API!" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -233,7 +239,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -241,7 +247,7 @@ "output_type": "stream", "text": [ "\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mWhat is the task performed by the `roberta-large-mnli` checkpoint?\u001b[0m\n", + "\u001b[37;1mHow can I push a model to the Hub?\u001b[0m\n", "\u001b[38;20mSystem prompt is as follows:\u001b[0m\n", "\u001b[38;20mYou are an expert assistant who can solve any task using JSON tool calls. You will be given a task to solve as best you can.\n", "To do so, you have been given access to the following tools: 'retriever', 'final_answer'\n", @@ -359,12 +365,12 @@ "}\n", "\n", "\n", - "Above example were using notional tools that might not exist for you. You only have acces to those tools:\n", + "Above example were using notional tools that might not exist for you. You only have access to those tools:\n", "\n", "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", "\n", - "- final_answer: Provides a final answer to the given problem.\n", + "- final_answer: Provides a final answer to the given problem\n", " Takes inputs: {'answer': {'type': 'text', 'description': 'The final answer to the problem'}}\n", "\n", "Here are the rules you should always follow to solve your task:\n", @@ -377,149 +383,98 @@ "\u001b[0m\n", "\u001b[38;20m===== New step =====\u001b[0m\n", "===== Calling LLM with this last message: =====\n", - "{'role': , 'content': 'Task: What is the task performed by the `roberta-large-mnli` checkpoint?'}\n", + "{'role': , 'content': 'Task: How can I push a model to the Hub?'}\n", "\u001b[38;20m===== Output message of the LLM: =====\u001b[0m\n", - "\u001b[38;20mThought: To answer this question, I first need to retrieve some information about the \"roberta-large-mnli\" checkpoint. I can do this using the retriever tool with the query \"roberta-large-mnli checkpoint task\".\n", - "\n", + "\u001b[38;20mThought: I will try to retrieve documents related to pushing a model to the hub with the retriever tool and then provide the answer based on this information.\n", "Action: ```json\n", "{\n", " \"action\": \"retriever\",\n", " \"action_input\": {\n", - " \"query\": \"roberta-large-mnli checkpoint task\"\n", + " \"query\": \"How can I push a model to the Hub?\"\n", " }\n", - "}\u001b[0m\n", + "}\n", + "\n", + "```\u001b[0m\n", "\u001b[38;20m===== Extracting action =====\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'roberta-large-mnli checkpoint task'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How can I push a model to the Hub?'}\u001b[0m\n", "Retrieved documents:\n", "===== Document 0 =====\n", - "!-- DISABLE-FRONTMATTER-SECTIONS -->\n", - "\n", - "# End-of-chapter quiz[[end-of-chapter-quiz]]\n", - "\n", - "\n", - "\n", - "This chapter covered a lot of ground! Don't worry if you didn't grasp all the details; the next chapters will help you understand how things work under the hood.\n", - "\n", - "First, though, let's test what you learned in this chapter!\n", - "\n", - "\n", - "### 1. Explore the Hub and look for the `roberta-large-mnli` checkpoint. What task does it perform?===== Document 1 =====\n", - "## Models\n", - "\n", - "### RoBERTa\n", - "\n", - "#### Load RoBERTa Checkpoints for the Classification Task\n", - "\n", - "We load the pre-trained RoBERTa model with a sequence classification head using the Hugging Face `AutoModelForSequenceClassification` class:\n", - "\n", - "```python\n", - "from transformers import AutoModelForSequenceClassification \n", - "roberta_model = AutoModelForSequenceClassification.from_pretrained(roberta_checkpoint, num_labels=2)\n", + "# Step 7. Push everything to the Hub\n", + " api.upload_folder(\n", + " repo_id=repo_id,\n", + " folder_path=repo_local_path,\n", + " path_in_repo=\".\",\n", + " )\n", + "\n", + " print(\"Your model is pushed to the Hub. You can view your model here: \", repo_url)\n", "```\n", "\n", + "### .\n", "\n", - "#### LoRA setup for RoBERTa classifier===== Document 2 =====\n", - "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", - "\n", - "\n", - "\n", - "## StableDiffusionInpaintPipeline\n", - "\n", - "[[autodoc]] StableDiffusionInpaintPipeline\n", - "\t- all\n", - "\t- __call__\n", - "\t- enable_attention_slicing\n", - "\t- disable_attention_slicing\n", - "\t- enable_xformers_memory_efficient_attention\n", - "\t- disable_xformers_memory_efficient_attention\n", - "\t- load_textual_inversion\n", - "\t- load_lora_weights\n", - "\t- save_lora_weights===== Document 3 =====\n", - "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", - "\n", - "\n", - "\n", - "## StableDiffusionDepth2ImgPipeline\n", - "\n", - "[[autodoc]] StableDiffusionDepth2ImgPipeline\n", - "\t- all\n", - "\t- __call__\n", - "\t- enable_attention_slicing\n", - "\t- disable_attention_slicing\n", - "\t- enable_xformers_memory_efficient_attention\n", - "\t- disable_xformers_memory_efficient_attention\n", - "\t- load_textual_inversion\n", - "\t- load_lora_weights\n", - "\t- save_lora_weights===== Document 4 =====\n", - "## Setup\n", - "\n", - "RoBERTa has a limitatiom of maximum sequence length of 512, so we set the `MAX_LEN=512` for all models to ensure a fair comparison.\n", - "\n", - "```python\n", - "MAX_LEN = 512 \n", - "roberta_checkpoint = \"roberta-large\"\n", - "mistral_checkpoint = \"mistralai/Mistral-7B-v0.1\"\n", - "llama_checkpoint = \"meta-llama/Llama-2-7b-hf\"\n", + "By using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.===== Document 1 =====\n", + "```py\n", + ">>> trainer.push_to_hub()\n", "```\n", - "\n", - "## Data preparation\n", - "### Data loading\n", - "\n", - "We will load the dataset from Hugging Face:\n", - "```python\n", - "from datasets import load_dataset\n", - "dataset = load_dataset(\"mehdiiraqui/twitter_disaster\")\n", + "\n", + "\n", + "Share a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:\n", + "\n", + "- An output directory for your model.\n", + "- A tokenizer.\n", + "- The `hub_model_id`, which is your Hub username and model name.\n", + "\n", + "```py\n", + ">>> from transformers import PushToHubCallback\n", + "\n", + ">>> push_to_hub_callback = PushToHubCallback(\n", + "... output_dir=\"./your_model_save_path\", tokenizer=tokenizer, hub_model_id=\"your-username/my-awesome-model\"\n", + "... )\n", + "```===== Document 2 =====\n", + "Let's pretend we've now fine-tuned the model. The next step would be to push it to the Hub! We can do this with the `timm.models.hub.push_to_hf_hub` function.\n", + "\n", + "```py\n", + ">>> model_cfg = dict(labels=['a', 'b', 'c', 'd'])\n", + ">>> timm.models.hub.push_to_hf_hub(model, 'resnet18-random', model_config=model_cfg)\n", "```\n", - " Now, let's split the dataset into training and validation datasets. Then add the test set:===== Document 5 =====\n", - "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", - "\n", - "\n", "\n", - "## StableDiffusionLatentUpscalePipeline\n", + "Running the above would push the model to `/resnet18-random` on the Hub. You can now share this model with your friends, or use it in your own code!\n", "\n", - "[[autodoc]] StableDiffusionLatentUpscalePipeline\n", - "\t- all\n", - "\t- __call__\n", - "\t- enable_sequential_cpu_offload\n", - "\t- enable_attention_slicing\n", - "\t- disable_attention_slicing\n", - "\t- enable_xformers_memory_efficient_attention\n", - "\t- disable_xformers_memory_efficient_attention\n", - "\n", - "## StableDiffusionPipelineOutput===== Document 6 =====\n", - "If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\n", - "\n", - "\n", + "## Loading a Model===== Document 3 =====\n", + "processor.push_to_hub(hub_model_id)\n", + "trainer.push_to_hub(**kwargs)\n", + "```\n", "\n", - "## StableDiffusionUpscalePipeline\n", + "# 4. Inference\n", "\n", - "[[autodoc]] StableDiffusionUpscalePipeline\n", - "\t- all\n", - "\t- __call__\n", - "\t- enable_attention_slicing\n", - "\t- disable_attention_slicing\n", - "\t- enable_xformers_memory_efficient_attention\n", - "\t- disable_xformers_memory_efficient_attention\n", + "Now comes the exciting part, using our fine-tuned model! In this section, we'll show how you can load your model from the hub and use it for inference.===== Document 4 =====\n", + "--push_to_hub\n", + "```===== Document 5 =====\n", + ". The second way to upload a model, though, is to call model.push_to_hub(). So this is more of a once-off method - it's not called regularly during training. You can just call this manually whenever you want to upload a model to the hub. So we recommend running this after the end of training, just to make sure that you have a commit message just to guarantee that this was the final version of the model at the end of training. And it just makes sure that you're working with the definitive end-of-training model and not accidentally using a model that's from a checkpoint somewhere along the way===== Document 6 =====\n", + "Finally, if you want, you can push your model up to the hub. Here, we'll push it up if you specified `push_to_hub=True` in the training configuration. Note that in order to push to hub, you'll have to have git-lfs installed and be logged into your Hugging Face account (which can be done via `huggingface-cli login`).\n", "\n", - "## StableDiffusionPipelineOutput\n", + "```python\n", + "kwargs = {\n", + " \"finetuned_from\": model.config._name_or_path,\n", + " \"tasks\": \"image-classification\",\n", + " \"dataset\": 'beans',\n", + " \"tags\": ['image-classification'],\n", + "}\n", "\u001b[38;20m===== New step =====\u001b[0m\n", "===== Calling LLM with this last message: =====\n", - "{'role': , 'content': 'Observation: Retrieved documents:\\n===== Document 0 =====\\n!-- DISABLE-FRONTMATTER-SECTIONS -->\\n\\n# End-of-chapter quiz[[end-of-chapter-quiz]]\\n\\n\\n\\nThis chapter covered a lot of ground! Don\\'t worry if you didn\\'t grasp all the details; the next chapters will help you understand how things work under the hood.\\n\\nFirst, though, let\\'s test what you learned in this chapter!\\n\\n\\n### 1. Explore the Hub and look for the `roberta-large-mnli` checkpoint. What task does it perform?===== Document 1 =====\\n## Models\\n\\n### RoBERTa\\n\\n#### Load RoBERTa Checkpoints for the Classification Task\\n\\nWe load the pre-trained RoBERTa model with a sequence classification head using the Hugging Face `AutoModelForSequenceClassification` class:\\n\\n```python\\nfrom transformers import AutoModelForSequenceClassification \\nroberta_model = AutoModelForSequenceClassification.from_pretrained(roberta_checkpoint, num_labels=2)\\n```\\n\\n\\n#### LoRA setup for RoBERTa classifier===== Document 2 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionInpaintPipeline\\n\\n[[autodoc]] StableDiffusionInpaintPipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\t- load_textual_inversion\\n\\t- load_lora_weights\\n\\t- save_lora_weights===== Document 3 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionDepth2ImgPipeline\\n\\n[[autodoc]] StableDiffusionDepth2ImgPipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\t- load_textual_inversion\\n\\t- load_lora_weights\\n\\t- save_lora_weights===== Document 4 =====\\n## Setup\\n\\nRoBERTa has a limitatiom of maximum sequence length of 512, so we set the `MAX_LEN=512` for all models to ensure a fair comparison.\\n\\n```python\\nMAX_LEN = 512 \\nroberta_checkpoint = \"roberta-large\"\\nmistral_checkpoint = \"mistralai/Mistral-7B-v0.1\"\\nllama_checkpoint = \"meta-llama/Llama-2-7b-hf\"\\n```\\n\\n## Data preparation\\n### Data loading\\n\\nWe will load the dataset from Hugging Face:\\n```python\\nfrom datasets import load_dataset\\ndataset = load_dataset(\"mehdiiraqui/twitter_disaster\")\\n```\\n Now, let\\'s split the dataset into training and validation datasets. Then add the test set:===== Document 5 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionLatentUpscalePipeline\\n\\n[[autodoc]] StableDiffusionLatentUpscalePipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_sequential_cpu_offload\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\n## StableDiffusionPipelineOutput===== Document 6 =====\\nIf you\\'re interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!\\n\\n\\n\\n## StableDiffusionUpscalePipeline\\n\\n[[autodoc]] StableDiffusionUpscalePipeline\\n\\t- all\\n\\t- __call__\\n\\t- enable_attention_slicing\\n\\t- disable_attention_slicing\\n\\t- enable_xformers_memory_efficient_attention\\n\\t- disable_xformers_memory_efficient_attention\\n\\n## StableDiffusionPipelineOutput'}\n", + "{'role': , 'content': '[OUTPUT OF STEP 0] Observation:\\nRetrieved documents:\\n===== Document 0 =====\\n# Step 7. Push everything to the Hub\\n api.upload_folder(\\n repo_id=repo_id,\\n folder_path=repo_local_path,\\n path_in_repo=\".\",\\n )\\n\\n print(\"Your model is pushed to the Hub. You can view your model here: \", repo_url)\\n```\\n\\n### .\\n\\nBy using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.===== Document 1 =====\\n```py\\n>>> trainer.push_to_hub()\\n```\\n\\n\\nShare a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:\\n\\n- An output directory for your model.\\n- A tokenizer.\\n- The `hub_model_id`, which is your Hub username and model name.\\n\\n```py\\n>>> from transformers import PushToHubCallback\\n\\n>>> push_to_hub_callback = PushToHubCallback(\\n... output_dir=\"./your_model_save_path\", tokenizer=tokenizer, hub_model_id=\"your-username/my-awesome-model\"\\n... )\\n```===== Document 2 =====\\nLet\\'s pretend we\\'ve now fine-tuned the model. The next step would be to push it to the Hub! We can do this with the `timm.models.hub.push_to_hf_hub` function.\\n\\n```py\\n>>> model_cfg = dict(labels=[\\'a\\', \\'b\\', \\'c\\', \\'d\\'])\\n>>> timm.models.hub.push_to_hf_hub(model, \\'resnet18-random\\', model_config=model_cfg)\\n```\\n\\nRunning the above would push the model to `/resnet18-random` on the Hub. You can now share this model with your friends, or use it in your own code!\\n\\n## Loading a Model===== Document 3 =====\\nprocessor.push_to_hub(hub_model_id)\\ntrainer.push_to_hub(**kwargs)\\n```\\n\\n# 4. Inference\\n\\nNow comes the exciting part, using our fine-tuned model! In this section, we\\'ll show how you can load your model from the hub and use it for inference.===== Document 4 =====\\n--push_to_hub\\n```===== Document 5 =====\\n. The second way to upload a model, though, is to call model.push_to_hub(). So this is more of a once-off method - it\\'s not called regularly during training. You can just call this manually whenever you want to upload a model to the hub. So we recommend running this after the end of training, just to make sure that you have a commit message just to guarantee that this was the final version of the model at the end of training. And it just makes sure that you\\'re working with the definitive end-of-training model and not accidentally using a model that\\'s from a checkpoint somewhere along the way===== Document 6 =====\\nFinally, if you want, you can push your model up to the hub. Here, we\\'ll push it up if you specified `push_to_hub=True` in the training configuration. Note that in order to push to hub, you\\'ll have to have git-lfs installed and be logged into your Hugging Face account (which can be done via `huggingface-cli login`).\\n\\n```python\\nkwargs = {\\n \"finetuned_from\": model.config._name_or_path,\\n \"tasks\": \"image-classification\",\\n \"dataset\": \\'beans\\',\\n \"tags\": [\\'image-classification\\'],\\n}'}\n", "\u001b[38;20m===== Output message of the LLM: =====\u001b[0m\n", - "\u001b[38;20mThought: In the retrieved documents, I found the answer: \"roberta-large-mnli performs a classification task\". Now I can provide the final answer.\n", - "\n", + "\u001b[38;20mThought: I will now provide the answer.\n", "Action: ```json\n", "{\n", " \"action\": \"final_answer\",\n", " \"action_input\": {\n", - " \"answer\": \"The roberta-large-mnli checkpoint performs a classification task.\"\n", + " \"answer\": \"To push a model to the Hub, you can make use of the push_to_hub API provided by Hugging Face. An example usage would be: model.push_to_hub() or trainer.push_to_hub(). Additionally, a function called timm.models.hub.push_to_hf_hub is also available to push models to the Hub.\"\n", " }\n", - "}\u001b[0m\n", + "}\n", + "\n", + "```\u001b[0m\n", "\u001b[38;20m===== Extracting action =====\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The roberta-large-mnli checkpoint performs a classification task.'}\u001b[0m\n" + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To push a model to the Hub, you can make use of the push_to_hub API provided by Hugging Face. An example usage would be: model.push_to_hub() or trainer.push_to_hub(). Additionally, a function called timm.models.hub.push_to_hf_hub is also available to push models to the Hub.'}\u001b[0m\n" ] }, { @@ -527,14 +482,12 @@ "output_type": "stream", "text": [ "Final output:\n", - "The roberta-large-mnli checkpoint performs a classification task.\n" + "To push a model to the Hub, you can make use of the push_to_hub API provided by Hugging Face. An example usage would be: model.push_to_hub() or trainer.push_to_hub(). Additionally, a function called timm.models.hub.push_to_hf_hub is also available to push models to the Hub.\n" ] } ], "source": [ - "agent_output = agent.run(\n", - " \"What is the task performed by the `roberta-large-mnli` checkpoint?\"\n", - ")\n", + "agent_output = agent.run(\"How can I push a model to the Hub?\")\n", "\n", "print(\"Final output:\")\n", "print(agent_output)" @@ -544,64 +497,22 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Evaluate with LLM judge\n", + "## Agentic RAG vs. standard RAG\n", "\n", - "Does the agent setup improve the performance of our RAG system? Well, let's measure it.\n", + "Does the agent setup give a performant RAG system? Well, let's measure it.\n", "\n", - "We'll compare the score of our agent system with the different RAG systems tested in [this other cookbook](advanced_rag).\n", - "\n", - "So we take care to use the same evaluation setup, with GPT-4-turbo as a judge for reproducibility. But nowadays I'd just be using [Llama-3](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)." + "We will use [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) for evaluation since it's one of the strongest OS models we tested for LLM judge use cases." ] }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "eval_dataset = datasets.load_dataset(\"m-ric/huggingface_doc_qa_eval\", split=\"train\")" ] }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [], - "source": [ - "EVALUATION_PROMPT = \"\"\"###Task Description:\n", - "An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n", - "1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n", - "2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n", - "3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n", - "4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n", - "\n", - "###The instruction to evaluate:\n", - "{instruction}\n", - "\n", - "###Response to evaluate:\n", - "{response}\n", - "\n", - "###Reference Answer (Score 5):\n", - "{reference_answer}\n", - "\n", - "###Score Rubrics:\n", - "[Is the response correct, accurate, and factual based on the reference answer?]\n", - "Score 1: The response is completely incorrect, inaccurate, and/or not factual.\n", - "Score 2: The response is mostly incorrect, inaccurate, and/or not factual.\n", - "Score 3: The response is somewhat correct, accurate, and/or factual.\n", - "Score 4: The response is mostly correct, accurate, and factual.\n", - "Score 5: The response is completely correct, accurate, and factual.\n", - "\n", - "###Feedback:\"\"\"\n", - "\n", - "from openai import OpenAI\n", - "import os\n", - "\n", - "client = OpenAI(\n", - " api_key=os.getenv(\"OPENAI_API_KEY\"),\n", - ")" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -611,7 +522,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "metadata": {}, "outputs": [], "source": [ @@ -622,7 +533,7 @@ }, { "cell_type": "code", - "execution_count": 79, + "execution_count": 56, "metadata": {}, "outputs": [ { @@ -633,20 +544,23 @@ "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What architecture is the `tokenizers-linux-x64-musl` binary designed for?\n", "\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What architecture is the `tokenizers-linux-x64-musl` binary designed for?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `tokenizers-linux-x64-musl` binary is designed for the **x86_64-unknown-linux-musl** architecture (Document 0).'}\u001b[0m\n", - " 2%|▏ | 1/65 [00:15<16:01, 15.02s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'tokenizers-linux-x64-musl binary architecture'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'tokenizers-linux-x64-musl architecture'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The tokenizers-linux-x64-musl binary is designed for the **x86_64-unknown-linux-musl** architecture.'}\u001b[0m\n", + " 2%|▏ | 1/65 [00:20<21:28, 20.13s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the purpose of the BLIP-Diffusion model?\n", @@ -660,7 +574,7 @@ "=======================================================\n", "Question: What architecture is the `tokenizers-linux-x64-musl` binary designed for?\n", "\n", - "Answer: The `tokenizers-linux-x64-musl` binary is designed for the **x86_64-unknown-linux-musl** architecture (Document 0).\n", + "Answer: The tokenizers-linux-x64-musl binary is designed for the **x86_64-unknown-linux-musl** architecture.\n", "True answer: x86_64-unknown-linux-musl\n" ] }, @@ -668,14 +582,17 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'BLIP-Diffusion model'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The BLIP-Diffusion model is a pre-trained subject representation for controllable text-to-image generation and editing. It enables zero-shot subject-driven generation and control-guided zero-shot generation.'}\u001b[0m\n", - " 3%|▎ | 2/65 [00:25<12:44, 12.13s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'BLIP-Diffusion purpose'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'BLIP-Diffusion role'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'BLIP-Diffusion is a text-to-image generation model that enables zero-shot subject-driven generation and control-guided zero-shot generation. It addresses limitations in existing models, particularly in terms of lengthy fine-tuning and difficulties in preserving subject fidelity. BLIP-Diffusion introduces a new multimodal encoder pre-trained to provide subject representation, allowing for efficient fine-tuning and novel subject-driven generation and editing applications.'}\u001b[0m\n", + " 3%|▎ | 2/65 [00:43<23:00, 21.92s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "How can a user claim authorship of a paper on the Hugging Face Hub?\n", @@ -689,7 +606,7 @@ "=======================================================\n", "Question: What is the purpose of the BLIP-Diffusion model?\n", "\n", - "Answer: The BLIP-Diffusion model is a pre-trained subject representation for controllable text-to-image generation and editing. It enables zero-shot subject-driven generation and control-guided zero-shot generation.\n", + "Answer: BLIP-Diffusion is a text-to-image generation model that enables zero-shot subject-driven generation and control-guided zero-shot generation. It addresses limitations in existing models, particularly in terms of lengthy fine-tuning and difficulties in preserving subject fidelity. BLIP-Diffusion introduces a new multimodal encoder pre-trained to provide subject representation, allowing for efficient fine-tuning and novel subject-driven generation and editing applications.\n", "True answer: The BLIP-Diffusion model is designed for controllable text-to-image generation and editing.\n" ] }, @@ -697,44 +614,16 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[31;20mCould not parse the given action: JSON is invalid: you probably tried to provide multiple tool calls in one action. PROVIDE ONLY ONE TOOL CALL..\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 83, in parse_json_blob\n", - " json_data = json.loads(json_blob, strict=False)\n", - " File \"/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py\", line 359, in loads\n", - " return cls(**kw).decode(s)\n", - " File \"/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py\", line 340, in decode\n", - " raise JSONDecodeError(\"Extra data\", s, end)\n", - "json.decoder.JSONDecodeError: Extra data: line 6 column 6 (char 165)\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 784, in step\n", - " tool_name, arguments = self.tool_parser(action)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 113, in parse_json_tool_call\n", - " tool_call = parse_json_blob(json_blob)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 88, in parse_json_blob\n", - " raise ValueError(\n", - "ValueError: JSON is invalid: you probably tried to provide multiple tool calls in one action. PROVIDE ONLY ONE TOOL CALL.\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 710, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 786, in step\n", - " raise AgentParsingError(f\"Could not parse the given action: {e}.\")\n", - "transformers.agents.agents.AgentParsingError: Could not parse the given action: JSON is invalid: you probably tried to provide multiple tool calls in one action. PROVIDE ONLY ONE TOOL CALL..\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How can a user claim authorship of a paper on the Hugging Face Hub?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the Hugging Face Hub?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The Hub attempts to automatically match papers to users based on their email. If a user's paper is not linked to their account, they can click on their name in the corresponding Paper page, select 'claim authorship', and confirm the request. Once the request is validated by the admin team, the Paper page will show as verified (Documents 0, 1, and 2).\"}\u001b[0m\n", - " 5%|▍ | 3/65 [00:57<21:53, 21.19s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Claim authorship of a paper on the Hugging Face Hub'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How does the Hugging Face Hub match papers to users?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The Hugging Face Hub will attempt to automatically match papers to users based on their email address. Users also have the option to manually control which papers show on their profile by visiting their settings. There, they can choose to hide or show specific papers.'}\u001b[0m\n", + " 5%|▍ | 3/65 [01:07<23:47, 23.02s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the purpose of the /healthcheck endpoint in the Datasets server API?\n", @@ -748,7 +637,7 @@ "=======================================================\n", "Question: How can a user claim authorship of a paper on the Hugging Face Hub?\n", "\n", - "Answer: The Hub attempts to automatically match papers to users based on their email. If a user's paper is not linked to their account, they can click on their name in the corresponding Paper page, select 'claim authorship', and confirm the request. Once the request is validated by the admin team, the Paper page will show as verified (Documents 0, 1, and 2).\n", + "Answer: The Hugging Face Hub will attempt to automatically match papers to users based on their email address. Users also have the option to manually control which papers show on their profile by visiting their settings. There, they can choose to hide or show specific papers.\n", "True answer: By clicking their name on the corresponding Paper page and clicking \"claim authorship\", then confirming the request in paper settings for admin team validation.\n" ] }, @@ -756,14 +645,16 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': '/healthcheck endpoint in the Datasets server API'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The /healthcheck endpoint in the Datasets server API is used to ensure the app is running.'}\u001b[0m\n", - " 6%|▌ | 4/65 [01:05<16:17, 16.03s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of /healthcheck endpoint in Datasets server API'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': '/healthcheck endpoint in Datasets server API functionality'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The purpose of the /healthcheck endpoint in the Datasets server API is to ensure that the application is running.'}\u001b[0m\n", + " 6%|▌ | 4/65 [01:23<20:41, 20.35s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the default context window size for Local Attention in the LongT5 model?\n", @@ -777,7 +668,7 @@ "=======================================================\n", "Question: What is the purpose of the /healthcheck endpoint in the Datasets server API?\n", "\n", - "Answer: The /healthcheck endpoint in the Datasets server API is used to ensure the app is running.\n", + "Answer: The purpose of the /healthcheck endpoint in the Datasets server API is to ensure that the application is running.\n", "True answer: Ensure the app is running\n" ] }, @@ -785,14 +676,19 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Default context window size for Local Attention in LongT5 model'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The default context window size for Local Attention in the LongT5 model is inferred to be 255 (127 tokens to the left and 127 tokens to the right, plus the self-token). This information is inferred from a description of Local Attention in one of the retrieved documents.'}\u001b[0m\n", - " 8%|▊ | 5/65 [01:16<14:17, 14.29s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'LongT5 model default context window size local attention'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'LongT5 model default attention window size'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'longt5 default configuration'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'longt5 local attention context window size'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 8%|▊ | 5/65 [01:46<21:08, 21.15s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What method is used to load a checkpoint for a task using `AutoPipeline`?\n", @@ -806,7 +702,7 @@ "=======================================================\n", "Question: What is the default context window size for Local Attention in the LongT5 model?\n", "\n", - "Answer: The default context window size for Local Attention in the LongT5 model is inferred to be 255 (127 tokens to the left and 127 tokens to the right, plus the self-token). This information is inferred from a description of Local Attention in one of the retrieved documents.\n", + "Answer: Sorry, despite my best efforts, I was unable to find the default context window size for Local Attention in the LongT5 model. The retriever was unable to provide clear and concise information on this matter.\n", "True answer: 127 tokens\n" ] }, @@ -814,14 +710,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'loading checkpoints with AutoPipeline'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The method used to load a checkpoint for a task using `AutoPipeline` is the `from_pretrained()` method. For seamless task switching without reallocating additional memory, the `from_pipe()` method is utilized (Source Document 0).'}\u001b[0m\n", - " 9%|▉ | 6/65 [01:25<12:18, 12.52s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'load a checkpoint with AutoPipeline'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The method used to load a checkpoint for a task using `AutoPipeline` is `from_pretrained()`.'}\u001b[0m\n", + " 9%|▉ | 6/65 [01:53<16:05, 16.37s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the purpose of Diffusers library?\n", @@ -835,7 +732,7 @@ "=======================================================\n", "Question: What method is used to load a checkpoint for a task using `AutoPipeline`?\n", "\n", - "Answer: The method used to load a checkpoint for a task using `AutoPipeline` is the `from_pretrained()` method. For seamless task switching without reallocating additional memory, the `from_pipe()` method is utilized (Source Document 0).\n", + "Answer: The method used to load a checkpoint for a task using `AutoPipeline` is `from_pretrained()`.\n", "True answer: from_pretrained()\n" ] }, @@ -843,7 +740,43 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'What is the purpose of the Diffusers library?'}}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of Diffusers library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of Models component in Diffusers library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of Scripts component in Diffusers library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of Tokenizers component in Diffusers library'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 11%|█ | 7/65 [02:22<19:39, 20.34s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What method does the EulerAncestralDiscreteScheduler use for sampling?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of Diffusers library?\n", + "\n", + "Answer: The Diffusers library is a comprehensive and accessible toolbox for working with diffusion models, focusing on state-of-the-art pretrained models for generating images, audio, and even 3D structures of molecules. It is designed with simplicity, accessibility, reproducibility, and responsibility in mind. The library offers three core components: Models, Scripts, and Tokenizers. \n", + "\n", + "The Models component provides pretrained diffusion models and custom diffusion system building blocks. The Scripts component includes example scripts and training examples to demonstrate how to effectively use the library for training and fine-tuning. Lastly, the Tokenizers component offers efficient and versatile implementations of popular tokenizers, a critical step in preparing inputs for diffusion models.\n", + "True answer: To serve as a modular toolbox for both inference and training of state-of-the-art pretrained diffusion models across multiple modalities.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'EulerAncestralDiscreteScheduler sampling method'}}\u001b[0m\n", "\u001b[31;20mError in tool call execution: Your search query must be a string\n", "You should only use this tool with a correct input.\n", "As a reminder, this tool's description is the following:\n", @@ -851,22 +784,22 @@ "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 478, in execute_tool_call\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", " observation = self.toolbox.tools[tool_name](**arguments)\n", " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_10832/4081367717.py\", line 22, in forward\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", " assert isinstance(query, str), \"Your search query must be a string\"\n", "AssertionError: Your search query must be a string\n", "\n", "During handling of the above exception, another exception occurred:\n", "\n", "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 710, in direct_run\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 803, in step\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 481, in execute_tool_call\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", " raise AgentExecutionError(\n", "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", "You should only use this tool with a correct input.\n", @@ -874,43 +807,108 @@ "\n", "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the purpose of the Diffusers library?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The Diffusers library is a generative AI tool principally designed to generate images, audio files and 3D molecular structures using diffusion models. [Source: documents 1,4,5]'}\u001b[0m\n", - " 11%|█ | 7/65 [01:49<15:53, 16.43s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'What is the EulerAncestralDiscreteScheduler sampling method?'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", "\n", - "Question:\n", - "What method does the EulerAncestralDiscreteScheduler use for sampling?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of Diffusers library?\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", "\n", - "Answer: The Diffusers library is a generative AI tool principally designed to generate images, audio files and 3D molecular structures using diffusion models. [Source: documents 1,4,5]\n", - "True answer: To serve as a modular toolbox for both inference and training of state-of-the-art pretrained diffusion models across multiple modalities.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What method does the EulerAncestralDiscreteScheduler use for sampling?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Ancestral sampling is used by the EulerAncestralDiscreteScheduler for sampling.'}\u001b[0m\n", - " 12%|█▏ | 8/65 [02:00<13:49, 14.55s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'EulerAncestralDiscreteScheduler sampling'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'How does EulerAncestralDiscreteScheduler work?'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 12%|█▏ | 8/65 [02:46<20:28, 21.55s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the name of the large multimodal model that can solve image-text tasks and is based on Flamingo?\n", @@ -924,7 +922,7 @@ "=======================================================\n", "Question: What method does the EulerAncestralDiscreteScheduler use for sampling?\n", "\n", - "Answer: Ancestral sampling is used by the EulerAncestralDiscreteScheduler for sampling.\n", + "Answer: Apologies, I encountered an error while trying to answer your question. May I try asking in a different way, such as \"What is the EulerAncestralDiscreteScheduler?\"\n", "True answer: Ancestral sampling with Euler method steps.\n" ] }, @@ -932,15 +930,16 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Flamingo image-text'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Flamingo model details'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The name of the large multimodal model based on Flamingo is OpenFlamingo. It is an open reproduction of Flamingo at a scale of 9 billion parameters. This information was found in document 1.'}\u001b[0m\n", - " 14%|█▍ | 9/65 [02:23<16:05, 17.25s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Large multimodal model based on Flamingo for image-text tasks'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Large multimodal model based on Flamingo'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The name of the large multimodal model based on Flamingo is IDEFICS.'}\u001b[0m\n", + " 14%|█▍ | 9/65 [03:02<18:40, 20.00s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the purpose of the `gradio.Blocks` API?\n", @@ -954,7 +953,7 @@ "=======================================================\n", "Question: What is the name of the large multimodal model that can solve image-text tasks and is based on Flamingo?\n", "\n", - "Answer: The name of the large multimodal model based on Flamingo is OpenFlamingo. It is an open reproduction of Flamingo at a scale of 9 billion parameters. This information was found in document 1.\n", + "Answer: The name of the large multimodal model based on Flamingo is IDEFICS.\n", "True answer: IDEFICS\n" ] }, @@ -962,14 +961,17 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of gradio.Blocks API'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `gradio.Blocks` API provides a low-level approach for designing web apps with flexible layouts and data flows, allowing control over component placement, handling complex data flows, and updating properties based on user interaction, all within a Python environment.'}\u001b[0m\n", - " 15%|█▌ | 10/65 [02:38<15:06, 16.47s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'gradio.Blocks purpose'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'gradio.blocks api'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'gradio component meaning'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `gradio.Blocks` API is a low-level approach for designing Gradio web apps that offers more flexibility than traditional Gradio demos. It lets developers create custom layouts, manage complex data flows, and control the visibility and behavior of UI components using Python code. For example, with `gradio.Blocks`, you can determine where specific UI components (such as input fields or output panels) appear on the page and how they interact with each other, enabling you to build sophisticated interactive web applications.'}\u001b[0m\n", + " 15%|█▌ | 10/65 [03:29<20:06, 21.94s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What is the purpose of the two-stage model proposed in the paper \"Hierarchical Text-Conditional Image Generation with CLIP Latents\"?\n", @@ -983,7 +985,7 @@ "=======================================================\n", "Question: What is the purpose of the `gradio.Blocks` API?\n", "\n", - "Answer: The `gradio.Blocks` API provides a low-level approach for designing web apps with flexible layouts and data flows, allowing control over component placement, handling complex data flows, and updating properties based on user interaction, all within a Python environment.\n", + "Answer: The `gradio.Blocks` API is a low-level approach for designing Gradio web apps that offers more flexibility than traditional Gradio demos. It lets developers create custom layouts, manage complex data flows, and control the visibility and behavior of UI components using Python code. For example, with `gradio.Blocks`, you can determine where specific UI components (such as input fields or output panels) appear on the page and how they interact with each other, enabling you to build sophisticated interactive web applications.\n", "True answer: The `gradio.Blocks` API allows you to have full control over the data flows and layout of your application, enabling the building of complex, multi-step applications.\n" ] }, @@ -991,14 +993,16 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hierarchical Text-Conditional Image Generation with CLIP Latents paper purpose of two-stage model'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The two-stage model proposed in the paper 'Hierarchical Text-Conditional Image Generation with CLIP Latents' by Ramesh et al. aims to generate images from text captions. It first turns text captions into CLIP image embeddings, which are then decoded into images using a diffusion model.\"}\u001b[0m\n", - " 17%|█▋ | 11/65 [02:55<14:53, 16.54s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hierarchical Text-Conditional Image Generation with CLIP Latents'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'two-stage model Hierarchical Text-Conditional Image Generation with CLIP Latents'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The purpose of the two-stage model is to generate images from text captions. The first stage of the model, the prior, turns a text caption into a CLIP image embedding. In the second stage, a diffusion model decodes this embedding into an image. This process enables the generation of images that preserve semantics and style while allowing variation in non-essential details. This two-stage approach improves image diversity while maintaining photorealism and caption similarity.'}\u001b[0m\n", + " 17%|█▋ | 11/65 [03:57<21:34, 23.98s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What command is used to install the requirements for a research project using 🤗 Transformers?\n", @@ -1012,7 +1016,7 @@ "=======================================================\n", "Question: What is the purpose of the two-stage model proposed in the paper \"Hierarchical Text-Conditional Image Generation with CLIP Latents\"?\n", "\n", - "Answer: The two-stage model proposed in the paper 'Hierarchical Text-Conditional Image Generation with CLIP Latents' by Ramesh et al. aims to generate images from text captions. It first turns text captions into CLIP image embeddings, which are then decoded into images using a diffusion model.\n", + "Answer: The purpose of the two-stage model is to generate images from text captions. The first stage of the model, the prior, turns a text caption into a CLIP image embedding. In the second stage, a diffusion model decodes this embedding into an image. This process enables the generation of images that preserve semantics and style while allowing variation in non-essential details. This two-stage approach improves image diversity while maintaining photorealism and caption similarity.\n", "True answer: The purpose of the two-stage model is to generate a CLIP image embedding given a text caption and then generate an image conditioned on the image embedding.\n" ] }, @@ -1020,74 +1024,239 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What command is used to install the requirements for a research project using 🤗 Transformers?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The command used to install the requirements for a research project using 🤗 Transformers is 'pip install -r requirements.txt' or 'pip install .' depending on the context.\"}\u001b[0m\n", - " 18%|█▊ | 12/65 [03:04<12:35, 14.25s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'installing requirements research project'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", "\n", - "Question:\n", - "What task does the `roberta-large-mnli` checkpoint perform?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What command is used to install the requirements for a research project using 🤗 Transformers?\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", "\n", - "Answer: The command used to install the requirements for a research project using 🤗 Transformers is 'pip install -r requirements.txt' or 'pip install .' depending on the context.\n", - "True answer: pip install -r requirements.txt\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What task does the `roberta-large-mnli` checkpoint perform?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'roberta-large-mnli task'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'roberta-large-mnli task natural language inference'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `roberta-large-mnli` checkpoint is used for Natural Language Inference (NLI).'}\u001b[0m\n", - " 20%|██ | 13/65 [03:32<16:10, 18.67s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "During handling of the above exception, another exception occurred:\n", "\n", - "Question:\n", - "What service is replacing the Paid tier of the Inference API at Hugging Face?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What task does the `roberta-large-mnli` checkpoint perform?\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", "\n", - "Answer: The `roberta-large-mnli` checkpoint is used for Natural Language Inference (NLI).\n", - "True answer: Text classification\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is replacing the Paid tier of the Inference API?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Inference Endpoints are replacing the Paid tier of the Inference API at Hugging Face, as stated in document 0.'}\u001b[0m\n", - " 22%|██▏ | 14/65 [03:44<14:08, 16.65s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'installing requirements research project 🤗 transformers'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'installing requirements research project hugging face transformers'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'installing requirements hugging face transformers'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 18%|█▊ | 12/65 [04:24<22:03, 24.97s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What task does the `roberta-large-mnli` checkpoint perform?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What command is used to install the requirements for a research project using 🤗 Transformers?\n", + "\n", + "Answer: Apologies, but despite my best efforts, I was unable to retrieve the relevant information from the knowledge base using the 'retriever' tool. I can suggest alternative phrasing for your query, but I cannot guarantee that it will yield the desired results without further context or information. \n", + "\n", + "An alternative query could be: \"command to install requirements research project 🤔 Transformers\". \n", + "\n", + "If there are specific errors or issues you are encountering with the 'retriever' tool, I can try to assist with those as well.\n", + "True answer: pip install -r requirements.txt\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'roberta-large-mnli checkpoint'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'roberta large mnli task'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'mnli meaning'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'multi genre natural language inference task'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 20%|██ | 13/65 [05:05<25:40, 29.63s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What service is replacing the Paid tier of the Inference API at Hugging Face?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What task does the `roberta-large-mnli` checkpoint perform?\n", + "\n", + "Answer: The 'roberta-large-mnli' checkpoint is related to the task of Multi-genre Natural Language Inference (MNLI), a classification task where the model predicts textual entailment between two sentences.\n", + "True answer: Text classification\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Changes to Inference API Paid tier Hugging Face'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Changes to Inference API Paid tier Hugging Face'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Inference Endpoints replacing Paid tier Inference API Hugging Face'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Inference Endpoints is the service that is replacing the Paid tier of the Inference API at Hugging Face.'}\u001b[0m\n", + " 22%|██▏ | 14/65 [05:32<24:34, 28.91s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What architectural feature does SqueezeBERT use instead of fully-connected layers for the Q, K, V, and FFN layers?\n", @@ -1101,7 +1270,7 @@ "=======================================================\n", "Question: What service is replacing the Paid tier of the Inference API at Hugging Face?\n", "\n", - "Answer: Inference Endpoints are replacing the Paid tier of the Inference API at Hugging Face, as stated in document 0.\n", + "Answer: Inference Endpoints is the service that is replacing the Paid tier of the Inference API at Hugging Face.\n", "True answer: Inference Endpoints\n" ] }, @@ -1109,14 +1278,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What architectural feature does SqueezeBERT use instead of fully-connected layers for the Q, K, V, and FFN layers?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'SqueezeBERT architecture design choices'}\u001b[0m\n", "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'SqueezeBERT uses grouped convolutions instead of fully-connected layers for the Q, K, V, and FFN layers.'}\u001b[0m\n", - " 23%|██▎ | 15/65 [03:53<11:55, 14.31s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + " 23%|██▎ | 15/65 [05:39<18:34, 22.29s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What type of license is the HuggingFace Team's software distributed under?\n", @@ -1138,20 +1308,3231 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'HuggingFace software license'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The HuggingFace Team's software is distributed under the Apache License, Version 2.0.\"}\u001b[0m\n", - " 25%|██▍ | 16/65 [04:06<11:10, 13.68s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face software license type'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Apache License, Version 2.0'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The HuggingFace software is distributed under the Apache License, Version 2.0.'}\u001b[0m\n", + " 25%|██▍ | 16/65 [05:52<15:56, 19.52s/it]\u001b[33;1m======== New task ========\u001b[0m\n", "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", "give a comprehensive answer to the question below.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", "\n", "Question:\n", "What are the two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed?\n", "\u001b[0m\n" ] }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What type of license is the HuggingFace Team's software distributed under?\n", + "\n", + "Answer: The HuggingFace software is distributed under the Apache License, Version 2.0.\n", + "True answer: Apache License, Version 2.0\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'ALBERT model parameter-reduction techniques for lower memory consumption and faster training'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The ALBERT model employed two parameter-reduction techniques to achieve lower memory consumption and faster training: \\n1. Splitting the embedding matrix into two smaller matrices\\n2. Utilizing repeating layers that are split among groups.'}\u001b[0m\n", + " 26%|██▌ | 17/65 [06:04<13:50, 17.31s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What are the three main steps for fine-tuning a model with the 🤗 Datasets library?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What are the two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed?\n", + "\n", + "Answer: The ALBERT model employed two parameter-reduction techniques to achieve lower memory consumption and faster training: \n", + "1. Splitting the embedding matrix into two smaller matrices\n", + "2. Utilizing repeating layers that are split among groups.\n", + "True answer: Splitting the embedding matrix into two smaller matrices and using repeating layers split among groups.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'steps to fine-tune a model Datasets library'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'steps to fine-tune a model Datasets library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'second and third steps fine-tune model datasets library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'information about fit method datasets library'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 28%|██▊ | 18/65 [06:44<18:44, 23.92s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What are the three main steps for fine-tuning a model with the 🤗 Datasets library?\n", + "\n", + "Answer: The three main steps for fine-tuning a model with the 🤗 Datasets library are:\n", + "\n", + "1. Load a dataset from the Hugging Face Hub.\n", + "2. Preprocess the data with the `Dataset.map()` method, and then pass it to the `fit()` method to start fine-tuning.\n", + "3. After fine-tuning, load and compute metrics.\n", + "True answer: 1. Load a dataset from the Hugging Face Hub. 2. Preprocess the data with `Dataset.map()`. 3. Load and compute metrics.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face Infinity throughput compared to vanilla transformers'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Hugging Face Infinity can deliver a maximum improvement in throughput of up to 800% compared to vanilla transformers.'}\u001b[0m\n", + " 29%|██▉ | 19/65 [06:51<14:33, 18.99s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the command to upload a spaCy pipeline to the Hugging Face Hub?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers?\n", + "\n", + "Answer: Hugging Face Infinity can deliver a maximum improvement in throughput of up to 800% compared to vanilla transformers.\n", + "True answer: +800%\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Upload spaCy pipeline to Hugging Face Hub'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Upload spaCy pipeline to Hugging Face Hub'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The command to upload a spaCy pipeline to the Hugging Face Hub is `python -m spacy huggingface-hub`.'}\u001b[0m\n", + " 31%|███ | 20/65 [07:10<14:17, 19.04s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the time and memory complexity of the Nyströmformer's approximation of self-attention?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the command to upload a spaCy pipeline to the Hugging Face Hub?\n", + "\n", + "Answer: The command to upload a spaCy pipeline to the Hugging Face Hub is `python -m spacy huggingface-hub`.\n", + "True answer: python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Nyströmformer self-attention time complexity'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Nyströmformer self-attention time complexity'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Nyströmformer self-attention memory complexity'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The Nyströmformer's approximation of self-attention has a time complexity of O(n) and a memory complexity of O(n) as well.\"}\u001b[0m\n", + " 32%|███▏ | 21/65 [07:37<15:33, 21.23s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the goal of the Named Entity Recognition task in token classification?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the time and memory complexity of the Nyströmformer's approximation of self-attention?\n", + "\n", + "Answer: The Nyströmformer's approximation of self-attention has a time complexity of O(n) and a memory complexity of O(n) as well.\n", + "True answer: O(n)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'What is the goal of the Named Entity Recognition task in token classification?'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Named Entity Recognition task goal'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Goal of Named Entity Recognition'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Named Entity Recognition information'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 34%|███▍ | 22/65 [08:00<15:40, 21.87s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the resolution of images used by the CLIPSeg model?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the goal of the Named Entity Recognition task in token classification?\n", + "\n", + "Answer: Apologies, I seem to be facing some technical difficulties with the tool. As a work-around, I can tell you that the goal of the Named Entity Recognition (NER) task in token classification is to identify and classify named entities in text, such as people, organisations, locations, dates, and more. NER is an important task in natural language processing as it enables machines to understand and extract relevant information from text, which can then be used for various downstream tasks such as information retrieval, question answering, and language understanding.\n", + "True answer: The goal of the Named Entity Recognition task is to find the entities in a piece of text, such as person, location, or organization.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'CLIPSeg model image resolution'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The CLIPSeg model uses images with a resolution of 352 x 352 pixels.'}\u001b[0m\n", + " 35%|███▌ | 23/65 [08:18<14:32, 20.77s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What can you use Gradio for?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the resolution of images used by the CLIPSeg model?\n", + "\n", + "Answer: The CLIPSeg model uses images with a resolution of 352 x 352 pixels.\n", + "True answer: 352 x 352 pixels\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is Gradio'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'import gradio as gr'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio building an application'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio use cases apart from an ML model'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 37%|███▋ | 24/65 [08:50<16:33, 24.23s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What TensorFlow API function is used to load a saved tensor file?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What can you use Gradio for?\n", + "\n", + "Answer: You can use Gradio to quickly create customizable web applications and demos for any type of machine learning model and not just NLP models. Using Gradio, you can also debug your model. However, Gradio is designed only for model inference and not for training a machine learning model.\n", + "True answer: Create a demo for your machine learning model, share your machine learning model with others, and debug your model.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'TensorFlow api function load saved tensor file'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'tensorflow api function load saved tensor file'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'TensorFlow load a saved tensor file'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Tensorflow load saved tensor file'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 38%|███▊ | 25/65 [09:12<15:35, 23.39s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "Where can you access the logs of your Endpoints in Hugging Face Endpoints?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What TensorFlow API function is used to load a saved tensor file?\n", + "\n", + "Answer: Thought: This error seems to be persistent and unrelated to the query format. I will proceed without the retriever tool and answer based on my pre-existing knowledge. \n", + "\n", + "The TensorFlow API function used to load a saved tensor file is `tf.load_file`. This function allows you to load a tensor from a file, and it takes the file name as an argument. \n", + "\n", + "You can use this function to load tensor files that have been previously saved using TensorFlow's saving mechanisms, such as `tf.save_tensor_slices` or checkpoint saving. The loaded tensor can then be used in your TensorFlow program for inference, training, or further processing.\n", + "True answer: safetensors.tensorflow.load_file\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Access logs Hugging Face Endpoints'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"You can access the logs of your Hugging Face Endpoints through the UI in the 'Logs' tab of your Endpoint. The Container Logs are available when your Endpoint is in the 'Running' state.\"}\u001b[0m\n", + " 40%|████ | 26/65 [09:26<13:21, 20.55s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the latest task added to Hugging Face AutoTrain for Computer Vision?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: Where can you access the logs of your Endpoints in Hugging Face Endpoints?\n", + "\n", + "Answer: You can access the logs of your Hugging Face Endpoints through the UI in the 'Logs' tab of your Endpoint. The Container Logs are available when your Endpoint is in the 'Running' state.\n", + "True answer: In the \"Logs\" tab of your Endpoint through the UI.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Latest Hugging Face AutoTrain task for Computer Vision'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Latest Hugging Face AutoTrain task for Computer Vision'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Image Classification is the latest task added to Hugging Face AutoTrain for Computer Vision.'}\u001b[0m\n", + " 42%|████▏ | 27/65 [09:46<12:55, 20.41s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the default repository type created by the `create_repo` function on Hugging Face Hub?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the latest task added to Hugging Face AutoTrain for Computer Vision?\n", + "\n", + "Answer: Image Classification is the latest task added to Hugging Face AutoTrain for Computer Vision.\n", + "True answer: Image Classification\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'What is the default repository type created by the `create_repo` function on Hugging Face Hub?'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the default repository type created by the `create_repo` function on Hugging Face Hub?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Default repository type created by create_repo function'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The default repository type created by the `create_repo` function on the Hugging Face Hub is a model repository.'}\u001b[0m\n", + " 43%|████▎ | 28/65 [10:08<12:56, 21.00s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "How many splits does the \"duorc\" dataset have?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the default repository type created by the `create_repo` function on Hugging Face Hub?\n", + "\n", + "Answer: The default repository type created by the `create_repo` function on the Hugging Face Hub is a model repository.\n", + "True answer: model\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'duorc dataset splits'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'number of duorc dataset splits'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The 'duorc' dataset has six splits.\"}\u001b[0m\n", + " 45%|████▍ | 29/65 [10:23<11:25, 19.05s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of Fully Sharded Data Parallel (FSDP) in distributed training?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How many splits does the \"duorc\" dataset have?\n", + "\n", + "Answer: The 'duorc' dataset has six splits.\n", + "True answer: Six\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of Fully Sharded Data Parallel FSDP in distributed training'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Benefits of Fully Sharded Data Parallel FSDP in distributed training'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Fully Sharded Data Parallel FSDP explanation'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Role of Fully Sharded Data Parallel FSDP in distributed training'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 46%|████▌ | 30/65 [10:46<11:54, 20.41s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What file format is used to save and store PyTorch model weights more securely than `.bin` files?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of Fully Sharded Data Parallel (FSDP) in distributed training?\n", + "\n", + "Answer: Apologies for the errors and delay. \n", + "\n", + "Fully Sharded Data Parallel (FSDP) is a technique used in distributed training to improve memory efficiency and training speed, especially for large models. It involves partitioning or sharding model parameters across data parallel processes, enabling more efficient utilization of compute resources. This sharding reduces memory consumption per process, allowing training of large models on a greater number of accelerator-equipped nodes. Memory bandwidth is also better utilized, as parameter updates are exchanged between nodes, improving the overall training speed. FSDP is particularly beneficial for training large language models, where memory requirements can be prodigious.\n", + "True answer: FSDP is developed for distributed training of large pretrained models up to 1T parameters by sharding the model parameters, gradients, and optimizer states across data parallel processes.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Secure file format for saving and storing PyTorch model weights'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Secure file format for saving and storing PyTorch model weights other than .bin'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'string', 'description': 'Secure file format for saving and storing PyTorch model weights other than .bin'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'string', 'description': 'File format for secure saving of PyTorch model weights'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 48%|████▊ | 31/65 [11:08<11:43, 20.69s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What type of security certification does Hugging Face have?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What file format is used to save and store PyTorch model weights more securely than `.bin` files?\n", + "\n", + "Answer: Apologies, I encountered errors with the retriever tool and was not able to generate an answer to your question. I can suggest some approaches you could take to troubleshoot and find a solution:\n", + "\n", + "1. Re-format the query: Try simplifying the query or breaking it down into multiple parts. For example: \"File formats for saving PyTorch model weights\" and \"Secure file formats for model weights\".\n", + "\n", + "2. Check tool inputs: Ensure that the tool is receiving the correct input format. In this case, the error indicates that the input must be a string, so confirm that your query is enclosed in quotation marks.\n", + "\n", + "3. Try a different tool: If the retriever tool continues to fail, consider using an alternative tool, such as a web search or a subject matter expert consultation, to find the answer.\n", + "\n", + "4. Expand the knowledge base: If the information you seek is not present in the knowledge base, you may need to add it. This could involve curating relevant documents or adding specific facts to ensure questions can be answered accurately in the future.\n", + "\n", + "I recommend a combination of these approaches to improve your chances of finding an answer.\n", + "True answer: `.safetensors`\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Hugging Face security certification'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face security certification'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Hugging Face has a SOC2 Type 2 security certification, which means they provide security certification to their customers and actively monitor and patch any security weaknesses.'}\u001b[0m\n", + " 49%|████▉ | 32/65 [11:32<11:55, 21.69s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What do RAG models combine to generate outputs?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What type of security certification does Hugging Face have?\n", + "\n", + "Answer: Hugging Face has a SOC2 Type 2 security certification, which means they provide security certification to their customers and actively monitor and patch any security weaknesses.\n", + "True answer: SOC2 Type 2 certified\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is combined in RAG models.'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'RAG model other inputs'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'RAG models combine pre-trained dense retrieval (DPR) and Seq2Seq models to generate outputs.'}\u001b[0m\n", + " 51%|█████ | 33/65 [11:47<10:37, 19.92s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What library does MarkupLMFeatureExtractor use to extract data from HTML and XML files?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What do RAG models combine to generate outputs?\n", + "\n", + "Answer: RAG models combine pre-trained dense retrieval (DPR) and Seq2Seq models to generate outputs.\n", + "True answer: Pretrained dense retrieval (DPR) and sequence-to-sequence models.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Which library does MarkupLMFeatureExtractor use to extract data from HTML and XML files?'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The MarkupLMFeatureExtractor uses the Beautiful Soup library to extract data from HTML and XML files.'}\u001b[0m\n", + " 52%|█████▏ | 34/65 [11:57<08:41, 16.82s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the file size limit for syncing to HF Spaces without using Git-LFS?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What library does MarkupLMFeatureExtractor use to extract data from HTML and XML files?\n", + "\n", + "Answer: The MarkupLMFeatureExtractor uses the Beautiful Soup library to extract data from HTML and XML files.\n", + "True answer: Beautiful Soup\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'sync large files to hf spaces'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'sync files larger than 10MB to hf spaces without git-lfs'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The file size limit for syncing to HF Spaces without using Git-LFS is 10MB.'}\u001b[0m\n", + " 54%|█████▍ | 35/65 [12:19<09:08, 18.29s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the title of the paper introducing the ByT5 model?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the file size limit for syncing to HF Spaces without using Git-LFS?\n", + "\n", + "Answer: The file size limit for syncing to HF Spaces without using Git-LFS is 10MB.\n", + "True answer: 10MB\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'ByT5'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'ByT5: Towards a token-free future with pre-trained byte-to-byte models'}\u001b[0m\n", + " 55%|█████▌ | 36/65 [12:27<07:23, 15.30s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the dimension of the feature vector for the base BERT model?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the title of the paper introducing the ByT5 model?\n", + "\n", + "Answer: ByT5: Towards a token-free future with pre-trained byte-to-byte models\n", + "True answer: ByT5: Towards a token-free future with pre-trained byte-to-byte models\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'dimension of BERT feature vector'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The dimension of the feature vector for the base BERT model is 768.'}\u001b[0m\n", + " 57%|█████▋ | 37/65 [12:39<06:41, 14.34s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What special identifier does the WordPiece Model use for continuing subwords?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the dimension of the feature vector for the base BERT model?\n", + "\n", + "Answer: The dimension of the feature vector for the base BERT model is 768.\n", + "True answer: 768\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'WordPiece Model special identifier continuing subwords'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'WordPiece continuation token'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The special identifier used by the WordPiece model for continuing subwords is '##'.\"}\u001b[0m\n", + " 58%|█████▊ | 38/65 [12:57<06:55, 15.40s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of the 🧨 Diffusers tutorials?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What special identifier does the WordPiece Model use for continuing subwords?\n", + "\n", + "Answer: The special identifier used by the WordPiece model for continuing subwords is '##'.\n", + "True answer: ##\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'urpose of the 🤨 Diffusers tutorials'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The 🧨 Diffusers tutorials aim to introduce users to diffusion models and the fundamentals of the Diffusers library. They teach users how to perform inference and train diffusion models, while also showcasing how the library can be used as a modular toolbox for building custom diffusion systems. In short, the tutorials provide a comprehensive guide to utilizing Diffusers for various applications, with a focus on usability and customizability.'}\u001b[0m\n", + " 60%|██████ | 39/65 [13:11<06:30, 15.01s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the default setting for the `allow_flagging` parameter in Gradio's `Interface`?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the 🧨 Diffusers tutorials?\n", + "\n", + "Answer: The 🧨 Diffusers tutorials aim to introduce users to diffusion models and the fundamentals of the Diffusers library. They teach users how to perform inference and train diffusion models, while also showcasing how the library can be used as a modular toolbox for building custom diffusion systems. In short, the tutorials provide a comprehensive guide to utilizing Diffusers for various applications, with a focus on usability and customizability.\n", + "True answer: To provide a gentle introduction to diffusion models and help understand the library fundamentals.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio Interface allow_flagging parameter'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The default setting for the `allow_flagging` parameter in Gradio's `Interface` is 'auto'.\"}\u001b[0m\n", + " 62%|██████▏ | 40/65 [13:21<05:39, 13.59s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "Where can the full code for the Stable Diffusion demo be found?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the default setting for the `allow_flagging` parameter in Gradio's `Interface`?\n", + "\n", + "Answer: The default setting for the `allow_flagging` parameter in Gradio's `Interface` is 'auto'.\n", + "True answer: \"manual\"\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Stable Diffusion demo code location'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Stable Diffusion demo code location'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Stable Diffusion demo code location'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Stable Diffusion demo code location'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 63%|██████▎ | 41/65 [13:52<07:30, 18.78s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What transformation does the FNet model use to replace the self-attention layer in a BERT model?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: Where can the full code for the Stable Diffusion demo be found?\n", + "\n", + "Answer: The full code for the Stable Diffusion demo can be found at the following links:\n", + "- https://hf.co/spaces/stabilityai/stable-diffusion/tree/main\n", + "- https://blog.problemsolversguild.com/technical/research/2022/11/02/DiffEdit-Implementation.html\n", + "- https://gitlab.com/juliensimon/huggingface-demos/-/tree/main/optimum/stable_diffusion_intel \n", + "\n", + "Additionally, the original codebases for Stable Diffusion v1.0 and v2.0 are available at:\n", + "- https://github.com/CompVis/stable-diffusion \n", + "- https://github.com/Stability-AI/stablediffusion\n", + "True answer: https://hf.co/spaces/stabilityai/stable-diffusion/tree/main\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'fnet model'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'FNet model transformation instead of BERT self attention'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'FNet model self-attention replacement'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'FNet model self-attention replacement'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 65%|██████▍ | 42/65 [14:35<09:55, 25.90s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What type of test should typically accompany a bug fix in Gradio's testing strategy?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What transformation does the FNet model use to replace the self-attention layer in a BERT model?\n", + "\n", + "Answer: The FNet model replaces the self-attention layer in a BERT model with a Fourier transform, specifically returning only the real parts of the transform. This transformation allows FNet to be faster, more memory-efficient, and to train much quicker than the original BERT model. While achieving 92-97% accuracy of BERT counterparts on the GLUE benchmark, the FNet model also has fewer parameters.\n", + "True answer: Fourier transform\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio testing strategy'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"Typically, a dynamic code test accompanies a bug fix in Gradio's testing strategy. This ensures that the fix resolves the issue and prevents future regressions. However, there can be exceptions to this rule, where other types of tests or no tests may be deemed more appropriate for specific cases.\"}\u001b[0m\n", + " 66%|██████▌ | 43/65 [14:51<08:23, 22.88s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "How can you force mixed precision training when initializing the Accelerator in 🤗 Accelerate?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What type of test should typically accompany a bug fix in Gradio's testing strategy?\n", + "\n", + "Answer: Typically, a dynamic code test accompanies a bug fix in Gradio's testing strategy. This ensures that the fix resolves the issue and prevents future regressions. However, there can be exceptions to this rule, where other types of tests or no tests may be deemed more appropriate for specific cases.\n", + "True answer: Dynamic code test\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'force mixed precision training initializing Accelerator 🤗 Accelerate'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'force mixed precision with --fp16'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Initializing the Accelerator --fp16'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'You can force mixed precision training when initializing the Accelerator in 🤗 Accelerate by specifying `fp16=True` when initializing the Accelerator object as shown in the code snippet below:\\npython\\nfrom accelerate import Accelerator\\n\\naccelerator = Accelerator(fp16=True)\\n\\n'}\u001b[0m\n", + " 68%|██████▊ | 44/65 [15:27<09:22, 26.79s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of tokenizers in the NLP pipeline?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How can you force mixed precision training when initializing the Accelerator in 🤗 Accelerate?\n", + "\n", + "Answer: You can force mixed precision training when initializing the Accelerator in 🤗 Accelerate by specifying `fp16=True` when initializing the Accelerator object as shown in the code snippet below:\n", + "python\n", + "from accelerate import Accelerator\n", + "\n", + "accelerator = Accelerator(fp16=True)\n", + "\n", + "\n", + "True answer: By passing `fp16=True` to the Accelerator init.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'purpose of tokenizers NLP'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'purpose of tokenizers NLP'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The purpose of tokenizers in the NLP pipeline is to convert raw text data into numerical data that can be processed by models. Tokenizers act as a pipeline, taking raw text as input and outputting an 'Encoding'. The tokenization process typically involves normalization, truncation, padding, and the addition of special tokens. Training a tokenizer involves a deterministic process of identifying the best subwords for a given corpus, which is algorithm-dependent.\"}\u001b[0m\n", + " 69%|██████▉ | 45/65 [15:51<08:42, 26.13s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of the Safety Checker in the Diffusers library?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of tokenizers in the NLP pipeline?\n", + "\n", + "Answer: The purpose of tokenizers in the NLP pipeline is to convert raw text data into numerical data that can be processed by models. Tokenizers act as a pipeline, taking raw text as input and outputting an 'Encoding'. The tokenization process typically involves normalization, truncation, padding, and the addition of special tokens. Training a tokenizer involves a deterministic process of identifying the best subwords for a given corpus, which is algorithm-dependent.\n", + "True answer: To translate text into data that can be processed by the model.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Safety Checker in Diffusers library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'The Safety Checker flags inappropriate content during inference.'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The Safety Checker in the Diffusers library is a component for screening against harmful and NSFW content. It flags inappropriate content generated by diffusion models, like Stable Diffusion, during inference. The component can be incorporated into models by their creators. It checks generated images against a set of hard-coded harmful concepts in the embedding space. This helps ensure that models interact responsibly and ethically with users, reducing the risk of harm.'}\u001b[0m\n", + " 71%|███████ | 46/65 [16:13<07:49, 24.70s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What Python class allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the Safety Checker in the Diffusers library?\n", + "\n", + "Answer: The Safety Checker in the Diffusers library is a component for screening against harmful and NSFW content. It flags inappropriate content generated by diffusion models, like Stable Diffusion, during inference. The component can be incorporated into models by their creators. It checks generated images against a set of hard-coded harmful concepts in the embedding space. This helps ensure that models interact responsibly and ethically with users, reducing the risk of harm.\n", + "True answer: The Safety Checker checks and compares the class probability of a set of hard-coded harmful concepts in the embedding space against an image after it has been generated to mitigate the risk of generating harmful content.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Python class to retrieve Discussions and Pull Requests from repository'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The Python class that allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub is called `HfApi`, which is part of the `huggingface_hub` library.'}\u001b[0m\n", + " 72%|███████▏ | 47/65 [16:49<08:26, 28.16s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the name of the new library introduced by Hugging Face for hosting scikit-learn models?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What Python class allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub?\n", + "\n", + "Answer: The Python class that allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub is called `HfApi`, which is part of the `huggingface_hub` library.\n", + "True answer: HfApi\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face new library hosting scikit-learn models'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Skops end to end example'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The new library introduced by Hugging Face for hosting scikit-learn models is called Skops. It allows users to host their models on the Hugging Face Hub, collaborate with others, and create model cards for model documentation. For more information, refer to the Skops introduction: https://huggingface.co/blog/skops.'}\u001b[0m\n", + " 74%|███████▍ | 48/65 [17:15<07:46, 27.44s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of Textual Inversion?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the name of the new library introduced by Hugging Face for hosting scikit-learn models?\n", + "\n", + "Answer: The new library introduced by Hugging Face for hosting scikit-learn models is called Skops. It allows users to host their models on the Hugging Face Hub, collaborate with others, and create model cards for model documentation. For more information, refer to the Skops introduction: https://huggingface.co/blog/skops.\n", + "True answer: Skops\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'purpose of Textual Inversion'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'purpose of Textual Inversion'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Textual Inversion is a training technique for personalizing image generation models by learning new text embeddings from a few example images.'}\u001b[0m\n", + " 75%|███████▌ | 49/65 [17:44<07:27, 27.95s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the recommended multiple of batch size for fp16 data type on an A100 GPU?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of Textual Inversion?\n", + "\n", + "Answer: Textual Inversion is a training technique for personalizing image generation models by learning new text embeddings from a few example images.\n", + "True answer: Textual Inversion is a training method for personalizing models by learning new text embeddings from a few example images.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'recommended batch size multiple fp16 data type'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'recommended batch size multiple fp16 data type A100 GPU'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The batch size multiple for fp16 data type on an A100 GPU depends on various factors such as the model size, number of GPUs, and hardware specifications. In the retrieved documents, batch sizes of 10 and 200 have been used successfully on A100 GPUs in different contexts. It is recommended to experiment with different batch sizes and hardware configurations to find the optimal setting for a specific scenario.'}\u001b[0m\n", + " 77%|███████▋ | 50/65 [18:24<07:52, 31.53s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "How do you run a Gradio Blocks app in reload mode using a Python IDE?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the recommended multiple of batch size for fp16 data type on an A100 GPU?\n", + "\n", + "Answer: The batch size multiple for fp16 data type on an A100 GPU depends on various factors such as the model size, number of GPUs, and hardware specifications. In the retrieved documents, batch sizes of 10 and 200 have been used successfully on A100 GPUs in different contexts. It is recommended to experiment with different batch sizes and hardware configurations to find the optimal setting for a specific scenario.\n", + "True answer: 64\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Running Gradio blocks app reload mode Python IDE'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To run a Gradio Blocks app in reload mode using a Python IDE, follow these steps:\\n1. Create an `app.py` file in the root directory, containing your Gradio Blocks code and the `launch()` method.\\n2. In your terminal, navigate to the project directory.\\n3. Run the command: `gradio app.py` instead of `python app.py`. This enables reload mode, which automatically reloads the app whenever you make changes to the file.\\n4. Your app will launch on port 7860 by default, e.g. http://127.0.0.1:7860.'}\u001b[0m\n", + " 78%|███████▊ | 51/65 [18:56<07:27, 31.94s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "How can you install the Hugging Face Unity API in your Unity project?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How do you run a Gradio Blocks app in reload mode using a Python IDE?\n", + "\n", + "Answer: To run a Gradio Blocks app in reload mode using a Python IDE, follow these steps:\n", + "1. Create an `app.py` file in the root directory, containing your Gradio Blocks code and the `launch()` method.\n", + "2. In your terminal, navigate to the project directory.\n", + "3. Run the command: `gradio app.py` instead of `python app.py`. This enables reload mode, which automatically reloads the app whenever you make changes to the file.\n", + "4. Your app will launch on port 7860 by default, e.g. http://127.0.0.1:7860.\n", + "True answer: Run `gradio run.py` in the terminal.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Install Hugging Face Unity API'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Install Hugging Face Unity API'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"To install the Hugging Face Unity API in your Unity project, follow these steps: \\n1. Open your Unity project\\n2. Go to 'Window' -> 'Package Manager'\\n3. Click '+' and select 'Add Package from git URL'\\n4. Enter the URL: 'https://github.com/huggingface/unity-api.git'\\n5. Once installed, the Unity API wizard should pop up. If not, go to 'Window' -> 'Hugging Face API Wizard' for further setup.\"}\u001b[0m\n", + " 80%|████████ | 52/65 [19:24<06:37, 30.61s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the pretraining objective of the Wav2Vec2 context network?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How can you install the Hugging Face Unity API in your Unity project?\n", + "\n", + "Answer: To install the Hugging Face Unity API in your Unity project, follow these steps: \n", + "1. Open your Unity project\n", + "2. Go to 'Window' -> 'Package Manager'\n", + "3. Click '+' and select 'Add Package from git URL'\n", + "4. Enter the URL: 'https://github.com/huggingface/unity-api.git'\n", + "5. Once installed, the Unity API wizard should pop up. If not, go to 'Window' -> 'Hugging Face API Wizard' for further setup.\n", + "True answer: To install the Hugging Face Unity API in your Unity project, go to `Window` -> `Package Manager`, click `+` and select `Add Package from git URL`, then enter `https://github.com/huggingface/unity-api.git`.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Wav2Vec2 context network pretraining objective'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The pretraining objective of the Wav2Vec2 context network is a contrastive task, where the model predicts the true quantized speech representation from a masked prediction.'}\u001b[0m\n", + " 82%|████████▏ | 53/65 [19:46<05:36, 28.03s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the pretraining objective of the Wav2Vec2 context network?\n", + "\n", + "Answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task, where the model predicts the true quantized speech representation from a masked prediction.\n", + "True answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task where the model has to predict the true quantized speech representation of the masked prediction from a set of false ones.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'default checkpoint sentiment analysis pipeline Transformers library'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'distilbert-base-uncased-finetuned-sst-2-english checkpoint'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english'. This checkpoint is a distilled version of the larger BERT model, specifically tailored for sentiment analysis tasks. It has been fine-tuned on the Stanford Sentiment Treebank (SST-2) dataset, which consists of English sentences labeled with their sentiment polarity (positive or negative). The distilled nature of the model makes it more lightweight and faster to use compared to its larger counterparts, without sacrificing much performance.\"}\u001b[0m\n", + " 83%|████████▎ | 54/65 [20:04<04:34, 24.92s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", + "\n", + "Answer: The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english'. This checkpoint is a distilled version of the larger BERT model, specifically tailored for sentiment analysis tasks. It has been fine-tuned on the Stanford Sentiment Treebank (SST-2) dataset, which consists of English sentences labeled with their sentiment polarity (positive or negative). The distilled nature of the model makes it more lightweight and faster to use compared to its larger counterparts, without sacrificing much performance.\n", + "True answer: distilbert base uncased finetuned sst2 english\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of notebook How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of notebook How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The notebook 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi' aims to showcase how to use DeepSpeed to pre-train and fine-tune the 1.6B-parameter GPT2-XL model for causal language modeling on Habana Gaudi, a training solution that delivers lower price/performance when compared to similar training solutions. The notebook demonstrates how to take advantage of the parallel computation capabilities of Habana Gaudi to efficiently train large language models.\"}\u001b[0m\n", + " 85%|████████▍ | 55/65 [20:30<04:14, 25.42s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What command line module does PyTorch provide to run a script on multiple GPUs?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", + "\n", + "Answer: The notebook 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi' aims to showcase how to use DeepSpeed to pre-train and fine-tune the 1.6B-parameter GPT2-XL model for causal language modeling on Habana Gaudi, a training solution that delivers lower price/performance when compared to similar training solutions. The notebook demonstrates how to take advantage of the parallel computation capabilities of Habana Gaudi to efficiently train large language models.\n", + "True answer: To show how to use DeepSpeed to pre-train/fine-tune the 1.6B-parameter GPT2-XL for causal language modeling on Habana Gaudi.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch command line module to run a script on multiple GPUs'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch command line multiple GPUs'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch command line module multiple GPUs'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch multi-gpu command line'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 86%|████████▌ | 56/65 [21:00<04:01, 26.87s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What command line module does PyTorch provide to run a script on multiple GPUs?\n", + "\n", + "Answer: Apologies for the delays and errors encountered in my attempts to answer your question. PyTorch provides the \"torch.distributed\" package and the \"torch.multiprocessing\" package to run a script on multiple GPUs. These packages enable you to distribute your training code across multiple GPUs or machines for faster processing. By utilizing these packages, you can take advantage of multiple GPUs to speed up your deep learning training workflows.\n", + "True answer: torchrun\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Most popular vision transformer image classification hugging face model hub'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'According to the retrieved documents, the most popular vision transformer model for image classification on the Hugging Face Model Hub is `google/vit-base-patch16-224`, which has been trained on ImageNet images with a resolution of 224x224 pixels.'}\u001b[0m\n", + " 88%|████████▊ | 57/65 [21:24<03:26, 25.86s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the command to upload an ESPnet model to a Hugging Face repository?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", + "\n", + "Answer: According to the retrieved documents, the most popular vision transformer model for image classification on the Hugging Face Model Hub is `google/vit-base-patch16-224`, which has been trained on ImageNet images with a resolution of 224x224 pixels.\n", + "True answer: google/vit-base-patch16-224\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'upload ESPnet model Hugging Face repository'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To upload an ESPnet model to a Hugging Face repository, you can use the command:\\n\\nbash\\n./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\\n\\n\\nReplace `username/model_repo` with your Hugging Face username and the desired repository name for your model.'}\u001b[0m\n", + " 89%|████████▉ | 58/65 [21:54<03:09, 27.08s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the command to upload an ESPnet model to a Hugging Face repository?\n", + "\n", + "Answer: To upload an ESPnet model to a Hugging Face repository, you can use the command:\n", + "\n", + "bash\n", + "./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n", + "\n", + "\n", + "Replace `username/model_repo` with your Hugging Face username and the desired repository name for your model.\n", + "True answer: ./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Python dependencies for Inference Endpoints'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Python dependencies for Inference Endpoints'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Custom Python dependencies Inference Endpoints models'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To install custom Python dependencies for Inference Endpoints, a `requirements.txt` file should be added to the model repository.'}\u001b[0m\n", + " 91%|█████████ | 59/65 [22:16<02:33, 25.52s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", + "\n", + "Answer: To install custom Python dependencies for Inference Endpoints, a `requirements.txt` file should be added to the model repository.\n", + "True answer: requirements.txt\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Confirm number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}\u001b[0m\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 92%|█████████▏| 60/65 [22:41<02:06, 25.33s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", + "\n", + "Answer: You need between 3 to 5 images to teach new concepts to Stable Diffusion using Textual Inversion.\n", + "True answer: 3-5 images\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'model checkpoint size limit before automatic sharding in transformers v4.18.0'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'automatic model checkpoint sharding size limit transformers v4.18.0'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Starting from version 4.18.0, model checkpoints exceed the 10GB size limit are automatically sharded into smaller pieces. This is done to reduce memory usage during model loading.'}\u001b[0m\n", + " 94%|█████████▍| 61/65 [23:09<01:45, 26.35s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", + "\n", + "Answer: Starting from version 4.18.0, model checkpoints exceed the 10GB size limit are automatically sharded into smaller pieces. This is done to reduce memory usage during model loading.\n", + "True answer: 10GB\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'purpose of Weights and Biases'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of Weights and Biases for data and machine learning scientists'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose Weights and Biases ML scientists'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of Weights and Biases for data scientists and ML scientists'}}\u001b[0m\n", + "\u001b[31;20mError in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", + " observation = self.toolbox.tools[tool_name](**arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", + " outputs = self.forward(*args, **kwargs)\n", + " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", + " assert isinstance(query, str), \"Your search query must be a string\"\n", + "AssertionError: Your search query must be a string\n", + "\n", + "During handling of the above exception, another exception occurred:\n", + "\n", + "Traceback (most recent call last):\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", + " step_logs = self.step()\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", + " observation = self.execute_tool_call(tool_name, arguments)\n", + " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", + " raise AgentExecutionError(\n", + "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", + "You should only use this tool with a correct input.\n", + "As a reminder, this tool's description is the following:\n", + "\n", + "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", + " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", + "\u001b[31;20mReached max iterations.\u001b[0m\n", + "NoneType: None\n", + " 95%|█████████▌| 62/65 [23:38<01:21, 27.13s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", + "\n", + "Answer: Apologies, there seems to be an error with my tools that is preventing me from retrieving the relevant information to answer your question. I will need further assistance to resolve this issue and provide you with the answer you seek.\n", + "True answer: To track their machine learning experiments at every stage, from training to production.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'hugging face open source library transformer acceleration'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum. It can be found and used at the following links: \\n- https://github.com/huggingface/optimum-intel\\n- https://github.com/huggingface/optimum\\n- https://huggingface.co/docs/optimum/index'}\u001b[0m\n", + " 97%|█████████▋| 63/65 [24:03<00:52, 26.46s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What parameter is used to ensure that elements in a row have the same height in Gradio?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", + "\n", + "Answer: The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum. It can be found and used at the following links: \n", + "- https://github.com/huggingface/optimum-intel\n", + "- https://github.com/huggingface/optimum\n", + "- https://huggingface.co/docs/optimum/index\n", + "True answer: Optimum\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio library same height parameter in row'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio equal_height parameter for row elements'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The parameter used to ensure that elements in a row have the same height in Gradio is 'equal_height'. It is passed to the '.style()' method of 'gr.Row()'.\"}\u001b[0m\n", + " 98%|█████████▊| 64/65 [24:23<00:24, 24.59s/it]\u001b[33;1m======== New task ========\u001b[0m\n", + "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "What is the command to install the latest version of Optimum with OpenVINO support?\n", + "\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What parameter is used to ensure that elements in a row have the same height in Gradio?\n", + "\n", + "Answer: The parameter used to ensure that elements in a row have the same height in Gradio is 'equal_height'. It is passed to the '.style()' method of 'gr.Row()'.\n", + "True answer: equal_height\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Install latest version Optimum OpenVINO'}\u001b[0m\n", + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The command to install the latest version of Optimum with OpenVINO support is: bash\\npip install --upgrade-strategy eager optimum['openvino']\\n\"}\u001b[0m\n", + "100%|██████████| 65/65 [24:39<00:00, 22.76s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the command to install the latest version of Optimum with OpenVINO support?\n", + "\n", + "Answer: The command to install the latest version of Optimum with OpenVINO support is: bash\n", + "pip install --upgrade-strategy eager optimum['openvino']\n", + "\n", + "True answer: pip install --upgrade-strategy eager optimum[\"openvino\"]\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "outputs_agentic_rag = []\n", + "\n", + "for example in tqdm(eval_dataset):\n", + " question = example[\"question\"]\n", + "\n", + " enhanced_question = f\"\"\"Using the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "{question}\"\"\"\n", + " answer = agent.run(enhanced_question)\n", + " print(\"=======================================================\")\n", + " print(f\"Question: {question}\")\n", + " print(f\"Answer: {answer}\")\n", + " print(f'True answer: {example[\"answer\"]}')\n", + "\n", + " results_agentic = {\n", + " \"question\": question,\n", + " \"true_answer\": example[\"answer\"],\n", + " \"source_doc\": example[\"source_doc\"],\n", + " \"generated_answer\": answer,\n", + " }\n", + " outputs_agentic_rag.append(results_agentic)" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 2%|▏ | 1/65 [00:01<01:33, 1.47s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What architecture is the `tokenizers-linux-x64-musl` binary designed for?\n", + "\n", + "Answer: The `tokenizers-linux-x64-musl` binary is designed for the x86_64-unknown-linux-musl architecture.\n", + "True answer: x86_64-unknown-linux-musl\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 3%|▎ | 2/65 [00:02<01:30, 1.44s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the BLIP-Diffusion model?\n", + "\n", + "Answer: BLIP-Diffusion is a model that enables zero-shot subject-driven generation and control-guided zero-shot image generation.\n", + "True answer: The BLIP-Diffusion model is designed for controllable text-to-image generation and editing.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 5%|▍ | 3/65 [00:04<01:41, 1.64s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: How can a user claim authorship of a paper on the Hugging Face Hub?\n", + "\n", + "Answer: A user can claim authorship of a paper by clicking on their name in the Paper page and selecting \"Claim authorship\". After confirming the request, the admin team will validate the claim, and the Paper page will be marked as verified.\n", + "True answer: By clicking their name on the corresponding Paper page and clicking \"claim authorship\", then confirming the request in paper settings for admin team validation.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 6%|▌ | 4/65 [00:08<02:38, 2.59s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the /healthcheck endpoint in the Datasets server API?\n", + "\n", + "Answer: The /healthcheck endpoint in the Datasets server API is to ensure that the app is running.\n", + "True answer: Ensure the app is running\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 8%|▊ | 5/65 [00:12<02:51, 2.86s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the default context window size for Local Attention in the LongT5 model?\n", + "\n", + "Answer: The window size is defined as `w` and is determined by `config.attention_window` in the Longformer model. For the LongT5 model specifically, there is no explicit mention of the default window size in the provided documents.\n", + "True answer: 127 tokens\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 9%|▉ | 6/65 [00:13<02:13, 2.26s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What method is used to load a checkpoint for a task using `AutoPipeline`?\n", + "\n", + "Answer: To load a checkpoint for a task using `AutoPipeline`, the `from_pretrained()` method is used.\n", + "True answer: from_pretrained()\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 11%|█ | 7/65 [00:14<01:55, 1.98s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of Diffusers library?\n", + "\n", + "Answer: The Diffusers library is a generative AI library for creating images, videos, and 3D structures from text or images using state-of-the-art pretrained diffusion models.\n", + "True answer: To serve as a modular toolbox for both inference and training of state-of-the-art pretrained diffusion models across multiple modalities.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 12%|█▏ | 8/65 [00:15<01:33, 1.64s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What method does the EulerAncestralDiscreteScheduler use for sampling?\n", + "\n", + "Answer: The EulerAncestralDiscreteScheduler uses ancestral sampling with Euler method steps.\n", + "True answer: Ancestral sampling with Euler method steps.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 14%|█▍ | 9/65 [00:19<02:05, 2.24s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the name of the large multimodal model that can solve image-text tasks and is based on Flamingo?\n", + "\n", + "Answer: The name of the large multimodal model that can solve image-text tasks and is based on Flamingo is IDEFICS.\n", + "True answer: IDEFICS\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 15%|█▌ | 10/65 [00:20<01:50, 2.01s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the `gradio.Blocks` API?\n", + "\n", + "Answer: The `gradio.Blocks` API allows designers to create web apps with more flexible layouts and complex data flows, all in Python.\n", + "True answer: The `gradio.Blocks` API allows you to have full control over the data flows and layout of your application, enabling the building of complex, multi-step applications.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 17%|█▋ | 11/65 [00:22<01:49, 2.02s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What is the purpose of the two-stage model proposed in the paper \"Hierarchical Text-Conditional Image Generation with CLIP Latents\"?\n", + "\n", + "Answer: To make a good language-specific text-to-image model using a 2-stage training process inspired by PITI.\n", + "\n", + "\n", + "\n", + "3\n", + "True answer: The purpose of the two-stage model is to generate a CLIP image embedding given a text caption and then generate an image conditioned on the image embedding.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 18%|█▊ | 12/65 [00:24<01:38, 1.86s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What command is used to install the requirements for a research project using 🤗 Transformers?\n", + "\n", + "Answer: The command used to install the requirements for a research project using 🤗 Transformers is `pip install -r requirements.txt` (1)\n", + "True answer: pip install -r requirements.txt\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 20%|██ | 13/65 [00:26<01:47, 2.07s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What task does the `roberta-large-mnli` checkpoint perform?\n", + "\n", + "Answer: Performs a text classification task\n", + "True answer: Text classification\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 22%|██▏ | 14/65 [00:27<01:24, 1.65s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What service is replacing the Paid tier of the Inference API at Hugging Face?\n", + "\n", + "Answer: Inference Endpoints.\n", + "True answer: Inference Endpoints\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 23%|██▎ | 15/65 [00:28<01:17, 1.54s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=======================================================\n", + "Question: What architectural feature does SqueezeBERT use instead of fully-connected layers for the Q, K, V, and FFN layers?\n", + "\n", + "Answer: Grouped convolutions\n", + "True answer: Grouped convolutions\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 25%|██▍ | 16/65 [00:29<01:11, 1.47s/it]" + ] + }, { "name": "stdout", "output_type": "stream", @@ -1167,19 +4548,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the ALBERT model?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Document 4 ALBERT parameter reduction techniques'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed are:\\n1. Splitting the embedding matrix into two smaller matrices.\\n2. Using repeating layers split among groups. (Document 4)'}\u001b[0m\n", - " 26%|██▌ | 17/65 [04:17<10:28, 13.08s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What are the three main steps for fine-tuning a model with the 🤗 Datasets library?\n", - "\u001b[0m\n" + " 26%|██▌ | 17/65 [00:34<01:48, 2.25s/it]" ] }, { @@ -1190,8 +4559,10 @@ "Question: What are the two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed?\n", "\n", "Answer: The two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed are:\n", - "1. Splitting the embedding matrix into two smaller matrices.\n", - "2. Using repeating layers split among groups. (Document 4)\n", + "\n", + "1. Separating the larger vocabulary embedding into two smaller matrices (Document 1) or splitting the embedding matrix into two smaller ones (Document 0).\n", + "\n", + "2. Allowing layers to share parameters by using repeating layers split among groups (Document 0, Document 1).\n", "True answer: Splitting the embedding matrix into two smaller matrices and using repeating layers split among groups.\n" ] }, @@ -1199,19 +4570,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Steps for fine-tuning a model with the Hugging Face Datasets library'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What are the steps for fine-tuning a model Hugging Face Datasets library'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The three main steps for fine-tuning a model with the Hugging Face Datasets library are: \\n\\n1. Finding a suitable pretrained model for fine-tuning on the Hugging Face Hub (Document 0).\\n2. Using the Hugging Face Datasets library to download and preprocess publicly available datasets (Document 3, Document 4).\\n3. Optimizing the fine-tuning process with tools like Weights & Biases or Hugging Face Optimum (Document 1, Document 5, Document 6).'}\u001b[0m\n", - " 28%|██▊ | 18/65 [04:39<12:22, 15.80s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers?\n", - "\u001b[0m\n" + " 28%|██▊ | 18/65 [00:40<02:44, 3.49s/it]" ] }, { @@ -1221,11 +4580,10 @@ "=======================================================\n", "Question: What are the three main steps for fine-tuning a model with the 🤗 Datasets library?\n", "\n", - "Answer: The three main steps for fine-tuning a model with the Hugging Face Datasets library are: \n", - "\n", - "1. Finding a suitable pretrained model for fine-tuning on the Hugging Face Hub (Document 0).\n", - "2. Using the Hugging Face Datasets library to download and preprocess publicly available datasets (Document 3, Document 4).\n", - "3. Optimizing the fine-tuning process with tools like Weights & Biases or Hugging Face Optimum (Document 1, Document 5, Document 6).\n", + "Answer: The three main steps for fine-tuning a model with the 🤗 Datasets library are: \n", + "1. Load a dataset from the Hugging Face Hub\n", + "2. Preprocess the data with `Dataset.map()`\n", + "3. Load and compute metrics\n", "True answer: 1. Load a dataset from the Hugging Face Hub. 2. Preprocess the data with `Dataset.map()`. 3. Load and compute metrics.\n" ] }, @@ -1233,18 +4591,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers is 800%.'}\u001b[0m\n", - " 29%|██▉ | 19/65 [04:48<10:32, 13.76s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the command to upload a spaCy pipeline to the Hugging Face Hub?\n", - "\u001b[0m\n" + " 29%|██▉ | 19/65 [00:41<02:01, 2.63s/it]" ] }, { @@ -1254,7 +4601,7 @@ "=======================================================\n", "Question: What is the maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers?\n", "\n", - "Answer: The maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers is 800%.\n", + "Answer: 800%.\n", "True answer: +800%\n" ] }, @@ -1262,18 +4609,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the command to upload a spaCy pipeline to the Hugging Face Hub?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The command to upload a spaCy pipeline to the Hugging Face Hub is: `python -m spacy huggingface-hub push [whl_path]`. Before running this command, ensure you have the `spacy-huggingface-hub` package installed and have logged in with `huggingface-cli login`. The `[whl_path]` should be the path to the .whl file you want to upload.'}\u001b[0m\n", - " 31%|███ | 20/65 [04:59<09:38, 12.85s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the time and memory complexity of the Nyströmformer's approximation of self-attention?\n", - "\u001b[0m\n" + " 31%|███ | 20/65 [00:45<02:20, 3.13s/it]" ] }, { @@ -1283,7 +4619,13 @@ "=======================================================\n", "Question: What is the command to upload a spaCy pipeline to the Hugging Face Hub?\n", "\n", - "Answer: The command to upload a spaCy pipeline to the Hugging Face Hub is: `python -m spacy huggingface-hub push [whl_path]`. Before running this command, ensure you have the `spacy-huggingface-hub` package installed and have logged in with `huggingface-cli login`. The `[whl_path]` should be the path to the .whl file you want to upload.\n", + "Answer: The command to upload a spaCy pipeline to the Hugging Face Hub is:\n", + "\n", + "```bash\n", + "python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]\n", + "```\n", + "\n", + "Make sure to install `spacy-huggingface-hub` before running this command.\n", "True answer: python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl\n" ] }, @@ -1291,18 +4633,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': \"What is the time and memory complexity of the Nyströmformer's approximation of self-attention?\"}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The Nyströmformer's approximation of self-attention has a time complexity of O(n) as mentioned in Document 1, and a memory complexity of O(n^2) as mentioned in Document 2.\"}\u001b[0m\n", - " 32%|███▏ | 21/65 [05:11<09:15, 12.63s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the goal of the Named Entity Recognition task in token classification?\n", - "\u001b[0m\n" + " 32%|███▏ | 21/65 [00:49<02:35, 3.53s/it]" ] }, { @@ -1312,7 +4643,7 @@ "=======================================================\n", "Question: What is the time and memory complexity of the Nyströmformer's approximation of self-attention?\n", "\n", - "Answer: The Nyströmformer's approximation of self-attention has a time complexity of O(n) as mentioned in Document 1, and a memory complexity of O(n^2) as mentioned in Document 2.\n", + "Answer: The Nyströmformer model approximates standard self-attention with linear or O(n) time and memory complexity.\n", "True answer: O(n)\n" ] }, @@ -1320,18 +4651,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the goal of the Named Entity Recognition task in token classification?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The goal of the Named Entity Recognition (NER) task in token classification is to find and label entities in a piece of text, such as person, location, or organization. Each token in the sentence is assigned a specific class or label corresponding to an entity, with an additional class for tokens that are not entities.'}\u001b[0m\n", - " 34%|███▍ | 22/65 [05:23<08:56, 12.47s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the resolution of images used by the CLIPSeg model?\n", - "\u001b[0m\n" + " 34%|███▍ | 22/65 [00:55<03:00, 4.20s/it]" ] }, { @@ -1341,7 +4661,7 @@ "=======================================================\n", "Question: What is the goal of the Named Entity Recognition task in token classification?\n", "\n", - "Answer: The goal of the Named Entity Recognition (NER) task in token classification is to find and label entities in a piece of text, such as person, location, or organization. Each token in the sentence is assigned a specific class or label corresponding to an entity, with an additional class for tokens that are not entities.\n", + "Answer: The goal of the Named Entity Recognition task in token classification is to identify named entities in a piece of text, such as a person, location, or organization, and classify each token in the text into one of these categories or a \"none of the above\" category.\n", "True answer: The goal of the Named Entity Recognition task is to find the entities in a piece of text, such as person, location, or organization.\n" ] }, @@ -1349,52 +4669,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'CLIPSeg model'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'image': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 478, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_10832/4081367717.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 710, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 803, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 481, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'CLIPSeg input image resolution'}\u001b[0m\n", - "\u001b[31;20mReached max iterations.\u001b[0m\n", - "NoneType: None\n", - " 35%|███▌ | 23/65 [05:54<12:28, 17.83s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What can you use Gradio for?\n", - "\u001b[0m\n" + " 35%|███▌ | 23/65 [00:59<02:56, 4.21s/it]" ] }, { @@ -1404,7 +4679,7 @@ "=======================================================\n", "Question: What is the resolution of images used by the CLIPSeg model?\n", "\n", - "Answer: The CLIPSeg model uses images with a resolution of 352 x 352 pixels. This information was mentioned in Document 0. The low-resolution output of the model is a limitation, leading to less precise segmentation masks. The model's authors suggest that, for more accurate segmentations, one could fine-tune a state-of-the-art segmentation model using CLIPSeg's rough segmentation labels. This two-step process can improve the final output's accuracy.\n", + "Answer: The CLIPSeg model uses images of 352x352 pixels.\n", "True answer: 352 x 352 pixels\n" ] }, @@ -1412,19 +4687,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is Gradio used for.'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio app use case'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Gradio is a Python library used for creating web applications to showcase machine learning models and data processing pipelines. It allows users to quickly build customized apps that can be deployed on platforms like Hugging Face Spaces.'}\u001b[0m\n", - " 37%|███▋ | 24/65 [06:13<12:31, 18.34s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What TensorFlow API function is used to load a saved tensor file?\n", - "\u001b[0m\n" + " 37%|███▋ | 24/65 [01:01<02:26, 3.58s/it]" ] }, { @@ -1434,7 +4697,7 @@ "=======================================================\n", "Question: What can you use Gradio for?\n", "\n", - "Answer: Gradio is a Python library used for creating web applications to showcase machine learning models and data processing pipelines. It allows users to quickly build customized apps that can be deployed on platforms like Hugging Face Spaces.\n", + "Answer: You can use Gradio, a Python library, to create customizable web apps for machine learning models and data-processing pipelines.\n", "True answer: Create a demo for your machine learning model, share your machine learning model with others, and debug your model.\n" ] }, @@ -1442,19 +4705,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'TensorFlow API function to load saved tensor file'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'TensorFlow function load tensor file'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The TensorFlow API function used to load a saved tensor file is safetensors.tensorflow.load or safetensors.tensorflow.load_file. This information is found in Document 0.'}\u001b[0m\n", - " 38%|███▊ | 25/65 [06:27<11:15, 16.89s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "Where can you access the logs of your Endpoints in Hugging Face Endpoints?\n", - "\u001b[0m\n" + " 38%|███▊ | 25/65 [01:04<02:11, 3.28s/it]" ] }, { @@ -1464,7 +4715,7 @@ "=======================================================\n", "Question: What TensorFlow API function is used to load a saved tensor file?\n", "\n", - "Answer: The TensorFlow API function used to load a saved tensor file is safetensors.tensorflow.load or safetensors.tensorflow.load_file. This information is found in Document 0.\n", + "Answer: safetensors.tensorflow.load_file is the API function used to load a saved tensor file.\n", "True answer: safetensors.tensorflow.load_file\n" ] }, @@ -1472,18 +4723,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face Endpoints logs'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"You can access the logs of your Hugging Face Endpoints in the UI, under the 'Logs' tab of your Endpoint. This feature is mentioned in Document 0.\"}\u001b[0m\n", - " 40%|████ | 26/65 [06:36<09:35, 14.75s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the latest task added to Hugging Face AutoTrain for Computer Vision?\n", - "\u001b[0m\n" + " 40%|████ | 26/65 [01:06<01:53, 2.91s/it]" ] }, { @@ -1493,7 +4733,7 @@ "=======================================================\n", "Question: Where can you access the logs of your Endpoints in Hugging Face Endpoints?\n", "\n", - "Answer: You can access the logs of your Hugging Face Endpoints in the UI, under the 'Logs' tab of your Endpoint. This feature is mentioned in Document 0.\n", + "Answer: You can access and read the logs of your Endpoints in Hugging Face Endpoints through the UI in the \"Logs\" tab of your Endpoint. This feature provides access to both the build logs of your Image artifacts and the Container Logs during inference.\n", "True answer: In the \"Logs\" tab of your Endpoint through the UI.\n" ] }, @@ -1501,18 +4741,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face AutoTrain for Computer Vision'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Image classification is the latest task added to Hugging Face AutoTrain for Computer Vision, as mentioned in the first retrieved document.'}\u001b[0m\n", - " 42%|████▏ | 27/65 [06:46<08:22, 13.22s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the default repository type created by the `create_repo` function on Hugging Face Hub?\n", - "\u001b[0m\n" + " 42%|████▏ | 27/65 [01:07<01:22, 2.18s/it]" ] }, { @@ -1522,7 +4751,7 @@ "=======================================================\n", "Question: What is the latest task added to Hugging Face AutoTrain for Computer Vision?\n", "\n", - "Answer: Image classification is the latest task added to Hugging Face AutoTrain for Computer Vision, as mentioned in the first retrieved document.\n", + "Answer: Image classification.\n", "True answer: Image Classification\n" ] }, @@ -1530,18 +4759,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the default repository type created by the `create_repo` function on Hugging Face Hub?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The default repository type created by the `create_repo` function on Hugging Face Hub is a model repository, as seen in Document 2.'}\u001b[0m\n", - " 43%|████▎ | 28/65 [07:00<08:15, 13.40s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "How many splits does the \"duorc\" dataset have?\n", - "\u001b[0m\n" + " 43%|████▎ | 28/65 [01:08<01:12, 1.95s/it]" ] }, { @@ -1551,7 +4769,7 @@ "=======================================================\n", "Question: What is the default repository type created by the `create_repo` function on Hugging Face Hub?\n", "\n", - "Answer: The default repository type created by the `create_repo` function on Hugging Face Hub is a model repository, as seen in Document 2.\n", + "Answer: By default, the `create_repo` method creates a model repository.\n", "True answer: model\n" ] }, @@ -1559,18 +4777,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'duorc dataset'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The 'duorc' dataset has three splits: 'train', 'validation', and 'test'.\"}\u001b[0m\n", - " 45%|████▍ | 29/65 [07:17<08:39, 14.44s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of Fully Sharded Data Parallel (FSDP) in distributed training?\n", - "\u001b[0m\n" + " 45%|████▍ | 29/65 [01:10<01:15, 2.09s/it]" ] }, { @@ -1580,7 +4787,7 @@ "=======================================================\n", "Question: How many splits does the \"duorc\" dataset have?\n", "\n", - "Answer: The 'duorc' dataset has three splits: 'train', 'validation', and 'test'.\n", + "Answer: Six\n", "True answer: Six\n" ] }, @@ -1588,18 +4795,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Fully Sharded Data Parallel (FSDP) in distributed training purpose'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The purpose of Fully Sharded Data Parallel (FSDP) in distributed training is to efficiently train large pre-trained models with up to 1T parameters by sharding model parameters, gradients, and optimizer states across data parallel processes, thereby allowing larger batch or model sizes to be used. This reduces memory usage by avoiding the need to replicate the entire model on each GPU.'}\u001b[0m\n", - " 46%|████▌ | 30/65 [07:29<08:02, 13.79s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What file format is used to save and store PyTorch model weights more securely than `.bin` files?\n", - "\u001b[0m\n" + " 46%|████▌ | 30/65 [01:13<01:16, 2.19s/it]" ] }, { @@ -1609,7 +4805,7 @@ "=======================================================\n", "Question: What is the purpose of Fully Sharded Data Parallel (FSDP) in distributed training?\n", "\n", - "Answer: The purpose of Fully Sharded Data Parallel (FSDP) in distributed training is to efficiently train large pre-trained models with up to 1T parameters by sharding model parameters, gradients, and optimizer states across data parallel processes, thereby allowing larger batch or model sizes to be used. This reduces memory usage by avoiding the need to replicate the entire model on each GPU.\n", + "Answer: Fully Sharded Data Parallel (FSDP) is a method that shards a model's parameters, gradients, and optimizer states across multiple GPUs. This reduces memory usage and allows for the training of larger models on fewer GPUs.\n", "True answer: FSDP is developed for distributed training of large pretrained models up to 1T parameters by sharding the model parameters, gradients, and optimizer states across data parallel processes.\n" ] }, @@ -1617,18 +4813,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': \"What file format is used to save and store PyTorch model weights more securely than '.bin' files?\"}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The file format used to save and store PyTorch model weights more securely than `.bin` files is the `safetensor` format.'}\u001b[0m\n", - " 48%|████▊ | 31/65 [07:43<07:55, 13.98s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What type of security certification does Hugging Face have?\n", - "\u001b[0m\n" + " 48%|████▊ | 31/65 [01:14<01:07, 1.98s/it]" ] }, { @@ -1638,7 +4823,7 @@ "=======================================================\n", "Question: What file format is used to save and store PyTorch model weights more securely than `.bin` files?\n", "\n", - "Answer: The file format used to save and store PyTorch model weights more securely than `.bin` files is the `safetensor` format.\n", + "Answer: .safetensors\n", "True answer: `.safetensors`\n" ] }, @@ -1646,18 +4831,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face security certification'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Hugging Face has SOC2 Type 2 security certification.'}\u001b[0m\n", - " 49%|████▉ | 32/65 [07:54<07:05, 12.89s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What do RAG models combine to generate outputs?\n", - "\u001b[0m\n" + " 49%|████▉ | 32/65 [01:15<00:53, 1.62s/it]" ] }, { @@ -1667,7 +4841,7 @@ "=======================================================\n", "Question: What type of security certification does Hugging Face have?\n", "\n", - "Answer: Hugging Face has SOC2 Type 2 security certification.\n", + "Answer: Hugging Face is SOC2 Type 2 certified (2).\n", "True answer: SOC2 Type 2 certified\n" ] }, @@ -1675,18 +4849,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What do RAG models combine to generate outputs?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'RAG models combine the capabilities of pre-trained dense retrieval (DPR) and sequence-to-sequence models. The models retrieve documents, pass them to a seq2seq model, and then marginalize the results to generate outputs.'}\u001b[0m\n", - " 51%|█████ | 33/65 [08:12<07:41, 14.43s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What library does MarkupLMFeatureExtractor use to extract data from HTML and XML files?\n", - "\u001b[0m\n" + " 51%|█████ | 33/65 [01:17<00:59, 1.86s/it]" ] }, { @@ -1696,7 +4859,7 @@ "=======================================================\n", "Question: What do RAG models combine to generate outputs?\n", "\n", - "Answer: RAG models combine the capabilities of pre-trained dense retrieval (DPR) and sequence-to-sequence models. The models retrieve documents, pass them to a seq2seq model, and then marginalize the results to generate outputs.\n", + "Answer: Retrieval-augmented generation (RAG) models combine the powers of pre-trained dense retrieval (DPR) and sequence-to-sequence models during the generation of outputs.\n", "True answer: Pretrained dense retrieval (DPR) and sequence-to-sequence models.\n" ] }, @@ -1704,18 +4867,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'MarkupLMFeatureExtractor'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The MarkupLMFeatureExtractor uses the Beautiful Soup python library for extracting data from HTML and XML files.'}\u001b[0m\n", - " 52%|█████▏ | 34/65 [08:30<07:57, 15.42s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the file size limit for syncing to HF Spaces without using Git-LFS?\n", - "\u001b[0m\n" + " 52%|█████▏ | 34/65 [01:19<00:52, 1.70s/it]" ] }, { @@ -1725,7 +4877,7 @@ "=======================================================\n", "Question: What library does MarkupLMFeatureExtractor use to extract data from HTML and XML files?\n", "\n", - "Answer: The MarkupLMFeatureExtractor uses the Beautiful Soup python library for extracting data from HTML and XML files.\n", + "Answer: Beautiful Soup.\n", "True answer: Beautiful Soup\n" ] }, @@ -1733,18 +4885,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'file size limit syncing HF Spaces without Git-LFS'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The file size limit for syncing to HF Spaces without using Git-LFS is less than or equal to 10MB (as mentioned in Document 1).'}\u001b[0m\n", - " 54%|█████▍ | 35/65 [08:41<07:04, 14.14s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the title of the paper introducing the ByT5 model?\n", - "\u001b[0m\n" + " 54%|█████▍ | 35/65 [01:20<00:47, 1.58s/it]" ] }, { @@ -1754,7 +4895,7 @@ "=======================================================\n", "Question: What is the file size limit for syncing to HF Spaces without using Git-LFS?\n", "\n", - "Answer: The file size limit for syncing to HF Spaces without using Git-LFS is less than or equal to 10MB (as mentioned in Document 1).\n", + "Answer: 10MB\n", "True answer: 10MB\n" ] }, @@ -1762,18 +4903,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Introduction paper ByT5 model'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The title of the paper introducing the ByT5 model is: 'ByT5: Towards a token-free future with pre-trained byte-to-byte models'.\"}\u001b[0m\n", - " 55%|█████▌ | 36/65 [08:50<06:06, 12.63s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the dimension of the feature vector for the base BERT model?\n", - "\u001b[0m\n" + " 55%|█████▌ | 36/65 [01:22<00:44, 1.54s/it]" ] }, { @@ -1783,7 +4913,7 @@ "=======================================================\n", "Question: What is the title of the paper introducing the ByT5 model?\n", "\n", - "Answer: The title of the paper introducing the ByT5 model is: 'ByT5: Towards a token-free future with pre-trained byte-to-byte models'.\n", + "Answer: ByT5: Towards a token-free future with pre-trained byte-to-byte models\n", "True answer: ByT5: Towards a token-free future with pre-trained byte-to-byte models\n" ] }, @@ -1791,18 +4921,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'feature vector dimension in base bert model'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The dimension of the feature vector for the base BERT model is 768 [source document 0].'}\u001b[0m\n", - " 57%|█████▋ | 37/65 [08:58<05:12, 11.17s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What special identifier does the WordPiece Model use for continuing subwords?\n", - "\u001b[0m\n" + " 57%|█████▋ | 37/65 [01:23<00:38, 1.38s/it]" ] }, { @@ -1812,7 +4931,7 @@ "=======================================================\n", "Question: What is the dimension of the feature vector for the base BERT model?\n", "\n", - "Answer: The dimension of the feature vector for the base BERT model is 768 [source document 0].\n", + "Answer: The dimension of the feature vector for the base BERT model is 768.\n", "True answer: 768\n" ] }, @@ -1820,18 +4939,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What special identifier does the WordPiece Model use for continuing subwords?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The special identifier used by the WordPiece Model to indicate continuing subwords is '##'.\"}\u001b[0m\n", - " 58%|█████▊ | 38/65 [09:05<04:28, 9.95s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of the 🧨 Diffusers tutorials?\n", - "\u001b[0m\n" + " 58%|█████▊ | 38/65 [01:26<00:56, 2.09s/it]" ] }, { @@ -1841,7 +4949,7 @@ "=======================================================\n", "Question: What special identifier does the WordPiece Model use for continuing subwords?\n", "\n", - "Answer: The special identifier used by the WordPiece Model to indicate continuing subwords is '##'.\n", + "Answer: The `##` prefix is used to identify the start of a subword that is part of a word.\n", "True answer: ##\n" ] }, @@ -1849,18 +4957,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of the Diffusers tutorials'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The 🧨 Diffusers tutorials serve as a gentle introduction to diffusion models, with a focus on the core components of the 🧨 Diffusers library. They aim to help beginners understand how to use the library effectively, both for inference and training, with a focus on usability, simplicity, and customizability. The tutorials also emphasize the project's philosophy, which includes values such as accessibility, reproducibility, and responsibility. (Sources: Documents 0, 1, 2, 3, 4, 6)\"}\u001b[0m\n", - " 60%|██████ | 39/65 [09:14<04:17, 9.90s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the default setting for the `allow_flagging` parameter in Gradio's `Interface`?\n", - "\u001b[0m\n" + " 60%|██████ | 39/65 [01:27<00:44, 1.70s/it]" ] }, { @@ -1870,7 +4967,7 @@ "=======================================================\n", "Question: What is the purpose of the 🧨 Diffusers tutorials?\n", "\n", - "Answer: The 🧨 Diffusers tutorials serve as a gentle introduction to diffusion models, with a focus on the core components of the 🧨 Diffusers library. They aim to help beginners understand how to use the library effectively, both for inference and training, with a focus on usability, simplicity, and customizability. The tutorials also emphasize the project's philosophy, which includes values such as accessibility, reproducibility, and responsibility. (Sources: Documents 0, 1, 2, 3, 4, 6)\n", + "Answer: To provide a beginner-friendly introduction to diffusion models.\n", "True answer: To provide a gentle introduction to diffusion models and help understand the library fundamentals.\n" ] }, @@ -1878,18 +4975,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Default allow_flagging parameter Gradio Interface.'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The default setting for the `allow_flagging` parameter in Gradio's `Interface` is 'manual'.\"}\u001b[0m\n", - " 62%|██████▏ | 40/65 [09:21<03:44, 8.97s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "Where can the full code for the Stable Diffusion demo be found?\n", - "\u001b[0m\n" + " 62%|██████▏ | 40/65 [01:30<00:49, 1.97s/it]" ] }, { @@ -1899,7 +4985,7 @@ "=======================================================\n", "Question: What is the default setting for the `allow_flagging` parameter in Gradio's `Interface`?\n", "\n", - "Answer: The default setting for the `allow_flagging` parameter in Gradio's `Interface` is 'manual'.\n", + "Answer: \"auto\"\n", "True answer: \"manual\"\n" ] }, @@ -1907,18 +4993,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'stable diffusion demo code'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The full code for the Stable Diffusion demo can be found at https://hf.co/spaces/stabilityai/stable-diffusion/tree/main or https://gitlab.com/juliensimon/huggingface-demos/-/tree/main/optimum/stable_diffusion_intel.'}\u001b[0m\n", - " 63%|██████▎ | 41/65 [09:31<03:39, 9.14s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What transformation does the FNet model use to replace the self-attention layer in a BERT model?\n", - "\u001b[0m\n" + " 63%|██████▎ | 41/65 [01:34<01:04, 2.70s/it]" ] }, { @@ -1928,7 +5003,7 @@ "=======================================================\n", "Question: Where can the full code for the Stable Diffusion demo be found?\n", "\n", - "Answer: The full code for the Stable Diffusion demo can be found at https://hf.co/spaces/stabilityai/stable-diffusion/tree/main or https://gitlab.com/juliensimon/huggingface-demos/-/tree/main/optimum/stable_diffusion_intel.\n", + "Answer: Document 0 and Document 5: https://hf.co/spaces/stabilityai/stable-diffusion/tree/main and https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion.\n", "True answer: https://hf.co/spaces/stabilityai/stable-diffusion/tree/main\n" ] }, @@ -1936,18 +5011,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What transformation does the FNet model use to replace the self-attention layer in a BERT model?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The FNet model replaces the self-attention layer in a BERT model with a Fourier transform. Specifically, it uses the real parts of the Fourier transform.'}\u001b[0m\n", - " 65%|██████▍ | 42/65 [09:40<03:30, 9.15s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What type of test should typically accompany a bug fix in Gradio's testing strategy?\n", - "\u001b[0m\n" + " 65%|██████▍ | 42/65 [01:38<01:11, 3.11s/it]" ] }, { @@ -1957,26 +5021,15 @@ "=======================================================\n", "Question: What transformation does the FNet model use to replace the self-attention layer in a BERT model?\n", "\n", - "Answer: The FNet model replaces the self-attention layer in a BERT model with a Fourier transform. Specifically, it uses the real parts of the Fourier transform.\n", + "Answer: The transformation used by the FNet model to replace the self-attention layer in a BERT model is a Fourier Transform.\n", "True answer: Fourier transform\n" ] }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': \"What type of test should typically accompany a bug fix in Gradio's testing strategy?\"}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"A dynamic code test should typically accompany a bug fix in Gradio's testing strategy (as mentioned in documents 0 and 1).\"}\u001b[0m\n", - " 66%|██████▌ | 43/65 [09:56<04:06, 11.20s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "How can you force mixed precision training when initializing the Accelerator in 🤗 Accelerate?\n", - "\u001b[0m\n" + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 66%|██████▌ | 43/65 [01:39<00:51, 2.34s/it]" ] }, { @@ -1986,7 +5039,7 @@ "=======================================================\n", "Question: What type of test should typically accompany a bug fix in Gradio's testing strategy?\n", "\n", - "Answer: A dynamic code test should typically accompany a bug fix in Gradio's testing strategy (as mentioned in documents 0 and 1).\n", + "Answer: Dynamic code test\n", "True answer: Dynamic code test\n" ] }, @@ -1994,19 +5047,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How can you force mixed precision training when initializing the Accelerator in Accelerate?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How to force mixed precision training with accelerate library'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'You can force mixed precision training when initializing the Accelerator in 🤗 Accelerate by using the `--fp16` flag when running your scripts. This enables mixed precision training for GPUs with Pascal architecture or newer, providing faster training speeds with the same final results.'}\u001b[0m\n", - " 68%|██████▊ | 44/65 [10:22<05:25, 15.51s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of tokenizers in the NLP pipeline?\n", - "\u001b[0m\n" + " 68%|██████▊ | 44/65 [01:40<00:41, 1.98s/it]" ] }, { @@ -2016,7 +5057,7 @@ "=======================================================\n", "Question: How can you force mixed precision training when initializing the Accelerator in 🤗 Accelerate?\n", "\n", - "Answer: You can force mixed precision training when initializing the Accelerator in 🤗 Accelerate by using the `--fp16` flag when running your scripts. This enables mixed precision training for GPUs with Pascal architecture or newer, providing faster training speeds with the same final results.\n", + "Answer: To force mixed precision training when initializing the accelerator in 🤗 Accelerate, use the command --fp16.\n", "True answer: By passing `fp16=True` to the Accelerator init.\n" ] }, @@ -2024,18 +5065,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'purpose of tokenizers in NLP pipeline'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Tokenizers are a crucial component of the NLP pipeline, responsible for converting raw text data into numerical data that can be processed by machine learning models. They serve as a bridge between text inputs and the numerical format required by models. Tokenizers also keep track of spans of text corresponding to each word and token, facilitating tasks like masked language modeling and token classification. They offer features like vocabulary training, normalization, truncation, padding, and addition of special tokens.'}\u001b[0m\n", - " 69%|██████▉ | 45/65 [10:37<05:12, 15.63s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of the Safety Checker in the Diffusers library?\n", - "\u001b[0m\n" + " 69%|██████▉ | 45/65 [01:41<00:36, 1.81s/it]" ] }, { @@ -2045,7 +5075,7 @@ "=======================================================\n", "Question: What is the purpose of tokenizers in the NLP pipeline?\n", "\n", - "Answer: Tokenizers are a crucial component of the NLP pipeline, responsible for converting raw text data into numerical data that can be processed by machine learning models. They serve as a bridge between text inputs and the numerical format required by models. Tokenizers also keep track of spans of text corresponding to each word and token, facilitating tasks like masked language modeling and token classification. They offer features like vocabulary training, normalization, truncation, padding, and addition of special tokens.\n", + "Answer: The purpose of tokenizers in the NLP pipeline is to translate text data into numerical data that can be processed by NLP models.\n", "True answer: To translate text into data that can be processed by the model.\n" ] }, @@ -2053,18 +5083,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the purpose of the Safety Checker in the Diffusers library?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The purpose of the Safety Checker in the Diffusers library is to flag and prevent inappropriate and harmful content generated during inference. It is a mandatory component for public-facing applications to ensure responsible usage and to avoid exposing unfiltered results to the public. The Safety Checker uses a set of hard-coded harmful concepts in the embedding space to compare and identify inappropriate content. This component can be disabled, but it is strongly recommended to keep it enabled unless for specific use cases like analyzing network behavior.'}\u001b[0m\n", - " 71%|███████ | 46/65 [10:50<04:40, 14.76s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What Python class allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub?\n", - "\u001b[0m\n" + " 71%|███████ | 46/65 [01:46<00:52, 2.74s/it]" ] }, { @@ -2074,7 +5093,7 @@ "=======================================================\n", "Question: What is the purpose of the Safety Checker in the Diffusers library?\n", "\n", - "Answer: The purpose of the Safety Checker in the Diffusers library is to flag and prevent inappropriate and harmful content generated during inference. It is a mandatory component for public-facing applications to ensure responsible usage and to avoid exposing unfiltered results to the public. The Safety Checker uses a set of hard-coded harmful concepts in the embedding space to compare and identify inappropriate content. This component can be disabled, but it is strongly recommended to keep it enabled unless for specific use cases like analyzing network behavior.\n", + "Answer: The **Safety Checker** is a component of Diffusers library that flags inappropriate content generated by models during inference. It screens against harmful content by checking and comparing the class probability of hardcoded harmful concepts in the embedding space against an image.\n", "True answer: The Safety Checker checks and compares the class probability of a set of hard-coded harmful concepts in the embedding space against an image after it has been generated to mitigate the risk of generating harmful content.\n" ] }, @@ -2082,18 +5101,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Python class retrieve Discussions and Pull Requests from repository Hugging Face Hub'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `HfApi` class allows you to retrieve Discussions and Pull Requests on a given repository (ref: Document 0).'}\u001b[0m\n", - " 72%|███████▏ | 47/65 [11:00<04:00, 13.37s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the name of the new library introduced by Hugging Face for hosting scikit-learn models?\n", - "\u001b[0m\n" + " 72%|███████▏ | 47/65 [01:51<01:00, 3.38s/it]" ] }, { @@ -2103,7 +5111,7 @@ "=======================================================\n", "Question: What Python class allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub?\n", "\n", - "Answer: The `HfApi` class allows you to retrieve Discussions and Pull Requests on a given repository (ref: Document 0).\n", + "Answer: HfApi\n", "True answer: HfApi\n" ] }, @@ -2111,18 +5119,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face new library for hosting scikit-learn models'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The name of the new library introduced by Hugging Face for hosting scikit-learn models is Skops.'}\u001b[0m\n", - " 74%|███████▍ | 48/65 [11:23<04:35, 16.21s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of Textual Inversion?\n", - "\u001b[0m\n" + " 74%|███████▍ | 48/65 [01:52<00:43, 2.55s/it]" ] }, { @@ -2132,7 +5129,7 @@ "=======================================================\n", "Question: What is the name of the new library introduced by Hugging Face for hosting scikit-learn models?\n", "\n", - "Answer: The name of the new library introduced by Hugging Face for hosting scikit-learn models is Skops.\n", + "Answer: huggingface_hub\n", "True answer: Skops\n" ] }, @@ -2140,18 +5137,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the purpose of Textual Inversion?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Textual Inversion is a training technique for personalising image generation models, allowing them to learn and update text embeddings to match example images provided. This method, detailed in [this paper](https://hf.co/papers/2208.01618), requires just a few example images to teach a model new concepts and tailor it towards specific ideas.'}\u001b[0m\n", - " 75%|███████▌ | 49/65 [11:40<04:24, 16.54s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the recommended multiple of batch size for fp16 data type on an A100 GPU?\n", - "\u001b[0m\n" + " 75%|███████▌ | 49/65 [01:54<00:41, 2.60s/it]" ] }, { @@ -2161,7 +5147,7 @@ "=======================================================\n", "Question: What is the purpose of Textual Inversion?\n", "\n", - "Answer: Textual Inversion is a training technique for personalising image generation models, allowing them to learn and update text embeddings to match example images provided. This method, detailed in [this paper](https://hf.co/papers/2208.01618), requires just a few example images to teach a model new concepts and tailor it towards specific ideas.\n", + "Answer: Textual inversion is a technique that allows a model to learn a new concept from a few sample images. This gives the user more control over the generated images and allows the model to be tailored towards specific concepts.\n", "True answer: Textual Inversion is a training method for personalizing models by learning new text embeddings from a few example images.\n" ] }, @@ -2169,19 +5155,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the recommended multiple of batch size for fp16 data type on an A100 GPU?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': \"Document containing 'fp16', 'A100 GPU', 'batch size' and '4'\"}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Based on the retrieved documents, it seems that a batch size of 4 is recommended for fp16 data type on an A100 GPU. This information was found in Document 6.'}\u001b[0m\n", - " 77%|███████▋ | 50/65 [12:06<04:49, 19.31s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "How do you run a Gradio Blocks app in reload mode using a Python IDE?\n", - "\u001b[0m\n" + " 77%|███████▋ | 50/65 [01:55<00:30, 2.01s/it]" ] }, { @@ -2191,7 +5165,7 @@ "=======================================================\n", "Question: What is the recommended multiple of batch size for fp16 data type on an A100 GPU?\n", "\n", - "Answer: Based on the retrieved documents, it seems that a batch size of 4 is recommended for fp16 data type on an A100 GPU. This information was found in Document 6.\n", + "Answer: 5\n", "True answer: 64\n" ] }, @@ -2199,18 +5173,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How to run a Gradio Blocks app in reload mode using a Python IDE?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To run a Gradio Blocks app in reload mode using a Python IDE, you can use the `gradio` command followed by the name of your Python script file, for example: `gradio app.py`. This will launch the app in reload mode, automatically reloading whenever you make changes to the file. The app will be accessible at http://127.0.0.1:7860 by default.'}\u001b[0m\n", - " 78%|███████▊ | 51/65 [12:21<04:11, 17.94s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "How can you install the Hugging Face Unity API in your Unity project?\n", - "\u001b[0m\n" + " 78%|███████▊ | 51/65 [01:58<00:33, 2.41s/it]" ] }, { @@ -2220,7 +5183,7 @@ "=======================================================\n", "Question: How do you run a Gradio Blocks app in reload mode using a Python IDE?\n", "\n", - "Answer: To run a Gradio Blocks app in reload mode using a Python IDE, you can use the `gradio` command followed by the name of your Python script file, for example: `gradio app.py`. This will launch the app in reload mode, automatically reloading whenever you make changes to the file. The app will be accessible at http://127.0.0.1:7860 by default.\n", + "Answer: To run a Gradio Blocks app in reload mode from a Python IDE, use the terminal and type in: \"gradio app.py\" to launch the demo.\n", "True answer: Run `gradio run.py` in the terminal.\n" ] }, @@ -2228,18 +5191,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How to install Hugging Face Unity API in Unity project'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"To install the Hugging Face Unity API in your Unity project, follow these steps: \\n1. Open your Unity project.\\n2. Go to 'Window' -> 'Package Manager'.\\n3. Click '+' and select 'Add Package from git URL'.\\n4. Enter the URL: 'https://github.com/huggingface/unity-api.git'.\\n5. Once installed, the Unity API wizard should appear. If not, go to 'Window' -> 'Hugging Face API Wizard'.\\n6. Enter your API key, which can be created in your Hugging Face account settings.\\n7. Test the API key using the 'Test API key' option in the wizard.\\n8. Optionally, change the model endpoints to select a different model to use.\\n9. Configure any advanced settings as desired.\\n10. Install examples to see how to use the API.\"}\u001b[0m\n", - " 80%|████████ | 52/65 [12:36<03:42, 17.14s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the pretraining objective of the Wav2Vec2 context network?\n", - "\u001b[0m\n" + " 80%|████████ | 52/65 [02:03<00:40, 3.08s/it]" ] }, { @@ -2250,16 +5202,12 @@ "Question: How can you install the Hugging Face Unity API in your Unity project?\n", "\n", "Answer: To install the Hugging Face Unity API in your Unity project, follow these steps: \n", - "1. Open your Unity project.\n", - "2. Go to 'Window' -> 'Package Manager'.\n", - "3. Click '+' and select 'Add Package from git URL'.\n", - "4. Enter the URL: 'https://github.com/huggingface/unity-api.git'.\n", - "5. Once installed, the Unity API wizard should appear. If not, go to 'Window' -> 'Hugging Face API Wizard'.\n", - "6. Enter your API key, which can be created in your Hugging Face account settings.\n", - "7. Test the API key using the 'Test API key' option in the wizard.\n", - "8. Optionally, change the model endpoints to select a different model to use.\n", - "9. Configure any advanced settings as desired.\n", - "10. Install examples to see how to use the API.\n", + "\n", + "1. Open your Unity project. \n", + "2. Navigate to 'Window' and select 'Package Manager'. \n", + "3. Click the '+' button and choose the option 'Add Package from git URL'. \n", + "4. In the field provided, enter 'https://github.com/huggingface/unity-api.git'. \n", + "5. Once installed, the Unity API wizard should open. If not,\n", "True answer: To install the Hugging Face Unity API in your Unity project, go to `Window` -> `Package Manager`, click `+` and select `Add Package from git URL`, then enter `https://github.com/huggingface/unity-api.git`.\n" ] }, @@ -2267,18 +5215,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the pretraining objective of the Wav2Vec2 context network?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The pretraining objective of the Wav2Vec2 context network is a contrastive task. The model predicts a true quantized speech representation, from a set of false ones, using masked prediction. By doing so, the model finds the most similar context vector, and the quantized speech unit is the target label.'}\u001b[0m\n", - " 82%|████████▏ | 53/65 [12:48<03:06, 15.51s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", - "\u001b[0m\n" + " 82%|████████▏ | 53/65 [02:05<00:33, 2.80s/it]" ] }, { @@ -2288,7 +5225,7 @@ "=======================================================\n", "Question: What is the pretraining objective of the Wav2Vec2 context network?\n", "\n", - "Answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task. The model predicts a true quantized speech representation, from a set of false ones, using masked prediction. By doing so, the model finds the most similar context vector, and the quantized speech unit is the target label.\n", + "Answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task, which involves predicting the true quantized speech representation of a masked prediction.\n", "True answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task where the model has to predict the true quantized speech representation of the masked prediction from a set of false ones.\n" ] }, @@ -2296,18 +5233,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Default checkpoint for sentiment analysis pipeline in Transformers library'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english' (source document: 0).\"}\u001b[0m\n", - " 83%|████████▎ | 54/65 [13:02<02:45, 15.08s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", - "\u001b[0m\n" + " 83%|████████▎ | 54/65 [02:08<00:31, 2.82s/it]" ] }, { @@ -2317,7 +5243,7 @@ "=======================================================\n", "Question: What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", "\n", - "Answer: The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english' (source document: 0).\n", + "Answer: The default checkpoint used by the sentiment analysis pipeline in the Transformers library is distilbert-base-uncased-finetuned-sst-2-english. (Source document: 0)\n", "True answer: distilbert base uncased finetuned sst2 english\n" ] }, @@ -2325,18 +5251,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': \"What is the purpose of the notebook 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'?\"}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The purpose of the notebook is to show how to use DeepSpeed to pre-train and fine-tune the 1.6 billion parameter GPT2-XL model for causal language modeling on Habana Gaudi. This is demonstrated using a specific notebook titled 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'.\"}\u001b[0m\n", - " 85%|████████▍ | 55/65 [13:18<02:34, 15.48s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What command line module does PyTorch provide to run a script on multiple GPUs?\n", - "\u001b[0m\n" + " 85%|████████▍ | 55/65 [02:14<00:38, 3.80s/it]" ] }, { @@ -2346,7 +5261,7 @@ "=======================================================\n", "Question: What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", "\n", - "Answer: The purpose of the notebook is to show how to use DeepSpeed to pre-train and fine-tune the 1.6 billion parameter GPT2-XL model for causal language modeling on Habana Gaudi. This is demonstrated using a specific notebook titled 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'.\n", + "Answer: The purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\" (Documents 0,1) is to demonstrate how to use DeepSpeed for pre-training and fine-tuning large-scale language models, specifically the 1.6B-parameter GPT2-XL, on Habana Gaudi hardware. The notebook provides a practical guide for utilizing DeepSpeed and Habana Gaudi to accelerate and optimize the training of models with a large number of parameters\n", "True answer: To show how to use DeepSpeed to pre-train/fine-tune the 1.6B-parameter GPT2-XL for causal language modeling on Habana Gaudi.\n" ] }, @@ -2354,19 +5269,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'PyTorch command line module to run a script on multiple GPUs'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How many nodes can you use with torchrun?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'PyTorch provides the command-line module `torchrun` to run a script on multiple GPUs. You can specify the number of GPUs using the argument `--nproc_per_node=integer`.'}\u001b[0m\n", - " 86%|████████▌ | 56/65 [13:35<02:21, 15.74s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", - "\u001b[0m\n" + " 86%|████████▌ | 56/65 [02:15<00:27, 3.01s/it]" ] }, { @@ -2376,7 +5279,7 @@ "=======================================================\n", "Question: What command line module does PyTorch provide to run a script on multiple GPUs?\n", "\n", - "Answer: PyTorch provides the command-line module `torchrun` to run a script on multiple GPUs. You can specify the number of GPUs using the argument `--nproc_per_node=integer`.\n", + "Answer: 'CUDA_VISIBLE_DEVICES` is the command line module that PyTorch provides to run a script on multiple GPUs.\n", "True answer: torchrun\n" ] }, @@ -2384,19 +5287,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Most popular vision transformer models on Hugging Face model hub'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'vision transformer model image classification'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The most popular vision transformer model on the Hugging Face Model Hub for image classification is `google/vit-base-patch16-224`.'}\u001b[0m\n", - " 88%|████████▊ | 57/65 [13:52<02:10, 16.30s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the command to upload an ESPnet model to a Hugging Face repository?\n", - "\u001b[0m\n" + " 88%|████████▊ | 57/65 [02:20<00:27, 3.46s/it]" ] }, { @@ -2406,7 +5297,7 @@ "=======================================================\n", "Question: What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", "\n", - "Answer: The most popular vision transformer model on the Hugging Face Model Hub for image classification is `google/vit-base-patch16-224`.\n", + "Answer: The most popular vision transformer model for image classification on the Hugging Face Model Hub is 'google/vit-base-patch16-224'.\n", "True answer: google/vit-base-patch16-224\n" ] }, @@ -2414,18 +5305,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How to upload an ESPnet model to a Hugging Face repository?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To upload an ESPnet model to a Hugging Face repository, ensure you have an account and are logged in. Then, use the `run.sh` script with the appropriate arguments, specifying the stage, whether to skip the upload, and the name of your repository. Here is an example command: \\nbash\\n./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\\n \\nAlternatively, you can use the Hugging Face CLI with the `push_to_hub` argument, specifying the model directory and repository name.'}\u001b[0m\n", - " 89%|████████▉ | 58/65 [14:12<02:00, 17.20s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", - "\u001b[0m\n" + " 89%|████████▉ | 58/65 [02:24<00:26, 3.79s/it]" ] }, { @@ -2435,11 +5315,11 @@ "=======================================================\n", "Question: What is the command to upload an ESPnet model to a Hugging Face repository?\n", "\n", - "Answer: To upload an ESPnet model to a Hugging Face repository, ensure you have an account and are logged in. Then, use the `run.sh` script with the appropriate arguments, specifying the stage, whether to skip the upload, and the name of your repository. Here is an example command: \n", - "bash\n", + "Answer: The command to upload an ESPnet model to a Hugging Face repository is:\n", + "```bash\n", "./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n", - " \n", - "Alternatively, you can use the Hugging Face CLI with the `push_to_hub` argument, specifying the model directory and repository name.\n", + "```\n", + "where `username` and `model_repo` should be replaced with the desired username and model repository name, respectively.\n", "True answer: ./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n" ] }, @@ -2447,18 +5327,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'A `requirements.txt` file should be added to a model repository to install custom Python dependencies for Inference Endpoints.'}\u001b[0m\n", - " 91%|█████████ | 59/65 [14:25<01:36, 16.11s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", - "\u001b[0m\n" + " 91%|█████████ | 59/65 [02:30<00:26, 4.35s/it]" ] }, { @@ -2476,18 +5345,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Between 3 and 5 images are needed to teach new concepts to Stable Diffusion using Textual Inversion.'}\u001b[0m\n", - " 92%|█████████▏| 60/65 [14:41<01:19, 15.99s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", - "\u001b[0m\n" + " 92%|█████████▏| 60/65 [02:32<00:17, 3.52s/it]" ] }, { @@ -2497,7 +5355,7 @@ "=======================================================\n", "Question: How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", "\n", - "Answer: Between 3 and 5 images are needed to teach new concepts to Stable Diffusion using Textual Inversion.\n", + "Answer: To teach new concepts to Stable Diffusion using Textual Inversion, you need 3 to 5 sample images.\n", "True answer: 3-5 images\n" ] }, @@ -2505,18 +5363,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Transformers version 4.18.0 model checkpoint automatic sharding'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'In Transformers version 4.18.0, model checkpoints larger than 10GB are automatically sharded. [Source Document 0]'}\u001b[0m\n", - " 94%|█████████▍| 61/65 [14:51<00:56, 14.07s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", - "\u001b[0m\n" + " 94%|█████████▍| 61/65 [02:34<00:13, 3.27s/it]" ] }, { @@ -2526,7 +5373,7 @@ "=======================================================\n", "Question: What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", "\n", - "Answer: In Transformers version 4.18.0, model checkpoints larger than 10GB are automatically sharded. [Source Document 0]\n", + "Answer: Since version 4.18.0, model checkpoints that end up taking more than 10GB of space are automatically sharded into smaller pieces.\n", "True answer: 10GB\n" ] }, @@ -2534,18 +5381,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Weights and Biases (W&B) is a tool that allows data scientists and machine learning scientists to track their experiments at every stage, from training to production. It provides a customisable dashboard to visualise metrics and samples, helping users to optimise their machine learning workflows.'}\u001b[0m\n", - " 95%|█████████▌| 62/65 [15:04<00:41, 13.97s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", - "\u001b[0m\n" + " 95%|█████████▌| 62/65 [02:36<00:08, 2.80s/it]" ] }, { @@ -2555,7 +5391,7 @@ "=======================================================\n", "Question: What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", "\n", - "Answer: Weights and Biases (W&B) is a tool that allows data scientists and machine learning scientists to track their experiments at every stage, from training to production. It provides a customisable dashboard to visualise metrics and samples, helping users to optimise their machine learning workflows.\n", + "Answer: The purpose of Weights and Biases (W&B) is to enable data scientists and machine learning scientists to track their machine learning experiments at every stage, from training to production.\n", "True answer: To track their machine learning experiments at every stage, from training to production.\n" ] }, @@ -2563,18 +5399,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Hugging Face open-source library Transformer acceleration'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum (also referred to as '🤗 Optimum' and 'Optimum Intel'). This information is derived from multiple documents in the knowledge base.\"}\u001b[0m\n", - " 97%|█████████▋| 63/65 [15:12<00:24, 12.18s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What parameter is used to ensure that elements in a row have the same height in Gradio?\n", - "\u001b[0m\n" + " 97%|█████████▋| 63/65 [02:38<00:05, 2.68s/it]" ] }, { @@ -2584,7 +5409,7 @@ "=======================================================\n", "Question: What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", "\n", - "Answer: The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum (also referred to as '🤗 Optimum' and 'Optimum Intel'). This information is derived from multiple documents in the knowledge base.\n", + "Answer: The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum.\n", "True answer: Optimum\n" ] }, @@ -2592,18 +5417,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'What parameter is used to ensure that elements in a row have the same height in Gradio?'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The `equal_height` parameter is used to ensure that elements in a row have the same height in Gradio.'}\u001b[0m\n", - " 98%|█████████▊| 64/65 [15:24<00:12, 12.17s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "Provide the number of the source document when relevant.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "\n", - "Question:\n", - "What is the command to install the latest version of Optimum with OpenVINO support?\n", - "\u001b[0m\n" + " 98%|█████████▊| 64/65 [02:39<00:02, 2.21s/it]" ] }, { @@ -2613,7 +5427,7 @@ "=======================================================\n", "Question: What parameter is used to ensure that elements in a row have the same height in Gradio?\n", "\n", - "Answer: The `equal_height` parameter is used to ensure that elements in a row have the same height in Gradio.\n", + "Answer: `equal_height`\n", "True answer: equal_height\n" ] }, @@ -2621,9 +5435,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Optimum installation command with OpenVINO support'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The command to install the latest version of Optimum with OpenVINO support is:\\n\\nbash\\npip install --upgrade-strategy eager optimum['openvino']\\n\\n\\nThis command ensures that the latest version of Optimum is installed with support for OpenVINO, enabling inference on a variety of Intel processors.\"}\u001b[0m\n", - "100%|██████████| 65/65 [15:49<00:00, 14.60s/it]" + "100%|██████████| 65/65 [02:42<00:00, 2.51s/it]" ] }, { @@ -2634,12 +5446,9 @@ "Question: What is the command to install the latest version of Optimum with OpenVINO support?\n", "\n", "Answer: The command to install the latest version of Optimum with OpenVINO support is:\n", - "\n", - "bash\n", - "pip install --upgrade-strategy eager optimum['openvino']\n", - "\n", - "\n", - "This command ensures that the latest version of Optimum is installed with support for OpenVINO, enabling inference on a variety of Intel processors.\n", + "```bash\n", + "pip install --upgrade-strategy eager optimum[\"openvino,nncf\"]\n", + "```\n", "True answer: pip install --upgrade-strategy eager optimum[\"openvino\"]\n" ] }, @@ -2652,127 +5461,184 @@ } ], "source": [ - "from transformers.agents.agents import AgentError\n", - "\n", - "outputs = []\n", + "from huggingface_hub import InferenceClient\n", "\n", + "reader_llm = InferenceClient(\"CohereForAI/c4ai-command-r-plus\")\n", "\n", - "def serialize_agent_error(obj):\n", - " if isinstance(obj, AgentError):\n", - " return {\"error_type\": obj.__class__.__name__, \"message\": obj.message}\n", - " else:\n", - " return str(obj)\n", - "\n", + "outputs_standard_rag = []\n", "\n", "for example in tqdm(eval_dataset):\n", " question = example[\"question\"]\n", - " if question in [el[\"question\"] for el in outputs]:\n", - " continue\n", + " context = retriever_tool(question)\n", "\n", - " enhanced_question = f\"\"\"Using the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", + " prompt = f\"\"\"Given the question and supporting documents below, give a comprehensive answer to the question.\n", "Respond only to the question asked, response should be concise and relevant to the question.\n", "Provide the number of the source document when relevant.\n", "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", "\n", "Question:\n", - "{question}\"\"\"\n", - " answer = agent.run(enhanced_question)\n", + "{question}\n", + "\n", + "{context}\n", + "\"\"\"\n", + " messages = [{\"role\": \"user\", \"content\": prompt}]\n", + " answer = reader_llm.chat_completion(messages).choices[0].message.content\n", + "\n", " print(\"=======================================================\")\n", " print(f\"Question: {question}\")\n", " print(f\"Answer: {answer}\")\n", " print(f'True answer: {example[\"answer\"]}')\n", "\n", - " result = {\n", + " results_agentic = {\n", " \"question\": question,\n", " \"true_answer\": example[\"answer\"],\n", " \"source_doc\": example[\"source_doc\"],\n", " \"generated_answer\": answer,\n", " }\n", - " outputs.append(result)\n", + " outputs_standard_rag.append(results_agentic)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The evaluation prompt follows some of the best principles shown in [our llm_judge cookbook](llm_judge): it follows a small integer Likert scale, has clear criteria, and a description for each score." + ] + }, + { + "cell_type": "code", + "execution_count": 68, + "metadata": {}, + "outputs": [], + "source": [ + "EVALUATION_PROMPT = \"\"\"You are a fair evaluator language model.\n", + "\n", + "You will be given an instruction, a response to evaluate, a reference answer that gets a score of 3, and a score rubric representing a evaluation criteria are given.\n", + "1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n", + "2. After writing a feedback, write a score that is an integer between 1 and 3. You should refer to the score rubric.\n", + "3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 3}}\\\"\n", + "4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n", + "5. Do not score conciseness: a correct answer that covers the question should receive max score, even if it contains additional useless information.\n", + "\n", + "The instruction to evaluate:\n", + "{instruction}\n", + "\n", + "Response to evaluate:\n", + "{response}\n", + "\n", + "Reference Answer (Score 3):\n", + "{reference_answer}\n", + "\n", + "Score Rubrics:\n", + "[Is the response complete, accurate, and factual based on the reference answer?]\n", + "Score 1: The response is completely incomplete, inaccurate, and/or not factual.\n", + "Score 2: The response is somewhat complete, accurate, and/or factual.\n", + "Score 3: The response is completely complete, accurate, and/or factual.\n", + "\n", + "Feedback:\"\"\"" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "metadata": {}, + "outputs": [], + "source": [ + "from huggingface_hub import InferenceClient\n", "\n", - " with open(\"outputs.jsonl\", \"w\") as f:\n", - " for d in outputs:\n", - " json.dump(d, f, default=serialize_agent_error)\n", - " f.write(\"\\n\") # add a newline for JSONL format" + "evaluation_client = InferenceClient(\"meta-llama/Meta-Llama-3-70B-Instruct\")" ] }, { "cell_type": "code", - "execution_count": 81, + "execution_count": 70, "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 65/65 [02:24<00:00, 2.23s/it]\n" + ] + }, { "name": "stdout", "output_type": "stream", "text": [ - "Loaded 65 answers.\n" + "Average score for agentic RAG: 78.5%\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 65/65 [05:02<00:00, 4.65s/it]\n" + "100%|██████████| 65/65 [02:17<00:00, 2.12s/it]" ] - } - ], - "source": [ - "for experiment in tqdm(outputs):\n", - " eval_prompt = EVALUATION_PROMPT.format(\n", - " instruction=experiment[\"question\"],\n", - " response=experiment[\"generated_answer\"],\n", - " reference_answer=experiment[\"true_answer\"],\n", - " )\n", - " messages = [\n", - " {\"role\": \"system\", \"content\": \"You are a fair evaluator language model.\"},\n", - " {\"role\": \"user\", \"content\": eval_prompt},\n", - " ]\n", - "\n", - " eval_result = (\n", - " client.chat.completions.create(\n", - " model=\"gpt-4-1106-preview\", messages=messages, temperature=0.5\n", - " )\n", - " .choices[0]\n", - " .message.content\n", - " )\n", - "\n", - " feedback, score = [item.strip() for item in eval_result.split(\"[RESULT]\")]\n", - " experiment[f\"eval_score_LLM_judge\"] = score\n", - " experiment[f\"eval_feedback_LLM_judge\"] = feedback" - ] - }, - { - "cell_type": "code", - "execution_count": 82, - "metadata": {}, - "outputs": [ + }, { "name": "stdout", "output_type": "stream", "text": [ - "Average score: 0.850\n" + "Average score for standard RAG: 70.0%\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" ] } ], "source": [ "import pandas as pd\n", "\n", - "result = pd.DataFrame.from_dict(outputs)\n", - "result = result.loc[~result[\"generated_answer\"].str.contains(\"Error\")]\n", - "result[\"eval_score_LLM_judge_int\"] = result[\"eval_score_LLM_judge\"].apply(\n", - " lambda x: int(x)\n", - ")\n", - "result[\"eval_score_LLM_judge_int\"] = (result[\"eval_score_LLM_judge_int\"] - 1) / 4\n", + "for type, outputs in [\n", + " (\"agentic\", outputs_agentic_rag),\n", + " (\"standard\", outputs_standard_rag),\n", + "]:\n", + " for experiment in tqdm(outputs):\n", + " eval_prompt = EVALUATION_PROMPT.format(\n", + " instruction=experiment[\"question\"],\n", + " response=experiment[\"generated_answer\"],\n", + " reference_answer=experiment[\"true_answer\"],\n", + " )\n", + " messages = [\n", + " {\"role\": \"system\", \"content\": \"You are a fair evaluator language model.\"},\n", + " {\"role\": \"user\", \"content\": eval_prompt},\n", + " ]\n", + "\n", + " eval_result = evaluation_client.text_generation(\n", + " eval_prompt, max_new_tokens=1000\n", + " )\n", + " try:\n", + " feedback, score = [item.strip() for item in eval_result.split(\"[RESULT]\")]\n", + " experiment[\"eval_score_LLM_judge\"] = score\n", + " experiment[\"eval_feedback_LLM_judge\"] = feedback\n", + " except:\n", + " print(f\"Parsing failed - output was: {eval_result}\")\n", "\n", - "print(f\"Average score: {result['eval_score_LLM_judge_int'].mean():.3f}\")" + " results = pd.DataFrame.from_dict(outputs)\n", + " results = results.loc[~results[\"generated_answer\"].str.contains(\"Error\")]\n", + " results[\"eval_score_LLM_judge_int\"] = (\n", + " results[\"eval_score_LLM_judge\"].fillna(1).apply(lambda x: int(x))\n", + " )\n", + " results[\"eval_score_LLM_judge_int\"] = (results[\"eval_score_LLM_judge_int\"] - 1) / 2\n", + "\n", + " print(\n", + " f\"Average score for {type} RAG: {results['eval_score_LLM_judge_int'].mean()*100:.1f}%\"\n", + " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "So we reach 85% score with this simple setup!" + "**Let us recap: the Agent setup improves scores by 8.5% compared to a standard RAG!**\n", + "\n", + "This is a great improvement, with a very simple setup 🚀\n", + "\n", + "(For a baseline, using Llama-3-70B without the knowledge base got 36%)" ] } ], From e563a666af4c43a9ddff741ff27d701dd1f0fa3a Mon Sep 17 00:00:00 2001 From: Aymeric Date: Fri, 5 Jul 2024 18:21:12 +0200 Subject: [PATCH 3/3] Remove cell outputs to get rid of bug --- notebooks/en/agent_rag.ipynb | 5077 +--------------------------------- 1 file changed, 119 insertions(+), 4958 deletions(-) diff --git a/notebooks/en/agent_rag.ipynb b/notebooks/en/agent_rag.ipynb index 86c1fabc..78c571db 100644 --- a/notebooks/en/agent_rag.ipynb +++ b/notebooks/en/agent_rag.ipynb @@ -30,11 +30,11 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "!pip install pandas langchain sentence-transformers faiss-cpu \"transformers[agents]\"" + "!pip install pandas langchain langchain-community sentence-transformers faiss-cpu \"transformers[agents]\"" ] }, { @@ -48,7 +48,16 @@ "cell_type": "code", "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/aymeric/Documents/Code/cookbook/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + } + ], "source": [ "import datasets\n", "\n", @@ -67,7 +76,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "metadata": {}, "outputs": [ { @@ -81,7 +90,9 @@ "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 2647/2647 [00:34<00:00, 75.69it/s] \n" + "100%|██████████| 2647/2647 [00:34<00:00, 76.04it/s] \n", + "/Users/aymeric/Documents/Code/cookbook/.venv/lib/python3.12/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 0.3.0. An updated version of the class exists in the langchain-huggingface package and should be used instead. To use it run `pip install -U langchain-huggingface` and import as `from langchain_huggingface import HuggingFaceEmbeddings`.\n", + " warn_deprecated(\n" ] }, { @@ -141,16 +152,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now that we have the database ready, let’s build a RAG system that answers user queries based on it!\n", + "Now the database is ready: let’s build our agentic RAG system!\n", "\n", - "We want our system to select only from the most relevant sources of information, depending on the query." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "👉 Let us build our RAG system as an agent that will be free to choose its query and can iterate on the retrieval results!" + "👉 We only need a `RetrieverTool` that our agent can leverage to retrieve information from the knowledge base." ] }, { @@ -159,7 +163,6 @@ "metadata": {}, "outputs": [], "source": [ - "import json\n", "from transformers.agents import Tool\n", "from langchain_core.vectorstores import VectorStore\n", "\n", @@ -214,7 +217,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 7, "metadata": {}, "outputs": [], "source": [ @@ -239,7 +242,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 8, "metadata": {}, "outputs": [ { @@ -385,16 +388,15 @@ "===== Calling LLM with this last message: =====\n", "{'role': , 'content': 'Task: How can I push a model to the Hub?'}\n", "\u001b[38;20m===== Output message of the LLM: =====\u001b[0m\n", - "\u001b[38;20mThought: I will try to retrieve documents related to pushing a model to the hub with the retriever tool and then provide the answer based on this information.\n", + "\u001b[38;20mThought: I can use the \"retriever\" tool to find documents relevant to the question, \"How can I push a model to the Hub?\" I will then read through the retrieved documents to find the relevant information and provide an answer to the question.\n", + "\n", "Action: ```json\n", "{\n", " \"action\": \"retriever\",\n", " \"action_input\": {\n", " \"query\": \"How can I push a model to the Hub?\"\n", " }\n", - "}\n", - "\n", - "```\u001b[0m\n", + "}\u001b[0m\n", "\u001b[38;20m===== Extracting action =====\u001b[0m\n", "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'How can I push a model to the Hub?'}\u001b[0m\n", "Retrieved documents:\n", @@ -461,20 +463,19 @@ "}\n", "\u001b[38;20m===== New step =====\u001b[0m\n", "===== Calling LLM with this last message: =====\n", - "{'role': , 'content': '[OUTPUT OF STEP 0] Observation:\\nRetrieved documents:\\n===== Document 0 =====\\n# Step 7. Push everything to the Hub\\n api.upload_folder(\\n repo_id=repo_id,\\n folder_path=repo_local_path,\\n path_in_repo=\".\",\\n )\\n\\n print(\"Your model is pushed to the Hub. You can view your model here: \", repo_url)\\n```\\n\\n### .\\n\\nBy using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.===== Document 1 =====\\n```py\\n>>> trainer.push_to_hub()\\n```\\n\\n\\nShare a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:\\n\\n- An output directory for your model.\\n- A tokenizer.\\n- The `hub_model_id`, which is your Hub username and model name.\\n\\n```py\\n>>> from transformers import PushToHubCallback\\n\\n>>> push_to_hub_callback = PushToHubCallback(\\n... output_dir=\"./your_model_save_path\", tokenizer=tokenizer, hub_model_id=\"your-username/my-awesome-model\"\\n... )\\n```===== Document 2 =====\\nLet\\'s pretend we\\'ve now fine-tuned the model. The next step would be to push it to the Hub! We can do this with the `timm.models.hub.push_to_hf_hub` function.\\n\\n```py\\n>>> model_cfg = dict(labels=[\\'a\\', \\'b\\', \\'c\\', \\'d\\'])\\n>>> timm.models.hub.push_to_hf_hub(model, \\'resnet18-random\\', model_config=model_cfg)\\n```\\n\\nRunning the above would push the model to `/resnet18-random` on the Hub. You can now share this model with your friends, or use it in your own code!\\n\\n## Loading a Model===== Document 3 =====\\nprocessor.push_to_hub(hub_model_id)\\ntrainer.push_to_hub(**kwargs)\\n```\\n\\n# 4. Inference\\n\\nNow comes the exciting part, using our fine-tuned model! In this section, we\\'ll show how you can load your model from the hub and use it for inference.===== Document 4 =====\\n--push_to_hub\\n```===== Document 5 =====\\n. The second way to upload a model, though, is to call model.push_to_hub(). So this is more of a once-off method - it\\'s not called regularly during training. You can just call this manually whenever you want to upload a model to the hub. So we recommend running this after the end of training, just to make sure that you have a commit message just to guarantee that this was the final version of the model at the end of training. And it just makes sure that you\\'re working with the definitive end-of-training model and not accidentally using a model that\\'s from a checkpoint somewhere along the way===== Document 6 =====\\nFinally, if you want, you can push your model up to the hub. Here, we\\'ll push it up if you specified `push_to_hub=True` in the training configuration. Note that in order to push to hub, you\\'ll have to have git-lfs installed and be logged into your Hugging Face account (which can be done via `huggingface-cli login`).\\n\\n```python\\nkwargs = {\\n \"finetuned_from\": model.config._name_or_path,\\n \"tasks\": \"image-classification\",\\n \"dataset\": \\'beans\\',\\n \"tags\": [\\'image-classification\\'],\\n}'}\n", + "{'role': , 'content': 'Observation: Retrieved documents:\\n===== Document 0 =====\\n# Step 7. Push everything to the Hub\\n api.upload_folder(\\n repo_id=repo_id,\\n folder_path=repo_local_path,\\n path_in_repo=\".\",\\n )\\n\\n print(\"Your model is pushed to the Hub. You can view your model here: \", repo_url)\\n```\\n\\n### .\\n\\nBy using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.===== Document 1 =====\\n```py\\n>>> trainer.push_to_hub()\\n```\\n\\n\\nShare a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:\\n\\n- An output directory for your model.\\n- A tokenizer.\\n- The `hub_model_id`, which is your Hub username and model name.\\n\\n```py\\n>>> from transformers import PushToHubCallback\\n\\n>>> push_to_hub_callback = PushToHubCallback(\\n... output_dir=\"./your_model_save_path\", tokenizer=tokenizer, hub_model_id=\"your-username/my-awesome-model\"\\n... )\\n```===== Document 2 =====\\nLet\\'s pretend we\\'ve now fine-tuned the model. The next step would be to push it to the Hub! We can do this with the `timm.models.hub.push_to_hf_hub` function.\\n\\n```py\\n>>> model_cfg = dict(labels=[\\'a\\', \\'b\\', \\'c\\', \\'d\\'])\\n>>> timm.models.hub.push_to_hf_hub(model, \\'resnet18-random\\', model_config=model_cfg)\\n```\\n\\nRunning the above would push the model to `/resnet18-random` on the Hub. You can now share this model with your friends, or use it in your own code!\\n\\n## Loading a Model===== Document 3 =====\\nprocessor.push_to_hub(hub_model_id)\\ntrainer.push_to_hub(**kwargs)\\n```\\n\\n# 4. Inference\\n\\nNow comes the exciting part, using our fine-tuned model! In this section, we\\'ll show how you can load your model from the hub and use it for inference.===== Document 4 =====\\n--push_to_hub\\n```===== Document 5 =====\\n. The second way to upload a model, though, is to call model.push_to_hub(). So this is more of a once-off method - it\\'s not called regularly during training. You can just call this manually whenever you want to upload a model to the hub. So we recommend running this after the end of training, just to make sure that you have a commit message just to guarantee that this was the final version of the model at the end of training. And it just makes sure that you\\'re working with the definitive end-of-training model and not accidentally using a model that\\'s from a checkpoint somewhere along the way===== Document 6 =====\\nFinally, if you want, you can push your model up to the hub. Here, we\\'ll push it up if you specified `push_to_hub=True` in the training configuration. Note that in order to push to hub, you\\'ll have to have git-lfs installed and be logged into your Hugging Face account (which can be done via `huggingface-cli login`).\\n\\n```python\\nkwargs = {\\n \"finetuned_from\": model.config._name_or_path,\\n \"tasks\": \"image-classification\",\\n \"dataset\": \\'beans\\',\\n \"tags\": [\\'image-classification\\'],\\n}'}\n", "\u001b[38;20m===== Output message of the LLM: =====\u001b[0m\n", - "\u001b[38;20mThought: I will now provide the answer.\n", + "\u001b[38;20mThought: I have found multiple ways to push a model to the Hub. In order to answer comprehensively, I will provide each code snippet that describes how to push a model to the Hub.\n", + "\n", "Action: ```json\n", "{\n", " \"action\": \"final_answer\",\n", " \"action_input\": {\n", - " \"answer\": \"To push a model to the Hub, you can make use of the push_to_hub API provided by Hugging Face. An example usage would be: model.push_to_hub() or trainer.push_to_hub(). Additionally, a function called timm.models.hub.push_to_hf_hub is also available to push models to the Hub.\"\n", + " \"answer\": \"There are multiple ways to push a model to the Hub. Here are a few examples using different libraries and functions:\\n\\nUsing the `api`:\\n```python\\napi.upload_folder(\\n repo_id=repo_id,\\n folder_path=repo_local_path,\\n path_in_repo=\\\".\\\",\\n)\\n\\nprint(\\\"Your model is pushed to the Hub. You can view your model here:\\\", repo_url)\\n```\\n\\nWith Transformers:\\n```python\\nfrom transformers import PushToHubCallback\\n\\n# Initialize the callback with the output directory,\\ntokenizer, and your Hub username and model name\\npush_to_hub_callback = PushToHubCallback(\\n output_dir=\\\"./your_model_save_path\\\",\\n tokenizer=tokenizer,\\n hub_model_id=\\\"your-username/my-awesome-model\\\"\\n)\\n\\n# Assuming `trainer` is your Trainer object\\ntrainer.add_callback(push_to_hub_callback)\\n```\\n\\nUsing `timm`:\\n```python\\nfrom timm.models.hub import push_to_hf_hub\\n\\n# Assuming `model` is your fine-tuned model\\nmodel_cfg = {\\\"labels\\\": [\\\"a\\\", \\\"b\\\", \\\"c\\\", \\\"d\\\"]}\\npush_to_hf_hub(model, 'resnet18-random', model_config=model_cfg)\\n```\\n\\nFor computer vision models, you can also use `push_to_hub`:\\n```python\\nprocessor.push_to_hub(hub_model_id)\\ntrainer.push_to_hub(**kwargs)\\n```\\n\\nYou can also manually push a model with `model.push_to_hub()`:\\n```python\\nmodel.push_to_hub()\\n```\\n\\nAdditionally, you can opt to push your model to the Hub at the end of training by specifying `push_to_hub=True` in the training configuration. Don't forget to have git-lfs installed and be logged into your Hugging Face account.\"\n", " }\n", - "}\n", - "\n", - "```\u001b[0m\n", + "}\u001b[0m\n", "\u001b[38;20m===== Extracting action =====\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To push a model to the Hub, you can make use of the push_to_hub API provided by Hugging Face. An example usage would be: model.push_to_hub() or trainer.push_to_hub(). Additionally, a function called timm.models.hub.push_to_hf_hub is also available to push models to the Hub.'}\u001b[0m\n" + "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"There are multiple ways to push a model to the Hub. Here are a few examples using different libraries and functions:\\n\\nUsing the `api`:\\npython\\napi.upload_folder(\\n repo_id=repo_id,\\n folder_path=repo_local_path,\\n path_in_repo='.',\\n)\\n\\nprint('Your model is pushed to the Hub. You can view your model here:', repo_url)\\n\\n\\nWith Transformers:\\npython\\nfrom transformers import PushToHubCallback\\n\\n# Initialize the callback with the output directory,\\ntokenizer, and your Hub username and model name\\npush_to_hub_callback = PushToHubCallback(\\n output_dir='./your_model_save_path',\\n tokenizer=tokenizer,\\n hub_model_id='your-username/my-awesome-model'\\n)\\n\\n# Assuming `trainer` is your Trainer object\\ntrainer.add_callback(push_to_hub_callback)\\n\\n\\nUsing `timm`:\\npython\\nfrom timm.models.hub import push_to_hf_hub\\n\\n# Assuming `model` is your fine-tuned model\\nmodel_cfg = {'labels': ['a', 'b', 'c', 'd']}\\npush_to_hf_hub(model, 'resnet18-random', model_config=model_cfg)\\n\\n\\nFor computer vision models, you can also use `push_to_hub`:\\npython\\nprocessor.push_to_hub(hub_model_id)\\ntrainer.push_to_hub(**kwargs)\\n\\n\\nYou can also manually push a model with `model.push_to_hub()`:\\npython\\nmodel.push_to_hub()\\n\\n\\nAdditionally, you can opt to push your model to the Hub at the end of training by specifying `push_to_hub=True` in the training configuration. Don't forget to have git-lfs installed and be logged into your Hugging Face account.\"}\u001b[0m\n" ] }, { @@ -482,7 +483,56 @@ "output_type": "stream", "text": [ "Final output:\n", - "To push a model to the Hub, you can make use of the push_to_hub API provided by Hugging Face. An example usage would be: model.push_to_hub() or trainer.push_to_hub(). Additionally, a function called timm.models.hub.push_to_hf_hub is also available to push models to the Hub.\n" + "There are multiple ways to push a model to the Hub. Here are a few examples using different libraries and functions:\n", + "\n", + "Using the `api`:\n", + "python\n", + "api.upload_folder(\n", + " repo_id=repo_id,\n", + " folder_path=repo_local_path,\n", + " path_in_repo='.',\n", + ")\n", + "\n", + "print('Your model is pushed to the Hub. You can view your model here:', repo_url)\n", + "\n", + "\n", + "With Transformers:\n", + "python\n", + "from transformers import PushToHubCallback\n", + "\n", + "# Initialize the callback with the output directory,\n", + "tokenizer, and your Hub username and model name\n", + "push_to_hub_callback = PushToHubCallback(\n", + " output_dir='./your_model_save_path',\n", + " tokenizer=tokenizer,\n", + " hub_model_id='your-username/my-awesome-model'\n", + ")\n", + "\n", + "# Assuming `trainer` is your Trainer object\n", + "trainer.add_callback(push_to_hub_callback)\n", + "\n", + "\n", + "Using `timm`:\n", + "python\n", + "from timm.models.hub import push_to_hf_hub\n", + "\n", + "# Assuming `model` is your fine-tuned model\n", + "model_cfg = {'labels': ['a', 'b', 'c', 'd']}\n", + "push_to_hf_hub(model, 'resnet18-random', model_config=model_cfg)\n", + "\n", + "\n", + "For computer vision models, you can also use `push_to_hub`:\n", + "python\n", + "processor.push_to_hub(hub_model_id)\n", + "trainer.push_to_hub(**kwargs)\n", + "\n", + "\n", + "You can also manually push a model with `model.push_to_hub()`:\n", + "python\n", + "model.push_to_hub()\n", + "\n", + "\n", + "Additionally, you can opt to push your model to the Hub at the end of training by specifying `push_to_hub=True` in the training configuration. Don't forget to have git-lfs installed and be logged into your Hugging Face account.\n" ] } ], @@ -499,7 +549,7 @@ "source": [ "## Agentic RAG vs. standard RAG\n", "\n", - "Does the agent setup give a performant RAG system? Well, let's measure it.\n", + "Does the agent setup make a better RAG system? Well, let's comapre it to a standard RAG system using LLM Judge!\n", "\n", "We will use [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) for evaluation since it's one of the strongest OS models we tested for LLM judge use cases." ] @@ -533,4933 +583,44 @@ }, { "cell_type": "code", - "execution_count": 56, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 0%| | 0/65 [00:00 'Package Manager'\\n3. Click '+' and select 'Add Package from git URL'\\n4. Enter the URL: 'https://github.com/huggingface/unity-api.git'\\n5. Once installed, the Unity API wizard should pop up. If not, go to 'Window' -> 'Hugging Face API Wizard' for further setup.\"}\u001b[0m\n", - " 80%|████████ | 52/65 [19:24<06:37, 30.61s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the pretraining objective of the Wav2Vec2 context network?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How can you install the Hugging Face Unity API in your Unity project?\n", - "\n", - "Answer: To install the Hugging Face Unity API in your Unity project, follow these steps: \n", - "1. Open your Unity project\n", - "2. Go to 'Window' -> 'Package Manager'\n", - "3. Click '+' and select 'Add Package from git URL'\n", - "4. Enter the URL: 'https://github.com/huggingface/unity-api.git'\n", - "5. Once installed, the Unity API wizard should pop up. If not, go to 'Window' -> 'Hugging Face API Wizard' for further setup.\n", - "True answer: To install the Hugging Face Unity API in your Unity project, go to `Window` -> `Package Manager`, click `+` and select `Add Package from git URL`, then enter `https://github.com/huggingface/unity-api.git`.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Wav2Vec2 context network pretraining objective'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The pretraining objective of the Wav2Vec2 context network is a contrastive task, where the model predicts the true quantized speech representation from a masked prediction.'}\u001b[0m\n", - " 82%|████████▏ | 53/65 [19:46<05:36, 28.03s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the pretraining objective of the Wav2Vec2 context network?\n", - "\n", - "Answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task, where the model predicts the true quantized speech representation from a masked prediction.\n", - "True answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task where the model has to predict the true quantized speech representation of the masked prediction from a set of false ones.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'default checkpoint sentiment analysis pipeline Transformers library'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'distilbert-base-uncased-finetuned-sst-2-english checkpoint'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english'. This checkpoint is a distilled version of the larger BERT model, specifically tailored for sentiment analysis tasks. It has been fine-tuned on the Stanford Sentiment Treebank (SST-2) dataset, which consists of English sentences labeled with their sentiment polarity (positive or negative). The distilled nature of the model makes it more lightweight and faster to use compared to its larger counterparts, without sacrificing much performance.\"}\u001b[0m\n", - " 83%|████████▎ | 54/65 [20:04<04:34, 24.92s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", - "\n", - "Answer: The default checkpoint used by the sentiment analysis pipeline in the Transformers library is 'distilbert-base-uncased-finetuned-sst-2-english'. This checkpoint is a distilled version of the larger BERT model, specifically tailored for sentiment analysis tasks. It has been fine-tuned on the Stanford Sentiment Treebank (SST-2) dataset, which consists of English sentences labeled with their sentiment polarity (positive or negative). The distilled nature of the model makes it more lightweight and faster to use compared to its larger counterparts, without sacrificing much performance.\n", - "True answer: distilbert base uncased finetuned sst2 english\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of notebook How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Purpose of notebook How to use DeepSpeed to train models with billions of parameters on Habana Gaudi'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The notebook 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi' aims to showcase how to use DeepSpeed to pre-train and fine-tune the 1.6B-parameter GPT2-XL model for causal language modeling on Habana Gaudi, a training solution that delivers lower price/performance when compared to similar training solutions. The notebook demonstrates how to take advantage of the parallel computation capabilities of Habana Gaudi to efficiently train large language models.\"}\u001b[0m\n", - " 85%|████████▍ | 55/65 [20:30<04:14, 25.42s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What command line module does PyTorch provide to run a script on multiple GPUs?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", - "\n", - "Answer: The notebook 'How to use DeepSpeed to train models with billions of parameters on Habana Gaudi' aims to showcase how to use DeepSpeed to pre-train and fine-tune the 1.6B-parameter GPT2-XL model for causal language modeling on Habana Gaudi, a training solution that delivers lower price/performance when compared to similar training solutions. The notebook demonstrates how to take advantage of the parallel computation capabilities of Habana Gaudi to efficiently train large language models.\n", - "True answer: To show how to use DeepSpeed to pre-train/fine-tune the 1.6B-parameter GPT2-XL for causal language modeling on Habana Gaudi.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch command line module to run a script on multiple GPUs'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch command line multiple GPUs'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch command line module multiple GPUs'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'PyTorch multi-gpu command line'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[31;20mReached max iterations.\u001b[0m\n", - "NoneType: None\n", - " 86%|████████▌ | 56/65 [21:00<04:01, 26.87s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What command line module does PyTorch provide to run a script on multiple GPUs?\n", - "\n", - "Answer: Apologies for the delays and errors encountered in my attempts to answer your question. PyTorch provides the \"torch.distributed\" package and the \"torch.multiprocessing\" package to run a script on multiple GPUs. These packages enable you to distribute your training code across multiple GPUs or machines for faster processing. By utilizing these packages, you can take advantage of multiple GPUs to speed up your deep learning training workflows.\n", - "True answer: torchrun\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Most popular vision transformer image classification hugging face model hub'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'According to the retrieved documents, the most popular vision transformer model for image classification on the Hugging Face Model Hub is `google/vit-base-patch16-224`, which has been trained on ImageNet images with a resolution of 224x224 pixels.'}\u001b[0m\n", - " 88%|████████▊ | 57/65 [21:24<03:26, 25.86s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the command to upload an ESPnet model to a Hugging Face repository?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", - "\n", - "Answer: According to the retrieved documents, the most popular vision transformer model for image classification on the Hugging Face Model Hub is `google/vit-base-patch16-224`, which has been trained on ImageNet images with a resolution of 224x224 pixels.\n", - "True answer: google/vit-base-patch16-224\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'upload ESPnet model Hugging Face repository'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To upload an ESPnet model to a Hugging Face repository, you can use the command:\\n\\nbash\\n./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\\n\\n\\nReplace `username/model_repo` with your Hugging Face username and the desired repository name for your model.'}\u001b[0m\n", - " 89%|████████▉ | 58/65 [21:54<03:09, 27.08s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the command to upload an ESPnet model to a Hugging Face repository?\n", - "\n", - "Answer: To upload an ESPnet model to a Hugging Face repository, you can use the command:\n", - "\n", - "bash\n", - "./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n", - "\n", - "\n", - "Replace `username/model_repo` with your Hugging Face username and the desired repository name for your model.\n", - "True answer: ./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Python dependencies for Inference Endpoints'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Python dependencies for Inference Endpoints'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Custom Python dependencies Inference Endpoints models'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'To install custom Python dependencies for Inference Endpoints, a `requirements.txt` file should be added to the model repository.'}\u001b[0m\n", - " 91%|█████████ | 59/65 [22:16<02:33, 25.52s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", - "\n", - "Answer: To install custom Python dependencies for Inference Endpoints, a `requirements.txt` file should be added to the model repository.\n", - "True answer: requirements.txt\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Confirm number of images for teaching new concepts to Stable Diffusion with Textual Inversion'}\u001b[0m\n", - "\u001b[31;20mReached max iterations.\u001b[0m\n", - "NoneType: None\n", - " 92%|█████████▏| 60/65 [22:41<02:06, 25.33s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", - "\n", - "Answer: You need between 3 to 5 images to teach new concepts to Stable Diffusion using Textual Inversion.\n", - "True answer: 3-5 images\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'model checkpoint size limit before automatic sharding in transformers v4.18.0'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'automatic model checkpoint sharding size limit transformers v4.18.0'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'Starting from version 4.18.0, model checkpoints exceed the 10GB size limit are automatically sharded into smaller pieces. This is done to reduce memory usage during model loading.'}\u001b[0m\n", - " 94%|█████████▍| 61/65 [23:09<01:45, 26.35s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", - "\n", - "Answer: Starting from version 4.18.0, model checkpoints exceed the 10GB size limit are automatically sharded into smaller pieces. This is done to reduce memory usage during model loading.\n", - "True answer: 10GB\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'purpose of Weights and Biases'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of Weights and Biases for data and machine learning scientists'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose Weights and Biases ML scientists'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': {'type': 'text', 'description': 'Purpose of Weights and Biases for data scientists and ML scientists'}}\u001b[0m\n", - "\u001b[31;20mError in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\u001b[0m\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 499, in execute_tool_call\n", - " observation = self.toolbox.tools[tool_name](**arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/tools.py\", line 134, in __call__\n", - " outputs = self.forward(*args, **kwargs)\n", - " File \"/var/folders/6m/9b1tts6d5w960j80wbw9tx3m0000gn/T/ipykernel_14585/2141085945.py\", line 22, in forward\n", - " assert isinstance(query, str), \"Your search query must be a string\"\n", - "AssertionError: Your search query must be a string\n", - "\n", - "During handling of the above exception, another exception occurred:\n", - "\n", - "Traceback (most recent call last):\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 737, in direct_run\n", - " step_logs = self.step()\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 923, in step\n", - " observation = self.execute_tool_call(tool_name, arguments)\n", - " File \"/Users/aymeric/Documents/Code/original_transformers/transformers/src/transformers/agents/agents.py\", line 502, in execute_tool_call\n", - " raise AgentExecutionError(\n", - "transformers.agents.agents.AgentExecutionError: Error in tool call execution: Your search query must be a string\n", - "You should only use this tool with a correct input.\n", - "As a reminder, this tool's description is the following:\n", - "\n", - "- retriever: Using semantic similarity, retrieves some documents from the knowledge base that have the closest embeddings to the input query.\n", - " Takes inputs: {'query': {'type': 'text', 'description': 'The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.'}}\n", - "\u001b[31;20mReached max iterations.\u001b[0m\n", - "NoneType: None\n", - " 95%|█████████▌| 62/65 [23:38<01:21, 27.13s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", - "\n", - "Answer: Apologies, there seems to be an error with my tools that is preventing me from retrieving the relevant information to answer your question. I will need further assistance to resolve this issue and provide you with the answer you seek.\n", - "True answer: To track their machine learning experiments at every stage, from training to production.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'hugging face open source library transformer acceleration'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': 'The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum. It can be found and used at the following links: \\n- https://github.com/huggingface/optimum-intel\\n- https://github.com/huggingface/optimum\\n- https://huggingface.co/docs/optimum/index'}\u001b[0m\n", - " 97%|█████████▋| 63/65 [24:03<00:52, 26.46s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What parameter is used to ensure that elements in a row have the same height in Gradio?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", - "\n", - "Answer: The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum. It can be found and used at the following links: \n", - "- https://github.com/huggingface/optimum-intel\n", - "- https://github.com/huggingface/optimum\n", - "- https://huggingface.co/docs/optimum/index\n", - "True answer: Optimum\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio library same height parameter in row'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Gradio equal_height parameter for row elements'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The parameter used to ensure that elements in a row have the same height in Gradio is 'equal_height'. It is passed to the '.style()' method of 'gr.Row()'.\"}\u001b[0m\n", - " 98%|█████████▊| 64/65 [24:23<00:24, 24.59s/it]\u001b[33;1m======== New task ========\u001b[0m\n", - "\u001b[37;1mUsing the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "What is the command to install the latest version of Optimum with OpenVINO support?\n", - "\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What parameter is used to ensure that elements in a row have the same height in Gradio?\n", - "\n", - "Answer: The parameter used to ensure that elements in a row have the same height in Gradio is 'equal_height'. It is passed to the '.style()' method of 'gr.Row()'.\n", - "True answer: equal_height\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[33;1mCalling tool: 'retriever' with arguments: {'query': 'Install latest version Optimum OpenVINO'}\u001b[0m\n", - "\u001b[33;1mCalling tool: 'final_answer' with arguments: {'answer': \"The command to install the latest version of Optimum with OpenVINO support is: bash\\npip install --upgrade-strategy eager optimum['openvino']\\n\"}\u001b[0m\n", - "100%|██████████| 65/65 [24:39<00:00, 22.76s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the command to install the latest version of Optimum with OpenVINO support?\n", - "\n", - "Answer: The command to install the latest version of Optimum with OpenVINO support is: bash\n", - "pip install --upgrade-strategy eager optimum['openvino']\n", - "\n", - "True answer: pip install --upgrade-strategy eager optimum[\"openvino\"]\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n" - ] - } - ], - "source": [ - "outputs_agentic_rag = []\n", - "\n", - "for example in tqdm(eval_dataset):\n", - " question = example[\"question\"]\n", - "\n", - " enhanced_question = f\"\"\"Using the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", - "give a comprehensive answer to the question below.\n", - "Respond only to the question asked, response should be concise and relevant to the question.\n", - "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", - "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", - "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", - "\n", - "Question:\n", - "{question}\"\"\"\n", - " answer = agent.run(enhanced_question)\n", - " print(\"=======================================================\")\n", - " print(f\"Question: {question}\")\n", - " print(f\"Answer: {answer}\")\n", - " print(f'True answer: {example[\"answer\"]}')\n", - "\n", - " results_agentic = {\n", - " \"question\": question,\n", - " \"true_answer\": example[\"answer\"],\n", - " \"source_doc\": example[\"source_doc\"],\n", - " \"generated_answer\": answer,\n", - " }\n", - " outputs_agentic_rag.append(results_agentic)" - ] - }, - { - "cell_type": "code", - "execution_count": 45, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 2%|▏ | 1/65 [00:01<01:33, 1.47s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What architecture is the `tokenizers-linux-x64-musl` binary designed for?\n", - "\n", - "Answer: The `tokenizers-linux-x64-musl` binary is designed for the x86_64-unknown-linux-musl architecture.\n", - "True answer: x86_64-unknown-linux-musl\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 3%|▎ | 2/65 [00:02<01:30, 1.44s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the BLIP-Diffusion model?\n", - "\n", - "Answer: BLIP-Diffusion is a model that enables zero-shot subject-driven generation and control-guided zero-shot image generation.\n", - "True answer: The BLIP-Diffusion model is designed for controllable text-to-image generation and editing.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 5%|▍ | 3/65 [00:04<01:41, 1.64s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How can a user claim authorship of a paper on the Hugging Face Hub?\n", - "\n", - "Answer: A user can claim authorship of a paper by clicking on their name in the Paper page and selecting \"Claim authorship\". After confirming the request, the admin team will validate the claim, and the Paper page will be marked as verified.\n", - "True answer: By clicking their name on the corresponding Paper page and clicking \"claim authorship\", then confirming the request in paper settings for admin team validation.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 6%|▌ | 4/65 [00:08<02:38, 2.59s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the /healthcheck endpoint in the Datasets server API?\n", - "\n", - "Answer: The /healthcheck endpoint in the Datasets server API is to ensure that the app is running.\n", - "True answer: Ensure the app is running\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 8%|▊ | 5/65 [00:12<02:51, 2.86s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the default context window size for Local Attention in the LongT5 model?\n", - "\n", - "Answer: The window size is defined as `w` and is determined by `config.attention_window` in the Longformer model. For the LongT5 model specifically, there is no explicit mention of the default window size in the provided documents.\n", - "True answer: 127 tokens\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 9%|▉ | 6/65 [00:13<02:13, 2.26s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What method is used to load a checkpoint for a task using `AutoPipeline`?\n", - "\n", - "Answer: To load a checkpoint for a task using `AutoPipeline`, the `from_pretrained()` method is used.\n", - "True answer: from_pretrained()\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 11%|█ | 7/65 [00:14<01:55, 1.98s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of Diffusers library?\n", - "\n", - "Answer: The Diffusers library is a generative AI library for creating images, videos, and 3D structures from text or images using state-of-the-art pretrained diffusion models.\n", - "True answer: To serve as a modular toolbox for both inference and training of state-of-the-art pretrained diffusion models across multiple modalities.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 12%|█▏ | 8/65 [00:15<01:33, 1.64s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What method does the EulerAncestralDiscreteScheduler use for sampling?\n", - "\n", - "Answer: The EulerAncestralDiscreteScheduler uses ancestral sampling with Euler method steps.\n", - "True answer: Ancestral sampling with Euler method steps.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 14%|█▍ | 9/65 [00:19<02:05, 2.24s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the name of the large multimodal model that can solve image-text tasks and is based on Flamingo?\n", - "\n", - "Answer: The name of the large multimodal model that can solve image-text tasks and is based on Flamingo is IDEFICS.\n", - "True answer: IDEFICS\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 15%|█▌ | 10/65 [00:20<01:50, 2.01s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the `gradio.Blocks` API?\n", - "\n", - "Answer: The `gradio.Blocks` API allows designers to create web apps with more flexible layouts and complex data flows, all in Python.\n", - "True answer: The `gradio.Blocks` API allows you to have full control over the data flows and layout of your application, enabling the building of complex, multi-step applications.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 17%|█▋ | 11/65 [00:22<01:49, 2.02s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the two-stage model proposed in the paper \"Hierarchical Text-Conditional Image Generation with CLIP Latents\"?\n", - "\n", - "Answer: To make a good language-specific text-to-image model using a 2-stage training process inspired by PITI.\n", - "\n", - "\n", - "\n", - "3\n", - "True answer: The purpose of the two-stage model is to generate a CLIP image embedding given a text caption and then generate an image conditioned on the image embedding.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 18%|█▊ | 12/65 [00:24<01:38, 1.86s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What command is used to install the requirements for a research project using 🤗 Transformers?\n", - "\n", - "Answer: The command used to install the requirements for a research project using 🤗 Transformers is `pip install -r requirements.txt` (1)\n", - "True answer: pip install -r requirements.txt\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 20%|██ | 13/65 [00:26<01:47, 2.07s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What task does the `roberta-large-mnli` checkpoint perform?\n", - "\n", - "Answer: Performs a text classification task\n", - "True answer: Text classification\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 22%|██▏ | 14/65 [00:27<01:24, 1.65s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What service is replacing the Paid tier of the Inference API at Hugging Face?\n", - "\n", - "Answer: Inference Endpoints.\n", - "True answer: Inference Endpoints\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 23%|██▎ | 15/65 [00:28<01:17, 1.54s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What architectural feature does SqueezeBERT use instead of fully-connected layers for the Q, K, V, and FFN layers?\n", - "\n", - "Answer: Grouped convolutions\n", - "True answer: Grouped convolutions\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 25%|██▍ | 16/65 [00:29<01:11, 1.47s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What type of license is the HuggingFace Team's software distributed under?\n", - "\n", - "Answer: The HuggingFace Team's software is distributed under the Apache License, Version 2.0.\n", - "True answer: Apache License, Version 2.0\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 26%|██▌ | 17/65 [00:34<01:48, 2.25s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What are the two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed?\n", - "\n", - "Answer: The two parameter-reduction techniques proposed in the ALBERT model to lower memory consumption and increase training speed are:\n", - "\n", - "1. Separating the larger vocabulary embedding into two smaller matrices (Document 1) or splitting the embedding matrix into two smaller ones (Document 0).\n", - "\n", - "2. Allowing layers to share parameters by using repeating layers split among groups (Document 0, Document 1).\n", - "True answer: Splitting the embedding matrix into two smaller matrices and using repeating layers split among groups.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 28%|██▊ | 18/65 [00:40<02:44, 3.49s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What are the three main steps for fine-tuning a model with the 🤗 Datasets library?\n", - "\n", - "Answer: The three main steps for fine-tuning a model with the 🤗 Datasets library are: \n", - "1. Load a dataset from the Hugging Face Hub\n", - "2. Preprocess the data with `Dataset.map()`\n", - "3. Load and compute metrics\n", - "True answer: 1. Load a dataset from the Hugging Face Hub. 2. Preprocess the data with `Dataset.map()`. 3. Load and compute metrics.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 29%|██▉ | 19/65 [00:41<02:01, 2.63s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the maximum improvement in throughput achieved by Hugging Face Infinity compared to vanilla transformers?\n", - "\n", - "Answer: 800%.\n", - "True answer: +800%\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 31%|███ | 20/65 [00:45<02:20, 3.13s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the command to upload a spaCy pipeline to the Hugging Face Hub?\n", - "\n", - "Answer: The command to upload a spaCy pipeline to the Hugging Face Hub is:\n", - "\n", - "```bash\n", - "python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]\n", - "```\n", - "\n", - "Make sure to install `spacy-huggingface-hub` before running this command.\n", - "True answer: python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 32%|███▏ | 21/65 [00:49<02:35, 3.53s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the time and memory complexity of the Nyströmformer's approximation of self-attention?\n", - "\n", - "Answer: The Nyströmformer model approximates standard self-attention with linear or O(n) time and memory complexity.\n", - "True answer: O(n)\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 34%|███▍ | 22/65 [00:55<03:00, 4.20s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the goal of the Named Entity Recognition task in token classification?\n", - "\n", - "Answer: The goal of the Named Entity Recognition task in token classification is to identify named entities in a piece of text, such as a person, location, or organization, and classify each token in the text into one of these categories or a \"none of the above\" category.\n", - "True answer: The goal of the Named Entity Recognition task is to find the entities in a piece of text, such as person, location, or organization.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 35%|███▌ | 23/65 [00:59<02:56, 4.21s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the resolution of images used by the CLIPSeg model?\n", - "\n", - "Answer: The CLIPSeg model uses images of 352x352 pixels.\n", - "True answer: 352 x 352 pixels\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 37%|███▋ | 24/65 [01:01<02:26, 3.58s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What can you use Gradio for?\n", - "\n", - "Answer: You can use Gradio, a Python library, to create customizable web apps for machine learning models and data-processing pipelines.\n", - "True answer: Create a demo for your machine learning model, share your machine learning model with others, and debug your model.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 38%|███▊ | 25/65 [01:04<02:11, 3.28s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What TensorFlow API function is used to load a saved tensor file?\n", - "\n", - "Answer: safetensors.tensorflow.load_file is the API function used to load a saved tensor file.\n", - "True answer: safetensors.tensorflow.load_file\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 40%|████ | 26/65 [01:06<01:53, 2.91s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: Where can you access the logs of your Endpoints in Hugging Face Endpoints?\n", - "\n", - "Answer: You can access and read the logs of your Endpoints in Hugging Face Endpoints through the UI in the \"Logs\" tab of your Endpoint. This feature provides access to both the build logs of your Image artifacts and the Container Logs during inference.\n", - "True answer: In the \"Logs\" tab of your Endpoint through the UI.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 42%|████▏ | 27/65 [01:07<01:22, 2.18s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the latest task added to Hugging Face AutoTrain for Computer Vision?\n", - "\n", - "Answer: Image classification.\n", - "True answer: Image Classification\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 43%|████▎ | 28/65 [01:08<01:12, 1.95s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the default repository type created by the `create_repo` function on Hugging Face Hub?\n", - "\n", - "Answer: By default, the `create_repo` method creates a model repository.\n", - "True answer: model\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 45%|████▍ | 29/65 [01:10<01:15, 2.09s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How many splits does the \"duorc\" dataset have?\n", - "\n", - "Answer: Six\n", - "True answer: Six\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 46%|████▌ | 30/65 [01:13<01:16, 2.19s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of Fully Sharded Data Parallel (FSDP) in distributed training?\n", - "\n", - "Answer: Fully Sharded Data Parallel (FSDP) is a method that shards a model's parameters, gradients, and optimizer states across multiple GPUs. This reduces memory usage and allows for the training of larger models on fewer GPUs.\n", - "True answer: FSDP is developed for distributed training of large pretrained models up to 1T parameters by sharding the model parameters, gradients, and optimizer states across data parallel processes.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 48%|████▊ | 31/65 [01:14<01:07, 1.98s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What file format is used to save and store PyTorch model weights more securely than `.bin` files?\n", - "\n", - "Answer: .safetensors\n", - "True answer: `.safetensors`\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 49%|████▉ | 32/65 [01:15<00:53, 1.62s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What type of security certification does Hugging Face have?\n", - "\n", - "Answer: Hugging Face is SOC2 Type 2 certified (2).\n", - "True answer: SOC2 Type 2 certified\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 51%|█████ | 33/65 [01:17<00:59, 1.86s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What do RAG models combine to generate outputs?\n", - "\n", - "Answer: Retrieval-augmented generation (RAG) models combine the powers of pre-trained dense retrieval (DPR) and sequence-to-sequence models during the generation of outputs.\n", - "True answer: Pretrained dense retrieval (DPR) and sequence-to-sequence models.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 52%|█████▏ | 34/65 [01:19<00:52, 1.70s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What library does MarkupLMFeatureExtractor use to extract data from HTML and XML files?\n", - "\n", - "Answer: Beautiful Soup.\n", - "True answer: Beautiful Soup\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 54%|█████▍ | 35/65 [01:20<00:47, 1.58s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the file size limit for syncing to HF Spaces without using Git-LFS?\n", - "\n", - "Answer: 10MB\n", - "True answer: 10MB\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 55%|█████▌ | 36/65 [01:22<00:44, 1.54s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the title of the paper introducing the ByT5 model?\n", - "\n", - "Answer: ByT5: Towards a token-free future with pre-trained byte-to-byte models\n", - "True answer: ByT5: Towards a token-free future with pre-trained byte-to-byte models\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 57%|█████▋ | 37/65 [01:23<00:38, 1.38s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the dimension of the feature vector for the base BERT model?\n", - "\n", - "Answer: The dimension of the feature vector for the base BERT model is 768.\n", - "True answer: 768\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 58%|█████▊ | 38/65 [01:26<00:56, 2.09s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What special identifier does the WordPiece Model use for continuing subwords?\n", - "\n", - "Answer: The `##` prefix is used to identify the start of a subword that is part of a word.\n", - "True answer: ##\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 60%|██████ | 39/65 [01:27<00:44, 1.70s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the 🧨 Diffusers tutorials?\n", - "\n", - "Answer: To provide a beginner-friendly introduction to diffusion models.\n", - "True answer: To provide a gentle introduction to diffusion models and help understand the library fundamentals.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 62%|██████▏ | 40/65 [01:30<00:49, 1.97s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the default setting for the `allow_flagging` parameter in Gradio's `Interface`?\n", - "\n", - "Answer: \"auto\"\n", - "True answer: \"manual\"\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 63%|██████▎ | 41/65 [01:34<01:04, 2.70s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: Where can the full code for the Stable Diffusion demo be found?\n", - "\n", - "Answer: Document 0 and Document 5: https://hf.co/spaces/stabilityai/stable-diffusion/tree/main and https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion.\n", - "True answer: https://hf.co/spaces/stabilityai/stable-diffusion/tree/main\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 65%|██████▍ | 42/65 [01:38<01:11, 3.11s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What transformation does the FNet model use to replace the self-attention layer in a BERT model?\n", - "\n", - "Answer: The transformation used by the FNet model to replace the self-attention layer in a BERT model is a Fourier Transform.\n", - "True answer: Fourier transform\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 66%|██████▌ | 43/65 [01:39<00:51, 2.34s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What type of test should typically accompany a bug fix in Gradio's testing strategy?\n", - "\n", - "Answer: Dynamic code test\n", - "True answer: Dynamic code test\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 68%|██████▊ | 44/65 [01:40<00:41, 1.98s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How can you force mixed precision training when initializing the Accelerator in 🤗 Accelerate?\n", - "\n", - "Answer: To force mixed precision training when initializing the accelerator in 🤗 Accelerate, use the command --fp16.\n", - "True answer: By passing `fp16=True` to the Accelerator init.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 69%|██████▉ | 45/65 [01:41<00:36, 1.81s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of tokenizers in the NLP pipeline?\n", - "\n", - "Answer: The purpose of tokenizers in the NLP pipeline is to translate text data into numerical data that can be processed by NLP models.\n", - "True answer: To translate text into data that can be processed by the model.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 71%|███████ | 46/65 [01:46<00:52, 2.74s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the Safety Checker in the Diffusers library?\n", - "\n", - "Answer: The **Safety Checker** is a component of Diffusers library that flags inappropriate content generated by models during inference. It screens against harmful content by checking and comparing the class probability of hardcoded harmful concepts in the embedding space against an image.\n", - "True answer: The Safety Checker checks and compares the class probability of a set of hard-coded harmful concepts in the embedding space against an image after it has been generated to mitigate the risk of generating harmful content.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 72%|███████▏ | 47/65 [01:51<01:00, 3.38s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What Python class allows you to retrieve Discussions and Pull Requests from a given repository on the Hugging Face Hub?\n", - "\n", - "Answer: HfApi\n", - "True answer: HfApi\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 74%|███████▍ | 48/65 [01:52<00:43, 2.55s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the name of the new library introduced by Hugging Face for hosting scikit-learn models?\n", - "\n", - "Answer: huggingface_hub\n", - "True answer: Skops\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 75%|███████▌ | 49/65 [01:54<00:41, 2.60s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of Textual Inversion?\n", - "\n", - "Answer: Textual inversion is a technique that allows a model to learn a new concept from a few sample images. This gives the user more control over the generated images and allows the model to be tailored towards specific concepts.\n", - "True answer: Textual Inversion is a training method for personalizing models by learning new text embeddings from a few example images.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 77%|███████▋ | 50/65 [01:55<00:30, 2.01s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the recommended multiple of batch size for fp16 data type on an A100 GPU?\n", - "\n", - "Answer: 5\n", - "True answer: 64\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 78%|███████▊ | 51/65 [01:58<00:33, 2.41s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How do you run a Gradio Blocks app in reload mode using a Python IDE?\n", - "\n", - "Answer: To run a Gradio Blocks app in reload mode from a Python IDE, use the terminal and type in: \"gradio app.py\" to launch the demo.\n", - "True answer: Run `gradio run.py` in the terminal.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 80%|████████ | 52/65 [02:03<00:40, 3.08s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How can you install the Hugging Face Unity API in your Unity project?\n", - "\n", - "Answer: To install the Hugging Face Unity API in your Unity project, follow these steps: \n", - "\n", - "1. Open your Unity project. \n", - "2. Navigate to 'Window' and select 'Package Manager'. \n", - "3. Click the '+' button and choose the option 'Add Package from git URL'. \n", - "4. In the field provided, enter 'https://github.com/huggingface/unity-api.git'. \n", - "5. Once installed, the Unity API wizard should open. If not,\n", - "True answer: To install the Hugging Face Unity API in your Unity project, go to `Window` -> `Package Manager`, click `+` and select `Add Package from git URL`, then enter `https://github.com/huggingface/unity-api.git`.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 82%|████████▏ | 53/65 [02:05<00:33, 2.80s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the pretraining objective of the Wav2Vec2 context network?\n", - "\n", - "Answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task, which involves predicting the true quantized speech representation of a masked prediction.\n", - "True answer: The pretraining objective of the Wav2Vec2 context network is a contrastive task where the model has to predict the true quantized speech representation of the masked prediction from a set of false ones.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 83%|████████▎ | 54/65 [02:08<00:31, 2.82s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the default checkpoint used by the sentiment analysis pipeline in the Transformers library?\n", - "\n", - "Answer: The default checkpoint used by the sentiment analysis pipeline in the Transformers library is distilbert-base-uncased-finetuned-sst-2-english. (Source document: 0)\n", - "True answer: distilbert base uncased finetuned sst2 english\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 85%|████████▍ | 55/65 [02:14<00:38, 3.80s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\"?\n", - "\n", - "Answer: The purpose of the notebook \"How to use DeepSpeed to train models with billions of parameters on Habana Gaudi\" (Documents 0,1) is to demonstrate how to use DeepSpeed for pre-training and fine-tuning large-scale language models, specifically the 1.6B-parameter GPT2-XL, on Habana Gaudi hardware. The notebook provides a practical guide for utilizing DeepSpeed and Habana Gaudi to accelerate and optimize the training of models with a large number of parameters\n", - "True answer: To show how to use DeepSpeed to pre-train/fine-tune the 1.6B-parameter GPT2-XL for causal language modeling on Habana Gaudi.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 86%|████████▌ | 56/65 [02:15<00:27, 3.01s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What command line module does PyTorch provide to run a script on multiple GPUs?\n", - "\n", - "Answer: 'CUDA_VISIBLE_DEVICES` is the command line module that PyTorch provides to run a script on multiple GPUs.\n", - "True answer: torchrun\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 88%|████████▊ | 57/65 [02:20<00:27, 3.46s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the most popular vision transformer model on the Hugging Face Model Hub for image classification?\n", - "\n", - "Answer: The most popular vision transformer model for image classification on the Hugging Face Model Hub is 'google/vit-base-patch16-224'.\n", - "True answer: google/vit-base-patch16-224\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 89%|████████▉ | 58/65 [02:24<00:26, 3.79s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the command to upload an ESPnet model to a Hugging Face repository?\n", - "\n", - "Answer: The command to upload an ESPnet model to a Hugging Face repository is:\n", - "```bash\n", - "./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n", - "```\n", - "where `username` and `model_repo` should be replaced with the desired username and model repository name, respectively.\n", - "True answer: ./run.sh --stage 15 --skip_upload_hf false --hf_repo username/model_repo\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 91%|█████████ | 59/65 [02:30<00:26, 4.35s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What file should be added to a model repository to install custom Python dependencies for Inference Endpoints?\n", - "\n", - "Answer: A `requirements.txt` file should be added to a model repository to install custom Python dependencies for Inference Endpoints.\n", - "True answer: requirements.txt\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 92%|█████████▏| 60/65 [02:32<00:17, 3.52s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: How many images are needed to teach new concepts to Stable Diffusion using Textual Inversion?\n", - "\n", - "Answer: To teach new concepts to Stable Diffusion using Textual Inversion, you need 3 to 5 sample images.\n", - "True answer: 3-5 images\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 94%|█████████▍| 61/65 [02:34<00:13, 3.27s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the maximum size of a model checkpoint before it is automatically sharded in Transformers version 4.18.0?\n", - "\n", - "Answer: Since version 4.18.0, model checkpoints that end up taking more than 10GB of space are automatically sharded into smaller pieces.\n", - "True answer: 10GB\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 95%|█████████▌| 62/65 [02:36<00:08, 2.80s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the purpose of Weights and Biases (W&B) for data scientists and machine learning scientists?\n", - "\n", - "Answer: The purpose of Weights and Biases (W&B) is to enable data scientists and machine learning scientists to track their machine learning experiments at every stage, from training to production.\n", - "True answer: To track their machine learning experiments at every stage, from training to production.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 97%|█████████▋| 63/65 [02:38<00:05, 2.68s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the name of the open-source library created by Hugging Face to simplify Transformer acceleration?\n", - "\n", - "Answer: The name of the open-source library created by Hugging Face to simplify Transformer acceleration is Optimum.\n", - "True answer: Optimum\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " 98%|█████████▊| 64/65 [02:39<00:02, 2.21s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What parameter is used to ensure that elements in a row have the same height in Gradio?\n", - "\n", - "Answer: `equal_height`\n", - "True answer: equal_height\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "100%|██████████| 65/65 [02:42<00:00, 2.51s/it]" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=======================================================\n", - "Question: What is the command to install the latest version of Optimum with OpenVINO support?\n", - "\n", - "Answer: The command to install the latest version of Optimum with OpenVINO support is:\n", - "```bash\n", - "pip install --upgrade-strategy eager optimum[\"openvino,nncf\"]\n", - "```\n", - "True answer: pip install --upgrade-strategy eager optimum[\"openvino\"]\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n" - ] - } - ], + "outputs": [], + "source": [ + "outputs_agentic_rag = []\n", + "\n", + "for example in tqdm(eval_dataset):\n", + " question = example[\"question\"]\n", + "\n", + " enhanced_question = f\"\"\"Using the information contained in your knowledge base, which you can access with the 'retriever' tool,\n", + "give a comprehensive answer to the question below.\n", + "Respond only to the question asked, response should be concise and relevant to the question.\n", + "If you cannot find information, do not give up and try calling your retriever again with different arguments!\n", + "Make sure to have covered the question completely by calling the retriever tool several times with semantically different queries.\n", + "Your queries should not be questions but affirmative form sentences: e.g. rather than \"How do I load a model from the Hub in bf16?\", query should be \"load a model from the Hub bf16 weights\".\n", + "\n", + "Question:\n", + "{question}\"\"\"\n", + " answer = agent.run(enhanced_question)\n", + " print(\"=======================================================\")\n", + " print(f\"Question: {question}\")\n", + " print(f\"Answer: {answer}\")\n", + " print(f'True answer: {example[\"answer\"]}')\n", + "\n", + " results_agentic = {\n", + " \"question\": question,\n", + " \"true_answer\": example[\"answer\"],\n", + " \"source_doc\": example[\"source_doc\"],\n", + " \"generated_answer\": answer,\n", + " }\n", + " outputs_agentic_rag.append(results_agentic)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from huggingface_hub import InferenceClient\n", "\n", @@ -5634,7 +795,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Let us recap: the Agent setup improves scores by 8.5% compared to a standard RAG!**\n", + "**Let us recap: the Agent setup improves scores by 8.5% compared to a standard RAG!** (from 70.0% to 78.5%)\n", "\n", "This is a great improvement, with a very simple setup 🚀\n", "\n", @@ -5658,7 +819,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.14" + "version": "3.12.0" } }, "nbformat": 4,