diff --git a/notebooks/en/_toctree.yml b/notebooks/en/_toctree.yml index 09825b25..ae6d9ef9 100644 --- a/notebooks/en/_toctree.yml +++ b/notebooks/en/_toctree.yml @@ -12,3 +12,5 @@ title: Advanced RAG on HuggingFace documentation using LangChain - local: rag_evaluation title: RAG Evaluation + - local: chain_of_verification + title: Chain-of-Verification Recipe - Prompt Engineering with AIConfig diff --git a/notebooks/en/chain_of_verification.ipynb b/notebooks/en/chain_of_verification.ipynb new file mode 100644 index 00000000..d5ebf8b8 --- /dev/null +++ b/notebooks/en/chain_of_verification.ipynb @@ -0,0 +1,490 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "_KIVLBz840Bi" + }, + "source": [ + "# Chain-of-Verification Recipe - Prompt Engineering\n", + "_Authored by: [Ankush Pala](https://github.com/Ankush-lastmile)_\n", + "\n", + "Chain-of-Verification (CoVe) is a **prompt engineering technique to reduce hallucinations!** An LLM generates a baseline response to a user query, but this might contain errors. CoVe helps by creating a plan comprising of verification questions that are used to validate the information. This process results in more accurate answers than the initial response. The final answer is revised based on these validations. **[ Link to Paper](https://arxiv.org/pdf/2309.11495.pdf)**\n", + "\n", + "**Check out the open-source tool used here! 🚀 [AIConfig Github Repo](https://github.com/lastmile-ai/aiconfig)**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "k3tsITZhVFp-" + }, + "outputs": [], + "source": [ + "# Install AIConfig package\n", + "!pip install python-aiconfig\n", + "!pip install aiconfig-extension-hugging-face" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "51w-3OZC_Z97" + }, + "outputs": [], + "source": [ + "# Import required modules from AIConfig and other dependencies\n", + "\n", + "import json\n", + "import pandas as pd\n", + "from IPython.display import display, Markdown\n", + "from aiconfig import AIConfigRuntime, CallbackManager, InferenceOptions\n", + "from aiconfig_extension_hugging_face import HuggingFaceTextGenerationRemoteInference" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Enables the Use of HuggingFace models for remote inference\n", + "def register_model_parsers() -> None:\n", + " \"\"\"Register model parsers for HuggingFace models.\"\"\"\n", + " # Register remote inference client for text generation\n", + " text_generation_remote = HuggingFaceTextGenerationRemoteInference()\n", + " AIConfigRuntime.register_model_parser(\n", + " text_generation_remote, \"Text Generation\"\n", + " )\n", + "register_model_parsers()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Z1Y91C-iIyq4" + }, + "source": [ + "**The cell below defines the CoVe prompt template config.**\n", + "\n", + "The next code cell sets up a 'CoVe prompt template' within a structure known as AIConfig. AIConfig is a data format to organize prompt templates and specific model settings. To generate an AIConfig, use the AIConfig Editor VSCode Extension. This gives you a user-friendly interface to create prompt templates across any model and store these templates in a config format. You can use the AIConfig SDK to execute the prompts in the config along with their settings in your application code.\n", + "\n", + "Alternatively, you can also download the config [here](https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/Chain-of-Verification/cove_template_config.json) and load the config with\n", + "\n", + "`config = AIConfigRuntime.load('cove_template_config.json')`.\n", + "\n", + "Check these links out for more background on AIConfig:\n", + "\n", + "[AIConfig Github](https://github.com/lastmile-ai/aiconfig) \n", + "\n", + "[AIConfig Vscode Extension](https://marketplace.visualstudio.com/items?itemName=lastmile-ai.vscode-aiconfig)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "cellView": "form", + "id": "8VQVZicOGN5b" + }, + "outputs": [], + "source": [ + "# @title\n", + "cove_template_config = {\n", + " \"name\": \"Chain-of-Verification (CoVe) Template\",\n", + " \"schema_version\": \"latest\",\n", + " \"metadata\": {\n", + " \"parameters\": {\n", + " \"baseline_prompt\": \"Name 20 programming languages that were developed in the United States. Include the developer name in parantheses.\",\n", + " \"verification_question\": \"Where was {{entity}} born? \"\n", + " },\n", + " \"models\": {\n", + " \"gpt-4\": {\n", + " \"model\": \"gpt-4\",\n", + " \"top_p\": 1,\n", + " \"temperature\": 0,\n", + " \"presence_penalty\": 0,\n", + " \"frequency_penalty\": 0\n", + " }\n", + " }\n", + " },\n", + " \"description\": \"\",\n", + " \"prompts\": [\n", + " {\n", + " \"name\": \"baseline_response_gen\",\n", + " \"input\": \"{{baseline_prompt}}\",\n", + " \"metadata\": {\n", + " \"model\": {\n", + " \"name\": \"Text Generation\",\n", + " \"settings\": {\n", + " \"system_prompt\": \"\",\n", + " \"model\": \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n", + " }\n", + " },\n", + " \"parameters\": {},\n", + " \"remember_chat_context\": False\n", + " },\n", + " \"outputs\": []\n", + " },\n", + " {\n", + " \"name\": \"verification\",\n", + " \"input\": \"{{verification_question}}\",\n", + " \"metadata\": {\n", + " \"model\": {\n", + " \"name\": \"gpt-4\",\n", + " \"settings\": {\n", + " \"system_prompt\": \"{{entity}}\"\n", + " }\n", + " },\n", + " \"parameters\": {\n", + " \"entity\": \"George Pataki\"\n", + " },\n", + " \"remember_chat_context\": False\n", + " },\n", + " \"outputs\": []\n", + " },\n", + " {\n", + " \"name\": \"final_response_gen\",\n", + " \"input\": \"Cross-check the provided list of verification data with the original baseline response that is supposed to accurately answer the baseline prompt. \\n\\nBaseline prompt: {{baseline_prompt}} \\nBaseline response: {{baseline_response_gen.output}}\\nVerification data: {{verification_results}}\",\n", + " \"metadata\": {\n", + " \"model\": {\n", + " \"name\": \"gpt-4\",\n", + " \"settings\": {\n", + " \"system_prompt\": \"For each entity from the baseline response, verify that the entity met the criteria asked for in the baseline prompt based on the verification data. \\n\\nOutput Format: \\n\\n### Revised Response \\nThis is the revised response after running chain-of-verification. \\n(Please output the revised response after the cross-check.)\\n\\n### Failed Entities \\nThese are the entities that failed the cross-check and are no longer included in revised response. \\n(List the entities that failed the cross-check with a concise reason why)\"\n", + " }\n", + " },\n", + " \"parameters\": {\n", + " \"verification_results\": \"Theodore Roosevelt was born in New York City, New York on October 27, 1858. Franklin D. Roosevelt was born in Hyde Park, New York on January 30, 1882. Alexander Hamilton was born in Charlestown, Nevis on January 11, 1755. John Jay was born in New York City, New York on December 12, 1745. DeWitt Clinton was born in Little Britain, New York on March 2, 1769. William H. Seward was born in Florida, New York on May 16, 1801. Charles Evans Hughes was born in Glens Falls, New York on April 11, 1862. Nelson Rockefeller was born in Bar Harbor, Maine on July 8, 1908. Robert F. Wagner Jr. was born in Manhattan, New York on April 20, 1910. Bella Abzug was born in New York City, New York on July 24, 1920. Shirley Chisholm was born in Brooklyn, New York on November 30, 1924. Geraldine Ferraro was born in Newburgh, New York on August 26, 1935. Eliot Spitzer was born in The Bronx, New York on June 10, 1959. Michael Bloomberg was born in Boston, Massachusetts on February 14, 1942. Andrew Cuomo was born in New York City, New York on December 6, 1957. Bill de Blasio was born in Manhattan, New York on May 8, 1961. Charles Rangel was born in Harlem, New York City on June 11, 1930. Daniel Patrick Moynihan was born in Tulsa, Oklahoma on March 16, 1927. Jacob Javits was born in New York City, New York on May 18, 1904. Al Smith was born in New York City, New York on December 30, 1873. Rudy Giuliani was born in Brooklyn, New York on May 28, 1944. George Pataki was born in Peekskill, New York on June 24, 1945. Kirsten Gillibrand was born in Albany, New York on December 9, 1966. Chuck Schumer was born in Brooklyn, New York on November 23, 1950. Alexandria Ocasio-Cortez was born in The Bronx, New York City, New York on October 13, 1989.\"\n", + " },\n", + " \"remember_chat_context\": False\n", + " },\n", + " \"outputs\": []\n", + " }\n", + " ],\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ykZE2ieO6ryn" + }, + "source": [ + "## 1. Baseline Response\n", + "Prompt LLM with user question that generates a list. The baseline response from the LLM might contain inaccuracies that we can verify.\n", + "\n", + "**Prompt: Name 20 programming languages that were developed in the United States.**" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "6cAw1ekXCxGn" + }, + "outputs": [], + "source": [ + "\n", + "config = AIConfigRuntime.create(**cove_template_config) # loads config (see code above)\n", + "config.callback_manager = CallbackManager([])\n", + "\n", + "inference_options = InferenceOptions() # setup streaming" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "nbolW2mVDeZD", + "outputId": "07cbaff4-3125-442a-cd2a-0126aa09b1b7" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "1. C (Dennis Ritchie)\n", + "2. C++ (Bjarne Stroustrup)\n", + "3. Java (James Gosling)\n", + "4. Python (Guido van Rossum)\n", + "5. JavaScript (Brendan Eich)\n", + "6. Swift (Chris Lattner)\n", + "7. Go (Robert Griesemer, Rob Pike, Ken Thompson)\n", + "8. Rust (Graydon Hoare)\n", + "9. PHP (Rasmus Lerdorf)\n", + "10. Ruby (Yukihiro Matsumoto)\n", + "11. Perl (Larry Wall)\n", + "12. Haskell (Simon Peyton Jones, Paul Hudak, John Hughes)\n", + "13. Lisp (John McCarthy)\n", + "14. Smalltalk (Alan Kay, Dan Ingalls, Adele Goldberg)\n", + "15. Ada (Jean Ichbiah)\n", + "16. Fortran (John Backus)\n", + "17. COBOL (Grace Hopper)\n", + "18. Lua (Roberto Ierusalimschy, Luiz Henrique de Figueiredo, Waldemar Celes)\n", + "19. Prolog (Alain Colmerauer, Philippe Roussel)\n", + "20. ML (Robin Milner)\n", + "\n", + "Note: While some of these languages were developed by individuals who were not American citizens, they were developed while they were working in the United States.\n" + ] + } + ], + "source": [ + "baseline_prompt = \"Name 20 programming languages that were developed in the United States. Include the developer name in parantheses.\"\n", + "\n", + "# Run baseline prompt to generate initial response which might contain errors\n", + "async def run_baseline_prompt(baseline_prompt):\n", + " config.update_parameter(\"baseline_prompt\", baseline_prompt)\n", + " config.save()\n", + "\n", + " await config.run(\"baseline_response_gen\", options=inference_options) # run baseline prompt\n", + " return config.get_output_text(\"baseline_response_gen\")\n", + "\n", + "baseline_response = await run_baseline_prompt(baseline_prompt)\n", + "print(baseline_response)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7OQNz9cM7Myv" + }, + "source": [ + "\n", + "## 2. Setup and Test Verification Question\n", + "Given both query and baseline response, generate a verification\n", + "question that could help to self-analyze if there are any mistakes in the original response. We will use one verification question here.\n", + "\n", + "**Verification Prompt: Where was this coding language developed: {{entity}}?**" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "jD9S3q5mMtqd", + "outputId": "6d3446a9-32ff-4fc1-9e3d-1f3600455b26" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Clojure was developed in the United States." + ] + } + ], + "source": [ + "# verification_question = \"Where was {{entity}} born?\"\n", + "verification_question = \"Where was this coding language developed: {{entity}}?\"\n", + "\n", + "# Run verification on a single entity from baseline response to test\n", + "async def run_single_verification(verification_question, entity):\n", + " params = {\"entity\": entity}\n", + " config.update_parameter(\"verification_question\", verification_question)\n", + " config.save()\n", + "\n", + " verification_completion = await config.run(\"verification\", params, options=inference_options)\n", + " return verification_completion\n", + "\n", + "verification_completion = await run_single_verification(verification_question, \"clojure\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "B9Zaypp075f9" + }, + "source": [ + "## 3. Execute Verifications\n", + "Answer each verification question for each entity from the the baseline response. Save the verification results in a single string." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "QFew6GhONR8X", + "outputId": "71af66ce-d410-49ab-c65b-87eaee836219" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The C programming language was developed at Bell Labs (Bell Telephone Laboratories Inc.) in the United States.\n", + "\n", + "The C++ coding language was developed at Bell Labs in Murray Hill, New Jersey, USA.\n", + "\n", + "Java was developed at Sun Microsystems.\n", + "\n", + "Python was developed in the Netherlands.\n", + "\n", + "JavaScript was developed at Netscape Communications Corporation.\n", + "\n", + "Swift was developed at Apple Inc.\n", + "\n", + "The Go coding language was developed at Google Inc. in the United States.\n", + "\n", + "The Rust coding language was developed at Mozilla Research.\n", + "\n", + "PHP was developed in Greenland.\n", + "\n", + "The coding language Ruby was developed in Japan.\n", + "\n", + "Perl was developed in the United States.\n", + "\n", + "Haskell was developed at the University of Glasgow, Scotland.\n", + "\n", + "Lisp was developed at the Massachusetts Institute of Technology (MIT).\n", + "\n", + "Smalltalk was developed at Xerox PARC (Palo Alto Research Center) in Palo Alto, California.\n", + "\n", + "The Ada programming language was developed in the United States by the Defense Advanced Research Projects Agency (DARPA) for the U.S. Department of Defense.\n", + "\n", + "Fortran was developed at IBM in the United States.\n", + "\n", + "COBOL (Grace Hopper) was developed in the United States.\n", + "\n", + "The Lua coding language was developed in Rio de Janeiro, Brazil.\n", + "\n", + "Prolog was developed at the University of Marseille in France.\n", + "\n", + "ML (Meta Language) was developed at the University of Edinburgh in Scotland.\n", + "\n", + "Sorry, I can't provide the information because the entity is not specified. Please provide the name of the coding language.\n", + "\n" + ] + } + ], + "source": [ + "# Extracts entity names from a given baseline response by processing each line with regex.\n", + "def gen_entities_list(baseline_response):\n", + " rows = baseline_response.split('\\n')\n", + " entities = []\n", + "\n", + " for row in rows:\n", + " if not row.strip():\n", + " continue\n", + " entities.append(pd.Series(row).str.extract(r'(\\d+\\.\\s)([^,]*)')[1].values[0])\n", + "\n", + " return entities\n", + "\n", + "# Run verification question for each entity and concatenates returned verifications into a single string.\n", + "async def gen_verification_results(entities):\n", + " verification_data = \"\"\n", + " for n in entities:\n", + " params = {\n", + " \"verification_question\": verification_question,\n", + " \"entity\": n\n", + " }\n", + " verification_completion = await config.run(\"verification\", params, options=inference_options)\n", + " single_verification_text = config.get_output_text(\"verification\")\n", + " verification_data += \" \" + single_verification_text\n", + " print(\"\\n\")\n", + "\n", + " return verification_data\n", + "\n", + "\n", + "entities = gen_entities_list(baseline_response)\n", + "verification_data = await gen_verification_results(entities)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ldof6NdR86qI" + }, + "source": [ + "## 4. Generate Revised Response\n", + "Given the discovered inconsistencies (if any), generate a revised response incorporating the verification results." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "4MiNxiJc9GPI" + }, + "outputs": [ + { + "data": { + "text/markdown": [ + "### Revised Response \n", + "1. C (Dennis Ritchie)\n", + "2. C++ (Bjarne Stroustrup)\n", + "3. Java (James Gosling)\n", + "4. JavaScript (Brendan Eich)\n", + "5. Swift (Chris Lattner)\n", + "6. Go (Robert Griesemer, Rob Pike, Ken Thompson)\n", + "7. Rust (Graydon Hoare)\n", + "8. Perl (Larry Wall)\n", + "9. Lisp (John McCarthy)\n", + "10. Smalltalk (Alan Kay, Dan Ingalls, Adele Goldberg)\n", + "11. Ada (Jean Ichbiah)\n", + "12. Fortran (John Backus)\n", + "13. COBOL (Grace Hopper)\n", + "\n", + "### Failed Entities \n", + "1. Python (Guido van Rossum) - Python was developed in the Netherlands, not the United States.\n", + "2. PHP (Rasmus Lerdorf) - PHP was developed in Greenland, not the United States.\n", + "3. Ruby (Yukihiro Matsumoto) - Ruby was developed in Japan, not the United States.\n", + "4. Haskell (Simon Peyton Jones, Paul Hudak, John Hughes) - Haskell was developed in Scotland, not the United States.\n", + "5. Lua (Roberto Ierusalimschy, Luiz Henrique de Figueiredo, Waldemar Celes) - Lua was developed in Brazil, not the United States.\n", + "6. Prolog (Alain Colmerauer, Philippe Roussel) - Prolog was developed in France, not the United States.\n", + "7. ML (Robin Milner) - ML was developed in Scotland, not the United States." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Generated the revised response using verification data\n", + "params = {\"verification_results\": verification_data}\n", + "revised_response = await config.run(\"final_response_gen\", params)\n", + "\n", + "# Display with Markdown\n", + "display(Markdown(config.get_output_text(\"final_response_gen\")))" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/en/index.md b/notebooks/en/index.md index b9b2a530..81f68054 100644 --- a/notebooks/en/index.md +++ b/notebooks/en/index.md @@ -1,12 +1,13 @@ # Open-Source AI Cookbook -The Open-Source AI Cookbook is a collection of notebooks illustrating practical aspects of building AI +The Open-Source AI Cookbook is a collection of notebooks illustrating practical aspects of building AI applications and solving various machine learning tasks using open-source tools and models. ## Latest notebooks -Check out the recently added notebooks: +Check out the recently added notebooks: +- [Chain-of-Verification Recipe - Prompt Engineering with AIConfig](chain_of_verification.ipynb) - [Simple RAG for GitHub issues using Hugging Face Zephyr and LangChain](rag_zephyr_langchain) - [Embedding multimodal data for similarity search using 🤗 transformers, 🤗 datasets and FAISS](faiss_with_hf_datasets_and_clip) - [Fine-tuning a Code LLM on Custom Code on a single GPU](fine_tuning_code_llm_on_single_gpu) @@ -17,6 +18,6 @@ You can also check out the notebooks in the cookbook's [GitHub repo](https://git ## Contributing -The Open-Source AI Cookbook is a community effort, and we welcome contributions from everyone! -Check out the cookbook's [Contribution guide](https://github.com/huggingface/cookbook/blob/main/README.md) to learn -how you can add your "recipe". +The Open-Source AI Cookbook is a community effort, and we welcome contributions from everyone! +Check out the cookbook's [Contribution guide](https://github.com/huggingface/cookbook/blob/main/README.md) to learn +how you can add your "recipe".