3_document_qa.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "TbEyM5LG7v"
      },
      "source": [
        "# Dependencies\n",
        "\n",
        "1. `PyPDFLoader` depends upon `pypdf` to process the pdfs\n",
        "2. `YoutubeAudioLoader` depends upon `yt_dlp`, `pydub` and `librosa`\n",
        "    - `yt_dlp`: To download the relevant audio transcripts of youtube videos\n",
        "    - `pydub`: To split the audio to adhere to OpenAI Whisper's 25mb limit\n",
        "\n",
        "(Here is the relevant list of all other document loaders)[https://python.langchain.com/docs/integrations/document_loaders]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "jROmUVO56V"
      },
      "source": [
        "from langchain.document_loaders import PyPDFLoader\n",
        "\n",
        "pdf_loader = PyPDFLoader('docs/pdf/hyperion.pdf')\n",
        "pdf_docs = pdf_loader.load()\n",
        "pdf_docs = pdf_docs\n",
        "print(f'Number of pages: {len(pdf_docs)}')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Number of pages: 570\n"
        }
      ],
      "execution_count": 1
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "S1hZu5RQBT"
      },
      "source": [
        "# Possible Bugs\n",
        "\n",
        "I've encountered the recent versions of openai (>=1.0.0) to be incompatible with the latest version of langchain (0.0.333), as a result I've had to make the following changes:\n",
        "\n",
        "1. run `openai migrate <path-to-langchain>/document_loaders/parsers/audio.py` replace `<path-to-langchain>` with the correct path to langchain\n",
        "2. change line 66 in `<path-to-langchain>/document_loaders/parsers/audio.py` to `transcript = client.audio.transcriptions.create(model=\"whisper-1\", file=file_obj)`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "XoRGWAQZ64"
      },
      "source": [
        "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader\n",
        "from langchain.document_loaders.generic import GenericLoader\n",
        "from langchain.document_loaders.parsers import OpenAIWhisperParser\n",
        "\n",
        "url = 'https://youtu.be/_PPWWRV6gbA?si=hQFeGBgt6yawfuPI'\n",
        "youtube_loader = GenericLoader(YoutubeAudioLoader([url],'docs/youtube'), OpenAIWhisperParser())\n",
        "youtube_docs = youtube_loader.load()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "Ms4oGwnKyA"
      },
      "source": [
        "# Preprocessing & Splitting\n",
        "\n",
        "### Preprocessing\n",
        "1. The `PyPDFLoader` returns a list of `Document` objects, each of which has a `page_content` and `metadata` attribute\n",
        "2. The `page_content` attribute is then preprocessed to add `#` before each chapter number\n",
        "3. The `metadata` attribute is then updated to include the chapter number, story and character name\n",
        "\n",
        "\n",
        "### Splitting\n",
        "Langchain provides us with numerous splitting options, some of most common ones are:\n",
        "1. `CharacterTextSplitter`: Splits the text into chunks of a fixed size, with a fixed overlap\n",
        "2. `RecursiveCharacterTextSplitter`: Simillar to `CharacterTextSplitter` but recursively splits the text into smaller chunks\n",
        "3. `MarkdownTextSplitter`: Splits the text into chunks based on markdown headers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "Cc7oqpJZQe"
      },
      "source": [
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "import re\n",
        "\n",
        "def preprocess_pdf(docs):\n",
        "    sc_mapping = {'Priest/Lenar Hoyt': [35, 114], 'Soldier/Fedmahn Kassad': [144, 204], 'Poet/Martin Silenus': [210, 271], 'Scholar/Sol Weintraub': [285, 356], 'Detective/Brawne Lamia': [376, 470], 'Consul/Consul': [484, 541]}\n",
        "    chapter = 0\n",
        "    for page in docs:\n",
        "        page_content = re.sub(r'^\\s*(\\d+)\\s*$', r'#\\1', page.page_content, flags=re.MULTILINE)\n",
        "        if '#' in page_content: chapter = int(page_content.split('#')[1].split('\\n')[0])\n",
        "        for k,v in sc_mapping.items():\n",
        "            if v[0] <= page.metadata['page']+1 <= v[1]:\n",
        "                story = k.split('/')[0]\n",
        "                character = k.split('/')[1]\n",
        "                break\n",
        "            story, character = 'Plot', 'None'\n",
        "        page.page_content = page_content\n",
        "        page.metadata['chapter'] = chapter\n",
        "        page.metadata['story'] = story\n",
        "        page.metadata['character'] = character\n",
        "        page.metadata['page'] += 1\n",
        "    return docs\n",
        "\n",
        "pdf_docs = preprocess_pdf(pdf_docs)\n",
        "pdf_docs_copy = pdf_docs.copy()\n",
        "for _ in range(3):\n",
        "    pdf_docs += pdf_docs_copy.copy()\n",
        "print(f'Number of pages: {len(pdf_docs)}')\n",
        "print(f'random page: {pdf_docs[320].page_content[:100]}')\n",
        "print(f'random page metadata: {pdf_docs[320].metadata}')\n",
        "\n",
        "text_splitter = RecursiveCharacterTextSplitter(\n",
        "    chunk_size=1500,\n",
        "    chunk_overlap=150,\n",
        "    separators=['#', '\\n', '\"', ' ', '']\n",
        ")\n",
        "splits = text_splitter.split_documents(pdf_docs)\n",
        "print(f'Number of splits: {len(splits)}')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Number of pages: 2280\nrandom page: last contact, Arundez had aged but little\u2014Sol guessed that\nhe was still in his late twenties. But th\nrandom page metadata: {'source': 'docs/pdf/hyperion.pdf', 'page': 321, 'chapter': 4, 'story': 'Scholar', 'character': 'Sol Weintraub'}\nNumber of splits: 4340\n"
        }
      ],
      "execution_count": 2
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "WedNBTtFMw"
      },
      "source": [
        "# Embedding & Vectorstore\n",
        "\n",
        "### Embedding\n",
        "Langchain provides us with numerous vectorization options, some of most common ones are:\n",
        "1. `HuggingFaceEmbeddings`: Uses the HuggingFace transformers library to generate embeddings\n",
        "2. `OpenAIEmbeddings`: Uses the OpenAI GPT library to generate embeddings\n",
        "\n",
        "We chose to use the `HuggingFaceEmbeddings` as OpenAI was rate limiting the number of requests we could make to their API\n",
        "\n",
        "### Vectorstore\n",
        "A vectorstore is a database of embeddings which corresponds to a set of documents. Langchain provides us with numerous vectorstore options, some of most common ones are:\n",
        "1. `FAISS`: Stands for Facebook AI similarity search\n",
        "2. `Chroma`: langchain's preferred vectorstore"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "olTlw6cSCu"
      },
      "source": [
        "from langchain.embeddings import HuggingFaceEmbeddings\n",
        "from langchain.vectorstores import FAISS, Chroma\n",
        "import os\n",
        "\n",
        "embedding = HuggingFaceEmbeddings()\n",
        "save_dir = r'docs/vectorstores'\n",
        "\n",
        "def retrieve_FAISS_vectorstore(documents):\n",
        "    store_name, _ = os.path.splitext(os.path.basename(documents[0].metadata['source']))\n",
        "    store_path = os.path.join(os.getcwd(), save_dir, 'faiss', store_name)\n",
        "    if not os.path.exists(store_path):\n",
        "        vectorstore = FAISS.from_documents(documents=documents, embedding=embedding)\n",
        "        vectorstore.save_local(store_path)\n",
        "        return vectorstore\n",
        "    else:\n",
        "        return FAISS.load_local(store_path, embedding)\n",
        "\n",
        "def retrieve_Chroma_vectorstore(documents):\n",
        "    store_name, _ = os.path.splitext(os.path.basename(documents[0].metadata['source']))\n",
        "    store_path = os.path.join(os.getcwd(), save_dir, 'chroma', store_name)\n",
        "    if not os.path.exists(store_path):\n",
        "        os.makedirs(store_path)\n",
        "        vectorstore = Chroma.from_documents(documents=documents, embedding=embedding, persist_directory=store_path)\n",
        "    else:\n",
        "        vectorstore = Chroma(persist_directory=store_path, embedding_function=embedding)\n",
        "    return vectorstore\n",
        "\n",
        "\n",
        "vectordb = retrieve_Chroma_vectorstore(splits)\n",
        "query = 'Who are the Ousters?'\n",
        "docs_ss = vectordb.similarity_search(query, k=2)\n",
        "print(docs_ss)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "[Document(page_content='the Ousters. The SDF forces have been running wild. Much of\\nthe carnage could be their doing.\u201d\\n\u201cWith no bodies?\u201d laughed Martin Silenus. \u201cWishful\\nthinking. Our absent hosts downstairs dangle now on the\\nShrike\u2019s steel tree. Where, ere long, we too will be.\u201d\\n\u201cShut up,\u201d Brawne Lamia said tiredly.', metadata={'chapter': 6, 'character': 'None', 'page': 476, 'source': 'docs/pdf/hyperion.pdf', 'story': 'Plot'}), Document(page_content='the Ousters. The SDF forces have been running wild. Much of\\nthe carnage could be their doing.\u201d\\n\u201cWith no bodies?\u201d laughed Martin Silenus. \u201cWishful\\nthinking. Our absent hosts downstairs dangle now on the\\nShrike\u2019s steel tree. Where, ere long, we too will be.\u201d\\n\u201cShut up,\u201d Brawne Lamia said tiredly.', metadata={'chapter': 6, 'character': 'None', 'page': 476, 'source': 'docs/pdf/hyperion.pdf', 'story': 'Plot'})]\n"
        }
      ],
      "execution_count": 3
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "g5vkwfMIU1"
      },
      "source": [
        "# Max Marginal Relevance\n",
        "To introduce diversity in the model responses"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "wNICNzcrFr"
      },
      "source": [
        "docs_mmr = vectordb.max_marginal_relevance_search(query, k=2, fetch_k=5)\n",
        "for i in range(2):\n",
        "    print(f'{i+1}. Similarity search: {docs_ss[i].metadata}')\n",
        "    print(f'{i+1}. Marginal relevance: {docs_mmr[i].metadata}')\n",
        "    print('')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "1. Similarity search: {'chapter': 0, 'character': 'None', 'page': 12, 'source': 'docs/pdf/hyperion.pdf', 'story': 'Plot'}\n1. Marginal relevance: {'chapter': 0, 'character': 'None', 'page': 12, 'source': 'docs/pdf/hyperion.pdf', 'story': 'Plot'}\n\n2. Similarity search: {'chapter': 0, 'character': 'None', 'page': 12, 'source': 'docs/pdf/hyperion.pdf', 'story': 'Plot'}\n2. Marginal relevance: {'chapter': 0, 'character': 'None', 'page': 13, 'source': 'docs/pdf/hyperion.pdf', 'story': 'Plot'}\n\n"
        }
      ],
      "execution_count": 4
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "1xz9Ar6H14"
      },
      "source": [
        "# Self Query\n",
        "Self query allows langchain to auto infer the metadata filter as well as the query which results in better inference overall\n",
        "\n",
        "For example: The query 'What are some movies about aliens made in 1980' would be passed through an llm and divided into te following parts:\n",
        "1. `query`: alien movies\n",
        "2. `metadata filter`: eq(\"year\" 1980)\n",
        "\n",
        "**Note**: Unfortunately, there was no way around using openai ver. 0.28 for the following part. Please downgrade it with the command pip install openai==0.28"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "Rr4bA8pa9t"
      },
      "source": [
        "from langchain.llms import OpenAI\n",
        "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
        "from langchain.chains.query_constructor.base import AttributeInfo\n",
        "\n",
        "metadata_field_info = [\n",
        "    AttributeInfo(\n",
        "        name = 'source',\n",
        "        description = 'File path of the document',\n",
        "        type = 'text',\n",
        "    ),\n",
        "    AttributeInfo(\n",
        "        name = 'chapter',\n",
        "        description = 'Chapter number of the 579-page document ranging from 0-6',\n",
        "        type = 'integer',\n",
        "    ),\n",
        "    AttributeInfo(\n",
        "        name = 'page',\n",
        "        description = 'Page number of the document ranging from 0-579',\n",
        "        type = 'integer',\n",
        "    ),\n",
        "    AttributeInfo(\n",
        "        name = 'story',\n",
        "        description = 'Profession of the character who is narrating their story. Plot is used in the case no character is narrating their story. Possible values are: Priest, Soldier, Poet, Scholar, Detective, Consul, Plot',\n",
        "        type = 'text',\n",
        "    ),\n",
        "    AttributeInfo(\n",
        "        name = 'character',\n",
        "        description = 'Name of the character who is narrating their story. None is used in the case no character is narrating their story. Possible values are: Lenar Hoyt, Fedmahn Kassad, Martin Silenus, Sol Weintraub, Brawne Lamia, Consul, None',\n",
        "        type = 'text',\n",
        "    ),\n",
        "]\n",
        "document_content_description = 'Hyperion book by Dan Simmons'\n",
        "llm = OpenAI(temperature=0)\n",
        "retriever = SelfQueryRetriever.from_llm(\n",
        "    llm,\n",
        "    vectordb,\n",
        "    document_content_description,\n",
        "    metadata_field_info,\n",
        "    search_type='mmr',\n",
        "    search_kwargs={'k': 5},\n",
        "    verbose=True,\n",
        ")\n",
        "query_2 = 'Who did Lenar Hoyt find while narrating his story?'\n",
        "docs_ss_2 = vectordb.similarity_search(query_2, k=2)\n",
        "docs_sqr = retriever.get_relevant_documents(query_2)\n",
        "print('Similarity search:\\n', [doc.metadata['page'] for doc in docs_ss_2])\n",
        "print('Self query retrieval:\\n', [doc.metadata['page'] for doc in docs_sqr])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Similarity search:\n [361, 361]\nSelf query retrieval:\n [37, 37, 35, 43, 41]\n"
        }
      ],
      "execution_count": 5
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "uqX6GlwxiL"
      },
      "source": [
        "# Document Compression\n",
        "\n",
        "With compression, we run all our documents through a language model and extract the most relevant segments of the documents and only pass those through the final llm call\n",
        "\n",
        "### Document Compressor\n",
        "Langchain provides us with numerous document compression options, some of most common ones are:\n",
        "1. `LLMChainExtractor`: Uses the LLM to compress the relevant document to adhere to the context window size\n",
        "2. `DocumentCompressorPipeline`: Allows us to chain multiple document compressors togethe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "jPtNcpzlwK"
      },
      "source": [
        "from langchain.retrievers import ContextualCompressionRetriever\n",
        "from langchain.retrievers.document_compressors import LLMChainExtractor\n",
        "from langchain.retrievers.document_compressors import DocumentCompressorPipeline\n",
        "from langchain.document_transformers import EmbeddingsRedundantFilter\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "llm = OpenAI(temperature=0)\n",
        "compressor_llm = LLMChainExtractor.from_llm(llm)\n",
        "filter = EmbeddingsRedundantFilter(embeddings=embedding)\n",
        "compressor_doc = DocumentCompressorPipeline(transformers=[filter])\n",
        "\n",
        "doc_compression_retriever = ContextualCompressionRetriever(\n",
        "    base_compressor=compressor_doc,\n",
        "    base_retriever=retriever,\n",
        ")\n",
        "llm_compression_retriever = ContextualCompressionRetriever(\n",
        "    base_compressor=compressor_llm,\n",
        "    base_retriever=retriever\n",
        ")\n",
        "\n",
        "query = 'What did Father Dure find strange about the Bikura?'\n",
        "compressed_docs = doc_compression_retriever.get_relevant_documents(query)\n",
        "print([doc.page_content for doc in compressed_docs])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "['glide over the rough ground like wraiths. From a distance,\\ntheir appearance reminded me of nothing so much as a\\ngaggle of diminutive Jesuits at a New Vatican enclave.\\nI almost giggled then, but realized that such a response\\nmight well be a sign of rising panic. The Bikura showed no\\noutward signs of aggression to cause such a panic; they\\ncarried no weapons, their small hands were empty. As empty\\nas their expressions.\\nTheir physiognomy is hard to describe succinctly. They are\\nbald. All of them. That baldness, the absence of any facial\\nhair, and the loose robes that fell in a straight line to the\\nground, all conspired to make it very di\ufb03cult to tell the men\\nfrom the women. The group now confronting me\u2014more than\\n\ufb01fty by this time\u2014looked to be all of roughly the same age:\\nsomewhere between forty and \ufb01fty standard years. Their\\nfaces were smooth, the skin tinged with a yellowish cast that\\nI guessed might be associated with generations of ingesting\\ntrace minerals in the astery chalma and other local plant\\nlife.\\nOne might be tempted to describe the round faces of the\\nBikura as cherubic until, upon closer inspection, that\\nimpression of sweetness fades and is replaced by another\\ninterpretation\u2014placid idiocy. As a priest, I have spent\\nenough time on backward worlds to see the e\ufb00ects of an\\nancient genetic disorder variously called Down\u2019s syndrome,\\nmongolism, or generation-ship legacy. This, then, was the\\noverall impression created by the sixty or so dark-robed little', 'and was radiating it toward us, into us. Then even the cross\\nwas dark and the winds died and in the sudden dimness\\nAlpha said softly, \u201cBring him along.\u201d\\nWe emerged onto the wide ledge of stone and Beta was\\nthere with torches. As Beta passed them out to a selected\\nfew, I wondered if the Bikura reserved \ufb01re for ritual purposes\\nonly. Then Beta was leading the way and we descended the\\nnarrow staircase carved into the stone.\\nAt \ufb01rst I crept along, terri\ufb01ed, clutching at the smooth rock\\nand searching for any reassuring projection of root or stone.\\nThe drop to our right was so sheer and endless that it\\nbordered on being absurd. Descending the ancient staircase', '\u201cWithin a month I found my way to Perecebo Plantation\\nupriver from Port Romance,\u201d he continued, forcing some\\nstrength into his voice. \u201cMy assumption was that the\\n\ufb01berplastic growers might tell me the truth even if they\\nwould have nothing to do with the consulate or Home Rule\\nAuthorities. I was right. The administrator at Perecebo, a\\nman named Orlandi, remembered Father Dur\u00e9, as did\\nOrlandi\u2019s new wife, the woman named Semfa whom Father\\nDur\u00e9 mentioned in his journals. The plantation manager had\\ntried to mount several rescue operations onto the Plateau,\\nbut an unprecedented series of active seasons in the \ufb02ame\\nforests had made them abandon their attempts. After\\nseveral years they had given up hope that Dur\u00e9 or their man\\nTuk could still be alive.\\n\u201cNonetheless, Orlandi recruited two expert bush pilots to\\n\ufb02y a rescue expedition up the Cleft in two plantation\\nskimmers. We stayed in the Cleft itself for as long as we\\ncould, trusting to terrain-avoidance instruments and luck to\\nget us to Bikura country. Even with by-passing most of the\\n\ufb02ame forest that way, we lost one of the skimmers and four\\npeople to tesla activity.\u201d\\nFather Hoyt paused and swayed slightly. Gripping the\\nedge of the table to steady himself, he cleared his throat\\nand said, \u201cThere\u2019s little else to tell. We located the Bikura\\nvillage. There were seventy of them, each as stupid and\\nuncommunicative as Dur\u00e9\u2019s notes had suggested. I managed\\nto ascertain from them that Father Dur\u00e9 had died while', '\u201cFather Dur\u00e9 had died the true death. We returned the\\nremains to the Perecebo Plantation where he was buried\\nfollowing a full funeral Mass.\u201d Hoyt took a deep breath.\\n\u201cOver my strong objections, M. Orlandi destroyed the Bikura\\nvillage and a section of the Cleft wail with shaped nuclear\\ncharges he had brought from the plantation. I do not believe\\nthat any of the Bikura could have survived. As far as we\\ncould tell, the entrance to the labyrinth and the so-called\\nbasilica also must have been destroyed in the landslide.\\n\u201cI had sustained several injuries during the expedition and\\nthus had to remain at the plantation for several months\\nbefore returning to the northern continent and booking\\npassage to Pacem. No one knows of these journals or their\\ncontents except M. Orlandi, Monsignor Edouard, and\\nwhichever of his superiors Monsignor Edouard chose to tell.\\nAs far as I know, the Church has issued no declaration\\nrelating to the journals of Father Paul Dur\u00e9.\u201d\\nFather Hoyt had been standing and now he sat. Sweat\\ndripped from his chin and his face was blue-white in the\\nre\ufb02ected light of Hyperion.\\n\u201cIs that\\xa0\u2026\\xa0all?\u201d asked Martin Silenus.\\n\u201cYes,\u201d managed Father Hoyt.\\n\u201cGentlemen and lady,\u201d said Het Masteen, \u201cit is late. I\\nsuggest that you gather your luggage and rendezvous at our\\nfriend the Consul\u2019s ship on sphere 11 in thirty minutes or\\nsooner. I will be using one of the tree\u2019s dropships to join you\\nlater.\u201d\\nMost of the group was assembled in less than \ufb01fteen', 'Del responded with the type of silence I had come to\\ninterpret as assent.\\nThe pattern seemed clear enough. The Bikura were quite\\nserious about their Three Score and Ten. They kept the tribal\\npopulation at seventy\u2014the same number recorded on the\\npassenger list of the dropship that crashed here four\\nhundred years ago. Little chance of coincidence there. When\\nsomeone died, they allowed a child to be born to replace the\\nadult. Simple.\\nSimple but impossible. Nature and biology do not work\\nthat neatly. Besides the problem of minimum-herd\\npopulation, there were other absurdities. Even though it is\\ndi\ufb03cult to tell the ages of these smooth-skinned people, it is\\nobvious that no more than ten years separates the oldest\\nfrom the youngest. Although they act like children, I would\\nguess their average age to be in the late thirties or mid-\\nforties in standard years. So where are the very old? Where\\nare the parents, aging uncles, and unmarried aunts? At this\\nrate, the entire tribe will enter old age at approximately the\\nsame time. What happens when they all pass beyond\\nchildbearing age and it comes time to replace members of\\nthe tribe?\\nThe Bikura lead dull, sedentary lives. The accident rate\u2014\\neven while living on the very edge of the Cleft\u2014must be low.\\nThere are no predators. The seasonal variations are minimal\\nand the food supply almost certainly remains stable. But,\\ngranted all this, there must have been times in the four-\\nhundred-year history of this ba\ufb04ing group when disease']\n"
        }
      ],
      "execution_count": 6
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jukit_cell_id": "SMMNEGk5YA"
      },
      "source": [
        "# Question Answering\n",
        "\n",
        "This is the final part of the pipeline where we pass the compressed documents through the llm to generate the final answer\n",
        "\n",
        "### Question Answering\n",
        "1. We use all the techniques mentioned above to obtain relevant splits of the document\n",
        "2. We do this by first loading the pdf\n",
        "3. We then preprocess the pdf and the metadata to include additional information\n",
        "4. We then split the document into smaller chunks\n",
        "5. We then vectorize the document\n",
        "6. We use Self Query to efficiently retrieve the relevant documents making use of the metadata\n",
        "7. We then apply Max Marginal Relevance to introduce diversity in the model responses\n",
        "8. We then use Document Compression to compress the relevant documents\n",
        "9. We then pass the compressed documents through the llm to generate the final answer\n",
        "\n",
        "**To generate the final answer, we use a custom template and ChatOpenAI model.**\n",
        "(Here's te reference medium article)[https://medium.com/@onkarmishra/using-langchain-for-question-answering-on-own-data-3af0a82789ed]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "jukit_cell_id": "uZncSN0sTb"
      },
      "source": [
        "from langchain.retrievers import ContextualCompressionRetriever\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.chains import RetrievalQA\n",
        "from langchain.prompts import PromptTemplate\n",
        "\n",
        "template = '''Use the following pieces of context to answer the question at the end. Answer the question in a human-like way. Make sure to be concise with your wording and answer the question directly.\n",
        "{context}\n",
        "Question: {question}\n",
        "Answer:'''\n",
        "\n",
        "QA_CHAIN_PROMPT = PromptTemplate.from_template(template)\n",
        "qa_chain = RetrievalQA.from_chain_type(\n",
        "    ChatOpenAI(temperature=0),\n",
        "    retriever=doc_compression_retriever,\n",
        "    return_source_documents=True,\n",
        "    chain_type_kwargs={'prompt': QA_CHAIN_PROMPT}\n",
        ")\n",
        "\n",
        "query = 'What did Father Dure find strange about the Bikura?'\n",
        "result = qa_chain({'query': query})\n",
        "print(result['result'])\n",
        "print(f'References: {\",\".join([str(doc.metadata[\"page\"]) for doc in result[\"source_documents\"]])}')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Father Dure found it strange that the Bikura did not have any elderly members or parents, aging uncles, and unmarried aunts. He also found it puzzling that the entire tribe would enter old age at the same time and questioned how they would replace members of the tribe when they reached old age.\nReferences: 67,92,115,116,78\n"
        }
      ],
      "execution_count": 7
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "python",
      "language": "python",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}