diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 776df0c..4cfbf0b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -55,4 +55,4 @@ jobs: # AZURE_OPENAI_DEPLOYMENT_NAME: ${{secrets.AZURE_OPENAI_DEPLOYMENT_NAME}} # OPENAI_API_VERSION: ${{secrets.OPENAI_API_VERSION}} run: | - pytest --nbval-lax python-recipes/RAG/00_intro_redispy.ipynb python-recipes/RAG/01_redisvl.ipynb python-recipes/RAG/02_langchain.ipynb python-recipes/RAG/03_llamaindex.ipynb python-recipes/RAG/04_advanced_redisvl.ipynb \ No newline at end of file + pytest --nbval-lax python-recipes/RAG/00_intro_redispy.ipynb python-recipes/RAG/01_redisvl.ipynb python-recipes/RAG/02_langchain.ipynb python-recipes/RAG/03_llamaindex.ipynb python-recipes/RAG/04_advanced_redisvl.ipynb python-recipes/RAG/06_ragas_evaluation.ipynb \ No newline at end of file diff --git a/python-recipes/RAG/06_ragas_evaluation.ipynb b/python-recipes/RAG/06_ragas_evaluation.ipynb new file mode 100644 index 0000000..7c9d0d2 --- /dev/null +++ b/python-recipes/RAG/06_ragas_evaluation.ipynb @@ -0,0 +1,1037 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", + "# Evaluating RAG\n", + "\n", + "This notebook uses the [ragas library](https://docs.ragas.io/en/stable/getstarted/index.html) and [Redis](https://redis.com) to evaluate the performance of sample RAG application. Also see the original [source paper](https://arxiv.org/pdf/2309.15217) to build a more detailed understanding.\n", + "\n", + "## Let's Begin!\n", + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To start, we need a RAG app to evaluate. Let's create one using LangChain and connect it with Redis as the vector DB.\n", + "\n", + "## Init redis, data prep, and populating the vector DB" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "# install deps\n", + "# NBVAL_SKIP\n", + "%pip install -q redis \"unstructured[pdf]\" sentence-transformers langchain langchain-community langchain-openai ragas datasets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Running Redis in Colab\n", + "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "%%sh\n", + "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", + "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", + "sudo apt-get update > /dev/null 2>&1\n", + "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", + "redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### For Alternative Environments\n", + "There are many ways to get the necessary redis-stack instance running\n", + "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", + "own version of Redis Enterprise running, that works too!\n", + "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", + "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "# Replace values below with your own if using Redis Cloud instance\n", + "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", + "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", + "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", + "\n", + "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", + "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", + "from langchain.document_loaders import UnstructuredFileLoader\n", + "\n", + "source_doc = \"resources/nke-10k-2023.pdf\"\n", + "\n", + "loader = UnstructuredFileLoader(\n", + " source_doc, mode=\"single\", strategy=\"fast\"\n", + ")\n", + "\n", + "text_splitter = RecursiveCharacterTextSplitter(\n", + " chunk_size=2500, chunk_overlap=0\n", + ")\n", + "\n", + "chunks = loader.load_and_split(text_splitter)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n", + "\n", + "embeddings = HuggingFaceEmbeddings(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "`index_schema` does not match generated metadata schema.\n", + "If you meant to manually override the schema, please ignore this message.\n", + "index_schema: {'vector': [{'name': 'chunk_vector', 'algorithm': 'HNSW', 'dims': 384, 'distance_metric': 'COSINE', 'datatype': 'FLOAT32'}], 'text': [{'name': 'source'}, {'name': 'content'}], 'content_vector_key': 'chunk_vector'}\n", + "generated_schema: {'text': [{'name': 'source'}], 'numeric': [], 'tag': []}\n", + "\n" + ] + } + ], + "source": [ + "from langchain.vectorstores.redis import Redis as LangChainRedis\n", + "\n", + "\n", + "# set the index name for this example\n", + "index_name = \"ragas_ex\"\n", + "\n", + "# with langchain we can manually modify the default vector schema configuration\n", + "vector_schema = {\n", + " \"name\": \"chunk_vector\", # name of the vector field in langchain\n", + " \"algorithm\": \"HNSW\", # could use HNSW instead\n", + " \"dims\": 384, # set based on the HF model embedding dimension\n", + " \"distance_metric\": \"COSINE\", # could use EUCLIDEAN or IP\n", + " \"datatype\": \"FLOAT32\",\n", + "}\n", + "\n", + "# here we can define the entire schema spec for our index in LangChain\n", + "index_schema = {\n", + " \"vector\": [vector_schema],\n", + " \"text\": [{\"name\": \"source\"}, {\"name\": \"content\"}],\n", + " \"content_vector_key\": \"chunk_vector\" # name of the vector field in langchain\n", + "}\n", + "\n", + "\n", + "# construct the vector store class from texts and metadata\n", + "rds = LangChainRedis.from_documents(\n", + " documents=chunks,\n", + " embedding=embeddings,\n", + " index_name=index_name,\n", + " redis_url=REDIS_URL,\n", + " index_schema=index_schema,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test the vector store" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Document(page_content='As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)', metadata={'id': 'doc:ragas_ex:f593fe9528d44f8c806cb2b1f2c5fede', 'source': 'resources/nke-10k-2023.pdf'})" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rds.similarity_search(\"What was nike's revenue last year?\")[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup RAG\n", + "\n", + "Now that the vector db is populated let's initialize our RAG app." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "import getpass\n", + "from langchain_openai import OpenAI\n", + "\n", + "if \"OPENAI_API_KEY\" not in os.environ:\n", + " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OPENAI_API_KEY\")\n", + "\n", + "llm = OpenAI(openai_api_key=os.getenv(\"OPENAI_API_KEY\"))" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "def get_prompt():\n", + " \"\"\"Create the QA chain.\"\"\"\n", + " from langchain.prompts import PromptTemplate\n", + "\n", + " prompt_template = \"\"\"Use the following pieces of context from financial 10k filings data to answer the user question at the end. If you don't know the answer, say that you don't know, don't try to make up an answer.\n", + "\n", + " This should be in the following format:\n", + "\n", + " Question: [question here]\n", + " Answer: [answer here]\n", + "\n", + " Begin!\n", + "\n", + " Context:\n", + " ---------\n", + " {context}\n", + " ---------\n", + " Question: {question}\n", + " Answer:\"\"\"\n", + "\n", + " prompt = PromptTemplate(\n", + " template=prompt_template,\n", + " input_variables=[\"context\", \"question\"]\n", + " )\n", + " return prompt" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.chains import RetrievalQA\n", + "\n", + "qa = RetrievalQA.from_chain_type(\n", + " llm=llm,\n", + " chain_type=\"stuff\",\n", + " retriever=rds.as_retriever(search_type=\"similarity_distance_threshold\",search_kwargs={\"distance_threshold\":0.5}),\n", + " return_source_documents=True,\n", + " chain_type_kwargs={\"prompt\": get_prompt()},\n", + " #verbose=True\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test it out" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", + "To disable this warning, you can either:\n", + "\t- Avoid using `tokenizers` before the fork if possible\n", + "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" + ] + }, + { + "data": { + "text/plain": [ + "{'query': \"What was nike's revenue last year?\",\n", + " 'result': '\\n$51,217 million.',\n", + " 'source_documents': [Document(page_content='As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)', metadata={'id': 'doc:ragas_ex:f593fe9528d44f8c806cb2b1f2c5fede', 'source': 'resources/nke-10k-2023.pdf'}),\n", + " Document(page_content=\"NIKE, INC. CONSOLIDATED STATEMENTS OF INCOME\\n\\n(In millions, except per share data)\\n\\nRevenues Cost of sales\\n\\nGross profit\\n\\nDemand creation expense Operating overhead expense\\n\\nTotal selling and administrative expense\\n\\nInterest expense (income), net\\n\\nOther (income) expense, net Income before income taxes\\n\\nIncome tax expense NET INCOME\\n\\nEarnings per common share:\\n\\nBasic Diluted\\n\\nWeighted average common shares outstanding:\\n\\nBasic Diluted\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n$\\n\\n$\\n\\n$ $\\n\\nYEAR ENDED MAY 31,\\n\\n2023\\n\\n2022\\n\\n2021\\n\\n51,217 $ 28,925\\n\\n46,710 $ 25,231\\n\\n44,538 24,576\\n\\n22,292 4,060 12,317\\n\\n21,479 3,850 10,954\\n\\n19,962 3,114 9,911\\n\\n16,377 (6)\\n\\n14,804 205\\n\\n13,025 262\\n\\n(280) 6,201\\n\\n(181) 6,651\\n\\n14 6,661\\n\\n1,131 5,070 $\\n\\n605 6,046 $\\n\\n934 5,727\\n\\n3.27 $ 3.23 $\\n\\n3.83 $ 3.75 $\\n\\n3.64 3.56\\n\\n1,551.6 1,569.8\\n\\n1,578.8 1,610.8\\n\\n1,573.0 1,609.4\\n\\n2023 FORM 10-K 55\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED STATEMENTS OF COMPREHENSIVE INCOME\\n\\nYEAR ENDED MAY 31,\\n\\n(Dollars in millions)\\n\\n2023\\n\\n2022\\n\\nNet income Other comprehensive income (loss), net of tax:\\n\\n$\\n\\n5,070 $\\n\\n6,046 $\\n\\nChange in net foreign currency translation adjustment\\n\\n267\\n\\n(522)\\n\\nChange in net gains (losses) on cash flow hedges Change in net gains (losses) on other\\n\\n(348) (6)\\n\\n1,214 6\\n\\nTotal other comprehensive income (loss), net of tax TOTAL COMPREHENSIVE INCOME\\n\\n$\\n\\n(87) 4,983 $\\n\\n698 6,744 $\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n2023 FORM 10-K 56\\n\\n2021\\n\\n5,727\\n\\n496\\n\\n(825) 5\\n\\n(324) 5,403\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED BALANCE SHEETS\\n\\n(In millions)\\n\\nASSETS\\n\\nCurrent assets:\\n\\nCash and equivalents Short-term investments\\n\\nAccounts receivable, net Inventories Prepaid expenses and other current assets\\n\\nTotal current assets\\n\\nProperty, plant and equipment, net\\n\\nOperating lease right-of-use assets, net Identifiable intangible assets, net Goodwill\\n\\nDeferred income taxes and other assets\\n\\nTOTAL ASSETS\\n\\nLIABILITIES AND SHAREHOLDERS' EQUITY Current liabilities:\\n\\nCurrent portion of long-term debt Notes payable Accounts payable\\n\\nCurrent portion of operating lease liabilities Accrued liabilities Income taxes payable\\n\\nTotal current liabilities\\n\\nLong-term debt\\n\\nOperating lease liabilities Deferred income taxes and other liabilities Commitments and contingencies (Note 16)\\n\\nRedeemable preferred stock Shareholders' equity: Common stock at stated value:\", metadata={'id': 'doc:ragas_ex:5a2d1550c0b344119d491680c29006b8', 'source': 'resources/nke-10k-2023.pdf'}),\n", + " Document(page_content=\"Tax (expense) benefit Gain (loss) net of tax\\n\\n5 (14)\\n\\n(9) 22\\n\\nTotal net gain (loss) reclassified for the period\\n\\n$\\n\\n463 $\\n\\n30\\n\\n2023 FORM 10-K 82\\n\\nTable of Contents\\n\\nNOTE 14 — REVENUES\\n\\nDISAGGREGATION OF REVENUES The following tables present the Company's Revenues disaggregated by reportable operating segment, major product line and distribution channel:\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nYEAR ENDED MAY 31, 2023 ASIA PACIFIC & LATIN (1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nAMERICA\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear\\n\\n$\\n\\n14,897 $\\n\\n8,260 $\\n\\n5,435 $\\n\\n4,543 $\\n\\n— $\\n\\n33,135 $\\n\\n2,155 $\\n\\n— $\\n\\n35,290\\n\\nApparel Equipment Other\\n\\n5,947 764 —\\n\\n4,566 592 —\\n\\n1,666 147 —\\n\\n1,664 224 —\\n\\n— — 58\\n\\n13,843 1,727 58\\n\\n90 28 154\\n\\n— — 27\\n\\n13,933 1,755 239\\n\\nTOTAL REVENUES\\n\\n$\\n\\n21,608 $\\n\\n13,418 $\\n\\n7,248 $\\n\\n6,431 $\\n\\n58 $\\n\\n48,763 $\\n\\n2,427 $\\n\\n27 $\\n\\n51,217\\n\\nRevenues by:\\n\\nSales to Wholesale Customers Sales through Direct to Consumer\\n\\n$\\n\\n11,273 $ 10,335\\n\\n8,522 $ 4,896\\n\\n3,866 $ 3,382\\n\\n3,736 $ 2,695\\n\\n— $ —\\n\\n27,397 $ 21,308\\n\\n1,299 $ 974\\n\\n— $ —\\n\\n28,696 22,282\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n21,608 $\\n\\n—\\n\\n13,418 $\\n\\n— 7,248 $\\n\\n— 6,431 $\\n\\n58 58 $\\n\\n58\\n\\n48,763 $\\n\\n154 2,427 $\\n\\n27 27 $\\n\\n239 51,217\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand businesses in its CASA territory to third-party distributors.\\n\\nYEAR ENDED MAY 31, 2022\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nASIA PACIFIC & LATIN AMERICA\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear Apparel\\n\\n$\\n\\n12,228 $ 5,492\\n\\n7,388 $ 4,527\\n\\n5,416 $ 1,938\\n\\n4,111 $ 1,610\\n\\n— $ —\\n\\n29,143 $ 13,567\\n\\n2,094 $ 103\\n\\n— $ —\\n\\n31,237 13,670\\n\\nEquipment Other\\n\\n633 —\\n\\n564 —\\n\\n193 —\\n\\n234 —\\n\\n— 102\\n\\n1,624 102\\n\\n26 123\\n\\n— (72)\\n\\n1,650 153\\n\\nTOTAL REVENUES Revenues by:\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\nSales to Wholesale Customers Sales through Direct to Consumer Other\\n\\n$\\n\\n9,621 $ 8,732 —\\n\\n8,377 $ 4,102 —\\n\\n4,081 $ 3,466 —\\n\\n3,529 $ 2,426 —\\n\\n— $ — 102\\n\\n25,608 $ 18,726 102\\n\\n1,292 $ 931 123\\n\\n— $ — (72)\\n\\n26,900 19,657 153\\n\\nTOTAL REVENUES\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\n2023 FORM 10-K 83\\n\\nTable of Contents\\n\\nYEAR ENDED MAY 31, 2021\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\", metadata={'id': 'doc:ragas_ex:96a581da1bd6466e99179e9d1bef4fe1', 'source': 'resources/nke-10k-2023.pdf'}),\n", + " Document(page_content=\"ASIA PACIFIC & LATIN AMERICA\\n\\n(1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE BRAND\\n\\nCONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by:\\n\\nFootwear Apparel Equipment\\n\\n$\\n\\n11,644 $ 5,028 507\\n\\n6,970 $ 3,996 490\\n\\n5,748 $ 2,347 195\\n\\n3,659 $ 1,494 190\\n\\n— $ — —\\n\\n28,021 $ 12,865 1,382\\n\\n1,986 $ 104 29\\n\\n— $ — —\\n\\n30,007 12,969 1,411\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n17,179 $\\n\\n—\\n\\n11,456 $\\n\\n— 8,290 $\\n\\n— 5,343 $\\n\\n25 25 $\\n\\n25\\n\\n42,293 $\\n\\n86 2,205 $\\n\\n40 40 $\\n\\n151 44,538\\n\\nRevenues by:\\n\\nSales to Wholesale Customers $\\n\\n10,186 $\\n\\n7,812 $\\n\\n4,513 $\\n\\n3,387 $\\n\\n— $\\n\\n25,898 $\\n\\n1,353 $\\n\\n— $\\n\\n27,251\\n\\nSales through Direct to Consumer Other\\n\\n6,993 —\\n\\n3,644 —\\n\\n3,777 —\\n\\n1,956 —\\n\\n— 25\\n\\n16,370 25\\n\\n766 86\\n\\n— 40\\n\\n17,136 151\\n\\nTOTAL REVENUES\\n\\n$\\n\\n17,179 $\\n\\n11,456 $\\n\\n8,290 $\\n\\n5,343 $\\n\\n25 $\\n\\n42,293 $\\n\\n2,205 $\\n\\n40 $\\n\\n44,538\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand business in Brazil to a third-party distributor.\\n\\nFor the fiscal years ended May 31, 2023, 2022 and 2021, Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment. Converse Other revenues were primarily attributable to licensing businesses. Corporate revenues primarily consisted of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse but managed through the Company's central foreign exchange risk management program.\\n\\nAs of May 31, 2023 and 2022, the Company did not have any contract assets and had an immaterial amount of contract liabilities recorded in Accrued liabilities on the Consolidated Balance Sheets.\\n\\nSALES-RELATED RESERVES\\n\\nAs of May 31, 2023 and 2022, the Company's sales-related reserve balance, which includes returns, post-invoice sales discounts and miscellaneous claims, was $994 million and $1,015 million, respectively, recorded in Accrued liabilities on the Consolidated Balance Sheets. The estimated cost of inventory for expected product returns was $226 million and $194 million as of May 31, 2023 and 2022, respectively, and was recorded in Prepaid expenses and other current assets on the Consolidated Balance Sheets.\\n\\nNOTE 15 — OPERATING SEGMENTS AND RELATED INFORMATION\", metadata={'id': 'doc:ragas_ex:43a81ddfeb804ddbbbf2dfabe8bcf453', 'source': 'resources/nke-10k-2023.pdf'})]}" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "query = \"What was nike's revenue last year?\"\n", + "res=qa.invoke(query)\n", + "res" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating a test set\n", + "\n", + "Now that our setup is complete and we have our RAG app to evaluate we need a test set to evaluate against. The ragas library provides a helpful class for generating a synthetic test set given our data as input that we will use here. The output of this generation is a set of `questions`, `contexts`, and `ground_truth`. \n", + "\n", + "The questions are generated by an LLM based on slices of context from the provided doc and the ground_truth is determined via a critic LLM. Note there is nothing special about this data itself and you can provide your own `questions` and `ground_truth` for evaluation purposes. When starting a project however, there is often a lack of quality human labeled data to be used for evaluation and a synthetic dataset is a valuable place to start if pre live user/process data (which should be incorporated as an ultimate goal).\n", + "\n", + "For more detail see [the docs](https://docs.ragas.io/en/stable/concepts/testset_generation.html)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "# source: https://docs.ragas.io/en/latest/getstarted/testset_generation.html\n", + "from ragas.testset.generator import TestsetGenerator\n", + "from ragas.testset.evolutions import simple, reasoning, multi_context\n", + "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n", + "\n", + "# generator with openai models\n", + "generator_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\")\n", + "critic_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\") # can use more advanced model here.\n", + "embeddings = OpenAIEmbeddings()\n", + "\n", + "generator = TestsetGenerator.from_langchain(\n", + " generator_llm,\n", + " critic_llm,\n", + " embeddings\n", + ")\n", + "\n", + "# generate testset\n", + "testset = generator.generate_with_langchain_docs(chunks, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Check testset output\n", + "\n", + "We will save this to a file as well as good practice to not constantly be regenerating examples." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
questioncontextsground_truthevolution_typemetadataepisode_done
0What potential consequences could securities c...[Our Class B Common Stock is traded publicly, ...Any litigation could result in reputational da...simple[{'source': 'resources/nke-10k-2023.pdf'}]True
1How do currency exchange rates impact the fina...[Economic factors beyond our control, and chan...Currency exchange rate fluctuations could disr...simple[{'source': 'resources/nke-10k-2023.pdf'}]True
2Why are cost-effective investments considered ...[From time to time, we may invest in technolog...Cost-effective investments are considered esse...simple[{'source': 'resources/nke-10k-2023.pdf'}]True
3What changes were made to the U.S. corporate i...[FISCAL 2023 COMPARED TO FISCAL 2022\\n\\nOther ...The Inflation Reduction Act of 2022 made chang...simple[{'source': 'resources/nke-10k-2023.pdf'}]True
4How does master netting arrangements impact re...[The Company records the assets and liabilitie...The Company's derivative financial instruments...multi_context[{'source': 'resources/nke-10k-2023.pdf'}]True
\n", + "
" + ], + "text/plain": [ + " question \\\n", + "0 What potential consequences could securities c... \n", + "1 How do currency exchange rates impact the fina... \n", + "2 Why are cost-effective investments considered ... \n", + "3 What changes were made to the U.S. corporate i... \n", + "4 How does master netting arrangements impact re... \n", + "\n", + " contexts \\\n", + "0 [Our Class B Common Stock is traded publicly, ... \n", + "1 [Economic factors beyond our control, and chan... \n", + "2 [From time to time, we may invest in technolog... \n", + "3 [FISCAL 2023 COMPARED TO FISCAL 2022\\n\\nOther ... \n", + "4 [The Company records the assets and liabilitie... \n", + "\n", + " ground_truth evolution_type \\\n", + "0 Any litigation could result in reputational da... simple \n", + "1 Currency exchange rate fluctuations could disr... simple \n", + "2 Cost-effective investments are considered esse... simple \n", + "3 The Inflation Reduction Act of 2022 made chang... simple \n", + "4 The Company's derivative financial instruments... multi_context \n", + "\n", + " metadata episode_done \n", + "0 [{'source': 'resources/nke-10k-2023.pdf'}] True \n", + "1 [{'source': 'resources/nke-10k-2023.pdf'}] True \n", + "2 [{'source': 'resources/nke-10k-2023.pdf'}] True \n", + "3 [{'source': 'resources/nke-10k-2023.pdf'}] True \n", + "4 [{'source': 'resources/nke-10k-2023.pdf'}] True " + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# NBVAL_SKIP\n", + "testset_df = testset.to_pandas()\n", + "testset_df.to_csv(\"resources/testset.csv\", index=False)\n", + "testset_df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Evaluation helper functions\n", + "\n", + "The following code takes a RetrievalQA chain, testset dataframe, and the metrics to be evaluated and returns a dataframe including the metrics calculated." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "# define reusable helper function for evaluating our test set against different chains\n", + "import pandas as pd\n", + "from datasets import Dataset\n", + "from ragas import evaluate\n", + "\n", + "def parse_contexts(source_docs):\n", + " return [doc.page_content for doc in source_docs]\n", + "\n", + "def create_evaluation_dataset(chain: RetrievalQA, testset: pd.DataFrame) -> dict:\n", + " res_set = {\n", + " \"question\": [],\n", + " \"answer\": [],\n", + " \"contexts\": [],\n", + " \"ground_truth\": []\n", + " }\n", + "\n", + " for _, row in testset.iterrows():\n", + " # call QA chain\n", + " result = chain.invoke(row[\"question\"])\n", + "\n", + " res_set[\"question\"].append(row[\"question\"])\n", + " res_set[\"answer\"].append(result[\"result\"])\n", + " res_set[\"contexts\"].append(parse_contexts(result[\"source_documents\"]))\n", + " res_set[\"ground_truth\"].append(str(row[\"ground_truth\"]))\n", + "\n", + " return res_set\n", + "\n", + "def evaluate_chain(chain: RetrievalQA, testset: pd.DataFrame, test_name: str, metrics: list):\n", + " eval_dataset = create_evaluation_dataset(chain, testset)\n", + "\n", + " parsed = Dataset.from_dict(eval_dataset)\n", + "\n", + " eval_result = evaluate(\n", + " parsed,\n", + " metrics=metrics\n", + " )\n", + "\n", + " eval_df = eval_result.to_pandas()\n", + " # store the results of our test for future reference in csv\n", + " eval_df.to_csv(f\"{test_name}.csv\")\n", + " return eval_df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# First let's evaluate generation metrics\n", + "Generation metrics quantify how well the RAG app did creating answers to the provided questions (i.e. the G in **R**etrival **A**ugments **G**eneration). We will calculate the generation metrics **faithfulness** and **answer relevancy** for this example.\n", + "\n", + "The ragas libary conveniently abstracts the calculation of these metrics so we don't have to write redundant code but please review the following definitions in order to build intuition around what these metrics actually measure.\n", + "\n", + "Note: the following examples are paraphrased from the [ragas docs](https://docs.ragas.io/en/stable/concepts/metrics/index.html)\n", + "\n", + "------\n", + "\n", + "### Faithfulness\n", + "\n", + "An answer to a question can be said to be \"faithful\" if the **claims** that are made in the answer **can be inferred** from the **context**.\n", + "\n", + "#### Mathematically:\n", + "\n", + "$$\n", + "Faithfullness\\ score = \\frac{Number\\ of\\ claims\\ in\\ the\\ generated\\ answer\\ that\\ can\\ be\\ inferred\\ from\\ the\\ given\\ context}{Total\\ number\\ of\\ claim\\ in\\ the\\ generated\\ answer}\n", + "$$\n", + "\n", + "#### Example process:\n", + "\n", + "> Question: Where and when was Einstein born?\n", + "> \n", + "> Context: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time\n", + ">\n", + "> answer: Einstein was born in Germany on 20th March 1879.\n", + "\n", + "Step 1: Use LLM to break generated answer into individual statements.\n", + "- “Einstein was born in Germany.”\n", + "- “Einstein was born on 20th March 1879.”\n", + "\n", + "Step 2: For each statement use LLM to verify if it can be inferred from the context.\n", + "- “Einstein was born in Germany.” => yes. \n", + "- “Einstein was born on 20th March 1879.” => no.\n", + "\n", + "Step 3: plug into formula\n", + "\n", + "Number of claims inferred from context = 1\n", + "Total number of claims = 2\n", + "Faithfulness = 1/2\n", + "\n", + "### Answer Relevance\n", + "\n", + "An answer can be said to be relevant if it directly addresses the question (intuitively).\n", + "\n", + "#### Example process:\n", + "\n", + "1. Use an LLM to generate \"hypothetical\" questions to a given answer with the following prompt:\n", + "\n", + " > Generate a question for the given answer.\n", + " > answer: [answer]\n", + "\n", + "2. Embed the generated \"hypothetical\" questions as vectors.\n", + "3. Calculate the cosine similarity of the hypothetical questions and the original question, sum those similarities, and divide by n.\n", + "\n", + "With data:\n", + "\n", + "> Question: Where is France and what is it’s capital?\n", + "> \n", + "> answer: France is in western Europe.\n", + "\n", + "Step 1 - use LLM to create 'n' variants of question from the generated answer.\n", + "\n", + "- “In which part of Europe is France located?”\n", + "- “What is the geographical location of France within Europe?”\n", + "- “Can you identify the region of Europe where France is situated?”\n", + "\n", + "Step 2 - Calculate the mean cosine similarity between the generated questions and the actual question.\n", + "\n", + "## Now let's implement using our helper functions\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "4956c411b3094d6895658b730cf9fae9", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Evaluating: 0%| | 0/12 [00:00\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
answer_relevancyfaithfulness
count6.0000006.000000
mean0.9559670.958333
std0.0397100.102062
min0.9011410.750000
25%0.9304291.000000
50%0.9596361.000000
75%0.9859291.000000
max1.0000001.000000
\n", + "" + ], + "text/plain": [ + " answer_relevancy faithfulness\n", + "count 6.000000 6.000000\n", + "mean 0.955967 0.958333\n", + "std 0.039710 0.102062\n", + "min 0.901141 0.750000\n", + "25% 0.930429 1.000000\n", + "50% 0.959636 1.000000\n", + "75% 0.985929 1.000000\n", + "max 1.000000 1.000000" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# NBVAL_SKIP\n", + "gen_basic_rag_test.describe()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Next let's evaluate the retrieval metrics\n", + "\n", + "Retrieval metrics quantify how well the system performed at fetching the best possible context for generation. Like before please review the definitions below to understand what happens under-the-hood when we execute the evaluation code. \n", + "\n", + "-----\n", + "\n", + "### Context Relevance\n", + "\n", + "\"The context is considered relevant to the extent that it exclusively contains information that is needed to answer the question.\"\n", + "\n", + "#### Example process:\n", + "\n", + "1. Use the following LLM prompt to extract a subset of sentences necessary to answer the question. The context is defined as the formatted search result from the vector database.\n", + "\n", + " > Please extract relevant sentences from\n", + " > the provided context that can potentially\n", + " > help answer the following `{question}`. If no\n", + " > relevant sentences are found, or if you\n", + " > believe the question cannot be answered\n", + " > from the given context, return the phrase\n", + " > \"Insufficient Information\". While extracting candidate sentences you’re not allowed to make any changes to sentences\n", + " > from given `{context}`.\n", + "\n", + "2. Compute the context relevance score = (number of extracted sentences) / (total number of sentences in context)\n", + "\n", + "Moving from the initial paper to the active evaluation library ragas there are a few more insightful metrics to evaluate. From the library [source](https://docs.ragas.io/en/stable/concepts/metrics/index.html) let's introduce `context precision` and `context recall`. \n", + "\n", + "### Context recall\n", + "Context can be said to have high recall if retrieved context aligns with the ground truth answer.\n", + "\n", + "#### Mathematically:\n", + "\n", + "$$\n", + "Context\\ recall = \\frac{Ground\\ Truth\\ sentences\\ that\\ can\\ be\\ attributed\\ to\\ context}{Total\\ number\\ of\\ sentences\\ in\\ the\\ ground\\ truth}\n", + "$$\n", + "\n", + "#### Example process:\n", + "\n", + "Data:\n", + "> question: Where is France and what is it’s capital?\n", + "> ground truth answer: France is in Western Europe and its capital is Paris.\n", + "> context: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history.\n", + ">\n", + "> Note: ground truth answer can be created by critic LLM of with own human labeled data set.\n", + "\n", + "Step 1 - use an LLM to break the ground truth down into individual statements:\n", + "- `France is in Western Europe`\n", + "- `Its capital is Paris`\n", + "\n", + "Step 2 - for each ground truth statement, use an LLM to determine if it can be attributed from the context.\n", + "- `France is in Western Europe` => yes\n", + "- `Its capital is Paris` => no\n", + "\n", + "\n", + "Step 3 - plug in to formula\n", + "\n", + "context recall = (1 + 0) / 2 = 0.5\n", + "\n", + "### Context precision\n", + "\n", + "This metrics relates to how chunks are ranked in a response. Ideally the most relevant chunks are at the top.\n", + "\n", + "#### Mathematically:\n", + "\n", + "$$\n", + "Context\\ Precision@k = \\frac{precision@k}{total\\ number\\ relevant\\ items\\ in\\ the\\ top\\ k\\ results}\n", + "$$\n", + "\n", + "$$\n", + "Precision@k = \\frac{true\\ positive@k}{true\\ positives@k + false\\ positives@k}\n", + "$$\n", + "\n", + "#### Example process:\n", + "\n", + "Data:\n", + "> Question: Where is France and what is it’s capital?\n", + "> \n", + "> Ground truth: France is in Western Europe and its capital is Paris.\n", + "> \n", + "> Context: [ “The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and”, “France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower”]\n", + "\n", + "Step 1 - for each chunk use the LLM to check if it's relevant or not to the ground truth answer.\n", + "\n", + "Step 2 - for each chunk in the context calculate the precision defined as: ``\n", + "- `“The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and”` => precision = 0/1 or 0.\n", + "- `“France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower”` => the precision would be (1) / (1 true positive + 1 false positive) = 0.5. \n", + "\n", + "\n", + "Step 3 - calculate the overall context precision = (0 + 0.5) / 1 = 0.5" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "1cc95ba38ffd4fe7bc76a3b33334e8e6", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Evaluating: 0%| | 0/12 [00:00\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
context_precisioncontext_recall
count6.0000006.000000
mean0.8333330.666667
std0.4082480.516398
min0.0000000.000000
25%1.0000000.250000
50%1.0000001.000000
75%1.0000001.000000
max1.0000001.000000
\n", + "" + ], + "text/plain": [ + " context_precision context_recall\n", + "count 6.000000 6.000000\n", + "mean 0.833333 0.666667\n", + "std 0.408248 0.516398\n", + "min 0.000000 0.000000\n", + "25% 1.000000 0.250000\n", + "50% 1.000000 1.000000\n", + "75% 1.000000 1.000000\n", + "max 1.000000 1.000000" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# NBVAL_SKIP\n", + "ret_basic_rag_test.describe()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Review\n", + "\n", + "- we initialized our RAG app with data from a 10k document\n", + "- generated a testset to evaluate \n", + "- calculated both retrieval and generation metrics\n", + "\n", + "## Analysis\n", + "- The generation metrics both scored >95% in on our small example. This means 2 things: 1) answers generated were fairly reliably attributable to the context provided. 2) the hypothetical question that could be asked of our generated answers align with the actual question asked.\n", + "- The retrieval metrics reveal the most to be improved. Looking at the data generated, we can see that in retrieval we fetched a lot of information that is not required for answering the questions provided. This indicates we may want to experiment with a smaller chunking size or creating dense propositions from the initial chunks to improve the quality of retrieval.\n", + "\n", + "## Next steps\n", + "\n", + "Now that we know how to measure our system we can quickly and easily experiment with different techniques with a baseline in place to improve our systems. " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/python-recipes/RAG/resources/generation_basic_rag_test.csv b/python-recipes/RAG/resources/generation_basic_rag_test.csv new file mode 100644 index 0000000..9eef310 --- /dev/null +++ b/python-recipes/RAG/resources/generation_basic_rag_test.csv @@ -0,0 +1,14 @@ +,question,answer,contexts,ground_truth,answer_relevancy,faithfulness +0,What potential consequences could securities class action litigation have on a company's operations and reputation?," Securities class action litigation could result in reputational damage, substantial costs, and a diversion of management's attention and resources needed to successfully run the business.","[""Our Class B Common Stock is traded publicly, and at any given time various securities analysts follow our financial results and issue reports on us. These reports include information about our historical financial results as well as analysts' opinions of our future performance, which may, in part, be based upon any guidance we have provided. Analysts' estimates are often different from our estimates or expectations. If our operating results are below the estimates or expectations of public market analysts and investors, our stock price could decline. In the past, securities class action litigation has been brought against NIKE and other companies following a decline in the market price of their securities. If our stock price is volatile for any reason, we may become involved in this type of litigation in the future. Any litigation could result in reputational damage, substantial costs and a diversion of management's attention and resources needed to successfully run our business.\n\n2023 FORM 10-K 23\n\nTable of Contents\n\nITEM 1B. UNRESOLVED STAFF COMMENTS\n\nNone.\n\nITEM 2. PROPERTIES\n\nThe following is a summary of principal properties owned or leased by NIKE:\n\nThe NIKE World Campus, owned by NIKE and located near Beaverton, Oregon, USA, is an approximately 400-acre site consisting of over 40 buildings which, together with adjacent leased properties, functions as our world headquarters and is occupied by approximately 11,400 employees engaged in management, research, design, development, marketing, finance and other administrative functions serving nearly all of our segments. We lease a similar, but smaller, administrative facility in Hilversum, the Netherlands, which serves as the headquarters for our Europe, Middle East & Africa geography and management of certain brand functions for our non-U.S. operations. We also lease an office complex in Shanghai, China, our headquarters for our Greater China geography, occupied by employees focused on implementing our wholesale, NIKE Direct and merchandising strategies in the region, among other functions.""]","Any litigation could result in reputational damage, substantial costs and a diversion of management's attention and resources needed to successfully run our business.",0.9344258387584756,1.0 +1,How do currency exchange rates impact the financial results of the company?," Currency exchange rates can have a significant impact on the company's financial results as a majority of their products are manufactured and sold outside of the United States. Fluctuations in inflation and currency exchange rates can result in lower revenues, higher costs, and decreased margins and earnings. The company hedges certain foreign currency exposures to lessen the impact of these fluctuations, but they still have a significant impact on the company's reported results. The translation of foreign subsidiaries' balance sheets and operational results into U.S. Dollars for consolidated reporting can also create volatility in the company's reported results.","[""Economic factors beyond our control, and changes in the global economic environment, including fluctuations in inflation and currency exchange rates, could result in lower revenues, higher costs and decreased margins and earnings.\n\nA majority of our products are manufactured and sold outside of the United States, and we conduct purchase and sale transactions in various currencies, which creates exposure to the volatility of global economic conditions, including fluctuations in inflation and foreign currency exchange rates. Central banks may deploy various strategies to combat inflation, including increasing interest rates, which may impact our borrowing costs. Additionally, there has been, and may continue to be, volatility in currency exchange rates that impact the U.S. Dollar value relative to other international currencies. Our international revenues and expenses generally are derived from sales and operations in foreign currencies, and these revenues and expenses are affected by currency fluctuations, specifically amounts recorded in foreign currencies and translated into U.S. Dollars for consolidated financial reporting, as weakening of foreign currencies relative to the U.S. Dollar adversely affects the U.S. Dollar value of the Company's foreign currency-denominated sales and earnings. Currency exchange rate fluctuations could also disrupt the business of the independent manufacturers that produce our products by making their purchases of raw materials more expensive and more difficult to finance. Foreign currency fluctuations have adversely affected and could continue to have an adverse effect on our results of operations and financial condition.\n\nWe hedge certain foreign currency exposures to lessen and delay, but not to completely eliminate, the effects of foreign currency fluctuations on our financial results. Since the hedging activities are designed to lessen volatility, they not only reduce the negative impact of a stronger U.S. Dollar or other trading currency, but they also reduce the positive impact of a weaker U.S. Dollar or other trading currency. Our future financial results have in the past been and could in the future be significantly affected by the value of the U.S. Dollar in relation to the foreign currencies in which we conduct business. The degree to which our financial results are affected for any given time period will depend in part upon our hedging activities."" + 'Many of our foreign subsidiaries operate in functional currencies other than the U.S. Dollar. Fluctuations in currency exchange rates create volatility in our reported results as we are required to translate the balance sheets, operational results and cash flows of these subsidiaries into U.S. Dollars for consolidated reporting. The translation of foreign subsidiaries\' non-U.S. Dollar denominated balance sheets into U.S. Dollars for consolidated reporting results in a cumulative translation adjustment to Accumulated other comprehensive income (loss) within Shareholders\' equity. In the translation of our Consolidated Statements of Income, a weaker U.S. Dollar in relation to foreign functional currencies benefits our consolidated earnings whereas a stronger U.S. Dollar reduces our consolidated earnings. The impact of foreign exchange rate fluctuations on the translation of our consolidated Revenues was a detriment of approximately $2,859 million, $295 million and a benefit of approximately $893 million for the years ended May 31, 2023, 2022 and 2021, respectively. The impact of foreign exchange rate fluctuations on the translation of our Income before income taxes was a detriment of approximately $824 million, $87 million and a benefit of approximately $260 million for the years ended May 31, 2023, 2022 and 2021, respectively.\n\nMANAGING TRANSLATIONAL EXPOSURES To minimize the impact of translating foreign currency denominated revenues and expenses into U.S. Dollars for consolidated reporting, certain foreign subsidiaries use excess cash to purchase U.S. Dollar denominated available-for-sale investments. The variable future cash flows associated with the purchase and subsequent sale of these U.S. Dollar denominated investments at non-U.S. Dollar functional currency subsidiaries creates a foreign currency exposure that qualifies for hedge accounting under generally accepted accounting principles in the United States of America (""U.S. GAAP""). We utilize forward contracts and/or options to mitigate the variability of the forecasted future purchases and sales of these U.S. Dollar investments. The combination of the purchase and sale of the U.S. Dollar investment and the hedging instrument has the effect of partially offsetting the year-over-year foreign currency translation impact on net earnings in the period the investments are sold. Hedges of the purchase of U.S. Dollar denominated available-for-sale investments are accounted for as cash flow hedges.' + ""As part of the Company's centrally managed foreign exchange risk management program, standard foreign currency rates are assigned twice per year to each NIKE Brand entity in the Company's geographic operating segments and to Converse. These rates are set approximately nine and twelve months in advance of the future selling seasons to which they relate (specifically, for each currency, one standard rate applies to the fall and holiday selling seasons, and one standard rate applies to the spring and summer selling seasons) based on average market spot rates in the calendar month preceding the date they are established. Inventories and Cost of sales for geographic operating segments and Converse reflect the use of these standard rates to record non-functional currency product purchases in the entity's functional currency. Differences between assigned standard foreign currency rates and actual market rates are included in Corporate, together with foreign currency hedge gains and losses generated from the Company's centrally managed foreign exchange risk management program and other conversion gains and losses.\n\nAccounts receivable, net, Inventories and Property, plant and equipment, net for operating segments are regularly reviewed by management and are therefore provided below.\n\n2023 FORM 10-K 85\n\nTable of Contents\n\n(Dollars in millions)\n\nREVENUES\n\nNorth America Europe, Middle East & Africa Greater China\n\nAsia Pacific & Latin America Global Brand Divisions\n\nTotal NIKE Brand\n\nConverse Corporate\n\nTOTAL NIKE, INC. REVENUES\n\nEARNINGS BEFORE INTEREST AND TAXES\n\nNorth America Europe, Middle East & Africa\n\nGreater China Asia Pacific & Latin America Global Brand Divisions\n\nConverse Corporate\n\nInterest expense (income), net\n\nTOTAL NIKE, INC. INCOME BEFORE INCOME TAXES\n\nADDITIONS TO PROPERTY, PLANT AND EQUIPMENT\n\nNorth America Europe, Middle East & Africa Greater China\n\nAsia Pacific & Latin America Global Brand Divisions\n\nTotal NIKE Brand\n\nConverse Corporate\n\nTOTAL ADDITIONS TO PROPERTY, PLANT AND EQUIPMENT\n\nDEPRECIATION North America Europe, Middle East & Africa\n\nGreater China Asia Pacific & Latin America Global Brand Divisions\n\nTotal NIKE Brand\n\nConverse\n\nCorporate\n\nTOTAL DEPRECIATION\n\n2023 FORM 10-K 86\n\n$\n\n$\n\n$\n\n$\n\n$\n\n$\n\n$\n\n$\n\nYEAR ENDED MAY 31,\n\n2023\n\n2022\n\n21,608 $ 13,418 7,248\n\n18,353 $ 12,479 7,547\n\n6,431 58\n\n5,955 102\n\n48,763 2,427 27\n\n44,436 2,346 (72)\n\n51,217 $\n\n46,710 $\n\n5,454 $ 3,531\n\n5,114 $ 3,293\n\n2,283 1,932 (4,841)\n\n2,365 1,896 (4,262)"" + 'recognized upon completion of the sale our entities in Argentina and Uruguay to a third-party distributor in the second quarter of fiscal 2023. This was partially offset by the one-time charge related to the deconsolidation of our Russian operations recognized in the prior year, with the net amount of these activities reported as a component of consolidated Other (income) expense, net; and\n\na favorable change in net foreign currency gains and losses of $174 million related to the remeasurement of monetary assets and liabilities denominated in non- functional currencies and the impact of certain foreign currency derivative instruments, reported as a component of consolidated Other (income) expense, net.\n\nFOREIGN CURRENCY EXPOSURES AND HEDGING PRACTICES\n\nOVERVIEW\n\nAs a global company with significant operations outside the United States, in the normal course of business we are exposed to risk arising from changes in currency exchange rates. Our primary foreign currency exposures arise from the recording of transactions denominated in non-functional currencies and the translation of foreign currency denominated results of operations, financial position and cash flows into U.S. Dollars.']",Currency exchange rate fluctuations could disrupt the business of the independent manufacturers that produce our products by making their purchases of raw materials more expensive and more difficult to finance. Foreign currency fluctuations have adversely affected and could continue to have an adverse effect on our results of operations and financial condition.,0.9848463917711548,1.0 +2,Why are cost-effective investments considered essential for business growth and profitability?," Cost-effective investments are considered essential for business growth and profitability because they allow companies to allocate their resources efficiently, maximize their return on investment, and reduce their overall expenses. By making smart and strategic investments, companies can increase their revenue, expand their operations, and stay competitive in the market. This ultimately leads to growth and higher profitability for the business.",[],"Cost-effective investments are considered essential for business growth and profitability because they contribute to the development of new businesses, expansion of existing businesses, and the improvement of product offerings and manufacturing innovation. These investments help drive growth and increase profitability. However, significant investments also come with risks and uncertainties.",0.9999999999999997,1.0 +3,What changes were made to the U.S. corporate income tax system by the Inflation Reduction Act of 2022?," The Inflation Reduction Act of 2022 made changes to the U.S. corporate income tax system, including revisions to profit allocation and nexus rules and the implementation of a global corporate minimum tax rate of 15% as proposed by the OECD and G20 Inclusive Framework on Base Erosion and Profit Shifting.","['Proposals to reform U.S. and foreign tax laws could significantly impact how U.S. multinational corporations are taxed on global earnings and could increase the U.S. corporate tax rate. For example, the Organization for Economic Co-operation and Development (OECD) and the G20 Inclusive Framework on Base Erosion and Profit Shifting (the ""Inclusive Framework"") has put forth two proposals—Pillar One and Pillar Two—that revise the existing profit allocation and nexus rules and ensure a minimal level of taxation, respectively. On December 12, 2022, the European Union member states agreed to implement the Inclusive Framework\'s global corporate minimum tax rate of 15%. Other countries are also actively considering changes to their tax laws to adopt certain parts of the Inclusive Framework\'s proposals. Although we cannot predict whether or in what form these proposals will be enacted into law, these changes, if enacted into law, could have an adverse impact on our effective tax rate, income tax expense and cash flows.\n\nPortions of our operations are subject to a reduced tax rate or are under a tax holiday. We also utilize tax rulings and other agreements to obtain certainty in treatment of certain tax matters. Tax holidays and rulings can expire from time to time and may be extended when certain conditions are met, or terminated if certain conditions are not met. The impact of any changes in conditions would be the loss of certainty in treatment thus potentially impacting our effective income tax rate. For example, in January 2019, the European Commission opened a formal investigation to examine whether the Netherlands has breached State Aid rules when granting certain tax rulings to the Company. If this matter is adversely resolved, the Netherlands may be required to assess additional amounts with respect to prior periods, and the Company\'s income taxes related to prior periods in the Netherlands could increase.' + 'A reconciliation from the U.S. statutory federal income tax rate to the effective income tax rate is as follows:\n\nYEAR ENDED MAY 31,\n\n2023\n\n2022\n\nFederal income tax rate\n\n21.0 %\n\n21.0 %\n\nState taxes, net of federal benefit Foreign earnings Subpart F deferred tax benefit\n\n1.5 % 1.7 % 0.0 %\n\n1.4 % -1.8 % -4.7 %\n\nForeign-derived intangible income benefit Excess tax benefits from stock-based compensation Income tax audits and contingency reserves\n\n6.1 % -1.1 % 1.0 %\n\n4.1 % -4.9 % 1.5 %\n\nU.S. research and development tax credit Other, net\n\nEFFECTIVE INCOME TAX RATE\n\n1.2 % 1.4 %\n\n18.2 %\n\n1.0 % 1.7 % 9.1 %\n\nOn December 22, 2017, the U.S. enacted the Tax Cuts and Jobs Act (the ""Tax Act""), which significantly changed U.S. tax law and included a provision to tax global intangible low-taxed income (""GILTI"") of foreign subsidiaries. The Company recognizes taxes due under the GILTI provision as a current period expense.\n\nThe effective tax rate for the fiscal year ended May 31, 2023 was higher than the effective tax rate for the fiscal year ended May 31, 2022. The increase was primarily due to decreased benefits from stock-based compensation and the prior year recognition of a non-cash, one-time benefit related to the onshoring of the Company\'s non-U.S. intangible property. During the fourth quarter of fiscal 2022, the Company onshored certain non-U.S. intangible property ownership rights and implemented changes in the Company\'s legal entity structure. The tax restructuring increases the possibility that foreign earnings in future periods will be subject to tax in the U.S. due to Subpart F of the Internal Revenue Code. The Company recognized a deferred tax asset and corresponding non-cash deferred income tax benefit of 4.7%, to establish the deferred tax deduction that is expected to reduce taxable income in future periods.\n\n2023 FORM 10-K 72\n\n2021\n\n5,723 938\n\n6,661\n\n2021\n\n328 134\n\n857 1,319\n\n(371)\n\n(34) 20\n\n(385) 934\n\n2021\n\n21.0 %\n\n1.3 % 0.2 % 0.0 %\n\n3.7 % -4.5 % 1.5 %\n\n0.9 % -0.9 %\n\n14.0 %\n\nTable of Contents\n\nThe effective tax rate for the fiscal year ended May 31, 2022 was lower than the effective tax rate for the fiscal year ended May 31, 2021. The decrease was primarily due to a shift in the Company\'s earnings mix and recognition of a non-cash, one-time benefit related to the onshoring of the Company\'s non-U.S. intangible property.\n\nDeferred tax assets and liabilities comprise the following as of:\n\nMAY 31,\n\n(Dollars in millions)\n\n2023\n\n2022\n\nDeferred tax assets:']","The Inflation Reduction Act of 2022 made changes to the U.S. corporate income tax system, including the implementation of a fifteen percent minimum tax based on 'adjusted financial statement income', effective for NIKE beginning June 1, 2023.",0.9862903842664424,0.75 +4,"How does master netting arrangements impact recording of assets and liabilities for derivative financial instruments, and how are cash collateral amounts for credit-related contingent features accounted for?"," Master netting arrangements allow for the offset of assets and liabilities in the event of default or early termination of the contract. Any amounts of cash collateral received or posted related to derivative instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents, Accrued liabilities, and Prepaid expenses and other current assets and may further offset against the Company's derivative asset or liability balances. The impact of master netting arrangements and cash collateral amounts is disclosed in the Company's financial statements and notes to the financial statements.","[""The Company records the assets and liabilities of its derivative financial instruments on a gross basis on the Consolidated Balance Sheets. The Company's derivative financial instruments are subject to master netting arrangements that allow for the offset of assets and liabilities in the event of default or early termination of the contract. Any amounts of cash collateral received related to these instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents and Accrued liabilities, the latter of which would further offset against the Company's derivative asset balance. Any amounts of cash collateral posted related to these instruments associated with the Company's credit-related contingent features are recorded in Prepaid expenses and other current assets, which would further offset against the Company's derivative liability balance. Cash collateral received or posted related to the Company's credit-related contingent features is presented in the Cash provided by operations component of the Consolidated Statements of Cash Flows. The Company does not recognize amounts of non-cash collateral received, such as securities, on the Consolidated Balance Sheets. For further information related to credit risk, refer to Note 12 — Risk Management and Derivatives.\n\n2023 FORM 10-K 68\n\nTable of Contents\n\nThe following tables present information about the Company's derivative assets and liabilities measured at fair value on a recurring basis and indicate the level in the fair value hierarchy in which the Company classifies the fair value measurement:\n\nMAY 31, 2023\n\nDERIVATIVE ASSETS\n\nDERIVATIVE LIABILITIES\n\n(Dollars in millions)\n\nASSETS AT FAIR VALUE\n\nOTHER CURRENT ASSETS\n\nOTHER LONG- TERM ASSETS\n\nLIABILITIES AT FAIR VALUE\n\nACCRUED LIABILITIES\n\nOTHER LONG- TERM LIABILITIES\n\nLevel 2:\n\nForeign exchange forwards and options\n\n(1)\n\n$\n\n557 $\n\n493 $\n\n64 $\n\n180 $\n\n128 $\n\n52\n\n(1)\n\nIf the foreign exchange derivative instruments had been netted on the Consolidated Balance Sheets, the asset and liability positions each would have been reduced by $178 million as of May 31, 2023. As of that date, the Company received $36 million of cash collateral from various counterparties related to foreign exchange derivative instruments. No amount of collateral was posted on the derivative liability balance as of May 31, 2023.\n\nMAY 31, 2022\n\nDERIVATIVE ASSETS\n\nDERIVATIVE LIABILITIES\n\n(Dollars in millions)\n\nASSETS AT FAIR VALUE"" + ""The Company's derivative contracts contain credit risk-related contingent features designed to protect against significant deterioration in counterparties' creditworthiness and their ultimate ability to settle outstanding derivative contracts in the normal course of business. The Company's bilateral credit-related contingent features generally require the owing entity, either the Company or the derivative counterparty, to post collateral for the portion of the fair value in excess of $50 million should the fair value of outstanding derivatives per counterparty be greater than $50 million. Additionally, a certain level of decline in credit rating of either the Company or the counterparty could trigger collateral requirements. As of May 31, 2023, the Company was in compliance with all credit risk-related contingent features, and derivative instruments with such features were in a net liability position of approximately $2 million. Accordingly, the Company posted no cash collateral as a result of these contingent features. Further, as of May 31, 2023, the Company had received $36 million in cash collateral from various counterparties to its derivative contracts. The Company considers the impact of the risk of counterparty default to be immaterial.\n\nFor additional information related to the Company's derivative financial instruments and collateral, refer to Note 4 — Fair Value Measurements.\n\n2023 FORM 10-K 80\n\nTable of Contents\n\nNOTE 13 — ACCUMULATED OTHER COMPREHENSIVE INCOME (LOSS)\n\nThe changes in Accumulated other comprehensive income (loss), net of tax, were as follows:\n\n(Dollars in millions)\n\nFOREIGN CURRENCY TRANSLATION (1) ADJUSTMENT\n\nCASH FLOW HEDGES\n\nNET INVESTMENT (1) HEDGES\n\nOTHER\n\nTOTAL\n\nBalance at May 31, 2022 Other comprehensive income (loss):\n\n$\n\n(520) $\n\n779 $\n\n115 $\n\n(56) $\n\n318\n\nOther comprehensive gains (losses) before reclassifications Reclassifications to net income of previously deferred (gains) losses\n\n(2)\n\n(3)\n\n(91) 358\n\n487 (835)\n\n— —\n\n(20) 14\n\n376 (463)\n\nTotal other comprehensive income (loss) Balance at May 31, 2023\n\n$\n\n267 (253) $\n\n(348) 431 $\n\n— 115 $\n\n(6) (62) $\n\n(87) 231\n\n(1)\n\nThe accumulated foreign currency translation adjustment and net investment hedge gains/losses related to an investment in a foreign subsidiary are reclassified to Net income upon sale or upon complete or substantially complete liquidation of the respective entity."" + ""ACCOUNTING FOR DERIVATIVES AND HEDGING ACTIVITIES The Company uses derivative financial instruments to reduce its exposure to changes in foreign currency exchange rates and interest rates. All derivatives are recorded at fair value on the Consolidated Balance Sheets and changes in the fair value of derivative financial instruments are either recognized in Accumulated other comprehensive income (loss), Long-term debt or Net income depending on the nature of the underlying exposure, whether the derivative is formally designated as a hedge and, if designated, the extent to which the hedge is effective. The Company classifies the cash flows at settlement from derivatives in the same category as the cash flows from the related hedged items. For undesignated hedges and designated cash flow hedges, this is primarily within the Cash provided by operations component of the Consolidated Statements of Cash Flows. For designated net investment hedges, this is within the Cash provided by investing activities component of the Consolidated Statements of Cash Flows. For the Company's fair value hedges, which are interest rate swaps used to mitigate the change in fair value of its fixed-rate debt attributable to changes in interest rates, the related cash flows from periodic interest payments are reflected within the Cash provided by operations component of the Consolidated Statements of Cash Flows.\n\nRefer to Note 12 — Risk Management and Derivatives for additional information on the Company's risk management program and derivatives.\n\nSTOCK-BASED COMPENSATION"" + ""NET INVESTMENT HEDGES The Company has, in the past, hedged and may, in the future, hedge the risk of variability in foreign currency-denominated net investments in wholly-owned international operations. All changes in fair value of the derivatives designated as net investment hedges are reported in Accumulated other comprehensive income (loss) along with the foreign currency translation adjustments on those investments. The Company had no outstanding net investment hedges as of May 31, 2023.\n\nUNDESIGNATED DERIVATIVE INSTRUMENTS\n\nThe Company may elect to enter into foreign exchange forwards to mitigate the change in fair value of specific assets and liabilities on the Consolidated Balance Sheets. These undesignated instruments are recorded at fair value as a derivative asset or liability on the Consolidated Balance Sheets with their corresponding change in fair value recognized in Other (income) expense, net, together with the remeasurement gain or loss from the hedged balance sheet position. The total notional amount of outstanding undesignated derivative instruments was $4.7 billion as of May 31, 2023.\n\nCREDIT RISK\n\nThe Company is exposed to credit-related losses in the event of nonperformance by counterparties to hedging instruments. The counterparties to all derivative transactions are major financial institutions with investment grade credit ratings; however, this does not eliminate the Company's exposure to credit risk with these institutions. This credit risk is limited to the unrealized gains in such contracts should any of these counterparties fail to perform as contracted. To manage this risk, the Company has established strict counterparty credit guidelines that are continually monitored.""]","The Company's derivative financial instruments are subject to master netting arrangements that allow for the offset of assets and liabilities in the event of default or early termination of the contract. Cash collateral received related to these instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents and Accrued liabilities, the latter of which would further offset against the Company's derivative asset balance. Cash collateral posted related to these instruments associated with the Company's credit-related contingent features are recorded in Prepaid expenses and other current assets, which would further offset against the Company's derivative liability balance.",0.929096467344117,1.0 +5,"How are gains and losses determined for short-term investments, specifically available-for-sale debt securities, according to the Company's accounting policies?", Realized gains and losses on the sale of securities are determined by specific identification.,"[""CASH AND EQUIVALENTS Cash and equivalents represent cash and short-term, highly liquid investments, that are both readily convertible to known amounts of cash and so near their maturity they present insignificant risk of changes in value because of changes in interest rates, with maturities three months or less at the date of purchase.\n\n2023 FORM 10-K 62\n\nTable of Contents\n\nSHORT-TERM INVESTMENTS\n\nShort-term investments consist of highly liquid investments with maturities over three months at the date of purchase. At May 31, 2023 and 2022, Short-term investments consisted of available-for-sale debt securities, which are recorded at fair value with unrealized gains and losses reported, net of tax, in Accumulated other comprehensive income (loss), unless unrealized losses are determined to be unrecoverable. Realized gains and losses on the sale of securities are determined by specific identification. The Company considers all available-for-sale debt securities, including those with maturity dates beyond 12 months, as available to support current operational liquidity needs and, therefore, classifies all securities with maturity dates beyond three months at the date of purchase as current assets within Short-term investments on the Consolidated Balance Sheets.\n\nRefer to Note 4 — Fair Value Measurements for more information on the Company's Short-term investments.\n\nALLOWANCE FOR UNCOLLECTIBLE ACCOUNTS RECEIVABLE Accounts receivable, net consist primarily of amounts due from customers. The Company makes ongoing estimates relating to the collectability of its accounts receivable and maintains an allowance for expected losses resulting from the inability of its customers to make required payments. In addition to judgments about the creditworthiness of significant customers based on ongoing credit evaluations, the Company considers historical levels of credit losses, as well as macroeconomic and industry trends to determine the amount of the allowance. The allowance for uncollectible accounts receivable was $35 million and $34 million as of May 31, 2023 and 2022, respectively.\n\nINVENTORY VALUATION""]",Realized gains and losses on the sale of securities are determined by specific identification.,0.9011408795782977,1.0 diff --git a/python-recipes/RAG/resources/retrieval_basic_rag_test.csv b/python-recipes/RAG/resources/retrieval_basic_rag_test.csv new file mode 100644 index 0000000..9e1ac35 --- /dev/null +++ b/python-recipes/RAG/resources/retrieval_basic_rag_test.csv @@ -0,0 +1,14 @@ +,question,answer,contexts,ground_truth,context_precision,context_recall +0,What potential consequences could securities class action litigation have on a company's operations and reputation?," Securities class action litigation could result in reputational damage, substantial costs, and a diversion of management's attention and resources needed to successfully run the business.","[""Our Class B Common Stock is traded publicly, and at any given time various securities analysts follow our financial results and issue reports on us. These reports include information about our historical financial results as well as analysts' opinions of our future performance, which may, in part, be based upon any guidance we have provided. Analysts' estimates are often different from our estimates or expectations. If our operating results are below the estimates or expectations of public market analysts and investors, our stock price could decline. In the past, securities class action litigation has been brought against NIKE and other companies following a decline in the market price of their securities. If our stock price is volatile for any reason, we may become involved in this type of litigation in the future. Any litigation could result in reputational damage, substantial costs and a diversion of management's attention and resources needed to successfully run our business.\n\n2023 FORM 10-K 23\n\nTable of Contents\n\nITEM 1B. UNRESOLVED STAFF COMMENTS\n\nNone.\n\nITEM 2. PROPERTIES\n\nThe following is a summary of principal properties owned or leased by NIKE:\n\nThe NIKE World Campus, owned by NIKE and located near Beaverton, Oregon, USA, is an approximately 400-acre site consisting of over 40 buildings which, together with adjacent leased properties, functions as our world headquarters and is occupied by approximately 11,400 employees engaged in management, research, design, development, marketing, finance and other administrative functions serving nearly all of our segments. We lease a similar, but smaller, administrative facility in Hilversum, the Netherlands, which serves as the headquarters for our Europe, Middle East & Africa geography and management of certain brand functions for our non-U.S. operations. We also lease an office complex in Shanghai, China, our headquarters for our Greater China geography, occupied by employees focused on implementing our wholesale, NIKE Direct and merchandising strategies in the region, among other functions.""]","Any litigation could result in reputational damage, substantial costs and a diversion of management's attention and resources needed to successfully run our business.",0.9999999999,1.0 +1,How do currency exchange rates impact the financial results of the company?," Currency exchange rates can significantly impact the company's financial results, as the translation of foreign currencies into U.S. Dollars can result in either a benefit or a detriment to the company's revenues and earnings. Fluctuations in exchange rates can also increase costs and decrease margins, resulting in lower revenues and profits. The company hedges certain foreign currency exposures to minimize the impact of these fluctuations, but they cannot completely eliminate the effects. ","[""Economic factors beyond our control, and changes in the global economic environment, including fluctuations in inflation and currency exchange rates, could result in lower revenues, higher costs and decreased margins and earnings.\n\nA majority of our products are manufactured and sold outside of the United States, and we conduct purchase and sale transactions in various currencies, which creates exposure to the volatility of global economic conditions, including fluctuations in inflation and foreign currency exchange rates. Central banks may deploy various strategies to combat inflation, including increasing interest rates, which may impact our borrowing costs. Additionally, there has been, and may continue to be, volatility in currency exchange rates that impact the U.S. Dollar value relative to other international currencies. Our international revenues and expenses generally are derived from sales and operations in foreign currencies, and these revenues and expenses are affected by currency fluctuations, specifically amounts recorded in foreign currencies and translated into U.S. Dollars for consolidated financial reporting, as weakening of foreign currencies relative to the U.S. Dollar adversely affects the U.S. Dollar value of the Company's foreign currency-denominated sales and earnings. Currency exchange rate fluctuations could also disrupt the business of the independent manufacturers that produce our products by making their purchases of raw materials more expensive and more difficult to finance. Foreign currency fluctuations have adversely affected and could continue to have an adverse effect on our results of operations and financial condition.\n\nWe hedge certain foreign currency exposures to lessen and delay, but not to completely eliminate, the effects of foreign currency fluctuations on our financial results. Since the hedging activities are designed to lessen volatility, they not only reduce the negative impact of a stronger U.S. Dollar or other trading currency, but they also reduce the positive impact of a weaker U.S. Dollar or other trading currency. Our future financial results have in the past been and could in the future be significantly affected by the value of the U.S. Dollar in relation to the foreign currencies in which we conduct business. The degree to which our financial results are affected for any given time period will depend in part upon our hedging activities."" + 'Many of our foreign subsidiaries operate in functional currencies other than the U.S. Dollar. Fluctuations in currency exchange rates create volatility in our reported results as we are required to translate the balance sheets, operational results and cash flows of these subsidiaries into U.S. Dollars for consolidated reporting. The translation of foreign subsidiaries\' non-U.S. Dollar denominated balance sheets into U.S. Dollars for consolidated reporting results in a cumulative translation adjustment to Accumulated other comprehensive income (loss) within Shareholders\' equity. In the translation of our Consolidated Statements of Income, a weaker U.S. Dollar in relation to foreign functional currencies benefits our consolidated earnings whereas a stronger U.S. Dollar reduces our consolidated earnings. The impact of foreign exchange rate fluctuations on the translation of our consolidated Revenues was a detriment of approximately $2,859 million, $295 million and a benefit of approximately $893 million for the years ended May 31, 2023, 2022 and 2021, respectively. The impact of foreign exchange rate fluctuations on the translation of our Income before income taxes was a detriment of approximately $824 million, $87 million and a benefit of approximately $260 million for the years ended May 31, 2023, 2022 and 2021, respectively.\n\nMANAGING TRANSLATIONAL EXPOSURES To minimize the impact of translating foreign currency denominated revenues and expenses into U.S. Dollars for consolidated reporting, certain foreign subsidiaries use excess cash to purchase U.S. Dollar denominated available-for-sale investments. The variable future cash flows associated with the purchase and subsequent sale of these U.S. Dollar denominated investments at non-U.S. Dollar functional currency subsidiaries creates a foreign currency exposure that qualifies for hedge accounting under generally accepted accounting principles in the United States of America (""U.S. GAAP""). We utilize forward contracts and/or options to mitigate the variability of the forecasted future purchases and sales of these U.S. Dollar investments. The combination of the purchase and sale of the U.S. Dollar investment and the hedging instrument has the effect of partially offsetting the year-over-year foreign currency translation impact on net earnings in the period the investments are sold. Hedges of the purchase of U.S. Dollar denominated available-for-sale investments are accounted for as cash flow hedges.' + ""As part of the Company's centrally managed foreign exchange risk management program, standard foreign currency rates are assigned twice per year to each NIKE Brand entity in the Company's geographic operating segments and to Converse. These rates are set approximately nine and twelve months in advance of the future selling seasons to which they relate (specifically, for each currency, one standard rate applies to the fall and holiday selling seasons, and one standard rate applies to the spring and summer selling seasons) based on average market spot rates in the calendar month preceding the date they are established. Inventories and Cost of sales for geographic operating segments and Converse reflect the use of these standard rates to record non-functional currency product purchases in the entity's functional currency. Differences between assigned standard foreign currency rates and actual market rates are included in Corporate, together with foreign currency hedge gains and losses generated from the Company's centrally managed foreign exchange risk management program and other conversion gains and losses.\n\nAccounts receivable, net, Inventories and Property, plant and equipment, net for operating segments are regularly reviewed by management and are therefore provided below.\n\n2023 FORM 10-K 85\n\nTable of Contents\n\n(Dollars in millions)\n\nREVENUES\n\nNorth America Europe, Middle East & Africa Greater China\n\nAsia Pacific & Latin America Global Brand Divisions\n\nTotal NIKE Brand\n\nConverse Corporate\n\nTOTAL NIKE, INC. REVENUES\n\nEARNINGS BEFORE INTEREST AND TAXES\n\nNorth America Europe, Middle East & Africa\n\nGreater China Asia Pacific & Latin America Global Brand Divisions\n\nConverse Corporate\n\nInterest expense (income), net\n\nTOTAL NIKE, INC. INCOME BEFORE INCOME TAXES\n\nADDITIONS TO PROPERTY, PLANT AND EQUIPMENT\n\nNorth America Europe, Middle East & Africa Greater China\n\nAsia Pacific & Latin America Global Brand Divisions\n\nTotal NIKE Brand\n\nConverse Corporate\n\nTOTAL ADDITIONS TO PROPERTY, PLANT AND EQUIPMENT\n\nDEPRECIATION North America Europe, Middle East & Africa\n\nGreater China Asia Pacific & Latin America Global Brand Divisions\n\nTotal NIKE Brand\n\nConverse\n\nCorporate\n\nTOTAL DEPRECIATION\n\n2023 FORM 10-K 86\n\n$\n\n$\n\n$\n\n$\n\n$\n\n$\n\n$\n\n$\n\nYEAR ENDED MAY 31,\n\n2023\n\n2022\n\n21,608 $ 13,418 7,248\n\n18,353 $ 12,479 7,547\n\n6,431 58\n\n5,955 102\n\n48,763 2,427 27\n\n44,436 2,346 (72)\n\n51,217 $\n\n46,710 $\n\n5,454 $ 3,531\n\n5,114 $ 3,293\n\n2,283 1,932 (4,841)\n\n2,365 1,896 (4,262)"" + 'recognized upon completion of the sale our entities in Argentina and Uruguay to a third-party distributor in the second quarter of fiscal 2023. This was partially offset by the one-time charge related to the deconsolidation of our Russian operations recognized in the prior year, with the net amount of these activities reported as a component of consolidated Other (income) expense, net; and\n\na favorable change in net foreign currency gains and losses of $174 million related to the remeasurement of monetary assets and liabilities denominated in non- functional currencies and the impact of certain foreign currency derivative instruments, reported as a component of consolidated Other (income) expense, net.\n\nFOREIGN CURRENCY EXPOSURES AND HEDGING PRACTICES\n\nOVERVIEW\n\nAs a global company with significant operations outside the United States, in the normal course of business we are exposed to risk arising from changes in currency exchange rates. Our primary foreign currency exposures arise from the recording of transactions denominated in non-functional currencies and the translation of foreign currency denominated results of operations, financial position and cash flows into U.S. Dollars.']",Currency exchange rate fluctuations could disrupt the business of the independent manufacturers that produce our products by making their purchases of raw materials more expensive and more difficult to finance. Foreign currency fluctuations have adversely affected and could continue to have an adverse effect on our results of operations and financial condition.,0.999999999975,1.0 +2,Why are cost-effective investments considered essential for business growth and profitability?," Cost-effective investments are considered essential for business growth and profitability because they allow companies to allocate resources efficiently and maximize their returns. By making smart and strategic investments, businesses can improve their operations, expand their market share, and increase their profits. This can also lead to increased shareholder value and attract potential investors. Additionally, cost-effective investments can help companies stay competitive in their industry and adapt to changing market conditions. ",[],"Cost-effective investments are considered essential for business growth and profitability because they contribute to the development of new businesses, expansion of existing businesses, and the improvement of product offerings and manufacturing innovation. These investments help drive growth and increase profitability. However, significant investments also come with risks and uncertainties.",0.0,0.0 +3,What changes were made to the U.S. corporate income tax system by the Inflation Reduction Act of 2022?," The Inflation Reduction Act of 2022 made significant changes to the U.S. corporate income tax system, including introducing new profit allocation and nexus rules and implementing a global corporate minimum tax rate of 15% as proposed by the OECD and G20 Inclusive Framework.","['Proposals to reform U.S. and foreign tax laws could significantly impact how U.S. multinational corporations are taxed on global earnings and could increase the U.S. corporate tax rate. For example, the Organization for Economic Co-operation and Development (OECD) and the G20 Inclusive Framework on Base Erosion and Profit Shifting (the ""Inclusive Framework"") has put forth two proposals—Pillar One and Pillar Two—that revise the existing profit allocation and nexus rules and ensure a minimal level of taxation, respectively. On December 12, 2022, the European Union member states agreed to implement the Inclusive Framework\'s global corporate minimum tax rate of 15%. Other countries are also actively considering changes to their tax laws to adopt certain parts of the Inclusive Framework\'s proposals. Although we cannot predict whether or in what form these proposals will be enacted into law, these changes, if enacted into law, could have an adverse impact on our effective tax rate, income tax expense and cash flows.\n\nPortions of our operations are subject to a reduced tax rate or are under a tax holiday. We also utilize tax rulings and other agreements to obtain certainty in treatment of certain tax matters. Tax holidays and rulings can expire from time to time and may be extended when certain conditions are met, or terminated if certain conditions are not met. The impact of any changes in conditions would be the loss of certainty in treatment thus potentially impacting our effective income tax rate. For example, in January 2019, the European Commission opened a formal investigation to examine whether the Netherlands has breached State Aid rules when granting certain tax rulings to the Company. If this matter is adversely resolved, the Netherlands may be required to assess additional amounts with respect to prior periods, and the Company\'s income taxes related to prior periods in the Netherlands could increase.' + 'A reconciliation from the U.S. statutory federal income tax rate to the effective income tax rate is as follows:\n\nYEAR ENDED MAY 31,\n\n2023\n\n2022\n\nFederal income tax rate\n\n21.0 %\n\n21.0 %\n\nState taxes, net of federal benefit Foreign earnings Subpart F deferred tax benefit\n\n1.5 % 1.7 % 0.0 %\n\n1.4 % -1.8 % -4.7 %\n\nForeign-derived intangible income benefit Excess tax benefits from stock-based compensation Income tax audits and contingency reserves\n\n6.1 % -1.1 % 1.0 %\n\n4.1 % -4.9 % 1.5 %\n\nU.S. research and development tax credit Other, net\n\nEFFECTIVE INCOME TAX RATE\n\n1.2 % 1.4 %\n\n18.2 %\n\n1.0 % 1.7 % 9.1 %\n\nOn December 22, 2017, the U.S. enacted the Tax Cuts and Jobs Act (the ""Tax Act""), which significantly changed U.S. tax law and included a provision to tax global intangible low-taxed income (""GILTI"") of foreign subsidiaries. The Company recognizes taxes due under the GILTI provision as a current period expense.\n\nThe effective tax rate for the fiscal year ended May 31, 2023 was higher than the effective tax rate for the fiscal year ended May 31, 2022. The increase was primarily due to decreased benefits from stock-based compensation and the prior year recognition of a non-cash, one-time benefit related to the onshoring of the Company\'s non-U.S. intangible property. During the fourth quarter of fiscal 2022, the Company onshored certain non-U.S. intangible property ownership rights and implemented changes in the Company\'s legal entity structure. The tax restructuring increases the possibility that foreign earnings in future periods will be subject to tax in the U.S. due to Subpart F of the Internal Revenue Code. The Company recognized a deferred tax asset and corresponding non-cash deferred income tax benefit of 4.7%, to establish the deferred tax deduction that is expected to reduce taxable income in future periods.\n\n2023 FORM 10-K 72\n\n2021\n\n5,723 938\n\n6,661\n\n2021\n\n328 134\n\n857 1,319\n\n(371)\n\n(34) 20\n\n(385) 934\n\n2021\n\n21.0 %\n\n1.3 % 0.2 % 0.0 %\n\n3.7 % -4.5 % 1.5 %\n\n0.9 % -0.9 %\n\n14.0 %\n\nTable of Contents\n\nThe effective tax rate for the fiscal year ended May 31, 2022 was lower than the effective tax rate for the fiscal year ended May 31, 2021. The decrease was primarily due to a shift in the Company\'s earnings mix and recognition of a non-cash, one-time benefit related to the onshoring of the Company\'s non-U.S. intangible property.\n\nDeferred tax assets and liabilities comprise the following as of:\n\nMAY 31,\n\n(Dollars in millions)\n\n2023\n\n2022\n\nDeferred tax assets:']","The Inflation Reduction Act of 2022 made changes to the U.S. corporate income tax system, including the implementation of a fifteen percent minimum tax based on 'adjusted financial statement income', effective for NIKE beginning June 1, 2023.",0.99999999995,0.0 +4,"How does master netting arrangements impact recording of assets and liabilities for derivative financial instruments, and how are cash collateral amounts for credit-related contingent features accounted for?"," Master netting arrangements allow for the offset of assets and liabilities in the event of default or early termination of derivative contracts. Cash collateral amounts related to credit-related contingent features are recorded in either Cash and equivalents or Prepaid expenses and other current assets, and are further offset against the derivative asset or liability balance. ","[""The Company records the assets and liabilities of its derivative financial instruments on a gross basis on the Consolidated Balance Sheets. The Company's derivative financial instruments are subject to master netting arrangements that allow for the offset of assets and liabilities in the event of default or early termination of the contract. Any amounts of cash collateral received related to these instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents and Accrued liabilities, the latter of which would further offset against the Company's derivative asset balance. Any amounts of cash collateral posted related to these instruments associated with the Company's credit-related contingent features are recorded in Prepaid expenses and other current assets, which would further offset against the Company's derivative liability balance. Cash collateral received or posted related to the Company's credit-related contingent features is presented in the Cash provided by operations component of the Consolidated Statements of Cash Flows. The Company does not recognize amounts of non-cash collateral received, such as securities, on the Consolidated Balance Sheets. For further information related to credit risk, refer to Note 12 — Risk Management and Derivatives.\n\n2023 FORM 10-K 68\n\nTable of Contents\n\nThe following tables present information about the Company's derivative assets and liabilities measured at fair value on a recurring basis and indicate the level in the fair value hierarchy in which the Company classifies the fair value measurement:\n\nMAY 31, 2023\n\nDERIVATIVE ASSETS\n\nDERIVATIVE LIABILITIES\n\n(Dollars in millions)\n\nASSETS AT FAIR VALUE\n\nOTHER CURRENT ASSETS\n\nOTHER LONG- TERM ASSETS\n\nLIABILITIES AT FAIR VALUE\n\nACCRUED LIABILITIES\n\nOTHER LONG- TERM LIABILITIES\n\nLevel 2:\n\nForeign exchange forwards and options\n\n(1)\n\n$\n\n557 $\n\n493 $\n\n64 $\n\n180 $\n\n128 $\n\n52\n\n(1)\n\nIf the foreign exchange derivative instruments had been netted on the Consolidated Balance Sheets, the asset and liability positions each would have been reduced by $178 million as of May 31, 2023. As of that date, the Company received $36 million of cash collateral from various counterparties related to foreign exchange derivative instruments. No amount of collateral was posted on the derivative liability balance as of May 31, 2023.\n\nMAY 31, 2022\n\nDERIVATIVE ASSETS\n\nDERIVATIVE LIABILITIES\n\n(Dollars in millions)\n\nASSETS AT FAIR VALUE"" + ""The Company's derivative contracts contain credit risk-related contingent features designed to protect against significant deterioration in counterparties' creditworthiness and their ultimate ability to settle outstanding derivative contracts in the normal course of business. The Company's bilateral credit-related contingent features generally require the owing entity, either the Company or the derivative counterparty, to post collateral for the portion of the fair value in excess of $50 million should the fair value of outstanding derivatives per counterparty be greater than $50 million. Additionally, a certain level of decline in credit rating of either the Company or the counterparty could trigger collateral requirements. As of May 31, 2023, the Company was in compliance with all credit risk-related contingent features, and derivative instruments with such features were in a net liability position of approximately $2 million. Accordingly, the Company posted no cash collateral as a result of these contingent features. Further, as of May 31, 2023, the Company had received $36 million in cash collateral from various counterparties to its derivative contracts. The Company considers the impact of the risk of counterparty default to be immaterial.\n\nFor additional information related to the Company's derivative financial instruments and collateral, refer to Note 4 — Fair Value Measurements.\n\n2023 FORM 10-K 80\n\nTable of Contents\n\nNOTE 13 — ACCUMULATED OTHER COMPREHENSIVE INCOME (LOSS)\n\nThe changes in Accumulated other comprehensive income (loss), net of tax, were as follows:\n\n(Dollars in millions)\n\nFOREIGN CURRENCY TRANSLATION (1) ADJUSTMENT\n\nCASH FLOW HEDGES\n\nNET INVESTMENT (1) HEDGES\n\nOTHER\n\nTOTAL\n\nBalance at May 31, 2022 Other comprehensive income (loss):\n\n$\n\n(520) $\n\n779 $\n\n115 $\n\n(56) $\n\n318\n\nOther comprehensive gains (losses) before reclassifications Reclassifications to net income of previously deferred (gains) losses\n\n(2)\n\n(3)\n\n(91) 358\n\n487 (835)\n\n— —\n\n(20) 14\n\n376 (463)\n\nTotal other comprehensive income (loss) Balance at May 31, 2023\n\n$\n\n267 (253) $\n\n(348) 431 $\n\n— 115 $\n\n(6) (62) $\n\n(87) 231\n\n(1)\n\nThe accumulated foreign currency translation adjustment and net investment hedge gains/losses related to an investment in a foreign subsidiary are reclassified to Net income upon sale or upon complete or substantially complete liquidation of the respective entity."" + ""ACCOUNTING FOR DERIVATIVES AND HEDGING ACTIVITIES The Company uses derivative financial instruments to reduce its exposure to changes in foreign currency exchange rates and interest rates. All derivatives are recorded at fair value on the Consolidated Balance Sheets and changes in the fair value of derivative financial instruments are either recognized in Accumulated other comprehensive income (loss), Long-term debt or Net income depending on the nature of the underlying exposure, whether the derivative is formally designated as a hedge and, if designated, the extent to which the hedge is effective. The Company classifies the cash flows at settlement from derivatives in the same category as the cash flows from the related hedged items. For undesignated hedges and designated cash flow hedges, this is primarily within the Cash provided by operations component of the Consolidated Statements of Cash Flows. For designated net investment hedges, this is within the Cash provided by investing activities component of the Consolidated Statements of Cash Flows. For the Company's fair value hedges, which are interest rate swaps used to mitigate the change in fair value of its fixed-rate debt attributable to changes in interest rates, the related cash flows from periodic interest payments are reflected within the Cash provided by operations component of the Consolidated Statements of Cash Flows.\n\nRefer to Note 12 — Risk Management and Derivatives for additional information on the Company's risk management program and derivatives.\n\nSTOCK-BASED COMPENSATION"" + ""NET INVESTMENT HEDGES The Company has, in the past, hedged and may, in the future, hedge the risk of variability in foreign currency-denominated net investments in wholly-owned international operations. All changes in fair value of the derivatives designated as net investment hedges are reported in Accumulated other comprehensive income (loss) along with the foreign currency translation adjustments on those investments. The Company had no outstanding net investment hedges as of May 31, 2023.\n\nUNDESIGNATED DERIVATIVE INSTRUMENTS\n\nThe Company may elect to enter into foreign exchange forwards to mitigate the change in fair value of specific assets and liabilities on the Consolidated Balance Sheets. These undesignated instruments are recorded at fair value as a derivative asset or liability on the Consolidated Balance Sheets with their corresponding change in fair value recognized in Other (income) expense, net, together with the remeasurement gain or loss from the hedged balance sheet position. The total notional amount of outstanding undesignated derivative instruments was $4.7 billion as of May 31, 2023.\n\nCREDIT RISK\n\nThe Company is exposed to credit-related losses in the event of nonperformance by counterparties to hedging instruments. The counterparties to all derivative transactions are major financial institutions with investment grade credit ratings; however, this does not eliminate the Company's exposure to credit risk with these institutions. This credit risk is limited to the unrealized gains in such contracts should any of these counterparties fail to perform as contracted. To manage this risk, the Company has established strict counterparty credit guidelines that are continually monitored.""]","The Company's derivative financial instruments are subject to master netting arrangements that allow for the offset of assets and liabilities in the event of default or early termination of the contract. Cash collateral received related to these instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents and Accrued liabilities, the latter of which would further offset against the Company's derivative asset balance. Cash collateral posted related to these instruments associated with the Company's credit-related contingent features are recorded in Prepaid expenses and other current assets, which would further offset against the Company's derivative liability balance.",0.999999999975,1.0 +5,"How are gains and losses determined for short-term investments, specifically available-for-sale debt securities, according to the Company's accounting policies?", Realized gains and losses on the sale of securities are determined by specific identification.,"[""CASH AND EQUIVALENTS Cash and equivalents represent cash and short-term, highly liquid investments, that are both readily convertible to known amounts of cash and so near their maturity they present insignificant risk of changes in value because of changes in interest rates, with maturities three months or less at the date of purchase.\n\n2023 FORM 10-K 62\n\nTable of Contents\n\nSHORT-TERM INVESTMENTS\n\nShort-term investments consist of highly liquid investments with maturities over three months at the date of purchase. At May 31, 2023 and 2022, Short-term investments consisted of available-for-sale debt securities, which are recorded at fair value with unrealized gains and losses reported, net of tax, in Accumulated other comprehensive income (loss), unless unrealized losses are determined to be unrecoverable. Realized gains and losses on the sale of securities are determined by specific identification. The Company considers all available-for-sale debt securities, including those with maturity dates beyond 12 months, as available to support current operational liquidity needs and, therefore, classifies all securities with maturity dates beyond three months at the date of purchase as current assets within Short-term investments on the Consolidated Balance Sheets.\n\nRefer to Note 4 — Fair Value Measurements for more information on the Company's Short-term investments.\n\nALLOWANCE FOR UNCOLLECTIBLE ACCOUNTS RECEIVABLE Accounts receivable, net consist primarily of amounts due from customers. The Company makes ongoing estimates relating to the collectability of its accounts receivable and maintains an allowance for expected losses resulting from the inability of its customers to make required payments. In addition to judgments about the creditworthiness of significant customers based on ongoing credit evaluations, the Company considers historical levels of credit losses, as well as macroeconomic and industry trends to determine the amount of the allowance. The allowance for uncollectible accounts receivable was $35 million and $34 million as of May 31, 2023 and 2022, respectively.\n\nINVENTORY VALUATION""]",Realized gains and losses on the sale of securities are determined by specific identification.,0.9999999999,1.0 diff --git a/python-recipes/RAG/resources/testset.csv b/python-recipes/RAG/resources/testset.csv new file mode 100644 index 0000000..fe91945 --- /dev/null +++ b/python-recipes/RAG/resources/testset.csv @@ -0,0 +1,7 @@ +question,contexts,ground_truth,evolution_type,metadata,episode_done +What potential consequences could securities class action litigation have on a company's operations and reputation?,"[""Our Class B Common Stock is traded publicly, and at any given time various securities analysts follow our financial results and issue reports on us. These reports include information about our historical financial results as well as analysts' opinions of our future performance, which may, in part, be based upon any guidance we have provided. Analysts' estimates are often different from our estimates or expectations. If our operating results are below the estimates or expectations of public market analysts and investors, our stock price could decline. In the past, securities class action litigation has been brought against NIKE and other companies following a decline in the market price of their securities. If our stock price is volatile for any reason, we may become involved in this type of litigation in the future. Any litigation could result in reputational damage, substantial costs and a diversion of management's attention and resources needed to successfully run our business.\n\n2023 FORM 10-K 23\n\nTable of Contents\n\nITEM 1B. UNRESOLVED STAFF COMMENTS\n\nNone.\n\nITEM 2. PROPERTIES\n\nThe following is a summary of principal properties owned or leased by NIKE:\n\nThe NIKE World Campus, owned by NIKE and located near Beaverton, Oregon, USA, is an approximately 400-acre site consisting of over 40 buildings which, together with adjacent leased properties, functions as our world headquarters and is occupied by approximately 11,400 employees engaged in management, research, design, development, marketing, finance and other administrative functions serving nearly all of our segments. We lease a similar, but smaller, administrative facility in Hilversum, the Netherlands, which serves as the headquarters for our Europe, Middle East & Africa geography and management of certain brand functions for our non-U.S. operations. We also lease an office complex in Shanghai, China, our headquarters for our Greater China geography, occupied by employees focused on implementing our wholesale, NIKE Direct and merchandising strategies in the region, among other functions.""]","Any litigation could result in reputational damage, substantial costs and a diversion of management's attention and resources needed to successfully run our business.",simple,[{'source': 'resources/nke-10k-2023.pdf'}],True +How do currency exchange rates impact the financial results of the company?,"[""Economic factors beyond our control, and changes in the global economic environment, including fluctuations in inflation and currency exchange rates, could result in lower revenues, higher costs and decreased margins and earnings.\n\nA majority of our products are manufactured and sold outside of the United States, and we conduct purchase and sale transactions in various currencies, which creates exposure to the volatility of global economic conditions, including fluctuations in inflation and foreign currency exchange rates. Central banks may deploy various strategies to combat inflation, including increasing interest rates, which may impact our borrowing costs. Additionally, there has been, and may continue to be, volatility in currency exchange rates that impact the U.S. Dollar value relative to other international currencies. Our international revenues and expenses generally are derived from sales and operations in foreign currencies, and these revenues and expenses are affected by currency fluctuations, specifically amounts recorded in foreign currencies and translated into U.S. Dollars for consolidated financial reporting, as weakening of foreign currencies relative to the U.S. Dollar adversely affects the U.S. Dollar value of the Company's foreign currency-denominated sales and earnings. Currency exchange rate fluctuations could also disrupt the business of the independent manufacturers that produce our products by making their purchases of raw materials more expensive and more difficult to finance. Foreign currency fluctuations have adversely affected and could continue to have an adverse effect on our results of operations and financial condition.\n\nWe hedge certain foreign currency exposures to lessen and delay, but not to completely eliminate, the effects of foreign currency fluctuations on our financial results. Since the hedging activities are designed to lessen volatility, they not only reduce the negative impact of a stronger U.S. Dollar or other trading currency, but they also reduce the positive impact of a weaker U.S. Dollar or other trading currency. Our future financial results have in the past been and could in the future be significantly affected by the value of the U.S. Dollar in relation to the foreign currencies in which we conduct business. The degree to which our financial results are affected for any given time period will depend in part upon our hedging activities.""]",Currency exchange rate fluctuations could disrupt the business of the independent manufacturers that produce our products by making their purchases of raw materials more expensive and more difficult to finance. Foreign currency fluctuations have adversely affected and could continue to have an adverse effect on our results of operations and financial condition.,simple,[{'source': 'resources/nke-10k-2023.pdf'}],True +Why are cost-effective investments considered essential for business growth and profitability?,"['From time to time, we may invest in technology, business infrastructure, new businesses or capabilities, product offering and manufacturing innovation and expansion of existing businesses, such as our NIKE Direct operations, which require substantial cash investments and management attention. We believe cost-effective investments are essential to business growth and profitability; however, significant investments are subject to typical risks and uncertainties inherent in developing a new business or expanding an existing business. The failure of any significant investment to provide expected returns or profitability could have a material adverse effect on our financial results and divert management attention from more profitable business operations. See also ""Our NIKE Direct operations have required and will continue to require a substantial investment and commitment of resources and are subject to numerous risks and uncertainties.""\n\nThe sale of a large number of shares of common stock by our principal shareholder could depress the market price of our common stock.\n\nAs of June 30, 2023, Swoosh, LLC beneficially owned approximately 77% of our Class A Common Stock. If, on June 30, 2023, all of these shares were converted into Class B Common Stock, Swoosh, LLC\'s commensurate ownership percentage of our Class B Common Stock would be approximately 16%. The shares are available for resale, subject to the requirements of the U.S. securities laws and the terms of the limited liability company agreement governing Swoosh, LLC. The sale or prospect of a sale of a substantial number of these shares could have an adverse effect on the market price of our common stock. Swoosh, LLC was formed by Philip H. Knight, our Chairman Emeritus, to hold the majority of his shares of Class A Common Stock. Mr. Knight does not have voting rights with respect to Swoosh, LLC, although Travis Knight, his son and a NIKE director, has a significant role in the management of the Class A Common Stock owned by Swoosh, LLC.\n\nChanges in our credit ratings or macroeconomic conditions may affect our liquidity, increasing borrowing costs and limiting our financing options.']","Cost-effective investments are considered essential for business growth and profitability because they contribute to the development of new businesses, expansion of existing businesses, and the improvement of product offerings and manufacturing innovation. These investments help drive growth and increase profitability. However, significant investments also come with risks and uncertainties.",simple,[{'source': 'resources/nke-10k-2023.pdf'}],True +What changes were made to the U.S. corporate income tax system by the Inflation Reduction Act of 2022?,"['FISCAL 2023 COMPARED TO FISCAL 2022\n\nOther (income) expense, net increased from $181 million of other income, net in fiscal 2022 to $280 million in the current fiscal year, primarily due to a net favorable change in foreign currency conversion gains and losses, including hedges, and the one-time charge related to the deconsolidation of our Russian operations recognized in the prior year. This increase was partially offset by net unfavorable activity related to our strategic distributor partnership transition within APLA, including the loss recognized upon the completion of the sale of our entities in Argentina and Uruguay to a third-party distributor in the second quarter of fiscal 2023.\n\nFor more information related to our distributor partnership transition within APLA, see Note 18 — Acquisitions and Divestitures within the accompanying Notes to the Consolidated Financial Statements.\n\nWe estimate the combination of the translation of foreign currency-denominated profits from our international businesses, and the year-over-year change in foreign currency-related gains and losses included in Other (income) expense, net had an unfavorable impact on our Income before income taxes of $1,023 million for fiscal 2023. INCOME TAXES\n\nFISCAL 2023\n\nFISCAL 2022\n\n% CHANGE\n\nFISCAL 2021\n\n% CHANGE\n\nEffective tax rate\n\n18.2 %\n\n9.1 %\n\n910 bps\n\n14.0 %\n\n(490) bps\n\nFISCAL 2023 COMPARED TO FISCAL 2022\n\nOur effective tax rate was 18.2% for fiscal 2023, compared to 9.1% for fiscal 2022, primarily due to decreased benefits from stock-based compensation and a non-cash, one-time benefit in the prior year related to the onshoring of certain non-U.S. intangible property ownership rights.\n\nOn August 16, 2022, the U.S. government enacted the Inflation Reduction Act of 2022 that includes, among other provisions, changes to the U.S. corporate income tax system, including a fifteen percent minimum tax based on ""adjusted financial statement income,"" which is effective for NIKE beginning June 1, 2023. Based on our current analysis of the provisions, we do not expect these tax law changes to have a material impact on our financial statements; however, we will continue to evaluate their impact as further information becomes available.\n\n2023 FORM 10-K 35\n\nTable of Contents\n\nOPERATING SEGMENTS']","The Inflation Reduction Act of 2022 made changes to the U.S. corporate income tax system, including the implementation of a fifteen percent minimum tax based on 'adjusted financial statement income', effective for NIKE beginning June 1, 2023.",simple,[{'source': 'resources/nke-10k-2023.pdf'}],True +"How does master netting arrangements impact recording of assets and liabilities for derivative financial instruments, and how are cash collateral amounts for credit-related contingent features accounted for?","[""The Company records the assets and liabilities of its derivative financial instruments on a gross basis on the Consolidated Balance Sheets. The Company's derivative financial instruments are subject to master netting arrangements that allow for the offset of assets and liabilities in the event of default or early termination of the contract. Any amounts of cash collateral received related to these instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents and Accrued liabilities, the latter of which would further offset against the Company's derivative asset balance. Any amounts of cash collateral posted related to these instruments associated with the Company's credit-related contingent features are recorded in Prepaid expenses and other current assets, which would further offset against the Company's derivative liability balance. Cash collateral received or posted related to the Company's credit-related contingent features is presented in the Cash provided by operations component of the Consolidated Statements of Cash Flows. The Company does not recognize amounts of non-cash collateral received, such as securities, on the Consolidated Balance Sheets. For further information related to credit risk, refer to Note 12 — Risk Management and Derivatives.\n\n2023 FORM 10-K 68\n\nTable of Contents\n\nThe following tables present information about the Company's derivative assets and liabilities measured at fair value on a recurring basis and indicate the level in the fair value hierarchy in which the Company classifies the fair value measurement:\n\nMAY 31, 2023\n\nDERIVATIVE ASSETS\n\nDERIVATIVE LIABILITIES\n\n(Dollars in millions)\n\nASSETS AT FAIR VALUE\n\nOTHER CURRENT ASSETS\n\nOTHER LONG- TERM ASSETS\n\nLIABILITIES AT FAIR VALUE\n\nACCRUED LIABILITIES\n\nOTHER LONG- TERM LIABILITIES\n\nLevel 2:\n\nForeign exchange forwards and options\n\n(1)\n\n$\n\n557 $\n\n493 $\n\n64 $\n\n180 $\n\n128 $\n\n52\n\n(1)\n\nIf the foreign exchange derivative instruments had been netted on the Consolidated Balance Sheets, the asset and liability positions each would have been reduced by $178 million as of May 31, 2023. As of that date, the Company received $36 million of cash collateral from various counterparties related to foreign exchange derivative instruments. No amount of collateral was posted on the derivative liability balance as of May 31, 2023.\n\nMAY 31, 2022\n\nDERIVATIVE ASSETS\n\nDERIVATIVE LIABILITIES\n\n(Dollars in millions)\n\nASSETS AT FAIR VALUE""]","The Company's derivative financial instruments are subject to master netting arrangements that allow for the offset of assets and liabilities in the event of default or early termination of the contract. Cash collateral received related to these instruments associated with the Company's credit-related contingent features are recorded in Cash and equivalents and Accrued liabilities, the latter of which would further offset against the Company's derivative asset balance. Cash collateral posted related to these instruments associated with the Company's credit-related contingent features are recorded in Prepaid expenses and other current assets, which would further offset against the Company's derivative liability balance.",multi_context,[{'source': 'resources/nke-10k-2023.pdf'}],True +"How are gains and losses determined for short-term investments, specifically available-for-sale debt securities, according to the Company's accounting policies?","[""CASH AND EQUIVALENTS Cash and equivalents represent cash and short-term, highly liquid investments, that are both readily convertible to known amounts of cash and so near their maturity they present insignificant risk of changes in value because of changes in interest rates, with maturities three months or less at the date of purchase.\n\n2023 FORM 10-K 62\n\nTable of Contents\n\nSHORT-TERM INVESTMENTS\n\nShort-term investments consist of highly liquid investments with maturities over three months at the date of purchase. At May 31, 2023 and 2022, Short-term investments consisted of available-for-sale debt securities, which are recorded at fair value with unrealized gains and losses reported, net of tax, in Accumulated other comprehensive income (loss), unless unrealized losses are determined to be unrecoverable. Realized gains and losses on the sale of securities are determined by specific identification. The Company considers all available-for-sale debt securities, including those with maturity dates beyond 12 months, as available to support current operational liquidity needs and, therefore, classifies all securities with maturity dates beyond three months at the date of purchase as current assets within Short-term investments on the Consolidated Balance Sheets.\n\nRefer to Note 4 — Fair Value Measurements for more information on the Company's Short-term investments.\n\nALLOWANCE FOR UNCOLLECTIBLE ACCOUNTS RECEIVABLE Accounts receivable, net consist primarily of amounts due from customers. The Company makes ongoing estimates relating to the collectability of its accounts receivable and maintains an allowance for expected losses resulting from the inability of its customers to make required payments. In addition to judgments about the creditworthiness of significant customers based on ongoing credit evaluations, the Company considers historical levels of credit losses, as well as macroeconomic and industry trends to determine the amount of the allowance. The allowance for uncollectible accounts receivable was $35 million and $34 million as of May 31, 2023 and 2022, respectively.\n\nINVENTORY VALUATION""]",Realized gains and losses on the sale of securities are determined by specific identification.,multi_context,[{'source': 'resources/nke-10k-2023.pdf'}],True diff --git a/requirements.txt b/requirements.txt index 0f4fcc7..3e1067a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,6 +4,7 @@ openai tiktoken langchain langchain-community +langchain-openai llama-index llama-index-vector-stores-redis llama-index-embeddings-cohere @@ -16,3 +17,6 @@ nbval redis redisvl pytest +ragas +datasets +