contrib/notebooks/qsv-describegpt-qa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f945346-0256-4a7d-8784-4894e4b74b3c",
   "metadata": {},
   "source": [
    "# Test Results for qsv describegpt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ab97c5c-01ef-473e-8f93-d1fc5cc8748b",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecd16ee0-9915-4130-b56c-301254b6a8b1",
   "metadata": {},
   "source": [
    "*Welcome to the \"Test Results\" page for qsv describegpt command. This page presents the outcomes and observations from the comprehensive Quality Assurance (QA) testing conducted on the \"describegpt\" command in Jupyter Lab.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2781e96f-d48b-4fdc-940b-d4ac691f6ce4",
   "metadata": {},
   "source": [
    "*This page contains detailed information and observations for each test case conducted during the QA process. You will find a summary of the tests performed, along with the outcomes and any noteworthy observations. The results will highlight the strengths and capabilities of the ``describegpt`` command, providing valuable insights to the development team and stakeholders.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc2accc9-bbca-40f7-a079-2ad6f2c09c01",
   "metadata": {},
   "source": [
    "*Before we begin the testing let's explore the various options that comes with describegpt. \n",
    "familiarizing ourselves with options ensures that we understand the available functionalities and tailor the tests to comprehensively evaluate each option's performance. This overview provides a solid foundation for conducting a thorough and effective QA testing process. Different options cater to different use cases and provide versatile functionality.\n",
    "``--help`` displays all the options of describegpt*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0588962a-6667-48a6-914a-84ddaeaf51e2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Infers extended metadata about a CSV using summary statistics.\n",
      "\n",
      "Note that this command uses OpenAI's LLMs for inferencing and is therefore prone to\n",
      "inaccurate information being produced. Verify output results before using them.\n",
      "\n",
      "For examples, see https://github.com/dathere/qsv/blob/master/tests/test_describegpt.rs.\n",
      "\n",
      "For more detailed info on how describegpt works and how to prepare a prompt file, \n",
      "see https://github.com/dathere/qsv/blob/master/docs/Describegpt.md\n",
      "\n",
      "Usage:\n",
      "    qsv describegpt [options] [<input>]\n",
      "    qsv describegpt --help\n",
      "\n",
      "describegpt options:\n",
      "    -A, --all              Print all extended metadata options output.\n",
      "    --description          Print a general description of the dataset.\n",
      "    --dictionary           For each field, prints an inferred type, a \n",
      "                           human-readable label, a description, and stats.\n",
      "    --tags                 Prints tags that categorize the dataset. Useful\n",
      "                           for grouping datasets and filtering.\n",
      "    --openai-key <key>     The OpenAI API key to use.\n",
      "                           If the QSV_OPENAI_KEY envvar is set, it will be used instead.                           \n",
      "    --max-tokens <value>   Limits the number of generated tokens in the output.\n",
      "                           [default: 50]\n",
      "    --json                 Return results in JSON format.\n",
      "    --jsonl                Return results in JSON Lines format.\n",
      "    --prompt-file <file>   The JSON file containing the prompts to use for inferencing.\n",
      "                           If not specified, default prompts will be used.\n",
      "    --model <model>        The model to use for inferencing.\n",
      "                           [default: gpt-3.5-turbo-16k]\n",
      "    --timeout <secs>       Timeout for OpenAI completions in seconds.\n",
      "                           [default: 60]\n",
      "    --user-agent <agent>   Specify custom user agent. It supports the following variables -\n",
      "                           $QSV_VERSION, $QSV_TARGET, $QSV_BIN_NAME, $QSV_KIND and $QSV_COMMAND.\n",
      "                           Try to follow the syntax here -\n",
      "                           https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent\n",
      "\n",
      "Common options:\n",
      "    -h, --help             Display this message\n",
      "    -o, --output <file>    Write output to <file> instead of stdout.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d331cbf6-ace1-4815-838f-5a87a7bd246b",
   "metadata": {},
   "source": [
    "*Let's now proceed to test each option and assess describegpt.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83b75706-ef07-433c-af4f-0077ae1581cd",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Integration Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395989d4-f091-4d10-93d0-adeaac949cce",
   "metadata": {},
   "source": [
    "*This testing phase focuses on verifying the seamless integration between the \"describegpt\" command and OpenAI's GPT chat completion models API. The objective is to ensure that the command effectively leverages the API to provide extended metadata inferences for CSV datasets.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5dd4a430-e4ad-4485-a113-a72fb90a08d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset contains information about individuals, addresses, and locations. It consists of multiple fields such as \"John,\" \"Doe,\" \"120 jefferson st.,\" \"Riverside,\" \"NJ,\" and \"08075,\" which are all"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from addresses.csv using qsv stats --everything...\n",
      "Generating frequency from addresses.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --openai-key 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 addresses.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb7cf287-d0d6-4162-8068-2d8ad1737285",
   "metadata": {},
   "source": [
    "As we can see describegpt seemlessly integrates with the OpenAI API with the API key to execute the command and give the output. describegpt effectively communicated with the API, utilizing the specified API key for inferencing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5739edd8-bbd8-4787-95bb-a10a9eb4ff88",
   "metadata": {},
   "source": [
    "Did not cause any conflicts or unexpected behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b03b5d-26ce-422a-b504-239b08bf6248",
   "metadata": {},
   "source": [
    "> ⚠️*Psst! Using an API key without any credits will result in describegpt not working. Make sure you use the correct API key and ensure you have enough credits associated with the key to continue. You can visit this link to check your available credits https://platform.openai.com/account/usage. \n",
    "If you don't yet have an API key you can generate one here https://platform.openai.com/account/api-keys*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf07aa8-36c3-4e9e-b65b-d404ac18b2cc",
   "metadata": {},
   "source": [
    "<a id='1'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "cbe385d3-d270-46df-a7d3-feb7cd54f86a",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from addresses.csv using qsv stats --everything...\n",
      "Generating frequency from addresses.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Error while requesting OpenAI models: Error response when making request: {\n",
      "  \"error\": {\n",
      "    \"message\": \"Incorrect API key provided: sk-n90nv**************************************eoKa. You can find your API key at https://platform.openai.com/account/api-keys.\",\n",
      "    \"type\": \"invalid_request_error\",\n",
      "    \"param\": null,\n",
      "    \"code\": \"invalid_api_key\"\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --openai-key sk-O.......w addresses.csv \n",
    "#wrong API key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4d036b-8692-4c54-971d-567967ada7c3",
   "metadata": {},
   "source": [
    "<a id='2'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "571d44b7-967d-4a6d-a31c-12f37eda6c23",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from addresses.csv using qsv stats --everything...\n",
      "Generating frequency from addresses.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Error response when making request: {\n",
      "    \"error\": {\n",
      "        \"message\": \"You exceeded your current quota, please check your plan and billing details.\",\n",
      "        \"type\": \"insufficient_quota\",\n",
      "        \"param\": null,\n",
      "        \"code\": \"insufficient_quota\"\n",
      "    }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --openai-key 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 addresses.csv \n",
    "#API key with 0 credits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "06b7cd1c-b2d1-42e6-a832-2996daac00e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error: QSV_OPENAI_KEY environment variable not found.\n",
      "Note that this command uses OpenAI's LLMs for inferencing and is therefore prone to inaccurate information being produced. Verify output results before using them.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description \"World Happiness Report.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9504a31-6c44-461d-bf05-728c01259e9c",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7af9e81-fadb-40f0-9c6c-94906a9d42d8",
   "metadata": {},
   "source": [
    "## Functional Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01abf0f4-96cf-495b-ba54-011b2f180ea7",
   "metadata": {},
   "source": [
    "Functional testing is a crucial phase of Quality Assurance QA for describegpt in qsv v0.109. This testing phase focuses on evaluating the individual functionalities and behaviors of the describegpt options ``--description`` ``--dictionary`` and ``--tags``. The objective is to ensure that each option operates as intended, providing accurate and reliable extended metadata inferences for CSV datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e1f61bd-75db-420a-a18a-6eb0999a2f3b",
   "metadata": {},
   "source": [
    "Here we'll be verifying if ``--all``, ``--description``, ``--dictionary``, ``--tags`` correctly produces the expected output and accurate metadata inferences. Our testing process will involve evaluating various datasets with different complexities and sizes to ensure that the generated descriptions are relevant, concise, and aligned with the dataset's content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2b13230-300a-4f0a-a00a-19c15f8bdfb2",
   "metadata": {},
   "source": [
    "Let's delve into the testing process, the test cases, and the types of CSV files we'll be using to ensure the reliability and robustness describegpt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751c3c20-4fae-405f-b477-6e1b6f4b2aee",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Small CSV"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0354580b-3a08-4e29-90aa-a39d5cd362fb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "\n",
    "We have a csv file which contains the ranking for the countries from the World Happiness Report. It contains the happiness scored according to economic production, social support, etc. It's a small csv file with 156 ranked on basis of different parameters with Numerical and text data types "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "838dc237-5edc-4c68-8145-b98d4534a355",
   "metadata": {},
   "source": [
    "Here's the link for the dataset https://www.kaggle.com/datasets/unsdsn/world- License https://creativecommons.org/publicdomain/zero/1.0/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "598b1d30-d1a3-44e9-a82d-a1c8f97b66d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "156\n"
     ]
    }
   ],
   "source": [
    "!qsv count \"World Happiness Report.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e00c6020-f95b-4ae3-9e4b-7a37bc5ba374",
   "metadata": {},
   "source": [
    "Let's try the --description option on this CSV file and let's see what comes out.\n",
    "Expected output: \"Description of the dataset\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "31d5eb87-271b-4523-a71a-2eb4811e2c68",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"QSV_OPENAI_KEY\"] = \"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\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "350008bb-da0b-4484-8463-5245b054d321",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset contains information on various countries and their corresponding scores on different factors contributing to happiness. The dataset includes overall rank, country or region name, score, GDP per capita, social support, healthy life expectancy, freedom to make life choices, generosity, and perceptions of corruption. The overall rank ranges from 1 to 156, with a mean of 78.5 and a standard deviation of 45.0324. The scores range from 2.853 to 7.769, with a mean of 5.4071 and a standard deviation of 1.1095. The dataset includes information on the frequency of each rank, country or region, score, GDP per capita, social support, healthy life expectancy, freedom to make\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 150 \"World Happiness Report.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "c64d205a-5e5a-4952-a746-05938d5a68fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"Overallrank\": 1, \"Countryorregion\": 1, \"Score\": 1, \"GDPpercapita\": 1, \"Socialsupport\": 1, \"Healthylifeexpectancy\": 1, \"Freedom\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags \"World Happiness Report.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "9356aad4-c5ea-4cbc-85fe-8ba983ea425a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"fields\": [\n",
      "    {\n",
      "      \"Name\": \"Overall rank\",\n",
      "      \"Type\": \"Integer\",\n",
      "      \"Label\": \"Overall rank\",\n",
      "      \"Description\": \"The overall rank of a country or region\"\n",
      "    },\n",
      "    {\n",
      "     \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --dictionary \"World Happiness Report.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6e86f76-7148-432d-b41b-454c6902551e",
   "metadata": {},
   "source": [
    "#### Medium Size CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b65818b7-ea59-4528-b5ec-d06846047e7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"summary_statistics\": {\n",
      "    \"field\": \"string\",\n",
      "    \"type\": \"string\",\n",
      "    \"sum\": \"string\",\n",
      "    \"min\": \"string\",\n",
      "    \"max\": \"string\",\n",
      "    \"range\": \"string\",\n",
      "    \"min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from addresses.csv using qsv stats --everything...\n",
      "Generating frequency from addresses.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --openai-key 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 addresses.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "9c24c165-2bc4-4dff-abc4-79c6463fff60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18249\n",
      "CPU times: total: 0 ns\n",
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "!qsv count avocado.csv\n",
    "%time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "b9165252-8353-430e-8a3f-ea64d0ad1b00",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset contains information about avocado sales, including the average price, total volume, quantity of specific types of avocados (4046, 4225, 4770), and the number of bags of avocados sold. The dataset also includes dates, regions, and the type of avocado (conventional or organic). The dataset spans from 2015 to 2018. The average price of avocados ranges from 0.44 to 3.25, with a mean of 1.406. The total volume of avocados sold ranges from 84.56 to 62,505,646.52, with a mean of 850,644.013. The dataset contains a variety of regions, ranging from Albany to WestTexNewMexico, with Albany being the most frequent region and WestTexNewMexico being the least frequent. The dataset has a total of 432 data points.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from avocado.csv using qsv stats --everything...\n",
      "Generating frequency from avocado.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 250 avocado.csv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "1cbd1152-bb4f-4c62-a370-7b5b2b45e0b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Marital status\": {\n",
      "    \"Name\": \"Marital status\",\n",
      "    \"Type\": \"Integer\",\n",
      "    \"Label\": \"Marital status\",\n",
      "    \"Description\": \"Marital status of the individual\"\n",
      "  },\n",
      "  \"Application\n",
      "{\n",
      "  \"Description\": \"This dataset contains information about individuals, including their marital status, application mode, application order, course, attendance, previous qualification, nationality, mother's qualification, father's qualification, mother's occupation, father's occupation, displacement\n",
      "{\n",
      "  \"Marital status\": [\"maritalstatus\", \"status\", \"relationship\"],\n",
      "  \"Application mode\": [\"applicationmode\", \"mode\", \"apply\"],\n",
      "  \"Application order\": [\"applicationorder\", \"order\"],\n",
      "  \"Course\": [\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Dropout Rate.csv using qsv stats --everything...\n",
      "Generating frequency from Dropout Rate.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all \"Dropout Rate.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c1876e-0ea3-4bb6-83d3-e9cc86fae5f9",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Large Size CSV\n",
    "> ###### ``Test Flagged 🚩``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "484f97a7-75eb-4c10-b63b-407063945aaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2467\n"
     ]
    }
   ],
   "source": [
    "!qsv count transcripts.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e374b836-835e-4c36-a97c-b503e9fc227f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from transcripts.csv using qsv stats --everything...\n",
      "Generating frequency from transcripts.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Error response when making request: {\n",
      "  \"error\": {\n",
      "    \"message\": \"This model's maximum context length is 16385 tokens. However, your messages resulted in 34746 tokens. Please reduce the length of the messages.\",\n",
      "    \"type\": \"invalid_request_error\",\n",
      "    \"param\": \"messages\",\n",
      "    \"code\": \"context_length_exceeded\"\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max-tokens 250 transcripts.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b24c1f82-d842-4289-bb26-fbb8b9f19de4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from transcripts.csv using qsv stats --everything...\n",
      "Generating frequency from transcripts.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Error response when making request: {\n",
      "    \"error\": {\n",
      "        \"message\": \"Rate limit reached for default-gpt-3.5-turbo-16k in organization org-ZPfnuhVb5tuJxnK1voGb2WQL on tokens per min. Limit: 40000 / min. Please try again in 1ms. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.\",\n",
      "        \"type\": \"tokens\",\n",
      "        \"param\": null,\n",
      "        \"code\": \"rate_limit_exceeded\"\n",
      "    }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 250 transcripts.csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd0a3111-0b69-4eb5-9702-41a9e20e5e5f",
   "metadata": {},
   "source": [
    "🚩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "bfab0060-39a9-4f40-a488-8758efaf2798",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"data\": [\n",
      "    {\n",
      "      \"Name\": \"field\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Field\",\n",
      "      \"Description\": \"The field name\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"type\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Type\",\n",
      "      \"Description\": \"The data type of the field\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"label\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Label\",\n",
      "      \"Description\": \"A human-friendly label for the field\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"description\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Description\",\n",
      "      \"Description\": \"A full description for the field\"\n",
      "    }\n",
      "  ],\n",
      "  \"rows\": [\n",
      "    {\n",
      "      \"field\": \"comments\",\n",
      "      \"type\": \"Integer\",\n",
      "      \"label\": \"Comments\",\n",
      "      \"description\": \"The number of comments on the talk\"\n",
      "    },\n",
      "    {\n",
      "      \"field\": \"description\",\n",
      "      \"type\": \"String\",\n",
      "      \"label\": \"Description\",\n",
      "      \"description\": \"The description of the talk\"\n",
      "    },\n",
      "    {\n",
      "      \"field\": \"duration\",\n",
      "      \"type\": \"Integer\",\n",
      "     \n",
      "The dataset contains information on different fields such as comments, description, duration, event, film date, languages, main speaker, name, number of speakers, published date, ratings, related talks, speaker occupation, tags, title, url, and views. It includes a variety of data types such as integers, strings, and lists. The dataset explores topics such as the number of comments on talks, the description of the talks, the duration of the talks, the language in which the talks are given, and the ratings and views of the talks. It also includes related talks, speaker occupations, tags, and URLs for further exploration.\n",
      "{\n",
      "  \"tags\": [\n",
      "    \"comments\",\n",
      "    \"description\",\n",
      "    \"duration\",\n",
      "    \"event\",\n",
      "    \"film_date\",\n",
      "    \"languages\",\n",
      "    \"main_speaker\",\n",
      "    \"name\",\n",
      "    \"num_speaker\",\n",
      "    \"published_date\",\n",
      "    \"ratings\",\n",
      "    \"related_talks\",\n",
      "    \"speaker_occupation\",\n",
      "    \"tags\",\n",
      "    \"title\",\n",
      "    \"url\",\n",
      "    \"views\"\n",
      "  ]\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from ted_main.csv using qsv stats --everything...\n",
      "Generating frequency from ted_main.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max-tokens 250 ted_main.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f1b2ab11-c16d-4e28-a044-019cac480806",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8807\n"
     ]
    }
   ],
   "source": [
    "!qsv count netflix_titles.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "abd81a80-f11e-4ccd-bd10-152b6bd1fc71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"fields\": [\n",
      "    {\n",
      "      \"Name\": \"show_id\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Show ID\",\n",
      "      \"Description\": \"A unique identifier for each show\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\n",
      "The dataset contains information about TV shows and movies, including their titles, directors, cast members, countries, date added, release years, ratings, durations, genres, and descriptions. There are a total of 8,807 shows in the dataset,\n",
      "{\n",
      "  \"tags\": [\n",
      "    \"keyword\",\n",
      "    \"label\",\n",
      "    \"categorizes\",\n",
      "    \"datasets\",\n",
      "    \"summary statistics\",\n",
      "    \"frequency data\",\n",
      "    \"CSV file\",\n",
      "    \"lowercase\",\n",
      "    \"whitespace\",\n",
      "   \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from netflix_titles.csv using qsv stats --everything...\n",
      "Generating frequency from netflix_titles.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all netflix_titles.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7b09ed81-7db0-4fbb-a72f-d57751a1d7b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23171\n"
     ]
    }
   ],
   "source": [
    "!qsv count Airline_review.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "386f5c01-f619-44c7-a782-8a83b955213a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Airline_review.csv using qsv stats --everything...\n",
      "Generating frequency from Airline_review.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Error response when making request: {\n",
      "  \"error\": {\n",
      "    \"message\": \"This model's maximum context length is 16385 tokens. However, your messages resulted in 24093 tokens. Please reduce the length of the messages.\",\n",
      "    \"type\": \"invalid_request_error\",\n",
      "    \"param\": \"messages\",\n",
      "    \"code\": \"context_length_exceeded\"\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all Airline_review.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "040432db-1916-4318-aaa7-a4640398cd57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36564\n"
     ]
    }
   ],
   "source": [
    "!qsv count research-and-development-survey-2022.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3c4162-3caf-4bef-af05-1c533ea33e68",
   "metadata": {},
   "source": [
    "<a id='3'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "ff14d54a-9539-4892-bb05-98c0809c9a82",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Unknown flag: '--max_tokens'. Did you mean '--max-tokens'?\n",
      "\n",
      "Usage:\n",
      "    qsv describegpt [options] [<input>]\n",
      "    qsv describegpt --help\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max_tokens 250 research-and-development-survey-2022.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "7d8481ba-7983-4a5e-834d-409ad3dde224",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"columns\": [\n",
      "    {\n",
      "      \"Name\": \"Variable\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Variable\",\n",
      "      \"Description\": \"The variable being measured\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"Breakdown\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Breakdown\",\n",
      "      \"Description\": \"The breakdown category\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"Breakdown_category\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Breakdown Category\",\n",
      "      \"Description\": \"The subcategory of the breakdown\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"Year\",\n",
      "      \"Type\": \"Integer\",\n",
      "      \"Label\": \"Year\",\n",
      "      \"Description\": \"The year of the data\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"RD_Value\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"RD Value\",\n",
      "      \"Description\": \"The value of RD\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"Status\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Status\",\n",
      "      \"Description\": \"The status of the data\"\n",
      "    },\n",
      "    {\n",
      "      \"Name\": \"Unit\",\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Unit\",\n",
      "\n",
      "The dataset consists of information about various variables related to research and development (RD) in different sectors and industries. It includes data for multiple years, with the most recent being 2022. The dataset also provides information on the breakdown of RD expenditure by different categories. The variables cover a wide range of aspects, including personnel, funding sources, engagement with external parties, and the purpose of research. The dataset includes information on the total RD expenditure as a proportion of GDP and the RD expenditure performed overseas. It also contains data on the count and distribution of businesses and employees involved in RD activities. The dataset provides a comprehensive view of RD activities in different sectors and can be used to analyze trends and patterns in RD expenditure and engagement.\n",
      "{\n",
      "  \"Variable\": [\n",
      "    \"totalrdexpenditure\",\n",
      "    \"currentandexpenditurewagessalaries\",\n",
      "    \"personnelbyoccupationfteresearchertotal\",\n",
      "    \"personnelbyqualificationheadcountpostgraduatestudents\",\n",
      "    \"typeofengagementaboutbusinessrdneedscontactmadethroughprofessionalcontactsbetweenstaffmembersengagedwithuniversities\",\n",
      "    \"purposeofresearchother\",\n",
      "    \"fundingofrdundertakenexternallybythegovernmentsectornewzealandcentralgovernment\",\n",
      "    \"typeofengagementaboutbusinessrdneedscontactmadethroughnetworkingeventsxengagedwithuniversities\",\n",
      "    \"currentandcapitalexpenditurewagessalaries\",\n",
      "    \"typeofresearchexperimental\",\n",
      "    \"personnelbyoccupationheadcountresearcheremployees\"\n",
      "  ],\n",
      "  \"Breakdown\": [\n",
      "    \"publishedindustry\",\n",
      "    \"anzsic1digit\",\n",
      "    \"size\",\n",
      "    \"industry\",\n",
      "    \"sector\",\n",
      "    \"country\",\n",
      "    \"total\",\n",
      "    \"fwholesaletrade\",\n",
      "    \"jinformationmediaandtelecommunications\",\n",
      "    \"11scientificresearchandtechnicalservices\"\n",
      "  ],\n",
      "  \"Breakdown_category\": [\n",
      "    \"fwholesaletrade\",\n",
      "    \"jinformationmediaandtelecommunications\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from research-and-development-survey-2022.csv using qsv stats --everything...\n",
      "Generating frequency from research-and-development-survey-2022.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max-tokens 250 research-and-development-survey-2022.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f1291d-6181-4144-bdd3-6b6586e27463",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Missing Data CSV type\n",
    "> ###### ``Test OK ✔️``"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e0c56bd-df5a-4ca0-a6cc-5df153f967df",
   "metadata": {},
   "source": [
    "In this CSV file, some rows have missing data in certain columns. For example, the third row has a missing value for the \"Revenue\" column, the fifth row has a missing value for the \"UnitsSold\" column, and the seventh row has missing values for both the \"CarModel\" and \"UnitsSold\" columns. These missing data points can occur in real-world scenarios when dealing with sales data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c00f8e-c0ee-424f-b1b5-828b3a6c37d8",
   "metadata": {},
   "source": [
    "| Date       | CarModel            | UnitsSold | Revenue |\n",
    "|------------|---------------------|-----------|---------|\n",
    "| 2023-01-01 | Toyota Camry        | 100       | 50000   |\n",
    "| 2023-01-02 | Ford Mustang        | 75        | 60000   |\n",
    "| 2023-01-03 | Honda Civic         |           | 40000   |\n",
    "| 2023-01-04 | Chevrolet Silverado | 120       | 80000   |\n",
    "| 2023-01-05 | Jeep Wrangler       | 90        |         |\n",
    "| 2023-01-06 | Nissan Altima       | 80        | 55000   |\n",
    "| 2023-01-07 |                     | 95        | 65000   |\n",
    "| 2023-01-08 | Ford Explorer       | 105       | 70000   |\n",
    "| 2023-01-09 | Toyota Corolla      | 70        |         |\n",
    "| 2023-01-10 | Honda CR-V          | 110       | 75000   |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab0a1fa0-ce90-4fb0-8a27-b20af1e9b58a",
   "metadata": {},
   "source": [
    "_+Additional More Rows, Source: AI Generated_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83481ed-f571-4c11-ae0b-9fc16014ac82",
   "metadata": {},
   "source": [
    "Here qsv should ignore the missing inputs of the set and move on with whats provided "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fae5393-2424-4252-bc8b-909c5392b6d4",
   "metadata": {},
   "source": [
    "###### Expected output:\n",
    "> This dataset contains information about car company sales. It includes details such as the CarModel, Date, UnitsSold and Revenue. Some rows have missing data for certain columns. The dataset appears to be a mix of sales records for different car models over multiple dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "5dc4fde7-7e9c-43d7-86fc-4c3da6a16d7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"dictionary\": {\n",
      "    \"error\": \"Error: Invalid JSON output for dictionary.\"\n",
      "  },\n",
      "  \"description\": {\n",
      "    \"error\": \"Error: Invalid JSON output for description.\"\n",
      "  },\n",
      "  \"tags\": {\n",
      "    \"error\": \"Error: Invalid JSON output for tags.\"\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Car_Sales_Data_With_Missing_Data.csv using qsv stats --everything...\n",
      "Generating frequency from Car_Sales_Data_With_Missing_Data.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "{\"error\":\"Error: Invalid JSON output for dictionary.\"}\n",
      "Output: {\n",
      "    \"Name\": \"Date\",\n",
      "    \"Type\": \"String\",\n",
      "    \"Label\": \"Date\",\n",
      "    \"Description\": \"The date of the record\"\n",
      "},\n",
      "{\n",
      "    \"Name\": \"CarModel\",\n",
      "    \"Type\": \"String\",\n",
      "    \"Label\": \"Car Model\",\n",
      "    \"Description\": \"The model of the car\"\n",
      "},\n",
      "{\n",
      "    \"Name\": \"UnitsSold\",\n",
      "    \"Type\": \"Integer\",\n",
      "    \"Label\": \"Units Sold\",\n",
      "    \"Description\": \"The number of units sold\"\n",
      "},\n",
      "{\n",
      "    \"Name\": \"Revenue\",\n",
      "    \"Type\": \"Integer\",\n",
      "    \"Label\": \"Revenue\",\n",
      "    \"Description\": \"The revenue generated\"\n",
      "}\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "{\"error\":\"Error: Invalid JSON output for description.\"}\n",
      "Output: The dataset contains information about car sales, including the date, car model, number of units sold, and the revenue generated. The data covers a time period from January 1, 2023, to February 10, 2023. There are a total of 41 records in the dataset, with 28 distinct car models. The units sold range from a minimum of 50 to a maximum of 120, with a mean of 89.41 and a standard deviation of 17.40. The revenue generated ranges from $40,000 to $90,000, with a mean of $63,181.82 and a standard deviation of $12,781.27.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n",
      "{\"error\":\"Error: Invalid JSON output for tags.\"}\n",
      "Output: [\n",
      "    \"date\",\n",
      "    \"carmodel\",\n",
      "    \"unitssold\",\n",
      "    \"revenue\",\n",
      "    \"sum\",\n",
      "    \"min\",\n",
      "    \"max\",\n",
      "    \"range\",\n",
      "    \"minlength\",\n",
      "    \"maxlength\",\n",
      "    \"mean\",\n",
      "    \"stddev\",\n",
      "    \"variance\",\n",
      "    \"nullcount\",\n",
      "    \"sparsity\",\n",
      "    \"mad\",\n",
      "    \"lowerouterfence\",\n",
      "    \"lowerinnerfence\",\n",
      "    \"q1\",\n",
      "    \"q2median\",\n",
      "    \"q3\",\n",
      "    \"iqr\",\n",
      "    \"upperinnerfence\",\n",
      "    \"upperouterfence\",\n",
      "    \"skewness\",\n",
      "    \"cardinality\",\n",
      "    \"mode\",\n",
      "    \"modecount\",\n",
      "    \"modeoccurrences\",\n",
      "    \"antimode\",\n",
      "    \"antimodecount\",\n",
      "    \"antimodeoccurrences\",\n",
      "    \"2023\",\n",
      "    \"01\",\n",
      "    \"23\",\n",
      "    \"31\",\n",
      "    \"02\",\n",
      "    \"10\",\n",
      "    \"17\",\n",
      "    \"14\",\n",
      "    \"03\",\n",
      "    \"05\",\n",
      "    \"07\",\n",
      "    \"15\",\n",
      "    \"29\",\n",
      "    \"null\",\n",
      "    \"bmw\",\n",
      "    \"x5\",\n",
      "    \"toyota\",\n",
      "    \"sienna\",\n",
      "    \"preview\",\n",
      "    \"chevrolet\",\n",
      "    \"equinox\",\n",
      "    \"tahoe\",\n",
      "    \"ford\",\n",
      "    \"bronco\",\n",
      "    \"escape\",\n",
      "    \"explorer\",\n",
      "    \"f-150\",\n",
      "    \"honda\",\n",
      "    \"cr-v\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max-tokens 300 --json Car_Sales_Data_With_Missing_Data.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "465bafeb-fe8a-4f92-a08e-c4794475d6d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset consists of information about car sales and revenue, spanning from January 1st, 2023 to February 10th, 2023. There are a total of 41 records in the dataset, with each record containing details such as the car model, units sold, and revenue generated. The car models range from BMW X5 to Toyota Sienna, with BMW X5 being the most commonly occurring car model in the dataset. The number of units sold ranges from 50 to 120, with an average of 89.41 units sold per record. Similarly, the revenue generated ranges from 40,000 to 90,000, with an average of 63,181.82. The dataset does not have any missing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Car_Sales_Data_With_Missing_Data.csv using qsv stats --everything...\n",
      "Generating frequency from Car_Sales_Data_With_Missing_Data.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 150 Car_Sales_Data_With_Missing_Data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f54126-ccb3-4d86-b106-9f703dc0247c",
   "metadata": {},
   "source": [
    "That's great to hear! describegpt passed successfully! It provided a meaningful and accurate summary of the CSV dataset, as expected"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99c3031e-f7a1-49cb-a7b5-8cef17548ff0",
   "metadata": {},
   "source": [
    "It is also worth noting that the ``--description`` option gives a different output as compared to when used with ``--all``"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39c0b53b-db84-43a7-91af-fd16222c853d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Complex Structure CSV files\n",
    "> ###### ``Test Flagged 🚩``"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e102b6-cb98-4695-b41b-e195aa175037",
   "metadata": {},
   "source": [
    "Here's an example of a CSV file with complex datasets containing nested structures and irregular formats:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5d9d3ef-778c-4a34-979d-227400700e64",
   "metadata": {},
   "source": [
    "In this CSV file, the dataset contains **complex nested structures** and **irregular formats**. For example, the \"CarDetails\" column contains nested data with car model and year information. The \"Orders\" column contains nested data with item names and quantities. Some rows have missing data for certain columns, and the data format varies in each row, making it a complex dataset with irregular structures. Handling such complex datasets requires careful data parsing and processing to extract meaningful information accurately."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfe96c14-3bbe-4438-8ee6-61aa1f3b5f67",
   "metadata": {},
   "source": [
    "| Name                | Age | Contact     | Address                 | CarDetails                              | Orders                                                 |\n",
    "|---------------------|-----|-------------|-------------------------|-----------------------------------------|--------------------------------------------------------|\n",
    "| John Doe            | 25  | 1234567890  | 123 Main St, City      | Model: Toyota Camry, Year: 2023        | Order1: Item1, Quantity: 2; Order2: Item3, Quantity: 1 |\n",
    "| Alice Nuno          | 32  | 9876543210  | 456 Elm St, Town       | Model: Ford Mustang, Year: 2022        | Order1: Item2, Quantity: 3; Order2: Item4, Quantity: 2; Order3: Item6, Quantity: 1 |\n",
    "| Emilie Levesque     | 28  | 5551234567  | 789 Oak St, Village    | Model: Honda Civic, Year: 2023         | Order1: Item1, Quantity: 1; Order2: Item5, Quantity: 4   |\n",
    "| Tomasz Wojcik       | 30  | 7893216540  | 321 Pine St, County    | Model: Chevrolet Silverado, Year: 2021 | Order1: Item3, Quantity: 2; Order2: Item5, Quantity: 3; Order3: Item7, Quantity: 1 |\n",
    "|                     |     | 4567891230  | 654 Maple St, Village  | Model: Jeep Wrangler, Year: 2023       | Order1: Item2, Quantity: 1; Order2: Item4, Quantity: 2   |\n",
    "| Sonke Muller        |     | 3216549870  | 987 Cherry St, City    | Model: Nissan Altima, Year: 2022      | Order1: Item1, Quantity: 2; Order2: Item6, Quantity: 1; Order3: Item7, Quantity: 3 |\n",
    "| Kathe Gruber        | 27  | 8765432109  |                         | Model: Ford Explorer, Year: 2023      | Order1: Item2, Quantity: 3; Order2: Item4, Quantity: 1; Order3: Item7, Quantity: 2 |\n",
    "| Inaki Lopez         | 31  | 2345678901  | 123 Pine St, City      | Model: Toyota Corolla, Year: 2022     | Order1: Item1, Quantity: 1; Order2: Item2, Quantity: 2; Order3: Item5, Quantity: 1 |\n",
    "| Andre Silva         | 26  | 7890123456  | 456 Oak St, Town       | Model: Honda CR-V, Year: 2021         | Order1: Item6, Quantity: 2; Order2: Item7, Quantity: 3   |\n",
    "| Jose Cordoba        | 30  | 6543210987  | 321 Maple St, City     | Model: Subaru Outback, Year: 2023     | Order1: Item1, Quantity: 2; Order2: Item2, Quantity: 1; Order3: Item3, Quantity: 3 |\n",
    "|                     |     | 2345678901  | 789 Cherry St, County  | Model: Mercedes-Benz C-Class, Year: 2022 | Order1: Item4, Quantity: 4; Order2: Item5, Quantity: 2; Order3: Item6, Quantity: 1 |\n",
    "| Mohan Gupta         | 33  | 6543210987  | 987 Elm St, Village    | Model: BMW X5, Year: 2023             | Order1: Item1, Quantity: 1; Order2: Item3, Quantity: 2   |\n",
    "| Hyunwoo Jo          | 28  | 7894561230  | 123 Maple St, City     | Model: Chevrolet Tahoe, Year: 2021    | Order1: Item2, Quantity: 2; Order2: Item4, Quantity: 1; Order3: Item7, Quantity: 3 |\n",
    "| An Jie Li           | 27  |             |                         | Model: Ford F-150, Year: 2022         | Order1: Item1, Quantity: 3; Order2: Item2, Quantity: 1; Order3: Item3, Quantity: 2 |\n",
    "| Mera Nam Abhishek Hai | 25 | 4567891230  | 456 Pine St, County    | Model: Honda CR-V, Year: 2022         | Order1: Item1, Quantity: 2; Order2: Item2, Quantity: 1; Order3: Item3, Quantity: 2 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e330513-77ec-4bd7-99d4-2c6bca0c5b42",
   "metadata": {},
   "source": [
    "The expected output for the ``--description`` should be a concise and informative summary of the CSV dataset. It should provide a general description that captures the overall theme or content of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8e7ef87-d5f2-43b0-876a-28d18f52e84e",
   "metadata": {},
   "source": [
    "###### Expected output:\n",
    "> This dataset contains information about car company sales. It includes details such as the name of the customer, their age, contact information, address, car model and year, and order details. Some rows have missing data for certain columns, and the data format varies in each row. The dataset appears to be a mix of sales records for different car models over multiple dates. Further analysis can provide more insights into the sales trends and customer preferences.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "32731eef-72bb-40c6-bf83-adacff7b1ce8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset, in JSON format, provides a comprehensive overview of the data from the CSV file. It includes various summary statistics and frequency information. The summary statistics offer insights into the central tendency, spread, and shape of the data, while the frequency data showcases the occurrence of values within the dataset. By analyzing this dataset, users can gain a holistic understanding of the data distribution and patterns.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Car_sales_data_with_complexstructure.csv using qsv stats --everything...\n",
      "Generating frequency from Car_sales_data_with_complexstructure.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 200 Car_sales_data_with_complexstructure.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "fbe33fa7-b566-46fe-8b8c-ae5da237e28a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"fields\": [\n",
      "    {\n",
      "      \"name\": \"column_name\",\n",
      "      \"type\": \"data_type\",\n",
      "      \"label\": \"human-friendly label\",\n",
      "      \"description\": \"full description\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"column_name\",\n",
      "      \"type\": \"data_type\",\n",
      "      \"label\": \"human-friendly label\",\n",
      "      \"description\": \"full description\"\n",
      "    },\n",
      "    ...\n",
      "  ]\n",
      "}\n",
      "{\n",
      "  \"description\": \"The dataset contains information about XYZ. It includes various fields such as A, B, and C. Field A represents XYZ, field B represents XYZ, and field C represents XYZ. The dataset provides valuable insights into XYZ and can be used for XYZ purposes.\"\n",
      "}\n",
      "{\n",
      "  \"fields\": [\n",
      "    {\n",
      "      \"name\": \"summary_statistic\",\n",
      "      \"type\": \"data_type\",\n",
      "      \"label\": \"human-friendly label\",\n",
      "      \"description\": \"full description\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"summary_statistic\",\n",
      "      \"type\": \"data_type\",\n",
      "      \"label\": \"human-friendly label\",\n",
      "      \"description\": \"full description\"\n",
      "    },\n",
      "    ...\n",
      "  ]\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Car_sales_data_with_complexstructure.csv using qsv stats --everything...\n",
      "Generating frequency from Car_sales_data_with_complexstructure.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max-tokens 200 Car_sales_data_with_complexstructure.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40bbac45-0fe3-43bd-bf39-b43266ef2f31",
   "metadata": {},
   "source": [
    "The output provided by the --description option in this case seems to be inaccurate and generic. It appears to be using placeholders like \"XYZ\" and providing redundant information without offering specific insights into the actual dataset's content."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ed69b9-15e5-4a84-9c5e-216ec865cab5",
   "metadata": {},
   "source": [
    "It is also worth Noting that the ``--description`` option gives a different output as compared to when used with ``--all``"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc0a7c44-7635-410d-8306-ebe19943ae1d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "This should raise a flag 🚩"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1b7f99f-1922-4799-922a-9db5d0bf4cbe",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Mismatched data CSV file\n",
    "> ###### ``Test Flagged 🚩``"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ec3d7a7-110b-4422-82b6-7905cf02d20c",
   "metadata": {},
   "source": [
    "In this CSV file the \"Age\" column has values like \"thirty-five\" and \"twenty-nine,\" which are not valid numeric entries. The \"Phone\" column has values like \"9999999999\" and \"abcde,\" which are not valid phone numbers. The \"IsSubscribed\" column has values like \"xyz\" and \"abcde,\" which are not valid boolean values (should be either \"true\" or \"false\")."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a12e29-15d2-43ce-ba2e-c22f7e152c86",
   "metadata": {},
   "source": [
    "| Name            | Age           | Email                    | Phone      | IsSubscribed |\n",
    "|-----------------|---------------|--------------------------|------------|--------------|\n",
    "| John Doe        | 25            | john.doe@example.com     | 1234567890 | true         |\n",
    "| Alice Nuno      | 32            | alice.nuno@example.com   | 9876543210 | true         |\n",
    "| Emilie Levesque | 28            | emilie.levesque@example.com | 5551234567 | 40000        |\n",
    "| Tomasz Wojcik   | 30            | tomasz.wojcik@example.com | 7893216540 | true         |\n",
    "|                 | 31            | jose.martinez@example.com | 4567891230 | false        |\n",
    "| Sofia Sanchez   | 24            | sofia.sanchez@example.com | 3216549870 | true         |\n",
    "| Laura Gonzalez  | 27            | laura.gonzalez@example.com| 8765432109 | true         |\n",
    "| Andre Silva     | 26            | andre.silva@example.com   | 7890123456 | 70000        |\n",
    "| Jose Cordoba    | 30            | jose.cordoba@example.com  | 6543210987 | true         |\n",
    "| Maria Ramos     | 25            | maria.ramos@example.com   | 4567891230 | true         |\n",
    "| Jane Smith      | thirty-two    | jane.smith@example.com    | 9876543210 | true         |\n",
    "| Michael Johnson | 28            | michael.johnson@example.com | 5551234567 | 90000        |\n",
    "| Sophie Anderson | 29            | sophie.anderson@example.com| 7893216540 | false        |\n",
    "| Robert          | 30            | robert@example.com        | 1234       | 123          |\n",
    "| Lucy            | 27            | lucy@example.com          | abcd       | true         |\n",
    "| Ella            | thirty-five  | ella@example.com          | 8765432109 | true         |\n",
    "| William         | 29            | william@example.com       | 9999999999 | xyz          |\n",
    "| Grace           | 22            | grace@example.com         | 4563217890 | true         |\n",
    "| Sophia          | 31            | sophia@example.com        | abcde      | false        |\n",
    "| Emma            | 26            | emma@example.com          | 9876543210 | true         |\n",
    "| Oliver          | twenty-nine  | oliver@example.com        | 2345678901 | true         |\n",
    "| Benjamin        | 24            | benjamin@example.com      | 6547893210 | false        |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86055803-091b-49f7-806c-da5da8f9eebc",
   "metadata": {},
   "source": [
    "###### Expected output:\n",
    "> This dataset dataset represents a list of individuals with various attributes such as \"Name,\" \"Age,\" \"Email,\" \"Phone,\" and \"IsSubscribed.\" It contains 21 records of individuals, each with different information. Few of fields contain unrelated datatypes probably are mismatched among other fields, it exhibits data inconsistencies and mismatched data types "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "439a7b7d-30e4-448e-ba84-3d2e6cfd41ee",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset consists of information about individuals, including their names, ages, email addresses, phone numbers, and subscription status. The names range from Alice Nuno to William, with a total of 22 unique names. The ages range from 22 to twenty-nine, with a mean age of 27 and a standard deviation of approximately 3.3. The dataset includes 16 unique email addresses and 27 unique phone numbers. The majority of individuals, 13 in total, have subscribed to something, while 4 have not. The most common phone number is 9876543210, with a count of 3. Overall, the dataset captures a diverse range of individuals and their corresponding attributes.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Mismatched.csv using qsv stats --everything...\n",
      "Generating frequency from Mismatched.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 200 Mismatched.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37537e8d-89cb-4b97-abe4-0534960bf411",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9d54c43c-42c5-411b-b7d7-1c8e2ea729dd",
   "metadata": {},
   "source": [
    "*Well. Not bad. describegpt did actually give us a good in-depth summary of the data set. However, it does not completely brings attention to the data quality issues that needs to be addressed, specifically the presence of non-numeric age entries and the inconsistent representation of subscription status. Addressing these issues will be essential to ensure the accuracy and reliability of any analyses or applications based on this dataset.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "b40e6860-8e22-4cac-9472-ba2e7bdcd581",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Name\": [\"alice\", \"nuno\", \"william\", \"emma\", \"michael\", \"johnson\", \"tomasz\", \"wojcik\", \"sophie\", \"anderson\", \"benjamin\", \"maria\", \"ramos\", \"oliver\", \"andre\", \"silva\"],\n",
      "  \"Age\": [\"22\", \"twenty-nine\", \"30\", \"31\", \"26\", \"27\", \"25\", \"28\", \"24\", \"thirty-five\"],\n",
      "  \"Email\": [\"alice.nuno@example.com\", \"william@example.com\", \"robert@example.com\", \"sofia.sanchez@example.com\", \"sophie.anderson@example.com\", \"emilie.levesque@example.com\", \"sophia@example.com\", \"jose.martinez@example.com\", \"john.doe@example.com\", \"andre.silva@example.com\", \"laura.gonzalez@example.com\"],\n",
      "  \"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Mismatched.csv using qsv stats --everything...\n",
      "Generating frequency from Mismatched.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --max-tokens 200 Mismatched.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "045527bb-e074-4b57-9b88-f6ecc0d10c24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Name\": [\"alice\", \"nuno\", \"william\", \"ella\", \"robert\", \"michael\", \"johnson\", \"jane\", \"smith\", \"andre\", \"silva\", \"oliver\", \"tomasz\", \"wojcik\", \"lucy\"],\n",
      "  \"Age\": [\"22\", \"thirty\", \"nine\", \"30\", \"28\", \"31\", \"25\", \"24\", \"26\", \"27\", \"twenty\", \"seven\", \"twenty\", \"nine\", \"thirty\", \"two\"],\n",
      "  \"Email\": [\"alice.nuno@example.com\", \"william@example.com\", \"jose.cordoba@example.com\", \"michael.johnson@example.com\", \"sophia@example.com\", \"robert@example.com\", \"maria.ramos@example.com\", \"emilie.levesque@example.com\", \"ella@example.com\", \"lucy@example.com\", \"laura.g\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Mismatched.csv using qsv stats --everything...\n",
      "Generating frequency from Mismatched.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --max-tokens 200 Mismatched.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "2972d9e4-1718-46bf-8458-3acc02ee9d10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"name\":[\"alice\",\"nuno\",\"william\",\"tomasz\",\"wojcik\",\"benjamin\",\"emma\",\"michael\",\"johnson\",\"sofia\",\"sanchez\",\"grace\",\"ella\",\"emilie\",\"levesque\",\"john\",\"doe\",\"andre\",\"silva\",\"benjamin\",\"andre\",\"silva\",\"tomasz\",\"wojcik\"],\"age\":[\"22\",\"thirtytwo\",\"twenty\",\"nine\",\"thirty\",\"five\",\"twenty\",\"nine\",\"twenty\",\"two\",\"thirty\",\"31\",\"24\",\"26\",\"27\",\"30\",\"28\",\"25\"],\"email\":[\"alice\",\"nuno\",\"examplecom\",\"william\",\"examplecom\",\"alice\",\"nuno\",\"examplecom\",\"john\",\"doe\",\"emilie\",\"levesque\",\"examplecom\",\"tomasz\",\"wojcik\",\"examplecom\",\"ella\",\"examplecom\",\"andre\",\"silva\",\"examplecom\",\"emma\",\"examplecom\",\"benjamin\",\"examplecom\",\"michael\",\"johnson\",\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Mismatched.csv using qsv stats --everything...\n",
      "Generating frequency from Mismatched.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --max-tokens 200 Mismatched.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "a22e7c16-c238-4e5f-aecd-23bb5ebb1aa8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"dictionary\": {\n",
      "    \"Name\": {\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Name\",\n",
      "      \"Description\": \"The name of the person\"\n",
      "    },\n",
      "    \"Age\": {\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Age\",\n",
      "      \"Description\": \"The age of the person\"\n",
      "    },\n",
      "    \"Email\": {\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Email\",\n",
      "      \"Description\": \"The email address of the person\"\n",
      "    },\n",
      "    \"Phone\": {\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"Phone\",\n",
      "      \"Description\": \"The phone number of the person\"\n",
      "    },\n",
      "    \"IsSubscribed\": {\n",
      "      \"Type\": \"String\",\n",
      "      \"Label\": \"IsSubscribed\",\n",
      "      \"Description\": \"Indicates whether the person is subscribed or not\"\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Mismatched.csv using qsv stats --everything...\n",
      "Generating frequency from Mismatched.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --dictionary --json --max-tokens 200 Mismatched.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b658352-7468-4923-8c84-8db3fda50121",
   "metadata": {},
   "source": [
    "All in all ``describegpt`` does a good job of providing a helpful overview of the dataset, giving insights into its contents and highlighting areas of interest. But the area where it goes wrong is in identifying the data type of the column. This leads to possiblilties of errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c150a72-4e21-4adc-a970-ae0f915e6d4b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "380cee27-c340-4f37-b965-78c1709465c1",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### Empty CSV file\n",
    "> ###### ``Test OK ✔️``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "6e27b72a-7b12-4828-8949-ed8c07e4042b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "!qsv count Empty.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "0399835e-fe7d-484e-8fbe-08c95c4555cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset consists of four columns, each with 0 null values. The dataset has a cardinality of 1 for each column, indicating that there is only one unique value in each. The dataset does not provide any summary statistics or frequency data about the actual values in each column.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Empty.csv using qsv stats --everything...\n",
      "Generating frequency from Empty.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 150 Empty.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "d8ee0fb3-c111-4fb1-8ed2-3c87b95fc698",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"field\": \"column1\",\n",
      "  \"type\": \"null\",\n",
      "  \"sum\": 0,\n",
      "  \"min\": 0,\n",
      "  \"max\": 0,\n",
      "  \"range\": 0,\n",
      "  \"min_length\": 0,\n",
      "  \"max_length\": 0,\n",
      "  \"mean\": 0,\n",
      "  \"stddev\": 0,\n",
      "  \"variance\": 0,\n",
      "  \"nullcount\": 0,\n",
      "  \"sparsity\": 0,\n",
      "  \"mad\": 0,\n",
      "  \"lower_outer_fence\": 0,\n",
      "  \"lower_inner_fence\": 0,\n",
      "  \"q1\": 0,\n",
      "  \"q2_median\": 0,\n",
      "  \"q3\": 0,\n",
      " \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Empty.csv using qsv stats --everything...\n",
      "Generating frequency from Empty.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --max-tokens 150 Empty.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "05735cd6-8931-4a70-9614-3d37554130b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"dictionary\": {\n",
      "    \"fields\": [\n",
      "      {\n",
      "        \"Name\": \"Column1\",\n",
      "        \"Type\": \"NULL\",\n",
      "        \"Label\": \"Column 1\",\n",
      "        \"Description\": \"\"\n",
      "      },\n",
      "      {\n",
      "        \"Name\": \"Column2\",\n",
      "        \"Type\": \"NULL\",\n",
      "        \"Label\": \"Column 2\",\n",
      "        \"Description\": \"\"\n",
      "      },\n",
      "      {\n",
      "        \"Name\": \"Column3\",\n",
      "        \"Type\": \"NULL\",\n",
      "        \"Label\": \"Column 3\",\n",
      "        \"Description\": \"\"\n",
      "      },\n",
      "      {\n",
      "        \"Name\": \"Column4\",\n",
      "        \"Type\": \"NULL\",\n",
      "        \"Label\": \"Column 4\",\n",
      "        \"Description\": \"\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Empty.csv using qsv stats --everything...\n",
      "Generating frequency from Empty.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --dictionary --json --max-tokens 150 Empty.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "a47c6e8c-0621-4d37-bbe0-4b711b340f75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Completly-Empty.csv using qsv stats --everything...\n",
      "Generating frequency from Completly-Empty.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "  \"type\": \"summary statistics\",\n",
      "  \"sum\": \"sum\",\n",
      "  \"min\": \"min\",\n",
      "  \"max\": \"max\",\n",
      "  \"range\": \"range\",\n",
      "  \"min_length\": \"minlength\",\n",
      "  \"max_length\": \"maxlength\",\n",
      "  \"mean\": \"mean\",\n",
      "  \"stddev\": \"stddev\",\n",
      "  \"variance\": \"variance\",\n",
      "  \"nullcount\": \"nullcount\",\n",
      "  \"sparsity\": \"sparsity\",\n",
      "  \"mad\": \"mad\",\n",
      "  \"lower_outer_fence\": \"lowerouterfence\",\n",
      "  \"lower_inner_fence\": \"lowerinnerfence\",\n",
      "  \"q1\": \"q1\",\n",
      "  \"q2_median\": \"q2median\",\n",
      "  \"\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --max-tokens 150 Completly-Empty.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "d2a158cc-97d5-4ca1-855f-1685efdc4dc6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"dictionary\": {\n",
      "    \"error\": \"Error: Invalid JSON output for dictionary.\"\n",
      "  },\n",
      "  \"description\": {\n",
      "    \"description\": \"The dataset contains information about various fields. It provides summary statistics such as sum, minimum, maximum, range, minimum length, maximum length, mean, standard deviation, variance, null count, sparsity, mean absolute deviation, lower outer fence, lower inner fence, first quartile, median, third quartile, interquartile range, upper inner fence, upper outer fence, skewness, cardinality, mode, mode count, mode occurrences, antimode, antimode count, antimode occurrences. Additionally, it includes frequency data for each field and its corresponding values. The dataset provides a comprehensive overview of the various fields and their statistical properties. \"\n",
      "  },\n",
      "  \"tags\": {\n",
      "    \"error\": \"Error: Invalid JSON output for tags.\"\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from Completly-Empty.csv using qsv stats --everything...\n",
      "Generating frequency from Completly-Empty.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "{\"error\":\"Error: Invalid JSON output for dictionary.\"}\n",
      "Output: I'm sorry, but I cannot generate the data dictionary based on the provided information. The data dictionary should include the columns of the dataset along with their corresponding types, labels, and descriptions.\n",
      "\n",
      "Please provide the actual columns of the dataset along with their respective information.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n",
      "{\"error\":\"Error: Invalid JSON output for tags.\"}\n",
      "Output: I apologize, but I cannot generate the tags based on the summary statistics and frequency data provided in the current format. \n",
      "\n",
      "The frequency data should include the actual values in the dataset along with their respective counts. Please provide the actual values and their counts for each field in the dataset.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --json --max-tokens 250 Completly-Empty.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c369582a-a476-4067-b8d7-23da7cf34089",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0b0f5ce8-c992-4e69-8966-d35a6b928166",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### CSV File with Special Characters\n",
    "> ###### ``Test OK ✔️``"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7d5f19-2b45-4aa8-9a6e-4facbdc6b84c",
   "metadata": {},
   "source": [
    "This CSV file I've integrated few of the special characters and non ASCII characters to test the commands ability to handle diverse files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2b42bd7-bb92-41f8-be46-068ac742e25c",
   "metadata": {},
   "source": [
    "| Name                    | Role       | Email                          | Phone        |\n",
    "|-------------------------|------------|--------------------------------|--------------|\n",
    "| John Dõe                | Engineer   | john.doe@example.com           | 123-456-7890 |\n",
    "| Alice Ñuñø              | Manager    | alice.ñuño@example.com         | 987-654-3210 |\n",
    "| Émilie Lévésqué         | Designer   | emilie.levesque@example.com    | 555-123-4567 |\n",
    "| Tomasz Wójcik           | Developer  | tomasz.wojcik@example.com      | 789-321-6540 |\n",
    "| Anita Jørgensén         | Analyst    | anita.jorgensen@example.com    | 456-789-1230 |\n",
    "| Sönke Müller            | Architect  | sonke.muller@example.com       | 321-654-9870 |\n",
    "| Käthe Grüber            | Sales      | kathe.gruber@example.com       | 876-543-2109 |\n",
    "| Iñaki López             | Support    | iñaki.lopez@example.com        | 234-567-8901 |\n",
    "| André Silvà             | Manager    | andre.silva@example.com        | 789-012-3456 |\n",
    "| José Córdoba            | Engineer   | jose.cordoba@example.com       | 654-321-0987 |\n",
    "| Łukasz Kowalski         | Developer  | lukasz.kowalski@example.com    | 987-654-3210 |\n",
    "| Jiří Novák              | Analyst    | jiri.novak@example.com         | 789-456-1230 |\n",
    "| Nikos Papadopoulos      | Architect  | nikos.papadopoulos@example.com | 321-654-7890 |\n",
    "| Çağatay Yılmaz          | Designer   | cagatay.yilmaz@example.com     | 234-567-8901 |\n",
    "| Δημήτρης Παπαδόπουλος   | Sales      | dimitris.papadopoulos@example.com | 876-543-2109 |\n",
    "| 🌟 Ruby Jones           | Manager    | ruby.jones@example.com         | 987-654-3210 |\n",
    "| 😊 Sam Smith            | Engineer   | sam.smith@example.com          | 555-123-4567 |\n",
    "| 🎉 Emily Chen           | Designer   | emily.chen@example.com         | 789-321-6540 |\n",
    "| 🔥 Alex Kim             | Developer  | alex.kim@example.com           | 456-789-1230 |\n",
    "| 🚀 Lisa Rodríguez       | Analyst    | lisa.rodriguez@example.com     | 321-654-9870 |\n",
    "| 🌈 Luca Müller          | Architect  | luca.muller@example.com        | 876-543-2109 |\n",
    "| 💡 Charlie Baker        | Sales      | charlie.baker@example.com      | 234-567-8901 |\n",
    "| 🌼 Luna Park            | Support    | luna.park@example.com          | 789-012-3456 |\n",
    "| 💻 Max Weber            | Manager    | max.weber@example.com          | 654-321-0987 |\n",
    "... (continues with around 70 more records)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0040a27a-167a-46b7-9fb2-2fd5193c05ec",
   "metadata": {},
   "source": [
    "As we can observe from the above CSV file that it contains various symbols like Emojis, few Greek letters and Unicode characters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f4ef00-e81e-418b-b840-63156a0f3f79",
   "metadata": {},
   "source": [
    "Let's try out describegpt for this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ccc9dea6-0bc2-46d0-9517-19b7acae2bcc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset is in JSON format and contains information about individuals, including their names, roles, emails, and phone numbers. The dataset has a total of 24 records. The names range from \"Alice Ã‘uÃ±Ã¸\" to \"ðŸš€ Lisa RodrÃ­guez\" and have a minimum length of 9 and a maximum length of 41 characters. The roles range from \"Analyst\" to \"Support\" and have a minimum length of 5 and a maximum length of 9 characters. The emails range from \"alex.kim@example.com\" to \"tomasz.wojcik@example.com\" and have a minimum length of 20 and a maximum length of 33 characters. The phone numbers range from \"123-456-7890\" to \"987-654-3210\" and have a fixed length of 12 characters. The dataset has no missing values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from specialchars_max.csv using qsv stats --everything...\n",
      "Generating frequency from specialchars_max.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 200 --openai-key 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 specialchars_max.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3f3baca-43d9-4b3e-87df-049d2ddcf07b",
   "metadata": {},
   "source": [
    "<a id='4'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "e1b9bec4-c1d2-4db8-9bbc-7415230b2c95",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Unknown flag: '--dscription'. Did you mean '--description'?\n",
      "\n",
      "Usage:\n",
      "    qsv describegpt [options] [<input>]\n",
      "    qsv describegpt --help\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --dscription --max-tokens 200 --openai-key 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 specialchars_max.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "278ba7c2-2c3e-4cf2-8c86-e8141236b743",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Name\": [\"aliceÃ±uÃ±Ã¸\", \"lisarodrÃ­guez\", \"ðŸ’¡charliebaker\", \"nikospapadopoulos\", \"Î”Î·Î¼Î®Ï„Ï�Î·Ï‚Ï€Î±Ï€Î±Î´ÏŒÏ€Î¿Ï…Î»Î¿Ï‚\", \"Ã©milielÃ©vÃ©squÃ©\", \"johndÃµe\", \"ðŸŒŸrubyjones\", \"anitajÃ¸rgensÃ©n\", \"Ã§aÄŸatayyÄ±lmaz\", \"jiÅ™Ã­novÃ¡k\"],\n",
      "  \"Role\": [\"analyst\", \"support\", \"manager\", \"architect\", \"engineer\", \"sales\", \"designer\", \"developer\"],\n",
      "  \"Email\": [\"alex.kim@example.com\", \"tomasz.wojcik@example.com\", \"max.weber@example.com\", \"luna.park@example.com\", \"cagatay.yilmaz@example.com\", \"nikos.papadopoulos@example.com\", \"jiri\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from specialchars_max.csv using qsv stats --everything...\n",
      "Generating frequency from specialchars_max.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags --max-tokens 200 --openai-key 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 specialchars_max.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a8fc960-940c-42c7-9801-4780664c5a10",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f0f87dd4-2150-4dca-97dc-6c53399488e9",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "#### CSV file with Non-English Data\n",
    "> ###### ``Test Flagged 🚩``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "56dd0c61-b0f6-476e-9656-ec6adfbc4cc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"QSV_OPENAI_KEY\"] = \"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\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2cd84d94-1371-4664-a98f-b210d8c3a8ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset, in JSON format, consists of a collection of data points with various fields. The summary statistics provide an overview of the dataset, while the frequency data shows the occurrence of values within the dataset. The dataset contains diverse information, showcasing range and variability across its fields. It allows for detailed analysis and exploration of the data points within, highlighting trends and patterns. The dataset's rich representation facilitates meaningful insights and enables comprehensive understanding of its contents. Through further exploration, one can gain valuable knowledge and derive actionable conclusions from this well-structured dataset.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from restaurant_reviews_in_arabic.csv using qsv stats --everything...\n",
      "Generating frequency from restaurant_reviews_in_arabic.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --description --max-tokens 200 --openai-key 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 restaurant_reviews_in_arabic.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9790a768-ed94-4d9b-afc3-0e9238b86735",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm sorry, but you haven't provided any summary statistics or frequency data. Please provide the required information so that I can generate the data dictionary for you.\n",
      "I'm sorry, but you haven't provided any summary statistics or frequency data for the dataset. Please provide the required information so that I can generate the data dictionary for you.\n",
      "I'm sorry, but without any summary statistics or frequency data, I am unable to generate the tags for the dataset. Please provide the necessary information for me to assist you further.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from restaurant_reviews_in_arabic.csv using qsv stats --everything...\n",
      "Generating frequency from restaurant_reviews_in_arabic.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all --max-tokens 200 restaurant_reviews_in_arabic.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19f84885-fcde-4dc4-9a58-6bd23f18bba4",
   "metadata": {},
   "source": [
    "!qsv describegpt --all --max-tokens 200 restaurant_reviews_in_spanish.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41026ca1-3dfd-4ac2-9c7c-a34d24e7fca8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1f4797e8-8304-4497-8c02-fb5b2b66c327",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Error Handling Testing:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59edf1dc-b80f-4d59-ace2-81e1dfefd92d",
   "metadata": {},
   "source": [
    "[Wrong API key](#1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee2ac149-4dac-4a0c-b428-fb1a90c5aad7",
   "metadata": {},
   "source": [
    "[API Key with 0 credits](#2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85cafc0c-390a-477d-bf05-fa4e436371d8",
   "metadata": {},
   "source": [
    "[Typo in Command #1](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69b3bb7b-a3dc-4ebe-b89c-eeead932e441",
   "metadata": {},
   "source": [
    "[Typo in Command #2](#4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecadf12d-696e-4695-9577-71eb538890f2",
   "metadata": {},
   "source": [
    "#### Default Option Behavior Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a080c20b-66c0-477a-8d11-b3b16a21f37e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error: No inference options specified.\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt netflix_titles.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d7e4a4-5d25-4f27-a0a5-c265b4f97b51",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Option Combinations Testing:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c24e2876-5f45-4915-9dff-ec15371a93f2",
   "metadata": {},
   "source": [
    "#### ⚗️Experimental"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "b00c7fff-df75-49ea-8ad5-5fb70701dfad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "\"  \"\"fields\"\": [\"\n",
      "    {\n",
      "\"      \"\"Name\"\": \"\"show_id\"\"\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from netflix_titles.csv using qsv stats --everything...\n",
      "Generating frequency from netflix_titles.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "CSV error: found record with 2 fields, but the previous record has 1 fields\n",
      "Received description completion.\n",
      "thread 'main' panicked at 'failed printing to stdout: The pipe is being closed. (os error 232)', library\\std\\src\\io\\stdio.rs:1019:9\n",
      "note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all netflix_titles.csv | qsv table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "75fd1663-3f2f-45d3-9fa3-48d0ace78098",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from WorldCups.csv using qsv stats --everything...\n",
      "Generating frequency from WorldCups.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n",
      "CSV error: found record with 11 fields, but the previous record has 1 fields\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\"  \"\"Year\"\": [\"\"1978\"\"\"  \" \"\"1986\"\"\"  \" \"\"1970\"\"\"  \" \"\"1994\"\"\"  \" \"\"1930\"\"\"  \" \"\"1998\"\"\"  \" \"\"1982\"\"\"  \" \"\"1962\"\"\"  \" \"\"1958\"\"\"  \" \"\"1990\"\"]\"\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags WorldCups.csv | qsv table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "e7b60a98-8004-4105-8a93-975bb5fb3a77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "\"    \"\"fields\"\": [\"\n",
      "        {\n",
      "\"            \"\"Name\"\": \"\"Overall rank\"\"\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "CSV error: found record with 2 fields, but the previous record has 1 fields\n",
      "Received description completion.\n",
      "thread 'main' panicked at 'failed printing to stdout: The pipe is being closed. (os error 232)', library\\std\\src\\io\\stdio.rs:1019:9\n",
      "note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all \"World Happiness Report.csv\" | qsv table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "850caef2-47ed-4efb-9e59-690034c2baad",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Could not match 'tables' with any of the allowed variants: [\"apply\", \"behead\", \"cat\", \"count\", \"dedup\", \"describegpt\", \"diff\", \"enum\", \"excel\", \"exclude\", \"explode\", \"extdedup\", \"extsort\", \"fetch\", \"fetchpost\", \"fill\", \"fixlengths\", \"flatten\", \"fmt\", \"frequency\", \"generate\", \"headers\", \"help\", \"index\", \"input\", \"join\", \"jsonl\", \"partition\", \"pseudo\", \"rename\", \"replace\", \"reverse\", \"safenames\", \"sample\", \"schema\", \"search\", \"searchset\", \"select\", \"slice\", \"snappy\", \"sniff\", \"sort\", \"sortcheck\", \"split\", \"stats\", \"table\", \"transpose\", \"tojsonl\", \"validate\"]\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "thread 'main' panicked at 'failed printing to stdout: The pipe is being closed. (os error 232)', library\\std\\src\\io\\stdio.rs:1019:9\n",
      "note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all \"World Happiness Report.csv\" | qsv tables\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "4a522e0d-6bc9-4139-bf94-5f3bab3accee",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Error: No input file specified.\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Received tags completion.\n",
      "thread 'main' panicked at 'failed printing to stdout: The pipe is being closed. (os error 232)', library\\std\\src\\io\\stdio.rs:1019:9\n",
      "note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --tags \"World Happiness Report.csv\" | qsv describegpt --description\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "be116be1-e055-497e-addd-5e9afeb06c4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset consists of information on various countries or regions, including their overall rank, score, GDP per capita, social support, healthy life expectancy, freedom to make life choices, generosity, and perceptions of corruption. The overall rank ranges from 1 to 156, with a mean of 78.5. The score ranges from 2.853 to 7.769, with a mean of 5.4071. The GDP per capita ranges from 0.0 to 1.684, with a mean of 0.9051. The social support ranges from 0.0 to 1.624, with a mean of 1.2088. The healthy life expectancy ranges from 0.0 to 1.141, with a mean of 0.7252. The freedom to make life choices ranges from 0.0 to 0.631, with a mean of 0.3926. The generosity ranges from 0.0 to 0.566, with a mean of 0.1848. The perceptions of corruption ranges from 0.0 to 0.453, with a mean of 0.1106. The dataset contains information on 156 countries or\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating tags from OpenAI API...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating description from OpenAI API...\n",
      "Received description completion.\n",
      "Received tags completion.\n",
      "thread 'main' panicked at 'failed printing to stdout: The pipe is being closed. (os error 232)', library\\std\\src\\io\\stdio.rs:1019:9\n",
      "note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt  --tags --max-tokens 250 \"World Happiness Report.csv\" | qsv describegpt --description --max-tokens 250 \"World Happiness Report.csv\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "b5e12a31-98c9-44ae-9238-9389fcd74250",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating stats from World Happiness Report.csv using qsv stats --everything...\n",
      "Generating frequency from World Happiness Report.csv using qsv frequency...\n",
      "Interacting with OpenAI API...\n",
      "\n",
      "Generating data dictionary from OpenAI API...\n",
      "Received dictionary completion.\n",
      "Generating description from OpenAI API...\n",
      "CSV error: record 2 (line: 3, byte: 14): found record with 2 fields, but the previous record has 1 fields\n",
      "Received description completion.\n",
      "thread 'main' panicked at 'failed printing to stdout: The pipe is being closed. (os error 232)', library\\std\\src\\io\\stdio.rs:1019:9\n",
      "note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
     ]
    }
   ],
   "source": [
    "!qsv describegpt --all \"World Happiness Report.csv\" | qsv count\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63edb5e4-3f5a-444b-a999-11e286b39242",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}