flights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Technical test for a Python Dev Junior position\n",
    "\n",
    "You have three types of files with information about flights: a JSON file with flight details, a CSV file with passenger and revenue data, and a LOG file with flight departure records. The objective is for the candidate to implement functions to process and analyze the data, and additionally, perform unit tests using pytest.\n",
    "\n",
    "# Part 1: Data processing\n",
    "\n",
    "<div style=\"text-align: justify;\"\n",
    "\n",
    "1. Create a function *`load_json_data(file_path: str) -> List[Dict[str, Any]]`* that loads data from the JSON file and returns a list of dictionaries, where each dictionary represents a flight record. Make sure to handle possible errors when loading the file and use the logging module to log the errors.  \n",
    "\n",
    "2. Create a function *`load_csv_data(file_path: str) -> List[Dict[str, Any]]`* that loads data from the CSV file and returns a list of dictionaries, where each dictionary represents a flight record with passenger and revenue information. Implement error handling to deal with issues while reading the CSV file and use the logging module to log them.\n",
    "\n",
    "3. Create a function *`parse_log_data(file_path: str) -> List[Dict[str, Any]]`* that parses the LOG file and returns a list of dictionaries, where each dictionary represents a flight record with date, flight ID, departure, and arrival airports information. Make sure to handle errors while parsing the LOG file and use the logging module to log the errors.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing libraries needed\n",
    "import json\n",
    "import csv\n",
    "import logging\n",
    "import re\n",
    "import pytest\n",
    "# Type Hints: Allow us to use annotations to indicate the types of variables and function return values\n",
    "# List type; Dictionary type; Any allows variables to be of any type\n",
    "from typing import List, Dict, Any"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading data from a JSON file\n",
    "### Functions\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "***`setup_custom_logger`*** creates and configures a custom logger for logging errors and messages. \n",
    " \n",
    "*<u>Parameters:</u> *`name`* of the logger, path to the *`log file`* where the log messages will be stored and *`logging level`* to be set for the logger. \n",
    "\n",
    "*<u>Actions:</u>\n",
    "\n",
    "- Creates a *`formatter`* object with a specific log message format that includes a timestamp, log level, and log message content. \n",
    "\n",
    "*`asctime`* is a format specifier used in the logging module. The format of asctime is as follows: yyyy-mm-dd HH:MM:SS.\n",
    "\n",
    "- Creates a *`FileHandler`* object to define the log file where the log messages will be written. The file is opened in write mode *`a`*, (append) mode is used to add data to the end of the file without deleting the existing content. If the file does not exist, a new file will be created. This is useful if you want to maintain a continuous log history. Depending on the need, the *`w`* (write) mode can be used, which one the existing content in app.log will be removed, and only the new log messages will be written to the file.  \n",
    "\n",
    "- Associates the Formatter created earlier with the FileHandler, specifying the log message format for the log file.  \n",
    "\n",
    "- Creates a *`logger`* object with the given name, sets the logging level to the provided level, and adds the FileHandler to the logger. Now the logger is ready to log messages to the specified log file with the configured settings.  \n",
    "\n",
    "- Returns the configured logger object.\n",
    "\n",
    "***`load_json_data`*** loads data from a JSON file and returns a list of dictionaries, where each dictionary represents a flight record.\n",
    "\n",
    "it defines a custom logger called json_logger to log errors or higher severity messages to the file *`app.log`*. Each time you run the code, it creates or overwrite the *`app.log`* file in the current directory and log any errors that occur during the execution of the load_json_data function. \n",
    "\n",
    "*<u>Parameter:</u> *`file_path`*: The path to the JSON file to be loaded.  \n",
    "\n",
    "*<u>Actions:</u>\n",
    "\n",
    "- Calls the setup_custom_logger function to create a logger named *`json_logger`*. This logger will log any errors that occur during the loading process to the log file *`app.log`*.  \n",
    "\n",
    "- Attempts to open the specified JSON file using a with statement to ensure proper file handling.  \n",
    "\n",
    "- Parses the JSON data from the file and stores it in a variable called data. Parses the JSON data: means to take the JSON data which is typically in string format and convert it into a Python data structure. \n",
    "\n",
    "- Extracts the list of dictionaries representing flight records from the data variable under the key *`flights`*.  \n",
    "\n",
    "If any errors occur during the loading process (e.g., file not found, invalid JSON format), the function logs the specific error message using the json_logger created earlier. This allows you to track and diagnose any issues that may occur while loading the JSON data.\n",
    "\n",
    "If no errors occur, the function returns the list of flight records obtained from the JSON data. This list of dictionaries contains the flight details such as flight ID, date, departure airport, arrival airport, duration, passengers, and revenue.\n",
    "\n",
    "If there are errors during the loading process, the function returns an empty list ([]) to indicate that the loading was not successful.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Flight_ID': 'FL111', 'Date': '2025-01-01', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'LDN', 'Duration_Minutes': 180, 'Passengers': 111, 'Revenue': 1000}\n",
      "{'Flight_ID': 'FL222', 'Date': '2025-02-02', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'BCN', 'Duration_Minutes': 30, 'Passengers': 222, 'Revenue': 2000}\n"
     ]
    }
   ],
   "source": [
    "def setup_custom_logger(name, log_file, level=logging.ERROR):\n",
    "    formatter = logging.Formatter(fmt='%(asctime)s - %(levelname)s - %(message)s')\n",
    "    handler = logging.FileHandler(log_file, mode='a')\n",
    "    handler.setFormatter(formatter)\n",
    "    logger = logging.getLogger(name)\n",
    "    logger.setLevel(level)\n",
    "    logger.addHandler(handler)\n",
    "    return logger\n",
    "\n",
    "# This function takes a string as input and returns a list of dictionaries\n",
    "# Where keys are strings and values can be of any type\n",
    "def load_json_data(file_path: str) -> List[Dict[str, Any]]:\n",
    "    logger = setup_custom_logger('json_logger', 'app.log', level=logging.ERROR)\n",
    "    \n",
    "    try:\n",
    "        with open(file_path, 'r') as file:\n",
    "            # load() allows us to convert json data (strings) into Python objects\n",
    "            data = json.load(file)\n",
    "        return data['flights']\n",
    "    except FileNotFoundError as e:\n",
    "        logger.error(f\"File '{file_path}' not found.\")\n",
    "    except json.JSONDecodeError as e:\n",
    "        logger.error(f\"Error decoding JSON in file '{file_path}': {e}\")\n",
    "    except Exception as e:\n",
    "        logger.error(f\"An unexpected error occurred while loading data: {e}\")\n",
    "    return []  # Return an empty list in case of any error.\n",
    "\n",
    "# Using data from local base. Change this route if data is loaded from another location\n",
    "file_path = '/Users/sheila/Desktop/Prueba_Tecnica/data.json'\n",
    "# Save into a variable called flight_records the list of dictionaries to check the data\n",
    "flight_records1 = load_json_data(file_path)\n",
    "# Each flight record is a dictionary that contains information about a specific flight\n",
    "# Use a for loop to print each flight on console and check the function's action:\n",
    "for flight in flight_records1:\n",
    "    print(flight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Loading data from a CSV file\n",
    "### Functions\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "***`load_csv_data`***\n",
    "\n",
    "*<u>Parameter:</u> *`file_path`*: The path to the CSV file to be loaded.  \n",
    "\n",
    "For CSV data, each row has to be read it separately and create a dictionary for each flight record. \n",
    "\n",
    "It creates a list *`flight_records2`* to store these individual records.\n",
    "\n",
    "During the parsing process, each row of the CSV file is read, converted into a dictionary representing a flight record, and then appended to the list *`flight_records2`* for further processing.\n",
    "\n",
    "For JSON data, the entire data structure is loaded as a Python data structure, and there's no need for an intermediary list to store individual records. \n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Flight_ID': 'FL111', 'Passengers': 111, 'Revenue': 1000}\n",
      "{'Flight_ID': 'FL222', 'Passengers': 222, 'Revenue': 2000}\n"
     ]
    }
   ],
   "source": [
    "def load_csv_data(file_path: str) -> List[Dict[str, Any]]:\n",
    "    logger = setup_custom_logger('csv_logger', 'app.log', level=logging.ERROR)\n",
    "\n",
    "    flight_records2 = []\n",
    "    try:\n",
    "        with open(file_path, 'r', newline='') as csvfile:\n",
    "            csv_reader = csv.DictReader(csvfile)\n",
    "            for row in csv_reader:\n",
    "                flight_record = {\n",
    "                    'Flight_ID': row['Flight_ID'],\n",
    "                    'Passengers': int(row['Passengers']),\n",
    "                    'Revenue': int(row['Revenue'])\n",
    "                }\n",
    "                flight_records2.append(flight_record)\n",
    "        return flight_records2\n",
    "    except FileNotFoundError as e:\n",
    "        logger.error(f\"File '{file_path}' not found.\")\n",
    "    except csv.Error as e:\n",
    "        logger.error(f\"Error reading CSV file '{file_path}': {e}\")\n",
    "    except Exception as e:\n",
    "        logger.error(f\"An unexpected error occurred while loading data: {e}\")\n",
    "    return []  # Return an empty list in case of any error.\n",
    "\n",
    "# Using data from local base. Change this route if data is loaded from another location\n",
    "file_path = '/Users/sheila/Desktop/Prueba_Tecnica/data.csv'\n",
    "flight_records2 = load_csv_data(file_path)\n",
    "\n",
    "# Print the list of flight records to check function's action\n",
    "for flight in flight_records2:\n",
    "    print(flight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Loading data from a LOG file\n",
    "### Functions\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "***`parse_log_data`***\n",
    "\n",
    "*<u>Parameter:</u> *`file_path`*: The path to the LOG file to be loaded.  \n",
    "\n",
    "It also creates a list *`flight_records2`* to store individual records and uses `regular expressions` to find a match. For this purpose, it is used the module re. \n",
    "\n",
    "It assigns the result of the regular expression match to the variable *`match`*.\n",
    "\n",
    "The *`re.match()`* function returns a match object if the regular expression pattern matches the beginning of the line. \n",
    "\n",
    "The regular expression pattern that the *`re.match()`* function will attempt to match against the line is:  \n",
    " \n",
    "*`r'(\\d{4}-\\d{2}-\\d{2}) (\\w+) from (\\w+) to (\\w+) departed.'`* \n",
    "\n",
    "- *`(\\d{4}-\\d{2}-\\d{2})`*: This part of the pattern matches the date in the format 'yyyy-mm-dd' and captures it using parentheses to create a group. The *`\\d`* matches any digit, and *`{4}`*, *`{2}`*, and *`{2}`* specify the number of digits expected for the year, month, and day, respectively.\n",
    "\n",
    "- *`(\\w+)`*: This part of the pattern matches and captures one or more word characters (letters, digits, or underscores). This is used to capture the flight ID.\n",
    "\n",
    "- *`from`*: This part of the pattern matches the literal word 'from'.\n",
    "\n",
    "- *`(\\w+)`*: This part of the pattern matches and captures one or more word characters. This is used to capture the departure airport.\n",
    "\n",
    "- *`to`*: This part of the pattern matches the literal word 'to'.\n",
    "\n",
    "- *`(\\w+)`*: This part of the pattern matches and captures one or more word characters. This is used to capture the arrival airport.\n",
    "\n",
    "- *`departed.`*: This part of the pattern matches the literal word 'departed.' at the end of the line.\n",
    "\n",
    "*`line`* is the current line being processed from the log file. \n",
    "\n",
    "<u>After this line of code executes:</u>\n",
    "\n",
    "If the regular expression pattern matches the line, the match variable will hold a match object, and the subsequent code block (the body of the if statement) will be executed.\n",
    "\n",
    "If the regular expression pattern does not match the line, the match variable will be assigned None, and the subsequent code block (inside the else statement) will be executed.\n",
    "\n",
    "In summary, it uses a regular expression pattern to match and capture specific parts of the line, which represents a log entry from the log file. If the log entry matches the expected format, the regular expression groups are used to extract the date, flight ID, departure airport, and arrival airport information. If the log entry doesn't match the expected format, an error message will be logged.\n",
    "\n",
    "The error message is formatted using an f-string (f\"Error parsing line: {line.strip()}\").\n",
    "\n",
    "*`line.strip()`* is used to remove whitespaces from the line. This ensures that any extra whitespaces around the log entry are not included in the error message.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Date': '2025-01-01', 'Flight_ID': 'FL111', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'LDN'}\n",
      "{'Date': '2025-02-02', 'Flight_ID': 'FL222', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'BCN'}\n"
     ]
    }
   ],
   "source": [
    "def parse_log_data(file_path: str) -> List[Dict[str, Any]]:\n",
    "    logger = setup_custom_logger('log_parser', 'app.log', level=logging.ERROR)\n",
    "\n",
    "    flight_records3 = []\n",
    "    try:\n",
    "        with open(file_path, 'r') as log_file:\n",
    "            for line in log_file:\n",
    "                match = re.match(r'(\\d{4}-\\d{2}-\\d{2}) (\\w+) from (\\w+) to (\\w+) departed.', line)\n",
    "                if match:\n",
    "                    flight_record = {\n",
    "                        'Date': match.group(1),\n",
    "                        'Flight_ID': match.group(2),\n",
    "                        'Departure_Airport': match.group(3),\n",
    "                        'Arrival_Airport': match.group(4)\n",
    "                    }\n",
    "                    flight_records3.append(flight_record)\n",
    "                else:\n",
    "                    logger.error(f\"Error parsing line: {line.strip()}\")\n",
    "        return flight_records3\n",
    "    except FileNotFoundError as e:\n",
    "        logger.error(f\"File '{file_path}' not found.\")\n",
    "    except Exception as e:\n",
    "        logger.error(f\"An unexpected error occurred while parsing data: {e}\")\n",
    "    return []  # Return an empty list in case of any error\n",
    "\n",
    "# Using data from local base. Change this route if data is loaded from another location\n",
    "file_path = '/Users/sheila/Desktop/Prueba_Tecnica/data.log'\n",
    "\n",
    "flight_records3 = parse_log_data(file_path)\n",
    "\n",
    "# Print the list of flight records\n",
    "for flight in flight_records3:\n",
    "    print(flight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Data Transformation\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "1. Create a function *`combine_data(json_data: List[Dict[str, Any]], csv_data: List[Dict[str, Any]], log_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]`* that combines the information from the three types of data into a list of dictionaries, where each dictionary represents a flight record with the complete information (Flight_ID, Date, Departure_Airport, Arrival_Airport, Duration_Minutes, Passengers, Revenue). Implement error handling to manage possible data combination issues and use the logging module to record any errors that occur.\n",
    "\n",
    "2. Create a function *`calculate_revenue_per_passenger(flights_data: List[Dict[str, Any]]) -> None`* that adds the \"Revenue_Per_Passenger\" column to the list of dictionaries, where the value of this column is the result of dividing the revenue (Revenue) by the number of passengers (Passengers) for each flight. This function does not have an expected output as it directly modifies the flight data in place. Make sure to handle possible errors when calculating the \"Revenue_Per_Passenger\" column and use the logging module to record any issues.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Function: ***`combine_data`*** \n",
    "\n",
    "*<u>Parameters:</u> \n",
    "*`json_data`*: it includes the type, thus it expects a list of dictionaries as an input. \n",
    "*`csv_data`*: it also expects a list of dictionaries as an input. \n",
    "*`log_data`*: it also expects a list of dictionaries as an input. \n",
    "\n",
    "*<u>Action:</u> \n",
    "\n",
    "The function receives as inputs the list of dictionaries originated by `load_json_data`, `load_csv_data` and `parse_log_data` and merge the data from these three sources (JSON, CSV, and LOG data) to create a final list of dictionaries, where each dictionary represents a flight record with complete information, including 'Flight_ID', 'Date', 'Departure_Airport', 'Arrival_Airport', 'Duration_Minutes', 'Passengers', and 'Revenue'. The function handles possible errors while combining the data and logs any errors using the custom logger *`data_combiner`*.\n",
    "\n",
    "It has similar structure but in this case, `list of comprehensions` are used within a for loop to match coincidences in all files for the same flight. Below the steps are commented in detailed within the code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Flight_ID': 'FL111', 'Date': '2025-01-01', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'LDN', 'Duration_Minutes': 180, 'Passengers': 111, 'Revenue': 1000}\n",
      "{'Flight_ID': 'FL222', 'Date': '2025-02-02', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'BCN', 'Duration_Minutes': 30, 'Passengers': 222, 'Revenue': 2000}\n"
     ]
    }
   ],
   "source": [
    "def combine_data(json_data: List[Dict[str, Any]], csv_data: List[Dict[str, Any]], log_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:\n",
    "    # Step 1: create a custom logger to log any errors that ocurr during data combination \n",
    "    logger = setup_custom_logger('data_combiner', 'app.log', level=logging.ERROR)\n",
    "    # Step 2: create an empty list to store the combined flight records\n",
    "    combined_data = []\n",
    "    try:\n",
    "        # Step 3: create a for loop to go through each flight in the JSON data\n",
    "        for json_flight in json_data:\n",
    "            # Step 4: get the `flight_ID` \n",
    "            flight_id = json_flight['Flight_ID']\n",
    "            # Step 5: find matching flight records in the CSV data based on the `flight_ID`\n",
    "            matching_csv_flights = [flight for flight in csv_data if flight['Flight_ID'] == flight_id]\n",
    "            # Step 6: find matching flight records in the LOG data based on the `flight_ID`\n",
    "            matching_log_flights = [flight for flight in log_data if flight['Flight_ID'] == flight_id]\n",
    "            # Step 7: check if matching flights were found in both files\n",
    "            if not matching_csv_flights:\n",
    "                # Step 8: if not found in CSV data, log an error\n",
    "                logger.error(f\"CSV data not found for Flight_ID '{flight_id}'. Skipping data combination.\")\n",
    "                continue\n",
    "            if not matching_log_flights:\n",
    "                # Step 9: if not found in LOG data, log an error\n",
    "                logger.error(f\"Log data not found for Flight_ID '{flight_id}'. Skipping data combination.\")\n",
    "                continue\n",
    "            # Step 10: if flight is found in both files, then combine the flight records in a dictionary\n",
    "            combined_flight = {\n",
    "                'Flight_ID': flight_id,\n",
    "                'Date': json_flight.get('Date'),\n",
    "                'Departure_Airport': json_flight.get('Departure_Airport'),\n",
    "                'Arrival_Airport': json_flight.get('Arrival_Airport'),\n",
    "                'Duration_Minutes': json_flight.get('Duration_Minutes'),\n",
    "                'Passengers': matching_csv_flights[0].get('Passengers'),\n",
    "                'Revenue': matching_csv_flights[0].get('Revenue')\n",
    "            }\n",
    "            # Step 11: append the combined records to the list `combined_data`\n",
    "            combined_data.append(combined_flight)\n",
    "    # Step 12: if any unexpected error occurs during data combination, log the error\n",
    "    except Exception as e:\n",
    "        logger.error(f\"An unexpected error occurred while combining data: {e}\")\n",
    "    # Step 13: return the list of combined flight records\n",
    "    return combined_data\n",
    "\n",
    "# Use of the function with the provided JSON, CSV, and log data:\n",
    "json_data = flight_records1 # list of dictionaries representing JSON flight data\n",
    "csv_data = flight_records2  # list of dictionaries representing CSV flight data\n",
    "log_data = flight_records3   # list of dictionaries representing log flight data\n",
    "\n",
    "combined_data = combine_data(json_data, csv_data, log_data)\n",
    "\n",
    "# Print the list of combined flight records to check results\n",
    "for flight in combined_data:\n",
    "    print(flight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Function: ***`calculate_revenue_per_passenger`*** \n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "*<u>Parameter:</u> \n",
    "\n",
    "*`flights_data`*: it's a list of dictionaries representing flight records as an input.   \n",
    "\n",
    "*`none`*: indicates that the function does not return any specific value. Instead, it directly modifies the flights_data list in place, adding the `Revenue_Per_Passenger` key to each flight record.\n",
    "\n",
    "*<u>Action:</u> \n",
    "\n",
    "The function takes the data originated by `combine_data` and add the extra column requested (`Revenue_Per_Passenger`). \n",
    "\n",
    "If passengers or revenue is None, it means that the corresponding key (`Passengers` or `Revenue`) is not present in the flight record, or its value is not valid.\n",
    "\n",
    "If passengers is 0 or a negative value, it means that the number of passengers is invalid for the flight.\n",
    "\n",
    "If any of these conditions are met, the function cannot calculate the `Revenue_Per_Passenger` for the flight, so it logs an error. The use of None and the conditional check help ensure that the function handles cases where the required data is missing or invalid and avoids potential errors during the calculation.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Flight_ID': 'FL111', 'Date': '2025-01-01', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'LDN', 'Duration_Minutes': 180, 'Passengers': 111, 'Revenue': 1000, 'Revenue_Per_Passenger': 9.00900900900901}\n",
      "{'Flight_ID': 'FL222', 'Date': '2025-02-02', 'Departure_Airport': 'MLG', 'Arrival_Airport': 'BCN', 'Duration_Minutes': 30, 'Passengers': 222, 'Revenue': 2000, 'Revenue_Per_Passenger': 9.00900900900901}\n"
     ]
    }
   ],
   "source": [
    "def calculate_revenue_per_passenger(flights_data: List[Dict[str, Any]]) -> None:\n",
    "    # Step 1: Create a custom logger to log any errors that occur during the calculation\n",
    "    logger = setup_custom_logger('revenue_calculator', 'app.log', level=logging.ERROR)\n",
    "\n",
    "    try:\n",
    "        # Step 2: Loop through each flight record in the flights_data\n",
    "        for flight in flights_data:\n",
    "            passengers = flight.get('Passengers')\n",
    "            revenue = flight.get('Revenue')\n",
    "\n",
    "            # Step 3: Check if both 'Passengers' and 'Revenue' are present in the flight record\n",
    "            if passengers is not None and revenue is not None and passengers > 0:\n",
    "                # Step 4: Calculate \"Revenue_Per_Passenger\" by dividing 'Revenue' by 'Passengers'\n",
    "                revenue_per_passenger = revenue / passengers\n",
    "\n",
    "                # Step 5: Add \"Revenue_Per_Passenger\" key and value to the flight record\n",
    "                flight['Revenue_Per_Passenger'] = revenue_per_passenger\n",
    "            else:\n",
    "                # Step 6: Log an error if any of the required data is missing or invalid\n",
    "                logger.error(f\"Invalid data for Flight_ID '{flight.get('Flight_ID')}'. Unable to calculate 'Revenue_Per_Passenger'.\")\n",
    "\n",
    "    except Exception as e:\n",
    "        # Step 7: If any unexpected error occurs during calculation, log the error\n",
    "        logger.error(f\"An unexpected error occurred while calculating 'Revenue_Per_Passenger': {e}\")\n",
    "\n",
    "flights_data = combined_data  # list of dictionaries representing flight records\n",
    "\n",
    "calculate_revenue_per_passenger(flights_data)\n",
    "\n",
    "# Print the list of flight records with the new column `Revenue_Per_Passenger`\n",
    "for flight in flights_data:\n",
    "    print(flight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Testing with pytest\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "Implement tests for each of the previous functions using pytest. Make sure to cover different scenarios, such as empty data, data with errors, and valid data. Also, verify that the functions return the expected results. For functions without an expected output, ensure to check that the flight data has been modified correctly after calling the `calculate_revenue_per_passenger()` function.\n",
    "\n",
    "Observations:\n",
    "- The functions should be independent and testable, with no side effects.\n",
    "- Use the `json`, `csv`, and file handling functions to load and parse the data.\n",
    "- Implement proper error handling using `try` and `except` in the functions and utilize the `logging` module to log errors and issues encountered during data processing.\n",
    "- The randomly generated data in the `generate_flight_data()` function from the previous question can also be used as input for the tests.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with pytest ***`load_json_data`*** function:\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "1. `test_load_json_data_valid_file`: This test function verifies the behavior of the `load_json_data` function when it is provided with a valid JSON file. It loads the data from the JSON file and checks that the function returns a non-empty list.\n",
    "\n",
    "2. `test_load_json_data_invalid_file`: This test function verifies the behavior of the `load_json_data` function when it is provided with a JSON file that is invalid. In this case, the function is expected to return an empty list.\n",
    "\n",
    "3. `test_load_json_data_invalid_json`: This test function verifies the behavior of the `load_json_data` function when it is provided with a JSON file that contains invalid data, i.e., data that cannot be decoded as a valid Python object. Again, the function is expected to return an empty list.\n",
    "\n",
    "These test functions help ensure that the `load_json_data` function handles various scenarios properly and returns the expected results, both for valid data and for incorrect data or non-existent files.\n",
    "\n",
    "### Test Assertions in Python Testing\n",
    "\n",
    "With `pytest`, you can define test functions using the def keyword and use assertions (assert statements) to check if the actual output of a function matches the expected output.\n",
    "\n",
    "`assert` is a keyword in Python used for making assertions. When an assertion is evaluated, Python checks if the statement following `assert` is `True`. If it is true, the program continues executing normally. If the assertion is `False`, an `AssertionError` exception is raised, and the program stops.\n",
    "\n",
    "- `isinstance(flight_records, list)`: `isinstance` is used to check if `flight_records` is an instance of the `list` class. In this case, we want to ensure that `flight_records` is a list since we expect the `load_json_data` function to return a list of dictionaries.\n",
    "\n",
    "- `len(flight_records) > 0`: This assertion checks that the `flight_records` list has a length greater than zero. In other words, we are checking that the list is not empty, which implies that the `load_json_data` function has loaded valid data from the JSON file.\n",
    "\n",
    "- `len(flight_records) == 0`: This assertion verifies that the `flight_records` list has a length equal to zero. In this case, we want to ensure that the list is empty, indicating that the `load_json_data` function has returned an empty list due to a non-existent JSON file or invalid data.\n",
    "\n",
    "In summary, the `assert` statements in the test functions allow us to verify if the obtained results align with our expectations for the function being tested. If any of the assertions fail, an `AssertionError` is raised in the test function, indicating that there might be an issue with the function being tested or with the test data used. This helps us detect problems and errors in our code during the development and testing process.\n",
    "\n",
    "For help on terminal: `python3 -m pydoc pytest`\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test loading JSON data function\n",
    "\n",
    "# def test_load_json_data_valid_file():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/data.json'  # Change this to your test data location\n",
    "#     flight_records = load_json_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) > 0\n",
    "\n",
    "# def test_load_json_data_invalid_file():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/invalid_jsonfile.json'\n",
    "#     flight_records = load_json_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) == 0\n",
    "\n",
    "# def test_load_json_data_invalid_json():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/invalid_jsondata.json'\n",
    "#     flight_records = load_json_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with pytest the ***`load_csv_data`*** function\n",
    "\n",
    "Same tests are carried out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test loading CSV data function\n",
    "\n",
    "# def test_load_csv_data_valid_file():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/data.csv'  \n",
    "#     flight_records = load_csv_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) > 0\n",
    "\n",
    "# def test_load_csv_data_invalid_file():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/invalid_csvfile.csv'\n",
    "#     flight_records = load_csv_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) == 0\n",
    "\n",
    "# def test_load_csv_data_invalid_csv():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/invalid_csvdata.csv'\n",
    "#     flight_records = load_csv_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with pytest the ***`parse_log_data`*** function\n",
    "\n",
    "Same tests are carried out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test loading LOG data function\n",
    "\n",
    "# def test_load_log_data_valid_file():  \n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/data.log'   \n",
    "#     flight_records = parse_log_data(file_path) \n",
    "#     assert isinstance(flight_records, list) \n",
    "#     assert len(flight_records) > 0 \n",
    "\n",
    "# def test_load_log_data_invalid_file():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/invalid_logfile.log'\n",
    "#     flight_records = parse_log_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) == 0\n",
    "\n",
    "# def test_load_csv_data_invalid_log():\n",
    "#     file_path = '/Users/sheila/Desktop/Prueba_Tecnica/invalid_logdata.log'\n",
    "#     flight_records = parse_log_data(file_path)\n",
    "#     assert isinstance(flight_records, list)\n",
    "#     assert len(flight_records) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with pytest the ***`combine_data`*** function\n",
    "\n",
    "On this part, I try to verify if the combination of data loaded from prevoius funcions has been performed properly. \n",
    "\n",
    "For this purpose, the below tests has been designed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if the length of the combined data is the same than the lengths of the original datasets\n",
    "# def test_lenght_combine_data():\n",
    "#     combined_data = combine_data(flight_records1, flight_records2, flight_records3)\n",
    "#     assert len(combined_data) == len(flight_records1) == len(flight_records2) == len(flight_records3)\n",
    "\n",
    "# # Check if the combined data has the correct structure \n",
    "# def test_keys_combine_data():\n",
    "#     combined_data = combine_data(flight_records1, flight_records2, flight_records3)\n",
    "#     expected_keys = {'Flight_ID', 'Date', 'Departure_Airport', 'Arrival_Airport', 'Duration_Minutes', 'Passengers', 'Revenue'}\n",
    "#     assert all(set(record.keys()) == expected_keys for record in combined_data)\n",
    "\n",
    "# # Chech if all `Flight_ID` are present in all files\n",
    "# def get_flight_ids(records):\n",
    "#     return [record['Flight_ID'] for record in records]\n",
    "\n",
    "# def test_combine_data_with_valid_data()``\n",
    "#     combined_data = combine_data(flight_records1, flight_records2, flight_records3)\n",
    "\n",
    "#     # Obtain lists of `Flight_ID` from each file \n",
    "#     flight_ids_1 = get_flight_ids(flight_records1)\n",
    "#     flight_ids_2 = get_flight_ids(flight_records2)\n",
    "#     flight_ids_3 = get_flight_ids(flight_records3)\n",
    "#     combined_flight_ids = get_flight_ids(combined_data)\n",
    "\n",
    "#     # Check if the Flight_IDs of each file are the same \n",
    "#     assert set(flight_ids_1) == set(flight_ids_2) == set(flight_ids_3)\n",
    "\n",
    "#     # Check if the Flight_IDs are the same in combined_data and the original files\n",
    "#     assert set(combined_flight_ids) == set(flight_ids_1)\n",
    "\n",
    "# # Check that there are not present duplicate data after the combination\n",
    "# def test_no_duplicates_in_combined_data():\n",
    "#     combined_data = combine_data(flight_records1, flight_records2, flight_records3)\n",
    "#     ids = [record['Flight_ID'] for record in combined_data]\n",
    "#     assert len(ids) == len(set(ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with pytest the ***`calculate_revenue_per_passenger`*** function\n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "On this section, I create test cases that cover different scenarios and verify if the function behaves as expected.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test when flights_data is an empty list\n",
    "# def test_calculate_revenue_per_passenger_empty_data():\n",
    "#     flights_data = []\n",
    "#     calculate_revenue_per_passenger(flights_data)\n",
    "#     assert len(flights_data) == 0  # The function should not add any columns\n",
    "\n",
    "# # Test when flights_data contains valid data\n",
    "# def test_calculate_revenue_per_passenger_valid_data():\n",
    "#     # Create some sample flight data with 'Passengers' and 'Revenue'\n",
    "#     flights_data = [\n",
    "#         {'Flight_ID': 1, 'Passengers': 100, 'Revenue': 2000},\n",
    "#         {'Flight_ID': 2, 'Passengers': 50, 'Revenue': 1000},\n",
    "#     ]\n",
    "#     calculate_revenue_per_passenger(flights_data)\n",
    "#     # Check if the 'Revenue_Per_Passenger' column has been added and calculated correctly\n",
    "#     assert 'Revenue_Per_Passenger' in flights_data[0]\n",
    "#     assert 'Revenue_Per_Passenger' in flights_data[1]\n",
    "#     assert flights_data[0]['Revenue_Per_Passenger'] == 20.0\n",
    "#     assert flights_data[1]['Revenue_Per_Passenger'] == 20.0\n",
    "\n",
    "# # Test when flights_data contains missing 'Passengers' or 'Revenue' information\n",
    "# def test_calculate_revenue_per_passenger_missing_data():\n",
    "#     # Create some sample flight data with missing 'Passengers' and 'Revenue'\n",
    "#     flights_data = [\n",
    "#         {'Flight_ID': 1, 'Revenue': 2000},\n",
    "#         {'Flight_ID': 2, 'Passengers': 100, 'Revenue': None},\n",
    "#     ]\n",
    "#     calculate_revenue_per_passenger(flights_data)\n",
    "#     # The function should not add the 'Revenue_Per_Passenger' column for missing data\n",
    "#     assert 'Revenue_Per_Passenger' not in flights_data[0]\n",
    "#     assert 'Revenue_Per_Passenger' not in flights_data[1]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To run the test functions: \n",
    "\n",
    "<div style=\"text-align: justify;\">\n",
    "\n",
    "Some files have been created to mock the cases of having invalid files as input and files with invalid data to test with the logging module included in each function.\n",
    "\n",
    "Additionally, a file `test_flights.py` has been created to make testing easier with pytest. This file includes the functions to test, as well as the test functions without comments. All explanations and clarifications are included in this notebook with the goal of making this project more didactic.\n",
    "\n",
    "Later on, the command on the terminal `pytest test_flights.py` will run the testing.\n",
    "\n",
    "Note: The `generate_flight_data()` function mentioned in part 3 has not been previously requested; therefore, nothing has been done with respect to it.\n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}