Comprehensive Update

Author: WillTheFarmer

.github/CHANGELOG.md

@@ -0,0 +1,32 @@
+- version 1.0.0 - 10/31/2024
+- version 1.1.0 - 11/18/2024 - major changes
+- version 1.1.1 - 11/20/2024 - keyword replacement
+- version 2.0.0 - 11/28/2024 - backward incompatible
+- [1.0.1] apache_logs.error_systemCodeID corrected line - INTO logsystemCode to INTO logsystemCodeID
+- [1.0.1] remove debugging - SELECT statement from apache_logs.process_access_import, process_error_import & normalize_useragent.
+- [1.0.1] remove whitespace and commented out old code on all stored programs
+- [1.0.1] set all table AUTO_INCREMENT=1. All future version releases will be the same - AUTO_INCREMENT=1.
+- [1.1.0] rename LOAD DATA TABLES, normalized access_log_useragent TABLE into 11 TABLES, added 13 VIEWS.
+- [1.1.0] resize LOAD DATA COLUMNS, added req_query COLUMN and seperated query strings from req_uri COLUMN.
+- [1.1.0] add access_log_reqquery TABLE, renamed access_log_session TABLE to access_log_cookie.
+- [1.1.0] add UPDATE TRIM statements for apachemessage COLUMNS.
+- [1.1.1] change word 'extended' to 'csv2mysql'
+- [2.0.0] fix the double backslash requirement on Windows platform for file paths. 
+- [2.0.0] move all stage table parsing from Python to MySQL Stored Procedures - process_access_parse & process_error_parse
+- [2.0.0] process_access_parse, process_access_import, process_error_parse, process_error_import have importloadid parameter.
+- [2.0.0] revamp parse & import processes and now provide 2 processing methods - by process_status and new importloadid.
+- [2.0.0] relate and control each client load, parsing and importing separately. Enable multiple upload clients simultaneously.  
+- [2.0.0] add or rename of COLUMNS in import_load, import_process, import_file. populate import_load and import_process with meaningful information.
+- [2.0.0] add SET importfileid COLUMN to LOAD DATA statement and remove UPDATE importfileid statement in Python. 
+- [2.0.0] create ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] %7F: %E: [client\ %a] %M% ,\ referer\ %{Referer}i , %v" to add %v-canonical ServerName.
+- [2.0.0] add ServerName & ServerPort on import Combined & Error logs stage tables. Option allow adding domains to logs.
+- [2.0.0] add ERROR_SERVERNAME,ERROR_SERVERPORT,COMBINED_SERVERNAME & COMBINED_SERVERPORT variables to settings.env. 
+- [2.0.0] add SET servername & serverport COLUMN values to LOAD DATA statements.
+- [2.0.0] create log_referer, log_remotehost, log_servername, log_serverport TABLES to assoicate Access and Error logs.
+- [2.0.0] add server_name & server_port COLUMNS to import_file TABLE. Provides second option to update Apache logs without %v. 
+- [2.0.0] modify process_access_import & process_error_import to populate empty server_name & server_port with ServerName & ServerPort from import_file TABLE. 
+- [2.0.0] add WATCH_LOG to setting Log Level in watch4logs.py. 0=no messages, 1=message when files found, 2=message when checking for files & files found
+- [2.0.0] add class bcolors to place RED BACKGROUND on all ERROR - messsages
+- [2.0.0] add file - mysql_user_and_grants.sql - MySQL USER and GRANTS file for CREATE USER apache_upload for Python module
+- [2.0.0] add file - call_processes.sql - description and CALL command lines for 5 Stored Procedures
+- [2.0.0] add file - CHANGLOG.md - for single place to list all code and database changes

.github/INSTALL.md

@@ -1,14 +1,14 @@
 ## Installation Instructions
-The steps are very important to make installation painless. Please follow instructions in order.
+The steps are important to make installation painless.
 
 ### 1. MySQL Steps
-Before running `apachLogs2MySQL.sql` open file in your favorite editor and do a ***Find and Replace*** of the following User Account with a User Account with DBA Role on server you are installing on. This will make everything much easier. Copy below:
+Before running `apachLogs2MySQL.sql` if `root`@`localhost` does not exist open file and do a ***Find and Replace*** of User Account with a User Account with DBA Role on installation server. Copy below:
 ```
-`root`@`%`
+`root`@`localhost`
 ```
-Rename above <sup>user</sup> to a <sup>user</sup> on your server. For example - `root`@`%` to `dbadmin`@`localhost`
+Rename above <sup>user</sup> to a <sup>user</sup> on your server. For example - `root`@`localhost` to `dbadmin`@`localhost`
 
-The easiest way to install is use MySQL Command Line Client. Login as User with DBA Role and execute the following:
+The easiest way to install is MySQL Command Line Client. Login as User with DBA Role and execute the following:
 ```
 source yourpath/apacheLogs2MySQL.sql
 ```
@@ -20,7 +20,7 @@ local-infile=1
 After these 3 steps MySQL server should be good to go.
 
 ### 2. Python Steps
-Install all modules:
+Install all modules (`requirements.txt` in repository):
 ```
 pip install -r requirements.txt
 ```
@@ -32,7 +32,7 @@ python3 -m ensurepip --upgrade
 If any issues with ***pip install*** occur use individual install commands below:
 
 ### 3. Required Python Modules
-Python module links & install command lines for each platform. Single quotes around module name are required on macOS. The simplest option is run the command line under '5. Python Steps'. If that works you are all set. The `requirements.txt` file is included in repository.
+Python module links & install command lines for each platform. Single quotes around module name are required on macOS.
 |Python Package|Windows 10 & 11|Ubuntu 24.04|macOS 15.0.1 Darwin 24.0.0|GitHub Repository|
 |--------------|---------------|------------|--------------------------|-----------------|
 |[PyMySQL](https://pypi.org/project/PyMySQL/)|python -m pip install PyMySQL[rsa]|sudo apt-get install python3-pymysql|python3 -m pip install 'PyMySQL[rsa]'|[PyMySQL/PyMySQL](https://github.com/PyMySQL/PyMySQL)|
@@ -47,48 +47,45 @@ By default the load_dotenv() is looking for a file name .env which is standard n
 ```
 load_dotenv() # Loads variables from .env into the environment
 ```
-Windows requires double backslash:
-```
-C:\\Users\\farmf\\Documents\\apacheLogs\\
-```
-Lunix & macOS require single frontslash:
-```
-/home/will/apacheLogs/
-```
 Below is settings.env with default settings for running on Windows 11 Pro workstation. Make sure the correct logFormats are in correct logFormat folders. The application does not currently detect logFormats. Data will not be imported properly if folder settings are not correct.
 ### 5. Settings.env Variables
 ```
-MYSQL_HOST=localhost # MySQL server
-MYSQL_PORT=3306 # MySQL server port
-MYSQL_USER=root # MySQL server User 
-MYSQL_PASSWORD=password # MySQL server User Password
-MYSQL_SCHEMA=apache_logs # MySQL database schema ApacheLogs2MySQL will create
-WATCH_PATH=C:\\Users\\farmf\\Documents\\apacheLogs\\  # watch folder for new files
-WATCH_RECURSIVE=1 # watch all subfolders - 0=no 1=yes
-WATCH_INTERVAL=15 # seconds between watching for new files
-ERROR=1 # process error logs - 0=no 1=yes
-ERROR_LOG=2 # display process error processing to console in python - 0=none 1=summary 2=summary & each file being processed
-ERROR_PATH=C:\\Users\\farmf\\Documents\\apacheLogs\\**/*error*.* # process folder & file patterns for Error Log files - MUST BE Watch folder and can be Subfolders
-ERROR_RECURSIVE=1 # watch all subfolders - 0=no 1=yes
-ERROR_PROCESS=1 # execute MySQL Stored Procedure to import Error log staging table - 0=no 1=yes
-COMBINED=1 # process Common and Combined Access logs - 0=no 1=yes
-COMBINED_LOG=2 # display Common and Combined Access log processing to console in python - 0=none 1=summary 2=summary & each file being processed
-COMBINED_PATH=C:\\Users\\farmf\\Documents\\apacheLogs\\combined\\**/*access*.* # process folder & file patterns for Common and Combined Access files - MUST BE Watch subfolder
-COMBINED_RECURSIVE=1 # watch all subfolders - 0=no 1=yes
-COMBINED_PROCESS=1 # execute MySQL Stored Procedure to import Common and Combined Access staging table - 0=no 1=yes
-VHOST=1 # process Vhost Access logs - 0=no 1=yes
-VHOST_LOG=2 # display Vhost Access log processing to console in python - 0=none 1=summary 2=summary & each file being processed
-VHOST_PATH=C:\\Users\\farmf\\Documents\\apacheLogs\\vhost\\**/*access*.* # process folder & file patterns for Vhost Access files - MUST BE Watch subfolder
-VHOST_RECURSIVE=1 # watch all subfolders - 0=no 1=yes
-VHOST_PROCESS=1 # execute MySQL Stored Procedure to import Vhost Access log staging table - 0=no 1=yes
-CSV2MYSQL=1 # process error logs - 0=no 1=yes
-CSV2MYSQL_LOG=2 # display Csv2mysql Access log processing to console in python - 0=none 1=summary 2=summary & each file being processed
-CSV2MYSQL_PATH=C:\\Users\\farmf\\Documents\\apacheLogs\\csv2mysql\\**/*access*.* # process folder & file patterns for Csv2mysql Access files - MUST BE Watch subfolder
-CSV2MYSQL_RECURSIVE=1 # watch all subfolders - 0=no 1=yes
-CSV2MYSQL_PROCESS=1 # execute MySQL Stored Procedure to import Csv2mysql Access log staging table - 0=no 1=yes
-USERAGENT=1 # process csv2mysql Access logs - 0=no 1=yes
-USERAGENT_LOG=2 # display userAgent processing to console in python - 0=none 1=summary 2=summary & each file being processed
-USERAGENT_PROCESS=1 # execute MySQL Stored Procedure process userAgent parsed table into 14 normalized userAgent tables - 0=no 1=yes
+MYSQL_HOST=localhost
+MYSQL_PORT=3306
+MYSQL_USER=apache_upload
+MYSQL_PASSWORD=password
+MYSQL_SCHEMA=apache_logs
+WATCH_PATH=C:\Users\farmf\Documents\apacheLogs\
+WATCH_RECURSIVE=1
+WATCH_INTERVAL=15
+WATCH_LOG=2
+ERROR=1
+ERROR_LOG=2
+ERROR_PATH=C:\Users\farmf\Documents\apacheLogs\**/*error*.*
+ERROR_RECURSIVE=1
+ERROR_PROCESS=2
+ERROR_SERVERNAME=yourdomain.com
+ERROR_SERVERPORT=443
+COMBINED=1
+COMBINED_LOG=2
+COMBINED_PATH=C:\Users\farmf\Documents\apacheLogs\combined\**/*access*.*
+COMBINED_RECURSIVE=1
+COMBINED_PROCESS=2
+COMBINED_SERVERNAME=yourdomain.com
+COMBINED_SERVERPORT=443
+VHOST=1
+VHOST_LOG=2
+VHOST_PATH=C:\Users\farmf\Documents\apacheLogs\vhost\**/*access*.*
+VHOST_RECURSIVE=1
+VHOST_PROCESS=2
+CSV2MYSQL=1
+CSV2MYSQL_LOG=2
+CSV2MYSQL_PATH=C:\Users\farmf\Documents\apacheLogs\csv2mysql\**/*access*.*
+CSV2MYSQL_RECURSIVE=1
+CSV2MYSQL_PROCESS=2
+USERAGENT=1
+USERAGENT_LOG=2
+USERAGENT_PROCESS=1
 ```
 ### 6. Run Application
 If MySQL steps completed successfully, successfully installed Python modules, renamed file `settings.env` to `.env`, and updated MySQL server connection and log folder variables it is time to run application.