You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-[1.0.1] remove whitespace and commented out old code on all stored programs
@@ -38,3 +39,14 @@
38
39
-[2.0.0] add file - CHANGLOG.md - for single place to list all code and database changes
39
40
-[2.0.0] The rest of changes made last month are many modifications and additions from previous version will remain undocumented.
40
41
-[2.0.0] This version is the application baseline
42
+
-[2.1.0] add request_log_id to access and error formats functionality. Enables easier association with access and error records.
43
+
-[2.1.0] add columns to load_error_default & load_access_csv2mysql TABLES
44
+
-[2.1.0] modify process_error_parse - replace POSITION function with LOCATE function, removed unrequired brackets, add parsing logic for %v and %L String Formats.
45
+
-[2.1.0] modify process_error_import - add normalization for request_log_id, replace POSITION function with LOCATE function
46
+
-[2.1.0] modify process_access_parse - add parsing for request_log_id, replace POSITION function with LOCATE function
47
+
-[2.1.0] modify process_access_import - add normalization for request_log_id, replace POSITION function with LOCATE function
48
+
-[2.1.0] modify process_access_import - cookie,remoteuser,remoteLogName,referer,requestLogID values of '-' set to NULL. No normalized table value is created.
49
+
-[2.1.0] modify process_access_import - add servernameid to requestlogid before normalization to avoid duplicate requestlogid on consolidation of multiple domain logs.
50
+
-[2.1.0] modify process_error_import - add servernameid to requestlogid before normalization to avoid duplicate requestlogid on consolidation of multiple domain logs.
51
+
-[2.1.0] modify apacheLogs2MySQL.py - add .replace('"', ' in.') to all useragent attributes before UPDATE statement execute. The " in attribute value breaks UPDATE String.
52
+
-[2.1.0] update README.md to describe and explain additional request_log_id functionality
Copy file name to clipboardExpand all lines: .github/CONTRIBUTING.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,14 @@
1
-
To contribute Apache Access or Log formats commonly used that the application is not setup to do please email [email protected] with subject <sup>Additional Import Formats</sup> and I'll consider incorporating the format into import process.
1
+
To contribute Apache Access or Log formats commonly used that application is not setup for please email [email protected] with subject <sup>Additional Import Formats</sup> and I'll consider incorporating the format.
2
2
3
3
To contribute ideas or comments email [email protected] with subject <sup>ideas or comments</sup> and I'll be sure to reply.
4
4
5
-
Any organizations, people or person with multiple Apache servers that find this application to be a godsend in log collection financial contributions are greatly appreciated.
5
+
Any organizations, people or person with multiple Apache servers that find application to be a godsend in log collection financial contributions are greatly appreciated.
6
6
7
7
### The Repository's ***- Sponsor this project -*** lists three methods to make financial contributions.
8
8
9
-
There's not alot of code in the final version but I have 800 plus hours of research, design, iteration & development into application. During October I worked crazy long hours around the clock on this. It is way more time then I intended to invest into this project and another project did get put on the back burner but it did produce my first open-source software.
9
+
There's not alot of code in the final version but I have 850 plus hours of research, design, iteration & development into application. During October I worked long hours around the clock on this. It is way more time then I intended to invest into this project and another project did get put on the back burner but it did produce my first open-source software.
10
10
11
-
I volunteer for a nonprofit organization that needed a simple solution to import Apache logs into MySQL. The Executive Director loves MySQL. First I installed the Apache log_sql_mysql modules which did create a single MySQL mostly empty table of the access log with no control or customization. Next I researched many available Apache logging solutions including GoAccess, Logstach, Apache Viewer, DataDog and others as well as CrowdStrike and Soloarwinds Loggly. I also reviewed several simple log file parsers but none that normalized the parsed log data into a MySQL database. After all my investigating I decided to write a simple solution which snowballed into this complete solution.
11
+
I volunteer for a nonprofit organization that needed a simple solution to import Apache logs into MySQL. The Executive Director loves MySQL. First I installed the Apache log_sql_mysql modules which did create a single MySQL mostly empty table of the access log with no control or customization and many other issues. Next I researched available Apache logging solutions including GoAccess, Logstach, Apache Viewer, DataDog and others as well as CrowdStrike and Soloarwinds Loggly. I also reviewed several simple log file parsers but none normalized the parsed log data into a MySQL database. After all my investigating I decided to write a simple solution which snowballed into this complete solution.
12
12
13
13
That's how volunteering, lack of a viable MySQL solution and a flexible schedule came together just right to allow me to dive deep into this project.
Copy file name to clipboardExpand all lines: .github/README.md
+14-9Lines changed: 14 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,8 +3,6 @@ ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate i
3
3
4
4
Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0.
5
5
6
-
Version 2.0.0 fixes issues encountered running 4 weeks on 7 VPS with 2 to 6 VirtualHosts on each VPS. Each VPS is running `watch4logs.py` in PM2 connecting to one MySQL server `apache_logs` schema. Application currently consolidates Access and Error logs from 32 domains.
7
-
8
6
Imports Access Logs in Apache LogFormats - ***common***, ***combined*** and ***vhost_combined*** & additional ***csv2mysql*** LogFormat defined below.
9
7
10
8
Imports Error Logs in Apache ***default*** ErrorLogFormat & ***additional*** ErrorLogFormat defined below performing data harmonization on Apache Codes & Messages, System Codes & Messages, and Log Messages to create a unified, standardized dataset. Error Log view images below.
|%v|The canonical ServerName of the server serving the request.|
74
72
|%p|The canonical port of the server serving the request.|
75
73
76
-
Application is designed to use the ***csv2mysql*** LogFormat. LogFormat has comma-separated values and adds 7 Format Strings. A complete list of Format Strings
74
+
Application is designed to use the ***csv2mysql*** LogFormat. LogFormat has comma-separated values and adds 8 Format Strings. A complete list of Format Strings
77
75
with descriptions indicating added Format Strings below.
|%{Referer}i|The "Referer" (sic) HTTP request header. This gives the site that the client reports having been referred from.|
102
100
|%{User-Agent}i|The User-Agent HTTP request header. This is the identifying information that the client browser reports about itself.|
103
101
|%{VARNAME}C|ADDED - The contents of cookie VARNAME in request sent to server. Only version 0 cookies are fully supported. Format String is optional.|
102
+
|%L|ADDED - The request log ID from the error log (or '-' if nothing has been logged to the error log for this request). Look for the matching error log line to see what request| caused what error.
104
103
## Two supported Error Log Formats
105
104
Application processes Error Logs with ***default format*** for threaded MPMs (Multi-Processing Modules). If you're running Apache 2.4 on any platform and ErrorLogFormat is not defined in config files this is the Error Log format.
106
105
|Format String|Description|
@@ -118,14 +117,19 @@ Application processes Error Logs with ***default format*** for threaded MPMs (Mu
Application also processes Error Logs with this ***additional format*** which adds `%v - The canonical ServerName`. Easiest way to identify error logs for each domain is add `%v` to ErrorLogFormat.
120
+
Application also processes Error Logs with ***additional format*** which adds:
121
+
1)`%v - The canonical ServerName` - This is easiest way to identify error logs for each domain is add `%v` to ErrorLogFormat.
122
+
2)`%L - Log ID of the request` - This is easiest way to associate Access record that created an Error record. Apache mod_unique_id.generate_log_id() only called when error occurs and will not cause performance degradation under error-free operations.
122
123
123
-
To use this format place the `ErrorLogFormat` before `ErrorLog` in `apache2.conf` to set error log format for Server and ALL VitualHosts on Server.
124
-
|Format String|Description - The spaces on each side of comma are required.|
124
+
To use this format place `ErrorLogFormat` before `ErrorLog` in `apache2.conf` to set error log format for ***Server*** and ***VitualHosts*** on Server.
125
+
|Format String|Description - `Space` required on left-side of `Commas` to parse data properly|
125
126
|-------------|-----------|
126
127
|%v|The canonical ServerName of the server serving the request.|
128
+
|%L|Log ID of the request. A %L format string is also available in `mod_log_config` to allow to correlate access log entries with error log lines. If `mod_unique_id` is loaded, its unique id will be used as log ID for requests.|
129
+
130
+
***Important:***`Space` required on left-side of `Commas` as defined below:
## Two options to attach ServerName & ServerPort to Access & Error logs
131
135
@@ -151,6 +155,7 @@ Log file naming conventions enable the use of UPDATE statements:
151
155
```
152
156
UPDATE apache_logs.import_file SET server_name='farmfreshsoftware.com', server_port=443 WHERE server_name IS NULL AND name LIKE '%farmfreshsoftware%';
153
157
UPDATE apache_logs.import_file SET server_name='farmwork.app', server_port=443 WHERE server_name IS NULL AND name LIKE '%farmwork%';
158
+
UPDATE apache_logs.import_file SET server_name='ip255-255-255-255.us-east.com', server_port=443 WHERE server_name IS NULL AND name LIKE '%error%';
154
159
```
155
160
First option requires uncommenting `os.getenv` to load variables at top of `apacheLogs2MySQL.py`. By default, variables are defined and set to an empty string. Below is a screenshot of the variables loaded at top of `apacheLogs2MySQL.py` with commented `os.getenv` code.
@@ -241,7 +246,7 @@ The second parameter enables Python Client modules to run simultaneously on mult
241
246
Database normalization is the process of organizing data in a relational database to improve data integrity and reduce redundancy.
242
247
Normalization ensures that data is organized in a way that makes sense for the data model and attributes, and that the database functions efficiently.
243
248
244
-
MySQL `apache_logs` schema has 46 tables, 772 columns, 128 indexes, 56 views, 7 stored procedures and 41 functions to process Apache Access log in 4 formats
249
+
MySQL `apache_logs` schema has 47 tables, 779 columns, 125 indexes, 56 views, 7 stored procedures and 42 functions to process Apache Access log in 4 formats
245
250
& Apache Error log in 2 formats. Database normalization at work!
246
251
## MySQL Access Log View by URI
247
252
MySQL View - apache_logs.access_requri_list - data from LogFormat: combined & csv2mysql
Copy file name to clipboardExpand all lines: .github/SUPPORT.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
### Please include Apache Logformat or ErrorLogFormat if a missing or malformed data issue.
4
4
5
-
### My office hours are 'almost always' unless I am doing farm chores, attending my kid's sporting events or sleeping. I rarely leave the farm so if I do not reply within 24 hours I am working on a project outside. If I do not reply within 48 hours something bad might have occurred.
5
+
### My office hours are 'almost always' unless I am doing farm chores, attending my kid's sporting events or sleeping. Rarely do I leave the farm so if I do not reply within 24 hours I am working on a project outside. If I do not reply within 48 hours something bad might have happened.
6
6
7
7
### I am designer, developer and support staff with vast knowledge of Apache, MySQL and Python capable of fixing any issue with application.
0 commit comments