How to handle huge trace files

[bobbingwide]

Whilst it's not advisable to trace in a production system, it's possible to forget to disable tracing.
If a trace file is not being reset for each request, ie. if the trace file is being appended to, then the file can get very large indeed.

Something needs to be done about this since it can lead to the server returning HTTP error code 500.

Note: There could be a problem with the trace file being written to even when tracing is only configured for a specific IP address. I'm looking into it.

[bobbingwide]

One possible solution to this is to implement some cycling logic using generations and sets of trace files.
Where

• a generation is a finite number of files with a particular file mask.
• sets are used to create multiple file masks.

The current logic for file naming is:

• bwtrace.loh is the default for 'normal' requests
• The trace file name and extension can be specified. e.g. bwtraces.loh
• AJAX requests can be traced to a different file e.g. bwtrace.ajax
• batch requests may be traced to a different file name.

The new file mask would be of the form: path/filename.set.generation
where

• path is optional and relative to ABSPATH
• filename defaults to bwtrace
• set defaults to the request type, but can be set per request type
• generation is an integer from 1 to limit.
• limit defaults to 100

The generation logic will depend on the value given for limit.

limit	Generation logic	Generation
blank/null	Not used.	null
0	Unlimited	.timestamp - from REQUEST_TIME_FLOAT
>0	Cycling	.generation

The cycling logic is intended to ensure that concurrent requests do not use the same trace file.

• The generation number picked is the first missing from the series 1 ... limit.
• If all generations are used then we pick the oldest file, based on file modification time.
• The logic to determine the oldest file needs to be fairly be efficient.
• The maximum value of limit should be a reasonable number.

Concurrently executed transactions may still choose the same file.
If this happens a lot then use the limit =0 option.
Trace file reset logic will apply to the selected trace file, except when the limit is 0.

reset	generation	Results in	Supports concurrency
n	n	One big file . i.e. the current situation.	Output mixed
n	y	Lots of ever increasing files	Should do
y	n	Trace file reset every transaction	Possible loss of data
y	y	Selected trace file reset	Should do

[bobbingwide]

This change should also take in account and/or help address.

• Improve support for tracing REST API processing #52
• Make bw_trace_reset_status() more context sensitive #49
• Reduce messages produced by bw_trace_reset() #46

[bobbingwide]

The trace admin page should be updated to reflect the generation logic

1. The trace file name will be wrong when the value of limit is set.
2. The reset capability of the [bwtrace] shortcode needs to be revisited.

[bobbingwide]

Using the generation limit of 0 for REST calls allows concurrent tracing of the REST requests issued by the new block editor ( Gutenberg ). But it's not necessary for the standard browser, AJAX or batch ( CLI ) requests. So, I'm going to make 4 versions of the "limit" setting; one per request type. The Trace options page will be updated. screenshot-1.jpg reflects the latest version.

https://github.com/bobbingwide/oik-bwtrace/blob/23b773c69d108fee586cd05b6c6be9334e34a6fd/screenshot-1.jpg

Note: I haven't updated the PHPUnit tests yet.

[bobbingwide]

I noticed that trace reset is not working when the generation limit is not set.
The logic in trace_file-selector::reset_as_required() is incorrect.
The solution is to change the switch statement to a simple if statement.

[bobbingwide]

There is still the possibility that the trace output files could use up a lot of disk space. The oik trace admin section for files shows the total space used for the files. And there’s logic to purge the files. This is documented, so I believe it’s safe to close the issue.