I looked at run-regression-tests.pl.in quickly and it seems to me that match_log might be the issue. Log flushing can differ significantly between systems, so it's possible that searching the log in each test takes a long time. I suggest you add some debug print statements to the Perl script so that you can see in the CI output which operations use a lot of time.

I tried to review the script again, and fixed two small things in bbe5239:

• switched to select() from sleep()
• clear the buffer which hold the read log

Also I removed the test regression/misc/10-tfn-cache.t, because it took many time (on my machine the whole test file was 1m 6secs). The reason is that it seems SecCacheTransformations directive is deprecated.

Until I write this comment, the tests has started, but it seems that no performance improvement. (And beside of that there are two failed tests...)

Okay, I think the reason is visible now but unknown: see the output of the last test. The Apache stop time is terrible slow.

And I don't know why...

@theseion, @dune73 do you have any idea?

If I remember correctly, Apache waits for connections to be closed or whatever. I do not think it sends a RST or something, but it really waits. So if it's slow to stop, I would look at the connection table during the stopping.

Stopping Apache in production is always slow. In a test situation that's not ideal. Maybe you want to kill it instead.

Take a look at my script https://www.netnea.com/files/apachex that I use for control of Apache in my lab. It's a very rough tool, more a mallet than a screwdriver, but it never fails me to kill and restore apache instantly.

Not sure this is the right path, but that's what I have in mind.

There are probably two things you can do:

1. Test request should be sent with Connection: close
2. Kill httpd

The first is just a nice-to-have of course. I'd absolutely simply kill httpd. There's no reason for handling open connections gracefully in this scenario.

Thanks for ideas,

1. Test request should be sent with `Connection: close`

All requests send Connection: close header (I checked with tcpdump), eg:

10:55:19.058098 IP localhost.omniorb > localhost.33008: Flags [P.], seq 1:225, ack 144, win 512, options [nop,nop,TS val 2093795869 ecr 2093795869], length 224
E...~d@.@..}............J..................
|...|...HTTP/1.1 200 OK
Date: Sat, 08 Nov 2025 09:55:19 GMT
Server: Apache/2.4.58 (Ubuntu)
Last-Modified: Sat, 08 Nov 2025 09:46:10 GMT
ETag: "5-643122b0cc4e7"
Accept-Ranges: bytes
Content-Length: 5
Connection: close

TEST

2. Kill httpd

I tried, but then many other weird issues came out. Eg. when httpd_start() function is called, eg:

Loaded 2 tests from /home/airween/src/modsecurity-apache/tests/regression/config/00-load-modsec.t
  1) config - module loaded: passed
Httpd start failed with error messages:
httpd (pid 10748) already running

The first is just a nice-to-have of course. I'd absolutely simply kill httpd. There's no reason for handling open connections gracefully in this scenario.

I'm afraid I have to reorganize the complete start/stop mechanism.

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

I'm afraid I have to reorganize the complete start/stop mechanism.

How are you killing the process(es)? It looks like you did kill and then start_httpd() immediately, so your requests start being processed before the old process has been killed. Maybe you need to put a waitpid in there.

I'm afraid I have to reorganize the complete start/stop mechanism.

How are you killing the process(es)?

here is the current routine which stops httpd.

This section builds and runs the command:

    my @p = (
        $HTTPD,
        -d => $opt{S},
        -f => $opt{C},
        (map { (-c => $_) } ("Listen localhost:$opt{p}", @_)),
        -k => "stop",
    );

    my $httpd_out;
    my $httpd_pid = open3(undef, $httpd_out, undef, @p) or quit(1);

@p holds the command and open3() runs that. It's approximately httpd -k stop.

It looks like you did kill and then start_httpd() immediately, so your requests start being processed before the old process has been killed. Maybe you need to put a waitpid in there.

There is a waitipd() already..

Edit: it seems like the waitpid() waits for terminating of httpd -k stop command. That takes a lot of time.

Edit: it seems like the waitpid() waits for terminating of httpd -k stop command. That takes a lot of time.

Yes, exactly. I was thinking of kill + waitpid (if necessary).

The shutdown time appears to have improved in Apache HTTP Server 2.4.62 and later:
apache/httpd@9005268

I tested with apache2 2.4.65-3ubuntu1 on Ubuntu 25.10, and the shutdown is fast.

In contrast, shutdown was slow with apache2 2.4.58-1ubuntu8.8 on Ubuntu 24.04.

As a workaround, I tried using -k graceful-shutdown instead of -k stop in httpd_stop in tests/run-regression-tests.pl. This allows the next test to start earlier; however, the old worker processes remain for some time, so increased memory usage could be an issue in CI environments.

Oops, sorry for my typo: it is graceful-stop, not graceful-shutdown.

Hi @hnakamur,

thanks for this great help,

The shutdown time appears to have improved in Apache HTTP Server 2.4.62 and later: apache/httpd@9005268

ahm, sorry to see this.

I tested with apache2 2.4.65-3ubuntu1 on Ubuntu 25.10, and the shutdown is fast.

unfortunately there is no Ubuntu 25.10 option for CI on GH.

In contrast, shutdown was slow with apache2 2.4.58-1ubuntu8.8 on Ubuntu 24.04.

yepp',

As a workaround, I tried using -k graceful-shutdown instead of -k stop in httpd_stop in tests/run-regression-tests.pl. This allows the next test to start earlier; however, the old worker processes remain for some time, so increased memory usage could be an issue in CI environments.

thanks again, see commit a90c4f5 where I changed the method. But it seems -k start is not available. I have to review the other changes again.

Huhh,

@hnakamur, so many thanks for your investigations again.

Now the regression test is so fast:

The total run time was 5:23:

I think this is good enough for us to integrate this test to our CI.

Btw I'm still curious, because I tried this test on one of my VM Ubuntu 24.04, and could reproduce the problem.

But I definitely remember, that when I tried mod_security2's regression test at first time, years ago, when Debian SID probably had this Apache version, the situation was the same. I think it was about summer of 2024. Then I was horrified and started to develop a new test case, MRTS.

Never mind, I'm very happy you find the solution, so we can apply this PR.

@fzipi, @theseion: could you take a review on this PR and approve it?

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud