Exponential backtracking in regex blocks Thread #425
Description
Hey,
I am currently facing problem with servers which have regular expressions within their robots.txt which require an unreasonable amount of computing time to match for using RobotstxtServer.java (https://www.regular-expressions.info/catastrophic.html). Eg:
User-agent: *
Disallow: /********************/
Disallow: /*******************
Matching for the url: "/asdjdsfsdfjkhejrhwjerhjkfdhksdjfhksjdfhjksdfhjksfdhjksdfasdasdd/js/jquery/jquery-migrate.min.js" completly freezes the thread. The console just shows:
10:53:34.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:39.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:39.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:44.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:44.438 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:49.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:49.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:54.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:54.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:53:59.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:53:59.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:04.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:04.439 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:09.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:09.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:14.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:14.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:19.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:19.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:24.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:24.440 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:29.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:29.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:34.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:34.441 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:39.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:39.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:44.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:44.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:49.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:49.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:54.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:54.442 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
10:54:59.443 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing expired connections
10:54:59.443 [Connection Manager] DEBUG e.u.i.c.f.SniPoolingHttpClientConnectionManager - Closing connections idle longer than 30 SECONDS
Is there a way around this?