Replace current regex engine with PCRE2 by Rot127 · Pull Request #4033 · rizinorg/rizin

Rot127 · 2023-12-13T23:03:52Z

Your checklist for this pull request

I've read the guidelines for contributing to this repository
I made sure to follow the project's coding style
I've documented or updated the documentation of every function and struct this PR changes. If not so I've explained why.
I've added tests that prove my fix is effective or that my feature works (if possible)
I've updated the rizin book with the relevant information (if needed)

Detailed description

Replace the current regex engine with PCRE2.

Test plan

All green.

Closing issues

closes #3730

Partially addresses #4055

Todo

Add a rz_vector_pop_ptr() function which doesn't use memcpy() and use it in match_all_flat(). Or add a rz_vector_concat() function with the same functionality.
Terminology for groups and matches is weird.
docs
Open bug report for tcc and pcre2 compilation and __asm
Test rz_regex_get_match_name

XVilka

Please send it from dist- branch from rizinorg account, to check that it builds and runs fine on all supported platforms and configurations
Once it's done, I think it is worth removing the old implementation as well.

wargio · 2023-12-14T03:17:19Z

would be nice to also allow PCRE (v1) for older distro

Rot127 · 2023-12-14T16:03:38Z

would be nice to also allow PCRE (v1) for older distro

Currently for every distro which has no pcre2 it is compiled as subproject. Shouldn't this be enough?

XVilka · 2023-12-14T16:24:37Z

would be nice to also allow PCRE (v1) for older distro

Currently for every distro which has no pcre2 it is compiled as subproject. Shouldn't this be enough?

Yes, I think it should be enough. We have the same strategy for Capstone dependency as well.

Rot127 · 2023-12-14T18:26:03Z

The PCRE2 defines a huge amount of options and flags. Is it fine if we do not add an RZ_REGEX_... variant for each of it? Simply to avoid duplicates?
Or is it always preferable to have our API consistently named?

wargio · 2023-12-15T04:25:42Z

you should only use PCRE2_CASELESS | PCRE2_MULTILINE
where caseless is applied only if the grep configuration is auto or case insensitive.

Rot127 · 2024-01-04T13:54:23Z

Tests/build still fail, but I'd like to know your opinion on the API first. Because now there are some changes in actual code now.

XVilka

The new API looks fine. Have you tried comparing the performance with and without these changes?

XVilka · 2024-01-04T14:42:49Z

@@ -0,0 +1,5 @@
+[wrap-git]
+url = https://github.com/PCRE2Project/pcre2.git
+revision = 52c08847921a324c804cabf2814549f50bce1265


Shouldn't we use release by default?

Looks like they are preparing to make a new release soon, which would be great if done before our release: https://github.com/PCRE2Project/pcre2/commits/master/

XVilka · 2024-01-04T14:44:58Z

+	pcre2_code_free(regex);
+}
+
+RZ_OWN RzRegexMatchData *rz_regex_match_data_new(const RzRegex *regex, RzRegexGeneralContext *context) {


Doxygen for these functions is more important than oxygen for _free() function ;-)

XVilka

Please also fix test\db\archos\windows-x64\dbg_dts - remove \n or just don't include it in the capture.
macOS (Darwin) regexes should be fixed too, in particular with whitespace and newlines handling: https://ci.rizin.re/repos/27/pipeline/4083/7#L616

OpenBSD error message is puzzling:

ERROR: Regex compilation failed at 0: no more memory

Same happens on NetBSD too.

Rot127 · 2024-02-03T04:34:02Z

Yep, aware of it. Just wait and bundle it with other fixes, so the CI is not triggered again and again.

Rot127 · 2024-02-03T04:44:33Z

BSD problems seem to be a bug in PCRE2 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276252). Hopefully only affecting the JIT compiler?

Going make a minimal example and open an issue later. If it is indeed a BSD only problem, I would just exclude BSD from JIT/

XVilka · 2024-02-03T04:53:38Z

@Rot127 yes, though FreeBSD works just fine. The problem occurs only in OpenBSD and NetBSD, please disable JIT only for those, but not the FreeBSD.

XVilka · 2024-02-03T04:56:34Z

Once this PR is green after the rebase, please send one from the dist-fuzz- branch, to check all configurations. Also, do you want to squash all those commits into one?

Rot127 · 2024-02-03T05:24:57Z

Also, do you want to squash all those commits into one?

Absolutely. The commit history is a mess

Rot127 · 2024-02-03T09:54:04Z

Should be fixed now.

PCRE2 has way better performance than the OpenBSD library (something around 20 times faster). The following flags are enabled for every pattern: - PCRE2_UTF - PCRE2_MATCH_INVALID_UTF - PCRE2_NO_UTF_CHECK All the others are optional. Changes made: - Adds PCRE2 as subproject. - Changes the API away from POSIX to PCRE2. - Edits many regex patterns because: - ' ' is skipped in patterns, if the EXTENDED flag is set for matching. '\s' must be set now. - '.' doesn't match newlines by default. - Changes the API so matches and their groups are bundled into PVectors. - Moves the regex component to rz_util.

XVilka · 2024-02-03T11:54:19Z

@Rot127 please send a new PR from inside the rizinorg account with the dist-fuzz- branch to trigger everything.

Rot127 requested review from XVilka, kazarmy, ret2libc, thestr4ng3r and wargio as code owners December 13, 2023 23:03

Rot127 marked this pull request as draft December 13, 2023 23:04

Rot127 changed the title ~~Add PCRE2 as dependency.~~ Replace current regex engine with PCRE2 Dec 13, 2023

Rot127 added the performance A performance problem/enhancement label Dec 13, 2023

XVilka requested changes Dec 14, 2023

View reviewed changes

github-actions bot added API RzUtil labels Dec 18, 2023

wargio reviewed Dec 20, 2023

View reviewed changes

Comment thread subprojects/packagefiles/pcre2/meson.build Outdated

wargio reviewed Dec 20, 2023

View reviewed changes

Comment thread subprojects/packagefiles/pcre2/meson.build

wargio reviewed Dec 20, 2023

View reviewed changes

Comment thread meson.build Outdated

wargio reviewed Dec 20, 2023

View reviewed changes

Comment thread librz/util/regex.c Outdated

Rot127 force-pushed the pcre2 branch 2 times, most recently from 558652e to 4168046 Compare January 3, 2024 17:28

github-actions bot added rz-test RzCore RzParse RzCons RzMagic RzSearch labels Jan 4, 2024

XVilka requested changes Jan 4, 2024

View reviewed changes

github-actions bot removed the infrastructure label Feb 2, 2024

Rot127 marked this pull request as ready for review February 2, 2024 08:06

wargio reviewed Feb 2, 2024

View reviewed changes

Comment thread librz/include/rz_util/rz_regex.h Outdated

wargio reviewed Feb 2, 2024

View reviewed changes

Comment thread librz/include/rz_util/rz_regex.h Outdated

XVilka requested changes Feb 2, 2024

View reviewed changes

Comment thread librz/cons/pager.c

Rot127 force-pushed the pcre2 branch from 892348b to 1ef41f3 Compare February 3, 2024 03:07

This comment was marked as resolved.

Sign in to view

Rot127 force-pushed the pcre2 branch 4 times, most recently from 5ff6680 to 5a5d5af Compare February 3, 2024 08:41

This comment was marked as resolved.

Sign in to view

Rot127 force-pushed the pcre2 branch from c781400 to aa54f54 Compare February 3, 2024 09:22

Rot127 force-pushed the pcre2 branch from 9ee4212 to 730621a Compare February 3, 2024 09:55

wargio reviewed Feb 3, 2024

View reviewed changes

Comment thread meson.build Outdated

wargio reviewed Feb 3, 2024

View reviewed changes

Comment thread subprojects/packagefiles/pcre2/meson.build Outdated

XVilka approved these changes Feb 3, 2024

View reviewed changes

Comment thread librz/magic/softmagic.c

Rot127 force-pushed the pcre2 branch from 730621a to d11a763 Compare February 3, 2024 11:36

thestr4ng3r reviewed Feb 3, 2024

View reviewed changes

Comment thread test/unit/test_str.c

Rot127 mentioned this pull request Feb 4, 2024

Replace current regex engine with PCRE2 #4185

Merged

10 tasks

Rot127 closed this Feb 4, 2024

Uh oh!

Conversation

Rot127 commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XVilka left a comment

Choose a reason for hiding this comment

Uh oh!

wargio commented Dec 14, 2023

Uh oh!

Rot127 commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XVilka commented Dec 14, 2023

Uh oh!

Rot127 commented Dec 14, 2023

Uh oh!

wargio commented Dec 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rot127 commented Jan 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XVilka left a comment

Choose a reason for hiding this comment

Uh oh!

XVilka Jan 4, 2024

Choose a reason for hiding this comment

Uh oh!

XVilka Jan 7, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

XVilka Jan 4, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XVilka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Rot127 commented Feb 3, 2024

Uh oh!

Rot127 commented Feb 3, 2024

Uh oh!

XVilka commented Feb 3, 2024

Uh oh!

XVilka commented Feb 3, 2024

Uh oh!

Rot127 commented Feb 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Rot127 commented Feb 3, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XVilka commented Feb 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Rot127 commented Dec 13, 2023 •

edited

Loading

Rot127 commented Dec 14, 2023 •

edited

Loading

Rot127 commented Jan 4, 2024 •

edited

Loading

Rot127 commented Feb 3, 2024 •

edited

Loading