Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.2.0 does not work with Python 3.10 #55

Open
dfarley1 opened this issue Apr 7, 2023 · 5 comments
Open

v0.2.0 does not work with Python 3.10 #55

dfarley1 opened this issue Apr 7, 2023 · 5 comments

Comments

@dfarley1
Copy link

dfarley1 commented Apr 7, 2023

We're currently using v0.2.0 since we have a customized version of libhs. Since trying to update to Python 3.10 we're seeing the following error (in a container based on python:3.10-bullseye:

$ python3 -m pip install hyperscan==0.2.0 --no-cache-dir
Defaulting to user installation because normal site-packages is not writeable
Collecting hyperscan==0.2.0
  Downloading hyperscan-0.2.0.tar.gz (13 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: hyperscan
  Building wheel for hyperscan (pyproject.toml) ... done
  Created wheel for hyperscan: filename=hyperscan-0.2.0-cp310-cp310-manylinux_2_31_x86_64.whl size=12566 sha256=c6665898f9cdb3b601ccbc21d0a432838502eb6644ab555e7220cf8e8a5bd341
  Stored in directory: /tmp/pip-ephem-wheel-cache-scq2qfof/wheels/4d/2a/88/2f04c0e9dfd098657e8617bff17d277bfa9eb5febc26b4ba33
Successfully built hyperscan
Installing collected packages: hyperscan
Successfully installed hyperscan-0.2.0
$ python3
Python 3.10.11 (main, Apr  5 2023, 23:35:11) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyperscan
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/hyperscan/__init__.py", line 5, in <module>
    from hyperscan._hyperscan import *
ModuleNotFoundError: No module named 'hyperscan._hyperscan'
>>> 

Upgrading to v0.4.0 fixes the problem, but we can't use it because we require dynamically linking libhs. I suppose this is either a bug report for the old version or a feature request for dynamic linking support in the newer version?

@dfarley1
Copy link
Author

dfarley1 commented Apr 7, 2023

cc @betterlch

@betterlch
Copy link

cc @betterlch
I just know that 0.2.0 works in python3.7~


Here are other work about python3.10:

I linked libhs.so as _hyperscn.so, and I get this

ImportError: dynamic module does not define module export function (PyInit__hyperscan)

And then I use

strace python3 -c 'import hyperscan'

It has no more help.
I don't how C extending works in python3, so I have no ideas.

@darvid
Copy link
Owner

darvid commented Apr 10, 2023

just added a new environment variable to enable dynamic linking when building, PYTHON_HYPERSCAN_STATIC (defaults to true). try pulling latest and building with:

PYTHON_HYPERSCAN_STATIC=false poetry install

@betterlch
Copy link

just added a new environment variable to enable dynamic linking when building, PYTHON_HYPERSCAN_STATIC (defaults to true). try pulling latest and building with:

PYTHON_HYPERSCAN_STATIC=false poetry install

Something maybe not as you wish:

running build_ext
building 'hyperscan._hyperscan' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/include/openssl11 -I/usr/include/openssl11 -fPIC -I/usr/include/hs -I/usr/include/hs -I/tmp/tmpf0_08_r3/.venv/include -I/usr/local/include/python3.10 -c src/hyperscan/hyperscanmodule.c -o /root/py/python-hyperscan/build/temp.linux-x86_64-cpython-310/src/hyperscan/hyperscanmodule.o
src/hyperscan/hyperscanmodule.c: In function ‘ch_match_handler’:
src/hyperscan/hyperscanmodule.c:124:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (unsigned int i = 0; i < size; i++) {
   ^
src/hyperscan/hyperscanmodule.c:124:3: note: use option -std=c99 or -std=gnu99 to compile your code
src/hyperscan/hyperscanmodule.c: In function ‘Database_compile’:
src/hyperscan/hyperscanmodule.c:270:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (uint64_t i = 0; i < elements; i++) {
   ^
src/hyperscan/hyperscanmodule.c:331:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
     for (uint64_t i = 0; i < elements; i++) {
     ^
src/hyperscan/hyperscanmodule.c:384:9: error: ‘for’ loop initial declarations are only allowed in C99 mode
         for (uint64_t i = 0; i < elements; i++) {
         ^
src/hyperscan/hyperscanmodule.c: In function ‘Database_scan’:
src/hyperscan/hyperscanmodule.c:547:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
     for (uint32_t i = 0; i < num_buffers; i++) {
     ^
error: command '/usr/bin/gcc' failed with exit code 1

Resolved by change code to std format c11

And new error is here:

Preparing build environment with build-system requirements setuptools, poetry>=0.12Building hyperscan (0.4.0)
A setup.py file already exists. Using it.
running build
running build_py
copying src/hyperscan/hyperscanmodule.c -> /root/py/python-hyperscan/build/lib.linux-x86_64-cpython-310/hyperscan
running build_ext
building 'hyperscan._hyperscan' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/include/openssl11 -I/usr/include/openssl11 -fPIC -I/usr/include/hs -I/usr/include/hs -I/tmp/tmp5r6qyfy_/.venv/include -I/usr/local/include/python3.10 -c src/hyperscan/hyperscanmodule.c -o /root/py/python-hyperscan/build/temp.linux-x86_64-cpython-310/src/hyperscan/hyperscanmodule.o
gcc -pthread -shared -L/usr/lib64/openssl11 -lssl -lcrypto -L/usr/lib64/openssl11 -lssl -lcrypto /root/py/python-hyperscan/build/temp.linux-x86_64-cpython-310/src/hyperscan/hyperscanmodule.o /opt/pcre/.libs/libpcre.a /opt/pcre/.libs/libpcre_la-pcre_byte_order.o /opt/pcre/.libs/libpcre_la-pcre_compile.o /opt/pcre/.libs/libpcre_la-pcre_config.o /opt/pcre/.libs/libpcre_la-pcre_dfa_exec.o /opt/pcre/.libs/libpcre_la-pcre_exec.o /opt/pcre/.libs/libpcre_la-pcre_fullinfo.o /opt/pcre/.libs/libpcre_la-pcre_get.o /opt/pcre/.libs/libpcre_la-pcre_globals.o /opt/pcre/.libs/libpcre_la-pcre_jit_compile.o /opt/pcre/.libs/libpcre_la-pcre_maketables.o /opt/pcre/.libs/libpcre_la-pcre_newline.o /opt/pcre/.libs/libpcre_la-pcre_ord2utf8.o /opt/pcre/.libs/libpcre_la-pcre_refcount.o /opt/pcre/.libs/libpcre_la-pcre_string_utils.o /opt/pcre/.libs/libpcre_la-pcre_study.o /opt/pcre/.libs/libpcre_la-pcre_tables.o /opt/pcre/.libs/libpcre_la-pcre_ucd.o /opt/pcre/.libs/libpcre_la-pcre_valid_utf8.o /opt/pcre/.libs/libpcre_la-pcre_version.o /opt/pcre/.libs/libpcre_la-pcre_xclass.o /opt/pcre/.libs/libpcre_la-pcre_chartables.o /opt/pcre/.libs/libpcreposix_la-pcreposix.o /opt/pcre/.libs/libpcrecpp_la-pcrecpp.o /opt/pcre/.libs/libpcrecpp_la-pcre_scanner.o /opt/pcre/.libs/libpcrecpp_la-pcre_stringpiece.o -lchimera -lhs -o /root/py/python-hyperscan/build/lib.linux-x86_64-cpython-310/hyperscan/_hyperscan.cpython-310-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libchimera.a(ch_compile.cpp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libchimera.a(ch_runtime.c.o): relocation R_X86_64_32S against undefined symbol `mmbit_maxlevel_direct_lut' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libchimera.a(ch_alloc.c.o): relocation R_X86_64_32 against symbol `malloc@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
error: command '/usr/bin/gcc' failed with exit code 1

@betterlch
Copy link

  1. download and make install hypercan 5.4.0 with doc guide
  2. use cmake -DCMAKE_POSITION_INDEPENDENT_CODE=ON .. remake hyperscan/build/chimera --> hyperscan/build/lib/libchimera.a
  3. add /opt/pcre/ and build pcre-8.4.5 to gen /opt/pcre/.libs for build python-hyperscan
  4. poetry build -vvv (maybe need update cmake an add c11 support in python-hyperscan sourcecode)

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants