Installation bug

[mmar30]

Describe the bug
I can’t install khiva, when I run make –j8 I get the following error after 32% :

CMakCe EMrraork aet /EAprprliocr ataiton s//CMAakpe.palppi/Ccoantteintosn/ssh/aCrMe/ackmeak.ea-3p.p14//CMoodnultees/nGtoosg/lesThesatrAded/Tcemsatkse.-c3m.a1k4e/:M4o0d (message):
u l eEsr/rGooro grluenTneisntgA dtdeTsets tesx.eccmuatkaeb:l4e0.
(
m e s s aPgateh): :'
/U s eErrsr/omra rriuan/nPiyncgha rtmePsrto jeexcetcsu/tkahbilvea.N
e
w / b u iPladt/bhi:n /'p/oUlsyenrosm/imaalr'i
a / P y cRheasrumlPtr:o jPercoctess/sk htievramNienwa/tbeudi lddu/eb itno/ vteirmseioount'

   O utRpuets:

u l t : P
r
o
cess termin
ated due to timeout
Output:

make[2]: *** [bin/polynomial] Error 1
make[2]: *** Deleting file bin/polynomial' make[2]: *** [bin/version] Error 1 make[2]: *** Deleting file bin/version'
make[1]: *** [test/CMakeFiles/polynomial.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [test/CMakeFiles/version.dir/all] Error 2
[ 32%] Built target statistics
[ 32%] Built target regularization
[ 32%] Built target normalization
[ 32%] Built target clustering
[ 32%] Built target matrix
fatal: not a git repository (or any of the parent directories): .git

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
File "/Users/maria/anaconda3/lib/python3.7/site-packages/sphinx/config.py", line 368, in eval_config_file
execfile_(filename, namespace)
File "/Users/maria/anaconda3/lib/python3.7/site-packages/sphinx/util/pycompat.py", line 150, in execfile_
exec_(code, _globals)
File "/Users/maria/PycharmProjects/khivaNew/doc/sphinx/source/conf.py", line 44, in
["git", "describe"]).strip().decode("utf-8").split('-')[0]
File "/Users/maria/anaconda3/lib/python3.7/subprocess.py", line 389, in check_output
**kwargs).stdout
File "/Users/maria/anaconda3/lib/python3.7/subprocess.py", line 481, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'describe']' returned non-zero exit status 128.

make[2]: *** [doc/CMakeFiles/documentation] Error 2
make[1]: *** [doc/CMakeFiles/documentation.dir/all] Error 2
make: *** [all] Error 2

To Reproduce
after running:
make -j8

Environment information:

• OS: 10.14.3
• Python 3.5
• ArrayFire 3.6.2
• Conan version 1.13.2

[avilchess]

Hi Maria,

It seems we are using a git command during the compilation process, to get the current version of khiva. I believe your source code is not in a git folder. Have you downloaded the code and moved it into a different folder? Could you check you have a hidden .git/ folder in your khiva source code folder. If it is not the case, please delete all files and do a git clone from that folder. It should solve this issue.

Let us know if it solves your problem.

[mmar30]

I've tried to install it after git clone, but I still get this error:

error: problems opening map file /Users/maria/PycharmProjects/khiva2/khiva/build/doc/doxygen/html/array_8h__dep__incl.map for inclusion in the docs!
If you installed Graphviz/dot after a previous failing run, 
try deleting the output directory and rerun doxygen.
Patching output file 2/40
error: problem writing FIG 0 figure!
lookup cache used 380/65536 hits=3570 misses=453
finished...

Extension error:
Could not import extension breathe (exception: No module named 'breathe')
make[2]: *** [doc/CMakeFiles/documentation] Error 2
make[1]: *** [doc/CMakeFiles/documentation.dir/all] Error 2
make: *** [all] Error 2

Even though I've installed breathe and deleted the output directory of doxygen and ran it again, I get this error.

[avilchess]

Hi again,

I see you have successfully compiled the source code. However, you are stopped in the documentation generation step.

1. First solution. Generate documentation.
   It seems it can´t find breathe, please check your python and pip versions are the same ( with python --version and pip --version). Sometimes, we have several versions of python installed in our system. Try to install sphinx and breathe using the corresponding pip version to your python installation ( i.e. it could be one of the followingpip pip2 pip2.7 pip3 pip3.6). I believe sphinx is trying to find breathe, but it is looking in the wrong environment.

2. Second Solution. Skip the documentation generation step.
   If you don't care about documentation, you can skip this step with the following command:

• Go to build folder
• cmake .. -DKHIVA_BUILD_DOCUMENTATION=OFF
• make all -j8

3. Third solution. Install khiva from a package in the release list.
   The easiest way to install Khiva is by using a released package, try the last version for macOs
   You just need to download and install it by doing double click.

Hope it helps.
Antonio Vilches.

[mmar30]

Hi Antonio,

I tried all three solutions. After I run the second without an error and installed the khiva package with pip install khiva, I get the fallowing error when I run the example code:

import numpy as np
from khiva.matrix import*
stomp_result = stomp(Array(np.array([11, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,11])),
                     Array(np.array([9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,9])),
                     3)
find_best_n_discords_result = find_best_n_discords(stomp_result[0],stomp_result[1], 2)
a = find_best_n_discords_result[2].to_numpy()
print(a)

ERROR:

libc++abi.dylib: terminating with uncaught exception of type af::exception: ArrayFire Exception (Failed to load dynamic library. :501):
Error in af_err unified::AFSymbolManager::call(const char *, CalleeArgs...) [CalleeArgs = <void **, const void *, unsigned int, const long long *, af_dtype>]
In file src/api/unified/symbol_manager.hpp:61
Failed to load dynamic library. See http://www.arrayfire.com/docs/unifiedbackend.htm for instructions to set up environment for Unified backend.

In function void af::initDataArray(af_array *, const T *, af::source, dim_t, dim_t, dim_t, dim_t) [T = float]
In file src/api/cpp/array.cpp:141

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

[avilchess]

Hi Maria,

I see you are getting a new error, I'm happy previous errors are solved. So, let's focus on this new one. You are getting an exception from arrayfire because we are using the unified interface and it can't find the corresponding arrayfire backend library in your SO at runtime, why is that?. This problem is raising from ArrayFire 3.6.2, because they moved the default installation dir from /usr/local/lib to /opt/arrayfire/lib. And the unified interface is looking in the old directory.

How do we solve this issue?. There are possible 2 solutions for this:

1. Copy all arrayfire files to /usr/local/lib, execute the following commands.

• cp /opt/arrayfire/lib/* /usr/local/lib/
• cp /opt/arrayfire/include/* /usr/local/include/
• cp /opt/arrayfire/share/* /usr/local/share/

2. Add /opt/arrayfire/lib folder to the environment variableLD_LIBRARY_PATH, execute the following command:

• export LD_LIBRARY_PATH=/opt/arrayfire/lib/:$LD_LIBRARY_PATH

Note: the second option will only work if you have disabled the SIP (System Integrity Protection) of APPLE, check this. Because this protection doesn't allow you to modify the env variable LD_LIBRARY_PATH.

[avilchess]

@mmar30 did it work?

[mmar30]

unfortunately both solutions throw the same error as before :(

[skoeb]

My installation is also failing in <make -j4>

[ 32%] Linking CXX executable ../bin/features
CMakeFiles/features.dir/featuresTest.cpp.o: In function length()': featuresTest.cpp:(.text+0x2addf): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)'
featuresTest.cpp:(.text+0x2ae76): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' CMakeFiles/features.dir/featuresTest.cpp.o: In function absoluteSumOfChanges()':
featuresTest.cpp:(.text+0x2b883): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' featuresTest.cpp:(.text+0x2b92a): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)'
featuresTest.cpp:(.text+0x2b9d1): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' CMakeFiles/features.dir/featuresTest.cpp.o:featuresTest.cpp:(.text+0x2c79e): more undefined references to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)' follow
collect2: error: ld returned 1 exit status
test/CMakeFiles/features.dir/build.make:98: recipe for target 'bin/features' failed
make[2]: *** [bin/features] Error 1
CMakeFiles/Makefile2:260: recipe for target 'test/CMakeFiles/features.dir/all' failed
make[1]: *** [test/CMakeFiles/features.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2
(base) root@sam-server:~/Downloads/khiva/build#

[skoeb]

I've tried installing from the release .deb file (ubuntu) too, but that is failing silently.

[otorreno]

My installation is also failing in <make -j4>

[ 32%] Linking CXX executable ../bin/features
CMakeFiles/features.dir/featuresTest.cpp.o: In function length()': featuresTest.cpp:(.text+0x2addf): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)'
featuresTest.cpp:(.text+0x2ae76): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' CMakeFiles/features.dir/featuresTest.cpp.o: In function absoluteSumOfChanges()':
featuresTest.cpp:(.text+0x2b883): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' featuresTest.cpp:(.text+0x2b92a): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)'
featuresTest.cpp:(.text+0x2b9d1): undefined reference to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' CMakeFiles/features.dir/featuresTest.cpp.o:featuresTest.cpp:(.text+0x2c79e): more undefined references to testing::internal::EqFailure(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)' follow
collect2: error: ld returned 1 exit status
test/CMakeFiles/features.dir/build.make:98: recipe for target 'bin/features' failed
make[2]: *** [bin/features] Error 1
CMakeFiles/Makefile2:260: recipe for target 'test/CMakeFiles/features.dir/all' failed
make[1]: *** [test/CMakeFiles/features.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2
(base) root@sam-server:~/Downloads/khiva/build#

Hi @skoeb, we are sorry you had problems installing our library.

Please take a look to #67, your problem might be related with the C++ ABI Conan linked the GTest library with. Follow the procedure mentioned in the issue mentioned just before. If the problem persists don’t hesitate to contact us again.

[skoeb]

I tried changing the setting in the conan default profile to <compiler.libcxx=libc++11>, but this wasn't recognized by conan, so I used <compiler.libcxx=libstdc++11>.

The error during <make all -j4> is now:

[ 26%] Linking CXX executable ../bin/library
/home/skoeb/Downloads/khiva/build/bin/clustering: error while loading shared libraries: libmkl_core.so: cannot open shared object file: No such file or directory
CMake Error at /usr/share/cmake-3.10/Modules/GoogleTestAddTests.cmake:39 (message):
  Error running test executable.

    Path: '/home/skoeb/Downloads/khiva/build/bin/clustering'
    Result: 127
    Output:
      



test/CMakeFiles/clustering.dir/build.make:98: recipe for target 'bin/clustering' failed
make[2]: *** [bin/clustering] Error 1
make[2]: *** Deleting file 'bin/clustering'
CMakeFiles/Makefile2:371: recipe for target 'test/CMakeFiles/clustering.dir/all' failed
make[1]: *** [test/CMakeFiles/clustering.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/home/skoeb/Downloads/khiva/build/bin/library: error while loading shared libraries: libmkl_core.so: cannot open shared object file: No such file or directory
CMake Error at /usr/share/cmake-3.10/Modules/GoogleTestAddTests.cmake:39 (message):
  Error running test executable.

    Path: '/home/skoeb/Downloads/khiva/build/bin/library'
    Result: 127
    Output:
      



test/CMakeFiles/library.dir/build.make:98: recipe for target 'bin/library' failed
make[2]: *** [bin/library] Error 1
make[2]: *** Deleting file 'bin/library'
CMakeFiles/Makefile2:297: recipe for target 'test/CMakeFiles/library.dir/all' failed
make[1]: *** [test/CMakeFiles/library.dir/all] Error 2
[ 27%] Linking CXX executable ../bin/dimensionality
/home/skoeb/Downloads/khiva/build/bin/dimensionality: error while loading shared libraries: libmkl_core.so: cannot open shared object file: No such file or directory
CMake Error at /usr/share/cmake-3.10/Modules/GoogleTestAddTests.cmake:39 (message):
  Error running test executable.

    Path: '/home/skoeb/Downloads/khiva/build/bin/dimensionality'
    Result: 127
    Output:
      



test/CMakeFiles/dimensionality.dir/build.make:98: recipe for target 'bin/dimensionality' failed
make[2]: *** [bin/dimensionality] Error 1
make[2]: *** Deleting file 'bin/dimensionality'
CMakeFiles/Makefile2:334: recipe for target 'test/CMakeFiles/dimensionality.dir/all' failed
make[1]: *** [test/CMakeFiles/dimensionality.dir/all] Error 2
[ 28%] Linking CXX executable ../bin/features
/home/skoeb/Downloads/khiva/build/bin/features: error while loading shared libraries: libmkl_core.so: cannot open shared object file: No such file or directory
CMake Error at /usr/share/cmake-3.10/Modules/GoogleTestAddTests.cmake:39 (message):
  Error running test executable.

    Path: '/home/skoeb/Downloads/khiva/build/bin/features'
    Result: 127
    Output:
      



test/CMakeFiles/features.dir/build.make:98: recipe for target 'bin/features' failed
make[2]: *** [bin/features] Error 1
make[2]: *** Deleting file 'bin/features'
CMakeFiles/Makefile2:260: recipe for target 'test/CMakeFiles/features.dir/all' failed
make[1]: *** [test/CMakeFiles/features.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2

Thank you for your quick response and help, I'm looking forward to using your library!

[skoeb]

Also, when I try installing with the release, it appears to be failing silently:

skoeb@sam-server:~/Downloads$ sudo gdebi khiva-khiva_0.2.3-1_amd64.deb 
[sudo] password for skoeb: 
Reading package lists... Done
Building dependency tree        
Reading state information... Done
Reading state information... Done

An open source C++ library which focus on providing efficient algorithms to perform analytics over time series of data.
Do you want to install the software package? [y/N]:y
(Reading database ... 266440 files and directories currently installed.)
Preparing to unpack khiva-khiva_0.2.3-1_amd64.deb ...
Unpacking khiva-khiva (0.2.3) over (0.2.3) ...
Setting up khiva-khiva (0.2.3) ...
skoeb@sam-server:~/Downloads$

python is 2.7 (64 bit), same version pip. I've also tried installing within a python 3.6.8 (64 bit) conda environment.

[otorreno]

Also, when I try installing with the release, it appears to be failing silently:

skoeb@sam-server:~/Downloads$ sudo gdebi khiva-khiva_0.2.3-1_amd64.deb 
[sudo] password for skoeb: 
Reading package lists... Done
Building dependency tree        
Reading state information... Done
Reading state information... Done

An open source C++ library which focus on providing efficient algorithms to perform analytics over time series of data.
Do you want to install the software package? [y/N]:y
(Reading database ... 266440 files and directories currently installed.)
Preparing to unpack khiva-khiva_0.2.3-1_amd64.deb ...
Unpacking khiva-khiva (0.2.3) over (0.2.3) ...
Setting up khiva-khiva (0.2.3) ...
skoeb@sam-server:~/Downloads$

python is 2.7 (64 bit), same version pip. I've also tried installing within a python 3.6.8 (64 bit) conda environment.

It looks like it was installed without problems. Could you check if you have the .so files under /usr/local/lib (libkhiva.so, libkhiva_c.so and libkhiva_jni.so)?

If the files are there, and you are planning to use the Python binding, install it using pip and try following the first example available in read the docs.

[skoeb]

Thanks again for the quick response. The installation appeared to be working, but my anaconda environment path wasn't looking in /usr/local/lib. I updated this and installed the python wrapper using pip.

I am now getting a similar error to @mmar30 though.

When I try to run khivia_test.py:

import numpy as np
from khiva.matrix import *
stomp_result = stomp(Array(np.array([11, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11])),
                         Array(np.array([9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 9])),
                         3)
find_best_n_discords_result = find_best_n_discords(stomp_result[0],
                                                       stomp_result[1], 2)
a = find_best_n_discords_result[2].to_numpy()
print(a)

I am getting the following error:

(spoilage) skoeb@sam-server:~/Desktop$ python khiva_test.py 
terminate called after throwing an instance of 'af::exception'
  what():  ArrayFire Exception (Internal error:998):
In function cuda::Kernel cuda::buildKernel(int, const string&, const string&, const std::vector<std::basic_string<char> >&, bool)
In file src/backend/cuda/nvrtc/cache.cpp:
In function af::array af::join(int, const af::array&, const af::array&)
In file src/api/cpp/data.cpp:209
Aborted (core dumped)
(spoilage) skoeb@sam-server:~/Desktop$ 

[skoeb]

One thing to note: the opt/arrayfire/lib file was named opt/arrayfire/lib64 in my installation. I still copyed the files to /usr/local/lib as detailed by @avilchess above.

[skoeb]

Making progress on figuring this out... setting CPU backend fixes the test:

import numpy as np
from khiva.matrix import *
from khiva.features import *
from khiva.library import *
set_backend(KHIVABackend.KHIVA_BACKEND_CPU)
set_device(0)

stomp_result = stomp(Array(np.array([11.0, 10.0, 11.0, 10.0, 10.0, 9.0, 11.0])),
                         Array(np.array([9.0, 10.0, 9.0, 8.0, 9.0, 10.0, 12.0])),
                         3)
find_best_n_discords_result = find_best_n_discords(stomp_result[0],
                                                       stomp_result[1], 2, 3)
a = find_best_n_discords_result[2].to_numpy()
print(a)

Outputs [3 1 0]

Additionally, Khiva is seeing my GPU:

set_backend(KHIVABackend.KHIVA_BACKEND_CUDA)
set_device(0)
get_backend_info()

'ArrayFire v3.6.4 (CUDA, 64-bit Linux, build 1b8030c)\nPlatform: CUDA Toolkit 10.0, Driver: 418.43\n[0] GeForce GTX 1060 6GB, 6076 MB, CUDA Compute 6.1\n'

but running the first block of code with the GPU as the backend results in:

(spoilage) skoeb@sam-server:~/Desktop$ python khiva_test.py 
terminate called after throwing an instance of 'af::exception'
  what():  ArrayFire Exception (Internal error:998):
In function cuda::Kernel cuda::buildKernel(int, const string&, const string&, const std::vector<std::basic_string<char> >&, bool)
In file src/backend/cuda/nvrtc/cache.cpp:
In function af::array af::join(int, const af::array&, const af::array&)
In file src/api/cpp/data.cpp:209
Aborted (core dumped)

[otorreno]

It looks like khiva has been installed properly. The error you are facing right now seems to be related with the CUDA backend of ArrayFire. Which CUDA toolkit version do you have in your system? We experienced some issues with previous versions of ArrayFire and CUDA toolkit v10. We haven’t tested with ArrayFire v3.6.4 as of yet, but it could be a similar issue.

Another thing you can try is compiling ArrayFire, it might take some time, but it should solve the problem. Take a look to the following Dockerfile for instructions on how to compile it.

A different alternative would be using the khiva docker image, which is available in Docker hub. Use the CUDA one to obtain the best possible performance.

[otorreno]

@skoeb did you manage to make it work?

[skoeb]

Partially, still having trouble with GPU compatibility in some modules, but others (like DTW) are working.

[skoeb]

I have some questions about functionalities (things like PSI for DTW), but I'll open another issue for those. Thanks for your help!

[otorreno]

@skoeb The modules which do not work could be because of the ArrayFire-CUDA toolkit versión issue I mentioned before.

For any questions about functionality or further support in the install process please use our Gitter channel. There we can have a more fluid conversation.

We use GitHub issues for actual issues/bugs and feature requests.