Package installation of 'vctrs' failed due to undefined symbol STDVEC_DATAPTR

[hsselman]

I am building a docker image based on Ubuntu 20.04, Java11 and GraalVM+FastR 20.3.0. In this docker image I am trying to install all needed R packages for our goal. I am installing package dplyr and all it's dependencies. One of it's dependencies is vctrs. Because fastR 20.3.0 is based on R 3.6.1 I am using vctrs_0.3.6.tar.gz (just downloaded from CRAN to load from source, so installing this version from CRAN will give the same result I suppose).

I am using LLVM to build this package:

$some_location/bin/R -e "fastr.setToolchain('llvm'); install.packages('$PKG', repos='$RpackagesRepo', type = 'source'); if (require('$PKG')) { q(status=0); } else { q(status=1); }"

where $PKG is the name of the package and $RpackagesRepo is my local miniCRAN repo.

I am getting the following error while installing the R package vctrs:

** testing if installed package can be loaded from temporary location
Error in polyglot evaluation : /opt/graalvm-builder/target/graalvm-ce-java11-20.3.0/languages/R/library/00LOCK-vctrs/00new/vctrs/libs/vctrs.so: undefined symbol: STDVEC_DATAPTR

I am no trying to install it again using toolchain in native. I'll post the result here when it is finished.

Stack trace for the install of this package:

* installing *source* package ?vctrs? ...
** package ?vctrs? successfully unpacked and MD5 sums checked
** using staged installation
** libs
... a lot of lines that are not needed for you guys...
installing to /opt/graalvm-builder/target/graalvm-ce-java11-20.3.0/languages/R/library/00LOCK-vctrs/00new/vctrs/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error in polyglot evaluation : /opt/graalvm-builder/target/graalvm-ce-java11-20.3.0/languages/R/library/00LOCK-vctrs/00new/vctrs/libs/vctrs.so: undefined symbol: STDVEC_DATAPTR
ERROR: loading failed
* removing ?/opt/graalvm-builder/target/graalvm-ce-java11-20.3.0/languages/R/library/vctrs?

[hsselman]

Same holds for toolchain set to native

[hsselman]

Also posted the issue to vctrs and this was his reply: r-lib/vctrs#1310 (comment)

I think fastr needs to implement support for ALTREP.

[lionel-]

If needed, I'm happy to condition ALTREP support in vctrs based on a preprocessor define. Currently we have:

#if (R_VERSION < R_Version(3, 5, 0)) ||                         \
    (defined(_WIN32) && R_VERSION == R_Version(3, 5, 0))
# define HAS_ALTREP 0
#else
# define HAS_ALTREP 1
#endif

[steve-s]

Hello,

we've added preliminary support for ALTREP in 20.3.0, but it seems we missed this function. It should be simple to add it, but unfortunately the fix won't make it into the next GraalVM release (21.0).

Putting aside the fact that this is indeed something we should fix in FastR. Do you have any specific reason you want to install vctrs 0.3.6?

What we do in FastR is that the default CRAN URL actually points to MRAN snapshot, currently it's https://mran.microsoft.com/snapshot/2019-11-03. This allows us to guarantee some degree of compatibility with the packages -- it protects us from changes that are out of our control. So simple install.packages("dplyr") should work fine (we do test it), but it will use https://cran.microsoft.com/snapshot/2019-11-03/web/packages/dplyr/index.html.

You can override this by passing actual CRAN URL to install.packages, e.g., install.packages("dplyr", repos="CRAN.r-project.org"), but in such case you are entering uncharted waters :-) You can also change the default URL in etc/DEFAULT_CRAN_MIRROR. It is possible that you may be able to mix newer version of some package with other packages from the snapshot, but that's again uncharted waters only slightly less uncharted.

We intend to update the snapshot date from time to time, but there is no fixed schedule for that.

[steve-s]

@lionel- thanks for looking into this. The condition should be fine for FastR as well, we'll only need to implement this one function.

(In FastR it will be simple alias to DATAPTR. My understanding of it is that it skips the ALTREP(x) check and branch if you know that the vector is definitely not ALTREP, but that does not apply to FastR. We have our own transparent internal representations that we need to check for anyway)

[hsselman]

Hi @steve-s

Thanks for your anwser! I was installing vctrs 0.3.6 as a dependency of dplyr 1.0.2. Because fastR 20.3.0 is based on R 3.6.1, we updated all R packages we needed that say they are compatible with R 3.6.1 (using the miniCRAN package on CRAN this is very straightforward to do).

I must say it is a surprise for me that fastr is testing against that snapshot. Since fastr says to be based on R 3.6.1 I was assuming that most common R packages that are compatible to R 3.6.1 would also be supported in fastr. I know that you guys have been working hard to support as much of R packages as possible and that it is a race to keep up with the community. For that I am very greatful. Also I'd like to state again the love for graalvm and fastr, and especially your quick responses on our questions and bugs. I just wish I would have read something about your package compatibility strategy somewhere in the documentation.

Anyway I was hoping to start using dplyr>1.0 after our upgrade to 20.3.0. We currently chose our update strategy for GraalVM: we upgrade one mayor with every x.3.0 and then use the patched versions x.3.1, x.3.2,..

Thanks again. I hope that the snapshot will be updated in the future. Maybe an idea to look at the strategy again. As I would recommend to test against all R 3.6.1 compatible packages when communicating to be based on R 3.6.1.

[steve-s]

FastR 20.3.0 is as much compatible with R 3.6.1 as we could make it. Any R 3.6.1 compatible package "should" work on FastR, but there are compatibility issues in FastR (known in our backlog, and also yet unknown) and in the packages (wrong use of the R API). At the moment, we do not have the resources to continuously check and maintain compatibility with all (or even subset of) the R 3.6.1 compatible packages, unfortunately.

The use of CRAN snapshot is mentioned in the compatibility documentation and also in GitHub readme file.

[hsselman]

Forgive my naive attitude. I can image the immense work that would take. See it as an expression of surprise. I did not fully understand the amount of work that would go in building a tool like fastr.

Ah found it in the documentation. I have read it, but did not notice there was a footnote.

Again, I'd like to express gratitude for all the effort that goes into maintaining and developing graalvm and fastr. Great job! And thanks again for all the patience to answer all questions and reports. Keep up the good work.

If I come across any other errors I will help you guys improve this product by reporting it. Stay safe and healthy!

[steve-s]

This specific issue is fixed in master and will be in 21.1.0. However, latest dplyr does not install yet on FastR. We plan to look into that as well eventually.

[hsselman]

Thanks @steve-s ! Just curious, will 20.3.x contain this fix as well?

And thanks for considering to look into dplyr on fastr in the future! Would be great!

[steve-s]

Happy to help :-) As experimental GraalVM component we are not back-porting fixes, everything goes only to the next release. This fix unfortunately did not make it to 21.0.0 in time, so it will be in 21.1.0.