Error R_Reprotect: only x protected items, can't reprotect index y

[nbenn]

This might be related to #41 but as the error is different, I'm reporting this as a separate issue. In fact, this was the problem I came across first but when trying to boil everything down to a reproducible example, I ran into what is reported in #41. What seems to trigger this more severe issue is simply a larger problem size when compared to what is reported in #41.

set.seed(11)

nrow <- 3000000
ncol <- 1000

mat <- bigmemory::big.matrix(nrow, ncol, type = "double", shared = TRUE)

for (i in seq_len(ncol)) {
 mat[, i] <- sample(c(0, 1), nrow, replace = TRUE)
}

res <- biglasso::cv.biglasso(
 mat, sample(c(TRUE, FALSE), nrow, replace = TRUE), family = "binomial",
 ncores = 8, nlambda = 50, verbose = TRUE
)

yielding

Preprocessing start: 2021-09-21 13:23:31.000
Preprocessing end: 2021-09-21 13:23:41.000

-----------------------------------------------
Lambda 1. Now time: 2021-09-21 13:23:42.000
...
Lambda 49. Now time: 2021-09-21 13:51:23.000
Error in structure(new.time - time, class = "proc_time") : 
 R_Reprotect: only -69 protected items, can't reprotect index -71
Calls: <Anonymous> -> biglasso -> system.time -> structure
Error during wrapup: R_Reprotect: only -53 protected items, can't reprotect index -54

and my sessionInfo()

R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /cluster/spack/apps/linux-centos7-x86_64/gcc-8.2.0/openblas-0.3.7-aapr6wps44sbmhbedmmigxjk5bhfy5p3/lib/libopenblas_haswell-r0.3.7.so

locale:
[1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
[3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
[5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
[7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
[9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] bigmemory.sri_0.1.3 bigmemory_4.5.36    compiler_4.0.2     
[4] Matrix_1.2-18       biglasso_1.4.1      parallel_4.0.2     
[7] Rcpp_1.0.7          grid_4.0.2          ncvreg_3.13.0      
[10] lattice_0.20-41    

Happy to share additional info and also to do some further probing of the issue if someone has any pointers.

[privefl]

Do you get the same issue when using a filebacked BM?

[nbenn]

@privefl, sorry for the delay on my side.

It appears as though the problem goes away when using a filebacked BM: If I instantiate my matrix as

mat <- bigmemory::filebacked.big.matrix(
  nrow, ncol, type = "double", backingfile = "test.mat",
  descriptorfile = "test.desc", backingpath = tempdir(),
  binarydescriptor = TRUE
)

the above runs through.

[privefl]

I always try to avoid using non-filebacked shared big.matrix objects as I also ran with some troubles using them.
Also try to avoid using the temporary directory to store large files.