`renv::restore()` looking in repo archive for current package

[asadow]

As per #209, I encountered renv trying to install the current version of Matrix from the archive.

minimal reprex

This happens with or without the suggested fix of adding repos = c(CRAN = 'https://cloud.r-project.org') to renv::restore().

 => ERROR [ 9/12] RUN R -q -e "options(warn=2); renv::restore(repos = c(CRAN = 'https://cloud.r-project.org'))"                                                                                 20.6s
------                                                                                                                                                                                                
 > [ 9/12] RUN R -q -e "options(warn=2); renv::restore(repos = c(CRAN = 'https://cloud.r-project.org'))":
0.634 > options(warn=2); renv::restore(repos = c(CRAN = 'https://cloud.r-project.org'))
0.996 The following package(s) will be updated:
0.996 
0.996 # CRAN -----------------------------------------------------------------------
0.996 - lattice        [0.22-5 -> 0.22-6]
0.996 - MASS           [7.3-60.0.1 -> 7.3-61]
0.996 - Matrix         [1.6-5 -> 1.7-1]
0.996 - nlme           [3.1-164 -> 3.1-166]
0.996 - renv           [repo: RSPM -> CRAN; ver: 1.0.11 -> 1.0.7]
0.996 - base64enc      [* -> 0.1-3]
0.996 - bslib          [* -> 0.7.0]
0.996 - cachem         [* -> 1.0.8]
0.996 - cli            [* -> 3.6.2]
0.996 - colorspace     [* -> 2.1-0]
0.996 - commonmark     [* -> 1.9.1]
0.996 - cpp11          [* -> 0.4.7]
0.996 - crayon         [* -> 1.5.3]
0.996 - crosstalk      [* -> 1.2.1]
0.996 - digest         [* -> 0.6.35]
0.996 - dplyr          [* -> 1.1.4]
0.996 - DT             [* -> 0.33]
0.996 - evaluate       [* -> 0.24.0]
0.996 - fansi          [* -> 1.0.6]
0.996 - farver         [* -> 2.1.2]
0.996 - fastmap        [* -> 1.1.1]
0.996 - fontawesome    [* -> 0.5.2]
0.996 - fs             [* -> 1.6.4]
0.996 - generics       [* -> 0.1.3]
0.996 - ggplot2        [* -> 3.5.1]
0.996 - glue           [* -> 1.7.0]
0.996 - gtable         [* -> 0.3.5]
0.996 - here           [* -> 1.0.1]
0.996 - highr          [* -> 0.11]
0.996 - htmltools      [* -> 0.5.8.1]
0.996 - htmlwidgets    [* -> 1.6.4]
0.996 - httpuv         [* -> 1.6.15]
0.996 - isoband        [* -> 0.2.7]
0.996 - jquerylib      [* -> 0.1.4]
0.996 - jsonlite       [* -> 1.8.8]
0.996 - knitr          [* -> 1.48]
0.996 - labeling       [* -> 0.4.3]
0.996 - later          [* -> 1.3.2]
0.996 - lazyeval       [* -> 0.2.2]
0.996 - lifecycle      [* -> 1.0.4]
0.996 - magrittr       [* -> 2.0.3]
0.996 - memoise        [* -> 2.0.1]
0.996 - mime           [* -> 0.12]
0.996 - munsell        [* -> 0.5.1]
0.996 - pillar         [* -> 1.9.0]
0.996 - pkgconfig      [* -> 2.0.3]
0.996 - promises       [* -> 1.3.0]
0.996 - purrr          [* -> 1.0.2]
0.996 - R6             [* -> 2.5.1]
0.996 - rappdirs       [* -> 0.3.3]
0.996 - RColorBrewer   [* -> 1.1-3]
0.996 - Rcpp           [* -> 1.0.12]
0.996 - rlang          [* -> 1.1.3]
0.996 - rmarkdown      [* -> 2.27]
0.996 - rprojroot      [* -> 2.0.4]
0.996 - sass           [* -> 0.4.9]
0.996 - scales         [* -> 1.3.0]
0.996 - shiny          [* -> 1.8.1.1]
0.996 - snakecase      [* -> 0.11.1]
0.996 - sourcetools    [* -> 0.1.7-1]
0.996 - stringi        [* -> 1.8.3]
0.996 - stringr        [* -> 1.5.1]
0.996 - tibble         [* -> 3.2.1]
0.996 - tidyr          [* -> 1.3.1]
0.996 - tidyselect     [* -> 1.2.1]
0.996 - tinytex        [* -> 0.52]
0.996 - utf8           [* -> 1.2.4]
0.996 - vctrs          [* -> 0.6.5]
0.996 - viridisLite    [* -> 0.4.2]
0.996 - withr          [* -> 3.0.0]
0.996 - xfun           [* -> 0.44]
0.996 - xtable         [* -> 1.8-4]
0.996 - yaml           [* -> 2.3.8]
0.996 
1.013 # Downloading packages -------------------------------------------------------
1.336 - Querying repositories for available source packages ... Done!
19.63 - Downloading Matrix from CRAN ...                ERROR [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/Matrix/Matrix_1.7-1.tar.gz']
20.49 Warning: failed to find source for 'Matrix 1.7-1' in package repositories
20.49 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/Matrix/Matrix_1.7-1.tar.gz' [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/Matrix/Matrix_1.7-1.tar.gz']
20.49 Error: failed to retrieve package 'Matrix@1.7-1'
20.50 Traceback (most recent calls last):
20.50 8: renv::restore(repos = c(CRAN = "https://cloud.r-project.org"))
20.50 7: renv_restore_run_actions(project, diff, current, lockfile, rebuild)
20.50 6: renv_retrieve_impl(packages)
20.50 5: handler(package, renv_retrieve_impl_one(package))
20.50 4: renv_retrieve_impl_one(package)
20.50 3: renv_retrieve_repos(record)
20.50 2: stopf("failed to retrieve package '%s'", remote)
20.50 1: stop(sprintf(fmt, ...), call. = call.)
20.50 Execution halted
------
Dockerfile:25
--------------------
  23 |     WORKDIR /build
  24 |     COPY snow/renv.lock /build/renv.lock
  25 | >>> RUN R -q -e "options(warn=2); renv::restore(repos = c(CRAN = 'https://cloud.r-project.org'))"
  26 |     
  27 |     # install R code
--------------------
ERROR: failed to solve: process "/bin/sh -c R -q -e \"options(warn=2); renv::restore(repos = c(CRAN = 'https://cloud.r-project.org'))\"" did not complete successfully: exit code: 1

[kevinushey]

I believe the problem here is that your Dockerfile is using R 4.3.3:

https://github.com/asadow/appSnowMinimal/blob/main/Dockerfile#L1

but Matrix 1.7-1 is only available for R (>= 4.4.0), as per the current description on CRAN. You can also see that with:

> packageDescription("Matrix", field = "Depends")
[1] "R (>= 4.4.0), methods"

[asadow]

I set R to 4.4.2 and the same thing happens with DT:

288.9 - Downloading bslib from CRAN ...                 ERROR [download from 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' failed]
349.2 - Downloading htmlwidgets from CRAN ...           ERROR [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/htmlwidgets/htmlwidgets_1.6.4.tar.gz']
350.1 - Downloading DT from CRAN ...                    ERROR [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/DT/DT_0.33.tar.gz']
350.8 Warning: failed to find source for 'digest 0.6.35' in package repositories
350.8 Warning: failed to find source for 'fastmap 1.1.1' in package repositories
350.8 Warning: failed to find source for 'rlang 1.1.3' in package repositories
350.8 Warning: failed to find source for 'jsonlite 1.8.8' in package repositories
350.8 Warning: failed to find source for 'knitr 1.48' in package repositories
350.8 Warning: failed to find source for 'evaluate 0.24.0' in package repositories
350.8 Warning: failed to find source for 'xfun 0.44' in package repositories
350.8 Warning: failed to find source for 'yaml 2.3.8' in package repositories
350.8 Warning: failed to find source for 'rmarkdown 2.27' in package repositories
350.8 Warning: failed to find source for 'bslib 0.7.0' in package repositories
350.8 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' [download from 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' failed]
350.8 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/htmlwidgets/htmlwidgets_1.6.4.tar.gz' [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/htmlwidgets/htmlwidgets_1.6.4.tar.gz']
350.8 Warning: failed to find source for 'jsonlite 1.8.8' in package repositories
350.8 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/DT/DT_0.33.tar.gz' [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/DT/DT_0.33.tar.gz']
350.8 Error: failed to retrieve package 'DT@0.33'

[kevinushey]

I tried to reproduce using an alternate version of your Dockerfile, and all of the packages installed successfully.

FROM openanalytics/r-ver:4.4.2

LABEL maintainer="Adam Sadowski <asadowsk@uoguelph.ca>"

RUN /rocker_scripts/setup_R.sh https://packagemanager.posit.co/cran/__linux__/jammy/latest
RUN echo "\noptions(shiny.port=3838, shiny.host='0.0.0.0')" >> /usr/local/lib/R/etc/Rprofile.site

# system libraries of general use
# find through pak::pkg_sysreqs("DT", sysreqs_platform = "ubuntu-22.04")
RUN apt-get update && apt-get install --no-install-recommends -y \
    pandoc \
    make \
    zlib1g-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR project
COPY renv.lock renv.lock

RUN R -q -e 'install.packages("renv")'
RUN R -q -e 'renv::init(bare = TRUE)'
RUN R -q -e 'renv::restore()'

[asadow]

Is the renv in your lockfile version 1.0.7?

I updated the repo with your Dockerfile and two things occurred:

1. I had to place renv::init(bare = TRUE) on the same line as the install, otherwise I get the error renv is not found. Is the install.packages() call doing something strange?

 => [ 7/10] RUN R -e 'install.packages("renv")'                                                                                                                                           77.3s
 => => # > install.packages("renv")                                                                                                                                                            
 => => # Installing package into ‘/usr/local/lib/R/site-library’                                                                                                                               
 => => # (as ‘lib’ is unspecified)                                                                                                                                                             
 => => # trying URL 'https://packagemanager.posit.co/cran/__linux__/jammy/latest/src/contrib/renv_1.0.11.tar.gz'                                                                               
 => => # Content type 'binary/octet-stream' length 2226439 bytes (2.1 MB)       

2. restore() again failed for DT. Interestingly, it downloads it successfully as seen below. But then it tries to download again from Archive?

 => ERROR [ 9/11] RUN R -q -e 'renv::restore()'                                                                                                                                          295.2s 
------                                                                                                                                                                                          
 > [ 9/11] RUN R -q -e 'renv::restore()':
1.014 - Project '/project' loaded. [renv 1.0.11]
1.023 - One or more packages recorded in the lockfile are not installed.
1.023 - Use `renv::status()` for more details.
1.064 > renv::restore()
1.215 The following package(s) will be updated:
1.215 
1.215 # CRAN -----------------------------------------------------------------------
1.215 - renv           [repo: RSPM -> CRAN; ver: 1.0.11 -> 1.0.7]
1.215 - base64enc      [* -> 0.1-3]
1.215 - bslib          [* -> 0.7.0]
1.215 - cachem         [* -> 1.0.8]
1.215 - cli            [* -> 3.6.2]
1.215 - colorspace     [* -> 2.1-0]
1.215 - commonmark     [* -> 1.9.1]
1.215 - crayon         [* -> 1.5.3]
1.215 - crosstalk      [* -> 1.2.1]
1.215 - digest         [* -> 0.6.35]
1.215 - dplyr          [* -> 1.1.4]
1.215 - DT             [* -> 0.33]
1.215 - evaluate       [* -> 0.24.0]
1.215 - fansi          [* -> 1.0.6]
1.215 - farver         [* -> 2.1.2]
1.215 - fastmap        [* -> 1.1.1]
1.215 - fontawesome    [* -> 0.5.2]
1.215 - fs             [* -> 1.6.4]
1.215 - generics       [* -> 0.1.3]
1.215 - ggplot2        [* -> 3.5.1]
1.215 - glue           [* -> 1.7.0]
1.215 - gtable         [* -> 0.3.5]
1.215 - here           [* -> 1.0.1]
1.215 - highr          [* -> 0.11]
1.215 - htmltools      [* -> 0.5.8.1]
1.215 - htmlwidgets    [* -> 1.6.4]
1.215 - httpuv         [* -> 1.6.15]
1.215 - isoband        [* -> 0.2.7]
1.215 - jquerylib      [* -> 0.1.4]
1.215 - jsonlite       [* -> 1.8.8]
1.215 - knitr          [* -> 1.48]
1.215 - labeling       [* -> 0.4.3]
1.215 - later          [* -> 1.3.2]
1.215 - lazyeval       [* -> 0.2.2]
1.215 - lifecycle      [* -> 1.0.4]
1.215 - magrittr       [* -> 2.0.3]
1.215 - memoise        [* -> 2.0.1]
1.215 - mime           [* -> 0.12]
1.215 - munsell        [* -> 0.5.1]
1.215 - pillar         [* -> 1.9.0]
1.215 - pkgconfig      [* -> 2.0.3]
1.215 - promises       [* -> 1.3.0]
1.215 - R6             [* -> 2.5.1]
1.215 - rappdirs       [* -> 0.3.3]
1.215 - RColorBrewer   [* -> 1.1-3]
1.215 - Rcpp           [* -> 1.0.12]
1.215 - rlang          [* -> 1.1.3]
1.215 - rmarkdown      [* -> 2.27]
1.215 - rprojroot      [* -> 2.0.4]
1.215 - sass           [* -> 0.4.9]
1.215 - scales         [* -> 1.3.0]
1.215 - shiny          [* -> 1.8.1.1]
1.215 - snakecase      [* -> 0.11.1]
1.215 - sourcetools    [* -> 0.1.7-1]
1.215 - stringi        [* -> 1.8.3]
1.215 - stringr        [* -> 1.5.1]
1.215 - tibble         [* -> 3.2.1]
1.215 - tidyselect     [* -> 1.2.1]
1.215 - tinytex        [* -> 0.52]
1.215 - utf8           [* -> 1.2.4]
1.215 - vctrs          [* -> 0.6.5]
1.215 - viridisLite    [* -> 0.4.2]
1.215 - withr          [* -> 3.0.0]
1.215 - xfun           [* -> 0.44]
1.215 - xtable         [* -> 1.8-4]
1.215 - yaml           [* -> 2.3.8]
1.215 
1.231 # Downloading packages -------------------------------------------------------
1.237 - Querying repositories for available source packages ... Done!
16.63 - Downloading renv from CRAN ...                OK [1.1 Mb in 29s]
45.99 - Downloading DT from CRAN ...                  OK [1.6 Mb in 25s]
71.19 - Downloading htmltools from CRAN ...           OK [132 Kb in 5.1s]
76.44 - Downloading base64enc from CRAN ...           OK [7.6 Kb in 0.93s]
77.48 - Downloading digest from CRAN ...              OK [225 Kb in 4.4s]
82.01 - Downloading fastmap from CRAN ...             OK [45.3 Kb in 0.93s]
83.04 - Downloading rlang from CRAN ...               OK [745.9 Kb in 19s]
102.5 - Downloading htmlwidgets from CRAN ...         OK [848.5 Kb in 22s]
124.6 - Downloading jsonlite from CRAN ...            OK [1 Mb in 18s]
142.9 - Downloading knitr from CRAN ...               OK [569.3 Kb in 16s]
159.5 - Downloading evaluate from CRAN ...            OK [28 Kb in 1.8s]
161.4 - Downloading highr from CRAN ...               OK [13.5 Kb in 1.0s]
162.7 - Downloading xfun from CRAN ...                OK [154.4 Kb in 6.5s]
169.4 - Downloading yaml from CRAN ...                OK [92.5 Kb in 13s]
182.9 - Downloading rmarkdown from CRAN ...           OK [2.1 Mb in 50s]
233.4 - Downloading bslib from CRAN ...                 ERROR [download from 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' failed]
293.6 - Downloading htmlwidgets from CRAN ...           ERROR [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/htmlwidgets/htmlwidgets_1.6.4.tar.gz']
294.4 - Downloading DT from CRAN ...                    ERROR [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/DT/DT_0.33.tar.gz']
295.1 Warning: failed to find source for 'digest 0.6.35' in package repositories
295.1 Warning: failed to find source for 'fastmap 1.1.1' in package repositories
295.1 Warning: failed to find source for 'rlang 1.1.3' in package repositories
295.1 Warning: failed to find source for 'jsonlite 1.8.8' in package repositories
295.1 Warning: failed to find source for 'knitr 1.48' in package repositories
295.1 Warning: failed to find source for 'evaluate 0.24.0' in package repositories
295.1 Warning: failed to find source for 'xfun 0.44' in package repositories
295.1 Warning: failed to find source for 'yaml 2.3.8' in package repositories
295.1 Warning: failed to find source for 'rmarkdown 2.27' in package repositories
295.1 Warning: failed to find source for 'bslib 0.7.0' in package repositories
295.1 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' [download from 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' failed]
295.1 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/htmlwidgets/htmlwidgets_1.6.4.tar.gz' [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/htmlwidgets/htmlwidgets_1.6.4.tar.gz']
295.1 Warning: failed to find source for 'jsonlite 1.8.8' in package repositories
295.1 Warning: error downloading 'https://cloud.r-project.org/src/contrib/Archive/DT/DT_0.33.tar.gz' [cannot open URL 'https://cloud.r-project.org/src/contrib/Archive/DT/DT_0.33.tar.gz']
295.1 Error: failed to retrieve package 'DT@0.33'
295.1 Traceback (most recent calls last):
295.1 8: renv::restore()
295.1 7: renv_restore_run_actions(project, diff, current, lockfile, rebuild)
295.1 6: renv_retrieve_impl(packages)
295.1 5: handler(package, renv_retrieve_impl_one(package))
295.1 4: renv_retrieve_impl_one(package)
295.1 3: renv_retrieve_repos(record)
295.1 2: stopf("failed to retrieve package '%s'", remote)
295.1 1: stop(sprintf(fmt, ...), call. = call.)
295.1 Execution halted
------
Dockerfile:22
--------------------
  20 |     RUN R -q -e 'install.packages("renv");renv::init(bare = TRUE)'
  21 |     #RUN R -q -e 'renv::init(bare = TRUE)'
  22 | >>> RUN R -q -e 'renv::restore()'
  23 |     
  24 |     # copy R code
--------------------
ERROR: failed to solve: process "/bin/sh -c R -q -e 'renv::restore()'" did not complete successfully: exit code: 1

[kevinushey]

I don't really understand how it could be succeeding for me but failing for you; aren't we using the same Dockerfile? Are you working on a machine with a spotty internet connection or something like that?

[kevinushey]

233.4 - Downloading bslib from CRAN ... ERROR [download from 'https://cloud.r-project.org/src/contrib/Archive/bslib/bslib_0.7.0.tar.gz' failed]

Note that this URL does exist; I think internet connectivity or some other issue prevented the download here.

[asadow]

I noticed that about bslib as well. Yet the call fails at the DT error.

I don't believe it is spotty internet as I repeatedly get the error at the DT download. I'll try to build the image locally as I can only assume this is a server issue.

[asadow]

I can confirm that the image builds successfully on my desktop, but fails on the server, using the same Dockerfile.

uname -a

Linux pr-posit 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[asadow]

I fixed this by removing

RUN /rocker_scripts/setup_R.sh https://packagemanager.posit.co/cran/__linux__/jammy/latest

and instead of CRAN = 'https://cloud.r-project.org' I used CRAN = 'https://cran.rstudio.com'

[benlake]

While I don't want to refresh old issues needlessly, I actually just experienced this same "internet connection" double-take with the MASS and Matrix packages at older versions. If I attempted install from https://cloud.r-project.org, which I saw being used by renv 1.1.4 (only because of the failure message, not because the normal output indicated it (kind sad about that)), installation would fail. If I specified https://cran.r-project.org, they both installed.

The thing is, when I navigated to https://cloud.r-project.org/src/contrib/Archive/MASS/ to check what versions were available (a URL presented to me by the original failure message), it 404'd. When I navigated to https://cran.r-project.org/src/contrib/Archive/MASS/ it 200'd and I saw the packages. Now here's where the gaslighting began! I'm guessing about 10 minutes later after I messed with changing the projects default repo list (to avoid the 404), the original cloud.r-project URL starting returning the same response as cran.r-project! Now I can install from either just fine.

I think the original 404 request to cloud.r-project triggered a mirror synchronization; which I could understand for archive directories. That's just a theory, but it explains why we all might have been squinting and questioning our sanity.