Update PPM package mapping database for R 4.4

[nickybell]

Description

When trying to install historical binaries using PPM on macOS using R 4.4, packages are installed from source rather than binaries. I believe this is because the database mapping package versions to snapshot dates has not been updated for R 4.4.

database <- renv:::renv_p3m_database_load()
key <- renv:::renv_p3m_database_key("macosx/big-sur-arm64", "4.4")
is.null(database[[key]])
#> [1] TRUE
key <- renv:::renv_p3m_database_key("macosx/big-sur-arm64", "4.3")
is.null(database[[key]])
#> [1] FALSE

Reprex

Based on #1699, using duckplyr as an example and attempting to install version 0.2.0 as a binary from the 2023-09-13 snapshot on PPM. (h/t @rszymanski)

Using R 4.4.1

withr::local_options(list(renv.settings.use.cache = FALSE)) # don't use cache to show how package is installed
    project_directory <- withr::local_tempdir()
    
    withr::with_dir(
        new = project_directory,
        code = {
            renv::init(restart = FALSE)
            
            message("Installing [email protected] from https://packagemanager.posit.co/cran/latest")
            # installs from source
            renv::install(
                "[email protected]", 
                prompt = FALSE
            ) 
            
            message("Installing [email protected] from P3M snapshot")
            # installs binary
            renv::install(
                "[email protected]", 
                prompt = FALSE,
                repos = c(
                    "CRAN" = "https://packagemanager.posit.co/cran/2023-09-13"
                )
            )
        }
    )

Output

#> The following package(s) will be updated in the lockfile:
#> 
#> # CRAN -----------------------------------------------------------------------
#> - renv   [* -> 1.0.7]
#> 
#> The version of R recorded in the lockfile will be updated:
#> - R      [* -> 4.4.1]
#> 
#> - Lockfile written to "/private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpWDsToA/fileb365693320c5/renv.lock".
#> Installing [email protected] from https://packagemanager.posit.co/cran/latest
#> # Downloading packages -------------------------------------------------------
#> - Downloading duckplyr from Repository ...      OK [216.8 Kb in 0.28s]
#> - Downloading cli from CRAN ...                 OK [file is up to date]
#> - Downloading collections from CRAN ...         OK [64.5 Kb in 0.23s]
#> - Downloading DBI from CRAN ...                 OK [905.4 Kb in 0.43s]
#> - Downloading dplyr from CRAN ...               OK [1.4 Mb in 0.56s]
#> - Downloading generics from CRAN ...            OK [81.7 Kb in 0.23s]
#> - Downloading glue from CRAN ...                OK [file is up to date]
#> - Downloading lifecycle from CRAN ...           OK [file is up to date]
#> - Downloading rlang from CRAN ...               OK [file is up to date]
#> - Downloading magrittr from CRAN ...            OK [219.9 Kb in 0.27s]
#> - Downloading pillar from CRAN ...              OK [643.7 Kb in 0.38s]
#> - Downloading fansi from CRAN ...               OK [306.3 Kb in 0.29s]
#> - Downloading utf8 from CRAN ...                OK [143.2 Kb in 0.24s]
#> - Downloading vctrs from CRAN ...               OK [1.3 Mb in 0.53s]
#> - Downloading R6 from CRAN ...                  OK [file is up to date]
#> - Downloading tibble from CRAN ...              OK [665.8 Kb in 0.37s]
#> - Downloading pkgconfig from CRAN ...           OK [18.3 Kb in 0.21s]
#> - Downloading tidyselect from CRAN ...          OK [221.5 Kb in 0.26s]
#> - Downloading withr from CRAN ...               OK [file is up to date]
#> - Downloading duckdb from CRAN ...              OK [10.4 Mb in 2.8s]
#> - Downloading purrr from CRAN ...               OK [490.2 Kb in 0.34s]
#> Successfully downloaded 21 packages in 13 seconds.
#> 
#> The following package(s) will be installed:
#> - cli         [3.6.3]
#> - collections [0.3.7]
#> - DBI         [1.2.3]
#> - dplyr       [1.1.4]
#> - duckdb      [1.1.0]
#> - duckplyr    [0.2.0]
#> - fansi       [1.0.6]
#> - generics    [0.1.3]
#> - glue        [1.8.0]
#> - lifecycle   [1.0.4]
#> - magrittr    [2.0.3]
#> - pillar      [1.9.0]
#> - pkgconfig   [2.0.3]
#> - purrr       [1.0.2]
#> - R6          [2.5.1]
#> - rlang       [1.1.4]
#> - tibble      [3.2.1]
#> - tidyselect  [1.2.1]
#> - utf8        [1.2.4]
#> - vctrs       [0.6.5]
#> - withr       [3.0.1]
#> These packages will be installed into "/private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpWDsToA/fileb365693320c5/renv/library/macos/R-4.4/aarch64-apple-darwin20".
#> 
#> # Installing packages --------------------------------------------------------
#> - Installing cli ...                            OK [installed binary in 0.42s]
#> - Installing collections ...                    OK [installed binary in 0.34s]
#> - Installing DBI ...                            OK [installed binary in 0.37s]
#> - Installing generics ...                       OK [installed binary in 0.33s]
#> - Installing glue ...                           OK [installed binary in 0.35s]
#> - Installing rlang ...                          OK [installed binary in 0.63s]
#> - Installing lifecycle ...                      OK [installed binary in 0.39s]
#> - Installing magrittr ...                       OK [installed binary in 0.75s]
#> - Installing fansi ...                          OK [installed binary in 0.33s]
#> - Installing utf8 ...                           OK [installed binary in 0.33s]
#> - Installing vctrs ...                          OK [installed binary in 0.48s]
#> - Installing pillar ...                         OK [installed binary in 0.69s]
#> - Installing R6 ...                             OK [installed binary in 0.94s]
#> - Installing pkgconfig ...                      OK [installed binary in 0.33s]
#> - Installing tibble ...                         OK [installed binary in 0.94s]
#> - Installing withr ...                          OK [installed binary in 0.8s]
#> - Installing tidyselect ...                     OK [installed binary in 0.91s]
#> - Installing dplyr ...                          OK [installed binary in 0.39s]
#> - Installing duckdb ...                         OK [installed binary in 1.2s]
#> - Installing purrr ...                          OK [installed binary in 0.85s]
#> - Installing duckplyr ...                       OK [built from source in 8.6s]
#> Successfully installed 21 packages in 21 seconds.
#> Installing [email protected] from P3M snapshot
#> # Downloading packages -------------------------------------------------------
#> - Downloading duckplyr from Repository ...      OK [372.9 Kb in 1.2s]
#> Successfully downloaded 1 package in 1.4 seconds.
#> 
#> The following package(s) will be installed:
#> - duckplyr [0.2.0]
#> These packages will be installed into "/private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpWDsToA/fileb365693320c5/renv/library/macos/R-4.4/aarch64-apple-darwin20".
#> 
#> # Installing packages --------------------------------------------------------
#> - Installing duckplyr ...                       OK [installed binary in 0.36s]
#> Successfully installed 1 package in 0.38 seconds.

^{Created on 2024-10-01 with reprex v2.1.1}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.1 (2024-06-14)
#>  os       macOS Sonoma 14.5
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/New_York
#>  date     2024-10-01
#>  pandoc   3.1.11 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package     * version date (UTC) lib source
#>  P cli           3.6.3   2024-06-21 [?] RSPM
#>  R digest        0.6.37  <NA>       [?] <NA>
#>  R evaluate      1.0.0   <NA>       [?] <NA>
#>  R fastmap       1.2.0   <NA>       [?] <NA>
#>  R fs            1.6.4   <NA>       [?] <NA>
#>  P glue          1.8.0   2024-09-30 [?] RSPM
#>  R htmltools     0.5.8.1 <NA>       [?] <NA>
#>  R knitr         1.48    <NA>       [?] <NA>
#>  P lifecycle     1.0.4   2023-11-07 [?] RSPM
#>  P renv          1.0.7   2024-04-11 [?] CRAN (R 4.4.1)
#>  R reprex        2.1.1   <NA>       [?] <NA>
#>  P rlang         1.1.4   2024-06-04 [?] RSPM
#>  R rmarkdown     2.28    <NA>       [?] <NA>
#>  R rstudioapi    0.16.0  <NA>       [?] <NA>
#>  R sessioninfo   1.2.2   <NA>       [?] <NA>
#>  P withr         3.0.1   2024-07-31 [?] RSPM
#>  R xfun          0.47    <NA>       [?] <NA>
#>  R yaml          2.3.10  <NA>       [?] <NA>
#> 
#>  [1] /private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpWDsToA/fileb365693320c5/renv/library/macos/R-4.4/aarch64-apple-darwin20
#>  [2] /Users/nickybell/Library/Caches/org.R-project.R/R/renv/sandbox/macos/R-4.4/aarch64-apple-darwin20/f7156815
#> 
#>  P ── Loaded and on-disk path mismatch.
#>  R ── Package was removed from disk.
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Using R 4.3.1

withr::local_options(list(renv.settings.use.cache = FALSE)) # don't use cache to show how package is installed
    project_directory <- withr::local_tempdir()
    
    withr::with_dir(
        new = project_directory,
        code = {
            renv::init(restart = FALSE)
            
            message("Installing [email protected] from https://packagemanager.posit.co/cran/latest")
            # installs from source
            renv::install(
                "[email protected]", 
                prompt = FALSE
            )
        }
    )

Output

#> The following package(s) will be updated in the lockfile:
#> 
#> # RSPM -----------------------------------------------------------------------
#> - renv   [* -> 1.0.7]
#> 
#> The version of R recorded in the lockfile will be updated:
#> - R      [* -> 4.3.1]
#> 
#> - Lockfile written to "/private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpSXShFq/filed7ba529bb07c/renv.lock".
#> Installing [email protected] from https://packagemanager.posit.co/cran/latest
#> # Downloading packages -------------------------------------------------------
#> - Downloading duckplyr from P3M ...             OK [377.3 Kb in 1.3s]
#> - Downloading cli from CRAN ...                 OK [file is up to date]
#> - Downloading collections from CRAN ...         OK [63.3 Kb in 0.38s]
#> - Downloading DBI from CRAN ...                 OK [878.6 Kb in 0.43s]
#> - Downloading dplyr from CRAN ...               OK [1.4 Mb in 0.57s]
#> - Downloading generics from CRAN ...            OK [74.5 Kb in 0.22s]
#> - Downloading glue from CRAN ...                OK [file is up to date]
#> - Downloading lifecycle from CRAN ...           OK [file is up to date]
#> - Downloading rlang from CRAN ...               OK [file is up to date]
#> - Downloading magrittr from CRAN ...            OK [212.9 Kb in 0.25s]
#> - Downloading pillar from CRAN ...              OK [634.5 Kb in 0.37s]
#> - Downloading fansi from CRAN ...               OK [296 Kb in 0.28s]
#> - Downloading utf8 from CRAN ...                OK [140.5 Kb in 0.24s]
#> - Downloading vctrs from CRAN ...               OK [1.2 Mb in 0.54s]
#> - Downloading R6 from CRAN ...                  OK [file is up to date]
#> - Downloading tibble from CRAN ...              OK [658 Kb in 0.38s]
#> - Downloading pkgconfig from CRAN ...           OK [17.7 Kb in 0.21s]
#> - Downloading tidyselect from CRAN ...          OK [217 Kb in 0.27s]
#> - Downloading withr from CRAN ...               OK [file is up to date]
#> - Downloading duckdb from CRAN ...              OK [10.4 Mb in 3.0s]
#> - Downloading purrr from CRAN ...               OK [477.1 Kb in 0.33s]
#> Successfully downloaded 21 packages in 14 seconds.
#> 
#> The following package(s) will be installed:
#> - cli         [3.6.3]
#> - collections [0.3.7]
#> - DBI         [1.2.3]
#> - dplyr       [1.1.4]
#> - duckdb      [1.1.0]
#> - duckplyr    [0.2.0]
#> - fansi       [1.0.6]
#> - generics    [0.1.3]
#> - glue        [1.8.0]
#> - lifecycle   [1.0.4]
#> - magrittr    [2.0.3]
#> - pillar      [1.9.0]
#> - pkgconfig   [2.0.3]
#> - purrr       [1.0.2]
#> - R6          [2.5.1]
#> - rlang       [1.1.4]
#> - tibble      [3.2.1]
#> - tidyselect  [1.2.1]
#> - utf8        [1.2.4]
#> - vctrs       [0.6.5]
#> - withr       [3.0.1]
#> These packages will be installed into "/private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpSXShFq/filed7ba529bb07c/renv/library/R-4.3/aarch64-apple-darwin20".
#> 
#> # Installing packages --------------------------------------------------------
#> - Installing cli ...                            OK [installed binary in 0.44s]
#> - Installing collections ...                    OK [installed binary in 0.33s]
#> - Installing DBI ...                            OK [installed binary in 0.86s]
#> - Installing generics ...                       OK [installed binary in 0.34s]
#> - Installing glue ...                           OK [installed binary in 0.34s]
#> - Installing rlang ...                          OK [installed binary in 0.38s]
#> - Installing lifecycle ...                      OK [installed binary in 0.35s]
#> - Installing magrittr ...                       OK [installed binary in 0.35s]
#> - Installing fansi ...                          OK [installed binary in 0.35s]
#> - Installing utf8 ...                           OK [installed binary in 0.33s]
#> - Installing vctrs ...                          OK [installed binary in 0.38s]
#> - Installing pillar ...                         OK [installed binary in 0.38s]
#> - Installing R6 ...                             OK [installed binary in 0.34s]
#> - Installing pkgconfig ...                      OK [installed binary in 0.36s]
#> - Installing tibble ...                         OK [installed binary in 0.92s]
#> - Installing withr ...                          OK [installed binary in 0.35s]
#> - Installing tidyselect ...                     OK [installed binary in 0.53s]
#> - Installing dplyr ...                          OK [installed binary in 0.39s]
#> - Installing duckdb ...                         OK [installed binary in 0.93s]
#> - Installing purrr ...                          OK [installed binary in 0.59s]
#> - Installing duckplyr ...                       OK [installed binary in 0.77s]
#> Successfully installed 21 packages in 10 seconds.

^{Created on 2024-10-01 with reprex v2.1.1}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16)
#>  os       macOS Sonoma 14.5
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/New_York
#>  date     2024-10-01
#>  pandoc   3.1.11 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package     * version date (UTC) lib source
#>  P cli           3.6.3   2024-06-21 [?] RSPM
#>  R digest        0.6.37  <NA>       [?] <NA>
#>  R evaluate      1.0.0   <NA>       [?] <NA>
#>  R fastmap       1.2.0   <NA>       [?] <NA>
#>  R fs            1.6.4   <NA>       [?] <NA>
#>  P glue          1.8.0   2024-09-30 [?] RSPM
#>  R htmltools     0.5.8.1 <NA>       [?] <NA>
#>  R knitr         1.48    <NA>       [?] <NA>
#>  P lifecycle     1.0.4   2023-11-07 [?] RSPM
#>  P renv          1.0.7   2024-04-11 [?] RSPM (R 4.3.1)
#>  R reprex        2.1.1   <NA>       [?] <NA>
#>  P rlang         1.1.4   2024-06-04 [?] RSPM
#>  R rmarkdown     2.28    <NA>       [?] <NA>
#>  R rstudioapi    0.16.0  <NA>       [?] <NA>
#>  R sessioninfo   1.2.2   <NA>       [?] <NA>
#>  P withr         3.0.1   2024-07-31 [?] RSPM
#>  R xfun          0.47    <NA>       [?] <NA>
#>  R yaml          2.3.10  <NA>       [?] <NA>
#> 
#>  [1] /private/var/folders/xh/crvwt7596l3b9p1s66q3fr1r0000gp/T/RtmpSXShFq/filed7ba529bb07c/renv/library/R-4.3/aarch64-apple-darwin20
#>  [2] /Users/nickybell/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.3/aarch64-apple-darwin20/ac5c2659
#> 
#>  P ── Loaded and on-disk path mismatch.
#>  R ── Package was removed from disk.
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[kevinushey]

Thanks -- I've updated the PPM database, but it looks like we didn't get the older versions of duckplyr for R 4.4.x.

> db <- renv:::renv_p3m_database_load()
> pkgs <- ls(envir = db$`/bin/macosx/big-sur-arm64/contrib/4.4`)
> grep("duckplyr", pkgs, value = TRUE)
[1] "duckplyr 0.3.2" "duckplyr 0.4.0" "duckplyr 0.4.1"

renv only synchronizes available packages from the release date of R, so you'd only see package releases from 2024-04-24 for R 4.4.x.

[nickybell]

Thanks! I am curious, what's the reason to only make available the package releases from the version release date? To ensure compatibility?

[kevinushey]

Mainly because I didn't realize that PPM was actually producing binary packages for snapshot dates that are older than the associated release of R. 😅

We might be able to expand this, but it would take some extra thought.

[shearerpmm]

Keeping old PPM binaries working would be very helpful for OS X users who don't keep up with the latest versions, e.g. people with production apps who fix dependency versions for stability.

Installing lots of packages from source is an tricky business, an endless loop of looking up and trying to solve obscure command line errors on StackOverflow, and not practical for many ordinary R users, even those with considerable programming and computer skills.