Error noaa_stage3() - forecast workshop on Intel chip

[jevanilla]

I'm getting the following error in the forecast_challenge_workshop_aquatics tutorial:

weather_past_s3 <- neon4cast::noaa_stage3()

Error in (function (anonymous, access_key, secret_key, session_token,  : 
  Cannot call fs___S3FileSystem__create(). See https://arrow.apache.org/docs/r/articles/install.html for help installing Arrow C++ libraries. 

I'm running on a 2020 MacBook Pro with a 2.3 GHz Quad-Core Intel Core i7 processor

> Sys.getenv()
__CF_USER_TEXT_ENCODING                  0x4A06D600:0x0:0x0
__CFBundleIdentifier                     com.rstudio.desktop
AWS_EC2_METADATA_DISABLED                TRUE
CLICOLOR_FORCE                           1
COMMAND_MODE                             unix2003
DISPLAY                                  :0
DYLD_FALLBACK_LIBRARY_PATH               /Library/Frameworks/R.framework/Resources/lib:/Library/Java/JavaVirtualMachines/jdk-11.0.18+10/Contents/Home/lib/server:/var/folders/s8/3dc07f2d57jg73jj_vs31jdn50dnh0/T/rstudio-fallback-library-path-1691195468
EDITOR                                   vi
GIT_ASKPASS                              rpostback-askpass
HOME                                     /Users/jevanilla
LANG                                     en_US.UTF-8
LC_CTYPE                                 en_US.UTF-8
LN_S                                     ln -s
LOGNAME                                  jevanilla
MAKE                                     make
MallocNanoZone                           0
MPLENGINE                                tkAgg
ORIGINAL_XDG_CURRENT_DESKTOP             undefined
PAGER                                    /usr/bin/less
PATH                                     /usr/local/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/jevanilla/Applications/quarto/bin:/Library/TeX/texbin:/usr/texbin:/Applications/RStudio.app/Contents/Resources/app/quarto/bin:/Applications/RStudio.app/Contents/Resources/app/bin/postback
PYTHONIOENCODING                         utf-8
R_ARCH                                   
R_BROWSER                                /usr/bin/open
R_BZIPCMD                                /usr/bin/bzip2
R_CLI_HAS_HYPERLINK_IDE_HELP             true
R_CLI_HAS_HYPERLINK_IDE_RUN              true
R_CLI_HAS_HYPERLINK_IDE_VIGNETTE         true
R_DOC_DIR                                /Library/Frameworks/R.framework/Resources/doc
R_GZIPCMD                                /usr/bin/gzip
R_HOME                                   /Library/Frameworks/R.framework/Resources
R_INCLUDE_DIR                            /Library/Frameworks/R.framework/Resources/include
R_LIBS_SITE                              /Library/Frameworks/R.framework/Resources/site-library
R_LIBS_USER                              /Users/jevanilla/Library/R/x86_64/4.3/library
R_PAPERSIZE                              a4
R_PDFVIEWER                              /usr/bin/open
R_PLATFORM                               x86_64-apple-darwin20
R_PRINTCMD                               lpr
R_QPDF                                   /Library/Frameworks/R.framework/Resources/bin/qpdf
R_RD4PDF                                 times,inconsolata,hyper
R_RUNTIME                                
R_SESSION_TMPDIR                         /var/folders/s8/3dc07f2d57jg73jj_vs31jdn50dnh0/T//RtmpDSZxUD
R_SHARE_DIR                              /Library/Frameworks/R.framework/Resources/share
R_STRIP_SHARED_LIB                       strip -x
R_STRIP_STATIC_LIB                       strip -S
R_SYSTEM_ABI                             macos,gcc,gxx,gfortran,gfortran
R_TEXI2DVICMD                            /opt/R/x86_64/bin/texi2dvi
R_UNZIPCMD                               /usr/bin/unzip
R_ZIPCMD                                 /usr/bin/zip
RETICULATE_PYTHON_FALLBACK               /usr/local/bin/python3
RMARKDOWN_MATHJAX_PATH                   /Applications/RStudio.app/Contents/Resources/app/resources/mathjax-27
RS_LOG_LEVEL                             WARN
RS_RPOSTBACK_PATH                        /Applications/RStudio.app/Contents/Resources/app/bin/rpostback
RS_SHARED_SECRET                         e6e0a0ee-7d30-418c-957b-fafd99040eec
RSTUDIO                                  1
RSTUDIO_CLI_HYPERLINKS                   true
RSTUDIO_CONSOLE_COLOR                    256
RSTUDIO_CONSOLE_WIDTH                    137
RSTUDIO_DESKTOP_EXE                      /Applications/RStudio.app/Contents/MacOS/RStudio
RSTUDIO_FALLBACK_LIBRARY_PATH            /var/folders/s8/3dc07f2d57jg73jj_vs31jdn50dnh0/T/rstudio-fallback-library-path-1691195468
RSTUDIO_PANDOC                           /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools
RSTUDIO_PROGRAM_MODE                     desktop
RSTUDIO_SESSION_PID                      28303
RSTUDIO_SESSION_PORT                     19543
RSTUDIO_USER_IDENTITY                    jevanilla
RSTUDIO_WINUTILS                         bin/winutils
SED                                      /usr/bin/sed
SHELL                                    /bin/zsh
SSH_ASKPASS                              rpostback-askpass
SSH_AUTH_SOCK                            /private/tmp/com.apple.launchd.sWnZ8hHNX9/Listeners
TAR                                      /usr/bin/tar
TERM                                     xterm-256color
TMPDIR                                   /var/folders/s8/3dc07f2d57jg73jj_vs31jdn50dnh0/T/
TZDIR                                    /var/db/timezone/zoneinfo
USER                                     jevanilla
XPC_FLAGS                                0x0
XPC_SERVICE_NAME                         application.com.rstudio.desktop.102038174.102038183

[jevanilla]

Is this related to #16 ?

[cboettig]

@jevanilla kind of related to #16, though that was on arm architecture (apple silicon) while you're on intel. but yes you just need the libraries.

Can you try:

 install.packages("arrow", type="binary")

then restart R and try again?

[jevanilla]

@cboettig I'm getting the same error after trying that.

[cboettig]

hmm.. what does arrow::arrow_with_s3() show?

[jevanilla]

> arrow::arrow_with_s3()
[1] FALSE

[cboettig]

@jevanilla okay thanks for bearing with me! Yes, apparently CRAN binaries are not being built with S3 right now, CRAN is giving the arrow team a hard time about external dependencies. The suggested work-around at present is to do:

install.packages('arrow', repos = 'https://apache.r-universe.dev')

instead. can you try this and let us know? Thanks so much!

[jevanilla]

Still no love for me using this install method...

[cboettig]

@jevanilla just to confirm, you do install.packages('arrow', repos = 'https://apache.r-universe.dev'), and you restart R, and then arrow::arrow_with_s3() still says FALSE?

[jevanilla]

That is correct. I just updated my version of R from 4.3.2 to the latest 4.4.1. I get a message when loading arrow to install using the method you describe, but that is in fact how I did it.

> library(arrow)
Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
The repository you retrieved Arrow from did not include all of Arrow's features.
You can install a fully-featured version by running:
`install.packages('arrow', repos = 'https://apache.r-universe.dev')`.

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

    timestamp
> arrow_info()
Arrow package version: 16.1.0

Capabilities:
               
acero      TRUE
dataset    TRUE
substrait FALSE
parquet    TRUE
json       TRUE
s3        FALSE
gcs       FALSE
utf8proc   TRUE
re2        TRUE
snappy     TRUE
gzip      FALSE
brotli    FALSE
zstd      FALSE
lz4        TRUE
lz4_frame  TRUE
lzo       FALSE
bz2       FALSE
jemalloc  FALSE
mimalloc   TRUE

Memory:
                  
Allocator mimalloc
Current    0 bytes
Max        0 bytes

Runtime:
                          
SIMD Level          avx512
Detected SIMD Level avx512

Build:
                                    
C++ Library Version           16.1.0
C++ Compiler              AppleClang
C++ Compiler Version 14.0.0.14000029

> arrow_with_s3()
[1] FALSE

[cboettig]

right, hmm. Just to confirm, you did also restart R after installing this way? (without restarting, R will not start using the newer version, and the message implies that it still loading the version of arrow installed from CRAN)

[jevanilla]

Yes, just tried it again to be double sure.

> remove.packages("arrow")
Removing package from ‘/Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library’
(as ‘lib’ is unspecified)
> install.packages('arrow', repos = 'https://apache.r-universe.dev')
trying URL 'https://apache.r-universe.dev/bin/macosx/big-sur-x86_64/contrib/4.4/arrow_16.1.0.tgz'
Content type 'application/x-gzip' length 13275496 bytes (12.7 MB)
==================================================
downloaded 12.7 MB


The downloaded binary packages are in
	/var/folders/s8/3dc07f2d57jg73jj_vs31jdn50dnh0/T//Rtmpedi58C/downloaded_packages

Restarting R session...

> arrow::arrow_with_s3()
[1] FALSE