Skip to content

Commit

Permalink
Point to the new PortalPredictions, portal-forecasts
Browse files Browse the repository at this point in the history
Portal-forecasts is a revised version of portalPredictions
Portal-forecasts contains forecasts as zipped files
  • Loading branch information
henrykironde committed Nov 17, 2023
1 parent a35a772 commit 582f1c7
Show file tree
Hide file tree
Showing 87 changed files with 106 additions and 10,604 deletions.
7 changes: 4 additions & 3 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: portalcasting
Title: Model and Forecast Portal Rodent Dynamics
Version: 0.60.1
Version: 0.60.2
Authors@R: c(
person(c("Juniper", "L."), "Simonis",
email = "[email protected]", role = c("aut", "cre"),
Expand All @@ -21,7 +21,7 @@ Authors@R: c(
comment = c(ORCID = "0000-0001-6728-7745")),
person(c("S. K.", "Morgan"), "Ernest", role = c("aut"),
comment = c(ORCID = "0000-0002-6026-8530")))
Description: Create a directory, populate it with data, build models, forecast rodent populations, and visualize the results. The user can spin-up a local lightweight sandbox or full-scale production environment from the same code underlying the continuously deployed Portal Predictions Project continuous integration system <https://github.com/weecology/portalpredictions> and website <https://portal.naturecast.org>.
Description: Create a directory, populate it with data, build models, forecast rodent populations, and visualize the results. The user can spin-up a local lightweight sandbox or full-scale production environment from the same code underlying the continuously deployed Portal Predictions Project continuous integration system <https://github.com/weecology/portal-forecasts> and website <https://portal.naturecast.org>.
URL: https://weecology.github.io/portalcasting/, https://github.com/weecology/portalcasting
BugReports: https://github.com/weecology/portalcasting/issues
Depends:
Expand All @@ -48,7 +48,8 @@ Imports:
shiny,
stats,
viridis,
yaml
yaml,
zip
Suggests:
pkgdown,
testthat
Expand Down
7 changes: 7 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

Version numbers follow [Semantic Versioning](https://semver.org/).

# [portalcasting 0.60.2](https://github.com/weecology/portalcasting/releases/tag/v0.60.2)
*11-16-2023*

## Repository moved
* Store forecasts by date using the zip format
* Move from portalPredictions to portal-forecasts due to size limitations.

# [portalcasting 0.60.1](https://github.com/weecology/portalcasting/releases/tag/v0.60.1)
*5-22-2023*

Expand Down
2 changes: 1 addition & 1 deletion R/directory.R
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,7 @@ update_directory_configuration <- function (main = ".") {
# fix this so it grabs the actual values when `latest`

settings$resources$PortalData_version <- settings$resources$PortalData$version
settings$resources$archive_version <- ifnull(settings$resources$portalPredictions$version, "")
settings$resources$archive_version <- ifnull(settings$resources$portal-forecasts$version, "")
settings$resources$climate_forecast_version <- settings$resources$climate_forecasts$version

if (settings$resources$PortalData_version == "latest") {
Expand Down
8 changes: 4 additions & 4 deletions R/download.R
Original file line number Diff line number Diff line change
Expand Up @@ -290,8 +290,8 @@ download_archive <- function(main = ".",
} else if (source == "github") {

url <- ifelse(version == "latest",
"https://api.github.com/repos/weecology/portalPredictions/releases/latest",
paste0("https://api.github.com/repos/weecology/portalPredictions/releases/tags/", version))
"https://api.github.com/repos/weecology/portal-forecasts/releases/latest",
paste0("https://api.github.com/repos/weecology/portal-forecasts/releases/tags/", version))

got <- GET(url = url)

Expand All @@ -308,8 +308,8 @@ download_archive <- function(main = ".",

}

temp <- file.path(tempdir(), "portalPredictions.zip")
final <- file.path(main, resources_sub, "portalPredictions")
temp <- file.path(tempdir(), "portal-forecasts")
final <- file.path(main, resources_sub, "portal-forecasts")
version_file <- file.path(final, "version.txt")

if (!force & file.exists(version_file)) {
Expand Down
4 changes: 2 additions & 2 deletions R/fill_dir.R
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,8 @@ fill_resources <- function (main = ".") {

download_archive(main = main,
resources_sub = settings$subdirectories$resources,
version = settings$resources$portalPredictions$version,
source = settings$resources$portalPredictions$source,
version = settings$resources$portal-forecasts$version,
source = settings$resources$portal-forecasts$source,
pause = settings$unzip_pause,
timeout = settings$download_timeout,
force = settings$force,
Expand Down
30 changes: 15 additions & 15 deletions R/settings.R
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@
#'
#' @param PortalData `list` of `source` and `version` elements of `character` values for the Portal Data download. Default values retrieve the latest data from github
#'
#' @param portalPredictions `list` of `source` and `version` elements of `character` values for the archive download. Default values point to github, but `verison = NULL` indicates no download.
#' @param portal-forecasts `list` of `source` and `version` elements of `character` values for the archive download. Default values point to github, but `verison = NULL` indicates no download.
#'
#' @param climate_forecasts `list` of `source`, `version`, and `data` elements of `character` values for the climate forecasts download. Default values retrieve the current day's forecast of min, mean, and max temperature and precipitation from the Northwest Knowledge Network's North American Multi-Model Ensemble (NMME) climate forecasts.
#'
Expand Down Expand Up @@ -133,7 +133,7 @@ directory_settings <- function (files = directory_files( ),
list(files = files,
subdirectories = subdirectories,
resources = resources,
repository = "portalPredictions",
repository = "portal-forecasts",
confidence_level = confidence_level,
nsamples = nsamples,
time = time,
Expand Down Expand Up @@ -222,11 +222,11 @@ directory_files <- function (directory_configuration = "directory_configuration.
#'
#' @export
#'
directory_subdirectories <- function (forecasts = "forecasts",
fits = "fits",
models = "models",
resources = "resources",
data = "data",
directory_subdirectories <- function (forecasts = "forecasts",
fits = "fits",
models = "models",
resources = "resources",
data = "data",
www = "www") {

list(forecasts = forecasts,
Expand All @@ -242,19 +242,19 @@ directory_subdirectories <- function (forecasts = "forecasts",
#'
#' @export
#'
directory_resources <- function (PortalData = list(source = "github",
directory_resources <- function (PortalData = list(source = "github",
version = "latest"),
portalPredictions = list(source = "github",
portal-forecasts = list(source = "github",
version = NULL),
climate_forecasts = list(source = "NMME",
version = as.character(Sys.Date()),
climate_forecasts = list(source = "NMME",
version = as.character(Sys.Date()),
data = list(mintemp = "tasmin",
meantemp = "tasmean",
maxtemp = "tasmax",
meantemp = "tasmean",
maxtemp = "tasmax",
precipitation = "pr"))) {

list(PortalData = PortalData,
portalPredictions = portalPredictions,
portal-forecasts = portal-forecasts,
climate_forecasts = climate_forecasts)

}
Expand All @@ -266,7 +266,7 @@ directory_resources <- function (PortalData = list(source = "github",
#'
production_settings <- function (download_timeout = max(getOption("timeout"), 600)) {

resources <- directory_resources(portalPredictions = list(source = "github",
resources <- directory_resources(portal-forecasts = list(source = "github",
version = "latest"))

directory_settings(resources = resources,
Expand Down
55 changes: 55 additions & 0 deletions R/zip_unzip_forecasts.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#' unZip and zip forecasts by forecast date
#'
#' @param type either zip or unzip
#'
#' @export
#'
zip_unzip <- function(type=NULL, forecast_path="forecasts/", date=NULL){
print("Preparing forecasts files")
proj_path <- forecast_path
forecasts_metadata = paste0(proj_path, "forecasts_metadata.csv")
metadata <- read.csv(forecasts_metadata)
unique_dates <- unique(metadata$forecast_date)
unique_dates = sort(unique_dates)

if (type=="zip"){
csv_file <- "_forecast_table.csv"
yaml_file <- "_metadata.yaml"
json_file <- "_model_forecast.json"
if (!is.null(date)) {
unique_dates = c(date)
print(paste0("Zipping forecasts files for ", date))
}
for (forecast_day in unique_dates){
id_date_files <- c()
zipfile <- paste0(proj_path, "forecast_id_", forecast_day, ".zip")
# Get all the values of that particular day in a data frame
newdata <- subset(metadata, forecast_date == forecast_day, select=c(forecast_id, forecast_date))
# for each forecast_id get 3 files
All_ids <- newdata$forecast_id
for (id in All_ids){
csv_file_path = paste0(proj_path, "forecast_id_", id, csv_file)
yaml_file_path = paste0(proj_path, "forecast_id_", id, yaml_file)
json_file_path = paste0(proj_path, "forecast_id_", id, json_file)
id_date_files <- c(id_date_files, csv_file_path, yaml_file_path, json_file_path)
}
# First remove old zip file if exists
unlink(zipfile)
# zip all id_date_files
zipr(zipfile, id_date_files, compression_level = 9)
unlink(id_date_files)
}
}

if (type=="unzip"){
print("Unzipping forecasts files")
# unzip files basing on unique_dates
for (forecast_day in unique_dates){
zipfile <- paste0(proj_path, "forecast_id_", forecast_day, ".zip")
if(file.exists(zipfile)){
unzip(zipfile, exdir = proj_path)
unlink(zipfile)
}
}
}
}
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Supporting [automated forecasting](https://github.com/weecology/portalPredictions) of [rodent populations](https://portal.weecology.org/)
# Supporting [automated forecasting](https://github.com/weecology/portal-forecasts) of [rodent populations](https://portal.weecology.org/)

<img src="man/figures/portalcasting.png" alt="hexagon software logo, light grey blue background, basic lettering at the top says portalcasting, main image is a drawn all black rodent standing on two feet with a fishing rod in hand and a brown fishing hat on head, standing next to a tan and green tackle box." width="200px" align="right">

Expand All @@ -7,7 +7,7 @@
[![Codecov test coverage](https://img.shields.io/codecov/c/github/weecology/portalcasting/main.svg)](https://app.codecov.io/github/weecology/portalcasting/branch/main)
[![Lifecycle:maturing](https://img.shields.io/badge/lifecycle-maturing-blue.svg)](https://lifecycle.r-lib.org/articles/stages.html)
[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
[![License](http://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/weecology/portalPredictions/master/LICENSE)
[![License](http://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/weecology/portal-forecasts/master/LICENSE)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3332973.svg)](https://doi.org/10.5281/zenodo.3332973)
[![NSF-1929730](https://img.shields.io/badge/NSF-1929730-blue.svg)](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1929730)
[![JOSS](https://joss.theoj.org/papers/10.21105/joss.03220/status.svg)](https://doi.org/10.21105/joss.03220)
Expand All @@ -17,24 +17,24 @@

The `portalcasting` package provides a model development, deployment, and evaluation system for forecasting how ecological systems change through time, with a focus on a widely used long-term study of mammal population and community dynamics, the [Portal Project](https://portal.weecology.org/).
It combines the [Portal Data Repository](https://github.com/weecology/PortalData) and [portalr data management package](https://github.com/weecology/portalr) in a pipeline to automate weekly forecasting.
Forecasts are archived on [GitHub](https://github.com/weecology/portalPredictions) and [Zenodo](https://doi.org/10.5281/zenodo.833438).
Forecasts are archived on [GitHub](https://github.com/weecology/portal-forecasts) and [Zenodo](https://doi.org/10.5281/zenodo.833438).
The [Portal Forecasting website](https://portal.naturecast.org/) provides a dynamic view of the results.

`portalcasting`'s functions are also portable, allowing users to set up a fully-functional replica repository on a local or remote machine.
This facilitates development and testing of new models
via a [sandbox](https://en.wikipedia.org/wiki/Sandbox_(software_development)) approach.
via a [sandbox](https://en.wikipedia.org/wiki/Sandbox_(software_development)) approach.

## Status: Deployed, Active Development

The `portalcasting` package is deployed for use within the [Portal Predictions repository](https://github.com/weecology/portalPredictions), providing the underlying R code to populate the directory with up-to-date data, analyze the data, produce new forecasts, generate new output figures, and render a new version of the [website](https://portal.naturecast.org/).
All of the code underlying the forecasting functionality has been migrated over from the [predictions repository](https://github.com/weecology/portalPredictions), which contains the code executed by the continuous integration.
Having relocated the code here, the `portalcasting` package is the location for active development of the model set and additional functionality.
The `portalcasting` package is deployed for use within the [Portal Predictions repository](https://github.com/weecology/portal-forecasts), providing the underlying R code to populate the directory with up-to-date data, analyze the data, produce new forecasts, generate new output figures, and render a new version of the [website](https://portal.naturecast.org/).
All of the code underlying the forecasting functionality has been migrated over from the [predictions repository](https://github.com/weecology/portal-forecasts), which contains the code executed by the continuous integration.
Having relocated the code here, the `portalcasting` package is the location for active development of the model set and additional functionality.

We leverage a [software container](https://en.wikipedia.org/wiki/Operating-system-level_virtualization) to enable reproducibility of the [predictions repository](https://github.com/weecology/portalPredictions).
We leverage a [software container](https://en.wikipedia.org/wiki/Operating-system-level_virtualization) to enable reproducibility of the [predictions repository](https://github.com/weecology/portal-forecasts).
Presently, we use a [Docker](https://hub.docker.com/r/weecology/portalcasting) image of the software environment to create a container for running the code.
The image is automatically rebuilt when there is a new `portalcasting` release, tagged with both the `latest` and version-specific (`vX.X.X`) tags, and pushed to [DockerHub](https://hub.docker.com/r/weecology/portalcasting).
The image is automatically rebuilt when there is a new `portalcasting` release, tagged with both the `latest` and version-specific (`vX.X.X`) tags, and pushed to [DockerHub](https://hub.docker.com/r/weecology/portalcasting).

Because the `latest` image is updated with releases, the current main branch code in `portalcasting` is typically, but not necessarily always, being executed within the [predictions repository](https://github.com/weecology/portalPredictions).
Because the `latest` image is updated with releases, the current main branch code in `portalcasting` is typically, but not necessarily always, being executed within the [predictions repository](https://github.com/weecology/portalPredictions).

The API is moderately well defined at this point, but is still evolving.

Expand Down
Loading

0 comments on commit 582f1c7

Please sign in to comment.