Skip to content
This repository has been archived by the owner on Jun 30, 2023. It is now read-only.

Fetch nhdv2 attributes from ScienceBase #82

Merged
merged 9 commits into from
Feb 17, 2022
Merged

Fetch nhdv2 attributes from ScienceBase #82

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b2860d5
add sb items to retrieve to VarsOfInterest csv
lkoenig-usgs Feb 4, 2022
1313d4e
change monthly precip sb links to child items representing CAT,ACC,TOT
lkoenig-usgs Feb 4, 2022
ecb008e
harmonize shorthand names across sb_ids in VarsOfInterest table
lkoenig-usgs Feb 4, 2022
117a029
download nhdv2 attributes from ScienceBase
lkoenig-usgs Feb 7, 2022
a3d020e
remove duplicated vars from VarsOfInterest table
lkoenig-usgs Feb 8, 2022
887bd66
edit year and file name parsing in fetch_nhdv2_attributes_from_sb.R
lkoenig-usgs Feb 8, 2022
b8e6a13
add documentation to nhdv2 attr data download target
lkoenig-usgs Feb 14, 2022
76f4596
download VarsOfInterest using dynamic branching
lkoenig-usgs Feb 15, 2022
7065cf3
minor edit to parse file name in downloaded sb data
lkoenig-usgs Feb 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions 1_fetch.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ source("1_fetch/src/get_nhdplusv2.R")
source('1_fetch/src/download_tifs_annual.R')
source("1_fetch/src/get_gf.R")
source("1_fetch/src/fetch_sb_data.R")
source("1_fetch/src/fetch_nhdv2_attributes_from_sb.R")

p1_targets_list <- list(

Expand Down Expand Up @@ -178,7 +179,7 @@ p1_targets_list <- list(
format = 'file'
),

# Read in NLCD datasets and subet by comid in DRB
# Read in NLCD datasets and subset by comid in DRB
## Note that this returns a vector of dfs if more than one NLCD data is in the p1_NLCD_data_unzipped
tar_target(
p1_NLCD_data,
Expand Down Expand Up @@ -227,8 +228,20 @@ p1_targets_list <- list(
tar_target(
p1_vars_of_interest,
read_csv(p1_vars_of_interest_csv, show_col_types = FALSE) %>%
# Parse sb_id from sb link
mutate(sb_id = str_extract(Science.Base.Link,"[^/]*$")) %>%
# Remove the NADP from this since we are loading that separately and no not need it in vars of interest
filter(!Theme %in% c('Chemical', 'Land Cover'))
filter(!Theme %in% c('Chemical', 'Land Cover'))
),

# Download Wieczorek dataset variables of interest from ScienceBase
tar_target(
p1_vars_of_interest_downloaded_csvs,
p1_vars_of_interest %>%
split(.,.$sb_id) %>%
lekoenig marked this conversation as resolved.
Show resolved Hide resolved
lapply(.,fetch_nhdv2_attributes_from_sb,save_dir = "1_fetch/out",comids=p1_nhdv2reaches_sf$COMID,delete_local_copies=TRUE) %>%
Copy link
Collaborator

@msleckman msleckman Feb 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I always default to using function(x) in apply() because I think that it doesn't like it when there are multiple arguments in the given function. Nice to see it works!

do.call('c',.),
format = "file"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you describe what is happening here with a couple comments. In particular, why are you using split()? (see comment suggestion above).
what does do.call(., 'c') do ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, thanks! I've added comments to briefly explain what each line is doing within the p1_vars_of_interest_downloaded_csvs target. In particular, fetch_nhdv2_attributes_from_sb returns the file output path for each unique sb_id, so do.call('c',.) concatenates those strings into one vector of class chr.

),

# # download NADP data
Expand Down
Loading