Skip to content
This repository has been archived by the owner on Jun 30, 2023. It is now read-only.

Add target for a table of ScienceBase date last modified #98

Open
jds485 opened this issue Mar 1, 2022 · 0 comments
Open

Add target for a table of ScienceBase date last modified #98

jds485 opened this issue Mar 1, 2022 · 0 comments
Labels
documentation Improvements or additions to documentation pipeline issues related to targets pipeline or tar_meta

Comments

@jds485
Copy link
Member

jds485 commented Mar 1, 2022

There is a way to check the date that a ScienceBase item was last modified with ​item_get_fields(<sb_id>, 'provenance')​.

So, I think we can make a target for a table containing the provenance and other ScienceBase item information, and pass that table into the download target. In the download target, we can re-download only those items whose date changed.

Suggested steps:

  • Add a target that adds a provenance column to NHDVarsOfInterest.csv. This column could also be added within the p1_vars_of_interest target, but I think a new target before p1_vars_of_interest would be better because the provenance should always be built when the pipeline is run. Using cue = tar_cue(mode = "always') should work to do that check.
  • Modify fetch_nhdv2_attributes_from_sb to check the provenance column before downloading items from ScienceBase

@lekoenig, @msleckman - tagging for your awareness of this option. Does this sound good to you?

@lekoenig lekoenig added documentation Improvements or additions to documentation pipeline issues related to targets pipeline or tar_meta labels Mar 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
documentation Improvements or additions to documentation pipeline issues related to targets pipeline or tar_meta
Projects
None yet
Development

No branches or pull requests

2 participants