Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ask DetectNodes to curate/organize input data files with different record lengths #18

Open
zarzycki opened this issue Jun 17, 2020 · 1 comment

Comments

@zarzycki
Copy link
Contributor

Many modeling groups "block" 3-D and 2-D variables differently.
Ex: 2-D PSL may have ntime=1460 but 3-D Z may have ntime = 360.

Currently, to track on PSL, files need to be "lined up" using NCO (ncrcat Z files together or split PSL using ncks) or other operators.

In an ideal world, one could pass a completely unordered list of files into TE. TE could then determine a "full time series" by ordering all files with the tracking var (i.e., PSL) and build a lookup table to match times in other files with different vars based on the required processing/outputs (ex: Z, U10). Perhaps this can be done on a master task when DetectNodes is first called but IANASE.

Probably logical addition: a quick check could also be implemented that either exits if a matching time isn't found for each required variable passed in or the "full time series" can be truncated to only include overlapping times between PSL, Z, U10. The former seems safer from a UI perspective.

@paullric
Copy link
Collaborator

Latest version of DetectNodes indexes on time, so as part of a single input you can include files where overlap between time indices is only partial. The first file specified on the line is used for master indexing.

I have to wonder how often it'll be the case that there is climate data that isn't organized using proper subsets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants