Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Establish a standardized catalog config specification and corresponding validation tests #448

Open
yymao opened this issue May 28, 2020 · 1 comment
Labels
configs Issues related to catalog config yaml files priority:medium register Issues related to the register module test Issues related to unit tests and integration tests

Comments

@yymao
Copy link
Member

yymao commented May 28, 2020

During the early development stage of GCRCatalogs, one of the guiding principles was to give as much flexibility as possible to the catalog providers, to lower the barrier for catalog providers who want to make their product accessible via GCRCatalogs.

With GCRCatalogs now being one of the main data access methods for a wide range of different data products in the DESC, the flexibility that we gave to catalog providers has gradually turned into maintenance costs. As of today there are 29 readers and more than 130 catalogs.

One step to reduce future maintenance cost is to make establish a standardized catalog config specification.

In fact, several recent updates are already moving toward that direction. In particular we have established a common root_dir for all the paths that appear in catalog configs, and have also established a few "reserved" config keywords. So we are indeed getting closer to having a more uniform, formally-defined catalog config structure.

There's still work, however. For example, we would need to homogenize different keywords used to define file/catalog paths. We may want to have a more structured content for deprecated key. We may want to homogenize the way people specify a subset of catalog files (for example, right now object catalogs use regular expression while cosmoDC2 uses a list of integers).

This issue supersedes #345.

@yymao yymao added register Issues related to the register module configs Issues related to catalog config yaml files priority:medium labels May 28, 2020
@yymao yymao changed the title Establish a standardized catalog config specification Establish a standardized catalog config specification and corresponding validation tests May 28, 2020
@yymao
Copy link
Member Author

yymao commented May 28, 2020

Equally important are the tests to validate catalog configs, ensuring that they do follow the specification.

There have been some discussion between @JoanneBogart and me in #446. We have now already established a few reserved keywords (and are documenting them inCONTRIBUTING.md, see #446), so in principle we can already add tests to validate catalog configs that include any of these reserved keywords.

That said, the design of a specification and the tests to validate it are of course highly related. So it would be good to think about these two aspects together when working on the design.

@yymao yymao added the test Issues related to unit tests and integration tests label May 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
configs Issues related to catalog config yaml files priority:medium register Issues related to the register module test Issues related to unit tests and integration tests
Projects
None yet
Development

No branches or pull requests

1 participant