Establish a standardized catalog config specification and corresponding validation tests #448
Labels
configs
Issues related to catalog config yaml files
priority:medium
register
Issues related to the register module
test
Issues related to unit tests and integration tests
During the early development stage of
GCRCatalogs
, one of the guiding principles was to give as much flexibility as possible to the catalog providers, to lower the barrier for catalog providers who want to make their product accessible viaGCRCatalogs
.With
GCRCatalogs
now being one of the main data access methods for a wide range of different data products in the DESC, the flexibility that we gave to catalog providers has gradually turned into maintenance costs. As of today there are 29 readers and more than 130 catalogs.One step to reduce future maintenance cost is to make establish a standardized catalog config specification.
In fact, several recent updates are already moving toward that direction. In particular we have established a common
root_dir
for all the paths that appear in catalog configs, and have also established a few "reserved" config keywords. So we are indeed getting closer to having a more uniform, formally-defined catalog config structure.There's still work, however. For example, we would need to homogenize different keywords used to define file/catalog paths. We may want to have a more structured content for
deprecated
key. We may want to homogenize the way people specify a subset of catalog files (for example, right now object catalogs use regular expression while cosmoDC2 uses a list of integers).This issue supersedes #345.
The text was updated successfully, but these errors were encountered: