Replies: 1 comment
-
GDAL maintainer speaking here. So is my understanding of the above proposal correct, that it would be ultimately stored in the GDAL_METADATA TIFF tag, and retrieved through GDALDataset::GetMetadataItem("Table_SunPosition", "USGS") ? For consistency with other general practice, I would rather suggest to use GetMetadata("json:USGS")[0] (or maybe "json:ISIS" ?) and capture all objects as top-level JSON members of that single json:USGS / json:ISIS metadata domain. |
Beta Was this translation helpful? Give feedback.
-
Terms
Summary
ISIS will be changing how libisis I/O handles images and move to leveraging GDAL as its primary engine. This is predominately to enable processing GeoTiffs/COGs which requires ISIS supported metadata to read, write and operate on GeoTiffs from ingestion. We have been successful in reading GeoTiffs using gdal. Here, we propose a structure for ISIS metadata when working with images that are not ISIS cubes.
Motivation
ISIS LTP, NASA SAT findings, and, by extension, the community has identified that a priority for ISIS is to have improved interoperability with other software and enhance our software capability to access data via the cloud, especially in supporting operations in HPC environments. Better interoperability demands we use community-supported standards wherever possible and establish metadata standards for operations novel to our software. We decided to emphasize support for COGs by using GDAL for the underlying I/O operations for DNs and labels. This is expected to allow users to stream images (including ISIS's 2.5D shape models) over the web using GDAL's VSI interface.
ISIS expects formatted label data to support any operation post-ingestion. These data do not match how GDAL works if you use
gdal_translateto convert from ISIS Cube to Geotiff or any other format. Beyond some inconsistencies in translation (we have found numbers can completely change on translation), the Cube is exploded into multiple files with a detached label, one for the label, one for each ISIS table, another for DNs. This is because GDAL does not have a metadata format compatible with both ISIS and other GDAL formats without using detached labels. For this reason, we are proposing a single-file metadata format that works with GDAL API. We expect to contribute these changes to GDAL's ISIS3 Cube driver to support it throughgdal_translatewhen converting ISIS3 Cubes to COGs.We expect users to benefit by:
Proposed Solution / Explanation
The proposed solution would allow ISIS to read and write GeoTiffs using GDAL, maintaining Cube reading and writting. To write GTiffs in ISIS, users can set the format of the output file using the same facilities that they would set the format of a Cube. By adding a "+GTiff" to the output image in any ISIS program, the output image produced from the program will be a GeoTiff with all data, and metadata necessary to work with ISIS. Bellow is an example of the format translation:
The process above, does a one-to-one translation from the ISIS Cube to a GeoTiff. The Cube PVL header is placed onto the GeoTiffs metadata as JSON. The binary blobs that would be placed on the ISIS Cube also have PVL headers that are converted to JSON and placed into the GeoTiff metadata.
Since GeoTiffs cannot store binary data like Cubes, the binary blobs that would be placed into the Cubes are converted into a format that can be written to the GeoTiff. Bellow is what the ISIS PVL blob metadata looks like vs the converted data that will be written to the GeoTiff:
VS.
{"Table_SunPosition": '{ "_container_name":"Table", "_type":"object", "Field_J2000X":{ "_container_name":"Field", "_type":"group", "Name":"J2000X", "Type":"Double", "Size":"1" }, "Field_J2000Y":{ "_container_name":"Field", "_type":"group", "Name":"J2000Y", "Type":"Double", "Size":"1" }, "Field_J2000Z":{ "_container_name":"Field", "_type":"group", "Name":"J2000Z", "Type":"Double", "Size":"1" }, "Field_J2000XV":{ "_container_name":"Field", "_type":"group", "Name":"J2000XV", "Type":"Double", "Size":"1" }, "Field_J2000YV":{ "_container_name":"Field", "_type":"group", "Name":"J2000YV", "Type":"Double", "Size":"1" }, "Field_J2000ZV":{ "_container_name":"Field", "_type":"group", "Name":"J2000ZV", "Type":"Double", "Size":"1" }, "Field_ET":{ "_container_name":"Field", "_type":"group", "Name":"ET", "Type":"Double", "Size":"1" }, "Name":"SunPosition", "StartByte":"1", "Bytes":"112", "Records":"2", "ByteOrder":"Lsb", "CacheType":"Linear", "SpkTableStartTime":"297088762.24158", "SpkTableEndTime":"297088808.37074", "SpkTableOriginalSize":"2.0", "Description":"Created by spiceinit", "Kernels":[ "$base/kernels/spk/de430.bsp", "$base/kernels/spk/mar097.bsp" ], "Data":"4cffffffd401ffffffe62effffffd3ffffffa8ffffffc11d03ffffffffffffff8525495dffffffc157fffffffb03ffffff89ffffff800c404114ffffffd1ffffffa7ffffffb6ffffffecffffffe7ffffffcaffffffbfffffffc6fffffffa48ffffffd7ffffffe1ffffffe637ffffffc0246fffffff93ffffffae39ffffffea25ffffffc074ffffffd83dfffffffa36ffffffb5ffffffb141fffffff6ffffffc264fffffff92effffffd3ffffffa8ffffffc1503cffffffb32a394a5dffffffc1fffffff5fffffff746ffffffceffffff830b4041ffffffc6066d1b4fffffffe3ffffffcaffffffbf6c4f1dffffff80ffffffe1ffffffe637ffffffc03fffffffb246ffffffde39ffffffea25ffffffc0ffffff8fffffffe85e2837ffffffb5ffffffb141" }' }The PVL metadata for each blob is converted into JSON with the binary data placed at the "Data" key within the JSON. The JSON is written to the GeoTiff under the Domain "USGS", and Keyed based on the name and type of the blob. In the example above the blob is an ISIS "Table". Its type is set to "Table", and name set to "SunPosition" making its metadata key "Table_SunPosition".
Users should not see a change in any workflow that uses Cubes with all data attached to them. Both Cubes and GTiffs should process the same in ISIS and produce the same end results for any given pipeline.
The GTiff format will not support either "Detached" or "External" labels that Cubes support.
Drawbacks
The main drawback for this functionality is adding new bugs into the existing software. If reading and writing GeoTiffs in ISIS or the interoperability between the GeoTiff format and other libraries is not important enough to the project, then these changes should not be considered. It is specifically for the interoperability between ISIS and other software libraries that we believe the risk is worth it.
Once the software is done, we plan to get it in users hands for testing before release.
Alternatives
There are a few alternatives to using GDAL and reading/writing GeoTiffs. There are plenty of other file formats that can manage image data (jpeg, png, gif, etc.), and a metadata sidecar file could we used to manage the data. This approach is bulky and requires users to manage both the image file and the side car file as they process data. Mainly, when moving data around directories, making sure that the image file and the metadata sidecar are collocated. Since ISIS blobs/metadata can be placed directly into the GeoTiff metadata, GeoTiffs do not have the issue that the alternative presents.
There are other Geospatial image formats that we could use as well like Cloud Raster Format, that support writing custom metadata. All of these formats offer the same suite of functionality that GeoTiffs offer.
While alternatives to GeoTiffs exist, we believe that they are the best format to support in ISIS. GeoTiffs are widely used by the planetary science community, are an open source format, and the format is not bogged down by sidecar files. Finally, GeoTiffs support all of the necessary functionality to be compatible with ISIS and support Geospatial image process in a wide varierty of other applications.
Unresolved Questions
Custom metadata will be written to ISIS produced GeoTiffs, so other libraries that want to take advantage of reading/writing that information will need to update. This poses a question to libraries like GDAL for proper metadata translation between file formats. The metadata that ISIS will write also differs from the metadata written by
gdal_translate(a gdal application that can translate between file formats). We will need to be in touch with the GDAL team and, ideally, change GDAL to write the same metadata that ISIS will be writing.Future Possibilities
Beta Was this translation helpful? Give feedback.
All reactions