Skip to content

Meeting Minutes 2019 11 12

Jump to bottom
Glen Rice edited this page Nov 12, 2019 · 1 revision

Attendees: Stuart MacGillivary, Jeff Adams, Giuseppe Masetti, Glen Rice

Meeting Notes:

  1. Discussion of overhead associated with HDF5 groups. Consensus is that this cannot be avoided so creating fewer groups may be better.

  2. The creation of fewer groups can be done by sorting the datasets by types (Elevation, Uncertainty, etc) rather than by tiles. Stuart started a script to do this but has yet not completed this effort.

  3. Giuseppe added georeferencing information as attributes to the tile groups from last week's approach. A conversation with a member of the GDAL team indicated this was a good way to go and that full CF conventions was probably not worth the effort.

  4. Investigation into compression indicates that a large number of groups is detrimental to efficient compression. This supports the potential rearrangement of datasets sorting into types rather than tiles.

  5. We agreed that moving the refinements to a different root would make sense for the overall structure. This is implemented in groups_by_super_cells_with_bag_tles_in_root. An interesting aspect of this structure is what to do with the XML metadata. Since we intend to put the georeferencing information into the root attributes, the metadata really contains just metadata, which is preferable. The library can export XML georeferencing information if needed, but this is not a good way to store this information. Information such as the type of uncertainty should be difference between the two nodes (tiles and single resolution) since the information may have been calculated differently.

  6. We think doubling of size might be worth the cost of a better structure and simpler code, but if we can make things slightly more complicated with only a slight size increase it would be more palatable.

Homework

A. Stuart is taking the lead on creating a sort by type structure based on the most recent script posted by Giuseppe. This will put the refinements on a separate root node.

B. Glen will find some large VR datasets.

C. We will look at the differences in compression between tiled grouping and type grouping.

D. We are targeting a final recommendation at next week's meeting.

Clone this wiki locally