Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix SIDD 3.0 validation error with J2K compression #741

Merged
merged 141 commits into from
Mar 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
69b3cc9
copy six.sidd to six.sidd30
Jan 30, 2023
2eb06d5
six.sidd30 project and unittests
Jan 30, 2023
c8de122
sidd -> sidd30
Jan 30, 2023
822b4ae
more sidd -> sidd30
Jan 30, 2023
0fa02ec
more sidd -> sidd30
Jan 30, 2023
0407911
begin removing SIDD 3.0 support from six.sidd; it is now (or will be)…
Jan 30, 2023
00bea9f
remove more SIDD 3.0 unittest code from six.sidd, it's not just in si…
Jan 30, 2023
757a806
remove last? traces of SIDD3.0 from six.sidd
Jan 30, 2023
a3da393
tweaks from feature/ISM
Jan 30, 2023
0e1c446
latest from coda-oss and nitro
Jan 30, 2023
41a3083
Squashed 'externals/coda-oss/' changes from 1582c6ac9..aabc5818e
Jan 30, 2023
35a73fd
Merge commit '41a30838e1d4dba6472d7211f3e76b671b48450d' into feature/…
Jan 30, 2023
54a36ce
Squashed 'externals/nitro/' changes from e1a64da1f..24ff32863
Jan 30, 2023
9bdb185
latest from coda-oss and nitro
Jan 30, 2023
b3df43d
latest from coda-oss and nitro
Jan 30, 2023
f6faa5b
use common code from six.sidd
Jan 30, 2023
1ea9562
consistent convertDerivedClassificationToXML() API
Jan 30, 2023
79091cb
use common code from six.sidd
Jan 30, 2023
9a0c703
compile SIDD30 Utilities.cpp
Jan 30, 2023
ba63e04
get six.sidd30 unittests working
Jan 30, 2023
1f4ba79
remove files duplicated between six.sidd and six.sidd30
Jan 30, 2023
06d8f23
build with CMake
Jan 31, 2023
5b68012
SIDD_V3.0.0 directory in "schema"
Feb 2, 2023
7446c7f
easy access to test XML files from Visual Studio
Feb 2, 2023
a687c3a
ValidatorXerces can be moved, but not copied
Feb 2, 2023
c37b15a
fix SIDD version bug when generating fake data
Feb 2, 2023
a579efa
set things up for different schema paths for SIDD 3.0 (not yet implem…
Feb 2, 2023
0ea02ca
more detailed output from schema validation failure
Feb 2, 2023
0dec03b
SIDD_V3.0.0
Feb 2, 2023
797d147
SIDD 3.0 schemas are in the SIDD_V3.0.0 directory
Feb 2, 2023
e5daf7f
no ISM-v201609 in this branch
Feb 2, 2023
4c76c9c
Merge branch 'main' into feature/SIDD-3.0
Apr 25, 2023
624f31f
IonoParameters/F2Height is optional
Apr 25, 2023
6b7cf7e
Merge branch 'feature/SIDD-3.0' of github.com:ngageoint/six-library i…
Apr 25, 2023
84ad444
Merge branch 'main' into feature/SIDD-3.0
Apr 25, 2023
47f9f69
remove six.sidd30
Apr 25, 2023
eaeb4e8
remove SIDD_V3.0.0
Apr 25, 2023
c06756b
slam in 'main'
Apr 25, 2023
84a4cf6
Merge branch 'main' into feature/SIDD-3.0
Apr 26, 2023
2d7838a
Merge branch 'main' into feature/SIDD-3.0
May 22, 2023
869666d
Merge branch 'main' into feature/SIDD-3.0
May 22, 2023
ebd7d8d
Merge branch 'feature/SIDD-3.0' of github.com:ngageoint/six-library i…
May 22, 2023
cc06d88
ISMVersion enum for 201609 and 13
May 22, 2023
d5e5577
need to be able to set the ISM version in more places
May 22, 2023
9725e04
start rippling changes for ISMVersion
May 22, 2023
e5fd009
add a "current" to make it easy for developers to switch between V201…
May 22, 2023
b439ab4
change the ISM namespace
May 22, 2023
1723ca3
restore SIDD-3.0_ISM-v13 to a separate directory
May 22, 2023
c710603
Create sidd300_ISM-v13.xml
May 22, 2023
9abfb68
wrap the xml::lite::validator in our own class
May 22, 2023
adfcf4d
pass a XmlLiteValidator instance
May 22, 2023
6367940
xml.lite changes from coda-oss
May 23, 2023
17bf734
despite various checks for a NULL logger, it really can't be NULL
May 23, 2023
401e703
use xml::lite::Validator directly
May 23, 2023
a3563df
move ISMVersion to the a sidd300 namespace to make it clear it's part…
May 23, 2023
eba3d9e
got a scoping issue somewhere :-(
May 23, 2023
f3eba0d
unittest for ISM-v13
May 23, 2023
d5bc547
hack to read ISM-v13 SIDD from unittest code; WIP
May 23, 2023
c357f82
don't need make_ utility, at least right now
May 24, 2023
9b4427d
log_errors_and_throw() utility
May 24, 2023
b5f88f6
prepare for validating a schema-path at a time
May 24, 2023
8245eef
latest xml.lite tweaks from coda-oss
May 24, 2023
54633a8
"new" code does XML validation one path at a time
May 24, 2023
bd431f7
adjust schema paths if ISMVersion is set
May 24, 2023
d2c26e9
automatically prepend the ISM schema path
May 24, 2023
4d74163
remove ISMVersion from DerivedXMLControl
May 24, 2023
8055430
allow the hard-coded "SIDD_V3.0.0_ISM-" name to be changed
May 24, 2023
6bb5937
first pass at finding the corresponding XSD for a XML
May 29, 2023
7f2de53
add a "hook" so that we can intercept validate() in SIDD code
May 29, 2023
18a57e5
provide standard validateXMLImpl() implementation
May 30, 2023
debc9d9
Revert "first pass at finding the corresponding XSD for a XML"
May 30, 2023
bd9238e
move schema finding code to six.sidd
May 30, 2023
aa81b50
new code only needs to work with std::vector<path>
May 30, 2023
7bdcec0
If we found the right XSD, use that directory first.
May 30, 2023
b08a864
Revert "allow the hard-coded "SIDD_V3.0.0_ISM-" name to be changed"
May 30, 2023
29d93c0
always validate one schema-path at a time
May 30, 2023
8c14122
try not to open so many XSDs
May 30, 2023
fb0c56c
getEnvIfSet()
May 30, 2023
e6ec603
look for SICommonTypes_V1.0.xsd to find the schema directory
May 30, 2023
b28f2d1
Merge branch 'main' into feature/SIDD-3.0
May 30, 2023
fd261c0
"hook" for validating created XML
May 30, 2023
6328464
find_SIDD_schema_V_files() utility
May 30, 2023
fd9704b
put the hard-code SIDD schema name in one place
May 30, 2023
37afd61
copy ISM-v201609 files to separate directory
May 30, 2023
0c5978b
There is now a separate directory for each SIDD 3.0 ISM
May 30, 2023
e5c785d
latest from coda-oss
May 30, 2023
e68140d
Merge commit '81533c2ebd8d743a2fd16116e0d365ea92862eb2' into feature/…
May 30, 2023
81533c2
Squashed 'externals/coda-oss/' changes from e87c32b4d..ff4f820ed
May 30, 2023
853b6f7
Squashed 'externals/nitro/' changes from c8ecbe9ae..2fd7a0bfa
May 30, 2023
f6c30df
copy_externals
May 30, 2023
9732f16
Merge branch 'main' into feature/SIDD-3.0
May 31, 2023
56b4497
toXMLImplValidate() isn't needed (at least not right now)
May 31, 2023
8b5a1bd
simplify validation code by converting to std::filesystem::path rathe…
May 31, 2023
1981611
No longer need ScoppedISMVersion to read ISM-v13 XML
May 31, 2023
842100e
comments to indicate the attributes for which we're looking
May 31, 2023
13831f0
be sure SIDD XML is really SIDD
May 31, 2023
b49336a
latest coda-oss from xml.lite branch
May 31, 2023
887b0f2
use std::vector::insert
May 31, 2023
432307c
create a SIDD Version enum rather than using strings
May 31, 2023
0eab307
expose ISMVersion in fromXML() and toXML()
May 31, 2023
9c0f159
don't need old ISM with SIDD 3.0
Jun 1, 2023
06c9115
Merge branch 'main' into feature/SIDD-3.0
Jun 8, 2023
3077749
use six::sidd::Version rather than a string for creating fake data
Jun 8, 2023
5d9d1e2
require ISMVersion for SIDD 3.0.0
Jun 8, 2023
4ebf05b
more use of six::sidd::Version
Jun 8, 2023
cc682ca
save ISMVersion for SIDD 3.0.0
Jun 8, 2023
70ac245
adjust desVersion for ISMVersion
Jun 8, 2023
c863746
use the ISMVersion from the environment, if any
Jun 8, 2023
f42bfa7
still more use of sidd::Version
Jun 8, 2023
7710dfd
still more use of sidd::Version instead of strVersion
Jun 8, 2023
d81dfdb
latest from CODA-OSS and NITRO
Jun 8, 2023
88fca03
Merge branch 'main' into feature/SIDD-3.0
Jun 8, 2023
50dbb92
latest from CODA-OSS
Jun 8, 2023
2d83223
Squashed 'externals/coda-oss/' changes from ff4f820ed..d1244a080
Jun 8, 2023
06ba41f
Merge commit '2d83223fc22d43f7d580b625ef78d2e04f42645c' into feature/…
Jun 8, 2023
2fdb1db
Squashed 'externals/nitro/' changes from 2fd7a0bfa..05dae18a3
Jun 8, 2023
091ca2a
sync_externals
Jun 8, 2023
5121522
latest from CODA-OSS
Jun 8, 2023
f9af2e7
Merge branch 'main' into feature/SIDD-3.0
Sep 12, 2023
2808fd4
Merge branch 'main' into feature/SIDD-3.0
Dec 6, 2023
ea57dca
latest from CODA-OSS
Dec 7, 2023
ba6cb11
xml.lite tweaks from CODA-OSS speed things up
Dec 7, 2023
a4dd0ce
allow schema paths to be found/validated separate from XML validation
Dec 7, 2023
aebe10f
XercesContext can't be "static" (yet)
Dec 7, 2023
2d25612
only initialize Xerces for multiple ValidatorXerces instances
Dec 7, 2023
ac17986
use sys::convertPaths() utility
Dec 7, 2023
76e19ce
remove unneeded ValidatorXerces() overloads
Dec 8, 2023
11de21b
don't expose findValidSchemaPaths() right now; no actual use-case
Dec 8, 2023
3560bad
account for xml.lite changes in CODA-OSS
Dec 8, 2023
f823a5f
latest from CODA-OSS
Dec 11, 2023
2b2de9f
Merge branch 'main' into feature/SIDD-3.0
Dec 11, 2023
4a4b43e
Merge branch 'main' into feature/SIDD-3.0
Dec 12, 2023
2a044c8
Merge branch 'main' into feature/SIDD-3.0
Mar 13, 2024
faa58ee
`positiveInteger` -> `nonNegativeInteger` in SIDD 3.0
Mar 13, 2024
68ce833
convertJ2KLayerInfoToXML() routine so the `index` attribute can be wr…
Mar 13, 2024
84307de
get rid of convertJ2KLayerInfoToXML(), it's just noise
Mar 13, 2024
0c3ec12
convertCompressionToXML() already existed, make it virtual
Mar 13, 2024
7ffac43
already had convertJ2KToXML(), make it `virtual`
Mar 13, 2024
71e9bdd
validate the value of the `index` attribute
Mar 13, 2024
7625be1
fix compiler warning
Mar 14, 2024
d01d9d6
update sample SIDD XML with <Compression>
Mar 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1442,7 +1442,7 @@
</xs:annotation>
</xs:element>
</xs:sequence>
<xs:attribute name="index" type="xs:positiveInteger" use="required"/>
<xs:attribute name="index" type="xs:nonNegativeInteger" use="required"/>
</xs:complexType>
<xs:complexType name="DigitalElevationDataType">
<xs:annotation>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1442,7 +1442,7 @@
</xs:annotation>
</xs:element>
</xs:sequence>
<xs:attribute name="index" type="xs:positiveInteger" use="required"/>
<xs:attribute name="index" type="xs:nonNegativeInteger" use="required"/>
</xs:complexType>
<xs:complexType name="DigitalElevationDataType">
<xs:annotation>
Expand Down
6 changes: 3 additions & 3 deletions six/modules/c++/six.sidd/include/six/sidd/Compression.h
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@ struct J2KCompression
{
Layer();

double bitRate;
double bitRate = 0.0;
};

J2KCompression();

size_t numWaveletLevels;
size_t numBands;
size_t numWaveletLevels = 0;
size_t numBands = 0;

std::vector<Layer> layerInfo;
};
Expand Down
6 changes: 2 additions & 4 deletions six/modules/c++/six.sidd/include/six/sidd/DerivedXMLParser.h
Original file line number Diff line number Diff line change
Expand Up @@ -194,10 +194,8 @@ struct DerivedXMLParser : public six::XMLParser
XMLElem parent = nullptr) const = 0;
XMLElem convertAnnotationToXML(const Annotation *a,
XMLElem parent = nullptr) const;
XMLElem convertCompressionToXML(const Compression *c,
XMLElem parent = nullptr) const;
void convertJ2KToXML(const J2KCompression* c, XMLElem& parent) const;
XMLElem convertSFAGeometryToXML(const SFAGeometry *g,
public: virtual void convertJ2KToXML(const J2KCompression&, xml::lite::Element&) const {}
protected: XMLElem convertSFAGeometryToXML(const SFAGeometry *g,
XMLElem parent = nullptr) const;
XMLElem convertGeographicCoordinateSystemToXML(
const SFAGeographicCoordinateSystem* geographicCoordinateSystem,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,6 @@ struct DerivedXMLParser200 final : public DerivedXMLParser
virtual void parseCompressionFromXML(const xml::lite::Element* compressionElem,
Compression& compression) const;

virtual XMLElem convertCompressionToXML(const Compression& compression,
XMLElem parent = nullptr) const;

virtual XMLElem convertDisplayToXML(const Display& display,
XMLElem parent = nullptr) const override;

Expand Down Expand Up @@ -141,8 +138,7 @@ struct DerivedXMLParser200 final : public DerivedXMLParser
static xml::lite::Element& convertFilterToXML(const DerivedXMLParser&,
const std::string& name, const Filter&, xml::lite::Element& parent);

static void convertJ2KToXML(const DerivedXMLParser&,
const J2KCompression&, xml::lite::Element& parent);
void convertJ2KToXML(const J2KCompression&, xml::lite::Element&) const override;

void parseJ2KCompression(const xml::lite::Element* j2kElem,
J2KCompression& j2k) const;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ struct DerivedXMLParser300 final : public DerivedXMLParser
void parseBankFromXML(const xml::lite::Element* bankElem, Filter::Bank&) const;
void parseFilterFromXML(const xml::lite::Element& filterELem, Filter& filter) const;
void parseJ2KCompression(const xml::lite::Element& j2kElem, J2KCompression& j2k) const;
void convertJ2KToXML(const J2KCompression&, xml::lite::Element&) const override;
void parseGeoDataFromXML(const xml::lite::Element& elem, GeoDataBase&) const;
void parseDigitalElevationDataFromXML(const xml::lite::Element& elem, DigitalElevationData&) const;
void parseProductGenerationOptionsFromXML(const xml::lite::Element& optionsElem, ProductGenerationOptions&) const;
Expand Down
7 changes: 2 additions & 5 deletions six/modules/c++/six.sidd/source/Compression.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,11 @@ namespace six
{
namespace sidd
{
J2KCompression::Layer::Layer() :
bitRate(0.0)
J2KCompression::Layer::Layer()
{
}

J2KCompression::J2KCompression() :
numWaveletLevels(0),
numBands(0)
J2KCompression::J2KCompression()
{
}
}
Expand Down
20 changes: 7 additions & 13 deletions six/modules/c++/six.sidd/source/DerivedXMLParser200.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ xml::lite::Document* DerivedXMLParser200::toXML(const DerivedData* derived) cons
// optional
if (derived->compression.get())
{
convertCompressionToXML(*derived->compression, root);
convertCompressionToXML(*this, *(derived->compression), *root);
}
// optional
if (derived->digitalElevationData.get())
Expand Down Expand Up @@ -1482,32 +1482,26 @@ xml::lite::Element& DerivedXMLParser200::convertFilterToXML(const DerivedXMLPars
return filterElem;
}

XMLElem DerivedXMLParser200::convertCompressionToXML(
const Compression& compression,
XMLElem parent) const
{
assert(parent != nullptr);
return &convertCompressionToXML(*this, compression, *parent);
}
xml::lite::Element& DerivedXMLParser200::convertCompressionToXML(const DerivedXMLParser& parser,
const Compression& compression, xml::lite::Element& parent)
{
auto& compressionElem = parser.newElement("Compression", parent);
auto& j2kElem = parser.newElement("J2K", compressionElem);
auto& originalElem = parser.newElement("Original", j2kElem);
convertJ2KToXML(parser, compression.original, originalElem);
parser.convertJ2KToXML(compression.original, originalElem);

if (compression.parsed.get())
{
auto& parsedElem = parser.newElement("Parsed", j2kElem);
convertJ2KToXML(parser, *compression.parsed, parsedElem);
parser.convertJ2KToXML(*(compression.parsed), parsedElem);
}
return compressionElem;
}

void DerivedXMLParser200::convertJ2KToXML(const DerivedXMLParser& parser,
const J2KCompression& j2k, xml::lite::Element& parent)
void DerivedXMLParser200::convertJ2KToXML(const J2KCompression& j2k, xml::lite::Element& parent) const
{
auto& parser = *this;

parser.createInt("NumWaveletLevels", j2k.numWaveletLevels, parent);
parser.createInt("NumBands", j2k.numBands, parent);

Expand All @@ -1518,7 +1512,7 @@ void DerivedXMLParser200::convertJ2KToXML(const DerivedXMLParser& parser,
for (size_t ii = 0; ii < numLayers; ++ii)
{
auto& layerElem = parser.newElement("Layer", layerInfoElem);
parser.setAttribute(layerElem, "index", ii + 1);
parser.setAttribute(layerElem, "index", ii + 1); // `positiveInteger` in SIDD 2.0; changed to `nonNegativeInteger` in SIDD 3.0
parser.createDouble("Bitrate", j2k.layerInfo[ii].bitRate, layerElem);
}
}
Expand Down
35 changes: 34 additions & 1 deletion six/modules/c++/six.sidd/source/DerivedXMLParser300.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@

#include <sstream>
#include <stdexcept>
#include <tuple>

#include <gsl/gsl.h>

Expand Down Expand Up @@ -222,7 +223,7 @@ xml::lite::Document* DerivedXMLParser300::toXML(const DerivedData* derived) cons
// optional
if (derived->compression.get())
{
DerivedXMLParser200::convertCompressionToXML(*this, *derived->compression, *root);
DerivedXMLParser200::convertCompressionToXML(*this, *(derived->compression), *root);
}
// optional
if (derived->digitalElevationData.get())
Expand Down Expand Up @@ -335,10 +336,42 @@ void DerivedXMLParser300::parseJ2KCompression(const xml::lite::Element& j2kElem,

for (size_t ii = 0; ii < layerElems.size(); ++ii)
{
// In SIDD 3.0, the `index` attribute type changed from `positiveInteger` to `nonNegativeInteger`
// (matching C-style indexing). Since we had a problem with this, use the opportunity to
// validate the `index` value.
const auto& attributes = layerElems[ii]->getAttributes();
std::string strIndex;
if (attributes.getValue("index", strIndex))
{
// The schema says this is required, but we might not be validating.
const size_t index = std::stoi(strIndex);
std::ignore = index; // compiler warning
assert(ii == index); // again, we might not be validating the XML
}

parseDouble(getFirstAndOnly(layerElems[ii], "Bitrate"), j2k.layerInfo[ii].bitRate);
}
}

void DerivedXMLParser300::convertJ2KToXML(const J2KCompression& j2k, xml::lite::Element& parent) const
{
auto& parser = *this;

parser.createInt("NumWaveletLevels", j2k.numWaveletLevels, parent);
parser.createInt("NumBands", j2k.numBands, parent);

const auto numLayers = j2k.layerInfo.size();
auto& layerInfoElem = parser.newElement("LayerInfo", parent);
parser.setAttribute(layerInfoElem, "numLayers", numLayers);

for (size_t ii = 0; ii < numLayers; ++ii)
{
auto& layerElem = parser.newElement("Layer", layerInfoElem);
parser.setAttribute(layerElem, "index", ii); // `ii + 1` in SIDD 2.0, `positiveInteger` is now `nonNegativeInteger`
parser.createDouble("Bitrate", j2k.layerInfo[ii].bitRate, layerElem);
}
}

void DerivedXMLParser300::parseDisplayFromXML(const xml::lite::Element& displayElem,
Display& display) const
{
Expand Down
25 changes: 25 additions & 0 deletions six/modules/c++/six.sidd/tests/sample_xml/sidd300.xml
Original file line number Diff line number Diff line change
Expand Up @@ -504,6 +504,31 @@
<si:Coef exponent1="0" exponent2="0">0</si:Coef>
</si:GammaZeroSFPoly>
</Radiometric>
<Compression>
<J2K>
<Original>
<NumWaveletLevels>1</NumWaveletLevels>
<NumBands>2</NumBands>
<LayerInfo numLayers="1">
<Layer index="0">
<Bitrate>3.14</Bitrate>
</Layer>
</LayerInfo>
</Original>
<Parsed>
<NumWaveletLevels>3</NumWaveletLevels>
<NumBands>4</NumBands>
<LayerInfo numLayers="2">
<Layer index="0">
<Bitrate>3.1415</Bitrate>
</Layer>
<Layer index="1">
<Bitrate>3.1415926</Bitrate>
</Layer>
</LayerInfo>
</Parsed>
</J2K>
</Compression>
<DigitalElevationData>
<GeographicCoordinates>
<LongitudeDensity>1</LongitudeDensity>
Expand Down
Loading