generated from NOAA-OWP/owp-open-source-project-template
-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GeoPackage input support #522
Merged
+1,986
−2
Merged
Changes from 61 commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
81ae5f1
initial geopackage support impl
program-- da0ff43
more work toward gpkg support; started wkb parser
program-- d5cc164
scaffold CMake for gpkg
program-- 484ee5a
implement (standard) wkb parser
program-- af82743
cleanup function declarations
program-- f251db2
add docs to wkb reader funcs
program-- 3945ae6
add generic geometry struct; refactor wkt funcs as visitor
program-- dbb1a47
clean up some WKB code
program-- 2278e0b
separate wkt visitor from wkb directly
program-- 2463e40
lots of refactoring -- streamlined wkb -> geojson types [no ci]
program-- d30f2b0
fixed wkb reading multi geometries and added more wkb tests
program-- 425128c
added sqlite dep and tests
program-- 8a87439
readd const modifier on member functions
program-- 5d899b6
separate GeoPackage impl from headers
program-- c92fd65
add geopackage tests
program-- 94229a8
move impl out to parent folder
program-- 8f54e95
reorg wkb impl; move wkb::copy_from -> utils::copy_from
program-- f5b6af9
separate sqlite impl from header
program-- 9fe5bae
finally got projections working, gpkg read success
program-- 8d3dc38
fix projection/bounding box issues; added test gpkg file; started on …
program-- 432a572
remove unnecessary headers
program-- 1ebfc13
improve documentation for gpkg and wkb
program-- 609a0a9
add explicit double initialization for copy_from; add SQLite cmake mo…
program-- 4545dfe
cleanup formatting/tests
program-- e287834
fix OOR error; better errors for geopackage::read
program-- 88885a6
allow read from gpkg or geojson when running ngen
program-- 6abf62a
added more docs; working on issue w/ projections for Release build type
program-- 9eee5d4
more docs to source files
program-- d2de692
add conditional SQLite support
program-- 8aab30a
add ifdef guard around GeoPackage.hpp include
program-- 8701bfe
fix ifdef -> if
program-- 71824c9
more ifdef -> if fixes
program-- 999c523
revert ifdef changes; fix adding compile defs for sqlite
program-- 97c6db4
disable optimizations for geometry.cpp if compiler is not intel
program-- aef9c56
rev: remove default construction for sqlite object
program-- ec3421d
rev: fix off-by-one in WKB reader; reorg WKB impl
program-- d114363
rev: remove extra return
program-- a78e9b7
rev: get_property takes in reference to string
program-- 79148ba
rev: add test for geopackage subsetting on read
program-- 4b82a22
rev: insert newlines to prevent git issues
program-- 64b3cfd
rev: rm sqlite3 cmake module in favor of built-in
program-- 0730d2f
cmake: remove test linking to NGen::core; make includes public
program-- fd548c9
rev: rename SQLite.hpp; move sqlite_error def
program-- 864a859
rev: move sqlite_t type alias into class
program-- 87618ca
rev: delete sqlite copy constructor/assignment
program-- e999531
rev: rm reinitialization in constructor
program-- fcd9d6d
rev: set sqlite move constructor/assignment to default
program-- 9dcecbf
rev: remove `stmt` from sqlite; shared -> unique
program-- 4e18ca2
rev: add comments regarding WKB implementation
program-- 9e44d42
rev: add geometry type guards in wkb reader
program-- f244bb4
rev: throw exception if invalid wkb type
program-- 74018aa
rev: rewrite `std::memcpy` call in `copy_from`
program-- 9ceed0c
rev: add newline to
program-- a562388
rev: handle FeatureCollection `shared_ptr` creation
program-- 19849fc
rev: remove statement output from sqlite_error calls
program-- 5698854
rev: move point bbox into switch
program-- 894e8ab
rev: use wkb_geom_t enum for switch
program-- 5419798
rev: scientific notation; cmake: update optimizer warning
program-- 5531a89
rev: change literal byte keys for clarity
program-- 8cbd253
rev: white-list check for layer names
program-- cf1d17f
rev: update cmakelists; feature exhaustive case
program-- bd30664
rev: update wkb test
program-- File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
#ifndef NGEN_GEOPACKAGE_H | ||
#define NGEN_GEOPACKAGE_H | ||
|
||
#include "FeatureCollection.hpp" | ||
#include "NGen_SQLite.hpp" | ||
|
||
namespace geopackage { | ||
|
||
/** | ||
* Build a geometry object from GeoPackage WKB. | ||
* | ||
* @param[in] row SQLite iterator at the row containing a geometry column | ||
* @param[in] geom_col Name of geometry column containing GPKG WKB | ||
* @param[out] bounding_box Bounding box of the geometry to output | ||
* @return geojson::geometry GPKG WKB converted and projected to a boost geometry model | ||
*/ | ||
geojson::geometry build_geometry( | ||
const sqlite_iter& row, | ||
const std::string& geom_col, | ||
std::vector<double>& bounding_box | ||
); | ||
|
||
/** | ||
* Build properties from GeoPackage table columns. | ||
* | ||
* @param[in] row SQLite iterator at the row containing the data columns | ||
* @param[in] geom_col Name of geometry column containing GPKG WKB to ignore | ||
* @return geojson::PropertyMap PropertyMap of properties from the given row | ||
*/ | ||
geojson::PropertyMap build_properties( | ||
const sqlite_iter& row, | ||
const std::string& geom_col | ||
); | ||
|
||
/** | ||
* Build a feature from a GPKG table row | ||
* | ||
* @param[in] row SQLite iterator at the row to build a feature from | ||
* @param[in] geom_col Name of geometry column containing GPKG WKB | ||
* @return geojson::Feature Feature containing geometry and properties from the given row | ||
*/ | ||
geojson::Feature build_feature( | ||
const sqlite_iter& row, | ||
const std::string& geom_col | ||
); | ||
|
||
/** | ||
* Build a feature collection from a GPKG layer | ||
* | ||
* @param[in] gpkg_path Path to GPKG file | ||
* @param[in] layer Layer name within GPKG file to create a collection from | ||
* @param[in] ids optional subset of feature IDs to capture (if empty, the entire layer is converted) | ||
* @return std::shared_ptr<geojson::FeatureCollection> | ||
*/ | ||
std::shared_ptr<geojson::FeatureCollection> read( | ||
const std::string& gpkg_path, | ||
const std::string& layer, | ||
const std::vector<std::string>& ids | ||
); | ||
|
||
} // namespace geopackage | ||
#endif // NGEN_GEOPACKAGE_H |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,230 @@ | ||
#ifndef NGEN_GEOPACKAGE_SQLITE_H | ||
#define NGEN_GEOPACKAGE_SQLITE_H | ||
|
||
#include <memory> | ||
#include <stdexcept> | ||
#include <vector> | ||
#include <string> | ||
|
||
#include <sqlite3.h> | ||
|
||
namespace geopackage { | ||
|
||
/** | ||
* Deleter used to provide smart pointer support for sqlite3 structs. | ||
*/ | ||
struct sqlite_deleter | ||
{ | ||
void operator()(sqlite3* db) { sqlite3_close_v2(db); } | ||
void operator()(sqlite3_stmt* stmt) { sqlite3_finalize(stmt); } | ||
}; | ||
|
||
/** | ||
* Smart pointer (shared) type for sqlite3 prepared statements | ||
*/ | ||
using stmt_t = std::unique_ptr<sqlite3_stmt, sqlite_deleter>; | ||
|
||
/** | ||
* SQLite3 row iterator | ||
* | ||
* Provides a simple iterator-like implementation | ||
* over rows of a SQLite3 query. | ||
*/ | ||
class sqlite_iter | ||
{ | ||
private: | ||
stmt_t stmt; | ||
int iteration_step = -1; | ||
bool iteration_finished = false; | ||
|
||
int column_count; | ||
std::vector<std::string> column_names; | ||
|
||
// vector of SQLITE data types, see: https://www.sqlite.org/datatype3.html | ||
std::vector<int> column_types; | ||
|
||
// returns the raw pointer to the sqlite statement | ||
sqlite3_stmt* ptr() const noexcept; | ||
|
||
// checks if int is out of range, and throws error if so | ||
void handle_get_index(int) const; | ||
|
||
public: | ||
sqlite_iter(stmt_t stmt); | ||
|
||
/** | ||
* Check if a row iterator is finished | ||
* | ||
* @return true if next() returned SQLITE_DONE | ||
* @return false if there is more rows available | ||
*/ | ||
bool done() const noexcept; | ||
|
||
/** | ||
* Step into the next row of a SQLite query | ||
* | ||
* If the query is finished, next() acts idempotently, | ||
* but will change done() to return true. | ||
* @return sqlite_iter& returns itself | ||
*/ | ||
sqlite_iter& next(); | ||
|
||
/** | ||
* Restart an iteration to its initial state. | ||
* next() must be called after calling this. | ||
* | ||
* @return sqlite_iter& returns itself | ||
*/ | ||
sqlite_iter& restart(); | ||
|
||
/** | ||
* Get the current row index for the iterator | ||
* | ||
* @return int the current row index, or -1 if next() hasn't been called | ||
*/ | ||
int current_row() const noexcept; | ||
|
||
/** | ||
* Get the number of columns within this iterator | ||
* @return int number of columns in query | ||
*/ | ||
int num_columns() const noexcept; | ||
|
||
/** | ||
* Return the column index for a named column | ||
* | ||
* @param name column name to search for | ||
* @return int index of given column name, or -1 if not found. | ||
*/ | ||
int column_index(const std::string& name) const noexcept; | ||
|
||
/** | ||
* Get a vector of column names | ||
* | ||
* @return const std::vector<std::string>& column names as a vector of strings | ||
*/ | ||
const std::vector<std::string>& columns() const noexcept { return this->column_names; } | ||
|
||
/** | ||
* Get a vector of column types | ||
* | ||
* See https://www.sqlite.org/datatype3.html for type values. The integers | ||
* are the affinity for data types. | ||
* @return const std::vector<int>& column types as a vector of ints | ||
*/ | ||
const std::vector<int>& types() const noexcept { return this->column_types; } | ||
|
||
/** | ||
* Get a column value from a row iterator by index | ||
* | ||
* @tparam T Type to parse value as, i.e. int | ||
* @param col Column index to parse | ||
* @return T value at column `col` | ||
*/ | ||
template<typename T> | ||
T get(int col) const; | ||
|
||
/** | ||
* Get a column value from a row iterator by name | ||
* | ||
* @tparam T Type to parse value as, i.e. int | ||
* @return T value at the named column | ||
*/ | ||
template<typename T> | ||
T get(const std::string&) const; | ||
}; | ||
|
||
template<typename T> | ||
inline T sqlite_iter::get(const std::string& name) const | ||
{ | ||
const int index = this->column_index(name); | ||
return this->get<T>(index); | ||
} | ||
|
||
/** | ||
* Wrapper around a SQLite3 database | ||
*/ | ||
class sqlite | ||
{ | ||
private: | ||
/** | ||
* Smart pointer (unique) type for sqlite3 database | ||
*/ | ||
using sqlite_t = std::unique_ptr<sqlite3, sqlite_deleter>; | ||
|
||
sqlite_t conn = nullptr; | ||
|
||
public: | ||
sqlite() = delete; | ||
|
||
/** | ||
* Construct a new sqlite object from a path to database | ||
* | ||
* @param path File path to sqlite3 database | ||
*/ | ||
sqlite(const std::string& path); | ||
|
||
sqlite(sqlite& db) = delete; | ||
|
||
sqlite& operator=(sqlite& db) = delete; | ||
|
||
/** | ||
* Take ownership of a sqlite3 database | ||
* | ||
* @param db sqlite3 database object | ||
*/ | ||
sqlite(sqlite&& db) = default; | ||
|
||
/** | ||
* Move assignment operator | ||
* | ||
* @param db sqlite3 database object | ||
* @return sqlite& reference to sqlite3 database | ||
*/ | ||
sqlite& operator=(sqlite&& db) = default; | ||
|
||
/** | ||
* Return the originating sqlite3 database pointer | ||
* | ||
* @return sqlite3* | ||
*/ | ||
sqlite3* connection() const noexcept; | ||
|
||
/** | ||
* Check if SQLite database contains a given table | ||
* | ||
* @param table name of table | ||
* @return true if table does exist | ||
* @return false if table does not exist | ||
*/ | ||
bool has_table(const std::string& table) noexcept; | ||
|
||
/** | ||
* Query the SQLite Database and get the result | ||
* @param statement String query | ||
* @return sqlite_iter SQLite row iterator | ||
*/ | ||
sqlite_iter query(const std::string& statement); | ||
|
||
/** | ||
* Query the SQLite Database with multiple boundable text parameters. | ||
* | ||
* @param statement String query with parameters | ||
* @param binds text parameters to bind to statement | ||
* @return sqlite_iter SQLite row iterator | ||
*/ | ||
sqlite_iter query(const std::string& statement, const std::vector<std::string>& binds); | ||
|
||
/** | ||
* Query the SQLite Database with a bound statement and get the result | ||
* @param statement String query with parameters | ||
* @param params parameters to bind to statement | ||
* @return sqlite_iter SQLite row iterator | ||
*/ | ||
template<typename... T> | ||
sqlite_iter query(const std::string& statement, T const&... params); | ||
}; | ||
|
||
} // namespace geopackage | ||
|
||
#endif // NGEN_GEOPACKAGE_SQLITE_H |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not thrilled with an API where "empty subset list" means "give me everything", but it looks like that's matching what our geojson reader already does, so that's probably unreasonable to change.
My preferred alternative would probably be to take something like
std::variant
that distinguishes the "give me everything" case from a specified subset.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is something we can potentially change when we get around to refactoring the feature handling.