Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GeoPackage input support #522

Merged
merged 62 commits into from
Jul 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
81ae5f1
initial geopackage support impl
program-- May 1, 2023
da0ff43
more work toward gpkg support; started wkb parser
program-- May 1, 2023
d5cc164
scaffold CMake for gpkg
program-- May 1, 2023
484ee5a
implement (standard) wkb parser
program-- May 1, 2023
af82743
cleanup function declarations
program-- May 1, 2023
f251db2
add docs to wkb reader funcs
program-- May 1, 2023
3945ae6
add generic geometry struct; refactor wkt funcs as visitor
program-- May 1, 2023
dbb1a47
clean up some WKB code
program-- May 1, 2023
2278e0b
separate wkt visitor from wkb directly
program-- May 1, 2023
2463e40
lots of refactoring -- streamlined wkb -> geojson types [no ci]
program-- May 1, 2023
d30f2b0
fixed wkb reading multi geometries and added more wkb tests
program-- May 1, 2023
425128c
added sqlite dep and tests
program-- May 1, 2023
8a87439
readd const modifier on member functions
program-- May 1, 2023
5d899b6
separate GeoPackage impl from headers
program-- May 1, 2023
c92fd65
add geopackage tests
program-- May 1, 2023
94229a8
move impl out to parent folder
program-- May 1, 2023
8f54e95
reorg wkb impl; move wkb::copy_from -> utils::copy_from
program-- May 1, 2023
f5b6af9
separate sqlite impl from header
program-- May 1, 2023
9fe5bae
finally got projections working, gpkg read success
program-- May 2, 2023
8d3dc38
fix projection/bounding box issues; added test gpkg file; started on …
program-- May 3, 2023
432a572
remove unnecessary headers
program-- May 3, 2023
1ebfc13
improve documentation for gpkg and wkb
program-- May 3, 2023
609a0a9
add explicit double initialization for copy_from; add SQLite cmake mo…
program-- May 3, 2023
4545dfe
cleanup formatting/tests
program-- May 8, 2023
e287834
fix OOR error; better errors for geopackage::read
program-- May 8, 2023
88885a6
allow read from gpkg or geojson when running ngen
program-- May 8, 2023
6abf62a
added more docs; working on issue w/ projections for Release build type
program-- May 9, 2023
9eee5d4
more docs to source files
program-- May 9, 2023
d2de692
add conditional SQLite support
program-- May 9, 2023
8aab30a
add ifdef guard around GeoPackage.hpp include
program-- May 9, 2023
8701bfe
fix ifdef -> if
program-- May 10, 2023
71824c9
more ifdef -> if fixes
program-- May 10, 2023
999c523
revert ifdef changes; fix adding compile defs for sqlite
program-- May 10, 2023
97c6db4
disable optimizations for geometry.cpp if compiler is not intel
program-- May 10, 2023
aef9c56
rev: remove default construction for sqlite object
program-- Jul 10, 2023
ec3421d
rev: fix off-by-one in WKB reader; reorg WKB impl
program-- Jul 10, 2023
d114363
rev: remove extra return
program-- Jul 10, 2023
a78e9b7
rev: get_property takes in reference to string
program-- Jul 10, 2023
79148ba
rev: add test for geopackage subsetting on read
program-- Jul 10, 2023
4b82a22
rev: insert newlines to prevent git issues
program-- Jul 10, 2023
64b3cfd
rev: rm sqlite3 cmake module in favor of built-in
program-- Jul 10, 2023
0730d2f
cmake: remove test linking to NGen::core; make includes public
program-- Jul 10, 2023
fd548c9
rev: rename SQLite.hpp; move sqlite_error def
program-- Jul 10, 2023
864a859
rev: move sqlite_t type alias into class
program-- Jul 10, 2023
87618ca
rev: delete sqlite copy constructor/assignment
program-- Jul 10, 2023
e999531
rev: rm reinitialization in constructor
program-- Jul 10, 2023
fcd9d6d
rev: set sqlite move constructor/assignment to default
program-- Jul 10, 2023
9dcecbf
rev: remove `stmt` from sqlite; shared -> unique
program-- Jul 11, 2023
4e18ca2
rev: add comments regarding WKB implementation
program-- Jul 11, 2023
9e44d42
rev: add geometry type guards in wkb reader
program-- Jul 11, 2023
f244bb4
rev: throw exception if invalid wkb type
program-- Jul 11, 2023
74018aa
rev: rewrite `std::memcpy` call in `copy_from`
program-- Jul 11, 2023
9ceed0c
rev: add newline to
program-- Jul 11, 2023
a562388
rev: handle FeatureCollection `shared_ptr` creation
program-- Jul 11, 2023
19849fc
rev: remove statement output from sqlite_error calls
program-- Jul 11, 2023
5698854
rev: move point bbox into switch
program-- Jul 11, 2023
894e8ab
rev: use wkb_geom_t enum for switch
program-- Jul 11, 2023
5419798
rev: scientific notation; cmake: update optimizer warning
program-- Jul 12, 2023
5531a89
rev: change literal byte keys for clarity
program-- Jul 12, 2023
8cbd253
rev: white-list check for layer names
program-- Jul 12, 2023
cf1d17f
rev: update cmakelists; feature exhaustive case
program-- Jul 13, 2023
bd30664
rev: update wkb test
program-- Jul 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,16 @@ endif()

find_package(Boost 1.72.0 REQUIRED)


find_package(SQLite3)
if(SQLite3_FOUND)
set(NGEN_WITH_SQLITE3 ON)
add_compile_definitions(NGEN_WITH_SQLITE3)
else()
set(NGEN_WITH_SQLITE3 OFF)
endif()
message(INFO " SQLite3 support is ${NGEN_WITH_SQLITE3}")

# UDUNITS
# Since UDUNITS is currently not really optional (yet) let's make it default to on...
if (NOT (DEFINED UDUNITS_ACTIVE))
Expand Down Expand Up @@ -259,6 +269,12 @@ endif()
add_subdirectory("src/core")
add_dependencies(core libudunits2)
add_subdirectory("src/geojson")

if(NGEN_WITH_SQLITE3)
add_subdirectory("src/geopackage")
target_link_libraries(ngen PUBLIC NGen::geopackage)
endif()

add_subdirectory("src/realizations/catchment")
add_subdirectory("src/models/tshirt")
add_subdirectory("src/models/kernels/reservoir")
Expand Down
62 changes: 62 additions & 0 deletions include/geopackage/GeoPackage.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#ifndef NGEN_GEOPACKAGE_H
#define NGEN_GEOPACKAGE_H

#include "FeatureCollection.hpp"
#include "NGen_SQLite.hpp"

namespace geopackage {

/**
* Build a geometry object from GeoPackage WKB.
*
* @param[in] row SQLite iterator at the row containing a geometry column
* @param[in] geom_col Name of geometry column containing GPKG WKB
* @param[out] bounding_box Bounding box of the geometry to output
* @return geojson::geometry GPKG WKB converted and projected to a boost geometry model
*/
geojson::geometry build_geometry(
const sqlite_iter& row,
const std::string& geom_col,
std::vector<double>& bounding_box
);

/**
* Build properties from GeoPackage table columns.
*
* @param[in] row SQLite iterator at the row containing the data columns
* @param[in] geom_col Name of geometry column containing GPKG WKB to ignore
* @return geojson::PropertyMap PropertyMap of properties from the given row
*/
geojson::PropertyMap build_properties(
const sqlite_iter& row,
const std::string& geom_col
);

/**
* Build a feature from a GPKG table row
*
* @param[in] row SQLite iterator at the row to build a feature from
* @param[in] geom_col Name of geometry column containing GPKG WKB
* @return geojson::Feature Feature containing geometry and properties from the given row
*/
geojson::Feature build_feature(
const sqlite_iter& row,
const std::string& geom_col
);

/**
* Build a feature collection from a GPKG layer
*
* @param[in] gpkg_path Path to GPKG file
* @param[in] layer Layer name within GPKG file to create a collection from
* @param[in] ids optional subset of feature IDs to capture (if empty, the entire layer is converted)
* @return std::shared_ptr<geojson::FeatureCollection>
*/
std::shared_ptr<geojson::FeatureCollection> read(
const std::string& gpkg_path,
const std::string& layer,
const std::vector<std::string>& ids
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not thrilled with an API where "empty subset list" means "give me everything", but it looks like that's matching what our geojson reader already does, so that's probably unreasonable to change.

My preferred alternative would probably be to take something like std::variant that distinguishes the "give me everything" case from a specified subset.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something we can potentially change when we get around to refactoring the feature handling.

);

} // namespace geopackage
#endif // NGEN_GEOPACKAGE_H
230 changes: 230 additions & 0 deletions include/geopackage/NGen_SQLite.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,230 @@
#ifndef NGEN_GEOPACKAGE_SQLITE_H
#define NGEN_GEOPACKAGE_SQLITE_H

#include <memory>
#include <stdexcept>
#include <vector>
#include <string>

#include <sqlite3.h>

namespace geopackage {

/**
* Deleter used to provide smart pointer support for sqlite3 structs.
*/
struct sqlite_deleter
{
void operator()(sqlite3* db) { sqlite3_close_v2(db); }
void operator()(sqlite3_stmt* stmt) { sqlite3_finalize(stmt); }
};

/**
* Smart pointer (shared) type for sqlite3 prepared statements
*/
using stmt_t = std::unique_ptr<sqlite3_stmt, sqlite_deleter>;

/**
* SQLite3 row iterator
*
* Provides a simple iterator-like implementation
* over rows of a SQLite3 query.
*/
class sqlite_iter
{
private:
stmt_t stmt;
int iteration_step = -1;
bool iteration_finished = false;

int column_count;
std::vector<std::string> column_names;

// vector of SQLITE data types, see: https://www.sqlite.org/datatype3.html
std::vector<int> column_types;

// returns the raw pointer to the sqlite statement
sqlite3_stmt* ptr() const noexcept;

// checks if int is out of range, and throws error if so
void handle_get_index(int) const;

public:
sqlite_iter(stmt_t stmt);

/**
* Check if a row iterator is finished
*
* @return true if next() returned SQLITE_DONE
* @return false if there is more rows available
*/
bool done() const noexcept;

/**
* Step into the next row of a SQLite query
*
* If the query is finished, next() acts idempotently,
* but will change done() to return true.
* @return sqlite_iter& returns itself
*/
sqlite_iter& next();

/**
* Restart an iteration to its initial state.
* next() must be called after calling this.
*
* @return sqlite_iter& returns itself
*/
sqlite_iter& restart();

/**
* Get the current row index for the iterator
*
* @return int the current row index, or -1 if next() hasn't been called
*/
int current_row() const noexcept;

/**
* Get the number of columns within this iterator
* @return int number of columns in query
*/
int num_columns() const noexcept;

/**
* Return the column index for a named column
*
* @param name column name to search for
* @return int index of given column name, or -1 if not found.
*/
int column_index(const std::string& name) const noexcept;

/**
* Get a vector of column names
*
* @return const std::vector<std::string>& column names as a vector of strings
*/
const std::vector<std::string>& columns() const noexcept { return this->column_names; }

/**
* Get a vector of column types
*
* See https://www.sqlite.org/datatype3.html for type values. The integers
* are the affinity for data types.
* @return const std::vector<int>& column types as a vector of ints
*/
const std::vector<int>& types() const noexcept { return this->column_types; }

/**
* Get a column value from a row iterator by index
*
* @tparam T Type to parse value as, i.e. int
* @param col Column index to parse
* @return T value at column `col`
*/
template<typename T>
T get(int col) const;

/**
* Get a column value from a row iterator by name
*
* @tparam T Type to parse value as, i.e. int
* @return T value at the named column
*/
template<typename T>
T get(const std::string&) const;
};

template<typename T>
inline T sqlite_iter::get(const std::string& name) const
{
const int index = this->column_index(name);
return this->get<T>(index);
}

/**
* Wrapper around a SQLite3 database
*/
class sqlite
{
private:
/**
* Smart pointer (unique) type for sqlite3 database
*/
using sqlite_t = std::unique_ptr<sqlite3, sqlite_deleter>;

sqlite_t conn = nullptr;

public:
sqlite() = delete;

/**
* Construct a new sqlite object from a path to database
*
* @param path File path to sqlite3 database
*/
sqlite(const std::string& path);

sqlite(sqlite& db) = delete;

sqlite& operator=(sqlite& db) = delete;

/**
* Take ownership of a sqlite3 database
*
* @param db sqlite3 database object
*/
sqlite(sqlite&& db) = default;

/**
* Move assignment operator
*
* @param db sqlite3 database object
* @return sqlite& reference to sqlite3 database
*/
sqlite& operator=(sqlite&& db) = default;

/**
* Return the originating sqlite3 database pointer
*
* @return sqlite3*
*/
sqlite3* connection() const noexcept;

/**
* Check if SQLite database contains a given table
*
* @param table name of table
* @return true if table does exist
* @return false if table does not exist
*/
bool has_table(const std::string& table) noexcept;

/**
* Query the SQLite Database and get the result
* @param statement String query
* @return sqlite_iter SQLite row iterator
*/
sqlite_iter query(const std::string& statement);

/**
* Query the SQLite Database with multiple boundable text parameters.
*
* @param statement String query with parameters
* @param binds text parameters to bind to statement
* @return sqlite_iter SQLite row iterator
*/
sqlite_iter query(const std::string& statement, const std::vector<std::string>& binds);

/**
* Query the SQLite Database with a bound statement and get the result
* @param statement String query with parameters
* @param params parameters to bind to statement
* @return sqlite_iter SQLite row iterator
*/
template<typename... T>
sqlite_iter query(const std::string& statement, T const&... params);
};

} // namespace geopackage

#endif // NGEN_GEOPACKAGE_SQLITE_H
Loading