Fix for Sorting and Upserts (#153)

* fix issue with upserts

* fix sorting issue #152

* bump version

* add tests for sorting by a property

* small change to kick ci

* add stac_extensions table

* make queryables aggregate enum, min, max, add tooling to get extensions into queryables

* update changelog

* reformat for black update

* add check_pgstac_settings function

* update settings function

* add procedures to validate constraints and analyze items partitions

* prep migration

* update docs

* add tests for paging without using fields extension

Author: bitner

CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [v0.6.12]
+
+### Added
+- Add ability to merge enum, min, and max from queryables where collections have different values.
+- Add tooling in pypgstac and pgstac to add stac_extension definitions to the database.
+- Modify missing_queryables function to try to use stac_extension definitions to populate queryable definitions from the stac_extension schemas.
+- Add validate_constraints procedure
+- Add analyze_items procedure
+- Add check_pgstac_settings function to check system and pgstac settings.
+
+### Fixed
+- Fix issue with upserts in the trigger for using the items_staging tables
+- Fix for generating token query for sorting. [152] (https://github.com/stac-utils/pgstac/pull/152)
+
 ## [v0.6.11]
 
 ### Fixed

docs/src/pgstac.md

@@ -72,6 +72,11 @@ ALTER ROLE <username> SET pgstac.context_estimated_cost TO '<estimated query cos
 ALTER ROLE <username> SET pgstac.context_stats_ttl TO '<an interval string ie "1 day" after which pgstac search will force recalculation of it's estimates>>';
 ```
 
+The check_pgstac_settings function can be used to check what pgstac settings are being used and to check recomendations for system settings. It takes a single parameter which should be the amount of memory available on the database system.
+```sql
+SELECT check_pgstac_settings('16GB');
+```
+
 #### Runtime Configurations
 
 Runtime configuration of variables can be made with search by passing in configuration in the search json "conf" item.
@@ -104,3 +109,10 @@ VALUES (<property name>, <property wrapper>, <index type>);
 Property wrapper should be one of to_int, to_float, to_tstz, or to_text. The index type should almost always be 'BTREE', but can be any PostgreSQL index type valid for the data type.
 
 **More indexes is note necessarily better.** You should only index the primary fields that are actively being used to search. Adding too many indexes can be very detrimental to performance and ingest speed. If your primary use case is delivering items sorted by datetime and you do not use the context extension, you likely will not need any further indexes.
+
+### Maintenance Procedures
+These are procedures that should be run periodically to make sure that statistics and constraints are kept up-to-date and validated. These can be made to run regularly using the pg_cron extension if available.
+```sql
+SELECT cron.schedule('0 * * * *', 'CALL validate_constraints();');
+SELECT cron.schedule('10, * * * *', 'CALL analyze_items();');
+```

pypgstac/pypgstac/db.py

@@ -174,12 +174,13 @@ def query(
             with conn.cursor(row_factory=row_factory) as cursor:
                 if args is None:
                     rows = cursor.execute(query, prepare=False)
-                    for row in rows:
-                        yield row
                 else:
                     rows = cursor.execute(query, args)
+                if rows:
                     for row in rows:
                         yield row
+                else:
+                    yield None
         except psycopg.errors.OperationalError as e:
             # If we get an operational error check the pool and retry
             logger.warning(f"OPERATIONAL ERROR: {e}")

pypgstac/pypgstac/hydration.py

@@ -12,6 +12,7 @@ def hydrate(base_item: Dict[str, Any], item: Dict[str, Any]) -> Dict[str, Any]:
     This will not perform a deep copy; values of the original item will be referenced
     in the return item.
     """
+
     # Merge will mutate i, but create deep copies of values in the base item
     # This will prevent the base item values from being mutated, e.g. by
     # filtering out fields in `filter_fields`.