Skip to content

HAME regional land use planning database compatible with national Ryhti data model

License

Notifications You must be signed in to change notification settings

GispoCoding/hame-ryhti

 
 

Repository files navigation

hame-ryhti

CI/CD pre-commit

HAME regional land use planning database compatible with national Ryhti data model

Architecture

HAME-Ryhti consists of

  1. a PostGIS database,
  2. various AWS Lambda functions to manage the database and import or export planning data, and
  3. QGIS project to connect to the database and create regional land use plans.

To manage Hame-Ryhti AWS resources, check the infra directory.

Development requirements

Development

  1. Create a Python virtual environment and activate it.
  2. pip install pip-tools
  3. pip-sync requirements.txt requirements-dev.txt
  4. pre-commit install

Database and functions

  1. Run tests with make pytest
  2. Build and start the development containers with docker-compose -f docker-compose.dev.yml up -d (or make rebuild).
  3. Fill the database with current data model by make test-create-db.
  4. Populate national code tables from koodistot.suomi.fi by make test-koodistot.
  5. Edit the lambda functions under database, run tests and rebuild again.

If test using pytest-docker get stuck, you can remove the dangling containers with:

docker ps --format '{{.Names}}' |grep pytest | awk '{print $1}' | xargs -I {} docker stop {}
docker ps --format '{{.Names}}' |grep pytest | awk '{print $1}' | xargs -I {} docker rm {}
docker network ls --format {{.Name}} |grep pytest | awk '{print $1}' | xargs -I {} docker network rm {}

Database changes

  1. Database is defined using SQLAlchemy, so familiarize yourself with SQLAlchemy declarative style.
  2. Database is divided into two schemas: codes contains all the Ryhti specific national code lists, while hame contains all the data tables (plans, plan objects, plan regulations etc.).
  3. If you want to change all tables in a schema (i.e. edit all the code tables, or add a field to all the data tables), the abstract base classes are in base.py.
  4. If you only want to change/add one code table or one data table, please edit/add the right table in codes.py or models.py.
  5. To get the changes tested and usable in your functions, create a new database revision with make revision name="describe_your_changes", e.g. make revision name="add_plan_object_table". This creates a new random id (uuid) for your migration, and a revision file YYYY-MM-DD-HHMM-uuid-add_plan_object_table in the alembic versions dir. Please check that the autogenerated revision file seems to do approximately sensible things.
    • Specifically, when adding geometry fields, please note GeoAlchemy2 bug with Alembic, which means you will have to manually remove op.create_index and op.drop_index in the revision file. This is because GeoAlchemy2 already automatically creates geometry index whenever adding a geometry column.
  6. Run tests with make pytest to check that the revision file runs correctly. At minimum, you may have to change the tested table counts (codes_count and hame_count) in database test setup to reflect the correct number of tables in the database.
  7. Run make rebuild and make test-create-db to start development instance with the new model.
  1. Commit your changes and the new revision file in alembic versions dir.

Adding requirements

To add new requirements:

  1. Add the Python library in requirements.in (if used in production) or requirements-dev.in (if used in development/CI/CD).
  2. pip-compile requirements.in or pip-compile requirements-dev.in
  3. pip-sync requirements.txt requirements-dev.txt

To update requirements to latest versions:

  1. pip-compile requirements.in --upgrade and pip-compile requirements-dev.in --upgrade
  2. pip-sync requirements.txt requirements-dev.txt