Skip to content

v7.0.0

Latest
Latest
Compare
Choose a tag to compare
@twelch twelch released this 26 Dec 06:48
· 2 commits to dev since this release
v7.0.0
74b41c2

What's Changed

This release has been about a year in the making and is finally ready (enough)!

  • Complete transition to ECMAScript modules aka ESM, including in geoprocessing projects and all the way to the browser and Lambdas.
  • Support for Node 22
  • Transition from Webpack to Vite, and Jest to Vitest
  • Upgrade to aws-sdk v3
  • Use latest versions of Turf.js, Geoblaze, Flatgeobuf, and GDAL
  • New documentation website maintained internally and published at https://seasketch.github.io/geoprocessing/

Performance improvements:

  • getFeaturesForSketchBBoxes - estimated 10-50% decrease in time to fetch features, when used instead of loadFgb for sketch collections covering large geographic area.
  • overlapFeatures - in addition to above, estimated 5-10% decrease in time to calculate collection metric.
  • Workder lambdas - allowing data fetching and analysis to spread out across multiple lambdas, usually one per data class, but potential for more. Can decrease time to run reports an estimated 80% in some cases.

Scaling improvements:

  • Task cache: Analysis results can be as large as you want, previously limited to 400KB.
  • LambdaStacks: Project can have any number of preprocessing/geoprocessing functions, previously limited to 12-15.

Upgrade path: significant breaking changes have accumulated for version 7 over 13 beta releases. To upgrade from 6.1.x, it is suggested to create a new 7.0 project and migrate your existing project assets one step at a time. The code you bring over will require additional migration. If you created your project using one of the project betas, then you should be able to use the upgrade process. As of beta.8, prettier code formatting will be run on all source files.

Geoprocessing Workspace: requires upgrade to at least sha-8f10ce5 published on 12/11/2024

  • Ubuntu 24.04 Noble
  • Node 22.12.0

Global Datasources: the global datasources project has been upgraded with new datasources. Vector datasources are subdivided. All preprocessing and geoprocessing functions are welcome to use them directly. Instructions are also provided to export a subset of these datasources for local import into projects.

💥 Breaking Changes

Move/remove/replace:

  • Drop deprecated loadCogWindow and bboxToPixelEdge functions
  • Drop overlapRaster function
  • Migrate dataprovider module into top-level library instead of having as a separate export (now that esm is here and flatgeobuf issues have been fixed), migrate base project to use it.
  • Drop genPreprocessor function. Replace with clipToPolygonFeatures, genClipToPolygonFeatures, clipToPolygonDatasources and genClipToPolygonDatasources.
  • Drop genRandomSketch, merge into genRandomFeature and refactor with more options
  • Drop genClipLoader, replaced with clipToPolygonFeatures/genClipToPolygonFeatures and clipToPolygonDatasources/genClipToPolygonDatasources. Default preprocessing functions have migrated to using clipToPolygonFeatures and fetching features in a separate step.
  • Drop JSON format from data import and precalc. It's no longer required now that precalc reads fgb from disk and fgb is superior for other uses.
  • All project devDependencies moved to dependencies. On your next upgrade, all dependencies will be added as dependencies and you will need to manually remove them from devDependencies because upgrade does not handle deletion.
  • Some CLI commands have been renamed. watch is now build:watch, clear-all-results has been removed, and clear-results is now clear:results and you can enter the option to delete all cached results. This has the benefit of limiting accidental deletes of the full cache, you have to specifically state you want to delete all.

Upgrade script:

  • Add new upgrade script for migrating to latest version of geoprocessing library. by @twelch in #301, improved in #306.
    • For any project running older than 7.0.0-beta.5 you must manually add the script to the geoprocessing.json scripts section - "upgrade": "geoprocessing upgrade". See the upgrade page for instruction on how to use it.
  • Move geoprocessing.json from top-level to project/geoprocessing.json.

LambdaStack:

  • Migrate lambda functions to NestedStacks under the top-level GeoprocessingStack to avoid cdk 500 resource limit. Scales to add more stacks as needed automatically by @twelch in #309 and #314.
    • The breaking change is that due to fixed logical ID's being used for each lambda function, a duplicate ID error will result when redeploying an existing project and CDK tries to create a function in a nested LambdaStack that already exists in the root stack. It doesn't appear to take care of the deletion first. Until a solution can be found, you will need to completely destroy and redeploy your project.
    • Functions and their workers are deployed to the same LambdaStack solving thrashing issue causing Duplicate resource error on deploy by @twelch in #353.

DynamoDB cache:

  • Switch tasks class to storing dynamodb results as JSON string chunks, drop correlationId and peerDependencies by @twelch in #330 and #336
    • A new algorithm is used for storing function results in DynamoDB that works regardless of the structure of your JSON. After deploy, you will need to clear your existing cache by running npm run clear-all-results. by @twelch in
      #313
  • Disable task cache for worker lambdas by default, improve logging by @twelch in #342

👎 Deprecated

  • Deprecate getFeatures (in name) and replace with same functionality in getDatasourceFeatures. This makes it clear that this is a datasource aware function.
  • Deprecate fgbFetchAll function for loadFgb
  • Deprecate genClipLoader function, no longer needed now that genClipToPolygonFeatures and genClipToPolygonDatasources is available
  • Deprecate use of bbox and planning area properties in basic.json.
  • Deprecate template-ocean-eez. template-blank-project is now installed by default. It may simply get merged with base-project before 7.0 release. createReport has been found to be a better way of letting developer install a variety of report types.
  • Add overlapPolygonArea and overlapPolygonSum, deprecating overlapFeatures by @twelch in #389.
    • The input parameters and algorithms for the two operations offered by overlapFeatures (area and sum) had diverged too far and it made sense to split them into separate functions. overlapFeatures only let you run one operation or the other so this should be a simple migration of code to call one or the other.
  • Split out toMultiPolygon and removeOverlap helper functions used as building blocks for these functions.

🚀 New Feature / Improvement

  • Add JSON schema for validation of project JSON config files. By @avmey in #283
  • Export genTaskCacheKey function in client-core library.
  • Adds overlapRasterGroupMetrics function by @avmey in #270
  • Add overlap report create:report by @avmey in #271
  • Add eslint unicorn plugin with fixes by @twelch in #355
  • Support childProperties for SketchCollection by @twelch in #341
  • ADA Accessibility by @avmey in #370

Worker lambdas:

  • Support parallel geoprocessing using worker lambda functions. Added permission for async function lambdas to invoke sync function lambdas. Provide geoprocessing functions access to original API parameters (e.g. sketch URL) so that it can be passed on to workers. By @twelch in #287.
    • Require explicit registration of worker functions, fix CDK circular dependency error by @twelch in #352. To register workers used by a geoprocessing function, simply pass a workers option to your GeoprocessingHandler with a list of the worker titles. See Workers guide doc.
    • Add runLambdaWorker helper for invoking sync worker functions by @avmey in #326
    • Support passing geobuf-encoded sketch parameter to geoprocessing function by @twelch in #294. Used by parent geoprocessing function to pass sketch to worker when sketch is small enough to be directly passed.

Dependency upgrades:

  • Upgrade to React 18 by @twelch in #286
  • Upgrade to stable Turf v7.0.
  • Upgrade to aws-sdk v3 by @twelch in #321
  • Upgrade to latest CDK lib and Node 22 lambda runtime by @twelch in #392
  • Upgrade to latest version of Storybook supporting Vite. Add genReportStories() that runs on storybook start and generates stories for each instance of *.stories.tsx and example sketch. This has diminished the storybook experience because it requires storybook to be restarted more often because the story index is not updated automatically, see #374.

i18n

  • Update i18next dependencies, update base translations, fix locale import, migrate importTerms and publishTerms to use built-in fetch API instead of request by @twelch in #295
  • Add Brazilian Portuguese to supported languages by @avmey in #354
  • upgrade: remove old translation scripts by @twelch in #360
  • Allow user to limit project i18n languages in project/basic.json, and change watch to build:watch by @twelch in #356

Storybook:

  • speed up storybook, merge existing project prettierignore with base on upgrade by @twelch in #361

CLI/Scripts

Helper functions:

  • add validatePolygon preprocessor, with ensureValidPolygon helper function
  • add roundDecimalFormat and numberFormat helper functions
  • add toSketchProperties helper function
  • add getMetricGroupDatasource and getMetricGroupClassKey methods to projectClientBase
  • Re-export toJsonFile helper for quickly writing JSON to disk.

Toolbox functions

  • add getFeaturesForSketchBboxes function with split feature, add splitBBoxAntimeridian and cleanBBox helpers,
  • export intersection functions - intersectInChunks, intersectInChunksArea, intersectSum
  • getDatasourceFeatures now accepts optional sketch option. When used it calls optimized getFeaturesForSketchBBoxes
  • getFeaturesForSketchBBoxes add support for all geometry types and allow user to extend them with generics support
  • Add lower-level getFeaturesForBBoxes that takes an array of bounding boxes
  • rasterStats no longer catches and swallows all errors. Improved to catch specific error thrown when a zero polygon is passed to geoblaze, and returns default raster stats, otherwise it rethrows the error.

Types:

  • Make MetricGroup type property optional.
  • improve metricGroupSchema in schemas.json

Project init:

  • Drop bbox, planning area, and template questions.
  • Remove template-ocean-eez from workspace so no longer tested (likely to be removed in future)
  • Clean up simple and blank reports (function + cards)
  • Drop ViabilityPage from TabReport and simply inline all report pages
  • dataImport: conditional ask additional properties question
  • Sort init language alphabetically
  • Add vitest-fetch-mock dep to base project
  • Migrate to childProperties
  • Add validatePolygon preprocessor
  • Add minSize and enforceMinSize options to preprocessors
  • Drop global datasources from basic-project template except for osm-land and eez-land-union
  • Remove default objective and metricgroup from project. objectives.json and metrics.json start as empty arrays now. This is inline with them being opt-in features rather than forced use.

createReport:

  • switch vectorFunction to use getFeaturesForSketchBboxes.
  • migrate vectorFunction/rasterFunction/OverlapCard from using NullSketch in result to get sketch properties to using useSketchProperties
  • createFunction and createReport now default to async geoprocessing function. The user often doesn't know which they need right away.

Datasources:

  • loadFgb: add retry on error with exponential backoff (3 times). This will hopefully avoid the Closed socket error seeing in testing environment, seemingly in high volume request situation.
  • Add callWithRetry function that will retry calling a function on thrown error, with progressively larger wait time in between each call. optional ifErrorMsgContains string, will now retry only if thrown error message contains that string otherwise.

Preprocessing functions:

  • Preprocessor update by @twelch in #393
    • Migrate all preprocessing functions away from using the clipToPolygonFeatures abstraction and directly calling clip and clipMultiMerge. This makes it easier for developer to reason about and test what is happening.
    • Directly use subdivided flatgeobuf global datasources instead of VectorDataSource bundles. no need to register them in datasources.json just directly use URL.
    • Switch from OSM land to Daylight coastline polygons.
    • Add biggestPolygon helper function used by preprocessing functions to return the largest polygon to SeaSketch when multiple remain after clipping.

🐛 Bug Fix

Storybook:

  • genReportStories - pass example sketch params to report context so that sketch is fully available in story
  • fix storybook error when sketch name has spaces or any characters not valid in a Javascript variable name
  • Fix storybook:install script.
  • remove .story-cache folders from dist build

i18n:

  • Fix translation:install command so that it doesn't wipe out project i18n configuration and remove translation install from translation:extract command by @twelch in #296.
  • fix importTranslations, fix BlankCard dynamic trans string, update translations
  • Switch to using raw option for glob import of locale files and parsing as JSON strings. Module loading was not working properly.
  • Stop overwriting src/i18n/config.ts on translation:install. remoteContext was being lost.
  • fix i18nAsync bug using base english strings instead of letting component render its own
  • Add json type assert to import of extraTerms.json files
  • Fix clear results scripts by @twelch in #298
  • Fix tooltip z-index issue and font styling. Document pdf printing by @avmey in #276
  • Fix template path bug on new project init by @twelch in #308
  • Handle no-overlap for categorical rasters, creating proper zero metrics by @avmey in #315
  • task metricGroup results not saved when just one group by @twelch in #319
  • Generate clipped bounding boxes when using clipToGeography by @avmey in #325
  • Fix task caching and improved logging by @twelch in #336
  • npm run url command now properly outputs all service URLs and bucket names used by the project for easy reference.
  • Fix clear-all-results command. Add retry to dynamodb bulk delete resolving ThroughputError, break out helper functions by @twelch in #344
  • init: fix setting package version, gpVersion override by @twelch in #359
  • fix genRandomPolygon name bug
  • genRandomPolygon - start id's at 1, was causing a bug downstream with 0 id value
  • avoid node-fetch error when fetching VectorDataSource metadata by migrating preprocessing functions to use new subdivided global datasources.
  • clipToLand: fix bug where if land feature fetch returned no polygons it would return the original sketch, instead of throwing that the sketch is outside the land boundary.
  • Fix multiple choice sketch attributes bug by @avmey in #368

🏠 Internal

  • Now that projects need their own .storybook directory, template packages now symlink in the base-project .geoprocessing directory.
  • Merge scripts directory build and launchers with src by @twelch in #297
  • Add separate export for non-english supported languages
  • Update github action to latest versions to support node 20
  • Drop wiki docs from repo including publish action. For now they will be managed in the wiki directly again.
  • Drop mentions of test:smoke
  • Drop all group and environment directives from tests
  • Extract getTemplatePackages helper function
  • Add node 22 to CI tests in prep for future use by @twelch in #305
  • Add End-to-end test of project build + CDK synth for a variety of scenarios by @twelch in #314
  • Add additional logging to GeoprocessingHandler to be able to assess functions are working as expected.
  • Add internal support for ESLint and Prettier by @twelch in #348
  • rename split.js to antimeridian.js.
  • storybook no longer auto-opens web browser, caused error in geoprocessing WSL.
  • migrate to loadFgb internally from deprecated fgbFetchAll
  • upgrade to flatgeobuf 3.36.0 and drop use of web streams polyfill and node-fetch now that bug fixed.
  • add Vite babel plugins for storybook
  • Remove antimeridian tests from getFeatures
  • add getFgbPath method to projectClientBase for loading of flatgeobuf from local filesystem
  • move all createReport assets over to now unused template-ocean-eez directory. eventually need to simplify this.
  • Drop old doc build workflow by @twelch in #382
  • Drop global datasources from base-project. No longer included with new project.
  • Contribute squares.fgb, squares.json with easy to reason about named polygons for testing. Develop getFeaturesForSketchBBoxes e2e tests to use them.
  • Update squareSketches fixtures to match squares.fgb/json for testing.
  • Add squaresReadme.txt ascii art image to see the layout of the polygons
  • refine polygonSmokeTest logging

📃 Docs

  • Add docusaurus website by @twelch in #362.
  • Fix 6.1.0 tutorial, referenced features not available in 6.x. Also some issues with ordering of steps.
  • Major refactor for Next (7.0) docs, complete with systemsetup, sampleproject and newproject tutorials, as well as a number of guides.
  • remove codespaces as documented tutorial option
  • Add printing reports docs to Extending by @avmey in #275
  • Drop macos direct install support from docs by @twelch in #383

Full Changelog: v6.1.2...v7.0.0

Contributors

twelch and avmey
Assets 2
Loading