v7.0.1

What's Changed

Removed template packages, simplifying project init

⚰️ Code Death

• Drop template packages and merge into base-project, convert to report-assets by @twelch in #396

Full Changelog: v7.0.0...v7.0.1

v7.0.0

What's Changed

This release has been about a year in the making and is finally ready (enough)!

• Complete transition to ECMAScript modules aka ESM, including in geoprocessing projects and all the way to the browser and Lambdas.
• Support for Node 22
• Transition from Webpack to Vite, and Jest to Vitest
• Upgrade to aws-sdk v3
• Use latest versions of Turf.js, Geoblaze, Flatgeobuf, and GDAL
• New documentation website maintained internally and published at https://seasketch.github.io/geoprocessing/

Performance improvements:

• getFeaturesForSketchBBoxes - estimated 10-50% decrease in time to fetch features, when used instead of loadFgb for sketch collections covering large geographic area.
• overlapFeatures - in addition to above, estimated 5-10% decrease in time to calculate collection metric.
• Workder lambdas - allowing data fetching and analysis to spread out across multiple lambdas, usually one per data class, but potential for more. Can decrease time to run reports an estimated 80% in some cases.

Scaling improvements:

• Task cache: Analysis results can be as large as you want, previously limited to 400KB.
• LambdaStacks: Project can have any number of preprocessing/geoprocessing functions, previously limited to 12-15.

Upgrade path: significant breaking changes have accumulated for version 7 over 13 beta releases. To upgrade from 6.1.x, it is suggested to create a new 7.0 project and migrate your existing project assets one step at a time. The code you bring over will require additional migration. If you created your project using one of the project betas, then you should be able to use the upgrade process. As of beta.8, prettier code formatting will be run on all source files.

Geoprocessing Workspace: requires upgrade to at least sha-8f10ce5 published on 12/11/2024

• Ubuntu 24.04 Noble
• Node 22.12.0

Global Datasources: the global datasources project has been upgraded with new datasources. Vector datasources are subdivided. All preprocessing and geoprocessing functions are welcome to use them directly. Instructions are also provided to export a subset of these datasources for local import into projects.

💥 Breaking Changes

Move/remove/replace:

• Drop deprecated loadCogWindow and bboxToPixelEdge functions
• Drop overlapRaster function
• Migrate dataprovider module into top-level library instead of having as a separate export (now that esm is here and flatgeobuf issues have been fixed), migrate base project to use it.
• Drop genPreprocessor function. Replace with clipToPolygonFeatures, genClipToPolygonFeatures, clipToPolygonDatasources and genClipToPolygonDatasources.
• Drop genRandomSketch, merge into genRandomFeature and refactor with more options
• Drop genClipLoader, replaced with clipToPolygonFeatures/genClipToPolygonFeatures and clipToPolygonDatasources/genClipToPolygonDatasources. Default preprocessing functions have migrated to using clipToPolygonFeatures and fetching features in a separate step.
• Drop JSON format from data import and precalc. It's no longer required now that precalc reads fgb from disk and fgb is superior for other uses.
• All project devDependencies moved to dependencies. On your next upgrade, all dependencies will be added as dependencies and you will need to manually remove them from devDependencies because upgrade does not handle deletion.
• Some CLI commands have been renamed. watch is now build:watch, clear-all-results has been removed, and clear-results is now clear:results and you can enter the option to delete all cached results. This has the benefit of limiting accidental deletes of the full cache, you have to specifically state you want to delete all.

Upgrade script:

• Add new upgrade script for migrating to latest version of geoprocessing library. by @twelch in #301, improved in #306.
  • For any project running older than 7.0.0-beta.5 you must manually add the script to the geoprocessing.json scripts section - "upgrade": "geoprocessing upgrade". See the upgrade page for instruction on how to use it.
• Move geoprocessing.json from top-level to project/geoprocessing.json.

LambdaStack:

• Migrate lambda functions to NestedStacks under the top-level GeoprocessingStack to avoid cdk 500 resource limit. Scales to add more stacks as needed automatically by @twelch in #309 and #314.
  • The breaking change is that due to fixed logical ID's being used for each lambda function, a duplicate ID error will result when redeploying an existing project and CDK tries to create a function in a nested LambdaStack that already exists in the root stack. It doesn't appear to take care of the deletion first. Until a solution can be found, you will need to completely destroy and redeploy your project.
  • Functions and their workers are deployed to the same LambdaStack solving thrashing issue causing Duplicate resource error on deploy by @twelch in #353.

DynamoDB cache:

• Switch tasks class to storing dynamodb results as JSON string chunks, drop correlationId and peerDependencies by @twelch in #330 and #336
  • A new algorithm is used for storing function results in DynamoDB that works regardless of the structure of your JSON. After deploy, you will need to clear your existing cache by running npm run clear-all-results. by @twelch in
    #313
• Disable task cache for worker lambdas by default, improve logging by @twelch in #342

👎 Deprecated

• Deprecate getFeatures (in name) and replace with same functionality in getDatasourceFeatures. This makes it clear that this is a datasource aware function.
• Deprecate fgbFetchAll function for loadFgb
• Deprecate genClipLoader function, no longer needed now that genClipToPolygonFeatures and genClipToPolygonDatasources is available
• Deprecate use of bbox and planning area properties in basic.json.
• Deprecate template-ocean-eez. template-blank-project is now installed by default. It may simply get merged with base-project before 7.0 release. createReport has been found to be a better way of letting developer install a variety of report types.
• Add overlapPolygonArea and overlapPolygonSum, deprecating overlapFeatures by @twelch in #389.
  • The input parameters and algorithms for the two operations offered by overlapFeatures (area and sum) had diverged too far and it made sense to split them into separate functions. overlapFeatures only let you run one operation or the other so this should be a simple migration of code to call one or the other.
• Split out toMultiPolygon and removeOverlap helper functions used as building blocks for these functions.

🚀 New Feature / Improvement

• Add JSON schema for validation of project JSON config files. By @avmey in #283
• Export genTaskCacheKey function in client-core library.
• Adds overlapRasterGroupMetrics function by @avmey in #270
• Add overlap report create:report by @avmey in #271
• Add eslint unicorn plugin with fixes by @twelch in #355
• Support childProperties for SketchCollection by @twelch in #341
• ADA Accessibility by @avmey in #370

Worker lambdas:

• Support parallel geoprocessing using worker lambda functions. Added permission for async function lambdas to invoke sync function lambdas. Provide geoprocessing functions access to original API parameters (e.g. sketch URL) so that it can be passed on to workers. By @twelch in #287.
  • Require explicit registration of worker functions, fix CDK circular dependency error by @twelch in #352. To register workers used by a geoprocessing function, simply pass a workers option to your GeoprocessingHandler with a list of the worker titles. See Workers guide doc.
  • Add runLambdaWorker helper for invoking sync worker functions by @avmey in #326
  • Support passing geobuf-encoded sketch parameter to geoprocessing function by @twelch in #294. Used by parent geoprocessing function to pass sketch to worker when sketch is small enough to be directly passed.

Dependency upgrades:

• Upgrade to React 18 by @twelch in #286
• Upgrade to stable Turf v7.0.
• Upgrade to aws-sdk v3 by @twelch in #321
• Upgrade to latest CDK lib and Node 22 lambda runtime by @twelch in #392
• Upgrade to latest version of Storybook supporting Vite. Add genReportStories() that runs on storybook start and generates stories for each instance of *.stories.tsx and example sketch. This has d...

v7.0.0-beta.13

What's Changed

🚀 New Feature / Improvement

• Preprocessor update by @twelch in #393
  • Migrate all preprocessing functions away from using the clipToPolygonFeatures abstraction and directly calling clip and clipMultiMerge. This makes it easier for developer to reason about and test what is happening.
  • Directly use subdivided flatgeobuf global datasources instead of VectorDataSource bundles. no need to register them in datasources.json just directly use URL.
  • Switch from OSM land to Daylight coastline polygons.
  • Add biggestPolygon helper function used by preprocessing functions to return the largest polygon to SeaSketch when multiple remain after clipping.
  • Re-export toJsonFile helper for quickly writing JSON to disk.

🐛 Bug Fix

• avoid node-fetch error when fetching VectorDataSource metadata by migrating preprocessing functions to use new subdivided global datasources.
• clipToLand: fix bug where if land feature fetch returned no polygons it would return the original sketch, instead of throwing that the sketch is outside the land boundary.

🏠 Internal

• refine polygonSmokeTest logging

Full Changelog: v7.0.0-beta.12...v7.0.0-beta.13

v7.0.0-beta.12

What's Changed

👎 Deprecated

• Add overlapPolygonArea and overlapPolygonSum, deprecating overlapFeatures by @twelch in #389.
  • The input parameters and algorithms for the two operations offered by overlapFeatures (area and sum) had diverged too far and it made sense to split them into separate functions. overlapFeatures only let you run one operation or the other so this should be a simple migration of code to call one or the other.
  • Split out toMultiPolygon and removeOverlap helper functions used as building blocks for these functions.

🚀 New Feature / Improvement

• callWithRetry updates, add back node-fetch polyfill due to unsolved SocketClosed error by @twelch in #388
  • callWithRetry - add optional ifErrorMsgContains string, will now retry only if thrown error message contains that string otherwise. add exponential backoff so that there is a an increasing wait before the next retry.
  • rasterStats no longer catches and swallows all errors. Improved to catch specific error thrown when a zero polygon is passed to geoblaze, and returns default raster stats, otherwise it rethrows the error.
  • add back node-fetch polyfill to cog and flatgeobuf to solve SocketError showing up during high volume network call situation (e.g. smoke tests).
• Upgrade to latest CDK lib and Node 22 lambda runtime by @twelch in #392

📃 Docs

• Update toolbox page with new overlap functions and deep links to API docs.

Full Changelog: v7.0.0-beta.11...v7.0.0-beta.12

v7.0.0-beta.11

What's Changed

💥 Breaking Changes

Users on upgrade should look at preprocessing functions now installed with new project and borrow their code. See global datasources for datasource listing.

• Drop genClipLoader, replaced with clipToPolygonFeatures/genClipToPolygonFeatures and clipToPolygonDatasources/genClipToPolygonDatasources. Default preprocessing functions have migrated to using clipToPolygonFeatures and fetching features in a separate step.
• Drop JSON format from data import and precalc. It's no longer required now that precalc reads fgb from disk and fgb is superior for other uses.

👎 Deprecated

• Deprecate getFeatures (in name) and replace with same functionality in getDatasourceFeatures. This makes it clear that this is a datasource aware function.

🚀 New Feature / Improvement

• getDatasourceFeatures now accepts optional sketch option. When used it calls optimized getFeaturesForSketchBBoxes
• getFeaturesForSketchBBoxes add support for all geometry types and allow user to extend them with generics support
• Add lower-level getFeaturesForBBoxes that takes an array of bounding boxes
• loadFgb: add retry on error with exponential backoff (3 times). This will hopefully avoid the Closed socket error seeing in testing environment, seemingly in high volume request situation.

🏠 Internal

• Drop old doc build workflow by @twelch in #382
• Drop global datasources from base-project. No longer included with new project.
• Contribute squares.fgb, squares.json with easy to reason about named polygons for testing. Develop getFeaturesForSketchBBoxes e2e tests to use them.
• Update squareSketches fixtures to match squares.fgb/json for testing.
• Add squaresReadme.txt ascii art image to see the layout of the polygons

📃 Docs

• Drop macos direct install support from docs by @twelch in #383
• Delete chunked cache results on clear:results by @avmey in #387

Full Changelog: v7.0.0-beta.10...v7.0.0-beta.11

v7.0.0-beta.10

What's Changed

Be sure to run the upgrade command after updating to this latest beta version. On upgrade:

• genRandomSketch script will be removed in favor of the new genRandomPolygon that supports both regular features and sketches. See sample tutorial for examples.

💥 Breaking Changes

• drop deprecated loadCogWindow and bboxToPixelEdge functions
• drop overlapRaster function
• re-export dataprovider in top-level library instead of having as a separate export (now that esm is here and flatgeobuf issues have been fixed), migrate base project to use it.
• drop genPreprocessor function. Replace with clipToPolygonFeatures, genClipToPolygonFeatures, clipToPolygonDatasources and genClipToPolygonDatasources.
• drop genRandomSketch, merge into genRandomFeature and refactor with more options
• drop use of node-fetch for the global fetch offered by Node. There is a "Socket closed" error that occurs when a high number of requests in test environment. This should be fixed with an upgrade to Node v22 (to come).

👎 Deprecated

• deprecated fgbFetchAll function for loadFgb
• deprecated genClipLoader function, no longer needed now that genClipToPolygonFeatures and genClipToPolygonDatasources is available
• deprecated use of bbox and planning area properties in basic.json.
• deprecated template-ocean-eez. template-blank-project is now installed by default. It may simply get merged with base-project before 7.0 release. createReport has been found to be a better way of letting developer install a variety of report types.

🚀 New Feature / Improvement

• speed up storybook, merge existing project prettierignore with base on upgrade by @twelch in #361
• Add docusaurus website by @twelch in #362
• Fix precalc cleanup script by @avmey in #363
• Fix multiple choice sketch attributes bug by @avmey in #368
• ADA Accessibility by @avmey in #370

• add validatePolygon preprocessor, with ensureValidPolygon helper function
• add roundDecimalFormat and numberFormat helper functions
• add toSketchProperties helper function
• add getFeaturesForSketchBboxes function with split feature, add splitBBoxAntimeridian and cleanBBox helpers,
• make MetricGroup type property optional.
• export intersection functions - intersectInChunks, intersectInChunksArea, intersectSum
• add getMetricGroupDatasource and getMetricGroupClassKey methods to projectClientBase
• improve metricGroupSchema in schemas.json

project init:

• drop bbox, planning area, and template questions.
• remove template-ocean-eez from workspace so no longer tested (likely to be removed in future)
• clean up simple and blank reports (function + cards)
• drop ViabilityPage from TabReport and simply inline all report pages
• dataImport: conditional ask additional properties question
• sort init language alphabetically
• add vitest-fetch-mock dep to base project
• migrate to childProperties
• add validatePolygon preprocessor
• add minSize and enforceMinSize options to preprocessors
• drop global datasources from basic-project template except for osm-land and eez-land-union
• remove default objective and metricgroup from project. objectives.json and metrics.json start as empty arrays now. This is inline with them being opt-in features rather than forced use.

createReport:

• switch vectorFunction to use getFeaturesForSketchBboxes.
• migrate vectorFunction/rasterFunction/OverlapCard from using NullSketch in result to get sketch properties to using useSketchProperties
• createFunction and createReport now default to async geoprocessing function. The user often doesn't know which they need right away.

🐛 Bug Fix

• fix genRandomPolygon name bug
• fix i18nAsync bug using base english strings instead of letting component render its own
• fix storybook error when sketch name has spaces or any characters not valid in a Javascript variable name
• remove .story-cache folders from dist build
• genRandomPolygon - start id's at 1, was causing a bug downstream with 0 id value

🏠 Internal

• rename split.js to antimeridian.js.
• storybook no longer auto-opens web browser, caused error in geoprocessing WSL.
• migrate to loadFgb internally from deprecated fgbFetchAll
• upgrade to flatgeobuf 3.36.0 and drop use of web streams polyfill and node-fetch now that bug fixed.
• add Vite babel plugins for storybook
• Remove antimeridian tests from getFeatures
• add getFgbPath method to projectClientBase for loading of flatgeobuf from local filesystem
• move all createReport assets over to now unused template-ocean-eez directory. eventually need to simplify this.

📃 Docs

• Fix 6.1.0 tutorial, referenced features not available in 6.x. Also some issues with ordering of steps.
• Major refactor and updates to Next docs, complete with systemsetup, sampleproject and newproject tutorials, as well as a number of guides.
• remove codespaces as documented tutorial option

Full Changelog: v7.0.0-beta.9...v7.0.0-beta.10

v7.0.0-beta.9

What's Changed

When you npm run upgrade

• you can expect a lot of changes to installed code due to use of the eslint unicorn plugin internally in the geoprocessing library.
• Project languages are now defined in project/basic.json and the English language code EN will have been added. If there are additional languages that your project offers translation, then manually add their language codes. You'll find the master list in src/i18n/languages.json.
  This will configure the translation publish and import commands to work with these languages. Some old i18n files such as src/i18n/supported.ts will be deleted, with the addition of src/i18n/languages.ts.

💥 Breaking Changes

• Allow user to limit project i18n languages, change watch to build:watch by @twelch in
• Some CLI commands have been renamed. watch is now build:watch, clear-all-results has been removed, and clear-results is now clear:results and you can enter the option to delete all cached results. This has the benefit of limiting accidental deletes of the full cache, you have to specifically state you want to delete all.

🚀 New Feature / Improvement
#356

• Add eslint unicorn plugin with fixes by @twelch in #355
• Add Brazilian Portuguese to supported languages by @avmey in #354
• upgrade: remove old translation scripts by @twelch in #360

🐛 Bug Fix

• init: fix setting package version, gpVersion override by @twelch in #359

Full Changelog: v7.0.0-beta.8...v7.0.0-beta.9

v7.0.0-beta.8

What's Changed

Running upgrade on your project will:

• Install and configure Prettier along with a new lint-staged command. If you are using VSCode you should already have the Prettier extension installed but this addition makes formatting rules consistent across developer machines and will auto-format all staged files on commit if not already.

💥 Breaking Changes

• Separate worker stacks and lock functions to one stack solving thrashing issue causing Duplicate resource error on deploy by @twelch in #353
• Require explicit registration of worker functions, fix CDK circular dependency error by @twelch in #352

To register workers used by a geoprocessing function, simply pass a workers option to your GeoprocessingHandler with a list of the worker titles.

export default new GeoprocessingHandler(kelpPersist, {
  title: "boundaryOverlap",
  description: "boundary overlap",
  executionMode: "async",
  workers: ['boundaryOverlapWorker']
});

The architecture has changed from beta.7 for nested LambdaStacks. Worker functions are now deployed to separate stacks than non-workers. This should allow multiple geoprocessing functions to register and use the same worker without issue.

Upgrade will require destroying your deployed project first. Steps in order include:

• npm run upgrade
• npm run destroy
• npm run build && npm run deploy
• npm run publish:data

🚀 New Feature / Improvement

• Support childProperties for SketchCollection by @twelch in #341

🐛 Bug Fix

• npm run url command now properly outputs all service URLs and bucket names used by the project for easy reference.
• Fix clear-all-results command. Add retry to dynamodb bulk delete resolving ThroughputError, break out helper functions by @twelch in #344

🏠 Internal

• Disable task cache for worker lambdas by default, improve logging by @twelch in #342
• Add support for ESLint and Prettier by @twelch in #348

Full Changelog: v7.0.0-beta.7...v7.0.0-beta.8

v7.0.0-beta.7

What's Changed

💥 Breaking Changes

• Switch tasks class to storing dynamodb results as JSON string chunks, drop correlationId and peerDependencies by @twelch in #330 and #336

A new algorithm is used for storing function results in DynamoDB that works regardless of the structure of your JSON. After deploy, you will need to clear your existing cache by running npm run clear-all-results

There are no longer devDependencies installed for a project whether using init or upgrade. On your next upgrade, all dependencies will be added as dependencies and you will need to manually remove them from devDependencies because upgrade does not handle deletion.

🚀 New Feature / Improvement

• Migrate to aws-sdk v3 by @twelch in #321
• Add runLambdaWorker helper for invoking sync worker functions by @avmey in #326

🐛 Bug Fix

• Generate clipped bounding boxes when using clipToGeography by @avmey in #325
• Fix task caching and improved logging by @twelch in #336

🏠 Internal

• update dependencies by @twelch in #304 and #324
• Add additional logging to GeoprocessingHandler to be able to assess functions are working as expected.

Full Changelog: v7.0.0-beta.6...v7.0.0-beta.7

v7.0.0-beta.6

What's Changed

💥 Breaking Changes

• Migrate lambdas to NestedStack to avoid cdk 500 resource limit. Scales to add more stacks as needed automatically by @twelch in #309 and #314. The breaking change is that due to fixed logical ID's being used for each lambda function, a duplicate ID error will result when redeploying an existing project and CDK tries to create a function in a nested LambdaStack that already exists in the root stack. It doesn't appear to take care of the deletion first. Until a solution can be found, you will need to completely destroy and redeploy your project.

See Migrating guide for more information.

🚀 New Feature / Improvement

• Overcome dynamoDB 400KB item size limit by splitting metric results across one or more additional metricGroup DB items in a way that auto-scales @twelch in #313

🐛 Bug Fix

• Fix template path bug on new project init by @twelch in #308
• Handle no-overlap for categorical rasters, creating proper zero metrics by @avmey in #315
• bug: task metricGroup results not saved when just one group by @twelch in #319

🏠 Internal

• Add node 22 to CI tests in prep for future use by @twelch in #305
• End-to-end test of project build + CDK synth for a variety of scenarios by @twelch in #314

Full Changelog: v7.0.0-beta.5...v7.0.0-beta.6