Skip to content

Releases: OasisLMF/OasisPlatform

Release 2.3.12

17 Jan 10:51
@awsbuild awsbuild
2.3.12
39b1de9
Compare
Choose a tag to compare
Release 2.3.12 Latest
Latest

Oasis Release v2.3.12

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

  • #1063 - Expose components versions in API

OasisLMF Changelog - 2.3.12

  • #1394 - Net RI losses do not use -z in summarycalc
  • #1618 - Fix testing failures (API Client)
  • #1611 - fix loss_out len in account level back allocation
  • #1615 - Non-useful error message for missing PolInceptionDate with RA Basis reinsurance

ODS_Tools Changelog - 3.2.10

  • #167 - Fix default for do_disaggregation in schema
  • #172 - CI Fix - branches renamed in OED repo

Release Notes

OasisPlatform Notes

Expose components versions in API - (PR #1152)

Add ods-tools version and oed spec version to the following endpoints:

  • /server_info : Version of these components on the server.
  • /models/{id}/versions: Versions of these components on model workers.

Issue: #1063

OasisLMF Notes

Calculate summarycalc reinsurance without all zeros flag - (PR #1601)

Running summarycalc with the -z flag for Net RI was running into significant performance issues.

To resolve this, this PR:

  • removes the output all zeros flag when running summarycalc or summarypy
  • adds an additional check on the the mean_idx to catch non-zeros that were previously being filtered out

Issue: #1394

Fix testing failures (API Client) - (PR #1618)

Objected returned from a mocked API post has changed between versions of the responses
Pinning to responses<=0.25.3 for now, but test should be updated later

Fix fmpy back allocation for account level term - (PR #1611)

Trim loss_out to the correct length before account level term back allocation

fixes error message for missing PolInceptionDate - (PR #1614)

For RA based reinsurance introduced in 2.3.11 OasisLMF/OasisLMF#1576
Solution is to look for both non '_x' and '_x' variants of these column numbers in the row if they exist and output them.

acc_info = {
    field: row[f'{field}_x'] if f'{field}_x' in row else row[f'{field}']
    for field in RISK_LEVEL_FIELD_MAP[oed.REINS_RISK_LEVEL_ACCOUNT]
    if f'{field}_x' in row or f'{field}' in row
}

Should output the correct error message as shown in the example

Error: PolInceptionDate missing for {'PortNumber': '1', 'AccNumber': 'A11111'}, cannot use AttachmentBasis [RA]. Please check the account file

ODS_Tools Notes

Fix default for do_disaggregation to true - (PR #169)

Match schema default to the oasislmf package

CI Fix - branches renamed in OED repo - (PR #172)

Assets 5
Loading

Release 2.3.11

28 Nov 18:07
@awsbuild awsbuild
2.3.11
01b1861
Compare
Choose a tag to compare
Release 2.3.11

Oasis Release v2.3.11

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.3.11

  • #1125 - Support Cyber models on OasisPlatform
  • #1134 - Fixed pip install on model worker images
  • #1139 - Fix/1135 model settings conf
  • #1129 - Platform schema generation - minor changes between versions.
  • #1140 - Add DB indexing to fields like Analyses status.
  • #1147 - Release notes script - skip merge commits

OasisLMF Changelog - 2.3.11

  • #1394 - Net RI losses do not use -z in summarycalc
  • #1250 - Support Risk Attaching 'RA' basis in reinsurance
  • #1581 - oasislmf code uses legacy correlation settings location in model settings
  • #1589 - Update platform API client for Cyber models
  • #1594 - improve the memory performance of il layer number continuity step
  • #1595 - Allow process perils with different resolution grids

ODS_Tools Changelog - 3.2.9

  • #148 - PLA options to be added into MS/AS schemas explicitly
  • #151 - Package does not include the validation files

Release Notes

OasisPlatform Notes

Add cybermodels support in the OasisAPI - V1 Only - (PR #1133)

  • Only for run_mode = V1 allow input generation to work without posting a location.csv file.
  • Update the check in portfolios to allow execution if Either a location or an account file is attached.

Fixed pip install on model worker images - (PR #1134)

Moving to ubuntu 24.04 uses a version of python with a stricter package manager. Added an env variable to base image as a workaround.

Removed the conf.ini variable MODEL_SETTINGS_FILE from run params - (PR #1139)

In run_mode = v1 the conf.ini MODEL_SETTINGS_FILE variable can override the oasislmf.json
model_settings path. This value has been removed from the run_params. its not ONLY used for a workers auto-registration and the only source of the model_settings path is given by the oasislmf config.

Fix API schema change, missing description fields - (PR #1143)

Issue seems to come from the update to package drf-yasg==1.21.8 from drf-yasg==1.21.5.
Note: this problem dosn't show on main only stable/2.3.x

Fix/1140 status db indexing (stable 2.3.x) - (PR #1145)

  • Added DB indexing to analyses/sub-task status fields #1140
  • Added CI testing for DB migration upgrades
  • Fixed DB migration clashes from main branch
  • Fixed CVE-2023-43804

Release notes script - skip merge commits - (PR #1147)

  • Update repo scraper to skip any merged commits from release notes

OasisLMF Notes

Calculate summarycalc reinsurance without all zeros flag - (PR #1601)

Running summarycalc with the -z flag for Net RI was running into significant performance issues.

To resolve this, this PR:

  • removes the output all zeros flag when running summarycalc or summarypy
  • adds an additional check on the the mean_idx to catch non-zeros that were previously being filtered out

Issue: #1394

Support Risk Attaching 'RA' basis in reinsurance - (PR #1576)

Implemented #1250 as shown below. Policies which meet the RA requirement are not set to the PASSTHROUGH_PROFILE_ID, and left as NO_LOSS_PROFILE_ID.
UseReinsDates is currently not considered for checking RA, only AttachmentBasis (LO/RA).

Added fix for correlation settings - (PR #1587)

  • Fixed #1581, where oasislmf code was looking for correlation in the legacy location, it now checks both.
  • Added testing to catch problems loading correlation from model settings.

Update Platform API client for Cyber models - (PR #1589)

Cyber models can run with only an account file, so no OED location input, updated API client to allow for this.

OOM issue for accounts with lots of layers for several levels - (PR #1594)

Improve the memory performance of level matching between levels of accounts that have lots of layers.

Feature - Unique grid resolution for each areal_peril - (PR #1596)

This PR does the following:

  • Introduces a new builtin lookup function fixed_size_geo_grid_multi_peril which allows for a unique resolution grid per peril.
  • Fixes an issue where fixed_size_geo_grid created area_peril_id starting at index 0. It now starts at index 1 as expected.

Related Issue: #1595

ODS_Tools Notes

Added lossfactor set options to settings files - (PR #162)

The oasislmf package accepts the pla_loss_factor_set from OasisLMF/OasisLMF#1558 Added these as options in the settings schema files (analysis_settings / model_settings) following the same format as other indexed files in the static dir lossfactors_{idx}.bin

Fix package build to include all ODTF data files - (PR #165)

Fixed setup.py so the package build adds all files under odtf/data/*

Assets 5
Loading

Release 1.28.10

27 Nov 11:37
@awsbuild awsbuild
1.28.10
c795f40
Compare
Choose a tag to compare
Release 1.28.10

Oasis Release v1.28.10

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 1.28.10

  • #1141 - Add setttings from platform v2 for running the v1.28.x server on azure cloud
  • #1146 - Fix CVE issues in release 1.28.10

OasisLMF Changelog - 1.28.10

  • #1506 - acc_idx read as category type from input/accounts.csv
  • #1556 - Fix CI build system failures
  • #1566 - Fiona package vulnerability issue
  • #1599 - Fixes for CI testing stable 1.28.x

Release Notes

OasisPlatform Notes

Add setttings from platform v2 for running the v1.28.x server on azure cloud - (PR #1141)

  • Added a fix to Django settings to avoid permission denied errors when upload files to azure files
  • Fixed 'Network error' when access swagger via an ingress using https

Fix CVE issues in release 1.28.10 - (PR #1146)

  • Added DB migration CI check
  • Updated Oasislmf to 1.28.10
  • Update python packages for CVE fixes

OasisLMF Notes

Fix CI build system failures - (PR #1556)

  • Updated GH actions to use artifact v4
  • Pinned Fiona package (the newer package has incompatibility with some older versions of Geopandas)
    update to geopandas 0.14.4 or pin fiona to version 1.9.6

Remove pin of fiona package - (PR #1565)

A package pin of fiona causes CVE issues for published docker images

Fixes for CI testing stable 1.28.x - (PR #1599)

  • Remove numpy max version pin and fix plat2 testing
Assets 3
Loading

Release 2.3.10

31 Oct 17:57
@awsbuild awsbuild
2.3.10
4576ad5
Compare
Choose a tag to compare
Release 2.3.10

Oasis Release v2.3.10

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.3.10

  • #1122 - Update python packages for 2.3.9
  • #1124 - Update docker base images and python packages
  • #1123 - Generate oasis files is not using the pre-analysis adjusted location file for V2 runs
  • #1125 - Support Cyber models on OasisPlatform

OasisLMF Changelog - 2.3.10

  • #1563 - Intensity Adjustments in gulmc for dynamic footprints
  • #1585, #1575 - Fix missing complex keys return without amplification
  • #1578 - fix: point to the correct source in error message for vulnerabiity_id…
  • #1580 - Hazard correlation defaults to 100% if missing
  • #1581 - oasislmf code uses legacy correlation settings location in model settings

ODS_Tools Changelog - 3.2.8

  • #139 - Look for validation files in custom paths
  • #147 - fix fill_empty method for field that end in XX or YYY
  • #157 - Both correlation_settings fields are marked as "Legacy" in desc

Release Notes

OasisPlatform Notes

Update python packages for 2.3.9 - (PR #1122)

Update docker base images and python packages - (PR #1124)

  • Update docker base images to ubuntu LTS 24.04
  • regenerate python package requirements files
  • Load CVE scanning DB from Github actions cache (Trivy)

Fixed pre-analysis exposure files not working in V2 worker runs - (PR #1130)

Add cybermodels support in the OasisAPI - V1 Only - (PR #1133)

  • Only for run_mode = V1 allow input generation to work without posting a location.csv file.
  • Update the check in portfolios to allow execution if Either a location or an account file is attached.

OasisLMF Notes

Fix missing complex keys return without amplification - (PR #1574)

  • Fix from complex keys return when running without PLA
  • Added testing for complex keys return + logging output

Fix vulnerability_id missing error message - (PR #1578)

Fix the error message so it points to the correct source when some vulnerability_id are missing from the vulnerability file.

Fix: undo 100% correlation if no correlation setting provided in model settings - (PR #1584)

The hazard_group_id where not generated if the correlation settings were not provided, causing a 100% correlation across all location.
This fix assure the hazard_group_id are generated and used correctly in gulmc.

Added fix for correlation settings - (PR #1587)

  • Fixed #1581, where oasislmf code was looking for correlation in the legacy location, it now checks both.
  • Added testing to catch problems loading correlation from model settings.

ODS_Tools Notes

Validation files in custom paths - (PR #138)

When a custom path is passed in the configuration file, validation files in that folder will be found and used.

fix default values for field ending in XX or YYY - (PR #147)

Some OED field end with XX or YYY to allow users to specify several type of the same field (for example for GeogSchemeXX)
Because of the mismatch between the column name and the field name (GeogScheme1 vs GeogSchemeXX), default value were not applied.

Fixed 'correlation_settings' description in model settings - (PR #157)

There are two locations of correlation_settings one is Legacy which at the root of the file. The new version is within the model_settings attribute (fixed the description field to avoid confusion

Assets 5
Loading

Release 2.3.9

08 Oct 18:32
@awsbuild awsbuild
2.3.9
f0f1653
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 2.3.9

Oasis Release v2.3.9

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.3.9

  • #1105 - Release 2.3.8
  • #1122 - Update python packages for 2.3.9

OasisLMF Changelog - 2.3.9

  • #1567 - Occurrence file not found when requesting output from ktools component alt_meanonly

  • #1563 - Intensity Adjustments in gulmc for dynamic footprints

  • #1379 - Performance issue in get_exposure_summary

  • #1557 - Release 2.3.8

  • #1558 - support for having several pla sets

  • #1560 - add check for vulnerability id and intensity bin boundary

  • #1566 - Fiona package vulnerability issue

  • #139 - Look for validation files in custom paths

Release Notes

OasisPlatform Notes

Update python packages for 2.3.9 - (PR #1122)

OasisLMF Notes

Fix performance issues with get exposure summary - (PR #1550)

No Return keys were included in keys errors file, but this was ballooning to a huge size when AA1 peril code was used, since we would see no returns for all peril codes. This was causing the performance of the generate exposure summary report to degrade massively.

The fix removes the no-returns from the returned errors, but counts them in the exposure summary.

support for having several pla sets - (PR #1558)

Allow user to specify a different pla set from the default one by adding loss_factors_set in the model_settings part of analysis_settings.json
if "loss_factors_set": "2" is added
the loss factor that will be taken into account will be lossfactors_2.bin in the static folder

Fix out of bound issue when vulnerability file contains more intensity bin than the footprint - (PR #1560)

In the model part, we want to only load intensity that will be relevant in the footprint, however there was an issue in the code where the out of bound intensity was overwriting the other part of the vulnerability table that was created.
Also add a check to verify that all vulnerability_id that are needed are present in the vulnerability static file

Remove pin of fiona package - (PR #1565)

A package pin of fiona causes CVE issues for published docker images

ODS_Tools Notes

Validation files in custom paths - (PR #138)

When a custom path is passed in the configuration file, validation files in that folder will be found and used.

Assets 5
Loading

Release 2.3.8

20 Sep 10:51
@awsbuild awsbuild
2.3.8
9fbcfe4
Compare
Choose a tag to compare
Release 2.3.8

Oasis Release v2.3.8

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.3.8

  • #1075 - Logging - V2 input generation and losses tar should include log files for all sub-tasks
  • #1104 - Fix params for V1 workers - so custom hooks are called
  • #1107 - Fix/gen log storage
  • #1109 - set artifact to v4
  • #1111 - Keycloak OIDC group permistions is broken.
  • #1110 - Add Endpoint for models - 'storage_links'

OasisLMF Changelog - 2.3.8

  • #1544 - dynamic footprint - slow performance for large hazard case
  • #1546 - Combus changes: 20240807
  • #1554 - drop duplicate il term lines before merging
  • #1547 - Running full_correlation + gulmc causes an execution to hang.
  • #1556 - Fix CI build system failures
  • #1559 - Read do_disaggregation from settings files
  • #1560 - add check for vulnerability id and intensity bin boundary
  • #1561 - Set Ktools 3.12.4
  • #1531 - fix effective deductible applied in minded calculation for calcrule 19

ODS_Tools Changelog - 3.2.7

  • #135 - Only run attach project issues workflow from Our repo - not forks
  • #136 - Triple quoting when saving fields containing multiple entries
  • #139 - Look for validation files in custom paths
  • #140 - Failed ODTF transformation with lark > 1.1.9
  • #142 - Update GH actions artifacts to v4
  • #143 - pin lark version
  • #145 - Add disaggregation-flag

ktools Changelog - v3.12.4

  • #387 - Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows
  • #390 - Average Loss Convergence Table (ALCT) fix
  • #391 - Fix GH actions artifact

Release Notes

OasisPlatform Notes

Fix docker compose UI - (PR #1099)

Fix log storage in result tar files. - (PR #1103)

  • Fixed log storage for input generation (run_mode=v1). workers now store a copy of run logs in the input.tar.gz file.
  • Fixed log storage for input generation (run_mode=v2). Workers copy in each sub-task log file to a log/ within a the input.tar.gz file
  • Fixed mismatch between v1 and v2 ktools log stored under /run_log_file/, v2 no longer nests the archive in a logs sub-dir.

Fix run params for V1 workers - (PR #1104)

Needed so that custom hooks are correctly called for workers in run_mode=V1, similar to #1087

Fix/gen log storage - (PR #1107)

Worker storage manager between main and stable/2.3.x is different, fix needed to work with both

Fixed group mapper class in Keyclock realm config - (PR #1112)

Issue #1111 - the users group information wasn't included in the JSON Web Token (JWT). This causes OasisAPI objects to be created without any groups attached.

Added model storage_links endpoint - (PR #1113)

Added the following endpoints to models, /v1/models/{id}/storage_links/ and /v2/models/{id}/storage_links/.
works in the same way as the other storage_links endpoints on (models and portfolios), returns the real filename or Object Store key of the model settings file.

example return

{
  "settings_file": "82cd9b3e16c44954a1ec98be6554440b.json"
}

OasisLMF Notes

Reduce memory usage in il generation step - (PR #1554)

Drop duplicated lines containing each specific level terms before merging with the input level do reduce memory consumption

Disable full_correlation when running gulmc - (PR #1555)

Gulmc provide a more fine grained correlation feature and make full_correlation flag not useful and created an issue where the run would hang
This change disable full_correlation in that case fixing the hanging issue.

Fix CI build system failures - (PR #1556)

  • Updated GH actions to use artifact v4
  • Pinned Fiona package (the newer package has incompatibility with some older versions of Geopandas)
    update to geopandas 0.14.4 or pin fiona to version 1.9.6

Read do_disaggregation from settings json files - (PR #1559)

Workaround to address ticket #1497 to give more time for creating a better option for overriding settings OasisLMF/OasisLMF#1552

Fix out of bound issue when vulnerability file contains more intensity bin than the footprint - (PR #1560)

In the model part, we want to only load intensity that will be relevant in the footprint, however there was an issue in the code where the out of bound intensity was overwriting the other part of the vulnerability table that was created.
Also add a check to verify that all vulnerability_id that are needed are present in the vulnerability static file

Fix issue in calcrule 19 when min deductible is triggered - (PR #1531)

When % of loss deductible is smaller than the minimum deductible, neither the minimum deductible nor the % of loss deductible gets applied. This overstates gross losses.
the deductible passed to the min deductible calculation was incorrect for calcrule 19, as we passed the percentage instead of the actual deductible, this fix the issue by passing the effective deductible for this calcrule.

ODS_Tools Notes

Only run attach project issues workflow from Our repo - not forks - (PR #135)

CI fix for external PRs

Fixes triple quoting when strings contain special characters - (PR #137)

... Strings containing special symbols (for example ;, ,, \t, \n) are now correctly quoted (only once with double quotes).

Validation files in custom paths - (PR #138)

When a custom path is passed in the configuration file, validation files in that folder will be found and used.

Fix failed transformations with lark > 1.1.9 - (PR #141)

Compatibility with all lark versions

Update GH actions artifacts to v4 - (PR #142)

Pin lark version to <=1.9 - (PR #143)

While testing the solution, lark is pinned to a working version to avoid failing tests.

Add do_disaggregation to setting schemas - (PR #145)

ktools Notes

Average Loss Convergence Table (ALCT) fix - (PR #390)

  • Changed the calculation of StandardError which is used for the confidence intervals for the AAL estimate to be equal to StandardErrorVuln from the ANOVA metrics
  • Reintroduced the ANOVA fields from the first version of the report
  • Set the lower and upper confidence threshold to zero where StandardError=0 (for SampleSize 1)
Assets 5
Loading

Release 1.28.9

19 Sep 14:24
@awsbuild awsbuild
1.28.9
268d0aa
Compare
Choose a tag to compare
Release 1.28.9

Oasis Release v1.28.9

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 1.28.9

  • #1117 - Updated Package Requirements

OasisLMF Changelog - 1.28.9

  • #1518 - Fix/pre analysis user dir
  • #1554 - drop duplicate il term lines before merging
  • #1547 - Running full_correlation + gulmc causes an execution to hang.
  • #1556 - Fix CI build system failures

Release Notes

OasisLMF Notes

Added user_data_dir to pre and post analysis hooks - (PR #1518)

Path for custom user assets added to hook class parameters

Reduce memory usage in il generation step - (PR #1554)

Drop duplicated lines containing each specific level terms before merging with the input level do reduce memory consumption

Disable full_correlation when running gulmc - (PR #1555)

Gulmc provide a more fine grained correlation feature and make full_correlation flag not useful and created an issue where the run would hang
This change disable full_correlation in that case fixing the hanging issue.

Fix CI build system failures - (PR #1556)

  • Updated GH actions to use artifact v4
  • Pinned Fiona package (the newer package has incompatibility with some older versions of Geopandas)
    update to geopandas 0.14.4 or pin fiona to version 1.9.6
Assets 3
Loading

Release 1.27.10

19 Sep 14:23
@awsbuild awsbuild
1.27.10
56a1810
Compare
Choose a tag to compare
Release 1.27.10

Oasis Release v1.27.10

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 1.27.10

  • #1116 - Updated Package Requirements

OasisLMF Changelog - 1.27.10

  • #1474 - Use billiard package for keys multiprocess if available
  • #1514 - manual merge of PR 1509 improve perf of file preparation
  • #1547 - Running full_correlation + gulmc causes an execution to hang.
  • #1556 - Fix CI build system failures

Release Notes

OasisLMF Notes

Use billiard package for keys multiprocess in workers - (PR #1474)

Needed for #994 otherwise running a keys lookup with multiprocessing will throw an exception:

[2024-03-14 12:19:54,399: ERROR/ForkPoolWorker-1] generate_input[d60b57a2-f8f9-4794-8f0a-831380a44ea0]: daemonic processes are not allowed to have children

improve perf of file preparation - (PR #1514)

  • improve speed of key server id missing check
  • improve gul loss generation speed and memory usage in exposure run

Disable full_correlation when running gulmc - (PR #1555)

Gulmc provide a more fine grained correlation feature and make full_correlation flag not useful and created an issue where the run would hang
This change disable full_correlation in that case fixing the hanging issue.

Fix CI build system failures - (PR #1556)

  • Updated GH actions to use artifact v4
  • Pinned Fiona package (the newer package has incompatibility with some older versions of Geopandas)
    update to geopandas 0.14.4 or pin fiona to version 1.9.6
Assets 3
Loading

Release 2.3.7

06 Aug 12:08
@awsbuild awsbuild
2.3.7
1b8d539
Compare
Choose a tag to compare
Release 2.3.7

Oasis Release v2.3.7

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.3.7

  • #1090 - Fixed CVEs from fiona package (backport)
  • #1091 - Fixed build error from worker-controller image
  • #1057, #1092 - Fix/1092 task cancellation issue
  • #1095 - Updated Package Requirements: twisted==24.7.0rc1
  • #1077 - FIx missing exception trace in V1 workers
  • #1076 - Fix return of types of create 'analyses' and 'model' POST in API spec.
  • #1081 - Fix/api responses v2
  • #1085 - Validation errors when posting a run incorrectly update an anaysis state to RUN_ERROR
  • #1071 - Add new custom code hooks into the V2 workflow

OasisLMF Changelog - 2.3.7

  • #1533 - API client names the downloaded output file .tar instead of .tar.gz
  • #1530 - Short flags unexpectedly changed in 2.3.6
  • #1539 - Allow keys files with both amplification_id and model_data columns
  • #1542 - Time and Memory performance issue for RI contract
  • #1531 - fix effective deductible applied in minded calculation for calcrule 19
  • #1532 - make all compute step run command oasis logged
  • #1468 - loss output at intermediate inuring priorities - new features
  • #1535 - add interval mapping to built in function

ODS_Tools Changelog - 3.2.6

  • #128 - Fix allow blank errors with OED v4 spec
  • #105 - Correlation Settings in Model settings
  • #129 - Explicitly select mapping file to be used (ODTF)
  • #133 - pla update
  • #134 - Update test for OED4 - test_field_required_allow_blank
  • #127 - Update readme

Release Notes

OasisPlatform Notes

Fixed CVEs from fiona package (backport) - (PR #1090)

Updated Package Requirements: fiona==1.10b1 for CVE-2023-45853 & CVE-2020-14152

Fixed build error from worker-controller image - (PR #1091)

A new version of setuptools fails with packaging<22.0, added new pin to requirements file.

From issue: pypa/setuptools#4483)

Fixed task cancellation issue when analysis is queued - (PR #1093)

Analyses cancellation silently fails when there are no workers on a model queue.

Example:

  1. POST /analyses/1/run/ ~ auto-scaller starts spinning up workers but none are ready yet, tasks on queue
  2. POST /analyses/1/cancel/ ~ analyses is marked as cancelled but sub-tasks are unrevoked and still on queue
  3. POST /analyses/1/run/ ~ a new set of sub-tasks are placed on queue.
  4. Bugs #1092 and #1057 occur

Fix

  • sub-tasks celery id's are stored by the task controller when a job is created
  • when a task is cancelled these are all marked as REVOKED when a task cancellation is issued.
  • The workers task guard check_task_redelivered has been updated to 'self kill' if a task detects itself as REVOKED
  • When set to a DEBUG log level the worker-controller lists all dispatched tasks, with linked celery id's
start_input_generation_task[-id-]: 'generate_inputs' - canvas dispatched, analyses=1, run_uuid=9b03e7bf29764b798d78abb59b81906b
start_input_generation_task[-id-]: Prepare input generation params = 7c74f4eb-4090-4238-953d-c04d47f8d25d
start_input_generation_task[-id-]: Pre analysis hook = af5feb49-0b62-41c6-a805-fd84ff854c2e                                                                                                                                                                    
start_input_generation_task[-id-]: Prepare keys file 0 = 2f326943-6f6d-412c-b4fb-0d007999e12a
start_input_generation_task[-id-]: Prepare keys file 1 = e6ebc3aa-06b3-4765-a4aa-98acdf3acece
start_input_generation_task[-id-]: Prepare keys file 2 = 5843cebb-9aea-44db-89c6-e4a99ed6d738
start_input_generation_task[-id-]: Prepare keys file 3 = 8d9367ec-7a56-4aab-83ea-b143117bcdc0
start_input_generation_task[-id-]: Prepare keys file 4 = 098e7dfd-e082-446e-918b-62387cf6d150
start_input_generation_task[-id-]: Prepare keys file 5 = e92d3730-46a0-4c5f-978a-ff25183f4186
start_input_generation_task[-id-]: Prepare keys file 6 = ae236118-96a5-4bac-af6f-6a5aa6c7b3aa
start_input_generation_task[-id-]: Prepare keys file 7 = f9842a6d-1b0e-45bf-a96c-1a1f9720832a
start_input_generation_task[-id-]: Prepare keys file 8 = 996344ae-c633-4215-920c-ae45debbc070
start_input_generation_task[-id-]: Prepare keys file 9 = e30240ff-7aa0-4648-9ba0-e9f631501839
start_input_generation_task[-id-]: Collect keys = ce1289f7-9a60-4fee-b019-0f3335c7cc04
start_input_generation_task[-id-]: Write input files = e88db9e3-2d7f-4883-88da-0da7e7bb0a8d
start_input_generation_task[-id-]: Record input files = 873b2872-b51e-4ebb-8524-53f826b0b1c7
start_input_generation_task[-id-]: Cleanup input generation = 7b152ff6-da9f-4118-a514-ee3d90d996bc

Updated Package Requirements: twisted==24.7.0rc1 - (PR #1095)

FIx missing exception trace in V1 workers - (PR #1077)

Fixed logger to show full trace when a V1 worker raises an exception

Fix API spec - return from create & update for 'analyses' and 'model' - (PR #1078)

Corrected the returns from /analyses/ and /model/ so that the settings resources are settings: <URL-to-file> and not settings: <JSON>

Note: This is the same behaviour as release prior to #1067

Fix api responses v2 - (PR #1081)

More fixes to align Analysis with AnalysisList

Fixed sever setting an analyses to an error state when submitting a bad request. - (PR #1086)

Fixed issue where analyses are marked as RUN_ERROR but no reason can be found in logs. This can happen due to issue #1085. Fixed by not setting ab analyses state to RUN_ERROR on validation issues.

Added new hoosk to v2 workflow (backport) - (PR #1087)

Added custom code hooks from OasisLMF/OasisLMF#1524

  • pre-analysis (unchanged) - runs in its own sub-task before keys lookup and modifies the exposure files.
  • post-generation-hook (new) - runs directly after write-oasis-files but before the inputs.tar.gz is created.
  • pre-losses-hook (new) - runs after the runs losses directory has been initialized, but before executing an analysis.
  • post-losses-hook (unchanged) - runs after the output calc has run, but before creating the results file output.tar.gz

OasisLMF Notes

Fix missing short flags from CLI - (PR #1538)

Fix for OasisLMF/OasisLMF#1530, new custom hooks from 2.3.6 caused the CLI to lose some flags.

Fix Cartesian product issue for RI Fac contract - (PR #1541)

With the introduction of all the scope oed column in the filter process of RI, the Fac contract had to have part of it's logic move to the filter level, however contrary to the other type of contract, we only use 1 layer_id for all FAC, this created a Cartesian product during the merge between info and scope.
this PR add some ...

Read more
Assets 5
Loading

Release 2.3.6

02 Jul 18:02
@awsbuild awsbuild
2.3.6
703fbe6
Compare
Choose a tag to compare
Release 2.3.6

Oasis Release v2.3.6

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.3.6

  • #1062 - Fix autoscaling ramping down when new analysis run is triggered
  • #1066 - Fix keycloak when running with Postgres Flexible server
  • #1056 - Passing analysis/model settings as a(n optional) parameter when creating a new analysis or model
  • #1068 - If conf.ini is missing a [celery] section OASIS_CELERY_BROKER_URL fails to load
  • #1072 - Always mark Sub-tasks as error when hitting problems

OasisLMF Changelog - 2.3.6

  • #1516 - Added support for occurrence attachment for Per Risk XL
  • #1517 - event ranges
  • #1518 - Fix/pre analysis user dir
  • #1520, #1519 - get_vulns update
  • #1258 - Document oasislmf.json configuration options
  • #1488 - Add post input-gen hook and pre exec hook
  • #1526 - Oasis Dynamic Model
  • #1529 - add user_data_dir to all the hooks

ODS_Tools Changelog - 3.2.5

  • #117 - Fix/fill empty
  • #114 - Add ODBC Connection support to ODTF
  • #120 - event ranges
  • #118 - Realign model and analys settings schema with actual options for correlation settings
  • #125 - Run transformation without config file
  • #126 - Set OED schema to 3.4.0

Release Notes

OasisPlatform Notes

Fix autoscaling ramping down when new analysis run is triggered - (PR #1062)

  • Fixed issue with auto-scaler send full queue state to web-socket when an analysis is submitted (not a filtered list)
  • Fixed bug in pre-analysis hook causing custom oed spec to be removed

Fix keycloak when running with Postgres Flexible server - (PR #1066)

Helm chart adjustment for running OasisAzureDeployment with OasisLMF/OasisAzureDeployment#23

Accept settings JSON on analysis or model creation - (PR #1067)

Models and Analyses V2 endpoints now accept settings json on POST and PATCH calls via an optional 'settings: { .. json ..}' field. These are validated before the new object is created, if valid the settings are loaded into the newly created (or updated) instance.

Example Analysis (POST / PATCH)

{
  "name": "string",
  "portfolio": 1,
  "model": 1,
  "settings": {
    "model_name_id": "PiWind",
    "model_supplier_id": "OasisLMF",
    "gul_threshold": 0,
    "gul_output": true,
    "model_settings": {
        "event_set": "p",
        "event_occurrence_id": "lt"
    },
    "gul_summaries": [
        {"id": 1, "eltcalc": true}
    ],
    "il_output": false,
    "il_summaries": [],
    "ri_output": 'foo',
    "ri_summaries": []
  }
}

Example Model (POST / PATCH)

{
  "supplier_id": "OasisLMF",
  "model_id": "PiWind",
  "version_id": "ver",
  "settings": {  
    "model_settings":{
        "event_set":{
            "name": "Event Set",
            "desc": "Piwind Event Set selection",
            "default": "p",
            "options":[
                {"id":"p", "desc": "Probabilistic", "number_of_events": 1447}
            ]
         },
        "event_occurrence_id":{
            "name": "Occurrence Set",
            "desc": "PiWind Occurrence selection",
            "default": "lt",
            "options":[
                {"id":"lt", "desc": "Long Term"}
            ]
        }
     },
    "lookup_settings":{},
    "model_default_samples" :10
  }
}

Ensure ConfigParser has required sections when not set in conf.ini file - (PR #1069)

Fix added - if sections are not set in a workers config file, environment variable will still load correctly.

Fix to handle exceptions which can't be serialized by celery - (PR #1073)

  • Celery can fail to log exceptions which are not pickable or json-serializable. Added exception logging within the V2 worker task file in case this happens (workaround).
  • Added check in worker-monitor v2 to make sure the trackback from sub-tasks is not set to None

OasisLMF Notes

Add support for occurrence attachment for a PerRisk XL (ReinsType=PR) - (PR #1516)

ReinsType 'PR' now supports both risk and occurrence attachment and limit, meaning that it can be used to model reinsurance with unusual features, like CatXL with risk terms for example.

Support for ranges of events in analysis settings - (PR #1517)

Adding support for additional ranges of events in analysis settings, in addition to the list of event ids which can already be provided.

Event id ranges should be submitted using the new "event_ranges" option in analysis settings

The string data will take the format "1-4,8,89-94" which will be parsed and an event list created.

It can be used in combination with the existing list, and unique values will be established

Added user_data_dir to pre and post analysis hooks - (PR #1518)

Path for custom user assets added to hook class parameters

oasislmf.json options now included in docs generated by sphinx - (PR #1490)

add post file gen and pre loss calculation step - (PR #1524)

Add the possibility for model developer to add two specific step in model run

  • post file gen: part of generate file, happen after the oasis files are generated (single worker on the platform)
  • pre loss : part of generate foll, happen before the loss are calculated (each loss calculation worker on the platform)

Oasis Dynamic Footprint Generation - (PR #1496)

Additional functionality added to generate footprint data in flow from underlying hazard maps and event definitions using the presented exposure data as a filter

New functionality

  • New Keys functionality to provide section_id as part of the keys which represents a sub-section of the geography of the model, aligned with the hazard map levels
  • model_data can also be included in non-complex model keys to house the data needed for defense adjustments
  • new hazard_case and event_definition model files accepted representing the hazard maps and event definitions
  • new functionality to read these files, filtered based on the locations in the exposure, and generate footprints

Outstanding tasks

  • overlapping sections - at the moment it is assumed that no sections will overlap, but there is a requirement for this within a single event, with functionality to allow the most severe return period to be chosen
  • adjustments for defenses - the framework is there to capture the data and present it at the relevant point, but the calculations do not yet use it
  • unit tests to be added to oasislmf. Manual testing has been done with OasisModels, but a small representative model needs to be added to unit tests

Add user_data_dir to all hooks - (PR #1529)

ODS_Tools Notes

improve performance and management of empty cells - (PR #117)

  • use np version of isin instead of pandas one, np version is faster and reliable for big dataframe
  • add function to get default value from column

DB as source for data transformations - (PR #119)

Data to be transformed can now be loaded from a database. Supported connections: mssql, postgres, sqlite

event range...

Read more
Assets 5
Loading