IDEA Release 3.1.7 (#162)

IDEA Release 3.1.7

See CHANGELOG

.gitattributes

@@ -1,3 +1,3 @@
 *.css linguist-vendored
 *.scss linguist-vendored
-*.js linguist-vendored
+*.js linguist-vendored

.github/renovate.json

@@ -0,0 +1,40 @@
+{
+    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+    "extends": ["config:best-practices"],
+    "assignees": ["cfsnate"],
+    "baseBranches": ["/^dev-.*/"],
+    "pip_requirements": {
+      "fileMatch": ["requirements\\/.*\\.(in)$"]
+    },
+    "labels": ["renovate"],
+    "packageRules": [
+      {
+        "matchDatasources": ["pypi"],
+        "labels": ["python"]
+      },
+      {
+        "matchDatasources": ["npm"],
+        "labels": ["node"]
+      },
+      {
+        "matchDepTypes": ["optionalDependencies"],
+        "addLabels": ["optional"]
+      },
+      {
+        "matchPackageNames": ["prompt-toolkit"],
+        "allowedVersions": "3.0.36"
+      },
+      {
+        "matchPackageNames": ["sqlalchemy"],
+        "allowedVersions": "1.4.52"
+      },
+      {
+        "matchPackageNames": ["typeguard"],
+        "allowedVersions": "2.13.3"
+      },
+      {
+        "matchPackageNames": ["urllib3"],
+        "allowedVersions": "1.26.18"
+      }
+    ]
+  }

.gitignore

@@ -9,6 +9,7 @@ open-source/
 deployment/global-s3-assets/
 deployment/regional-s3-assets/
 
+
 # C extensions
 *.so
 
@@ -191,3 +192,10 @@ codebuild_build.sh
 
 # Test outputs
 integration-test-results/
+
+package.json
+package-lock.json
+yarn.lock
+
+region_ami_config_backup_*.yml
+base-software-stack-config_backup_*.yaml

CHANGELOG.md

@@ -4,6 +4,75 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.1.7] - 2024-06-01
+
+### Notes
+* This upgrade does require an update to the global settings. Please review [Global Settings Upgrade](https://docs.idea-hpc.com/first-time-users/cluster-operations/update-idea-cluster/update-idea-backend-resource#global-settings-backup-and-upgrade) before upgrading.
+* Pre-upgrade script `scripts/pre-upgrade-317.sh` will assist by outputting the cluster settings command to update Base OS AMI as well as rename occurrences in settings for schedule `STOP_ALL_DAY` to `STOP_ON_IDLE`.
+* Schedule Update script included in scripts to be able to update existing sessions to `STOP_ON_IDLE` via API
+
+### Features
+* First truly Open Source IDEA Release!
+* Documentation combined into IDEA monorepo under `docs` folder
+* Multi Architecture Administrator Docker Image
+  * Added build-push-multi task to use buildx to create the IDEA Admin container and auto-push to ECR
+* Added DCV Checks in the eVDI Auto Power off event. This will check CPU, DCV, and SSH connections based on the idle time setup before stopping eVDI hosts
+  * Renamed schedule `STOP_ALL_DAY` to `STOP_ON_IDLE` to better represent stop schedule behavior
+  * Set default schedule for all days to `STOP_ON_IDLE`
+* Login page will show a Login With SSO button when SSO is enabled.
+
+### Changes
+* Update AWS CDK from `2.95.1` to `2.317.0`
+* Update Node from `16.20.2` to `18.20.2`
+* Update Python from `3.9.18` to `3.9.19`
+* Update NVM from `0.39.5` to `0.39.7`
+* Update NPM from `9.8.1` to `10.5.2`
+* Update OpenMPI from `4.1.5` to `5.0.3`
+* Update EFA Driver from `1.25.1` to `1.31.0`
+* Update DCV and components from `2023.0` to `2023.1`
+* Update NVIDIA GPU drivers used during installation
+  * LTSB from  `470.199.02` to `470.239.06`
+  * Production `535.104.05` to `550.54.15`
+* Update all AMIs for Base OS and Software Stacks. Update Amazon Linux 2 AMI to Kernel 5.10
+  * Pre-Upgrade script can assist in updating AMI in DDB Settings. Software Stacks will remain untouched during upgrade
+* Implement Renovate for dependency tracking
+  * Update all applicable Python packages per Renovate best-practices config
+    * `sanic` left at `23.6.0` for socket authorization issues in `ideactl`
+  * Dependency pinning and some dependency updates for Cluster Manager Web App node packages
+* Added gcc and python3-dev to Docker image for building Python requirements
+* Removed AWS Corporate ECR dependency in `idea-admin.sh`
+* New Active Directory users will get their account name applied to the givenName and sn attributes to satisfy directory import syncing with Okta requirements
+* Updated Help menu in Cluster Web App to direct issues to GitHub
+* Update references to old Github Repo
+* Update references to old Docs URL
+* Added AMI update scripts in `scripts/dev` to update AMI versions for base and software stacks. These will be worked into devtool at a later time
+
+### Bug Fixes
+* Fixed FSx for Lustre allowed size mismatch between AWS and IDEA
+  * This likely needs a re-work for different FSx Lustre types.
+* Fixed NVIDIA GPU Driver install for Amazon Linux 2 with Kernel 5.10
+* Remove trailing whitespace from ENI IDS for tagging in instance bootstrap
+* Fixed HPC Job Name pydantic validation to allow strings, integers, and floats instead of just strings for HPC job names submitted from CLI
+* Set PBS config on eVDI Nodes to use the Rt53 scheduler record instead of the ec2 hostname. This will enable existing eVDI nodes to access PBS with an upgraded scheduler with a new ip.
+* Fix root storage size calculation for new eVDI nodes with hibernation enabled
+* Fix resuming Hibernated eVDI nodes
+* Fixed ID Token claims for SSO enabled clusters. Added Cognito pre token generation Lambda to stuff custom attributes to ID token
+* Fix License Server Check script permissions
+  * Cluster home directory `/apps/idea-<cluster name>` from `700` to `701` to allow traverse to script directory
+* Fix Home Dir permissions for New Users from `700` to `710` to allow file sharing when using `Add User to My Group` feature
+* Fix scratch storage mount permissions for HPC jobs on Linux
+* In `idea-admin.sh` sso configuration, multiple scopes can be submitted seperated by `,` this will be replaced with a space in Cognito
+
+### Known Caveats
+* Data model was updated in 3.1.6, devtool web-portal.typings needs to be re-worked to support changes in Pydantic v2 and IDEA data model
+* Developer documentation needs some re-work / updates
+* When using OpenLDAP and doing an upgrade, replacement of the directoryservice instance removes existing directory entries. This can be avoided by backing up and restoring post upgrade or using ideactl on cluster manager to sync groups and users from DynamoDB into OpenLDAP. Generally there is not an explicit need to upgrade the directoryservice module.
+* After upgrading the scheduler module, job id's reset to 0. This is purely cosmetic and does not affect functionality. However, users tracking analytics by job id may want to be aware of this upon upgrade.
+* Tests need some re-work
+* Node dependencies need updating in more places. Cluster Manager Web App
+* Python needs updating. 3.9 is EOL October 2025
+
+
 ## [3.1.6] - 2023-10-20
 
 ### Features

CONTRIBUTING.md

@@ -11,7 +11,7 @@ information to effectively respond to your bug report or contribution.
 
 We welcome you to use the GitHub issue tracker to report bugs or suggest features.
 
-When filing an issue, please check [existing open](https://github.com/awslabs/solution-for-scale-out-computing-on-aws/issues), or [recently closed](https://github.com/awslabs/solution-for-scale-out-computing-on-aws/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
+When filing an issue, please check [existing open](https://github.com/cfs-energy/idea/issues), or [recently closed](https://github.com/cfs-energy/idea/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
 reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
 
 * A reproducible test case or series of steps
@@ -41,7 +41,7 @@ GitHub provides additional document on [forking a repository](https://help.githu
 
 
 ## Finding contributions to work on
-Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/solution-for-scale-out-computing-on-aws/labels/help%20wanted) issues is a great place to start.
+Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/cfs-energy/idea/labels/help%20wanted) issues is a great place to start.
 
 
 ## Code of Conduct
@@ -51,11 +51,11 @@ [email protected] with any additional questions or comments.
 
 
 ## Security issue notifications
-If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
+If you discover a potential security issue in this project we ask that you notify CFS Security via email [[email protected]](mailto:[email protected]). Please do **not** create a public github issue.
 
 
 ## Licensing
 
-See the [LICENSE](https://github.com/awslabs/solution-for-scale-out-computing-on-aws/blob/main/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+See the [LICENSE](https://github.com/cfs-energy/idea/blob/main/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
 
 We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

IDEA_VERSION.txt

@@ -1 +1 @@
-3.1.6
+3.1.7

README.md

@@ -1,12 +1,37 @@
 # Integrated Digital Engineering on AWS
 
+**Read Me First**
+
+IDEA is **NOT** supported by AWS. Do **NOT** expect support from AWS for IDEA. If you're looking for a supported solution, please look to AWS RES.
+
+AWS has decided to take a different direction with IDEA and pursue RES (https://github.com/aws/res). Until RES achieves feature parity with IDEA, this repository will be the authoritative source for IDEA updates, patches, and information.
+
+This project is NOT AWS RES (https://github.com/aws/res). RES is a fork of IDEA and not all IDEA features made it to RES. Specifically HPC and other features. RES Documentation is available on https://docs.aws.amazon.com/res/latest/ug/overview.html
+
+This code was originally written by AWS but has been abandoned and taken open source as of version 3.1.6 respecting the original Apache 2.0 license terms.
+
 ## Documentation
 
-https://docs.ide-on-aws.com/
+https://docs.idea-hpc.com/
+
+## What is IDEA?
+
+Integrated Digital Engineering on AWS (IDEA) is an evolution of SOCA ([Scale Out Computing on AWS](https://awslabs.github.io/scale-out-computing-on-aws/)). IDEA empowers teams of engineers, scientists and researchers with a cloud environment to host engineering tools required for end-to-end product development workloads.
+
+IDEA supports two main computation workstyles. **eVDI** (Virtual Desktops) & **Scale Out Compute / HPC** (OpenPBS based job submission)
+
+IDEA provides all the backend infrastructure to support these worksryles on AWS with a simple to use web interface, SSO, and more advanced user features.
+
+IDEA can be a good solution for any compute or VDI workflow but often specializes in:
+* Computer Aided Design – CAD
+* Computer Aided Engineering - CAE
+* Model Based Systems Engineering - MBSE
+* Electronic Design Automation – EDA Using this solution, research and development (R&D) leaders enable engineers and designers to break-through development silos and accelerate the product development process.
+* Large distributed compute and simulation jobs. 1-100k+ CPU and GPU cores. CFD/EFA/etc
 
 ## Installation
 
-Refer to [IDEA Installation](https://docs.ide-on-aws.com/idea/first-time-users/install-idea) for installation instructions.
+Refer to [IDEA Installation](https://docs.idea-hpc.com/first-time-users/install-idea) for installation instructions.
 
 This solution collects anonymous operational metrics to help AWS improve the quality of the solution.
 

THIRD_PARTY_LICENSES.txt

@@ -1,5 +1,5 @@
 ** cloudwatch-fluent-metrics; version 0.5.2 -- https://github.com/awslabs/cloudwatch-fluent-metrics
-** botocore; version 1.27 -- https://github.com/boto/botocore
+** botocore; version 1.34.84 -- https://github.com/boto/botocore
 
 Apache License
 Version 2.0, January 2004
@@ -320,7 +320,7 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 ------
 
-** sanic; version 22.3.2 -- https://sanicframework.org/
+** sanic; version 23.6.0 -- https://sanicframework.org/
 Copyright (c) 2016-present Sanic Community
 
 MIT License

deployment/ecr/idea-administrator/Dockerfile

@@ -1,4 +1,8 @@
-FROM public.ecr.aws/docker/library/python:3.9.18-slim
+FROM public.ecr.aws/docker/library/python:3.9.19-slim@sha256:fd8693fc6115345452a19654c1388bc9a1029cda7b98e1bce45ede67e74190f0
+
+ARG TARGETARCH
+
+ENV OS_ARCH="${TARGETARCH:-amd64}"
 
 WORKDIR /root
 
@@ -8,6 +12,8 @@ RUN apt-get update && \
     tar \
     unzip \
     locales \
+    gcc \
+    python3-dev \
     && apt-get clean
 
 
@@ -21,20 +27,39 @@ RUN sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen \
     && dpkg-reconfigure locales
 
 # install aws cli
-RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
-    unzip -qq awscliv2.zip && \
+RUN if [ "${OS_ARCH}" = "amd64" ]; then \
+        curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" ; \
+    elif [ "${OS_ARCH}" = "arm64" ]; then \
+        curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip" ; \
+    fi && \
+    unzip awscliv2.zip && \
     ./aws/install && \
     rm -rf ./aws awscliv2.zip
 
+
 # install node and npm
-RUN NODE_MAJOR=16 && rm -f /etc/apt/trusted.gpg.d/nodesource.gpg /etc/apt/sources.list.d/nodesource.list \
-    && apt-get install -y ca-certificates curl gnupg \
-    && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/trusted.gpg.d/nodesource.gpg \
-    && echo "deb [signed-by=/etc/apt/trusted.gpg.d/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list \
-    && echo "Package: nodejs" >> /etc/apt/preferences.d/preferences \
-    && echo "Pin: origin deb.nodesource.com" >> /etc/apt/preferences.d/preferences \
-    && echo "Pin-Priority: 1001" >> /etc/apt/preferences.d/preferences\
-    && apt update && apt-get install -y nodejs
+# RUN NODE_MAJOR=16 && rm -f /etc/apt/trusted.gpg.d/nodesource.gpg /etc/apt/sources.list.d/nodesource.list \
+#     && apt-get install -y ca-certificates curl gnupg \
+#     && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/trusted.gpg.d/nodesource.gpg \
+#     && echo "deb [signed-by=/etc/apt/trusted.gpg.d/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list \
+#     && echo "Package: nodejs" >> /etc/apt/preferences.d/preferences \
+#     && echo "Pin: origin deb.nodesource.com" >> /etc/apt/preferences.d/preferences \
+#     && echo "Pin-Priority: 1001" >> /etc/apt/preferences.d/preferences\
+#     && apt update && apt-get install -y nodejs
+# install nvm and node
+RUN set -uex && \
+    apt-get update && \
+    apt-get install -y ca-certificates curl gnupg && \
+    mkdir -p /etc/apt/keyrings && \
+    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key \
+    | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
+    NODE_MAJOR=18 && \
+    echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" \
+    > /etc/apt/sources.list.d/nodesource.list && \
+    apt-get update && \
+	apt-get install nodejs -y
+
+
 
 # add all packaged artifacts to container
 ARG PUBLIC_ECR_TAG