[GITFLOW]merging 'release-1.67.0' into 'master'

.github/workflows/ci.yml

@@ -6,17 +6,21 @@ name: Java CI with Maven
 env:
   JAVA: 11
   PRIVILEGED_RUN: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/development') || github.event.pull_request.head.repo.full_name == github.repository }}
-
+  CODEQL_LANGUAGES: 'java' # FIXME(@JonasCir) add 'javascript'
 on:
   push:
     branches: [ development, master, hotfix* ]
   pull_request:
     branches: [ development, hotfix* ]
-
+  workflow_dispatch: # run it manually from the GH Actions web console
+  schedule:
+    - cron: '35 1 * * 0'
 jobs:
-  test:
-    name: mvn verify
+  ci:
+    name: SORMAS CI
     runs-on: ubuntu-latest
+    permissions:
+      security-events: write
 
     steps:
       - name: Checkout repository (with token)
@@ -35,6 +39,11 @@ jobs:
         if: ${{ !fromJSON(env.PRIVILEGED_RUN) }}
         uses: actions/checkout@v2
 
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v1
+        with:
+          languages: ${{ env.CODEQL_LANGUAGES }}
+
       - name: Set up JDK ${{ env.JAVA }}
         uses: actions/setup-java@v1
         with:
@@ -56,6 +65,9 @@ jobs:
         working-directory: ./sormas-base
         run: mvn verify -B -ntp
 
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v1
+
       - name: Commit external visits API spec to development
         # Privileged action needing a secret token. Since this only runs on development in our own repo
         # the token will be available through a privileged checkout.
@@ -64,12 +76,17 @@ jobs:
         run: |
             git config --global user.name "sormas-vitagroup"
             git config --global user.email "[email protected]"
+          
+            mkdir /tmp/openapi
+            cp sormas-rest/target/external_visits_API.* /tmp/openapi
+            
+            git fetch
+            git checkout development
+            git pull
 
             rm -rf openapi
-            mkdir openapi/
-            cp sormas-rest/target/external_visits_API.* openapi
+            cp -r /tmp/openapi .
 
             git add openapi
             git commit -m "[GitHub Actions] Update external visits API spec files"
-            git pull --rebase
             git push

.github/workflows/skip_required_status.yml

@@ -0,0 +1,14 @@
+# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
+name: Skip Required Status
+
+on:
+  push:
+    branches: [development]
+
+jobs:
+  mergefreeze:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Make mergefreeze pass if we want to commit external visits API spec to development
+        if: github.event_name == 'push' && github.ref == 'refs/heads/development'
+        run: 'echo "Skipping mergefreeze check for bot"'

README.md

@@ -33,7 +33,7 @@ Please also make sure that you've read the [*Development Contributing Guidelines
 If you want to report a **security issue**, please read and follow our [*Security Policies*](docs/SECURITY.md). For bugs without security implications, change and feature requests, please [create a new issue](https://github.com/hzi-braunschweig/SORMAS-Project/issues/new/choose) and
 read the [*Submitting an Issue*](docs/CONTRIBUTING.md#submitting-an-issue) guide for more detailed instructions. We appreciate your help!
 
-### Which Browsers and Android Versions Are Supported?
+### Which Browsers and Android Versions are Supported?
 SORMAS officially supports and is tested on **Chromium-based browsers** (like Google Chrome) and **Mozilla Firefox**, and all Android versions starting from **Android 7.0** (Nougat). In principle, SORMAS should be usable with all web browsers that are supported by Vaadin 8 (Chrome, Firefox, Safari, Edge, Internet Explorer 11; see <https://vaadin.com/faq>).
 
 Making use of the SORMAS web application through a mobile device web browser is possible and acceptable also in countries that are subject to the General Data Protection Regulation (GDPR) as enforced by the European Union. However, in such countries that are subject to the GDPR, the Android application (.apk file) for SORMAS should not be used on mobile devices until further notice.
@@ -42,6 +42,11 @@ Making use of the SORMAS web application through a mobile device web browser is
 Yes! Please download the [latest release](https://github.com/hzi-braunschweig/SORMAS-Project/releases/latest) and copy the content of /deploy/openapi/sormas-rest.yaml to an editor that generates a visual API documentation(e.g. <https://editor.swagger.io/>).
 A runtime Swagger documentation of the External Visits Resource (used by external symptom journals such as CLIMEDO or PIA) is available at ``<<host>>/sormas-rest/openapi.json`` or ``<<host>>/sormas-rest/openapi.yaml``
 
+### Who is responsible for Data Protection and Data Security?
+We herewith explicitly would like to draw your attention to the fact, that the respective public health agency running SORMAS is in charge of data security and data protection and has to ensure compliance with national data protection and data security regulations in their respective jurisdiction.
+It has to ensure that state-of-the art requirements for data protection and data security are fulfilled. All those prerequisites and examinations have to be done in the context of the country and its respective legal framework.
+For these reasons, HZI cannot take the responsibility from the respective public health agency running the SORMAS systems and is not liable for any violation of data protection of the agency as the data generated by SORMAS belong to that very agency.
+
 <p align="center"><img src="https://user-images.githubusercontent.com/23701005/74659600-ebb8fc00-5194-11ea-836b-a7ca9d682301.png"/></p>
 
 ## Guidelines and Resources

docs/DEVELOPMENT_ENVIRONMENT.md

@@ -126,3 +126,25 @@ Optional, but strongly recommended:
 6. M2_HOME need to be set. By default, for newer version, it is set to MAVEN_HOME. But Ant script is looking for M2_HOME
 
 7. For eclipse formatted plugin, there is an issue for Idea: <https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter> - `cannot save settings Path to custom eclipse folder is not valid` - it works only when settings were saved from down to up. And not vice versa.
+
+## Avoid redeployment problems
+
+**Problem**: Due to currently a not mitigated problem, it is only possible to deploy the `sormas-ear.ear` (contains `sormas-backend`) once without problems. If you undeploy it and deploy `sormas-ear.ear` again, the other artifacts `sormas-ui`and `sormas-rest` cannot successfully call the backend.
+
+**Workaround**: Undeploy `sormas-ear` and all other sormas artifacts, restart the Payara domain, deploy `sormas-ear` again (the same or changed version).
+
+**Symptom**: This exception occurs when `sormas-ui` or `sormas-rest` calls the `sormas-backend`.
+```java
+Caused by: java.lang.IllegalArgumentException: Can not set java.util.Properties field de.symeda.sormas.backend.common.ConfigFacadeEjb.props to de.symeda.sormas.backend.common.ConfigFacadeEjb
+    at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
+    at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
+    at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
+    at java.base/jdk.internal.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:75)
+    at java.base/java.lang.reflect.Field.set(Field.java:780)
+   at com.sun.enterprise.container.common.impl.util.InjectionManagerImpl._inject(InjectionManagerImpl.java:594)
+```
+
+**Additional info**:
+- You can undeploy and deploy all other modules without restarting the Payara domain, as long as nothing changes on `sormas-ear` (implicates `sormas-api` and `sormas-backend`).
+- The problem occurs no matter if you deploy directly from your IDE or as packaged ears/wars into the autodeploy directory.
+- Related ticket: #2511

docs/SERVER_SETUP.md

@@ -142,13 +142,13 @@ Keycloak can be set up in two ways:
 
 **Setup**
 
-Setting Keycloak up as a standalone installation [Server Installation and Configuration Guide](https://www.keycloak.org/docs/11.0/server_installation/#installation)
-* Make sure to configure Keycloak with PostgreSQL Database [Relational Database Setup](https://www.keycloak.org/docs/11.0/server_installation/#_database)
+Setting Keycloak up as a standalone installation [Server Installation and Configuration Guide](https://www.keycloak.org/docs/16.1/server_installation/#installation)
+* Make sure to configure Keycloak with PostgreSQL Database [Relational Database Setup](https://www.keycloak.org/docs/16.1/server_installation/#_database)
 * Set up an Admin User
-* Copy the `themes` folder content to `${KEYCLOAK_HOME}/themes` [Deploying Themes](https://www.keycloak.org/docs/11.0/server_development/#deploying-themes)
-* Deploy the `sormas-keycloak-service-provider` [Using Keycloak Deployer](https://www.keycloak.org/docs/11.0/server_development/#using-the-keycloak-deployer)
+* Copy the `themes` folder content to `${KEYCLOAK_HOME}/themes` [Deploying Themes](https://www.keycloak.org/docs/16.1/server_development/#deploying-themes)
+* Deploy the `sormas-keycloak-service-provider` [Using Keycloak Deployer](https://www.keycloak.org/docs/16.1/server_development/#using-the-keycloak-deployer)
 * Update the `sormas-base/setup/keycloak/SORMAS.json` file by replacing the following placeholders: `${SORMAS_SERVER_URL}`, `${KEYCLOAK_SORMAS_UI_SECRET}`, `${KEYCLOAK_SORMAS_BACKEND_SECRET}`, `${KEYCLOAK_SORMAS_REST_SECRET}`
-* Create the SORMAS Realm by importing `sormas-base/setup/keycloak/SORMAS.json` see [Create a New Realm](https://www.keycloak.org/docs/11.0/server_admin/#_create-realm)
+* Create the SORMAS Realm by importing `sormas-base/setup/keycloak/SORMAS.json` see [Create a New Realm](https://www.keycloak.org/docs/16.1/server_admin/#proc-creating-a-realm_server_administration_guide)
 * Update the `sormas-*` clients by generating new secrets for them
 * Update the realm's email settings to allow sending emails to users
 
@@ -179,7 +179,7 @@ Then update `sormas.properties` file in the SORMAS domain with the property `aut
 *after setting up Keycloak as one of the described options above*
 
 In case Keycloak is set up alongside an already running instance of SORMAS, these are the steps to follow to make sure already existing users can access the system:
-1. Manually create an admin user in Keycloak for the SORMAS realm [Creating a user](https://www.keycloak.org/docs/11.0/getting_started/index.html#creating-a-user) *(username has to be the same as admin's username in SORMAS)*
+1. Manually create an admin user in Keycloak for the SORMAS realm [Creating a user](https://www.keycloak.org/docs/16.1/server_admin/#proc-creating-user_server_administration_guide) *(username has to be the same as admin's username in SORMAS)*
 2. Login to SORMAS and trigger the **Sync Users** button from the **Users** page
 3. This will sync users to Keycloak keeping their original password - see [SORMAS Keycloak Service Provider](sormas-keycloak-service-provider/README.md) for more information about this
 

docs/SERVER_UPDATE.md

@@ -3,6 +3,9 @@
 SORMAS releases starting from 1.21.0 contain a script that automatically updates and deploys the server. If you are using an older version and therefore need to do a manual server update, please download the 1.21.0 release files and use the commands specified in the server-update.sh script.
 
 ## Preparations
+Note: At some versions it is mandatory to switch to a new Payara Server. If your version bump does apply to the listing below, please proceed with [Payara migration](SERVER_UPDATE.md#how-to-migrate-to-new-payara-server).
+* Switching from <=v1.66.4 to v1.67.0 or newer
+
 Note: You can skip this step if you've just set up your SORMAS server and have already downloaded the latest release.
 
 * Get the latest release files (deploy.zip) from <https://github.com/hzi-braunschweig/SORMAS-Project/releases/latest>
@@ -56,7 +59,66 @@ These are the default users for most user roles, intended to be used on developm
 
 ### Standalone installation
 
-Upgrading from Keycloak 11 to 12 following the steps from here <https://www.keycloak.org/docs/latest/upgrading/#_upgrading>
+Upgrading from Keycloak 12 to 16 following the steps from here <https://www.keycloak.org/docs/16.1/upgrading/#_upgrading>
+
+*16.1.0 doesn't provide a way to upgrade host based installations as there were a lot of changes due to the Wildfly update <https://www.keycloak.org/docs/16.1/upgrading/#migrating-to-16-0-0>*
+
+To update follow this steps:
+
+1. Prerequisites
+* Backup the DB
+* Backup the current Keycloak configuration
+* Download 16.1.0 zip from <https://www.keycloak.org/downloads>
+* Extract everything from the archive somewhere on your disk (will call this `KEYCLOAK_HOME_16`)
+
+2. From you current installation (will call this `KEYCLOAK_HOME_12`) directory copy the following into the new installation
+* Copy directory `KEYCLOAK_HOME_12/themes/sormas` over to `KEYCLOAK_HOME_16/themes`
+* Copy directory `KEYCLOAK_HOME_12/modules/system/layers/keycloak/org/postgresql` over into `KEYCLOAK_HOME_16/modules/system/layers/keycloak/org`
+* Copy `KEYCLOAK_HOME_12/standalone/deployments/sormas-keycloak-service-provider-*.jar` over to `KEYCLOAK_HOME_16/standalone/deployments`
+
+3. Edit the `KEYCLOAK_HOME_16/standalone/configuration/standalone.xml`
+* Search for `java:jboss/datasources/KeycloakDS` and you should find something like this
+```xml
+<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
+```
+* Replace it's content with the content from `KEYCLOAK_HOME_12/standalone/configuration/standalone.xml` and you should end up with something like this
+```xml
+<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
+    <connection-url>jdbc:postgresql://host:5432/keycloak-db-name</connection-url>
+    <driver>postgresql</driver>
+     <pool>
+         <max-pool-size>20</max-pool-size>
+     </pool>
+     <security>
+          <user-name>keycloak-db-username</user-name>
+          <password>keycloak-db-password</password>
+      </security>
+ </datasource>
+```
+* Make sure that you replace `keycloak-db-name`, `keycloak-db-username` and `keycloak-db-password` with your actual values
+* In the section `<drivers>` bellow, add also this option
+```xml
+<driver name="postgresql" module="org.postgresql">
+    <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
+</driver>
+```
+* You should end up with something like this
+
+```xml
+<drivers>
+    <driver name="postgresql" module="org.postgresql">
+        <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
+    </driver>
+    <driver name="h2" module="com.h2database.h2">
+        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
+    </driver>
+</drivers>
+```
+4. Start Keycloak
+* Database will be migrated automatically
+
+
+Upgrading from Keycloak 11 to 12 following the steps from here <https://www.keycloak.org/docs/12.0/upgrading/#_upgrading>
 
 1. Stop the old server and make sure to remove any open connections to the DB
 2. Backup the DB *(once the upgrade is done the old version cannot be used with the new DB version)*
@@ -76,3 +138,26 @@ The docker installation is automatically upgraded to the latest version specifie
 **Prerequisites:** Make sure the DB is backed up, because once the upgrade is done the new DB won't be usable with the old version of Keycloak.
 
 For more info see the [Keycloak Docker Documentation](https://github.com/hzi-braunschweig/SORMAS-Docker/blob/development/keycloak/README.md).
+
+## How to migrate to new Payara Server
+
+### Step 1: Shutdown existing domain
+```bash
+# Stop domain
+service payara-sormas stop
+
+# Move existing domain
+DOMAIN_PATH=/opt/domains
+DOMAIN_NAME="sormas"
+DOMAIN_BACKUP_NAME="sormas_backup"
+mv $DOMAIN_PATH/$DOMAIN_NAME $DOMAIN_PATH/$DOMAIN_BACKUP_NAME
+```
+
+### Step 2: Setup Payara domain
+Please follow the [server setup](SERVER_SETUP.md#sormas-server): Create the payara domain under the same path as before, use the same directory paths and the same database settings.
+
+### Step 3: Apply your config file changes
+Transfer your settings from `sormas.properties`, `logback.xml` or changes in the domain setup. Use the new provided files and copy your changes in, don't reuse old files!
+
+### Step 4: Install new SORMAS version
+To install the new SORMAS version in the Payara domain, proceed with the [automatic update](SERVER_UPDATE.md#automatic-server-update) or for developers: Deploy SORMAS via the IDE as usual.

docs/TROUBLESHOOTING.md

@@ -87,3 +87,14 @@ If the problem occurred right after you've pulled new code from GitHub, your saf
 ### News Feeds Polling
 
 When running eclipse with JDK 11, you might encounter the following error message: `An internal error occurred during: "Polling news feeds".  javax/xml/bind/JAXBContext`. To fix it, disable `Window --> Preferences --> General --> News --> "Enable automatic news polling"`.
+
+## Redeployment problems
+
+If you face problems that `sormas-ui` or `sormas-rest` cannot call the backend anymore after redeploying, please follow [this instruction](DEVELOPMENT_ENVIRONMENT.md#avoid-redeployment-problems).
+
+## Malware detection triggers
+It might happen that a defensive program on your system falsely recognizes files needed to run SORMAS as vulnerability.
+
+Please ignore the following known findings (no quarantine, no deletion):
+* File: payara-5.2021.10.zip, Recognized: Trojan:Script/Oneeva.A!ml (found by Windows Defender). Has rarely happened when running server-setup.sh which downloads that file. The script subsequently fails because zip file cannot be extracted.
+* File: glassfish/modules/war-util.jar, Recognized: Exploit:Java/CVE-2012-0507.D!ldr (found by Windows Defender in payara-5.2021.10). The deployed OSGi bundle might also be recognized, for example under this path: osgi-cache/felix/bundle365/version0.0/bundle.jar . If the file is quarantined, the paraya domain fails to start, without any exception in the log.