Database load pagination (#5183)

* Added script to run sequelizer in windows - updated README Signed-off-by: George Dias <[email protected]> * db pagination work Signed-off-by: George Dias <[email protected]> * pagination WIP Signed-off-by: George Dias <[email protected]> * WIP - almost done Signed-off-by: George Dias <[email protected]> * WIP - completed samples UI retrofit Signed-off-by: George Dias <[email protected]> * WIP - completed db UI retrofit Signed-off-by: George Dias <[email protected]> * WIP-Implementing the search capability logic, 50% completed Signed-off-by: George Dias <[email protected]> * WIP-Implementing the search capability logic, 85% completed Signed-off-by: George Dias <[email protected]> * Completed pagination, working on linting Signed-off-by: George Dias <[email protected]> * testing linting and e2e tests Signed-off-by: George Dias <[email protected]> * 2e2 tests updates Signed-off-by: George Dias <[email protected]> * added search instructions Signed-off-by: George Dias <[email protected]> * minor UI updates Signed-off-by: George Dias <[email protected]> * removed commented out code Signed-off-by: George Dias <[email protected]> * updated supported formats dialog Signed-off-by: George Dias <[email protected]> * added a bonus capability Signed-off-by: George Dias <[email protected]> * updates in response to sonalclould findings Signed-off-by: George Dias <[email protected]> --------- Signed-off-by: George Dias <[email protected]>
mitre · Jan 2, 2024 · 94ee29c · 94ee29c
1 parent c36f01c
commit 94ee29c
Show file tree

Hide file tree

Showing 39 changed files with 1,609 additions and 272 deletions.
diff --git a/.github/workflows/hdfconverter-tests.yml b/.github/workflows/hdfconverter-tests.yml
@@ -1,4 +1,4 @@
-name: Run HDF-Converters Tests
+name: Run OHDF-Converters Tests
 
 on:
   push:

diff --git a/.github/workflows/push-to-npm.yml b/.github/workflows/push-to-npm.yml
@@ -36,7 +36,7 @@ jobs:
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
 
-      - name: Publish HDF Converters to NPM
+      - name: Publish OHDF Converters to NPM
         if: always()
         run: npm publish --access public libs/hdf-converters/mitre-hdf-converters*.tgz
         env:

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -0,0 +1,25 @@
+{
+  "cSpell.words": [
+    "apikeygoeshere",
+    "casl",
+    "CREATEDB",
+    "DISA",
+    "distro",
+    "Evals",
+    "FEDRAMP",
+    "FISMA",
+    "fontface",
+    "headerapikey",
+    "headerprops",
+    "ldapauth",
+    "nestjs",
+    "nodesource",
+    "npmjs",
+    "openidconnect",
+    "prismjs",
+    "psql",
+    "srcset",
+    "Webserver",
+    "XCCDF"
+  ]
+}
diff --git a/CHANGELOG b/CHANGELOG
@@ -401,7 +401,7 @@ v2.8.1
 
 - Node Version Update to 18 - requires minimum of node v18 now @HenryXiaoHX @charleshu-8 @Amndeep7 (#3850)
 - Conveyor mapper updates @rbogren-brock (#4892)
-- Move HTML export feature to HDF Converters @charleshu-8 (#4834)
+- Move HTML export feature to OHDF Converters @charleshu-8 (#4834)
 
 ## Dependency Updates
 
@@ -491,9 +491,9 @@ v2.7.1
 v2.7.0
 
 - Unique Group Names @Rlin232 (#4722) - make sure to run the db migration scripts and be ready for your group names potentially being changed in order to enforce the uniqueness constraint
-- Using Semver Compare for ASFF "Previously HDF" Special Casing @Rlin232 (#4767)
+- Using Semver Compare for ASFF "Previously OHDF" Special Casing @Rlin232 (#4767)
 - Implemented OIDC external group import @akey77 (#3922)
-- Map XCCDF result value of "notselected" to HDF impact of 0 @candrews (#4758)
+- Map XCCDF result value of "notselected" to OHDF impact of 0 @candrews (#4758)
 - Remove jQuery import @charleshu-8 (#4752)
 
 ## Dependency Updates
@@ -538,7 +538,7 @@ v2.6.58
 
 - Group Descriptions @Rlin232 (#4695)
 - Reassign group owners upon user deletion @Rlin232 (#4403)
-- Improve xccdf_results_mapper when converting XCCDF Results to HDF Results @candrews (#4255)
+- Improve xccdf_results_mapper when converting XCCDF Results to OHDF Results @candrews (#4255)
 - Update Splunk links @charleshu-8 (#4716)
 
 

diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 ![Run E2E Backend + Frontend Tests](https://github.com/mitre/heimdall2/workflows/Run%20E2E%20Backend%20+%20Frontend%20Tests/badge.svg) ![Run Frontend Tests](https://github.com/mitre/heimdall2/workflows/Run%20Frontend%20Tests/badge.svg) ![Run Backend Tests](https://github.com/mitre/heimdall2/workflows/Run%20Backend%20Tests/badge.svg)
 
-This repository contains the source code for Heimdall's [Backend](https://github.com/mitre/heimdall2/tree/master/apps/backend), [Frontend (AKA Heimdall Lite)](https://github.com/mitre/heimdall2/tree/master/apps/frontend), [HDF Converters](https://github.com/mitre/heimdall2/tree/master/libs/hdf-converters), and [InSpecJS](https://github.com/mitre/heimdall2/tree/master/libs/inspecjs).
+This repository contains the source code for Heimdall's [Backend](https://github.com/mitre/heimdall2/tree/master/apps/backend), [Frontend (AKA Heimdall Lite)](https://github.com/mitre/heimdall2/tree/master/apps/frontend), [OHDF Converters](https://github.com/mitre/heimdall2/tree/master/libs/hdf-converters), and [InSpecJS](https://github.com/mitre/heimdall2/tree/master/libs/inspecjs).
 
 ## Contents
 
@@ -296,7 +296,13 @@ If you would like to change Heimdall to your needs, you can use Heimdall's 'Deve
      sudo apt install nano                        # recommended installation
      sudo npm install -g yarn
      ```
-
+
+   **NOTES** 
+
+     - The installation scripts setup_XX.x are no longer supported and are not needed anymore, as the installation process is straightforward for any RPM and DEB distro.
+
+     - See the [Debian and Ubuntu based distributions](https://github.com/nodesource/distributions#debian-and-ubuntu-based-distributions) nodesource for nodejs supported version and additional installation information
+
    OSX:
 
    - ```bash
@@ -305,6 +311,25 @@ If you would like to change Heimdall to your needs, you can use Heimdall's 'Deve
      sudo npm install -g yarn
      ```
 
+   WINDOWS:
+    - Install Node.js via MSI Installer
+      - Download the node release 18.xx installer (msi) from the [nodejs site](https://nodejs.org/en/blog/release)
+      - Open and run (double-click) the .msi file, the installation process begins, follow the installation instructions
+      - Node.js offers you options to install tools for native modules, we recommend checking the Automatically install the necessary tools check box.
+      - Verify the Node and npm version
+      ```shell
+      node --version 
+      npm --version
+      ```
+
+    - Install Yarn via MSI Installer
+      - Download the Yarn installation file from [GitHub](https://github.com/yarnpkg/yarn/releases/)
+      - Open and run the installation file, follow the installation instructions
+      - Run the following command in the PowerShell to verify the installation:
+      ```shell
+      yarn --version
+      ```
+
 2. Clone this repository:
 
    - ```bash
@@ -351,7 +376,31 @@ If you would like to change Heimdall to your needs, you can use Heimdall's 'Deve
       # Switch back to your original OS user
       exit
       ```   
-
+   WINDOWS:
+   - Start the postgres server base on the installation method
+     - Starting Postgres Server Using `net start`
+       ```sql
+       net start postgresql-[x32 or x64]-[version]
+       ```
+     - Starting Postgres Server Using `pg_ctl`
+       ```sql 
+       pg_ctl -D "C:\[path-to-postgres-installation]\PostgreSQL\[version]\data" start
+       ```
+     - Starting Postgres Server Using Services Manager
+       - Press the `win key + R` to launch the `Run` window.
+       - Type `services.msc` and hit the `OK` button to open the Services Manager:
+       - Search for `Postgresql-[x32 or x64]-[version]`, select the service, and hit the `Start/play` button to start
+   - Create the database user 
+     - Recommend using pgAdmin and follow instruction listed here 
+     - Open a postgres shell terminal (path to postgres executable directory must be set)
+       ```sql
+       # Start the terminal
+       psql -U postgres  
+       # Create the database user
+       CREATE USER <username> with encrypted password '<password>';
+       ALTER USER <username> CREATEDB;
+       \q
+       ```
 
 4. Install project dependencies:
 
@@ -360,13 +409,19 @@ If you would like to change Heimdall to your needs, you can use Heimdall's 'Deve
      yarn install
      ```
 
-5. Edit your apps/backend/.env file using the provided `setup-dev-env.sh` script. Make sure to set a DATABASE_USERNAME and DATABASE_PASSWORD that match what you set for the PostgresDB in step 3.
+5. Edit your apps/backend/.env file using the provided `setup-dev-env.sh or setup-dev-env.bat` script. Make sure to set a DATABASE_USERNAME and DATABASE_PASSWORD that match what you set for the PostgresDB in step 3.
 
-You can also open the apps/backend/.env file in a text editor and set additional optional configuration values. For more info on configuration values see [Enviroment Variables Configuration](https://github.com/mitre/heimdall2/wiki/Environment-Variables-Configuration).
+You can also open the apps/backend/.env file in a text editor and set additional optional configuration values. For more info on configuration values see [Environment Variables Configuration](https://github.com/mitre/heimdall2/wiki/Environment-Variables-Configuration).
 
 6. Create the database:
 
    - ```bash
+     # Windows
+     yarn backend sequelize-cli-windows db:create
+     yarn backend sequelize-cli-windows db:migrate
+     yarn backend sequelize-cli-windows db:seed:all
+
+     # All other OSs
      yarn backend sequelize-cli db:create
      yarn backend sequelize-cli db:migrate
      yarn backend sequelize-cli db:seed:all
@@ -382,7 +437,7 @@ This will start both the frontend and backend in development mode, meaning any c
 
 ### Debugging Heimdall Server
 
-If you are using Visual Studio Code, it is very simple to debug this application locally. First open up the Visual Studio Code workspace and ensure the [Node debuger Auto Attach](https://code.visualstudio.com/docs/nodejs/nodejs-debugging#_auto-attach) feature in Visual Studio Code is enabled. Next, open the integrated Visual Studio Code terminal and run:
+If you are using Visual Studio Code, it is very simple to debug this application locally. First open up the Visual Studio Code workspace and ensure the [Node debugger Auto Attach](https://code.visualstudio.com/docs/nodejs/nodejs-debugging#_auto-attach) feature in Visual Studio Code is enabled. Next, open the integrated Visual Studio Code terminal and run:
 
 ```
 yarn backend start:debug

diff --git a/apps/backend/package.json b/apps/backend/package.json
@@ -15,6 +15,7 @@
     "build": "nest build",
     "lint": "eslint \"{src,migrations,seeders,test}/**/*.ts\" --fix",
     "lint:ci": "eslint \"{src,migrations,seeders,test}/**/*.ts\" --max-warnings 0",
+    "sequelize-cli-windows": "node --experimental-loader ts-node/esm node_modules/sequelize-cli/lib/sequelize",
     "sequelize-cli": "node --experimental-loader ts-node/esm node_modules/.bin/sequelize",
     "start": "node dist/src/main",
     "start:debug": "nest start --debug --watch",

diff --git a/apps/backend/src/app.service.ts b/apps/backend/src/app.service.ts
@@ -14,13 +14,27 @@ export class AppService
     BeforeApplicationShutdown,
     OnApplicationShutdown
 {
+  private readonly line = '____________________________________________\n';
+  private colors = winston.addColors({
+    info: 'cyan',
+    warn: 'yellow',
+    error: 'red',
+    verbose: 'blue'
+  });
+
   public logger = winston.createLogger({
     transports: [new winston.transports.Console()],
     format: winston.format.combine(
+      winston.format.colorize({all: true}),
       winston.format.timestamp({
         format: 'MMM-DD-YYYY HH:mm:ss Z'
       }),
-      winston.format.printf((info) => `[${[info.timestamp]}] ${info.message}`)
+      winston.format.errors({stack: true}),
+      winston.format.align(),
+      winston.format.printf(
+        (info) =>
+          `${this.line}[${[info.timestamp]}] (App Service): ${info.message}`
+      )
     )
   });
 

diff --git a/apps/backend/src/authn/authn.service.ts b/apps/backend/src/authn/authn.service.ts
@@ -21,14 +21,18 @@ import {UsersService} from '../users/users.service';
 
 @Injectable()
 export class AuthnService {
+  private readonly line = '_______________________________________________\n';
   public loggingTimeFormat = 'MMM-DD-YYYY HH:mm:ss Z';
   public logger = winston.createLogger({
     transports: [new winston.transports.Console()],
     format: winston.format.combine(
       winston.format.timestamp({
         format: this.loggingTimeFormat
       }),
-      winston.format.printf((info) => `[${[info.timestamp]}] ${info.message}`)
+      winston.format.printf(
+        (info) =>
+          `${this.line}[${[info.timestamp]}] (Authn Service): ${info.message}`
+      )
     )
   });
 

diff --git a/apps/backend/src/casl/casl-ability.factory.ts b/apps/backend/src/casl/casl-ability.factory.ts
@@ -1,9 +1,9 @@
 import {
-  Ability,
   AbilityBuilder,
-  AbilityClass,
+  createMongoAbility,
   ExtractSubjectType,
-  InferSubjects
+  InferSubjects,
+  MongoAbility
 } from '@casl/ability';
 import {Injectable} from '@nestjs/common';
 import {Evaluation} from '../evaluations/evaluation.model';
@@ -13,7 +13,8 @@ import {User} from '../users/user.model';
 
 type AllTypes = typeof User | typeof Evaluation | typeof Group;
 
-type Subjects = InferSubjects<AllTypes> | 'all';
+type Subjects = InferSubjects<AllTypes, true> | 'all';
+type PossibleAbilities = [Action, Subjects];
 
 export enum Action {
   Manage = 'manage', // manage is a special keyword in CASL which represents "any" action.
@@ -49,16 +50,14 @@ interface EvaluationQuery extends Evaluation {
   'groups.users.id': User['id'];
 }
 
-export type AppAbility = Ability<[Action, Subjects]>;
+export type AppAbility = MongoAbility<PossibleAbilities>;
 
 @Injectable()
 export class CaslAbilityFactory {
-  createForUser(user: User): Ability {
-    const {can, cannot, build} = new AbilityBuilder<
-      Ability<[Action, Subjects]>
-    >(Ability as AbilityClass<AppAbility>);
-
+  createForUser(user: User): MongoAbility {
+    const {can, cannot, build} = new AbilityBuilder(createMongoAbility);
     if (user.role === 'admin') {
+      // all is a special keyword in CASL that represents "any subject".
       // read-write access to everything
       can(Action.Manage, 'all');
       // Read statistics about this heimdall deployment
@@ -94,7 +93,7 @@ export class CaslAbilityFactory {
     // it requires every evaluation to have a join on Groups and then another join on Users
     can([Action.Create], Evaluation);
 
-    can([Action.Read], Evaluation, {public: true});
+    can(Action.Read, Evaluation, {public: true});
 
     can([Action.Manage], Evaluation, {
       userId: user.id
@@ -119,11 +118,8 @@ export class CaslAbilityFactory {
   // This provides the ability to use the same codepath for validating
   // user abilities and non-registered user abilities. Useful for the
   // few anonymous endpoints we have.
-  createForAnonymous(): Ability {
-    const {cannot, build} = new AbilityBuilder<Ability<[Action, Subjects]>>(
-      Ability
-    );
-
+  createForAnonymous(): MongoAbility {
+    const {cannot, build} = new AbilityBuilder(createMongoAbility);
     cannot(Action.Manage, 'all');
 
     return build();

diff --git a/apps/backend/src/database/database.module.ts b/apps/backend/src/database/database.module.ts
@@ -5,13 +5,22 @@ import {ConfigModule} from '../config/config.module';
 import {ConfigService} from '../config/config.service';
 import {DatabaseService} from './database.service';
 
+const line = '________________________________________________\n';
 const logger = winston.createLogger({
   transports: [new winston.transports.Console()],
   format: winston.format.combine(
+    winston.format.colorize({
+      all: true
+    }),
     winston.format.timestamp({
       format: 'MMM-DD-YYYY HH:mm:ss Z'
     }),
-    winston.format.printf((info) => `[${[info.timestamp]}] ${info.message}`)
+    winston.format.errors({stack: true}),
+    winston.format.align(),
+    winston.format.printf(
+      (info) =>
+        `${line}[${info.timestamp}] Query(${info.queryType}): ${info.message}`
+    )
   )
 });
 
@@ -49,7 +58,8 @@ function logQuery(
   logger.info({
     message: `${sql} [${sanitize(connection.fields, connection.bind).join(
       ', '
-    )}]`
+    )}]`,
+    queryType: connection.type
   });
 }
 

diff --git a/apps/backend/src/evaluations/dto/evaluation.dto.ts b/apps/backend/src/evaluations/dto/evaluation.dto.ts
@@ -52,3 +52,8 @@ export class EvaluationDto implements IEvaluation {
     this.shareURL = shareURL;
   }
 }
+
+export interface IEvaluationResponse {
+  evaluations: EvaluationDto[];
+  totalCount: number;
+}