Merge pull request #16 from atlassian/release-1.0.5-beta

Release 1.0.5 beta

README.md

@@ -6,7 +6,9 @@ This repository contains Taurus scripts for performance testing of Atlassian Dat
 At the moment, Jira DC support is in beta. Confluence DC and Bitbucket DC support is coming soon.
 
 ## Known issues/limitations
-* Jira version 8.0.3 is only supported. Version 7.13.x support is coming soon.
+* Supported Jira versions: 
+    * Latest platform release: 8.0.3
+    * Latest enterprise release: 7.13.6
 * The SQL import is flaky. In case of a failure, run it again.
 
 ## Installation and set up

jira/README.md

@@ -1,6 +1,6 @@
 # Running tests
 ## Pre-requisites
-* Working Jira Software, version 8.0.3 with users, issues, projects, and boards, etc.
+* Working Jira Software of supported version (see toolkit README.md for a list of supported Jira versions) with users, issues, projects, and boards, etc.
 * Client machine with 4 CPUs and 16 GBs of RAM to run the Toolkit.
 * Virtual environment with Python3.6+ and bzt installed. See the root `README.md` file for more details.
 

jira/dc-apps-performance-toolkit-user-guide.md

@@ -47,12 +47,12 @@ All important parameters are listed and described in this section. For all other
 | Parameter | Recommended Value |
 | --------- | ----------------- |
 | Jira Product | Software |
-| Jira Version | 8.0.3 |
+| Jira Version | 8.0.3 or 7.13.6 |
 
 The Data Center App Performance Toolkit officially supports:
 
-- The latest Jira GA (General Availability release) version: 8.0.3
-- The latest [Enterprise Release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html): 7.13
+- The latest Jira Platform Release version: 8.0.3
+- The latest Jira [Enterprise Release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html): 7.13.6
 
 **Cluster nodes**
 
@@ -159,7 +159,7 @@ All the datasets use the standard `admin`/`admin` credentials.
 
 Pre-loading the dataset is a three-step process:
 
-1. [Importing the main dataset](#importingdataset). To help you out, we provide an enterprise-scale dataset you can import either via the [populate_db.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/populate_db.sh) script or [xml_backup.zip](https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large/xml_backup.zip).
+1. [Importing the main dataset](#importingdataset). To help you out, we provide an enterprise-scale dataset you can import either via the [populate_db.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/populate_db.sh) script or restore from xml backup file.
 1. [Restoring attachments](#copyingattachments). We also provide attachments, which you can pre-load via an [upload_attachments.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/upload_attachments.sh) script.
 1. [Re-indexing Jira Data Center](#reindexing). For more information, go to [Re-indexing Jira](https://confluence.atlassian.com/adminjiraserver/search-indexing-938847710.html).
 
@@ -185,11 +185,11 @@ To populate the database with SQL:
 
     For Windows, use Putty to connect to the Jira node over SSH.
     For Linux or MacOS:
-    ``` bash
+    ```bash
     ssh-add path_to_your_private_key_pem
     export BASTION_IP=bastion_instance_public_ip
     export NODE_IP=node_private_ip
-    ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP"
+    ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP}
     ```
     For more information, go to [Connecting your nodes over SSH](https://confluence.atlassian.com/adminjiraserver/administering-jira-data-center-on-aws-938846969.html#AdministeringJiraDataCenteronAWS-ConnectingtoyournodesoverSSH).
 1. Download the [populate_db.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/populate_db.sh) script and make it executable:
@@ -203,14 +203,13 @@ To populate the database with SQL:
     INSTALL_PSQL_CMD="amazon-linux-extras install -y postgresql10"
     DB_CONFIG="/var/atlassian/application-data/jira/dbconfig.xml"
     JIRA_CURRENT_DIR="/opt/atlassian/jira-software/current"
-    STOP_JIRA="${JIRA_CURRENT_DIR}/bin/stop-jira.sh"
-    START_JIRA="${JIRA_CURRENT_DIR}/bin/start-jira.sh"
     CATALINA_PID_FILE="${JIRA_CURRENT_DIR}/work/catalina.pid"
     JIRA_DB_NAME="jira"
     JIRA_DB_USER="postgres"
     JIRA_DB_PASS="Password1!"
-    DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large"
-    DB_DUMP_NAME="db.dump"
+    JIRA_SETENV_FILE="${JIRA_CURRENT_DIR}/bin/setenv.sh"
+    JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version"
+    DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
     ```
 1. Run the script:
 
@@ -236,17 +235,18 @@ We recommend that you only use this method if you are having problems with the [
 
     For Windows, use Putty to connect to the Jira node over SSH.
     For Linux or MacOS:
-    ``` bash
+    ```bash
     ssh-add path_to_your_private_key_pem
     export BASTION_IP=bastion_instance_public_ip
     export NODE_IP=node_private_ip
-    ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP"
+    ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" eec2-user@${NODE_IP}
     ```
     For more information, go to [Connecting your nodes over SSH](https://confluence.atlassian.com/adminjiraserver/administering-jira-data-center-on-aws-938846969.html#AdministeringJiraDataCenteronAWS-ConnectingtoyournodesoverSSH).
-1. Download the [xml_backup.zip](https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large/xml_backup.zip) file.
+1. Download the xml_backup.zip file corresponding to your Jira version.
 
     ``` bash
-    sudo su jira -c "wget https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large/xml_backup.zip -O /media/atl/jira/shared/import"
+    JIRA_VERSION=$(sudo su jira -c "cat /media/atl/jira/shared/jira-software.version")
+    sudo su jira -c "wget https://centaurus-datasets.s3.amazonaws.com/jira/${JIRA_VERSION}/large/xml_backup.zip -O /media/atl/jira/shared/import/xml_backup.zip"
     ```
 1. From a different computer, log in as a user with the **Jira System Administrators** [global permission](https://confluence.atlassian.com/adminjiraserver/managing-global-permissions-938847142.html).
 1. Go to **![cog icon](/platform/marketplace/images/cog.png) > System > Restore System.** from the menu.
@@ -261,11 +261,11 @@ After [Importing the main dataset](#importingdataset), you'll now have to pre-lo
 
     For Windows, use Putty to connect to the Jira node over SSH.
     For Linux or MacOS:
-    ``` bash
+    ```bash
     ssh-add path_to_your_private_key_pem
     export BASTION_IP=bastion_instance_public_ip
     export NODE_IP=node_private_ip
-    ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP"
+    ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP}
     ```
     For more information, go to [Connecting your nodes over SSH](https://confluence.atlassian.com/adminjiraserver/administering-jira-data-center-on-aws-938846969.html#AdministeringJiraDataCenteronAWS-ConnectingtoyournodesoverSSH).
 1. Download the [upload_attachments.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/upload_attachments.sh) script and make it executable:
@@ -276,7 +276,7 @@ After [Importing the main dataset](#importingdataset), you'll now have to pre-lo
 1. Review the following `Variables section` of the script:
 
     ``` bash
-    DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large"
+    DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
     ATTACHMENTS_TAR="attachments.tar.gz"
     ATTACHMENTS_DIR="attachments"
     TMP_DIR="/tmp"
@@ -514,7 +514,7 @@ To receive scalability benchmark results for two-node Jira DC with app-specific
     ssh-add path_to_your_private_key_pem
     export BASTION_IP=bastion_instance_public_ip
     export NODE_IP=node_private_ip
-    ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP"
+    ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP}
     ```
 1. Once you're in the second node, download the [index-sync.sh](https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/jira/util/index-sync.sh) file. Then, make it executable and run it:
 

jira/jira.yml

@@ -84,4 +84,4 @@ modules:
       httpsampler.ignore_failed_embedded_resources: "true"
   selenium:
     chromedriver:
-      version: "76.0.3809.68" # Supports Chrome version 76. You can refer to http://chromedriver.chromium.org/downloads
+      version: "77.0.3865.40" # Supports Chrome version 77. You can refer to http://chromedriver.chromium.org/downloads

jira/util/populate_db.sh

@@ -1,6 +1,5 @@
 #!/bin/bash
 
-
 ###################    Variables section         ###################
 # Command to install psql client for Amazon Linux 2.
 # In case of different distributive, please adjust accordingly or install manually.
@@ -14,40 +13,79 @@ JIRA_CURRENT_DIR="/opt/atlassian/jira-software/current"
 STOP_JIRA="${JIRA_CURRENT_DIR}/bin/stop-jira.sh"
 START_JIRA="${JIRA_CURRENT_DIR}/bin/start-jira.sh"
 CATALINA_PID_FILE="${JIRA_CURRENT_DIR}/work/catalina.pid"
+JIRA_SETENV_FILE="${JIRA_CURRENT_DIR}/bin/setenv.sh"
+JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version"
+SHUT_DOWN_TOMCAT="${JIRA_CURRENT_DIR}/bin/shutdown.sh"
 
 # DB admin user name, password and DB name
 JIRA_DB_NAME="jira"
 JIRA_DB_USER="postgres"
 JIRA_DB_PASS="Password1!"
 
-# Datasets AWS bucket and db dupm name
-DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large"
+# Jira version variables
+SUPPORTED_JIRA_VERSIONS=(8.0.3 7.13.6)
+JIRA_VERSION=$(sudo su jira -c "cat ${JIRA_VERSION_FILE}")
+echo "Jira Version: ${JIRA_VERSION}"
+
+# Datasets AWS bucket and db dump name
+DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
+DATASETS_SIZE="large"
 DB_DUMP_NAME="db.dump"
+DB_DUMP_URL="${DATASETS_AWS_BUCKET}/${JIRA_VERSION}/${DATASETS_SIZE}/${DB_DUMP_NAME}"
+
 ###################    End of variables section  ###################
 
 
+# Check if Jira version is supported
+if [[ ! "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "${JIRA_VERSION}" ]]; then
+  echo "Jira Version: ${JIRA_VERSION} is not officially supported by DCAPT."
+  echo "Supported Jira Versions: ${SUPPORTED_JIRA_VERSIONS[@]}"
+  echo "If you want to force apply an existing datasets to your Jira, use --force flag with version of dataset you want to apply:"
+  echo "e.g. ./populate_db.sh --force 8.0.3"
+  echo "!!! Warning !!! This may broke your Jira instance."
+  # Check if --force flag is passed into command
+  if [[ "$1" == "--force" ]]; then
+    # Check if passed Jira version is in list of supported
+    if [[ "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "$2" ]]; then
+      DB_DUMP_URL="${DATASETS_AWS_BUCKET}/$2/${DATASETS_SIZE}/${DB_DUMP_NAME}"
+      echo "Force mode. Dataset URL: ${DB_DUMP_URL}"
+      # If there is no DOWNGRADE_OPT - set it
+      DOWNGRADE_OPT="Djira.downgrade.allowed=true"
+      if sudo su jira -c "! grep -q ${DOWNGRADE_OPT} $JIRA_SETENV_FILE"; then
+        sudo sed -i "s/JVM_SUPPORT_RECOMMENDED_ARGS=\"/&-${DOWNGRADE_OPT} /" "${JIRA_SETENV_FILE}"
+        echo "Flag -${DOWNGRADE_OPT} was set in ${JIRA_SETENV_FILE}"
+      fi
+    else
+      echo "Correct dataset version was not specified after --force flag."
+      echo "Available datasets: ${SUPPORTED_JIRA_VERSIONS[@]}"
+      exit 1
+    fi
+  else
+    # No force flag
+    exit 1
+  fi
+fi
+
 echo "!!! Warning !!!"
-echo    # move to a new line
+echo # move to a new line
 echo "This script restores Postgres DB from SQL DB dump for Jira DC created with AWS Quickstart defaults."
 echo "You can review or modify default variables in 'Variables section' of this script."
-echo    # move to a new line
+echo # move to a new line
 echo "Variables:"
 echo "JIRA_CURRENT_DIR=${JIRA_CURRENT_DIR}"
 echo "DB_CONFIG=${DB_CONFIG}"
 echo "JIRA_DB_NAME=${JIRA_DB_NAME}"
 echo "JIRA_DB_USER=${JIRA_DB_USER}"
 echo "JIRA_DB_PASS=${JIRA_DB_PASS}"
-echo "DB_DUMP=${DATASETS_AWS_BUCKET}/${DB_DUMP_NAME}"
-echo    # move to a new line
+echo "DB_DUMP_URL=${DB_DUMP_URL}"
+echo # move to a new line
 read -p "I confirm that variables are correct and want to proceed (y/n)?  " -n 1 -r
-echo    # move to a new line
-if [[ ! $REPLY =~ ^[Yy]$ ]]
-then
-    echo "Script was canceled."
-    exit 1
+echo # move to a new line
+if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+  echo "Script was canceled."
+  exit 1
 fi
 
-
 echo "Step1: Check Postgres Client"
 if ! [[ -x "$(command -v psql)" ]]; then
   echo "Install Postgres client"
@@ -64,50 +102,68 @@ fi
 
 echo "Step2: Download DB dump"
 rm -rf ${DB_DUMP_NAME}
-wget ${DATASETS_AWS_BUCKET}/${DB_DUMP_NAME}
+wget ${DB_DUMP_URL}
 if [[ $? -ne 0 ]]; then
   echo "DB dump download failed! Pls check available disk space."
   exit 1
 fi
 
 echo "Step3: Stop Jira"
 CATALINA_PID=$(pgrep -f "catalina")
+echo "CATALINA_PID=${CATALINA_PID}"
 if [[ -z ${CATALINA_PID} ]]; then
   echo "Jira is not running"
+  sudo su -c "rm -rf ${CATALINA_PID_FILE}"
 else
   echo "Stopping Jira"
-  sudo su jira -c "echo ${CATALINA_PID} > ${CATALINA_PID_FILE}"
-  sudo su jira -c "${STOP_JIRA}"
-  sleep 5
+  sudo su -c "echo ${CATALINA_PID} > ${CATALINA_PID_FILE}"
+  sudo su -c "${SHUT_DOWN_TOMCAT}"
+  COUNTER=0
+  TIMEOUT=5
+  ATTEMPTS=20
+  while [[ "${COUNTER}" -lt "${ATTEMPTS}" ]]; do
+    if [[ -z $(pgrep -f "catalina") ]]; then
+      echo Jira is stopped
+      break
+    fi
+    echo "Waiting for Jira stop, attempt ${COUNTER} at waiting ${TIMEOUT} seconds."
+    sleep ${TIMEOUT}
+    let COUNTER++
+  done
+  if [ ${COUNTER} -eq ${ATTEMPTS} ]; then
+    echo "Jira stop was not finished in $ATTEMPTS attempts with $TIMEOUT sec timeout."
+    echo "Try to rerun script."
+    exit 1
+  fi
 fi
 
-echo "Step4: Get DB_URL"
-DB_URL=$(sudo su -c "cat ${DB_CONFIG} | grep 'jdbc:postgresql' | cut -d'/' -f3 | cut -d':' -f1")
-if [[ -z ${DB_URL} ]]; then
+echo "Step4: Get DB Host"
+DB_HOST=$(sudo su -c "cat ${DB_CONFIG} | grep 'jdbc:postgresql' | cut -d'/' -f3 | cut -d':' -f1")
+if [[ -z ${DB_HOST} ]]; then
   echo "DataBase URL was not found in ${DB_CONFIG}"
   exit 1
 fi
-echo "DB_URL=${DB_URL}"
+echo "DB_HOST=${DB_HOST}"
 
 echo "Step5: SQL Restore"
 echo "Check DB connection"
-PGPASSWORD=${JIRA_DB_PASS} pg_isready -U ${JIRA_DB_USER} -h ${DB_URL}
+PGPASSWORD=${JIRA_DB_PASS} pg_isready -U ${JIRA_DB_USER} -h ${DB_HOST}
 if [[ $? -ne 0 ]]; then
   echo "Connection to DB failed. Please check correctness of following variables:"
   echo "JIRA_DB_NAME=${JIRA_DB_NAME}"
   echo "JIRA_DB_USER=${JIRA_DB_USER}"
   echo "JIRA_DB_PASS=${JIRA_DB_PASS}"
-  echo "DB_URL=${DB_URL}"
+  echo "DB_HOST=${DB_HOST}"
   exit 1
 fi
 echo "Drop DB"
-PGPASSWORD=${JIRA_DB_PASS} dropdb -U ${JIRA_DB_USER} -h ${DB_URL} ${JIRA_DB_NAME}
+PGPASSWORD=${JIRA_DB_PASS} dropdb -U ${JIRA_DB_USER} -h ${DB_HOST} ${JIRA_DB_NAME}
 sleep 5
 echo "Create DB"
-PGPASSWORD=${JIRA_DB_PASS} createdb -U ${JIRA_DB_USER} -h ${DB_URL} -T template0 ${JIRA_DB_NAME}
+PGPASSWORD=${JIRA_DB_PASS} createdb -U ${JIRA_DB_USER} -h ${DB_HOST} -T template0 ${JIRA_DB_NAME}
 sleep 5
 echo "PG Restore"
-time PGPASSWORD=${JIRA_DB_PASS} pg_restore -v -U ${JIRA_DB_USER} -h ${DB_URL} -d ${JIRA_DB_NAME} ${DB_DUMP_NAME}
+time PGPASSWORD=${JIRA_DB_PASS} pg_restore -v -U ${JIRA_DB_USER} -h ${DB_HOST} -d ${JIRA_DB_NAME} ${DB_DUMP_NAME}
 if [[ $? -ne 0 ]]; then
   echo "SQL Restore failed!"
   exit 1

jira/util/upload_attachments.sh

@@ -2,13 +2,44 @@
 
 
 ###################    Variables section         ###################
-DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large"
+# Jira version variables
+JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version"
+SUPPORTED_JIRA_VERSIONS=(8.0.3 7.13.6)
+JIRA_VERSION=$(sudo su jira -c "cat ${JIRA_VERSION_FILE}")
+echo "Jira Version: ${JIRA_VERSION}"
+
+DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
 ATTACHMENTS_TAR="attachments.tar.gz"
 ATTACHMENTS_DIR="attachments"
+DATASETS_SIZE="large"
+ATTACHMENTS_TAR_URL="${DATASETS_AWS_BUCKET}/${JIRA_VERSION}/${DATASETS_SIZE}/${ATTACHMENTS_TAR}"
 TMP_DIR="/tmp"
 EFS_DIR="/media/atl/jira/shared/data"
 ###################    End of variables section  ###################
 
+# Check if Jira version is supported
+if [[ ! "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "${JIRA_VERSION}" ]]; then
+  echo "Jira Version: ${JIRA_VERSION} is not officially supported by DCAPT."
+  echo "Supported Jira Versions: ${SUPPORTED_JIRA_VERSIONS[@]}"
+  echo "If you want to force apply an existing datasets to your Jira, use --force flag with version of dataset you want to apply:"
+  echo "e.g. ./upload_attachments --force 8.0.3"
+  echo "!!! Warning !!! This may broke your Jira instance."
+  # Check if --force flag is passed into command
+  if [[ "$1" == "--force" ]]; then
+    # Check if passed Jira version is in list of supported
+    if [[ "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "$2" ]]; then
+      ATTACHMENTS_TAR_URL="${DATASETS_AWS_BUCKET}/$2/${DATASETS_SIZE}/${ATTACHMENTS_TAR}"
+      echo "Force mode. Dataset URL: ${ATTACHMENTS_TAR_URL}"
+    else
+      echo "Correct dataset version was not specified after --force flag."
+      echo "Available datasets: ${SUPPORTED_JIRA_VERSIONS[@]}"
+      exit 1
+    fi
+  else
+    # No force flag
+    exit 1
+  fi
+fi
 
 echo "!!! Warning !!!"
 echo    # move to a new line
@@ -17,7 +48,7 @@ echo "You can review or modify default variables in 'Variables section' of this
 echo    # move to a new line
 echo "Variables:"
 echo "EFS_DIR=${EFS_DIR}"
-echo "ATTACHMENTS_TAR_URL=${DATASETS_AWS_BUCKET}/${ATTACHMENTS_TAR}"
+echo "ATTACHMENTS_TAR_URL=${ATTACHMENTS_TAR_URL}"
 echo    # move to a new line
 read -p "I confirm that variables are correct and want to proceed (y/n)?  " -n 1 -r
 echo    # move to a new line
@@ -39,7 +70,7 @@ fi
 
 echo "Step2: Download attachments"
 sudo su -c "rm -rf ${ATTACHMENTS_TAR}"
-sudo su jira -c "wget ${DATASETS_AWS_BUCKET}/${ATTACHMENTS_TAR}"
+sudo su jira -c "wget ${ATTACHMENTS_TAR_URL}"
 
 echo "Step3: Untar attachments to tmp folder"
 sudo su -c "rm -rf ${ATTACHMENTS_DIR}"