diff --git a/README.md b/README.md index 479765833..3a2c95951 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,9 @@ This repository contains Taurus scripts for performance testing of Atlassian Dat At the moment, Jira DC support is in beta. Confluence DC and Bitbucket DC support is coming soon. ## Known issues/limitations -* Jira version 8.0.3 is only supported. Version 7.13.x support is coming soon. +* Supported Jira versions: + * Latest platform release: 8.0.3 + * Latest enterprise release: 7.13.6 * The SQL import is flaky. In case of a failure, run it again. ## Installation and set up diff --git a/jira/README.md b/jira/README.md index ed7240119..57fe66dac 100644 --- a/jira/README.md +++ b/jira/README.md @@ -1,6 +1,6 @@ # Running tests ## Pre-requisites -* Working Jira Software, version 8.0.3 with users, issues, projects, and boards, etc. +* Working Jira Software of supported version (see toolkit README.md for a list of supported Jira versions) with users, issues, projects, and boards, etc. * Client machine with 4 CPUs and 16 GBs of RAM to run the Toolkit. * Virtual environment with Python3.6+ and bzt installed. See the root `README.md` file for more details. diff --git a/jira/dc-apps-performance-toolkit-user-guide.md b/jira/dc-apps-performance-toolkit-user-guide.md index 5904de6a0..a179bdb8d 100644 --- a/jira/dc-apps-performance-toolkit-user-guide.md +++ b/jira/dc-apps-performance-toolkit-user-guide.md @@ -47,12 +47,12 @@ All important parameters are listed and described in this section. For all other | Parameter | Recommended Value | | --------- | ----------------- | | Jira Product | Software | -| Jira Version | 8.0.3 | +| Jira Version | 8.0.3 or 7.13.6 | The Data Center App Performance Toolkit officially supports: -- The latest Jira GA (General Availability release) version: 8.0.3 -- The latest [Enterprise Release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html): 7.13 +- The latest Jira Platform Release version: 8.0.3 +- The latest Jira [Enterprise Release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html): 7.13.6 **Cluster nodes** @@ -159,7 +159,7 @@ All the datasets use the standard `admin`/`admin` credentials. Pre-loading the dataset is a three-step process: -1. [Importing the main dataset](#importingdataset). To help you out, we provide an enterprise-scale dataset you can import either via the [populate_db.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/populate_db.sh) script or [xml_backup.zip](https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large/xml_backup.zip). +1. [Importing the main dataset](#importingdataset). To help you out, we provide an enterprise-scale dataset you can import either via the [populate_db.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/populate_db.sh) script or restore from xml backup file. 1. [Restoring attachments](#copyingattachments). We also provide attachments, which you can pre-load via an [upload_attachments.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/upload_attachments.sh) script. 1. [Re-indexing Jira Data Center](#reindexing). For more information, go to [Re-indexing Jira](https://confluence.atlassian.com/adminjiraserver/search-indexing-938847710.html). @@ -185,11 +185,11 @@ To populate the database with SQL: For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS: - ``` bash + ```bash ssh-add path_to_your_private_key_pem export BASTION_IP=bastion_instance_public_ip export NODE_IP=node_private_ip - ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" + ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP} ``` For more information, go to [Connecting your nodes over SSH](https://confluence.atlassian.com/adminjiraserver/administering-jira-data-center-on-aws-938846969.html#AdministeringJiraDataCenteronAWS-ConnectingtoyournodesoverSSH). 1. Download the [populate_db.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/populate_db.sh) script and make it executable: @@ -203,14 +203,13 @@ To populate the database with SQL: INSTALL_PSQL_CMD="amazon-linux-extras install -y postgresql10" DB_CONFIG="/var/atlassian/application-data/jira/dbconfig.xml" JIRA_CURRENT_DIR="/opt/atlassian/jira-software/current" - STOP_JIRA="${JIRA_CURRENT_DIR}/bin/stop-jira.sh" - START_JIRA="${JIRA_CURRENT_DIR}/bin/start-jira.sh" CATALINA_PID_FILE="${JIRA_CURRENT_DIR}/work/catalina.pid" JIRA_DB_NAME="jira" JIRA_DB_USER="postgres" JIRA_DB_PASS="Password1!" - DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large" - DB_DUMP_NAME="db.dump" + JIRA_SETENV_FILE="${JIRA_CURRENT_DIR}/bin/setenv.sh" + JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version" + DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira" ``` 1. Run the script: @@ -236,17 +235,18 @@ We recommend that you only use this method if you are having problems with the [ For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS: - ``` bash + ```bash ssh-add path_to_your_private_key_pem export BASTION_IP=bastion_instance_public_ip export NODE_IP=node_private_ip - ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" + ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" eec2-user@${NODE_IP} ``` For more information, go to [Connecting your nodes over SSH](https://confluence.atlassian.com/adminjiraserver/administering-jira-data-center-on-aws-938846969.html#AdministeringJiraDataCenteronAWS-ConnectingtoyournodesoverSSH). -1. Download the [xml_backup.zip](https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large/xml_backup.zip) file. +1. Download the xml_backup.zip file corresponding to your Jira version. ``` bash - sudo su jira -c "wget https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large/xml_backup.zip -O /media/atl/jira/shared/import" + JIRA_VERSION=$(sudo su jira -c "cat /media/atl/jira/shared/jira-software.version") + sudo su jira -c "wget https://centaurus-datasets.s3.amazonaws.com/jira/${JIRA_VERSION}/large/xml_backup.zip -O /media/atl/jira/shared/import/xml_backup.zip" ``` 1. From a different computer, log in as a user with the **Jira System Administrators** [global permission](https://confluence.atlassian.com/adminjiraserver/managing-global-permissions-938847142.html). 1. Go to **![cog icon](/platform/marketplace/images/cog.png) > System > Restore System.** from the menu. @@ -261,11 +261,11 @@ After [Importing the main dataset](#importingdataset), you'll now have to pre-lo For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS: - ``` bash + ```bash ssh-add path_to_your_private_key_pem export BASTION_IP=bastion_instance_public_ip export NODE_IP=node_private_ip - ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" + ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP} ``` For more information, go to [Connecting your nodes over SSH](https://confluence.atlassian.com/adminjiraserver/administering-jira-data-center-on-aws-938846969.html#AdministeringJiraDataCenteronAWS-ConnectingtoyournodesoverSSH). 1. Download the [upload_attachments.sh](https://github.com/atlassian/dc-app-performance-toolkit/blob/master/jira/util/upload_attachments.sh) script and make it executable: @@ -276,7 +276,7 @@ After [Importing the main dataset](#importingdataset), you'll now have to pre-lo 1. Review the following `Variables section` of the script: ``` bash - DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large" + DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira" ATTACHMENTS_TAR="attachments.tar.gz" ATTACHMENTS_DIR="attachments" TMP_DIR="/tmp" @@ -514,7 +514,7 @@ To receive scalability benchmark results for two-node Jira DC with app-specific ssh-add path_to_your_private_key_pem export BASTION_IP=bastion_instance_public_ip export NODE_IP=node_private_ip - ssh ec2-user@$NODE_IP -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" + ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP} ``` 1. Once you're in the second node, download the [index-sync.sh](https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/jira/util/index-sync.sh) file. Then, make it executable and run it: diff --git a/jira/jira.yml b/jira/jira.yml index e88c96b95..4543f56bb 100644 --- a/jira/jira.yml +++ b/jira/jira.yml @@ -84,4 +84,4 @@ modules: httpsampler.ignore_failed_embedded_resources: "true" selenium: chromedriver: - version: "76.0.3809.68" # Supports Chrome version 76. You can refer to http://chromedriver.chromium.org/downloads \ No newline at end of file + version: "77.0.3865.40" # Supports Chrome version 77. You can refer to http://chromedriver.chromium.org/downloads \ No newline at end of file diff --git a/jira/util/populate_db.sh b/jira/util/populate_db.sh index 5819ac912..9c45f98cf 100755 --- a/jira/util/populate_db.sh +++ b/jira/util/populate_db.sh @@ -1,6 +1,5 @@ #!/bin/bash - ################### Variables section ################### # Command to install psql client for Amazon Linux 2. # In case of different distributive, please adjust accordingly or install manually. @@ -14,40 +13,79 @@ JIRA_CURRENT_DIR="/opt/atlassian/jira-software/current" STOP_JIRA="${JIRA_CURRENT_DIR}/bin/stop-jira.sh" START_JIRA="${JIRA_CURRENT_DIR}/bin/start-jira.sh" CATALINA_PID_FILE="${JIRA_CURRENT_DIR}/work/catalina.pid" +JIRA_SETENV_FILE="${JIRA_CURRENT_DIR}/bin/setenv.sh" +JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version" +SHUT_DOWN_TOMCAT="${JIRA_CURRENT_DIR}/bin/shutdown.sh" # DB admin user name, password and DB name JIRA_DB_NAME="jira" JIRA_DB_USER="postgres" JIRA_DB_PASS="Password1!" -# Datasets AWS bucket and db dupm name -DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large" +# Jira version variables +SUPPORTED_JIRA_VERSIONS=(8.0.3 7.13.6) +JIRA_VERSION=$(sudo su jira -c "cat ${JIRA_VERSION_FILE}") +echo "Jira Version: ${JIRA_VERSION}" + +# Datasets AWS bucket and db dump name +DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira" +DATASETS_SIZE="large" DB_DUMP_NAME="db.dump" +DB_DUMP_URL="${DATASETS_AWS_BUCKET}/${JIRA_VERSION}/${DATASETS_SIZE}/${DB_DUMP_NAME}" + ################### End of variables section ################### +# Check if Jira version is supported +if [[ ! "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "${JIRA_VERSION}" ]]; then + echo "Jira Version: ${JIRA_VERSION} is not officially supported by DCAPT." + echo "Supported Jira Versions: ${SUPPORTED_JIRA_VERSIONS[@]}" + echo "If you want to force apply an existing datasets to your Jira, use --force flag with version of dataset you want to apply:" + echo "e.g. ./populate_db.sh --force 8.0.3" + echo "!!! Warning !!! This may broke your Jira instance." + # Check if --force flag is passed into command + if [[ "$1" == "--force" ]]; then + # Check if passed Jira version is in list of supported + if [[ "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "$2" ]]; then + DB_DUMP_URL="${DATASETS_AWS_BUCKET}/$2/${DATASETS_SIZE}/${DB_DUMP_NAME}" + echo "Force mode. Dataset URL: ${DB_DUMP_URL}" + # If there is no DOWNGRADE_OPT - set it + DOWNGRADE_OPT="Djira.downgrade.allowed=true" + if sudo su jira -c "! grep -q ${DOWNGRADE_OPT} $JIRA_SETENV_FILE"; then + sudo sed -i "s/JVM_SUPPORT_RECOMMENDED_ARGS=\"/&-${DOWNGRADE_OPT} /" "${JIRA_SETENV_FILE}" + echo "Flag -${DOWNGRADE_OPT} was set in ${JIRA_SETENV_FILE}" + fi + else + echo "Correct dataset version was not specified after --force flag." + echo "Available datasets: ${SUPPORTED_JIRA_VERSIONS[@]}" + exit 1 + fi + else + # No force flag + exit 1 + fi +fi + echo "!!! Warning !!!" -echo # move to a new line +echo # move to a new line echo "This script restores Postgres DB from SQL DB dump for Jira DC created with AWS Quickstart defaults." echo "You can review or modify default variables in 'Variables section' of this script." -echo # move to a new line +echo # move to a new line echo "Variables:" echo "JIRA_CURRENT_DIR=${JIRA_CURRENT_DIR}" echo "DB_CONFIG=${DB_CONFIG}" echo "JIRA_DB_NAME=${JIRA_DB_NAME}" echo "JIRA_DB_USER=${JIRA_DB_USER}" echo "JIRA_DB_PASS=${JIRA_DB_PASS}" -echo "DB_DUMP=${DATASETS_AWS_BUCKET}/${DB_DUMP_NAME}" -echo # move to a new line +echo "DB_DUMP_URL=${DB_DUMP_URL}" +echo # move to a new line read -p "I confirm that variables are correct and want to proceed (y/n)? " -n 1 -r -echo # move to a new line -if [[ ! $REPLY =~ ^[Yy]$ ]] -then - echo "Script was canceled." - exit 1 +echo # move to a new line +if [[ ! $REPLY =~ ^[Yy]$ ]]; then + echo "Script was canceled." + exit 1 fi - echo "Step1: Check Postgres Client" if ! [[ -x "$(command -v psql)" ]]; then echo "Install Postgres client" @@ -64,7 +102,7 @@ fi echo "Step2: Download DB dump" rm -rf ${DB_DUMP_NAME} -wget ${DATASETS_AWS_BUCKET}/${DB_DUMP_NAME} +wget ${DB_DUMP_URL} if [[ $? -ne 0 ]]; then echo "DB dump download failed! Pls check available disk space." exit 1 @@ -72,42 +110,60 @@ fi echo "Step3: Stop Jira" CATALINA_PID=$(pgrep -f "catalina") +echo "CATALINA_PID=${CATALINA_PID}" if [[ -z ${CATALINA_PID} ]]; then echo "Jira is not running" + sudo su -c "rm -rf ${CATALINA_PID_FILE}" else echo "Stopping Jira" - sudo su jira -c "echo ${CATALINA_PID} > ${CATALINA_PID_FILE}" - sudo su jira -c "${STOP_JIRA}" - sleep 5 + sudo su -c "echo ${CATALINA_PID} > ${CATALINA_PID_FILE}" + sudo su -c "${SHUT_DOWN_TOMCAT}" + COUNTER=0 + TIMEOUT=5 + ATTEMPTS=20 + while [[ "${COUNTER}" -lt "${ATTEMPTS}" ]]; do + if [[ -z $(pgrep -f "catalina") ]]; then + echo Jira is stopped + break + fi + echo "Waiting for Jira stop, attempt ${COUNTER} at waiting ${TIMEOUT} seconds." + sleep ${TIMEOUT} + let COUNTER++ + done + if [ ${COUNTER} -eq ${ATTEMPTS} ]; then + echo "Jira stop was not finished in $ATTEMPTS attempts with $TIMEOUT sec timeout." + echo "Try to rerun script." + exit 1 + fi fi -echo "Step4: Get DB_URL" -DB_URL=$(sudo su -c "cat ${DB_CONFIG} | grep 'jdbc:postgresql' | cut -d'/' -f3 | cut -d':' -f1") -if [[ -z ${DB_URL} ]]; then +echo "Step4: Get DB Host" +DB_HOST=$(sudo su -c "cat ${DB_CONFIG} | grep 'jdbc:postgresql' | cut -d'/' -f3 | cut -d':' -f1") +if [[ -z ${DB_HOST} ]]; then echo "DataBase URL was not found in ${DB_CONFIG}" exit 1 fi -echo "DB_URL=${DB_URL}" +echo "DB_HOST=${DB_HOST}" echo "Step5: SQL Restore" echo "Check DB connection" -PGPASSWORD=${JIRA_DB_PASS} pg_isready -U ${JIRA_DB_USER} -h ${DB_URL} +PGPASSWORD=${JIRA_DB_PASS} pg_isready -U ${JIRA_DB_USER} -h ${DB_HOST} if [[ $? -ne 0 ]]; then echo "Connection to DB failed. Please check correctness of following variables:" echo "JIRA_DB_NAME=${JIRA_DB_NAME}" echo "JIRA_DB_USER=${JIRA_DB_USER}" echo "JIRA_DB_PASS=${JIRA_DB_PASS}" - echo "DB_URL=${DB_URL}" + echo "DB_HOST=${DB_HOST}" exit 1 fi echo "Drop DB" -PGPASSWORD=${JIRA_DB_PASS} dropdb -U ${JIRA_DB_USER} -h ${DB_URL} ${JIRA_DB_NAME} +PGPASSWORD=${JIRA_DB_PASS} dropdb -U ${JIRA_DB_USER} -h ${DB_HOST} ${JIRA_DB_NAME} sleep 5 echo "Create DB" -PGPASSWORD=${JIRA_DB_PASS} createdb -U ${JIRA_DB_USER} -h ${DB_URL} -T template0 ${JIRA_DB_NAME} +PGPASSWORD=${JIRA_DB_PASS} createdb -U ${JIRA_DB_USER} -h ${DB_HOST} -T template0 ${JIRA_DB_NAME} sleep 5 echo "PG Restore" -time PGPASSWORD=${JIRA_DB_PASS} pg_restore -v -U ${JIRA_DB_USER} -h ${DB_URL} -d ${JIRA_DB_NAME} ${DB_DUMP_NAME} +time PGPASSWORD=${JIRA_DB_PASS} pg_restore -v -U ${JIRA_DB_USER} -h ${DB_HOST} -d ${JIRA_DB_NAME} ${DB_DUMP_NAME} if [[ $? -ne 0 ]]; then echo "SQL Restore failed!" exit 1 diff --git a/jira/util/upload_attachments.sh b/jira/util/upload_attachments.sh index 078a367a9..74e234855 100755 --- a/jira/util/upload_attachments.sh +++ b/jira/util/upload_attachments.sh @@ -2,13 +2,44 @@ ################### Variables section ################### -DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira/8.0.3/large" +# Jira version variables +JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version" +SUPPORTED_JIRA_VERSIONS=(8.0.3 7.13.6) +JIRA_VERSION=$(sudo su jira -c "cat ${JIRA_VERSION_FILE}") +echo "Jira Version: ${JIRA_VERSION}" + +DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira" ATTACHMENTS_TAR="attachments.tar.gz" ATTACHMENTS_DIR="attachments" +DATASETS_SIZE="large" +ATTACHMENTS_TAR_URL="${DATASETS_AWS_BUCKET}/${JIRA_VERSION}/${DATASETS_SIZE}/${ATTACHMENTS_TAR}" TMP_DIR="/tmp" EFS_DIR="/media/atl/jira/shared/data" ################### End of variables section ################### +# Check if Jira version is supported +if [[ ! "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "${JIRA_VERSION}" ]]; then + echo "Jira Version: ${JIRA_VERSION} is not officially supported by DCAPT." + echo "Supported Jira Versions: ${SUPPORTED_JIRA_VERSIONS[@]}" + echo "If you want to force apply an existing datasets to your Jira, use --force flag with version of dataset you want to apply:" + echo "e.g. ./upload_attachments --force 8.0.3" + echo "!!! Warning !!! This may broke your Jira instance." + # Check if --force flag is passed into command + if [[ "$1" == "--force" ]]; then + # Check if passed Jira version is in list of supported + if [[ "${SUPPORTED_JIRA_VERSIONS[@]}" =~ "$2" ]]; then + ATTACHMENTS_TAR_URL="${DATASETS_AWS_BUCKET}/$2/${DATASETS_SIZE}/${ATTACHMENTS_TAR}" + echo "Force mode. Dataset URL: ${ATTACHMENTS_TAR_URL}" + else + echo "Correct dataset version was not specified after --force flag." + echo "Available datasets: ${SUPPORTED_JIRA_VERSIONS[@]}" + exit 1 + fi + else + # No force flag + exit 1 + fi +fi echo "!!! Warning !!!" echo # move to a new line @@ -17,7 +48,7 @@ echo "You can review or modify default variables in 'Variables section' of this echo # move to a new line echo "Variables:" echo "EFS_DIR=${EFS_DIR}" -echo "ATTACHMENTS_TAR_URL=${DATASETS_AWS_BUCKET}/${ATTACHMENTS_TAR}" +echo "ATTACHMENTS_TAR_URL=${ATTACHMENTS_TAR_URL}" echo # move to a new line read -p "I confirm that variables are correct and want to proceed (y/n)? " -n 1 -r echo # move to a new line @@ -39,7 +70,7 @@ fi echo "Step2: Download attachments" sudo su -c "rm -rf ${ATTACHMENTS_TAR}" -sudo su jira -c "wget ${DATASETS_AWS_BUCKET}/${ATTACHMENTS_TAR}" +sudo su jira -c "wget ${ATTACHMENTS_TAR_URL}" echo "Step3: Untar attachments to tmp folder" sudo su -c "rm -rf ${ATTACHMENTS_DIR}"