netdata/ci: nits and fixes around package release workflow (netdata#6914)

* netdata/packaging: bring el/6 first, because this appears on all cases. Otherwise travis craps out the steps

* netdata/ci: Attempt to sanitise workflow for master branch commits

1) Rename Packaging for release, now named: Support activities on main branch
2) Adjust instruction messages to reflect more realistically the activities
3) Put labeler first on the stage, since this will always run
4) Make changelog and tag for release conditionally run, so that it does not confuse people

* netdata/ci: Sanitise workflow for release more (explained below)

We need to clean up tagging logic more and in order to do this we need to bring tagger logic to travis.yml
Then we have the complete control of tagging and triggering for changelog generation in travis.
To mitigate this, we need to:
1) make tagger script a lirbary. Keep the tagging logic in a method and remove the things needed when it was an executable (checks, executions etc)
2) make travis handle GIT_TAG checking and setting at the beginning. We need this outside of any conditionals so that we have GIT_TAG result available inside the steps
3) COMMIT_TAG_POINTS_AT can now become GIT_TAG, since this is what we actually wanted. We just named differently before to avoid any weird conflicts
4) make changelog generation for release to NOT source tagger any more. Also make it fail in case GIT_TAG is not there, we wont run the script if git tag not in place so that should be error if it happens
5) Rename .travis/generate_changelog_for_release.sh -> .travis/create_changelog_for_release.sh
6) Rename .travis/generate_changelog_and_tag_release.sh -> .travis/generate_changelog_for_release.sh

* netdata/ci: reinstate filename here. we basically do the git tag operation here still

* netdata/ci: simplify more the changelog generation process

1) Rename create_changelog_for_release.sh -> create_changelog.sh
2) Add comments and the standard path checks in create_changelog.sh
3) Do some minor changes to help diff with nightlies changelog (no functional change though, primarily messages and move flags lower)
4) Inside generate_changelog_for_nightlies.sh, use create_changelog.sh and remove code related to create changelog process
5) in nightlies.sh, use standard shebang adn fix some messages

Note: might follow up with more changes, this is just a first batch

* netdata/packaging: reinstate accidentally removed OPTS from other iterationns (different engineer)

Author: paulkatsoulakis

Diff for: .travis.yml

@@ -31,6 +31,10 @@ install:
   - if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export DEPLOY_REPO="netdata-edge"; fi;
   - export PACKAGE_CLOUD_RETENTION_DAYS=30
   - if [ ! "${TRAVIS_REPO_SLUG}" = "netdata/netdata" ]; then export DEPLOY_REPO="netdata-devel";  fi;
+  # These are release-related artifacts and have to be evaluated before we start doing conditional checks inside stages
+  - source ".travis/tagger.sh"
+  - export GIT_TAG="$(git tag --points-at)"
+  - if [[ -z "${GIT_TAG}" ]]; then  echo "Running set tag for release" && set_tag_for_release; fi;
 
 
 
@@ -58,7 +62,7 @@ stages:
     if: branch = master AND type = cron
 
     # Scheduled releases
-  - name: Packaging for release
+  - name: Support activities on main branch
     if: branch = master AND type != pull_request AND type != cron
 
   - name: Publish for release
@@ -240,22 +244,22 @@ jobs:
 
 
 
-    - stage: Packaging for release
+    - stage: Support activities on main branch
+      name: Run labeler on github issues
+      script: .travis/labeler.sh  # labeler should be replaced with GitHub Actions when they hit GA
 
-      name: Generate changelog and TAG the release (only on special commit msg)
-      before_script: post_message "TRAVIS_MESSAGE" "Packaging step for release initiated" "${NOTIF_CHANNEL}"
+    - name: Generate changelog for release (only on special and tagged commit msg)
+      before_script: post_message "TRAVIS_MESSAGE" "Support activities on main branch initiated" "${NOTIF_CHANNEL}"
       script:
         - echo "GIT Branch:" && git branch
         - echo "Last commit:" && git log -1
         - echo "GIT Describe:" && git describe
         - echo "packaging/version:" && cat packaging/version
         - .travis/generate_changelog_and_tag_release.sh
-      after_failure: post_message "TRAVIS_MESSAGE" "<!here> Packaging for release failed"
+      after_failure: post_message "TRAVIS_MESSAGE" "<!here> Changelog generation and tag of release, failed"
       git:
         depth: false
-
-    - name: Run labeler on github issues
-      script: .travis/labeler.sh  # labeler should be replaced with GitHub Actions when they hit GA
+      if: commit_message =~ /\[netdata (release candidate|(major|minor|patch) release)\]/ AND tag !~ /(-rc)/ OR (env(GIT_TAG) IS present AND NOT env(GIT_TAG) IS blank)
 
 
 
@@ -411,19 +415,19 @@ jobs:
           - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
           - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
 
-      name: "Build & Publish RPM package for Enterprise Linux 7"
+      name: "Build & Publish RPM package for Enterprise Linux 6"
       <<: *RPM_TEMPLATE
-      if: commit_message =~ /\[Package (amd64|arm64) RPM( Enterprise Linux)?\]/
+      if: commit_message =~ /\[Package (amd64|arm64|i386) RPM( Enterprise Linux)?\]/
       env:
-        - BUILDER_NAME="builder" BUILD_DISTRO="centos" BUILD_RELEASE="7" BUILD_STRING="el/7"
+        - BUILDER_NAME="builder" BUILD_DISTRO="centos" BUILD_RELEASE="6" BUILD_STRING="el/6"
         - PACKAGE_TYPE="rpm" REPO_TOOL="yum"
         - ALLOW_SOFT_FAILURE_HERE=true
 
-    - name: "Build & Publish RPM package for Enterprise Linux 6"
+    - name: "Build & Publish RPM package for Enterprise Linux 7"
       <<: *RPM_TEMPLATE
-      if: commit_message =~ /\[Package (amd64|arm64|i386) RPM( Enterprise Linux)?\]/
+      if: commit_message =~ /\[Package (amd64|arm64) RPM( Enterprise Linux)?\]/
       env:
-        - BUILDER_NAME="builder" BUILD_DISTRO="centos" BUILD_RELEASE="6" BUILD_STRING="el/6"
+        - BUILDER_NAME="builder" BUILD_DISTRO="centos" BUILD_RELEASE="7" BUILD_STRING="el/7"
         - PACKAGE_TYPE="rpm" REPO_TOOL="yum"
         - ALLOW_SOFT_FAILURE_HERE=true
 

Diff for: .travis/generate_changelog_for_release.sh renamed to .travis/create_changelog.sh

@@ -1,24 +1,32 @@
-#!/bin/bash
-
+#!/usr/bin/env bash
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author  : Pavlos Emm. Katsoulakis ([email protected])
 set -e
 
-if [ ! -f .gitignore ]; then
-	echo "Run as ./travis/$(basename "$0") from top level directory of git repository"
-	exit 1
+# If we are not in netdata git repo, at the top level directory, fail
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
+CWD=$(git rev-parse --show-cdup || echo "")
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
+    echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
+    echo "Changelog creation aborted"
+    exit 1
 fi
 
 ORGANIZATION=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $1}')
 PROJECT=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $2}')
 GIT_MAIL=${GIT_MAIL:-"[email protected]"}
 GIT_USER=${GIT_USER:-"netdatabot"}
+
 if [ -z ${GIT_TAG+x} ]; then
 	OPTS=""
 else
 	OPTS="--future-release ${GIT_TAG}"
 fi
 
 if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
-	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do"
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do here"
 	exit 0
 fi
 
@@ -31,8 +39,8 @@ docker run -it -v "$(pwd)":/project markmandel/github-changelog-generator:latest
 	--project "${PROJECT}" \
 	--token "${GITHUB_TOKEN}" \
 	--since-tag "v1.10.0" \
-	--no-issues \
 	--unreleased-label "**Next release**" \
+	--no-issues \
 	--exclude-labels "stale,duplicate,question,invalid,wontfix,discussion,no changelog" \
-        --max-issues 500 \
-        --bug-labels IGNOREBUGS
+	--max-issues 500 \
+	--bug-labels IGNOREBUGS ${OPTS}

Diff for: .travis/generate_changelog_and_tag_release.sh

@@ -30,14 +30,11 @@ if [ ! -f .gitignore ]; then
 	exit 1
 fi
 
-echo "--- Executing Tagging facility to determine TAG ---"
-source .travis/tagger.sh
-
-echo "--- Changelog generator and tagger script starting ---"
-# If tagger script hasn't produced a TAG, there is nothing to do so bail out happy
+echo "--- Changelog generator script starting ---"
+# If we dont have a produced TAG there is nothing to do, so bail out happy
 if [ -z "${GIT_TAG}" ]; then
-	echo "GIT_TAG is empty, nothing to do for now (Value: $GIT_TAG)"
-	exit 0
+	echo "GIT_TAG is empty, that is not suppose to happen (Value: $GIT_TAG)"
+	exit 1
 fi
 
 if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
@@ -55,8 +52,8 @@ echo "---- UPDATE VERSION FILE ----"
 echo "$GIT_TAG" >packaging/version
 git add packaging/version
 
-echo "---- GENERATE CHANGELOG -----"
-./.travis/generate_changelog_for_release.sh
+echo "---- Create CHANGELOG -----"
+./.travis/create_changelog.sh
 git add CHANGELOG.md
 
 echo "---- COMMIT AND PUSH CHANGES ----"

Diff for: .travis/generate_changelog_for_nightlies.sh

@@ -1,6 +1,6 @@
-#!/bin/bash
+#!/usr/bin/env bash
 #
-# Changelog generation scriptlet.
+# Changelog generation scriptlet, for nightlies
 #
 # Copyright: SPDX-License-Identifier: GPL-3.0-or-later
 #
@@ -20,39 +20,18 @@ fi
 LAST_TAG="$1"
 COMMITS_SINCE_RELEASE="$2"
 NEW_VERSION="${LAST_TAG}-$((COMMITS_SINCE_RELEASE + 1))-nightly"
-ORG=$(echo "$TRAVIS_REPO_SLUG" | cut -d '/' -f1)
-PROJECT=$(echo "$TRAVIS_REPO_SLUG" | cut -d '/' -f 2)
 GIT_MAIL=${GIT_MAIL:-"[email protected]"}
 GIT_USER=${GIT_USER:-"netdatabot"}
 PUSH_URL=$(git config --get remote.origin.url | sed -e 's/^https:\/\///')
 FAIL=0
-if [ -z ${GIT_TAG+x} ]; then
-	OPTS=""
-else
-	OPTS="--future-release ${GIT_TAG}"
-fi
-echo "We got $COMMITS_SINCE_RELEASE changes since $LAST_TAG, re-generating changelog"
 
 if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
 	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do here"
 	exit 0
 fi
 
-git checkout master
-git pull
-
-echo "Running project markmandel for github changelog generation"
-#docker run -it --rm -v "$(pwd)":/usr/local/src/your-app ferrarimarco/github-changelog-generator:1.14.3 \
-docker run -it -v "$(pwd)":/project markmandel/github-changelog-generator:latest \
-	--user "${ORG}" \
-	--project "${PROJECT}" \
-	--token "${GITHUB_TOKEN}" \
-	--since-tag "v1.10.0" \
-	--unreleased-label "**Next release**" \
-	--no-issues \
-	--exclude-labels "stale,duplicate,question,invalid,wontfix,discussion,no changelog" \
-	--max-issues 500 \
-	--bug-labels IGNOREBUGS
+echo "Running changelog creation mechanism"
+.travis/create_changelog.sh
 
 echo "Changelog created! Adding packaging/version(${NEW_VERSION}) and CHANGELOG.md to the repository"
 echo "${NEW_VERSION}" > packaging/version

Diff for: .travis/nightlies.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 #
 # This is the nightly changelog generation script
 # It is responsible for two major activities:
@@ -29,7 +29,7 @@ PREVIOUS_NIGHTLY_COUNT="$(rev <packaging/version | cut -d- -f 2 | rev)"
 
 # If no commits since release, just stop
 if [ "${COMMITS_SINCE_RELEASE}" == "${PREVIOUS_NIGHTLY_COUNT}" ]; then
-	echo "No changes since last nighthly release"
+	echo "No changes since last nighthly release, nothing else to do"
 	exit 0
 fi
 
@@ -39,6 +39,7 @@ if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
 fi
 
 echo "--- Running Changelog generation ---"
+echo "We got $COMMITS_SINCE_RELEASE changes since $LAST_TAG, re-generating changelog"
 NIGHTLIES_CHANGELOG_FAILED=0
 .travis/generate_changelog_for_nightlies.sh "${LAST_TAG}" "${COMMITS_SINCE_RELEASE}" || NIGHTLIES_CHANGELOG_FAILED=1
 

Diff for: .travis/tagger.sh

@@ -1,6 +1,4 @@
-#!/bin/bash
-#
-# Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh
+# #BASH library
 #
 # Tags are generated by searching for a keyword in last commit message. Keywords are:
 #  - [patch] or [fix] to bump patch number
@@ -12,33 +10,18 @@
 #   - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo
 #   - git-semver python package (pip install git-semver)
 #
-# Note: Exported variables needed by .travis/draft_release.sh
-#
 # Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh
 #
 # Copyright: SPDX-License-Identifier: GPL-3.0-or-later
 #
 # Author  : Pawel Krupa (paulfantom)
 # Author  : Pavlos Emm. Katsoulakis ([email protected])
 
-set -e
-
-
-# If we are not in netdata git repo, at the top level directory, fail
-TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
-CWD=$(git rev-parse --show-cdup || echo "")
-if [ -n "${CWD}" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
-    echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
-    echo "Changelog generation process aborted"
-    exit 1
-fi
-
-
 # Figure out what will be new release candidate tag based only on previous ones.
 # This assumes that RELEASES are in format of "v0.1.2" and prereleases (RCs) are using "v0.1.2-rc0"
 function set_tag_release_candidate() {
 	LAST_TAG=$(git semver)
-	echo "Last tag found is: ${LAST_TAG}"
+	echo "${0}: Last tag found is: ${LAST_TAG}"
 
 	if [[ $LAST_TAG =~ -rc* ]]; then
 		VERSION=$(echo "$LAST_TAG" | cut -d'-' -f 1)
@@ -47,31 +30,32 @@ function set_tag_release_candidate() {
 	else
 		VERSION="$(git semver --next-minor)"
 		RC=0
-		echo "Warning: Will set version to ${VERSION} (Last tag: ${LAST_TAG}) while tagged for release candidate generation"
+		echo "${0}: Warning: Will set version to ${VERSION} (Last tag: ${LAST_TAG}) while tagged for release candidate generation"
 	fi
+
 	GIT_TAG="v${VERSION}-rc${RC}"
+	echo "${0}: Generated a new tag, set to: (${GIT_TAG})"
 }
 
-echo "Determining TAG"
-
-# Check if current commit is tagged or not
-GIT_TAG=$(git tag --points-at)
-
-if [ -z "${GIT_TAG}" ]; then
-	git semver
-	# Figure out next tag based on commit message
-	echo "Last commit message: ${TRAVIS_COMMIT_MESSAGE}"
-	case "${TRAVIS_COMMIT_MESSAGE}" in
-	*"[netdata patch release]"*) GIT_TAG="v$(git semver --next-patch)" ;;
-	*"[netdata minor release]"*) GIT_TAG="v$(git semver --next-minor)" ;;
-	*"[netdata major release]"*) GIT_TAG="v$(git semver --next-major)" ;;
-	*"[netdata release candidate]"*) set_tag_release_candidate ;;
-	*)
-		echo "Keyword not detected. Exiting..."
-		exit 0
-		;;
-	esac
-fi
-
-echo "Setting up GIT_TAG to ${GIT_TAG}"
-export GIT_TAG
+function set_tag_for_release() {
+	echo "${0}: Checking for tag existence"
+	if [ -z "${GIT_TAG}" ]; then
+		echo "${0}: No tag was found, generating a new tag"
+		git semver
+
+		echo "${0}: Last commit message: ${TRAVIS_COMMIT_MESSAGE}"
+
+		# Figure out next tag based on commit message
+		case "${TRAVIS_COMMIT_MESSAGE}" in
+		*"[netdata patch release]"*) GIT_TAG="v$(git semver --next-patch)" ;;
+		*"[netdata minor release]"*) GIT_TAG="v$(git semver --next-minor)" ;;
+		*"[netdata major release]"*) GIT_TAG="v$(git semver --next-major)" ;;
+		*"[netdata release candidate]"*) set_tag_release_candidate ;;
+		*)
+			echo "${0}: Keyword not detected. Nothing to set for GIT_TAG"
+			;;
+		esac
+	else
+		echo "${0}: We seem to already have a GIT_TAG set to (${GIT_TAG})"
+	fi
+}