Skip to content

Commit 1bcaa3d

Browse files
TebaleloSbenedekimiroslavpojerlsulak
authored
Feature/#63 create build matrix and build action (#64)
* fixes #63 - added some configs variables for stb and jacoco code coverage. * fixes #63 - added more configs variables for stb * fixes #63 - added GitHub workflow Actions * fixes #63 - added license * fixes #63 * fixes #63 * * Fixed `ProjectMatrix` * * fixed bugs preventing project load and compilation * Fixes #63 - Emplementing jacoco setup * Fixes #63 - implementing jacoco setup * Fixes #63 - implementing jacoco setup * Fixes #63 - implementing jacoco setup * Fixes #63 - Fixing license * Fixes #63 - Fixing license * Fixes #63 - removed license_check and CODEOWNERS files * Update project/plugins.sbt Co-authored-by: miroslavpojer <[email protected]> * Update project/plugins.sbt Co-authored-by: miroslavpojer <[email protected]> * Fixes #63 - Added jacoco workflow * Fixes #63 - Added dependent_items.yml * Fixes #63 - Added dependent_items.yml * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * * Proposal of adding Jacoco support for server module. * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * * Proposal for change. * * Return back the title with spark major version now. * Fixes #63 * Fixes #63 - added license * Update .github/workflows/jacoco_check.yml Co-authored-by: Ladislav Sulak <[email protected]> * Update .github/workflows/jacoco_check.yml Co-authored-by: Ladislav Sulak <[email protected]> * Fixes #63 - update build.sbt * Fixes #63 - update publish.sbt * Fixes #63 - update publish.sbt * Fixes #63 * Fixes #63 * Fixes #63 * Fixes #63 * Closes #63 * Closes #63 - Added notes for jacoco coverage and realse in the readme file --------- Co-authored-by: David Benedeki <[email protected]> Co-authored-by: miroslavpojer <[email protected]> Co-authored-by: Ladislav Sulak <[email protected]>
1 parent c0cdbfc commit 1bcaa3d

22 files changed

+620
-41
lines changed

‎.github/workflows/assign_issue_to_project.yml

+15
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,18 @@
1+
#
2+
# Copyright 2021 ABSA Group Limited
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
#
15+
116
name: Auto Assign Issue to Project
217

318
on:

‎.github/workflows/build.yml

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#
2+
# Copyright 2021 ABSA Group Limited
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
name: Build
18+
19+
on:
20+
pull_request:
21+
branches: [ master ]
22+
types: [ opened, synchronize, reopened ]
23+
24+
jobs:
25+
test:
26+
runs-on: ubuntu-latest
27+
strategy:
28+
fail-fast: false
29+
30+
steps:
31+
- name: Checkout code
32+
uses: actions/checkout@v2
33+
- uses: coursier/cache-action@v5
34+
- name: Setup Scala
35+
uses: olafurpg/setup-scala@v10
36+
with:
37+
java-version: "[email protected]"
38+
- name: Build and run tests
39+
run: sbt test doc

‎.github/workflows/dependent_items.yml

+16
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
#
2+
# Copyright 2021 ABSA Group Limited
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
117
name: Dependent Issues
218

319
on:

‎.github/workflows/format_check.yml

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#
2+
# Copyright 2021 ABSA Group Limited
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
name: FormatCheck
18+
19+
on:
20+
pull_request:
21+
branches: [ master, develop ]
22+
types: [ assigned, opened, synchronize, reopened, labeled ]
23+
24+
jobs:
25+
scalafmt:
26+
runs-on: ubuntu-latest
27+
name: Scalafmt Check
28+
steps:
29+
- name: Checkout code
30+
uses: actions/checkout@v2
31+
with:
32+
fetch-depth: 0
33+
ref: ${{ github.event.pull_request.head.ref }}
34+
35+
- name: Setup Scala
36+
uses: olafurpg/setup-scala@v10
37+
with:
38+
java-version: "[email protected]"
39+
40+
- name: Run scalafmt And Print Diff
41+
continue-on-error: true
42+
run: sbt scalafmt scalafmtSbt && git diff --exit-code

‎.github/workflows/jacoco_check.yml

+88
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
#
2+
# Copyright 2021 ABSA Group Limited
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
name: JaCoCo report
18+
19+
on:
20+
pull_request:
21+
branches: [ master ]
22+
types: [ opened, edited, synchronize, reopened ]
23+
24+
jobs:
25+
test:
26+
name: Build and test
27+
runs-on: ubuntu-latest
28+
strategy:
29+
matrix:
30+
include:
31+
# The project supports Scala 2.11, 2.12 and the default version of spark2
32+
# The CI runs all tests suites for all supported Scala versions at build.yml
33+
# The codebase for all Scala versions is the same, so the coverage is calculated only once
34+
# Scala 2.12 is chosen since it is supported by the most wide range of Spark versions and
35+
# vendor distributions.
36+
- scala: 2.12.12
37+
scalaShort: "2.12"
38+
overall: 80.0
39+
changed: 80.0
40+
steps:
41+
- name: Checkout code
42+
uses: actions/checkout@v2
43+
- name: Setup Scala
44+
uses: olafurpg/setup-scala@v10
45+
with:
46+
java-version: "[email protected]"
47+
- name: Build and run tests
48+
run: sbt ++${{matrix.scala}} jacoco
49+
# Agent module code coverage
50+
- name: Add coverage to PR
51+
id: jacoco-agent
52+
uses: madrapps/[email protected]
53+
with:
54+
name: agent-jacoco-report
55+
paths: ${{ github.workspace }}/agent/target/spark2-jvm-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml
56+
token: ${{ secrets.GITHUB_TOKEN }}
57+
min-coverage-overall: ${{matrix.overall }}
58+
min-coverage-changed-file: ${{ matrix.changed }}
59+
title: JaCoCo agent module code coverage report - spark:2 - scala ${{ matrix.scala }}
60+
update-comment: true
61+
# server module code coverage
62+
- name: Add coverage to PR
63+
id: jacoco-server
64+
uses: madrapps/[email protected]
65+
with:
66+
name: server-jacoco-report
67+
paths: ${{ github.workspace }}/server/target/jvm-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml
68+
token: ${{ secrets.GITHUB_TOKEN }}
69+
min-coverage-overall: ${{matrix.overall }}
70+
min-coverage-changed-file: ${{ matrix.changed }}
71+
title: JaCoCo server module code coverage report - scala ${{ matrix.scala }}
72+
update-comment: true
73+
- name: Get the Coverage info
74+
run: |
75+
echo "Total agent module coverage ${{ steps.jacoco-agent.outputs.coverage-overall }}"
76+
echo "Changed Files coverage ${{ steps.jacoco-agent.outputs.coverage-changed-files }}"
77+
echo "Total sever module coverage ${{ steps.jacoco-server.outputs.coverage-overall }}"
78+
echo "Changed Files coverage ${{ steps.jacoco-server.outputs.coverage-changed-files }}"
79+
- name: Fail PR if changed files coverage is less than ${{ matrix.changed }}%
80+
if: |
81+
${{ steps.jacoco-agent.outputs.coverage-changed-files < 80.0 }} &&
82+
${{ steps.jacoco-server.outputs.coverage-changed-files < 80.0 }}
83+
uses: actions/github-script@v6
84+
with:
85+
script: |
86+
core.setFailed('Changed files coverage is less than ${{ matrix.changed }}%!')
87+
88+

‎.github/workflows/release.yml

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#
2+
# Copyright 2021 ABSA Group Limited
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
name: Release
18+
on:
19+
workflow_dispatch:
20+
21+
jobs:
22+
publish:
23+
runs-on: ubuntu-latest
24+
steps:
25+
- uses: actions/[email protected]
26+
with:
27+
fetch-depth: 0
28+
- uses: olafurpg/setup-scala@v13
29+
- run: sbt ci-release
30+
env:
31+
PGP_PASSPHRASE: ${{ secrets.PGP_PASSPHRASE }}
32+
PGP_SECRET: ${{ secrets.PGP_SECRET }}
33+
SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
34+
SONATYPE_USERNAME: ${{ secrets.SONATYPE_USERNAME }}

‎README.md

+15
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,18 @@ See `agent/README.md`.
1010
### Server `server/`
1111
An API under construction that communicates with AtumAgent and with the persisting storage. It also provides measure configuration to the `AtumAgent`.
1212
See `server/README.md`.
13+
14+
## How to generate Code coverage report
15+
```sbt
16+
sbt jacoco
17+
```
18+
Code coverage wil be generated on path:
19+
```
20+
{project-root}/atum-service/target/spark{spark_version}-jvm-{scala_version}/jacoco/report/html
21+
{project-root}/atum-service-test/target/jvm-{scala_version}/jacoco/report/html
22+
```
23+
24+
25+
## How to Release
26+
27+
Please see [this file](RELEASE.md) for more details.

‎RELEASE.md

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Release
2+
3+
Releases of this library are currently handled by [sbt-ci-release](https://github.com/sbt/sbt-ci-release).
4+
Please see its documentation for more details about how it works if you are interested to know more.
5+
6+
The actual deployments are triggered manually by the maintainers of this repository, using `workflow_dispatch` event
7+
trigger.
8+
9+
Once changes from a PR were reviewed and merged into the master branch, follow these steps:
10+
1. Create a new Git Tag and push it to the repository, to the master branch. For example,
11+
if you want to release a version 0.4.0 (note that we are using [Semantic Versioning](https://semver.org/)):
12+
13+
```shell
14+
git tag -a v0.4.0 -m "v0.4.0"
15+
git push origin v0.4.0
16+
```
17+
18+
2. In GitHub UI, go to the repository's **Actions** -> **Release** -> **Run workflow**, and under **Use workflow from**
19+
use **Tags** and find the tag you created in the previous step.
20+
21+
> **Important note**: don't run the workflow against the master branch, but against the tag.
22+
> `sbt-ci-release` plugin won't be able to correctly find tag, and it will think that you are trying
23+
> to do a snapshot release, not an actual release that should be synchronized with Maven Central.

‎agent/src/main/scala/za/co/absa/atum/agent/AtumAgent.scala

+16
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
/*
2+
* Copyright 2021 ABSA Group Limited
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
117
package za.co.absa.atum.agent
218

319
import za.co.absa.atum.agent.model.MeasureResult

‎agent/src/main/scala/za/co/absa/atum/agent/AtumContext.scala

+16
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
/*
2+
* Copyright 2021 ABSA Group Limited
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
117
package za.co.absa.atum.agent
218

319
import org.apache.spark.sql.DataFrame

‎agent/src/main/scala/za/co/absa/atum/agent/core/MeasurementProcessor.scala

+16
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
/*
2+
* Copyright 2021 ABSA Group Limited
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
117
package za.co.absa.atum.agent.core
218

319
import org.apache.spark.sql.DataFrame

‎agent/src/main/scala/za/co/absa/atum/agent/model/MeasureResult.scala

+16
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
/*
2+
* Copyright 2021 ABSA Group Limited
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
117
package za.co.absa.atum.agent.model
218

319
case class MeasureResult(measurement: Measurement, result: String)

‎agent/src/main/scala/za/co/absa/atum/agent/model/Measurement.scala

+16
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
/*
2+
* Copyright 2021 ABSA Group Limited
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
117
package za.co.absa.atum.agent.model
218

319
import org.apache.spark.sql.functions._

0 commit comments

Comments
 (0)