Skip to content

Commit

Permalink
Organize CLI by subcmd (#601)
Browse files Browse the repository at this point in the history
<!--
⚠️ If you do not respect this template, your pull request will be
closed.
⚠️ Your pull request title should be short detailed and understandable
for all.
⚠️ Also, please add a release note file using reno if the change needs
to be
  documented in the release notes.
⚠️ If your pull request fixes an open issue, please link to the issue.

- [ ] I have added the tests to cover my changes.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the CONTRIBUTING document.
-->

### Summary

Clean up `manager.py` by re-organizing it by subcmd.

### Details and comments

- [x] ci subcmd
- [x] tests subcmd
- [x] website subcmd
- [x] members subcmd
- [x] Update docs
- [x] Update tests
- [x] Update CI/tox

---
Closes #230

---------

Co-authored-by: Eric Arellano <[email protected]>
  • Loading branch information
mickahell and Eric-Arellano committed Nov 28, 2023
1 parent 7a50000 commit a8bcecc
Show file tree
Hide file tree
Showing 20 changed files with 426 additions and 332 deletions.
10 changes: 5 additions & 5 deletions .github/actions/run-tests/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,35 +28,35 @@ runs:
run: |
if [ "${{ inputs.test_type }}" == "standard" ]; then
echo "Running repository tests for ${{inputs.repo_url}}..."
python manager.py python_standard_tests ${{inputs.repo_url}} \
python manager.py tests python_standard_tests ${{inputs.repo_url}} \
--run_name=${{inputs.run_name}} \
--python_version=${{inputs.tox_env}} \
--tier=${{ inputs.tier }} \
--logs_link=${{ inputs.logs_link }}
fi
if [ "${{ inputs.test_type }}" == "stable" ]; then
echo "Running Qiskit stable tests for ${{inputs.repo_url}}..."
python manager.py python_stable_tests ${{inputs.repo_url}} \
python manager.py tests python_stable_tests ${{inputs.repo_url}} \
--run_name=${{inputs.run_name}} \
--python_version=${{inputs.tox_env}} \
--tier=${{ inputs.tier }} \
--logs_link=${{ inputs.logs_link }}
fi
if [ "${{ inputs.test_type }}" == "development" ]; then
echo "Running Qiskit dev tests for ${{inputs.repo_url}}..."
python manager.py python_dev_tests ${{inputs.repo_url}} \
python manager.py tests python_dev_tests ${{inputs.repo_url}} \
--run_name=${{inputs.run_name}} \
--python_version=${{inputs.tox_env}} \
--tier=${{ inputs.tier }} \
--logs_link=${{ inputs.logs_link }}
fi
if [ "${{ inputs.test_type }}" == "lint" ]; then
echo "Running lint tests for ${{inputs.repo_url}}..."
python manager.py python_styles_check ${{inputs.repo_url}} --run_name=${{inputs.run_name}} --tier=${{inputs.tier}} --style_type="pylint"
python manager.py tests python_styles_check ${{inputs.repo_url}} --run_name=${{inputs.run_name}} --tier=${{inputs.tier}} --style_type="pylint"
fi
if [ "${{ inputs.test_type }}" == "coverage" ]; then
echo "Running coverage tests for ${{inputs.repo_url}}..."
python manager.py python_coverage ${{inputs.repo_url}} --run_name=${{inputs.run_name}} --tier=${{inputs.tier}} --coverage_type=""
python manager.py tests python_coverage ${{inputs.repo_url}} --run_name=${{inputs.run_name}} --tier=${{inputs.tier}} --coverage_type=""
fi
shell: bash
outputs:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ecosystem-badge-and-stars-update.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,10 @@ jobs:
pip install -r requirements.txt
- name: Update badges
run: python manager.py update_badges
run: python manager.py members update_badges

- name: Update stars
run: python manager.py update_stars
run: python manager.py members update_stars

- name: Create PR for stars and badges update
id: cpr
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/ecosystem-batch-repo-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,18 +37,18 @@ jobs:
branch: ${{ steps.vars.outputs.pr_branch_name }}
- name: Get project for check
id: repos
run: python manager.py expose_all_project_to_actions
run: python manager.py ci expose_all_project_to_actions
- name: Launch separate workflows for each project check
run: |
IFS=',' read -r -a repositories <<< ${{ steps.repos.outputs.repositories }}
IFS=',' read -r -a tiers <<< ${{ steps.repos.outputs.tiers }}
for (( i=0; i<${#repositories[*]}; ++i)); do
echo "Launching separate workflow for ${repositories[$i]} in ${tiers[$i]} tier"
python manager.py dispatch_check_workflow \
--repo_url=${repositories[$i]} \
--branch_name=${{ steps.vars.outputs.pr_branch_name }} \
--tier=${tiers[$i]} \
--token=${{ secrets.GITHUB_TOKEN }} \
--owner=${{ github.repository_owner }}
python manager.py ci dispatch_check_workflow \
--repo_url=${repositories[$i]} \
--branch_name=${{ steps.vars.outputs.pr_branch_name }} \
--tier=${tiers[$i]} \
--token=${{ secrets.GITHUB_TOKEN }} \
--owner=${{ github.repository_owner }}
done
2 changes: 1 addition & 1 deletion .github/workflows/ecosystem-batch-save-temp-res-to-db.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Merge files and save to db
run: python manager.py process_temp_test_results_files --folder_name=${{ github.event.inputs.batch_folder }}
run: python manager.py tests process_temp_test_results_files --folder_name=${{ github.event.inputs.batch_folder }}
- name: Commit changes
run: |
git pull --rebase --autostash
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ecosystem-main_repos_fetch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
pip install -r requirements-dev.txt
- name: Fetch test results
run: python manager.py fetch_and_update_main_tests_results
run: python manager.py tests fetch_and_update_main_tests_results

- name: Create PR for batch checks
id: cpr
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ecosystem-project-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
logs_link: https://github.com/${{github.repository}}/actions/runs/${{ github.run_id }}

- name: Merge files and save to db
run: python manager.py process_temp_test_results_files --folder_name=${{ github.event.client_payload.branch_name }}
run: python manager.py tests process_temp_test_results_files --folder_name=${{ github.event.client_payload.branch_name }}

- name: Commit changes
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ecosystem-recompile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:

- name: Recompile and push
run: |
python -m manager recompile
python -m manager members recompile
git config user.name github-actions
git config user.email [email protected]
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ecosystem-submission.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
id: parse-issue
env:
ISSUE_BODY: ${{ github.event.issue.body }}
run: python manager.py parser_issue --body="$ISSUE_BODY"
run: python manager.py ci parser_issue --body="$ISSUE_BODY"
- name: Tests stable check
id: stable
uses: ./.github/actions/run-tests
Expand Down Expand Up @@ -81,7 +81,7 @@ jobs:
SUBMISSION_LABELS: ${{ steps.parse-issue.outputs.SUBMISSION_LABELS }}
SUBMISSION_WEBSITE: ${{ steps.parse-issue.outputs.SUBMISSION_WEBSITE }}
run: |
python manager.py add_repo_2db --repo_name="$SUBMISSION_NAME" \
python manager.py members add_repo_2db --repo_name="$SUBMISSION_NAME" \
--repo_link="$SUBMISSION_REPO" \
--repo_description="$SUBMISSION_DESCRIPTION" \
--repo_licence="$SUBMISSION_LICENCE" \
Expand Down
2 changes: 1 addition & 1 deletion .pylintrc
Original file line number Diff line number Diff line change
Expand Up @@ -612,7 +612,7 @@ max-returns=6
max-statements=50

# Minimum number of public methods for a class (see R0903).
min-public-methods=2
min-public-methods=1


[EXCEPTIONS]
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@ pip install -r requirements-dev.txt

# Running the tests
1. To run tests against the stable version of qiskit <br/>
<code> python manager.py python_stable_tests <url_of_the github_repository> --python_version=py39 --run_name="stable"</code>
<code> python manager.py tests python_stable_tests <url_of_the github_repository> --python_version=py39 --run_name="stable"</code>
2. To run tests against the dev version of qiskit <br/>
<code> python manager.py python_dev_tests <url_of_the github_repository> --python_version=py39 --run_name="dev"</code>
<code> python manager.py tests python_dev_tests <url_of_the github_repository> --python_version=py39 --run_name="dev"</code>
3. To run tests within repository <br/>
<code> python manager.py python_standard_tests <url_of_the github_repository> --python_version=py39 --run_name="standard"</code>
<code> python manager.py tests python_standard_tests <url_of_the github_repository> --python_version=py39 --run_name="standard"</code>

# Performing style checks
- Run for style checks
Expand Down
16 changes: 8 additions & 8 deletions docs/project_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@ Entrypoint is ``manager.py`` file in the root of repository.

Example of commands:
```shell
python manager.py python_dev_tests https://github.com/IceKhan13/demo-implementation --python_version=py39
python manager.py python_stable_tests https://github.com/IceKhan13/demo-implementation --python_version=py39
python manager.py tests python_dev_tests https://github.com/IceKhan13/demo-implementation --python_version=py39
python manager.py tests python_stable_tests https://github.com/IceKhan13/demo-implementation --python_version=py39
```
or in general
```shell
python manager.py <NAME_OF_FUNCTION_IN_MANAGER_FILE> <POSITIONAL_ARGUMENT> [FLAGS]
python manager.py <CMD> <NAME_OF_FUNCTION_IN_MANAGER_FILE> <POSITIONAL_ARGUMENT> [FLAGS]
```

### Adding project to the ecosystem
Expand Down Expand Up @@ -119,13 +119,13 @@ file. This file should be compiled automatically by an action on pushing to
`main`, but you can also compile it locally (e.g. for testing) using

```sh
python -m manager recompile
python -m manager members recompile
```

You shouldn't edit `members.json` manually.

If you somehow get a merge conflict in `members.json`, don't try to manually
resolve the conflict. Instead, merge the branch, then run `python -m manager
resolve the conflict. Instead, merge the branch, then run `python -m manager members
recompile` and add the file to resolve the conflict.

### Tests
Expand All @@ -136,21 +136,21 @@ There are 3 type of tests for project: `STANDARD`, `DEV` and `STABLE`.

CLI command:
```shell
python manager.py python_standard_tests <REPO_URL> --run_name=<NAME_OF_RUN> --python_version=py39 --tier=<TIER>
python manager.py tests python_standard_tests <REPO_URL> --run_name=<NAME_OF_RUN> --python_version=py39 --tier=<TIER>
```

`DEV` - runs tests with default requirements for project + dev version of qiskit-terra installed

CLI command:
```shell
python manager.py python_dev_tests <REPO_URL> --run_name=<NAME_OF_RUN> --python_version=py39 --tier=<TIER>
python manager.py tests python_dev_tests <REPO_URL> --run_name=<NAME_OF_RUN> --python_version=py39 --tier=<TIER>
```

`STABLE` - runs tests with default requirements for project + latest stable version of qiskit-terra installed

CLI command:
```shell
python manager.py python_stable_tests <REPO_URL> --run_name=<NAME_OF_RUN> --python_version=py39 --tier=<TIER>
python manager.py tests python_stable_tests <REPO_URL> --run_name=<NAME_OF_RUN> --python_version=py39 --tier=<TIER>
```

You can see full setup on test running in [GitHub Action](https://github.com/qiskit-community/ecosystem/blob/main/.github/actions/run-tests/action.yml)
Expand Down
1 change: 0 additions & 1 deletion ecosystem/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
"""Ecosystem main module."""
from .manager import Manager
5 changes: 5 additions & 0 deletions ecosystem/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"""CLI."""
from .members import CliMembers
from .website import CliWebsite
from .ci import CliCI
from .tests import CliTests
121 changes: 121 additions & 0 deletions ecosystem/cli/ci.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
"""CliCI class for controlling all CLI functions."""
import os
from typing import Optional

import requests

from ecosystem.daos import DAO
from ecosystem.models import Tier
from ecosystem.utils import logger, parse_submission_issue
from ecosystem.utils.utils import set_actions_output


class CliCI:
"""CliCI class.
Entrypoint for all CLI CI commands.
Each public method of this class is CLI command
and arguments for method are options/flags for this command.
Ex: `python manager.py ci parser_issue --body="<SOME_MARKDOWN>"`
"""

def __init__(self, root_path: Optional[str] = None):
"""CliCI class."""
self.current_dir = root_path or os.path.abspath(os.getcwd())
self.resources_dir = "{}/ecosystem/resources".format(self.current_dir)
self.dao = DAO(path=self.resources_dir)
self.logger = logger

def dispatch_check_workflow(
self,
repo_url: str,
branch_name: str,
tier: str,
token: str,
owner: str = "qiskit-community",
repo: str = "ecosystem",
) -> bool:
"""Dispatch event to trigger check workflow.
Args:
repo_url: url of the repo
branch_name: name of the branch
tier: tier of the project
token: token base on the date
owner: "qiskit-community" parameters
repo: "ecosystem"
Return: true
"""
url = "https://api.github.com/repos/{owner}/{repo}/dispatches".format(
owner=owner, repo=repo
)
repo_split = repo_url.split("/")
repo_name = repo_split[-1]

# run each type of tests in same workflow
response = requests.post(
url,
json={
"event_type": "check_project",
"client_payload": {
"repo_url": repo_url,
"repo_name": repo_name,
"branch_name": branch_name,
"tier": tier,
},
},
headers={
"Authorization": "token {}".format(token),
"Accept": "application/vnd.github.v3+json",
},
)
if response.ok:
self.logger.info("Success response on dispatch event. %s", response.text)
else:
self.logger.warning(
"Something wend wrong with dispatch event: %s", response.text
)
return True

def expose_all_project_to_actions(self):
"""Exposes all project for github actions."""
repositories = []
tiers = []
for tier in Tier.non_main_tiers():
for repo in self.dao.get_repos_by_tier(tier):
if not repo.skip_tests:
repositories.append(repo.url)
tiers.append(repo.tier)
set_actions_output(
[("repositories", ",".join(repositories)), ("tiers", ",".join(tiers))]
)

@staticmethod
def parser_issue(body: str) -> None:
"""Command for calling body issue parsing function.
Args:
body: body of the created issue
Returns:
logs output
We want to give the result of the parsing issue to the GitHub action
"""

parsed_result = parse_submission_issue(body)

to_print = [
("SUBMISSION_NAME", parsed_result.name),
("SUBMISSION_REPO", parsed_result.url),
("SUBMISSION_DESCRIPTION", parsed_result.description),
("SUBMISSION_LICENCE", parsed_result.licence),
("SUBMISSION_CONTACT", parsed_result.contact_info),
("SUBMISSION_ALTERNATIVES", parsed_result.alternatives),
("SUBMISSION_AFFILIATIONS", parsed_result.affiliations),
("SUBMISSION_LABELS", parsed_result.labels),
("SUBMISSION_WEBSITE", parsed_result.website),
]

set_actions_output(to_print)
Loading

0 comments on commit a8bcecc

Please sign in to comment.