Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AZ ACR CSSC Extension #7749

Draft
wants to merge 69 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
d9e88db
add code for cssc
pwalecha Jun 5, 2024
cdb363c
# Correct the Delete repository method - it is only deleting Tag
pwalecha Jun 5, 2024
b1f89dc
update new yamls
pwalecha Jun 6, 2024
a898153
add support for streaming logging as well
pwalecha Jun 6, 2024
5a0ce0d
add support for logging
pwalecha Jun 6, 2024
7b86532
add support for streamed logs only giving acr-cli logs
pwalecha Jun 6, 2024
5c783c9
Delete old files
pwalecha Jun 10, 2024
968fafd
fix dry run
pwalecha Jun 10, 2024
d7538ba
remove redundant files
pwalecha Jun 10, 2024
add2048
simplify print code
pwalecha Jun 10, 2024
71680a3
add user confirmation before deletion
cegraybl Jun 10, 2024
dbd5113
Merge branch 'users/puwalech/acrcssc' of https://github.com/pwalecha/…
cegraybl Jun 10, 2024
be52009
allow update to be done on cadence or config (or both)
cegraybl Jun 10, 2024
1848240
fix:
pwalecha Jun 10, 2024
8a72e2d
merge latest
pwalecha Jun 10, 2024
94403bb
remove redundant code
pwalecha Jun 10, 2024
69788e2
add template file paths to the extension build
cegraybl Jun 10, 2024
8919cb8
Merge branch 'users/puwalech/acrcssc' of https://github.com/pwalecha/…
cegraybl Jun 10, 2024
2bfad35
fix minor bugs:
pwalecha Jun 11, 2024
e04fd35
fix update issue
pwalecha Jun 11, 2024
63cd2f9
update from warning to print
pwalecha Jun 11, 2024
13b1373
fix:
pwalecha Jun 12, 2024
84fc89e
fix minor issues, supress stderror from acr login
cegraybl Jun 12, 2024
d9360b3
add logging for better experience
pwalecha Jun 12, 2024
01667f2
Merge branch 'users/puwalech/acrcssc' of https://github.com/pwalecha/…
pwalecha Jun 12, 2024
f535211
remove redundant line
pwalecha Jun 12, 2024
f0e71c3
fix minor bugs:
pwalecha Jun 12, 2024
d48e9d7
update yaml for trigger task to 0.11, update filter parameter for dryrun
cegraybl Jun 13, 2024
ffa4998
fix acr-cli version and env variable for dryrun yaml
cegraybl Jun 13, 2024
e87f0ec
fix alot of style checks, some pylint issues
cegraybl Jun 13, 2024
f96a295
fix linter issue with missing help
cegraybl Jun 13, 2024
6dcbbbd
use download logs for dry_run
pwalecha Jun 14, 2024
90147ea
Merge branch 'users/puwalech/acrcssc_v2' of https://github.com/pwalec…
pwalecha Jun 14, 2024
081dda3
Fix temporary directory creation
pwalecha Jun 14, 2024
12e5bc4
add more unit test cases
pwalecha Jun 15, 2024
b1b8603
add a sample scenario test
pwalecha Jun 16, 2024
5d9834a
fix minor verbiage issues
pwalecha Jun 17, 2024
abb0683
fix issue where the 'supply-chain' section is not marked as 'preview'…
cegraybl Jun 17, 2024
cea7d4f
use the Task client to get values for OS & platform from the centrali…
cegraybl Jun 17, 2024
a028585
fix another handful of lint and style issues
cegraybl Jun 17, 2024
6570d07
add a scenario test.
pwalecha Jun 18, 2024
4537b3e
Merge branch 'users/puwalech/acrcssc_v2' of https://github.com/pwalec…
pwalecha Jun 18, 2024
946ab51
fix pylint issues
pwalecha Jun 21, 2024
51b795b
update task yaml files, update version to use latest cssc acr-cli image
cegraybl Jun 24, 2024
22d0d4d
fix per review comments:
pwalecha Jun 25, 2024
ad12541
merge latest
pwalecha Jun 25, 2024
7cefb1b
update to initial version
pwalecha Jun 27, 2024
32fb023
update to the same initial version
pwalecha Jun 27, 2024
b5b2f86
fix the build issues
pwalecha Jul 1, 2024
1d81086
Fix style issue
pwalecha Jul 1, 2024
4bbc6b5
comment failing test case
pwalecha Jul 1, 2024
ef073ad
fix the style issue
pwalecha Jul 1, 2024
0080d77
fix breaking unit test cases
pwalecha Jul 1, 2024
49df558
fix the dry-run yaml
pwalecha Jul 2, 2024
b7c1cde
fix 28610631, improve output message to list a specific cli command t…
cegraybl Jul 9, 2024
537eac6
fix 28610548: create with dry run will now check if the workflow alre…
cegraybl Jul 9, 2024
caea45f
add timeout for 60 minutes
pwalecha Jul 9, 2024
38e1a8e
fix 28610649, re-running workflow delete after a failed deletion shou…
cegraybl Jul 10, 2024
507d8be
standardize string replacement in the extension
cegraybl Jul 10, 2024
f1e4209
Multiple bug fixes in yamls related to patch failing with repos havin…
Jul 16, 2024
b2706a4
Changes to skip patching if image has eosl=true and fixed some other …
Jul 19, 2024
d422e09
Merge branch 'users/puwalech/acrcssc_v2' of https://github.com/pwalec…
pwalecha Jul 19, 2024
8d51d2a
fix breaking test case
pwalecha Jul 19, 2024
f664af9
Remove help for dry-run
pwalecha Jul 19, 2024
d93502f
fix style issues
pwalecha Jul 19, 2024
f1adf73
fix linter and style issues
pwalecha Jul 19, 2024
f1aa1b0
increase copatimeout to 30 m
pwalecha Jul 19, 2024
d553231
Update help for cadence.
pwalecha Jul 25, 2024
641705d
Update recommendation message to reflect max value of cadence
pwalecha Jul 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/acrcssc/HISTORY.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
.. :changelog:

Release History
===============

0.1.0
++++++
* Initial release.
5 changes: 5 additions & 0 deletions src/acrcssc/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Microsoft Azure CLI 'acrcssc' Extension
==========================================

This package is for the 'acrcssc' extension.
i.e. 'az acrcssc'
32 changes: 32 additions & 0 deletions src/acrcssc/azext_acrcssc/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------

from azure.cli.core import AzCommandsLoader

from azext_acrcssc._help import helps # pylint: disable=unused-import


class AcrcsscCommandsLoader(AzCommandsLoader):

def __init__(self, cli_ctx=None):
from azure.cli.core.commands import CliCommandType
from azext_acrcssc._client_factory import cf_acr
acrcssc_custom = CliCommandType(
operations_tmpl='azext_acrcssc.custom#{}',
client_factory=cf_acr)
super(AcrcsscCommandsLoader, self).__init__(cli_ctx=cli_ctx,
custom_command_type=acrcssc_custom)

def load_command_table(self, args):
from azext_acrcssc.commands import load_command_table
load_command_table(self, args)
return self.command_table

def load_arguments(self, command):
from azext_acrcssc._params import load_arguments
load_arguments(self, command)


COMMAND_LOADER_CLS = AcrcsscCommandsLoader
61 changes: 61 additions & 0 deletions src/acrcssc/azext_acrcssc/_client_factory.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------
from azure.cli.core.commands.client_factory import get_mgmt_service_client
from azure.cli.core.profiles import ResourceType
from azure.mgmt.containerregistry import ContainerRegistryManagementClient
from .helper._constants import (
ACR_API_VERSION_2023_01_01_PREVIEW,
ACR_API_VERSION_2019_06_01_PREVIEW
)

from azure.mgmt.authorization import AuthorizationManagementClient


def cf_acr(cli_ctx, *_) -> ContainerRegistryManagementClient:
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_CONTAINERREGISTRY,
api_version=ACR_API_VERSION_2023_01_01_PREVIEW)


def cf_acr_registries(cli_ctx, *_) -> ContainerRegistryManagementClient:
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_CONTAINERREGISTRY,
api_version=ACR_API_VERSION_2023_01_01_PREVIEW).registries


def cf_acr_tasks(cli_ctx, *_):
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_CONTAINERREGISTRY,
api_version=ACR_API_VERSION_2019_06_01_PREVIEW).tasks


def cf_acr_registries_tasks(cli_ctx, *_):
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_CONTAINERREGISTRY,
api_version=ACR_API_VERSION_2019_06_01_PREVIEW).registries


def cf_acr_taskruns(cli_ctx, *_):
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_CONTAINERREGISTRY,
api_version=ACR_API_VERSION_2019_06_01_PREVIEW).task_runs


def cf_acr_runs(cli_ctx, *_):
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_CONTAINERREGISTRY,
api_version=ACR_API_VERSION_2019_06_01_PREVIEW).runs


def cf_resources(cli_ctx, subscription_id=None):
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_RESOURCE_RESOURCES,
subscription_id=subscription_id)


def cf_authorization(cli_ctx, subscription_id=None) -> AuthorizationManagementClient:
return get_mgmt_service_client(cli_ctx,
ResourceType.MGMT_AUTHORIZATION,
subscription_id=subscription_id, api_version="2022-04-01")
50 changes: 50 additions & 0 deletions src/acrcssc/azext_acrcssc/_help.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# coding=utf-8
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------

from knack.help_files import helps # pylint: disable=unused-import

helps['acr supply-chain'] = """
type: group
short-summary: Commands to manage acr supply chain resources.
"""

helps['acr supply-chain workflow'] = """
type: group
short-summary: Commands to manage acr supply chain workflows.
"""

helps['acr supply-chain workflow create'] = """
type: command
short-summary: Create acr supply chain workflow.
examples:
- name: Create acr supply chain workflow
text: az acr supply-chain workflow create -r $MyRegistry -g $MyResourceGroup \
--type continuouspatchv1 --cadence 1d --config path-to-config-file
"""
helps['acr supply-chain workflow update'] = """
type: command
short-summary: Update acr supply chain workflow.
examples:
- name: Updates acr supply chain workflow
text: az acr supply-chain workflow update -r $MyRegistry -g $MyResourceGroup --type \
continuouspatchv1 --cadence 1d --config path-to-config-file
"""

helps['acr supply-chain workflow show'] = """
type: command
short-summary: Show acr supply chain workflow tasks.
examples:
- name: Show all acr supply chain workflow
text: az acr supply-chain workflow show -r $MyRegistry -g $MyResourceGroup --type continuouspatchv1
"""

helps['acr supply-chain workflow delete'] = """
type: command
short-summary: Delete acr supply chain workflow.
examples:
- name: Delete acr supply chain workflow and associated configuration files
text: az acr supply-chain workflow delete -r $MyRegistry -g $MyResourceGroup --type continuouspatchv1
"""
32 changes: 32 additions & 0 deletions src/acrcssc/azext_acrcssc/_params.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------
# pylint: disable=line-too-long
from azure.cli.core import AzCommandsLoader
from azure.cli.core.commands.parameters import (get_resource_name_completion_list, get_three_state_flag, get_enum_type)

from azure.cli.command_modules.acr._constants import REGISTRY_RESOURCE_TYPE

from azure.cli.command_modules.acr._validators import validate_registry_name


def load_arguments(self: AzCommandsLoader, _):
from .helper._constants import CSSCTaskTypes

with self.argument_context("acr supply-chain workflow") as c:
c.argument('resource_group', options_list=['--resource-group', '-g'], help='Name of resource group.You can configure the default group using `az configure --defaults group=<name>`', completer=get_resource_name_completion_list(REGISTRY_RESOURCE_TYPE), configured_default='acr', validator=validate_registry_name)
c.argument('registry_name', options_list=['--registry', '-r'], help='The name of the container registry. It should be specified in lower case. You can configure the default registry name using `az configure --defaults acr=<registry name>`', completer=get_resource_name_completion_list(REGISTRY_RESOURCE_TYPE), configured_default='acr', validator=validate_registry_name)
c.argument("workflow_type", arg_type=get_enum_type(CSSCTaskTypes), options_list=['--type', '-t'], help="Type of workflow task.", required=True)

with self.argument_context("acr supply-chain workflow create") as c:
c.argument("config", options_list=["--config"], help="Configuration file path containing the json schema for the list of repositories and tags to filter within the registry. Schema example:{\"repositories\":[{\"repository\":\"alpine\",\"tags\":[\"tag1\",\"tag2\"],\"enabled\":true},{\"repository\":\"python\",\"tags\":[\"*\"],\"enabled\":false}], \"version\": \"v1\"}", required=True)
c.argument("cadence", options_list=["--cadence"], help="Cadence to run the scan and patching task. E.g. `<n>d` where <n> is the number of days between each run. Max value is 30d.", required=True)
c.argument("defer_immediate_run", options_list=["--defer-immediate-run"], help="Use this flag to defer immediately running of selected workflow task. Default value: false.", arg_type=get_three_state_flag(), required=False)
c.argument("dryrun", options_list=["--dry-run"], help="Use this flag to see the qualifying repositories and tags that would be affected by the workflow. Default value: false. 'config' parameter is mandatory to provide with dry-run", arg_type=get_three_state_flag(), required=False)

with self.argument_context("acr supply-chain workflow update") as c:
c.argument("config", options_list=["--config"], help="Configuration file path containing the json schema for the list of repositories and tags to filter within the registry. Schema example:{\"repositories\":[{\"repository\":\"alpine\",\"tags\":[\"tag1\",\"tag2\"],\"enabled\":true},{\"repository\":\"python\",\"tags\":[\"*\"],\"enabled\":false}], \"version\": \"v1\"}", required=False)
c.argument("cadence", options_list=["--cadence"], help="Cadence to run the scan and patching task. E.g. `<n>d` where n is the number of days between each run. Max value is 30d.", required=True)
c.argument("defer_immediate_run", options_list=["--defer-immediate-run"], help="Use this flag to defer immediately running of selected workflow task. Default value: false.", arg_type=get_three_state_flag(), required=False)
c.argument("dryrun", options_list=["--dry-run"], help="Use this flag to see the qualifying repositories and tags that would be affected by the workflow. Default value: false. 'config' parameter is mandatory to provide with dry-run", arg_type=get_three_state_flag(), required=False)
135 changes: 135 additions & 0 deletions src/acrcssc/azext_acrcssc/_validators.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------
# pylint: disable=line-too-long
# pylint: disable=broad-exception-caught
# pylint: disable=logging-fstring-interpolation
import json
import os
import re
from knack.log import get_logger
from azure.cli.command_modules.acr.repository import acr_repository_show
from .helper._constants import (
BEARER_TOKEN_USERNAME,
CSSC_WORKFLOW_POLICY_REPOSITORY,
CONTINUOSPATCH_OCI_ARTIFACT_CONFIG,
CONTINUOUSPATCH_CONFIG_SCHEMA_V1,
CONTINUOUSPATCH_CONFIG_SCHEMA_SIZE_LIMIT,
CONTINUOSPATCH_ALL_TASK_NAMES,
ERROR_MESSAGE_INVALID_TIMESPAN_FORMAT,
ERROR_MESSAGE_INVALID_TIMESPAN_VALUE,
RESOURCE_GROUP,
SUBSCRIPTION)
from .helper._constants import CSSCTaskTypes, ERROR_MESSAGE_INVALID_TASK, RECOMMENDATION_CADENCE
from .helper._ociartifactoperations import _get_acr_token
from azure.mgmt.core.tools import (parse_resource_id)
from azure.cli.core.azclierror import InvalidArgumentValueError
from ._client_factory import cf_acr_tasks
logger = get_logger(__name__)


def validate_continuouspatch_config_v1(config_path):
_validate_continuouspatch_file(config_path)
_validate_continuouspatch_json(config_path)


def _validate_continuouspatch_file(config_path):
if not os.path.exists(config_path):
raise InvalidArgumentValueError(f"Config path file: {config_path} does not exist in the path specified")
if not os.path.isfile(config_path):
raise InvalidArgumentValueError(f"Config path file: {config_path} is not a valid file")
if os.path.getsize(config_path) > CONTINUOUSPATCH_CONFIG_SCHEMA_SIZE_LIMIT:
raise InvalidArgumentValueError(f"Config path file: {config_path} is too large. Max size limit is 10 MB")
if os.path.getsize(config_path) == 0:
raise InvalidArgumentValueError(f"Config path file: {config_path} is empty")
if not os.access(config_path, os.R_OK):
raise InvalidArgumentValueError(f"Config path file: '{config_path}' is not readable")


def _validate_continuouspatch_json(config_path):
from jsonschema import validate
try:
with open(config_path, 'r') as f:
config = json.load(f)
validate(config, CONTINUOUSPATCH_CONFIG_SCHEMA_V1)
except Exception as e:
logger.debug(f"Error validating the continuous patch config file: {e}")
raise InvalidArgumentValueError("File used for --config is not a valid config JSON file. Use --help to see the schema of the config file.")
finally:
f.close()


def check_continuous_task_exists(cmd, registry):
exists = False
for task_name in CONTINUOSPATCH_ALL_TASK_NAMES:
exists = exists or _check_task_exists(cmd, registry, task_name)
return exists


def check_continuous_task_config_exists(cmd, registry):
# A client cannot be used in this situation because the 'show registry/image'
# is a data plane operation and the az cli does not include the data plane API.
subscription = parse_resource_id(registry.id)[SUBSCRIPTION]
try:
token = _get_acr_token(registry.name, subscription)
acr_repository_show(
cmd=cmd,
registry_name=registry.name,
repository=f"{CSSC_WORKFLOW_POLICY_REPOSITORY}/{CONTINUOSPATCH_OCI_ARTIFACT_CONFIG}",
username=BEARER_TOKEN_USERNAME,
password=token)
except Exception as exception:
if hasattr(exception, 'status_code') and exception.status_code == 404:
return False
# report on the error only if we get something other than 404
logger.debug(f"Failed to find config {CSSC_WORKFLOW_POLICY_REPOSITORY}/{CONTINUOSPATCH_OCI_ARTIFACT_CONFIG} from registry {registry.name} : {exception}")
raise
return True


def _check_task_exists(cmd, registry, task_name=""):
acrtask_client = cf_acr_tasks(cmd.cli_ctx)
resourceid = parse_resource_id(registry.id)
resource_group = resourceid[RESOURCE_GROUP]

try:
task = acrtask_client.get(resource_group, registry.name, task_name)
except Exception as exception:
logger.debug(f"Failed to find task {task_name} from registry {registry.name} : {exception}")
return False

if task is not None:
return True
return False


def _validate_cadence(cadence):
# during update, cadence can be null if we are only updating the config
if cadence is None:
return
# Extract the numeric value and unit from the timespan expression
match = re.match(r'(\d+)(d)$', cadence)
if not match:
raise InvalidArgumentValueError(error_msg=ERROR_MESSAGE_INVALID_TIMESPAN_FORMAT, recommendation=RECOMMENDATION_CADENCE)
if match is not None:
value = int(match.group(1))
unit = match.group(2)
if unit == 'd' and (value < 1 or value > 30): # day of the month
raise InvalidArgumentValueError(error_msg=ERROR_MESSAGE_INVALID_TIMESPAN_VALUE, recommendation=RECOMMENDATION_CADENCE)


def validate_inputs(cadence, config_file_path=None):
_validate_cadence(cadence)
if config_file_path is not None:
validate_continuouspatch_config_v1(config_file_path)


def validate_task_type(task_type):
if (task_type not in [item.value for item in CSSCTaskTypes]):
raise InvalidArgumentValueError(error_msg=ERROR_MESSAGE_INVALID_TASK)


def validate_cssc_optional_inputs(cssc_config_path, cadence):
if cssc_config_path is None and cadence is None:
raise InvalidArgumentValueError(error_msg="Provide at least one parameter to update: --cadence or --config")
4 changes: 4 additions & 0 deletions src/acrcssc/azext_acrcssc/azext_metadata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"azext.isPreview": true,
"azext.minCliCoreVersion": "2.15.0"
}
18 changes: 18 additions & 0 deletions src/acrcssc/azext_acrcssc/commands.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------

# pylint: disable=line-too-long
from azext_acrcssc._client_factory import cf_acr


def load_command_table(self, _):
# required to mark the command as 'preview', can be expanded if additional commands are added
self.command_group("acr supply-chain", client_factory=cf_acr, is_preview=True)

with self.command_group("acr supply-chain workflow", client_factory=cf_acr, is_preview=True) as g:
g.custom_command("create", "create_acrcssc")
g.custom_command("update", "update_acrcssc")
g.custom_command("delete", "delete_acrcssc")
g.custom_show_command("show", "show_acrcssc")
Loading
Loading