[Monitor Control Service] Upgrade to latest API version (2023-03-11) + Fix current gaps

[AfstoneBharfon]

Describe the bug

• We have a new swagger available and we would like to update the cli extension accordingly
  Link to swagger

• When using --rule-file with the current version of the CLI, certain properties of the payload are being ignored and my guess is that is probably due to the fact the Data Collection Rule model definition in the CLI is very outdated. Is there a process in place to make sure this model always reflect what we have in our swagger?

Related command

az monitor data-collection-rule create

Errors

A server side error due to missing properties in the request payload

Issue script & Debug output

cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/6f1b91a9-5242-4017-bc01-e262558ddab7/resourceGroups/bmeadam-rg/providers/Microsoft.Insights/dataCollectionRules/CLI-DCR-001?api-version=2022-06-01'
cli.azure.cli.core.sdk.policies: Request method: 'PUT'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'Content-Length': '74'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'f83e13e4-f847-11ee-9253-00155d7f94c4'
cli.azure.cli.core.sdk.policies: 'CommandName': 'monitor data-collection rule create'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--location --resource-group --name --description --rule-file --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.59.0 (RPM) azsdk-python-core/1.28.0 Python/3.9.19 (Linux-5.10.102.2-microsoft-standard-x86_64-with-glibc2.35) cloud-shell/1.0'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: {"location": "eastus", "properties": {"description": "DCR create by CLI"}}
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/6f1b91a9-5242-4017-bc01-e262558ddab7/resourceGroups/bmeadam-rg/providers/Microsoft.Insights/dataCollectionRules/CLI-DCR-001?api-version=2022-06-01 HTTP/1.1" 400 348
cli.azure.cli.core.sdk.policies: Response status: 400
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Length': '348'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-resource-requests': '149'
cli.azure.cli.core.sdk.policies: 'Request-Context': 'appId=cid-v1:2bbfbac8-e1b0-44af-b9c6-3a40669d37e3'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '079e3de3-c47e-48e3-bf60-aa4457b301cc'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'f83e13e4-f847-11ee-9253-00155d7f94c4'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTUS:20240411T211050Z:079e3de3-c47e-48e3-bf60-aa4457b301cc'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '931024a7-66d4-4717-b30a-c2076e028e8f'
cli.azure.cli.core.sdk.policies: 'api-supported-versions': '2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11, 2024-03-11'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: E9EAF5FAEEAF492786912FF7E6FAB280 Ref B: SJC211051205027 Ref C: 2024-04-11T21:10:49Z'
cli.azure.cli.core.sdk.policies: 'Date': 'Thu, 11 Apr 2024 21:10:50 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"MissingDestinations","message":"'Destinations' must not be empty.","target":"Properties.Destinations"},{"code":"MissingDataFlows","message":"'Data Flows' item count should be 1 or more. Specified list has 0 items.","target":"Properties.DataFlows"}]}}

Expected behavior

The monitor-control-service CLI extension should deserialize all the properties in the rule payload

Environment Summary

azure-cli 2.56.0 *

core 2.56.0 *
telemetry 1.1.0

Extensions:
interactive 0.4.5
monitor-control-service 1.0.1

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[AllyW]

hi @AfstoneBharfon , for the two requests you made:

1. api-version upgrade to 2023-03-11 is marked down and we'll make this update in recent sprint.

2. about the --rule-file including whole DCR model, actually currently monitor-control-service is already migrated into auto-codegen code, so you may want to refer to the code here:

   azure-cli-extensions/src/monitor-control-service/azext_amcs/custom.py

   Line 73 in 4f4ed09

   class RuleCreate(_RuleCreate):

And here is the thing about --rule-file: it is not a parameter from service API and was designed previously for complex data obj input, which only supports --data-sources, --data-flows, --destinations and --stream-declarations now. Other parameters are already exposed and supposed to be put in from cmd line. You can check all the parameters you need when running az monitor data-collection rule create -h

As this module is migrated into auto-generation code base, all parameters in future api-version releases will be automatically supported using cli codegen tool and exposed to customers to use, therefore cli does not plan to add other parameters into --rule-file manually.

If other concerns exists, please let us know.