az containerapp revision copy returns provisioningState:Succeeded even if container fails to start

[henrikwh]

Related command

az containerapp revision copy

Extension name (the extension in question)

containerapp

Description of issue (in as much detail as possible)

Setup: container app, running in single instance mode.
Isssue: CDCI deploys to that container instance. If a container fails to start, az containerapp revision copy reports {"provisioningState": "Succeeded"}back, even though container failed to provision. Since success is reported back, CICD pipeline continues. It seems like revision copy worked, hence success, the goal was not successful (a new running revision).

Running in multi revision mode, az containerapp revision activate does the activation.

Expected outcome: deploying a revision to an container app running in single instance mode, should not return provisioningState=succeeded if deployment fails.

Repro
Create an image that throws an unhandled exception:

if(mySettings.MyNumber==5)
    throw new Exception($"Bad error by Henrik: {mySettings.MyNumber}");
app.Run();

Run az

json="$( az containerapp revision copy  -n "${containerapp}" -g "${resourceGroupName}" --cpu 0.5 --memory 1.0Gi  --set-env-vars MySettings__MyNumber=5 )"
echo "${json}" | jq > ./deployment_status.json

The only workaround which seems to be available is to inspect the response from az containerapp revision copy to see if latestReadyRevisionName != latestRevisionName. If that is the case (!=), then you still need to verify by calling az containerapp revision list -n "${containerapp}" -g "${resourceGroupName}" and then again test the same properties for the lastest deployment.

  "latestReadyRevisionName": "xxx--su93uxl",
  "latestRevisionName": "xxx--7ip72po",

Full script that will test if container succeeded.

resourceGroupName=somecontainertest-we23
containerapp=somecontainertestapp-we23

json="$( az containerapp revision copy  -n "${containerapp}" -g "${resourceGroupName}" --cpu 0.5 --memory 1.0Gi  --set-env-vars MySettings__MyNumber=1 )"
echo "${json}" | jq > ./deployment_status.json

latestRevisionName="$( echo "${json}" | jq -r .properties.latestRevisionName )"
latestReadyRevisionName="$( echo "${json}" | jq -r .properties.latestReadyRevisionName )"

provisioningStateFromNewDeployment="$( echo "${json}" | jq -r .properties.provisioningState )"
echo "copy revision \"${latestRevisionName}\" status: \"${provisioningStateFromNewDeployment}\" )"


if [ $latestRevisionName == $latestReadyRevisionName ]; then
    echo $"All good! This is what i want: ${latestRevisionName},${latestReadyRevisionName}. Devops pipeline can continue."
else
    echo "Issue: latestRevision is NOT the latestReadyRevisoin. That means, what was just deployed is not ready (IT SEEMS FROM THE 'az containerapp revision copy' RESPONSE ) ${latestRevisionName} ${provisioningState}"
    echo "But dont trust 'az containerapp revision copy' response. Look at  az containerapp revision list, and see if the container just deployed is actually ready"
    revisionList="$( az containerapp revision list  -n "${containerapp}" -g "${resourceGroupName}" )"
    echo "${revisionList}" |jq > revisionlist.json

    provisioningState="$( echo "${revisionList}" |  jq  ".[]  | select(.name == \"${latestRevisionName}\" ) |  select( .properties.active==true)  | .properties.provisioningState " | sed 's/.//;s/.$//'  )" 


    echo "${latestRevisionName} ${provisioningState}"

    if [ $provisioningState != "Provisioned" ]; then
      
      echo $"Error, latestrevision (just deployed) is not the same as latestreadyrevision. Latest: ${latestRevisionName} Ready: ${latestReadyRevisionName}" 
      echo $"'az containerapp revision list' ALSO failed.. Stop pipeline"
      #      exit 1
    else
        echo $"'az containerapp revision list' shows that the deployment actually succeeded. So, pipeline can continue "

    fi
fi

---

[yonzhan]

Thank you for opening this issue, we will look into it.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @calvinsID.

Issue Details

---

Related command

az containerapp revision copy

Extension name (the extension in question)

containerapp

Description of issue (in as much detail as possible)

Setup: container app, running in single instance mode.
Isssue: CDCI deploys to that container instance. If a container fails to start, az containerapp revision copy reports {"provisioningState": "Succeeded"}back, even though container failed to provision. Since success is reported back, CICD pipeline continues. It seems like revision copy worked, hence success, the goal was not successful (a new running revision).

Running in multi revision mode, az containerapp revision activate does the activation.

Expected outcome: deploying a revision to an container app running in single instance mode, should not return provisioningState=succeeded if deployment fails.

Repro
Create an image that throws an unhandled exception:

if(mySettings.MyNumber==5)
    throw new Exception($"Bad error by Henrik: {mySettings.MyNumber}");
app.Run();

Run az

json="$( az containerapp revision copy  -n "${containerapp}" -g "${resourceGroupName}" --cpu 0.5 --memory 1.0Gi  --set-env-vars MySettings__MyNumber=5 )"
echo "${json}" | jq > ./deployment_status.json

The only workaround which seems to be available is to inspect the response from az containerapp revision copy to see if latestReadyRevisionName != latestRevisionName. If that is the case (!=), then you still need to verify by calling az containerapp revision list -n "${containerapp}" -g "${resourceGroupName}" and then again test the same properties for the lastest deployment.

  "latestReadyRevisionName": "xxx--su93uxl",
  "latestRevisionName": "xxx--7ip72po",

Full script that will test if container succeeded.

resourceGroupName=somecontainertest-we23
containerapp=somecontainertestapp-we23

json="$( az containerapp revision copy  -n "${containerapp}" -g "${resourceGroupName}" --cpu 0.5 --memory 1.0Gi  --set-env-vars MySettings__MyNumber=1 )"
echo "${json}" | jq > ./deployment_status.json

latestRevisionName="$( echo "${json}" | jq -r .properties.latestRevisionName )"
latestReadyRevisionName="$( echo "${json}" | jq -r .properties.latestReadyRevisionName )"

provisioningStateFromNewDeployment="$( echo "${json}" | jq -r .properties.provisioningState )"
echo "copy revision \"${latestRevisionName}\" status: \"${provisioningStateFromNewDeployment}\" )"


if [ $latestRevisionName == $latestReadyRevisionName ]; then
    echo $"All good! This is what i want: ${latestRevisionName},${latestReadyRevisionName}. Devops pipeline can continue."
else
    echo "Issue: latestRevision is NOT the latestReadyRevisoin. That means, what was just deployed is not ready (IT SEEMS FROM THE 'az containerapp revision copy' RESPONSE ) ${latestRevisionName} ${provisioningState}"
    echo "But dont trust 'az containerapp revision copy' response. Look at  az containerapp revision list, and see if the container just deployed is actually ready"
    revisionList="$( az containerapp revision list  -n "${containerapp}" -g "${resourceGroupName}" )"
    echo "${revisionList}" |jq > revisionlist.json

    provisioningState="$( echo "${revisionList}" |  jq  ".[]  | select(.name == \"${latestRevisionName}\" ) |  select( .properties.active==true)  | .properties.provisioningState " | sed 's/.//;s/.$//'  )" 


    echo "${latestRevisionName} ${provisioningState}"

    if [ $provisioningState != "Provisioned" ]; then
      
      echo $"Error, latestrevision (just deployed) is not the same as latestreadyrevision. Latest: ${latestRevisionName} Ready: ${latestReadyRevisionName}" 
      echo $"'az containerapp revision list' ALSO failed.. Stop pipeline"
      #      exit 1
    else
        echo $"'az containerapp revision list' shows that the deployment actually succeeded. So, pipeline can continue "

    fi
fi

---

Author:	henrikwh
Assignees:	-
Labels:	Service Attention, Auto-Assign, ContainerApp
Milestone:	-

[navba-MSFT]

Non customer-reported issue. Adding Service attention.

[AurimasNav]

I am using this with az containerapp update
modified the script a bit since it was not working for me in some scenarios:

json="$(az containerapp update --name $BITBUCKET_REPO_SLUG --resource-group $RESOURCE_GROUP_NAME --image $FULL_IMAGE_NAME)"
latestRevisionName="$(echo "${json}" | jq -r .properties.latestRevisionName)"
latestReadyRevisionName="$(echo "${json}" | jq -r .properties.latestReadyRevisionName)"
provisioningStateFromNewDeployment="$( echo "${json}" | jq -r .properties.provisioningState )"
echo "updated revision \"${latestRevisionName}\" status: \"${provisioningStateFromNewDeployment}\" )"
if [ $latestRevisionName != $latestReadyRevisionName ]; then
    echo "Issue: latestRevision is NOT the latestReadyRevision. That means, what was just deployed is potentially not ready, latest: ${latestRevisionName} ready: ${latestReadyRevisionName}"
    echo "Don't trust 'az containerapp update' response. Look at az containerapp revision list, and see if the container just deployed is actually ready"
    healthState="None"
    count=1
    while [ "$healthState" = "None" -a $count -le 10 ]
    do
      sleep 5
      revisionList="$(az containerapp revision list -n "$BITBUCKET_REPO_SLUG" -g "$RESOURCE_GROUP_NAME")"
      echo get revision list $count
      echo "${revisionList}" |jq  
      healthState="$(echo "${revisionList}" |  jq  ".[]  | select(.name == \"${latestRevisionName}\" ) |  select( .properties.active==true)  | .properties.healthState " | sed 's/.//;s/.$//')" 
      count=$[$count +1]
    done
    echo "Latest revision: ${latestRevisionName} health state:${healthState}"
    if [ $healthState != "Healthy" ]; then
      echo $"Error, deployed revision is not ready. Latest: ${latestRevisionName} Ready: ${latestReadyRevisionName}" 
      exitCode=1
    else
      exitCode=0
    fi
fi

looking forward to solution.