[BUG] Error in submitting Automatic Evaluation in Azure AI Studio due to promptflow

[usamabinrauf]

Description of issue:

I'm facing the error "We encountered a problem during your evaluation. Please review the raw JSON file or visit the 'Logs' tab for further details." upon submitting the automatic evaluation on Azure AI Studio.

I have added shortened versions of promptflow-automatic.log, promptflow-service.log and screenshots of the error below. Let me know if you need raw JSON file as well.

promptflow-service.log:

2024-10-24 15:56:45 +0 Information Start processing flow run with run id: 7da9ce2e-0822-433c-bf81-89522ff31d5c, flow id: 7da9ce2e-0822-433c-bf81-89522ff31d5c
2024-10-24 15:56:45 +0 Information Created Run 7da9ce2e-0822-433c-bf81-89522ff31d5c under experiment e6e25bdf-1d86-4f93-b008-35b7bf7b57e3, experiment name as 7da9ce2e-0822-433c-bf81-89522ff31d5c.
2024-10-24 16:17:42 +0 Information Submit run response parsed for BulkTest run 7da9ce2e-0822-433c-bf81-89522ff31d5c. Wait for submit.
2024-10-24 16:17:54 +0 Error       Submit BulkTest run 7da9ce2e-0822-433c-bf81-89522ff31d5c failed. Mark it as failed. Exception: Microsoft.MachineLearning.Common.Core.ServiceInvocationException: Service invocation failed!
Request: POST flowmt.designer-eastus.svc/flow/api/subscriptions/781c6dac-9e70-4f55-81ac-4333384d4c07/resourceGroups/XVGroup77119/providers/Microsoft.MachineLearningServices/workspaces/XVUser47822-0711/FlowRunsAdmin/7da9ce2e-0822-433c-bf81-89522ff31d5c/bulkRuns/7da9ce2e-0822-433c-bf81-89522ff31d5c/submit
Status Code: 400 BadRequest
Error Code: UserError/BadArgument/ArgumentInvalid/AutomaticRuntimeFailed
Reason Phrase: Automatic runtime provision failed, please check promptflow-automatic.log under current run for deta
Response Body: {
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "Automatic runtime provision failed, please check promptflow-automatic.log under current run for details.",
    "messageFormat": "Automatic runtime provision failed, please check promptflow-automatic.log under current run for details.",
    "messageParameters": {},
    "referenceCode": null,
    "detailsUri": null,
    "target": null,
    "details": [],
    "innerError": {
      "code": "BadArgument",
      "innerError": {
        "code": "ArgumentInvalid",
        "innerError": {
          "code": "AutomaticRuntimeFailed",
          "innerError": null
        }
      }
    },
    "debugInfo": null,
    "additionalInfo": null
  },
  "correlation": {
    "operation": "ffc532364b994185b244aa4b05d3fefa",
    "request": "5a0f704053e709ec"
  },
  "environment": "eastus",
  "location": "eastus",
  "time": "2024-10-24T16:17:53.878645+00:00",
  "componentName": "PromptFlowService",
  "statusCode": 400
}
   at Microsoft.MachineLearning.Common.WebApi.Client.ServiceInvoker._MakeRequest(UriBuilder builder, MethodDetails details, CancellationToken cancellationToken, Nullable`1 initialStreamPosition, Uri fallbackServiceAddress, Object[] parameters) in /mnt/vss/_work/1/s/src/azureml-api/src/Common/WebApi.Client/ServiceInvoker.cs:line 906
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.MachineLearning.Common.WebApi.Client.ServiceInvoker._Invoke[T](String methodId, Object[] parameters) in /mnt/vss/_work/1/s/src/azureml-api/src/Common/WebApi.Client/ServiceInvoker.cs:line 262
   at Microsoft.MachineLearning.PromptFlow.Orchestrator.RequestManagement.<>c__DisplayClass13_0.<<SubmitBulkRunInner>b__1>d.MoveNext() in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/PromptFlow/Orchestrator/RequestManagement.cs:line 611
--- End of stack trace from previous location ---
   at Microsoft.MachineLearning.Studio.MiddleTierCommon.Helper.ExternalApiCaller.<>c__DisplayClass4_0`1.<<ExecuteWithRetryAsync>b__0>d.MoveNext() in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/MiddleTier/MiddleTierCommon/Helper/ExternalApiCaller.cs:line 104
--- End of stack trace from previous location ---
   at Microsoft.MachineLearning.Studio.MiddleTierCommon.Helper.ExternalApiCaller.<>c__DisplayClass4_0`1.<<ExecuteWithRetryAsync>b__0>d.MoveNext() in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/MiddleTier/MiddleTierCommon/Helper/ExternalApiCaller.cs:line 104
--- End of stack trace from previous location ---
   at Microsoft.MachineLearning.Studio.MiddleTierCommon.Helper.ExternalApiCaller.<>c__DisplayClass4_0`1.<<ExecuteWithRetryAsync>b__0>d.MoveNext() in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/MiddleTier/MiddleTierCommon/Helper/ExternalApiCaller.cs:line 104
--- End of stack trace from previous location ---
   at Microsoft.MachineLearning.Studio.MiddleTierCommon.Helper.ExternalApiCaller.ExecuteWithRetryAsync[T](Func`1 func, RetryPolicy retryPolicy, ExternalApiEnum externalApiEnum, Func`2 exceptionHandler, String callerName, String callerFile, Int32 callerLine, Boolean logResponseDetail, Boolean logArgumentDetail, LoggingValuesBuildMethod logValuesBuildMethod, LoggingValuesBuildMethod logArgumentsBuildMethod, Int32 maxRetryCountForTooManyRequest) in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/MiddleTier/MiddleTierCommon/Helper/ExternalApiCaller.cs:line 112
   at Microsoft.MachineLearning.PromptFlow.Orchestrator.RequestManagement.SubmitBulkRunInner(String s2sToken, SubmitBulkRunToServiceBusRequest toServiceBusRequest) in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/PromptFlow/Orchestrator/RequestManagement.cs:line 639
   at Microsoft.MachineLearning.PromptFlow.Orchestrator.RequestManagement.SubmitBulkRunAndUpdateTable(String s2sToken, SubmitBulkRunToServiceBusRequest toServiceBusRequest) in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/PromptFlow/Orchestrator/RequestManagement.cs:line 563
   at Microsoft.MachineLearning.PromptFlow.Orchestrator.RequestManagement.SubmitBulkRun(SubmitBulkRunToServiceBusRequest toServiceBusRequest) in /mnt/vss/_work/1/s/src/azureml-api/src/Designer/src/PromptFlow/Orchestrator/RequestManagement.cs:line 429
2024-10-24 16:19:54 +0 Information Mark BulkTest run 7da9ce2e-0822-433c-bf81-89522ff31d5c as Failed.

promptflow-automatice.log:

2024-10-24 16:17:42 +0 Information Automatic runtime creation is in progress, id: 0ebcbc1d-9483-53e5-893b-fc39c3fd5cba.
2024-10-24 16:17:53 +0 Error       Failed to provision Automatic runtime, id: 0ebcbc1d-9483-53e5-893b-fc39c3fd5cba, error code: ValidationError/ValidationError, message: Custom application deployment failed after retry with new compute sessionaef731c26f4e, abort session creation for Session ID: 0ebcbc1d-9483-53e5-893b-fc39c3fd5cba.

Screenshots: