Include child instance IDs in execution history

[StephenAstrumU]

Durable Function Monitor is a very useful tool for debugging durable function executions, but the functionality for durable function apps using netherite is limited. Unlike the default backend, you can't browse to child instances from the Execution History Table, and the graphical representations are limited to one level (i.e. only sub-orchestrations and activities triggered directly are displayed)

If it doesn't make sense to apply this across the board, enabling through a configuration would be great.

[scale-tone]

Just putting the link to the relevant discussion on DfMon's side: microsoft/DurableFunctionsMonitor#126 (comment)

[sebastianburckhardt]

Instance Ids of suborchestrations are already included as part of the List<History> returned by Netherite. Execution histories are not interpreted or modified in any way by Netherite, this is all just represented exactly like DurableTask.Core defines it.

I think the problem may be that the DFMon implementation is not using the DurableTask.Core representation of histories, which is the same for all storage providers, but a reverse engineering of the Azure Table storage format which is specific to the Azure Table Storage provider?

[scale-tone]

@sebastianburckhardt , DfMon does not do any 'reverse engineering' , when bound to Netherite. For Netherite DfMon falls back to IDurableOrchestrationClient.GetStatusAsync() and renders precisely what that method returns.

Have a test Netherite-based orchestration like this:

Running it locally, with Azurite as a storage and an Event Hub in Azure.

Testing .GetStatusAsync() like this:

        [FunctionName("HttpStatusTest")]
        public static async Task<IActionResult> HttpStatusTest(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestMessage req,
            [DurableClient] IDurableOrchestrationClient starter,
            ILogger log)
        {
            var status = await starter.GetStatusAsync("57f8e7d45e3540ed9e83cfc0f6636812", true, true, true);
            return new ObjectResult(status);
        }

Here is the output:

{
    "name": "DurableFunctionsOrchestrationCSharp1",
    "instanceId": "57f8e7d45e3540ed9e83cfc0f6636812",
    "createdTime": "2023-08-16T15:32:11.6209051Z",
    "lastUpdatedTime": "2023-08-16T15:32:52.4618299Z",
    "input": null,
    "output": [
        "Hello Oslo!",
        "Hello Reykjavik!"
    ],
    "runtimeStatus": 1,
    "customStatus": null,
    "history": [
        {
            "EventType": "ExecutionStarted",
            "Input": null,
            "Correlation": null,
            "ScheduledStartTime": null,
            "Generation": null,
            "Timestamp": "2023-08-16T15:32:11.6209051Z",
            "FunctionName": "DurableFunctionsOrchestrationCSharp1"
        },
        {
            "EventType": "SubOrchestrationInstanceCompleted",
            "Result": [
                "Hello Tokyo Subway!",
                "Hello London Subway!"
            ],
            "Timestamp": "2023-08-16T15:32:32.2081703Z",
            "ScheduledTime": "2023-08-16T17:32:11.810787+02:00",
            "FunctionName": "SubOrchestrationCSharp1",
            "Input": null
        },
        {
            "EventType": "TaskCompleted",
            "Result": "Hello Oslo!",
            "Timestamp": "2023-08-16T15:32:42.3814044Z",
            "ScheduledTime": "2023-08-16T17:32:32.3416229+02:00",
            "FunctionName": "SayHello",
            "Input": null
        },
        {
            "EventType": "TaskCompleted",
            "Result": "Hello Reykjavik!",
            "Timestamp": "2023-08-16T15:32:52.4449048Z",
            "ScheduledTime": "2023-08-16T17:32:42.424239+02:00",
            "FunctionName": "SayHello",
            "Input": null
        },
        {
            "EventType": "ExecutionCompleted",
            "OrchestrationStatus": "Completed",
            "Result": [
                "Hello Oslo!",
                "Hello Reykjavik!"
            ],
            "FailureDetails": null,
            "Timestamp": "2023-08-16T15:32:52.4614817Z"
        }
    ]
}

You see, no child instanceId (which is supposed to be ea459c8948d74e6fb86635a48c2517a1:0) returned.
What are we missing?

[sebastianburckhardt]

Thanks for the quick response! Indeed a mystery. Will try to figure it out.

[sebastianburckhardt]

What seems to happen is that IDurableOrchestrationClient.GetStatusAsync is not returning the original history, but a processed version that removes some events and does some event correlation.

I was not aware that this is happening (the code that does this processing is not part of Netherite, it is in the DF extension).

Here is the code that does it:
https://github.com/Azure/azure-functions-durable-extension/blob/2455636cca50cfb4d1543633d28657437bbb5173/src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs#L850

Not sure why it does that. Seems unnecessarily complicated.

[StephenAstrumU]

@sebastianburckhardt does it make sense to raise an issue with the DF extension, then, if that's the root cause of this issue?

[sebastianburckhardt]

Yes, I think the appropriate place to modify this would be the code I linked to above.