.NET: execute_tool doesn't capture the failure in telemetry #2211
Description
Description
Microsoft Agent Framework allows to link available tools using AIFunctionFactory.Create(<delegate>). If this delegate throws an exception, then LLM gets tool failure:
{"type":"tool_call_response","id":"call_rmxOJ31uq6qJnV15KpSLh7pg","response":"Error: Function failed."}
But this exception is swallowed and not reported to Open Telemetry. The only thing which gets reported is error.type on execute_tool span which is far from enough to troubleshoot failures:
Expectation
The actual exception (message, call stack) to be reported as well. Either as a part of tool span or as a part of nested log entry.
Example
If I modify this
AIFunction aiFunction = AIFunctionFactory.Create(tool.ExecuteCommand);
[Description("Execute a command (cmd.exe) in Windows terminal.")]
public Task<string> ExecuteCommand(
[Description("The command to execute (e.g., 'kubectl', 'docker', 'git')")]
string command,
[Description("Arguments to pass to the command (optional)")]
string arguments = "")
{
throw new Exception("This is induced failure");
}
With a wrapper:
[Description("Execute a command (cmd.exe) in Windows terminal.")]
public Task<string> ExecuteCommand(
[Description("The command to execute (e.g., 'kubectl', 'docker', 'git')")]
string command,
[Description("Arguments to pass to the command (optional)")]
string arguments = "")
{
try
{
throw new Exception("This is induced failure");
}
catch (Exception ex)
{
Program.logger.LogError(0, ex, ex.Message); // ILogger
throw;
}
}
Then user will be able to see how this tool actually failed:
Repro
Here is the agent setup which produced above traces:
CmdSecuredTool tool = new(commandReviewAgent);
AIFunction aiFunction = AIFunctionFactory.Create(tool.ExecuteCommand);
IChatClient instrumentedChatClient = new AzureOpenAIClient(endpoint: endpoint, credential)
.GetChatClient(model)
.AsIChatClient()
.AsBuilder()
.UseFunctionInvocation()
.UseOpenTelemetry(loggerFactory: loggerFactory, sourceName: openTelemetrySourceName, otel =>
{
otel.EnableSensitiveData = true;
})
.Build();
return new ChatClientAgent(
chatClient: instrumentedChatClient,
name: agentName,
instructions: instructions,
tools: new List<AITool>(tools))
.AsBuilder()
.UseOpenTelemetry(sourceName: openTelemetrySourceName, cfg => cfg.EnableSensitiveData = true)
.Build();