[BFCL] Multi-turn evaluation problem.

[lucenzhong]

For multi_turn_base_186 at turn 3, the user's question is

There has been a problem with my booking and I previously reached out to support without any feedback yet. Kindly contact customer support on my behalf, emphasizing the smooth facilitation of my travel arrangements.

The model is supposed to call the customer_support_message function with a specific user message. I didn't find any definition of the standard message, as I understand it, the meesage parameter is a generated by the model. The evaluation process considerd the only correct message as the ground truth. I'm wondering if my gernerated message is also correct.

My result:

"contact_customer_support(booking_id='3426812',message='I am experiencing an issue with my booking and require immediate assistance to ensure the smooth facilitation of my travel arrangements. Please address this matter urgently.')"

groud_truth:

"contact_customer_support(booking_id='3426812', message='No feedback yet on my inquiry regarding my flight arrangements. Please expedite the process.')"

[lucenzhong]

For multi_turn_base_79 at turn 2.
My result:

"setCruiseControl(activate=True,distanceToNextVehicle=100.0,speed=65.0)"

Ground truth:

"setCruiseControl(speed=65, activate=True, distanceToNextVehicle=100)"`.

error message:

Model response execution results so far does not contain all the ground truth response execution results for turn 2."

Since the setCruiseControl function define speed and distanceToNextVehicle as float. I think the gound truth may be wrong.

  def setCruiseControl(
      self, speed: float, activate: bool, distanceToNextVehicle: float
  ) -> Dict[str, Union[str, float]]:
      """
      Sets the cruise control of the vehicle.
      Args:
          speed (float): The speed to set in m/h. The speed should be between 0 and 120 and a multiple of 5.
          activate (bool): True to activate the cruise control, False to deactivate.
          distanceToNextVehicle (float): The distance to the next vehicle in meters.
      Returns:
          cruiseStatus (str): The status of the cruise control. [Enum]: ["active", "inactive"]
          currentSpeed (float): The current speed of the vehicle in km/h.
          distanceToNextVehicle (float): The distance to the next vehicle in meters.
      """

[lucenzhong]

For multi_turn_base_24 at turn 0, the user's question is:

Look for draft and final report in my current directory. Compare the content difference of both.

The ground_truth is

diff(file_name1='report_draft.txt', file_name2='report_final.txt')

The report_draft.txt and report_final.txt is not specified in the question. I don't think it's reasonable to let the model to guess the full file name.

[HuanzhiMao]

Hey @lucenzhong,

Sorry for the (very!) late response—this issue slipped past my radar until I was triaging open tickets recently.

1. Regarding multi_turn_base_186 & multi_turn_base_79

   I completely agree with your reasoning about both cases. We relaxed the evaluation logic and fixed the ground-truth parameters in PR [BFCL] Contact Customer Support Multi Turn & Vehicle Control #914 #1110 , so these two tasks should now pass with your outputs.

2. Regardingmulti_turn_base_24

   Here the model is expected to explore the file system state and discover file names by calling the provided ls function, not by guessing. While the ground truth doesn't create false negatives during eval, it does read a bit strange to humans, so we have addressed it in the PR as well.

Thanks for flagging these details and for your patience. Let me know if anything is still off, and I'll jump on it right away.