You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
when I try to set uo locally the following benchmark:
importargparsefromagentimportget_crewfromswekit.benchmark.run_evaluationimportevaluatefromswekit.config.storeimportIssueConfigdefbench(workspace_id: str, issue_config: IssueConfig) ->str:
"""Run benchmark on the agent."""# Get required toolscrew, composio_toolset=get_crew(
repo_path=issue_config.repo_name,
workspace_id=workspace_id,
)
# Set the workspace for the tools to run.composio_toolset.set_workspace_id(workspace_id)
# kick off the crew on the issue.returncrew.kickoff(
inputs={
"repo": issue_config.repo_name,
"issue": issue_config.issue_desc,
}
)
if__name__=="__main__":
parser=argparse.ArgumentParser(
description="Run benchmark on the agent.",
)
group=parser.add_mutually_exclusive_group()
group.add_argument(
"--test-split",
type=str,
default="1:2",
help="Test split ratio (e.g. 1:2, 1:500) Maximum 500 tests per project.",
)
group.add_argument(
"--test-instance-ids",
type=str,
default="",
help="Test instance ids (comma-separated)",
)
args=parser.parse_args()
ifargs.test_instance_ids:
test_instance_ids_list= [
id.strip() foridinargs.test_instance_ids.split(",")
]
test_range="1:500"else:
test_instance_ids_list= []
test_range=args.test_splitevaluate(
bench,
dry_run=False,
test_range=test_range,
test_instance_ids=test_instance_ids_list,
image_name="composio/composio:latest", # if you are doing local dev
)
I get the following error
when I try to set uo locally the following benchmark:
Agent code:
The text was updated successfully, but these errors were encountered: