Skip to content

Add ProjectDev to the Agent section#8

Open
minhngh wants to merge 1 commit into
mlabonne:mainfrom
minhngh:main
Open

Add ProjectDev to the Agent section#8
minhngh wants to merge 1 commit into
mlabonne:mainfrom
minhngh:main

Conversation

@minhngh

@minhngh minhngh commented Aug 27, 2024

Copy link
Copy Markdown

Dear Mr. mlabonne,

I would like to request the inclusion of the ProjectDev dataset in the Agent section. This dataset is used to evaluate the performance of multi-agent systems, such as ChatDev, MetaGPT, and AgileCoder, in solving software development challenges.

Thank you very much.

@mlabonne

Copy link
Copy Markdown
Owner

Hey, is it this dataset https://huggingface.co/datasets/Fsoft-AIC/ProjectDev? If that's the case, sorry but this is too small to include in this repo. I'm mostly interested in large datasets (>10k, preferably >100k).

@minhngh

minhngh commented Aug 30, 2024

Copy link
Copy Markdown
Author

Yes, that’s correct. The dataset is pretty small because it requires manual evaluation to assess the capabilities of multi-agent systems like ChatDev and AgileCoder. Tasks within the dataset involve generating software based on user input, with user interfaces that feature complex interactions, such as chains of mouse and keyboard events, and generated programs can vary significantly in terms of code element names and implementations. Thus, it's hard to scale the dataset beyond 10,000 samples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants