Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: prioritize training over who it labeled it (human vs auto by bot) #13

Open
maryamariyan opened this issue Nov 6, 2020 · 2 comments
Milestone

Comments

@maryamariyan
Copy link
Member

Enhancement suggested by @jkotas

If an issue was labelled by human we can assume that the label is right, but if assigned by labeler automatically there is a chance of error. Therefore, training the bot on its past decisions using incorrectly labeled issues which were never updated, may lead to amplification of wrong behaviors.

In general, it may be a good idea to give higher training priority for cases where the label is assigned by human.

@maryamariyan
Copy link
Member Author

Note for future:

I don't know of a github API to query name of users who last updated a label on an issue/PRs. I could either use that for this case or start keeping track of last updated area-* label event per issue/pr in the issue labeler instead.

@maryamariyan maryamariyan changed the title Enhancement: prefer manual label assignment to automatic when training Enhancement: prioritize training over who it labeled it (human vs auto by bot) Nov 6, 2020
@maryamariyan maryamariyan modified the milestones: Next, Future Apr 13, 2022
@danmoseley
Copy link
Member

I'm not sure this is feasible through the API but I'm also not sure there is much of a useful signal here. Sometimes humans make mistakes; also when the bot gets it wrong a human will fix it, almost always. So if a label was written by a bot, and not changed by human, it ought to be about as reliable as if it was changed by a human.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants