Add AI tooling disclosure text to contributor guide and fields to PR templates

[pepijnve]

Is your feature request related to a problem or challenge?

Inspired by ghostty-org/ghostty#8289 I think it might be a good idea to add some text to the contributor guide requesting usage of AI tooling to be disclosed. Additionally, adding fields to PR templates encourages people to fill in this information proactively.

Use of AI tools should not be discouraged, but as a courtesy to the human being reviewing changes it would be nice if any output from an AI system is marked as such. The referenced example explains this much better than I can.

Describe the solution you'd like

See https://github.com/ghostty-org/ghostty/blob/main/CONTRIBUTING.md#ai-assistance-notice for a possible source of inspiration.

Describe alternatives you've considered

Do nothing.

I'm not sure to what extent AI slop PRs are already a problem in this project. Maybe adding the PR template fields is too much of a nuisance for regular contributors.

Additional context

No response

[2010YOUY01]

Maybe the PR template should include something like: “If you're using an AI assistant, do you fully understand the generated code and its context?” — instead of just asking whether AI was used.

To be honest, I’m more willing to review PRs when the author explicitly states that they fully understand the generated code. Sometimes it’s hard to review a large AI-generated PR if the author doesn’t fully grasp the rationale behind it — whether the code is fully generated seems less important to me.

[Jefffrey]

If PR responses are being generated by an AI, disclose that as well.

I also like & support this line from ghostty

[codetyri0n]

I think this is a good example too - apache/opendal#6666

[alamb]

To be honest, I’m more willing to review PRs when the author explicitly states that they fully understand the generated code. Sometimes it’s hard to review a large AI-generated PR if the author doesn’t fully grasp the rationale behind it — whether the code is fully generated seems less important to me.

think this is a really great point

It is really tough (takes lots of mental focus time) to review large PRs, and I think one challenge is that AI tools make it much easer to create such large PRs.

Thus, a related suggestion would be to emphasize breaking PRs down into small pieces that are easier to understand (both for submitter and reviewer).

[petern48]

My two cents as a contributor: Let's remember that there's no guarantee that people will tell the truth on the internet. I don't think this will help much with the most problematic PRs where all the code is fully AI-generated and the PR feedback is handled by AI, too. I think the same people who author these PRs will also lie about their use GenAI use.

Maybe the PR template should include something like: “If you're using an AI assistant, do you fully understand the generated code and its context?”

I like the intention of this suggestion, but I don't think anyone would ever say "No" to this. It might just be extra noise for regular contributors.

I can definitely recall one time when I used agent mode to fix something for me (in a different repo). The hard part was finding where to do the fix, but the fix was trivial (2 lines), and I fully understood it. In that case, I decided to lie (oh no, how could I 😱) and say "No" to the question asking if GenAI is used because I just wanted my PR in, and I knew I was ready for feedback if there was any. (For the record, if I ever did make any large GenAI changes that I didn't understand, I probably wouldn't submit the PR, please still trust me as a contributor 🙏 haha)

I do think this change could be helpful to encourage more comments like this #17988 (comment), but again, I think it will be from the people who won't be a problem. I guess how relevant my argument is largely dependent on whether DF gets more PRs like that or more PRs that are AI spam, like the example PR below.

I think what might be more useful is to decide on how reviewers should handle PRs that are suspected to be AI-generated. e.g. Do you call it out explicitly or not? Do you ghost them, or do you close the PR? I've seen PRs where the reviewer calls the author out, asking if it's AI-generated, and then there's no more activity on the PR, but then the issue never gets fulfilled because there's a linked PR, and it's 80% there. In fact, here's an example in DF.

[pepijnve]

My two cents as a contributor: Let's remember that there's no guarantee that people will tell the truth on the internet. I don't think this will help much with the most problematic PRs where all the code is fully AI-generated and the PR feedback is handled by AI, too.

It’s not a magic solution, fully agree with that. But being able to tap the contributor guide sign might still be useful for when people are called out. It changes the feedback from “this is the reviewer’s individual opinion” to “this is how we would like to do things in the project”.

[alamb]

and it's 80% there. In fact, here's an #17314 (comment) in DF.

FWIW my interpretation of the conversation on that PR is that it was reviewed, but then feedback was not actually addressed even though there were responses (aka interactions like this #17314 (comment) was wasting @milenkovicm 's time)

That is my only real concern about using AI tools -- that they can potentially waste a bunch of reviewer time (one of the most constrained resources we have).

[2010YOUY01]

My two cents as a contributor: Let's remember that there's no guarantee that people will tell the truth on the internet. I don't think this will help much with the most problematic PRs where all the code is fully AI-generated and the PR feedback is handled by AI, too. I think the same people who author these PRs will also lie about their use GenAI use.

I think people are not telling the truth because of misunderstandings rather than malice. We should better explain why large AI-generated PRs without understanding are not helpful.

Here are my opinions. If we can reach agreements, they can be added to the contributor guide:

If the PR author only understands a small part of the implementation details, such PR should be rejected
The PR and code review system exists for two major reasons:

• Get the job done, obviously.
• Share knowledge about the codebase between author and reviewers, which is a long-term investment in the project. For this reason, even if someone familiar with the codebase can finish a task quickly, it’s still worth helping a new contributor work on it even if it takes longer.

This doesn’t work for fully AI-generated PRs without understanding, because reviewers are mostly prompting through a pass-through human proxy. They can solve the problem faster by using AI directly, and there is almost no knowledge sharing since the PR author doesn’t want to understand the solution.

A better way to help is to write a better issue to make it easier for others to contribute, instead of generating a full solution.

If the author knows most of the solution but is missing some corners, such PRs are welcome
One case is implementing a large trait, where the author knows the high-level idea but isn’t sure if a certain interface’s default implementation is okay to use. This might be due to under-documentation of the API.
They're encouraged to point out exactly which part of the PR they don’t fully understand, the review can focus on that and potentially improve the documentation.

[alamb]

Here are my opinions. If we can reach agreements, they can be added to the contributor guide:

@2010YOUY01 -- I think this is the best description I have ever read about the nuances and tradeoffs of AI generated PRs. It captures the tradeoffs really well I think it would be an amazing addition to our contribution guides.

Do you have time to create a PR to add it? If not I will find time to do so

[2010YOUY01]

Here are my opinions. If we can reach agreements, they can be added to the contributor guide:

@2010YOUY01 -- I think this is the best description I have ever read about the nuances and tradeoffs of AI generated PRs. It captures the tradeoffs really well I think it would be an amazing addition to our contribution guides.

Do you have time to create a PR to add it? If not I will find time to do so

I have filed #18237, looking forward to feedbacks.