one >53 errrors to report, reviewdog aborts GitHubActionLogWriter: "reported too many annotation"

[bernhardkaindl]

TODO / Remaining bug:

When more than 53 errors are reported by mypy and are not filtered, reviewdog gives up and aborts with the message:

GitHubActionLogWriter: "reported too many annotation"

The error is buried in the middle of the long output:

  /home/runner/work/xen-api/xen-api/scripts/plugins/extauth-hook-AD.py:70:19: error: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes  [str-bytes-safe]
  Warning: [mypy] reported by reviewdog 🐶
  In function "fresh_name":
  
  Raw Output:
  /home/runner/work/xen-api/xen-api/scripts/static-vdis: note: In function "fresh_name":
  reviewdog: GitHubActionLogWriter: reported too many annotation (N=53)
  + reviewdog_exit_val=1
  + echo ::endgroup::
  + [[ false == \t\r\u\e ]]
  Error: [mypy] reported by reviewdog 🐶
  Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
  
  Raw Output:

I know the current GitHub limits, they are also printed in the log:

   Error: reviewdog: Too many results (annotations) in diff.
  You may miss some annotations due to GitHub limitation for annotation created by logging command.
  Please check GitHub Actions log console to see all results.
  
  Limitation:
  - 10 warning annotations and 10 error annotations per step
  - 50 annotations per job (sum of annotations from all the steps)
  - 50 annotations per run (separate from the job annotations, these annotations aren't created by users)

However, reviewdog should do it's best to report 10 error annotations and 10 warning annotations (and it should also support 10 notice annotations, btw) and if > 10 annotations, the 10th annotation of each kind should be a message that there are more than 10 annotations, which GitHub does not support.

The same may also apply to review comments, but I am not aware of such limit. However, to prevent a DoS from reviewdog, it is likely best if reviewdog replaces the last comment it would be able to add with a "Giving up due to GitHub limit".

In case of too many errors, it may also be nice to merge several erros of the same kind into one error and add "more of the same are omitted due to GitHub annotation limits of 10 annotations per type and step".

Another nice feature would be to mark less important errors not as errors but as warings and notices, so in total 30 annotations could be issued per step.

The biggest problem is that the action-mypy does not fail CI in this case, so the error goes unnoticed:

https://github.com/xenserver-next/xen-api/actions/runs/7706543512/job/21002295649

[tsuyoshicho]

Thank you, I'll try to remember it.

[tsuyoshicho]

JSON support merged