Add support for doing code coverage reporting entirely on GitHub

[Cadair]

It seems not that complex to upload a coverage report from each job and then combine them in an extra job: https://hynek.me/articles/ditch-codecov-python/ I am finding codecov to be more and more annoying so it might be worth having a second option?

[mhvk]

Yes, please!!!

[Cadair]

It's unlikely I am getting to this anytime soon if anyone else wants to take a run at it.

[pllim]

Looks like pydata-sphinx-theme did the hard work here (uses https://github.com/py-cov-action/python-coverage-comment-action):

• Try to implement coverage action instead of using codecov. pydata/pydata-sphinx-theme#1756
• purge codecov pydata/pydata-sphinx-theme#1809

Though, @weaverba137 also said Coveralls still work (at least for DESI), if that is preferable.

I don't quite remember now why we switched from Coveralls to Codecov in the first place. (Something was broken?)

[pllim]

That said, given how much my attempt at #199 broke other repos running CI in different ways, maybe it is better for astropy to also ditch the reusable workflow here for CI and only keep it for wheels. That way, we only have to worry about ditching codecov in a way that works for us.

[astrofrog]

Why don't we just try and figure it out in the reusable workflows then we would get it to work in all coordinated packages etc too for free?

[pllim]

It would be a breaking change. I see very specific codecov settings.

[Cadair]

There's a generic coverage key: https://github-actions-workflows.openastronomy.org/en/latest/tox.html#coverage so we can add a coverage: github which people can opt-in to. I don't think this has to be a breaking change.

[Cadair]

From the coverage.py docs I just found this: https://github.com/Bachmann1234/diff_cover we could use that to fail the job based on the patch coverage rather than the whole project.

[zacharyburnett]

here's a usage of diff-cover (assuming the coverage files are uploaded as artifacts earlier in the run)
https://github.com/spacetelescope/stpipe/pull/160/files#r1653419802

  report_diff_coverage:
     needs: [ test ]
     if: github.event_name == 'pull_request'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/setup-python@v5
       - run: pip install diff-cover coverage
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - run: pip install .
       - uses: actions/download-artifact@v4
         with:
           pattern: ".coverage*"
           merge-multiple: true
       - run: coverage xml
       - run: diff-cover coverage.xml --compare-branch origin/${{ github.base_ref }} --markdown-report $GITHUB_STEP_SUMMARY

[zacharyburnett]

made it even better by having it comment the report on the PR and update that comment with every commit. However, it requires using pull_request_target, so the workflow has to already be on main

  report_diff_coverage:
     needs: [ test ]
     if: github.event_name == 'pull_request_target'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/setup-python@v5
         with:
           python-version: 3
       - run: pip install diff-cover coverage
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - run: pip install .
       - uses: actions/download-artifact@v4
         with:
           pattern: ".coverage*"
           merge-multiple: true
       - run: coverage xml
       - run: diff-cover coverage.xml --compare-branch origin/${{ github.base_ref }} --markdown-report diff_coverage.md
       - run: cat diff_coverage.md >> $GITHUB_STEP_SUMMARY
       - uses: thollander/[email protected]
         with:
           filePath: diff_coverage.md
           pr_number: ${{ github.event.number }}
           comment_tag: diff-coverage

[pllim]

Re: pull_request_target -- I can always test on my astropy fork instead of astropy proper in that case.

tl;dr -- Does it report patch coverage, or just project?

[pllim]

Actually, I have concern... Please see https://github.com/astropy/astropy/pull/16619/files#r1655466104