GitHub CodeBuild runners - job is not picked up by runner

[lukas665]

Hi,

We are experiencing problems with GH runners hosted on AWS Codebuild:

• job is starting, we can see in the job log the message: "Waiting for a runner to pick up this job..."
• CodeBuild project is starting, it's status is "In progress"
• on CodeBuild side we have a log entry: "Listening for Jobs"
• in GH, there is CodeBuild runner in "idle" status visible in under "Settings -> Actions -> Runners"
• after few minutes the workflow fails with errors on GH side:
  • Internal server error. Correlation ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  • The job was not acquired by Runner of type self-hosted even after multiple attempts
• on AWS CodeBuild side:
  • An error occurred: Access denied. System:ServiceIdentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD needs View permissions to perform the action.
    --
    Runner listener exit with retryable error, re-launch runner in 5 seconds.
    Restarting runner...
     
    √ Connected to GitHub
     
    Failed to create a session. The runner registration has been deleted from the server, please re-configure. Runner registrations are automatically deleted for runners that have not connected to the service recently.
    Runner listener exit with terminated error, stop the service, no retry needed.
    Exiting runner...

This is happening randomly, but most of the jobs are failing. This is the second time we are having such issues. we had the similar problems about a month ago, runners were failing with similar symptoms for whole day.

We are on GHE.

Is anyone else experiencing this?

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[adonisratepay]

We are facing the same issue, what I have noticed is that the labels get applied correctly however the job stay queued and never matches with the runner.
The runner itself would stay in idle mode.

[christos-loukas]

We have a very similar issue but let me try to elaborate in case I can help, since we are facing a really weird one.

We have the setup of ghe and codebuild runners. At some point in the last few days, on the newly created repos the codebuild runners would trigger and stuck on "Listening for jobs". The old repos would still have actions that would trigger codebuild runners that would get the job and run successfully, we got the "Listening for jobs" ONLY on the newly created repos.

[rojosinalma]

I'm facing EXACTLY the same issue with Codebuild in AWS.

This was working last week, so something must have changed either in the Github Enterprise API or the token workflow for runner registration.

I'm using Github Enterprise Cloud with data residency (.ghe.com).

[lukas665]

I can confirm that this time it is happening only with newly created repositories on hge.com. On already existing ones, jobs are processing fine. The issue still persists.

[christos-loukas]

@lukas665 did you by any chance find a fix or a way around?

[lukas665]

@christos-loukas we have raised a ticket on GHE Cloud Support today, but there is no solution provided yet.

[commixon]

@lukas665 we went through the same process. But in the meantime we found a potential fix (it is working now). If you are using a Github App with codebuild then apparently there was a missing permission in this app.

By adding Read-Write to the Actions permission it seemed to do the trick.

Now, this does not fully explain the weird situation with "old repos working, new ones stuck" but we are on GHE and AWS support and still trying to figure this out.

[lukas665]

Hi @commixon

It seems that changing RO permissions to RW on Actions solved problems on our side as well, now it works for newly created applications.

GHE support also suggested checking the permissions for Actions, but they asked to check the if the 'view' permissions are granted:

• Ensure that the service identity (often linked to your OAuth app registration for GitHub Actions) has the “View” permission required for GET and DELETE operations on pipeline messages.
• Confirm that no recent changes (such as updates in OAuth registration or changes to API access scopes) have affected these permissions.

But you are right, this does not explain why old repositories were working just fine.

[christos-loukas]

Same here. Changed the permissions to Read-Write and this did the trick.

Then we uninstalled/deleted everything (aws connections, codebuild project, github app, etc). Did the whole process from the beginning and now the Github APP has Read permissions on Actions and this seems to be enough for the Codebuild runners to pick up the jobs.

Something must have changed from AWS's side or GHE's side or maybe we hit an edge case where both implementations changed some kind of defaults under the hood?