Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gitlab MR pipeline status is not updated #5228

Open
rhariady opened this issue Jan 10, 2025 · 1 comment · May be fixed by #5259
Open

Gitlab MR pipeline status is not updated #5228

rhariady opened this issue Jan 10, 2025 · 1 comment · May be fixed by #5259
Labels
bug Something isn't working provider/gitlab

Comments

@rhariady
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

In gitlab, when i create 2 MR using 2 difference branch but with same HEAD commit, atlantis will only update MR pipeline status for first MR. This is blocking us when we set gitlab repo to enforce pipeline success before merge.

Reproduction Steps

  • Checkout branch A (head at commit X), and push to gitlab
  • Open a new MR 1 from branch A
  • Run atlantis plan on MR 1
    • Atlantis post a comment with plan result to MR 1
    • Atlantis update MR 1 pipeline status to success
  • Run atlantis unlock from MR 1
  • Checkout another branch B (head also at commit X), and push to gitlab
  • Create another MR 2 from branch B
  • Run atlantis plan on MR 2
    • Atlantis post a comment with plan result to MR 2
    • Atlantis do not update MR 2 pipeline status (it keep showing Checking pipeline status.

image

Logs

Logs 
[DBUG] Getting GitLab merge request 84
[DBUG] GET /projects/<repo_url>/merge_requests/84 returned: 200
[INFO] Pre-workflow hooks configured, running...
[DBUG] got workspace lock
[DBUG] clone directory '/home/atlantis/.atlantis/repos/<repo_url>/84/default' already exists, checking if it's at the right commit
[DBUG] repo is at correct commit '888407c1cf7a31adb584c17b0a1bfc6c6efef6df' so will not re-clone
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[DBUG] Processing pre workflow hook 'Generate repo config', Command 'plan', Target commands []
[DBUG] Running pre workflow hook: 'Generate repo config'
[DBUG] Setting shell to default: ''
[DBUG] Setting shellArgs to default: ''
[INFO] Updating GitLab commit status for 'atlantis-nonprod/pre_workflow_hook: Generate repo config' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] Updating GitLab commit status for 'atlantis-nonprod/pre_workflow_hook: Generate repo config' to 'success'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] Pre-workflow hooks completed successfully
[DBUG] Checking if GitLab merge request 84 is approved
[DBUG] GET /projects/<repo_url>/merge_requests/84/approvals returned: 200
[DBUG] Checking if GitLab merge request 84 is mergeable
[DBUG] GET /projects/<repo_url>/merge_requests/84 returned: 200
[DBUG] GET /projects/42481065 returned: 200
[DBUG] GET /projects/42481065/commits/888407c1cf7a31adb584c17b0a1bfc6c6efef6df/statuses returned: 200
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[WARN] unable to update commit status: POST https://gitlab.com/api/v4/projects/<repo_url>/statuses/888407c1cf7a31adb584c17b0a1bfc6c6efef6df: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via "run")}
[DBUG] Building plan command for all affected projects
[DBUG] Getting modified files for GitLab merge request 84
[DBUG] GET projects/<repo_url>/merge_requests/84/changes returned: 200
[DBUG] 1 files were modified in this pull request. Modified files: [nonprod.terraform.tfvars.yaml]
[DBUG] got workspace lock
[DBUG] clone directory '/home/atlantis/.atlantis/repos/<repo_url>/84/default' already exists, checking if it's at the right commit
[DBUG] repo is at correct commit '888407c1cf7a31adb584c17b0a1bfc6c6efef6df' so will not re-clone
[INFO] successfully parsed atlantis.yaml file
[DBUG] moduleInfo for '/home/atlantis/.atlantis/repos/<repo_url>/84/default' (matching '') = map[]
[DBUG] found downstream projects for "nonprod.terraform.tfvars.yaml": []
[DBUG] checking if project at dir "." workspace "nonprod" was modified
[DBUG] file "nonprod.terraform.tfvars.yaml" matched pattern
[INFO] 1 projects are to be planned based on their when_modified config
[DBUG] determining config for project at dir: '.' workspace: 'nonprod'
[DBUG] MergeProjectCfg started
[DBUG] setting import_requirements: [approved,mergeable,undiverged] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting repo_locks: this is a bug from default server config
[DBUG] setting plan_requirements: [undiverged] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting workflow: "default" from default server config
[DBUG] setting allowed_overrides: [workflow] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting allow_custom_workflows: false from default server config
[DBUG] setting delete_source_branch_on_merge: true from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] setting policy_check: false from default server config
[DBUG] setting custom_policy_check: false from default server config
[DBUG] setting apply_requirements: [approved,mergeable,undiverged] from repos[1], id: /gitlab\.com\/<repo_group>\/.*/
[DBUG] overriding server-defined workflow with repo-specified workflow: "bpv1"
[DBUG] MergeProjectCfg completed
[DBUG] final settings: plan_requirements: [undiverged], apply_requirements: [approved,mergeable,undiverged], import_requirements: [approved,mergeable,undiverged], workflow: bpv1, delete_source_branch_on_merge: true, repo_locking: on_plan, policy_check: false, custom_policy_check: false, silence_pr_comments: []
[DBUG] Building project command context for plan
[INFO] cannot determine which version to use from terraform configuration, detected 0 possibilities.
[DBUG] deleting previous plans and locks
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan: nonprod' to 'running'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] Acquired lock with id '<repo_url>/./nonprod'
[DBUG] acquired lock for project
[DBUG] clone directory '/home/atlantis/.atlantis/repos/<repo_url>/84/nonprod' already exists, checking if it's at the right commit
[DBUG] repo is at correct commit '888407c1cf7a31adb584c17b0a1bfc6c6efef6df' so will not re-clone
[INFO] Updating GitLab commit status for 'atlantis-nonprod/plan: nonprod' to 'success'
[DBUG] GET /projects/gitlab.com/<repo_url>/repository/commits/%!d(string=888407c1cf7a31adb584c17b0a1bfc6c6efef6df): 200
[INFO] Pipeline found for commit 888407c1cf7a31adb584c17b0a1bfc6c6efef6df, setting pipeline ID to 1618879816
[INFO] plan success. output available at: https://gitlab.com/<repo_url>/-/merge_requests/84
[DBUG] hiding previous plan comments for command: 'Plan', directory: ''
[DBUG] Hiding previous command comments on GitLab merge request 84
[DBUG] /projects/<repo_url>/merge_requests/84/notes
[DBUG] GET /projects/<repo_url>/merge_requests/84/notes returned: 200
[DBUG] Updating merge request note: Repo: '<repo_url>', MR: '84', comment ID: '2290094251'
[DBUG] PUT /projects/<repo_url>/merge_requests/84/notes/2290094251 returned: 200

PS: I have retract some information from the log

Environment details

  • Atlantis version: v0.32.0 (commit: ea838ac) (build date: 2024-12-20T02:51:00.531Z)
  • Deployment method: docker on google cloud run
  • If not running the latest Atlantis version have you tried to reproduce this issue on the latest version:
  • Atlantis flags:

Atlantis server-side config file:

ATLANTIS_WEB_BASIC_AUTH=true
ATLANTIS_AUTOMERGE=true
ATLANTIS_EMOJI_REACTION=trident
ATLANTIS_REPO_CONFIG=/app/atlantis/cfg/atlantis-cfg-repos/atlantis-cfg-repos.yaml
ATLANTIS_HIDE_PREV_PLAN_COMMENTS=true
ATLANTIS_SILENCE_NO_PROJECTS=true
ATLANTIS_FAIL_ON_PRE_WORKFLOW_HOOK_ERROR=true
ATLANTIS_VCS_STATUS_NAME=atlantis-nonprod
ATLANTIS_CONFIG=/app/atlantis/cfg/atlantis-cfg-config/atlantis-cfg-config.yaml
ATLANTIS_WRITE_GIT_CREDS=true
ATLANTIS_CHECKOUT_STRATEGY=merge
ATLANTIS_ENABLE_DIFF_MARKDOWN_FORMAT=true
ATLANTIS_SILENCE_VCS_STATUS_NO_PLANS=true

Additional Context

I try to investigate from the log, and it seems this bug is happen because atlantis found out that there was another pipeline for the commit X already (despite it's from another branch), and will try to use it instead of creating another one for different MR.

atlantis/server/events/vcs/gitlab_client.go

Lines 424 to 446 in de1d8dc

for i := 0; i <= retries; i++ {
commit, resp, err = g.Client.Commits.GetCommit(repo.FullName, pull.HeadCommit, nil)
if resp != nil {
logger.Debug("GET /projects/%s/repository/commits/%d: %d", pull.BaseRepo.ID(), pull.HeadCommit, resp.StatusCode)
}
if err != nil {
return err
}
if commit.LastPipeline != nil {
logger.Info("Pipeline found for commit %s, setting pipeline ID to %d", pull.HeadCommit, commit.LastPipeline.ID)
// Set the pipeline ID to the last pipeline that ran for the commit
setCommitStatusOptions.PipelineID = gitlab.Ptr(commit.LastPipeline.ID)
break
}
if i != retries {
logger.Info("No pipeline found for commit %s, retrying in %s", pull.HeadCommit, delay)
time.Sleep(delay)
} else {
// If we've exhausted all retries, set the Ref to the branch name
logger.Info("No pipeline found for commit %s, setting Ref to %s", pull.HeadCommit, pull.HeadBranch)
setCommitStatusOptions.Ref = gitlab.Ptr(pull.HeadBranch)
}
}

I think we can fix this by checking if the last pipeline was indeed from the same ref first. If the last pipeline was from different branch (ref), then try to create one instead.

example Gitlab API GET /projects/%s/repository/commits/%d response

{
  "id": "6104942438c14ec7bd21c6cd5bd995272b3faff6",
  "short_id": "6104942438c",
  "title": "Sanitize for network graph",
  "author_name": "randx",
  "author_email": "[email protected]",
  "committer_name": "Dmitriy",
  "committer_email": "[email protected]",
  "created_at": "2021-09-20T09:06:12.300+03:00",
  "message": "Sanitize for network graph",
  "committed_date": "2021-09-20T09:06:12.300+03:00",
  "authored_date": "2021-09-20T09:06:12.420+03:00",
  "parent_ids": [
    "ae1d9fb46aa2b07ee9836d49862ec4e2c46fbbba"
  ],
  "last_pipeline" : {
    "id": 8,
    "ref": "main",
    "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0",
    "status": "created"
  },
  "stats": {
    "additions": 15,
    "deletions": 10,
    "total": 25
  },
  "status": "running",
  "web_url": "https://gitlab.example.com/janedoe/gitlab-foss/-/commit/6104942438c14ec7bd21c6cd5bd995272b3faff6"
}
@rhariady rhariady added the bug Something isn't working label Jan 10, 2025
@dosubot DosuBot
Copy link

dosubot bot commented Jan 10, 2025

@orhan89 Hello! I'm here to help you with any issues, answer questions, and even guide you on how to become a contributor. Let me know how I can assist you with your current concern.

I found a similar issue that was previously reported and closed: Atlantis - Gitlab integration failing when updating MR with new commits on version v0.29.0. This issue was resolved by a pull request that changed the GitLab UpdateStatus function to get the pipeline from the commit rather than the merge request [1].

In your case, the problem seems to be that Atlantis is reusing an existing pipeline for a commit shared across multiple branches, which leads to the pipeline status not being updated for the second merge request. The proposed solution is to modify the logic to ensure that the last pipeline belongs to the same branch before reusing it. If the last pipeline is from a different branch, Atlantis should create a new pipeline for the second merge request. This involves checking the last_pipeline field in the GitLab API response to verify the branch (ref) before proceeding with the status update.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@rhariady rhariady linked a pull request Jan 21, 2025 that will close this issue
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working provider/gitlab
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: gitlab - ensure lastpipeline ref is the same as head branch rhariady/atlantis
1 participant
@rhariady