Custom Docker image not rebuild correctly after merging

[JuJup]

Bug description

We have a very strange behavior with our Gitpod Setup using a custom Dockerfile.
We have a main branch A and a feature branch B. In B we're testing some new things so the Dockerfile differs slightly but is mainly equal to A. From time to time, we merge the main branch A into the feature branch B to keep it up to date.
In this case, the Docker Image is rebuild for B, but with the Dockerfile from A.
Sounds confusing? Yes it is :lau

I described the steps using a minimal example to show the problem:

Steps to reproduce

1. Create a project with these two files (main):

# .gitpod.Dockerfile
FROM gitpod/workspace-base:latest

RUN touch main-branch.info

# .gitpod.yml

image:
  file: .gitpod.Dockerfile

2. Start a Gitpod workspace for main
   -> ls ~ should show only main-branch.info

3. Create a Feature Branch (test-branch) and change the Dockerfile to:

# .gitpod.Dockerfile
FROM gitpod/workspace-base:latest

RUN touch main-branch.info

RUN touch test-branch.info

4. Start a Gitpod workspace for test-branch
   -> ls ~ should show main-branch.info and test-branch.info
   -> So far everything is fine

5. Make a change to Dockerfile in main:

# .gitpod.Dockerfile
FROM gitpod/workspace-base:latest

RUN touch main-branch-refactored.info

6. Start a Gitpod workspace for main
   -> ls ~ should show only main-branch-refactored.info

7. Merge main into test-branch:

8. Start a Gitpod workspace for test-branch
   -> ls ~ shows main-branch-refactored.info but not test-branch.info ⚠️

That's already confusing, as the code now doesn't represent the state anymore. So I tried rebuilding the image explicitely:

9. Open https://gitpod.io/#imagebuild/https://gitlab.com/<group>/<project>/-/tree/test-branch
   -> Image is rebuild (you can see the logs, but you can see it's the wrong Dockerfile from main)
   -> ls ~ shows main-branch-refactored.info but not test-branch.info ⚠️

The only thing that helps now is changing some line in the Dockerfile in test-branch. But this can't be the solution honestly.

Workspace affected

No response

Expected behavior

• When the Dockerfile changes (no matter if directly in the branch or through a merge from main branch), the Image should be rebuild correctly from the branch's Dockerfile
• When I use https://gitpod.io/#imagebuild/, I expect the rebuild to be base on the branch's Dockerfile

Example repository

No response

Anything else?

No response

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JuJup]

Did anybody had a chance to take a look at that?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shintonik]

@JuJup Thanks for bringing up the issue. We are dealing with the same problem.

Every time we merge the main branch into a feature branch it often breaks our Gitpod workspaces which then needs a manual fix. The workaround is basically making a commit with a pseudo change to gitpod.Dockerfile to break the cache and trigger a prebuild. But this is really tedious. We would really appreciate if someone could look into this bug.

Thank in advance.

[axonasif]

Hi @shintonik, are you using prebuilds?

[shintonik]

Yes, we use prebuilds.

[JuJup]

Hey @axonasif,
can you reopen the issue?

[axonasif]

Hi @JuJup

Done

[axonasif]

@shintonik this could be another potential workaround:

• https://www.gitpod.io/docs/configure/workspaces/workspace-image#manually-rebuild-a-workspace-image

[JuJup]

@axonasif, see step 9. in my Steps to reproduce - at least when I created the ticket that was what surprised me most.

I have an assumption what might be the problem..
When I check the docs it says:

When a workspace is started, Gitpod will search the last 100 ancestor commits for a successful prebuild to use as a base for that workspace.

In relation to my example above:
When main is merged into test-branch (Step 7), and we open a new workspace, Gitpod checks for ancestor commits and finds a prebuild image in the main branch, which is a true ancestor. However, that the Dockerfile in test-branch might differ from main, is not considered at that point.
Not completely sure about this theory, but might be a starting point for further bug analysis?

[axonasif]

@JuJup, sorry, I missed it.

[JuJup]

No worries :)

[filiptronicek]

Hey there @JuJup 👋, I've tried reproducing according to your steps in https://github.com/filiptronicek/gitpod-test-image-builds and was not able to: when starting a new workspace on my feature branch, I see both feature-branch.info and main-branch-updated.info under /home/gitpod/. To that end, I should note that I handled the merge conflict so that it resulted in the following Dockerfile:

FROM gitpod/workspace-base:latest

RUN touch main-branch-updated.info
RUN touch feature-branch.info

Any ideas on what I'm doing wrong in my approach?

[JuJup]

Hey @filiptronicek,
thanks for checking on the issue. I also tried to reproduce things with the given steps, as it's some time ago I wrote it :D
The result was quite surprising. First, the result were equivalent to what I wrote in the reproduction steps when I tried it again in Gitlab, as I did before. So i tried the same on Github as you did - there I have the same behaviour as you (the "desired" behaviour).

Therefore, I assume this might be a platform depending issue?

[filiptronicek]

@JuJup Therefore, I assume this might be a platform depending issue?

It totally could be. I will try reproducing with GitLab when I find some time. For now, it's helpful to know that it works fine for GitHub.