[BUG] Using multiple compose files with their own includes will only load relatively to the first compose file

[MetalArend]

Description

Running docker compose with two compose files, both including another yaml file, it will only look for the included files relatively to the first loaded compose file.

I would expect both includes to happen based on the path of the compose.yaml file being loaded.

Steps To Reproduce

1. Create a dir1/compose.yaml file and a dir1/dir1-included.yaml file - add some services to them, and use include in the compose.yaml file to include the dir1-included.yaml file

dir1/compose.yaml

include:
  - ./dir1-included.yaml

services:
  dir1-whoami:
    image: traefik/whoami

dir1/dir1-included.yaml

services:
  dir1-included-whoami:
    image: traefik/whoami

2. Create a dir2/compose.yaml file and a dir2/dir2-included.yaml file - add some services to them, and use include in the compose.yaml file to include the dir2-included.yaml file

dir2/compose.yaml

include:
  - ./dir2-included.yaml

services:
  dir2-whoami:
    image: traefik/whoami

dir2/dir2-included.yaml

services:
  dir2-included-whoami:
    image: traefik/whoami

3. Run docker compose -f dir1/compose.yaml -f dir2/compose.yaml up and it will complain about the dir2-included.yamlfile not being found
4. Run docker compose -f dir2/compose.yaml -f dir1/compose.yaml up and it will complain about the dir1-included.yamlfile not being found
5. Use a .env file to set the COMPOSE_FILE variable to load dir1/compose.yaml and dir2/compose.yaml to see the same thing happening, depending on the order of files

Compose Version

Docker Compose version v2.27.1

Docker Environment

Client: Docker Engine - Community
 Version:    26.1.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 10
  Running: 4
  Paused: 0
  Stopped: 6
 Images: 73
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-41-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31GiB
 Name: mammi
 ID: ZECH:RGJA:RAZH:K6SZ:L4PM:WQOR:RZQP:F3KE:EMO4:OUFC:RUE6:3CT3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[MetalArend]

Same issue occurs on Docker Compose version v2.28.1

docker info:

Client: Docker Engine - Community
 Version:    27.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 10
  Running: 0
  Paused: 0
  Stopped: 10
 Images: 73
 Server Version: 27.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-41-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31GiB
 Name: mammi
 ID: ZECH:RGJA:RAZH:K6SZ:L4PM:WQOR:RZQP:F3KE:EMO4:OUFC:RUE6:3CT3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[ndeloof]

relative paths always apply based on the "project directory" with is the parent folder for the first compose file. There's unfortunately no syntax to tell compose to consider a path "relative to the current file". Same rule applies to all relative paths, not just include.

[MetalArend]

Can this please be reopened, because the issue is not as simple as you describe. Include explicitly is being referenced as loading files relatively to the included file. So the first included file in a relative directory is found correctly, the second one is not. This difference between both includes is not predictable, and not at all what is documented in the documentation.

[MetalArend]

I am not asking for relative paths here, I'm asking that if relative paths are being documented, that they work in a predictable way.

Relative paths only working when adding compose files in a certain order is not predictable at all. This issue happens both when adding command line flags for multiple files as being loaded by COMPOSE_FILE with a separator.

[MetalArend]

For anyone running into this, what worked for us is adding a compose.yaml file at the root of your project that is completely empty, and include it as the first COMPOSE_FILE. Somehow it makes all relative paths work against that "root" directory.