Add Compose

[Mcrich23]

There has been a lot of discussion around a docker-compose counterpart for Container. This moves that discussion a bit further by introducing a basic translation layer for compose.

This pr introduces a new command group into the cli: Compose. It has two subcommands: up and down.

The origin behind this pr being made comes from the desire to run Vapor on Apple Containers. Hopefully, efforts like this make it possible quite soon.

Note: This is an early implementation of compose and there are features missing in Container that makes it not up to complete feature parity. This is bound to change as features such as port customization are introduced.

[Mcrich23]

Referencing the discussions/issues about how best to support compose: #55 #194 #101

[stevenanthonyrevo]

I've reviewed as much as I can!

I'm fairly limited due to my current MacOS version as anything below 6.2.0 will just error out.

Although, thanks for your work on this feature request as I was really hopeful for such a feature!

So on Good Faith, I'm going to Approve! If anyone else wants to review, feel free!

[Mcrich23]

Thank you! I am also planning to add support for the newfound networking capabilities in the coming days.

Also, if anyone knows how to restructure compose to call these commands without invoking the command line, I would be very grateful to hear about it!

[Mcrich23]

To address these things before the merge, I am going to make this pull request into a draft. If you want to try the tool compiled, you can get it at https://github.com/mcrich23/Container-Compose.

[jglogan]

@jglogan Why not support the other stuff i've already built that work consistently like networking and drive linking?

It's simply what I explained above. Let's say I'm submit a contribution to a big open source project, say Linux, or an Apache Foundation or CNCF project, and you're a maintainer making decisions about all the contributions that come in, including mine. You need to make sure that anything that is contributed is maintainable and reliable over time, and not only by the original contributor who may or may not be engaged with the project in the future.

The way to do that is to avoid large PRs like this one, and instead break epic features into small deliverable units just as one would in most any agile software development process. It's far easier to ensure that cross-cutting concerns (logging, error handling, security, and so on) are addressed properly with fewer files, and once those are addressed it establishes a positive pattern for the future commits. It makes everyone's job easier, especially in the case that others see your promising work and want to contribute as well.

From there we add container, network, and storage features one item at a time, with README documentation so it's crystal clear what's currently supported and what isn't.

I can't be sure that anything "works consistently" yet because there are 35 files in the PR and zero tests. Again, with a smaller commit, it's simpler to include unit tests for what can be unit tested (granted, much of what container does is about side effects, but there may be some code here that could benefit from unit coverage).

I hope this explains why, but please follow up if I'm not communicating the need clearly!

[jglogan]

I couldn't figure out how to run code on shutdown which is why compose up actually launches everything and then attaches to containers logs. To shutdown the containers, you have to stop the compose up command and then run compose down.

With a small initial PR, we can focusing on solving that rather large usability problem so that while it's a very limited feature set to start with, it's one that works very well.

[Mcrich23]

Thanks for that explanation @jglogan. I am pretty busy for the next month or two, but I can send a sort of "State of Compose" surrounding how key components work and what is supported if you or others want to start pulling my work apart into smaller PRs. Just let me know 😄.

I tried to keep everything documented, which should help this effort.

Also, I totally agree that tests are needed, I just have not had times to write them. Hopefully that will happen as this is broken down into smaller PRs.

[jglogan]

@Mcrich23 All good! You've got a great proof of concept in place. There's definitely interest in this feature so I hope we'll all be able to find a little time in coming weeks to take what's here and build out a really cool compose plugin. Better to have it move slowly and thoughtfully than create something too quickly that's difficult for users and a maintenance burden.

[Mcrich23]

I have noticed that some repos also do decision markdown files to talk about stuff. Is that something we want to do for compose?

Also, would it be better to do this as a git submodule and maintain a separate issue and pr list for compose?

[Mcrich23]

The State of Compose

This is just a brief overview of current support and how certain things work!

cc: @jglogan @stevenanthonyrevo @niteshbalusu11

Note: For all of these, shortcut definitions may not parse correctly.

Full/Semi-Supported Features

Networking

Feature	Supported	Notes
Attaching Containers to Networks	✅
All Other Features/Configs	❌

Environment

Feature	Supported	Notes
Reading from .env file properties passed through compose file	✅
Shared environment values	✅
Resolving Named Services to IP Addresses	✅	Cross-container communication is easily supported by passing in the ip address through the environment. Just name a container and compose will automatically resolve the IP address of the created container. Make sure to list it as a dependency so the container is already running.

Depended By

Feature	Supported	Notes
Basic Service Listing	✅
All Other Features/Configs for listed services	❌

Volume Linking

Feature	Supported	Notes
Anonymous Linking	✅	When linking a volume not backed by a location on the host, compose fills that in by linking it to ~/.containers/Volumes/projectName/actualVolumeName.
Named Volumes	✅

Images

Feature	Supported	Notes
Pulling Image	✅	Does easily support private images
Auto Detecting Local Source to Build	❌
Normal Dedicated Build Parameters	✅	Use like build: context: . dockerfile: ./Dockerfile
Shorthand and Abnormal Build Parameter setups	❌

Documented Unsupported Features

• Docker Compose Spec Version Switching
• Deploy Config

[mazdak]

To address these things before the merge, I am going to make this pull request into a draft. If you want to try the tool compiled, you can get it at https://github.com/mcrich23/Container-Compose.

@Mcrich23 I have a similar branch with a complete implementation of "compose" with tests and everything. I will try to make this into a standalone plugin as suggested by @jglogan

https://github.com/mazdak/container/tree/mazdak/container-compose

p.s. how are you working around the permission issues with Postgres (not being allowed to chown/chmod)?

[Mcrich23]

@mazdak Sounds good! What do you mean about the issues with Postgres and chown?

[mazdak]

@mazdak Sounds good! What do you mean about the issues with Postgres and chown?

I am unable to get the standard Postgres or redis images to boot up because their entrypoint script tries to modify the mounted volume permissions. This is a limitation of VirtioFS if I understand correctly. You won't notice this issue if you are not using a persistent mounted volume.

[Mcrich23]

It could be cause i'm expressly setting the container to be owned by the user? But haven't encountered that error in my testing. Maybe take a look at my implementation

[jglogan]

p.s. how are you working around the permission issues with Postgres (not being allowed to chown/chmod)?

#333 (comment)

[jglogan]

Finally, as of right now, inter-container communication is supported via passing in ips through the environment. If you name a container in the environment, it can replace the value with the ip. This was done so people didn't have to mess with dns if they don't want to, but I can see how it could become challenging.

Adding the feature for requesting a specific IP (#282) isn't terribly hard, it's just that DNS names work today (but requires configuration as you note) so it's not our top priority.

[Mcrich23]

p.s. how are you working around the permission issues with Postgres (not being allowed to chown/chmod)?

#333 (comment)

Yeah. I guess the way that Vapor does it by default avoids this issue so I never encountered that specifically.

[jglogan]

Regarding Compose types, have you looked into generating the Swift code from the spec itself using quicktype. If that worked then we could automate that (running quicktype under container since it could be a manual step that we perform only when updates are required).

This is a rough sketch of how one could create a standalone swift package that provides a reader using Compose types generated from the JSON schema specification (the one under compose-spec/compose-go seems to be the canonical copy).

swift-docker-compose-types.tgz

[Mcrich23]

@jglogan I totally agree that the ideal way to do this is by autogenerated codable structures via quicktype. My wonder is if totally valid yaml that slightly differs from the mainstream definition will throw errors. One example is:

depends_on: foo

versus

depends_on: 
  - foo

[jglogan]

totally valid yaml

This particular case doesn't seem totally valid.

% docker compose -f docker-compose-syntax.yaml up -d
validating docker-compose-syntax.yaml: services.server.depends_on must be a list
% echo $?
1

docker-compose-syntax.yaml:

% cat docker-compose-syntax.yaml 
name: compose-test
services:
  db:
    image: postgres:latest
    restart: always
    environment:
      POSTGRES_USER: todo
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: todos
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  server:
    image: server
    depends_on: db
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgres://todo:[email protected]:5432/todos

volumes:
  pgdata:

[Mcrich23]

I apologize, that is a bad example, but it is true for other fields.

[jglogan]

I apologize, that is a bad example, but it is true for other fields.

No apologies needed! I'm just curious to see cases where the parser for Docker Compose would admit cases that don't comply to the JSON schema.

This doc describes everything the Compose parser does to transform YAML text into bound types: https://github.com/compose-spec/compose-go/blob/main/parsing.md.

[Mcrich23]

@jglogan So sorry for the delay. A lot going on personally including preparing for college (whoohoo!). Anyway, this is one example that comes to mind.

I feel like we maybe start with QuickType and find a way to fix these one off issues as we go. The good news is that people won't have an issue as long as people follow typical syntax instead of shortcuts.

[Mcrich23]

I wonder if we can make a general decoding pattern that allows all string arrays to be a single string outside of an array, etc.

Maybe member attribute or accessory macros could auto fill it so that just gets prepended to structs and we can still use QuickType.