Zest 2.0

[chrisaxiom]

Features

Minimal Bash / Rewrite

• Why?
  • bash not easy to work on
  • bash limited in expression
  • bash not truly portable across platforms

Improve Peelfile

• Add ability to specify order of containers (or have zest work out dependencies based on Peelfile, Zestfile's and docker-compose.yml)
  • for the case where the BUILD_CONTAINER is an image that needs to be built and can be from something in the project
• Add ability to specify where and what containers to build
  • With cmd pattern, containers will be nested in arbitrarily deep folders; current strategy of maxdepth is not tenable
• Ability to pass down docker build & run args to certain projects

Improve Useability / Workflow

• Zest should know when containers need to be rebuilt when iterating based on files changing, etc.
  • With cmd pattern, the command for rebuilding containers is again too long
• Review common usage patterns and improve those
• Be smart with caching for golang or provide easy way to disable caching
• Allow for separate integration tests & global integration test binaries
• Allow use of Dockerfile
• Implement the help command

Note:

• Prefer GoLang due to no dependency

Existing Functionality To Keep:

Everything zest does now, including:

• Cache Dirs
• Pass Build Args to Docker Run & Docker Build
• All Pre/Post hooks
• Ability to run from anywhere (self contained)
• Ability to pull code from outside the context (so keep build and bundle steps)
• Ensure ability to run individual service (zest run <service_name>)

[chrisaxiom]

cc @axiomzen/back-end

[AndrewBurian]

Are we still keeping Zest language agnostic or should we start creeping in some Go specific enhancements?

[chrisaxiom]

The "builder pattern" usually is a benefit to compiled languages (Java, c.c++, etc), not so much for JS/Python...

In other words, we should be able to get the cacheing to work for those languages... or were you thinking of something else?

[Kay-Zee]

I wouldn't mind go specific stuff since that's more of what we use at the moment anyway. but probably not too much

[chrisaxiom]

LibCompose still looks like a garbage fire

[turbolent]

I'll try out using docker-compose as a Python library.

The "builder pattern" usually is a benefit to compiled languages (Java, c.c++, etc), not so much for JS/Python...

Python now has optional types, and IDEs support them / use them for completion, so no reason not to use the builder pattern there

[zhangchiqing]

Why not making it as simple as a Makefile template? Every project just implement it's own make build, make run and make test?

And in the root folder, we have another Makefile that has a make build which cd each folder and run make build in it?

[chrisaxiom]

Python now has optional types, and IDEs support them / use them for completion, so no reason not to use the builder pattern there

👍 🎉

Why not making it as simple as a Makefile template? Every project just implement it's own make build, make run and make test?

And in the root folder, we have another Makefile that has a make build which cd each folder and run make build in it?

Yup; we could bake that into the existing Zestfile's , or come up with a better master Peelfile (yml) that all that is specified from.

[chrisaxiom]

Posted in slack, just for reference:

I think I remember the reason why we still use the zest build && zest bundle scheme rather than a single multi-stage zest bundle; we needed to pull in code from outside the context because of the cmd pattern and that is done via a volume fed to a docker run command (zest build), and zest bundle (which is docker build) doesn’t allow that yet (moby/moby#14080)

[omatvejev]

Could we add implementing help command to the list above?

[chrisaxiom]

@omatvejev added

[chrisaxiom]

Because we are keeping the zest build and zest bundle steps separate (because of golang caching/out of context code mounting for build containers), this has implications.

zest needs to be aware of services; it could determine this from the compose file (v3), but v3 doesn't allow multiple tags; latest tag is nice as it keeps the compose file static; git commit hash tag is also nice but only when building for CI or pushing images locally. (we should add auto pruning of these images). However, it is nice to tag it with the commit hash at the time of building, instead of after the fact. So zest should remain in control of building containers; we can leverage the build section in the compose file however by simply feeding a --no-build option to docker-compose up.

Thinking of new names;

zest build => zest create-artifact or zest create
zest bundle => zest build (runs docker build)

Thoughts?

Support the following commands, all from root directory:

• zest run all => docker-compose up --no-build // brings up all services
• zest build => builds all containers found in docker-compose.yml
  • also support zest build <svc> variant
• zest bundle => bundles (docker build) all containers found in docker-compose.yml
  • also support zest bundle <svc> variant
• zest bb => zest build && zest bundle
  • also support zest bb <svc> variant
• zest recreate <svc> => zest build <svc>, zest bundle <svc>, and docker-compose up `
• zest reintegrate => zest build integrator, zest bundle integrator, and docker-compose up integrator` // keeps up containers upon failure of integration tests
• zest iterate <svc> => zest recreate <svc> then zest reintegrate // keeps up containers upon failure of integration tests
• zest all => same as before, but uses compose-file to locate services

Ability to pass down docker build & run args to certain projects

Zest will now feed in all the args and cmd stuff found in the compose file, as well as inject the REVISION tag.

Peelfile:

Will convert to yml

• keep integrator
  -run : support all option; Peelfile checked in but .gitignored to support a different default for run; perhaps service specified in run can be the default for naked commands, with the more specific commands overriding that.
• keep compose and integrate file options

Zestfile:

To bash or not to bash;

Advantage of Bash:

• Don't have to worry about escaping commands, etc. (ala yml files)

Disadvantage:

• its bash

For now, will move all vars to a yml file, but all hooks as bash (so Test() and Build() for example) until I figure out how to do this cleanly with just go/yml.

Zest:

Will port all logic to go; will possibly use go-basher to run shell scripts, and moby-client to run docker commands.

Will just use go's built in command stuff to run docker-compose commands for now, as the go library is a hot mess still.

_zester:

Will use go-bindata to build into go binary (keep as bash for now)

Not Supported

For now, not supporting building a container thats used for BUILD_CONTAINER from the project itself; Will have to prebuild and upload these to our private registry beforehand.

[Kay-Zee]

zest dev <srv> to either hit local for that particular service or to have a dev container that mounts the code and hot swaps.

[Kay-Zee]

Also, is there a way to specify a list of folders to watch, and do not bother building if nothing in those folders have changed? some sort of hashing similar to git's maybe? could piggyback on their infra as well

[chrisaxiom]

Also, is there a way to specify a list of folders to watch, and do not bother building if nothing in those folders have changed? some sort of hashing similar to git's maybe? could piggyback on their infra as well

That would be cool - but perhaps in v2.1?

[turbolent]

Another improvement could be to make @Kay-Zee's improvements to the versioning/tagging the default, see see https://github.com/dapperlabs/dapper-api/pull/753:

git log -n 1 --pretty=format:%h -- .

[chrisaxiom]

Parallel builds might see an improvement now