Proposal: operations/bbl.yml

[evanfarrar]

Hello Concoursers!

On the CF Infrastructure team we'd like to really up our support for concourse. Lots of people are using concourse, lots of people are making their first BOSH to deploy concourse to deploy Cloud Foundry. It really seems to be the easiest way, and it isn't all that easy. Would you consider, as a pilot program, allowing us to programmatically update (I'm ok with PR, I guess, as long as you are ok with PRs from robots) a single file "operations/bbl.yml" that enables most of the functionality we see as helpful for operators, particularly operators just getting started.

This ops file would:

• Autogenerate a self-signed cert
• Deploy credhub and UAA and configure it to be used for auth/secrets
• Use an external database, provided by cross deployment links, provisioned by BBL
• Use a domain name, provided by cross deployment links, with DNS provisioned by BBL
• Preset the network to default, the one that bbl provides via cloud-config
• Preset the web vm extension to a name so that bbl can provide it via cloud-config
• Use compiled releases
• Use bosh release versions
• Use calculated VM sizes instead of VM type names
• Use a particular stemcell that has been tested in a pipeline instead of "default", which sometimes has issues (current "latest" is incompatible with garden).

[jama22]

Interesting suggestion @evanfarrar. Sounds like a mighty powerful ops file. Apologies in advance if my follow questions are a bit noob; I'm not too familiar with bbl and how it sets up Concourse:

• I assume this operations/bbl.yml would be to enhance the steps described here?
• You describe this is a pilot, what are the uncertainties and/or assumptions that you are testing for by doing this programmatically?
• Follow up: what are the specific uncertainties that you want to test that make it necessary to be put under this repo (concourse-deployment)

[vito]

(@evanfarrar Sorry, I realize after writing this that it sounds pretty harsh. It's not meant to be directed at you in particular, there's just a lot of frustration here.)

After living in the CF bubble for years I'm extremely wary of new tools like bbl that seem to augment BOSH without it feeling like an official part of BOSH or a recommended part of its ecosystem (i.e. prominently in https://bosh.io docs, not an afterthought). There have been so many of these invented over the years that I'd rather not expend the effort to embrace any of them until BOSH does.

I'm even more wary of tools that also require work for individual release authors. If and when BOSH's ecosystem grows to the point I want it to, that'll be a whole lot of busywork. I appreciate that it's something your team would like to do/automate for us, but the point still stands. What is it about the bbl.yml file that couldn't just be provided by other, more generic, ops files? (Or, as @jama-pivotal said, it could be in your own repos.)

I want to see BOSH succeed as an open-source project. So I'm trying to have Concourse remain an exemplary use of BOSH, as a user knowing only BOSH (further, knowing nothing about BOSH) would see it. For this approach, https://bosh.io, its documentation, and the bosh CLI is the source of truth. There's enough of a stigma around BOSH's learning curve that I'd rather not perpetuate it by making it appear that you have to use a third-party tool (even if it's within CF) just to get started. It makes BOSH's open-source users feel like an afterthought.

[evanfarrar]

Sorry, I missed this for a long time.

I assume this operations/bbl.yml would be to enhance the steps described here?

Yes, this PR is the ops file that those instructions tell you to write out and then use when deploying. But there are still problems, even if this little YML file editing weren't adding another step to the process: it takes a million years to compile, it still requires 17 totally static command line arguments to bosh deploy, and there is a bit of an uncertain leap to figuring out how to make this more production-friendly with external databases. That's why I went on to suggest maybe this one YML file shouldn't the end of BBL's relationship with concourse-deployment. We could show some new features in one "experimental, community maintained" corner of the manifest, and y'all could decide if and when you want to make them mainstream.

You describe this is a pilot, what are the uncertainties and/or assumptions that you are testing for by doing this programmatically?

By programmatically, I mean, via Continuous Integration. I could compile releases, and add them to "my" ops file in this repo, but I'd rather have a test suite do that only after verifying that the latest versions actually work then commit the result. Nobody should have to argue with robots though, so I'm suggesting this hypothetical CI robot get commit rights to your repo so that it can edit that ops file (bbl.yml) without a PR.

Follow up: what are the specific uncertainties that you want to test that make it necessary to be put under this repo (concourse-deployment)

I have very few uncertainties about this ops file, or the potential for continuing to close the gap concourse-deployment has with the operator friendly features of cf-deployment. Having it in this repo enhances the odds that users (and maintainers) might discover this new, easier experience, and allows me to point users to a single repo with everything necessary to conveniently deploy Concourse.

I'm even more wary of tools that also require work for individual release authors. If and when BOSH's ecosystem grows to the point I want it to, that'll be a whole lot of busywork. I appreciate that it's something your team would like to do/automate for us, but the point still stands. What is it about the bbl.yml file that couldn't just be provided by other, more generic, ops files? (Or, as @jama-pivotal said, it could be in your own repos.)

I would want a bbl.yml file to pack it with even more opinionated defaults in the future. I could have just PRed these particular changes into the base manifest, but I assume you knew this was possible, and had reasons for not doing it. Likewise, there are like 17 other vars that you could have made sane defaults for, but for some reason do not. That's fine, but I'd like to supply as few flags as possible to bosh deploy, even if it means some duplication and even if you want to name this file experimental/bbl-community-maintained-experimental-do-not-use.yml.

Nothing about deploying Concourse to a BBL created BOSH director requires sustained work for manifest authors. There are some things that require sustained work that are unrelated to BBL, but I would happily do them just to make the experience of Concourse on BOSH as good as it can possibly be: compiled releases, new IaaS features, and new opinions about the best configuration of the IaaS. I'd also like to supply release versions that I know have been smoke tested, because that has not been my experience.

Maintaining a separate concourse-deployment manifest was what you suggested originally (concourse/concourse#1034). I was thrilled when Concourse finally made a manifest, but I kind of assumed you read mine or cf-deployment when you made this one? I want to point users at a single source of truth. I don't want to verbally describe editing YAML files to my users. I want version controlled, declarative descriptions of our software that we know work. I don't want to describe the recipe for creating that declarative description.

I'd be the first to admit that there is a ton of busywork involved in making a good BOSH deployment manifest, and ironically Concourse has been the only way out of that mess. By demonstrating the patterns we follow in automation and higher level abstractions, we have collectively made BOSH much better. There is still a long way to go.

I want to see BOSH succeed as an open-source project. So I'm trying to have Concourse remain an exemplary use of BOSH, as a user knowing only BOSH (further, knowing nothing about BOSH) would see it. For this approach, https://bosh.io, its documentation, and the bosh CLI is the source of truth. There's enough of a stigma around BOSH's learning curve that I'd rather not perpetuate it by making it appear that you have to use a third-party tool (even if it's within CF) just to get started. It makes BOSH's open-source users feel like an afterthought.

We could merge into both the BOSH cli as bosh bootload and update the docs (and already do merge our features into the BOSH cli from time to time), but we'd love it if some more teams who are such paragons of BOSH usage as yourself would try the new experience and give feedback on it before we throw it at users who don't get a paycheck to use BOSH.