Why is this deployment so needlessly complicated?

[rdvansloten]

Recently, a client of mine has started using this on Microsoft's advice, and I am anything but impressed. A lot of straight-forward components such as Terraform have been wrapped in and endless barrage of shell scripts, wrappers and the repo has some very questionable practices, such as storing access keys in the repository when first deploying the deployer VMS, not having an approval gate before running Terraform apply, etc. We are a small team of devops engineers/architects, and we are really struggling to figure out sometimes what goes where (due to a lot of the repo committing to self and merging/moving folders during deployment) and to make simple changes without the real risk of breaking things, as there is no dry-run or plan happening during deployment.

On top of this, we found out that this repository cannot be deployed on any agent that it did not built itself. We are using both cloud and self-hosted versions of the official Azure DevOps/GitHub agent, and the pipelines will simply not run due to a flood of different errors. It will look in hardcoded places for the agent, the cloned repo, but these places only exist in this custom runner. Even figuring out where these locations came from took a long time, jumping from script to script, then into a Terraform template. Variables get passed through and into things so much that it's really difficult to piece things together.

Why is a project centered around Terraform and Ansible not idempotent? Why has all this vigorous scripting and wrapping been done, knowing it makes adding custom snippets of code or editing this so difficult? Are maintainers aware of this and are there any plans to change this, or will this core be built upon for years to come? I'm not asking because I have such a large vested interest, claim to know better or do better, but I can't in good conscience recommend this product to my clients, even if a Microsoft rep is telling them to use this.

Who was originally the target audience for this? In practice, the target audience for this is a team of SRE/devops that deploys this and then hands it over to SAP developers who are quite green when it comes to IaC and cloud deployment. They'll often understand the basic tenets of Terraform, Linux, cloud and Ansible, but not the rigorous scripting and wrapping done in this repository. If I don't understand all of it after weeks of trying to work with this, how will I teach less experienced SAP developers to run these pipelines and occasionally solve their own deployment errors after my work is done?

[UncleBats]

Hello @rdvansloten ,

I do not have all the answers to your questions, a lot of your questions we have for ourselves as well however, we managed last week to deploy using azdo pipeline completely (no use of the deployer). All required changes to run on default azdo agents has been provide with PR to the experimental branches. Ofc there could be some misconfiguration but we are able to deploy and provision a distributed 1DB: 1APP: 1SCS and a standalone system successful and are currently looking into HA setup.

For the agent configuration we are running our own private azdo agents. Build from the packer files each night. There is nothing custom about that but it is required for us due to the use of private-endpoints.

[rdvansloten]

Hi @ErickSegaar,

Thanks for the reply!

All required changes to run on default azdo agents has been provide with PR to the experimental branches

That's very good news, that will cut some setup pains at least.

For the agent configuration we are running our own private azdo agents. Build from the packer files each night. There is nothing custom about that but it is required for us due to the use of private-endpoints.

Yeah, same here. We run our pipelines on the official DevOps/Github agent repo: https://github.com/actions/runner-images. On the current main branch though, these do not work yet, so there are some dedicated SAP deployer VMs for now.

[UncleBats]

What are you running into on the unable to run the devops images. We have solved all the packer issues but every now and then that nightly build fails. So no 100% score but a 80% should be achievable. If you want more info on how to set it up, just reach out.

[rdvansloten]

Sorry it may have been confusing, the agents build fine from that actions repo. It's that the SAP Framework code expects things to be there that aren't available on the public/private agents. This script: https://github.com/Azure/sap-automation/blob/e35265c658a214fa79f8a8f73a0db6ac944794e6/deploy/scripts/configure_deployer.sh

Does some things that are not done on the AzDo agents, such as:

git clone "${asad_repo}" "${asad_dir}"

I tried running it on both the public and private agents, it will throw this as one of the issues.

[UncleBats]

ah, your trying to setup the deployer. What we did is just disable the deployer. You can run everything from the azdo pipeline and just skip the deployer setup.

just add the following to your TF of the management layer

deployer_enable_public_ip=false
deployer_count=0

[UncleBats]

This will make your pipeline succeed to at least the 05 pipeline. After that you need some ansible fixes that are in the following PR's
#294
#288
#297