Support Jenkins as build executor

[JasperDeLaat94]

I have UET running in a basic, non-distributed form on my Jenkins server. I would like to see if a Jenkins executor can be implemented.

I want to give it a shot implementing this myself using their Pipeline DSL system, but I think it would be good to open an issue here to share some knowledge, perhaps you already looked into this?

[hach-que]

Yeah I'd love to see a Jenkins executor implemented. From memory when I was thinking about this, the way forward was for UET to output Jenkinsfile files for the Job DSL plugin to process. Thus you'd basically have a "seed job" that runs uet build -x jenkins ... to generate the Jenkinsfile, and then a Job DSL step after that to generate the downstream pipeline/jobs that would actually run.

[JasperDeLaat94]

I've only briefly looked at Job DSL and Pipeline DSL, but haven't figured out yet how to 're-enter' them with additional generated pipeline code (at least, not in an easy way like GitLab allows). Is there a specific reason Job DSL caught your interest?

[JasperDeLaat94]

I think I have two potential approaches to the 'generate, then execute' approach for Groovy-based Jenkins DSL's (I don't think it matters if it's implemented in a scripted Pipeline DSL or Jobs DSL?):

1. Let UET generate a JSON file, then read it using readJSON and generate the groovy code. I prefer this as generating a JSON file from .NET is quite trivial.
2. Let UET generate a groovy script and load/execute it directly. This probably complicates things a lot on the UET side since you'd have to write a custom serializer so might not be a good idea.

I am quite new to a lot of concepts here so I'm checking it with you to see if I'm not spitting out complete nonsense. What are your thoughts?

[hach-que]

The load/execute of a Groovy file is maybe the best option, since the readJSON approach would mean having the JSON -> step generator inside Jenkins, and we want to minimize the amount of set up/maintainence on the Jenkins side - ideally we should be able to push out fixes to Jenkins generation in UET, and not require people to update their job configurations in Jenkins itself.

Job DSL might not be the right approach - I just remember years ago that was how we ended up generating very complex pipelines for Unreal because the pipeline DSL wasn't quite powerful enough. Though since Job DSL does generate jobs at the Jenkins level, maybe it isn't the right approach for dynamic jobs (because then we'd need to uniquely name them per build and clean them up afterwards?)

The other issue we'll have with Jenkins/pipeline DSL is that it doesn't look like it yet supports arbitrary dependencies on pipeline steps, only trivial parallelization. https://stackoverflow.com/a/59923355/529646 mentions using locks to block parts of the parallel graph, but I don't know whether this will acquire build nodes in Jenkins and then block (which would not be ideal because you could then deadlock without enough nodes).

Then there is always the nuclear option of having UET implement a build executor that directly calls https://ci.jenkins.io/createItem with the Jenkins XML for that particular step of BuildGraph and then submits it to the build queue. The downside of this would be that the equivalent "generate" step on GitLab would actually be a "build" step on Jenkins, where a node is allocated just to run the uet build -x jenkins process as it runs and monitors the build (delegating out each BuildGraph node to Jenkins as UET itself is aware that the dependencies in the graph are met). The upside however, is that you'd be able to run uet build -x jenkins on your local dev machine and have the build work delegated out to Jenkins (provided you had the Jenkins API key), which would be extremely useful for development/testing.

[hach-que]

The more I think about it, the more I'm thinking the "UET build executor directly calls the remote access API" is the best way to go. We don't need to deal with pipeline DSL limitations, Groovy syntax or require developers to update their Jenkins build configurations to fix bugs.

https://ci.jenkins.io/api/ unfortunately doesn't let me see the actual endpoints, but a real Jenkins instance should surface the whole API there. The endpoints we'd be interested in are:

• createItem, where we submit the Jenkins XML that is just a single execution of uet internal ci-build ... for a single BuildGraph node. There's the potential we could share the same job across all BuildGraph nodes of the same platform and just parameterize it, but I think it might be better if we do generate a job per BuildGraph node to make it easier to see in the Jenkins UI what is failing/passing.
• Submitting a build job to the queue, and getting the ID of the new build that was queued.
• Reading the logs/status of the new build that was queued (so that UET can "proxy" the build logs back to the job that is running uet build -x jenkins ....
• Deleting the job item when we're done with it? (Maybe we don't do this so that individual steps can still be seen? But then I worry about job clutter so we'd have to be a bit smart about how we generate job items)

If we can't figure these endpoints out from the Jenkins API endpoint, we might need to reverse engineer the calls from something like https://github.com/arangamani/jenkins_api_client which implements them in an SDK.

[JasperDeLaat94]

That sounds like a proper challenge, I hope you don't mind me taking a shot at it! :)