On the State of Terra in 2022

[elliottslaughter]

Hey all,

I thought it would be fun to write an annual-ish summary of where Terra has been in the last year. This is my update for 2022. Feel free to discuss here.

Release Updates

In case you haven't been following along, we've released a number of betas over the last couple of years. We are now quite close to releasing 1.0.0. If you want to help, please refer to this issue on the latest beta.

Some of the notable changes in recent betas include:

• Support for more LLVM versions. We're currently up to 14 (the most recent version as of this time of writing). I've been happy that we've been able to keep up to date; it's not such a long time ago when we were many versions behind on the LLVM treadmill.

• Support for AMD GPUs. This is very experimental, but has already been used in Regent to provide seamless GPU code generation for NVIDIA and AMD hardware. Preliminary performance looks good, and we're pretty much set to run on the Frontier supercomputer when it arrives.

• Fixes for performance regressions in CUDA. It used to be that you had to use LLVM 3.8 if you wanted to get the best performance on NVIDIA GPUs. Now, at least in my testing, we're able to do this with LLVM 13.

• Support for (experimental) concurrency primitives. Right now this includes the LLVM atomicrmw, fence, and cmpxchg instructions.

• Much better cross-version compatibility for Linux. You can now build on Ubuntu 18.04 and expect it to work on 20.04, 22.04, etc. I have yet to test the cross-distro compatibility, but I'm optimistic that this may work as well now.

• Some older changes include CMake support, substantially revamped Windows support, Nix support, a dramatically better CI and release infrastructure, and many, many bug fixes.

A Short Update on Terra Users

I'm personally most plugged into the Regent community. Regent users---by virtue of Regent being built on top of Terra---are (indirectly) Terra users.

Terra (via Regent) has been used on many of the top supercomputers (at least those in the USA, and some in Europe). Regent uses Terra to implement seamless support for generating efficient GPU kernels. We've been doing this for NVIDIA GPUs for many years now, and have recently started supporting AMD as well. Intel GPU support is also on the roadmap. The result is that, in a fairly short amount of time, Terra/Regent have grown into a powerful way to write performance-portable GPU code. That's pretty awesome.

I'd love to hear more about what other users are doing with Terra. Swing by our Zulip instance and let us know what you're working on!

Maintainership

Let me state the obvious up front: I'm not Zach. Zach moved on from his postdoc position a number of years ago, and I stepped into the gap to help keep Terra going.

Terra is a relatively small community. At this time, we have no corporate sponsors and no full-time (or even significant part-time) staff. I count myself lucky to be able to put about 5% of my time into the project, but it's still not a lot. I've been focusing mainly on keeping Terra up to date with the ecosystem changes (e.g., LLVM versions) and making sure Terra is able to target the hardware that comes along (particularly for USA-based supercomputers). I also try to do what I can to mentor other users who are interested in working on Terra.

As usual, if you want to support Terra, the best way is to get involved in the community. There are plenty of things to work on, and we love community contributions! If you need ideas, feel free to pop over to Zulip and let us know you're interested.

Goals for Terra

I believe strongly that Terra goes where the community wants it to go, where "community" opinion is weighted in favor of those those who actually put in the effort to do something about it. Maybe some day we'll be large enough to have a dedicated team with a more formal process, but right now that's not where Terra is at. First and foremost, Terra has to serve the community it has.

However, as one of those people who are putting active effort into Terra, I suppose I should say something about my goals. To be clear, these are my personal goals. I do not claim to speak for anyone else on the project (or for any past, present or future employer).

In roughly decreasing order of priority, I want Terra to have:

1. Best in class code generation support for AMD, Intel and NVIDIA GPUs (and of course CPUs too). You should never get to a point where you have to switch to CUDA, HIP, SYCL, etc. to accomplish what you need to do, and performance should be at least as good as any of those platforms.

2. Best in class support for HPC platforms and machines (especially those in the USA). I am biased in this regard by my employer, which as a U.S. DOE laboratory, has an interest in running on current and upcoming DOE supercomputers. But I also put effort into running on machines used by my collaborators, for example in Europe.

3. Best in class support for HPC codes and applications. That's not to say I don't or wouldn't support other classes of applications, but HPC has needs that are often not well-represented in the popular programming languages. My usage of Terra is mostly HPC-focused, so that's where my effort goes.

4. No unnecessary breakage. I am first and foremost a user of Terra. For me, this is not an exercise in language design, it's about getting work done. If Terra breaks, it gets in the way of me doing that.

5. And lastly, I aim to support community efforts as best I can in whatever time is remaining after fulfilling my other obligations.

Getting Involved

I think this is an exciting time to get involved with Terra, with some of the things that are around the corner. If you're interested, we'd love to chat and discuss how to get started.

I have also cross-posted this on my home page.