Rust CUDA 0.3

Top: upcoming MPM engine that runs on CPU and GPU using rust-cuda, Bottom: toy path tracer that can run on CPU, GPU, and GPU (hardware raytracing) using recent experiments with OptiX

Today marks an exciting milestone for the Rust CUDA Project, over the past couple of months, we have made significant advancements in supporting many of the fundamental CUDA ecosystem libraries. The main changes in this release are the changes to cust to make future library support possible, but we will also be highlighting some of the WIP experiments we have been conducting.

Cust changes

This release is likely to be the biggest and most breaking change to cust ever, we had to fundamentally rework how many things work to:

• Fix some unsoundness.
• Remove some outdated and inconsistent things.
• Rework how contexts work to be interoperable with the runtime API.

Therefore this release is guaranteed to break your code, however, the changes should not break too much unless you did a lot of lower-level work with device memory constructs.

Cust 0.3 changes

TLDR

This release is gigantic, so here are the main things you need to worry about:

Context::create_and_push(FLAGS, device) -> Context::new(device).
Module::from_str(PTX) -> Module::from_ptx(PTX, &[]).

Context handling overhaul

The way that contexts are handled in cust has been completely overhauled, it now
uses primary context handling instead of the normal driver API context APIs. This
is aimed at future-proofing cust for libraries such as cuBLAS and cuFFT, as well as
overall simplifying the context handling APIs. This does mean that the API changed a bit:

• create_and_push is now new and it only takes a device, not a device and flags.
• set_flags is now used for setting context flags.
• ContextStack, UnownedContext, and other legacy APIs are gone.

The old context handling is fully present in cust::context::legacy for anyone who needs it for specific reasons. If you use quick_init you don't need to worry about
any breaking changes, the API is the same.

cust_core

DeviceCopy has now been split into its own crate, cust_core. The crate is #![no_std], which allows you to
pull in cust_core in GPU crates for deriving DeviceCopy without cfg shenanigans.

Removed

• DeviceBox::wrap, use DeviceBox::from_raw.
• DeviceSlice::as_ptr and DeviceSlice::as_mut_ptr. Use DeviceSlice::as_device_ptr then DevicePointer::as_(mut)_ptr.
• DeviceSlice::chunks and consequently DeviceChunks.
• DeviceSlice::chunks_mut and consequently DeviceChunksMut.
• DeviceSlice::from_slice and DeviceSlice::from_slice_mut because it was unsound.
• DevicePointer::as_raw_mut (use DevicePointer::as_mut_ptr).
• DevicePointer::wrap (use DevicePointer::from_raw).
• DeviceSlice no longer implements Index and IndexMut, switching away from [T] made this impossible to implement.
  Instead you can now use DeviceSlice::index which behaves the same.
• vek is no longer re-exported.

Deprecated

• Module::from_str, use Module::from_ptx and pass &[] for options.
• Module::load_from_string, use Module::from_ptx_cstr.

Added

• cust::memory::LockedBox, same as LockedBuffer except for single elements.
• cust::memory::cuda_malloc_async.
• cust::memory::cuda_free_async.
• impl AsyncCopyDestination<LockedBox<T>> for DeviceBox<T> for async HtoD/DtoH memcpy.
• DeviceBox::new_async.
• DeviceBox::drop_async.
• DeviceBox::zeroed_async.
• DeviceBox::uninitialized_async.
• DeviceBuffer::uninitialized_async.
• DeviceBuffer::drop_async.
• DeviceBuffer::zeroed.
• DeviceBuffer::zeroed_async.
• DeviceBuffer::cast.
• DeviceBuffer::try_cast.
• DeviceSlice::set_8 and DeviceSlice::set_8_async.
• DeviceSlice::set_16 and DeviceSlice::set_16_async.
• DeviceSlice::set_32 and DeviceSlice::set_32_async.
• DeviceSlice::set_zero and DeviceSlice::set_zero_async.
• the bytemuck feature which is enabled by default.
• mint integration behind impl_mint.
• half integration behind impl_half.
• glam integration behind impl_glam.
• experimental linux external memory import APIs through cust::external::ExternalMemory.
• DeviceBuffer::as_slice.
• DeviceVariable, a simple wrapper around DeviceBox<T> and T which allows easy management of a CPU and GPU version of a type.
• DeviceMemory, a trait describing any region of GPU memory that can be described with a pointer + a length.
• memcpy_htod, a wrapper around cuMemcpyHtoD_v2.
• mem_get_info to query the amount of free and total memory.
• DevicePointer::as_ptr and DevicePointer::as_mut_ptr for *const T and *mut T.
• DevicePointer::from_raw for CUdeviceptr -> DevicePointer<T> with a safe function.
• DevicePointer::cast.
• dependency on cust_core for DeviceCopy.
• ModuleJitOption, JitFallback, JitTarget, and OptLevel for specifying options when loading a module. Note that
  ModuleJitOption::MaxRegisters does not seem to work currently, but NVIDIA is looking into it.
  You can achieve the same goal by compiling the ptx to cubin using nvcc then loading that: nvcc --cubin foo.ptx -maxrregcount=REGS
• Module::from_fatbin.
• Module::from_cubin.
• Module::from_ptx and Module::from_ptx_cstr.
• Stream, Module, Linker, Function, Event, UnifiedBox, ArrayObject, LockedBuffer, LockedBox, DeviceSlice, DeviceBuffer, and DeviceBox all now impl Send and Sync, this makes
  it much easier to write multigpu code. The CUDA API is fully thread-safe except for graph objects.

Changed

• zeroed functions on DeviceBox and others are no longer unsafe and instead now require T: Zeroable. The functions are only available with the bytemuck feature.
• Stream::add_callback now internally uses cuLaunchHostFunc anticipating the deprecation and removal of cuStreamAddCallback per the driver docs. This does however mean that the function no longer takes a device status as a parameter and does not execute on context error.
• Linker::complete now only returns the built cubin, and not the cubin and a duration.
• Features such as vek for implementing DeviceCopy are now impl_cratename, e.g. impl_vek, impl_half, etc.
• DevicePointer::as_raw now returns a CUdeviceptr instead of a *const T.
• num-complex integration is now behind impl_num_complex, not num-complex.
• DeviceBox now requires T: DeviceCopy (previously it didn't but almost all its methods did).
• DeviceBox::from_raw now takes a CUdeviceptr instead of a *mut T.
• DeviceBox::as_device_ptr now requires &self instead of &mut self.
• DeviceBuffer now requires T: DeviceCopy.
• DeviceBuffer is now repr(C) and is represented by a DevicePointer<T> and a usize.
• DeviceSlice now requires T: DeviceCopy.
• DeviceSlice is now represented as a DevicePointer<T> and a usize (and is repr(C)) instead of [T] which was definitely unsound.
• DeviceSlice::as_ptr and DeviceSlice::as_ptr_mut now both return a DevicePointer<T>.
• DeviceSlice is now Clone and Copy.
• DevicePointer::as_raw now returns a CUdeviceptr, not a *const T (use DevicePointer::as_ptr).
• Fixed typo in CudaError, InvalidSouce is now InvalidSource, no more invalid sauce 🍅🥣

Line tables

The libnvvm codegen can now generate line tables while optimizing (previously it could generate debug info but not optimize), which allows you to debug and profile kernels much better in tools like Nsight Compute. You can enable debug info creation using .debug(DebugInfo::LineTables) with cuda_builder.

OptiX

Using the generous work of @anderslanglands, we were able to get rust-cuda to target hardware raytracing completely in rust (both for the host and the device). The toy path tracer example has been ported to be able to use hardware rt as a backend, however, optix and optix_device are not published on crates.io yet since they are still highly experimental.

using hardware rt to render a simple mesh

cuBLAS

Work on supporting cuBLAS through a high-level wrapper library has started, a lot of work needed to be done in cust to interop with cuBLAS which is a runtime API based library. This required some changes with how cust handles contexts to avoid dropping context resources cuBLAS was using. The library is not yet published but eventually will be once it is more complete. cuBLAS is a big piece of neural network training on the GPU so it is critical to support it.

cuDNN

@frjnn has been generously working on wrapping the cuDNN library. cuDNN is the primary tool used to train neural networks on the GPU, and the primary tool used by pytorch and tensorflow. High level bindings to cuDNN are a major step to making Machine Learning in Rust a viable option. This work is still very in-progress so it is not published yet, it will be published once it is usable and will likely first be used in neuronika for GPU neural network training.

Atomics

Work on supporting GPU-side atomics in cuda_std has started, some preliminary work is already published in cuda_std, however, it is still very in-progress and subject to change. Atomics are a difficult issue due to the vast amount of options available for GPU atomics, including:

• Different atomic scopes, device, system, or block.
• Specialized instructions or emulated depending on the compute capability target.
• Hardware float atomics (which core does not have)

You can read more about it here.