HPC & Scientific Computing (pointer restrictions, etc.)

[veritas9872]

Hello, I have a question/suggestion concerning pointer restrictions in Carbon.
Currently, the C99 standard implements the restrict keyword to prevent pointer aliasing. Although restrict is not part of the C++ standard, most compilers implement it as __restrict, which has the same effect.
This is a very important issue in High-Performance Computing (HPC), where the compiler's inability to determine if a pointer uniquely refers to a memory location prevents it from performing several valuable optimizations. This is often cited as the reason that naive C cannot be as efficient as Fortran.
A simple solution would be to re-implement the restrict keyword in Carbon. However, this would be very verbose and difficult to read in HPC applications. I give an example below.

template<typename T, bool IS_OUTPUT, bool IS_BIAS, int RESIDUAL_NUM, bool IS_BETA, int UNROLL_FACTOR>
void generalAddBiasResidualLayerNormOpt(T* normed_output,
                                        T* output,
                                        const T* __restrict bias,
                                        const T* __restrict residual1,
                                        const T* __restrict residual2,
                                        const T* __restrict gamma,
                                        const T* __restrict beta,
                                        const float layernorm_eps,
                                        int         m,
                                        int         n)

I am curious as to whether there are better methods of making such optimizations without introducing bugs. It would be an improvement to at least have shorter keywords.

[geoffromer]

@chandlerc I have a vague recollection that you had some thoughts about restrict.

[mossaiby]

@geoffromer, @chandlerc: +1 for optimizing Carbon for HPC. Please consider this group of users in your designs.

[veritas9872]

Perhaps a special type of pointer, similar to unique_pointer could be implemented in the standard library instead of the core language. It could also raise an error during compilation if the uniqueness of the pointer is violated, whereas const keyword currently has undefined behavior when the pointer is not unique.
Also, considering that this scenario is most common in arrays, it may be preferable to have a special type of array with restricted pointers. This could also be implemented in the standard library instead of the core language.

[veritas9872]

This special array type could also have optimizations for loop unrolling and other common optimizations.

[OlaFosheimGrostad]

Have you looked at https://chapel-lang.org/ ?

[veritas9872]

Or maybe Fortran18 could give some hints. It is similarly low-level.