Issues with floating-point atomics

[OsKnoth]

I run my code under oneAPI on dawn (Cambridge, UK) and

for a kernel with

@atomic :monotonic M[Iz,ind,1] += JJ[ID,1,Iz,IF]

we get the error

ERROR: LoadError: Failed to translate LLVM code to SPIR-V.
RequiresExtension: Feature requires the following SPIR-V extension:
 SPV_EXT_shader_atomic_float_add

[OsKnoth]

After some more insights to oneAPI

I tried now the following short program on dawn (University of Cambridge)

using KernelAbstractions
using Atomix
using Adapt
using oneAPI

@kernel function mykernel(x, y)
    i = @index(Global)
    Atomix.@atomic x[1] += y[i]
end

backend = oneAPIBackend()
@show backend

x_ = zeros(Float64, 1)
y_ = rand(Float64, 100)
x = Adapt.adapt(backend, x_)
y = Adapt.adapt(backend, y_)

mykernel(backend)(x, y, ndrange=length(y))

@info "" sum(x) sum(y)

and get the following traceback

ERROR: LoadError: InvalidIRError: compiling MethodInstance for gpu_mykernel(::KernelAbstractions.CompilerMetadata{…}, ::oneDeviceVector{…}, ::oneDeviceVector{…}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to atomic_add!)
Stacktrace:
 [1] modify!
   @ ~/.julia/packages/Atomix/UFuJD/ext/AtomixoneAPIExt.jl:38
 [2] gpu_mykernel
   @ ~/.julia/packages/KernelAbstractions/HqVoT/src/macros.jl:324
 [3] gpu_mykernel
   @ ./none:0
Reason: unsupported call to an unknown function (call to gpu_malloc)
Stacktrace:
 [1] malloc
   @ ~/.julia/packages/GPUCompiler/Ecaql/src/runtime.jl:85
 [2] gc_pool_alloc
   @ ~/.julia/packages/GPUCompiler/Ecaql/src/runtime.jl:116
 [3] modify!
   @ ~/.julia/packages/Atomix/UFuJD/ext/AtomixoneAPIExt.jl:38
 [4] gpu_mykernel
   @ ~/.julia/packages/KernelAbstractions/HqVoT/src/macros.jl:324
 [5] gpu_mykernel
   @ ./none:0
Hint: catch this exception as `err` and call `code_typed(err; interactive = true)` to introspect the erroneous code with Cthulhu.jl
Stacktrace:
Stacktrace:
  [1] check_ir(job::GPUCompiler.CompilerJob{GPUCompiler.SPIRVCompilerTarget, oneAPI.oneAPICompilerParams}, args::LLVM.Module)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/validation.jl:167
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:385 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/Tracy/slmNc/src/tracepoint.jl:163 [inlined]
  [4] macro expansion
    @ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:384 [inlined]
  [5] emit_llvm(job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/utils.jl:116
  [6] emit_llvm(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/utils.jl:114
  [7] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:95
  [8] compile_unhooked
    @ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:80 [inlined]
  [9] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:67
 [10] compile
    @ ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:55 [inlined]
 [11] #37
    @ ~/.julia/packages/oneAPI/eeEyI/src/compiler/compilation.jl:81 [inlined]
 [12] JuliaContext(f::oneAPI.var"#37#38"{GPUCompiler.CompilerJob{…}}; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:34
 [13] JuliaContext(f::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/driver.jl:25
 [14] compile(job::GPUCompiler.CompilerJob)
    @ oneAPI ~/.julia/packages/oneAPI/eeEyI/src/compiler/compilation.jl:80
 [15] actual_compilation(cache::Dict{…}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{…}, compiler::typeof(oneAPI.compile), linker::typeof(oneAPI.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/execution.jl:245
 [16] cached_compilation(cache::Dict{…}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{…}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Ecaql/src/execution.jl:159
 [17] macro expansion
    @ ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:203 [inlined]
 [18] macro expansion
    @ ./lock.jl:273 [inlined]
 [19] zefunction(f::typeof(gpu_mykernel), tt::Type{Tuple{…}}; kwargs::@Kwargs{})
    @ oneAPI ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:198
 [20] zefunction(f::typeof(gpu_mykernel), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{…}, oneDeviceVector{…}, oneDeviceVector{…}}})
    @ oneAPI ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:195
 [21] macro expansion
    @ ~/.julia/packages/oneAPI/eeEyI/src/compiler/execution.jl:66 [inlined]
 [22] (::KernelAbstractions.Kernel{…})(::oneArray{…}, ::Vararg{…}; ndrange::Int64, workgroupsize::Nothing)
    @ oneAPI.oneAPIKernels ~/.julia/packages/oneAPI/eeEyI/src/oneAPIKernels.jl:89
 [23] top-level scope
    @ ~/GitHub/CliMA/CGDycore.jl/TestKernels/OneAPIAtomic.jl:19
 [24] include(fname::String)
    @ Main ./sysimg.jl:38
 [25] top-level scope
    @ REPL[1]:1
in expression starting at /home/dn-knot1/GitHub/CliMA/CGDycore.jl/TestKernels/OneAPIAtomic.jl:19
Some type information was truncated. Use `show(err)` to see complete types.

[michel2323]

Can you retry with this branch? #542 Once it passes the test, I'll merge and do another release.

[OsKnoth]

How should I change my environment to use this branch.

[michel2323]

I dug a bit into this. SPIRVIntrinsics.jl only generates routines for integers https://github.com/JuliaGPU/OpenCL.jl/blob/95039b8c271c0fc0734ed490102fe9ee72986ac0/lib/intrinsics/src/atomic.jl#L6-L17. SPIRV has an extension for it. I don't know where to add that though @maleadt @vchuravy https://github.khronos.org/SPIRV-Registry/extensions/EXT/SPV_EXT_shader_atomic_float_add.html

[michel2323]

I've just activated the extension #544 and added the missing methods in JuliaGPU/OpenCL.jl#399 . That seems to work on your example.