Making LLVM::ABI a private API

[HertzDevil]

LLVM::ABI is the type used by the compiler to determine the correct LLVM IR signature of C functions, by adding the appropriate sret and byval parameter attributes and turning those parameters into indirectly passed values, according to a given target's calling convention. It however has two problems:

• It is the only part of LLVM that consists mainly of Crystal code (in fact, a Rust source port) rather than bindings and wrappers.
• Since it operates on LLVM types only, and since LLVM types cannot represent C unions natively, no ABI can account for special rules used by C unions.

As an example, the RISC-V hardfloat calling convention roughly states that, if a struct has at most two nested fields of floats smaller than ABI_FLEN bits or integers smaller than XLEN bits, that entire struct may be passed directly in registers. In particular, for ILP32D (32-bit integers and 64-bit floats), an otherwise indirect parameter could be passed directly this way for e.g. a struct with 2 double fields. This rule doesn't apply to unions though; they are not considered as nested fields in the same way, and always use the integer calling convention. This C snippet demonstrates the differences. (syntacore-scr4-rv32 has 32-bit and 64-bit hardfloat support.)

Crystal and Clang both work around the lack of union support in LLVM by creating a different LLVM struct type with the same size and alignment. Ours looks like this:

crystal/src/compiler/crystal/codegen/llvm_typer.cr

Lines 320 to 395 in 558ce7b

	private def create_llvm_c_union_struct_type(type, wants_size)
	llvm_name = llvm_name(type, wants_size)
	if s = @structs[llvm_name]?
	return s
	end
	@llvm_context.struct(llvm_name) do |a_struct|
	if wants_size
	@wants_size_struct_cache[type] = a_struct
	else
	@struct_cache[type] = a_struct
	@structs[llvm_name] = a_struct
	end
	# We are going to represent the union like this:
	# 1. Find out what's the type with the largest alignment
	# 2. Find out what's the type's size
	# 3. Have the first member of the union be an array
	# of ints that match that alignment, up to that type's
	# size, followed by another member that is just bytes
	# to fill the rest of the union's size.
	#
	# So for example if we have this:
	#
	# struct Foo
	# x : Int8
	# y : Int32
	# end
	#
	# union Bar
	# foo : Foo
	# padding : UInt8[24]
	# end
	#
	# We have that for Bar, the largest alignment of its types
	# is 4 (for Foo's Int32). Foo's size is 8 bytes (4 for x, 4 for y).
	# Then for the first union member we'll have [2 x i32].
	# The total size of the union is 24 bytes. We already filled
	# 8 bytes so we still need 16 bytes.
	# The resulting union is { [2 x i32], [16 x i8] }.
	max_size = 0
	max_align = 0
	max_align_type = nil
	max_align_type_size = 0
	type.instance_vars.each do |name, var|
	var_type = var.type
	unless var_type.void?
	llvm_type = llvm_embedded_c_type(var_type, wants_size: true)
	size = size_of(llvm_type)
	align = align_of(llvm_type)
	if size > max_size
	max_size = size
	end
	if align > max_align
	max_align = align
	max_align_type = llvm_type
	max_align_type_size = size
	end
	end
	end
	filler = @llvm_context.int(max_align * 8)
	filler_size = max_align_type_size // max_align
	union_fill = [filler.array(filler_size)] of LLVM::Type
	if max_align_type_size < max_size
	union_fill << @llvm_context.int8.array(max_size - max_align_type_size)
	end
	union_fill
	end
	end

Now, given the following types:

lib Lib
  union Foo
    x : Float32
  end

  struct Bar
    x : Float64
    y : Foo
  end
end

Crystal will produce the following LLVM IR:

%"struct.Lib::Bar" = type { double, %"union.Lib::Foo" }
%"union.Lib::Foo" = type { [1 x i32] }

So Lib::Bar, when flattened, consists of a Float64 field and an Int32 field according to LLVM, and therefore would be passed directly. But this is incorrect; since the union inhibits flattening, and the whole struct exceeds twice XLEN in size, it must be passed indirectly. LLVM::ABI will fail to handle this if it only deals with LLVM::Types. We could circumvent this by assuming all LLVM types whose name starts with union. are C unions, but this is Crystal's convention, not LLVM's, and besides it is still impossible to recover the original variant types in case a calling convention requires this information.

The upstream Rust code uses its own type to represent the C function type, instead of what would correspond to a wrapper over LLVMTypeRef. We should follow suit, but this is only possible by first making LLVM::ABI deprecated and then copying the whole hierarchy into the Crystal namespace.

[HertzDevil]

For the record, C APIs in the wild rarely pass unions by value, but this absolutely happens in the Win32 API. See also #12550 (comment) and libffi/libffi#33 (comment)