Static Linking

[daxpedda]

Is there something preventing statically linking the DirectXShaderCompiler?

My understanding is that hassle doesn't actually build the compiler, but only provides an FFI interface, so statically linking would require building it as a library.

I've also read that shader validation is performed by the GPU/Driver, so is that why getting around the official dll's is not possible?

[MarijnS95]

See the dynamic-link branch. That could be extended to statistically link if DXC is built as a static library too (not sure if they support that), but dxil.dll for validation is closed source and only provided as shared/dynamic library.

[daxpedda]

Right, that would definitely be the most straightforward way to improve the situation.
Is there something holding this back?
Is there any intention by hassle to package or download the dll's during buildtime to make this as straightforward as possible for developers?
Apologies, I didn't deal with dynamic linking in a long-time, my questions don't make any sense.
I guess without static linking we can't avoid having to ship them with the binary.

but dxil.dll for validation is closed source and only provided as shared/dynamic library

Ah I see, that explains the issue.

[daxpedda]

Thanks for the explanation!
I guess as long as validation is closed source we will have to rely on the dll Microsoft provides.

[MarijnS95]

Right, that would definitely be the most straightforward way to improve the situation.

Not sure I follow: runtime linking (and dynamic linking) is rather straightforward if the final application provides the library, not in the least because that's the only thing distributed by Microsoft.

Is there something holding this back?

Distribution (and perhaps even support) to build (lib)dxcompiler.so/dll as static library (and all the system libraries it itself depends on)?

Is there any intention by hassle to package or download the dll's during buildtime to make this as straightforward as possible for developers?

I have no intent to make dynamic linking (as opposed to runtime linking) the default but can definitely submit my efforts on the dynamic-link branch to main after guarding it behind a feature flag. Static linking is a different beast though, and I definitely don't want to download anything during the build process.

Apologies, I didn't deal with dynamic linking in a long-time, my questions don't make any sense. I guess without static linking we can't avoid having to ship them with the binary.

As long as we're clear on runtime vs dynamic vs static linking :)

but dxil.dll for validation is closed source and only provided as shared/dynamic library

Ah I see, that explains the issue.

We've got our own implementation for dxil.dll though, but I am not sure if it "is enough" for modern drivers:

hassle-rs/src/fake_sign/mod.rs

Line 41 in f8c26ae

pub fn fake_sign_dxil_in_place(dxil: &mut [u8]) -> bool {

[daxpedda]

Right, that would definitely be the most straightforward way to improve the situation.

Not sure I follow: runtime linking (and dynamic linking) is rather straightforward if the final application provides the library, not in the least because that's the only thing distributed by Microsoft.

Is there something holding this back?

Distribution (and perhaps even support) to build (lib)dxcompiler.so/dll as static library (and all the system libraries it itself depends on)?

Is there any intention by hassle to package or download the dll's during buildtime to make this as straightforward as possible for developers?

I have no intent to make dynamic linking (as opposed to runtime linking) the default but can definitely submit my efforts on the dynamic-link branch to main after guarding it behind a feature flag. Static linking is a different beast though, and I definitely don't want to download anything during the build process.

Apologies, I didn't deal with dynamic linking in a long-time, my questions don't make any sense. I guess without static linking we can't avoid having to ship them with the binary.

As long as we're clear on runtime vs dynamic vs static linking :)

Yeah, sry about the confusion about all of this.
Personally I have no interest in dynamic linking, my interest is purely avoiding to ship the dll's with the binary.

but dxil.dll for validation is closed source and only provided as shared/dynamic library

Ah I see, that explains the issue.

We've got our own implementation for dxil.dll though, but I am not sure if it "is enough" for modern drivers:

hassle-rs/src/fake_sign/mod.rs

Line 41 in f8c26ae

pub fn fake_sign_dxil_in_place(dxil: &mut [u8]) -> bool {

This is very interesting. So potentially a use-case for static linking is still on the table, but a lot of work still, obviously.

I wonder what naga plans to do here when it implements DXBC, but if this method is reliable then maybe they can avoid having to sign shaders with dxil.dll.

[daxpedda]

Just checked MESA, they also just rely on dxil.dll:
https://gitlab.freedesktop.org/mesa/mesa/-/blob/8491b1fd5e7c97edc104a3ac3b8ece5464e652b0/src/microsoft/spirv_to_dxil/spirv2dxil.c#L141-167

[MarijnS95]

Yeah, sry about the confusion about all of this. Personally I have no interest in dynamic linking, my interest is purely avoiding to ship the dll's with the binary.

Ack, yes, that's a valid "does not work out of the box otherwise" justification.

but dxil.dll for validation is closed source and only provided as shared/dynamic library

Ah I see, that explains the issue.

We've got our own implementation for dxil.dll though, but I am not sure if it "is enough" for modern drivers:

hassle-rs/src/fake_sign/mod.rs

Line 41 in f8c26ae

pub fn fake_sign_dxil_in_place(dxil: &mut [u8]) -> bool {

This is very interesting. So potentially a use-case for static linking is still on the table, but a lot of work still, obviously.

Not a lot if DXC itself supports being compiled statically, as this implementation already exists and lives purely in hassle-rs' Rust code.

I wonder what naga plans to do here when it implements DXBC, but if this method is reliable then maybe they can avoid having to sign shaders with dxil.dll.

Yup, keep us posted; we use this method internally at Traverse for quite some time already.

---

Just checked MESA, they also just rely on dxil.dll: https://gitlab.freedesktop.org/mesa/mesa/-/blob/8491b1fd5e7c97edc104a3ac3b8ece5464e652b0/src/microsoft/spirv_to_dxil/spirv2dxil.c#L141-167

That's just Microsoft's tool to run on top of DirectX though, they have no reason to expose the internals of this implementation themselves if the systems they run on have access to dxil.dll 😉 - this shouldn't be used in any other way by mesa.

[MarijnS95]

@daxpedda I just started looking into this for #62 again, and it was actually quite trivial to modify DXC to build a static library, just drop SHARED here (and BUILD_SHARED_LIBS has to be kept at default OFF too).

https://github.com/microsoft/DirectXShaderCompiler/blob/667fb773cc2fb40bc31dba6d53e5230a196838ff/tools/clang/tools/dxcompiler/CMakeLists.txt#L127

Then unfortunately CMake emits a link "config" if we were to "install" this project, telling you all the many libraries to link rather than providing a single rearchived shared library. But once we list them all to be linked, and make sure DllMain stays around so that it'll be called before our executable opens (and if you call it yourself, it'll init twice causing havoc), you can actually use hassle-rs with static-linked DXC!:

dynamic-link...static-link

This isn't going to fit in my time-plan at all, but to approach this I would have to:

1. Set the dynamic-link branch up for cleaning (move it behind a feature flag) and merging, as a step-up to this;
2. Reply in How can I build dxil and dxcompiler as a static library? microsoft/DirectXShaderCompiler#4766 how easy it is to achieve this, and discuss if/how we can proceed to make it more convenient:
   • rearchive all deps into a single ar file;
   • provide a config option to enable this;
   • figure out a clean-er way for DllMain to be called exactly once (it got deadstripped on my end unless called manually, then it's called twice...);
3. Land the static linking support as another crate feature here.

[daxpedda]

That's awesome, thank you for the update!

I guess the work necessary will need some time (microsoft/DirectXShaderCompiler#4766), but it seems to me that at the end there won't be a way around a new crate feature that will compile the DirectXShaderCompiler with the newly introduced option, which would be a significant burden on compile time. Or is there?

In any case, that's good news, and I'm looking forward to what happens in microsoft/DirectXShaderCompiler#4766.

This isn't going to fit in my time-plan at all

I have to say that as eager as I am to look more into this and spend some time to contribute, but I would rather die then touch CMake again, so not gonna touch this. Happy to contribute to hassle though to help this along, which probably won't be much.

[MarijnS95]

but it seems to me that at the end there won't be a way around a new crate feature that will compile the DirectXShaderCompiler with the newly introduced option, which would be a significant burden on compile time. Or is there?

As I explained to @Makogan in #62 we have many ways of going at this:

• Dynamic library:
  • Provided by the crate/application author to its users together with their compiled Rust project;
  • Or expected to be available on the system;
  • Or somehow bundled and provided within hassle-rs (but don't think we'll do this);
  • Note that there's also a difference between dynamic linking and runtime loading. While both operate on the same binary (on Linux, on Windows you also need a special .lib file for the dynamic linking stage), they have different ways of finding libdxcompiler.so at runtime;
• Static library:
  • Provided by the users of hassle-rs when their project is compiled;
  • maybe pre-packaged in the hassle-rs crate by us, but we typically swap libraries quite often and rather keep it detached from hassle-rs releases (also to not conflate the crate, it is literally massive);
  • Compile DXC on the spot with a hassle-rs build script (very unlikely to happen as this is tricky to set up and we have a bit of an allergy towards cmake, especially within Rust crates).

Effectively my plan is to not rule out any of these cases but support them all via some simple feature flags (some will have to be mutually exclusive I think) and/or guidelines (e.g. set the right library search path via println!("cargo:rustc-link-search={}") to make linking successful).

TL;DR: No, I have no plans/time at all to compile DXC on the spot within the Rust crate. Will static-link against a precompiled / pre-provided libdxcompiler.a exclusively.

And again, the main downside of prebuilts provided by hassle-rs is that we have to make sure there are no erratic library dependencies (stdc++ or ncurses5 to name a few) that can only be satisfied on a handful of machines, and every target OS and architecture needs a separate blob to once again explode crate sizes.

I have to say that as eager as I am to look more into this and spend some time to contribute

Thanks!

I would rather die then touch CMake again, so not gonna touch this

Don't worry, I already have the proof-of-concept oneliner change made, and hiding it behind an option() will be trivial if needed at all. Getting such a feature accepted upstream will be the hardest, not in the least because of response times and having to justify a change. Making one big AR blob will be secondary, but nothing too complicated.

Happy to contribute to hassle though to help this along, which probably won't be much.

Thanks again, but here also the branches and changes have already been made and just need to be cleaned after the features are switchable via feature flags.

[daxpedda]

Thank you for all the detailed responses, it really clears up a lot of misconceptions I still have.

• Or expected to be available on the system;

I briefly looked into this for Windows, but as far as I understand that would require users to install the Windows SDK, which I didn't actually confirm contains these libraries, but I confirmed that a clean Windows 10 install with all drivers doesn't include them.

• Provided by the users of hassle-rs when their project is compiled;

That sounds totally reasonable to me.

• maybe pre-packaged in the hassle-rs crate by us, but we typically swap libraries quite often and rather keep it detached from hassle-rs releases (also to not conflate the crate, it is literally massive);

I agree that this is probably not a good idea because of the size.

• Compile DXC on the spot with a hassle-rs build script (very unlikely to happen as this is tricky to set up and we have a bit of an allergy towards cmake, especially within Rust crates).

Considering option one is preferable (to us) anyway, not doing this is very reasonable.

Making one big AR blob will be secondary, but nothing too complicated.

I was more concerned about distribution. My hope is, that after getting generating the AR blob upstream, Microsoft will actually distribute it alongside their releases ... otherwise we will have to build it ourselves, on some Windows machine, or get cross-compiling going ... which I'm not looking forward to. Compiling on CI might also turn out to be frustrating, but I didn't try so I should stop rambling now.

[MarijnS95]

• Or expected to be available on the system;

I briefly looked into this for Windows, but as far as I understand that would require users to install the Windows SDK, which I didn't actually confirm contains these libraries, but I confirmed that a clean Windows 10 install with all drivers doesn't include them.

This is more a case of the user providing it themselves, either in PWD, or via an env var (that may have been installed by another tool, the Vulkan SDK for example includes this library and I think also sets up the env vars such that it can be found). But on Linux, yes there are distros that provide a package OOTB!

Making one big AR blob will be secondary, but nothing too complicated.

I was more concerned about distribution. My hope is, that after getting generating the AR blob upstream, Microsoft will actually distribute it alongside their releases ... otherwise we will have to build it ourselves, on some Windows machine, or get cross-compiling going ... which I'm not looking forward to. Compiling on CI might also turn out to be frustrating, but I didn't try so I should stop rambling now.

Fair enough, might be a thing to take up with Microsoft, who after all provides the rest of DirectX pre-installed on Windows 😬 - but then again all games come with precompiled shaders for obvious reasons, so it is really a development tool whereas GPU drivers are ... obviously ... for the runtime 😬

[daxpedda]

This is more a case of the user providing it themselves, either in PWD, or via an env var (that may have been installed by another tool, the Vulkan SDK for example includes this library and I think also sets up the env vars such that it can be found).

Just to make sure, there are no typical consumer tools that one can expect a regular gamer to have installed that includes those libraries?

but then again all games come with precompiled shaders for obvious reasons, so it is really a development tool whereas GPU drivers are ... obviously ... for the runtime 😬

I'm all for pre-compiled shaders, but unfortunately wgpu doesn't currently have a good story around that, see gfx-rs/wgpu#3103. I guess if they did I wouldn't be here, then just pre-compiling all shaders on Linux for DX12 targets would be the way to go indeed.

EDIT:

Fair enough, might be a thing to take up with Microsoft, who after all provides the rest of DirectX pre-installed on Windows 😬

After reading your response again I realized there might be a misunderstanding: I meant released as part of their DirectXShaderCompiler releases.

[MarijnS95]

Just to make sure, there are no typical consumer tools that one can expect a regular gamer to have installed that includes those libraries?

Strongly doubt it, but I'm not a Windows user nor a game developer to tell :)

I'm all for pre-compiled shaders, but unfortunately wgpu doesn't currently have a good story around that, see gfx-rs/wgpu#3103. I guess if they did I wouldn't be here, then just pre-compiling all shaders on Linux for DX12 targets would be the way to go indeed.

(Without reading context in that issue) this effort could still allow you to more easily embed a hassle-rs shader compilation invocation in a buildstep, or in a minimal Rust tool to compile your shaders up front (we have something like that for a few boilerplate critical shaders, end result goes into include_bytes!() 🎉). You'll still need DXC in some form on your system and/or close to the crate.

EDIT:

Fair enough, might be a thing to take up with Microsoft, who after all provides the rest of DirectX pre-installed on Windows grimacing

After reading your response again I realized there might be a misunderstanding: I meant released as part of their DirectXShaderCompiler releases.

The static library specifically? Yes, that's the desired outcome (and CMake should be able to do it OOTB) just like the .dll+.pdb+.lib on Windows.

[daxpedda]

(Without reading context in that issue) this effort could still allow you to more easily embed a hassle-rs shader compilation invocation in a buildstep, or in a minimal Rust tool to compile your shaders up front (we have something like that for a few boilerplate critical shaders, end result goes into include_bytes!() tada). You'll still need DXC in some form on your system and/or close to the crate.

Good point! I guess I didn't think it through yet 😅

The static library specifically? Yes, that's the desired outcome (and CMake should be able to do it OOTB) just like the .dll+.pdb+.lib on Windows.

Our goals are aligned then! So this now just needs time and work, looking forward 🚀!

[JMS55]

https://github.com/hexops/mach-dxcompiler figured out how to do statically linked DXC builds, without cmake.