Decompiling and fixing the base game shaders

These instructions are for the decompiler found here: https://github.com/etnlGD/HLSLDecompiler. Other decompilers in the future could possibly make this process easier, but right now shader decompilers are really low level and barely give any readable and working output. This one will produce code that is at least 99% functional but is no more readable than asm.

This will work on all basic vertex and pixel shaders (Like the ones in geometry), I have not yet really messed around with the fx or compute shaders. Note that there are things similar to the errors described below which are not quite the same as what this guide shows how to fix. For any variations of things saying "manual fix required" check out Microsoft's docs. If you still cannot figure it out, leave an issue and I will see if I can fix it and extend the guide.

Step 1: Decompiling

Decompile and disassemble the shader. To do this run ./cmd_Decompiler.exe -D <compiled shader> and then ./cmd_Decompiler.exe -d <compiled shader>
You should now have the files .hlsl and .asm

Step 2: Initial cleanup

Inside the .hlsl file, move the globals out of the cbuffer structure. All you need to do is delete the enclosing structure definition e.g.

cbuffer _Globals : register(b0)  <--
{   <--                |--------------|  <--
    float4 flagParams  : packoffset(c0);
    float4 colorObjMin : packoffset(c1);
    ..... yada yada ....
    ....
}   <--

and remove the : packoffset(c0)s at the end of the variable definitions.

IGNORE STEPS 3 AND 4 IF THE .hlsl FILE DOES NOT HAVE ANY 'ld_structured_indexable' COMMANDS

Step 3: Adding and registering missing structs

Add missing indexed structs to the .hlsl file. This is done by first going to the 'Resource Bindings' section of the .asm file.
Then for each row where the type is 'struct', search for the part of the document where it says 'Resource bind info for ' and copy no more than from the struct until the last '}'. Do not copy anything like '$Element' or the brackets enclosing the entire struct!
Paste it into the .hlsl file before where the 'StructuredBuffer's are defined.

WARNING: Do not forget to add a closing ';' right after the last '}' in the struct, it should look similar to the pre-defined structs in the .hlsl file.

Then make sure that there is a line similar to:
StructuredBuffer<GpuShaderConstantSet> shaderConstantSetBuffer : register(t5);
for each of the structs right before the main() function of the shader.
The name inside of the <> must be the name right after struct, the name after the <> must be the name found in the 'Resource Bindings' section and the value for 't4' must be the 'HLSL Bind' in the bindings.
Repeat for every struct defined in the 'Resource Bindings' section.

You can think of these structs as arrays of objects.

Step 4: Manually fixing ld_structured_indexable

Now we need the fix the ld_structured_indexable commands. They will look something like this with some variable assignments afterwards:

// Known bad code for instruction (needs manual fix):
ld_structured_indexable(structured_buffer, stride=64)(mixed,mixed,mixed,mixed) r6.xyzw, r2.x, l(16), t5.xyzw
r6.x = modelInstanceBuffer[]..swiz;
r6.y = modelInstanceBuffer[]..swiz;
r6.z = modelInstanceBuffer[]..swiz;
r6.w = modelInstanceBuffer[]..swiz;

All this command does is access the structs we just copied over and assigns some variables with the values in the struct.
The important part is the 4 comma seperated arguments at the end e.g. r6.xyzw, r2.x, l(16), t5.xyzw:

• r6.xyzw is the destination/variable being assigned
• r2.x is the index of the 'object' in the structs
• l(16) is the index of the field in the struct
• t5.xyzw is the struct 'type' or binding we want to access and which parts we want

First go back to where you defined the 'StructuredBuffer's and grab the correct name for the binding. For an example if you had
StructuredBuffer<GpuShaderConstantSet> shaderConstantSetBuffer : register(t5);
then the name you want is shaderConstantSetBuffer.

Note: the names below the ld_structured command are frequently incorrect (in this case 'modelInstanceBuffer').

Now you need to find which field is being accessed; do this by going to the struct that matches the name/binding and (in this example) find the offset that matches the l(16):
float4 scriptVector1; // Offset: 16
Now that we know at offset 16 the field is 'scriptVector1' we just re-write the command to be (be sure to get rid of the old stuff):
r6.xyzw = shaderConstantSetBuffer[r2.x].scriptVector1.xyzw;

Note: In situations where the field being accessed is also a matrix or an array you will need to know a little about the datatype byte-sizes to manually calculate the offsets. A common example is:
float4 mat[3]; // Offset: 0
where l(0) would be field.mat[0], l(16) -> field.mat[1], l(32) -> field.mat[2] and so on...

---

Step 5: Fixing the cmps

Replace any 'cmp' calls and any flow control commands with valid versions. For an example:

r2.x = cmp(v5.x != 0.000000);
    if (r2.x != 0) {

is replaced with: if (v5.x != 0.000000) NOTE: operations such as r3.xy = cmp(r0.zx >= r1.yx) are evaluated for each component!

Step 6: Compiling and fixing errors

Try to compile the shader and see if there are any errors (See techset tutorial to compile). Very rarely are there still syntax errors or commands that were not translated over from asm but, if you run into that refer to https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx9-graphics-reference-asm to help you find the equivalent HLSL commands.
If you run into an error which occures on a line assigning/accessing the input or output variables make sure that the datatype is large enough e.g. v1.xyzw will not work if v1 is only a float2, so change it to a float4.

Leave an issue if you still can't get it to work. Chances are I forgot something or have not ran into some problems yet.