BO3 shaders, techsets, materials and adding them to maps

Shaders come in different parts and files which are executed in different stages of the rendering process. Pixel (ps) and Vertex (vs) shaders are the most common and the ones we will mostly be dealing with. So, you will want a (shader name)_vs.hlsl and a (shader name)_ps.hlsl. We will get into how you can get the boilerplate and base stuff for BO3 in the decompilation tutorial so you have something to start with and modify. Just know that you will want to decompile either or both a ps and vs file to start with when the time comes.

Shaders have different inputs and outputs and resources/structs depending on the passes and other techset variables that are defined.

Techsets tell the game what resources to provide the shader, what passes to render the shader on as well as additional data. Each techset with 'Globals()' is its own Material Type which can be viewed in APE by creating a material asset and then going to the correct Material Category. GDT's directly interact with techsets; Techsets define what textures slots (e.g. normal map, specular map, etc.) and properties (tint, normal intensity, etc.) are available while the GDT defines which textures and properties to send to the shaders. Techsets are found in the root/share/raw/techsetdefs_stable and root/share/raw/techsetdefs_stable_toolsgfx directories. Similar to the shader directories techsets are supposed to have different versions for both the editor and the game. Unlike the shaders, I think you need to have a copy of the same techset with the same contents in both directories to ensure everything works properly.

Techsets have some preprocessor directives. For an example things placed inside the directive #if TOOLSGFX != "1" will not be used in the mod tools but will be used in-game

In techsets, Techniques are used to determine which shaders to use during certain render passes, although to me a lot of the technique names are vague and unclear and I do not know when they are used. In the geometry/lit.techsetdef, I think that the gbuffer technique is the last pass before the material/model is rendered:

Technique( "gbuffer" )
{
    defines = "BASE_TEXTURES", "USE_COLOR_TINT"
    state = "gbuffer opaque"
    source = "gbuffer_lit.hlsl"
    vs = "vs_generic"
    ps = "ps_generic"
}

I believe what this does is

• Sets the preprocessor directives "BASE_TEXTURES", "USE_COLOR_TINT" on the actual shader before it compiles
• Says to use gbuffer_lit_vs.hlsl and gbuffer_lit_ps.hlsl as the shaders
• Uses generic/base inputs and outputs

Another way this could be done (which I use) is like:

Technique( "gbuffer" )
{
    defines = "BASE_TEXTURES", "USE_COLOR_TINT"
    state = "gbuffer opaque"
    // You can set the parameter to be the i/o type, generic is default e.g. VertexShader("vs_generic")
    vs = VertexShader()
    {
        source = "gbuffer_custom_lit_vs.hlsl"
    }
    ps = PixelShader()
    {
        // Say you didn't want to use a custom pixel shader, just do `source = "gbuffer_lit.hlsl"` in here
        source = "gbuffer_custom_lit_ps.hlsl"
    }
}

To get a test material set up I recommend copying the geometry/lit.techsetdef and decompiling and modifying gbuffer_lit_vs.hlsl and gbuffer_lit_ps.hlsl and then making the above changes to the techset. You do not need the full path of the shader, if it is in the correct directory it will be detected by the linker.

User defined Shader Globals/constants

Shader globals are constant values set by the table in a material's GDT and defined by the techset. I'm not sure what happens if you add custom keys to the GDT but, most of the keys in a basic GDT file with one material should be available. Some examples of keys or properties are like "cg00_w" or "glossRangeMin".

The keys and values in the GDT are used by the techset to create the constant values sent to the shader. For an example, if you want to have access to glossRangeMin and glossRangeMax in the shader you could do this in your techset:

float2( "glossRange" )
{
    x = <glossRangeMin>
    y = <glossRangeMax>
}

now if you define a variable float2 glossRange; in the root scope of the shader, you can then do glossRange.x to access glossRangeMin and glossRange.y to access glossRangeMax.

Note: This works with textures and samplers too but, you need the proper bindings on the shader end e.g.:

SamplerState colorSampler : register(s1);
Texture2D<float4> colorMap : register(t0);

If you want to be able to edit your constants in APE, then you need to define a tweak like this:

float2( "glossRange" )
{
    x = <glossRangeMin>
    y = <glossRangeMax>
    tweak = Tweak()
    {
        category  = "Specular and Gloss"
        title     = "Gloss Range"
        sortindex = "45"
     }
}

APE will automatically register any constants you have with the tweak and select the correct types. To see more about what you can do with the constants look at some of the pre-made techsets.