Pixel art anti aliasing

[alex5nader]

Objective

Better anti aliasing for upscaled pixel art.

Pixel art often uses nearest-neighbor sampling when upscaling. This works well for 2D environments with an integer scale factor, but otherwise leads to aliasing artifacts.

Solution

I'm proposing an implementation of the solution described in this video: https://youtu.be/d6tp43wZqps

The video explains it well and includes graphics to better illustrate why the process works, but here's a simplified summary if you'd prefer:

Simplified explanation

1. A linear sample at a given UV can be thought of as the average color within a texel-sized box centered at that UV.
2. Ideally, pixel art would instead use a pixel-sized box at each UV.
3. It is possible to scale the pixel-sized box to texel-size while preserving its average color. This moves its center.
4. Because the average color is the same, a linear sample at the new center can provide the correct color for the original pixel.
5. If the sprite is rotated, a pixel-sized box is no longer a rectangle, so the AABB of the box is used instead.
6. The AABB is not a perfect match, but inaccuracy can be decreased by weighing the center of the box more than the edges when finding the average color.

I've implemented this in my fork here and have a sample project using it here. Because of the following questions, though, I don't think it's ready for a PR quite yet.

Unanswered questions

For the following questions, I'm not sure which approach is best. In the current implementation, I arbitrarily chose to add pbr_functions::texture_sample and use a shader define PIXEL_ART_ANTI_ALIASING. This can be easily changed depending on the discussion.

When to check if enabled

To actually implement this strategy within Bevy, all relevant instances of texture sampling must be replaced with this function. I'll call it pixel_art_texture_sample. There are a few ways to approach this:

1. Add another new function (texture_sample) that checks if this feature is enabled. Use this to replace all relevant calls to a built-in texture sample.

Pseudocode

fn texture_sample(texture, sampler, uv) {
  if enabled {
    return pixel_art_texture_sample(texture, sampler, uv);
  else {
    return textureSample(texture, sampler, uv);
  }
}

2. Every time a texture is sampled, check if this feature is enabled, either using the built-in or pixel_art_texture_sample.

Example

From pbr.wgsl :

if ((pbr_bindings::material.flags & pbr_types::STANDARD_MATERIAL_FLAGS_BASE_COLOR_TEXTURE_BIT) != 0u) {
    if enabled {
        output_color = output_color * pixel_art_texture_sample(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv, view.mip_bias);
    } else {
        output_color = output_color * textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv, view.mip_bias);
    }
}

(Most of the builtin texture samples use a mip bias. This can be compensated for as explained here)

How to check if enabled

There are also two approaches for the enabled check itself. We can either add a new shader define, or add a new flag to the standard material's flags.

What is a "relevant texture sample"?

There's also the question of what texture samples should use the new method. Bevy's PBR shaders have many different textures, and any time one of them is sampled, we have to choose whether this new behaviour is necessary.

Obviously, base color texture should use it. Using it for the normal map and emissive textures also seems useful. The metallic/roughness and occlusion textures don't seem particularly likely to be provided with pixel art, but if they were, I think using this technique would produce more consistent results.

There are more texture samples than these in the PBR shaders, though. A grep shows the following files:

• src/environment_map/environment_map.wgsl
• src/render/parallax_mapping.wgsl
• src/render/shadow_sampling.wgsl
• src/render/shadows.wgsl
• src/ssao/gtao.wgsl

I don't really know enough about PBR to judge if these should also use this technique. But, since this technique shouldn't change the overall shape of the texture, perhaps shadows don't need to use it.

Possible issues

• Because this is implementing a sample function within the shader itself (as opposed to the builtin linear and nearest functions), shader authors have to remember this feature exists and not just use the builtin sample functions.

[dmyyy]

Relevant: #9349

[alex5nader]

I actually initially looked into both FilterMode and ImageSampler, but after looking into the WebGPU spec I'm pretty sure it's not possible to instruct the GPU to use anything other than linear or nearest (hence the need to replace calls to a built-in textureSample).

I suppose we could add a pixel art option to one of these, but then they're no longer equivalent to the wgpu types and would have to be special cased inside Bevy somehow.

[mnemotic]

Likewise relevant: #1856 (comment)

[alex5nader]

Yep, this technique solves the same issue as handmade hero and the shadertoy. It is more complicated though, to also help with 3d environments where sprites aren't just scaled by a non-integer amount, but may also be rotated away from the camera.

[mnemotic]

For what it counts, I'd love to have this in Bevy.

[dmyyy]

I want this in Bevy as well - however with 0.12 on the horizon I think people are going to be busy preparing for the new release + reviewing the mountain of in-flight rendering features.

@alex5nader I think it would make sense to ask about this in the #rendering-dev channel on discord if you don't get a response by then.

[alex5nader]

Yeah, my timing is pretty bad haha. I'm perfectly fine waiting to let people prepare for the release, but figured I'd go ahead and make my work public in the meantime. Once things have calmed down I do plan to reach out.

[agausmann]

Unanswered questions are perfectly fine for a PR! Opening a PR would be the best way to make sure your questions don't get accidentally overlooked by the project maintainers. If you don't want it merged yet, just mark as draft.