Why can't LayerNorm pass the result verification

[GeezYuven]

When I use the command MIOpenDriver layernorm --input 2x256x64x128 -o 0 -m 0 -F 1 -V 1 -i 1 on the gfx906 platform, the LayerNorm operation fails during the verification stage. However, when I change the command type to layernormfp16 or layernormbfp16, it passes the verification stage. Additionally, for the parameters that fail verification, when I change the -o parameter to 3, it passes the verification. What could be the reason for this?

[ruizhe-ops]

The verification failure arises from FP32’s sensitivity to numerical instability in MIOpen’s LayerNorm implementation for specific configurations. Lower-precision types (FP16/BF16) or parameter adjustments (-o 3) mask these issues by altering computation paths or relaxing tolerances.

[GeezYuven]

Is the solution you mentioned above also applicable to AddLayerNorm? Additionally, T5LayerNorm also encounters verification failures, but this operator does not have the -o parameter. What causes these issues?

[ruizhe-ops]

The solutions mentioned for LayerNorm might partially apply to AddLayerNorm, but there are nuances to consider. For AddLayerNorm, since it combines addition and normalization, numerical instability could stem from both operations.
For T5LayerNorm, which lacks the -o parameter, the verification failures likely arise from:

1. T5LayerNorm often uses a simplified variant (e.g., no bias term, fixed scaling) that might amplify FP32 sensitivity.
2. Without optimization flags, the implementation might rely on algorithms prone to FP32 instability (e.g., non-vectorized operations).
3. The verification tool may use stricter tolerances for FP32, exposing minor discrepancies in T5’s unique normalization logic.
   In both cases, numerical instability in FP32 remains the root cause, but fixes depend on implementation details and available configuration options.

[GeezYuven]

From this perspective, it seems that the released LayerNorm operators still need to be fixed, right? Additionally, in your first response, you mentioned 'the verification failures are due to FP32 being more sensitive to numerical instability in certain configurations of MIOpen's LayerNorm implementation.' Could you specify which particular configurations you are referring to?

[GeezYuven]

It seems to be the impact of verification accuracy. Modifying the verification type of the operator in the driver to double, and using the double type to store intermediate results in MIOpenLayerNorm, can solve the issue.