We got different accuracy for identical onnx models. Models have identical graphs and parameters. Nodes of both models are topologically sorted but order of nodes in files is different.

Steps to reproduce bag:

1. Download models and weights (~23GB) to onnx_models dir.

2. Build engine for model_1.onnx:

   python build.py --onnx_model onnx_models/model_1.onnx --output engines/model_1.onnx.engine

3. Build engine for model_2.onnx:

   python build.py --onnx_model onnx_models/model_2.onnx --output engines/model_2.onnx.engine

4. Validate model_1

   python validate.py --engine engines/model_1.onnx.engine --max_batches 1000

5. Validate model_2

   python validate.py --engine engines/model_2.onnx.engine --max_batches 1000

Result with TRT 8.4.1.5 + RTX3080ti:

build	accuracy
model_1	57.04%
model_2	61.74%

Expected result:

Accuracy of model_1 should be equal accuracy of model_2.