Does "fp16_run = true" allow users of nvidia gpus to use the tensor cores? (when supported by the gpu)

[vertexgamer]

Does "fp16_run = true" allow users of nvidia gpus to use the tensor cores? (when supported by the gpu)

[34j]

I have tried, but have not seen much improvement in speed, and would rather benefit from the reduction in VRAM usage. Looking for additional information.

[vertexgamer]

Ok thx, nice to know.
I will leave the discussion open in case more info is available

[quantumbalance]

how do i even get this repo to use my 3090?!?

INFO: GPU available: False, used: False
INFO [17:43:27] GPU available: False, used: False rank_zero.py:48
INFO: TPU available: False, using: 0 TPU cores
INFO [17:43:27] TPU available: False, using: 0 TPU cores rank_zero.py:48
INFO: IPU available: False, using: 0 IPUs
INFO [17:43:27] IPU available: False, using: 0 IPUs rank_zero.py:48
INFO: HPU available: False, using: 0 HPUs
INFO [17:43:27] HPU available: False, using: 0 HPUs

[vertexgamer]

mmmm it should works just fine. Check this tutorial to see if you missed any step https://www.youtube.com/watch?v=tZn0lcGO5OQ

[upColor]

@vertexgamer @34j

Well, now our dream might come true, because soon Nvidia is going to release a new GPU driver, which uses RTX tensor cores to boost generative AI performance 2X:

According to Nvidia, we have to thank Microsoft to recently release new tool which can convert Pytorch models to ONNX, which enables RTX tensor cores.

https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvidia-ai-windows-rtx/
Quote:

Microsoft released the Microsoft Olive toolchain for optimization and conversion of PyTorch models to ONNX, enabling developers to automatically tap into GPU hardware acceleration such as RTX Tensor Cores. Developers can optimize models via Olive and ONNX, and deploy Tensor Core-accelerated models to PC or cloud. Microsoft continues to invest in making PyTorch and related tools and frameworks work seamlessly with WSL to provide the best AI model development experience.