"use_safetensors" not enforced with "local_files_only", loads bin file

[troy-baker-aumni]

System Info

• transformers version: 4.41.2
• Platform: macOS-14.5-arm64-arm-64bit
• Python version: 3.11.7
• Huggingface_hub version: 0.23.1
• Safetensors version: 0.4.3
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.3.0 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. clone model repo (flan-t5-large)
2. remove model.safetensors from target directory
3. run code

from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained(
    pretrained_model_name_or_path="path/to/flan-t5-large",
    local_files_only=True,
    use_safetensors=True
)

4. observe from_pretrained falls back to the pytorch_model.bin file and loads the model

Expected behavior

from_pretrained function should raise an error since no safetensors file is present in the target. If this is the target behavior, perhaps this is a feature request rather than a bug, but I find this to be misleading at best or unsafe at worst. Additionally, it is unclear how to manually direct the from_pretrained function to only load models with a specific name and extension, or to do so with a file loaded using the safetensors library. It is unclear to the user how a model file is chosen from the dir, so at the very minimum, a user should be able to declare the safetensors file themselves as the model file.