arch error with tokenizers dependency

[sclausen]

Somehow the tokenizers library isn't built for arm64.
I don't know if this log output gets swallowed, but I don't see anything getting compiled with rust.
I'm building on mac 12.5.

Am I missing something?

RuntimeError: Failed to import transformers.models.auto because of the following error (look up to see its traceback):
Failed to import transformers.tokenization_utils because of the following error (look up to see its traceback):
dlopen(/Users/username/miniforge3/envs/ldm/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so, 0x0002): tried: '/usr/local/opt/openssl/lib/tokenizers.cpython-310-darwin.so' (no such file), '/tokenizers.cpython-310-darwin.so' (no such file), '/Users/username/miniforge3/envs/ldm/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

PS: Same error regardless of miniforge3, miniconda3 or anaconda3 (the folder changes off course 😉 )

[evancoons22]

I am having the same issue... I already had rust installed as well.

[magnusviri]

Please use this fork/branch.