SNOW-1825673: root can not be checked for existence of SNOWFLAKE_HOME due to permissions

[mikita-sakalouski]

Python version

3.11

Operating system and processor architecture

Linux-5.15.0-1071-aws-aarch64-with-glibc2.35

Installed packages

annotated-types==0.7.0
asn1crypto==1.5.1
asttokens==2.0.5
astunparse==1.6.3
azure-core==1.30.1
azure-storage-blob==12.19.1
azure-storage-file-datalake==12.14.0
backcall==0.2.0
black==23.3.0
blinker==1.4
boto3==1.34.39
botocore==1.34.39
brickflows==1.2.1
cachetools==5.3.3
certifi==2023.7.22
cffi==1.15.1
chardet==4.0.0
charset-normalizer==2.0.4
click==8.1.7
cloudpickle==2.2.1
comm==0.1.2
contourpy==1.0.5
cryptography==43.0.3
cycler==0.11.0
Cython==0.29.32
databricks-sdk==0.20.0
databricks-sql-connector==3.6.0
dbus-python==1.2.18
debugpy==1.6.7
decorator==5.1.1
delta-spark==3.2.1
distlib==0.3.8
distro==1.7.0
distro-info==1.1+ubuntu0.2
entrypoints==0.4
et_xmlfile==2.0.0
executing==0.8.3
facets-overview==1.1.1
filelock==3.13.4
fonttools==4.25.0
gitdb==4.0.11
GitPython==3.1.43
google-api-core==2.18.0
google-auth==2.29.0
google-cloud-core==2.4.1
google-cloud-storage==2.16.0
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.63.0
grpcio==1.60.0
grpcio-status==1.60.0
httplib2==0.20.2
idna==3.4
importlib-metadata==6.0.0
ipyflow-core==0.0.198
ipykernel==6.25.1
ipython==8.15.0
ipython-genutils==0.2.0
isodate==0.6.1
jedi==0.18.1
jeepney==0.7.1
Jinja2==3.1.2
jmespath==0.10.0
joblib==1.2.0
jsonpickle==4.0.0
jupyter_client==7.4.9
jupyter_core==5.3.0
keyring==23.5.0
kiwisolver==1.4.4
koheesio==0.9.0rc3
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lz4==4.3.3
MarkupSafe==3.0.2
matplotlib==3.7.2
matplotlib-inline==0.1.6
mlflow-skinny==2.11.4
more-itertools==8.10.0
multimethod==1.12
mypy-extensions==0.4.3
nest-asyncio==1.5.6
networkx==3.1
numpy==1.26.4
oauthlib==3.2.0
openpyxl==3.1.5
packaging==23.2
pandas==1.5.3
pandas-stubs==2.2.3.241009
parso==0.8.3
pathspec==0.10.3
patsy==0.5.3
pendulum==2.1.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.4.0
platformdirs==3.10.0
plotly==5.9.0
pluggy==1.5.0
prompt-toolkit==3.0.36
proto-plus==1.23.0
protobuf==4.24.1
psutil==5.9.0
psycopg2==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
py4j==0.10.9.7
pyarrow==14.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyccolo==0.0.52
pycparser==2.21
pydantic==2.10.1
pydantic_core==2.27.1
Pygments==2.15.1
PyGObject==3.42.1
PyJWT==2.3.0
pyodbc==4.0.38
pyOpenSSL==24.2.1
pyparsing==3.0.9
pyspark==3.5.3
python-apt==2.4.0+ubuntu3
python-dateutil==2.8.2
python-decouple==3.8
python-lsp-jsonrpc==1.1.1
pytz==2024.2
pytzdata==2020.1
PyYAML==6.0
pyzmq==23.2.0
requests==2.31.0
rsa==4.9
s3transfer==0.10.1
scikit-learn==1.3.0
scipy==1.11.1
seaborn==0.12.2
SecretStorage==3.3.1
six==1.16.0
smmap==5.0.1
snowflake-connector-python==3.12.3
sortedcontainers==2.4.0
sqlparse==0.5.0
ssh-import-id==5.11
stack-data==0.2.0
statsmodels==0.14.0
tenacity==8.2.2
threadpoolctl==2.2.0
thrift==0.20.0
tokenize-rt==4.2.1
tomli==2.1.0
tomlkit==0.13.2
tornado==6.3.2
traitlets==5.7.1
types-pytz==2024.2.0.20241003
typing_extensions==4.12.2
tzdata==2022.1
ujson==5.4.0
unattended-upgrades==0.1
urllib3==1.26.16
virtualenv==20.24.2
wadllib==1.3.6
wcwidth==0.2.5
zipp==3.11.0

What did you do?

Try to just perform import of connector `from snowflake import connector as snowflake_connector`, this will trigger the resolution of DIRS from `snowflake.connector.constants`.

And I'm facing the error: 

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/snowflake/connector/sf_dirs.py", line 37, in _resolve_platform_dirs
    if snowflake_home.exists():
       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1236, in exists
    self.stat()
  File "/usr/lib/python3.11/pathlib.py", line 1014, in stat
    return os.stat(self, follow_symlinks=follow_symlinks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/root/.snowflake'

What did you expect to see?

Correct handling of denied access to root folder

Can you set logging to DEBUG and collect the logs?

No response

[sfc-gh-sghosh]

Hello @mikita-sakalouski ,

Thanks for raising the issue. This is a configuration issue.

The error indicates the process is attempting to resolve SNOWFLAKE_HOME as /root/.snowflake , this can happen when
a) running via root user
b) restricted permission to /root directory

If the script runs under the root user, it will try to use /root/.snowflake. If the process lacks permissions to access this path (common in restricted environments), the PermissionError occur.

Also, please check the verified path for the connector
pip show snowflake-connector-python

So, please run as a non-root user with appropriate permissions or run via sudo command
Change the environment variable to redirect directories

os.environ["SNOWFLAKE_HOME"] = "/tmp/snowflake"

Regards,
Sujan

[gramakri]

I hit this issue when deploying superset. Superset tries to load snowflake-sqlalchemy and it fails because of permission denied with /root/.snowflake . In our deployment, superset runs as non-root user but it still tries to load /root/.snowflake anyway. The real question though is why a library requires some directory to be accessible ?

The workaround of setting SNOWFLAKE_HOME does work.

[mikita-sakalouski]

Agree, the directory check looks like in wrong place, especially taking into account that is resolved during import of connector and not during real access, i will prefer snowflake library to handle it correctly with two options:

• Resolve directory when it is needed and not during import
• fallback to tmp folder with using tmpfile module of Python if there is real need in dummy folder for just passing the resolution of directories

[gramakri]

Exactly. Superset is not even trying to create a connection and the load fails.

The relevant superset code is https://github.com/apache/superset/blob/dbcb473040b7e8ef1d41a81ba594387b66c04ebe/superset/db_engine_specs/__init__.py#L155 and that load() throws an exception of Permission Denied.

[sfc-gh-sghosh]

Hello @mikita-sakalouski @gramakri ,

Thanks for the update and glad to hear setting SNOWFLAKE_HOME does work.
Will dicuss with team about the feedback Snowflake Python Connector attempts to resolve platform directories (snowflake_home) during the import phase of the snowflake.connector module

Regards,
Sujan

[sfc-gh-mkeller]

It's because we resolve the folder as a module level constant only once here.
As suggested we could make this lazy, although that will only delay your error and not resolve it.
I think the real solution is to make our error handling more robust!
PRs are welcome for this change

[mikita-sakalouski]

It's because we resolve the folder as a module level constant only once here. As suggested we could make this lazy, although that will only delay your error and not resolve it. I think the real solution is to make our error handling more robust! PRs are welcome for this change

Please have a look at #2111