Usage of target_options vs storage_options

[beltra909]

I am facing some issues using fsspec within pandas to read parquet file from S3.
Firstly if I pass the filesystem to the pandas fucntion read_parquet everything works as it should:

s3_filesystem= fsspec.filesystem("s3",
                                 endpoint_url="https://my-edn-point:10444",
                                 key="key",
                                 secret="secret",
                                 client_kwargs={
                                    "verify": False,
                                 })

 df=pd.read_parquet(
    "s3://bucket/path.parquet",
    engine='auto',
    filesystem=s3_filesystem,
   )

However when I do not specify the filesystem and I let fsspec create one I get an exception from aiobotocore.

df=pd.read_parquet(
    's3:///bucket/path.parquet',
    storage_options={
                'key': "key",
                'secret': "secret",
                'endpoint_url' : '"https://my-edn-point:10444',
                'client_kwargs':{ 
                    'verify': False,
                },
        },
    )

Traceback (most recent call last):
File "/workspaces/test.py", line 26, in
df=pd.read_parquet(
File "/home/vscode/.local/lib/python3.9/site-packages/pandas/io/parquet.py", line 504, in read_parquet
return impl.read(
File "/home/vscode/.local/lib/python3.9/site-packages/pandas/io/parquet.py", line 252, in read
result = self.api.parquet.read_table(
File "/home/vscode/.local/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 2926, in read_table
dataset = _ParquetDatasetV2(
File "/home/vscode/.local/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 2452, in init
finfo = filesystem.get_file_info(path_or_paths)
File "pyarrow/_fs.pyx", line 571, in pyarrow._fs.FileSystem.get_file_info
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/_fs.pyx", line 1490, in pyarrow._fs._cb_get_file_info
File "/home/vscode/.local/lib/python3.9/site-packages/pyarrow/fs.py", line 330, in get_file_info
info = self.fs.info(path)
File "/home/vscode/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 115, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/home/vscode/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 100, in sync
raise return_result
File "/home/vscode/.local/lib/python3.9/site-packages/fsspec/asyn.py", line 55, in _runner
result[0] = await coro
File "/home/vscode/.local/lib/python3.9/site-packages/s3fs/core.py", line 1248, in _info
out = await self._call_s3(
File "/home/vscode/.local/lib/python3.9/site-packages/s3fs/core.py", line 341, in _call_s3
await self.set_session()
File "/home/vscode/.local/lib/python3.9/site-packages/s3fs/core.py", line 502, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: init() got an unexpected keyword argument 'target_options'

The target_options argument is introduced by the url_to_fs function. I am a bit confused on what is exactly that I am doing wrong. Why does the url_to_fs function add the target_options argument whereas the fsspec.filesystem uses storage_options? is there a way to get the read_paquet function to work without explicitly specifying the filesystem?

[martindurant]

Using pd.read_parquet is certainly the normal route used by most people.
What versions of s3fs, fsspec, pandas and pyarrow are you using?

[beltra909]

I am using Python 3.9.16 with these version of the packages.
I have tried to test more combinations but without success.

Package Version

---

aiobotocore 2.5.0
fsspec 2023.4.0
pandas 1.5.3
s3fs 2023.4.0
pyarrow 11.0.0

[martindurant]

df=pd.read_parquet(
    's3:///bucket/path.parquet',
    storage_options={
                'key': "key",

I see three "/" characters where there should be only two. Is this real, or just transcription error? It could explain what you are seeing.

[martindurant]

(to be sure, I tried a similar read_parquet and it worked OK for me)

[beltra909]

It is a transcription error :)
At this point I guess we can be fairly certainly say that is a problem with my environment although I am still puzzled by what that might be.