You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Traceback (most recent call last):
File "/Users/ppatterson/src/b2_zip_files/app.py", line 40, in <module>
files = fs.ls('/')
^^^^^^^^^^
File "/Users/ppatterson/live_read_demo/b2_zip_files/lib/python3.11/site-packages/fsspec/implementations/dirfs.py", line 253, in ls
entry["name"] = self._relpath(entry["name"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ppatterson/live_read_demo/b2_zip_files/lib/python3.11/site-packages/fsspec/implementations/dirfs.py", line 70, in _relpath
assert path.startswith(prefix)
AssertionError
The problem is that DirFileSystem.__init__ sets self.path via fs._strip_protocol(path), which returns /bucket_name. However, S3FileSystem._ls returns filenames of the form bucket_name/path/to/file.ext (no leading /). Hence, in the following code from DirFileSystem._relpath, the assertion fails:
prefix=self.path+self.fs.sep# '/bucket_name/'assertpath.startswith(prefix) # 'bucket_name/path/to/file.ext' does not start with '/bucket-name/'!
It doesn't look like it's possible to set a flag to have S3FileSystem prepend a / to paths, so I think the solution is to make the leading / optional in DirFileSystem._relpath. Something like:
def_relpath(self, path):
ifisinstance(path, str):
ifnotself.path:
returnpath# We need to account for S3FileSystem returning paths that do not# start with a '/'ifpath==self.pathor (self.path.startswith(self.fs.sep) andpath==self.path[1:]):
return""prefix=self.path+self.fs.sepifself.path.startswith(self.fs.sep) andnotpath.startswith(self.fs.sep):
prefix=prefix[1:]
assertpath.startswith(prefix)
returnpath[len(prefix) :]
return [self._relpath(_path) for_pathinpath]
I'll file a PR with this change.
The text was updated successfully, but these errors were encountered:
metadaddy
added a commit
to metadaddy/filesystem_spec
that referenced
this issue
Jun 27, 2024
This is a similar issue to #924.
Code
Error
The problem is that
DirFileSystem.__init__
setsself.path
viafs._strip_protocol(path)
, which returns/bucket_name
. However,S3FileSystem._ls
returns filenames of the formbucket_name/path/to/file.ext
(no leading/
). Hence, in the following code fromDirFileSystem._relpath
, the assertion fails:It doesn't look like it's possible to set a flag to have S3FileSystem prepend a
/
to paths, so I think the solution is to make the leading/
optional inDirFileSystem._relpath
. Something like:I'll file a PR with this change.
The text was updated successfully, but these errors were encountered: