Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

download: more consistent and exhaustive logging, new DANDI_DEVEL_AGGRESSIVE_RETRY mode, respect (?) Retry-After #1509

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

download: more consistent and exhaustive logging, new DANDI_DEVEL_AGGRESSIVE_RETRY mode, respect (?) Retry-After #1509

wants to merge 14 commits into from

Commits on Sep 27, 2024

  1. ENH: log dandischema version and ensure we log (consistently) path fo… 

    …r log messages in download
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c784048 View commit details
    Browse the repository at this point in the history

  2. Log download directory path and also either we are entering __exit__ … 

    …with or without exception etc
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    65144ae View commit details
    Browse the repository at this point in the history

  3. Fix typing check - move assert before use of fp.tell() + shorten the … 

    …message while at it
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    cf3fc19 View commit details
    Browse the repository at this point in the history

  4. Add unit-testing for DownloadDirectory to ensure expected operation 

    Also shortened the log line to not include traceback
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    9ccd782 View commit details
    Browse the repository at this point in the history

  5. Add a check that DownloadDirectory moves the file instead of copying … 

    …(via inode matching)
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    390a657 View commit details
    Browse the repository at this point in the history

  6. Move helper function to module level so it could be pickled 

    not sure why was not failing for me locally but fails on CI
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0e2134e View commit details
    Browse the repository at this point in the history

  7. Add handling of PermissionError on Windows for an existing directory 

    That is my guess for what is happening in

    ________________________ test_DownloadDirectory_basic _________________________
    dandi\tests\test_download.py:1048: in test_DownloadDirectory_basic
        with DownloadDirectory(tmp_path, digests={}) as dl:
    dandi\download.py:889: in __exit__
        self.writefile.replace(self.filepath)
    C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\pathlib.py:1247: in replace
        self._accessor.replace(self, target)
    E   PermissionError: [WinError 5] Access is denied: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_DownloadDirectory_basic0.dandidownload\\file' -> 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_DownloadDirectory_basic0'
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    519ea3a View commit details
    Browse the repository at this point in the history

  8. Log details on how much was actually downloaded and in how many chunks

    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    3eaef37 View commit details
    Browse the repository at this point in the history

  9. Make message more "factual" 

    Very unlikely but it could be that directory already existed but without
checksum file for some reason.
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c9b4dfe View commit details
    Browse the repository at this point in the history

  10. Log also versions of requests and urllib3

    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    945acaa View commit details
    Browse the repository at this point in the history

  11. RF+ENH: also report requests and urllib3 versions, do not prefix with v 

    no v prefix since is not providing any information on top; sorting for deterministic order
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    a4997d4 View commit details
    Browse the repository at this point in the history

  12. Also report python version

    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    4d9fd06 View commit details
    Browse the repository at this point in the history

  13. Demand urllib3 to be no less than 2.0.0 in which enforce_content_leng… 

    …th was set to default to True
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    4d0dad3 View commit details
    Browse the repository at this point in the history

  14. ENH: respect/log separately Retry-After + support DANDI_DEVEL_AGGRESS… 

    …IVE_RETRY mode

This is all to address that odd case with 000026 where connection keeps interrupting.
Unclear why so adding more specific cases handling and allowing for such an aggressive
retrying where we would proceed as long as we are getting something (but sleep would also increase)
    @yarikoptic
    yarikoptic committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    bfb73c0 View commit details
    Browse the repository at this point in the history