Skip to content

Latest commit

 

History

History
2679 lines (1655 loc) · 129 KB

CHANGELOG.md

File metadata and controls

2679 lines (1655 loc) · 129 KB
Raw

0.63.1 (Mon Sep 09 2024)

🐛 Bug Fix

  • Disallow hdmf 3.14.4, make organize to not parallelize for a single file, log information about all exceptions while reading metadata for organize #1496 (@yarikoptic)
  • BF: use correct variable in exception upon unknown how to handle value of "species" #1482 (@yarikoptic)
  • move: Immediately authenticate client after creation #1480 (@jwodder)

🧪 Tests

  • Dump audit table to file specified by DANDI_TESTS_AUDIT_CSV envvar #1486 (@jwodder)
  • Fix host used in minio URLs #1487 (@jwodder)

Authors: 2

0.63.0 (Thu Aug 08 2024)

🚀 Enhancement

🐛 Bug Fix

  • Fix handling of --preserve-tree for asset ID-only URLs #1476 (@jwodder)

📝 Documentation

🧪 Tests

Authors: 2

0.62.4 (Mon Jul 15 2024)

🔩 Dependency Updates

  • Revert "Temporarily restrict dandischema requirement to < 0.10.2" #1459 (@jwodder)

Authors: 1

0.62.3 (Fri Jul 12 2024)

🔩 Dependency Updates

  • Temporarily restrict dandischema requirement to < 0.10.2 #1458 (@jwodder)

Authors: 1

0.62.2 (Fri Jun 14 2024)

🐛 Bug Fix

  • Normalize path while requesting list of assets from the server #1454 (@yarikoptic)
  • OPT+RF of zarr downloads: do not wait for full files listing + compute %done from total zarr size #1443 (@yarikoptic)
  • Separate datetime components in logfile names with punctuation #1445 (@jwodder)

Authors: 2

0.62.1 (Thu May 23 2024)

🐛 Bug Fix

🏠 Internal

  • ENH: add/use codespell-project/codespell-problem-matcher to annotate PRs on where typos added #1429 (@yarikoptic)

🧪 Tests

Authors: 3

0.62.0 (Fri May 03 2024)

🚀 Enhancement

  • Convert DOI URLs in related_publications to related resources #1417 (@jwodder)

🐛 Bug Fix

🧪 Tests

  • Fix spelling of --scheduled option used in scheduled tests #1428 (@jwodder)
  • Xfail flaky ontobee tests, unless running daily tests #1423 (@jwodder)

Authors: 5

0.61.2 (Fri Mar 22 2024)

🧪 Tests

Authors: 1

0.61.1 (Thu Mar 21 2024)

🐛 Bug Fix

  • Revert "Do not allow dandischema 0.10.1 - new schema not yet supporteed by dandi-archive" #1420 (@yarikoptic)

Authors: 1

0.61.0 (Tue Mar 19 2024)

🚀 Enhancement

  • Add arguments for API query parameters when fetching all Dandisets; support creating embargoed Dandisets #1414 (@jwodder)

🐛 Bug Fix

  • Do not allow dandischema 0.10.1 - new schema not yet supported by dandi-archive #1419 (@yarikoptic)

🏠 Internal

🔩 Dependency Updates

Authors: 2

0.60.0 (Thu Feb 29 2024)

🚀 Enhancement

🏠 Internal

Authors: 2

0.59.1 (Fri Feb 02 2024)

🏠 Internal

📝 Documentation

  • Add notes to docs about get_metadata() vs. get_raw_metadata() #1398 (@jwodder)

🧪 Tests

  • Ignore irrelevant deprecation warning from pandas #1391 (@jwodder)

🔩 Dependency Updates

Authors: 3

0.59.0 (Tue Jan 09 2024)

🚀 Enhancement

  • dandi download dandi://…/dandiset.yaml now downloads dandiset.yaml #1384 (@jwodder)

🐛 Bug Fix

  • Double-check file sizes before & after uploading #1374 (@jwodder)

🏠 Internal

  • Adopt dandischema.digests.zarr.get_checksum() to dandi-cli: use zarr_checksum library constructs instead of copies in dandischema #1371 (@jwodder)
  • gh-actions: Bump github/codeql-action from 2 to 3 #1379 (@dependabot[bot])

📝 Documentation

🧪 Tests

  • Fix service script tests for change in autogenerated date #1385 (@jwodder)

Authors: 4

0.58.2 (Wed Dec 13 2023)

🐛 Bug Fix

  • Properly open filehandles for RemoteReadableAssets #1376 (@jwodder)
  • Repeatedly double-check return values of zero when spying on super_len() #1370 (@jwodder)
  • Set 30-second connect & read timeout when downloading files #1369 (@jwodder)

🏠 Internal

Authors: 2

0.58.1 (Mon Nov 27 2023)

🐛 Bug Fix

🏠 Internal

  • Remove unused configuration settings #1346 (@jwodder)
  • find_files: Simplify check for VCS and DataLad paths #1355 (@jwodder)
  • Convert dandiarchive URL classes from pydantic models to dataclasses #1356 (@jwodder)
  • Replace _pytest imports with modern pytest imports #1347 (@jwodder)
  • Remove _order_ attribute from Severity enum #1348 (@jwodder)
  • Use dataclass instead of auto_repr #1349 (@jwodder)
  • Run pyupgrade --py38-plus on source #1350 (@jwodder)
  • Modernize type annotations and improve mypy config #1343 (@jwodder)
  • Move words ignored by codespell from .codespellignore to setup.cfg #1344 (@jwodder)

🧪 Tests

🔩 Dependency Updates

Authors: 3

0.58.0 (Wed Nov 08 2023)

🚀 Enhancement

🧪 Tests

Authors: 2

0.57.0 (Wed Nov 01 2023)

🚀 Enhancement

🐛 Bug Fix

🏠 Internal

📝 Documentation

🧪 Tests

Authors: 4

0.56.2 (Fri Sep 29 2023)

🐛 Bug Fix

  • Fix uploading Zarr within a BIDS dataset; typing-check guided fix for handling requests exception #1331 (@yarikoptic)

Authors: 1

0.56.1 (Mon Sep 11 2023)

🐛 Bug Fix

Authors: 1

0.56.0 (Fri Aug 11 2023)

🚀 Enhancement

  • ENH: Add support for _desc- entity to label different processings of a file #1315 (@yarikoptic)
  • Accept URLs as Dandi instance specifiers #1298 (@jwodder)

🐛 Bug Fix

🏠 Internal

🧪 Tests

  • TST: fix and not announce xfailed test which checks /server-info - version is provided now #1319 (@yarikoptic)
  • Fix typing errors due to keyring and cv externals changes. #1316 (@jwodder)
  • Fix test for newer hdf5 #1301 (@jwodder)

🔩 Dependency Updates

  • Drop dependency on importlib-metadata backport for Python 3.7 #1299 (@jwodder)

Authors: 4

0.55.1 (Fri May 19 2023)

🔩 Dependency Updates

Authors: 2

0.55.0 (Thu May 18 2023)

🚀 Enhancement

  • Support uploading a subset of files from a BIDS dataset #1286 (@jwodder)
  • Add dandi service-scripts update-dandiset-from-doi command #1283 (@jwodder)

🐛 Bug Fix

  • Use inspect_nwbfile() instead of inspect_nwb(); address bug in recent ruamel.yaml versions #1285 (@jwodder)

🏠 Internal

  • Cancel any still-running tests when pushing to a branch or PR #1287 (@jwodder)

🧪 Tests

Authors: 1

0.54.0 (Fri Apr 21 2023)

🚀 Enhancement

🐛 Bug Fix

Authors: 3

0.53.0 (Mon Apr 17 2023)

🚀 Enhancement

  • Give DandiAPIClient.authenticate() a save_to_keyring parameter #1268 (@jwodder)

🐛 Bug Fix

  • Suppress log messages when downloading with pyout, even if no logger has been configured #1279 (@jwodder)
  • Accept Other sex as o abbreviation #1275 (@bendichter)
  • validate: resolved issues with relative path input #1260 (@TheChymera)
  • Log results of requests.utils.super_len() when DANDI_DEVEL_INSTRUMENT_REQUESTS_SUPERLEN is set #1267 (@jwodder)

🧪 Tests

Authors: 4

0.52.0 (Wed Apr 05 2023)

🚀 Enhancement

🐛 Bug Fix

  • Log more information about retried HTTP requests #1264 (@jwodder)

📝 Documentation

  • Add an example for streaming using as_readable + reference to PyNWB examples #1250 (@yarikoptic)

🧪 Tests

Authors: 4

0.51.0 (Fri Mar 10 2023)

🚀 Enhancement

  • validate: Yield error if path is not inside a Dandiset #1237 (@jwodder)
  • Add service script for re-extracting remote NWB metadata #1236 (@jwodder)

📝 Documentation

🧪 Tests

🔩 Dependency Updates

Authors: 3

0.50.1 (Fri Mar 03 2023)

🧪 Tests

Authors: 1

0.50.0 (Wed Mar 01 2023)

🚀 Enhancement

  • Add --ignore ID_REGEX option to dandi validate #1234 (@jwodder)
  • download: support for downloading multiple URLs at once; dandiset path is now included in the report #1231 (@jwodder)
  • Merging bids_validate into validate, stop validating nested BIDS. #1203 (@TheChymera @jwodder)
  • Support re-extracting metadata from remote assets #1214 (@jwodder)

🐛 Bug Fix

🏠 Internal

📝 Documentation

🧪 Tests

  • Update parse_purlobourl() test for change to rat metadata #1227 (@jwodder)
  • Revert "Temporarily disable all tests which talk to obolibrary" - ontobee said to be back #1226 (@yarikoptic)
  • Fix tests: avoid obolibrary (for now) and pynwb 2.3.0 (forever) #1225 (@yarikoptic)
  • Test against Python 3.11 #1143 (@jwodder)

Authors: 4

0.49.1 (Wed Feb 22 2023)

🐛 Bug Fix

🧪 Tests

  • Restore testing against Windows with Python 3.10 #1213 (@jwodder)

Authors: 1

0.49.0 (Fri Feb 10 2023)

🚀 Enhancement

🐛 Bug Fix

  • Make RemoteDandiset.get_version() return a VersionInfo instance with validation error fields #1210 (@jwodder)
  • Improve messages displayed when an upload fails #1204 (@jwodder)
  • RF/BF: make find_parent_directory_containing operate on absolute path #1188 (@yarikoptic)

🏠 Internal

📝 Documentation

  • Move doc/design under docs/ where the rest of docs reside #1198 (@yarikoptic)
  • Further improve README.md: point to specific sections of handbook, remove lengthy dandi ls --help example #1193 (@melster1010)

🧪 Tests

🔩 Dependency Updates

  • Blacklisting the most recent hdmf version 3.5.0 forbidding its use and breaking CI/testing. #1190 (@TheChymera)

Authors: 4

0.48.1 (Tue Jan 17 2023)

🐛 Bug Fix

  • Fail upload only for validation ERRORs and not necessarily WARNINGs or HINTs + log all errors #1186 (@yarikoptic)
  • Don't warn on unexpected formats for "next" page URLs #1184 (@jwodder)
  • Upload test to check for metadata #1180 (@TheChymera)

Authors: 3

0.48.0 (Mon Dec 19 2022)

🚀 Enhancement

Authors: 1

0.47.0 (Mon Dec 19 2022)

🚀 Enhancement

🐛 Bug Fix

🏠 Internal

📝 Documentation

🧪 Tests

  • No longer mark test_rename_type_mismatch as xfailing #1161 (@jwodder)
  • Fix typing error under mypy 0.990 #1156 (@jwodder)
  • Update test_validate_nwb_path_grouping test #1157 (@jwodder)
  • Fixed logic and commented a temporarily broken BIDS (lacking README) dataset #1148 (@TheChymera)
  • Installing hdf5 for Python 3.10 is no longer needed #1145 (@jwodder)
  • Fix a typing issue involving the outdated tmpdir_factory #1144 (@jwodder)

Authors: 7

0.46.6 (Fri Oct 21 2022)

🐛 Bug Fix

📝 Documentation

🧪 Tests

Authors: 2

0.46.5 (Fri Oct 14 2022)

🐛 Bug Fix

🏠 Internal

  • Make internal Zarr-checksumming tree types support empty Zarrs #1138 (@jwodder)
  • Simplify implementations of get_asset() methods #1136 (@jwodder)
  • Update GitHub Actions action versions #1135 (@jwodder)

Authors: 3

0.46.4 (Thu Oct 06 2022)

🏠 Internal

🧪 Tests

  • increase number of trials in docker_compose_setup to 25 #1122 (@djarecka)
  • Pull latest Docker images before spinning up Compose #1121 (@jwodder)

Authors: 2

0.46.3 (Mon Sep 19 2022)

🐛 Bug Fix

  • Detect link type for organize by linking from first src dir to dest dir #1115 (@jwodder)
  • Check for usage of latest NWB Inspector release #1108 (@CodyCBakerPhD)

🧪 Tests

Authors: 2

0.46.2 (Thu Sep 01 2022)

🐛 Bug Fix

  • Use upstream bidsschematools for BIDS validation #1095 (@TheChymera)
  • get_content_url(): If a HEAD fails, return the failing URL #1107 (@jwodder)

🏠 Internal

  • BF(workaround): blacklist 23.9.0 of keyring since introduced regression #1112 (@yarikoptic)

Authors: 3

0.46.1 (Thu Aug 18 2022)

🐛 Bug Fix

📝 Documentation

🧪 Tests

Authors: 3

0.46.0 (Thu Aug 18 2022)

🚀 Enhancement

🐛 Bug Fix

  • ENH: list localhost:8085 as the web UI for dandi-api-local-docker-tests #1003 (@yarikoptic)
  • BF: guard download_generator to not propagate errors #1008 (@yarikoptic)
  • User notification if datasets are invalid. #1080 (@TheChymera)
  • Updating BIDS validator and schema to contemporary upstream equivalent #1050 (@TheChymera)
  • Support DANDI:/draft and stop using identifiers.org for any of the DANDI: URIs #1066 (@yarikoptic)

🏎 Performance

  • Do not traverse .ome.zarr and .ngff folders while looking for BIDS datasets #1071 (@yarikoptic)

⚠️ Pushed to master

🏠 Internal

  • Factor out common fields in nwb2asset() and get_default_metadata() #1088 (@jwodder)

📝 Documentation

🧪 Tests

Authors: 4

0.45.1 (Fri Jul 15 2022)

🐛 Bug Fix

  • dandi move: Fix resolution of nonexistent dests when in a subdirectory #1063 (@jwodder)
  • dandi move: Include all mismatches in error messages #1064 (@jwodder)

📝 Documentation

  • Mention in help text that 'draft' is not supported in DANDI:/ #1062 (@jwodder)

Authors: 1

0.45.0 (Wed Jul 13 2022)

🚀 Enhancement

🐛 Bug Fix

🏠 Internal

Authors: 3

0.44.1 (Fri Jul 08 2022)

🐛 Bug Fix

  • Retry Zarr entry uploads that fail with "header ... implies functionality ... not implemented" #1048 (@jwodder)

Authors: 1

0.44.0 (Fri Jul 08 2022)

🚀 Enhancement

🐛 Bug Fix

  • Don't ingest a Zarr if nothing changed while uploading #1047 (@jwodder)

Authors: 1

0.43.0 (Tue Jul 05 2022)

🚀 Enhancement

Authors: 1

0.42.0 (Fri Jul 01 2022)

🚀 Enhancement

  • Reuse "loose" Zarrs that conflict with uploaded path #1035 (@jwodder)

🐛 Bug Fix

  • Include leading text of response bodies in HTTP error messages #1023 (@jwodder)

🏠 Internal

🧪 Tests

  • Make test_move_command_too_few_paths support DANDI_DEVEL=1 #1040 (@jwodder)

Authors: 1

0.41.0 (Mon Jun 27 2022)

🚀 Enhancement

Authors: 1

0.40.1 (Fri Jun 03 2022)

🐛 Bug Fix

  • Fix calculation of download paths for asset path prefix URLs #1026 (@jwodder)
  • organize - sanitize ; as well within the entity value #1009 (@yarikoptic)

Authors: 2

0.40.0 (Tue May 31 2022)

🚀 Enhancement

🐛 Bug Fix

🏎 Performance

Authors: 4

0.39.6 (Mon May 09 2022)

🐛 Bug Fix

Authors: 2

0.39.5 (Thu May 05 2022)

🐛 Bug Fix

🏠 Internal

Authors: 4

0.39.4 (Tue Apr 26 2022)

🏎 Performance

  • Remove size comparison check when uploading Zarr entries #986 (@jwodder)

Authors: 1

0.39.3 (Mon Apr 25 2022)

🐛 Bug Fix

🏠 Internal

🧪 Tests

Authors: 2

0.39.2 (Tue Apr 19 2022)

🐛 Bug Fix

  • Validator fixes: fallback to our version, skip dotdirs on windows, etc #955 (@TheChymera)
  • Better detection of empty directories when download-syncing Zarrs #961 (@jwodder)

Authors: 2

0.39.1 (Thu Apr 14 2022)

🐛 Bug Fix

Authors: 1

0.39.0 (Thu Apr 14 2022)

🚀 Enhancement

🐛 Bug Fix

⚠️ Pushed to master

🏠 Internal

🧪 Tests

Authors: 7

0.38.0 (Mon Apr 04 2022)

🚀 Enhancement

🏠 Internal

Authors: 2

0.37.0 (Tue Mar 22 2022)

🚀 Enhancement

  • Recognize video files as non-generic assets #922 (@jwodder)

🐛 Bug Fix

  • Update for change in Zarr checksum format #933 (@jwodder)
  • Trigger Zarr ingestion after performing an upload #938 (@jwodder)
  • check for existence of video path else assume relative path #931 (@Saksham20)
  • dandi upload: Make --jobs default to 5:5 and document it #940 (@jwodder)
  • checking for duplicate video files in external_files #928 (@Saksham20)
  • skip handling URLs in external files references as we do for local paths #932 (@Saksham20)
  • Let dandi delete take paths to local Zarrs #925 (@jwodder)

🏎 Performance

  • Minimize/optimize Zarr digestion when uploading #923 (@jwodder)
  • Don't repeat API requests when finding subdir files to delete before Zarr upload #919 (@jwodder)

📝 Documentation

  • Correct the docstring for BasePath.parents #936 (@jwodder)

Authors: 2

0.36.0 (Thu Feb 17 2022)

🚀 Enhancement

🐛 Bug Fix

  • Make dandi upload look for dandiset.yaml in parents of common path #916 (@jwodder)
  • Mint Zarr assets at start of upload and sync Zarr contents #907 (@jwodder)
  • Use full asset path as name of Zarr object #912 (@jwodder)
  • Specify Dandiset ID when creating a Zarr #911 (@jwodder)
  • Cancel a Zarr upload if an error occurs partway through #893 (@jwodder)
  • Make get_raw_digest() actually default to the proper digest type for Zarrs #894 (@jwodder)
  • Fix neural data type to metadata extractor #891 (@satra @yarikoptic)

🏠 Internal

  • Add dedicated HTTP404Error exception class #908 (@jwodder)
  • Make paths argument to upload() optional #899 (@jwodder)
  • Add type-annotations to various files #887 (@jwodder)
  • Delete functions (get_asset_metadata, find_dandi_files) obsoleted by PR #853 #885 (@jwodder)
  • Use a pydantic model to store a RemoteDandiset's cached data #876 (@jwodder)

📝 Documentation

  • improve DandiApiClient.init docstring #909 (@bendichter)
  • Document new dandi organize options in Read the Docs site #901 (@jwodder)
  • Organize & fine-tune dandi.dandiapi Sphinx docs #888 (@jwodder)

🧪 Tests

🔩 Dependency Updates

Authors: 6

0.35.0 (Mon Jan 24 2022)

🚀 Enhancement

  • Make navigate() and navigate_url() auto-authenticate for embargoed Dandisets #870 (@jwodder)

🐛 Bug Fix

🏠 Internal

Authors: 2

0.34.1 (Mon Jan 17 2022)

🔩 Dependency Updates

  • upgrade dandischema requirement from 0.4.1 to 0.5.1 (~=) #868 (@yarikoptic)

Authors: 1

0.34.0 (Wed Jan 12 2022)

🚀 Enhancement

  • Include "dandiset" field in /uploads/initialize/ requests #863 (@jwodder)

🐛 Bug Fix

  • ENH: update species interpretation from NWB file #857 (@satra)

Authors: 2

0.33.0 (Tue Jan 11 2022)

🚀 Enhancement

🐛 Bug Fix

  • Add DJANGO_DANDI_DANDISETS_EMBARGO_BUCKET_NAME setting to test docker-compose.yml #854 (@dchiquito)

🏠 Internal

📝 Documentation

🧪 Tests

  • Replace the use of pyfakefs with a tmp_home fixture #861 (@jwodder)

Authors: 2

0.32.2 (Mon Dec 06 2021)

🐛 Bug Fix

  • Improve UX of failed redirects while resolving resource identifiers #830 (@jwodder)

🏠 Internal

🧪 Tests

Authors: 2

0.32.1 (Mon Nov 29 2021)

🐛 Bug Fix

Authors: 1

0.32.0 (Fri Nov 19 2021)

🚀 Enhancement

  • Support setting --dandi-instance via a DANDI_INSTANCE envvar #822 (@jwodder)
  • Give the resource classes api_url properties #816 (@jwodder)

🐛 Bug Fix

  • Convert metadata 404's when downloading into an "error" status #839 (@jwodder)
  • Don't error when redownloading the same dandiset.yaml file #824 (@jwodder)
  • Make download fail immediately on nonexistent resources #820 (@jwodder)

🏠 Internal

📝 Documentation

  • Add Read the Docs badge to README #832 (@jwodder)
  • Improve dandi ls --help and dandi download --help output #827 (@jwodder)

Authors: 1

0.31.0 (Mon Oct 25 2021)

🚀 Enhancement

  • Support requesting sorted assets #807 (@jwodder)
  • Make logfile always contain all DEBUG or higher log messages #813 (@jwodder)
  • Publicly expose classmethods for constructing API resource instances #808 (@jwodder)
  • Make --dandi-instance public and add "instances" command #804 (@jwodder)
  • dandi.dandiapi: Convert 404 responses to NotFoundErrors #801 (@jwodder)

🐛 Bug Fix

  • Fix a minor bug in a handler for a rare condition #806 (@jwodder)
  • Show valid instance names when given a dandi:// URL with an unknown instance #815 (@jwodder)

🏠 Internal

📝 Documentation

Authors: 1

0.30.3 (Mon Oct 11 2021)

🏠 Internal

📝 Documentation

  • Document & type-annotate all public values in dandi.dandiapi #799 (@jwodder)
  • Set Sphinx default_role to "py:obj" #795 (@jwodder)

🧪 Tests

  • Apply "skipif_no_network" to all relevant tests #797 (@jwodder)

Authors: 1

0.30.2 (Mon Oct 11 2021)

🐛 Bug Fix

  • Retry more requests; wait longer for Dandiset validity #798 (@jwodder)

Authors: 1

0.30.1 (Thu Oct 07 2021)

📝 Documentation

🧪 Tests

Authors: 2

0.30.0 (Wed Oct 06 2021)

🚀 Enhancement

  • Log a message if a request succeeds after one or more failures [take 2] #792 (@jwodder)

🧪 Tests

  • Make CI tests dump Docker Compose logs on failure #793 (@jwodder)

Authors: 1

0.29.0 (Wed Oct 06 2021)

🚀 Enhancement

Authors: 1

0.28.0 (Mon Oct 04 2021)

🚀 Enhancement

  • Give validate command an --allow-any-path option #783 (@jwodder)

🐛 Bug Fix

  • Fix an AttributeError raised when a URL is used as a species #782 (@jwodder)

🧪 Tests

Authors: 2

0.27.3 (Tue Sep 28 2021)

🐛 Bug Fix

🧪 Tests

🔩 Dependency Updates

Authors: 1

0.27.2 (Mon Sep 13 2021)

🧪 Tests

  • Update Docker Compose spin-up code for loss of --username option #768 (@jwodder)

Authors: 1

0.27.1 (Fri Sep 03 2021)

🐛 Bug Fix

  • Stop uploading a file if no metadata can be extracted #767 (@jwodder)
  • Retry following redirects on 400 (not 404), 502, 503, & 504 up to 5 times #762 (@jwodder)
  • Add dandischema into fscacher tokens for validate #760 (@yarikoptic)

📝 Documentation

🧪 Tests

  • Use new published version of 000027 in tests #763 (@jwodder)

Authors: 2

0.27.0 (Thu Aug 26 2021)

🚀 Enhancement

  • Make download_url RemoteAsset-only; give BaseRemoteAsset base_download_url #759 (@jwodder)
  • Give API resource classes __str__ methods #753 (@jwodder)

🐛 Bug Fix

🧪 Tests

  • Remove test against rc/2.0.0 branch of pynwb #754 (@jwodder)

Authors: 1

0.26.1 (Mon Aug 09 2021)

🐛 Bug Fix

  • Boost dandischema to ~= 0.3.1 as it provides dandischema 0.5.1 required by dandi-api #749 (@yarikoptic)

Authors: 1

0.26.0 (Wed Aug 04 2021)

🚀 Enhancement

🧪 Tests

Authors: 2

0.25.0 (Wed Jul 28 2021)

🚀 Enhancement

  • Add replace_asset parameter to iter_upload_raw_asset() #743 (@jwodder)
  • Rename get_assets_under_path() to get_assets_with_path_prefix() #741 (@jwodder)

🐛 Bug Fix

  • Correctly set User-Agent for client requests #742 (@jwodder)

🧪 Tests

Authors: 2

0.24.0 (Fri Jul 23 2021)

🚀 Enhancement

  • Python API rewrite, part 2 #676 (@jwodder)
  • RF: attempt nwb metadata extraction only on .nwb, if fails -- warning #733 (@yarikoptic)
  • Invoke etelemetry when constructing a DandiAPIClient; honor DANDI_NO_ET #728 (@jwodder)
  • Make upload() fail if client & server schema versions are not in sync #724 (@jwodder)

🐛 Bug Fix

  • Log "error" statuses while uploading as ERROR log messages #737 (@jwodder)
  • Retry following redirects that return 404 #734 (@jwodder)
  • ENH: exit with non-0 when "bad_version" of dandi-cli is used #725 (@yarikoptic)

📝 Documentation

Authors: 2

0.23.2 (Tue Jul 20 2021)

🐛 Bug Fix

🏠 Internal

📝 Documentation

🧪 Tests

  • BF(TST): Allow for IteratorWithAggregation to get nothing if reraise_immediately #707 (@yarikoptic)

Authors: 3

0.23.1 (Thu Jul 08 2021)

🧪 Tests

  • Skip shell completion test entirely on Windows #702 (@jwodder)
  • BF+RF(TST): populate contentUrl to satisfy Asset requirement in 0.5.0 #705 (@yarikoptic)

Authors: 2

0.23.0 (Tue Jul 06 2021)

🚀 Enhancement

🐛 Bug Fix

  • fix: raise error when unable to parse age and set session name properly #669 (@satra @yarikoptic)
  • fix: set CLI version to align with schema base version #694 (@satra)
  • adding valueReference to extract_age return PropertyValue; adding test #689 (@djarecka @satra)

Authors: 4

0.22.0 (Wed Jun 30 2021)

🚀 Enhancement

🐛 Bug Fix

  • Create valid Dandiset metadata when testing; create valid asset metadata for non-NWB files; wait for validation before publishing #683 (@jwodder)
  • changing unitText #686 (@djarecka)
  • Fix a failing test on Windows on conda-forge (again) #681 (@jwodder)
  • Remove references to "dandi register" command #684 (@jwodder)

🧪 Tests

  • Use dandischema.metadata.validate() instead of validate*_json() #685 (@jwodder)

🔩 Dependency Updates

Authors: 3

0.21.0 (Thu Jun 24 2021)

🚀 Enhancement

🐛 Bug Fix

  • Don't pass "asset" field from upload iterator to pyout #679 (@jwodder)

🧪 Tests

  • Fix a failing test on Windows on conda-forge #680 (@jwodder)

Authors: 1

0.20.0 (Mon Jun 14 2021)

🚀 Enhancement

  • ENH: more metadata to reconstruct filename, upgrade to use dandischema 0.2.3 (schema 0.4.0) #644 (@satra @yarikoptic)

🐛 Bug Fix

  • ls --schema: Calculate digest for local assets #666 (@jwodder)
  • updating parse age to cover more formats #633 (@djarecka)
  • fix: add a default name when using an unknown session id #662 (@satra)

🏠 Internal

  • Run test workflow on pushes only on master #667 (@yarikoptic)
  • Address LGTM alerts #657 (@jwodder)
  • Update pre-commit repo versions and configure isort to properly handle "from . import" lines #656 (@jwodder)

Authors: 4

0.19.0 (Mon Jun 07 2021)

🚀 Enhancement

  • Reprompt or fail on attempt to use an invalid API token #655 (@jwodder)
  • dandi ls: Make json and json_pp formats output arrays; add json_lines for old json format #654 (@jwodder)
  • Change download --existing default to "error"; add "overwrite-different" option; handle git-annex repos #646 (@jwodder)

🏠 Internal

Authors: 1

0.18.0 (Wed May 26 2021)

🚀 Enhancement

🐛 Bug Fix

🏠 Internal

  • Include CHANGELOG.md and tox.ini in sdists #648 (@jwodder)

Authors: 4

0.17.0 (Wed May 12 2021)

🚀 Enhancement

🐛 Bug Fix

⚠️ Pushed to master

  • RM: .github/workflows/test-populate-dandiset-yaml.yml (@yarikoptic)

🧪 Tests

  • move test_get_metaadata to test_metadata.py #634 (@bendichter)
  • Error on PRs that modify existing schemata instead of creating a new version #626 (@jwodder)

Authors: 4

0.16.0 (Tue May 04 2021)

🚀 Enhancement

🐛 Bug Fix

  • DOC: Extend description for delete to point that it could be URL etc #609 (@yarikoptic)

🏠 Internal

  • Cut down on some code duplication in delete.py #610 (@jwodder)

🧪 Tests

  • Make test_server_info use Docker fixture if DANDI_REDIRECTOR_BASE is set #612 (@jwodder)
  • Add DANDI_DEVEL=1 job to GitHub Actions tests #607 (@jwodder)

Authors: 2

0.15.0 (Fri Apr 30 2021)

🚀 Enhancement

🐛 Bug Fix

  • Include list of supported URL patterns in dandi ls --help #601 (@jwodder)
  • Recognize "DANDI:" strings as URL-likes #602 (@jwodder)

🏠 Internal

  • Support specifying the dandi redirector via an env var #581 (@jwodder)
  • a script to validate dandi-api collection listing against girder #589 (@yarikoptic)

🧪 Tests

Authors: 2

0.14.2 (Thu Apr 22 2021)

🐛 Bug Fix

  • BF: "girder" record might be there but "url" might be None #591 (@yarikoptic)
  • Retry upload requests that result in 500 responses #585 (@jwodder)

🏠 Internal

Authors: 2

0.14.1 (Fri Apr 16 2021)

🐛 Bug Fix

Authors: 2

0.14.0 (Thu Apr 15 2021)

🚀 Enhancement

  • Models: define id, add various additional types (genotype, etc), boost model version to 0.3.0 #560 (@satra)
  • Switch default dandi instance to dandi-api based on redirector #565 (@jwodder)

🧪 Tests

  • Add further tests of get_instance() and server-info #571 (@jwodder)

Authors: 2

0.13.2 (Tue Apr 13 2021)

🐛 Bug Fix

  • BF: do not assume that service record of redirector is present/has url #567 (@yarikoptic)
  • Fix a typo in the display string for one of the known URL patterns #564 (@jwodder)
  • Error with a decent message when trying to delete() a path not in a Dandiset #563 (@jwodder)
  • Fix & test for downloading by asset ID URL #561 (@jwodder)
  • Strip trailing slash from API URL used by delete() #559 (@jwodder)
  • Refresh dandiset.yaml on download if out of date #556 (@jwodder)
  • Support "…/assets/?path=" URLs #555 (@jwodder)
  • Get hdmf, pynwb, h5py versions without importing #553 (@jwodder)

Authors: 2

0.13.1 (Sat Apr 10 2021)

🐛 Bug Fix

📝 Documentation

  • DOC: minor tune up to README.md on installation instructions and WiP #551 (@yarikoptic)

Authors: 3

0.13.0 (Thu Apr 08 2021)

🚀 Enhancement

🐛 Bug Fix

⚠️ Pushed to master

  • DOC: provide instructions in DEVELOPMENT.md for interaction with dandi-api instance (@yarikoptic)

🏠 Internal

🧪 Tests

  • Add tests of json_dict() method #526 (@jwodder)
  • Store metadata test JSON in files #525 (@jwodder)
  • Run "provision" container in the foreground #506 (@jwodder)
  • Ignore warnings from ruamel.yaml caused by hdmf using deprecated functions #507 (@jwodder)

Authors: 5

0.12.1 (Thu Mar 25 2021)

🐛 Bug Fix

  • Lowercase "sha256" #493 (@jwodder)
  • Validate uploads before digesting #495 (@jwodder)
  • Check for already-uploaded blobs via /uploads/initialize/ instead of /blobs/digest/ #496 (@jwodder)
  • Update upload code for changes in API #479 (@jwodder)
  • dandi ls: Error if --schema is given with remote resource of different version #489 (@jwodder)

🧪 Tests

  • Add option for only running new Dandi API tests #500 (@jwodder)
  • Don't hardcode DANDI_SCHEMA_VERSION value in tests #491 (@jwodder)

Authors: 1

0.12.0 (Tue Mar 23 2021)

🚀 Enhancement

🐛 Bug Fix

🏠 Internal

Authors: 2

0.11.0 (Mon Mar 08 2021)

🚀 Enhancement

🐛 Bug Fix

  • Add dateModified to asset metadata #452 (@jwodder)
  • fix: change enum to const for single enums #449 (@satra)
  • Add --new-schema option to ls command for converting output #445 (@jwodder)
  • Eliminate check for session_start_time preceding date_of_birth #440 (@jwodder)
  • Eliminate DANDI_SCHEMA; add get_schema_version() #442 (@jwodder)
  • Discard empty "sex" and "species" fields on conversion #438 (@jwodder)
  • schema: minor spotted typo fixes #435 (@yarikoptic)
  • Retry requests on ConnectionErrors #437 (@jwodder)
  • Include HDMF version as well into the token #434 (@yarikoptic)
  • Error if sha256 digest is missing from asset being downloaded #428 (@jwodder)
  • Report dandi version in User-Agent header #424 (@jwodder)
  • Remove misleading log message about authenticating with new API #425 (@jwodder)
  • Distinguish between pre- and post-validation when uploading #420 (@jwodder)
  • Log failed HTTP connections; include PID and TID in logs; include asset path in upload log messages #418 (@jwodder)
  • Revert PR #409 (Content-MD5 header) #419 (@jwodder)
  • Set Content-MD5 header when uploading asset parts #409 (@jwodder)
  • upload(): Only yield first "validating" status to pyout #417 (@jwodder)
  • Add more logging when uploading & downloading #412 (@jwodder)
  • Sleep increasing amounts while waiting for uploaded assets to validate #408 (@jwodder)
  • Populate wasDerivedFrom #386 (@jwodder)
  • FIX: use authorized checkout for actions #403 (@satra)
  • enh: account for samples, sessions and participants #392 (@satra @jwodder @yarikoptic)
  • Disabling logging to stderr while using pyout #401 (@jwodder)
  • Cache file digests and check for change in digest when uploading #391 (@jwodder)
  • Make existing="refresh" a synonym for "overwrite" for new upload #390 (@jwodder)
  • RF/NF: Identifiable and BareAssetMeta to describe an asset anywhere #373 (@yarikoptic @jwodder)
  • Give known_urls human-readable display strings #384 (@jwodder)
  • Make dandi download -i <instance> run in a Dandiset download that Dandiset #383 (@jwodder)
  • Give validate command a --schema VERSION option for validating assets and dandiset.yaml #379 (@jwodder)
  • Support downloading folders and latest Dandiset version #377 (@jwodder @yarikoptic)
  • Add include_metadata=False parameter to asset-listing DandiAPIClient methods #378 (@jwodder @yarikoptic)
  • Discard invalid asset identifiers when extracting metadata #374 (@jwodder)
  • Handle uploading already-present files in new API #347 (@jwodder)
  • Adjust license metadata conversion #364 (@jwodder @yarikoptic)
  • Give AssetMeta and DandiMeta json_dict() methods for better dictification #346 (@jwodder)
  • BF: allow to handle an item with multiple files #342 (@yarikoptic)
  • ENH: devel upload dandiset metadata #341 (@yarikoptic)
  • Try self.listFile() again on ConnectionErrors #335 (@jwodder)
  • Add functions for validating metadata against JSON Schema and use in tests #338 (@jwodder)
  • Fix AttributeError: 'Resource' object has no attribute 'values' #336 (@jwodder)
  • [DATALAD RUNCMD] Swap order of str and AnyUrl to be from specific to generic #334 (@yarikoptic)
  • cleaning up biosample and participant #312 (@satra @jwodder)
  • Tee all logs to user log directory #318 (@jwodder)
  • Update for new API at https://api.dandiarchive.org/api #283 (@jwodder)

⚠️ Pushed to master

  • Merge branch 'gh-320' (@yarikoptic)
  • DOC: provide description for both DANDI_API_KEY and DANDI_GIRDER_API_KEY (@yarikoptic)
  • ENH: log at DEBUG result.text from a failed response (@yarikoptic)
  • ENH: allow for DeprecationWarning to come from requests_toolbelt, not our problem (@yarikoptic)

🏠 Internal

📝 Documentation

🧪 Tests

Authors: 3

0.10.0 (Tue Dec 08 2020)

🚀 Enhancement

🐛 Bug Fix

  • Autogenerate schema element titles #304 (@jwodder)
  • Compare uploaded file size against what download headers report #306 (@jwodder)
  • fix: rat to common lab rat #307 (@satra)

Authors: 3

0.9.0 (Fri Dec 04 2020)

🚀 Enhancement

🐛 Bug Fix

🏠 Internal

Authors: 6

0.8.0 (Tue Dec 01 2020)

🚀 Enhancement

  • Add rudimentary duecredit support using zenodo's dandi-cli DOI #285 (@yarikoptic)

🐛 Bug Fix

🏠 Internal

Authors: 3

0.7.2 (Thu Nov 19 2020)

🐛 Bug Fix

🏠 Internal

  • Add healthchecks for the Postgres and minio Docker containers #272 (@jwodder)

Authors: 1

0.7.1 (Thu Nov 05 2020)

🐛 Bug Fix

  • Use oldest file when race condition causes multiple files per item #265 (@jwodder)

🏠 Internal

  • Set up workflow with auto for releasing & PyPI uploads #257 (@jwodder)

📝 Documentation

Authors: 1

[0.7.0] - 2020-11-04

Added

  • Files are now stored in temporary directories while downloading alongside metadata for use in resuming interrupted downloads

Changed

  • Moved code for navigating Dandi Archive into new dandiarchive submodule
  • YAML output now sorts keys
  • dandiset.yaml is no longer uploaded to the archive
  • Restrict h5py dependency to pre-v3.0

[0.6.4] - 2020-09-04

Primarily a range of bugfixes to ensure correct operation with current state of other components of DANDI, and use of the client on Windows OS.

Added

  • Initial DANDI schema files
  • More tests for various code paths
  • download: new option --download [assets,dandiset.yaml,all]

Fixed

  • download - account for changes in DANDI API (relevant only for released datasets, of which we do not have any "real" ones yet)
  • upload - various Windows specific fixes

Note: [0.6.3] was released under missing some of the fixes, so overall abandoned.

[0.6.2] - 2020-08-19

Fixed

  • organize treatment of paths on window (gh-204)

[0.6.1] - 2020-08-18

Changed

  • CLI modules RF to avoid circular imports
  • pytest default traceback style is short and shows 10 slowest tsts

Fixed

  • download of draft datasets from Windows (gh-202)
  • upload and other tests to account for new web UI

[0.6.0] - 2020-08-12

A variety of improvements and bug fixes, with major changes toward support of a new DANDI API, and improving DX (Development eXperience).

Added

  • Support for WiP DANDI API service. download now can download from "published" (versioned) dandisets.
  • A wide range of development enhancements
    • tox setup
    • code linting via tox and on github workflows
    • testing against Python 3.8
    • testing against a local instance of the archive via docker-compose, which is used against
  • Locking of the dandiset during upload to prevent multiple sessions modifying the same dandiset in the archive
  • upload now adds uploaded_by field into the item metadata

Changed

  • download was refactored and new UI also uses pyout (as upload and ls) so there will be no tqdm progress bar indicators. download also does "on-the-fly" integrity of the data as received (whenever corresponding metadata provided from the archive)
  • --log-level could be numeric or specified in lower-case
  • Unified YAML operations to ruamel.yaml
  • Avoid hardcoded URLs for dandiarchive components by querying /server-info
  • Improved logging for interactions with girder server

Fixed

  • minor compatibility issues across OSes

[0.5.0] - 2020-06-04

Added

  • metadata and organize: extract and use probe_ids metadata to disambiguate (if needed)
  • organize: --devel-debug option to perform metadata extraction serially
  • upload:
    • --allow-any-path development option to allow upload of DANDI not yet 'unsupported' file types/paths
    • compute 4 digests (all are checksums ATM): md5, sha1, sha256, sha512 and upload as a part of the metadata record
  • download:
    • use the "fastest" available digest (sha1) to validate correctness of the download
    • follow redirections from arbitrary redirector (e.g., bit.ly). Succeeds only if the final URL is known to DANDI.

Fixed

  • upload: a crash while issuing a record to update about deleted empty item

Refactored

  • organize: disambiguation process now could use a flexible list of metadata fields (ATM only probe_ids and obj_id)
  • download: handling of redirection - now uses HEAD request instead of GET

[0.4.6] - 2020-05-07

Fixed

  • invoke etelemetry only in command line (at click interface level)
  • download of updated dandiset landing page url (/dandiset not /dandiset-meta)

[0.4.5] - 2020-05-01

Added

  • support for downloading dandisets and files in the just released gui.dandiarchive.org UI refactor

Fixed

  • validate should no longer crash if loading metadata raises an exception

Refactored

  • the way URLs are mapped into girder instances. Now more regex driven

[0.4.4] - 2020-04-14

Added

  • validate now will report absent subject_id as an error

Fixed

  • Caching of multiple functions re-using the same cache -- it could have resulted in our case neural data types returned where full metadata was requested, or vice versa
  • Tolerate outdated (before 2.0.0) etelemetry

[0.4.3] - 2020-04-14

Added

  • Ability to download (multiple) individual files (using URL from gui.dandiarchive.org having files selected)

Changed

  • DANDI_CACHE_CLEAR -> DANDI_CACHE=(ignore|clear) env variable.
  • Sanitize and tollerate better incorrect nwb_version field.

Fixed

  • Test to not invoke Popen with shell=True to avoid stalling.
  • Explicit NO_ET=1 in workflows to avoid overreporting to etelemetry.

[0.4.2] - 2020-03-18

Added

  • Use of etelemetry for informing about new (or bad) versions

Changed

  • Fixed saving into yaml so it is consistently not using a flow style (#59)
  • All file names starting with a period are not considered (#63)

[0.4.1] - 2020-03-16

Changed

  • organize -- now would add _obj- key with the crc32 checksum of the nwb file object_id if files could not be otherwise disambiguated
  • variety of small tune ups and fixes

Removed

  • organize -- not implemented option --format
  • upload -- not properly implemented option -d|--dandiset-path

[0.4.0] - 2020-03-13

Provides interfaces for a full cycle of dandiset preparation, registration, upload, and download.

Added

  • caching of read metadata and validation results for .nwb files. Typically those take too long and as long as dandi and pynwb versions do not change -- results should not change. Set DANDI_DEVEL variable to forcefully reset all the caches.

Changed

  • DEVELOPMENT.md provides more information about full local test setup of the dandiarchive, and description of environment variables which could assist in development.

[0.3.0] - 2020-02-28

Added

  • organize: organize files into hierarchy using metadata. ATM operates only in "simulate" mode using .json files dumped by ls

Changed

  • various refactorings and minor improvements (docs, testing, etc).

[0.2.0] - 2020-02-04

Improvements to ls and upload commands

Added

  • ls: include a list (with counts) of neural datatypes in the file
  • upload:
    • ability to reupload files (by removing already existing ones)
    • ability to "sync" (skip if not modified) to girder based on mtime and size
  • CI (github actions): testing on macos-latest

Changed

  • removed hdmf != statement in setup.cfg to not confuse pypi.

Fixed

  • upload - assure string for an error message
  • mitigated crashes in pynwb if neural data type schema is not cached in the file and requires import of the extension module. ATM the known/handled only the AIBS_ecephys from allensdk