From 473a247fc67537dca99248db249985826ed28430 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 17 Apr 2024 20:45:44 -0500 Subject: [PATCH 01/48] Update supported python version for DANDI CLI/Python API --- docs/10_using_dandi.md | 2 +- docs/12_download.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/10_using_dandi.md b/docs/10_using_dandi.md index 2cc86897..78981568 100644 --- a/docs/10_using_dandi.md +++ b/docs/10_using_dandi.md @@ -90,7 +90,7 @@ The [DANDI Python client](https://pypi.org/project/dandi/) allows you to: * Organize your data locally before upload * Upload Dandisets -Before you can use the DANDI Python client, you have to install the package with `pip install dandi` in a Python 3.7+ +Before you can use the DANDI Python client, you have to install the package with `pip install dandi` in a Python 3.8+ environment. You should check the [Dandi Debugging section](./15_debugging.md) in case of any problems. diff --git a/docs/12_download.md b/docs/12_download.md index 3b3a13be..dcf6bd4e 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -34,7 +34,7 @@ Each file in the Dandiset has a download icon next to it, clicking the icon will The [DANDI Python client](https://pypi.org/project/dandi/) gives you more options, such as downloading entire Dandisets. -**Before You Begin**: You need to have Python 3.7+ and install the DANDI Python Client using `pip install dandi`. +**Before You Begin**: You need to have Python 3.8+ and install the DANDI Python Client using `pip install dandi`. If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md). ### Download a Dandiset From 28977c4817ad01f162c72fae37419a002e8e1808 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 17 Apr 2024 22:27:22 -0500 Subject: [PATCH 02/48] Rename `About This Doc` to `Contributing Documentation` --- docs/{100_about_this_doc.md => 100_contribute_docs.md} | 2 +- docs/index.md | 2 +- mkdocs.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) rename docs/{100_about_this_doc.md => 100_contribute_docs.md} (97%) diff --git a/docs/100_about_this_doc.md b/docs/100_contribute_docs.md similarity index 97% rename from docs/100_about_this_doc.md rename to docs/100_contribute_docs.md index 99ff5795..e65af75e 100644 --- a/docs/100_about_this_doc.md +++ b/docs/100_contribute_docs.md @@ -1,4 +1,4 @@ -# About This Documentation +# Contributing Documentation This documentation is a work in progress and we welcome all input: if something is missing or unclear, let us know by [opening an issue on our helpdesk](https://github.com/dandi/helpdesk/issues/new/choose). diff --git a/docs/index.md b/docs/index.md index ac520be6..c007a421 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,7 +43,7 @@ registered an account on the archive. If you want to get started right away and contribute directly to this documentation, see the [About This Documentation](. -/100_about_this_doc.md) section. +/100_contribute_docs.md) section. ## License diff --git a/mkdocs.yml b/mkdocs.yml index 10211565..cade3f66 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -37,11 +37,11 @@ nav: - Notes: "40_development.md" - REST API Swagger: https://api.dandiarchive.org/swagger - REST API Redoc: https://api.dandiarchive.org/redoc + - Contributing Documentation: "100_contribute_docs.md" - DANDI Hub: "50_hub.md" - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" - - About This Doc: "100_about_this_doc.md" # List of extensions markdown_extensions: From 29be38890475fd79481d0f6883783f042ebac376 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 20 May 2024 19:16:14 -0400 Subject: [PATCH 03/48] Update 16_account.md --- docs/16_account.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/16_account.md b/docs/16_account.md index 61aac25d..63516c2a 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -2,7 +2,7 @@ A DANDI account is only required for specific features. Without a DANDI account, you can search, view, and download files. -With a DANDI account, you can additionally create Dandisets and access the DANDI Hub to analyze existing data. +With a DANDI account, you can additionally create and edit Dandisets and access the DANDI Hub to analyze existing data. ## Instructions @@ -10,4 +10,4 @@ With a DANDI account, you can additionally create Dandisets and access the DANDI 1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). 1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours. - **note**: Requests from new GitHub accounts and from emails that are not an `.edu` domain might take longer to review and are more likely to be rejected, especially if there are no plans described to upload data to the archive. -1. If your request for an account is approved, you will be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file +1. If your request for an account is approved, you will be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. From 4cfef62399db75854257f21983fb16bfdb4ed3b9 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 20 May 2024 19:40:27 -0400 Subject: [PATCH 04/48] add details about intended users --- docs/16_account.md | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/docs/16_account.md b/docs/16_account.md index 61aac25d..d6e14266 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -1,13 +1,27 @@ # Create a DANDI Account -A DANDI account is only required for specific features. -Without a DANDI account, you can search, view, and download files. -With a DANDI account, you can additionally create Dandisets and access the DANDI Hub to analyze existing data. +A DANDI account enhances your capabilities within the DANDI Archive. +Without an account, users can freely search, view, and download available datasets. +With an account, users can create and edit Dandisets, and use the DANDI Hub to analyze data. -## Instructions +DANDI provides two servers: +- **Main server**: [https://dandiarchive.org/](https://dandiarchive.org/) - This is the primary platform for most users. +- **Staging server**: [https://gui-staging.dandiarchive.org/](https://gui-staging.dandiarchive.org/) - Ideal for training and testing purposes. -1. To create a DANDI account, first [create a GitHub account](https://github.com/) if you don't have one. -1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). -1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours. - - **note**: Requests from new GitHub accounts and from emails that are not an `.edu` domain might take longer to review and are more likely to be rejected, especially if there are no plans described to upload data to the archive. -1. If your request for an account is approved, you will be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file +Accounts are independently managed on each server, allowing users to register on one or both, depending on their testing and deployment needs. + +DANDI is freely accessible to the neuroscience research community. +Membership is usually granted automatically to GitHub accounts with a `.edu` or similar academic email. +If your registration is denied: +- With an academic email not linked to your GitHub, please contact [help@dandiarchive.org](mailto:help@dandiarchive.org) for assistance using this email address. +- Without an academic email, account approval is still possible under specific circumstances. Appeal decisions at [help@dandiarchive.org](mailto:help@dandiarchive.org). + +## How to Register for a DANDI Account + +1. **Create a GitHub Account**: If not already a GitHub user, [sign up here](https://github.com/). +2. **Register on DANDI**: Navigate to the DANDI homepage and click the `LOG IN WITH GITHUB` button to register using your GitHub account. +3. **Confirmation of Review**: Post-registration, you will receive an email confirming that your account is under review. Your request will be reviewed within 24 hours. + - **Note**: Reviews may extend beyond 24 hours for new GitHub accounts or non-.edu email addresses, particularly if the registration does not describe immediate plans to contribute data. +4. **Accessing DANDI**: Upon approval, access DANDI by logging in through the `LOG IN WITH GITHUB` button. + +For support or further inquiries, reach out to [help@dandiarchive.org](mailto:help@dandiarchive.org). From 763d7bd53a777a5a27c7a44809833176dd9ad6ae Mon Sep 17 00:00:00 2001 From: "John T. Wodder II" Date: Thu, 27 Jun 2024 08:56:09 -0400 Subject: [PATCH 05/48] Document location of encrypted keyfile --- docs/13_upload.md | 50 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 13 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 25d70af2..3d56f902 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -109,21 +109,45 @@ There are two options for storing your DANDI access credentials. and a plaintext (unencrypted) keyfile. - Specifying the `keyring` backend - - You can set the backend the `keyring` library uses either by setting the - `PYTHON_KEYRING_BACKEND` environment variable or by filling in the `keyring` - library's [configuration file](https://github.com/jaraco/keyring#configuring). - - IDs for the available backends can be listed by running `keyring --list`. - - If no backend is specified in this way, the library will use the available - backend with the highest priority. - - If the DANDI CLI encounters an error while attempting to fetch the API key - from the default keyring backend, it will fall back to using an encrypted - keyfile (the `keyrings.alt.file.EncryptedKeyring` backend). If the keyfile - does not already exist, the CLI will ask you for confirmation; if you answer - "yes," the `keyring` configuration file (if it does not already exist; see - above) will be configured to use `EncryptedKeyring` as the default backend. - If you answer "no," the CLI will exit with an error, and you must store the + - You can set the backend the `keyring` library uses either by setting + the `PYTHON_KEYRING_BACKEND` environment variable or by filling in + the `keyring` library's [configuration + file](https://github.com/jaraco/keyring#configuring). + + - IDs for the available backends can be listed by running `keyring + --list`. + + - If no backend is specified in this way, the library will use the + available backend with the highest priority. + + - If the DANDI CLI encounters an error while attempting to fetch the + API key from the default keyring backend, it will fall back to using + an encrypted keyfile (the `keyrings.alt.file.EncryptedKeyring` + backend). If the keyfile does not already exist, the CLI will ask + you for confirmation; if you answer "yes," the `keyring` + configuration file (if it does not already exist; see above) will be + configured to use `EncryptedKeyring` as the default backend. If you + answer "no," the CLI will exit with an error, and you must store the API key somewhere accessible to the CLI on your own. + - Unless a different location is set via the `keyring` + configuration file, the encrypted keyfile will be located at the + following path: + + - On Linux and macOS, if the `XDG_DATA_HOME` environment + variable is set to a nonempty string, the keyfile will be at + `$XDG_DATA_HOME/python_keyring/crypted_pass.cfg`; otherwise, + it will be at + `~/.local/share/python_keyring/crypted_pass.cfg`. + + - On Windows, if the `LOCALAPPDATA` environment variable is + set, the keyfile will be at `%LOCALAPPDATA%\Python + Keyring\crypted_pass.cfg`; otherwise, if the `ProgramData` + environment variable is set, the keyfile will be at + `%ProgramData%\Python Keyring\crypted_pass.cfg`; otherwise, + it will be at `Python Keyring\crypted_pass.cfg` within the + current directory. + - Storing the API key with `keyring` 1. You can store your API key where the `keyring` library can find it by using the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the From 0b63a6408031b6dc644bbd01193986d3c5627479 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 1 Jul 2024 15:56:08 -0500 Subject: [PATCH 06/48] Add `Real-time Status` to left nagivation menu --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index 6d577897..dd4dffec 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -39,6 +39,7 @@ nav: - REST API Swagger: https://api.dandiarchive.org/swagger - REST API Redoc: https://api.dandiarchive.org/redoc - DANDI Hub: "50_hub.md" + - Real-time Status: https://www.dandiarchive.org/upptime - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" From e0d7ed5a4b2022373f42715a115079f4f2570f6c Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 1 Jul 2024 16:01:27 -0500 Subject: [PATCH 07/48] Update heading --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index dd4dffec..ffa4f0fa 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -39,7 +39,7 @@ nav: - REST API Swagger: https://api.dandiarchive.org/swagger - REST API Redoc: https://api.dandiarchive.org/redoc - DANDI Hub: "50_hub.md" - - Real-time Status: https://www.dandiarchive.org/upptime + - Real-Time Status: https://www.dandiarchive.org/upptime - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" From dabf2b424f1b564f18b6b3e915a31925403b96ef Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 1 Jul 2024 17:01:27 -0500 Subject: [PATCH 08/48] Update mkdocs.yml Co-authored-by: Yaroslav Halchenko --- mkdocs.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index ffa4f0fa..180fb29e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -39,7 +39,9 @@ nav: - REST API Swagger: https://api.dandiarchive.org/swagger - REST API Redoc: https://api.dandiarchive.org/redoc - DANDI Hub: "50_hub.md" - - Real-Time Status: https://www.dandiarchive.org/upptime + - Health Status: + - Dandisets: https://github.com/dandi/dandisets-healthstatus + - Services: https://www.dandiarchive.org/upptime - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" From 0f78f7739e03d35453ea71921fdc3d8e9fa58f6b Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Sun, 7 Jul 2024 07:34:28 -1000 Subject: [PATCH 09/48] Add sections on DataLad and WebDAV access in download chapter --- docs/12_download.md | 57 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/docs/12_download.md b/docs/12_download.md index dcf6bd4e..24f0de50 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -1,12 +1,12 @@ # Downloading Data and Dandisets -You can download the content of a Dandiset using the DANDI Web application (such a specific file) or entire +You can download the content of a Dandiset using the DANDI Web application (such a specific file) or entire Dandisets using the DANDI Python CLI. ## Using the DANDI Web Application Once you have the Dandiset you are interested in (see more in [the Dandiset View section](./11_view.md)), you can download the content of the Dandiset. -On the landing page of each Dandiset, you can find `Download` button on the right-hand panel. After clicking the +On the landing page of each Dandiset, you can find `Download` button on the right-hand panel. After clicking the button, you will see the specific command you can use with DANDI Python CLI (as well as the information on how to download the CLI). Date: Sun, 7 Jul 2024 14:14:47 -0400 Subject: [PATCH 10/48] Wording and formatting tune ups from review Co-authored-by: John T. Wodder II --- docs/12_download.md | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/12_download.md b/docs/12_download.md index 24f0de50..cb765775 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -69,30 +69,31 @@ an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of support ## Using DataLad -All dandisets are regularly mirrored to DataLad datasets which are made available from https://github.com/dandisets GitHub organization. -Where present, individual [Zarr](https://zarr.dev/) files are included as subdatasets ([git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules)) from the https://github.com/dandizarrs/ GitHub organization. +All dandisets are regularly mirrored to DataLad datasets which are made available at the GitHub organization https://github.com/dandisets. +Where present, individual [Zarr](https://zarr.dev/) files are included as subdatasets ([git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules)) hosted in the GitHub organization . -Dandiset releases are tagged with Git tags and a history of modifications is available at the time intervals of execution of the mirroring job. +The Git revision histories of each dataset reflect the Dandiset's draft state as of each execution of the mirroring job. +Published Dandiset versions are tagged with Git tags. -With DataLad you can -- clone the entire dataset, +With DataLad, you can: +- clone an entire dataset, - use a specific version of it, - explore history of modifications, -- download content of the files of interest, -- drop the content of no longer needed files, +- download content of files of interest, +- locally discard the content of no-longer-needed files, - use the dataset in a reproducible manner, - include it as a subdataset in your own DataLad dataset, -- use https://github.com/datalad/datalad-fuse/ to FUSE mount individual dandisets you cloned locally so that files content is made transparently streamed to your DANDI/DataLad unaware tools, +- use https://github.com/datalad/datalad-fuse/ to [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)-mount individual locally-cloned dandisets so that their files' contents are transparently streamed to your DANDI/DataLad-unaware tools, - etc. -Learn more about DataLad from its handbook at https://handbook.datalad.org/ . +Learn more about DataLad from its handbook at . -**Developers note:** DataLad datasets are created using [dandi/backups2datalad](https://github.com/dandi/backups2datalad/) tool which is also available for use by the community for independent DANDI deployments to similarly maintain mirror of DataLad datasets. +**Developers' note:** DataLad datasets are created using the [dandi/backups2datalad](https://github.com/dandi/backups2datalad/) tool which is also available for use by the community to similarly maintain mirrors of independent DANDI deployments as DataLad datasets. ## Using WebDAV -DANDI provides WebDAV service at https://webdav.dandiarchive.org/ to access the data in the DANDI archive. -You can use any WebDAV client or simply a browser to access the data - any dandiset, any version, any file or collection of them. +DANDI provides a [WebDAV](https://en.wikipedia.org/wiki/WebDAV) service at https://webdav.dandiarchive.org/ for accessing the data in the DANDI archive. +You can use any WebDAV client or even a web browser to access the data - any dandiset, any version, any file or collection of files. You can use any web download tool to download the data from the DANDI archive, e.g. ````commandline @@ -101,7 +102,7 @@ wget -r -np -nH --cut-dirs=3 https://webdav.dandiarchive.org/dandisets/000027/re for a download of a specific release `0.210831.2033` of the `000027` dandiset. -**Note:** WebDAV service does not provide any data, and relies on redirects to AWS S3 storage where the data is stored. -You might need to configure your WebDAV client to follow redirects, e.g. set `follow_redirect` to `1` in `/etc/davfs2/davfs2.conf` for [davfs2](https://savannah.nongnu.org/projects/davfs2) WebDAV client. +**Note:** The WebDAV service does not directly serve any file contents; it instead relies on redirects to AWS S3 storage where the contents are stored. +You might need to configure your WebDAV client to follow redirects; e.g., for the [davfs2](https://savannah.nongnu.org/projects/davfs2) WebDAV client, set `follow_redirect` to `1` in `/etc/davfs2/davfs2.conf`. -**Developers note:** WebDAV service code is available at https://github.com/dandi/dandidav/ and can also be used for independent DANDI deployments. +**Developers' note:** The WebDAV service's code is available at https://github.com/dandi/dandidav/ and can also be used for independent DANDI deployments. From 645d1ab3308b1dea653d95b95c02c5cf7c2ae6ab Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 9 Jul 2024 14:25:13 -0400 Subject: [PATCH 11/48] fix indent formatting --- docs/16_account.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/16_account.md b/docs/16_account.md index f913df4e..8479f559 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -5,6 +5,7 @@ Without an account, users can freely search, view, and download available datase With an account, users can create and edit Dandisets, and use the DANDI Hub to analyze data. DANDI provides two servers: + - **Main server**: [https://dandiarchive.org/](https://dandiarchive.org/) - This is the primary platform for most users. - **Staging server**: [https://gui-staging.dandiarchive.org/](https://gui-staging.dandiarchive.org/) - Ideal for training and testing purposes. @@ -13,6 +14,7 @@ Accounts are independently managed on each server, allowing users to register on DANDI is freely accessible to the neuroscience research community. Membership is usually granted automatically to GitHub accounts with a `.edu` or similar academic email. If your registration is denied: + - With an academic email not linked to your GitHub, please contact [help@dandiarchive.org](mailto:help@dandiarchive.org) for assistance using this email address. - Without an academic email, account approval is still possible under specific circumstances. Appeal decisions at [help@dandiarchive.org](mailto:help@dandiarchive.org). From ff63c567144ba2672b46dd27cbae72e63ae525e4 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 10:45:02 -0400 Subject: [PATCH 12/48] update upload page to point to the NWB GUIDE for conversion and for upload --- docs/13_upload.md | 144 ++++++++++++++++++++++------------------------ 1 file changed, 68 insertions(+), 76 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 3d56f902..bf13ca22 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -1,91 +1,83 @@ # Creating Dandisets and Uploading Data -To create a new Dandiset and upload your data, you need to have a DANDI account. +This page provides instructions for creating a new Dandiset and uploading data to DANDI. + +## **Prerequisites** +1. **Convert data to NWB.** Your data should already be in NWB format (2.1+). We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. + This step can be complex depending on your data. Consider using the following tools: + 1. [NWB Graphical User Interface for Data Entry (GUIDE)](https://nwb-guide.readthedocs.io/en/stable/) is a cross-platform desktop application for converting data from common proprietary formats to NWB and uploading it to DANDI. + 2. [NeuroConv](https://neuroconv.readthedocs.io/) is a Python library that automates conversion to NWB from a variety of popular formats. See the [Conversion Gallery](https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/index.html) for example conversion scripts. + 3. [PyNWB](https://pynwb.readthedocs.io/en/stable/) and [MatNWB](https://github.com/NeurodataWithoutBorders/matnwb) are APIs in Python and MATLAB that allow full flexibility in reading and writing data. ([PyNWB tutorials](https://pynwb.readthedocs.io/en/stable/tutorials/index.html), [MatNWB tutorials](https://github.com/NeurodataWithoutBorders/matnwb?tab=readme-ov-file#tutorials)) + 4. [NWB Overview Docs](https://nwb-overview.readthedocs.io) points to more tools helpful for working with NWB files. + + Feel free to [reach out to us for help](https://github.com/dandi/helpdesk/discussions). + +1. **Choose a server.** + - **Production server**: https://dandiarchive.org. This is the main server for DANDI and should be used for sharing neuroscience data. + When you create a Dandiset, a permanent ID is automatically assigned to it. + This Dandiset can be fully public or embargoed according to NIH policy. + All data are uploaded as draft and can be adjusted before publishing on the production server. + - **Development server**: https://gui-staging.dandiarchive.org. This server is for testing and learning how to use DANDI. + It is not recommended for sharing data, but is recommended for testing the DANDI CLI and GUI or as a testing platform for developers. + Note that the development server should not be used to stage your data. + + The below instructions will alert you to where the commands for interacting with these two different servers differ slightly. +1. **Register for DANDI and copy the API key.** To create a new Dandiset and upload your data, you need to have a DANDI account. + * If you do not already have an account, see [Create a DANDI Account](./16_account.md) page for instructions. + * Once you are logged in, copy your API key. + Click on your user initials in the top-right corner after logging in. + Production (https://dandiarchive.org) and staging (https://gui-staging.dandiarchive.org) servers have different API keys and different logins. + * Store your API key somewhere that the CLI can find it; see ["Storing Access Credentials"](#storing-access-credentials) below. -## Create a Dandiset and Add Data - -You can create a new Dandiset at https://dandiarchive.org. This Dandiset can be fully -public or embargoed -according to NIH policy. -When you create a Dandiset, a permanent ID is automatically assigned to it. -To prevent the production server from being inundated with test Dandisets, we encourage developers to develop -against the development server (https://gui-staging.dandiarchive.org/). Note -that the development server -should not be used to stage your data. All data are uploaded as draft and can be adjusted before publishing on -the production server. The development server is primarily used by users learning to use DANDI or by developers. - -The below instructions will alert you to where the commands for interacting with these -two different servers differ slightly. +### **Data upload/management workflow** -### **Setup** +The NWB GUIDE provides a graphical interface for inspecting and validating NWB files, as well as for uploading data to +DANDI. See the [NWB GUIDE Dataset Publication Tutorial](https://nwb-guide.readthedocs.io/en/latest/tutorials/dataset_publication.html) for more information. -1. To create a new Dandiset and upload your data, you need to have a DANDI account. See the [Create a DANDI Account](./16_account.md) page. -1. Log in to DANDI and copy your API key. Click on your user initials in the - top-right corner after logging in. Production (dandiarchive.org) and staging (gui-staging.dandiarchive.org) servers - have different API keys and different logins. -1. Locally: - 1. Create a Python environment. This is not required, but strongly recommended; e.g. [miniconda](https://conda. - io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands), - [virtualenv](https://docs.python.org/3/library/venv.html). - 2. Install the DANDI CLI into your Python environment: +The below instructions show how to do the same thing programmatically using the command line interface (CLI). +The CLI approach may be more suitable for users who are comfortable with the command line or who need to automate the process, or for advanced use-cases. - pip install -U dandi +1. **Create a new Dandiset.** + * Click `NEW DANDISET` in the Web application (top right corner) after logging in. + * You will be asked to enter basic metadata: a name (title) and description (abstract) for your dataset. + * After you provide a name and description, the dataset identifier will be created; we will call this ``. +1. Check your files for [NWB Best Practices](https://nwbinspector.readthedocs.io/en/dev/best_practices/best_practices_index.html) with [NWBInspector](https://nwbinspector.readthedocs.io/en/dev/user_guide/user_guide_index.html). + Run NWB Inspector programmatically. Install the Python library (`pip install -U nwbinspector`) and run: - 3. Store your API key somewhere that the CLI can find it; see ["Storing - Access Credentials"](#storing-access-credentials) below. + nwbinspector --config dandi + + If the report is too large to efficiently navigate in your console, you can save a report using -### **Data upload/management workflow** + nwbinspector --config dandi --report-file-path .txt + + For more details and other options, run: + + nwbinspector --help -1. Register a Dandiset to generate an identifier. You will be asked to enter - basic metadata: a name (title) and description (abstract) for your dataset. - Click `NEW DANDISET` in the Web application (top right corner) after logging in. - After you provide a name and description, the dataset identifier will be created; - we will call this ``. -1. NWB format: - 1. Convert your data to NWB 2.1+ in a local folder. Let's call this ``. - We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. - This step can be complex depending on your data. - [NeuroConv](https://neuroconv.readthedocs.io/) automates - conversion to NWB from a variety of popular formats. - [nwb-overview.readthedocs.io](https://nwb-overview.readthedocs.io) - points to more tools helpful for working with NWB files, and [BIDS - converters](https://bids.neuroimaging.io/benefits.html#converters) - if you are preparing a BIDS dataset containing NWB files. - Feel free to [reach out to us for help](https://github.com/dandi/helpdesk/discussions). - 2. Check your files for [NWB Best Practices](https://nwbinspector.readthedocs.io/en/dev/best_practices/best_practices_index.html) by installing - the [NWBInspector](https://nwbinspector.readthedocs.io/en/dev/user_guide/user_guide_index.html) (`pip install -U nwbinspector`) and running - - nwbinspector --config dandi - - 3. Thoroughly read the NWBInspector report and try to address as many issues as possible. **DANDI will prevent validation and upload of any issues - labeled as level 'CRITICAL' or above when using the `--config dandi` option.** + Thoroughly read the NWBInspector report and try to address as many issues as possible. + **DANDI will prevent validation and upload of any issues labeled as level 'CRITICAL' or above when using the `--config dandi` option.** See ["Validation Levels for NWB Files"](./135_validation.md) for more information about validation criteria for uploading NWB - files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. - Note that some autodetected violations, such as `check_data_orientation`, may be safely ignored in the event - that the data is confirmed to be in the correct form; this can be done using either the `--ignore ` flag or a config file. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more details and other options, or type `nwbinspector --help`. - If the report is too large to efficiently navigate in your console, you can save a report using - - nwbinspector --config dandi --report-file-path .txt - - 4. Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: `dandi validate --ignore DANDI.NO_DANDISET_FOUND `. - **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** - 5. Now, prepare and fully validate again within the dandiset folder used for upload: - - dandi download https://dandiarchive.org/dandiset//draft - cd - dandi organize -f dry - dandi organize - dandi validate . - dandi upload - - Note that the `organize` steps should not be used if you are preparing a BIDS dataset with the NWB files. - Uploading to the development server is controlled via `-i` option, e.g. - `dandi upload -i dandi-staging`. - Note that validation is also done during `upload`, but ensuring compliance using `validate` prior upload helps avoid interruptions of the lengthier upload process due to validation failures. - 6. Add metadata by visiting your Dandiset landing page: - `https://dandiarchive.org/dandiset//draft` and clicking on the `METADATA` link. + files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event + that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. +1. Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: `dandi validate --ignore DANDI.NO_DANDISET_FOUND `. + **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** +1. Upload the data to DANDI. This can either be done through the NWB GUIDE, or programmatically: + + dandi download https://dandiarchive.org/dandiset//draft + cd + dandi organize -f dry + dandi organize + dandi validate . + dandi upload + + Note that the `organize` steps should not be used if you are preparing a BIDS dataset with the NWB files. + Uploading to the development server is controlled via `-i` option, e.g. + `dandi upload -i dandi-staging`. + Note that validation is also done during `upload`, but ensuring compliance using `validate` prior upload helps avoid interruptions of the lengthier upload process due to validation failures. +1. Add metadata by visiting your Dandiset landing page: + `https://dandiarchive.org/dandiset//draft` and clicking on the `METADATA` link. If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md). From 59bdee3d14fcad1e80f3fdaecedceee072167d11 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 10:46:31 -0400 Subject: [PATCH 13/48] language --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index bf13ca22..c54454da 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -3,7 +3,7 @@ This page provides instructions for creating a new Dandiset and uploading data to DANDI. ## **Prerequisites** -1. **Convert data to NWB.** Your data should already be in NWB format (2.1+). We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. +1. **Convert data to NWB.** You should start by converting your data to NWB format (2.1+). We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. This step can be complex depending on your data. Consider using the following tools: 1. [NWB Graphical User Interface for Data Entry (GUIDE)](https://nwb-guide.readthedocs.io/en/stable/) is a cross-platform desktop application for converting data from common proprietary formats to NWB and uploading it to DANDI. 2. [NeuroConv](https://neuroconv.readthedocs.io/) is a Python library that automates conversion to NWB from a variety of popular formats. See the [Conversion Gallery](https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/index.html) for example conversion scripts. From 825834281f74d10e262a1179d2584476d4125213 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 10:48:45 -0400 Subject: [PATCH 14/48] formatting --- docs/13_upload.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index c54454da..a4506cac 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -41,8 +41,8 @@ The CLI approach may be more suitable for users who are comfortable with the com * Click `NEW DANDISET` in the Web application (top right corner) after logging in. * You will be asked to enter basic metadata: a name (title) and description (abstract) for your dataset. * After you provide a name and description, the dataset identifier will be created; we will call this ``. -1. Check your files for [NWB Best Practices](https://nwbinspector.readthedocs.io/en/dev/best_practices/best_practices_index.html) with [NWBInspector](https://nwbinspector.readthedocs.io/en/dev/user_guide/user_guide_index.html). - Run NWB Inspector programmatically. Install the Python library (`pip install -U nwbinspector`) and run: +1. **Check your files for [NWB Best Practices](https://nwbinspector.readthedocs.io/en/dev/best_practices/best_practices_index.html).** + Run [NWB Inspector](https://nwbinspector.readthedocs.io/en/dev/user_guide/user_guide_index.html) programmatically. Install the Python library (`pip install -U nwbinspector`) and run: nwbinspector --config dandi @@ -61,9 +61,9 @@ The CLI approach may be more suitable for users who are comfortable with the com uploading NWB files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. -1. Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: `dandi validate --ignore DANDI.NO_DANDISET_FOUND `. +1. **Validate NWB files.** Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: `dandi validate --ignore DANDI.NO_DANDISET_FOUND `. **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** -1. Upload the data to DANDI. This can either be done through the NWB GUIDE, or programmatically: +1. **Upload the data to DANDI.** This can either be done through the NWB GUIDE, or programmatically: dandi download https://dandiarchive.org/dandiset//draft cd @@ -76,8 +76,8 @@ The CLI approach may be more suitable for users who are comfortable with the com Uploading to the development server is controlled via `-i` option, e.g. `dandi upload -i dandi-staging`. Note that validation is also done during `upload`, but ensuring compliance using `validate` prior upload helps avoid interruptions of the lengthier upload process due to validation failures. -1. Add metadata by visiting your Dandiset landing page: - `https://dandiarchive.org/dandiset//draft` and clicking on the `METADATA` link. +1. **Add metadata to Dandiset.** Visit your Dandiset landing page: + `https://dandiarchive.org/dandiset//draft` and click on the `METADATA` link. If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md). From b2eb9dd8bb4b2f2be6933ecd547f2554c9ec2cc4 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 18:20:24 -0400 Subject: [PATCH 15/48] improve formatting --- docs/13_upload.md | 51 +++++++++++++++++++++++++---------------------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index a4506cac..3e7d15ed 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -5,14 +5,15 @@ This page provides instructions for creating a new Dandiset and uploading data t ## **Prerequisites** 1. **Convert data to NWB.** You should start by converting your data to NWB format (2.1+). We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. This step can be complex depending on your data. Consider using the following tools: - 1. [NWB Graphical User Interface for Data Entry (GUIDE)](https://nwb-guide.readthedocs.io/en/stable/) is a cross-platform desktop application for converting data from common proprietary formats to NWB and uploading it to DANDI. - 2. [NeuroConv](https://neuroconv.readthedocs.io/) is a Python library that automates conversion to NWB from a variety of popular formats. See the [Conversion Gallery](https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/index.html) for example conversion scripts. - 3. [PyNWB](https://pynwb.readthedocs.io/en/stable/) and [MatNWB](https://github.com/NeurodataWithoutBorders/matnwb) are APIs in Python and MATLAB that allow full flexibility in reading and writing data. ([PyNWB tutorials](https://pynwb.readthedocs.io/en/stable/tutorials/index.html), [MatNWB tutorials](https://github.com/NeurodataWithoutBorders/matnwb?tab=readme-ov-file#tutorials)) - 4. [NWB Overview Docs](https://nwb-overview.readthedocs.io) points to more tools helpful for working with NWB files. - Feel free to [reach out to us for help](https://github.com/dandi/helpdesk/discussions). + 1. **[NWB Graphical User Interface for Data Entry (GUIDE)](https://nwb-guide.readthedocs.io/en/stable/)** is a cross-platform desktop application for converting data from common proprietary formats to NWB and uploading it to DANDI. + 2. **[NeuroConv](https://neuroconv.readthedocs.io/)** is a Python library that automates conversion to NWB from a variety of popular formats. See the [Conversion Gallery](https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/index.html) for example conversion scripts. + 3. **[PyNWB](https://pynwb.readthedocs.io/en/stable/)** and **[MatNWB](https://github.com/NeurodataWithoutBorders/matnwb)** are APIs in Python and MATLAB that allow full flexibility in reading and writing data. ([PyNWB tutorials](https://pynwb.readthedocs.io/en/stable/tutorials/index.html), [MatNWB tutorials](https://github.com/NeurodataWithoutBorders/matnwb?tab=readme-ov-file#tutorials)) + 4. **[NWB Overview Docs](https://nwb-overview.readthedocs.io)** points to more tools helpful for working with NWB files. -1. **Choose a server.** + Feel free to [reach out to us for help](https://github.com/dandi/helpdesk/discussions). + +2. **Choose a server.** - **Production server**: https://dandiarchive.org. This is the main server for DANDI and should be used for sharing neuroscience data. When you create a Dandiset, a permanent ID is automatically assigned to it. This Dandiset can be fully public or embargoed according to NIH policy. @@ -21,38 +22,39 @@ This page provides instructions for creating a new Dandiset and uploading data t It is not recommended for sharing data, but is recommended for testing the DANDI CLI and GUI or as a testing platform for developers. Note that the development server should not be used to stage your data. - The below instructions will alert you to where the commands for interacting with these two different servers differ slightly. -1. **Register for DANDI and copy the API key.** To create a new Dandiset and upload your data, you need to have a DANDI account. - * If you do not already have an account, see [Create a DANDI Account](./16_account.md) page for instructions. - * Once you are logged in, copy your API key. - Click on your user initials in the top-right corner after logging in. - Production (https://dandiarchive.org) and staging (https://gui-staging.dandiarchive.org) servers have different API keys and different logins. - * Store your API key somewhere that the CLI can find it; see ["Storing Access Credentials"](#storing-access-credentials) below. + The below instructions will alert you to where the commands for interacting with these two different servers differ slightly. + +3. **Register for DANDI and copy the API key.** To create a new Dandiset and upload your data, you need to have a DANDI account. + * If you do not already have an account, see [Create a DANDI Account](./16_account.md) page for instructions. + * Once you are logged in, copy your API key. + Click on your user initials in the top-right corner after logging in. + Production (https://dandiarchive.org) and staging (https://gui-staging.dandiarchive.org) servers have different API keys and different logins. + * Store your API key somewhere that the CLI can find it; see ["Storing Access Credentials"](#storing-access-credentials) below. -### **Data upload/management workflow** +## **Data upload/management workflow** The NWB GUIDE provides a graphical interface for inspecting and validating NWB files, as well as for uploading data to -DANDI. See the [NWB GUIDE Dataset Publication Tutorial](https://nwb-guide.readthedocs.io/en/latest/tutorials/dataset_publication.html) for more information. +DANDI. See the **[NWB GUIDE Dataset Publication Tutorial](https://nwb-guide.readthedocs.io/en/latest/tutorials/dataset_publication.html)** for more information. The below instructions show how to do the same thing programmatically using the command line interface (CLI). The CLI approach may be more suitable for users who are comfortable with the command line or who need to automate the process, or for advanced use-cases. 1. **Create a new Dandiset.** - * Click `NEW DANDISET` in the Web application (top right corner) after logging in. - * You will be asked to enter basic metadata: a name (title) and description (abstract) for your dataset. - * After you provide a name and description, the dataset identifier will be created; we will call this ``. + * Click `NEW DANDISET` in the Web application (top right corner) after logging in. + * You will be asked to enter basic metadata: a name (title) and description (abstract) for your dataset. + * After you provide a name and description, the dataset identifier will be created; we will call this ``. 1. **Check your files for [NWB Best Practices](https://nwbinspector.readthedocs.io/en/dev/best_practices/best_practices_index.html).** Run [NWB Inspector](https://nwbinspector.readthedocs.io/en/dev/user_guide/user_guide_index.html) programmatically. Install the Python library (`pip install -U nwbinspector`) and run: - nwbinspector --config dandi - - If the report is too large to efficiently navigate in your console, you can save a report using + nwbinspector --config dandi + + If the report is too large to efficiently navigate in your console, you can save a report using - nwbinspector --config dandi --report-file-path .txt + nwbinspector --config dandi --report-file-path .txt - For more details and other options, run: + For more details and other options, run: - nwbinspector --help + nwbinspector --help Thoroughly read the NWBInspector report and try to address as many issues as possible. **DANDI will prevent validation and upload of any issues labeled as level 'CRITICAL' or above when using the `--config dandi` option.** @@ -61,6 +63,7 @@ The CLI approach may be more suitable for users who are comfortable with the com uploading NWB files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. + 1. **Validate NWB files.** Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: `dandi validate --ignore DANDI.NO_DANDISET_FOUND `. **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** 1. **Upload the data to DANDI.** This can either be done through the NWB GUIDE, or programmatically: From a8f2bc9046fdefe840a4a55d54c83b98897b9f7c Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 23:18:20 -0400 Subject: [PATCH 16/48] Update docs/13_upload.md Co-authored-by: Kabilar Gunalan --- docs/13_upload.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 3e7d15ed..fed2331e 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -64,7 +64,8 @@ The CLI approach may be more suitable for users who are comfortable with the com files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. -1. **Validate NWB files.** Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: `dandi validate --ignore DANDI.NO_DANDISET_FOUND `. +1. **Validate NWB files.** Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: + dandi validate --ignore DANDI.NO_DANDISET_FOUND **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** 1. **Upload the data to DANDI.** This can either be done through the NWB GUIDE, or programmatically: From baff8ee230f415d2d8499a77c6a6ca8f7eff9c10 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 23:18:33 -0400 Subject: [PATCH 17/48] Update docs/13_upload.md Co-authored-by: Kabilar Gunalan --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index fed2331e..c4be78c5 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -79,7 +79,7 @@ The CLI approach may be more suitable for users who are comfortable with the com Note that the `organize` steps should not be used if you are preparing a BIDS dataset with the NWB files. Uploading to the development server is controlled via `-i` option, e.g. `dandi upload -i dandi-staging`. - Note that validation is also done during `upload`, but ensuring compliance using `validate` prior upload helps avoid interruptions of the lengthier upload process due to validation failures. + Note that validation is also done during `upload`, but ensuring compliance using `validate` prior to upload helps avoid interruptions of the lengthier upload process due to validation failures. 1. **Add metadata to Dandiset.** Visit your Dandiset landing page: `https://dandiarchive.org/dandiset//draft` and click on the `METADATA` link. From a49002bcbf1b2a2e20528f91f574c6cc45422c90 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 23:18:46 -0400 Subject: [PATCH 18/48] Update docs/13_upload.md Co-authored-by: Kabilar Gunalan --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index c4be78c5..fbd5e3dc 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -83,7 +83,7 @@ The CLI approach may be more suitable for users who are comfortable with the com 1. **Add metadata to Dandiset.** Visit your Dandiset landing page: `https://dandiarchive.org/dandiset//draft` and click on the `METADATA` link. -If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md). +If you have an issue using the DANDI Client, see the [DANDI Debugging section](./15_debugging.md). ## Storing Access Credentials From 032f947d6880730f1305ab807858ab61875b4bb7 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 23:18:54 -0400 Subject: [PATCH 19/48] Update docs/13_upload.md Co-authored-by: Kabilar Gunalan --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index fbd5e3dc..299f6b10 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -80,7 +80,7 @@ The CLI approach may be more suitable for users who are comfortable with the com Uploading to the development server is controlled via `-i` option, e.g. `dandi upload -i dandi-staging`. Note that validation is also done during `upload`, but ensuring compliance using `validate` prior to upload helps avoid interruptions of the lengthier upload process due to validation failures. -1. **Add metadata to Dandiset.** Visit your Dandiset landing page: +1. **Add metadata to the Dandiset.** Visit your Dandiset landing page: `https://dandiarchive.org/dandiset//draft` and click on the `METADATA` link. If you have an issue using the DANDI Client, see the [DANDI Debugging section](./15_debugging.md). From 17b9804c1469c4f40976bfac812216d725fca78e Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 23:18:59 -0400 Subject: [PATCH 20/48] Update docs/13_upload.md Co-authored-by: Kabilar Gunalan --- docs/13_upload.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/13_upload.md b/docs/13_upload.md index 299f6b10..60118120 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -64,6 +64,8 @@ The CLI approach may be more suitable for users who are comfortable with the com files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. +1. **Install the [DANDI Client](https://pypi.org/project/dandi/).** + pip install -U dandi 1. **Validate NWB files.** Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: dandi validate --ignore DANDI.NO_DANDISET_FOUND **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** From f769d5dc87591f752c3e3f7f56845d69f05fa9d6 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 18 Jul 2024 23:29:43 -0400 Subject: [PATCH 21/48] improve formatting --- docs/13_upload.md | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 60118120..02667235 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -2,7 +2,7 @@ This page provides instructions for creating a new Dandiset and uploading data to DANDI. -## **Prerequisites** +## Prerequisites 1. **Convert data to NWB.** You should start by converting your data to NWB format (2.1+). We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. This step can be complex depending on your data. Consider using the following tools: @@ -31,7 +31,7 @@ This page provides instructions for creating a new Dandiset and uploading data t Production (https://dandiarchive.org) and staging (https://gui-staging.dandiarchive.org) servers have different API keys and different logins. * Store your API key somewhere that the CLI can find it; see ["Storing Access Credentials"](#storing-access-credentials) below. -## **Data upload/management workflow** +## Data upload/management workflow The NWB GUIDE provides a graphical interface for inspecting and validating NWB files, as well as for uploading data to DANDI. See the **[NWB GUIDE Dataset Publication Tutorial](https://nwb-guide.readthedocs.io/en/latest/tutorials/dataset_publication.html)** for more information. @@ -56,19 +56,22 @@ The CLI approach may be more suitable for users who are comfortable with the com nwbinspector --help - Thoroughly read the NWBInspector report and try to address as many issues as possible. - **DANDI will prevent validation and upload of any issues labeled as level 'CRITICAL' or above when using the `--config dandi` option.** - See - ["Validation Levels for NWB Files"](./135_validation.md) for more information about validation criteria for - uploading NWB - files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event - that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. + Thoroughly read the NWBInspector report and try to address as many issues as possible. + **DANDI will prevent validation and upload of any issues labeled as level 'CRITICAL' or above when using the `--config dandi` option.** + See ["Validation Levels for NWB Files"](./135_validation.md) for more information about validation criteria for + uploading NWB files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some auto-detected violations, such as `check_data_orientation`, may be safely ignored in the event + that the data is confirmed to be in the correct form. See [the NWBInspector CLI documentation](https://nwbinspector.readthedocs.io/en/dev/user_guide/using_the_command_line_interface.html) for more information. 1. **Install the [DANDI Client](https://pypi.org/project/dandi/).** + pip install -U dandi -1. **Validate NWB files.** Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running: + +1. **Validate NWB files.** Perform a validation of the NWB files by running: + dandi validate --ignore DANDI.NO_DANDISET_FOUND - **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** + + **If you are having trouble with validation, make sure the conversions were run with the most recent version of `dandi`, `PyNWB` and `MatNWB`.** + 1. **Upload the data to DANDI.** This can either be done through the NWB GUIDE, or programmatically: dandi download https://dandiarchive.org/dandiset//draft @@ -78,14 +81,14 @@ The CLI approach may be more suitable for users who are comfortable with the com dandi validate . dandi upload - Note that the `organize` steps should not be used if you are preparing a BIDS dataset with the NWB files. - Uploading to the development server is controlled via `-i` option, e.g. - `dandi upload -i dandi-staging`. - Note that validation is also done during `upload`, but ensuring compliance using `validate` prior to upload helps avoid interruptions of the lengthier upload process due to validation failures. + Note that the `organize` steps should not be used if you are preparing a BIDS dataset with the NWB files. + Uploading to the development server is controlled via `-i` option, e.g. `dandi upload -i dandi-staging`. + Note that validation is also done during `upload`, but ensuring compliance using `validate` prior to upload helps avoid interruptions of the lengthier upload process due to validation failures. + If you have an issue using the DANDI Client, see the [DANDI Debugging section](./15_debugging.md). + 1. **Add metadata to the Dandiset.** Visit your Dandiset landing page: `https://dandiarchive.org/dandiset//draft` and click on the `METADATA` link. -If you have an issue using the DANDI Client, see the [DANDI Debugging section](./15_debugging.md). ## Storing Access Credentials @@ -109,8 +112,7 @@ There are two options for storing your DANDI access credentials. - Specifying the `keyring` backend - You can set the backend the `keyring` library uses either by setting the `PYTHON_KEYRING_BACKEND` environment variable or by filling in - the `keyring` library's [configuration - file](https://github.com/jaraco/keyring#configuring). + the `keyring` library's [configuration file](https://github.com/jaraco/keyring#configuring). - IDs for the available backends can be listed by running `keyring --list`. From 49f0842088116c99a2676429e96c0ce113a79039 Mon Sep 17 00:00:00 2001 From: Roni Choudhury Date: Mon, 22 Jul 2024 12:41:47 -0400 Subject: [PATCH 22/48] Retitle section --- docs/40_development.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/40_development.md b/docs/40_development.md index 8c9021f4..56815f7c 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -126,7 +126,7 @@ However, this is not strictly required. You can contribute using the standard fork-and-pull-request model, but under this workflow we will lose the benefit of those previews. -## Email Lists +## Email Services The project's email domain name services are managed via Terraform as AWS Route 53 entries. This allows the API server to send emails to users, etc. It also @@ -168,7 +168,7 @@ there. ### Why do incoming emails to dandiarchive.org look crazy? When a user emails help@dandiarchive.org or info@dandiarchive.org, those -messages are forwarded to dandi@mit.edu (see [above](#email-lists)) so that the +messages are forwarded to dandi@mit.edu (see [above](#email-services)) so that the dev team sees them. However, these emails arrive with a long, spammy-looking From address with a Heroku DNS domain; this seems to be an artifact of how mit.edu processes emails, and does not occur in general (e.g. messages sent From b8d2f3bdfe20bf82e67fa2794014ebc45ca09315 Mon Sep 17 00:00:00 2001 From: Roni Choudhury Date: Mon, 22 Jul 2024 12:42:52 -0400 Subject: [PATCH 23/48] Add intro section (plus a subheading for first section) --- docs/40_development.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/40_development.md b/docs/40_development.md index 56815f7c..01c9ff62 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -128,10 +128,24 @@ those previews. ## Email Services -The project's email domain name services are managed via Terraform as AWS Route -53 entries. This allows the API server to send emails to users, etc. It also -means we need a way to forward incoming emails to the proper mailing list--this -is accomplished with a service called [ImprovMX](https://improvmx.com/). +DANDI Archive maintains several email services to implement the following +facilities: + +- **Public email.** Users of the archive can reach the developers for help or to + report problems by sending email to info@dandiarchive.org and + help@dandiarchive.org. These are "virtual" email addresses managed by DNS + entries. +- **Transactional email.** The Archive sends email to users to manage the signup + process and to inform about special situations, long running operations, etc. + such as registration reject/approval, Dandiset embargo and unembargo, changes to + ownership, etc. These are sent via Amazon Simple Email Service (SES), + programmatically from the Archive code. +- **Mass email.** The maintainers of the Archive infrequently send + mass email to all users of the Archive to inform about downtime + or other notifications of mass appeal. This function is managed through a + Mailchimp account that requires special procedures to keep it up to date. + +### DNS Entries for public email addresses The email addresses info@dandiarchive.org and help@dandiarchive.org are advertised to users as general email addresses to use to ask for information or From a8c1278e969e57c796ee8420d1fffba42a3df49f Mon Sep 17 00:00:00 2001 From: Roni Choudhury Date: Mon, 22 Jul 2024 12:43:50 -0400 Subject: [PATCH 24/48] Remix existing content for clarity --- docs/40_development.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/40_development.md b/docs/40_development.md index 01c9ff62..422a6d65 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -149,10 +149,11 @@ facilities: The email addresses info@dandiarchive.org and help@dandiarchive.org are advertised to users as general email addresses to use to ask for information or -help; both of them are forwarded to dandi@mit.edu, a mailing list containing the -leaders and developers of the project. The forwarding is done by the ImprovMX -service, and more such email addresses can be created as needed within that -service. +help. These are managed via Terraform as AWS Route 53 MX entries. We use +[ImprovMX](https://improvmx.com/) to forward emails sent to these addresses to +dandi@mit.edu, a mailing list containing the leaders and developers of the +project. (Other virtual addresses within the dandiarchive.org domain can be +created as needed.) If you need the credentials for logging into ImprovMX, speak to Roni Choudhury (). From c7d16dd70718de1c49db03372b61228263f349aa Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 1 Aug 2024 10:09:16 -0500 Subject: [PATCH 25/48] Add `DANDI Datasets` page to navigation --- mkdocs.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 180fb29e..96416454 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -39,6 +39,8 @@ nav: - REST API Swagger: https://api.dandiarchive.org/swagger - REST API Redoc: https://api.dandiarchive.org/redoc - DANDI Hub: "50_hub.md" + - Tutorials: + - Dandiset Examples: https://www.dandiarchive.org/example-notebooks - Health Status: - Dandisets: https://github.com/dandi/dandisets-healthstatus - Services: https://www.dandiarchive.org/upptime From 98b2e88f7fac9dc4067cb7ec5af6397953c9d537 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 1 Aug 2024 08:46:46 -0700 Subject: [PATCH 26/48] add page on editing dandiset metadata --- docs/136_metadata.md | 66 ++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 67 insertions(+) create mode 100644 docs/136_metadata.md diff --git a/docs/136_metadata.md b/docs/136_metadata.md new file mode 100644 index 00000000..e708dd17 --- /dev/null +++ b/docs/136_metadata.md @@ -0,0 +1,66 @@ +# Dandiset Metadata + +The Dandiset Landing Page (DLP) displays metadata about the Dandiset and the data within. +Some of this metadata is automatically extracted from the files, for example the `Size` in the top panel. +The Assets Summary panel at the bottom of the DLP displays additional information +that is automatically extracted from NWB files, such as the species of the subject and the types of recordings. +This metadata will automatically be re-computed if you make any changes to the data files. + +## Metadata editor + +DANDI also has metadata that you must set manually using the `METADATA` button on the right panel. +Any `Owner` of a Dandiset has the ability to edit the manual Dandiset metadata through this editor. +Several fields here are essential for publication, and the rest provide opportunities to make your Dandiset +more FAIR, enabling secondary analysis. + +### General + +The General section is for high-level metadata. +The title should be about as descriptive as the title of a journal article. +If this Dandiset is associated with a specific journal article or preprint, you may give the Dandiset the same title. +You may also use the paper abstract as the Dandiset Description, though you are encouraged to provide more +detailed information if appropriate. + +Note that funding information should be entered in the Dandiset Contributors section, **not** in the `Acknowledgements` +field of `General`. + + +### Contributors + +The Contributors section allows you to provide attribution to personnel and organizations that contributed to the +publication of this data, metadata that is essential for Publication of the Dandiset. For each person, you +have the ability to enter rich metadata. +We highly encourage the use of ORCID identifiers for each person. +For each person, you have the ability to annotate multiple roles. +The "Author" role is often appropriate for contributors. +Marking a contributor as "Author" will make their name appear on the DLP. +You can also add contributors that are non-authors, such as data curators. +One of the personnel must be listed as the Contact, and this Person must have contact information. + + +**The Contributors tab is also where you enter funding information.** +Choose "Organization" and proceed to fill in information. +ror.org is a platform that provides unique identifiers for institutions. +It is highly recommended to search http://ror.org and include the unique identifiers for each funding agency. +Make sure to inspect the metadata for each institution on the ror.org website, as many organizations have similar names +(e.g. many different countries have an "NIH"). + +### Subject Matter + +The Subject Matter section allows you to annotate the Dandiset with links to terms in ontologies. Use `Generic Type` +for any type other than `Anatomy` or `Disorder`. + +### Ethics Approvals + +The Ethics Approvals section allows you to provide information about the ethics approvals that were obtained for the +experiment. It is highly recommended to include this information. + +### Related Resources + +This section allows you to annotate the Dandiset with links to related resources such as publications and code repositories. +It is highly recommended to add links to the following resources (if they exist): + * The associated publication + * The public code repository used to convert the data + * A data analysis library associated with the publication that can take this data as input + * An example notebook submitted to http://github.com/dandi/example-notebooks that demonstrates how to use the data + * Associated datasets published on DANDI or on other archives. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 180fb29e..b06b36c2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ nav: - Downloading Data and Dandisets: "12_download.md" - Creating Dandisets and Uploading Data: "13_upload.md" - Validation Levels for NWB Files: "135_validation.md" + - Dandiset Metadata: "136_metadata.md" - Publishing Dandisets: "14_publish.md" - Debugging: "15_debugging.md" - DANDI CLI and Python API: https://dandi.readthedocs.io From 94e98794a76a2450e03b0b11301e0d2ea286605b Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Sun, 4 Aug 2024 15:23:32 -0500 Subject: [PATCH 27/48] Add site analytics --- mkdocs.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 180fb29e..1049196f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -64,6 +64,9 @@ plugins: # Customize theme extra: homepage: https://dandiarchive.org + analytics: + provider: google + property: G-15WQLCLQ3L social: - icon: fontawesome/brands/slack link: https://dandiarchive.slack.com From 87658029f2b32f525eeef451462d04c703a6e2e8 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 7 Aug 2024 13:22:54 -0500 Subject: [PATCH 28/48] Fix formatting of list --- docs/136_metadata.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/136_metadata.md b/docs/136_metadata.md index e708dd17..4a30f54e 100644 --- a/docs/136_metadata.md +++ b/docs/136_metadata.md @@ -59,6 +59,7 @@ experiment. It is highly recommended to include this information. This section allows you to annotate the Dandiset with links to related resources such as publications and code repositories. It is highly recommended to add links to the following resources (if they exist): + * The associated publication * The public code repository used to convert the data * A data analysis library associated with the publication that can take this data as input From 72ba0c15a09af6168b6bf7cd61bedf42e4c5d85d Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 7 Aug 2024 13:26:22 -0500 Subject: [PATCH 29/48] Add `Resource Identifiers` link --- docs/12_download.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/12_download.md b/docs/12_download.md index cb765775..6c15686a 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -64,7 +64,7 @@ application, e.g.: `dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/0.210914.1900/assets/1a93dc97-327d-4f9c-992d-c2149e7810ae/download/` -**Hint:** `dandi download` supports a number of resource identifiers to point to a Dandiset, folder, or file. Providing +**Hint:** `dandi download` supports a number of [Resource Identifiers](https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids) to point to a Dandiset, folder, or file. Providing an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of supported identifiers. ## Using DataLad From c66553ccd57aa527dfbdebf9f07c18fdcb8925e2 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 7 Aug 2024 13:44:46 -0500 Subject: [PATCH 30/48] Add instructions for the `--preserve-tree` option --- docs/12_download.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/12_download.md b/docs/12_download.md index 6c15686a..dad95418 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -67,6 +67,16 @@ application, e.g.: **Hint:** `dandi download` supports a number of [Resource Identifiers](https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids) to point to a Dandiset, folder, or file. Providing an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of supported identifiers. +### Download a specific file and preserve the directory tree of the Dandiset +In the command below, replace the ``, ``, and asset ``. +The `` can be found by selecting the `View asset metadata` icon next to an asset on the web app and locating the `path` key. + + dandi download --preserve-tree dandi://dandi/@/ + +For example: + + dandi download --preserve-tree dandi://dandi/000026@draft/sub-I58/ses-Hip-CT/micr/sub-I58_sample-01_chunk-01_hipCT.json + ## Using DataLad All dandisets are regularly mirrored to DataLad datasets which are made available at the GitHub organization https://github.com/dandisets. From f4c8dba763253d1293bf007257fb57b2c0ff3ca2 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 7 Aug 2024 13:47:33 -0500 Subject: [PATCH 31/48] Add url --- docs/12_download.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/12_download.md b/docs/12_download.md index dad95418..fdb7e0c1 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -69,7 +69,7 @@ an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of support ### Download a specific file and preserve the directory tree of the Dandiset In the command below, replace the ``, ``, and asset ``. -The `` can be found by selecting the `View asset metadata` icon next to an asset on the web app and locating the `path` key. +The `` can be found by selecting the `View asset metadata` icon next to an asset on https://dandiarchive.org and locating the `path` key. dandi download --preserve-tree dandi://dandi/@/ From 954524fe57832585f4ae0336f51702cd851bce72 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 7 Aug 2024 13:59:14 -0500 Subject: [PATCH 32/48] Convert inline code highlighting to code blocks --- docs/12_download.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/12_download.md b/docs/12_download.md index cb765775..cf771f6c 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -40,7 +40,7 @@ If you have an issue using the Python CLI, see the [Dandi Debugging section](./1 ### Download a Dandiset To download an entire Dandiset, you can use the same command as suggested by DANDI web application, e.g.: -`dandi download DANDI:000023` + dandi download DANDI:000023 ### Download data for a specific subject from a Dandiset You can download data for specific subjects. @@ -48,20 +48,20 @@ Names of the subjects can be found on DANDI web application or by running a comm DANDI:000023`. Once you have the subject ID, you can download the data, e.g.: -`dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/_draft_/assets/?path=sub-811677083` + dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/_draft_/assets/?path=sub-811677083 You should replace `_draft_` with a specific version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset). You can also use the link from DANDI web application, e.g.: -`dandi download https://dandiarchive.org/dandiset/000023/0.210914.1900/files?location=sub-541516760%2F` + dandi download https://dandiarchive.org/dandiset/000023/0.210914.1900/files?location=sub-541516760%2F ### Download a specific file from a Dandiset You can download a specific file from a Dandiset when the link for the specific file can be found on the DANDI web application, e.g.: -`dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/0.210914.1900/assets/1a93dc97-327d-4f9c-992d-c2149e7810ae/download/` + dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/0.210914.1900/assets/1a93dc97-327d-4f9c-992d-c2149e7810ae/download/ **Hint:** `dandi download` supports a number of resource identifiers to point to a Dandiset, folder, or file. Providing From f4cec4e2bf4f5e65793126cdf313877038fc37b3 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 8 Aug 2024 08:29:06 -0500 Subject: [PATCH 33/48] Add version information --- docs/12_download.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/12_download.md b/docs/12_download.md index fdb7e0c1..8b3aba29 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -68,6 +68,7 @@ application, e.g.: an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of supported identifiers. ### Download a specific file and preserve the directory tree of the Dandiset +Now available in version `0.63.0` is the `--preserve-tree` option. In the command below, replace the ``, ``, and asset ``. The `` can be found by selecting the `View asset metadata` icon next to an asset on https://dandiarchive.org and locating the `path` key. From c706389686610c407c5ce8dc09fd2f2a34a4f951 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Fri, 9 Aug 2024 16:17:47 -0400 Subject: [PATCH 34/48] Make example copy-pasteable by using "draft" instead of "_draft_" placeholder And expanded on the description underneath --- docs/12_download.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/12_download.md b/docs/12_download.md index cf771f6c..c6a655ef 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -48,9 +48,9 @@ Names of the subjects can be found on DANDI web application or by running a comm DANDI:000023`. Once you have the subject ID, you can download the data, e.g.: - dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/_draft_/assets/?path=sub-811677083 + dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/draft/assets/?path=sub-811677083 -You should replace `_draft_` with a specific version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset). +You could replace `draft` with a specific non-draft version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset), if you are not interested in the latest, possibly different state of the dandiset. You can also use the link from DANDI web application, e.g.: From 16c82abe92176287ff551a32a2f3df484174d49c Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Fri, 9 Aug 2024 17:14:21 -0400 Subject: [PATCH 35/48] Use capitalized Dandiset Co-authored-by: Kabilar Gunalan --- docs/12_download.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/12_download.md b/docs/12_download.md index c6a655ef..6501ebe4 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -50,7 +50,7 @@ Once you have the subject ID, you can download the data, e.g.: dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/draft/assets/?path=sub-811677083 -You could replace `draft` with a specific non-draft version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset), if you are not interested in the latest, possibly different state of the dandiset. +You could replace `draft` with a specific non-draft version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset), if you are not interested in the latest, possibly different state of the Dandiset. You can also use the link from DANDI web application, e.g.: From e02a0e601b8898c6f4983c8c1abae8e2727084b1 Mon Sep 17 00:00:00 2001 From: Roni Choudhury <2903332+waxlamp@users.noreply.github.com> Date: Mon, 12 Aug 2024 11:28:01 -0400 Subject: [PATCH 36/48] Capitalize "Archive" Co-authored-by: Kabilar Gunalan --- docs/40_development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/40_development.md b/docs/40_development.md index 422a6d65..405f8c68 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -131,7 +131,7 @@ those previews. DANDI Archive maintains several email services to implement the following facilities: -- **Public email.** Users of the archive can reach the developers for help or to +- **Public email.** Users of the Archive can reach the developers for help or to report problems by sending email to info@dandiarchive.org and help@dandiarchive.org. These are "virtual" email addresses managed by DNS entries. From c292f545e53103470ad9c182d61f5c973de40d88 Mon Sep 17 00:00:00 2001 From: Roni Choudhury <2903332+waxlamp@users.noreply.github.com> Date: Mon, 12 Aug 2024 11:28:32 -0400 Subject: [PATCH 37/48] Improve wording Co-authored-by: Kabilar Gunalan --- docs/40_development.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/40_development.md b/docs/40_development.md index 405f8c68..9958918d 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -136,8 +136,8 @@ facilities: help@dandiarchive.org. These are "virtual" email addresses managed by DNS entries. - **Transactional email.** The Archive sends email to users to manage the signup - process and to inform about special situations, long running operations, etc. - such as registration reject/approval, Dandiset embargo and unembargo, changes to + process and to inform about special situations such as long running operations, + registration reject/approval, Dandiset embargo and unembargo, changes to ownership, etc. These are sent via Amazon Simple Email Service (SES), programmatically from the Archive code. - **Mass email.** The maintainers of the Archive infrequently send From 4c5c49da7bc1249ad21d0b00fd912524c80a6921 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Mon, 12 Aug 2024 18:30:33 -0400 Subject: [PATCH 38/48] Adjust section name a bit --- docs/100_contribute_docs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/100_contribute_docs.md b/docs/100_contribute_docs.md index e65af75e..cffef85d 100644 --- a/docs/100_contribute_docs.md +++ b/docs/100_contribute_docs.md @@ -1,4 +1,4 @@ -# Contributing Documentation +# Contributing to this Handbook This documentation is a work in progress and we welcome all input: if something is missing or unclear, let us know by [opening an issue on our helpdesk](https://github.com/dandi/helpdesk/issues/new/choose). From 61ce49c6a8cc58b1c2b328a392eca8b723a41fd5 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 15 Aug 2024 23:01:13 -0500 Subject: [PATCH 39/48] Update section title --- docs/12_download.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/12_download.md b/docs/12_download.md index 8b3aba29..ef8f9fc6 100644 --- a/docs/12_download.md +++ b/docs/12_download.md @@ -67,7 +67,7 @@ application, e.g.: **Hint:** `dandi download` supports a number of [Resource Identifiers](https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids) to point to a Dandiset, folder, or file. Providing an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of supported identifiers. -### Download a specific file and preserve the directory tree of the Dandiset +### Download the `dandiset.yaml` file and a specific file within the directory tree of the Dandiset Now available in version `0.63.0` is the `--preserve-tree` option. In the command below, replace the ``, ``, and asset ``. The `` can be found by selecting the `View asset metadata` icon next to an asset on https://dandiarchive.org and locating the `path` key. From 00505ae743106842b3ca10341e5de727c55eeaca Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 19 Aug 2024 22:40:47 -0400 Subject: [PATCH 40/48] add page on citing Dandisets --- docs/citing.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 docs/citing.md diff --git a/docs/citing.md b/docs/citing.md new file mode 100644 index 00000000..6282d132 --- /dev/null +++ b/docs/citing.md @@ -0,0 +1,30 @@ +Citing a Dandiset +----------------- + +If you use a Dandiset in your research, please acknowledge the Dandiset by citing it, just as you would a publication, +including the DOI. +The DOI can be found in the Dandiset's landing page on the DANDI Archive website. +An example formatted citation can also be found on the Dandiset's landing page at the "CITE AS" button. This citation +uses the DataCite citation style, which is a widely accepted standard for citing datasets, but you may need to adapt it +to the citation style required by the journal you are submitting to. + +If the Dandiset has an associated publication, it may also be appropriate to cite the publication, but this does not +replace the need to cite the Dandiset itself. + +Citing the Dandiset is important because it provides a direct link to the data used in your research, which can be used +to verify your results and to facilitate reproducibility. It also provides credit to the data contributors and helps +us track the impact of the data DANDI hosts. + + +Data availability statements +---------------------------- +It is common for journals to require a Data Availability Statement in the manuscript, which should include the DOI of +the Dandiset used in the research. Here is an example of a well formatted Data Availability Statement: + +``` +The data that support the findings of this study are openly available on the DANDI Archive at [DOI of Dandiset]. +``` + +It is important to note that a Data Availability Statement does not replace the need for a full citation in the +manuscript's references section. +Both elements serve different purposes and are typically required for comprehensive documentation of data sources. \ No newline at end of file From 3ad8a8825e0cca899610c81e8cd0976808169393 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 27 Aug 2024 13:49:03 -0400 Subject: [PATCH 41/48] Update docs/citing.md Co-authored-by: Dorota Jarecka --- docs/citing.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/citing.md b/docs/citing.md index 6282d132..20e6a87b 100644 --- a/docs/citing.md +++ b/docs/citing.md @@ -11,9 +11,11 @@ to the citation style required by the journal you are submitting to. If the Dandiset has an associated publication, it may also be appropriate to cite the publication, but this does not replace the need to cite the Dandiset itself. -Citing the Dandiset is important because it provides a direct link to the data used in your research, which can be used -to verify your results and to facilitate reproducibility. It also provides credit to the data contributors and helps -us track the impact of the data DANDI hosts. +Citing the Dandiset and other datasets is important because it provides a direct link to the data used in your research. That is crucial, because it: + - allows others to better understand and verify your results, and facilitates reproducibility, + - connects your work to other research using the same dataset, + - provides credit to the data collectors and maintainers, + - helps track the impact of DANDI and other data archives. Data availability statements From 1193151e2b8c949624f0948041e4607ff1ff9729 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 27 Aug 2024 13:49:17 -0400 Subject: [PATCH 42/48] Update docs/citing.md Co-authored-by: Dorota Jarecka --- docs/citing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/citing.md b/docs/citing.md index 20e6a87b..1fdc3d0a 100644 --- a/docs/citing.md +++ b/docs/citing.md @@ -8,8 +8,8 @@ An example formatted citation can also be found on the Dandiset's landing page a uses the DataCite citation style, which is a widely accepted standard for citing datasets, but you may need to adapt it to the citation style required by the journal you are submitting to. -If the Dandiset has an associated publication, it may also be appropriate to cite the publication, but this does not -replace the need to cite the Dandiset itself. +**If the Dandiset has an associated publication, it may also be appropriate to cite the publication, but this does not +replace the need to cite the Dandiset itself.** Citing the Dandiset and other datasets is important because it provides a direct link to the data used in your research. That is crucial, because it: - allows others to better understand and verify your results, and facilitates reproducibility, From df7f3d361b18aba788fa9568729bddfd46d5072b Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 27 Aug 2024 13:49:31 -0400 Subject: [PATCH 43/48] Update docs/citing.md Co-authored-by: Dorota Jarecka --- docs/citing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/citing.md b/docs/citing.md index 1fdc3d0a..531ec07d 100644 --- a/docs/citing.md +++ b/docs/citing.md @@ -27,6 +27,6 @@ the Dandiset used in the research. Here is an example of a well formatted Data A The data that support the findings of this study are openly available on the DANDI Archive at [DOI of Dandiset]. ``` -It is important to note that a Data Availability Statement does not replace the need for a full citation in the -manuscript's references section. +**It is important to note that a Data Availability Statement does not replace the need for a full citation in the +manuscript's references section.** Both elements serve different purposes and are typically required for comprehensive documentation of data sources. \ No newline at end of file From 0e929fe83a9e9a24e6b1e2ad45034db44d92eb34 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 25 Sep 2024 10:32:53 -0400 Subject: [PATCH 44/48] Update docs/citing.md --- docs/citing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/citing.md b/docs/citing.md index 531ec07d..eac7e5ae 100644 --- a/docs/citing.md +++ b/docs/citing.md @@ -24,7 +24,7 @@ It is common for journals to require a Data Availability Statement in the manusc the Dandiset used in the research. Here is an example of a well formatted Data Availability Statement: ``` -The data that support the findings of this study are openly available on the DANDI Archive at [DOI of Dandiset]. +The data that support the findings of this study are openly available on the DANDI Archive (RRID:SCR_017571) at [DOI of Dandiset]. ``` **It is important to note that a Data Availability Statement does not replace the need for a full citation in the From 36b3bb834144ec86eeafb8745f9f629314bcbb73 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 25 Sep 2024 10:34:16 -0400 Subject: [PATCH 45/48] Update docs/citing.md --- docs/citing.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/citing.md b/docs/citing.md index eac7e5ae..b39e1d19 100644 --- a/docs/citing.md +++ b/docs/citing.md @@ -20,8 +20,9 @@ Citing the Dandiset and other datasets is important because it provides a direct Data availability statements ---------------------------- -It is common for journals to require a Data Availability Statement in the manuscript, which should include the DOI of -the Dandiset used in the research. Here is an example of a well formatted Data Availability Statement: +It is common for journals to require a Data Availability Statement in the manuscript, which should include the +DANDI Archive RRID and the DOI of the Dandiset used in the research. +Here is an example of a well formatted Data Availability Statement: ``` The data that support the findings of this study are openly available on the DANDI Archive (RRID:SCR_017571) at [DOI of Dandiset]. From 8956f5574bcbac567a0c1ba51b07e04435bda371 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 25 Sep 2024 09:47:54 -0500 Subject: [PATCH 46/48] add citing.md to toc, modify and format the page a bit --- docs/citing.md | 11 ++++------- mkdocs.yml | 1 + 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/citing.md b/docs/citing.md index b39e1d19..c916e09e 100644 --- a/docs/citing.md +++ b/docs/citing.md @@ -1,5 +1,4 @@ -Citing a Dandiset ------------------ +# Citing a Dandiset If you use a Dandiset in your research, please acknowledge the Dandiset by citing it, just as you would a publication, including the DOI. @@ -12,21 +11,19 @@ to the citation style required by the journal you are submitting to. replace the need to cite the Dandiset itself.** Citing the Dandiset and other datasets is important because it provides a direct link to the data used in your research. That is crucial, because it: + - allows others to better understand and verify your results, and facilitates reproducibility, - connects your work to other research using the same dataset, - provides credit to the data collectors and maintainers, - helps track the impact of DANDI and other data archives. +## Data availability statement -Data availability statements ----------------------------- It is common for journals to require a Data Availability Statement in the manuscript, which should include the DANDI Archive RRID and the DOI of the Dandiset used in the research. Here is an example of a well formatted Data Availability Statement: -``` -The data that support the findings of this study are openly available on the DANDI Archive (RRID:SCR_017571) at [DOI of Dandiset]. -``` +> The data that support the findings of this study are openly available on the DANDI Archive (RRID:SCR_017571) at [DOI of Dandiset] (citation of Dandiset). **It is important to note that a Data Availability Statement does not replace the need for a full citation in the manuscript's references section.** diff --git a/mkdocs.yml b/mkdocs.yml index 13435274..06c75b2d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -32,6 +32,7 @@ nav: - Validation Levels for NWB Files: "135_validation.md" - Dandiset Metadata: "136_metadata.md" - Publishing Dandisets: "14_publish.md" + - Citing a Dandiset: "citing.md" - Debugging: "15_debugging.md" - DANDI CLI and Python API: https://dandi.readthedocs.io - Developer Guide: From 1b7839721b2b82ec4d152c5412defb82ee277947 Mon Sep 17 00:00:00 2001 From: Roni Choudhury Date: Mon, 22 Jul 2024 12:44:26 -0400 Subject: [PATCH 47/48] Add section about Mailchimp --- docs/40_development.md | 53 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/40_development.md b/docs/40_development.md index 9958918d..9239144c 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -158,6 +158,59 @@ created as needed.) If you need the credentials for logging into ImprovMX, speak to Roni Choudhury (). +### Mass emails with Mailchimp + +The Archive maintainers are able to send email to all users through the mass +emailing functions of a dedicated Mailchimp account. In technical parlance, +these communications are also known as "marketing email", though as a rule the +maintainers do not conduct any actual marketing through this channel. +nonetheless, such communications are governed by laws and regulations such as +the American CAN-SPAM Act, the California-specific CCPA, and the European +Union's GDPR; Mailchimp helps the maintainers comply with these rules and +regulations. + +Our major use case for mass email is to notify the userbase of upcoming downtime +(as is needed for, e.g., a major data migration or maintenance windows). + +If you need to mass email the DANDI Archive userbase, speak to Roni Choudhury +(). + +#### Updating the DANDI userbase audience in Mailchimp + +Follow these steps before sending a mass email through Mailchimp to ensure that +the Mailchimp-maintained DANDI userbase audience is up to date. + +1. Log into the DANDI admin panel and navigate to the dashboard page (at, e.g., + `api.dandiarchive.org/dashboard`). +2. Click on the `Mailchimp CSV` link in the navbar to download the CSV file to + disk. +3. Log into Mailchimp, click on the `Audience` section in the sidebar, then + click on the `Manage Audience` dropdown and select `Manage contacts`. +4. Click on the `Manage audience` dropdown and select `Archive all contacts`. + Then follow the confirmation prompt to carry out the archiving operation. +5. Click on the `Add contacts` dropdown and select `Import contacts`. +6. Select the `Upload a file` option, and follow the wizard steps, uploading the + CSV file from step 2 when prompted. Activate the `Update any existing + contacts` checkbox under the `Organize your contacts` step. Do not set any + tags during the `Tag your contacts` step. In the `Match column labels to + contact information` step, visually verify that the email address, first + name, and last name columns look correct matched. In the `Subscribe contacts + to marketing` step, ensure that `Subscribed` is selected in the dropdown. In + the `Review and complete your import`, read over the summary and ensure it is + correct before clicking the `Complete Import` button to finish the process. + +It is necessary to "deactivate" the entire userbase before reimporting the +current slate of users from the freshly computed CSV file because Mailchimp does +not have a way to perform a PUT-like operation during import (to borrow a term +from RESTful API design), only to add new users and update existing ones. + +The reason to archive the contacts instead of deleting them has to do with +Mailchimp's semantics for those actions. Deleting a user means they cannot be +re-added to the audience, while archiving is a reversible action that retains +all data and history and merely removes that user from the audience for purposes +of receiving emails. Thus, we use an archive-and-reimport procedure to emulate +the PUT-like operation we actually need. + ## Miscellaneous Tips and Information ### Use email address to log into dev Django admin panel From b87ac5c9040c8689727c69f8b61605025d953947 Mon Sep 17 00:00:00 2001 From: Roni Choudhury <2903332+waxlamp@users.noreply.github.com> Date: Thu, 17 Oct 2024 10:31:24 -0400 Subject: [PATCH 48/48] Fix grammatical typos Co-authored-by: Kabilar Gunalan --- docs/40_development.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/40_development.md b/docs/40_development.md index 9239144c..49deaf95 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -164,7 +164,7 @@ The Archive maintainers are able to send email to all users through the mass emailing functions of a dedicated Mailchimp account. In technical parlance, these communications are also known as "marketing email", though as a rule the maintainers do not conduct any actual marketing through this channel. -nonetheless, such communications are governed by laws and regulations such as +Nonetheless, such communications are governed by laws and regulations such as the American CAN-SPAM Act, the California-specific CCPA, and the European Union's GDPR; Mailchimp helps the maintainers comply with these rules and regulations. @@ -194,7 +194,7 @@ the Mailchimp-maintained DANDI userbase audience is up to date. contacts` checkbox under the `Organize your contacts` step. Do not set any tags during the `Tag your contacts` step. In the `Match column labels to contact information` step, visually verify that the email address, first - name, and last name columns look correct matched. In the `Subscribe contacts + name, and last name columns look correctly matched. In the `Subscribe contacts to marketing` step, ensure that `Subscribed` is selected in the dropdown. In the `Review and complete your import`, read over the summary and ensure it is correct before clicking the `Complete Import` button to finish the process.