merge upstream

docs/100_about_this_doc.md → docs/100_contribute_docs.md

@@ -1,4 +1,4 @@
-# About This Documentation
+# Contributing to this Handbook
 
 This documentation is a work in progress and we welcome all input: if something
 is missing or unclear, let us know by [opening an issue on our helpdesk](https://github.com/dandi/helpdesk/issues/new/choose).

docs/10_using_dandi.md

@@ -90,7 +90,7 @@ The [DANDI Python client](https://pypi.org/project/dandi/) allows you to:
 * Organize your data locally before upload
 * Upload Dandisets
 
-Before you can use the DANDI Python client, you have to install the package with `pip install dandi` in a Python 3.7+
+Before you can use the DANDI Python client, you have to install the package with `pip install dandi` in a Python 3.8+
 environment.
 
 You should check the [Dandi Debugging section](./15_debugging.md) in case of any problems.

docs/12_download.md

@@ -1,12 +1,12 @@
 # Downloading Data and Dandisets
 
-You can download the content of a Dandiset using the DANDI Web application (such a specific file) or entire 
+You can download the content of a Dandiset using the DANDI Web application (such a specific file) or entire
 Dandisets using the DANDI Python CLI.
 
 ## Using the DANDI Web Application
 
 Once you have the Dandiset you are interested in (see more in [the Dandiset View section](./11_view.md)), you can download the content of the Dandiset.
-On the landing page of each Dandiset, you can find `Download` button on the right-hand panel. After clicking the 
+On the landing page of each Dandiset, you can find `Download` button on the right-hand panel. After clicking the
 button, you will see the specific command you can use with DANDI Python CLI (as well as the information on how to download the CLI).
 
 <img
@@ -31,38 +31,89 @@ Each file in the Dandiset has a download icon next to it, clicking the icon will
 
 ## Using the Python CLI Client
 
-The [DANDI Python client](https://pypi.org/project/dandi/) gives you more options, such as downloading entire 
+The [DANDI Python client](https://pypi.org/project/dandi/) gives you more options, such as downloading entire
 Dandisets.
 
-**Before You Begin**: You need to have Python 3.7+ and install the DANDI Python Client using `pip install dandi`.
+**Before You Begin**: You need to have Python 3.8+ and install the DANDI Python Client using `pip install dandi`.
 If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md).
 
 ### Download a Dandiset
-To download an entire Dandiset, you can use the same command as suggested by DANDI web application, e.g.: 
+To download an entire Dandiset, you can use the same command as suggested by DANDI web application, e.g.:
 
-`dandi download DANDI:000023`
+    dandi download DANDI:000023
 
 ### Download data for a specific subject from a Dandiset
-You can download data for specific subjects. 
-Names of the subjects can be found on DANDI web application or by running a command with the DANDI CLI: `dandi ls -r 
+You can download data for specific subjects.
+Names of the subjects can be found on DANDI web application or by running a command with the DANDI CLI: `dandi ls -r
 DANDI:000023`.
 Once you have the subject ID, you can download the data, e.g.:
 
-`dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/_draft_/assets/?path=sub-811677083`
+    dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/draft/assets/?path=sub-811677083
 
-You should replace `_draft_` with a specific version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset).
+You could replace `draft` with a specific non-draft version you are interested in (e.g. `0.210914.1900` in the case of this Dandiset), if you are not interested in the latest, possibly different state of the Dandiset.
 
 You can also use the link from DANDI web application, e.g.:
 
-`dandi download https://dandiarchive.org/dandiset/000023/0.210914.1900/files?location=sub-541516760%2F`
+    dandi download https://dandiarchive.org/dandiset/000023/0.210914.1900/files?location=sub-541516760%2F
 
 
-### Download a specific file from a Dandiset 
-You can download a specific file from a Dandiset when the link for the specific file can be found on the DANDI web 
+### Download a specific file from a Dandiset
+You can download a specific file from a Dandiset when the link for the specific file can be found on the DANDI web
 application, e.g.:
 
-`dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/0.210914.1900/assets/1a93dc97-327d-4f9c-992d-c2149e7810ae/download/`
+    dandi download https://api.dandiarchive.org/api/dandisets/000023/versions/0.210914.1900/assets/1a93dc97-327d-4f9c-992d-c2149e7810ae/download/
 
 
-**Hint:** `dandi download` supports a number of resource identifiers to point to a Dandiset, folder, or file.  Providing 
+**Hint:** `dandi download` supports a number of [Resource Identifiers](https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids) to point to a Dandiset, folder, or file.  Providing
 an incorrect URL (e.g. `dandi download wrongurl`) will provide a list of supported identifiers.
+
+### Download the `dandiset.yaml` file and a specific file within the directory tree of the Dandiset
+Now available in version `0.63.0` is the `--preserve-tree` option.
+In the command below, replace the `<dandiset-id>`, `<version>`, and asset `<path>`.
+The `<path>` can be found by selecting the `View asset metadata` icon next to an asset on https://dandiarchive.org and locating the `path` key.
+
+    dandi download --preserve-tree dandi://dandi/<dandiset-id>@<version>/<path>
+
+For example:
+
+    dandi download --preserve-tree dandi://dandi/000026@draft/sub-I58/ses-Hip-CT/micr/sub-I58_sample-01_chunk-01_hipCT.json
+
+## Using DataLad
+
+All dandisets are regularly mirrored to DataLad datasets which are made available at the GitHub organization https://github.com/dandisets.
+Where present, individual [Zarr](https://zarr.dev/) files are included as subdatasets ([git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules)) hosted in the GitHub organization <https://github.com/dandizarrs/>.
+
+The Git revision histories of each dataset reflect the Dandiset's draft state as of each execution of the mirroring job.
+Published Dandiset versions are tagged with Git tags.
+
+With DataLad, you can:
+- clone an entire dataset,
+- use a specific version of it,
+- explore history of modifications,
+- download content of files of interest,
+- locally discard the content of no-longer-needed files,
+- use the dataset in a reproducible manner,
+- include it as a subdataset in your own DataLad dataset,
+- use https://github.com/datalad/datalad-fuse/ to [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)-mount individual locally-cloned dandisets so that their files' contents are transparently streamed to your DANDI/DataLad-unaware tools,
+- etc.
+
+Learn more about DataLad from its handbook at <https://handbook.datalad.org/>.
+
+**Developers' note:** DataLad datasets are created using the [dandi/backups2datalad](https://github.com/dandi/backups2datalad/) tool which is also available for use by the community to similarly maintain mirrors of independent DANDI deployments as DataLad datasets.
+
+## Using WebDAV
+
+DANDI provides a [WebDAV](https://en.wikipedia.org/wiki/WebDAV) service at https://webdav.dandiarchive.org/ for accessing the data in the DANDI archive.
+You can use any WebDAV client or even a web browser to access the data - any dandiset, any version, any file or collection of files.
+You can use any web download tool to download the data from the DANDI archive, e.g.
+
+````commandline
+wget -r -np -nH --cut-dirs=3 https://webdav.dandiarchive.org/dandisets/000027/releases/0.210831.2033/
+````
+
+for a download of a specific release `0.210831.2033` of the `000027` dandiset.
+
+**Note:** The WebDAV service does not directly serve any file contents; it instead relies on redirects to AWS S3 storage where the contents are stored.
+You might need to configure your WebDAV client to follow redirects; e.g., for the [davfs2](https://savannah.nongnu.org/projects/davfs2) WebDAV client, set `follow_redirect` to `1` in `/etc/davfs2/davfs2.conf`.
+
+**Developers' note:** The WebDAV service's code is available at https://github.com/dandi/dandidav/ and can also be used for independent DANDI deployments.

docs/136_metadata.md

@@ -0,0 +1,67 @@
+# Dandiset Metadata
+
+The Dandiset Landing Page (DLP) displays metadata about the Dandiset and the data within.
+Some of this metadata is automatically extracted from the files, for example the `Size` in the top panel.
+The Assets Summary panel at the bottom of the DLP displays additional information
+that is automatically extracted from NWB files, such as the species of the subject and the types of recordings.
+This metadata will automatically be re-computed if you make any changes to the data files.
+
+## Metadata editor
+
+DANDI also has metadata that you must set manually using the `METADATA` button on the right panel.
+Any `Owner` of a Dandiset has the ability to edit the manual Dandiset metadata through this editor.
+Several fields here are essential for publication, and the rest provide opportunities to make your Dandiset
+more FAIR, enabling secondary analysis.
+
+### General
+
+The General section is for high-level metadata.
+The title should be about as descriptive as the title of a journal article.
+If this Dandiset is associated with a specific journal article or preprint, you may give the Dandiset the same title.
+You may also use the paper abstract as the Dandiset Description, though you are encouraged to provide more
+detailed information if appropriate.
+
+Note that funding information should be entered in the Dandiset Contributors section, **not** in the `Acknowledgements`
+field of `General`.
+
+
+### Contributors
+
+The Contributors section allows you to provide attribution to personnel and organizations that contributed to the
+publication of this data, metadata that is essential for Publication of the Dandiset. For each person, you
+have the ability to enter rich metadata.
+We highly encourage the use of ORCID identifiers for each person.
+For each person, you have the ability to annotate multiple roles.
+The "Author" role is often appropriate for contributors.
+Marking a contributor as "Author" will make their name appear on the DLP.
+You can also add contributors that are non-authors, such as data curators.
+One of the personnel must be listed as the Contact, and this Person must have contact information.
+
+
+**The Contributors tab is also where you enter funding information.**
+Choose "Organization" and proceed to fill in information.
+ror.org is a platform that provides unique identifiers for institutions.
+It is highly recommended to search http://ror.org and include the unique identifiers for each funding agency.
+Make sure to inspect the metadata for each institution on the ror.org website, as many organizations have similar names
+(e.g. many different countries have an "NIH").
+
+### Subject Matter
+
+The Subject Matter section allows you to annotate the Dandiset with links to terms in ontologies. Use `Generic Type`
+for any type other than `Anatomy` or `Disorder`.
+
+### Ethics Approvals
+
+The Ethics Approvals section allows you to provide information about the ethics approvals that were obtained for the
+experiment. It is highly recommended to include this information.
+
+### Related Resources
+
+This section allows you to annotate the Dandiset with links to related resources such as publications and code repositories.
+It is highly recommended to add links to the following resources (if they exist):
+
+  * The associated publication
+  * The public code repository used to convert the data
+  * A data analysis library associated with the publication that can take this data as input
+  * An example notebook submitted to http://github.com/dandi/example-notebooks that demonstrates how to use the data
+  * Associated datasets published on DANDI or on other archives.