Merge remote-tracking branch 'IQSS/develop' into DANS-CSL

doc/release-notes/10883-license improvements.md

@@ -0,0 +1,12 @@
+- License metadata enhancements (#10883):
+  - Added new fields to licenses: rightsIdentifier, rightsIdentifierScheme, schemeUri, languageCode
+  - Updated DataCite metadata export to include rightsIdentifier, rightsIdentifierScheme, and schemeUri consistent with the DataCite 4.5 schema and examples
+  - Enhanced metadata exports to include all new license fields
+  - Existing licenses from the example set included with Dataverse will be automatically updated with new fields
+  - Existing API calls support the new optional fields
+
+  Setup: For existing published datasets, the additional license metadata will not be available from DataCite or in metadata exports until the dataset is republished or
+  - the /api/admin/metadata/{id}/reExportDataset is run for the dataset
+  - the api/datasets/{id}/modifyRegistrationMetadata API is run for the dataset,
+   or the global version of these api calls (/api/admin/metadata/reExportAll, /api/datasets/modifyRegistrationPIDMetadataAll) are used. 
+

doc/release-notes/11242-fix-oricid-recognition.md

@@ -0,0 +1,3 @@
+A bug that caused ORCIDs start with https://orcid.org/ entered as author identifier to be ignored when creating the DataCite metadata has been fixed. This primarily affected users of the ORCID external vocabulary script as the manual entry form recommends not using the URL form.
+
+The display of authorIdentifier, when not using any external vocabulary scripts, has been improved so that either the plain identifier (e.g. "0000-0002-1825-0097") or its URL form (e.g. "https://orcid.org/0000-0002-1825-0097") will result in valid links in the display (for identifier types that have a URL form). The URL form is now recommended when doing manual entry.

doc/sphinx-guides/source/_static/api/add-license.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution 4.0 International License.",
   "iconUrl": "https://i.creativecommons.org/l/by/4.0/88x31.png",
   "active": true,
-  "sortOrder": 2
-}
+  "sortOrder": 2,
+  "rightsIdentifier": "CC-BY-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
+}

doc/sphinx-guides/source/api/native-api.rst

@@ -1116,14 +1116,12 @@ This endpoint expects a JSON with the following format::
     {
       "datasetFieldTypeName": "datasetFieldTypeName1",
       "required": true,
-      "include": true,
-      "displayOnCreate": false
+      "include": true
     },
     {
       "datasetFieldTypeName": "datasetFieldTypeName2",
       "required": true,
-      "include": true,
-      "displayOnCreate": true
+      "include": true
     }
   ]
 
@@ -1132,7 +1130,6 @@ Parameters:
 - ``datasetFieldTypeName``: Name of the metadata field
 - ``required``: Whether the field is required (boolean)
 - ``include``: Whether the field is included (boolean)
-- ``displayOnCreate`` (optional): Whether the field is displayed during dataset creation, even when not required (boolean)
 
 .. code-block:: bash
 
@@ -1502,7 +1499,12 @@ It returns a list of versions with their metadata, and file list:
         "createTime": "2015-04-20T09:57:32Z",
         "license": {
           "name": "CC0 1.0",
-          "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+          "uri": "http://creativecommons.org/publicdomain/zero/1.0",
+          "iconUri": "https://licensebuttons.net/p/zero/1.0/88x31.png",
+          "rightsIdentifier": "CC0",
+          "rightsIdentifierScheme": "Creative Commons",
+          "schemeUri": "https://creativecommons.org/",
+          "languageCode": "en",
         },
         "termsOfAccess": "You need to request for access.",
         "fileAccessRequest": true,
@@ -1523,7 +1525,12 @@ It returns a list of versions with their metadata, and file list:
         "createTime": "2015-04-20T09:43:45Z",
         "license": {
           "name": "CC0 1.0",
-          "uri": "http://creativecommons.org/publicdomain/zero/1.0"
+          "uri": "http://creativecommons.org/publicdomain/zero/1.0",
+          "iconUri": "https://licensebuttons.net/p/zero/1.0/88x31.png",
+          "rightsIdentifier": "CC0",
+          "rightsIdentifierScheme": "Creative Commons",
+          "schemeUri": "https://creativecommons.org/",
+          "languageCode": "en",
         },
         "termsOfAccess": "You need to request for access.",
         "fileAccessRequest": true,
@@ -6929,6 +6936,10 @@ View the details of the standard license with the database ID specified in ``$ID
 
 Superusers can add a new license by posting a JSON file adapted from this example :download:`add-license.json <../_static/api/add-license.json>`. The ``name`` and ``uri`` of the new license must be unique. Sort order field is mandatory. If you are interested in adding a Creative Commons license, you are encouarged to use the JSON files under :ref:`adding-creative-commons-licenses`:
 
+Licenses must have a "name" and "uri" and may have the following optional fields: "shortDescription", "iconUri", "rightsIdentifier", "rightsIdentifierScheme", "schemeUri", "languageCode", "active", "sortOrder".
+The "name" and "uri" are used to display the license in the user interface, with "shortDescription" and "iconUri" being used to enhance the display if available.
+The "rightsIdentifier", "rightsIdentifierScheme", and "schemeUri" should be added if the license is available from https://spdx.org . "languageCode" should be sent if the language is not in English ("en"). "active" is a boolean indicating whether the license should be shown to users as an option. "sortOrder" is a numeric value - licenses are shown in the relative numeric order of this value.
+ 
 .. code-block:: bash
 
   export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

doc/sphinx-guides/source/user/appendix.rst

@@ -22,8 +22,8 @@ Supported Metadata
 
 Detailed below are what metadata schemas we support for Citation and Domain Specific Metadata in the Dataverse Project:
 
-- Citation Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/citation.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.0 <https://schema.datacite.org/meta/kernel-4.0/doc/DataCite-MetadataKernel_v4.0.pdf>`__, and Dublin Core's `DCMI Metadata Terms <https://dublincore.org/documents/dcmi-terms/>`__ . Language field uses `ISO 639-1 <https://www.loc.gov/standards/iso639-2/php/English_list.php>`__ controlled vocabulary.
-- Geospatial Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/geospatial.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.0 <https://schema.datacite.org/meta/kernel-4.0/doc/DataCite-MetadataKernel_v4.0.pdf>`__, and Dublin Core. Country / Nation field uses `ISO 3166-1 <https://en.wikipedia.org/wiki/ISO_3166-1>`_ controlled vocabulary.
+- Citation Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/citation.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.5 <https://schema.datacite.org/meta/kernel-4.5/>`__, and Dublin Core's `DCMI Metadata Terms <https://dublincore.org/documents/dcmi-terms/>`__ . Language field uses `ISO 639-1 <https://www.loc.gov/standards/iso639-2/php/English_list.php>`__ controlled vocabulary.
+- Geospatial Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/geospatial.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.5 <https://schema.datacite.org/meta/kernel-4.5/>`__, and Dublin Core. Country / Nation field uses `ISO 3166-1 <https://en.wikipedia.org/wiki/ISO_3166-1>`_ controlled vocabulary.
 - Social Science & Humanities Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/social_science.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, and Dublin Core.
 - Astronomy and Astrophysics Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/astrophysics.tsv>`__): These metadata elements can be mapped/exported to the International Virtual Observatory Alliance’s (IVOA) 
   `VOResource Schema format <https://www.ivoa.net/documents/latest/RM.html>`__ and is based on 

doc/sphinx-guides/source/user/dataset-management.rst

@@ -54,7 +54,7 @@ Adding a New Dataset
 #. Click on the "Add Data" button and select "New Dataset" in the dropdown menu. **Note:** If you are on the root Dataverse collection, your My Data page or click the "Add Data" link in the navbar, the dataset you create will be hosted in the root Dataverse collection. You can change this by selecting another Dataverse collection you have proper permissions to create datasets in, from the Host Dataverse collection dropdown in the create dataset form. This option to choose will not be available after you create the dataset.
 #. To quickly get started, enter at minimum all the required fields with an asterisk (e.g., the Dataset Title, Author Name, Description Text, Point of Contact Email, and Subject) to get a Data Citation with a DOI.
 
-   #. When entering author identifiers, select the type from the dropdown (e.g. "ORCID") and under "Identifier" enter just the unique identifier (e.g. "0000-0002-1825-0097") rather than the full URL (e.g. "https://orcid.org/0000-0002-1825-0097").
+   #. When entering author identifiers, select the type from the dropdown (e.g. "ORCID") and under "Identifier" enter the full URL (e.g. "https://orcid.org/0000-0002-1825-0097") for identifiers that have a URL form. The shorter form of the unique identifier (e.g. "0000-0002-1825-0097") can also be entered, but URL form is preferred when available.
 
 #. Scroll down to the "Files" section and click on "Select Files to Add" to add all the relevant files to your Dataset. 
    You can also upload your files directly from your Dropbox. **Tip:** You can drag and drop or select multiple files at a time from your desktop

scripts/api/data/licenses/licenseApache-2.0.json

@@ -1,8 +1,11 @@
 {
-    "name": "Apache-2.0",
-    "uri": "http://www.apache.org/licenses/LICENSE-2.0",
-    "shortDescription": "Apache License 2.0",
-    "active": true,
-    "sortOrder": 9
-  }
-
+  "name": "Apache-2.0",
+  "uri": "http://www.apache.org/licenses/LICENSE-2.0",
+  "shortDescription": "Apache License 2.0",
+  "active": true,
+  "sortOrder": 9,
+  "rightsIdentifier": "Apache-2.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
+}

scripts/api/data/licenses/licenseCC-BY-4.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution 4.0 International License.",
   "iconUrl": "https://licensebuttons.net/l/by/4.0/88x31.png",
   "active": true,
-  "sortOrder": 2
-}
+  "sortOrder": 2,
+  "rightsIdentifier": "CC-BY-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
+}

scripts/api/data/licenses/licenseCC-BY-NC-4.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution-NonCommercial 4.0 International License.",
   "iconUrl": "https://licensebuttons.net/l/by-nc/4.0/88x31.png",
   "active": true,
-  "sortOrder": 4
+  "sortOrder": 4,
+  "rightsIdentifier": "CC-BY-NC-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

scripts/api/data/licenses/licenseCC-BY-NC-ND-4.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.",
   "iconUrl": "https://licensebuttons.net/l/by-nc-nd/4.0/88x31.png",
   "active": true,
-  "sortOrder": 7
+  "sortOrder": 7,
+  "rightsIdentifier": "CC-BY-NC-ND-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

scripts/api/data/licenses/licenseCC-BY-NC-SA-4.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.",
   "iconUrl": "https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png",
   "active": true,
-  "sortOrder": 3
+  "sortOrder": 3,
+  "rightsIdentifier": "CC-BY-NC-SA-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

scripts/api/data/licenses/licenseCC-BY-ND-4.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution-NoDerivatives 4.0 International License.",
   "iconUrl": "https://licensebuttons.net/l/by-nd/4.0/88x31.png",
   "active": true,
-  "sortOrder": 6
+  "sortOrder": 6,
+  "rightsIdentifier": "CC-BY-ND-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

scripts/api/data/licenses/licenseCC-BY-SA-4.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons Attribution-ShareAlike 4.0 International License.",
   "iconUrl": "https://licensebuttons.net/l/by-sa/4.0/88x31.png",
   "active": true,
-  "sortOrder": 5
+  "sortOrder": 5,
+  "rightsIdentifier": "CC-BY-SA-4.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

scripts/api/data/licenses/licenseCC0-1.0.json

@@ -4,5 +4,9 @@
   "shortDescription": "Creative Commons CC0 1.0 Universal Public Domain Dedication.",
   "iconUrl": "https://licensebuttons.net/p/zero/1.0/88x31.png",
   "active": true,
-  "sortOrder": 1
+  "sortOrder": 1,
+  "rightsIdentifier": "CC0-1.0",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

scripts/api/data/licenses/licenseMIT.json

@@ -3,5 +3,9 @@
   "uri": "https://opensource.org/licenses/MIT",
   "shortDescription": "MIT License",
   "active": true,
-  "sortOrder": 8
+  "sortOrder": 8,
+  "rightsIdentifier": "MIT",
+  "rightsIdentifierScheme": "SPDX",
+  "schemeUri": "https://spdx.org/licenses/",
+  "languageCode": "en"
 }

src/main/java/edu/harvard/iq/dataverse/DatasetAuthor.java

@@ -97,8 +97,13 @@ public static String getIdentifierAsUrl(String idType, String idValue) {
         if (idType != null && !idType.isEmpty() && idValue != null && !idValue.isEmpty()) {
             try {
               ExternalIdentifier externalIdentifier = ExternalIdentifier.valueOf(idType);
-              if (externalIdentifier.isValidIdentifier(idValue))
-                return externalIdentifier.format(idValue);
+              if (externalIdentifier.isValidIdentifier(idValue)) {
+                  String uri = externalIdentifier.format(idValue);
+                  //The DAI identifier is a URI starting with "info" - we don't want to return it as a URL (we assume non-null URLs should be links in the display)
+                  if(uri.startsWith("http")) {
+                      return uri;
+                  }
+              }
             } catch (Exception e) {
                 // non registered identifier
             }

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

@@ -941,12 +941,6 @@ private Predicate buildFieldPresentInDataversePredicate(Dataverse dataverse, boo
                 criteriaBuilder.isTrue(datasetFieldTypeInputLevelJoin.get("required"))
         );
 
-        // Predicate for displayOnCreate in input level
-        Predicate displayOnCreateInputLevelPredicate = criteriaBuilder.and(
-            criteriaBuilder.equal(datasetFieldTypeRoot, datasetFieldTypeInputLevelJoin.get("datasetFieldType")),
-            criteriaBuilder.isTrue(datasetFieldTypeInputLevelJoin.get("displayOnCreate"))
-        );
-
         // Create a subquery to check for the absence of a specific DataverseFieldTypeInputLevel.
         Subquery<Long> subquery = criteriaQuery.subquery(Long.class);
         Root<DataverseFieldTypeInputLevel> subqueryRoot = subquery.from(DataverseFieldTypeInputLevel.class);
@@ -969,19 +963,10 @@ private Predicate buildFieldPresentInDataversePredicate(Dataverse dataverse, boo
         // Otherwise, use an always-true predicate (conjunction).
         Predicate displayedOnCreatePredicate = onlyDisplayedOnCreate
                 ? criteriaBuilder.or(
-                // 1. Field marked as displayOnCreate in input level
-                displayOnCreateInputLevelPredicate,
-
-                // 2. Field without input level that is marked as displayOnCreate or required
-                criteriaBuilder.and(
-                    hasNoInputLevelPredicate,
-                    criteriaBuilder.or(
+                criteriaBuilder.or(
                         criteriaBuilder.isTrue(datasetFieldTypeRoot.get("displayOnCreate")),
                         fieldRequiredInTheInstallation
-                    )
                 ),
-
-                // 3. Field required by input level
                 requiredAsInputLevelPredicate
         )
                 : criteriaBuilder.conjunction();

src/main/java/edu/harvard/iq/dataverse/DatasetFieldValueValidator.java

@@ -241,10 +241,6 @@ private boolean isValidDate(String dateString, String pattern) {
         return valid;
     }
 
-    public boolean isValidAuthorIdentifier(String userInput, Pattern pattern) {
-        return pattern.matcher(userInput).matches();
-    }
-
     // Validate child fields against each other and return failure message or Optional.empty() if success
     public Optional<String> validateChildConstraints(DatasetField dsf) {
         final String fieldName = dsf.getDatasetFieldType().getName() != null ? dsf.getDatasetFieldType().getName() : "";

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -1855,7 +1855,6 @@ private void updateDatasetFieldInputLevels() {
                 if (dsf != null){
                     // Yes, call "setInclude"
                     dsf.setInclude(oneDSFieldTypeInputLevel.isInclude());
-                    dsf.getDatasetFieldType().setDisplayOnCreate(oneDSFieldTypeInputLevel.isDisplayOnCreate());
                     // remove from hash
                     mapDatasetFields.remove(oneDSFieldTypeInputLevel.getDatasetFieldType().getId());
                 }

src/main/java/edu/harvard/iq/dataverse/Dataverse.java

@@ -438,12 +438,6 @@ public boolean isDatasetFieldTypeInInputLevels(Long datasetFieldTypeId) {
                 .anyMatch(inputLevel -> inputLevel.getDatasetFieldType().getId().equals(datasetFieldTypeId));
     }
 
-    public boolean isDatasetFieldTypeDisplayOnCreateAsInputLevel(Long datasetFieldTypeId) {
-        return dataverseFieldTypeInputLevels.stream()
-                .anyMatch(inputLevel -> inputLevel.getDatasetFieldType().getId().equals(datasetFieldTypeId) 
-                         && inputLevel.isDisplayOnCreate());
-    }
-
     public Template getDefaultTemplate() {
         return defaultTemplate;
     }

src/main/java/edu/harvard/iq/dataverse/DataverseFieldTypeInputLevel.java

@@ -58,16 +58,14 @@ public class DataverseFieldTypeInputLevel implements Serializable {
     private DatasetFieldType datasetFieldType;
     private boolean include;
     private boolean required;
-    private boolean displayOnCreate;
 
     public DataverseFieldTypeInputLevel () {}
 
-    public DataverseFieldTypeInputLevel (DatasetFieldType fieldType, Dataverse dataverse, boolean required, boolean include, boolean displayOnCreate) {
+    public DataverseFieldTypeInputLevel (DatasetFieldType fieldType, Dataverse dataverse, boolean required, boolean include) {
         this.datasetFieldType = fieldType;
         this.dataverse = dataverse;
         this.required = required;
         this.include = include;
-        this.displayOnCreate = displayOnCreate;
     }    
 
     public Long getId() {
@@ -117,14 +115,6 @@ public void setRequired(boolean required) {
         this.required = required;
     }
 
-    public boolean isDisplayOnCreate() {
-        return displayOnCreate;
-    }
-
-    public void setDisplayOnCreate(boolean displayOnCreate) {
-        this.displayOnCreate = displayOnCreate;
-    }
-
     @Override
     public boolean equals(Object object) {
         // TODO: Warning - this method won't work in the case the id fields are not set