CLDR-11874 Add conversion of subdivisions to json #4255

arjanmels · 2024-12-22T18:32:25Z

CLDR-11874

This PR completes the ticket.

ALLOW_MANY_COMMITS=true

CLAassistant · 2024-12-22T18:32:31Z

All committers have signed the CLA.

jira-pull-request-webhook · 2024-12-22T18:35:11Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2025-01-11T13:07:33Z

Notice: the branch changed across the force-push!

tools/cldr-code/src/main/java/org/unicode/cldr/json/LdmlConvertRules.java is now changed in the branch
tools/cldr-code/src/main/resources/org/unicode/cldr/json/JSON_config_supplemental.txt is now changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

arjanmels · 2025-01-11T13:08:27Z

Also added containment for subdivisions.

jira-pull-request-webhook · 2025-01-12T15:31:13Z

Notice: the branch changed across the force-push!

tools/cldr-code/src/main/java/org/unicode/cldr/json/Ldml2JsonConverter.java is different
tools/cldr-code/src/main/java/org/unicode/cldr/json/LdmlConvertRules.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

srl295

Code looks great need to test it

srl295 · 2025-01-13T18:31:04Z

        "autas": "Tasmania",
        "auvic": "Victoria",
        "auwa": "Western Australia",
        "AW": "Aruba",
        "AX": "Åland Islands",
        "azabs": "Absheron",
        "azaga": "Agstafa",
        "azagc": "Aghjabadi",
        "azagm": "Agdam",

looks like it includes the territories also, not just subdivisions.

srl295 · 2025-01-13T18:42:10Z

Files generated

`cldr-core/supplemental/subdivisionContainment.json`

{
  "supplemental": {
    "version": {
      "_unicodeVersion": "16.0.0",
      "_cldrVersion": "46"
    },
    "subdivisionContainment": {
      "AD": {
        "_contains": [
          "ad02",
          "ad03",
          "ad04",
          "ad05",
          "ad06",
          "ad07",
          "ad08"
        ]
      },
…

cldr-subdivisions/subdivisions/en.json

{
  "subdivisions": {
    "localeDisplayNames": {
      "subdivisions": {
        "ad02": "Canillo",
        "ad03": "Encamp",
        "ad04": "La Massana",
        "ad05": "Ordino",
        "ad06": "Sant Julià de Lòria",
        "ad07": "Andorra la Vella",
        "ad08": "Escaldes-Engordany",
        "aeaj": "Ajman",
        "aeaz": "Abu Dhabi",
        "aedu": "Dubai",
        "aefu": "Fujairah",
        "aerk": "Ras al-Khaimah",
        "aesh": "Sharjah",
        "aeuq": "Umm al-Quwain",
        "afbal": "Balkh",
…

Oddly it only generated en.json and ha.json - do you see something different?

I think those are the only two that aren't provisional.

But also it seems like the common/main and common/subdivision trees aren't being merged together properly, because we should see this data from French for example

                <subdivisions>
                        <subdivision type="gbeng">Angleterre</subdivision>
                        <subdivision type="gbsct">Écosse</subdivision>
                        <subdivision type="gbwls">Pays de Galles</subdivision>
                </subdivisions>

arjanmels · 2025-01-13T19:17:14Z

Oddly it only generated en.json and ha.json - do you see something different?
I think those are the only two that aren't provisional.

You're right for my local builds I set: DRAFTSTATUS=provisional in my local-config.sh

arjanmels · 2025-01-13T19:29:25Z

looks like it includes the territories also, not just subdivisions.

This is actually already in the source .xml. Not all territories are included (e.g. "NL", "FR", "GB), it seems only territories, which are not countries. Anyway the json is a direct reproduction of the xml.

srl295 · 2025-01-13T19:36:34Z

@macchiati indeed it does seem like this is in the source - is this a bug?

https://github.com/unicode-org/cldr/blob/main/common/subdivisions/en.xml#L211-L212

macchiati · 2025-01-13T22:19:09Z

        "autas": "Tasmania",
        "auvic": "Victoria",
        "auwa": "Western Australia",
        "AW": "Aruba",
        "AX": "Åland Islands",
        "azabs": "Absheron",
        "azaga": "Agstafa",
        "azagc": "Aghjabadi",
        "azagm": "Agdam",

looks like it includes the territories also, not just subdivisions.

I think what is happening is that some subdivisions are aliases of country codes. We're treating the country codes as predominant (we'll have better data for them). For the JSON version, I think we can skip the country codes; implementers are on notice that they should use the country codes.

srl295 · 2025-01-13T22:25:44Z

I think what is happening is that some subdivisions are aliases of country codes. We're treating the country codes as predominant (we'll have better data for them). For the JSON version, I think we can skip the country codes; implementers are on notice that they should use the country codes.

The XML has

			<subdivision type="AW">Aruba</subdivision>
			<subdivision type="AX">Åland Islands</subdivision>
			<!-- AW : Aruba : NO SUBDIVISIONS
				 AX : Åland Islands : NO SUBDIVISIONS
				 AZ : Azerbaijan
			-->

The spec doesn't address it. It looks like it's just included incorrectly. Unless the goal was to define a subdivision that was just "all of aruba"

The spec by the way at https://unicode.org/reports/tr35/dev/tr35-general.html#Contents gives examples with hyphens in the subdivision codes, and doesn't mention the special subdivisions in common/main.

<!-- from the spec -->
<subdivision type="AL-04">Fier County</subdivision>
<subdivision type="AL-FR">Fier</subdivision> <!-- in AL-04 : Fier County -->
<subdivision type="AL-LU">Lushnjë</subdivision> <!-- in AL-04 : Fier County -->
<subdivision type="AL-MK">Mallakastër</subdivision> <!-- in AL-04 : Fier County -->

macchiati · 2025-01-13T22:37:20Z

The spec doesn't address it. It looks like it's just included incorrectly.

That's a bug. We'll want to exclude the country codes from the XML data (and via that, the JSON), and also modify the spec to make it clear that implementations need to look at the subdivision aliases, such as

<subdivisionAlias type="usgu" replacement="GU" reason="overlong"/> <!-- Guam => Guam -->

The spec by the way at https://unicode.org/reports/tr35/dev/tr35-general.html#Contents has dashes in the subdivision codes, and doesn't mention the special subdivisions in common/main.

I was confused at first, because the TOC doesn't contain any subdivision codes. But I think you mean https://unicode.org/reports/tr35/dev/tr35-general.html#locale-display-name-fields, notably

<subdivision type="AL-04">Fier County</subdivision>
<subdivision type="AL-FR">Fier</subdivision> <!-- in AL-04 : Fier County -->
<subdivision type="AL-LU">Lushnjë</subdivision> <!-- in AL-04 : Fier County -->
<subdivision type="AL-MK">Mallakastër</subdivision> <!-- in AL-04 : Fier County -->

Is that what you meant?

Anyway, can you file a ticket for these two items? (They are related since we'll want to change the spec to document the relation between the subdivisions and codes.

srl295 · 2025-01-13T22:47:10Z

@arjanmels Can you look into:

supporting subdivisionAlias in the supplemental data (see above) from supplementalMetaData.xml - see https://unicode.org/reports/tr35/dev/tr35-info.html#supplemental-alias-information
I think the Arbua issue is a data bug as mentioned, so does not need to be suppressed in this PR.

srl295 · 2025-01-13T22:49:53Z

For the data issue: https://unicode-org.atlassian.net/browse/CLDR-18219 - looks like this was fixed before but is a regression

arjanmels · 2025-01-14T17:51:48Z

supporting subdivisionAlias in the supplemental data (see above) from supplementalMetaData.xml - see https://unicode.org/reports/tr35/dev/tr35-info.html#supplemental-alias-information

This was already being properly converted together with the language, territory etc. aliases (before this PR):

       "subdivisionAlias": {
          "cn11": {
            "_reason": "deprecated",
            "_replacement": "cnbj"
          },

srl295 · 2025-01-14T18:54:19Z

@arjanmels OK, sorry i missed it.

I'm going to approve this and look at the coverage issue separately

arjanmels · 2025-01-14T21:05:42Z

@srl295 great, thank you!

github-actions bot assigned arjanmels Dec 22, 2024

arjanmels changed the title ~~CLDR-11874: add conversion of subdivisions to json~~ CLDR-11874 Add conversion of subdivisions to json Dec 22, 2024

arjanmels force-pushed the convert_subdivisions_to_json branch from 3b9404d to 2be4ca9 Compare December 22, 2024 18:35

arjanmels force-pushed the convert_subdivisions_to_json branch from 2be4ca9 to 5b10a04 Compare January 11, 2025 13:07

arjanmels removed their assignment Jan 11, 2025

CLDR-11874 Add conversion of subdivisions to json

9afc856

arjanmels force-pushed the convert_subdivisions_to_json branch from 5b10a04 to 9afc856 Compare January 12, 2025 15:31

srl295 reviewed Jan 13, 2025

View reviewed changes

srl295 self-assigned this Jan 13, 2025

srl295 requested a review from macchiati January 13, 2025 18:42

srl295 approved these changes Jan 14, 2025

View reviewed changes

srl295 merged commit 7e1b129 into unicode-org:main Jan 14, 2025
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDR-11874 Add conversion of subdivisions to json #4255

CLDR-11874 Add conversion of subdivisions to json #4255

arjanmels commented Dec 22, 2024 •

edited by srl295

Loading

CLAassistant commented Dec 22, 2024 •

edited

Loading

jira-pull-request-webhook bot commented Dec 22, 2024

jira-pull-request-webhook bot commented Jan 11, 2025

arjanmels commented Jan 11, 2025

jira-pull-request-webhook bot commented Jan 12, 2025

srl295 left a comment

srl295 commented Jan 13, 2025

srl295 commented Jan 13, 2025

arjanmels commented Jan 13, 2025

arjanmels commented Jan 13, 2025 •

edited

Loading

srl295 commented Jan 13, 2025

macchiati commented Jan 13, 2025

srl295 commented Jan 13, 2025 •

edited

Loading

macchiati commented Jan 13, 2025

srl295 commented Jan 13, 2025

srl295 commented Jan 13, 2025

arjanmels commented Jan 14, 2025

srl295 commented Jan 14, 2025

arjanmels commented Jan 14, 2025

CLDR-11874 Add conversion of subdivisions to json #4255

CLDR-11874 Add conversion of subdivisions to json #4255

Conversation

arjanmels commented Dec 22, 2024 • edited by srl295 Loading

CLAassistant commented Dec 22, 2024 • edited Loading

jira-pull-request-webhook bot commented Dec 22, 2024

jira-pull-request-webhook bot commented Jan 11, 2025

arjanmels commented Jan 11, 2025

jira-pull-request-webhook bot commented Jan 12, 2025

srl295 left a comment

Choose a reason for hiding this comment

srl295 commented Jan 13, 2025

srl295 commented Jan 13, 2025

Files generated

cldr-core/supplemental/subdivisionContainment.json

cldr-subdivisions/subdivisions/en.json

arjanmels commented Jan 13, 2025

arjanmels commented Jan 13, 2025 • edited Loading

srl295 commented Jan 13, 2025

macchiati commented Jan 13, 2025

srl295 commented Jan 13, 2025 • edited Loading

macchiati commented Jan 13, 2025

srl295 commented Jan 13, 2025

srl295 commented Jan 13, 2025

arjanmels commented Jan 14, 2025

srl295 commented Jan 14, 2025

arjanmels commented Jan 14, 2025

arjanmels commented Dec 22, 2024 •

edited by srl295

Loading

CLAassistant commented Dec 22, 2024 •

edited

Loading

`cldr-core/supplemental/subdivisionContainment.json`

arjanmels commented Jan 13, 2025 •

edited

Loading

srl295 commented Jan 13, 2025 •

edited

Loading