Enhancements/Questions on Localization

[HermesSbicego-Laser]

Hi guys, I have a suggestion/question for you, related to how localization in Orchard works: in particular I would like to know which plans does the community have in order to improve it... some examples, to be more specific:

1. Container/Containable items have no automations in order to keep container and containable items synchronized based on their culture
2. Same for Taxonomies and terms, Taxonomy picker field, Content Picker Fileds and so on

It would be nice to have a site settings (or any more granular setting) that tells to our site/part/field what to do if it finds translated contents

We did something for taxonomies/terms/taxonomy field and ContentPickerField for 1.8.2; our approach unfortunately has some negative effects because we had to suppress some default behaviour of Orchard (drivers and handlers); for example a behaviour we had to override was in taxonomy: when a translation for a taxonomy is created the modules creates a new

taxnomy.Name + " Term"

content Type, so we had a proliferation of unnecessary ContentTypes "ProductTypeTerm", "TipoProdottiTerm", "TypesDeProduitsTerm", and we could not accept this effect (collateral effect due to how taxonomies was designed)...

However, being localization a crucial feature of CMS, we would like to know your opinion on that in order to plan our next steps

Another philosphical question: should be the Orchard.Localization Module manage how other modules should react with localizable related contents? Or should every module be designed to consistently manage localization?
Personally I prefer the second approach, maybe with some point of extension. Not sure how: IEventHandler called by LocalizationController? adding ContentPartDriver method?
What do you think?

Thank you for sharing your opinion with me.

[sebastienros]

I will setup a design meeting to talk about localization.

[HermesSbicego-Laser]

@sebastienros , nice. We will partecipate, probably @MatteoPiovanelli-Laser and me. If it's possible we will prefer a date after November 4... tnks

[MatteoPiovanelli]

Hello. @sebastienros, is there any news on the discussion about localization?

[sebastienros]

Is November 4th ok for everyone, anytime between 10am PST to 1pm PST (18 to 21 in Central Europe right now)

[sebastienros]

/cc @jersiovic @urbanit

[urbanit]

It works for me.

[urbanit]

I mean 10am pst works for me...

[jersiovic]

Ok 10am pst also works for me

[jersiovic]

To be sure we don't forget points that IMO are interesting here it is a list of needs that IMO is desirable localization solves:

LOCALIZATION NEEDS

1. To localize data for some fields of a content item.

2. To share unlocalized data between all cultures of a content item.

3. To have permissions per culture (or unlocalized) of a content item.

4. To be able of publishing and set author data only of one culture (or unlocalized)

5. When localizing taxonomies that has exactly same terms for each culture but with translated captions:

   • a. To don't need one taxonomy type per culture
   • b. To be able of change the hierarchy of terms easily for all the cultures at the same time.
   • c. To be able of removing a term and its localized versions easily.

6. To be able of query localized content items by culture through Projections and through Content Manager

7. To be able of query content items filtering by its pointed localized content through Projections and through Content Manager (We need all localized contents of a pointed content is considered the same content).

   Example: You have a non-localizable BoatOffer type with:

   • A Content Picker field pointing to a localizable content items of Boat type.
   • A Taxonomy field pointing to a localizable term of a taxonomy.
     It is desirable Orchard out of the box allows us to filter boatoffers by a term of a localized taxonomy through Projections and through ContentManager without adding extra complexity.

8. To be able of grouping content items by its pointed localized content.

   Example: For the previous example we need an efficient solution which provides BoatOffers grouped by pointed taxonomy term considering same term all those localized terms of the same term.

9. Help to maintain things simple in scenarios where you reference localized content items.

Example: You have class OrderType containing OrderLines with content items ids pointing to content items of type Product. It is desirable code used to deal with Products (filter, group, access data) would be the same if Products is non localizable initially and in the future is changed to localizable. With current solution problem after you make Product localizable with another culture different than the one used by default is you need all products duplicated and need to deal with this duplication everywhere.
10. Allow to work with taxonomies with different hierarchies per culture.
11. Allow to work easily with content items that are not localized in all cultures.

LOCALIZATION SCENARIOS

1. Contents that is expected they will have all its localized fields fulfilled for all the cultures of the site.
2. Contents that are only available for certain cultures.

[jersiovic]

This is my current view of the two solutions discussed by the moment in relation to the NEEDS defined previously. Maybe it is interesting for the Friday discussion.

LOCALIZATION SOLUTIONS

CulturePart pointing to related localized Content Items

Pros:

• Allows Need 1 and Need 2 based on redundancy. Requires extra process for syncing unlocalized data between content items of each culture. Writes are less efficient.
• Allows without extra work Need 4.
• Allows without extra work Need 10.
• Allows without extra work Need 11.

To-Do:

• Need 3: Doesn't allow to set access permission per culture. Currently you can provide access to owner of a content item not by culture. We would need to implement a mechanism for enable that feature. If we want to controlate permissions for unlocalized fields, this solution doesn't fit with that need.

Cons:

• Need 3: It doesn't provide solution for permissions, author and publishing of only unlocalized data.
• Need 5: It doesn't provide solution for Need 5.b and 5.c. We would need to distinguish in some way taxonomies that have same hierarchy of terms for all its cultures from those that not. For the first ones Orchard should offer ways of managing them in as simpler way despite they are separated content items.
• It doesn't provide a solution for Need 7 and Need 8 leaving the user and developer alone with this complexity. A possible solution can be to store always the master content item id in fields used for relate a content item with another.
  Problems are that extra accesses to db will be needed in this cases:
  • When saving a content item to get the master one.
  • When content item is rendered for showing the localized term with the same culture as the one required by the user. When you have 12 taxonomy fields in a content item those things matter.
• It doesn't provide a solution for Need 9, so complexity arise to other parts of you app that are agnostic of orchard.

A content item contains all its localized data

Pros:

• Allows Need 1 and Need 2 in a very simple and efficient way.
• Allows without extra work Need 5.
• Allows without extra work Need 6
• Allows without extra work Need 7.
• Allows without extra work Need 8.
• Allows without extra work Need 9.
• Allows without extra work Need 10 (if Need 4 is solved as commented in cons).
• Allows without extra work Need 11 (if Need 4 is solved as commented in cons).

To-Do:

• Need 3: As in previous solution we need a mechanism to set Permissions per culture at Role level. However If we want to controlate permissions for unlocalized fields, this solution fit very well with that need.

Cons:

• For Need 4) we need to stablish a mechanism to set author and publication of data per culture within a content item. Maybe we need an extra table called ContentItemCultureRecord with PK(ContentItemId, CultureId) that contains columns Author and Published cultures.

When we edit a content item with localizable data through Admin UI we will see non localized data and localized data can be seen/edited using a combo to choose the culture in which we want to see those fields.

When editing a content item, those fields we don't have permissions to modify will appear in readonly mode.

-- EDITED --

Save and Publish buttons will store or Publish content item of only data in the culture for which the user has permissions. Only data of cultures with any non-blank data fulfilled will be marked as published. That means when you don't fulfill data for one culture that Content item won't be considered as published for that cultured.
Alternatively we can offer a setting at conten type level that mark its Content items always available for all cultures no matter what we fulfill when we store it.

[jersiovic]

Please take a look to this interesting video from the Drupal community related with this topic:
Vimeo presents: Drupal 7/8 Content Translation models http://vimeo.com/27068202

[HermesSbicego-Laser]

Works also for me 10am pst.

[sebastienros]

Meeting notes

default: en-US

Page content type With no fields (body, title, url)
Home page / -> en-US (id = 10) -> (Id=10 , TSId = 10)
Pagina de inicio /pagina-de-inicio -> es-ES (micId: 10) Id =11, TSId = 10
Pagina di inicia /pagina-di-inicia -> it-IT (micId: 10) Id =12, TSId = 10

Product, Title, Weight field, not localizable, Draftable

Motorbike -> Weight: 50kg -> Helmet
Motocicleta (es-ES) -> Weight: 50kg -> Casco
Motocicleta (it-IT) -> Weight: 50kg -> Casci

O1

Master content item id === Translation sets ids

• Field localization doesn't work
  • Synchronize data across translation set
    • Question: Can we change it (a neutral) in any translation, can we display the value in every localization, or just on the main one
      • Yes -> propagates on "publish", and also updates all the drafts if any
    • If we rollback a localized item, we need to syncronize its fields also
    • Field Setting to mark it "culture neutral" / "non localizable"
    • Content handler to detect the fields, on publication/rollback
• You can delete any translation independently
  • Unpublish master content item
• How to select a content neutral content item (from any culture, not culture specific, i.e. the TS id)
  • We need to select the MCI id
• The Content Item Picker field shows a drop down with all languages (can/should we store the MCId also?)
  • Option (Settings) to ensure the selected items have the same culture as the current culture
    • Sub-option to translate the selection automatically when a localized version is created
• Should we localize menus
  • We can have different menus for each language, in text and in structure
  • Today we can create different menus, and use widget layers to render for each language
    • Solution: Add a LocalizationPart to Widgets, each widget in the Widgets Translation Set, will point to a different menu, when the widget is rendered, it will be filtered without the need of a Layer. It works today if you create new Widgets in different languages in different Translation sets, not by translating an existing widget (see bug BUG: localized widgets do not seem to be shown correctly #7347)
  • Optionally Be able to add a LocalizationPart to Menu and add a culture selector in the Menu editor using this part, so that synchronizing menus across languages is easier. It's independent from the previous item.
• When a Clone is created to create a new localization, we should be able to customize the cloning event to create
  automatic translations. Example: Convert a Content Item Picker to its translation, or Taxonomy terms.
  • Add a property in the ClonedContentContext for the target culture

-- Different menu structures

"Menu English"
mi1, mi2

"Menu Italian"
mi1, mi2

(Localization Part to Menu Widget)

New Menu Widget

"English menu everywhere"
Menu -> Menu English
Localization part Culture -> English

New translation
"English menu everywhere"
Menu -> Menu Italian
Localization part Culture -> English

-- Same menu structure (TODO)
Translate menu items
Localization part to "Menu Items"

Taxonomies: Same as Menu items?

---

Issue, we only have a MSCId if the localization part is defined

Cid Mcid
1 1
2 2
3 2
4 NULL
5 NULL

• Since 1.10 all master contents have their MCIds to 0, should it be the same id as the content item id.

[MatteoPiovanelli]

Hi,

@HermesSbicego-Laser and I debriefed offline a bit. We have some comments on what we discussed regarding the ContentPicker. Specifically, the following points from @sebastienros' notes:

How to select a content neutral content item (from any culture, not culture specific, i.e. the TS id)

• We need to select the MCI id

The Content Item Picker field shows a drop down with all languages (can/should we store the MCId also?)

• Option (Settings) to ensure the selected items have the same culture as the current culture
  • Sub-option to translate the selection automatically when a localized version is created

Adding the dropdown with the language selection in the "list" window that appears when clicking "Add" for a ContentPickerField is pretty straightforward, and does not actually require any dependency with the Orchard.Localization module (like it's done in the contents list view).

The Option and Sub-option I think will require a dependency from Orchard.Localization. To avoid introducing that dependency to the Orchard.ContentPicker feature (thus avoid back-compatibility issues) we would go and create another feature in the Orchard.ContentPicker module. Something along the lines of Orchard.ContentPickerLocalizationExtensions.
In that feature, we would add the settings we discussed, and the corresponding functionalities. A lot of those we actually have already as a custom extension we had built in O1.8, so it should not take too long to get a PR out for that.
Some questions remain for the sub-option:

• When should the translation happen? I think it should be on a save/publish, since when we are doing the translation (click on the "new translation" link) we have not yet selected the target localization. Also, since the clone feature is not yet in 1.10.x, that should come later anyway.
• How should we handle selected content items for which there is no translation? I would put a further setting to allow the users to choose whether they want to keep the content in the "wrong" language, or get rid of it (i.e. references to those items for which we find no pre-existing localization to the "correct" culture should be removed entirely, rather than replaced). The first case would be default for me.

That feature, as described, should pose no compatibility issue. I think it may not even have to change any existing code-files in the Orchard.ContentPicker module.

Tomorrow I will start working on this, because I have previous work we did to start from and adding the rest that we discussed should not be too much work. Any input and comments are appreciated.

Best,
Matteo