Using update_form on a custom form with duplicate fields

[spwoodcock]

Describe the bug

• We have a mandatory_fields.xlsx XLSForm.

• If this is merged via update_forms with a custom form that contains duplicate fields, then we get some duplicate columns.

• For example the mandatory_fields.xls contains the status field. The current buildings.xls also contains a status field.

• When these forms are merged, we get a duplicate save_to column called save_to.1:

  

To Reproduce

• Run update_form.merge_sheets on buildings.xls in this repo.
• Save the output to a file and view.

Expected behavior

• We should warn users they have a duplicate field when running the function.
• Ideally we return an error and ask them to rename / remove the field.
• In the end, the mandatory forms field should take precedence.

Additional context

• The form still works fine with the save_to.1 column.
• This isn't high priority as it doesn't break anything (as far as I am aware).

[manjitapandey]

Encountered cases:
First: I have encountered forms with only label field not specifying which language is it. Its mostly seen in forms where they are not concerned more about different language and only include english as default language. This can be one of the error where fields won't be visible in odk, though its very rare.
Second: The cases where label field may include spaces label::english (en) Though its not the standard behavior, odk doesn't rise error during validation and mostly beginner tend to make this kind of error. - Handled
Third: The cases where language may include number like label::english(1). - Handled
Forth: The cases where language short form are not included like label::English - Not handled
Fifth: The cases where label field contains capital letter like label::English(en) - Handled

[spwoodcock]

Thanks for investigating these!

In scenario 1, if the user doesn't care about languages and wants to set it to their device default, they will simply use label. Its not preferred over setting the language explicitly, but its possible.

However we can't identify which language they want to use, so should just select English as a default. The simplest way for us to handle that is to remove all translation columns, and update column label::english(en) to be label for the column matching.

Scenario 4 could be solvable by fuzzy matching the language name and keeping a dictionary matching language names to their two letter iso code. But to be honest, that's just user error and I dont think we should go to so much effort for it. Instead, this problem is aided when we create some sort of form builder directly in FMTM.