Automatic translation for languages with plurals

[ditwoo]

Hello!

I'm currently exploring the possibility of using automatic translation (with OpenAI) within the project. I was wondering if it's feasible to handle languages that have different pluralization rules?

I've successfully run automatic translations for singular forms, but it seems Weblate is defaulting to translating the "many" form for plurals.

Would it be possible to point me towards any documentation or a guide that explains how to configure automatic translations via OpenAI, specifically for handling plural forms? Any help would be greatly appreciated!

Thanks so much!

[nijel]

Weblate has a logic to map source and translation plurals reasonably:

weblate/weblate/lang/models.py

Lines 1154 to 1281 in 163eb55

	class PluralMapper:
	instances: WeakValueDictionary[tuple[str, str], PluralMapper] = (
	WeakValueDictionary()
	)
	def __new__(cls, source_plural: Plural, target_plural: Plural):
	key = (source_plural.formula, target_plural.formula)
	obj = cls.instances.get(key)
	if obj is None:
	obj = cls.instances[key] = super().__new__(cls)
	return obj
	def __init__(self, source_plural: Plural, target_plural: Plural) -> None:
	self.source_plural = source_plural
	self.target_plural = target_plural
	self.same_plurals = source_plural.same_as(target_plural)
	@cached_property
	def target_map(self) -> tuple[tuple[int | None, int | None], ...]:
	exact_source_map: dict[int, int] = {}
	all_source_map: dict[int, int] = {}
	for i, examples in self.source_plural.examples.items():
	if len(examples) == 1:
	exact_source_map[int(examples[0])] = i
	else:
	for example in examples:
	try:
	value = int(example)
	except ValueError:
	continue
	all_source_map[value] = i
	target_plural = self.target_plural
	result: list[tuple[int | None, int | None]] = []
	last = target_plural.number - 1
	for i in range(target_plural.number):
	examples = target_plural.examples.get(i, [])
	if len(examples) == 1:
	number = int(examples[0])
	if number in exact_source_map:
	result.append((exact_source_map[number], None))
	elif number in all_source_map:
	result.append((all_source_map[number], number))
	else:
	result.append((-1, number))
	elif i == last:
	result.append((-1, None))
	else:
	result.append((None, None))
	return tuple(result)
	def map(self, unit: Unit, other_unit: Unit | None = None) -> list[str]:
	if other_unit is not None:
	source_strings = other_unit.get_target_plurals()
	else:
	source_strings = unit.get_source_plurals()
	if self.same_plurals or len(source_strings) == 1:
	strings_to_translate = source_strings
	elif self.target_plural.number == 1:
	strings_to_translate = [source_strings[-1]]
	else:
	strings_to_translate = []
	format_check = next(
	(
	check
	for check in CHECKS.values()
	if (
	isinstance(check, BaseFormatCheck)
	and check.enable_string in unit.all_flags
	and check.plural_parameter_regexp
	)
	),
	None,
	)
	for source_index, number_to_interpolate in self.target_map:
	s = "" if source_index is None else source_strings[source_index]
	if s and number_to_interpolate is not None and format_check:
	s = format_check.interpolate_number(s, number_to_interpolate)
	strings_to_translate.append(s)
	return strings_to_translate
	@staticmethod
	def get_other_units(
	units: list[Unit] | UnitQuerySet, language: Language
	) -> dict[int, Unit]:
	if not units:
	return {}
	component = units[0].translation.component
	try:
	translation = component.translation_set.get(language=language)
	except ObjectDoesNotExist:
	return {}
	return {
	other.id_hash: other
	for other in translation.unit_set.filter(
	state__gte=STATE_TRANSLATED,
	id_hash__in={unit.id_hash for unit in units},
	)
	}
	def map_units(
	self, units: list[Unit] | UnitQuerySet, other_units: dict[int, Unit] | None
	) -> None:
	other: Unit | None
	for unit in units:
	if other_units is None:
	other = None
	elif unit.id_hash not in other_units:
	# Unit strings not available
	unit.plural_map = []
	continue
	else:
	other = other_units[unit.id_hash]
	unit.plural_map = self.map(unit, other)
	def zip(self, sources: list[str], targets: list[str], unit: Unit):
	if len(sources) != self.source_plural.number:
	msg = "length of `sources` doesn't match the number of source plurals"
	raise ValueError(msg)
	if len(targets) != self.target_plural.number:
	msg = "length of `targets` doesn't match the number of target plurals"
	raise ValueError(msg)
	if self.same_plurals:
	return zip(sources, targets, strict=True)
	return [
	(sources[-1 if i is None else i], targets[j])
	for (i, _), j in zip(self.target_map, range(len(targets)), strict=True)
	]

For LLM services like OpenAI, it might be useful to pass the actual plural info as well while translating, but Weblate currently doesn't do that. The current prompt is optimized for cost and performance and cannot pass additional per-string context.

Related topics:

• OpenAI: providing more context while translating #13398
• https://github.com/orgs/WeblateOrg/discussions/14273

[ditwoo]

Thank you, @nijel, for your helpful answer. It appears that PluralMapper may encounter some challenges with translation mappings in this instance.

I've attached a GIF that illustrates the issue – it shows that while OpenAI can suggest translations for the latest plural form, those suggestions don't seem to apply to other plural forms consistently.

In other words, when I click on "Clone to translation", I expect the translation to be placed into the correct plural input text field.

So, I thought, would it be possible to implement a solution where a translation request is made to OpenAI for each plural form (similar to how it works for singular forms) and then show OpenAI suggestions with the plural indicator?

If so, could you please provide some high-level instructions on what modifications would be needed in the codebase to achieve this functionality? Asking this because I could create a PR with a fix.
Any guidance would be greatly appreciated!

[nijel]

The BaseMachineTranslation.translate generates text for each plural using PluralMapper and passes that to the _translate method:

weblate/weblate/machinery/base.py

Lines 506 to 519 in 7d7c86d

	plural_mapper = PluralMapper(source_plural, target_plural)
	alternate_units: dict[int, Unit] | None = None
	if not translating_from_source:
	alternate_units = plural_mapper.get_other_units([unit], source_language)
	plural_mapper.map_units([unit], alternate_units)
	translations = self._translate(
	mapped_source_language,
	target_language,
	[(text, unit) for text in unit.plural_map],
	user,
	threshold=threshold,
	)
	return [translations[text] for text in unit.plural_map]

If there are no cached results, download_multiple_translations is called what is actually implemented in the BaseOpenAITranslation class and that in turn constructs a single prompt for all these strings using the _get_prompt method and it finally reaches OpenAI in the _download method.

The plural information from the PluralMapper is currently not passed along (only the source string and unit), so the API of all methods currently consuming tuple[str, Unit | None] (or a list of that) would have to be extended to pass plural info. Then OpenAI _download method could utilize this info.

[ditwoo]

Thank you for your response and the helpful information.

I'll be working on extending this functionality and will share my progress with you as soon as I have results. Hopefully, this will work as expected.

[ditwoo]

@nijel I've an update. I've implemented the manual translation of plurals using OpenAI suggestions, and it seems to be working as expected from my perspective.

As you suggested, I've added an extra argument with a PluralMapper object for some functions (this argument is always None for singulars). I'm planning to create a pull request with these changes soon.

On a related note, I had a question about the automatic translations. Could you point me in the right direction? I'm trying to understand which functions need to be adjusted to ensure automatic translations handle plurals correctly. It seems like the automatic translations via OpenAI integration work well with singular translations, but it's currently skipping the plural forms.
I'm considering the updating of BatchMachineTranslation.batch_translate to enable automatic translation for plurals.
Any guidance would be greatly appreciated!

[ditwoo]

Looks like I've finished with plural translations in "manual mode" and using the Automatic translations via OpenAI.

@nijel could you please check my draft PR when you will have free time.

[nijel]

There should be no difference in handling plurals between batch and regular mode. I'll get to the PR, but not this week.