Including Cyrillic letters, both the usual set (taken from kpv) as we…

…ll as the special Buryaad letters (taken from the phonology.twolc declaration).
giellalt · Jun 29, 2023 · f46344a · f46344a
1 parent 2648166
commit f46344a
Showing 1 changed file with 2 additions and 0 deletions.
diff --git a/tools/tokenisers/tokeniser-disamb-gt-desc.pmscript b/tools/tokenisers/tokeniser-disamb-gt-desc.pmscript
@@ -66,6 +66,8 @@ Define alphabet "a-z" !! * lower-case ASCII
                |"A-Z" !! * upper-case ASCII
                |Lst({àáâãāăȧäảåǎȁȃąạḁæǽǣèéêẽēĕėëẻěȅȇẹȩęḙḛìíîĩīīĭi̇ïỉǐịįȉȋḭɨòóôõōŏȯöỏőǒȍȏơǫọɵøờớỡởợǭộǿœùúûũūŭüủůűǔȕȗưụṳųṷṵừứữửựʉỳýŷỹȳẏÿỷƴỵɏÀÁÂÃĀĂȦÄẢÅǍȀȂĄẠḀÆǼǢÈÉÊẼĒĔĖËẺĚȄȆẸȨĘḘḚÌÍÎĨĪĪĬİÏỈǏỊĮȈȊḬƗÒÓÔÕŌŎȮÖỎŐǑȌȎƠǪỌƟØỜỚỠỞỢǬỘǾŒÙÚÛŨŪŬÜỦŮŰǓȔȖƯỤṲŲṶṴỪỨỮỬỰɄỲÝŶỸȲẎŸỶƳỴɎšžčđðíŋňŧñńŠŽČĐÐÍŊŇŦÑ})
                !! * select extended latin symbols
+               | Lst({абвгдеёжӝзӟиӥйкАБВГДЕЁЖӜЗӞИӤЙКлмноөпрстуүфхһцчӵшЛМНОӨПРСТУҮФХҺЦЧӴШщъыьэюяіӧЩЪЫЬЭЮЯІӦ})
+               !! * extended cyrillic
                | "0-9" !! ASCII digits
                | Lst({_§°†}) !! * select symbols
                !! * Combining diacritics as individual symbols,