How to transliterate from UTF8 to ASCII?

[kodersha]

Hey.

Yellow is a great cms, but I have a problem.
I want to create my blog on russian language. But since my notes are named in cyrillic, their url are also generated in cyrillic. Look like: myblog.site/blog/привет-мир
Is it possible add automatic transliteration to url? To make links look like this: myblog.site/blog/privet-mir
Thanks.

[schulle4u]

Yes, this is possible, at least half-automaticaly. Use the TitleSlug setting in your page to manually rewrite the URL slug of your post.

---
Title: привет мир
TitleSlug: privet-mir
---

To automatically add this setting upon page creation, you can edit your system/config/page-new-*.txt files and add it as an empty setting.

HTH
Steffen

[markseuffert]

And with some help of the API you can add a TitleSlug setting, if there isn't already one.

Here's an example extension file system/extensions/transliterate.php. It transliterates UTF8 characters to ASCII, for example Привет to Privet. I looked at different built-in PHP functions, in the end this is the solution that worked best on my web server. It can be extended to transliterate more languages if needed.

<?php
// Transliterate extension, experimental

class YellowTransliterate {
    const VERSION = "0.9.3";
    public $yellow;         //access to API

    // Handle initialisation
    public function onLoad($yellow) {
        $this->yellow = $yellow;
    }
    
    // Handle content file editing
    public function onEditContentFile($page, $action, $email) {
        if (($action=="precreate" || $action=="preedit") && !$page->isExisting("titleSlug")) {
            $replaceData = $this->getReplaceData($page->get("language"));
            $titleSlug = str_replace(array_keys($replaceData), array_values($replaceData), $page->get("title"));
            $titleSlug = preg_replace("/[^\pL\d\-\_\ ]/u", "-", $titleSlug);
            if ($page->get("title")!=$titleSlug) {
                $page->set("titleSlug", $titleSlug);
                $page->rawData = $this->yellow->toolbox->setMetaData($page->rawData, "titleSlug", $titleSlug);
            }
        }
    }
    
    // Return text replace data, UTF8 to ASCII
    public function getReplaceData($language) {
        $replaceData = array();
        if ($language=="de") {
            $replaceData = array(
                "Ä" => "Ae", "Ö" => "Oe", "Ü" => "Ue", "ẞ" => "Ss",
                "ä" => "ae", "ö" => "oe", "ü" => "ue", "ß" => "ss");
        } elseif ($language=="sv") {
            $replaceData = array(
                "Ä" => "A", "Ö" => "O", "Å" => "A",
                "ä" => "a", "ö" => "o", "å" => "a");
        } elseif ($language=="ru") {
            $replaceData = array(
                "А" => "A", "Б" => "B", "В" => "V", "Г" => "G", "Д" => "D", "Е" => "E", "Ё" => "Yo", "Ж" => "Zh",
                "З" => "Z", "И" => "I", "Й" => "J", "К" => "K", "Л" => "L", "М" => "M", "Н" => "N", "О" => "O",
                "П" => "P", "Р" => "R", "С" => "S", "Т" => "T", "У" => "U", "Ф" => "F", "Х" => "H", "Ц" => "C",
                "Ч" => "Ch", "Ш" => "Sh", "Щ" => "Sh", "Ъ" => "", "Ы" => "Y", "Ь" => "", "Э" => "E", "Ю" => "Yu",
                "Я" => "Ya",
                "а" => "a", "б" => "b", "в" => "v", "г" => "g", "д" => "d", "е" => "e", "ё" => "yo", "ж" => "zh",
                "з" => "z", "и" => "i", "й" => "j", "к" => "k", "л" => "l", "м" => "m", "н" => "n", "о" => "o",
                "п" => "p", "р" => "r", "с" => "s", "т" => "t", "у" => "u", "ф" => "f", "х" => "h", "ц" => "c",
                "ч" => "ch", "ш" => "sh", "щ" => "sh", "ъ" => "", "ы" => "y", "ь" => "", "э" => "e", "ю" => "yu",
                "я" => "ya");
        }
        // Add more languages here, see also:
        // https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php
        // https://github.com/gosimple/slug/blob/master/languages_substitution.go
        return $replaceData;
    }
}

Tested with Datenstrom Yellow 0.9. Hope it helps. 😄

[markseuffert]

I updated the example extension for the latest API. The code above supports German, Swedish and Russian. Please note that you have to transliterate based on the language. Or at least with the main language first. For example the character Ä is transliterated differently in different European languages.... German "Ä" => "Ae" and Swedish "Ä" => "A".

[kodersha]

Very well! That's what I need. Thanks you.