How to build third party Mozc client independently?

[kenhys]

背景

現在、DebianではMozcのクライアントとして、uim(uim-mozc)やfcitx5(fcitx5-mozc)をサポートしています。
そのサポート方法としては、Mozc本体にパッチをあててビルドする方式で実現しています。
このような方式ではなく、クライアントごとに独立してメンテナンスできるようにするにはどうするのがよいかというのが質問の背景です。

実現したいこと

• Mozc本体とは独立して、uimやfcitx5のクライアントモジュールをビルドして提供できるようにしたい
• ここでの独立して、というのはlibmozcclient-devのようなヘッダや再利用可能なライブラリーを含む開発パッケージを利用してビルドできるようになること（Mozc本体を同梱しないとビルドできないのを回避したい）

上記のようなことを実現するのに現実的なやりかただったり、進め方はどのようなものでしょうか？

関連情報

• DebianのMozcではまだupstreamにバージョンを追従できていないので、仮にupstreamであるgoogle/mozcで開発パッケージを提供しやすいかたちに(Bazelで)なっても、GYP向けにバックポートが必要な状況と認識しています。この認識はあっているでしょうか？
• uimでの本件と関連したissue Import Mozc support from MacUIM uim/uim#226

[kenhys]

issueとdiscussionのどちらにしようか迷いましたが、ふわっとした内容なので、いったんこちらに投げました。

[yukawa]

上記のようなことを実現するのに現実的なやりかただったり、進め方はどのようなものでしょうか？

My honest suggestion is to start from revisiting the current architecture of uim-mozc if Debian's policy strictly requires protobuf to be dynamically linked as a system library (e.g. libprotobuf.so).

Here is my understanding of how libuim-mozc.so works in Debian.

• libuim-mozc.so is dynamically loaded into the target application process (e.g. chromium)
• libuim-mozc.so is dynamically linked to the system-provided libprotobuf (e.g. libprotobuf.so.32).

Unfortunately, the above architecture has never been officially supported by Protocol Buffers.

• AddDescriptors is not thread safe, but multi-thread access may occur when the SO is opened through dlopen protocolbuffers/protobuf#22425

I don't know if we have this documented anywhere, but we don't support using dlopen this way unfortunately. We assume that all this initialization happens before main() and as a result, we can't handle protobuf generated code trying to register itself in our global data structures (such as the global descriptor pool) later on.

I think that if you want to dlopen a shared object containing protobuf gencode, then the only way to do it would be for that SO to have its own isolated copy of the protobuf runtime, with symbols hidden to prevent ODR violations. I have not personally tried this, but I believe it would work. You would not be able to pass protos across the SO boundary, though.

Perhaps what mozc_emacs_helper is doing would be a good starting point to explore how to decouple Mozc's code dependency. Once built, mozc_emacs_helper can work as a simple console tool to make an IPC to the mozc_server, which can be called from any process and any programing language without depending on the Mozc's client code.

See also the following request, which indeed came from the original uim-mozc author.

• Use static linking against protobuf even for Linux version of Mozc  #14

In related to the previous issue #13, please consider using statically linking with protobuf even for the Linux version of Mozc.

I request this because of the design of protobuf not to handle "bundle modules linking with protobuf" by dlopen(3) decently (related to issue #128 of protobuf). Specifically for mozc_tool in Linux, which uses Qt4 library as a GUI frontend and of course linking with shared protobuf library, it loads bundle modules with dlopen(3) when creating a input context. And as input module of uim for Qt do dlopen uim-mozc bundle (which is also linking with shared protobuf library dynamically) at that time, it will cause crash of protobuf as follows.

libprotobuf ERROR google/protobuf/descriptor_database.cc:56] File already exists in database: session/config.proto
libprotobuf FATAL google/protobuf/descriptor.cc:857] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):

I think this can be solved by using mozc_tool with protobuf library linked statically.

Here are other reports from uim-mozc users in Debian.

• mocz_tool with fatal errors (debian jessie) #387
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=721791

[kou]

At this moment, we don't know #1369 is sufficient or not to re-impalement uim-mozc.

I think that it's sufficient.

The current uim-mozc uses mozc::client::Client: https://github.com/e-kato/macuim/blob/7beac7ba000e0459a4dc933f3873b521664d2665/Mozc/uim/mozc.cc#L56

#1369 provides only mozc::client::Client::Call() and doesn't provide convenient wrappers on Call(). It may be inconvenient than using mozc::client::Client directly but there will not be feature differences.

it would make people start requesting Mozc team to update Mozc version every time JSON format changes.

I'm not familiar with Mozc's versioning policy but Mozc bumps version when new Mozc is released, right? I think that the normal release operation is sufficient. If people who use non released version request it, they should care about it by themselves. (I think that the Mozc team can say "wait for the next release".)

Currently it's 1:N mapping. Making it 1:1 mapping is a non-trivial feature creep.

Sorry. What does "1:N mapping" mean here?

mozc.commands.Output.6 may be named as candidates or candidate_window by ec4ff27#diff-26f5c72a08798c1f898bdaeaefc6a7752ad05cb0dd00f6f6392f66deecee388fR1342 , right? (6 -> candidates/candidate_window, 1:2)

[yukawa]

I think that it's sufficient.

Perhaps it's sufficient for uim-mozc, but what I wanted to discuss is what needs to be changed not only about the code but also about the entire process and/or compatibility promise from the viewpoint of Mozc team. In that sense, I probably should have said I wanted to fully understand what assumptions were going to be made in the uim-mozc side.

If people who use non released version request it, they should care about it by themselves. (I think that the Mozc team can say "wait for the next release".)

"they should care about it by themselves" is exactly what I suggested by asking client users to maintain protobuf descriptor that they use. I feel that's the most natural approach in the protobuf ecosystem. Perhaps one solution is to embed multiple versions of proto descriptor into mozc_client_cli so that its users can choose which version of proto descriptor to be used. In other words, I don't think trying to avoid the concept of proto descriptor itself just because it sounds complicated is the right approach.

Embedding multiple versions of proto descriptor into mozc_client_cli here means:

1. Define the concept of version (or "Edition") of Mozc's protocol. (E.g. "Edition 2025", "Ver 2.31.5851.102", ...)
2. Generate proto descriptor for each version (or "Edition")

protoc protocol/commands.proto --include_imports -o descriptors.binpb

3. Embed descriptors.binpb into the source

constexpr uint8 kVer2_29_5544[] = {0x0a, 0xda, 0x13, 0x0a, 0x1f, 0x70, 0x72, 0x6f, ...};
constexpr uint8 kVer2_32_5851[] = {0x0a, 0xda, 0x13, 0x0a, 0x1f, 0x70, 0x72, 0x6f, ...};

constexpr uint8 kEdition2025[] = {0x0a, 0xda, 0x13, 0x0a, 0x1f, 0x70, 0x72, 0x6f, ...};

4. Each client of mozc_client_cli specifies the highest protocol version (or protocol "Edition") that is compatible with the client. With that, the JSON format remain stable no matter which version of mozc_server is actually used.

That said, I have a feeling that just having uim-mozc source code under src/unix/uim would be much simpler though.

mozc.commands.Output.6 may be named as candidates or candidate_window by ec4ff27#diff-26f5c72a08798c1f898bdaeaefc6a7752ad05cb0dd00f6f6392f66deecee388fR1342 , right? (6 -> candidates/candidate_window, 1:2)

Yes. If a similar renaming happened N times in a single version, there are effectively N different ID-name mappings.

[kou]

I wanted to fully understand what assumptions were going to be made in the uim-mozc side

What information should I provide for it?

All client APIs used by uim-mozc?

Embedding multiple versions of proto descriptor into mozc_client_cli here means:

Thanks for the detailed approach. I feel that it's complex for Mozc team...

That said, I have a feeling that just having uim-mozc source code under src/unix/uim would be much simpler though.

I feel that the following parts in #1362 (reply in thread) isn't desired for Mozc team. (If I am a Mozc maintainer, I don't want to maintain uim integration...)

• Need to take care of random compile errors.
• May start receiving feature requests about uim-mozc.

I understand that Mozc team doesn't want to provide Mozc client API/tool. I'll try re-implementing mozc_client_cli #1369 like tool without Mozc client API. I think that I just need to re-implement https://github.com/google/mozc/blob/master/src/client/server_launcher.cc part.

[yukawa]

What information should I provide for it?

I think most of information I wanted to understand is already clarified, so further prototyping in the uim-mozc side is no longer necessary. Thank you for helping me understand what would be needed when introducing tools like mozc_client_cli.

Embedding multiple versions of proto descriptor into mozc_client_cli here means:

Thanks for the detailed approach. I feel that it's complex for Mozc team...

My understanding is that that complexity is inevitable as long as people want to use the JSON format instead of binary format for protocols that defined with protobuf. If it's not handled by the Mozc team, then each user of mozc_client_cli would start solving the same problem with their own approach. For instance, uim-mozc would simply ask the user to always use the latest version, but it's just one of possible solutions. Other users of mozc_client_cli may come up with different solutions in the end. The problem is that what solution would be picked up is completely out of the control from the viewpoint of the Mozc team. In such a situation, changing the behavior of mozc_client_cli becomes quite difficult. It's all about trade-offs.

That said, I have a feeling that just having uim-mozc source code under src/unix/uim would be much simpler though.

I feel that the following parts in #1362 (reply in thread) isn't desired for Mozc team. (If I am a Mozc maintainer, I don't want to maintain uim integration...)

• Need to take care of random compile errors.
• May start receiving feature requests about uim-mozc.

Again, it's all about trade-offs. What I feel is that just fixing random compile errors is still much easier task than maintaining JSON format against protobuf best practice. I'm still OK with maintaining uim-mozc under src/unix/uim as long as other Mozc maintainers are OK with it.

I understand that Mozc team doesn't want to provide Mozc client API/tool. I'll try re-implementing mozc_client_cli #1369 like tool without Mozc client API. I think that I just need to re-implement https://github.com/google/mozc/blob/master/src/client/server_launcher.cc part.

Just to make it clear, introducing tools like mozc_client_cli is still in my TODO list. What I cannot fully promise is when I can find time to work on it. Here are examples of things in my long TODO list:

• Implement a universal installer that supports both Windows x64 and ARM64.
• Dark theme support on Windows, macOS, and Linux.
• Redesigning of the IPC transport protocol on Windows.
• Redesigning of Windows candidate window UI.
• Investigation of using MSIX as a new installer format on Window.
• Investigation of new Windows IME API preview that Microsoft recently published.
• Investigation of cross platform ML runtime (e.g. LiteRT).

For now, my suggestion for the short term solution is to start from the simplest approach, which I believe is just putting uim-mozc into src/unix/uim directory. Then revisit this area later to explore to see if there is any other solution.

[kou]

FYI: uim implemented a mozc_client_cli like tool with mozc_emacs_helper and *.proto in Mozc as a workaround: uim/uim#318
uim will migrate to mozc_client_cli (or libmozc_client.a) once it's implemented.

[yukawa]

そのサポート方法としては、Mozc本体にパッチをあててビルドする方式で実現しています。

BTW, have you really checked if your latest commit doesn't introduce ABI breakage?

--- mozc-2.29.5160.102+dfsg.orig/src/unix/fcitx5/fcitx5.gyp
+++ mozc-2.29.5160.102+dfsg/src/unix/fcitx5/fcitx5.gyp
@@ -89,7 +89,7 @@
         'fcitx5_build_environment',
       ],
       'cflags_cc': [
-        '-std=c++17',
+        '-std=c++20',
       ],
       'cflags_cc!': [
         '-std=gnu++0x'

Seems that you were trying to set -std=c++20 only when building fcitx-mozc, but in general it's not guaranteed to be ABI compatible unless all the dependencies were carefully designed (and tested) to support such use cases. This is one of the reasons why the Mozc project prefers static linking over dynamic linking for C++ libraries. Enforcing the same compile options and macros significantly simplifies ABI compatibility stories.

See the following issues for instance. Mixing -std=c++17 and -std=c++20 is definitely not officially supported in abseil-cpp. Even mixing with and without NDEBUG macro have caused ABI mismatch there.

• Introducing C++20, dropping C++11 abseil/abseil-cpp#1127
• [Bug]: missing Mutex::Dtor on linux? abseil/abseil-cpp#1624
• dev-libs/re2: add 2023.08.01, mulitlibify abseil-cpp, add abseil-cpp-20230125.3 gentoo/gentoo#32281

If Debian wants to build fcitx-mozc with -std=c++20, the safest approach is to rebuild abseil-cpp and its all reverse dependencies with -std=c++20, or introduce a new variant of abseil-cpp which is built with-std=c++20.

[hiroyuki-komatsu]

First, thank you for initiating the discussion about improving package maintenance and for your efforts in updating the Mozc version.

I'm wondering if Mozc clients (e.g., uim, fcitx) really need to fork the Mozc repository. My thinking is that Mozc clients could instead use the Mozc repository as a third-party source code, similar to how Mozc uses Abseil and Protobuf, rather than forking it.

I'm also open to providing libmozc_client.a as a client library with minimal dependencies.

Here is a proof-of-concept (POC) of libmozc_client.a. With this POC, the libmozc-client-dev package could potentially include libmozc_client.a, mozc_client.h, commands.pb.h, and config.pb.h.

hiroyuki-komatsu@c82eabc

Please let me know your thoughts.

[yukawa]

As a combination of A and D, the clients can use Mozc as an external library without forking and merging it.
bzlmod is one of good options and it's option A. git submodule is also another option. Downloading an archived file is too (as our update_deps.py does).

Let me summarize what you are proposing.

E. Maintaining uim-mozc in a separate repository with some way to fetch Mozc source code as a build time dependency

• Mozc maintainers:
  • No major change from the current situation.
    • May need to send notices when making a breaking changes that affect downstream users (e.g. Notice: Build macros for Bazel will be changed #679)
• uim-mozc maintainers:
  • Need to figure out what's the best way to fetch Mozc source code as a build dependency (e.g. git submodule, downloading mozc source archive, ...)
  • Need to figure out how often the target Mozc version is updated. For example the Mozc team updates protobuf and abseil versions every few months, but uim-mozc may want to update the target Mozc git revisions more frequently.
  • Need to figure out whether bazel external dependencies should be included in the release tarball or not. If they are not included, such dependencies need to be downloaded later in the build environment to build uim-mozc, or some special actions need to be taken to support offline build.
• Linux distributions maintainers:
  • Need to figure out how to package mozc-server, uim-mozc, and fcitx-mozc. The tricky thing is that both uim-mozc and fcitx-mozc now pull Mozc source code as an external dependencies. In that sense, 3 copies of Mozc source code will be used there.
    • One options is just keeping doing the same approach, that is, use google/mozc source tarball with a series of patches.
  • Need to figure out how to address version-skew. In this approach, there is no guarantee that uim-mozc upstream and fcitx-mozc upstream use the same version of Mozc.

[hiroyuki-komatsu]

Thank you for the summary!

For option E, IIUC, what uim-mozc maintainers and distributions maintainers need for Mozc are conceptually same as cases of other libraries.

[kou]

Thanks for the libmozc_client.a idea. libmozc_client.a has ProtoBuf dependency, right?

It may not work for uim-mozc because uim-mozc is dlopen()-ed. I'm not sure whether hiding symbols from libmozc_client.a and libprotobuf.a works with uim-mozc use case...
See also: #1362 (comment)

Can "F: Providing mozc_client_cli" be a candidate for this request?

[kou]

Ah, we can create a mozc_client_cli like third-party tool with libmozc_client.a. libmozc_client.a may be enough.

"can you support pkgconf for libmozc_client?")

BTW, I want it...