[jjj-n]

Description

Step 1 (#3455) lets an IM channel use a fixed reply locale. A multilingual channel still cannot make fallback answers, session titles, and other prompts follow the writer's language. This Step 2 adds language_mode: fixed | follow_user (default fixed) to channel configuration and the admin UI.

In follow_user mode, the shared IM message path resolves the reply language before title generation and QA:

confident language from current text message
  -> last confidently detected language in this IM session
  -> DefaultLanguage()

The last language is stored on im_channel_sessions, so /clear resets it with the session mapping. Detector output is separate from the five UI locales; Vietnamese is covered. Webhook, WebSocket, and long-poll adapters all reach the shared HandleMessage path. The IM platform's Accept-Language is not used as the writer's language. API and IM integration docs are updated.

Real Feishu IM traffic validation is complete for the follow-user path; this PR is ready for review. The resolver and detector have unit tests, but there is not yet a transport-level assertion that each adapter delivers the resolved LanguageContextKey to title generation and QA. The language state follows the existing ChannelSession identity: user mode keys by platform/user/chat/tenant/agent, and thread mode by platform/chat/thread/tenant/agent. Two channel configurations that resolve to the same existing session key also share its language state; changing that session identity is outside this PR.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📚 Documentation update
• 🎨 Refactor
• ⚡ Performance improvement
• 🧪 Test
• 🔧 Configuration / Build / CI

Related Issue

Refs #3315. Follows merged Step 1 in #3455.

Testing

• Linux/WSL: go test ./internal/im ./internal/database -count=1 — passed.
• Linux/WSL: go test ./internal/handler -run TestCreateIMChannel -count=1 — passed.
• Linux/WSL: go vet ./internal/im ./internal/database ./internal/handler — passed.
• Frontend: npm run type-check and npm run check-i18n — passed.
• git diff --check origin/main...HEAD — passed.
• 2026-09-29: after merging current origin/main, Docker Go tests go test ./internal/im ./internal/database -count=1 and go test ./internal/handler -run TestCreateIMChannel -count=1, plus go vet ./internal/im ./internal/database ./internal/handler, passed. Frontend npm run type-check and npm run check-i18n passed. Full-repository Go tests were not run locally.

Checklist

• git diff --check origin/main...HEAD passes
• Changed source files are formatted
• Targeted tests for the changed packages/components pass
• Diff-scoped lint passes where applicable (for Go: golangci-lint run --new-from-rev=origin/main ./...)
• Full-repository checks were run, or any unrelated/environment-dependent failures are documented above
• Self-reviewed the code
• Added/updated tests covering the change
• Updated related documentation (README, website-docs/, Swagger annotations, etc.)
• Breaking changes are clearly called out in the description above

Live Feishu IM validation (2026-09-29)

• Feishu WebSocket received real im.message.receive_v1 events; a Chinese private-chat message produced a visible Chinese streaming-card reply. The active follow_user channel persisted Mandarin.
• A Vietnamese sentence produced a Vietnamese streaming-card reply and persisted Vietnamese. A subsequent ? in the same session produced a Vietnamese clarification, validating the short-message session fallback.
• /clear removed the old channel-session mapping; a new ? used the deployment default en-US and replied in English.
• The local test environment needs open.feishu.cn in its exact SSRF whitelist because its DNS resolves to a restricted proxy IP. After granting the test app cardkit:card:write, the Chinese turn, Vietnamese turn, short ?, /clear, and post-clear ? all completed the streaming-card path (Streaming started and Streaming ended in app logs). These are test-environment settings, not PR source changes.
• After syncing origin/main, renumbered this PR's migrations to SQLite 000034 and versioned 000115 to avoid upstream migration-number collisions. GitHub reports the branch mergeable and all seven checks passed. Live IM traffic ran on the pre-merge application image at 0034ec8; targeted tests, vet, frontend type-check and i18n ran on merged head e86ea50.

Screenshots / Recordings

The local Docker app and Feishu test bot are running. No admin UI screenshot is attached yet. The selector shows the fixed locale only in fixed mode.

Summary

IM channel setting -> shared HandleMessage path -> language context
                                           |-> current confident detection
                                           |-> session's last detection
                                           `-> deployment default

Evidence

• Before: IM channels could only use the fixed locale from feat(im): add per-channel reply locale #3455; an uncertain follow-up such as ok would have no per-conversation language to reuse.
  After: a Vietnamese turn stores Vietnamese; ok in the same session uses it; a new session after /clear uses the deployment default until a language is detected.

Merge Danger

Door: Two-way. The mode defaults to fixed; reverting the migration removes the new mode and saved language state.

Blast Radius: IM channels. Existing channels retain fixed-language behavior; follow_user adds one session-row read or write per handled message.

[jjj-n]

I synchronized this branch with the current \main\ and pushed head \

[vinhchuquang]

@jjj-n I don't have a non-production IM bot connected yet, so I haven't run cases 1–5 end to end. Your Feishu run covers that path. What I could do is run langdetect.Detect at e86ea50 against the kind of text Vietnamese users actually send, since that is where I expected the risk. I used 17 messages, each in NFC and NFD, with whatlanggo v1.0.1 on Go 1.26.

Where it holds up

• Long accented Vietnamese returns Vietnamese. So does long unaccented Vietnamese (cho minh hoi chinh sach doi tra san pham...), which I did not expect.
• Short unaccented text and teencode (shop oi con hang khong a, ko bik lam sao de dang nhap dc nua) come back unknown, not as a wrong language. whatlanggo's raw guesses there are Cebuano and Indonesian, so the reliability gate is doing its job.
• ok, ?, links and ok thanks bạn come back unknown, as intended.

Where it misses: ordinary short Vietnamese questions

Message	Letters	whatlanggo raw (NFC)	Detect
Giá bao nhiêu vậy?	14	Vietnamese, reliable	"" (letters < 20)
Shop ơi còn hàng không?	18	Vietnamese, reliable	"" (letters < 20)
Cảm ơn bạn nhiều nha	16	Vietnamese, unreliable	""
Em muốn đổi mật khẩu thì làm sao ạ	26	Latvian, unreliable	""
Mình bị lỗi timeout khi gọi endpoint upload file, check giúp mình với	56	Haitian Creole, unreliable	""

These are not the ok turns. They are typical first messages. When one of them opens a session, or comes right after /clear, there is no previous language to fall back to, so the reply goes out in DefaultLanguage(). The session fallback covers short turns in the middle of a conversation, but it cannot cover the first one.

A side finding on Unicode normalisation: whatlanggo scores NFD input differently from NFC. The last two rows come back as reliable Vietnamese when the input is NFD. Separately, the vietnameseMarks check only matches precomposed runes. Under NFD it counts close to zero, so the mixed-language guard turns off. Some Vietnamese input methods still send NFD. Normalising the text first with norm.NFC makes the check behave the same for both forms, and golang.org/x/text is already a dependency.

Suggestion. On short text, letters that only Vietnamese uses are a stronger signal than the n-gram score. I tried this variant in a throwaway test, not in the PR:

// ơ ư and the tone+circumflex/breve/horn composites occur only in Vietnamese.
// ă (Romanian), đ (Croatian) and plain â ê ô (French, Portuguese) are left out.
const vietnameseOnly = "ơưƠƯấầẩẫậắằẳẵặếềểễệốồổỗộớờởỡợứừửữựẤẦẨẪẬẮẰẲẴẶẾỀỂỄỆỐỒỔỖỘỚỜỞỠỢỨỪỬỮỰ"

text = norm.NFC.String(text)
if strings.ContainsAny(text, vietnameseOnly) {
	return whatlanggo.Langs[whatlanggo.Vie]
}

I placed it before the letters threshold. With it, all five rows above return Vietnamese in both NFC and NFD, and so do Đi đâu đấy and Bạn ăn cơm chưa.

The guard sentences kept their current result: French, Portuguese, Romanian, Croatian and English. ok, ?, ok thanks bạn and unaccented text stay unknown.

One behaviour does change. How do I reset my password? Mình quên mật khẩu rồi now returns Vietnamese instead of unknown. That is arguably right, because the user did write in Vietnamese, but it is a change to the mixed case.

If you would rather keep the detector generic, a Vietnamese-only deployment can work around the first-turn problem on the operator side. LanguageLocaleName passes unknown values through, so WEKNORA_LANGUAGE=Vietnamese makes the first-turn fallback land in the right language. That does not help a channel that serves both Vietnamese and other users.

I'm happy to run it on live IM traffic once I have a bot connected.

[jjj-n]

@vinhchuquang Thank you for the concrete Vietnamese samples and the NFC/NFD finding. I reproduced the five first-turn misses and pushed fe1976c to this PR.

The detector now normalizes to NFC and uses a conservative set of Vietnamese-distinctive letters when a message is below the 20-letter threshold or whatlanggo marks its result unreliable. It still keeps the existing unknown result for ok, links, ok thanks bạn, and the existing reliable English/Vietnamese mixed guard. I added NFC and NFD regression cases for all five messages, plus non-Vietnamese guard cases. go test ./internal/im/langdetect -count=1, go test ./internal/im ./internal/database -count=1, and go vet ./internal/im/... passed locally; GitHub CI is rerunning.

One deliberate trade-off matches your observation: How do I reset my password? Mình quên mật khẩu rồi now selects Vietnamese rather than unknown. I pinned that behavior in a test so it is visible to reviewers. The live Feishu run was on the earlier image; this detector change has not yet been retested through live IM.

[vinhchuquang]

@jjj-n Re-ran the probe at fe1976c with the same 17 messages, each in NFC and NFD, and added 17 more. All five first-turn misses now return Vietnamese in both forms. go test ./internal/im/langdetect ./internal/im passes.

New cases that hold

• All-caps Vietnamese (GIÁ BAO NHIÊU VẬY?, CHO MÌNH HỎI ĐỊA CHỈ CỬA HÀNG) returns Vietnamese, because the uppercase composites are in vietnameseOnly.
• Mở cửa lúc mấy giờ?, Hủy đơn giúp mình and Sao lâu vậy return Vietnamese.
• Guards are unchanged. Portuguese and Croatian return their own language. French and Romanian return unknown. A Yoruba sentence with ẹ, ọ and ṣ returns unknown. ok, ?, links, ok thanks bạn and short unaccented text return unknown.

One small gap: hook-above letters

Bạn khỏe không? and Có ship tỉnh không are under 20 letters and still return unknown. Their only distinctive letters are ỏ and ỉ, the hook-above tone. Hook-above (ả ẻ ỉ ỏ ủ ỷ) is used only in Vietnamese, so it can join the set:

const vietnameseOnly = "ơưƠƯấầẩẫậắằẳẵặếềểễệốồổỗộớờởỡợứừửữựẤẦẨẪẬẮẰẲẴẶẾỀỂỄỆỐỒỔỖỘỚỜỞỠỢỨỪỬỮỰảẻỉỏủỷẢẺỈỎỦỶ"

I tried an equivalent hook-above check in a throwaway test. Both messages then return Vietnamese in NFC and NFD. None of the guard sentences contain these letters, so their results do not change.

I'd leave dot-below (ạ ẹ ị ọ ụ ỵ) out. Yoruba and Igbo use ẹ, ị, ọ and ụ, and ok thanks bạn would flip to Vietnamese, which your tests pin as unknown.

Expected misses. Bạn tên gì?, Xin chào and Còn size M không have no Vietnamese-only letter, so they stay unknown. The session fallback covers them in the middle of a conversation. On a first turn, WEKNORA_LANGUAGE=Vietnamese covers a Vietnamese-only deployment.

Still no live IM bot on my side, so this covers the detector only.

[lyingbug]

Thanks @jjj-n. Following the user's language is the natural next step after #3455, and I'm happy with the direction. One thing has to change before this can land, and I have one question.

1. LanguageContextKey must carry a locale code, not a language name. When detection succeeds, withIMReplyLanguage stores the detected name in both last_detected_language and the context (internal/im/reply_language.go:22,38). langdetect.Detect returns names from whatlanggo.Langs[...] (internal/im/langdetect/detect.go:37,44,53); your Chinese test expects "Mandarin" (internal/im/langdetect/detect_test.go:16).

   The key's documented convention is a code like "zh-CN" or "en-US" (internal/types/const.go:65). Existing locale consumers rely on that representation: the Chinese branches in internal/application/service/knowledgebase.go:1342 and internal/types/artifact_versions.go:12 would not match "Mandarin". LanguageLocaleName does pass unknown values through (internal/types/context_helpers.go:425-427), so a name can still work in prompts, but that does not make it interchangeable with a locale code for other consumers.

   Please map detected languages to locale codes, covering at least zh-CN, en-US, ja-JP, ko-KR and ru-RU, and use standard BCP-47 language tags such as vi for other languages. Store the code in last_detected_language too. Extend LanguageLocaleName for any newly emitted tags it does not already support, so prompts still receive readable names. Please update the detector/session tests and add regression checks for persisted codes, remembered-language fallback, and the new prompt-name mappings.

2. Why whatlanggo? This PR adds github.com/abadojack/whatlanggo v1.0.1 (go.mod:8). Could you explain why you picked it, which alternatives you considered, and what you found about its current maintenance status? I'd also like to understand how we'd handle detector bugs or compatibility issues if upstream support is unavailable.

[jjj-n]

1. Locale codes instead of language names — done.

langdetect.Detect now returns a locale code. zh-CN, en-US, ja-JP, ko-KR and ru-RU come from an explicit override table so the detector matches the codes already used by internal/types/locale.go; every other detected language falls back to its BCP-47 primary subtag via golang.org/x/text/language (already a dependency), e.g. vi for Vietnamese. withIMReplyLanguage stores the code in both LanguageContextKey and last_detected_language, so the Chinese branches in knowledgebase.go and artifact_versions.go now see zh-CN rather than "Mandarin".

LanguageLocaleName gained a vi/vi-VN → Vietnamese mapping so prompts still receive a readable name; unknown tags keep passing through unchanged. Tests are updated: the detector suite asserts codes (vi, en-US, zh-CN), and a new TestFollowUserLanguagePersistsLocaleCode covers the persisted column value and the remembered-language fallback on the next turn. All changes are in 37345d5.

Also folded in @vinhchuquang's hook-above suggestion from the round-2 probe: ả ẻ ỉ ỏ ủ ỷ (and uppercase) joined vietnameseOnly, so Bạn khỏe không? and Có ship tỉnh không now detect as vi in both NFC and NFD. Dot-below letters stay out for the Yoruba/Igbo reasons given there.

2. Why whatlanggo.

I picked it for three reasons: (a) it is a small, dependency-free statistical detector with a closed language table, so no message text leaves the process and no network call is added to the IM hot path; (b) its trigram scoring returns a reliability flag, which the letters < 20 and vietnameseOnly guards in internal/im/langdetect build on — this PR never trusts a raw whatlanggo verdict; (c) MIT, pure Go, no cgo, so it drops into the existing build without new toolchain requirements.

Alternatives considered: pemistahl/lingua-go (more accurate on short text and still maintained, but a much larger model bundle and heavier API surface than this path needs) and hosted detection APIs such as detectlanguage (ruled out because IM message content must not leave the deployment).

Maintenance status: upstream has been quiet since March 2023 with open issues unanswered, so I'm treating it as frozen rather than maintained. That is why the detector is isolated in internal/im/langdetect behind a ~40-line Detect function: it is the only import site, it has its own unit tests pinning the guards, and swapping the engine (e.g. for lingua-go) would be a package-internal change with no impact on the IM service, the session schema, or this PR's resolution flow. If a detector bug shows up, the fix lands inside this package; the fallback chain (current detection → session's last detection → deployment default) also means a degraded detector fails closed to unknown, not to a wrong language.

[lyingbug]

Review of 37345d539d02.

The locale-code correction in 37345d5 is present, and the focused IM/locale checks and detector suite pass. Two integration items remain:

1. [P2] Add the Traditional Chinese keys when rebasing onto current main. IMChannelPanel.vue references agentEditor.im.replyLanguageFixed, replyLanguageFollowUser, and replyLanguageFollowHint, but current main's zh-TW.ts has none of them and this branch does not add them. Please supply all three translations and run the current locale-key audit after resolving the conflict in frontend/src/api/agent/index.ts.
2. Coordinate migration numbers with feat(toolbox): add shared tags for Skills and MCP #4053 and fix(sqlite): add MCP metadata migration #4060. All three PRs add SQLite version 000034; this PR and feat(toolbox): add shared tags for Skills and MCP #4053 also both use PostgreSQL version 000115. Loading the actual combined migration files reproduces a duplicate-migration error before migrations can run. The application logs the migration failure and continues startup, so it may operate with missing schema. Please assign unique versions to whichever migrations land later, and validate both a fresh database and the upgrade sequence.

The migration collision is a cross-PR integration problem; these versions need to remain unique in the final combined tree.

[jjj-n]

Both items are addressed in 78c7ae24.

1. zh-TW keys. The branch now merges current main, resolving the frontend/src/api/agent/index.ts conflict by keeping both sides (upstream's zh-TW locale plus language_mode). All three keys are added to zh-TW.ts:

• replyLanguageFixed: '固定語言'
• replyLanguageFollowUser: '跟隨使用者語言'
• replyLanguageFollowHint: '按訊息偵測回覆語言。短訊息或無法判斷時沿用本會話上次識別的語言，再回退到部署預設語言。'

The current locale-key audit passes: npm run check-i18n — 14/14; npm run type-check (vue-tsc) also passes.

2. Migration numbers. This PR's migrations are renumbered to SQLite 000035_im_follow_user_language and versioned 000116_im_follow_user_language, so they no longer collide with main's merged 000034_mcp_metadata, nor with the SQLite 000034 in #4053/#4060 or the versioned 000115 in #4053. Both directories load with no duplicate versions, and expectedSQLiteMigrationVersion is bumped to 35 so the fresh-schema and upgrade-sequence tests run through the new numbering.

Targeted checks on the merged head: go test ./internal/im/langdetect ./internal/im ./internal/handler -run TestCreateIMChannel -count=1 and go vet on the touched packages pass; GitHub CI is rerunning on 78c7ae24.

[jjj-n]

Merged current main in e8ac04c and resolved the conflicts. Renumbered the migrations to SQLite 000042 and PostgreSQL 000125. Added SQLite upgrade/rollback regression coverage and validated the PostgreSQL migration SQL. All seven checks on the updated head pass. Could you take another look when convenient?