enhance: add debug logs, Chinese stop words, and fnmatch optimization #45927

murataslan1 · 2025-11-28T00:05:30Z

Summary

This PR addresses three enhancement issues:

1. Add debug logs for collection load path (#45864)

Add debug logs at key checkpoints in the collection load process:

LoadCollectionJob.Execute() entry with metadata
Replica spawning start/completion
Collection description from broker
Target update operations
Collection observer task registration

2. Add default Chinese stop words (#45576)

Add _chinese_ keyword support for stop words filter with a comprehensive Chinese stop words list including:

Pronouns (我, 你, 他, etc.)
Auxiliary words (的, 地, 得, 了, etc.)
Prepositions and conjunctions (和, 与, 但是, etc.)
Adverbs (不, 很, 非常, etc.)
Common verbs and measure words
Time and location words

3. Use fnmatch for simple LIKE patterns (#44415)

Optimize pattern matching performance by using fnmatch for simple patterns that only contain % and _ wildcards without escape sequences. For complex patterns, fall back to boost::regex.

fnmatch is typically faster than regex for simple wildcard matching.

Closes

Type of change

Enhancement

sre-ci-robot · 2025-11-28T00:05:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: murataslan1
To complete the pull request process, please assign jiaoew1991 after the PR has been reviewed.
You can assign the PR to them by writing /assign @jiaoew1991 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

client/OWNERS
internal/core/OWNERS
internal/querycoordv2/job/OWNERS
internal/querycoordv2/observers/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2025-11-28T00:06:05Z

@murataslan1 Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.

sre-ci-robot · 2025-11-28T00:06:15Z

[ci-v2-notice]
Notice: We are gradually rolling out the new ci-v2 system.

Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-ut-integration // for ci-v2/ut-integration
/ci-rerun-ut-go // for ci-v2/ut-go
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
/ci-rerun-e2e-arm // for ci-v2/e2e-arm [master branch only]
/ci-rerun-e2e-default // for ci-v2/e2e-default [master branch only]

If you have any questions or requests, please contact @zhikunyao.

mergify · 2025-11-28T00:49:31Z

@murataslan1 cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-28T00:54:52Z

@murataslan1 go-sdk check failed, comment rerun go-sdk can trigger the job again.

…client Add support for HNSW_SQ, HNSW_PQ, and HNSW_PRQ index types in the Go client to match the Python SDK functionality. - Add HNSWSQ, HNSWPQ, and HNSWPRQ index type constants - Implement NewHNSWSQIndex with sq_type and optional refine support - Implement NewHNSWPQIndex with pqM, nbits and optional refine support - Implement NewHNSWPRQIndex with pqM, nbits and optional refine support - Add NewHNSWQuantAnnParam for search with refine_k parameter Fixes milvus-io#44635 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Signed-off-by: Murat Aslan <[email protected]>

Add debug logs at key checkpoints in the collection load process: - LoadCollectionJob.Execute() entry with metadata - Replica spawning start/completion - Collection description from broker - Target update operations - Collection observer task registration Closes milvus-io#45864 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Signed-off-by: Murat Aslan <[email protected]>

Add '_chinese_' keyword support for stop words filter with a comprehensive Chinese stop words list including: - Pronouns (我, 你, 他, etc.) - Auxiliary words (的, 地, 得, 了, etc.) - Prepositions and conjunctions (和, 与, 但是, etc.) - Adverbs (不, 很, 非常, etc.) - Common verbs and measure words - Time and location words Based on common Chinese NLP stop words lists (Baidu, HIT). Closes milvus-io#45576 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Signed-off-by: Murat Aslan <[email protected]>

Optimize pattern matching performance by using fnmatch for simple patterns that only contain % and _ wildcards without escape sequences. For complex patterns (with escape sequences or fnmatch special chars), fall back to boost::regex. Changes: - Add is_simple_pattern() to detect simple wildcard patterns - Add translate_pattern_match_to_fnmatch() to convert SQL wildcards to fnmatch - Update RegexMatcher to use fnmatch when applicable - fnmatch is typically faster than regex for simple wildcard matching Closes milvus-io#44415 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Signed-off-by: Murat Aslan <[email protected]>

Signed-off-by: Murat Aslan <[email protected]>

mergify · 2025-11-29T14:52:08Z

@murataslan1 cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-29T14:54:49Z

@murataslan1 go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2025-11-29T15:26:47Z

Codecov Report

❌ Patch coverage is 97.72727% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.46%. Comparing base (b69cd23) to head (f409e80).
⚠️ Report is 9 commits behind head on master.

⚠️ Current head f409e80 differs from pull request most recent head dec7e3f

Please upload reports for the commit dec7e3f to get more accurate results.

Files with missing lines	Patch %	Lines
internal/querycoordv2/job/job_load.go	85.71%	3 Missing ⚠️

❌ Your project check has failed because the head coverage (73.46%) is below the target coverage (77.00%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (b69cd23) and HEAD (f409e80). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (b69cd23) HEAD (f409e80)

3 2

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #45927       +/-   ##
===========================================
- Coverage   82.73%   73.46%    -9.28%     
===========================================
  Files         524     1360      +836     
  Lines       82326   212386   +130060     
===========================================
+ Hits        68111   156021    +87910     
- Misses      14215    48940    +34725     
- Partials        0     7425     +7425

Components	Coverage Δ
Client	`78.50% <100.00%> (∅)`
Core	`∅ <ø> (∅)`
Go	`74.11% <74.50%> (∅)`

Files with missing lines	Coverage Δ
client/index/hnsw.go	`100.00% <100.00%> (ø)`
...rnal/querycoordv2/observers/collection_observer.go	`83.77% <100.00%> (ø)`
internal/querycoordv2/job/job_load.go	`82.26% <85.71%> (ø)`

... and 1881 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Fix RegexMatcher to properly handle LIKE patterns vs regex patterns - Add FromLikePattern factory method that uses fnmatch for simple patterns - Update all callers to use FromLikePattern for LIKE pattern matching - Add PatternMatchQuery method to BitmapIndex for fnmatch optimization - Add comprehensive tests for the new fnmatch functionality Signed-off-by: Murat Aslan <[email protected]>

mergify · 2025-11-29T19:18:58Z

@murataslan1 cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-29T19:18:59Z

@murataslan1 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-11-29T20:48:16Z

@murataslan1 Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Signed-off-by: Murat Aslan <[email protected]>

mergify · 2025-11-29T21:06:44Z

@murataslan1 cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-29T21:07:51Z

@murataslan1 go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: Murat Aslan <[email protected]>

murataslan1 · 2025-11-30T08:38:06Z

/run-cpu-e2e

murataslan1 · 2025-11-30T08:38:07Z

rerun go-sdk

murataslan1 · 2025-11-30T08:38:21Z

/ci-rerun-code-check

murataslan1 · 2025-11-30T08:38:22Z

/ci-rerun-build

mergify · 2025-11-30T08:39:07Z

@murataslan1 cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-30T08:39:08Z

@murataslan1 go-sdk check failed, comment rerun go-sdk can trigger the job again.

murataslan1 · 2025-11-30T08:53:13Z

/ci-rerun-code-check

murataslan1 · 2025-11-30T08:53:15Z

/ci-rerun-build

murataslan1 · 2025-11-30T08:53:16Z

/run-cpu-e2e

murataslan1 · 2025-11-30T08:53:17Z

rerun go-sdk

mergify · 2025-11-30T09:01:50Z

@murataslan1 go-sdk check failed, comment rerun go-sdk can trigger the job again.

sre-ci-robot requested review from bigsheeper and congqixia November 28, 2025 00:05

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Nov 28, 2025

mergify bot added needs-dco DCO is missing in this pull request. kind/enhancement Issues or changes related to enhancement labels Nov 28, 2025

murataslan1 and others added 5 commits November 29, 2025 17:07

test(client): add unit tests for HNSW_SQ, HNSW_PQ, and HNSW_PRQ indexes

f409e80

Signed-off-by: Murat Aslan <[email protected]>

murataslan1 force-pushed the feat/debug-logs-and-enhancements branch from c3d9656 to f409e80 Compare November 29, 2025 14:08

mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Nov 29, 2025

sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Nov 29, 2025

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Nov 29, 2025

mergify bot added the needs-dco DCO is missing in this pull request. label Nov 29, 2025

mergify bot removed the dco-passed DCO check passed. label Nov 29, 2025

style: fix clang-format issues

792b715

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Signed-off-by: Murat Aslan <[email protected]>

murataslan1 force-pushed the feat/debug-logs-and-enhancements branch from 5205322 to 792b715 Compare November 29, 2025 20:50

mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Nov 29, 2025

style: fix rustfmt formatting issues in Chinese stop words

dec7e3f

Signed-off-by: Murat Aslan <[email protected]>

enhance: add debug logs, Chinese stop words, and fnmatch optimization #45927

Are you sure you want to change the base?

enhance: add debug logs, Chinese stop words, and fnmatch optimization #45927

Uh oh!

Conversation

murataslan1 commented Nov 28, 2025

Summary

1. Add debug logs for collection load path (#45864)

2. Add default Chinese stop words (#45576)

3. Use fnmatch for simple LIKE patterns (#44415)

Closes

Type of change

Uh oh!

sre-ci-robot commented Nov 28, 2025

Uh oh!

mergify bot commented Nov 28, 2025

Uh oh!

sre-ci-robot commented Nov 28, 2025

Uh oh!

mergify bot commented Nov 28, 2025

Uh oh!

mergify bot commented Nov 28, 2025

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

codecov bot commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

mergify bot commented Nov 29, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

mergify bot commented Nov 30, 2025

Uh oh!

mergify bot commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

murataslan1 commented Nov 30, 2025

Uh oh!

mergify bot commented Nov 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Nov 29, 2025 •

edited

Loading