[Bug]: `delimiter` does not work for any document when `general`

[lidongshengxdayu]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

RAGFlow workspace code commit ID

d9fe279

RAGFlow image version

d55f446(v0.20.3)

Other environment information

Actual behavior

This commit causes no delimiter to work when using general. Delimiter will only be used after chunk_len reaches chunk_token_num.

Expected behavior

Use delimiter to split and double split when chunk_token_num is exceeded

Steps to reproduce

1. Upload excel document
2. Use the `general` method to execute `parse`
3. Modify `delimiter` to the characters that exist in other known documents and rerun `parse`
4. Check the number of chunks, there will be no changes

Additional information

def naive_merge(sections, chunk_token_num=128, delimiter="\n。；！？", overlapped_percent=0):
    from deepdoc.parser.pdf_parser import RAGFlowPdfParser
    if not sections:
        return []
    if isinstance(sections[0], type("")):
        sections = [(s, "") for s in sections]
    cks = [""]
    tk_nums = [0]

    def add_chunk(t, pos):
        nonlocal cks, tk_nums, delimiter
        tnum = num_tokens_from_string(t)
        if not pos:
            pos = ""
        if tnum < 8:
            pos = ""
        # Ensure that the length of the merged chunk does not exceed chunk_token_num  
        if cks[-1] == "" or tk_nums[-1] > chunk_token_num * (100 - overlapped_percent)/100.:
            if cks:
                overlapped = RAGFlowPdfParser.remove_tag(cks[-1])
                t = overlapped[int(len(overlapped)*(100-overlapped_percent)/100.):] + t
            if t.find(pos) < 0:
                t += pos
            cks.append(t)
            tk_nums.append(tnum)
        else:
            if cks[-1].find(pos) < 0:
                t += pos
            cks[-1] += t
            tk_nums[-1] += tnum

    dels = get_delimiters(delimiter)
    for sec, pos in sections:
        # this if ignore any delimiter
        if num_tokens_from_string(sec) < chunk_token_num:
            add_chunk(sec, pos)
            continue
        splited_sec = re.split(r"(%s)" % dels, sec, flags=re.DOTALL)
        for sub_sec in splited_sec:
            if re.match(f"^{dels}$", sub_sec):
                continue
            add_chunk(sub_sec, pos)

[dosubot]

Hi @lidongshengxdayu! I'm Dosu and I’m helping the ragflow team.

The root cause is that in the 'general' method, delimiter-based splitting only happens if a section's token count exceeds chunk_token_num; otherwise, the section is added as a chunk directly and the delimiter is ignored. This means for many documents, especially those with short sections, changing the delimiter will have no effect unless you lower chunk_token_num to force splitting by delimiter. Also, for some formats (like Word), there's a known issue where the 'general' method ignores custom delimiters and always splits by line breaks, while TXT documents respect custom delimiters as expected. As a workaround for Word and similar documents, you can use the 'law chunk method' for segmentation instead. For best results with custom delimiters, wrap them in backticks (e.g., ##) and set chunk_token_num to a very low value (like 2) so the system prioritizes delimiter-based splitting source source.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lidongshengxdayu]

@dosu
Reducing chunk_token_num will cause more other problems, which should need to be fixed

1. Delete the default separator of web and function under the general method, and only use it manually if the user sets it manually.
2. Delete existing code that ignores delimiter
   The final code should be：

def naive_merge(sections, chunk_token_num=128, delimiter="", overlapped_percent=0):
    from deepdoc.parser.pdf_parser import RAGFlowPdfParser
    if not sections:
        return []
    if isinstance(sections[0], type("")):
        sections = [(s, "") for s in sections]
    cks = [""]
    tk_nums = [0]

    def add_chunk(t, pos):
        nonlocal cks, tk_nums, delimiter
        tnum = num_tokens_from_string(t)
        if not pos:
            pos = ""
        if tnum < 8:
            pos = ""
        # Ensure that the length of the merged chunk does not exceed chunk_token_num
        if cks[-1] == "" or tk_nums[-1] > chunk_token_num * (100 - overlapped_percent)/100.:
            if cks:
                overlapped = RAGFlowPdfParser.remove_tag(cks[-1])
                t = overlapped[int(len(overlapped)*(100-overlapped_percent)/100.):] + t
            if t.find(pos) < 0:
                t += pos
            cks.append(t)
            tk_nums.append(tnum)
        else:
            if cks[-1].find(pos) < 0:
                t += pos
            cks[-1] += t
            tk_nums[-1] += tnum

    dels = get_delimiters(delimiter)
    for sec, pos in sections:
        if dels:
            split_sec = re.split(r"(%s)" % dels, sec, flags=re.DOTALL)
            for sub_sec in split_sec:
                if re.match(f"^{dels}$", sub_sec):
                    if cks[-1] != "":
                        cks.append("")
                    continue
                add_chunk(sub_sec, pos)
        else:
            add_chunk(sec, pos)

    return cks

[dosubot]

Your proposal to remove default delimiters and only use user-specified ones in the general method is feasible and aligns with making delimiter handling more explicit and predictable. Currently, RAGFlow applies a default delimiter (\n。；！？) unless the user sets it to an empty string, which can lead to unexpected chunking—especially for documents with short sections, where the delimiter is ignored unless the token count exceeds the threshold source.

Switching to your approach would make chunking entirely dependent on user input, so segmentation only happens if the user sets a delimiter. This change would make parsing more consistent with user intent but less automatic, and could confuse users who expect line or paragraph breaks by default. No recent PRs address this change, so it would mainly affect backend modules (chunker, nlp) and not the frontend source.

If you move forward, consider documenting this behavior clearly to avoid confusion. Also, test with various document types (web, Word, TXT, PDF) to ensure chunking works as expected when no delimiter is set.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}