Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support chunk spliter settting on knowledge base creation #527

Open
sykp241095 opened this issue Dec 23, 2024 · 0 comments
Open

Support chunk spliter settting on knowledge base creation #527

sykp241095 opened this issue Dec 23, 2024 · 0 comments
Assignees
@Mini256
Labels
area/index-building
Milestone
Release v0.5.0

Comments

@sykp241095
Copy link
Member

sykp241095 commented Dec 23, 2024 •
edited by Mini256
Loading

Description

  • Support knowledge base level and data source level setting
  • Support select different parser for different data source
  • Using MarkdownNodeParser for text/markdown type documents by default

Splitters

  • SentenceSplitter
    • separator
    • chunk_size
    • chunk_overlap
    • tokenizer
  • MarkdownNodeParser
    • chunk_size
@sykp241095 sykp241095 added this to the Release v0.5.0 milestone Dec 23, 2024
@Mini256 Mini256 changed the title core(chunk): move CHUNK_SIZE settings from .env to creating kb scenario Support node parser settting on knowledge base creation Jan 3, 2025
@Mini256 Mini256 changed the title Support node parser settting on knowledge base creation Support chunk spliter settting on knowledge base creation Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Mini256 Mini256

Labels
area/index-building
Projects
None yet
Milestone
Release v0.5.0
Development

No branches or pull requests
2 participants
@sykp241095 @Mini256