Chunk optimization

[pacmanincarnate]

I recommend a more advanced chunking system. You ideally want to break text up by sentence or paragraph where possible. chunking by words will split sentences and break the meaning of those sentences. Even breaking up a paragraph can have that effect. Best option would maybe be to look for new lines and if that creates a good chunk, chunk it, if it’s too large, break it at a period in the middle and if it’s too small, check if two paragraphs fits in your context window. It’ll take a little more processing, but it should be fairly arbitrary.

[jckpn]

Good shout, shouldn't be too difficult to implement either. Ideally this would be a customisable option but I'm worried about adding too many parameters... a GUI would make this much easier for the user but I won't be able to work on that for a couple weeks.

[pacmanincarnate]

For what it’s worth I had chatGPT write the changes, but haven’t had a chance to test them or push them as a pull request. Might try to do that this evening.

[jckpn]

I think the main difficulty would be detecting where a chunk should end. It can be pretty ambiguous to tell with bullet-point note, lecture slides, video captions etc. where there's no paragraph spacing and inconsistent punctuation. A GUI could help with this too by showing some 'previews' of the chunk start/ends before running the operation