Adding Punctuation in Devnagari Script (Hindi) reduces 'Confidence'.

[abhishekkr]

I tried giving it a test for Hindi (in Devnagari Script).

With a random sentence the confidence was 39.x%.
So I tested it with a sentence "It is What Language.", I guessed might be close to home for this.

In Devnagari Script, FullStop is written as '|'. When I tested the sentence without it, confidence was 100%. But on including it, the confidence dropped few points.

I'm a n00b when it comes to Rust, but can try fixing it if you don't have time and can point me in right direction.

Also, if you need any help training models and have a guide I can follow.

PS: Thanks for this. I was looking for an interesting project to try restart Rust journey.

[greyblake]

Hi @abhishekkr ,
thank you for the report!
Right now whatlang is under heavy refactoring and improvements.
The old version is based on trigrams only.
The new one will also take alphabetics into considerations (see: https://github.com/greyblake/whatlang-rs/tree/alphabet/src/alphabets)

The reason for your results can be the following: | is not recognized as punctuation and it is used to build trigrams, what delutes a confidence in the final result.

Unfortunately I have zero knowledge about Hindi, so if you're available for assisting with Devnagari languages, that would be very helpful!

[abhishekkr]

Sure, I'd be glad to help with that (Hindi being my first language) and any other feature you might not be able to pick due to schedule.

Just 2 things; I've only done minimal introductory Rust in it's pre-stable releases and might not be always on-time due to pre-commitments. I wouldn't need hand holding with Rust & project in general, that I'll manage (with feedback)... but I can't do away with commitments.