Allow to use a different type of SudachiPy's dictionary for the Japanese model

[BLKSerene]

HI, there are 3 types of dictionaries for SudachiPy: sudachidict-small, sudachidict-core, sudachidict-full, and currently the Japanese support requires sudachidict-core to be installed. Is there any way that, e.g. the full version of SudachiPy's dictionary could be used instead.

[svlandeg]

Hi! As far as I know, we use sudachidict-core by default in our pretrained pipelines as it achieves the best performance/footprint trade-off. If you would install sudachidict-small instead of core, I expect that the existing ja_core_* Japanese models will still run, but with degraded performance. I'm not sure sure how it would look if you'd install sudachidict-full instead of core and then run predictions, have you tried that?

Either way - you could most likely train a new model from scratch while using full instead of core, and ensure the model requirements specify sudachidict-full.

If you do try that - let us know how you go?