How to speed up for large dataset

[linWujl]

Hello, my corpus is 700G, is there any way to speed up？

[AngledLuffa]

More threads? Better hardware?

…

On Wed, Apr 19, 2023, 6:48 PM Linlp ***@***.***> wrote: Hello, my corpus is 700G, is there any way to speed up？ — Reply to this email directly, view it on GitHub <#214>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWMCP43W4LZYUFGZGDLXCCIXZANCNFSM6AAAAAAXE2JWEU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[linWujl]

The coocur step has cost about 7500mins and it stills at the merge step.

Is it possible that use spark to construct the cooccurrence statistics and train it with tensorflow?

[AngledLuffa]

We did try converting it to torch at one point, but it wound up being significantly slower than the C version. We may try again sometime. You are welcome to try...

Do you have enough memory? Might be worth checking top