Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can I predict with crf model using multiple processes? I tried, But I got errors -_-! #110

Open
huwen2117 opened this issue Aug 20, 2019 · 1 comment

Comments

@huwen2117
Copy link

huwen2117 commented Aug 20, 2019

I tried, and I got Error:
File "stringsource", line 2, in pycrfsuite._pycrfsuite.Tagger.reduce_cython
TypeError: self.c_tagger cannot be converted to a Python object for pickling

my plan like this:

model_list = [model_1,model_2, ... ,model_n]

def do_tagging(args):
    model ,sent = args
    tagger = crfs.Tagger()
    tagger.open(model)
    labels = tagger.tag(sent)

with Pool(processes=num_proc) as p:
    predict_result = p.map(do_tagging, zip(model_list, repeat(sent))))
@huwen2117 huwen2117 changed the title Can I predict with crf model using multiple thread? I tried, But I got errors -_-! Can I predict with crf model using multiple processes? I tried, But I got errors -_-! Aug 20, 2019
@DomHudson
Copy link

Open the model within the process rather than passing the model in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants