Triton Inference Server

[B-M-S-West]

Anyone tried to use the python backend and deploy this on the triton inference server?

[geraldstanje]

hi, yes - join tensorrt llm channel on discord for more infos: https://discord.gg/xUEggt5xqn

[manickavela29]

Hey @geraldstanje

has there been any update on this?
the link doesn't seems to be working

[smblsm]

Just bumping this—any more info? I run into this error converting from onnx regardless of whether dimensions are fixed or dynamic:

['\x1b[38;5;9m'][E] IBuilder::buildSerializedNetwork: Error Code 1: Myelin ([backpropagation.cpp:throw_errors:199] The following checks failed:

    Error during shape inference of
    /span_rep_layer/span_rep_layer/Reshape_2_output_0' : f32[1, __mye55371_proxy, 12, 512] = move(/span_rep_layer/span_rep_layer/Reshape_2_reshape_output), name=/span_rep_layer/span_rep_layer/Reshape_2_final_copy
    Error is:
    In op '/span_rep_layer/span_rep_layer/Reshape_2_final_copy(u_pw:move)', expected shape [proxy___mye241145xxx,proxy___mye241147xxx,proxy___mye241149xxx,proxy___mye2411)

Had to lowercase the equation in the final einsum to get there. Any help would be appreciated! Love the project.