Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple TritonInferenceStage from pipeline mode #1402

Merged
merged 58 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
f3265cb
First pass at making needs_logits a constructor argument [no ci]
dagardner-nv Dec 1, 2023
0c0dd77
Add missing doc strings [no ci]
dagardner-nv Dec 1, 2023
ed4788c
Update log parsing inference class to make needs_logits an argument […
dagardner-nv Dec 1, 2023
ea0cabc
Update log parsing tests [no ci]
dagardner-nv Dec 1, 2023
7804987
Add more tests [no ci]
dagardner-nv Dec 1, 2023
474bc87
Lint fixes
dagardner-nv Dec 1, 2023
8b0e1ce
Allow for a worker-class argument
dagardner-nv Dec 11, 2023
77e2a67
Use the public InferenceWorker as the type hint
dagardner-nv Dec 11, 2023
8d51010
Specify worker class, since we now validate the worker class in the c…
dagardner-nv Dec 11, 2023
db5cc53
Fix & update tests
dagardner-nv Dec 11, 2023
52a6778
Lint fixes
dagardner-nv Dec 11, 2023
de00f7a
wip
dagardner-nv Dec 13, 2023
ee18ee4
Merge branch 'branch-24.03' of github.com:nv-morpheus/Morpheus into t…
dagardner-nv Dec 13, 2023
d852879
Remove the AE Triton worker which isn't supported, and remove the sta…
dagardner-nv Dec 14, 2023
de406e6
Rename _TritonInferenceWorker and make it public [no ci]
dagardner-nv Dec 14, 2023
fbbda24
wip [no ci]
dagardner-nv Dec 14, 2023
aba1f96
Add expand_dims as constructor argument, and implement the _build_res…
dagardner-nv Dec 14, 2023
10ff5cd
update tests [no ci]
dagardner-nv Dec 14, 2023
1171917
Merge branch 'branch-24.03' of github.com:nv-morpheus/Morpheus into t…
dagardner-nv Dec 14, 2023
fd726e1
Fix inout_mapping [no ci]
dagardner-nv Dec 14, 2023
2d3c213
Fix CLI tests [no ci]
dagardner-nv Dec 14, 2023
e847536
Remove expand_dims constructor argument, just always perform it [no ci]
dagardner-nv Dec 14, 2023
e6770df
Remove unused import [no ci]
dagardner-nv Dec 14, 2023
b35a21a
Remove redundant code by mazking LogParsingInferenceStage a subclass …
dagardner-nv Dec 14, 2023
1eb6241
wip [no ci]
dagardner-nv Dec 14, 2023
d83a7c5
Allow subclassing a stage already registered with the CLI [no ci]
dagardner-nv Dec 14, 2023
955e94f
Remove unused imports [no ci]
dagardner-nv Dec 14, 2023
f785c90
Add type hints [no ci]
dagardner-nv Dec 14, 2023
cd472c0
cleanups [no ci]
dagardner-nv Dec 14, 2023
746ad6c
Add support for dictionary types by configuring the click type as a t…
dagardner-nv Dec 14, 2023
54c09e9
Remove temportary print statement
dagardner-nv Dec 14, 2023
4868d09
Document --inout_mapping flag
dagardner-nv Dec 14, 2023
bf2508b
Update tests
dagardner-nv Dec 14, 2023
4dd39f3
Make INFERENCE_WORKER_DEFAULT_INOUT_MAPPING a class property
dagardner-nv Dec 20, 2023
e9634a3
Merge branch 'branch-24.03' of github.com:nv-morpheus/Morpheus into t…
dagardner-nv Dec 20, 2023
fd092ba
Replace ResponseMemoryLogParsing with ResponseMemory
dagardner-nv Dec 20, 2023
bfaef1c
Replace PostprocMemoryLogParsing with InferenceMemory [no ci]
dagardner-nv Dec 20, 2023
c40b0ff
Remove unused MultiResponseLogParsingMessage [no ci]
dagardner-nv Dec 20, 2023
5cb45b8
Remove un-needed property methods
dagardner-nv Dec 20, 2023
eff5960
Give a more descriptive message to the Attribute error
dagardner-nv Dec 20, 2023
9a6f8bf
WIP: the dataclass decorator creates a constructor that doesn't call …
dagardner-nv Dec 20, 2023
a52e534
Remove unused message class [no ci]
dagardner-nv Dec 20, 2023
b85776c
Update inference stage to emit the propper output messages [no ci]
dagardner-nv Dec 20, 2023
3821243
Update post-processing tests
dagardner-nv Dec 20, 2023
6ea799b
Replace usage of deprecated ResponseMemory class
dagardner-nv Dec 20, 2023
bee89d3
Fix type-hint
dagardner-nv Dec 20, 2023
ddd6572
Update test
dagardner-nv Dec 20, 2023
f100479
Remove _get_worker_class method
dagardner-nv Dec 20, 2023
342a004
Remove _get_worker_class method
dagardner-nv Dec 20, 2023
6916db6
Set include_nulls=True and na_rep=null to match Python json serializer
dagardner-nv Dec 21, 2023
d6e6341
Support running the pipeline in C++ mode
dagardner-nv Dec 21, 2023
64b77c3
Fix test to enable c++ mode
dagardner-nv Dec 21, 2023
1f5ae81
WIP: Need to fix tests to work in C++ mode
dagardner-nv Dec 21, 2023
923f4b5
Merge branch 'branch-24.03' of github.com:nv-morpheus/Morpheus into t…
dagardner-nv Dec 26, 2023
326fdb8
Avoid redundant calls to get_tensor
dagardner-nv Dec 26, 2023
bbc0ee3
Fix setting the range of the output tensor
dagardner-nv Dec 26, 2023
01f0fe8
Fix tests to opperate in C++ mode
dagardner-nv Dec 26, 2023
2ad239f
Lint fixes
dagardner-nv Dec 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions examples/log_parsing/inference.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ class TritonInferenceLogParsing(_TritonInferenceWorker):
use_shared_memory: bool, default = True
Whether or not to use CUDA Shared IPC Memory for transferring data to Triton. Using CUDA IPC reduces network
transfer time but requires that Morpheus and Triton are located on the same machine

needs_logits : bool, default = True
Determines whether a logits calculation is needed for the value returned by the Triton inference response.
"""

def __init__(self,
Expand All @@ -69,7 +70,8 @@ def __init__(self,
server_url: str,
force_convert_inputs: bool,
use_shared_memory: bool,
inout_mapping: typing.Dict[str, str] = None):
inout_mapping: typing.Dict[str, str] = None,
needs_logits: bool = True):
# Some models use different names for the same thing. Set that here but allow user customization
default_mapping = {
"attention_mask": "input_mask",
Expand All @@ -83,11 +85,8 @@ def __init__(self,
server_url=server_url,
force_convert_inputs=force_convert_inputs,
use_shared_memory=use_shared_memory,
inout_mapping=default_mapping)

@classmethod
def needs_logits(cls):
return True
inout_mapping=default_mapping,
needs_logits=needs_logits)

@classmethod
def default_inout_mapping(cls) -> typing.Dict[str, str]:
Expand Down Expand Up @@ -149,15 +148,17 @@ class LogParsingInferenceStage(InferenceStage):
use_shared_memory: bool, default = False, is_flag = True
Whether or not to use CUDA Shared IPC Memory for transferring data to Triton. Using CUDA IPC reduces network
transfer time but requires that Morpheus and Triton are located on the same machine

needs_logits : bool, default = True, is_flag = True
Determines whether a logits calculation is needed for the value returned by the Triton inference response.
"""

def __init__(self,
c: Config,
model_name: str,
server_url: str,
force_convert_inputs: bool = False,
use_shared_memory: bool = False):
use_shared_memory: bool = False,
needs_logits: bool = True):
super().__init__(c)

self._config = c
Expand All @@ -167,6 +168,7 @@ def __init__(self,
"server_url": server_url,
"force_convert_inputs": force_convert_inputs,
"use_shared_memory": use_shared_memory,
"needs_logits": needs_logits
}

self._requires_seg_ids = False
Expand Down
Loading
Loading