[BUG]: abp fil benchmark broken

[dagardner-nv]

Version

22.11

Which installation method(s) does this occur on?

Source

Describe the bug.

FAILED test_bench_e2e_pipelines.py::test_abp_fil_e2e - AssertionError: Number of features in preprocessing 18, does not match configuration 29

Minimum reproducible example

cd tests/benchmarks
pytest -s --run_benchmark --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_pipelines.py

Relevant log output

================================================================================= FAILURES =================================================================================
_____________________________________________________________________________ test_abp_fil_e2e _____________________________________________________________________________

benchmark = <pytest_benchmark.fixture.BenchmarkFixture object at 0x7fb000021e40>, tmp_path = PosixPath('/tmp/pytest-of-dagardner/pytest-12/test_abp_fil_e2e0')

    @pytest.mark.benchmark
    def test_abp_fil_e2e(benchmark, tmp_path):
    
        config = Config()
        config.mode = PipelineModes.FIL
        config.num_threads = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["num_threads"]
        config.pipeline_batch_size = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["pipeline_batch_size"]
        config.model_max_batch_size = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["model_max_batch_size"]
        config.feature_length = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["feature_length"]
        config.edge_buffer_size = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["edge_buffer_size"]
        config.class_labels = ["mining"]
        config.fil = ConfigFIL()
        fil_cols_filepath = os.path.join(TEST_DIRS.data_dir, 'columns_fil.txt')
        config.fil.feature_columns = load_labels_file(fil_cols_filepath)
        CppConfig.set_should_use_cpp(True)
    
        input_filepath = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["file_path"]
        repeat = E2E_TEST_CONFIGS["test_abp_fil_e2e"]["repeat"]
        output_filepath = os.path.join(tmp_path, "abp_fil_e2e_output.csv")
        model_name = "abp-nvsmi-xgb"
    
>       benchmark(fil_pipeline, config, input_filepath, repeat, output_filepath, model_name)

test_bench_e2e_pipelines.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../conda/envs/m2/lib/python3.10/site-packages/pytest_benchmark/fixture.py:125: in __call__
    return self._raw(function_to_benchmark, *args, **kwargs)
../../../conda/envs/m2/lib/python3.10/site-packages/pytest_benchmark/fixture.py:147: in _raw
    duration, iterations, loops_range = self._calibrate_timer(runner)
../../../conda/envs/m2/lib/python3.10/site-packages/pytest_benchmark/fixture.py:275: in _calibrate_timer
    duration = runner(loops_range)
../../../conda/envs/m2/lib/python3.10/site-packages/pytest_benchmark/fixture.py:90: in runner
    function_to_benchmark(*args, **kwargs)
test_bench_e2e_pipelines.py:85: in fil_pipeline
    pipeline.add_stage(PreprocessFILStage(config))
../../morpheus/pipeline/stream_wrapper.py:45: in inner
    func(self, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <preprocess-fil-74; PreprocessFILStage()>
c = Config(debug=False, log_level=30, log_config_file=None, plugins=None, mode=<PipelineModes.FIL: 'FIL'>, feature_length=....gpu.max_clocks.graphics_clock', 'nvidia_smi_log.gpu.max_clocks.sm_clock', 'nvidia_smi_log.gpu.max_clocks.mem_clock']))

    def __init__(self, c: Config):
        super().__init__(c)
    
        self._fea_length = c.feature_length
        self.features = c.fil.feature_columns
    
>       assert self._fea_length == len(self.features), \
            f"Number of features in preprocessing {len(self.features)}, does not match configuration {self._fea_length}"
E       AssertionError: Number of features in preprocessing 18, does not match configuration 29

../../morpheus/stages/preprocess/preprocess_fil_stage.py:57: AssertionError

Full env printout

No response

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[nyck33]

I had some similar output when trying to run this example.

I was just trying to run the example by running the morpheus container, cloning the repo into it, downloading lfs files for models and examples and then

morpheus) root@d83ffc14cc9f:/workspace/Morpheus# morpheus --log_level=DEBUG    `# Run a pipeline with 8 threads and a model batch size of 32 (Must be equal or less than Triton config)`    run --num_threads=8 --pipeline_batch_size=1024 --model_max_batch_size=1024    `# Specify a NLP pipeline with 256 sequence length (Must match Triton config)`    pipeline-fil --columns_file=${MORPHEUS_ROOT}/morpheus/data/columns_fil.txt    `# 1st Stage: Read from file`    from-file --filename=examples/data/nvsmi.jsonlines    `# 2nd Stage: Deserialize from JSON strings to objects`    deserialize    `# 3rd Stage: Preprocessing converts the input data into BERT tokens`    preprocess    `# 4th Stage: Send messages to Triton for inference. Specify the model loaded in Setup`    inf-triton --model_name=abp-nvsmi-xgb --server_url=localhost:8000    `# 5th Stage: Monitor stage prints throughput information to the console`    monitor --description "Inference Rate" --smoothing=0.001 --unit inf    `# 6th Stage: Add results from inference to the messages`    add-class    `# 7th Stage: Convert from objects back into strings. Ignore verbose input data`    serialize --include 'mining'    `# 8th Stage: Write out the JSON lines to the detections.jsonlines file`    to-file --filename=detections.jsonlines --overwrite
Configuring Pipeline via CLI
Loaded columns. Current columns: [['nvidia_smi_log.gpu.fb_memory_usage.used', 'nvidia_smi_log.gpu.fb_memory_usage.free', 'nvidia_smi_log.gpu.utilization.gpu_util', 'nvidia_smi_log.gpu.utilization.memory_util', 'nvidia_smi_log.gpu.temperature.gpu_temp', 'nvidia_smi_log.gpu.temperature.gpu_temp_max_threshold', 'nvidia_smi_log.gpu.temperature.gpu_temp_slow_threshold', 'nvidia_smi_log.gpu.power_readings.power_draw', 'nvidia_smi_log.gpu.clocks.graphics_clock', 'nvidia_smi_log.gpu.clocks.sm_clock', 'nvidia_smi_log.gpu.clocks.mem_clock', 'nvidia_smi_log.gpu.applications_clocks.graphics_clock', 'nvidia_smi_log.gpu.applications_clocks.mem_clock', 'nvidia_smi_log.gpu.default_applications_clocks.graphics_clock', 'nvidia_smi_log.gpu.default_applications_clocks.mem_clock', 'nvidia_smi_log.gpu.max_clocks.graphics_clock', 'nvidia_smi_log.gpu.max_clocks.sm_clock', 'nvidia_smi_log.gpu.max_clocks.mem_clock']]
Traceback (most recent call last):
  File "/opt/conda/envs/morpheus/bin/morpheus", line 11, in <module>
    sys.exit(run_cli())
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/cli/run.py", line 20, in run_cli
    cli(obj={}, auto_envvar_prefix='MORPHEUS', show_default=True, prog_name="morpheus")
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1719, in invoke
    rv.append(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/cli/utils.py", line 114, in new_func
    return f(ctx, *args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/cli/register_stage.py", line 330, in command_callback
    stage = stage_class(**kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/pipeline/stream_wrapper.py", line 45, in inner
    func(self, *args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/stages/preprocess/preprocess_fil_stage.py", line 57, in __init__
    assert self._fea_length == len(self.features), \
AssertionError: Number of features in preprocessing 18, does not match configuration 29