[BUG]: Intermittent failures with LLM agents example

[dagardner-nv]

Version

24.03

Which installation method(s) does this occur on?

Source

Describe the bug.

Hitting an intermittent error:

 Exception msg: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

Minimum reproducible example

python examples/llm/main.py --log_level=debug agents simple

### Relevant log output

<details><summary>Click here to see error details</summary><pre>

> Entering new AgentExecutor chain...
 I should use a search engine to find information about Leo DiCaprio's girlfriend.
Action: Search
Action Input: "Leo DiCaprio girlfriend"
Observation: Leonardo DiCaprio and Girlfriend Vittoria Ceretti Are 'In Love': Source | UsWeekly. By clicking Sign Up, you agree to our Terms and Conditions and that you have read our Privacy Policy.
Source rate[Complete]: 1 questions [00:00, 110.96 questions/s] I should use a calculator to calculate the girlfriend's current age raised to the 0.43 power.
Action: Calculator
Source rate[Complete]: 1 questions [00:00, 110.96 questions/s]E20231211 15:41:10.464584 187172 runnable.hpp:112] /linear_segment_0/llm-engine-3; rank: 0; size: 1; tid: 140383245805248 Unhandled exception occurred. Rethrowing
E20231211 15:41:10.464756 187172 context.cpp:124] /linear_segment_0/llm-engine-3; rank: 0; size: 1; tid: 140383245805248: set_exception issued; issuing kill to current runnable. Exception msg: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result

E20231211 15:41:10.465035 187043 runner.cpp:189] RuE20231211 15:41:10.465302 187043 segment_instance.cpp:273] segment::SegmentInstance - an exception was caught while awaiting on one or more nodes - rethrowing[00:00, ? events/s]
E20231211 15:41:10.465325 187043 service.cpp:224] Service[segment::SegmentInstance]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
E20231211 15:41:10.465353 187043 pipeline_instance.cpp:230] pipeline::PipelineInstance - an exception was caught while awaiting on segments - rethrowing
E20231211 15:41:10.465374 187043 service.cpp:224] Service[pipeline::PipelineInstance]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
E20231211 15:41:10.465499 187043 service.cpp:224] Service[pipeline::Manager]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
E20231211 15:41:10.465521 187043 service.cpp:224] Service[ExecutorDefinition]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
Source rate[Complete]: 1 questions [00:00, 110.96 questions/s]
Upload rate[Complete]: 0 events [00:00, ? events/s]
Exception occurred in pipeline. Rethrowing
Traceback (most recent call last):
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 760, in getArguments
    a = local_dict[name]
KeyError: 'age'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 80, in _evaluate_expression
    numexpr.evaluate(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 975, in evaluate
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 874, in validate
    arguments = getArguments(names, local_dict, global_dict, _frame_depth=_frame_depth)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 762, in getArguments
    a = global_dict[name]
KeyError: 'age'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dagardner/work/morpheus/morpheus/pipeline/pipeline.py", line 419, in join
    await executor.join_async()
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/dagardner/work/morpheus/morpheus/llm/nodes/langchain_agent_node.py", line 75, in execute
    results = await self._run_single(**input_dict)
  File "/home/dagardner/work/morpheus/morpheus/llm/nodes/langchain_agent_node.py", line 61, in _run_single
    results = await asyncio.gather(*results_async)
  File "/home/dagardner/work/morpheus/morpheus/llm/nodes/langchain_agent_node.py", line 69, in _run_single
    return await self._agent_executor.arun(**kwargs)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 266, in arun
    return (await self.acall(kwargs, callbacks=callbacks))[self.output_keys[0]]
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 178, in acall
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 172, in acall
    await self._acall(inputs, run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/agents/agent.py", line 1002, in _acall
    next_step_output = await self._atake_next_step(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/agents/agent.py", line 928, in _atake_next_step
    result = await asyncio.gather(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/agents/agent.py", line 909, in _aperform_agent_action
    observation = await tool.arun(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/tools/base.py", line 358, in arun
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/tools/base.py", line 330, in arun
    await self._arun(*tool_args, run_manager=run_manager, **tool_kwargs)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/tools/base.py", line 430, in _arun
    await self.coroutine(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 263, in arun
    return (await self.acall(args[0], callbacks=callbacks))[self.output_keys[0]]
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 178, in acall
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 172, in acall
    await self._acall(inputs, run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 163, in _acall
    return await self._aprocess_llm_result(llm_output, _run_manager)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 125, in _aprocess_llm_result
    output = self._evaluate_expression(expression)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 87, in _evaluate_expression
    raise ValueError(
ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression


</pre></details>


### Full env printout

<details><summary>Click here to see environment details</summary><pre>

 [Paste the results of print_env.sh here, it will be hidden by default]

</pre></details>


### Other/Misc.

_No response_

### Code of Conduct

- [X] I agree to follow Morpheus' Code of Conduct
- [X] I have searched the [open bugs](https://github.com/nv-morpheus/Morpheus/issues?q=is%3Aopen+is%3Aissue+label%3Abug) and have found no duplicates for this bug report