Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA]: Support different output data types in the HttpSourceStage #1811

Open
2 tasks done
mdemoret-nv opened this issue Jul 12, 2024 · 1 comment · May be fixed by #1834
Open
2 tasks done

[FEA]: Support different output data types in the HttpSourceStage #1811

mdemoret-nv opened this issue Jul 12, 2024 · 1 comment · May be fixed by #1834
Assignees
Labels
feature request New feature or request

Comments

@mdemoret-nv
Copy link
Contributor

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Currently, the HttpSourceStage only supports outputting a MessageMeta object. This is very limiting in situations where some additional metadata needs to be attached to the message that is only available with the HTTP request object.

Describe your ideal solution

Support outputting different data types by the HttpSourceStage class. Ideally, it should be capable of outputting any type which is returned by the endpoint processing lambda. At a minimum, ControlMessages should be able to be generated by this stage.

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
  • I have searched the open feature requests and have found no duplicates for this feature request
@mdemoret-nv mdemoret-nv added the feature request New feature or request label Jul 12, 2024
@dagardner-nv
Copy link
Contributor

dagardner-nv commented Aug 2, 2024

Ideally, it should be capable of outputting any type which is returned by the endpoint processing lambda.

Both HttpServerSourceStage and HttpClientSourceStage support a payload_to_df_fn method, however the output is a DataFrame.

If we wanted to do this I think there are two options:

  1. Break API compatibility and change this to a request_to_msg_fn method
  2. Add a second optional method to convert both the payload and the DataFrame to a message.

However currently for HttpServerSourceStage specifying payload_to_df_fn forces the Python impl of the stage (although the server itself is still implemented in C++).

I'm thinking instead doing something similar to what the DeserializeStage does and do:

  1. Add a message_type & task_type constructor arguments
  2. Add a request_to_task_payload_fn constructor argument. When None a default C++ impl will be used. When not-none the Python source stage impl will be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
Status: Review - Ready for Review
2 participants