Connect to SageMaker-hosted LLM

[mayonesa]

I would like to use Semantic Kernel Processes (Python) for my workflow where some steps will require AI. I would like to use my sagemaker-hosted LLM for this. However, I don't see how to do that in the documentation, the code samples, or internet. Can someone help me make this connection happen?

[moonbox3]

Hi @mayonesa, thanks for your question. We have an Amazon Bedrock AI Service connector available in Python. From what I see this is one component of Amazon SageMaker AI. I found the following article produced by Amazon in December of 2024 that may allow one to call a SageMaker deployed model via Bedrock. Can you have a look, please?

It may be as easy as:

• Deploy a Model in SageMaker JumpStart:
  • Navigate to SageMaker Studio and select a model from the JumpStart catalog.
  • Deploy the chosen model to create an endpoint.
• Register the SageMaker Endpoint with Amazon Bedrock:
  • In the SageMaker Studio, locate your deployed endpoint.
  • Choose the option to register this endpoint with Amazon Bedrock.

Now, in Semantic Kernel Python, you can follow the instructions to configure the AI service connector and connect to that model.

https://aws.amazon.com/blogs/machine-learning/use-amazon-bedrock-tooling-with-amazon-sagemaker-jumpstart-models/?utm_source=chatgpt.com

[mayonesa]

Thank you so much! Unfortunately, there will be extra costs associated w/ invoking my SageMaker-hosted model through Bedrock.

[moonbox3]

I understand, @mayonesa. I turned this discussion into an issue so we can triage/prioritize it as a team. You can follow along with #10175 if you'd like. We'll post any update there.