[Serve] Expose load balancer and services via TLS.

[maxmele]

Recently, I've been testing SkyPilot to expose a LLM API using sky serve. Everything is working great and I really like the project, thank you for the great work!

However, I couldn't find any reference in the documentation on how to expose the services and load-balancers via TLS.

The prompts we send to the API contain sensitive data that we prefer not to transmit in plain text over the internet. So it would be really nice to be able to configure TLS for both the load balancer and services.

Do you have any plans to implement this feature in the future?

[concretevitamin]

Thanks for this report and glad to see SkyPilot is working @maxmele!

We're actively looking into the security aspects of SkyServe. A few questions:

• Is it possible for the serving app to handle encryption + decryption? Or is it too much of a burden?
• Is any of these options good enough (they are not directly about traffic encryption)?
  • (Supported in main already) Launch an entire serve deployment (controller + replicas) in a private VPC, exposing private IPs only
  • Use a VPN service (e.g., Tailscale) to put an entire serve deployment (controller + replicas) in
  • Some other options we're brainstorming

[maxmele]

Hi, thanks for the response!

Absolutely! We were tinking about implementing encryption/decryption between ends, but we're hoping to find an alternative solution to avoid that.

Regarding the option of using private IPs within a VPC... doesn't it limit the ability to be multicloud and even multi-region? Unless you establish connections between different VPCs beforehand.

For me, it would be absolutely amazing if skypilot could connect nodes to a Tailscale network, and even better if it could use a custom control server like Headscale.

[concretevitamin]

Regarding the option of using private IPs within a VPC... doesn't it limit the ability to be multicloud and even multi-region? Unless you establish connections between different VPCs beforehand.

Yep, that's the tradeoff of that approach. Peering between VPCs needed for multi-region, and it doesn't support multi-cloud out of the box.

The team actually has been brainstorming quite a few options (cc @Michaelvll @cblmemo). Feel free to join https://slack.skypilot.co/ Slack as we'd love to learn more about your deployment requirements!

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[Michaelvll]

This is being added by #3380

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.