Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kube integration #1436

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

nickjiang2378
Copy link
Contributor

@nickjiang2378 nickjiang2378 commented Sep 2, 2024

This PR sets up kubernetes within Hydro Deploy. Users can create Pod hosts and deploy their programs on the machines just as they would with localhost and cloud providers. Note that we're assuming here that users will already have a Kubernetes cluster set up somewhere. This is still a pretty manual process for users and is just the start of auto-scaling within Hydro.

@nickjiang2378 nickjiang2378 changed the title Kube integration rebased [DO NOT MERGE] Kube integration rebased Sep 2, 2024
@nickjiang2378 nickjiang2378 marked this pull request as draft September 2, 2024 19:11
@nickjiang2378 nickjiang2378 changed the title [DO NOT MERGE] Kube integration rebased Kube integration Sep 4, 2024
@@ -47,9 +47,3 @@ inherits = "release"
debug = 2
lto = "off"
strip = "none"

[profile.dev.package.website_playground]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why were these removed?

Copy link
Member

@shadaj shadaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass

memo-map = "0.3.0"
nameof = "1.0.0"
nanoid = "0.4.0"
nix = { version = "0.29.0", features = [ "signal" ] }
once_cell = "1.17"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check again if these are necessary

@@ -8,8 +8,8 @@ use tokio::sync::RwLock;

use super::gcp::GcpNetwork;
use super::{
progress, CustomService, GcpComputeEngineHost, Host, LocalhostHost, ResourcePool,
ResourceResult, Service,
progress, CustomService, GCPComputeEngineHost, Host, LocalhostHost, ResourcePool,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this was copied over from before @MingweiSamuel's refactor

@@ -91,6 +91,11 @@ pub async fn build_crate_memoized(params: BuildParams) -> Result<&'static BuildO
tokio::task::spawn_blocking(move || {
let mut command = Command::new("cargo");
command.args(["build"]);
// command.args([
// "zigbuild".to_string(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we want to avoid use of zigbuild. Using the musl target with rust-lld for linking should get us sufficiently far.

@@ -213,7 +213,7 @@ impl GcpComputeEngineHost {
#[async_trait]
impl Host for GcpComputeEngineHost {
fn target_type(&self) -> HostTargetType {
HostTargetType::Linux
HostTargetType::Linux(crate::LinuxArchitecture::AARCH64)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we should probably default to x86_64, and also have a way for users to pass this in when constructing the host (until later when we can auto-infer based on the machine type).

#[cfg(unix)]
use std::sync::Arc;

use async_channel::{Receiver, Sender};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @MingweiSamuel replaced this with a tokio builtin if I remember correctly?? Will need to check against the SSH implementation.

async fn provision(&mut self, _resource_result: &Arc<ResourceResult>) -> Arc<dyn LaunchedHost> {
if self.launched.is_none() {
let client = Client::try_default().await.unwrap();
let alphabet = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should move this to a top-level constant in this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants