fix(cluster): write install script for worker pool #2009

MasterPtato · 2025-02-11T02:32:59Z

Changes

cloudflare-workers-and-pages · 2025-02-11T02:33:01Z

Deploying rivet with Cloudflare Pages

Latest commit:	`257ad3e`
Status:	⚡️ Build in progress...

View logs

MasterPtato · 2025-02-11T02:33:24Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

greptile-apps

PR Summary

This PR implements installation scripts and configurations for worker pools in the Rivet cluster management system, focusing on proper service connectivity and resource management.

Added new worker pool type in sdks/api/fern/definition/provision/common.yml with corresponding SDK updates
Implemented worker pool configuration in packages/common/config/src/config/server/rivet/cluster_provision.rs with proper service URLs and port mappings
Added FoundationDB configuration support in packages/edge/infra/client/isolate-v8-runner/src/main.rs for worker pools
Added datacenter_name_id field to server info responses for better datacenter identification
Configured tunnel services with specific ports (CRDB: 5040, Redis: 5041/5042, ClickHouse: 5043/5044, S3: 5045, NATS: 5046, Prometheus: 5047) in packages/core/services/cluster/src/workflows/server/install/install_scripts/components/traefik.rs

_{131 file(s) reviewed, 28 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-02-14T23:43:05Z

docker/universal/Dockerfile

+    --mount=type=cache,target=/usr/local/cargo/git,id=univseral-cargo-git \
+    --mount=type=cache,target=/usr/local/cargo/registry,id=univseral-cargo-registry \
+    --mount=type=cache,target=/app/target,id=univseral-target \


syntax: typo in cache mount ids: 'univseral' should be 'universal'

greptile-apps · 2025-02-14T23:43:45Z

packages/common/config/src/config/server/mod.rs

-			connection: "fdb:[email protected]:4500".to_string(),
+			cluster_description: "fdb".to_string(),
+			cluster_id: "fdb".to_string(),
+			addresses: Addresses::Static(vec!["127.0.0.1:4500".to_string()]),


style: default address includes port 4500 but should use the FDB constant from default_ports to stay DRY

greptile-apps · 2025-02-14T23:44:47Z

packages/common/pools/src/db/fdb.rs

+				async move {
+					let joined = servers
+						.into_iter()
+						.filter_map(|server| server.lan_ip)


logic: No handling for empty server list - could cause invalid connection string if all servers are filtered out

greptile-apps · 2025-02-14T23:44:47Z

packages/common/pools/src/db/fdb.rs

+						cluster_id = fdb_config.cluster_id,
+					);
+
+					fs::write(temp_path, connection.as_bytes()).await?;


logic: Multiple concurrent service discovery callbacks could race when writing to the same file

greptile-apps · 2025-02-14T23:44:57Z

packages/common/service-discovery/Cargo.toml

+rand = "0.8"
+reqwest = { version = "0.11", features = ["json"] }
+serde = { version = "1.0", features = ["derive"] }
+tokio = { version = "1.40", features = ["full"] }


style: tokio 1.40 is very new and may cause compatibility issues. Consider using a more established version like 1.32 which is used in other parts of the codebase.

greptile-apps · 2025-02-15T00:12:16Z

packages/edge/infra/edge-server/Dockerfile

+FROM scratch
+COPY --from=rust /app/dist/ /


logic: Using scratch base image without libfdb_c.so will cause runtime errors since the FoundationDB client library is required. Need to copy libfdb_c.so from builder stage.

greptile-apps · 2025-02-15T00:12:16Z

packages/edge/infra/edge-server/Dockerfile

+COPY --from=rust /app/dist/ /
+
+# Allows `docker create` to work even though this fails
+CMD [""]


logic: Empty CMD will cause container to fail immediately. Should set proper entrypoint to run rivet-edge-server.

greptile-apps · 2025-02-15T00:12:20Z

packages/edge/infra/edge-server/Dockerfile.dockerignore

@@ -0,0 +1,8 @@
+*


style: Consider adding a comment header explaining the purpose of this dockerignore file

greptile-apps · 2025-02-15T00:13:06Z

packages/edge/services/pegboard/Cargo.toml

@@ -33,6 +34,7 @@ util.workspace = true
 uuid = "1.11.0"

 server-spec = { workspace = true, optional = true }
+token-create.workspace = true


style: token-create is not marked as optional despite being likely only needed for certain features like the other dependencies

greptile-apps · 2025-02-15T00:14:24Z

packages/edge/services/pegboard/src/workflows/client/mod.rs

+		base_path: util::url::to_string_without_slash(&edge.intercom_endpoint),
+		bearer_access_token: Some(token),
 		..Default::default()
 	};


style: Consider adding error handling for URL parsing failures

cloudflare-workers-and-pages · 2025-02-19T05:16:40Z

Deploying rivet-hub with Cloudflare Pages

Latest commit:	`9936cb1`
Status:	⚡️ Build in progress...

View logs

cloudflare-workers-and-pages · 2025-02-20T22:09:27Z

Deploying rivet with Cloudflare Pages

Latest commit:	`257ad3e`
Status:	✅ Deploy successful!
Preview URL:	https://c8c5ec4d.rivet.pages.dev
Branch Preview URL:	https://graphite-base-2020.rivet.pages.dev

View logs

cloudflare-workers-and-pages · 2025-02-21T05:27:54Z

Deploying rivet-hub with Cloudflare Pages

Latest commit:	`9936cb1`
Status:	✅ Deploy successful!
Preview URL:	https://9930d981.rivet-hub-7jb.pages.dev
Branch Preview URL:	https://02-11-fix-cluster-write-inst.rivet-hub-7jb.pages.dev

View logs

cloudflare-workers-and-pages · 2025-02-21T05:28:39Z

Deploying rivet with Cloudflare Pages

Latest commit:	`9936cb1`
Status:	✅ Deploy successful!
Preview URL:	https://b9ab0ff9.rivet.pages.dev
Branch Preview URL:	https://02-11-fix-cluster-write-inst.rivet.pages.dev

View logs

This was referenced Feb 11, 2025

feat(clusters): add worker pool type #2008

Open

fix: get stuff building #2020

Open

fix: get tunnels working, fix tls, wf bug fix #2025

Open

fix: get actors running e2e on edge #2027

Open

MasterPtato marked this pull request as ready for review February 14, 2025 23:42

MasterPtato requested a review from NathanFlurry February 14, 2025 23:42

MasterPtato mentioned this pull request Feb 14, 2025

fix: get edge api access over gg #2032

Open

greptile-apps bot reviewed Feb 15, 2025

View reviewed changes

MasterPtato mentioned this pull request Feb 15, 2025

feat(workflows): abstract debug trait #2033

Open

MasterPtato force-pushed the 02-08-feat_clusters_add_worker_pool_type branch from 24ae5de to 02f4f81 Compare February 15, 2025 01:52

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 741ef37 to 0618a18 Compare February 15, 2025 01:52

MasterPtato force-pushed the 02-08-feat_clusters_add_worker_pool_type branch from 02f4f81 to 7bdc5b6 Compare February 18, 2025 02:28

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 0618a18 to 72e6e1e Compare February 18, 2025 02:32

MasterPtato mentioned this pull request Feb 18, 2025

feat(workflows): impl db debug for fdb driver #2040

Open

NathanFlurry changed the base branch from 02-08-feat_clusters_add_worker_pool_type to graphite-base/2009 February 19, 2025 02:56

NathanFlurry force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 72e6e1e to d4e7b15 Compare February 19, 2025 02:59

NathanFlurry force-pushed the graphite-base/2009 branch from 7bdc5b6 to e6dd614 Compare February 19, 2025 02:59

This was referenced Feb 19, 2025

chore: add native fdb library to path #2045

Open

chore: add fdb-backed sqlite driver #2046

Open

This was referenced Feb 19, 2025

chore: add timeout flag to docker-compose down #2048

Open

fix: force amd64 platform for FDB tests #2049

Open

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from d4e7b15 to b439967 Compare February 19, 2025 20:16

MasterPtato force-pushed the graphite-base/2009 branch from e6dd614 to c333240 Compare February 19, 2025 20:16

MasterPtato mentioned this pull request Feb 19, 2025

feat: add fdb cli #2051

Open

MasterPtato force-pushed the graphite-base/2009 branch from c333240 to 0bb9a93 Compare February 19, 2025 20:25

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from b439967 to 1ca83db Compare February 19, 2025 20:25

MasterPtato mentioned this pull request Feb 20, 2025

fix: move actor state to fdb #2056

Open

MasterPtato force-pushed the graphite-base/2009 branch from 0bb9a93 to 9f0f985 Compare February 20, 2025 01:55

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 1ca83db to 257ad3e Compare February 20, 2025 01:55

fix(cluster): write install script for worker pool

9936cb1

NathanFlurry force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 257ad3e to 9936cb1 Compare February 20, 2025 02:28

NathanFlurry force-pushed the graphite-base/2009 branch from 9f0f985 to 9763a09 Compare February 20, 2025 02:28

MasterPtato force-pushed the graphite-base/2009 branch from 9763a09 to 9f0f985 Compare February 20, 2025 02:36

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 9936cb1 to 257ad3e Compare February 20, 2025 02:36

MasterPtato mentioned this pull request Feb 20, 2025

feat(workflows): implement metrics for fdb driver #2057

Open

NathanFlurry force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 257ad3e to 9936cb1 Compare February 20, 2025 02:55

NathanFlurry force-pushed the graphite-base/2009 branch from 9f0f985 to 9763a09 Compare February 20, 2025 02:55

This was referenced Feb 20, 2025

fix(docker): fix corepack not enabled error #2058

Open

feat: add support for JSON5/JSONC config files #2059

Open

chore: add basic load test #2060

Open

MasterPtato force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 9936cb1 to 257ad3e Compare February 20, 2025 22:00

MasterPtato force-pushed the graphite-base/2009 branch from 9763a09 to 9f0f985 Compare February 20, 2025 22:00

MasterPtato mentioned this pull request Feb 20, 2025

fix(workflows): fix activity error backoff, in memory polling #2062

Open

MasterPtato mentioned this pull request Feb 21, 2025

fix(fdb): conflict ranges #2063

Open

MasterPtato changed the base branch from graphite-base/2009 to 02-08-feat_clusters_add_worker_pool_type February 21, 2025 02:25

NathanFlurry force-pushed the 02-11-fix_cluster_write_install_script_for_worker_pool branch from 257ad3e to 9936cb1 Compare February 21, 2025 05:17

NathanFlurry force-pushed the 02-08-feat_clusters_add_worker_pool_type branch from 9f0f985 to 9763a09 Compare February 21, 2025 05:17

NathanFlurry mentioned this pull request Feb 21, 2025

fix(job-runner): fix build pack in job-runner Dockerfile #2064

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(cluster): write install script for worker pool #2009

fix(cluster): write install script for worker pool #2009

MasterPtato commented Feb 11, 2025

cloudflare-workers-and-pages bot commented Feb 11, 2025 •

edited

Loading

MasterPtato commented Feb 11, 2025 •

edited by NathanFlurry

Loading

greptile-apps bot left a comment

greptile-apps bot Feb 14, 2025

greptile-apps bot Feb 14, 2025

greptile-apps bot Feb 14, 2025

greptile-apps bot Feb 14, 2025

greptile-apps bot Feb 14, 2025

greptile-apps bot Feb 15, 2025

greptile-apps bot Feb 15, 2025

greptile-apps bot Feb 15, 2025

greptile-apps bot Feb 15, 2025

greptile-apps bot Feb 15, 2025

cloudflare-workers-and-pages bot commented Feb 19, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Feb 20, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Feb 21, 2025

cloudflare-workers-and-pages bot commented Feb 21, 2025

fix(cluster): write install script for worker pool #2009

Are you sure you want to change the base?

fix(cluster): write install script for worker pool #2009

Conversation

MasterPtato commented Feb 11, 2025

Changes

cloudflare-workers-and-pages bot commented Feb 11, 2025 • edited Loading

Deploying rivet with Cloudflare Pages

MasterPtato commented Feb 11, 2025 • edited by NathanFlurry Loading

How to use the Graphite Merge Queue

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Feb 14, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 14, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 14, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 14, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 14, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 15, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 15, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 15, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 15, 2025

Choose a reason for hiding this comment

greptile-apps bot Feb 15, 2025

Choose a reason for hiding this comment

cloudflare-workers-and-pages bot commented Feb 19, 2025 • edited Loading

Deploying rivet-hub with Cloudflare Pages

cloudflare-workers-and-pages bot commented Feb 20, 2025 • edited Loading

Deploying rivet with Cloudflare Pages

cloudflare-workers-and-pages bot commented Feb 21, 2025

Deploying rivet-hub with Cloudflare Pages

cloudflare-workers-and-pages bot commented Feb 21, 2025

Deploying rivet with Cloudflare Pages

cloudflare-workers-and-pages bot commented Feb 11, 2025 •

edited

Loading

MasterPtato commented Feb 11, 2025 •

edited by NathanFlurry

Loading

cloudflare-workers-and-pages bot commented Feb 19, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Feb 20, 2025 •

edited

Loading