Run cloud project locally #12198

4e6 · 2025-01-30T17:13:34Z

Pull Request Description

implements https://github.com/enso-org/cloud-v2/issues/1651

This is a POC of the Hybrid execution feature https://github.com/enso-org/cloud-v2/issues/1650

Changelog:

update: add Run Locally button for Cloud projects that downloads the project into a temporary directory and starts it using the local backend
update: add Hybrid Execution feature that enables Run Locally button for Cloud projects. The feature is enabled for dev builds
update: recreate missing .enso/project.json metadata when listing projects using the project-manager-shim-middleware

Important Notes

Uploading logic is implemented but blocked by https://github.com/enso-org/cloud-v2/issues/1715

enso-12198-run-locally.mp4

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.

github-actions · 2025-01-30T17:18:56Z

🧪 Storybook is successfully deployed!

📊 Dashboard:

👀 Review changes: https://www.chromatic.com/build?appId=673738989fc3e8430364197e&number=1489
👨‍🎨 Preview storybook: https://673738989fc3e8430364197e-hupsgxqahl.chromatic.com/

app/gui/project-manager-shim-middleware/index.ts

app/gui/project-manager-shim-middleware/projectManagement.ts

MrFlashAccount · 2025-02-13T08:01:10Z

app/gui/project-manager-shim-middleware/projectManagement.ts

+
+/** Unpack a .tar.gz enso-project bundle into a temporary directory */
+export async function unpackBundle(bundle: stream.Readable): Promise<string> {
+  const ensoProjectsDirectory =


is it correct that in shim we don't respect the root directory setting ?

I updated the logic. Right now we create a {projects-root}/cloud-{projectId} directory for the cloud project. In the future, we will make it hidden. I suppose it requires using a native library for Windows.

MrFlashAccount · 2025-02-13T08:04:38Z

app/gui/src/dashboard/hooks/projectHooks.ts

-      if (projectsStore.getState().launchedProjects.length > 0) {
-        closeAllProjects()
+  return eventCallbacks.useEventCallback(
+    (project: LaunchedProject & { cloudProjectId?: backendModule.ProjectId }) => {


What's the purpose of keeping cloudProjectId property separately from the LaunchedProject prop?

This is an extra parameter I need to pass when starting the project in the Hybrid mode. We need an original id of the cloud project to upload it after it is closed.

I mean, why we won't add it to LaunchedProject interface, or introduce another interface that extends the LaunchedProject ?

Ok, I updated the schema

MrFlashAccount · 2025-02-13T08:11:25Z

app/gui/src/dashboard/hooks/projectHooks.ts

      await client.resetQueries({ queryKey: createGetProjectDetailsQuery.getQueryKey(id) })
      setProjectAsset(type, id, parentId, (asset) => ({
        ...asset,
        projectState: { ...asset.projectState, type: backendModule.ProjectState.closed },
      }))
+
+      // If the project runs in hybrid execution mode
+      if (cloudProjectId) {


I case if user closes local project in offline mode, the mutation will fail because remoteBackend is not available. This might lead to unexpected behavior.

Not sure how to handle the failure properly. I'll leave a TODO comment

i think typical for dashboard isa FIXME with a link to a new issue in cloud-v2

MrFlashAccount · 2025-02-13T08:16:18Z

app/gui/src/dashboard/layouts/AssetContextMenu.tsx

+            doAction={async () => {
+              invariant(localBackend != null, 'Local Backend is null')
+              const parentId = await remoteBackend.downloadProject(asset.id)
+              const assets = await localBackend.listDirectory({


prefer ensureQueryData over calling methods manually: https://tanstack.com/query/latest/docs/reference/QueryClient#queryclientensurequerydata

Prefer mutations over direct calls here. Also, consider moving this out of the function for better reusability

MrFlashAccount · 2025-02-13T08:17:32Z

app/gui/src/dashboard/services/RemoteBackend.ts

+  async downloadProject(id: backend.ProjectId): Promise<DirectoryId> {
+    const details = await this.getProjectDetails(id, true)
+
+    invariant(details.url != null, 'The download URL of the project must be present.')


prefer passing id directly using parameters instead.

The invariant method only silences the compiler. getPropertyDetails should always return url property when called with the parameter true. It is used this way in the download method above, for example

MrFlashAccount · 2025-02-13T08:19:06Z

app/gui/src/dashboard/services/RemoteBackend.ts

+    const response = await this.client.get(`./api/cloud/download-project?${queryString}`)
+    const path = await response.text()
+
+    invariant(response.ok, 'The download-project response must have status OK.')


As I mentioned above, the invariant is supposed for uh-oh situations. Prefer throwing an error here instead

MrFlashAccount · 2025-02-13T08:22:23Z

app/gui/src/dashboard/services/RemoteBackend.ts

+      directory: extractIdFromDirectoryId(directoryId),
+    })
+
+    await this.client.get(`./api/cloud/upload-project?${queryString}`)


I think it's better to have / instead of ./ here. Also we prefer url constructor functions instead of bare bone strings. Please take a look at similar methods and do it similarly

It looks like this module uses strings. I wasn't able to find any examples of URL usages

It was inspired by

enso/app/gui/src/dashboard/layouts/AssetContextMenu.tsx

Lines 302 to 304 in 28a1b03

const projectResponse = await fetch(

`./api/project-manager/projects/${localBackendModule.extractTypeAndId(asset.id).id}/enso-project`,

)

yeah fwiw:

i prefer ./ so that it works when proxied with a base path (e.g. through code-server or openvscode-server)

however i guess that's not super robust - instead we could use APP_BASE_URL: https://github.com/enso-org/enso/blob/e002d942a5d49ec2d275fdeb8f0c3e62ebaa2906/app/gui/src/dashboard/utilities/appBaseUrl.ts

dunno about url constructors here. maybe it's possible for it to work but i think this is more understandable. after all, only the path needs changing so idk if there are many advantages to using new URL instead (if there were query string changes involved i would absolutely recommend new URL though)

app/gui/src/dashboard/layouts/AssetContextMenu.tsx

PabloBuchu · 2025-02-19T08:42:37Z

Can you add mocked support for calling endpoints:

POST /projects/{PROJECT_ID}//open
POST /projects/{PROJECT_ID}/set_open
POST /projects/{PROJECT_ID}/close

4e6 · 2025-02-20T17:56:40Z

@PabloBuchu what's the difference between open and set_open endpoints? When should each of them be called?

4e6 added the CI: No changelog needed Do not require a changelog entry for this PR. label Jan 30, 2025

4e6 self-assigned this Jan 30, 2025

4e6 added 3 commits February 4, 2025 16:43

feat: RemoteBackend upload

e1402ba

DRAFT: run locally

058af73

feat: download project middleware

0b02904

4e6 force-pushed the wip/db/1651-cloud-sync branch from 5b95795 to aebbad9 Compare February 4, 2025 18:14

fix: project downloading

3697a86

4e6 force-pushed the wip/db/1651-cloud-sync branch from aebbad9 to 3697a86 Compare February 5, 2025 13:49

4e6 added 10 commits February 5, 2025 15:11

feat: recover missing project metadata

2164575

fix: open project

d9e774f

update: move project opening logic to menu

ae3d755

misc: format

16d374f

fix: lint

b07cd08

fix: open project

55fc8b5

feat: add feature flag

ff2745a

feat: upload project

690596a

Merge branch 'develop' into wip/db/1651-cloud-sync

2a393c3

misc: cleanup

64f5249

4e6 marked this pull request as ready for review February 10, 2025 16:42

4e6 requested review from PabloBuchu, indiv0, somebody1234, MrFlashAccount, Frizi, farmaazon, vitvakatu, kazcw and AdRiley as code owners February 10, 2025 16:42

MrFlashAccount requested changes Feb 13, 2025

View reviewed changes

4e6 added 8 commits February 13, 2025 14:26

Merge branch 'develop' into wip/db/1651-cloud-sync

2ca910f

update: simplify download request

315d325

update: simplify upload

0f727ee

update: download directory name

da86fff

update: use crypto

00ef2df

update: failure handling todo

abd07d9

update: list projects with ensureQueryData

304a970

update: downloadProject error

a905044

4e6 requested a review from MrFlashAccount February 14, 2025 09:04

4e6 added 2 commits February 14, 2025 09:52

misc: format

5291418

update: project schema

eb5c2e9

Merge branch 'develop' into wip/db/1651-cloud-sync

1d8d4c0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run cloud project locally #12198

Run cloud project locally #12198

4e6 commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

MrFlashAccount Feb 13, 2025

4e6 Feb 13, 2025

MrFlashAccount Feb 13, 2025

4e6 Feb 13, 2025

MrFlashAccount Feb 14, 2025

4e6 Feb 17, 2025

MrFlashAccount Feb 13, 2025

4e6 Feb 13, 2025

somebody1234 Feb 24, 2025

MrFlashAccount Feb 13, 2025

MrFlashAccount Feb 13, 2025

4e6 Feb 13, 2025

MrFlashAccount Feb 13, 2025

MrFlashAccount Feb 13, 2025

4e6 Feb 13, 2025

4e6 Feb 13, 2025

somebody1234 Feb 24, 2025

PabloBuchu commented Feb 19, 2025

4e6 commented Feb 20, 2025

	const projectResponse = await fetch(
	`./api/project-manager/projects/${localBackendModule.extractTypeAndId(asset.id).id}/enso-project`,
	)

Run cloud project locally #12198

Are you sure you want to change the base?

Run cloud project locally #12198

Conversation

4e6 commented Jan 30, 2025 • edited Loading

Pull Request Description

Important Notes

Checklist

github-actions bot commented Jan 30, 2025 • edited Loading

🧪 Storybook is successfully deployed!

📊 Dashboard:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PabloBuchu commented Feb 19, 2025

4e6 commented Feb 20, 2025

4e6 commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading