feat: add support for VNC using x11vnc and novnc #15

synacktraa · 2025-02-04T19:36:44Z

Added new custom template with e2b wallpaper and vnc support

New parameters:

"""
:param resolution: Startup the desktop with custom screen resolution. Defaults to (1024, 768)
:param dpi: Startup the desktop with custom DPI. Defaults to 96
:param display: Startup the desktop with custom display. Defaults to ":0"
:param vnc_port: Port number for VNC server. Defaults to 5900
:param novnc_port: Port number for noVNC server. Defaults to 6080
:param enable_novnc_auth: Enable noVNC server authentication. Defaults to False
"""

~~This don't really do anything now because environment variables passed to the sandbox are not available to start-up.sh script. Please look into this issue.~~

Support for password protected VNC. ~~It doesn't work right now because of the above mentioned issue.~~

~~Todo: Update js-sdk~~

mlejva · 2025-02-04T19:53:31Z

Thank you for the PR @synacktraa! I'll review the PR soon.

Can you provide steps to test the VNC support?

synacktraa · 2025-02-04T19:55:24Z

Thank you for the PR @synacktraa! I'll review the PR soon.

Can you provide steps to test the VNC support?

desktop = Desktop()
desktop.vnc_url

That's it!

synacktraa · 2025-02-04T20:02:42Z

@mlejva A suggestion: Build the template and publish it from your account as currently the template is under my account.

mlejva · 2025-02-04T20:05:33Z

Thank you for the PR @synacktraa! I'll review the PR soon.
Can you provide steps to test the VNC support?
desktop = Desktop()
desktop.vnc_url
That's it!

This is beautiful. I just tried it and it works without issues. Nice PR! I'll dedicate some time to review it in the near future. Thank you again!

synacktraa · 2025-02-04T20:07:04Z

Thank you for the PR @synacktraa! I'll review the PR soon.
Can you provide steps to test the VNC support?
desktop = Desktop()
desktop.vnc_url
That's it!
This is beautiful. I just tried it and it works without issues. Nice PR! I'll dedicate some time to review it in the near future. Thank you again!

Alright, In the meantime I will learn some ts and update the js-sdk

synacktraa · 2025-02-05T12:17:43Z

Hi @mlejva, I have added browser only templates. They are lightweight (~1.34gb) unlike the desktop (~3.3gb) and works just fine with the SDK. It also supports VNC.

mishushakov · 2025-02-05T15:31:12Z

Hey, thanks for the PR, looks very cool! Did you also test connecting via a VNC client or does it only work through the web browser currently?

synacktraa · 2025-02-05T15:34:55Z

Hey, thanks for the PR, looks very cool! Did you also test connecting via a VNC client or does it only work through the web browser currently?

Yes I didn't knew about novnc initially. I tried exposing port 5900 but wasn't able to connect to it through VNC client (I used TightVNC viewer). I think novnc is already good enough, we can add this as an iframe (If we build something like openai's operator). Is there any particular reason why you want to use VNC client?

If you're testing it rn, you can try once - sandbox.get_host(5900)

mlejva · 2025-02-05T21:03:59Z

Hi @mlejva, I have added browser only templates. They are lightweight (~1.34gb) unlike the desktop (~3.3gb) and works just fine with the SDK. It also supports VNC.

Hey @synacktraa please keep this in a separate PR/discussion and leave this one only for the VNC support

synacktraa · 2025-02-05T21:08:49Z

Okay yeah, I am sorry about this. I will remove this and create a PR from another branch.

mlejva

@synacktraa can you leave this PR for purely VNC support? I see you made several unrelated changes in this PR like removing pyautogui and using xdotool instead.

I'd like to keep things strictly separate in their own PRs. This PR should only add VNC support and we can move from there.

synacktraa · 2025-02-05T21:19:14Z

@synacktraa can you leave this PR for purely VNC support? I see you made several unrelated changes in this PR like removing pyautogui and using xdotool instead.

I'd like to keep things strictly separate in their own PRs. This PR should only add VNC support and we can move from there.

I get it, but the ubuntu-desktop template doesn't includes the pyautogui dependency. The changes in SDK is totally based on how the template is written.

mlejva · 2025-02-05T21:32:11Z

@synacktraa can you leave this PR for purely VNC support? I see you made several unrelated changes in this PR like removing pyautogui and using xdotool instead.
I'd like to keep things strictly separate in their own PRs. This PR should only add VNC support and we can move from there.

I get it, but the ubuntu-desktop template doesn't includes the pyautogui dependency. The changes in SDK is totally based on how the template is written.

I see. Using xdotool could be even better thinking about it - no need to do the hoops of writing code into a python file and then running it. Instead we can just run the command directly.

The game plan:

Test if everything works with xdotool
Check how the VNC works in this PR
Check how we can support making it easy for users to embed the VNC streams into their apps
Explore the security model around the VNC streams
Move all logic to our desktop template that's already being used
Add needed changes to the JS SDK
Merge

Side note - we won't be able to merge this without updates to the JS SDK as well.

synacktraa · 2025-02-05T21:40:09Z

Sounds good. We can discuss about 3rd and 4th point after I am done with the first two.

mlejva · 2025-02-05T22:24:36Z

templates/ubuntu-desktop/start-up.sh

+start_x11vnc() {
+    (x11vnc -display :0 \
+        -forever \
+        $VNC_SHARED_FLAG \
+        -wait 50 \
+        -rfbport 5900 \
+        $VNC_PW_FLAG \
+        2>/tmp/x11vnc_stderr.log) &
+
+    x11vnc_pid=$!
+
+    # Wait for x11vnc to start
+    timeout=10
+    while [ $timeout -gt 0 ]; do
+        if netstat -tuln | grep ":5900 "; then
+            break
+        fi
+        sleep 1
+        ((timeout--))
+    done
+
+    (
+        while true; do
+            if ! kill -0 $x11vnc_pid 2>/dev/null; then
+                echo "x11vnc process crashed, restarting..." >&2
+                if [ -f /tmp/x11vnc_stderr.log ]; then
+                    echo "x11vnc stderr output:" >&2
+                    cat /tmp/x11vnc_stderr.log >&2
+                    rm /tmp/x11vnc_stderr.log
+                fi
+                start_x11vnc
+                return
+            fi
+            sleep 5
+        done
+    ) &
+}


Can you explain how this function works and why it's being called recursively?

I understand that the point is to check if vnc crashed and then try to restart it but would like to have it explained in the codebase as well.
Also, is this even required? If the vnc is failing shouldn't we just fail completely? What are the use cases when vnc can fail to start and we should keep trying starting it?

@synacktraa @mlejva Another way to approach this would be to not start the VNC server automatically on startup but make it a method in the SDK.

This would:

Prevent unnecessary overhead when VNC isn't being used.

Allow developers to handle the VNC server failing in their preferred manner.

Can you explain how this function works and why it's being called recursively?

I understand that the point is to check if vnc crashed and then try to restart it but would like to have it explained in the codebase as well.

Also, is this even required? If the vnc is failing shouldn't we just fail completely? What are the use cases when vnc can fail to start and we should keep trying starting it?

So first we run the x11vnc to start the vnc server, then we just give it some time to start. After that we use "kill -0" to check if the server is running without actually killing it - This happens in background, so if the server crashes or stops for some reason we just try to restart it.

I have not yet faced any fails when starting it. This was just to ensure If anything goes wrong it should start the service again.

@synacktraa @mlejva Another way to approach this would be to not start the VNC server automatically on startup but make it a method in the SDK.

This would:

Prevent unnecessary overhead when VNC isn't being used.

Allow developers to handle the VNC server failing in their preferred manner.

Makes sense. If you want this to be implemented just give a thumbs up to this reply. @jamesmurdza @mlejva

@synacktraa

@synacktraa @mlejva Another way to approach this would be to not start the VNC server automatically on startup but make it a method in the SDK.

This would:

Prevent unnecessary overhead when VNC isn't being used.

Allow developers to handle the VNC server failing in their preferred manner.

@synacktraa can you test how long a user would wait during runtime to start the VNC server after calling something like sandbox.start_vnc_server()? I like this approach more than having it enabled by default.

mlejva · 2025-02-05T22:29:52Z

templates/ubuntu-desktop/start-up.sh

+        -forever \
+        $VNC_SHARED_FLAG \
+        -wait 50 \
+        -rfbport 5900 \


Please put things like the 5900 VNC port number into a variable so it can be reused in subsequent calls

mlejva · 2025-02-06T00:50:41Z

templates/ubuntu-desktop/e2b.Dockerfile

@synacktraa please expand on what packages you added and why they're needed

I'm getting following error when trying to install xfce4-goodies. Did it work for you? Is that package required?

ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get -y upgrade && apt-get -y install xfce4 xfce4-panel xfce4-goodies xfce4-session xvfb xterm xdotool scrot imagemagick sudo mutter x11vnc build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl wget git libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev net-tools netcat software-properties-common && sudo add-apt-repository ppa:mozillateam/ppa && sudo apt-get install -y --no-install-recommends libreoffice firefox-esr x11-apps xpdf gedit xpaint tint2 galculator pcmanfm unzip && apt-get clean" did not complete successfully: exit code: 100 Error: Command failed: docker build . -f e2b.dev.Dockerfile --pull --platform linux/amd64 -t docker.e2b.dev/e2b/custom-envs/xrlk1xkhq3q708rhhqjg:70cb8d1b-b738-4711-90c5-2e20d9b7b85d

The same is happening when trying to install xfce4-terminal.

Yeah it did install without any issue or else It wouldn't be possible to build the image.

It contains some misc software and artwork. https://packages.debian.org/buster/xfce/xfce4-goodies

The same is happening when trying to install xfce4-terminal.

Okay, this is unexpected. Are there any more packages that is throwing the same error?

I'm getting following error when trying to install xfce4-goodies. Did it work for you? Is that package required?

ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get -y upgrade && apt-get -y install xfce4 xfce4-panel xfce4-goodies xfce4-session xvfb xterm xdotool scrot imagemagick sudo mutter x11vnc build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl wget git libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev net-tools netcat software-properties-common && sudo add-apt-repository ppa:mozillateam/ppa && sudo apt-get install -y --no-install-recommends libreoffice firefox-esr x11-apps xpdf gedit xpaint tint2 galculator pcmanfm unzip && apt-get clean" did not complete successfully: exit code: 100 Error: Command failed: docker build . -f e2b.dev.Dockerfile --pull --platform linux/amd64 -t docker.e2b.dev/e2b/custom-envs/xrlk1xkhq3q708rhhqjg:70cb8d1b-b738-4711-90c5-2e20d9b7b85d

I don't any see particular error message stating that goodies package has failed. Can you provide me with the whole error?

@synacktraa please expand on what packages you added and why they're needed

I am going to separate the installs in different RUN blocks so it easier to see and faster to rebuild.

The errors look like it was my issue. You can ignore it. Sorry.

Can you still do this please?

please expand on what packages you added and why they're needed

Yes, give me some time. I will reply when I get to my laptop.

@synacktraa please expand on what packages you added and why they're needed

I am going to separate the installs in different RUN blocks so it easier to see and faster to rebuild.

I just did this in my latest commit

mlejva · 2025-02-06T04:31:06Z

@synacktraa I created a separate template for development called desktop-dev. It's essentially a copy of your ubuntu-desktop. Can you please for now revert the changes you did to the original template? We'll keep the work in the desktop-dev template.

synacktraa · 2025-02-06T04:35:14Z

Sure, I will restore the original template directory.

mlejva · 2025-02-06T04:44:15Z

Besides my comments above, there are two additional things I'd like to resolve before getting merged:

The new env vars are used a bit all over the place. Start up script, Dockerfile, inside code. At least some of them seems unnecessary (Dockerfile?)
We need to make the VNC view in the browser literally just a view without anything else so our users can embed it in their apps and not see noVNC's branding and controls. Essentially just see a video when you open the VNC URL. Can we do that?

I also added a multi-client support in my previous commit

synacktraa · 2025-02-06T04:52:51Z

Besides my comments above, there are two additional things I'd like to resolve before getting merged:

The new env vars are used a bit all over the place. Start up script, Dockerfile, inside code. At least some of them seems unnecessary (Dockerfile?)

We need to make the VNC view in the browser literally just a view without anything else so our users can embed it in their apps and not see noVNC's branding and controls. Essentially just see a video when you open the VNC URL. Can we do that?

I also added a multi-client support in my previous commit

Especially the DISPLAY env was not available even after writing it to .bashrc so I ended up adding then before running xdotool commands. We can remove envs from dockerfile.
There is a way to customise it. I'll look into it.

synacktraa · 2025-02-06T05:27:16Z

We need to make the VNC view in the browser literally just a view without anything else so our users can embed it in their apps and not see noVNC's branding and controls. Essentially just see a video when you open the VNC URL. Can we do that?

@mlejva What about password authentication? Are we including it?

synacktraa · 2025-02-06T05:28:28Z

And If we're starting up vnc through code, why don't we do the same with xvfb? That way we can decide the screen resolution when creating sandbox instance. This can also bypass the issue with providing env vars to start up script.

jamesmurdza · 2025-03-07T14:41:57Z

By the way, I also added a new branch called dev-template-id which changes the template ID making it easier to test.

mishushakov

The DX is like 80% there. Same comments apply to Python

mishushakov · 2025-03-07T14:48:01Z

README.md

are the examples up-to-date?

Yes. Usage: #15 (comment)

template/e2b.Dockerfile

template/e2b.toml

packages/js-sdk/src/sandbox.ts

mishushakov · 2025-03-07T15:22:05Z

packages/js-sdk/src/sandbox.ts

   */
-  async moveMouse(x: number, y: number) {
-    return this.runPyautoguiCode(`pyautogui.moveTo(${x}, ${y})`)
+  private async setVncCommand(): Promise<void> {


setVncCommand(novncAuthEnabled)

Do this later.

mishushakov · 2025-03-07T15:22:31Z

packages/js-sdk/src/sandbox.ts

-    return this.runPyautoguiCode(`pyautogui.moveTo(${x}, ${y})`)
+  private async setVncCommand(): Promise<void> {
+    let pwdFlag = "-nopw";
+    if (this.desktop.novncAuthEnabled) {


if (novncAuthEnabled) {}

mishushakov · 2025-03-07T15:23:11Z

packages/js-sdk/src/sandbox.ts

-    return out
-  }
+    if (this.vncCommand === "") {
+      await this.setVncCommand();


this.setVncCommand({ novncAuthEnabled })

…getScreenSize()

jamesmurdza · 2025-03-07T21:09:57Z

I've made the eight changes (separate commits above) that @mishushakov and I talked about, and also deleted the extra README file for the Python SDK.

All of our tests and our example files are working. Ideally would test a bit more, but I think it is ready to ship.

mishushakov · 2025-03-07T21:55:23Z

@mlejva we're ready for release here, but before you should change "desktop" template alias to point to template hnvd1lpiwnmcvzz9hfih

Resolved

mlejva · 2025-03-08T09:43:22Z

@mlejva we're ready for release here, but before you should change "desktop" template alias to point to template hnvd1lpiwnmcvzz9hfih

We can't do that because that's not E2B's (team owner) template. I'll update the desktop template based on the hnvd1lpiwnmcvzz9hfih one.

mlejva

I rebuilt the desktop template and it works with this branch.

✅ ~~Please move back the Python readme (and make sure it's updated to reflect the changes). Without readme poetry install won't work and the published package won't have a readme.~~
✅ ~~Add changeset so the new SDKs get released.~~

mlejva · 2025-03-08T10:53:38Z

README.md

+
+# Start the stream
+desktop.stream.start(
+    port=6080,  # Custom HTTPS port for streaming


When is setting this useful?

The comments should explain when a user would want this

I removed an example of how to change the port from the readmes.
@jamesmurdza @mishushakov correct me if I'm wrong but I don't think changing the port is something a user would do without going deep into customizing the sandbox template and changing the port of the noVNC server itself?

it's okay to leave out

README.md

…by invalidating Docker cache

…d in JS SDK

mlejva

I ended up implementing the changes I requested. This is good to get merged and released.

feat: add ubuntu-desktop template with custom wallpaper and vnc support

c8f0fd7

synacktraa requested a review from mlejva as a code owner February 4, 2025 19:36

synacktraa force-pushed the dev branch from c8caf55 to c8f0fd7 Compare February 5, 2025 21:16

mlejva requested changes Feb 5, 2025

View reviewed changes

mlejva previously requested changes Feb 6, 2025

View reviewed changes

Cleanup dockerfile; create dev template

c17d815

synacktraa and others added 3 commits February 6, 2025 10:37

Merge branch 'e2b-dev:main' into dev

e7c84c2

feat: restore original template directory

0e08053

feat: update env vars in desktop-dev template

33f8d5f

mishushakov requested changes Mar 7, 2025

View reviewed changes

jamesmurdza added 9 commits March 7, 2025 21:01

Throw an error instead of returning null for getCursorPosition() and …

64f77bf

…getScreenSize()

Remove README for Python SDK

95d9072

Remove desktop.hotkey()

37e97c3

Rename takeScreenshot() to screenshot()

0de885a

Make password member private in the TypeScript SDK

b5bb082

Throw an error on stream.start() if the stream is already running

34c9ccb

Always use Sandbox class name instead of Desktop

c5691f3

Move stream parameters to stream.start()

4103ecc

Remove desktop.refresh()

0288e85

mishushakov approved these changes Mar 7, 2025

View reviewed changes

mishushakov added 2 commits March 7, 2025 22:40

added changeset

6f45deb

changed default sandbox template

a2b6e06

mlejva added 2 commits March 8, 2025 11:33

Add apt-update

4ce4632

Remove old streaming web page

787a4ab

mlejva requested changes Mar 8, 2025

View reviewed changes

mlejva added 2 commits March 8, 2025 12:13

Make sure to always pull the latest git repo in the sandbox template …

3e48c2f

…by invalidating Docker cache

Add readmes to each package

71a06d8

mlejva removed the request for review from ben-fornefeld March 9, 2025 16:46

mlejva added 3 commits March 9, 2025 17:51

Explain better how stream auth works

684cde8

Remove example of customizing streaming port from readmes

db172e5

Change positional params to an options object for the write() metho…

7a84ba9

…d in JS SDK

mlejva approved these changes Mar 9, 2025

View reviewed changes

jamesmurdza merged commit 4e580db into e2b-dev:main Mar 9, 2025

feat: add support for VNC using x11vnc and novnc #15

feat: add support for VNC using x11vnc and novnc #15

Conversation

synacktraa commented Feb 4, 2025 • edited Loading

mlejva commented Feb 4, 2025

synacktraa commented Feb 4, 2025

synacktraa commented Feb 4, 2025

mlejva commented Feb 4, 2025

synacktraa commented Feb 4, 2025

synacktraa commented Feb 5, 2025 • edited Loading

mishushakov commented Feb 5, 2025

synacktraa commented Feb 5, 2025 • edited Loading

mlejva commented Feb 5, 2025

synacktraa commented Feb 5, 2025

mlejva left a comment

Choose a reason for hiding this comment

synacktraa commented Feb 5, 2025

mlejva commented Feb 5, 2025 • edited Loading

synacktraa commented Feb 5, 2025 • edited Loading

mlejva Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlejva Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlejva Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlejva Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

synacktraa Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlejva Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

mlejva commented Feb 6, 2025

synacktraa commented Feb 6, 2025

mlejva commented Feb 6, 2025 • edited Loading

synacktraa commented Feb 6, 2025 • edited Loading

synacktraa commented Feb 6, 2025

synacktraa commented Feb 6, 2025 • edited Loading

jamesmurdza commented Mar 7, 2025

mishushakov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamesmurdza commented Mar 7, 2025

mishushakov commented Mar 7, 2025

mlejva commented Mar 8, 2025 • edited Loading

mlejva left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlejva Mar 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlejva left a comment

Choose a reason for hiding this comment

synacktraa commented Feb 4, 2025 •

edited

Loading

synacktraa commented Feb 5, 2025 •

edited

Loading

synacktraa commented Feb 5, 2025 •

edited

Loading

mlejva commented Feb 5, 2025 •

edited

Loading

synacktraa commented Feb 5, 2025 •

edited

Loading

mlejva Feb 5, 2025 •

edited

Loading

mlejva Feb 6, 2025 •

edited

Loading

mlejva Feb 5, 2025 •

edited

Loading

mlejva Feb 6, 2025 •

edited

Loading

synacktraa Feb 6, 2025 •

edited

Loading

mlejva Feb 6, 2025 •

edited

Loading

mlejva commented Feb 6, 2025 •

edited

Loading

synacktraa commented Feb 6, 2025 •

edited

Loading

synacktraa commented Feb 6, 2025 •

edited

Loading

mlejva commented Mar 8, 2025 •

edited

Loading

mlejva left a comment •

edited

Loading

mlejva Mar 9, 2025 •

edited

Loading