feat: add support for VNC using x11vnc and novnc (#15)

* feat: add ubuntu-desktop template with custom wallpaper and vnc support

* Cleanup dockerfile; create dev template

* feat: restore original template directory

* feat: update env vars in desktop-dev template

* feat: add wait_for_port function to verify vnc and novnc status

* feat: full control over startup and vnc server

* feat: refresh desktop and unique password generation

* fix: remove vnc and novnc handles on stop

* feat: add js sdk

* feat: add auto-connect parameter in novnc URL

* fix: resolve extended sandboxOpts issue

* fix: resolve e2b wallpaper not applied issue

* fix: catch command exit error in wait and verify method

* Update README

* feat: update the original template

* Revert changes made to template to add VNC support

* Restore necessary changes to template for VNC support

* Use E2B fork of noVNC without control bar and branding

* Upgrade typedoc package to 0.27.9

* Add type definitions for NodeJS

* Remove autoconnect parameter from examples

* Add the password to the stream URL when auth is enabled

* Don't print the password in the Python example

* Enable stream auth in the TypeScript example

* Rename methods and documentation to use "stream" instead of "VNC"

* Throw an error instead of returning null for getCursorPosition() and getScreenSize()

* Remove README for Python SDK

* Remove desktop.hotkey()

* Rename takeScreenshot() to screenshot()

* Make password member private in the TypeScript SDK

* Throw an error on stream.start() if the stream is already running

* Always use Sandbox class name instead of Desktop

* Move stream parameters to stream.start()

* Remove desktop.refresh()

* added changeset

* changed default sandbox template

* Add apt-update

* Remove old streaming web page

* Make sure to always pull the latest git repo in the sandbox template by invalidating Docker cache

* Add readmes to each package

* Explain better how stream auth works

* Remove example of customizing streaming port from readmes

* Change positional params to an options object for the `write()` method in JS SDK

---------

Co-authored-by: Vasek Mlejnsky <vasek.mlejnsky@gmail.com>
Co-authored-by: James Murdza <james@jamesmurdza.com>
Co-authored-by: Mish Ushakov <mishushakov@users.noreply.github.com>

Author: 4 people

.changeset/lucky-rules-walk.md

@@ -0,0 +1,6 @@
+---
+"@e2b/desktop-python": minor
+"@e2b/desktop": minor
+---
+
+Added VNC support

README.md

@@ -7,78 +7,150 @@ Each sandbox is isolated from the others and can be customized with any dependen
 ![Desktop Sandbox](readme-assets/screenshot.png)
 
 ### Example app using Computer Use with Anthropic's Claude
-Check out the [example open-source app](https://github.com/e2b-dev/secure-computer-use) in a separate repository.
 
+Check out the [example open-source app](https://github.com/e2b-dev/secure-computer-use) in a separate repository.
 
 ## 🚀 Getting started
+
 The E2B Desktop Sandbox is built on top of [E2B Sandbox](https://e2b.dev/docs).
 
 ### 1. Get E2B API key
+
 Sign up at [E2B](https://e2b.dev) and get your API key.
 Set environment variable `E2B_API_KEY` with your API key.
 
 ### 2. Install SDK
+
 **Python**
+
 ```bash
 pip install e2b-desktop
 ```
 
 **JavaScript**
+
 ```bash
 npm install @e2b/desktop
 ```
 
 ### 3. Create Desktop Sandbox
+
 **Python**
+
 ```python
 from e2b_desktop import Sandbox
 
+# Basic initialization
 desktop = Sandbox()
+
+# With custom configuration
+desktop = Sandbox(
+    display=":0",  # Custom display (defaults to :0)
+    resolution=(1920, 1080),  # Custom resolution
+    dpi=96,  # Custom DPI
+)
 ```
 
 **JavaScript**
+
 ```javascript
 import { Sandbox } from '@e2b/desktop'
 
+// Basic initialization
 const desktop = await Sandbox.create()
-```
 
-## Stream virtual desktop screen
-You can enable streaming the desktop screen by passing `videoStream: true` to the `Sandbox.create` function in JavaScript and `video_stream=True` to the `Sandbox` constructor in Python.
+// With custom configuration
+const desktop = await Sandbox.create({
+  display: ':0', // Custom display (defaults to :0)
+  resolution: [1920, 1080], // Custom resolution
+  dpi: 96, // Custom DPI
+})
+```
 
-Then call `getVideoStreamUrl` in JS and `get_video_stream_url` method in Python to get the stream URL that will look like this: `https://e2b.dev/stream/sandbox/<sandbox-id>?token=<secret-token>` and open it in your browser.
+## Features
 
-You'll need to wait a couple of seconds for the stream to buffer the first frames.
+### Streaming desktop's screen
 
 **Python**
+
 ```python
 from e2b_desktop import Sandbox
+desktop = Sandbox()
+
+# Start the stream
+desktop.stream.start()
 
-desktop = Sandbox(video_stream=True)
-stream_url = desktop.get_video_stream_url()
-print(stream_url)
-# Open stream_url in your browser
-# You'll need to wait a couple of seconds for the stream to buffer the first frames
+# Get stream URL
+url = desktop.stream.get_url()
+print(url)
+
+# Stop the stream
+desktop.stream.stop()
 ```
 
 **JavaScript**
+
 ```javascript
 import { Sandbox } from '@e2b/desktop'
 
-const desktop = await Sandbox.create({ videoStream: true, onVideoStreamStart: (streamUrl) => {
-  console.log(streamUrl)
-}})
-// Open streamUrl in your browser
-// You'll need to wait a couple of seconds for the stream to buffer the first frames
+const desktop = await Sandbox.create()
+
+// Start the stream
+await desktop.stream.start()
+
+// Get stream URL
+const url = desktop.stream.getUrl()
+console.log(url)
+
+// Stop the stream
+await desktop.stream.stop()
 ```
 
-![Desktop Sandbox](readme-assets/video-stream.png)
+### Streaming with password protection
 
-## Features
+**Python**
+
+```python
+from e2b_desktop import Sandbox
+desktop = Sandbox()
+
+# Start the stream
+desktop.stream.start(
+    enable_auth=True  # Enable authentication with an auto-generated password that will be injected in the stream URL
+)
+
+# Get stream URL
+url = desktop.stream.get_url()
+print(url)
+
+# Stop the stream
+desktop.stream.stop()
+```
+
+**JavaScript**
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Start the stream
+await desktop.stream.start({
+  enableAuth: true, // Enable authentication with an auto-generated password that will be injected in the stream UR
+})
+
+// Get stream URL
+const url = desktop.stream.getUrl()
+console.log(url)
+
+// Stop the stream
+await desktop.stream.stop()
+```
 
 ### Mouse control
 
 **Python**
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
@@ -92,6 +164,7 @@ desktop.mouse_move(100, 200) # Move to x, y coordinates
 ```
 
 **JavaScript**
+
 ```javascript
 import { Sandbox } from '@e2b/desktop'
 
@@ -108,39 +181,70 @@ await desktop.moveMouse(100, 200) // Move to x, y coordinates
 ### Keyboard control
 
 **Python**
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
 
-desktop.write("Hello, world!") # Write text at the current cursor position
-desktop.hotkey("ctrl", "c") # Press ctrl+c
+# Write text at the current cursor position with customizable typing speed
+desktop.write("Hello, world!")  # Default: chunk_size=25, delay_in_ms=75
+desktop.write("Fast typing!", chunk_size=50, delay_in_ms=25)  # Faster typing
+
+# Press keys
+desktop.press("enter")
+desktop.press("space")
+desktop.press("backspace")
+desktop.press("ctrl+c")
+```
+
+**JavaScript**
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Write text at the current cursor position with customizable typing speed
+await desktop.write('Hello, world!')
+await desktop.write('Fast typing!', { chunkSize: 50, delayInMs: 25 }) // Faster typing
+
+// Press keys
+await desktop.press('enter')
+await desktop.press('space')
+await desktop.press('backspace')
+await desktop.press('ctrl+c') // Copy
 ```
 
 ### Screenshot
+
+**Python**
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
 
 # Take a screenshot and save it as "screenshot.png" locally
-image = desktop.take_screenshot()
+image = desktop.screenshot()
 # Save the image to a file
 with open("screenshot.png", "wb") as f:
     f.write(image)
 ```
 
 **JavaScript**
+
 ```javascript
 import { Sandbox } from '@e2b/desktop'
 
 const desktop = await Sandbox.create()
-const image = await desktop.takeScreenshot()
+const image = await desktop.screenshot()
 // Save the image to a file
-fs.writeFileSync("screenshot.png", image)
+fs.writeFileSync('screenshot.png', image)
 ```
 
 ### Open file
 
 **Python**
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
@@ -151,18 +255,21 @@ desktop.open("/home/user/index.js") # Then open it
 ```
 
 **JavaScript**
+
 ```javascript
 import { Sandbox } from '@e2b/desktop'
 
 const desktop = await Sandbox.create()
 
 // Open file with default application
-await desktop.files.write("/home/user/index.js", "console.log('hello')") // First create the file
-await desktop.open("/home/user/index.js") // Then open it
+await desktop.files.write('/home/user/index.js', "console.log('hello')") // First create the file
+await desktop.open('/home/user/index.js') // Then open it
 ```
 
 ### Run any bash commands
+
 **Python**
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
@@ -173,45 +280,18 @@ print(out)
 ```
 
 **JavaScript**
+
 ```javascript
 import { Sandbox } from '@e2b/desktop'
 
 const desktop = await Sandbox.create()
 
 // Run any bash command
-const out = await desktop.commands.run("ls -la /home/user")
+const out = await desktop.commands.run('ls -la /home/user')
 console.log(out)
 ```
 
-### Run PyAutoGUI commands
-**Python**
-```python
-from e2b_desktop import Sandbox
-desktop = Sandbox()
-
-# Run any PyAutoGUI command
-desktop.pyautogui("pyautogui.click()")
-```
-
-**JavaScript**
-```javascript
-import { Sandbox } from '@e2b/desktop'
-
-const desktop = await Sandbox.create()
-
-// Run any PyAutoGUI command
-await desktop.runPyautoguiCode("pyautogui.click()")
-```
-
-<!-- ### Customization
-```python
-from e2b_desktop import Sandbox
-desktop = Sandbox()
-``` -->
-
 ## Under the hood
-You can use [PyAutoGUI](https://pyautogui.readthedocs.io/en/latest/) to control the whole environment programmatically.
 
 The desktop-like environment is based on Linux and [Xfce](https://www.xfce.org/) at the moment. We chose Xfce because it's a fast and lightweight environment that's also popular and actively supported. However, this Sandbox template is fully customizable and you can create your own desktop environment.
 Check out the sandbox template's code [here](./template/).
-

package-lock.json

@@ -0,0 +1,155 @@
+# E2B Desktop Sandbox - Virtual Computer for Computer Use
+
+E2B Desktop Sandbox is a secure virtual desktop ready for Computer Use. Powered by [E2B](https://e2b.dev).
+
+Each sandbox is isolated from the others and can be customized with any dependencies you want.
+
+![Desktop Sandbox](../../readme-assets/screenshot.png)
+
+### Example app using Computer Use with Anthropic's Claude
+
+Check out the [example open-source app](https://github.com/e2b-dev/secure-computer-use) in a separate repository.
+
+## 🚀 Getting started
+
+The E2B Desktop Sandbox is built on top of [E2B Sandbox](https://e2b.dev/docs).
+
+### 1. Get E2B API key
+
+Sign up at [E2B](https://e2b.dev) and get your API key.
+Set environment variable `E2B_API_KEY` with your API key.
+
+### 2. Install SDK
+
+```bash
+npm install @e2b/desktop
+```
+
+### 3. Create Desktop Sandbox
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+// Basic initialization
+const desktop = await Sandbox.create()
+
+// With custom configuration
+const desktop = await Sandbox.create({
+  display: ':0', // Custom display (defaults to :0)
+  resolution: [1920, 1080], // Custom resolution
+  dpi: 96 // Custom DPI
+})
+```
+
+## Features
+
+### Streaming desktop's screen
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Start the stream
+await desktop.stream.start()
+
+// Get stream URL
+const url = desktop.stream.getUrl()
+console.log(url)
+
+// Stop the stream
+await desktop.stream.stop()
+```
+
+### Streaming with password protection
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Start the stream
+await desktop.stream.start({
+  enableAuth: true, // Enable authentication with an auto-generated password that will be injected in the stream UR
+})
+
+// Get stream URL
+const url = desktop.stream.getUrl()
+console.log(url)
+
+// Stop the stream
+await desktop.stream.stop()
+```
+
+### Mouse control
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+await desktop.doubleClick()
+await desktop.leftClick()
+await desktop.rightClick()
+await desktop.middleClick()
+await desktop.scroll(10) // Scroll by the amount. Positive for up, negative for down.
+await desktop.moveMouse(100, 200) // Move to x, y coordinates
+```
+
+### Keyboard control
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Write text at the current cursor position with customizable typing speed
+await desktop.write('Hello, world!')
+await desktop.write('Fast typing!', { chunkSize: 50, delayInMs: 25 }) // Faster typing
+
+// Press keys
+await desktop.press('enter')
+await desktop.press('space')
+await desktop.press('backspace')
+await desktop.press('ctrl+c') // Copy
+```
+
+### Screenshot
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+const image = await desktop.screenshot()
+// Save the image to a file
+fs.writeFileSync('screenshot.png', image)
+```
+
+### Open file
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Open file with default application
+await desktop.files.write('/home/user/index.js', "console.log('hello')") // First create the file
+await desktop.open('/home/user/index.js') // Then open it
+```
+
+### Run any bash commands
+
+```javascript
+import { Sandbox } from '@e2b/desktop'
+
+const desktop = await Sandbox.create()
+
+// Run any bash command
+const out = await desktop.commands.run('ls -la /home/user')
+console.log(out)
+```
+
+## Under the hood
+
+The desktop-like environment is based on Linux and [Xfce](https://www.xfce.org/) at the moment. We chose Xfce because it's a fast and lightweight environment that's also popular and actively supported. However, this Sandbox template is fully customizable and you can create your own desktop environment.
+Check out the sandbox template's code [here](./template/).

packages/js-sdk/README.md

@@ -2,50 +2,46 @@ import { config } from 'dotenv'
 
 config()
 import { Sandbox } from './dist'
+import { writeFileSync } from 'fs';
 
 
-const sbx = await Sandbox.create({
-  videoStream: true,
-  onVideoStreamStart: (url) => console.log('Video stream started:', url)
-})
+console.log("Starting desktop sandbox...")
 
-// const command = "ffmpeg -video_size 1024x768 -f x11grab -i :99 -c:v libx264 -c:a aac  -g 50 -b:v 4000k -maxrate 4000k -bufsize 8000k -f flv rtmp://global-live.mux.com:5222/app/stream-key"
-// const ffmpeg = await sbx.commands.run(command, { background: true, onStdout: (data) => console.log(data.toString()) })
+console.time('--> Sandbox creation time');
+const desktop = await Sandbox.create();
+console.timeEnd('--> Sandbox creation time');
 
-for (let i = 0; i < 30; i++) {
-  const x = Math.floor(Math.random() * 1024);
-  const y = Math.floor(Math.random() * 768);
-  await sbx.moveMouse(x, y);
-  await new Promise(resolve => setTimeout(resolve, 2000));
-  await sbx.rightClick();
-  console.log('right clicked', i)
-}
+console.log("Desktop Sandbox started, ID:", desktop.sandboxId)
+console.log("Screen size:", await desktop.getScreenSize())
+
+await desktop.stream.start({
+  enableAuth: true
+})
+
+console.log("Stream URL:", desktop.stream.getUrl())
 
+await new Promise(resolve => setTimeout(resolve, 5000));
 
-// await sbx.kill()
+console.log("Moving mouse to 'Applications' and clicking...")
+await desktop.moveMouse(100, 100)
+await desktop.leftClick()
+console.log("Cursor position:", await desktop.getCursorPosition())
 
+await new Promise(resolve => setTimeout(resolve, 1000));
 
-// // await sbx.rightClick()
-// let imageData = await sbx.takeScreenshot()
-// await processImage(imageData)
-// // const pos = await sbx.locateTextOnScreen('Applications')
-// // if (!pos) throw new Error('Text not found on screen')
-// await sbx.moveMouse(384 + 150, 1024 - 80)
-// // await new Promise(resolve => setTimeout(resolve, 5000));
-// await sbx.doubleClick()
-// // await sbx.leftClick()
-// console.log('clicked')
-// await new Promise(resolve => setTimeout(resolve, 2000));
-// console.log('screenshot')
-// imageData = await sbx.takeScreenshot()
-// await processImage(imageData)
-// await sbx.kill()
+const screenshot = await desktop.screenshot("bytes")
+writeFileSync('1.png', Buffer.from(screenshot));
 
 
+for (let i = 0; i < 20; i++) {
+  const x = Math.floor(Math.random() * 1024);
+  const y = Math.floor(Math.random() * 768);
+  await desktop.moveMouse(x, y);
+  await new Promise(resolve => setTimeout(resolve, 2000));
+  await desktop.rightClick();
+  console.log('right clicked', i)
+}
 
 
-// {
-//   text: 'File System',
-//   confidence: 95.43375396728516,
-//   bbox: { x0: 33, y0: 228, x1: 107, y1: 241 }
-// }
+await desktop.stream.stop()
+await desktop.kill()

packages/js-sdk/example.mts

@@ -51,11 +51,12 @@
     "generate-ref": "./scripts/generate_sdk_ref.sh"
   },
   "devDependencies": {
+    "@types/node": "^22.13.9",
     "dotenv": "^16.4.5",
-    "typedoc": "^0.26.8",
-    "typedoc-plugin-markdown": "^4.2.7",
     "tsup": "^8.3.5",
     "tsx": "^4.19.2",
+    "typedoc": "^0.27.9",
+    "typedoc-plugin-markdown": "^4.2.7",
     "typescript": "^5.6.3",
     "vitest": "^3.0.5"
   },

packages/js-sdk/package-lock.json

@@ -1,13 +1,13 @@
-/**
- * Generate a random ID with 16 characters.
- * @returns A random ID.
- */
-export function generateRandomId() {
-  const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
-  let result = '';
-  for (let i = 0; i < 16; i++) {
-    const randomIndex = Math.floor(Math.random() * characters.length);
-    result += characters[randomIndex];
-  }
-  return result;
-}
+import { randomBytes } from 'crypto';
+
+export function generateRandomString(length: number = 16): string {
+    const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
+    const bytes = randomBytes(length);
+    let result = '';
+
+    for (let i = 0; i < length; i++) {
+        result += characters[bytes[i] % characters.length];
+    }
+
+    return result;
+}

packages/js-sdk/package.json

@@ -7,6 +7,6 @@ sandboxTest('get screen size', async ({ sandbox }) => {
 })
 
 sandboxTest('take screenshot', async ({ sandbox }) => {
-  const screenshot = await sandbox.takeScreenshot()
+  const screenshot = await sandbox.screenshot()
   expect(screenshot.length).toBeGreaterThan(0)
 })

packages/js-sdk/src/sandbox.ts

@@ -1,46 +1,86 @@
-# E2B Desktop Sandbox (beta)
+# E2B Desktop Sandbox - Virtual Computer for Computer Use
 
-E2B Desktop Sandbox is an isolated cloud environment with a desktop-like interface powered by [E2B](https://e2b.dev).
+E2B Desktop Sandbox is a secure virtual desktop ready for Computer Use. Powered by [E2B](https://e2b.dev).
 
-Launching E2B Sandbox takes about 300-500ms. You can customize the desktop environment and preinstall any dependencies you want using our [custom sandbox templates](https://e2b.dev/docs/sandbox-template).
+Each sandbox is isolated from the others and can be customized with any dependencies you want.
 
-![Desktop Sandbox](screenshot.png)
+![Desktop Sandbox](../../readme-assets/screenshot.png)
 
-**Work in progress**
-This repository is a work in progress. We welcome feedback and contributions. Here's the list of features we're working on:
-- [ ] JavaScript SDK
-- [ ] Streaming live desktop
-- [ ] Tests
-- [ ] Docstrings
+### Example app using Computer Use with Anthropic's Claude
+
+Check out the [example open-source app](https://github.com/e2b-dev/secure-computer-use) in a separate repository.
+
+## 🚀 Getting started
 
-## Getting started
 The E2B Desktop Sandbox is built on top of [E2B Sandbox](https://e2b.dev/docs).
 
 ### 1. Get E2B API key
+
 Sign up at [E2B](https://e2b.dev) and get your API key.
 Set environment variable `E2B_API_KEY` with your API key.
 
 ### 2. Install SDK
-**Python**
-```bash
-pip install e2b-desktop
-```
 
-**JavaScript**
 ```bash
-Coming soon
+pip install e2b-desktop
 ```
 
 ### 3. Create Desktop Sandbox
+
 ```python
 from e2b_desktop import Sandbox
 
+# Basic initialization
 desktop = Sandbox()
+
+# With custom configuration
+desktop = Sandbox(
+    display=":0",  # Custom display (defaults to :0)
+    resolution=(1920, 1080),  # Custom resolution
+    dpi=96,  # Custom DPI
+)
 ```
 
 ## Features
 
+### Streaming desktop's screen
+
+```python
+from e2b_desktop import Sandbox
+desktop = Sandbox()
+
+# Start the stream
+desktop.stream.start()
+
+# Get stream URL
+url = desktop.stream.get_url()
+print(url)
+
+# Stop the stream
+desktop.stream.stop()
+```
+
+### Streaming with password protection
+
+```python
+from e2b_desktop import Sandbox
+desktop = Sandbox()
+
+# Start the stream
+desktop.stream.start(
+    enable_auth=True  # Enable authentication with an auto-generated password that will be injected in the stream URL
+)
+
+# Get stream URL
+url = desktop.stream.get_url()
+print(url)
+
+# Stop the stream
+desktop.stream.stop()
+```
+
 ### Mouse control
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
@@ -53,36 +93,38 @@ desktop.scroll(10) # Scroll by the amount. Positive for up, negative for down.
 desktop.mouse_move(100, 200) # Move to x, y coordinates
 ```
 
-### Locate on screen
-```python
-from e2b_desktop import Sandbox
-desktop = Sandbox()
-
-# Find "Home" text on the screen and return the coordinates
-x, y = desktop.locate_on_screen("Home")
-# Move the mouse to the coordinates
-desktop.mouse_move(x, y)
-```
-
 ### Keyboard control
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
 
-desktop.write("Hello, world!") # Write text at the current cursor position
-desktop.hotkey("ctrl", "c") # Press ctrl+c
+# Write text at the current cursor position with customizable typing speed
+desktop.write("Hello, world!")  # Default: chunk_size=25, delay_in_ms=75
+desktop.write("Fast typing!", chunk_size=50, delay_in_ms=25)  # Faster typing
+
+# Press keys
+desktop.press("enter")
+desktop.press("space")
+desktop.press("backspace")
+desktop.press("ctrl+c")
 ```
 
 ### Screenshot
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
 
 # Take a screenshot and save it as "screenshot.png" locally
-desktop.screenshot("screenshot.png")
+image = desktop.screenshot()
+# Save the image to a file
+with open("screenshot.png", "wb") as f:
+    f.write(image)
 ```
 
 ### Open file
+
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
@@ -93,31 +135,17 @@ desktop.open("/home/user/index.js") # Then open it
 ```
 
 ### Run any bash commands
-```python
-from e2b_desktop import Sandbox
-desktop = Sandbox()
-
-# Run any bash command
-desktop.commands.run("ls -la /home/user")
-```
 
-### Run PyAutoGUI commands
 ```python
 from e2b_desktop import Sandbox
 desktop = Sandbox()
 
-# Run any PyAutoGUI command
-desktop.pyautogui("pyautogui.click()")
+# Run any bash command
+out = desktop.commands.run("ls -la /home/user")
+print(out)
 ```
 
-<!-- ### Customization
-```python
-from e2b_desktop import Sandbox
-desktop = Sandbox()
-``` -->
-
 ## Under the hood
-You can use [PyAutoGUI](https://pyautogui.readthedocs.io/en/latest/) to control the whole environment programmatically.
 
 The desktop-like environment is based on Linux and [Xfce](https://www.xfce.org/) at the moment. We chose Xfce because it's a fast and lightweight environment that's also popular and actively supported. However, this Sandbox template is fully customizable and you can create your own desktop environment.
-Check out the code [here](./template/)
+Check out the sandbox template's code [here](./template/).

packages/js-sdk/src/utils.ts

@@ -1,3 +1,3 @@
 from e2b import *
 
-from .main import Sandbox
+from .main import Sandbox

packages/js-sdk/tests/screen.test.ts

@@ -7,27 +7,29 @@
 load_dotenv()
 
 print("Starting desktop sandbox...")
-desktop = Sandbox(
-    video_stream=True,
-)
-stream_url = desktop.get_video_stream_url()
-print("Video stream URL:", stream_url)
+desktop = Sandbox()
+print("Screen size:", desktop.get_screen_size())
+
+desktop.stream.start(enable_auth=True)
+
+print("Stream URL:", desktop.stream.get_url())
+
+input("Press enter to continue...")
+
 print("Desktop Sandbox started, ID:", desktop.sandbox_id)
 
-screenshot = desktop.take_screenshot(format="bytes")
+screenshot = desktop.screenshot(format="bytes")
+
 with open("1.png", "wb") as f:
     f.write(screenshot)
 
 print("Moving mouse to 'Applications' and clicking...")
 desktop.move_mouse(100, 100)
 desktop.left_click()
-desktop.commands.run(
-    cmd="code /home/user",
-    background=True,
-)
+print("Cursor position:", desktop.get_cursor_position())
 
 time.sleep(1)
-screenshot = desktop.take_screenshot(format="bytes")
+screenshot = desktop.screenshot(format="bytes")
 with open("2.png", "wb") as f:
     f.write(screenshot)
 
@@ -40,4 +42,5 @@
     print("Right clicked", i)
     time.sleep(2)
 
+desktop.stream.stop()
 desktop.kill()

packages/python-sdk/README.md

@@ -11,4 +11,4 @@
     "pretest": "poetry install",
     "generate-ref": "poetry install && ./scripts/generate_sdk_ref.sh"
   }
-}
+}

packages/python-sdk/e2b_desktop/__init__.py

@@ -17,7 +17,7 @@ def images_are_equal(img1, img2):
 
 def test_right_click(sandbox: Sandbox):
     # Capture the initial screenshot
-    initial_screenshot_bytes = sandbox.take_screenshot()
+    initial_screenshot_bytes = sandbox.screenshot()
     initial_image = Image.open(io.BytesIO(initial_screenshot_bytes))
 
     # Get cursor position and perform right click
@@ -26,7 +26,7 @@ def test_right_click(sandbox: Sandbox):
     time.sleep(5)  # Wait for UI to respond
 
     # Capture and process the second screenshot
-    post_click_screenshot_bytes = sandbox.take_screenshot()
+    post_click_screenshot_bytes = sandbox.screenshot()
     post_click_image = Image.open(io.BytesIO(post_click_screenshot_bytes))
 
     # Crop both images around the cursor position
@@ -37,7 +37,7 @@ def test_right_click(sandbox: Sandbox):
     assert not images_are_equal(cropped_image_1, cropped_image_2), "The image around the cursor did not change after right-click."
 
 def test_screenshot(sandbox: Sandbox):
-    image = sandbox.take_screenshot()
+    image = sandbox.screenshot()
     assert image, "Screenshot was not taken successfully"
 
     # Check if the image is a valid image format

packages/python-sdk/e2b_desktop/main.py

@@ -4,49 +4,47 @@ ENV DEBIAN_FRONTEND=noninteractive
 ENV DEBIAN_PRIORITY=high
 
 RUN yes | unminimize
-RUN apt-get --reinstall install -y python3-jwt python3-oauthlib python3-lazr.restfulclient \
+RUN apt-get update && apt-get --reinstall install -y python3-jwt python3-oauthlib python3-lazr.restfulclient \
     python3-launchpadlib python3-apport xserver-xorg apport xorg
 
 RUN apt-get update && apt-get install -y \
-  python3-xlib \
-  x11-xserver-utils \
-  xfce4 \
-  xfce4-goodies \
-  xvfb \
-  xubuntu-icon-theme \
-  scrot \
-  python3-pip \
-  python3-tk \
-  python3-dev \
-  x11-utils \
-  gnumeric \
-  python-is-python3 \
-  build-essential \
-  util-linux \
-  locales \
-  xauth \
-  gnome-screenshot \
-  xserver-xorg \
-  ffmpeg \
-  vim \
-  xorg
+    python3-xlib \
+    x11-xserver-utils \
+    xfce4 \
+    xfce4-goodies \
+    xvfb \
+    xubuntu-icon-theme \
+    scrot \
+    python3-pip \
+    python3-tk \
+    python3-dev \
+    x11-utils \
+    gnumeric \
+    python-is-python3 \
+    build-essential \
+    util-linux \
+    locales \
+    xauth \
+    gnome-screenshot \
+    xserver-xorg \
+    ffmpeg \
+    vim \
+    xorg
 
 RUN pip3 install mux_python requests
 
 # Install vscode
 RUN apt update -y \
-  && apt install -y software-properties-common apt-transport-https wget \
-  && wget -qO- https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
-  && add-apt-repository -y "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" \
-  && apt update -y \
-  && apt install -y code
+    && apt install -y software-properties-common apt-transport-https wget \
+    && wget -qO- https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
+    && add-apt-repository -y "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" \
+    && apt update -y \
+    && apt install -y code
 
 ENV PIP_DEFAULT_TIMEOUT=100 \
-  PIP_DISABLE_PIP_VERSION_CHECK=1 \
-  PIP_NO_CACHE_DIR=1 \
-  DEBIAN_FRONTEND=noninteractive
-
-RUN echo "export DISPLAY=:99" >> /etc/environment
+    PIP_DISABLE_PIP_VERSION_CHECK=1 \
+    PIP_NO_CACHE_DIR=1 \
+    DEBIAN_FRONTEND=noninteractive
 
 COPY ./requirements.txt requirements.txt
 RUN pip3 install --no-cache-dir -r requirements.txt
@@ -55,54 +53,66 @@ COPY ./45-allow-colord.pkla /etc/polkit-1/localauthority/50-local.d/45-allow-col
 
 COPY ./Xauthority /home/user/.Xauthority
 
-COPY ./start-up.sh /
-RUN chmod +x /start-up.sh
-
 RUN apt-get update && \
-  apt-get -y upgrade && \
-  apt-get -y install \
-  build-essential \
-  # UI Requirements
-  xvfb \
-  xterm \
-  xdotool \
-  scrot \
-  imagemagick \
-  sudo \
-  mutter \
-  x11vnc \
-  # Python/pyenv reqs
-  build-essential \
-  libssl-dev  \
-  zlib1g-dev \
-  libbz2-dev \
-  libreadline-dev \
-  libsqlite3-dev \
-  curl \
-  git \
-  libncursesw5-dev \
-  xz-utils \
-  tk-dev \
-  libxml2-dev \
-  libxmlsec1-dev \
-  libffi-dev \
-  liblzma-dev \
-  # Network tools
-  net-tools \
-  netcat \
-  # PPA req
-  software-properties-common && \
-  # Userland apps
-  sudo add-apt-repository ppa:mozillateam/ppa && \
-  sudo apt-get install -y --no-install-recommends \
-  libreoffice \
-  firefox-esr \
-  x11-apps \
-  xpdf \
-  gedit \
-  xpaint \
-  tint2 \
-  galculator \
-  pcmanfm \
-  unzip && \
-  apt-get clean
+    apt-get -y upgrade && \
+    apt-get -y install \
+    build-essential \
+    # UI Requirements
+    xvfb \
+    xterm \
+    xdotool \
+    scrot \
+    imagemagick \
+    sudo \
+    mutter \
+    x11vnc \
+    # Python/pyenv reqs
+    build-essential \
+    libssl-dev  \
+    zlib1g-dev \
+    libbz2-dev \
+    libreadline-dev \
+    libsqlite3-dev \
+    curl \
+    git \
+    libncursesw5-dev \
+    xz-utils \
+    tk-dev \
+    libxml2-dev \
+    libxmlsec1-dev \
+    libffi-dev \
+    liblzma-dev \
+    # Network tools
+    net-tools \
+    netcat \
+    # PPA req
+    software-properties-common && \
+    # Userland apps
+    sudo add-apt-repository ppa:mozillateam/ppa && \
+    sudo apt-get install -y --no-install-recommends \
+    libreoffice \
+    firefox-esr \
+    x11-apps \
+    xpdf \
+    gedit \
+    xpaint \
+    tint2 \
+    galculator \
+    pcmanfm \
+    unzip && \
+    apt-get clean
+
+# Install numpy which is used by websockify: https://github.com/novnc/websockify/issues/337
+RUN pip install numpy
+
+# Install noVNC and websockify
+#
+# Invalidate cache so we always pull the latest noVNC and websockify
+# otherwise, Docker might not execute the `RUN` command if it's cached from the previous build
+ARG CACHEBUST=1
+RUN git clone --branch e2b-desktop https://github.com/e2b-dev/noVNC.git /opt/noVNC && \
+    git clone --branch v0.12.0 https://github.com/novnc/websockify /opt/noVNC/utils/websockify && \
+    ln -s /opt/noVNC/vnc.html /opt/noVNC/index.html
+
+# Copy E2B desktop wallpaper
+COPY ./wallpaper.png /usr/share/backgrounds/xfce/wallpaper.png

packages/python-sdk/example.py

@@ -10,10 +10,9 @@
 # import { Sandbox } from 'e2b'
 # const sandbox = await Sandbox.create('desktop')
 
-template_id = "k0wmnzir0zuzye6dndlw"
+team_id = "460355b3-4f64-48f9-9a16-4442817f79f5"
+memory_mb = 4_096
+cpu_count = 4
 dockerfile = "e2b.Dockerfile"
 template_name = "desktop"
-start_cmd = "/start-up.sh"
-cpu_count = 4
-memory_mb = 4_096
-team_id = "460355b3-4f64-48f9-9a16-4442817f79f5"
+template_id = "k0wmnzir0zuzye6dndlw"