Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for VNC using x11vnc and novnc #15

Merged
merged 47 commits into from
Mar 9, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
c8f0fd7
feat: add ubuntu-desktop template with custom wallpaper and vnc support
synacktraa Feb 4, 2025
c17d815
Cleanup dockerfile; create dev template
mlejva Feb 6, 2025
e7c84c2
Merge branch 'e2b-dev:main' into dev
synacktraa Feb 6, 2025
0e08053
feat: restore original template directory
synacktraa Feb 6, 2025
33f8d5f
feat: update env vars in desktop-dev template
synacktraa Feb 6, 2025
ff920e7
feat: add wait_for_port function to verify vnc and novnc status
synacktraa Feb 6, 2025
07b8945
feat: full control over startup and vnc server
synacktraa Feb 7, 2025
93ed2f2
feat: refresh desktop and unique password generation
synacktraa Feb 7, 2025
4efffa6
fix: remove vnc and novnc handles on stop
synacktraa Feb 8, 2025
47505a8
feat: add js sdk
synacktraa Feb 8, 2025
5c299f9
feat: add auto-connect parameter in novnc URL
synacktraa Feb 8, 2025
3524787
fix: resolve extended sandboxOpts issue
synacktraa Feb 8, 2025
30a75a5
Merge branch 'js-sdk' into dev
synacktraa Feb 8, 2025
c53fdec
fix: resolve e2b wallpaper not applied issue
synacktraa Feb 10, 2025
4dcae11
fix: catch command exit error in wait and verify method
synacktraa Feb 12, 2025
dfba7f8
Update README
jamesmurdza Feb 12, 2025
0fb4544
feat: update the original template
synacktraa Feb 28, 2025
280609c
Merge branch 'dev' of github.com:synacktraa/desktop into dev
synacktraa Feb 28, 2025
5bdf00a
Merge branch 'e2b-dev:main' into dev
synacktraa Feb 28, 2025
d9db262
Revert changes made to template to add VNC support
jamesmurdza Mar 7, 2025
483b807
Restore necessary changes to template for VNC support
jamesmurdza Mar 7, 2025
eea1159
Use E2B fork of noVNC without control bar and branding
jamesmurdza Mar 7, 2025
070c494
Upgrade typedoc package to 0.27.9
jamesmurdza Mar 7, 2025
7acd091
Add type definitions for NodeJS
jamesmurdza Mar 7, 2025
869399a
Remove autoconnect parameter from examples
jamesmurdza Mar 7, 2025
21c63dd
Add the password to the stream URL when auth is enabled
jamesmurdza Mar 7, 2025
d533660
Don't print the password in the Python example
jamesmurdza Mar 7, 2025
e4ac029
Enable stream auth in the TypeScript example
jamesmurdza Mar 7, 2025
89c5795
Rename methods and documentation to use "stream" instead of "VNC"
jamesmurdza Mar 7, 2025
64f77bf
Throw an error instead of returning null for getCursorPosition() and …
jamesmurdza Mar 7, 2025
95d9072
Remove README for Python SDK
jamesmurdza Mar 7, 2025
37e97c3
Remove desktop.hotkey()
jamesmurdza Mar 7, 2025
0de885a
Rename takeScreenshot() to screenshot()
jamesmurdza Mar 7, 2025
b5bb082
Make password member private in the TypeScript SDK
jamesmurdza Mar 7, 2025
34c9ccb
Throw an error on stream.start() if the stream is already running
jamesmurdza Mar 7, 2025
c5691f3
Always use Sandbox class name instead of Desktop
jamesmurdza Mar 7, 2025
4103ecc
Move stream parameters to stream.start()
jamesmurdza Mar 7, 2025
0288e85
Remove desktop.refresh()
jamesmurdza Mar 7, 2025
6f45deb
added changeset
mishushakov Mar 7, 2025
a2b6e06
changed default sandbox template
mishushakov Mar 7, 2025
4ce4632
Add apt-update
mlejva Mar 8, 2025
787a4ab
Remove old streaming web page
mlejva Mar 8, 2025
3e48c2f
Make sure to always pull the latest git repo in the sandbox template …
mlejva Mar 8, 2025
71a06d8
Add readmes to each package
mlejva Mar 9, 2025
684cde8
Explain better how stream auth works
mlejva Mar 9, 2025
db172e5
Remove example of customizing streaming port from readmes
mlejva Mar 9, 2025
7a84ba9
Change positional params to an options object for the `write()` metho…
mlejva Mar 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/lucky-rules-walk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@e2b/desktop-python": minor
"@e2b/desktop": minor
---

Added VNC support
188 changes: 134 additions & 54 deletions README.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are the examples up-to-date?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Usage: #15 (comment)

Original file line number Diff line number Diff line change
Expand Up @@ -7,78 +7,150 @@ Each sandbox is isolated from the others and can be customized with any dependen
![Desktop Sandbox](readme-assets/screenshot.png)

### Example app using Computer Use with Anthropic's Claude
Check out the [example open-source app](https://github.com/e2b-dev/secure-computer-use) in a separate repository.

Check out the [example open-source app](https://github.com/e2b-dev/secure-computer-use) in a separate repository.

## 🚀 Getting started

The E2B Desktop Sandbox is built on top of [E2B Sandbox](https://e2b.dev/docs).

### 1. Get E2B API key

Sign up at [E2B](https://e2b.dev) and get your API key.
Set environment variable `E2B_API_KEY` with your API key.

### 2. Install SDK

**Python**

```bash
pip install e2b-desktop
```

**JavaScript**

```bash
npm install @e2b/desktop
```

### 3. Create Desktop Sandbox

**Python**

```python
from e2b_desktop import Sandbox

# Basic initialization
desktop = Sandbox()

# With custom configuration
desktop = Sandbox(
display=":0", # Custom display (defaults to :0)
resolution=(1920, 1080), # Custom resolution
dpi=96, # Custom DPI
)
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

// Basic initialization
const desktop = await Sandbox.create()
```

## Stream virtual desktop screen
You can enable streaming the desktop screen by passing `videoStream: true` to the `Sandbox.create` function in JavaScript and `video_stream=True` to the `Sandbox` constructor in Python.
// With custom configuration
const desktop = await Sandbox.create({
display: ':0', // Custom display (defaults to :0)
resolution: [1920, 1080], // Custom resolution
dpi: 96, // Custom DPI
})
```

Then call `getVideoStreamUrl` in JS and `get_video_stream_url` method in Python to get the stream URL that will look like this: `https://e2b.dev/stream/sandbox/<sandbox-id>?token=<secret-token>` and open it in your browser.
## Features

You'll need to wait a couple of seconds for the stream to buffer the first frames.
### Streaming desktop's screen

**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()

# Start the stream
desktop.stream.start()

desktop = Sandbox(video_stream=True)
stream_url = desktop.get_video_stream_url()
print(stream_url)
# Open stream_url in your browser
# You'll need to wait a couple of seconds for the stream to buffer the first frames
# Get stream URL
url = desktop.stream.get_url()
print(url)

# Stop the stream
desktop.stream.stop()
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create({ videoStream: true, onVideoStreamStart: (streamUrl) => {
console.log(streamUrl)
}})
// Open streamUrl in your browser
// You'll need to wait a couple of seconds for the stream to buffer the first frames
const desktop = await Sandbox.create()

// Start the stream
await desktop.stream.start()

// Get stream URL
const url = desktop.stream.getUrl()
console.log(url)

// Stop the stream
await desktop.stream.stop()
```

![Desktop Sandbox](readme-assets/video-stream.png)
### Streaming with password protection

## Features
**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()

# Start the stream
desktop.stream.start(
enable_auth=True # Enable authentication with an auto-generated password that will be injected in the stream URL
)

# Get stream URL
url = desktop.stream.get_url()
print(url)

# Stop the stream
desktop.stream.stop()
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create()

// Start the stream
await desktop.stream.start({
enableAuth: true, // Enable authentication with an auto-generated password that will be injected in the stream UR
})

// Get stream URL
const url = desktop.stream.getUrl()
console.log(url)

// Stop the stream
await desktop.stream.stop()
```

### Mouse control

**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()
Expand All @@ -92,6 +164,7 @@ desktop.mouse_move(100, 200) # Move to x, y coordinates
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

Expand All @@ -108,39 +181,70 @@ await desktop.moveMouse(100, 200) // Move to x, y coordinates
### Keyboard control

**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()

desktop.write("Hello, world!") # Write text at the current cursor position
desktop.hotkey("ctrl", "c") # Press ctrl+c
# Write text at the current cursor position with customizable typing speed
desktop.write("Hello, world!") # Default: chunk_size=25, delay_in_ms=75
desktop.write("Fast typing!", chunk_size=50, delay_in_ms=25) # Faster typing

# Press keys
desktop.press("enter")
desktop.press("space")
desktop.press("backspace")
desktop.press("ctrl+c")
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create()

// Write text at the current cursor position with customizable typing speed
await desktop.write('Hello, world!')
await desktop.write('Fast typing!', { chunkSize: 50, delayInMs: 25 }) // Faster typing

// Press keys
await desktop.press('enter')
await desktop.press('space')
await desktop.press('backspace')
await desktop.press('ctrl+c') // Copy
```

### Screenshot

**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()

# Take a screenshot and save it as "screenshot.png" locally
image = desktop.take_screenshot()
image = desktop.screenshot()
# Save the image to a file
with open("screenshot.png", "wb") as f:
f.write(image)
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create()
const image = await desktop.takeScreenshot()
const image = await desktop.screenshot()
// Save the image to a file
fs.writeFileSync("screenshot.png", image)
fs.writeFileSync('screenshot.png', image)
```

### Open file

**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()
Expand All @@ -151,18 +255,21 @@ desktop.open("/home/user/index.js") # Then open it
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create()

// Open file with default application
await desktop.files.write("/home/user/index.js", "console.log('hello')") // First create the file
await desktop.open("/home/user/index.js") // Then open it
await desktop.files.write('/home/user/index.js', "console.log('hello')") // First create the file
await desktop.open('/home/user/index.js') // Then open it
```

### Run any bash commands

**Python**

```python
from e2b_desktop import Sandbox
desktop = Sandbox()
Expand All @@ -173,45 +280,18 @@ print(out)
```

**JavaScript**

```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create()

// Run any bash command
const out = await desktop.commands.run("ls -la /home/user")
const out = await desktop.commands.run('ls -la /home/user')
console.log(out)
```

### Run PyAutoGUI commands
**Python**
```python
from e2b_desktop import Sandbox
desktop = Sandbox()

# Run any PyAutoGUI command
desktop.pyautogui("pyautogui.click()")
```

**JavaScript**
```javascript
import { Sandbox } from '@e2b/desktop'

const desktop = await Sandbox.create()

// Run any PyAutoGUI command
await desktop.runPyautoguiCode("pyautogui.click()")
```

<!-- ### Customization
```python
from e2b_desktop import Sandbox
desktop = Sandbox()
``` -->

## Under the hood
You can use [PyAutoGUI](https://pyautogui.readthedocs.io/en/latest/) to control the whole environment programmatically.

The desktop-like environment is based on Linux and [Xfce](https://www.xfce.org/) at the moment. We chose Xfce because it's a fast and lightweight environment that's also popular and actively supported. However, this Sandbox template is fully customizable and you can create your own desktop environment.
Check out the sandbox template's code [here](./template/).

Loading