Create sandbox event endpoint and handlers #784

0div · 2025-06-16T21:14:05Z

Description

We currently do not have a way to send events from inside a sandbox and programmatically handle them from the host, this is an attempt to solve that.

Setup networking in sandbox that routes http://events.e2b.dev requests to a server listening in orchestrator with customizable handlers.

add internal routing (/etc/hosts) in sandbox
add ip table rule to route sbx event requests to event server
parametrize event handler registering for event server
appends events to redis sorted set
abstract event store interface with getters and setters
track sandbox IPs via store on sandbox resume
use it to map back to sandbox ID in event handler
wire to ClickHouse
build API routes to fetch events

Test

[infra]
$ make build-and-upload/envd
$ make build-and-upload/template-manager
$ make build-and-upload/orchestrator
$ make plan
$ make apply 

[E2B/template/base]
$ e2b template build

[Sandbox]
$ e2b sbx sp base
user@e2b:~$ curl -X POST http://events.e2b.dev/test -d '{"your": "data"}'

{"event_ack": true, "path": "/test"}

user@e2b:~$ curl -X GET http://events.e2b.dev

[{"path":"/test","body":{"your": "data"}}]

linear · 2025-06-16T21:14:08Z

E2B-2486 Create internal sandbox event endpoint

…er validation and register some event handlers

…nternal-sandbox-event-endpoint-e2b-2486

packages/orchestrator/internal/server/main.go

packages/orchestrator/cmd/build-template/main.go

packages/orchestrator/main.go

packages/orchestrator/internal/template/build/rootfs.go

packages/orchestrator/internal/sandbox/network/network_linux.go

packages/orchestrator/internal/sandbox/event/server.go

dobrac · 2025-06-24T18:00:10Z

packages/orchestrator/internal/sandbox/network/network_linux.go

 	}

+	// Redirect http://event.e2b.dev traffic destined to event server
+	err = tables.Append("nat", "PREROUTING", "-i", s.VethName(), "-p", "tcp", "-d", "8.8.8.7", "--dport", "80", "-j", "REDIRECT", "--to-port", "5010")


use different IP than 8.8.8.7, I guess any IP from these should be more appropriate:

var blockedRanges = []string{ "10.0.0.0/8", "169.254.0.0/16", "192.168.0.0/16", "172.16.0.0/12", }

ideally yes, i first tried private IPs, they don't leave the sandbox through the network bridge with current setup, i'd have to figure out a way to route one of the private ones this way.

just note, these ranges are blocked by default, you need to enable the target IP address (maybe that might be why you haven't seen them routed)

Can we maybe not use public IP, but something like metadata IP used by envd for accessing Firecracker VM metadata?

More details on why Firecracker and cloud providers are using 169.254/16 https://datatracker.ietf.org/doc/html/rfc3927

Perhaps provide the domain as an environment variable via envd so we are not locked into it for the future? Then does not matter what domain we will use now

It does matter in a sense where it will be encoded in all envds

It does matter in a sense where it will be encoded in all envds

That was my point with Perhaps provide the domain as an environment variable via envd so we are not locked into it for the future

Oh, you mean through MMDS, not environment variable, correct?

Yes, sorry for confusion

dobrac · 2025-06-24T18:01:12Z

Also, how will we handle events like sandbox creation, deletion, etc? Basically events emitted from outside of the sandbox

0div · 2025-06-24T20:17:24Z

Also, how will we handle events like sandbox creation, deletion, etc? Basically events emitted from outside of the sandbox

This is not meant to replace events emitted from outside the sandbox.

dobrac · 2025-06-24T20:18:19Z

I've meant that we don't have events emitted from the outside, so the question is if it maybe makes sense to have it united

0div · 2025-06-24T20:21:31Z

I've meant that we don't have events emitted from the outside, so the question is if it maybe makes sense to have it united

any event that can be listened to from the outside should continue to be—especially if it's uncertain the VM is running—any event we would have to poll for inside the sandbox ideally shouldn't.

…nternal-sandbox-event-endpoint-e2b-2486

…ng to network slots and use it to map back to sandbox ID in event handler

…route according to RFC

sitole · 2025-07-11T09:49:49Z

packages/orchestrator/internal/sandbox/event/server.go

+	server := &http.Server{
+		Addr:    fmt.Sprintf(":%d", port),
+		Handler: mux,
+	}


What about running Gin here so we can use a strict Golang client in envd and track all version changes?

sure, we can use gin, but i'm not sure it's more strict than the std lib and also what do u mean by tracking version changes in this case?

but i'm not sure it's more strict than the std lib and also what do u mean by tracking version changes in this case?

I was thinking that the "metrics server" can be Gin-backed by the OpenAPI scheme, as we are using for the API, so all breaking changes will be clear (because of the re-generated schema). You will also be able to use the generated Go client from EnvD to call metrics service endpoints.

The point was just make it strong types on both caller and receiver

Agree with @sitole

sitole · 2025-07-11T09:51:30Z

packages/orchestrator/internal/sandbox/sandbox.go

+	sandboxIP := ips.slot.HostIPString()
+	eventStore.SetSandboxIP(config.SandboxId, sandboxIP)
+	cleanup.AddPriority(func(ctx context.Context) error {
+		return eventStore.DelSandboxIP(sandboxIP)
+	})
+


Why can we not directly get the sandbox from the orchestrator cache and get the slot IP on it?

Correct me if i'm wrong but that map uses sandbox ID as key? in my use case I need sandbox IP as key to get the ID.

Yes, you are right. But still, using Redis may not be needed here? You don't need to share state to a 3rd party service, because you will always receive requests just from sandboxes that are managed by the orchestrator itself, right? This way on Orchestrator, you should always know about all sandboxes without fetching them from Redis.

Probably there will be <300 sandboxes, which means the sandboxes map is not that big to iterate it every time you need to find an IP in it. The number of sandboxes will probably not grow, because of networking, memory, and CPU limitations. If you feel that it can be issue, then we can do some pattern that will create in-memory map "sbx ip -> sbx id" and make it synced with sandboxes map.

Agree with @sitole here

sitole · 2025-07-11T09:52:24Z

packages/orchestrator/internal/template/build/rootfs.go

 	hostname := "e2b.local"
+	eventProxyHostname := "events.e2b.dev"
+
+	eventIP := env.GetEnv("SANDBOX_EVENT_IP", "203.0.113.0")


Lets make IP constant in some shared package

Yeah, i was gonna introduce a consts.go like in other packages.

…nternal-sandbox-event-endpoint-e2b-2486

jakubno

Will we use redis or clickhouse here?

jakubno · 2025-07-21T10:36:49Z

packages/orchestrator/internal/template/build/rootfs.go

 	hostname := "e2b.local"
+	eventProxyHostname := "events.e2b.dev"
+
+	eventIP := internal.GetSandboxEventIP()


do we want to bake in into the template? Can't we use a mmds server here?

jakubno · 2025-07-21T10:45:39Z

packages/orchestrator/internal/consts.go

+)
+
+const (
+	defaultSandboxEventIP = "203.0.113.0"


why do you use TEST-NET-3 here?

jakubno · 2025-07-21T10:47:21Z

packages/orchestrator/internal/sandbox/event/server.go

+	server := &http.Server{
+		Addr:    fmt.Sprintf(":%d", port),
+		Handler: mux,
+	}


Agree with @sitole

sitole · 2025-08-12T16:20:48Z

@0div is there some update?

I would like to use the sandbox -> orchestrator HTTP server you introduced here.
I'm thinking we could split it into separate PRs to make the review quicker?

0div · 2025-08-13T23:51:24Z

@0div is there some update?

I would like to use the sandbox -> orchestrator HTTP server you introduced here. I'm thinking we could split it into separate PRs to make the review quicker?

@sitole , i put this back as draft bc i would start using it once i can consume it with the webhooks system i'm implenting.
What do you need this for exactly and how urgent do you need it?

0div added 2 commits June 16, 2025 11:52

WIP sandbox event endpoint boilerplate

d8dc71f

fix merge conflict with main

d7363e9

0div changed the title ~~Create internal sandbox event endpoint e2b 2486~~ Create internal sandbox event endpoint Jun 16, 2025

0div added 2 commits June 16, 2025 14:44

WIP working sandbox event endpoint boilerplate

587f92e

renaming and changes in file structure

61c8fe9

0div self-assigned this Jun 17, 2025

0div added the feature New feature label Jun 17, 2025

0div added 4 commits June 19, 2025 14:16

fix merge conflict with main

67abc7c

WIP abstract event server in its own sandbox submodule; add some head…

b39dd56

…er validation and register some event handlers

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

4aefc4a

…nternal-sandbox-event-endpoint-e2b-2486

cosmetic edits

78bfc5a

0div changed the title ~~Create internal sandbox event endpoint~~ Create sandbox event hook Jun 20, 2025

0div added 2 commits June 20, 2025 16:30

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

599531e

…nternal-sandbox-event-endpoint-e2b-2486

address gofumpt errors

6f1aa77

0div marked this pull request as ready for review June 20, 2025 23:58

0div requested review from ValentaTomas, dobrac and jakubno as code owners June 20, 2025 23:58

0div added 2 commits June 23, 2025 13:05

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

c47a330

…nternal-sandbox-event-endpoint-e2b-2486

store events in redis via a catchall endpoint

087c421

dobrac reviewed Jun 24, 2025

View reviewed changes

0div added 4 commits June 24, 2025 14:18

create event store interface

19a18a7

fix event store

d8ca2e6

go mod tidy & go work sync

5aa2b58

fix merge conflict with main

59ea4ad

0div added 6 commits June 24, 2025 17:07

go mod tidy api

be5c2ad

use sorted set for events time-series

eb5ca83

go-lint problematic file

3179789

add redis to start-services action

30e2cd8

add redis_url to template-manager iac

881fbb0

remove proto from redis url in github actions

89eb6fb

0div changed the title ~~Create sandbox event hook~~ Create sandbox event endpoint and handlers Jun 25, 2025

0div added 2 commits June 25, 2025 18:10

update openapi spec for sandbox events endpoint

fdc5cdb

fix merge conflict with main

333bb17

sitole self-requested a review July 3, 2025 03:15

0div added 8 commits July 3, 2025 14:25

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

e2e6e2c

…nternal-sandbox-event-endpoint-e2b-2486

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

6abcc04

…nternal-sandbox-event-endpoint-e2b-2486

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

fd4e9a1

…nternal-sandbox-event-endpoint-e2b-2486

track Sandbox IPs via store during key Sandbox lifecycle steps relati…

b4bb3f6

…ng to network slots and use it to map back to sandbox ID in event handler

remove sandbox id middleware

7535b6e

use public ip address for sandbox event routing that is not meant to …

d25a9f3

…route according to RFC

fix merge conflict with main

60de6a2

golint

4309142

sitole reviewed Jul 11, 2025

View reviewed changes

0div added 3 commits July 11, 2025 13:49

introduce internal const.go to orchestrator package

78ffeba

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

0b69432

…nternal-sandbox-event-endpoint-e2b-2486

Merge branch 'main' of https://github.com/e2b-dev/infra into create-i…

6d666e6

…nternal-sandbox-event-endpoint-e2b-2486

jakubno reviewed Jul 21, 2025

View reviewed changes

0div marked this pull request as draft July 29, 2025 19:19

ValentaTomas unassigned 0div Sep 25, 2025

Create sandbox event endpoint and handlers #784

Are you sure you want to change the base?

Create sandbox event endpoint and handlers #784

Uh oh!

Conversation

0div commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test

Uh oh!

linear bot commented Jun 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sitole Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dobrac Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sitole Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dobrac commented Jun 24, 2025

Uh oh!

0div commented Jun 24, 2025

Uh oh!

dobrac commented Jun 24, 2025

Uh oh!

0div commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sitole Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0div commented Jun 16, 2025 •

edited

Loading

sitole Jul 14, 2025 •

edited

Loading

dobrac Jul 14, 2025 •

edited

Loading

sitole Jul 14, 2025 •

edited

Loading

0div commented Jun 24, 2025 •

edited

Loading

sitole Jul 14, 2025 •

edited

Loading