Restructure guest/host error handling #868

ludfjig · 2025-09-05T22:38:01Z

This PR improves error handling between host and guest functions to prevent memory leaks and ensure more reliable deserialization.

Previously, when a host function (invoked from the guest) returned an error, the error was reported immediately without unwinding the guest stack. This left guest-side allocations in an inconsistent state, leading to memory leaks on subsequent entries into the guest.

In addition, error reporting relied on a fragile mechanism: guest errors were manually serialized into a buffer, and the host would attempt to detect them by trying to deserialize an error. If deserialization succeeded, an error was assumed to have occurred. This approach is risky because there was nothing preventing GuestError and FunctionCallResult from possibly having the same serialized format since they are completely separate.

Changes

Guest/host function calls now always return a FunctionCallResult, which explicitly represents either Ok or Err.
If host function returns an error, it's serialized back into the guest, and the guest will properly unwind, and report it back to the host, fixing a memory leak.

TODO:

For C guests: If host function returns an error, guest will panic when trying to read an expected good value (because there is only an error instead), and leak memory as a result. Will fix this in follow up PR

Closes #826
Closes #497

Signed-off-by: Ludvig Liljenberg <[email protected]>

- Add all.fbs to include all schema files in one place - Restructure function_call_result.fbs to use Result-like union - Add HostError variant to ErrorCode enum in guest_error.fbs - Update flatbuffer generation command in Justfile to use all.fbs - Update documentation for new generation process Signed-off-by: Ludvig Liljenberg <[email protected]>

Update all generated Rust code based on the new schema definitions. This includes new types for error handling and result structures. Signed-off-by: Ludvig Liljenberg <[email protected]>

- Update function_types.rs to handle Result-like return values - Simplify guest_error.rs wrapper implementation - Update util.rs for new generated types - Update mod.rs for new generated types Signed-off-by: Ludvig Liljenberg <[email protected]>

- Remove guest_err.rs from hyperlight_host (replaced by new error handling) - Remove guest_err.rs from hyperlight_guest_bin (replaced by new error handling) - Update func/mod.rs to remove obsolete import Signed-off-by: Ludvig Liljenberg <[email protected]>

- Update initialized_multi_use.rs to use new Result-like error handling - Update mem/mgr.rs to handle host function errors properly - Update sandbox/outb.rs for new error propagation pattern Signed-off-by: Ludvig Liljenberg <[email protected]>

- Update guest/host_comm.rs to use new Result-like return values - Update guest_bin/call.rs to properly handle host function errors - Update guest_bin/lib.rs to remove obsolete error handling import and make GUEST_HANDLE public (for use in C-API) - Update guest_capi/error.rs to support new error types Signed-off-by: Ludvig Liljenberg <[email protected]>

Update sandbox_host_tests.rs to use the new Result-like error handling pattern. Signed-off-by: Ludvig Liljenberg <[email protected]>

Update Cargo.lock and Cargo.toml files to reflect the dependency changes needed for the new error handling implementation. Signed-off-by: Ludvig Liljenberg <[email protected]>

Signed-off-by: Ludvig Liljenberg <[email protected]>

jsturtevant

Great catch! overall this looks like an improvement but others should probably take a look as this was my first time reviewing this code.

src/hyperlight_host/src/func/guest_err.rs

jsturtevant · 2025-09-12T17:24:23Z

src/hyperlight_guest_bin/src/guest_function/call.rs

@@ -81,10 +82,12 @@ pub(crate) fn call_guest_function(function_call: FunctionCall) -> Result<Vec<u8>

 // This function is marked as no_mangle/inline to prevent the compiler from inlining it , if its inlined the epilogue will not be called
 // and we will leak memory as the epilogue will not be called as halt() is not going to return.
+//
+// This function may panic, as we have no other ways of dealing with errors at this level


This seems like a tricky change, although the previous version looks to have just ignored errors?

jsturtevant · 2025-09-12T17:25:48Z

src/hyperlight_host/tests/sandbox_host_tests.rs

@@ -374,6 +374,7 @@ fn host_function_error() -> Result<()> {
            assert!(
                matches!(&res, HyperlightError::GuestError(_, msg) if msg == "Host function error!") // rust guest
            || matches!(&res, HyperlightError::GuestAborted(_, msg) if msg.contains("Host function error!")) // c guest
+            || matches!(&res, HyperlightError::StackOverflow()) // c guest. TODO fix this. C guest leaks when host func returns error guest panics.


do we have an issue for this? was this the case prior?

jsturtevant · 2025-09-12T17:26:57Z

src/hyperlight_common/src/flatbuffer_wrappers/function_types.rs

+//             .map_err(|e| anyhow!("Failed to get ReturnValue from bytes: {:?}", e))?;
+//         function_call_result_fb.try_into()
+//     }
+// }


remove this?

jsturtevant · 2025-09-12T17:32:39Z

src/hyperlight_common/src/flatbuffer_wrappers/function_types.rs

+    /// The builder should not be reused after a call to encode, since this function
+    /// does not reset the state of the builder. If you want to reuse the builder,
+    /// you'll need to reset it first.
+    pub fn encode<'a>(&self, builder: &'a mut flatbuffers::FlatBufferBuilder) -> &'a [u8] {


should we have some sanity tests like we do with

hyperlight/src/hyperlight_common/src/flatbuffer_wrappers/function_call.rs

Line 283 in b419bde

fn read_from_flatbuffer() -> Result<()> {

?

jsturtevant · 2025-09-12T17:38:21Z

src/hyperlight_host/tests/sandbox_host_tests.rs

+
+        for _ in 0..1000 {
+            let res = init_sandbox
+                .call::<i32>("GuestMethod1", msg.to_string())


could we also add a test that demonstrates how a guest would successfully handle a host function and continue processing with this new type? Do we have that already?

ludfjig force-pushed the host_error_leak_fix branch 2 times, most recently from d346b32 to 2e45037 Compare September 8, 2025 18:07

ludfjig added the kind/bugfix For PRs that fix bugs label Sep 8, 2025

ludfjig force-pushed the host_error_leak_fix branch 4 times, most recently from 2e83739 to fac5e92 Compare September 9, 2025 17:55

ludfjig added 9 commits September 9, 2025 10:55

Add test that fails due memory leaking when host function error occurs

82a763d

Signed-off-by: Ludvig Liljenberg <[email protected]>

Regenerate FlatBuffer generated code

f535594

Update all generated Rust code based on the new schema definitions. This includes new types for error handling and result structures. Signed-off-by: Ludvig Liljenberg <[email protected]>

Update host-side error handling

fde8fd1

- Update initialized_multi_use.rs to use new Result-like error handling - Update mem/mgr.rs to handle host function errors properly - Update sandbox/outb.rs for new error propagation pattern Signed-off-by: Ludvig Liljenberg <[email protected]>

Fix up test to work with new error handling

58900b6

Update sandbox_host_tests.rs to use the new Result-like error handling pattern. Signed-off-by: Ludvig Liljenberg <[email protected]>

Update Cargo dependencies

6b76ba0

Update Cargo.lock and Cargo.toml files to reflect the dependency changes needed for the new error handling implementation. Signed-off-by: Ludvig Liljenberg <[email protected]>

ludfjig force-pushed the host_error_leak_fix branch from fac5e92 to 8818cf5 Compare September 9, 2025 17:56

Fix test so it passes

ae355d7

Signed-off-by: Ludvig Liljenberg <[email protected]>

ludfjig force-pushed the host_error_leak_fix branch from 8818cf5 to ae355d7 Compare September 9, 2025 18:10

ludfjig marked this pull request as ready for review September 9, 2025 18:20

ludfjig requested review from danbugs, dblnz, devigned, syntactically, marosset, jprendes and simongdavies as code owners September 9, 2025 18:20

ludfjig mentioned this pull request Sep 9, 2025

Improve Error handling between the guest and the host #203

Open

jsturtevant reviewed Sep 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Restructure guest/host error handling #868

Restructure guest/host error handling #868

Uh oh!

ludfjig commented Sep 5, 2025 •

edited

Loading

Uh oh!

jsturtevant left a comment

Uh oh!

Uh oh!

jsturtevant Sep 12, 2025

Uh oh!

jsturtevant Sep 12, 2025

Uh oh!

jsturtevant Sep 12, 2025

Uh oh!

jsturtevant Sep 12, 2025

Uh oh!

jsturtevant Sep 12, 2025

Uh oh!

Uh oh!

Restructure guest/host error handling #868

Are you sure you want to change the base?

Restructure guest/host error handling #868

Uh oh!

Conversation

ludfjig commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsturtevant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsturtevant Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

jsturtevant Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

jsturtevant Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

jsturtevant Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

jsturtevant Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ludfjig commented Sep 5, 2025 •

edited

Loading