Use `glutin` to render to `Surface`/`NativeWindow` without `winit`/`ndk_glue`

[lattice0]

I want to render OpenGL inside Flutter. I've already made it work in Java and I know it's possible in C++.

Flutter has this external texture concept where it creates a SurfaceTexture surfaceTexture that I can render into. Given the surfaceTexture, this is how I render things in Java:

eglSurface = egl.eglCreateWindowSurface(eglDisplay, eglConfig, surfaceTexture, null);
egl.eglMakeCurrent(eglDisplay, eglSurface, eglSurface, eglContext)

after that, it's all a matter of calling opengl functions like glTexImage to send textures, glDraw to draw, etc.

Can I use this lib or even glium to simply create an eglSurface from a surfaceTexture and then draw things on it?

I don't think glium or glutin have the power to do anything with a surfaceTexture. But maybe I can create the eglSurface (from a surfaceTexture) in java calling from Rust, then it returns and then I can issue opengl calls using glutin? Something like this:

flutter engine calls java code
java code creates the surfaceTexture from flutter call and gives it an id
java starts rust code
rust code starts glutin or glium and fills vertex buffers, etc
rust code calls java with the id and thus java code can create the egl surface and make it current
java code call returns, now glium can issue opengl calls like glTexImage and glDraw, since the eglSurface is current, things will be drawn to it

[MarijnS95]

@lattice0 Wow, I didn't expect you to have an issue open on this for well over 1.5 years :)

Is it resolved now that you've submitted a SurfaceTexture implementation to android-ndk-rs? If so, please document here how you've solved it (little codesnippets using that implementation) for posterity, and feel free to close. Thanks!

[lattice0]

@MarijnS95 I didn't solve it yet. I'm implementing SurfaceTexture rendering for glium right now: https://github.com/lattice0/glium/blob/surface_texture/src/backend/surface_texture/mod.rs, using my SurfaceTexture implementation from android-ndk-rs and also using other small PRs I made, but having lots of OpenGL errors since I'm not an OpenGL expert.

I also have a Kotlin Android POC of my SurfaceTexture backend: https://github.com/lattice0/glium_surface_texture_poc/blob/master/app/src/main/rust/surface_texture_glium/src/lib.rs

I don't know to which functions should I map glium's make_current or swap_buffers, and also important functions from SurfaceTexture like update_tex_image have no equivalent on the glium backend side.

I presumed SurfaceTexture's attach_to_gl_context was glium's make_current, so I kind of made a static holder for the current SurfaceTexture and called detach on the old one when a new one should be attached, but I got

2022-05-25 20:40:42.434 8404-8404/com.example.gliumsurfacetexturepoc E/rfacetexturepo: [SurfaceTexture-0-8404-0] attachToContext: invalid current EGLDisplay
2022-05-25 20:40:42.439 8404-8404/com.example.gliumsurfacetexturepoc A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 8404 (rfacetexturepoc), pid 8404 (rfacetexturepoc)

If someone could help me with this I'd be very glad, I'm already deep into lots of small PRs to add functionality to support thtis

[MarijnS95]

Going through the trail of your SurfaceTexture PR this sparked a nice discussion that ended in maplibre/maplibre-rs#53 (comment) and finally a PR with that working: maplibre/maplibre-rs#99.

In short you probably don't want all this complication with SurfaceTexture and the "update" functions. You should be able to get hold of a Surface - which is really just a NativeWindow - and pass that into glutin. You can't yet, but that's what I'm working on while adding supporting the Suspended/Resumed events, in order to also support generic Surfaces without winit (though it may be complicated on the API front).

[lattice0]

I forgot to say. The reason I'm doing this is to render with glium in Flutter, and the only way to render in Flutter is to a SurfaceTexture, because its API gives me one 😩

…

-------- Mensagem Original -------- Ativo 27 de mai. de 2022 06:38, Marijn Suijten escreveu:

Going through the trail of your SurfaceTexture PR this sparked a nice discussion that ended in [maplibre/maplibre-rs#53 (comment)](maplibre/maplibre-rs#53 (comment)) and finally a PR with that working: [maplibre/maplibre-rs#99](maplibre/maplibre-rs#99). In short you probably don't want all this complication with SurfaceTexture and the "update" functions. You should be able to get hold of a Surface - which is really just a NativeWindow - and pass that into glutin. You can't yet, but that's what I'm working on while adding supporting the Suspended/Resumed events, in order to also support generic Surfaces without winit (though it may be complicated on the API front). — Reply to this email directly, [view it on GitHub](#1324 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABSTHAJ2DJFGNQPIPHILXP3VMCJX7ANCNFSM4R5N77YA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[lattice0]

@MarijnS95 here's an example: https://stackoverflow.com/questions/58531692/how-to-render-opengl-in-flutter-for-android

Flutter gives you a SurfaceTexture to render into: https://github.com/mogol/opengl_texture_widget_example/blob/master/android/src/main/java/com/yourcompany/opengltexture/OpenglTexturePlugin.java#L35 and I think there's no other way. I think it would be nice to have SurfaceTexture as a Backend for glium

[MarijnS95]

@lattice0 Right, I'm curious about that example directly passing the SurfaceTexture into eglCreateWindowSurface: https://github.com/mogol/opengl_texture_widget_example/blob/83930bd774505dec0a909abba53bc4c12c3d4d1c/android/src/main/java/com/yourcompany/opengltexture/OpenGLRenderer.java#L77 - I thought that has to go through an ANativeWindow / Surface. To me it feels like this system should be usable one way or another without having to upload "non-GL" data to the OpenGL ES texture with updateTexImage() since you already want to write to a GL texture directly and/or move that written texture into the SurfaceTexture.

[lattice0]

@MarijnS95 yes, when I tried to glue attachToGlContext to make_current I get attachToContext: invalid current EGLDisplay and then Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 8548 (rfacetexturepoc), pid 8548 (rfacetexturepoc) then looking at this error I found https://android.googlesource.com/platform/frameworks/native/+/a0db308/libs/gui/SurfaceTexture.cpp#387 which suggests that there's no EGL display, so I had to create one using an EGL rust wrapper. I then tried using https://github.com/timothee-haudebourg/khronos-egl

If you want to see my very crazy try: https://github.com/lattice0/glium_surface_texture_poc/blob/1e8f669f9dee7d6ef8dd6589fac31754f9c0aea9/app/src/main/rust/surface_texture_glium/src/lib.rs this is where I try to create the EGL display before glium calls some stuff. Also it has my submodules with my modifications to glium and android-ndk-rs for which I'm maintaining until/if the PRs get accepted.

However I don't feel like this is correct at all

[MarijnS95]

@lattice0 I could point out many errors in your codebase, starting with:

• JNI arguments don't match up (passing a surface + surfacetexture from Kotlin, but only accepting a SurfaceTexture in the Rust signature);
• Using JNI sys types instead of the core types (I don't know if they're layout-compatible).

I will later test if/how this works with your SurfaceTexture attempt. But as said many times before, even linked to the known-working maplibre WIP PR, try using a SurfaceView + Surface instead of a SurfaceTexture (again, this seems to contain extra tooling to upload image data from various producers into a GL texture... You can render into a GL texture directly).

I've cooked up an example for you, together with a small WIP change in glutin that replaces winit::Window/winit::WindowBuilder from winit with a RawWindowHandle. This works "by magic" because I finally implemented HasRawWindowHandle on ndk::NativeWindow. Which you can create using the also-recently-added from_surface() function:

https://github.com/MarijnS95/AndroidNativeSurface

Enjoy! 🥳