Add GL upload function to Java interface

[javagl]

(This builds upon #957, so that should be merged first)

This adds bindings for the ktxTexture_GLUpload function to the Java interface.

There certainly are a few degrees of freedom about how exactly that should be offered. And there is a trade-off between "trying to closely resemble the existing API" and "trying to create a nice API for Java". The main point here being that the function receives three pointers (to int values, essentially). And pointers don't exist in Java.

A common (although not pretty) way is to emulate these with single-element arrays. So the call currently looks like this:

int texture[] = { 0 };
int target[] = { 0 };
int glError[] = { 0 };
int result = ktxTexture.glUpload(texture, target, glError);

This will fill the given arrays with the values that are returned from the native layer, accordingly.

Proper testing may be difficult. The basic call conditions are checked in a unit test. But beyond that, really using the function will require an OpenGL context to be current, so that 1. requires an OpenGL binding for the test, and 2. can hardly happen during the standard test runs.

---

I tried it out in a very basic experiment. This experiment currently used https://www.lwjgl.org/ , but I'll probably also try it with https://jogamp.org/ . Given that LWJGL is the library behind https://libgdx.com/ and https://jmonkeyengine.org/ , that's likely to be a primary goal.

The result of that experiment is shown here...

... and I frankly couldn't care less that it's upside down. It works 🙃

[MarkCallow]

Just curious. Why are these parameters arrays?

[javagl]

That's mentioned in the first post. And it is the most important "design decision" here. I'll elaborate that in a comment below.

[MarkCallow]

and I frankly couldn't care less that it's upside down.

The upside-down image is hugely more likely to be an application issue than an issue with glUpload.

[javagl]

A detail:

The upside-down image is hugely more likely to be an application issue than an issue with glUpload.

I'm aware of the abundance of issues related to "textures being upside-down" (and crutches like UNPACK_FLIP_Y_WEBGL didn't make it better, but maybe worse - another possible source of this issue). What I find interesting, though, is that

• Dragging-and-dropping a PNG file into https://sandbox.babylonjs.com/ shows it properly
• Dragging-and-dropping the KTX file that was directly created from that PNG into into https://sandbox.babylonjs.com/ shows it upside down

But that's all (probably) unrelated to glUpload itself.

---

About the inlined comment:

Just curious. Why are these parameters arrays?

In the C version, these parameters are pointers, like GLuint *pTexture. And the function can conveniently be called by passing in the address of a local variable, as in

GLuint texture = 0;
ktxTexture_GLUpload(kTexture, &texture, ...);

In Java, there are no pointers or addresses. And we have to return (up to) three int values from this function - in addition to the return value that represents the KtxErrorCode.

There are different options for this:

Using arrays (as it is done here)

This is a (somewhat quirky, but IMHO reasonable) way of emulating "pointers". It is still somewhat convenient:

int texture[] = { 0 };
int result = ktxTexture.glUpload(texture, ...);
// The GL texture ID is now in `texture[0]`

It also most closely resembles the original API.

Defining a structure to carry these values

One could create a struct-like class, like

class GlUploadParameters {
    int texture;
    int target;
    int error;
}

And then call the function like this:

GlUploadParameters p = new GlUploadParameters();
int result = ktxTexture.glUpload(texture, r);
// The GL texture ID is now in `p.texture`

But defining, documenting, and accessing such a structure always involves some overhead of which I'm not sure whether it brings reasonable benefit here.

---

In the JavaScript bindings, the glUpload function seems to return an "anonymous" (untyped) object with the result properties:

            val texObject = val::module_property("GL")["textures"][texname];
            ret.set("object", texObject);
            ret.set("target", target);
            ret.set("error", error);
            return ret;

This does raise some questions about the error handling - the KTX_error_code is not present there. But untyped objects don't exist in Java either.

The Python bindings do not seem to offer the glUpload function at all.

[javagl]

(A small note: The build failure at https://github.com/KhronosGroup/KTX-Software/actions/runs/11837211122/job/32983720702?pr=959#step:17:53 looks like it is unrelated to this PR, but in doubt, I'll re-start/investigate)

[MarkCallow]

I'm aware of the abundance of issues related to "textures being upside-down" (and crutches like UNPACK_FLIP_Y_WEBGL didn't make it better, but maybe worse - another possible source of this issue). What I find interesting, though, is that

* Dragging-and-dropping a PNG file into https://sandbox.babylonjs.com/ shows it properly

* Dragging-and-dropping the KTX file that was _directly_ created from that PNG into into https://sandbox.babylonjs.com/ shows it upside down

But that's all (probably) unrelated to glUpload itself.

It is indeed unrelated to glUpload. It uploads the image data in the same memory order it is found in the input file which in KTX defaults to the first byte being the top-left corner (equivalent to KTXorientation of "rd"). But OpenGL's commonly used orientation has the first byte being the bottom left corner.

Applications need to check the image orientation of the KTX file and, if differs from the coordinate system, they are using, take steps to show the image with the correct orientation. That can be by y-flipping the image (ugh!), changing the texture coordinates or using a texture transform. Your observation suggests that Babylon is not properly handling KTX files. It is obviously taking one of the steps I suggested when handling PNG files. Most likely it is y-flipping the image before uploading to WebGL. It is not doing anything for KTX files. You can test this theory by creating the file with toktx using the --lower_left_maps_to_s0t0 option which will flip the image and try loading that with Babylon.

[MarkCallow]

Thanks for the explanation about the arrays. Agree they seem the best way to return the values in Java.

The JS binding is done the way it is due to EMBIND. Maybe there is a better way. If so, now would be a good time to fix it as the upcoming release contains a major rewrite of the wrapper.

Is there a python wrapper for OpenGL?

[MarkCallow]

(A small note: The build failure at https://github.com/KhronosGroup/KTX-Software/actions/runs/11837211122/job/32983720702?pr=959#step:17:53 looks like it is unrelated to this PR, but in doubt, I'll re-start/investigate)

Why do you say that? The error is due to this warning

D:\a\KTX-Software\KTX-Software\interface\java_binding\src\main\cpp\KtxTexture.cpp(251,13): warning C4189: 'pTexture': local variable is initialized but not referenced [D:\a\KTX-Software\KTX-Software\build\interface\java_binding\ktx-jni.vcxproj]
  KtxTexture1.cpp

which is turned into an error because CI uses -Werror or whatever the MSVC equivalent is.

[javagl]

Your observation suggests that Babylon is not properly handling KTX files

It seems to be handling them properly when they are part of glTF and used as textures. The preview being upside down may just be a detail. (And the fact that it is even possible to simply drag-and-drop a KTX file into an online viewer and have a preview is already tremendously helpful - at least, until I get my Java application together that is the reason for all the stuff that I'm doing here...). Maybe the fact that it's shown upside down could be tracked somewhere in https://github.com/babylonjs , but it doesn't seem to be critical.

---

The JS binding is done the way it is due to EMBIND.

I cannot say much about the JS bindings, and would have to take a closer look at the actual .js binding files to better understand the options. (I'd mainly be curious about the error handling here...)

---

Is there a python wrapper for OpenGL?

There seem to be bindings at https://pyopengl.sourceforge.net/ but I'm even less familiar with Python than with JavaScript.

---

Why do you say that? The error is due to this warning

I see, indeed, that's before that "Update test log" failure (and likely causes it). I removed the unused variable - let's see whether the build passes now.

[MarkCallow]

I guess merging #957 caused the conflict.

The Travis-CI failure is because the Java wrapper test is failing on macOS for some reason.

[MarkCallow]

Maybe the fact that it's shown upside down could be tracked somewhere in https://github.com/babylonjs , but it doesn't seem to be critical.

I'll open an issue but first please run ktx info on the file and tell me if it has KTXorientation metadata, if so, the value, and, for completeness, the tool used to create the KTX file.

[javagl]

I guess merging #957 caused the conflict.

The merge conflict is a bit surprising (as this branch should be based on the other one), but it is resolved now in any case.

---

The Travis-CI failure is because the Java wrapper test is failing on macOS for some reason.

Two questions about the build failure on MacOS:

1. Could it be that just calling ktxTexture_GLUpload (without an active GL context) causes a crash (or hard exit) on MacOS?
2. Could it be that calling this function twice causes a crash?

(If you cannot answer them from the tip of your head, then don't waste your time - I can try to investigate it further).

The relevant parts from the build output are

The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error occurred in starting fork, check output in log

Usually, a VM crash causes a file with the name hs_err_pid<number>.log to be written. And there may be additional log files, But I think that they cannot easily be accessed inside the Travis build.

---

About the flipped texture:

(That may be a bit off-topic here, but we can move this information wherever it belongs)

I converted a PNG into KTX2 using different commands, and looked at the resulting files with the babylon viewer, and their ktx info output:

• ktx create: Texture is upside down. No KTXorientation entry.
• toktx: Texture is upside down. It has KTXorientation: rd
• toktx --lower_left_maps_to_s0t0: Texture is shown properly. It has KTXorientation: ru

I had a short look at the documentation and the source code, searching for left_maps_to, yflip, and ktxorientation, but right now, it looks like there is no way to modify the orientation when using ktx create - can you confirm that?

In any case, I'm not sure what is "right" or "wrong" here, and whether it's an issue of Babylon or of KTX-Software. The fact that ktx create creates an output that is "upside down", and there does not seem to be a functionality to change that, may raise questions, though....

Here is an archive that contains...

• A Notes.txt with the command lines for creating the tests
• The input- and output images
• The _info.txt for each output image, containing the ktx-info output (including the KTX-Software version etc)

FlipYTest 2024-11-15.zip

[MarkCallow]

Two questions about the build failure on MacOS:

1. Could it be that just _calling_ `ktxTexture_GLUpload` (without an active GL context) causes a crash (or hard exit) on MacOS?

ktxTexture_GLUpload expects a context to be current. If there isn't one it will most likely crash. The behavior should be the same on all platforms but it is somewhat dependent on the OpenGL implementation. It does not check for an active context because that is not only a platform dependent operation and depends on what is being used to set up contexts.

2. Could it be that calling this function _twice_ causes a crash?

If the first time doesn't I can't see any reason the second time would.

(If you cannot answer them from the tip of your head, then don't waste your time - I can try to investigate it further).

The relevant parts from the build output are

The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error occurred in starting fork, check output in log

Usually, a VM crash causes a file with the name hs_err_pid<number>.log to be written. And there may be additional log files, But I think that they cannot easily be accessed inside the Travis build.

Update test log was an attempt to upload logs on interest in the event of a build failure. Unfortunately the site it was uploading to seems to no longer be available. Also it does not currently attempt to upload the vm log because I was unaware of it.

About the flipped texture:

(That may be a bit off-topic here, but we can move this information wherever it belongs)

I converted a PNG into KTX2 using different commands, and looked at the resulting files with the babylon viewer, and their ktx info output:

* `ktx create`: Texture is upside down. No `KTXorientation` entry.

* `toktx`: Texture is upside down. It has `KTXorientation: rd`

* `toktx --lower_left_maps_to_s0t0`: Texture is shown properly. It has `KTXorientation: ru`

I had a short look at the documentation and the source code, searching for left_maps_to, yflip, and ktxorientation, but right now, it looks like there is no way to modify the orientation when using ktx create - can you confirm that?

In any case, I'm not sure what is "right" or "wrong" here, and whether it's an issue of Babylon or of KTX-Software. The fact that ktx create creates an output that is "upside down", and there does not seem to be a functionality to change that, may raise questions, though....

It's Babylon. The spec says the default orientation is top down (a.k.a "rd"). So your first and second cases are the same. Babylon should be flipping it, if it using WebGL's default coordinate system. Otherwise it will be upside down, exactly as you are seeing. There is no bug in the KTX software.

In the third case toktx is y-flipping the input PNG so the texture has the default WebGL orientation of "ru". Yes ktx create does not have an option to flip. I just noticed that myself yesterday. No one has asked for it since release so I'm not sure we should add one .

[javagl]

ktxTexture_GLUpload expects a context to be current. If there isn't one it will most likely crash.

That would explain it. To me, this means that this function cannot reaonably be tested as part of the unit tests. I will keep the test cases that pass in invalid parameters (because the point of these tests is to ensure that these cases are cauught by throwing a Java exception before even trying to call the native function). But the case where it actually does call ktxTexture_GLUpload (and therefore, would expect a context) have to be omitted.

[javagl]

It's Babylon. [,,,] Babylon should be flipping it, if it using WebGL's default coordinate system

I did a quick search, and there are some related threads/issues. But I think that this drag-and-drop preview for KTX is not really "specified" in any way. They are not making claims about the "texture coordinates of that ('virtual') rectangle" that the texture is rendered on.

I hope it's OK to ping @bghgary here:

We're talking about the drag-and-drop preview of KTX files in the sandbox. And it could be confusing that the preview appears to be displayed upside down, compared to the corresponding PNG file:

Not a big deal, but maybe worth mentioning...

(EDIT: Test data and further infos are contained in the archive at the bottom of a previous comment)

[MarkCallow]

ktxTexture_GLUpload expects a context to be current. If there isn't one it will most likely crash.

That would explain it. To me, this means that this function cannot reaonably be tested as part of the unit tests. I will keep the test cases that pass in invalid parameters (because the point of these tests is to ensure that these cases are cauught by throwing a Java exception before even trying to call the native function). But the case where it actually does call ktxTexture_GLUpload (and therefore, would expect a context) have to be omitted.

That seems reasonable. On the native and JS side I test these functions using the glloadtests and vkloadtests apps which use SDL. Emscripten compiles these apps so I can use them in Javascript. Java versions would have to be written.

Isn't it possible to create an OpenGL context in the unittests?

One other thing I forgot to mention is that the ktxTexture*_GLUpload needs to load the function pointers for the GL functions it uses. The preferred way is for the application to call ktxLoadOpenGL passing it a pointer to the function to use to query the function pointers. If that has not been called by the first time ktxTexture*_GLUpload is called, an attempt is made to load the OpenGL library then dlsym is used on Linux and macOS, emscripten_GetProcAddress on the web and WglGetProcAddress or GetProcAddress on Windows. If the function pointers aren't found an error is returned. It shouldn't crash.

Is it not clear to me at all why the test crashed only on macOS before. Perhaps you aren't correctly handling a failure to get the function pointers and they weren't found on macOS.

[MarkCallow]

Not a big deal, but maybe worth mentioning...

The KTX spec is very clear that the default orientation has s increasting to the right and t increasing down. This is the same orientation as a PNG file. This can be modified by including KTXorientation metadata in the file. The default corresponds to a value of "rd". An application should check the orientation when loading the file and flip the image, modify the texture coordinates or use a texture transform to ensure the image is presented correctly. If using libktx, the orientation is found in the orientation field of the ktxTexture struct after loading. Otherwise the metadata must be read.

In my view this is a bug that should be fixed. It is indeed confusing that the preview for PNG and KTX differs.

[javagl]

About the tests and GL context:

Isn't it possible to create an OpenGL context in the unittests?

That might be possible (with the disclaimer that I don't know much about how many of these things actually work in the ~"headless" world of CI builds). I think that it would require to declare a dependency to a Java library that provides OpenGL bindings, to actually call the functions that create a GL context to begin with. This also might be possible (and of course, this dependency would only be required for the unit tests, and it would not be a dependency for the Java KTX bindings in general!). But there are too many unknowns for me to ~"just try this out" right now.

One other thing I forgot to mention is that the ktxTexture*_GLUpload needs to load the function pointers for the GL functions it uses. The preferred way is for the application to call ktxLoadOpenGL passing it a pointer to the function to use to query the function pointers. If that has not been called by the first time ktxTexture*_GLUpload is called, an attempt is made to load the OpenGL library then dlsym is used on Linux and macOS, emscripten_GetProcAddress on the web and WglGetProcAddress or GetProcAddress on Windows. If the function pointers aren't found an error is returned. It shouldn't crash.

I have a rough idea about how some of these things work (related to some some stuff that I did in my OpenCL bindings). But it's not clear for me how that translates to KTX and OpenGL. You mentioned ktxLoadOpenGL, but this is not really documented, and even it it was, it would be difficult to "map" something like this to the Java world (with function pointers, and considering that the OpenGL bindings for Java are just a different library that probably already hides much of the low-level functionality here...)

Is it not clear to me at all why the test crashed only on macOS before. Perhaps you aren't correctly handling a failure to get the function pointers and they weren't found on macOS.

There is the call to ktxTexture_GLUpload in the JNI layer - I could probably just try to wrap some blanket try ... catch around that, but that's probably not what you meant. And if this call itself causes a crash or exit, then there's not much that can be done to prevent the Java VM from bailing out...

---

About the flipped texture:

In my view this is a bug that should be fixed.

Preferably yes, because it is confusing. The reason why I said that it is "not a big deal" is that this only applies to the drag-and-drop preview. When used as textures, everything seems to be fine, so it looks like they are already properly handling the metadata/orientation in that case. (I haven't looked into the implementation details, though).
Let's see what @bghgary says.

[bghgary]

From a quick look, it does look like the preview creates a texture that is Y-inverted from what the glTF loader is doing. I will confirm and make a fix.

[MarkCallow]

I have a rough idea about how some of these things work (related to some some stuff that I did in my OpenCL bindings). But it's not clear for me how that translates to KTX and OpenGL. You mentioned ktxLoadOpenGL, but this is not really documented,

It is documented. Maybe you were looking at the on-line docs which will be for the last release. The function was added after the last release.

and even it it was, it would be difficult to "map" something like this to the Java world (with function pointers, and considering that the OpenGL bindings for Java are just a different library that probably already hides much of the low-level functionality here...)

The caller never sees any function pointers but does have to provide a pointer to the function to use to query OpenGL pointers. I'm pretty sure failure to call ktxLoadOpenGl is not the reason for the crash and your test program works without it so it is not worth trying to make it work from Java.

Is it not clear to me at all why the test crashed only on macOS before. Perhaps you aren't correctly handling a failure to get the function pointers and they weren't found on macOS.

There is the call to ktxTexture_GLUpload in the JNI layer - I could probably just try to wrap some blanket try ... catch around that, but that's probably not what you meant. And if this call itself causes a crash or exit, then there's not much that can be done to prevent the Java VM from bailing out...

If an OpenGL library or the function pointers could not be found, the call to ktxTexture*_GLUpload would return either KTX_LIBRARY_NOT_LINKED or KTX_NOT_FOUND. If it is crashing at this point it could only be due to the underlying system call to get the library, e.g. dlopen, or the call to obtain a symbol from it. Both are unlikely. More likely this part was satisfied but calling a GL function without a current context caused the crash.

I don't want to burden the unit tests with needing an OpenGL implementation so I'm going to merge this as is.

How do you obtain JOGL or whatever you are using in your app? Do you have to download an additional package?

[javagl]

How do you obtain JOGL or whatever you are using in your app?

Yeah, there are two-and-a-half OpenGL bindings for Java:

• LWJGL from https://www.lwjgl.org/
  • This comes in "Version 2.x" and "Version 3.x"
• JOGL from https://jogamp.org/cgit/jogl.git/about/

In all cases, it should usually be sufficient to add the respective dependency to the Maven POM. But unfortunately, there sometimes still is the hassle of the native libraries.

(This is strongly related to #624 where we talked about how this could be solved for KTX. The task to draft a solution for the native library handling for the KTX Maven release is still on my plate. I have some clear ideas here. I "only" need to allocate time for that...)

For the OpenGL bindings, the current state is roughly:

• For LWJGL 2.x (which I'm currently using), it is still necessary to manually add the lwjgl64.dll to the project directory. For end-user applications, this is a no-go...
• For LWJGL 3.x, this should no longer be necessary. It should automatically resolve the native libraries from JARs that are published in Maven
• JOGL should also be able to automatically resolve the libraries

The last two points remain to be verified. But if the last point is correct, then the answer to

Do you have to download an additional package?

is: No! - ideally, you just have to add it as a dependency to the Maven POM.xml, and it magically works 🤞

A while ago, created viewers for glTF, once based on JOGL, once based on LWJGL version 2 and (not published) based on LWJGL version 3. For my KTX experiments, I intend to do something similar: I started a jktx-viewer library that I intend to publish eventually, with different GL rendering backends. But that's still in an early stage...

[bghgary]

From a quick look, it does look like the preview creates a texture that is Y-inverted from what the glTF loader is doing. I will confirm and make a fix.

This is fixed in Babylon.js Sandbox now.

[javagl]

Thanks @bghgary !