Real actual GLSL shader support

[bud11]

Describe the project you are working on

A 3D game that I want to resemble a PS2 game.

Describe the problem or limitation you are having in your project

..So many limitations for such a simple visual style. It basically cant be done. Cant do accumulation motion blur without a hacky renderingdevice glsl workaround someone made, can't do stencil shadows without using someone's outdated stencil branch. Have also had a ton of trouble with rendering modes, alpha scissoring, attuentation/shadowmaps, it just sucks.
..Why am I sat here struggling to achieve rendering techniques from over 20 years ago, at this point for over 4 years? I seriously hate the godot shader language with a passion, it feels like a toy, its just barely good enough to do what I want about 60% of the time.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

GLSL shaders. Just, real glsl shaders. Theyd be so much more flexible from my understanding, and Im sure many of us intermediate/advanced users would be more than okay with the extra required effort to get them working.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Ideally just a GLSL option on shadermaterials.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No idea. Renderingdevice is all I can think of, but last I checked thats super undocumented and might even be more difficult than this. Not like Im unwilling to learn new things, but.. well, Id refer to this issue.

Is there a reason why this should be core and not an add-on in the asset library?

Self explanatory

[AThousandShips]

Are you talking about using glsl instead of gdshader in the existing pipeline, or about doing your own rendering from scratch?

[huwpascoe]

Well at a glance it's already available
shader_compile_spirv_from_source ( RDShaderSource shader_source )

Why am I sat here struggling to achieve rendering techniques from over 20 years ago, at this point for over 4 years?

Because those old techniques are complicated.

It'd be easiest to get it working with straight OpenGL and then port to Godot.
https://learnopengl.com/

[Calinou]

Moving to discussion, as there is no concrete technical implementation provided.

Note that there are already numerous proposals about making the renderer more flexible without giving up on usability (which is the main reason why GLSL is not recommended as the default option):

• Expose an intuitive subset of stencil operations #7174
• Implement a Rendering Compositor #7916
• Add support for shader templates to override built-in shader code #8366

As per the Best practices for engine contributors, Godot prefers offering tailored solutions to each problem, as opposed to big and flexible solutions that can be difficult to work with (including for basic use cases).

[bud11]

Then I'm confused as to what this means other than that?

Youve got me there, my definition and ideas have been changing over the whole discussion. So for the record, at this point, Im accepting that serious pipeline flexibility improvements are difficult and already on the general long term gdshader roadmap, and Im suggesting a hackier lower level solution that just lets me get at viewport and pipeline buffers and inbuilts and kind of do things myself. In whatever the least integrated, lowest effort way would be for the rendering team to implement that kind of access. Leaving the door open to people like me.

Well how would you solve it I ask again?

I dont know for sure. I havent poured serious time looking at the renderingdevice api to try to do something that I know isnt possible in multiple ways. But in my head, Im imagining pseudocode along the lines of..

in editor..
add glslshadermaterial to whatever material slot I want. takes a glsl shader file and has no other properties besides which render pass you want it to get drawn in and priority if godot is doing the draw calls. would probably personally base this on the exact standardmaterial glsl from godot's source.
might be nice if godot could cache this but doesnt really matter for me.

on viewport ready..
get viewport depth/stencil buffer from godot
send whatever stencil setup commands I have to send with renderingdevice, if I do

on model ready..
get model vertex/normal buffer from godot, or make my own if possible, exposure/implementation here doesnt matter as much as the screen buffers
bind buffers/uniforms to glsl shader material slot with some incredibly minimal glslshadermaterial api or with renderingdevice

in process or in some renderingserver frame draw callback or something..
get transforms, bone matrices and other constantly updating uniforms/built ins, ideally as much from godot as possible but Id be fine with reimplementing some stuff myself, send to shader in same way as just described
if I have to send manual draw calls then I can do that here too sure, feels like glslshadermaterial should do that on its own though

on model freeing..
free up model specific buffers if needed

Assuming again Im not massively dumbing down or misunderstanding something here

[huwpascoe]

Assuming again Im not massively dumbing down or misunderstanding something here

Thought experiment: all the features you want, full pipeline access, stencil buffers, vanilla glsl. Imagine a Godot scene: A directional light, a camera and three mesh instances (plane, sphere, torus) positioned in an aesthetically pleasing way.

What are the steps in pseudo-code to draw stencil shadows for that scene?

for mesh in scene.get_all_mesh_instances():
  # what do?

[bud11]

Thought experiment: all the features you want, full pipeline access, stencil buffers, vanilla glsl. Imagine a Godot scene: A directional light, a camera and three mesh instances (plane, sphere, torus) positioned in an aesthetically pleasing way.

What are the steps in pseudo-code to draw stencil shadows for that scene?

for mesh in scene.get_all_mesh_instances():
  # what do?

Id really prefer you just tell me if Im missing something, but to answer the question anyway.. in that specific for loop, after buffers are already set up and referenced in script and stuff, I really cant imagine doing much more than just:

1. Gather stuff like object's matrices, camera/projection matrices, light details/vectors/whatever, etc, convert a little if needed, send over to shader with uniform calls
2. Send vertex buffer draw calls using the shadow shader if needed, with vert index consideration if needed

Would have to make sure this happened in transparent/later pass or something so the scene geometry was done
In my case I dont care about shadows being per-light or anything, I just want one basic shadow for each object no matter what.

From the looks of it there might also be bindings/commands in or around the loop to properly start/end using specific buffers, but again I dont know specifics because Ive never had to do this, and I dont know how exactly renderingdevice works or compares to something like opengl.
With a little googling Im seeing stuff like this. Im imagining that kind of thing with maybe half the init/binding stuff cut out because godot already makes them, and obviously a lot more in the way of scene detail communication, in a loop.

If what youre getting at is that it starts becoming harder and harder the more standard rendering features you want to include, Id argue it really doesnt matter. I think the people using this wouldn't be the types to care. This would be a way to hack together very specific pipeline supplementations/advanced effects, with a lot of power and flexibility, but without source editing or complete reinvention

[huwpascoe]

I needed to ask because I had to confirm what you don't know.

for mesh in scene.get_all_mesh_instances():
  for light in scene.get_all_lights():
    for tri in _recompose_the_triangles(mesh):
      create a quad for every edge of the triangle using
      the normal of the triangle created from the light 
      source to the edge and store this in a buffer
      cache the quad data for static instances

This is Step 1, using the most naive approach. With GDScript the performance would be abysmal. I think the PS2 had something not unlike compute shaders to accelerate the task.

Step 2 render the shadow volumes front cull then back cull with the appropriate comparison function to generate an accumulated stencil while compensating for the camera frustum itself being inside or outside a volume.

Step 3 render ambient lighting only as first pass, render the lighting as a second pass using the depth stencil created in step 2 to discard fragments in shadow.

That's based on what I don't know. So... It's still simplified and probably wrong.

[bud11]

I needed to ask because I had to confirm what you don't know.
...
That's based on what I don't know. So... It's still simplified and probably wrong.

Fair, sorry. Kinda just assumed you all knew better than me.

Next thing, talk of quads and mesh generation. In past Ive got something very close to stencil shadows just by having a shader on the next pass that stretches verts super super far away if the dot between the light normal and the global vert normal is lower than some threshold, and then using some really stupid depth buffer + alpha depth trickery to get something that slightly works but isnt remotely project viable. Heres an image of that in godot 3.

This did look kind of ugly from a threshold standpoint too though, so yeah lets say generating quads in like a perfect stretched way is nessecary, and the other steps exactly as you describe too.
The great thing about what Im suggesting is that just becomes almost entirely my responsibility, its not extra extensive godot pipeline feature work. And I could go with a geometry shader, a compute shader, c#, whatever I want.

Infact, if anything, if its a little more complex of an effect than I was initially assuming, thats MORE reason for this stuff to be exposed. I know godot is set to just never include geometry shaders.

[bud11]

Alright, Ive spent a little time and looked into the details of what Im actually asking for here. Sorry if I came off as rude.
I'd be willing to learn and use renderingdevice to implement my own 3d glsl shaders in the likely case that the rendering compositor is just way off into the indeterminate ever shifting future, but it seems like it'd be needlessly difficult, primarily due to the fact that it doesnt seem like I can just access godot's buffers to use with renderingdevice? For example I couldnt find a way to just access the vertex buffer of a mesh or something. I only have a surface level understanding at this point but it seems like if I wanted to integrate my own glsl shaders into the rest of the godot world, or even just make much use of godot mesh/visual types, Id need those buffers.

So I think asking for a way to get internal buffer RIDs without c++ would be a better more reasonable request, assuming Ive not missed a way to do that.

[bud11]

...it still doesn't mean you can just add custom buffers, because of hardware limits and details, and you wouldn't have control over the data fed to the pipeline either

Why not, arent buffers just arbitrarily definable to some degree? And isnt renderingdevice supposed to abstract away a lot of the device/hardware/api complexity and platform specificness? Doesnt godot actually use renderingdevice for its own rendering at this stage?

My understanding was pretty much that as long as the glsl exists with proper buffer/uniform bindings and stuff and the rendering api digests it and gets told to draw with it, it works. Is that wrong? If so I guess Im massively underestimating either how godot's pipeline works or how rendering/pipelines work in general.

But I think what you seem to be proposing isn't realistic based on my experience, it sounds like you're wanting a fully configurable pipeline or one where you can inject code into in ways that I doubt the pipeline can be flexible enough for, simply because of the practical realities of writing pipelines

Im not nessecarily saying it'd need to be this amazing transparent super flexible generic yet somehow easy solution that works on every platform, every renderer, or even in editor (assuming no toolscripts). Im imagining something equal parts script and glsl based, where completely non-godot-standard buffers and uniforms and the like have to be entirely created and managed by the user, but you can rely on godot in areas where there is shader precedent and pipeline support. Kind of like a dirtier, more involved, but completely flexible way of getting at scene rendering, without having to start from nothing.
As for sending information TO the godot pipeline, I cant imagine why you'd need to. I dont get what that'd entail. I think what Im getting at is the idea of your code acting as an extension to the pipeline, like the pipeline and your code are working together to pass bindings and calls and stuff, if thats possible or remotely makes any sense at all.
From what I understand and can see in other discussions, the rendering compositor and its effects are kind of vaguely getting at the same idea, but theyre not as flexible.

This issue goes into the idea of letting you implement your own entire pipeline and goes into some of the same general issues we're talking about, but I think what Ive just described might be a genuinely viable middle ground, if I understand what Im talking about remotely correctly.

And yeah, Im talking very hypothetically. Ive looked at renderingdevice docs and googled and asked around but Ive not been able to make much progress learning with this. As a matter of fact I learnt what a buffer properly was like 2 days ago. Thank you for engaging in the conversation so far even if I'm sounding a little stupid

[bud11]

Found a render setting:

It's not quite perfect in a "we have shadow volumes at home" sort of way, but combined with a 480p buffer it's pretty convincing, I think.

Thats what Im doing now, hard high res shadows, but it falls flat in some aspects and has virtually zero stylization ability. I wanted to:

• Put subtle gradients on character shadows.
• Have shadow volumes that dont nessecarily make sense or have a clear casting object.
• Have individual shadows overlap and get darker instead of merging, which shadowmaps cant do.

Im also going for more of a ps2 aesthetic superficially, like a HD remaster of a ps2/wii game, so 480p being the intended/default resolution wouldn't be ideal.

[bud11]

Thinking about it more, I think I do just want more pipeline control, but specifically with the full glsl suite of features.

Itd be really cool if I could just take the 3d standard shaders in glsl that godot already has in the source, modify them a little, and shove those into a new shader type on material slots ("GLSLShaderMaterial") that doesnt actually do anything besides store a glsl reference and tell the engine that you're going to handle it yourself. It could just appear as a pink error texture or something, or offer incredibly basic vanilla placeholder functionality like being able to set an unshaded single albedo texture so the editor view doesnt end up a mess.

Then in script, with renderingdevice, Id set up bindings and uniforms and buffers and basically my own little specialized middle portion of a pipeline, ideally that could use some of the preexisting godot buffers by exposing the RIDs. Not even nessecarily to mix and match gdshaders and glsl shaders, but rather so I didnt have to create new identical buffers and waste vram or face hardware limitations something.

I can see the Material base class's doc that shadermaterial and standardmaterial inherit from already mentions being able to create custom material types. But something tells me making my own GLSLShaderMaterial in script to use like this isnt as simple as I'd hope..? Something something pipeline integration/exposure.
That and I cant get at godot buffers, again unless Im mistaken, either about them being exposed or that just not being how that works.

[huwpascoe]

Then in script, with renderingdevice, Id set up bindings and uniforms and buffers and basically my own little specialized middle portion of a pipeline, ideally that could use some of the preexisting godot buffers by exposing the RIDs. Not even nessecarily to mix and match gdshaders and glsl shaders, but rather so I didnt have to create new identical buffers and waste vram or face hardware limitations something.

If that much control is needed then go engineless with c++/opengl or even bgfx which has bindings for every programming language. (lol don't you dare make a feature request to integrate it with Godot)

That and I cant get at godot buffers, again unless Im mistaken, either about them being exposed or that just not being how that works.

viewport.get_texture().get_rid(); that's the buffer.

The upcoming 4.3 will allow you to run pure glsl compute shaders on the pixels. Maybe this is the way? https://docs.godotengine.org/en/latest/tutorials/rendering/compositor.html

[bud11]

If that much control is needed then go engineless with c++/opengl or even bgfx which has bindings for every programming language. (lol don't you dare make a feature request to integrate it with Godot)

I did think about switching to bgfx or something, or modifying godot myself, and I might try in future, but for now it'd be too much. Cant keep expanding scope forever. I feel like my realistic options here are either waiting for godot or throwing together a hacky little static rendering augmentation class that can just about handle the additions I want with renderingdevice. I dont know nearly enough about c++ or godot/engine internals yet.

viewport.get_texture().get_rid(); that's the buffer.

Thats just the final screen buffer is it not? Rather than the entire frame buffer object with access to depth/stencil and stuff. And would that even really work directly as a buffer on renderingserver, or is that more like the rid pointing towards a cpu godot texture reference or something?
Also can't get at the aforementioned vertex buffers, though Im less concerned about that since I can probably use arraymesh like you said.
Part of why this seems so feasible to me is that I heard that godot already makes a stencil buffer internally on the fbos, but just does nothing with it, iirc.

The upcoming 4.3 will allow you to run pure glsl compute shaders on the pixels. Maybe this is the way? https://docs.godotengine.org/en/latest/tutorials/rendering/compositor.html

Ive heard about the compositor and rendering hooks and stuff but Im a little confused on what it all actually is or what degree of control it offers. It all seems great for compute and post processing but not great for actually getting at the scene.
That, and arent compute shaders really slow or impossible on older/lower end hardware? Really not great if an apparently ps2 style game cant guarantee running on hardware from the last decade. Ive heard compute shaders suck on switch for example but I have no idea how true that is.