Custom code injection to Shader source #24
Conversation
|
Hey this is super interesting! Apologies for the slow reply, met me review and get back to you! |
|
So im trying to reason about if this makes sense in Satin main repo - is the goal here to:
If so, the shader has an existing parameter group, and I think introspecting the parameter group maybe a better way to do that (separation of concerns, reusability outside of the shader)
if the latter, id be curious why not either edit the shader, make a material instance yourself? If you dont mind, can you walk through the use cases of this functionality, as i could be missing something useful, but im fearful it introduces some specific use case functionality that might be confusing for folks working with Satin and using different entry points to expose UI controls to Satin |
|
hi, thanks for the reply! But other instance I found it a bit useful is the situation like below. In case you think this is a bit too much of the edge case, or if I'm missing other ways to handle these cases in Satin, please feel free dismiss / close this branch. I setup PDAL library which I exposed its cpp api to swift so I can parse point cloud file (.ply/.xyz/.las/.geotiff...) and render them via Satin. It returns the interleaved buffer and variable length list of lazy var calculateDepthProcessor: CalculateDepthProcessor = {
let device = context!.device
let dimensions = pointCloud.dimensions
let label = "\(self.label.filter { $0.isLetter })_CalculateDepth"
ComputeShaderLibrarySourceCache.registerInjector({( source: inout String, config: ComputeShaderLibraryConfiguration) -> Void in
source = source.replacingOccurrences(of: "// InterleavedBufferInjection", with: dimensions.constructMTLStruct())
source = self.injectCode(source: source)
}, for: label)
let processor = CalculateDepthProcessor(label: label, device: device, pipelinesURL: RasterisedPointCloud.pipelinesURL, live: true)
processor.delegate = self
DispatchQueue.main.async {
processor.shader?.recompile()
}
return processor
}()And this is corresponding metal file that gets processed by the injection. #include <metal_stdlib>
using namespace metal;
#include "../../Satin/VertexUniforms.metal"
// InterleavedBufferInjection <--- **this gets replaced by the custom injection**
struct CalculateDepthUniforms {
uint pointCount;
uint pointsOffset;
float pointSize; // slider,0.0,10.0,1.0
};
kernel void calculateDepthUpdate(
device const InterleavedPoint *interleavedBuffer [[buffer(ComputeBufferCustom1)]],
device atomic<uint> *depthBuffer [[buffer(ComputeBufferCustom2)]],
device atomic<uint> *attributeBuffer [[buffer(ComputeBufferCustom3)]],
const device VertexUniforms &vertexUniforms [[ buffer( ComputeBufferCustom10 ) ]],
const device CalculateDepthUniforms &uniforms [[ buffer( ComputeBufferUniforms ) ]],
uint3 gid [[thread_position_in_grid]],
uint simd_lane_id [[thread_index_in_simdgroup]],
uint simd_group_id [[simdgroup_index_in_threadgroup]]
)
{
uint index = gid.x;
if (index >= uniforms.pointCount) {
return;
}
uint pointDataIndex = index + uniforms.pointsOffset;
InterleavedPoint point = interleavedBuffer[pointDataIndex];
...
// Prepare color once
#ifdef COLOR_USHORT
uint r = uint(clamp(float(point.red) / float(USHRT_MAX) * 255.0, 0.0, 255.0));
uint g = uint(clamp(float(point.green) / float(USHRT_MAX) * 255.0, 0.0, 255.0));
uint b = uint(clamp(float(point.blue) / float(USHRT_MAX) * 255.0, 0.0, 255.0));
#elif defined(COLOR_UCHAR)
uint r = uint(point.red);
uint g = uint(point.green);
uint b = uint(point.blue);
#elif defined(NO_COLOR)
uint r = 255;
uint g = 255;
uint b = 255;
#else
uint r = uint(clamp(point.red * 255.0, 0.0, 255.0));
uint g = uint(clamp(point.green * 255.0, 0.0, 255.0));
uint b = uint(clamp(point.blue * 255.0, 0.0, 255.0));
#endif
uint a = 255;
uint packedColor = r | (g << 8) | (b << 16) | (a << 24);
uint uintDepth = as_type<uint>(clipW);
// for certain case of ply
struct InterleavedPoint {
float x; // offset 0, size 4
float y; // offset 4, size 4
float z; // offset 8, size 4
float red; // offset 12, size 4
float green; // offset 16, size 4
float blue; // offset 20, size 4
float uvmap_x; // offset 24, size 4
float uvmap_y; // offset 28, size 4
}; // Total Stride: 32
/// -----------------------------------------
// for the case of .laz file, no colours,
#define NO_COLOR 1
struct InterleavedPoint {
float x; // offset 0, size 4
float y; // offset 4, size 4
float z; // offset 8, size 4
ushort intensity; // offset 12, size 2
uchar returnNumber; // offset 14, size 1
uchar numberOfReturns; // offset 15, size 1
uchar scanDirectionFlag; // offset 16, size 1
uchar edgeOfFlightLine; // offset 17, size 1
uchar classification; // offset 18, size 1
uchar synthetic; // offset 19, size 1
uchar keyPoint; // offset 20, size 1
uchar withheld; // offset 21, size 1
uchar overlap; // offset 22, size 1
uchar __padding23[1];
float scanAngleRank; // offset 24, size 4
uchar userData; // offset 28, size 1
uchar __padding29[1];
ushort pointSourceId; // offset 30, size 2
float gpsTime; // offset 32, size 4
uchar scanChannel; // offset 36, size 1
uchar __paddingEnd[3]; // offset 37
}; // Total Stride: 40
Metal struct construction: extension Array where Element == PDALDimensionInfo {
func constructMTLStruct() -> String {
var structString = ""
// MARK: Construct COLOUR TYPE DECLARATION
if let redDim = first(where: { String($0.name) == "Red" }) {
if redDim.mtlTypeName == "ushort" {
structString += "#define COLOR_USHORT 1\n"
} else if redDim.mtlTypeName == "uchar" {
structString += "#define COLOR_UCHAR 1\n"
}
} else {
structString += "#define NO_COLOR 1\n"
}
if first(where: { String($0.name).lowercased() == "z" }) == nil {
structString += "#define NO_Z 1\n"
}
structString += "struct InterleavedPoint {\n"
var currentOffset = 0
var maxAlignment = 1 // Track the largest alignment
forEach { dim in
let typeSize = dim.outputSize
// This is the correct C-style alignment rule (size of the type)
let alignment = typeSize
// Track the largest alignment for final stride calculation
if alignment > maxAlignment {
maxAlignment = alignment
}
// Add padding if needed for alignment
let misalignment = currentOffset % alignment
if misalignment != 0 {
let paddingNeeded = alignment - misalignment
// Use a unique name for each padding
structString += " uchar __padding\(currentOffset)[\(paddingNeeded)];\n"
currentOffset += paddingNeeded
}
let typeName = dim.mtlTypeName // Assumes this is "float", "ushort", etc.
let varName = String(dim.name).camelCase // Assumes you have this utility
structString += " \(typeName) \(varName); // offset \(currentOffset), size \(typeSize)\n"
currentOffset += typeSize
}
// Pad to the *largest member alignment* (which is 4)
let remainder = currentOffset % maxAlignment
if remainder != 0 {
let paddingNeeded = maxAlignment - remainder
structString += " uchar __paddingEnd[\(paddingNeeded)]; // offset \(currentOffset)\n"
currentOffset += paddingNeeded
}
structString += "}; // Total Stride: \(currentOffset)\n"
return structString
}
} |
|
Thats super interesting - i see how that could be interesting and useful. Thanks for explaining it. Let me review now that i have a better sense of capabilities. Thanks for this! |
|
One quick question, could this not be handled without shader injection by upcasting to uint16 or 32 for some of the buffer contents, so theres a consistent buffer type? |
|
that's also an option, too! I wanted to avoid upscaling on my use case of handling large number of pointcloud like millions of points, where its memory consumption is also increasing if I normalise colours, or other properties by upcasting them, so I ended up not upcasting... just as the example above, when I import // for certain case of ply
struct InterleavedPoint {
float x; // offset 0, size 4
float y; // offset 4, size 4
float z; // offset 8, size 4
float red; // offset 12, size 4
float green; // offset 16, size 4
float blue; // offset 20, size 4
float uvmap_x; // offset 24, size 4
float uvmap_y; // offset 28, size 4
}; // Total Stride: 32and if I load the struct InterleavedPoint {
float x; // offset 0, size 4
float y; // offset 4, size 4
float z; // offset 8, size 4
ushort intensity; // offset 12, size 2
uchar returnNumber; // offset 14, size 1
uchar numberOfReturns; // offset 15, size 1
uchar scanDirectionFlag; // offset 16, size 1
uchar edgeOfFlightLine; // offset 17, size 1
uchar classification; // offset 18, size 1
uchar synthetic; // offset 19, size 1
uchar keyPoint; // offset 20, size 1
uchar withheld; // offset 21, size 1
uchar overlap; // offset 22, size 1
uchar __padding23[1];
float scanAngleRank; // offset 24, size 4
uchar userData; // offset 28, size 1
uchar __padding29[1];
ushort pointSourceId; // offset 30, size 2
float gpsTime; // offset 32, size 4
uchar scanChannel; // offset 36, size 1
uchar __paddingEnd[3]; // offset 37
}; // Total Stride: 40I also agree the framework should avoid modifying the hand-written shader code, but I couldn't find a good solution to inject those dynamic custom snippet otherwise so far and I may be missing something. it could be my niche use case that it's completely fine not to merge this PR for now, too. maybe there are better ways to handle this... |
|
definitely, i want to handle both point cloud as well as splats in Satin, and construct a nice reusable set of loaders, materials etc that operate natively on those types. So i think the use case is helpful guidance on solutions. I hear you about memory usage as well. Thanks for the dialogue on this, maybe we can find a nice general solution here that solves a few use cases together? I had done some minor work on https://github.com/scier/MetalSplatter and have been background processing converting it to Satin native solutions. Theres a fork or two which implements spherical harmonics, which does bloat memory in some sense. Id be curious if there are additional use cases for these dynamic inserts, im sure there are. |
|
I was also working on 3dgs on satin a bit referring to MetalSplatter too! but as you said Spherical coefficients props such I managed to bring it onto Satin, but I'm not well versed into graphics enough to render it efficiently... but served a purpose. Other metal based 3dgs renderer like gsm-renderer seemed to be composed of series of compute shaders, and I was wondering how I could incorporate those pipelines flexibly on top of Satin. |
|
Awesome! I found this port of MetalSplatter which seems to leverage spherical harmonics: (in git commits you can find it) https://github.com/lanxinger/MetalSplatter Satin def supports Compute, it should be doable:) |
2 participants
Not sure if anyone needs this, but putting this here in case someone may find it useful...
This change allows injecting custom code to shader before being cached.
registerInjector/unregisterInjectorglobally using the "*" wildcard
[Example]:
I had OSC parameter group I can add/remove the parameters, then I wanted to insert the metal struct dynamically so I dont have to maintain it in the metal source code while prototyping...