Modularize Manifold

[elalish]

We've been getting more requests to have just core Manifold functionality (usually Booleans) with minimal dependencies (like Clipper, convex hulls, SDFs, etc). Our code is largely already partitioned this way, but I think we need to make some build flags to allow it to build modularly. Context: godotengine/godot#91748 (comment)

I'm curious if anyone knows any best practices for how to set this up? ASSIMP does a lot of this. @kintel perhaps this would also help for OpenSCAD? Curious if you have any thoughts.

[pca006132]

I feel that getting rid of Clipper can be quite painful. If we want to do something about clipper dependency, it will be a large breaking change and should be in the 3.0 plan. I think there were comments about using our own type so it is easier to refactor.

[elalish]

Isn't clipper pretty much walled-off within CrossSection? I think we should be able to build Manifold without CrossSection if we want.

[elalish]

Also relevant to Blender integration: https://projects.blender.org/blender/blender/issues/120182#issuecomment-1170472

[pca006132]

ah, I was conflating it with glm, not something optional.

probably indicating I should sleep now :P

[kintel]

Some thou^H^H^H^Hramblings:

• Splitting up into feature sets is a bit of a slippery slope, as the number of possible permutations may explode, making testing very annoying. Can be mitigated by creating some automated test "matrixes". If there is a way of splitting into distinct sub-libraries, that might be preferable. Not sure how feasible that is.
• Some larger libraries do this pretty well (e.g. Qt, boost). They pretty much split into completely distinct libraries, possibly with some interdependencies.
• I kind of assume that people asking for this are mostly thinking in terms of dependencies, not features, but it would be nice if this is surfaced as feature selectors where the dependencies follow.
• In general, be prepared to spend some quality time with CMake :)
• Once you get sufficiently fed up with CMake, don't fall for the temptation to create a build system generator-generator script, as you'll eventually reinvent one of the 86 different build systems already created to solve this problem.
• Don't skimp on automated testing

Something which would make sense to keep in mind: There seems to be a beginning movement to start packaging Manifold into Linux distros. This currently comes implicitly through the OpenSCAD distro packages, but may become strengthened by other OSS projects adopting manifold. This can be a bit of a pivotal moment, as you'll be held to API versioning/compatibility standards (if not you, the people ending up owning your packages). Good API design and semantic versioning gets you a long way here, but having multiple variants of the same binary library is a bad thing in this domain (unless there is some sort of ultimate build containing everything that makes it into distros). Could be worth keeping in the back of your heads.

[pca006132]

Splitting into different packages should be simple. In fact we have that internally. But probably need less generic naming, as well as more tests to make sure some packages can be optional and things can still work.

[fire]

In Godot Engine engine's build we use these dependencies.

• manifold
• quickhull
• libcudacxx
• thrust
• glm

The only reasonable dependency we can remove is [thrust, libcudacxx]. quickhull is fairly short.

As mentioned earlier glm is load bearing and if you didn't pick glm, another implementation of math in C++ would probably be as big.

[elalish]

As we've said before, Manifold has absolutely no dependence on libcudacxx anymore - we're not using that backend of Thrust at all. Is there a reason you're including it?

[fire]

I switched to the newest cccl (thrust, libcudacxx). https://github.com/nvidia/cccl and it seems to still requires cccl/libcudacxx.

Note: that https://github.com/NVIDIA/thrust has been archived on Mar 21, 2024

Edited:

Compile log where I removed the cccl/libcudacxx folder.

log.txt

Edited:

There seems to be chain of includes from thrust that go to libcudacxx.

[elalish]

Interesting, it does seem like they pulled libcudacxx deeper into Thrust at some point, probably in preparation for this CCCL thing it's become. Looks like we're not the only ones displeased.

Maybe it's time to finally get off of Thrust - deprecation is a pretty good reason. @pca006132 already did a lot of work to make it easier for us to slot in a different parallel library underneath. @pca006132 Do you have any thoughts on what we should switch to? It's still hard for me to tell if PSTL encompasses TBB or vice versa, or if there's a yet more universal API we should be using.

[cjmayo]

If installing CCCL only to build Manifold (or OpenSCAD) then it can configured with:

-DCCCL_ENABLE_LIBCUDACXX=OFF
-Dlibcudacxx_ENABLE_INSTALL_RULES=ON
-DCCCL_ENABLE_CUB=OFF
-DCUB_ENABLE_INSTALL_RULES=ON
-DCCCL_ENABLE_THRUST=OFF
-DTHRUST_ENABLE_INSTALL_RULES=ON
-DCCCL_ENABLE_TESTING=OFF

Perhaps the issue is in using it as a submodule without building (just copying the files).

[elalish]

Is there a better way for manifold to package our Thrust (or CCCL) dependency than as a submodule? Curious if there's best practices here...

[cjmayo]

Personally I'm not using the submodule, but instead installing separately into my system and using it for Manifold and OpenSCAD.

But my guess is if using the submodule perhaps there is a need to run cmake etc. and install into a local directory and then use the installed directory when building Manifold.

[fire]

I like https://github.com/ingydotnet/git-subrepo for my projects as git submodules are difficult to use. Especially when for example manifold does manifold -> thrust -> imgui

[elalish]

I have to admit, the problems you're running into @fire (copying dependencies instead of pulling the submodule) makes me think the submodule approach is better. I don't want that giant pile of unused files in my repo any more than you do (which is what I get with git-subrepo). Thrust and CCCL still don't link in the actual CUDA code in our case, so all that cub and libcudacxx business doesn't actually end up in any of our binaries anyway.

I don't really see what all the submodule hate is about anyway - I find them pretty great for managing large 3rd-party dependencies that I don't ever want to touch, besides to update to the latest release occasionally. I feel like everyone who dislikes them is using them when they control both repos. Am I missing something?

[fire]

My previous monorepo had like 40+ submodules in various states of nesting. Your use cases may vary.

[pca006132]

I think we probably want a todo list for this one. cross_section is now optional, we plan to make quickhull optional as well (the PR was stalled, not sure what was blocking it), and thrust will be gone after #856 is merged. What else do we need? Remove nanobind and use the system version instead?

[fire]

nanobind and use the system version

I am not familiar with nanobind. What does it do?

[pca006132]

python binding

[elalish]

Using system dependencies is always best if we can, though I doubt nanobind concerns the C++ users since they won't bother compiling the Python bindings anyway.

[elalish]

It does seem like #810 is pretty much the last thing. It mostly got delayed to v3, since it's a breaking change. But there's also been a bunch of discussion about whether my technique makes sense or not (relying on an optional linked-in library - hull - to implement an API that's always defined in the manifold.h header). I'd appreciate some more feedback from our C++ users on whether they accept that pattern or not.

[elalish]

Actually, we're considering just forking quickhull and pulling it in so we have more opportunity to parallelize it. Then #810 won't be necessary at all and we'll still be down a dependency.

[starseeker]

@elalish is there a discussion somewhere about reworking quickhull for inclusion in Manifold? Want to make sure I'm subscribed if there is ;-)

[elalish]

Yes, in fact we have a GSOC student working on it right now: #781. More in the investigation phase just now, but moving into more code writing.

[elalish]

Okay, quickhull dependency has been removed in #881 by pulling in our own version instead. With everything else addressed, I'm going to go ahead and close this as completed.