Julia bindings

[eschnett]

This adds bindings for the Julia language to openPMD. These bindings are still a work in progress; they are incomplete and/or inefficient.

Julia bindings consist of two parts:

• A C++ wrapper that defines the functions that can be called from Julia
• A Julia module that exports these functions in a way that looks "natural" for Julia

Since Julia is a different language, one probably should not wrap all features one-to-one. For example, types are first class objects in Julia, so that much of the Datatype class is not necessary. Also, Julia is not object-oriented, and thus all function are free functions (and not member functions), etc.

I'm posting this here in case someone wants to add a comment.

Related to #625

To Do

• add example(s)/tests
• add to CI
• document requirements in README and docs
• discuss packaging plans / continuous delivery options

[ax3l]

Memo to myself: generalize.

• Control RPATH install (default: on) via option.
• Add JlCxx_location only to openPMD_jl target

[ax3l]

Prepared in #1105

[franzpoeschel]

I have pushed a commit adapting to this change d427b19
Would probably be good if you look at that again @ax3l @eschnett

[ax3l]

That is really cool. Wuhu!

Let me ping potentially @williamfgc, @berceanu and @MaxThevenet as they are excited about Julia as well. Maybe one of you is interested to add a review? :) (Otherwise sorry for the noise, please don't hesitate to unsubscribe from the thread.)

[eschnett]

Regarding packaging:

• Almost all Julia packages are available from their package registry. The Julia package should be split out from this repository and be published there, so that it can be installed in the Julia-usual way. I have begun to move it here.
• Binary dependencies for Julia packages are often published on Yggdrasil. This is in principle a generic package manager, but it's mostly used for Julia. The one defining feature of Yggdrasil is that it delivers binaries that are very quick to install. I have already created an openPMD-api package there. This package does not yet have Julia bindings, though.

I think the best path forward would be to release openPMD-api binaries on Yggdrasil, with the Julia bindings enabled, and to use it from the openPMD.jl Julia package. I'm working on this at the moment.

Regarding examples and tests: Tests are there in the Julia package. Examples are completely missing.

Regarding docs: Simple docs are there (the API is documented). There is no introduction or overview. The general approach of how the C++ functions are mapped to Julia functions is also not yet described.

Regarding CI: The Julia package will have CI, once it's set up properly.

[williamfgc]

@ax3l @eschnett thanks for the great work here. I'll probably be mostly learning since @eschnett did all the things I was planning to use in another project. Just a few things to think about that are collaterals:

• Julia allows pre-compilation for first-time use heavy package with PackageCompiler.jl. Just something to consider if it becomes a problem.
• Jupyter, since NERSC and OLCF have JupyterHub services, the bindings can be tested on both platforms for post-processing. For OLCF some pre-configuration is needed to get Julia going, there is a v1.4 module on Summit, there is a bunch of bug-fixes and improvements with Julia v1.6 (Now v1.7 I believe). Something to consider to establish a minimal version of Julia that is supported.
• Since Python and Julia bindings are both "lightweight" layers on top of C++ (in the runtime sense, this is a great effort). I'd look into performance penalties, if any.

@eschnett for the C++-object to Julia-type wrapper are you just translatin foo.bar(...) into bar(foo, ..)?

[eschnett]

@williamfgc Performance isn't good yet. At the moment, I'm copying data in some places where I really should pass references. This is just because I'm fighting other battles at the moment. I've marked most of the places where performance is bad with TODO.

Yes. foo.bar(...) becomes bar(foo, ...). I've taken the liberty to make a few other changes as well:

• fooBar becomes foo_bar (it's a Julia convention to use lower_case for function names)
• certain functions have no standard name in C++, but have a standard name in Julia. If so, I use these names (e.g. getDimensionality -> ndims, extent -> size, dtype -> eltype)
• setPosition becomes set_position!, with a trailing exclamation mark
• Small arrays (e.g. Extent) become tuples in Julia
• C++ just loves using different integer types for different purposes. These mostly become just Int in Julia.

I want the code to look "natural" in Julia.

[franzpoeschel]

I have pushed three commits:

1. Merged the current dev
2. Set some CMake properties for libopenPMD.jl.so
3. Added a minimal lowlevel test (I will probably expand this one a bit, just want to see what the CI thinks of it)

[franzpoeschel]

I hadn't noticed that you added the minimal lowlevel example, too. I've merged them both now, they now run from within ctest, and the test script is inside the test directory now.

I will expand the test a little and have a more in-depth look at your changes next week.

[franzpoeschel]

My only remaining question before merging this PR is how we should go about documentation?
I feel like we should mark this as experimental until we also have the high-level bindings merged?
Should we add documentation apart from install.rst, too? I think that we can keep it short due to the experimental status.

[eschnett]

That sounds fine. I would point to openPMD.jl as an existing package that uses these bindings, so that people are not put off by the experimental status – it's not openPMD that is experimental, it's only the way things are currently glued together.

[eschnett]

I have grown quite dissatisfied with libcxxwrap_julia. This library is not distributed with Julia, yet it depends on the current Julia version in a fragile way. I have not managed to build a version of the openPMD Julia bindings that reliably work with beta releases or with the development version of Julia. It seems that each new Julia release will require updating the bindings.

At the same time, the bindings produced by this library are not directly usable from Julia. The API it creates does not look "natural" for Julia, and requires another wrapper (written in Julia) to make it usable.

Thus #1537. This creates C bindings for openPMD, and wrapping them from Julia will be about the same amount of work as using libcxxwrap_julia. This approach has worked well for ADIOS2, without any headaches. (Of course there needs to be a C API, and it needs to be maintained, but that's not terribly difficult.)