[WIP] pybind11 wrapper for LandmarkDetector

[a-hurst]

As mentioned in issue #849. Could use a bit of cleanup before being merged, and would ideally be expanded to bind the GazeAnalyzer and FaceAnalyzer modules as well, but it installs easily and works fairly nicely for my uses.

Installation

Currently, the Python API isn't part of the main CMakeLists.txt, and won't be installed when doing a normal build of OpenFace. To build/install it, first do a regular build of OpenFace using the normal instructions, then cd into the python_api folder and run python setup.py install (installation with pip should also work). This should install the Python module 'pyopenface'.

Of course, OpenFace isn't much use without the required model files. To facilitate installation and distribution, I've written a module for pyopenface that provides two main functions: download_models() and install_models().

• download_models() downloads all the required models to run OpenFace (both the CEN patches and the ones included in the OpenFace git) to a directory called "openface_models" in the current working directory. pyopenface will check for this folder in the cwd on launch and, if it exists, loads the models from there.

• install_models() does the same thing as download_models(), but instead downloads the models to an invisible folder ".openface" in the root of the current user's home directory. Once this is run, pyopenface can be used from any path on the system without issue. An "openface_models" folder in the cwd will take precedence, however (in case you want tweaked models for a specific project).

Separating the models from the rest of the package this way dramatically cuts down on the size of the pyopenface install, making building and distributing pre-build wheels via PyPi a lot easier once the rest of the bindings are in good shape.

Basic structure

The bindings are broken into two main parts: the actual pybind11 bindings (in src/openface.cpp) and a pure-Python wrapper for those low-level bindings (in pyopenface/openface.py). I did things this way because type-checking for things like numpy arrays and arguments is a lot easier (for me, anyway) in pure Python, and because it lets me keep the pybind11 wrapper a pretty direct wrapping for the OpenFace API while allowing the user-facing API to be a little friendlier and more Pythonic.

Caveats

I've only tested this on macOS 10.14, with Python 3.7 installed via Homebrew. Other platforms and older versions of Python 3 (and maybe even 2.7) might very well work, but I haven't tested them out. Ideally if this ever gets to a PyPi-ready state, we can set up CI across platforms and Python versions using cibuildwheel or another similar tool.

General Notes

This project is the first time I've ever worked with pybind11, CMake, or C++. As such, I've probably made some major rookie mistakes that hopefully people with some more experience can spot.

The module should work fine with any package whose output you can coerce into a numpy array (e.g. Pillow, PyOpenCV). I need to add better type-checking for the input arrays, since the pybind11 bindings only accept uint8 images for the time being.

Currently, the FaceModelParameters object doesn't seem to have a way of specifying a non-standard path for both the HAAR and MTCNN classifiers on init, so it prints out

Could not find the HAAR face detector location
Could not find the MTCNN face detector location

when created from Python. The Python wrapper sets the correct paths after initialization, but the C++ code itself would need to be changed to fix this. Similarly, creating a CNLF object dumps a whole bunch of text into the console that I don't see any way to disable.

Let me know what you think!

[johnbone25]

Is it possible to invoke the head position or eye tracking in Python? And how to do?
Thanks

[a-hurst]

Is it possible to invoke the head position or eye tracking in Python? And how to do?
Thanks

By 'head position tracking' do you mean 3D orientation or just landmarks? The only part I needed for my project was the landmarks so I only really wrapped enough of the API to do that, but given that the core of the Python API is in-place, adding wrapping the head orientation and eye orientation APIs shouldn't be terribly difficult.

I'd be willing to work on it myself if the developers were interested in reviewing the basics of my PR here, but sadly it looks like they've been too busy to work on it for the past year and their custom open-source licence is restrictive/threatening enough that I wouldn't risk making a proper fork of the project.

[johnbone25]

@a-hurst I basically need gaze tracking in Python to check if the face is looking up, down, right or left using gaze tracking or head pose tracking.
How could I do? Thanks so much

[a-hurst]

@johnbone25 The face/head itself, or just the eyes? If you're trying to get head orientation (from photos or videos?), you might be able to infer that using the relative distances of the facial landmarks (i.e., if the points on the left side are spaced out further than the points on the right, they're looking towards the right).

If you need full head pose and eye gaze detection via PyOpenFace, you would need to wrap the relevant C++ functions from OpenFace in the python_api/src/openface.cpp file and add corresponding Python functions in `python_api/pyopenface/openface.py'. You can look at how I implemented the LandmarkDetection functions for an illustration of how to do that, a lot of it would just be copy/pasting and reworking definitions.

That said, in my (limited) testing OpenFace's eye gaze estimation seemed pretty noisy and erratic (at least for the video input i was giving it), so if that's your goal you may want to look into some other options in case they fit your use-case better.

Hope that helps!

[johnbone25]

@a-hurst my task is from photos. I need to know where the person looks (up, down, right or left), so face/head itself or just the eyes are good for me (I'll use the one that works best so either one).
What do you suggest me? Openface or an another good system?

[a-hurst]

@a-hurst I'm not fully sure what the current alternatives are, but you could give OpenFace a shot! First, I'd recommend trying out the command-line version of OpenFace on some files first, just to get a sense of how well it works for your use-case. It's not super well-documented, but it does provide useful illustrations of what's being detected and print out a .csv with the relevant data. If that works well enough, you can try these Python bindings out for easier scripting.

What platform are you running on? I've never tested these bindings on Windows, but they should run on Linux and macOS without issues.

[johnbone25]

@a-hurst I am running it on Windows. Ok, but how can i make bindings with python? I don't understand well your message on March 30, 2020

[johnbone25]

@a-hurst i obtained this values (namely gaze_angle, pose_T and so on) but how can i use them to understand if looking right, left, down or up? What are the intervals to determine if person looks right, left, up or down?

[a-hurst]

@johnbone25 Did you look through this page of the wiki yet? That explains the output columns of OpenFace in detail. Using the visualizations there you should be able to figure out what you'd need to get head position from an image.

As for making the Python bindings, I think you'll need to be able to build OpenFace from source on Windows first. The instructions for this should be on the Wiki. Then, in a terminal, cd into the python_api folder that I added in my fork of OpenFace and run python setup.py install to build and install the Python bindings. Again, I have zero clue if this will work on Windows at all, so you may find it easier to use Linux on a VM or spare computer for this project!

[johnbone25]

@a-hurst Yes, but these values in the image below, which CSV values exactly correspond to or these values are obtained from calculations taking into account gaze_0_x, gaze_0_y, gaze_0_z, gaze_angle_x, gaze_angle_y? If so how can you get the below image values from the values in the CSV? Thanks so much

[a-hurst]

@johnbone25 Do you have the OpenFace command-line tools set up to give you an image file as well as the CSV? It should have an option to output the image with all the landmark/gaze direction/head position overlays on top. Then, you should be able to feed it some prototype images of people looking in different directions and figure out which values of pos_Rx/pos_Ry/pos_Rz correspond to people looking in those directions.

Same thing with the gaze values, though again I'm not sure how much I trust them via OpenFace: unless the lighting and image resolution is really good, eye position estimation is not going to be all that great or stable. I've worked with research-grade camera-based eye tracking equipment before and that still sometimes has difficulty inferring gaze direction if conditions aren't ideal, so sanity-checking your images with the OpenFace overlays is probably a good idea.

[ghost]

#750 #1001 #858 build ALL THE NEEDED things and files for patched and the secured links connected into my self (Sofia Sköld) and all of my females workflow must be encoded linked and make all copies and version copyright protection for only use or make control remotely connections to all the females girls between Ambjörn Sköld or any devices or phones etc. All this is only for private use or any actions as well as changes or copying this work or samples.