Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build wheel with Manylinux #473

Merged
merged 4 commits into from
Oct 15, 2023
Merged

Conversation

dvadym
Copy link
Collaborator

@dvadym dvadym commented Oct 10, 2023

This PR implements wheel build with Manylinux for Linux.

Why is it needed?

When a Linux binary is built , it is linked against the standard libraries (libc, libstdc++ etc) of versions that are present on the built machine. Those libraries are forward compatible, namely it's possible to run the binary on machines which have standard libraries newer than ones against which this binary was built. But it's not backward compatible, i.e. the binary might fail on the machines with older libraries with errors like [usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29 not found].

Currently since PyDP is built on Ubuntu 22, it fails to run on many older machines. We can choose Ubuntu 20, but there will be still problems with older machines.

What is the solution?

The common solution for Python packages which have C++ parts are built C++ with Manylinux docker images. Those are Docker images which contain old enough standard libraries, which improves compatibility.

For building cibuildwheel is used. This libraries manages build in Docker with Manylinux images.

Note: many common Python libs (e.g. Numpy, Matplolib) use cibuildwheel. Here) are some other examples.

What in this PR

It contains extending publish workflow with build on manylinux for Linux platform. Other platforms are not changed.
Namely it contains

  1. Introducing build_PyDP_linux.sh which installs bazel and run build_PyDP.sh.
  2. Extending setup.py to call build_PyDP_linux.sh for Linux.
  3. Updating publish.yml to use cibuildwheel on Linux

@dvadym dvadym requested a review from chinmayshah99 October 10, 2023 09:48
Copy link
Member

@chinmayshah99 chinmayshah99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! But for linux, we're creating two bazel cache which we can avoid.

@chinmayshah99 chinmayshah99 merged commit 703c4c4 into OpenMined:dev Oct 15, 2023
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants