You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a use case for lakectl running in glibc based containers. glibc containers are required because we'll be running pytorch.
Although ChatGPT is suggesting pytorch can be installed on alpine by installing glibc on alpine, I am not ready to introduce that complication if it is unnecessary. In particular, I have a feeling leveraging a GPU from alpine w/ pytorch could be nearly impossible, and, even if it is possible, solutions would tend to be unique across GPU versions, driver versions, and glibc/alpine versions.
So, we are going to prefer glibc images as base images. But, we still want to use LakeFS. The main problem is that while musl images work out of the box on arm64, they do not work at all on x86_64 base images. I have a few solutions:
wget the correct binaries from github during the build. This is serviceable, but problematic in some contexts.
git-lfs store the correct binaries in the project being built
artifactory distribute the binaries from our artifactory
Or, build the lakeFS image in our normal workflow. Of all the options, building the LakeFS image in our normal workflow seems to be the lowest touch, most robust, and simplest solution over the next few years.
So the current Dockerfile only builds for alpine in linux-musl-amd64 architecture. It'll have to be generalized with build-args to allow for other platforms, such as glibc, to be used.
The text was updated successfully, but these errors were encountered:
We have a use case for
lakectl
running inglibc
based containers.glibc
containers are required because we'll be runningpytorch
.Although ChatGPT is suggesting
pytorch
can be installed onalpine
by installingglibc
onalpine
, I am not ready to introduce that complication if it is unnecessary. In particular, I have a feeling leveraging a GPU fromalpine
w/pytorch
could be nearly impossible, and, even if it is possible, solutions would tend to be unique across GPU versions, driver versions, and glibc/alpine versions.So, we are going to prefer
glibc
images as base images. But, we still want to use LakeFS. The main problem is that whilemusl
images work out of the box onarm64
, they do not work at all onx86_64
base images. I have a few solutions:wget
the correct binaries from github during the build. This is serviceable, but problematic in some contexts.git-lfs
store the correct binaries in the project being builtartifactory
distribute the binaries from our artifactoryOr, build the lakeFS image in our normal workflow. Of all the options, building the LakeFS image in our normal workflow seems to be the lowest touch, most robust, and simplest solution over the next few years.
So the current
Dockerfile
only builds foralpine
inlinux-musl-amd64
architecture. It'll have to be generalized withbuild-args
to allow for other platforms, such asglibc
, to be used.The text was updated successfully, but these errors were encountered: