Skip to content

Commit

Permalink
Refine binary archive.
Browse files Browse the repository at this point in the history
This streamlines the resulting archive in three ways:

1. It only includes mcrouter and mcpiper from /bin (no thrift, zfmt,
etc)
2. It only includes the libraries that mcrouter and mcpiper actually
link against.  This removes a bunch of boost libraries from the archive.
3. It includes the dynamic libraries not only from INSTALL_DIR, but
also from the host system (e.g. libssl and libz).

This archive is *NOT PORTABLE* in any meaningful way, it is only
meant to run on essentially the same target system, amazon linux 2.
  • Loading branch information
Paul Groudas committed May 21, 2021
1 parent 4a245b9 commit 5fa72da
Show file tree
Hide file tree
Showing 3 changed files with 39 additions and 4 deletions.
2 changes: 1 addition & 1 deletion mcrouter/scripts/INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ N.B These scripts *only* support Amazon Linux 2

MY_INSTALL_DIR=path/to/install/dir
TARGET=mcrouter
./get_and_build_by_make $MY_INSTALL_DIR mcrouter
./get_and_build_by_make.sh $MY_INSTALL_DIR mcrouter

You can substitute individual dependencies as the value of TARGET in order to debug.
9 changes: 6 additions & 3 deletions mcrouter/scripts/Makefile_amazon-linux-2
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ mcrouter: .fbthrift-done .folly-done .fizz-done .wangle-done .fmt-done .zstd-don
${RECIPES_DIR}/mcrouter.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

archive: mcrouter
touch $@
tar czfv "mcrouter.$$(git rev-parse --short=12 HEAD).$$(uname -m).tar.gz" -C $(INSTALL_DIR) aux bin lib lib64 share
MCROUTER_ARCHIVE := mcrouter.$(shell git rev-parse --short=12 HEAD).$(shell uname -m).tar.gz

$(MCROUTER_ARCHIVE): mcrouter
./slim-archive.sh $(INSTALL_DIR) $@

archive: $(MCROUTER_ARCHIVE)
32 changes: 32 additions & 0 deletions mcrouter/scripts/slim-archive.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env bash

# This creates an archive suitable for distribution to other hosts with the same system.

set -euo pipefail

INSTALL_DIR="$1"
ARCHIVE="$2"

function die() {
printf "%s: %s\n" "$0" "$@"
exit 1
}

[ -n "$1" ] || die "INSTALL_DIR missing"
[ -n "$2" ] || die "ARCHIVE missing"

ARCHIVE_INSTALL_DIR=$(mktemp -d)
mkdir "${ARCHIVE_INSTALL_DIR}"/lib
mkdir "${ARCHIVE_INSTALL_DIR}"/bin

# This expression uses "ldd" to locate all dynamically linked library dependencies.
# We then copy them to a tempdir so we can simply create an archive of only the needed dependencies.
# This will suck in libraries both from the "compiled from source" libraries (e.g. folly, thrift) as libraries
# from the host system.

LD_LIBRARY_PATH="${INSTALL_DIR}"/lib ldd "${INSTALL_DIR}"/bin/mcrouter "${INSTALL_DIR}"/bin/mcpiper \
| grep "=>" | tr -s "[:blank:]" " " | cut -d " " -f 4 | sort -u \
| xargs -I '%' cp '%' "${ARCHIVE_INSTALL_DIR}"/lib
cp "${INSTALL_DIR}"/bin/mcrouter "${INSTALL_DIR}"/bin/mcpiper "${ARCHIVE_INSTALL_DIR}"/bin/

tar czfv "${ARCHIVE}" -C "${ARCHIVE_INSTALL_DIR}" .

0 comments on commit 5fa72da

Please sign in to comment.