Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: facebook/mcrouter
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: useshortcut/mcrouter
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.

Commits on Apr 30, 2021

  1. Add top level .gitignore

    This just ignores intelliJ config files
    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    5c4f50e View commit details
  2. Add Amazon Linux 2 Build

    - "Install_amazon-linux-2.sh" will install required build tools and
      development libraries.
    
    - Adds "Makefile_amazon-linux-2" to describe the required dependency
      compilation order.
    
    - Updates individual recipes for compiling each dependency to generally
      be more consistent.
    
    - Fixes all dependencies to a specific version.
    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    065f705 View commit details
  3. Remove unused scripts.

    We only support Amazon Linux 2
    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    3b5fa20 View commit details
  4. Document build instructions.

    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    f257238 View commit details
  5. Update README

    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    fc04355 View commit details
  6. Reset to specific tags / revisions

    This just ensures that the way we manipulate the dependencies' git
    repositories is consistent.
    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    c2a07bc View commit details
  7. make target to create archive of binary artifacts

    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    5779ea5 View commit details
  8. Set boost libdir for mcrouter build

    The mcrouter configure script does not look for boost via
    LD_LIBRARY_PATH nor CPPFLAGS '-H' value, instead it has a dedicated
    cmdline option.
    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    588107e View commit details
  9. Fail on systems with insufficient memory.

    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    a7f2c03 View commit details
  10. Fix typo in comment.

    Paul Groudas committed Apr 30, 2021
    Copy the full SHA
    2f33be1 View commit details
  11. Copy the full SHA
    c798225 View commit details

Commits on May 3, 2021

  1. Fix Makefile

    I am unsure as to why, but the dependency expressed as "mcrouter: deps"
    doesn't seem to work correctly.
    
    When invoking the equivalent of "make mcrouter", I expect it to build
    all the transitive dependencies of "deps", but it only builds the first
    dependency that it determines it needs to build, and then exits with
    a return value of "2".
    
    I'm just reverting this to the previous style that is redundant but
    works correctly.
    Paul Groudas committed May 3, 2021
    Copy the full SHA
    b79d697 View commit details
  2. "git clean" during checkout of dependency

    This repo defines "recipes" for dependencies which are essentially build
    scripts.  Some of the build systems used (or at least the way they are
    used) will cache some state between invocations, which is confusing when
    testing different values for things like "LDFLAGS" and
    "LD_LIBRARY_PATH".  Specifically, the cmake based build scripts.
    
    This change ensures that when force re-running a given recipe, we do a
    full clean of the repo.
    
    n.b.: we are *not* cleaning the target install dir.
    Paul Groudas committed May 3, 2021
    Copy the full SHA
    abbe369 View commit details
  3. Build jemalloc from source.

    This introduces "jemalloc" as a library dependency that we build from
    source.
    
    While this library is available from our runtime system's package
    respository, mcrouter would log an error at startup implying that there
    is a page size mis-match between the current kernel, and the system on
    which jemalloc was compiled.
    
    As this package available via the EPEL[1] and not Amazon Linux directly,
    this isn't entirely surprising.
    
    Compiling "jemalloc" is very straightforward, but getting all the
    dependencies to correctly link against it during their builds was a bit
    challenging.  Ultimately I made the following changes:
    
    1. The default value we set for *all* recipes is:
    
            LDFLAGS="-L$INSTALL_DIR/lib -ljemalloc
    
    - "-ljemalloc" means "link against the 'jemalloc'" library.
    - "-L$INSTALL_DIR/lib" search "$INSTALL_DIR/lib" when looking for
    libraries.
    
    2. I removed redundant LDFLAGS from each recipe following the change to
    the default.
    
    3. Some builds, notable those that produce binary executables (not just
    shared libraries), also need to include "-Wl,-rpath=$INSTALL_DIR/lib",
    which embeds metadata about the dynamic libraries' location into the
    executable.  Conceptually, this doesn't make sense to me, as we are
    expecting users to manage their own library path, but not setting this
    causes the builds to fail, and I have a limited appetite for debugging
    and understanding these build tools.
    
    [1]: https://fedoraproject.org/wiki/EPEL
    Paul Groudas committed May 3, 2021
    Copy the full SHA
    06fc230 View commit details
  4. Merge pull request #2 from clubhouse/pgroudas/compile-jemalloc-from-s…

    …ource
    Paul Groudas authored May 3, 2021
    Copy the full SHA
    064eaac View commit details

Commits on May 17, 2021

  1. Use relative path in symlink

    Specifying the absolute path as the target to the symlink causes
    portability problems because when we eventually archive up the result
    for distribution, the absolute paths will be incorrect on the target
    systems.
    Paul Groudas committed May 17, 2021
    Copy the full SHA
    656dd4e View commit details
  2. Exclude "include/" distributable archive.

    We are really only packaging the runtime dependencies, we do not need
    the development headers.
    
    This saves ~200mb on the target systems.
    Paul Groudas committed May 17, 2021
    Copy the full SHA
    4a245b9 View commit details

Commits on May 21, 2021

  1. Refine binary archive.

    This streamlines the resulting archive in three ways:
    
    1. It only includes mcrouter and mcpiper from /bin (no thrift, zfmt,
    etc)
    2. It only includes the libraries that mcrouter and mcpiper actually
    link against.  This removes a bunch of boost libraries from the archive.
    3. It includes the dynamic libraries not only from INSTALL_DIR, but
    also from the host system (e.g. libssl and libz).
    
    This archive is *NOT PORTABLE* in any meaningful way, it is only
    meant to run on essentially the same target system, amazon linux 2.
    Paul Groudas committed May 21, 2021
    Copy the full SHA
    5fa72da View commit details

Commits on May 24, 2021

  1. [mcrouter] adjust tarball construction

    `tar cf "ARCHIVE_FILE" .` has a strange side effect that it includes a
    reference to the current dir (which given the syntax, isn't unexpected I
    guess), but in practice that mangles the paths in the archive to prefix
    them all with "./" and confounds the permissions on the target directory
    when you extract the archive.
    Paul Groudas committed May 24, 2021
    Copy the full SHA
    a9db3dd View commit details

Commits on Sep 30, 2021

  1. Updating makefile

    virgofx committed Sep 30, 2021
    Copy the full SHA
    cfb5953 View commit details
  2. Copy the full SHA
    fb1e3df View commit details
  3. Update fbthrift

    virgofx committed Sep 30, 2021
    Copy the full SHA
    9fea264 View commit details
  4. Update epel deps

    virgofx committed Sep 30, 2021
    Copy the full SHA
    7716e49 View commit details

Commits on Oct 1, 2021

  1. Copy the full SHA
    61b3519 View commit details
  2. Merge pull request #7 from useshortcut/mark/sc-170835/fix-mcrouter-in…

    …-latest-ami-v2
    
    Update Mcrouter to work with latest Linux Amazon 2
    virgofx authored Oct 1, 2021
    Copy the full SHA
    4179985 View commit details

Commits on Aug 15, 2022

  1. Copy the full SHA
    78038ff View commit details
  2. Copy the full SHA
    8658bce View commit details
Showing with 408 additions and 631 deletions.
  1. +1 −0 .gitignore
  2. +11 −20 README.md
  3. +1 −0 mcrouter/Makefile.am
  4. +1 −0 mcrouter/lib/network/test/Makefile.am
  5. +49 −0 mcrouter/scripts/INSTALL.md
  6. +74 −0 mcrouter/scripts/Makefile_amazon-linux-2
  7. +0 −41 mcrouter/scripts/Makefile_ubuntu-18.04
  8. +0 −41 mcrouter/scripts/Makefile_ubuntu-20.04
  9. +0 −31 mcrouter/scripts/clean_ubuntu_14.04.sh
  10. +12 −2 mcrouter/scripts/common.sh
  11. +0 −17 mcrouter/scripts/docker/Dockerfile
  12. +24 −7 mcrouter/scripts/get_and_build_by_make.sh
  13. +0 −28 mcrouter/scripts/get_and_build_everything.sh
  14. +0 −45 mcrouter/scripts/install_centos_7.2.sh
  15. +63 −0 mcrouter/scripts/install_deps_amazon-linux-2.sh
  16. +0 −60 mcrouter/scripts/install_ubuntu_14.04.sh
  17. +0 −43 mcrouter/scripts/install_ubuntu_15.04.sh
  18. +0 −45 mcrouter/scripts/install_ubuntu_16.04.sh
  19. +0 −48 mcrouter/scripts/install_ubuntu_18.04.sh
  20. +0 −49 mcrouter/scripts/install_ubuntu_20.04.sh
  21. +0 −1 mcrouter/scripts/order_centos-7.2/10_folly
  22. +0 −1 mcrouter/scripts/order_centos-7.2/11_fizz
  23. +0 −1 mcrouter/scripts/order_centos-7.2/12_wangle
  24. +0 −1 mcrouter/scripts/order_centos-7.2/13_rsocket-yarpl
  25. +0 −1 mcrouter/scripts/order_centos-7.2/15_fbthrift
  26. +0 −1 mcrouter/scripts/order_centos-7.2/20_mcrouter
  27. +0 −11 mcrouter/scripts/order_ubuntu-14.04/09_openssl
  28. +0 −1 mcrouter/scripts/order_ubuntu-14.04/10_folly
  29. +0 −1 mcrouter/scripts/order_ubuntu-14.04/11_fizz
  30. +0 −1 mcrouter/scripts/order_ubuntu-14.04/12_wangle
  31. +0 −1 mcrouter/scripts/order_ubuntu-14.04/13_rsocket-yarpl
  32. +0 −1 mcrouter/scripts/order_ubuntu-14.04/14_fmt
  33. +0 −1 mcrouter/scripts/order_ubuntu-14.04/15_fbthrift
  34. +0 −1 mcrouter/scripts/order_ubuntu-14.04/16_gtest
  35. +0 −1 mcrouter/scripts/order_ubuntu-14.04/20_mcrouter
  36. +0 −1 mcrouter/scripts/order_ubuntu-15.04/10_folly
  37. +0 −1 mcrouter/scripts/order_ubuntu-15.04/11_fizz
  38. +0 −1 mcrouter/scripts/order_ubuntu-15.04/12_wangle
  39. +0 −1 mcrouter/scripts/order_ubuntu-15.04/13_rsocket-yarpl
  40. +0 −1 mcrouter/scripts/order_ubuntu-15.04/14_fmt
  41. +0 −1 mcrouter/scripts/order_ubuntu-15.04/15_fbthrift
  42. +0 −1 mcrouter/scripts/order_ubuntu-15.04/16_gtest
  43. +0 −1 mcrouter/scripts/order_ubuntu-15.04/20_mcrouter
  44. +0 −1 mcrouter/scripts/order_ubuntu-16.04/10_folly
  45. +0 −1 mcrouter/scripts/order_ubuntu-16.04/11_fizz
  46. +0 −1 mcrouter/scripts/order_ubuntu-16.04/12_wangle
  47. +0 −1 mcrouter/scripts/order_ubuntu-16.04/13_rsocket-yarpl
  48. +0 −1 mcrouter/scripts/order_ubuntu-16.04/14_fmt
  49. +0 −1 mcrouter/scripts/order_ubuntu-16.04/15_fbthrift
  50. +0 −1 mcrouter/scripts/order_ubuntu-16.04/16_gtest
  51. +0 −1 mcrouter/scripts/order_ubuntu-16.04/20_mcrouter
  52. +19 −0 mcrouter/scripts/recipes/boost.sh
  53. +12 −10 mcrouter/scripts/recipes/fbthrift.sh
  54. +7 −10 mcrouter/scripts/recipes/fizz.sh
  55. +8 −5 mcrouter/scripts/recipes/fmtlib.sh
  56. +10 −36 mcrouter/scripts/recipes/folly.sh
  57. +24 −0 mcrouter/scripts/recipes/gflags.sh
  58. +12 −8 mcrouter/scripts/recipes/glog.sh
  59. +0 −18 mcrouter/scripts/recipes/gtest.sh
  60. +19 −0 mcrouter/scripts/recipes/jemalloc.sh
  61. +3 −3 mcrouter/scripts/recipes/mcrouter.sh
  62. +0 −13 mcrouter/scripts/recipes/python.sh
  63. +7 −10 mcrouter/scripts/recipes/wangle.sh
  64. +19 −0 mcrouter/scripts/recipes/zstd.sh
  65. +32 −0 mcrouter/scripts/slim-archive.sh
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/.idea/
31 changes: 11 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -12,27 +12,9 @@ See https://github.com/facebook/mcrouter/wiki to get started.

## Quick start guide

### New! Ubuntu package available

Currently, we support Ubuntu Bionic (18.04) amd64.
Here is how to install it:

Add the repo key:

$ wget -O - https://facebook.github.io/mcrouter/debrepo/bionic/PUBLIC.KEY | sudo apt-key add

Add the following line to apt sources file /etc/apt/sources.list

deb https://facebook.github.io/mcrouter/debrepo/bionic bionic contrib

Update the local repo cache:

$ sudo apt-get update

Install mcrouter:

$ sudo apt-get install mcrouter
### Building on Amazon Linux 2

See [INSTALL.md](/mcrouter/scripts/INSTALL.md)

### Installing From Source

@@ -87,6 +69,15 @@ the simplest mcrouter setup is:
Documentation: https://github.com/facebook/mcrouter/wiki
Engineering discussions and support: https://www.facebook.com/groups/mcrouter

## Development

This is an internal fork, aimed only to add packaging support for Amazon Linux 2.

Branches:

- upstream: This should track facebook/master
- main: PRs can target this for review

## License

Copyright (c) Facebook, Inc. and its affiliates.
1 change: 1 addition & 0 deletions mcrouter/Makefile.am
Original file line number Diff line number Diff line change
@@ -242,6 +242,7 @@ mcrouter_LDADD = \
-ltransport \
-lthriftprotocol \
-lrpcmetadata \
-lthriftmetadata \
-lasync \
-lconcurrency \
-lthrift-core \
1 change: 1 addition & 0 deletions mcrouter/lib/network/test/Makefile.am
Original file line number Diff line number Diff line change
@@ -20,6 +20,7 @@ mock_mc_server_LDADD = \
-ltransport \
-lthriftprotocol \
-lrpcmetadata \
-lthriftmetadata \
-lasync \
-lconcurrency \
-lthrift-core \
49 changes: 49 additions & 0 deletions mcrouter/scripts/INSTALL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Mcrouter Installation on Amazon Linux 2

We need to do this for both x86_64 architecture and graviton. Repeat the following steps
for each architecture

1. Launch a temporary instance for each x86_64 and ARM instance type and use Session Manager to connect.
- Amazon Linux 2 AMI [ebs, hvm]
- x86: m5ad.2xlarge
- arm: m6gd.2xlarge

1. Switch to root for all subsequent commands
```shell
sudo su -
```

1. Attach the EBS volume. Mcrouter needs a little more space than default 8GB volume.
```shell
file -s /dev/nvme1n1
mkfs -t xfs /dev/nvme1n1
mkdir /build
mount /dev/nvme1n1 /build
```

1. Clone the `useshortcut\mcrouter` repo.
```shell
yum -y update
yum -y install git
cd /build
git clone https://github.com/useshortcut/mcrouter.git
cd mcrouter
# Switch to branch if necessary
# git checkout <branch>
```

1. Install Pre-built Dependencies
```shell
cd /build/mcrouter/mcrouter/scripts
./install_deps_amazon-linux-2.sh
```

1. Compile additional dependencies, mcrouter and then archive
```shell
cd /build/mcrouter/mcrouter/scripts
INSTALL_DIR=/build/output
# Build dependencies and mcrouter
./get_and_build_by_make.sh $INSTALL_DIR all
# Now archive
./get_and_build_by_make.sh $INSTALL_DIR archive
```
74 changes: 74 additions & 0 deletions mcrouter/scripts/Makefile_amazon-linux-2
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Copyright (c) Facebook, Inc. and its affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

RECIPES_DIR := ./recipes
MCROUTER_ARCHIVE := mcrouter.$(shell git rev-parse --short=12 HEAD).$(shell uname -m).tar.gz

deps: .jemalloc-done .boost-done .zstd-done .gflags-done .glog-done .fmt-done .folly-done .fizz-done .wangle-done .fbthrift-done
touch $@

mcrouter: deps
${RECIPES_DIR}/mcrouter.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

all: .jemalloc-done .boost-done .zstd-done .gflags-done .glog-done .fmt-done .folly-done .fizz-done .wangle-done .fbthrift-done mcrouter
touch $@

archive:
./slim-archive.sh $(INSTALL_DIR) $(MCROUTER_ARCHIVE)

# jemalloc is available from the package repositories, but then we get this warning at mcrouter startup:
# <jemalloc>: Error in munmap(): Invalid argument which corresponds to https://github.com/jemalloc/jemalloc/issues/467
# As we build on the same system we deploy to (our amazon linux AMI), I hope that this results with consistent
# page sizes and no error.
.jemalloc-done:
# We set "LDFLAGS" in the "get_and_build_by_make.sh" script that tells all
# projects to link against jemalloc. We unset that when we actually compile jemalloc.
LDFLAGS="" ${RECIPES_DIR}/jemalloc.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

# Boost available from Amazon Linux is not recent enough.
# There *are* more recent versions available via EPEL, but they introduce python dependency conflicts.
.boost-done: .jemalloc-done
${RECIPES_DIR}/boost.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

.zstd-done: .jemalloc-done .boost-done
${RECIPES_DIR}/zstd.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

# The version of gflags in the repo is insufficient for compiling mcrouter
.gflags-done: .boost-done .jemalloc-done .zstd-done
${RECIPES_DIR}/gflags.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

# The version of glog in the repo is insufficient for compiling folly.
# Additionally, the most recent (HEAD / v0.5.0.rc2) revision from upstream is incompatible for compiling mcrouter.
# Fortunately, v0.4.0 works for both, so we just check out that revision.
.glog-done: .gflags-done .boost-done .jemalloc-done
${RECIPES_DIR}/glog.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

# The version of fmt from the package repositories is insufficient to satisfy the "folly" build.
.fmt-done: .jemalloc-done .boost-done .glog-done .gflags-done
${RECIPES_DIR}/fmtlib.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

.folly-done: .jemalloc-done .boost-done .fmt-done .glog-done .gflags-done
${RECIPES_DIR}/folly.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

.fizz-done: .boost-done .jemalloc-done .folly-done .glog-done .gflags-done
${RECIPES_DIR}/fizz.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

.wangle-done: .folly-done .fizz-done .glog-done .gflags-done .boost-done .jemalloc-done
# Wangle explicity depends on Fizz. Keep these versions aligned
${RECIPES_DIR}/wangle.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@

.fbthrift-done: .folly-done .fizz-done .wangle-done .fmt-done .glog-done .gflags-done .boost-done .jemalloc-done
${RECIPES_DIR}/fbthrift.sh $(PKG_DIR) $(INSTALL_DIR) $(INSTALL_AUX_DIR)
touch $@
41 changes: 0 additions & 41 deletions mcrouter/scripts/Makefile_ubuntu-18.04

This file was deleted.

41 changes: 0 additions & 41 deletions mcrouter/scripts/Makefile_ubuntu-20.04

This file was deleted.

31 changes: 0 additions & 31 deletions mcrouter/scripts/clean_ubuntu_14.04.sh

This file was deleted.

14 changes: 12 additions & 2 deletions mcrouter/scripts/common.sh
Original file line number Diff line number Diff line change
@@ -7,17 +7,27 @@ set -ex

function die { printf "%s: %s\n" "$0" "$@"; exit 1; }

# Ensures a git repository working tree is set to the specific commit
function gitEnsureTreeish {
# In the case of a commit or tag, this will put the repo into DETACHED HEAD state
git checkout "$1" --
# Ensures that there are edited files in git index or working directory
git reset --hard HEAD
git clean -fdx .
}

[ -n "$1" ] || die "PKG_DIR missing"
[ -n "$2" ] || die "INSTALL_DIR missing"

PKG_DIR="$1"
INSTALL_DIR="$2"
INSTALL_AUX_DIR="$3"
shift $#
MAKE_ARGS="$@"

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
export SCRIPT_DIR

mkdir -p "$PKG_DIR" "$INSTALL_DIR"
mkdir -p "$PKG_DIR" "$INSTALL_DIR"

if [ -n "$3" ]; then
mkdir -p "$INSTALL_AUX_DIR"
17 changes: 0 additions & 17 deletions mcrouter/scripts/docker/Dockerfile

This file was deleted.

Loading