Support Conda on macOS; limit gym to 0.21.0; and other changes (#313)

* Support Conda on macOS * Multiple changes: - Improve development installation, - Fix Boost.Bind compilation warnings, - Replace scipy dependency with numpy, - Add GitHub workflow to build on macOS. * Use gym<=0.21.0 * Make windows-2016 builds experimental, to avoid failure
google-research · Apr 19, 2022 · 9a9e35b · 9a9e35b
1 parent ba99521
commit 9a9e35b
Show file tree

Hide file tree

Showing 18 changed files with 250 additions and 121 deletions.
diff --git a/.github/workflows/build-mac.yml b/.github/workflows/build-mac.yml
@@ -0,0 +1,34 @@
+name: Build gfootball on Mac
+
+on:
+  push:
+    branches:
+      - master
+    paths-ignore:
+      - 'gfootball/doc/**'
+      - '*.md'
+      - '.dockerignore'
+      - '.gitignore'
+  workflow_dispatch:
+
+jobs:
+  build-mac:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ macos-10.15, macos-latest ]
+    steps:
+      - name: Check out football repository
+        uses: actions/checkout@v2
+
+      - name: Install brew dependencies
+        run: brew install python3 cmake sdl2 sdl2_image sdl2_ttf sdl2_gfx boost boost-python3
+
+      - name: Upgrade pip and install dependencies
+        run: |
+          python3 -m pip install --upgrade pip setuptools wheel build psutil
+          python3 -m pip install -r requirements.txt
+
+      - name: Build package
+        run: python3 -m build --no-isolation --wheel
diff --git a/.github/workflows/publish-wheels.yml b/.github/workflows/publish-wheels.yml
@@ -8,17 +8,38 @@ on:
 
 jobs:
   build-windows:
-    # TODO: Change back to windows-latest once the SDK problem is resolved
-    runs-on: windows-2016
+    runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ '3.6', '3.7', '3.8', '3.9', '3.10' ]
+        os: [windows-latest]
+        python-version: [ '3.9', '3.10' ]
         architecture: ['x86', 'x64']
-        # SciPy doesn't provide a wheel for Python 3.10 x86
-        exclude:
-          - python-version: '3.10'
+        include:
+          - os: windows-2016
+            python-version: '3.6'
             architecture: 'x86'
+            experimental: true
+          - os: windows-2016
+            python-version: '3.6'
+            architecture: 'x64'
+            experimental: true
+          - os: windows-2016
+            python-version: '3.7'
+            architecture: 'x86'
+            experimental: true
+          - os: windows-2016
+            python-version: '3.7'
+            architecture: 'x64'
+            experimental: true
+          - os: windows-2016
+            python-version: '3.8'
+            architecture: 'x86'
+            experimental: true
+          - os: windows-2016
+            python-version: '3.8'
+            architecture: 'x64'
+            experimental: true
 
     env:
       VCPKG_ROOT: "${{ github.workspace }}/vcpkg"
@@ -35,7 +56,7 @@ jobs:
         python-version: ${{ matrix.python-version }}
         architecture: ${{ matrix.architecture }}
 
-    - name: Upgrade pip and essential dependencies
+    - name: Upgrade pip and install dependencies
       run: |
         python -m pip install --upgrade pip
         python -m pip install -U build setuptools psutil wheel

diff --git a/.gitignore b/.gitignore
@@ -1,19 +1,29 @@
-third_party/gfootball_engine/build*
+# When updating this file don't forget to update MANIFEST.in
+
+# Python related files
 build/
-*.egg-info
+*.egg-info/
 __pycache__
+dist/
+# virtual environment
+football-env*/
+
+# CMake related directories and files
+third_party/gfootball_engine/build*
+third_party/gfootball_engine/CMakeFiles/
+third_party/gfootball_engine/Testing/
+third_party/gfootball_engine/.cmake/
+cmake-build-*/
+vcpkg_installed/
+C[M,m]akeCache.txt
+Makefile
+cmake_install.cmake
 *.a
 *.so
 *.dylib
-C[M,m]akeCache.txt
-third_party/gfootball_engine/CMakeFiles/
-third_party/gfootball_engine/Makefile
-third_party/gfootball_engine/cmake_install.cmake
-third_party/gfootball_engine/gameplayfootball
-footballenv/
-football-env*/
-cmake-build-debug/
+*.pyd
+*.cbp
+
+# IDE
 .idea
-.vs
-vcpkg_installed/
-dist/
+.vs*
diff --git a/CHANGELOG b/CHANGELOG
@@ -5,6 +5,14 @@ should not change, as modifications made to the environment are either
 new features or backward compatible bug fixes. We will maintain vX branches
 pointing at the most recent vX.Y.
 
+v2.10.3
+- Add Conda support on macOS.
+- Improve development setup on all platforms (symlink instead of copying).
+- Fix Boost.Bind compilation warnings.
+- Replace scipy dependency with numpy.
+- Limit Gym version to v0.21.0 for now.
+- Add GitHub workflow for building on macOS.
+
 v2.10.2
 - Replace deprecated FindPythonLibs CMake module.
 - Support Apple Silicon.

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -4,13 +4,14 @@ recursive-include gfootball/doc *
 recursive-include gfootball/eval_server/proto *
 recursive-include gfootball/examples *
 
-include Dockerfile .dockerignore CHANGELOG CONTRIBUTING.md run_docker_test.sh
+include Dockerfile Dockerfile_examples .dockerignore CHANGELOG CONTRIBUTING.md requirements.txt run_docker_test.sh
 
 recursive-include third_party/fonts *
 recursive-include third_party/gfootball_engine *
-recursive-exclude third_party/gfootball_engine *.a libgame.so cmake_install.cmake Makefile
+recursive-exclude third_party/gfootball_engine *.a *.so cmake_install.cmake Makefile
+
 prune third_party/gfootball_engine/CMakeFiles
-prune third_party/gfootball_engine/cmake-build-debug
+prune third_party/gfootball_engine/cmake-build-*
 prune third_party/gfootball_engine/build*
-prune third_party/gfootball_engine/.vs
+prune third_party/gfootball_engine/.vs*
 prune third_party/gfootball_engine/.idea
diff --git a/README.md b/README.md
@@ -25,7 +25,7 @@ This method doesn't support game rendering on screen - if you want to see the ga
 
 ### Using Docker
 
-This is the recommended way to avoid incompatible package versions.
+This is the recommended way for Linux-based systems to avoid incompatible package versions.
 Instructions are available [here](gfootball/doc/docker.md).
 
 ### On your computer
@@ -66,7 +66,7 @@ python3 -m pip install gfootball
 ```
 
 #### Option b. Installing from sources using GitHub repository 
-(On Windows you have to install additional tools and set environment variables, see 
+(On Windows you have to install additional tools and set an environment variable, see 
 [Compiling Engine](gfootball/doc/compile_engine.md#windows) for detailed instructions.)
 
 ```shell

diff --git a/gfootball/build_game_engine.sh b/gfootball/build_game_engine.sh
@@ -21,6 +21,7 @@ if [[ "$OSTYPE" == "darwin"* ]] ; then
 fi
 
 # Take into account # of cores and available RAM for deciding on compilation parallelism.
+# TODO: Try importing psutil and if failed fall back to 1 thread
 PARALLELISM=$(python3 -c 'import psutil; import multiprocessing as mp; print(int(max(1,min((psutil.virtual_memory().available/1000000000-1)/0.5, mp.cpu_count()))))')
 
 # Delete pre-existing version of CMakeCache.txt to make 'python3 -m pip install' work.

diff --git a/gfootball/doc/compile_engine.md b/gfootball/doc/compile_engine.md
@@ -58,6 +58,7 @@ python -m pip install .
 
 ## macOS (both Intel processors and Apple Silicon)
 
+### Install prerequisites
 First, install [brew](https://brew.sh/). It should automatically download Command Line Tools.
 Next, install the required packages:
 ```shell
@@ -72,41 +73,40 @@ git clone https://github.com/google-research/football.git
 cd football
 ```
 
-### Installation with brew version of Python
-It is recommended to use Python shipped with `brew`, because `boost-python3` is compiled against the same version.
-To check which Python 3 is used by default on your setup, execute `which python3`.
-If you have a different path, and you don't want to change symlinks, create a virtual environment with
-`/usr/local/bin/python3 -m venv football-env` or `$(brew --prefix python3)/bin/python3.9 -m venv football-env`.
+### Choosing Python distribution
+It is recommended to use Python shipped with `brew` because `boost-python3` is compiled against the same version.
+To check which Python 3 is used by default on your setup, execute `which python3`. If the output is `/usr/local/bin/python3`
+or `/opt/homebrew/bin/python3`, you're good to go.  
+If you see a different path, e.g. `/Library/Frameworks/Python.framework/Versions/3.9/bin/python3` or `...conda...`, 
+create a virtual environment with `$(brew --prefix python3)/bin/python3 -m venv football-env`. If you have `conda`, 
+you might need to deactivate base environment beforehand `conda deactivate`. 
 
+It is possible to use `conda` to create a virtual environment, but make sure that you select the same version of Python 
+that was used to build `boost-python3` (`3.9` as of February 2022).
+
+### Create virtual environment and build the game
 Use [virtual environment](https://docs.python.org/3/tutorial/venv.html) to avoid messing up with global dependencies:
 
 ```shell
 python3 -m venv football-env
 source football-env/bin/activate
-# update pip and setuptools
-python3 -m pip install --upgrade pip setuptools wheel
-python3 -m pip install psutil
 ```
+If you decide to use `conda` environment, use the following commands instead:
 
-Finally, build the game environment:
+```shell
+conda create --name football-env python=3.9 -y
+conda activate football-env
+```
 
+Upgrade `pip`, `setuptools`, `wheel` and install `psutil` inside the virtual environment:
 ```shell
-python3 -m pip install .
+python3 -m pip install --upgrade pip setuptools wheel
+python3 -m pip install psutil
 ```
 
-### Installation with conda
+Finally, build the game environment:
 
-If you installed the engine using `conda`, you might encounter the following error:
-`TypeError: __init__() should return None, not 'NoneType'` when trying to run the game.
-It may happen because the `boost-python3` installed with `brew` is compiled against a different
-version of Python (see the [discussion](https://github.com/google-research/football/issues/156)).
-If you successfully installed and ran Google Research Football using `conda` please update this guide.
-For now, the easiest way is to deactivate `conda` environment and install GRF with `brew` version of Python:
 ```shell
-conda deactivate
-$(brew --prefix python3)/bin/python3.9 -m venv football-env
-source football-env/bin/activate
-python3 -m pip install --upgrade pip setuptools psutil wheel
 python3 -m pip install .
 ```
 
@@ -122,6 +122,13 @@ libdirectfb-dev libst-dev mesa-utils xvfb x11vnc python3-pip
 python3 -m pip install --upgrade pip setuptools wheel
 python3 -m pip install psutil
 ```
+
+Clone the repository and navigate to the directory:
+```shell
+git clone https://github.com/google-research/football.git
+cd football
+```
+
 Use [virtual environment](https://docs.python.org/3/tutorial/venv.html) to avoid messing up with global dependencies:
 
 ```shell
@@ -148,5 +155,5 @@ in the [development](https://packaging.python.org/guides/distributing-packages-u
 python3 -m pip install -e .
 ```
 
-In such case, Python source files in projects can be edited in-place without reinstallation,
+In such case, Python source files can be edited in-place without reinstallation,
 the changes will be reflected the next time an interpreter process is started.
diff --git a/gfootball/doc/docker.md b/gfootball/doc/docker.md
@@ -6,7 +6,10 @@ git clone https://github.com/google-research/football.git
 cd football
 ```
 
-## Configure Docker
+For **rendering** the game on macOS and Windows, we recommend installing the game according to
+the instructions for your platform in [README](https://github.com/google-research/football#on-your-computer).
+
+## Configure Docker for Rendering (Linux only)
 In order to see rendered game you need to allow Docker containers access X server:
 
 ```
@@ -38,7 +41,7 @@ docker run --gpus all -e DISPLAY=$DISPLAY -it -v /tmp/.X11-unix:/tmp/.X11-unix:r
 
 If you get errors related to `--gpus all` flag, you can replace it with `--device /dev/dri/[X]`
 adding this flag for every file in the `/dev/dri/` directory. It makes sure that GPU is
-visibile inside the Docker image. You can also drop it altogether (environment
+visible inside the Docker image. You can also drop it altogether (environment
 will try to perform software rendering).
 
 ## Run environment

diff --git a/requirements.txt b/requirements.txt
@@ -2,7 +2,7 @@
 pygame>=1.9.6
 opencv-python
 psutil
-scipy
-gym>=0.11.0
+numpy
+gym<=0.21.0
 absl-py
 wheel