🔥 🔧 Remove/depreciated pkg_resource usage and other depreciated usage (#1384)

* Add Support for Graviton 2 / ARM Architecture

* Changed parameter in README to architecture

* Removed redundant blank line

* Bugfix + black compliance

* Fix: architecture argument not used properly

* Fixed black compliance

* Fixed architecture label for wheels, added check for invalid architecture

* Testing architecture

* Added tests for ARM64 architecture

* Only allow ARM64 arch if Python version >= 3.8

* Removed redundant references to architecture

* Removed redundant references to architecture

* Added test for default architecture setting

* added .pre-commit hook for black-check and isort-check

* Fixed test failure  in Python 3.7

* 🔧 re-work regex for abi3

* 🔧 fix Zappa.create_lambda_function to pass a list to 'Architectures' in `create_function()` call.

* 🔧 replace usage of os.listdir()/os.path.join() with pathlib.Path object usage.
✅ update

* :whtie_check_mark: Update `test_copy_editable_packages` testcase for update to use Path objects

* 🔥 remove unnecessary comments

* 🔧 replace (depreciated) click.BaseCommand usage with click.Command

* 🔥 remove bad json 'comma'

* 🔧 fix typeerror 'list' object is not callable in `get_deps_list()`
🔧 replace datetime.utcnow() -> datetime.now(timezone.utc)

* 🔧 fix bad reference in get_deps_list()

* 🔧 update ci.yml to use pipenv in place of manual venv creation/prep in attempt to resolve `Run .venv/bin/coveralls --service=github` ImportError: cannot import name 'Reporter' from 'coverage.report' error.

* 🔧 add install `pipenv` step to ci.yml

* 🔧 refactor `test_zappa_init` and `cli.py` to allow cleaner mocking of ZappaCLI user inputs.

* Update tests/tests.py

add clear prefix for temporarydirectory

Co-authored-by: Copilot <[email protected]>

* 🔧 replace os.path related items with pathlib Path usage
🔧 change print() to LOG statement

* 🔧 attempt to resolve coveralls error

* 🔧 attempt to migrate to coveralls github action

* 🔧 add carryforward

* 🔧 explicitly set github-token for coverallapp/github-action

* 🔧 turn on coverallsapp/github-action@v2 debug

* 🔧 add --cov-report=xml in Makefile tests command
🔧 add `format: python` to ci.yml for coveralls

* 🔧 change coveralls format from python -> cobertura

* 🔧 add missing line-feed slash

* 🔥 remove commented out "old coveralls support"

---------

Co-authored-by: DS992 <>
Co-authored-by: Gabor Schulz <[email protected]>
Co-authored-by: shane <[email protected]>
Co-authored-by: Copilot <[email protected]>

Author: 4 people

.github/workflows/ci.yml

@@ -26,29 +26,34 @@ jobs:
           restore-keys: ${{ runner.os }}-pip${{ matrix.python }}-
       - name: make sure we have version tags
         run: git fetch --unshallow --tags
-      - name: Setup Virtualenv
-        run: python -m venv .venv
-      - name: Install
-        run: source .venv/bin/activate && make requirements && python setup.py install
-      - name: Lint
-        run: source .venv/bin/activate && make flake black-check isort-check
+      - name: Install pipenv
+        run: python -m pip install --upgrade pip pipenv
+      - name: Setup/install dependencies (pipenv)
+        run: python -m pipenv install --dev
+      - name: Lint (flake/black/isort)
+        run: pipenv run make flake black-check isort-check
       - name: Test
-        run: source .venv/bin/activate && make tests
-      - name: Upload Coverage
-        run: .venv/bin/coveralls --service=github
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          COVERALLS_FLAG_NAME: py${{ matrix.python }}
-          COVERALLS_PARALLEL: true
-          COVERALLS_SERVICE_JOB_ID: ${{ github.run_id }}
+        run: pipenv run make tests
+      - name: Coveralls Parallel
+        uses: coverallsapp/github-action@v2
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          flag-name: py${{ matrix.python }}
+          parallel: true
+          format: cobertura
+          debug: true
 
   coverage:
     needs: test
     runs-on: ubuntu-latest
     steps:
       - name: Set up Python
         uses: actions/setup-python@v5
-      - name: Finalise Coverage
-        run: pip3 install --upgrade coveralls && coveralls --service=github --finish
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Coveralls Finished
+        uses: coverallsapp/github-action@v2
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          parallel-finished: true
+          format: cobertura
+          carryforward: "run-3.9,run-3.10,run-3.11,run-3.12,run-3.13"
+          debug: true

.gitignore

@@ -262,5 +262,6 @@ Temporary Items
 ##########################
 
 *.zip
+*-slim-handler-*.tar.gz
 README.test.md
 zappa_settings.json

Makefile

@@ -91,6 +91,7 @@ tests:
 		tests/tests_async.py \
 		tests/tests.py \
 		tests/tests_utilities.py \
-		--cov=zappa
+		--cov=zappa \
+		--cov-report=xml \
 		--durations=0
 

tests/tests.py

@@ -1,4 +1,3 @@
-# -*- coding: utf8 -*-
 import base64
 import collections
 import hashlib
@@ -25,6 +24,7 @@
 import botocore.stub
 import flask
 import mock
+import pytest
 from click.exceptions import ClickException
 from click.globals import resolve_color_default
 from packaging import version
@@ -308,34 +308,35 @@ def test_get_exclude_glob__file_not_deleted(self):
 
     def test_getting_installed_packages(self, *args):
         z = Zappa(runtime="python3.8")
+        sample_package = mock.MagicMock()
+        sample_package.name = "super_package"
+        sample_package.version = "0.1"
+        sample_package.locate_file = mock.MagicMock()
+        sample_package.locate_file.return_value = "/venv/site-packages"
 
-        # mock pkg_resources call to be same as what our mocked site packages dir has
-        mock_package = collections.namedtuple("mock_package", ["project_name", "version", "location"])
-        mock_pip_installed_packages = [mock_package("super_package", "0.1", "/venv/site-packages")]
+        site_packages = mock.MagicMock()
+        site_packages.is_dir = mock.MagicMock(return_value=True)
+        site_packages.glob = mock.MagicMock(return_value=[Path("/venv/site-packages/super_package")])
 
-        with mock.patch("os.path.isdir", return_value=True):
-            with mock.patch("os.listdir", return_value=["super_package"]):
-                import pkg_resources  # this gets called in non-test Zappa mode
-
-                with mock.patch("pkg_resources.WorkingSet", return_value=mock_pip_installed_packages):
-                    self.assertDictEqual(z.get_installed_packages("", ""), {"super_package": "0.1"})
+        mock_pip_installed_packages = [sample_package]
+        with mock.patch("importlib.metadata.distributions", return_value=mock_pip_installed_packages):
+            self.assertDictEqual(z.get_installed_packages(site_packages=site_packages), {"super_package": "0.1"})
 
     def test_get_current_venv(self, *args):
         z = Zappa()
 
-        expected = "/expected/versions/path"
-
         # VIRTUL_ENV test
+        expected = "/expected/versions/path"
         os_env = {"VIRTUAL_ENV": expected}
         with mock.patch.dict("os.environ", os_env):
             current_venv = z.get_current_venv()
-            self.assertEqual(current_venv, expected)
+            self.assertEqual(str(current_venv), expected)
 
         # pyenv test
         with mock.patch.dict("os.environ", {}, clear=True):
             with mock.patch("subprocess.check_output", side_effect=[None, b"/expected", b"path"]):
                 current_venv = z.get_current_venv()
-                self.assertEqual(current_venv, expected)
+                self.assertEqual(str(current_venv), expected)
 
             with mock.patch("subprocess.check_output", side_effect=OSError("No pyenv!")):
                 current_venv = z.get_current_venv()
@@ -344,39 +345,52 @@ def test_get_current_venv(self, *args):
     def test_getting_installed_packages_mixed_case_location(self, *args):
         z = Zappa(runtime="python3.8")
 
-        # mock pip packages call to be same as what our mocked site packages dir has
-        mock_package = collections.namedtuple("mock_package", ["project_name", "version", "location"])
-        mock_pip_installed_packages = [
-            mock_package("SuperPackage", "0.1", "/Venv/site-packages"),
-            mock_package("SuperPackage64", "0.1", "/Venv/site-packages64"),
-        ]
-
-        with mock.patch("os.path.isdir", return_value=True):
-            with mock.patch("os.listdir", return_value=[]):
-                import pkg_resources  # this gets called in non-test Zappa mode
-
-                with mock.patch("pkg_resources.WorkingSet", return_value=mock_pip_installed_packages):
-                    self.assertDictEqual(
-                        z.get_installed_packages("/venv/Site-packages", "/venv/site-packages64"),
-                        {
-                            "superpackage": "0.1",
-                            "superpackage64": "0.1",
-                        },
-                    )
+        mock_pip_installed_packages = []
+        for package_name, version, location in ("SuperPackage", "0.1", "/Venv/site-packages"), (
+            "SuperPackage64",
+            "0.1",
+            "/Venv/site-packages64",
+        ):
+            sample_package = mock.MagicMock()
+            sample_package.name = package_name
+            sample_package.version = version
+            sample_package.locate_file = mock.MagicMock()
+            sample_package.locate_file.return_value = location
+            mock_pip_installed_packages.append(sample_package)
+
+        site_packages = mock.MagicMock()
+        site_packages.is_dir = mock.MagicMock(return_value=True)
+        site_packages.glob = mock.MagicMock(return_value=[Path("/venv/site-packages/SuperPackage")])
+
+        site_packages64 = mock.MagicMock()
+        site_packages64.is_dir = mock.MagicMock(return_value=True)
+        site_packages64.glob = mock.MagicMock(return_value=[Path("/venv/site-packages64/SuperPackage64")])
+
+        with mock.patch("importlib.metadata.distributions", return_value=mock_pip_installed_packages):
+            self.assertDictEqual(
+                z.get_installed_packages(site_packages, site_packages64),
+                {
+                    "superpackage": "0.1",
+                    "superpackage64": "0.1",
+                },
+            )
 
     def test_getting_installed_packages_mixed_case(self, *args):
         z = Zappa(runtime="python3.8")
 
-        # mock pkg_resources call to be same as what our mocked site packages dir has
-        mock_package = collections.namedtuple("mock_package", ["project_name", "version", "location"])
-        mock_pip_installed_packages = [mock_package("SuperPackage", "0.1", "/venv/site-packages")]
+        sample_package = mock.MagicMock()
+        sample_package.name = "SuperPackage"
+        sample_package.version = "0.1"
+        sample_package.locate_file = mock.MagicMock()
+        sample_package.locate_file.return_value = "/venv/site-packages"
 
-        with mock.patch("os.path.isdir", return_value=True):
-            with mock.patch("os.listdir", return_value=["superpackage"]):
-                import pkg_resources  # this gets called in non-test Zappa mode
+        site_packages = mock.MagicMock()
+        site_packages.is_dir = mock.MagicMock(return_value=True)
+        site_packages.glob = mock.MagicMock(return_value=[Path("/venv/site-packages/superpackage")])
 
-                with mock.patch("pkg_resources.WorkingSet", return_value=mock_pip_installed_packages):
-                    self.assertDictEqual(z.get_installed_packages("", ""), {"superpackage": "0.1"})
+        mock_pip_installed_packages = [sample_package]
+        with mock.patch("importlib.metadata.distributions", return_value=mock_pip_installed_packages):
+            self.assertDictEqual(z.get_installed_packages(site_packages=site_packages), {"superpackage": "0.1"})
 
     def test_load_credentials(self):
         z = Zappa()
@@ -1353,26 +1367,49 @@ def test_wsgi_from_v2_event_with_iam_authorizer(self):
     # CLI
     ##
 
+    # @pytest.fixture(autouse=True)
     def test_zappa_init(self):
+
         # delete if file exists
-        if os.path.exists("zappa_settings.json"):
-            os.remove("zappa_settings.json")
+        current_dir = os.getcwd()
+        with tempfile.TemporaryDirectory(prefix="zappa_test") as tempdir:
+            try:
+                os.chdir(tempdir)
+                tempdir = Path(tempdir)
 
-        process = subprocess.Popen(
-            ["zappa", "init"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True
-        )
-        process.communicate("dev\nmy-zappa-bucket\ntest_settings\ndefault\nn\ny\n")
-        self.assertTrue(os.path.exists("zappa_settings.json"))
+                settings_filepath = tempdir / "zappa_settings.json"
+                if settings_filepath.exists():
+                    settings_filepath.unlink()  # delete the file if it exists
+                assert settings_filepath.exists() is False
 
-        with open("zappa_settings.json", "r") as f:
-            zappa_settings = json.load(f)
-            self.assertEqual(zappa_settings["dev"]["s3_bucket"], "my-zappa-bucket")
-            self.assertEqual(zappa_settings["dev"]["django_settings"], "test_settings")
-            self.assertEqual(zappa_settings["dev"]["exclude"], ["boto3", "dateutil", "botocore", "s3transfer", "concurrent"])
+                zappa_cli = ZappaCLI()
+                with mock.patch("zappa.cli.ZappaCLI._get_init_env", return_value="dev"), mock.patch(
+                    "zappa.cli.ZappaCLI._get_init_profile", return_value=("default", {"region": "us-west-2"})
+                ), mock.patch("zappa.cli.ZappaCLI._get_init_bucket", return_value="my-zappa-bucket"), mock.patch(
+                    "zappa.cli.ZappaCLI._get_init_django_settings", return_value="test_settings"
+                ), mock.patch(
+                    "zappa.cli.ZappaCLI._get_init_global_settings", return_value=["n", False]
+                ), mock.patch(
+                    "zappa.cli.ZappaCLI._get_init_confirm", return_value="y"
+                ):
+                    zappa_cli.init()
+
+                # make sure the expected `zappa_settings.json` file was created in the current directory
+                self.assertTrue(settings_filepath.exists())
+
+                with settings_filepath.open("r") as f:
+                    zappa_settings = json.load(f)
+                    self.assertEqual(zappa_settings["dev"]["s3_bucket"], "my-zappa-bucket")
+                    self.assertEqual(zappa_settings["dev"]["django_settings"], "test_settings")
+                    self.assertEqual(
+                        zappa_settings["dev"]["exclude"], ["boto3", "dateutil", "botocore", "s3transfer", "concurrent"]
+                    )
 
-        # delete the file
-        if os.path.exists("zappa_settings.json"):
-            os.remove("zappa_settings.json")
+                # delete the file
+                if os.path.exists("zappa_settings.json"):
+                    os.remove("zappa_settings.json")
+            finally:
+                os.chdir(current_dir)
 
     def test_cli_sanity(self):
         zappa_cli = ZappaCLI()
@@ -2274,8 +2311,6 @@ def test_remote_env_package(self):
         with zipfile.ZipFile(zappa_cli.zip_path, "r") as lambda_zip:
             content = lambda_zip.read("zappa_settings.py")
         zappa_cli.remove_local_zip()
-        # m = re.search("REMOTE_ENV='(.*)'", content)
-        # self.assertEqual(m.group(1), 's3://lmbda-env/dev/env.json')
 
         zappa_cli = ZappaCLI()
         zappa_cli.api_stage = "remote_env"
@@ -2285,8 +2320,6 @@ def test_remote_env_package(self):
         with zipfile.ZipFile(zappa_cli.zip_path, "r") as lambda_zip:
             content = lambda_zip.read("zappa_settings.py")
         zappa_cli.remove_local_zip()
-        # m = re.search("REMOTE_ENV='(.*)'", content)
-        # self.assertEqual(m.group(1), 's3://lmbda-env/prod/env.json')
 
     def test_package_only(self):
         for delete_local_zip in [True, False]: