Fix CI failures

[shchur]

Motivation

Fixing CI issues described in #2841.

CI failures fixed by this PR

• lint-3.7
  • Pinned importlib-metadata<5.0 for Python <= 3.7 in requirements/dev.txt
• lint_plugins-3.8(hydra-nevergrad-sweeper)
  • Remove # type: ignore comments in plugins/hydra_nevergrad_sweeper/hydra_plugins/hydra_nevergrad_sweeper/_impl.py
  • Ignore nevergrad module in .mypy.ini
• lint_plugins-3.8(hydra-rq-launcher)
  • Add / remove # type: ignore comments in plugins/hydra_rq_launcher/hydra_plugins/hydra_rq_launcher/_core.py
  • Ignore rq module in .mypy.ini
• test_plugins-3.8(hydra-ax-sweeper)
  • Pinned pandas<1.5.0 in plugins/hydra_ax_sweeper/setup.py. This dependency should be unpinned when upgrading ax to >=0.2.8.
• test_plugins-3.8(hydra-optuna-sweeper)

  • Pinned sqlalchemy~=1.3.0 in plugins/hydra_optuna_sweeper/setup.py. This dependency should be unpinned in Upgrade Optuna from v2.x.x to v3.0.0 #2360 when upgrading optuna to >=3.0.

    Alternatively, we could go for a more permissive version range sqlalchemy>=1.3.0,<2.0.0, but that would require adding ignore: statement for warnings produced by sqlalchemy to pytest.ini

• test_plugins-3.8(hydra-rq-launcher)
  • Pinned rq>=1.5.1,<1.12 in plugins/hydra_rq_launcher/setup.py. This is the highest version for which the tests pass, <1.13 leads to failures.
• test_plugins-3.8(hydra-submitit-launcher)
  • Set allow_warnings=True in test_example in plugins/hydra_submitit_launcher/tests/test_submitit_launcher.py
  • Add line ignore:.*pkg_resources is deprecated* in pytest.ini. DeprecationWarning comes from other package (submitit), so we have no way to fix them on the hydra side.
• test_plugins_vs_core-3.8
  • Use importlib.metadata.version instead of pkg_resources.get_distribution in plugins/hydra_ray_launcher/hydra_plugins/hydra_ray_launcher/_config.py.

Errors that I was unable to fix

• test_plugins-3.8(hydra-ray-launcher), test_plugins-3.9(hydra-ray-launcher)

  • Setting default object_store_memory to 78643200, as recommended in ValueError: After taking into account object store and redis memory usage, the amount of memory on this node available for tasks and actors (-0.01 GB) is less than 0% of total. You can adjust these settings with ray.init(memory=<bytes>, object store memory=<bytes> ray-project/ray#11966 (comment), did not help.

  Possible solutions:

  • Using instances with more memory (as recommended here). This is most likely the problem, seeing how ray tests pass on macos and windows.
  • Upgrading to ray 2.0 (as recommended here)

• lint_plugins-3.7(hydra-nevergrad-sweeper), lint_plugins-3.7(hydra-ray-launcher), test_plugins-3.7(hydra-ray-launcher), test_plugins_vs_core-3.7

  • Not prioritizing these issues since Python 3.7 support will be dropped.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

This PR does not introduce any new functionality, so no new tests are needed. Only existing tests will be fixed.

Related Issues and PRs

This PR resolves #2841

[shchur]

@Jasha10 all the tests except test_plugins-3.8(hydra-ray-launcher) are now passing for Python 3.8, 3.9, 3.10, 3.11. Do you think this is sufficient to unblock #2733?

[Jasha10]

Yes, I think so. Thank you, @shchur.

[jlopezpena]

I will rebase #2733 to include these changes.