Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for multiple ssh backends #282

Merged
merged 3 commits into from
Apr 15, 2024

Conversation

tpapaioa
Copy link
Collaborator

@tpapaioa tpapaioa commented Apr 8, 2024

  • Add support for hussh and ansible-pylibssh libraries
  • Add ssh_backend: ssh2-python312 default setting to broker_settings.yaml.example. To use hussh or ansible-pylibssh, install the package and then override the setting in the local broker_settings.yaml file, or with the BROKER_SSH_BACKEND environment variable:
$ pip install hussh
$ BROKER_SSH_BACKEND="hussh" broker inventory
[INFO 240408 14:39:58] SSH_BACKEND='hussh'
[...]

$ pip install ansible-pylibssh
$ BROKER_SSH_BACKEND="ansible-pylibssh" broker inventory
[INFO 240408 16:15:32] SSH_BACKEND='ansible-pylibssh'
[...]

Verified functional tests:

$ BROKER_SSH_BACKEND="hussh" pytest -k test_satlab
=== test session starts ===
[...]
collected 68 items / 58 deselected / 10 selected

tests/functional/test_satlab.py::test_checkout_scenarios[args_file0] PASSED [ 10%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file1] PASSED [ 20%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file2] PASSED [ 30%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file3] PASSED [ 40%]
tests/functional/test_satlab.py::test_execute_scenarios[args_file0] PASSED  [ 50%]
tests/functional/test_satlab.py::test_inventory_sync PASSED                 [ 60%]
tests/functional/test_satlab.py::test_workflows_list PASSED                 [ 70%]
tests/functional/test_satlab.py::test_workflow_query PASSED                 [ 80%]
tests/functional/test_satlab.py::test_tower_host PASSED                     [ 90%]
tests/functional/test_satlab.py::test_tower_host_mp PASSED                  [100%]
=== 10 passed, 58 deselected, 49 warnings in 4216.30s (1:10:16) ===

****

$ BROKER_SSH_BACKEND="ansible-pylibssh" pytest -k test_satlab
=== test session starts ===
[...]
collected 68 items / 58 deselected / 10 selected

tests/functional/test_satlab.py::test_checkout_scenarios[args_file0] PASSED   [ 10%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file1] PASSED   [ 20%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file2] PASSED   [ 30%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file3] PASSED   [ 40%]
tests/functional/test_satlab.py::test_execute_scenarios[args_file0] PASSED    [ 50%]
tests/functional/test_satlab.py::test_inventory_sync PASSED                   [ 60%]
tests/functional/test_satlab.py::test_workflows_list PASSED                   [ 70%]
tests/functional/test_satlab.py::test_workflow_query PASSED                   [ 80%]
tests/functional/test_satlab.py::test_tower_host PASSED                       [ 90%]
tests/functional/test_satlab.py::test_tower_host_mp PASSED                    [100%]
[...]
=== 10 passed, 58 deselected, 43 warnings in 3985.79s (1:06:25) ===

@tpapaioa
Copy link
Collaborator Author

tpapaioa commented Apr 9, 2024

Functional tests on ssh2-python312:

$ BROKER_SSH_BACKEND="ssh2-python312" pytest -k test_satlab
[...]
collected 68 items / 58 deselected / 10 selected

tests/functional/test_satlab.py::test_checkout_scenarios[args_file0] PASSED  [ 10%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file1] PASSED  [ 20%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file2] PASSED  [ 30%]
tests/functional/test_satlab.py::test_checkout_scenarios[args_file3] PASSED  [ 40%]
tests/functional/test_satlab.py::test_execute_scenarios[args_file0] PASSED   [ 50%]
tests/functional/test_satlab.py::test_inventory_sync PASSED                  [ 60%]
tests/functional/test_satlab.py::test_workflows_list PASSED                  [ 70%]
tests/functional/test_satlab.py::test_workflow_query PASSED                  [ 80%]
tests/functional/test_satlab.py::test_tower_host PASSED                      [ 90%]
tests/functional/test_satlab.py::test_tower_host_mp PASSED                   [100%]
[...]
=== 10 passed, 58 deselected, 45 warnings in 4045.79s (1:07:25) ===

@tpapaioa
Copy link
Collaborator Author

tpapaioa commented Apr 9, 2024

Updated functional test ids for parametrized scenario tests, to show the scenario file name instead of the default, less informative, ids created by pytest.

Before:

$ pytest --co -k 'test_checkout_scenarios or test_execute_scenarios'
[...]
collected 68 items / 58 deselected / 10 selected

<Dir broker>
  <Dir tests>
    <Dir functional>
      <Module test_containers.py>
        <Function test_checkout_scenarios[args_file0]>
        <Function test_checkout_scenarios[args_file1]>
        <Function test_execute_scenarios[args_file0]>
      <Module test_rh_beaker.py>
        <Function test_checkout_scenarios[args_file0]>
        <Function test_checkout_scenarios[args_file1]>
      <Module test_satlab.py>
        <Function test_checkout_scenarios[args_file0]>
        <Function test_checkout_scenarios[args_file1]>
        <Function test_checkout_scenarios[args_file2]>
        <Function test_checkout_scenarios[args_file3]>
        <Function test_execute_scenarios[args_file0]>

After:

$ pytest --co -k 'test_checkout_scenarios or test_execute_scenarios'
[...]
collected 68 items / 58 deselected / 10 selected

<Dir broker>
  <Dir tests>
    <Dir functional>
      <Module test_containers.py>
        <Function test_checkout_scenarios[checkout_ch-d_ubi8]>
        <Function test_checkout_scenarios[checkout_ch-d_ubi8_2]>
        <Function test_execute_scenarios[execute_ch-d_ubi8]>
      <Module test_rh_beaker.py>
        <Function test_checkout_scenarios[checkout_test_job-2]>
        <Function test_checkout_scenarios[checkout_test_job]>
      <Module test_satlab.py>
        <Function test_checkout_scenarios[checkout_latest_sat]>
        <Function test_checkout_scenarios[checkout_latest_rhel]>
        <Function test_checkout_scenarios[checkout_rhel91]>
        <Function test_checkout_scenarios[checkout_sat_613]>
        <Function test_execute_scenarios[execute_templates_list]>

Copy link
Member

@JacobCallahan JacobCallahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work overall! Just some relatively minor notes.

broker/session.py Outdated Show resolved Hide resolved
Comment on lines 27 to 32
if SSH_BACKEND == "ansible-pylibssh":
from broker.ssh_session_pylibssh import InteractiveShell, Session
elif SSH_BACKEND == "hussh":
from broker.ssh_session_hussh import Session
elif SSH_BACKEND in ("ssh2-python", "ssh2-python312"):
from broker.ssh_session_ssh2 import InteractiveShell, Session # noqa: F401
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more of an organization suggestion, but I think these shouldn't live at this level. I think it makes sense to move these under the binds directory and strip ssh_session from the module name.

Suggested change
if SSH_BACKEND == "ansible-pylibssh":
from broker.ssh_session_pylibssh import InteractiveShell, Session
elif SSH_BACKEND == "hussh":
from broker.ssh_session_hussh import Session
elif SSH_BACKEND in ("ssh2-python", "ssh2-python312"):
from broker.ssh_session_ssh2 import InteractiveShell, Session # noqa: F401
if SSH_BACKEND == "ansible-pylibssh":
from broker.binds.pylibssh import InteractiveShell, Session
elif SSH_BACKEND == "hussh":
from broker.binds.hussh import Session
elif SSH_BACKEND in ("ssh2-python", "ssh2-python312"):
from broker.binds.ssh2 import InteractiveShell, Session # noqa: F401

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved modules to binds directory.

@contextmanager
def tail_file(self, filename):
"""Simulate tailing a file on the remote host.
from broker.ssh_session import BaseSession as Session
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't the ssh_session module's contents live in this module?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had originally thought of using BaseSession as a base class for the individual backend classes. I've updated the import logic to create the default session class if a backend can't be imported.

broker/session.py Show resolved Hide resolved
broker/ssh_session_hussh.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
@tpapaioa tpapaioa force-pushed the ssh_backends branch 2 times, most recently from 919c03b to 7f317dc Compare April 10, 2024 20:33
Copy link
Member

@JacobCallahan JacobCallahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK, thanks for taking this on!

@JacobCallahan JacobCallahan merged commit ae07c9c into SatelliteQE:0.5 Apr 15, 2024
4 checks passed
@tpapaioa tpapaioa deleted the ssh_backends branch May 3, 2024 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants