Custom socket connect timeout in Session()

[akhil-jha]

What?
Let the user give a timeout for socket.connect(). Default is still 60.

Why?
In cloud VMs, we get Connection Refused as the machine is not ready to take ssh connection just yet. Increasing the timeouts(reties) solves the issue.

Usage:

with Session(username="ec2-user",  hostname=hostname,  key_filename=private_key_path, socket_connect_timeout=500) as host:
      assert "hostname" in host.run("rpm -q hostname").stdout

[akhil-jha]

Test:

ipdb> with BrokerSession(username="ec2-user", hostname=ip, key_filename=pubkey_data["private_key_path"], socket_connect_timeout=500,) as host:
                assert "hostname" in host.run("rpm -q hostname").stdout
                host.run("hostname").stdout
 

[WARNING 230502 13:07:53] Tried cmd=<built-in method connect of socket object at 0x7f0ecc1353a0> with cmd_args=[('54.197.154.60', 22)], cmd_kwargs={} but received err=ConnectionRefusedError(111, 'Connection refused')
    Trying again in 1 seconds.
[WARNING 230502 13:07:54] Tried cmd=<built-in method connect of socket object at 0x7f0ecc1353a0> with cmd_args=[('54.197.154.60', 22)], cmd_kwargs={} but received err=ConnectionRefusedError(111, 'Connection refused')
    Trying again in 2 seconds.
[WARNING 230502 13:07:56] Tried cmd=<built-in method connect of socket object at 0x7f0ecc1353a0> with cmd_args=[('54.197.154.60', 22)], cmd_kwargs={} but received err=ConnectionRefusedError(111, 'Connection refused')
    Trying again in 4 seconds.
[WARNING 230502 13:08:00] Tried cmd=<built-in method connect of socket object at 0x7f0ecc1353a0> with cmd_args=[('54.197.154.60', 22)], cmd_kwargs={} but received err=ConnectionRefusedError(111, 'Connection refused')
    Trying again in 8 seconds.
[WARNING 230502 13:08:09] Tried cmd=<built-in method connect of socket object at 0x7f0ecc1353a0> with cmd_args=[('54.197.154.60', 22)], cmd_kwargs={} but received err=ConnectionRefusedError(111, 'Connection refused')
    Trying again in 16 seconds.
[WARNING 230502 13:08:25] Tried cmd=<built-in method connect of socket object at 0x7f0ecc1353a0> with cmd_args=[('54.197.154.60', 22)], cmd_kwargs={} but received err=ConnectionRefusedError(111, 'Connection refused')
    Trying again in 32 seconds.

ipdb>

'ip-172-31-93-71.ec2.internal\n'

[ogajduse]

ACK
@JacobCallahan would it make sense to propagate this attribute to broker_settings.yaml?

[akhil-jha]

ACK @JacobCallahan would it make sense to propagate this attribute to broker_settings.yaml?

I don't think so. When you're checking out the VM, AT is turning it on anyway(if I recall correctly). This setting will mostly be used in cloud VMs.

[JacobCallahan]

@akhil-jha are you using Broker sessions directly instead of through a host object?

[akhil-jha]

@akhil-jha are you using Broker sessions directly instead of through a host object?

Yea, directly .

I did checkout(didn't practically tried) init() of Broker(), I am probably gonna hit No data load from file exception. Since I don't really need any provider setup and I don't think I need to have a broker_settings.yaml for any reason at all. That would be just another thing to maintain.

[akhil-jha]

Oops, I didn't checkout Host() 🤦‍♂️

EDIT: I saw that, I think using directly would be simpler. no? To accommodate socket_connect_timeout arg in Host() would require more changes.

[JacobCallahan]

It would indeed, which is why I asked. Most users make connections from the Host objects themselves instead of directly with a Session object. Perhaps, if you make it a configurable option (with a sane default), then that would help them. Otherwise, it would need to be passed down during host object creation.

[akhil-jha]

from broker.hosts import Host

        for ip in host_ips:
            host = Host(
                username="ec2-user",
                hostname=ip,
                key_filename=pubkey_data["private_key_path"],
                socket_connect_timeout=500,
            )
            host.execute("rpm -q hostname").stdout

$ ENV_FOR_DYNACONF=stage_proxy iqe tests plugin provisioning -k test_aws_reservations_e2e_launch_template -k test_aws_reservations_e2e_launch_template
...
======================================================== 1 passed, 22 deselected in 137.50s (0:02:17) ========================================================

[JacobCallahan]

@akhil-jha is this ready for merge, or did you want to do some more testing on it?

[JacobCallahan]

Let's switch from this implementation to simply passing this timeout
https://github.com/SatelliteQE/broker/blob/master/broker/hosts.py#L57
to the Sessions created here
https://github.com/SatelliteQE/broker/blob/master/broker/hosts.py#L65

I don't know why I removed it in #151

[akhil-jha]

Dependent on settings.HOST_CONNECTION_TIMEOUT. If not , it'll be none. That means no retry.

[JacobCallahan]

If None, simple_retry should still get your default of 60 so I think we're good here!

[akhil-jha]

Ah right.okay yes.

[JacobCallahan]

ACK, thanks for the revision!

[JacobCallahan]

If None, simple_retry should still get your default of 60 so I think we're good here!

[akhil-jha]

So you want to keep it? or should I remove it?

[JacobCallahan]

let's try to keep this version, but we'll need to pay attention to automation after this release is live.