PBS: detection of qstat failures no longer working

[dpmatthews]

#2691 added code to detect qstat failures by searching for "Connection refused" in stderr. However this is not working on our new system which is resulting in jobs being incorrectly reported as failed when polled.

Information at the time indicated we could expect to see errors like this if qstat failed to contact the server:

Connection refused
qstat: cannot connect to server xxxxxx (errno=111)

However, we now seeing errors like this from PBS 2022.1.7:

Connection timed out
qstat: cannot connect to server xxxxxx (errno=xxxxx)

For the moment I think we would be safe to change the search string to "cannot connect" (or possibly "errno").

Longer term we should consider other ways to make the polling more robust, see #3436

[dpmatthews]

I've found I can trigger various PBS errors by setting the PBS_SERVER env variable:

PBS 2022.1.7:

$ PBS_SERVER=localhost qstat
Connection refused
qstat: cannot connect to server localhost (errno=15010)

$ PBS_SERVER=dummy qstat
Unknown Host.
qstat: cannot connect to server dummy (errno=15008)

$ PBS_SERVER=google.com qstat
Connection timed out
qstat: cannot connect to server google.com (errno=15010)

PBS 18.2.6:

$ PBS_SERVER=localhost qstat
Connection refused
qstat: cannot connect to server localhost (errno=111)

$ PBS_SERVER=dummy qstat
Unknown Host.
qstat: cannot connect to server dummy (errno=15008)

$ PBS_SERVER=google.com qstat
Connection timed out
qstat: cannot connect to server google.com (errno=110)

I think we should be safe to change the search string to "cannot connect to server".