Execute Script Over SSH via a Jumpbox

[trevrobwhite]

Hi @merlinthemagic
Your repo looks fantastic just what I need, but I was after some help. (I wasn't sure if to post here or in MTM-SSH, sorry if it's wrong)

My problem, I need to run a script (Bash, Perl, etc) on a remote machine but I can only access it via a Jumpbox over SSH using Key authentication.
A <-> B <-> C

ServerA contains the script and the key to access ServerB, need to connect as nagios
ServerB contains the key to connect to server ServerC, need to connect as root

PHP is version 7.0, running on Ubuntu 16.04 LTS, OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 (Sadly no -J support)

As of today, I'm limited to a one liner via SSH and quotes are always a problem, as mentioned I would ideally like to create a PHP script which reads the contents on the script file and is able to execute them directly on ServerC, capturing the output (and return code if possible) without the need for multiple SCP/temporary files.

ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10 -t nagios@ServerB ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@ServerC 'command'

Thanks in advance.

[merlinthemagic]

Hi,

First off, thank you for the praise. Second ya, being on a linux distro you are going to want MTM-SSH, however lets address your question here. It took me a bit to back to you because key authentication was not supported when chaining ssh connections, i had to build that, its mostly untested so download the newest versions of MTM-Shells and MTM-SSH directly from master on github. One note, you will need all keys on server A. MTM currently does not support referencing keys located on remote file systems. However rest assured keys are only moved over ssh tunnels and only when encrypted. Basically the system sets an emphemeral 40 char random string as the passphrase regardless if the original RSA key has it before transporting it.

That aside, here we go:

Get Shell: A -> B:

$keyPath		= "/etc/pki/tls/private/someKey.pem"; //path local on A
//if the key does not have a passphrase:
$key		= file_get_contents($keyPath); 
//if the key is encrypted:
$passphrase	= "somethingVerySecret";
$key		= \MTM\Encrypt\Factories::getRSA()->getPrivateKey(file_get_contents($keyPath), $passphrase);

//get a shell on B from A:
$bShellObj	= \MTM\SSH\Factories::getShells()->keyAuthentication("server-b.example.com", "nagios", $key, null, 22, 10000);

Get Shell: B -> C:

$keyPath2	= "/etc/pki/tls/private/someKey2.pem"; //path local on A, this is not want you wanted.
//if the key does not have a passphrase:
$key2		= file_get_contents($keyPath2); 
//if the key is encrypted:
$passphrase2	= "somethingVerySecret2";
$key2		= \MTM\Encrypt\Factories::getRSA()->getPrivateKey(file_get_contents($keyPath2), $passphrase2);

//get a shell on C using the shell from B:
$cShellObj	= \MTM\SSH\Factories::getShells()->keyAuthentication("server-c.example.com", "root", $key2, $bShellObj, 22, 5000);

Now execute any commands you need to do (note: $bShellObj is now void as the session is being used to relay commands to C)
echo $cShellObj->getCmd("whoami")->get(); // root

You state you want to transport a full script to the remote server and execute it, you can, but i strongly suggest you break up the script into individual commands and handle error logic in PHP. You can do that with MTM because you have a persistent session as long as the shell is live. Its just like logging in using e.g. putty. Handling error logic in PHP is far easier than in BASH. The only time i would do a full scrip is if there is significant latency between server A and C.

Good luck and feel free to open issues as you work your way through the project

[trevrobwhite]

Many thanks, I'm working through this now and working out if having all the keys on the primary server is going to be an issue, I was hoping it could use the key on the gateway server as that's the way ssh -t does it, I might have to get the remote key up front as server C has a trust with Server B (not A) and each gateway server B connects to a different cluster of Server C servers using different keys.

As an aside, I don't know if it's an error or not but phpStorm is complaining about Models/Commands/Base.php line 161, Thrown object must be an instance of 'Exception' or 'Throwable', are you missing a "new" keyword?

Thanks again.

[merlinthemagic]

1. It would be trivial to add a method to the ssh class that would let you use a key file on a remote server. The reason its not there is that we use a key management service for our keys rather than have them distributed across servers. But for a static setup local keys make sense. May i suggest you test by simply reading the key off server B?

//replace:
$keyPath2	= "/etc/pki/tls/private/someKey2.pem"; //path local on A, this is not want you wanted.
$key2		= file_get_contents($keyPath2);

//With:
$keyPath2	= "/path/to/my/key/key.pem"; //path local on B
$key2		= $bShellObj->getCmd("cat ".$keyPath2)->get(); //read the key and haul it back to A

2. Storm is likely warning the class attribute ending up being thrown is initialized as null. In reality the only time it will end up thrown is if an exception was caught as part of the command execution, stored in the attribute and then re-thrown on Line 161. Probably better to type hint, but thats for another day.

[trevrobwhite]

That seems to work great for one liners and I was able to connect right through, however for a bash script that is on ServerA, using getCmd just prints the lines of the script, execute errors:
Uncaught Error: Call to a member function setRunning() on string

I know you said it's better to run the commands one by one, but we have a couple of BASH scripts all pre-built it would be good to run them directly if possible.

[merlinthemagic]

Issue you are going to run into is newline breaks and escaping quotes correctly. why not just base64 and push to file on remote?

$scriptPath	= "/path/to/script.sh"; //local to server A
$encData	= base64_encode(file_get_contents($scriptPath));

$remotePath	= "/tmp/myScript.sh"; //Valid path on server C
$cShellObj->getCmd("echo "'".$encData."' | base64 -d > ".$remotePath)->get();
$cShellObj->getCmd("chmod +x ".$remotePath)->get();

$timeout	= 60000; //max milisecs for script to complete, by default 25000.
$returnRegEx	= null; //regex to match when scrip is complete, by default this is the PS1 prompt, null === default, false === continue until timeout
$scriptReturn	= $cShellObj->getCmd("/usr/bin/bash ".$remotePath, $returnRegEx, $timeout)->get();
echo  $scriptReturn;

[trevrobwhite]

This is great many thanks and is working well, may I ask one other quick question.

Is it possible to get the return code of the command just run, so I can check if it was successful (as well as getting the data)?

Similar to exec:
exec($cmd, $output, $error_code);

Thank you again for all your help.

[merlinthemagic]

You are now 100% in bash scripting territory :).

Only you can validate if the return data is formatted as expected. If the return contains large sets, non printable or binary data then make sure to encode it. Base64 is natively available and it does a great job (e.g. echo "something" | base64 -w0 ). Terminal breaks can do "funny" things to outputs.

As for the exit status of the most recently executed foreground pipeline you can run the following command after completing your script execution:
echo $cShellObj->getCmd("echo \$?")->get();

if you get a "0" return you can be reasonably sure the process exited normally. I say reasonably because it depends on your error handling and if you are invoking logic to deal with errors (think ||). $? will only expand on the last command exit status.

Good luck

[trevrobwhite]

Of course, I need to get in the mindset that this is a virtual shell, thanks.

[trevrobwhite]

I think this will be the last question, as I am in a loop connecting to different machines to run scripts, what is the correct way to close a connection when moving from one server to the next without the script stopping?

I'm assuming when the script ends the tunnels are closed automatically?

[merlinthemagic]

Yes on script shutdown all connections are un-winded and terminated.
You can terminate any shell by issuing:

$shellObj->terminate();

In your case if you are done with C but need to connect to D from B you can reuse the connection to B and just terminate the connection B-C like so:

//Terminate C
$cShellObj->terminate();

//Connect to D, using the original B shell obj
$keyPath3	= "/path/to/my/key/key3.pem"; //path local on B for server D
$key3		= $bShellObj->getCmd("cat ".$keyPath3)->get(); //read the key and haul it back to A
$dShellObj	= \MTM\SSH\Factories::getShells()->keyAuthentication("server-d.example.com", "root", $key3, $bShellObj, 22, 5000);

//issue commands on D

If you are in a loop there are a few things you can do besides looping over connections one by one if you have a lot of long running jobs to run (nagios hits at such a scenario). One possibility is to parallelize many connections kick off the commands and then wait for all to finish. In this case each open connection will require its own dedicated connection e.g. you cannot use a single connection to B to then build connections to C and D at the same time. You would need to make two connections to B then use 1 for C and 2 for D. Upside is you can execute the jobs async of retrieving the return (not yet possible to check if a command has finished, but its trivial to implement).

$keyPath	= "/etc/pki/tls/private/serverKeyB.pem";
$key		= file_get_contents($keyPath); 
$bObj1		= \MTM\SSH\Factories::getShells()->keyAuthentication("server-b.example.com", "nagios", $key, null, 22, 10000);
$bObj2		= \MTM\SSH\Factories::getShells()->keyAuthentication("server-b.example.com", "nagios", $key, null, 22, 10000);

$keyPath2	="/etc/pki/tls/private/serverKeyC.pem"; 
$key2		= $bObj1->getCmd("cat ".$keyPath2)->get(); 
$cObj1		= \MTM\SSH\Factories::getShells()->keyAuthentication("server-c.example.com", "root", $key2, $bObj1, 22, 10000);

$keyPath3	="/etc/pki/tls/private/serverKeyD.pem"; 
$key3		= $bObj2->getCmd("cat ".$keyPath3)->get(); 
$dObj1		= \MTM\SSH\Factories::getShells()->keyAuthentication("server-d.example.com", "root", $key3, $bObj2, 22, 10000);

$cmdObjC	= $cObj1->getCmd("/path/to/my/script/on-c.sh", null, 130000)->exec(); //takes 120s to run
$cmdObjD	= $dObj1->getCmd("/path/to/my/script/on-d.sh", null, 130000)->exec();  //takes 120s to run

echo $cmdObjC->get();
echo $cmdObjD->get();
//done in 120s, not 240

[trevrobwhite]

Brilliant thanks, it's strange the script was working but now I'm finding I'm getting loads of timeouts, stepping through the code it looks like the script it stuck here:

PHP Fatal error:  Uncaught Exception: Read timeout in /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/Bash/Processing.php:68
Stack trace:
#0 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/Bash/Processing.php(28): MTM\Shells\Models\Shells\Bash\Processing->rawRead(Object(MTM\Shells\Models\Commands\Bash))
#1 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/Bash/Processing.php(30): MTM\Shells\Models\Shells\Bash\Processing->read(Object(MTM\Shells\Models\Commands\Bash))
#2 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Commands/Base.php(155): MTM\Shells\Models\Shells\Bash\Processing->read(Object(MTM\Shells\Models\Commands\Bash))
#3 /home/transfer/vendor/merlinthemagic/mtm-ssh/Tools/Shells/Bash/PublicKeyAuthentication.php(80): MTM\Shells\Models\Commands\Base->get()
#4 /home/transfer/vendor/merlinthemagic/mtm-ssh/Tools/Shells/Bash/PublicKeyAuthentication.php(15): MTM\SSH\Tools\Shells\Bash\PublicKeyAuthentication->keyConnectFromBash(Object(MTM\SSH\Models\Shells\Bash\Actions), Object(MTM in /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/Bash/Processing.php on line 68

Warning: Permanently added '192.168.5.4' (ECDSA) to the list of known hosts.
Enter passphrase for key '/dev/shm/MTM-Temp-6050ec13e7646/6050ec13e76f2/6050ec13e7874.pem':

If I run the SSH command manually and enter the passphrase from the code it works, but I wondered if the prompt warning was causing the issue, so I tried adding a -q to PublicKeyAuthentication.php line 39, however it didn't like that it resulted it loads of

PHP Fatal error:  Uncaught Exception: Not a File: /dev/shm/6050f263f1354/stdOut in /home/transfer/vendor/merlinthemagic/mtm-fs/Tools/Files/Local.php:16
Stack trace:
#0 /home/transfer/vendor/merlinthemagic/mtm-fs/Tools/Files/Local.php(124): MTM\FS\Tools\Files\Local->isFile(Object(MTM\FS\Models\File), true)
#1 /home/transfer/vendor/merlinthemagic/mtm-fs/Models/File.php(97): MTM\FS\Tools\Files\Local->getByteCount(Object(MTM\FS\Models\File))
#2 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/ProcessPipe.php(54): MTM\FS\Models\File->getByteCount()
#3 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/ProcessPipe.php(76): MTM\Shells\Models\Shells\ProcessPipe->getStdOutMax()
#4 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/Bash/Processing.php(49): MTM\Shells\Models\Shells\ProcessPipe->read()
#5 /home/transfer/vendor/merlinthemagic/mtm-shells/Models/Shells/Bash/Processing.php(28): MTM\Shells\Models\Shells\Bash\Processing->rawRead(Object(MTM\Shells\Models\Commands\Bash))
#6 /home/transfer/vendor in /home/transfer/vendor/merlinthemagic/mtm-fs/Tools/Files/Local.php on line 16

I did check /dev/shm and found loads of files left behind but clearing this out didn't help

[merlinthemagic]

How many tunnels are you instantiating at once? You likely have tons of Python processes left behind (used to spawn a pty shell). Please update MTM-Shells and MTM-SSH to newest dev versions.

Either reboot or kill all the orphaned python processes (pkill python), then monitor the /dev/shm folder when you execute a tunnel, does it clean up after itself?

[trevrobwhite]

Just the one, I haven't tried to done the parallel approach because I'm on shared bandwidth.

There were a couple of stuck pythons, killing them restored one of my servers, the one which issues a warning "Warning: Permanently added" seems to cause the timeout