Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent SIGSEGV when running multiple ssh_channel.exec_command() #645

Open
kucharskim opened this issue Sep 20, 2024 · 4 comments
Open

Intermittent SIGSEGV when running multiple ssh_channel.exec_command() #645

kucharskim opened this issue Sep 20, 2024 · 4 comments

Comments

@kucharskim
Copy link

SUMMARY

On OpenBSD -current as of 2024-09-04 I have following backtrace from a core dump:

Core was generated by `python3.11'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000008cf814426a8 in __pyx_type_11pylibsshext_7channel_Channel ()
   from /usr/local/lib/python3.11/site-packages/pylibsshext/channel.cpython-311.so
(gdb) bt
#0  0x000008cf814426a8 in __pyx_type_11pylibsshext_7channel_Channel ()
   from /usr/local/lib/python3.11/site-packages/pylibsshext/channel.cpython-311.so
#1  0x000008cf979ac6c4 in ssh_packet_socket_controlflow_callback () from /usr/local/lib/libssh.so.4.2
#2  0x000008cf979b842b in ssh_socket_pollcallback () from /usr/local/lib/libssh.so.4.2
#3  0x000008cf979b4365 in ssh_poll_ctx_dopoll () from /usr/local/lib/libssh.so.4.2
#4  0x000008cf979b5bb6 in ssh_handle_packets () from /usr/local/lib/libssh.so.4.2
#5  0x000008cf979b595b in ssh_handle_packets_termination () from /usr/local/lib/libssh.so.4.2
#6  0x000008cf97993195 in channel_open () from /usr/local/lib/libssh.so.4.2
#7  0x000008cf8143a3ac in __pyx_pf_11pylibsshext_7channel_7Channel_24exec_command (__pyx_v_self=0x8cf82114ac0, 
    __pyx_v_command=<error reading variable: Cannot access memory at address 0x0>) at pylibsshext/channel.c:7221
#8  __pyx_pw_11pylibsshext_7channel_7Channel_25exec_command (
    __pyx_v_self=<pylibsshext.channel.Channel at remote 0x8cf82114ac0>, __pyx_args=<optimized out>, 
    __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>) at pylibsshext/channel.c:7138
#9  0x000008cedc9a264d in PyObject_Vectorcall () from /usr/local/lib/libpython3.11.so.0.0
#10 0x000008cedcabb1cd in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.11.so.0.0
#11 0x000008cedcaaab1e in PyEval_EvalCode () from /usr/local/lib/libpython3.11.so.0.0
#12 0x000008cedcb1b6e9 in run_mod () from /usr/local/lib/libpython3.11.so.0.0
#13 0x000008cedcb1b1d5 in _PyRun_SimpleFileObject () from /usr/local/lib/libpython3.11.so.0.0
#14 0x000008cedcb1a3b3 in _PyRun_AnyFileObject () from /usr/local/lib/libpython3.11.so.0.0
#15 0x000008cedcb44172 in Py_RunMain () from /usr/local/lib/libpython3.11.so.0.0
#16 0x000008cedcb45095 in pymain_main () from /usr/local/lib/libpython3.11.so.0.0
#17 0x000008cedcb454bc in Py_BytesMain () from /usr/local/lib/libpython3.11.so.0.0
#18 0x000008ccc9b7c94b in _start ()
ISSUE TYPE
  • Bug Report
PYLISSH and LIBSSH VERSION
$ python3 version.py
__full_version__='<pylibsshext v1.2.2 with libssh v0.10.6>'
__libssh_version__='0.10.6'
__version__='1.2.2'
__version_info__=(1, 2, 2)
OS / ENVIRONMENT
OpenBSD 7.6-beta (GENERIC.MP) #310: Wed Sep  4 11:59:45 MDT 2024
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

# pkg_info -qI python
python-3.11.10p0

# pkg_info -qI libssh
libssh-0.10.6

# pkg_info -qI py3-ansible-libssh
py3-ansible-libssh-1.2.2
STEPS TO REPRODUCE
$ cat test1.py
#!/usr/bin/env python3

from pylibsshext.errors import LibsshSessionException
from pylibsshext.session import Session

import logging

HOST = "examplemachine1"
USER = "root"
TIMEOUT = 30
PORT = 22

ssh = Session()

# ssh.set_log_level(logging.DEBUG)

try:
    ssh.connect(
        host=HOST,
        user=USER,
        timeout=TIMEOUT,
        port=PORT,
        # proxycommand="ssh -q -W %h:%p ks2",
    )
except LibsshSessionException as ex:
    print(f"Failed to connect to {HOST}:{PORT} over SSH: {ex!s}")

print(f"ssh.is_connected={ssh.is_connected}")


def run_cmd(ssh_channel, cmd):
    print(f"Running {cmd}...", flush=True)
    print(f"Executing exec_command()...", flush=True)
    cmd_resp = ssh_channel.exec_command(cmd)
    print(f"Executing exec_command()... done", flush=True)
    print(f"stdout type: {type(cmd_resp.stdout)}")
    print(f"stdout:\n{cmd_resp.stdout.decode()}\n")
    print(f"stderr type: {type(cmd_resp.stderr)}")
    print(f"stderr:\n{cmd_resp.stderr.decode()}\n")
    print(f"return code: {cmd_resp.returncode}")


ssh_channel = ssh.new_channel()

run_cmd(ssh_channel, "uptime")
run_cmd(ssh_channel, "ls")
run_cmd(ssh_channel, "hostname")

print(f"Executing ssh_channel.close()...", flush=True)
ssh_channel.close()

print("Closing connection...", flush=True)
ssh.close()
EXPECTED RESULTS

Execution of below script should work all the time, but it code dumps intermittently.

python3 test1.py
ACTUAL RESULTS

Core dumps every now and then. Always on second command. First command always works.

$ python3 test1.py  
ssh.is_connected=1
Running uptime...
Executing exec_command()...
Executing exec_command()... done
stdout type: <class 'bytes'>
stdout:
 7:17PM  up 1 day, 15:59, 0 users, load averages: 0.01, 0.01, 0.00


stderr type: <class 'bytes'>
stderr:


return code: 0
Running ls...
Executing exec_command()...
Segmentation fault (core dumped)
@kucharskim
Copy link
Author

When I enable logging via ssh.set_log_level(logging.DEBUG) (as commented out above in test1.py), I see this:

...
[2024/09/21 07:09:35.914803, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:35.914841, 3] packet_send2:  packet: wrote [type=50, len=1136, padding_size=5, comp=1130, payload=1130]
[2024/09/21 07:09:35.965620, 3] ssh_packet_socket_callback:  packet: read type 52 [len=8,padding=6,comp=1,payload=1]
[2024/09/21 07:09:35.965631, 3] ssh_packet_process:  Dispatching handler for packet type 52
[2024/09/21 07:09:35.965635, 3] ssh_packet_userauth_success:  Authentication successful
[2024/09/21 07:09:35.965639, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=221, in_blocks=107]
ssh.is_connected=1
[2024/09/21 07:09:35.966108, 2] channel_open:  Creating a channel 43 with 64000 window and 32768 max packet
[2024/09/21 07:09:35.966114, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=224, in_blocks=110]
[2024/09/21 07:09:35.966133, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:35.966142, 3] packet_send2:  packet: wrote [type=90, len=32, padding_size=7, comp=24, payload=24]
[2024/09/21 07:09:35.966146, 3] channel_open:  Sent a SSH_MSG_CHANNEL_OPEN type session for channel 43
[2024/09/21 07:09:36.000845, 3] ssh_packet_socket_callback:  packet: read type 80 [len=480,padding=4,comp=475,payload=475]
[2024/09/21 07:09:36.000860, 3] ssh_packet_process:  Dispatching handler for packet type 80
[2024/09/21 07:09:36.000864, 2] ssh_packet_global_request:  Received SSH_MSG_GLOBAL_REQUEST packet
[2024/09/21 07:09:36.000871, 2] ssh_packet_global_request:  UNKNOWN SSH_MSG_GLOBAL_REQUEST [email protected], want_reply = 0
[2024/09/21 07:09:36.000874, 3] ssh_packet_global_request:  The requester doesn't want to know the request failed!
[2024/09/21 07:09:36.000879, 1] ssh_packet_global_request:  Invalid SSH_MSG_GLOBAL_REQUEST packet
[2024/09/21 07:09:36.000882, 3] ssh_packet_socket_callback:  Processing 280 bytes left in socket buffer
[2024/09/21 07:09:36.000889, 3] ssh_packet_socket_callback:  packet: read type 4 [len=120,padding=7,comp=112,payload=112]
[2024/09/21 07:09:36.000892, 3] ssh_packet_process:  Dispatching handler for packet type 4
[2024/09/21 07:09:36.000895, 2] ssh_packet_ignore_callback:  Received SSH_MSG_DEBUG packet
[2024/09/21 07:09:36.000898, 3] ssh_packet_socket_callback:  Processing 140 bytes left in socket buffer
[2024/09/21 07:09:36.000902, 3] ssh_packet_socket_callback:  packet: read type 4 [len=120,padding=7,comp=112,payload=112]
[2024/09/21 07:09:36.000905, 3] ssh_packet_process:  Dispatching handler for packet type 4
[2024/09/21 07:09:36.000908, 2] ssh_packet_ignore_callback:  Received SSH_MSG_DEBUG packet
[2024/09/21 07:09:36.000911, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=224, in_blocks=194]
[2024/09/21 07:09:36.000914, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=224, in_blocks=194]
[2024/09/21 07:09:36.000917, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=224, in_blocks=194]
[2024/09/21 07:09:36.228678, 3] ssh_packet_socket_callback:  packet: read type 91 [len=24,padding=6,comp=17,payload=17]
[2024/09/21 07:09:36.228691, 3] ssh_packet_process:  Dispatching handler for packet type 91
[2024/09/21 07:09:36.228695, 3] ssh_packet_channel_open_conf:  Received SSH2_MSG_CHANNEL_OPEN_CONFIRMATION
[2024/09/21 07:09:36.228699, 2] ssh_packet_channel_open_conf:  Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
[2024/09/21 07:09:36.228702, 2] ssh_packet_channel_open_conf:  Remote window : 0, maxpacket : 32768
[2024/09/21 07:09:36.228706, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=224, in_blocks=196]
Running uptime...
Executing exec_command()...
[2024/09/21 07:09:36.228773, 2] channel_open:  Creating a channel 44 with 64000 window and 32768 max packet
[2024/09/21 07:09:36.228777, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=227, in_blocks=199]
[2024/09/21 07:09:36.228793, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:36.228797, 3] packet_send2:  packet: wrote [type=90, len=32, padding_size=7, comp=24, payload=24]
[2024/09/21 07:09:36.228800, 3] channel_open:  Sent a SSH_MSG_CHANNEL_OPEN type session for channel 44
[2024/09/21 07:09:36.263447, 3] ssh_packet_socket_callback:  packet: read type 91 [len=24,padding=6,comp=17,payload=17]
[2024/09/21 07:09:36.263460, 3] ssh_packet_process:  Dispatching handler for packet type 91
[2024/09/21 07:09:36.263464, 3] ssh_packet_channel_open_conf:  Received SSH2_MSG_CHANNEL_OPEN_CONFIRMATION
[2024/09/21 07:09:36.263467, 2] ssh_packet_channel_open_conf:  Received a CHANNEL_OPEN_CONFIRMATION for channel 44:1
[2024/09/21 07:09:36.263470, 2] ssh_packet_channel_open_conf:  Remote window : 0, maxpacket : 32768
[2024/09/21 07:09:36.263476, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=227, in_blocks=198]
[2024/09/21 07:09:36.263494, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=230, in_blocks=201]
[2024/09/21 07:09:36.263514, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:36.263518, 3] packet_send2:  packet: wrote [type=98, len=32, padding_size=7, comp=24, payload=24]
[2024/09/21 07:09:36.263522, 3] channel_request:  Sent a SSH_MSG_CHANNEL_REQUEST exec
[2024/09/21 07:09:36.298556, 3] ssh_packet_socket_callback:  packet: read type 93 [len=16,padding=6,comp=9,payload=9]
[2024/09/21 07:09:36.298568, 3] ssh_packet_process:  Dispatching handler for packet type 93
[2024/09/21 07:09:36.298573, 2] channel_rcv_change_window:  Adding 2097152 bytes to channel (44:1) (from 0 bytes)
[2024/09/21 07:09:36.298576, 3] ssh_packet_socket_callback:  Processing 36 bytes left in socket buffer
[2024/09/21 07:09:36.298581, 3] ssh_packet_socket_callback:  packet: read type 99 [len=16,padding=10,comp=5,payload=5]
[2024/09/21 07:09:36.298583, 3] ssh_packet_process:  Dispatching handler for packet type 99
[2024/09/21 07:09:36.298587, 3] ssh_packet_channel_success:  Received SSH_CHANNEL_SUCCESS on channel (44:1)
[2024/09/21 07:09:36.298590, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=230, in_blocks=199]
[2024/09/21 07:09:36.298593, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=230, in_blocks=199]
[2024/09/21 07:09:36.298597, 2] channel_request:  Channel request exec success
[2024/09/21 07:09:36.298629, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=230, in_blocks=199]
[2024/09/21 07:09:36.298647, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:36.298651, 3] packet_send2:  packet: wrote [type=96, len=16, padding_size=10, comp=5, payload=5]
[2024/09/21 07:09:36.298654, 3] ssh_channel_send_eof:  Sent a EOF on client channel (44:1)
[2024/09/21 07:09:36.303498, 3] ssh_packet_socket_callback:  packet: read type 94 [len=88,padding=11,comp=76,payload=76]
[2024/09/21 07:09:36.303512, 3] ssh_packet_process:  Dispatching handler for packet type 94
[2024/09/21 07:09:36.303517, 3] channel_rcv_data:  Channel receiving 67 bytes data in 0 (local win=64000 remote win=2097152)
[2024/09/21 07:09:36.303521, 3] channel_default_bufferize:  placing 67 bytes into channel buffer (stdout)
[2024/09/21 07:09:36.303527, 3] channel_rcv_data:  Channel windows are now (local win=63933 remote win=2097152)
[2024/09/21 07:09:36.303538, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=231, in_blocks=209]
[2024/09/21 07:09:36.303548, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:36.303552, 3] packet_send2:  packet: wrote [type=93, len=16, padding_size=6, comp=9, payload=9]
[2024/09/21 07:09:36.303556, 2] grow_window:  growing window (channel 44:1) to 1280000 bytes
[2024/09/21 07:09:36.303559, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=231, in_blocks=208]
[2024/09/21 07:09:36.303694, 3] ssh_packet_socket_callback:  packet: read type 98 [len=32,padding=6,comp=25,payload=25]
[2024/09/21 07:09:36.303702, 3] ssh_packet_process:  Dispatching handler for packet type 98
[2024/09/21 07:09:36.303708, 3] channel_rcv_request:  received exit-status 0
[2024/09/21 07:09:36.303711, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=231, in_blocks=211]
[2024/09/21 07:09:36.303717, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=231, in_blocks=211]
[2024/09/21 07:09:36.303724, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2024/09/21 07:09:36.303727, 3] packet_send2:  packet: wrote [type=97, len=16, padding_size=10, comp=5, payload=5]
[2024/09/21 07:09:36.303731, 3] ssh_channel_close:  Sent a close on client channel (44:1)
Executing exec_command()... done
stdout type: <class 'bytes'>
stdout:
 7:09AM  up 2 days,  3:50, 1 user, load averages: 0.06, 0.03, 0.00


stderr type: <class 'bytes'>
stderr:


return code: 0
Running ls...
Executing exec_command()...
[2024/09/21 07:09:36.303797, 2] channel_open:  Creating a channel 45 with 64000 window and 32768 max packet
[2024/09/21 07:09:36.303802, 3] ssh_packet_need_rekey:  rekey: [data_rekey_needed=0, out_blocks=234, in_blocks=214]
[2024/09/21 07:09:36.303808, 3] packet_send2:  packet: wrote [type=90, len=32, padding_size=7, comp=24, payload=24]
[2024/09/21 07:09:36.303811, 3] channel_open:  Sent a SSH_MSG_CHANNEL_OPEN type session for channel 45
[2024/09/21 07:09:36.303819, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
Segmentation fault (core dumped)

@kucharskim
Copy link
Author

Version of remote SSH daemon (if that makes any difference):

$ nc -w5 -4 -v examplemachine1 22 
Connection to examplemachine1 (xxx.xxx.xxx.xxx) 22 port [tcp/ssh] succeeded!
SSH-2.0-OpenSSH_9.7

@webknjaz
Copy link
Member

@Jakuje ideas?

@Jakuje
Copy link
Contributor

Jakuje commented Oct 21, 2024

Having log level trace output (see #597) would help to investigate the issue. From the current debug log it is not clear what is going on. I can just guess from the backtrace, that after the first channel got closed, either the callback or something is probing the structure that might have been freed. Is there way to install debuginfo on openbsd to see some more information through gdb about variables that make it crash or does it look like crashing inside the cpython?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@webknjaz @Jakuje @kucharskim