tcmu-runner buffer overflow detected

[sabaini]

I'm seeing a *** buffer overflow detected ***: terminated message for tcmu-runner

Excerpt from journalctl:

Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 systemd[1]: Starting tcmu-runner.service - LIO Userspace-passthrough daemon...
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: log file path now is '/var/log/tcmu-runner.log'
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: main:1302: Starting...
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: load_our_module:575: Inserted module 'target_core_user'
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler fbo is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler zbc is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler qcow is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler rbd is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 systemd[1]: Started tcmu-runner.service - LIO Userspace-passthrough daemon.
Jul 09 09:24:00 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmu_rbd_open:1164 rbd/zaza_ec_metadata_pool.disk_ec_1: address: {10.5.3.45:0/3297752322}
Jul 09 09:24:10 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: *** buffer overflow detected ***: terminated

In dmesg there's some faults printed which might be knock-on effects:

[Tue Jul  9 09:13:30 2024] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[Tue Jul  9 09:25:11 2024] ABORT_TASK: Found referenced iSCSI task_tag: 5
[Tue Jul  9 09:26:01 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:01 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:26:18 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:18 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:26:35 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:35 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:26:51 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:51 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:27:09 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:27:09 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:27:26 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:27:26 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:27:43 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:27:43 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:28:00 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:28:00 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:28:16 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:28:16 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:28:22 2024] INFO: task kworker/0:0:8 blocked for more than 122 seconds.
[Tue Jul  9 09:28:22 2024]       Not tainted 6.8.0-36-generic #36-Ubuntu
[Tue Jul  9 09:28:22 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Jul  9 09:28:22 2024] task:kworker/0:0     state:D stack:0     pid:8     tgid:8     ppid:2      flags:0x00004000
[Tue Jul  9 09:28:22 2024] Workqueue: events target_tmr_work [target_core_mod]
[Tue Jul  9 09:28:22 2024] Call Trace:
[Tue Jul  9 09:28:22 2024]  <TASK>
[Tue Jul  9 09:28:22 2024]  __schedule+0x27c/0x6b0
[Tue Jul  9 09:28:22 2024]  schedule+0x33/0x110
[Tue Jul  9 09:28:22 2024]  schedule_timeout+0x157/0x170
[Tue Jul  9 09:28:22 2024]  wait_for_completion+0x88/0x150
[Tue Jul  9 09:28:22 2024]  target_put_cmd_and_wait+0x73/0xc0 [target_core_mod]
[Tue Jul  9 09:28:22 2024]  core_tmr_abort_task+0x29d/0x300 [target_core_mod]
[Tue Jul  9 09:28:22 2024]  target_tmr_work+0xe1/0x120 [target_core_mod]
[Tue Jul  9 09:28:22 2024]  process_one_work+0x16f/0x350
[Tue Jul  9 09:28:22 2024]  worker_thread+0x306/0x440
[Tue Jul  9 09:28:22 2024]  ? __pfx_worker_thread+0x10/0x10
[Tue Jul  9 09:28:22 2024]  kthread+0xf2/0x120
[Tue Jul  9 09:28:22 2024]  ? __pfx_kthread+0x10/0x10
[Tue Jul  9 09:28:22 2024]  ret_from_fork+0x47/0x70
[Tue Jul  9 09:28:22 2024]  ? __pfx_kthread+0x10/0x10
[Tue Jul  9 09:28:22 2024]  ret_from_fork_asm+0x1b/0x30
[Tue Jul  9 09:28:22 2024]  </TASK>
[Tue Jul  9 09:28:22 2024] INFO: task iscsi_trx:12919 blocked for more than 122 seconds.
[Tue Jul  9 09:28:22 2024]       Not tainted 6.8.0-36-generic #36-Ubuntu

This is reproducible, i.e. if I restart the tcmu-runner the buffer overflow comes back.

Versions:
tcmu-runner 1.5.4-6ubuntu1

uname -a
Linux juju-d407b0-zaza-6e2e71fc48f7-16 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Please let me know if I can provide more diagnostics or other info.