Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tcmu-runner buffer overflow detected #703

Open
sabaini opened this issue Jul 9, 2024 · 0 comments
Open

tcmu-runner buffer overflow detected #703

sabaini opened this issue Jul 9, 2024 · 0 comments

Comments

@sabaini
Copy link

sabaini commented Jul 9, 2024

I'm seeing a *** buffer overflow detected ***: terminated message for tcmu-runner

Excerpt from journalctl:

Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 systemd[1]: Starting tcmu-runner.service - LIO Userspace-passthrough daemon...
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: log file path now is '/var/log/tcmu-runner.log'
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: main:1302: Starting...
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: load_our_module:575: Inserted module 'target_core_user'
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler fbo is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler zbc is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler qcow is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmur_register_handler:92: Handler rbd is registered
Jul 09 09:13:18 juju-d407b0-zaza-6e2e71fc48f7-16 systemd[1]: Started tcmu-runner.service - LIO Userspace-passthrough daemon.
Jul 09 09:24:00 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: tcmu_rbd_open:1164 rbd/zaza_ec_metadata_pool.disk_ec_1: address: {10.5.3.45:0/3297752322}
Jul 09 09:24:10 juju-d407b0-zaza-6e2e71fc48f7-16 tcmu-runner[5905]: *** buffer overflow detected ***: terminated

In dmesg there's some faults printed which might be knock-on effects:

[Tue Jul  9 09:13:30 2024] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[Tue Jul  9 09:25:11 2024] ABORT_TASK: Found referenced iSCSI task_tag: 5
[Tue Jul  9 09:26:01 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:01 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:26:18 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:18 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:26:35 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:35 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:26:51 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:26:51 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:27:09 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:27:09 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:27:26 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:27:26 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:27:43 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:27:43 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:28:00 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:28:00 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:28:16 2024] tx_data returned -32, expecting 48.
[Tue Jul  9 09:28:16 2024] iSCSI Login negotiation failed.
[Tue Jul  9 09:28:22 2024] INFO: task kworker/0:0:8 blocked for more than 122 seconds.
[Tue Jul  9 09:28:22 2024]       Not tainted 6.8.0-36-generic #36-Ubuntu
[Tue Jul  9 09:28:22 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Jul  9 09:28:22 2024] task:kworker/0:0     state:D stack:0     pid:8     tgid:8     ppid:2      flags:0x00004000
[Tue Jul  9 09:28:22 2024] Workqueue: events target_tmr_work [target_core_mod]
[Tue Jul  9 09:28:22 2024] Call Trace:
[Tue Jul  9 09:28:22 2024]  <TASK>
[Tue Jul  9 09:28:22 2024]  __schedule+0x27c/0x6b0
[Tue Jul  9 09:28:22 2024]  schedule+0x33/0x110
[Tue Jul  9 09:28:22 2024]  schedule_timeout+0x157/0x170
[Tue Jul  9 09:28:22 2024]  wait_for_completion+0x88/0x150
[Tue Jul  9 09:28:22 2024]  target_put_cmd_and_wait+0x73/0xc0 [target_core_mod]
[Tue Jul  9 09:28:22 2024]  core_tmr_abort_task+0x29d/0x300 [target_core_mod]
[Tue Jul  9 09:28:22 2024]  target_tmr_work+0xe1/0x120 [target_core_mod]
[Tue Jul  9 09:28:22 2024]  process_one_work+0x16f/0x350
[Tue Jul  9 09:28:22 2024]  worker_thread+0x306/0x440
[Tue Jul  9 09:28:22 2024]  ? __pfx_worker_thread+0x10/0x10
[Tue Jul  9 09:28:22 2024]  kthread+0xf2/0x120
[Tue Jul  9 09:28:22 2024]  ? __pfx_kthread+0x10/0x10
[Tue Jul  9 09:28:22 2024]  ret_from_fork+0x47/0x70
[Tue Jul  9 09:28:22 2024]  ? __pfx_kthread+0x10/0x10
[Tue Jul  9 09:28:22 2024]  ret_from_fork_asm+0x1b/0x30
[Tue Jul  9 09:28:22 2024]  </TASK>
[Tue Jul  9 09:28:22 2024] INFO: task iscsi_trx:12919 blocked for more than 122 seconds.
[Tue Jul  9 09:28:22 2024]       Not tainted 6.8.0-36-generic #36-Ubuntu

This is reproducible, i.e. if I restart the tcmu-runner the buffer overflow comes back.

Versions:
tcmu-runner 1.5.4-6ubuntu1

uname -a
Linux juju-d407b0-zaza-6e2e71fc48f7-16 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Please let me know if I can provide more diagnostics or other info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant