[BUG] 150-165% CPU busy-wait spinloop during idle state caused by leaked /dev/ptmx master descriptor in epoll event loop

[Tamila-2017]

Severity: High (Performance Degradation, Battery Drain, Thermal Throttling)
Affected Component: Terminal Tool Execution / PTY Lifecycle Management / Go Netpoller Integration

---

1. Title

[BUG] 150–165% CPU busy-wait spinloop during idle state caused by leaked /dev/ptmx master descriptor in epoll event loop

---

2. Environment

• Binary: /home/ai/.local/bin/agy (ELF 64-bit LSB executable, x86-64, dynamically linked)
• Runtime: Go runtime (multi-threaded, with netpoller / epoll integration)
• Operating System: Linux x86_64 (Kernel 6.x)
• Execution Context: Terminal CLI session via agy --conversation <id>

---

3. Problem Summary

During an interactive CLI session where multiple commands are executed via the run_command tool, the agy process enters an infinite, non-yielding busy-wait spinloop.

Even after all commands have completed and the assistant is completely idle waiting for user input, the agy process consumes 150%–165% CPU (pegging 1.5 to 2 physical CPU cores at 100%). This state persists indefinitely until the session is killed.

---

4. Root Cause Analysis

System-level kernel tracing (strace, /proc/<PID>/fd, /proc/<PID>/fdinfo/<epfd>) identified the exact sequence:

1. PTY Master Allocation:
   When executing subprocesses for tools, agy allocates a pseudo-terminal master via /dev/ptmx and registers the descriptor in its internal epoll instance (fd 5) with flags EPOLLIN | EPOLLOUT | EPOLLET | EPOLLRDHUP (event mask 0x8000201d).

2. Descriptor Leak on Subprocess Termination:
   When the child process exits, the slave side (/dev/pts/X) is closed by the OS kernel. However, agy's cleanup routine fails to:

   • Deregister the master file descriptor from epoll (epoll_ctl(epfd, EPOLL_CTL_DEL, master_fd, NULL));
   • Close the master descriptor (close(master_fd)).

3. Linux Kernel PTY Invariant (drivers/tty/pty.c):
   Under Linux, once the slave end of a PTY is closed, the master descriptor (/dev/ptmx) enters an orphaned hangup state. The kernel immediately and continuously asserts EPOLLIN | EPOLLOUT | EPOLLHUP. Any subsequent read on the master returns EIO (Input/output error).

4. Event Loop Spinlock / Thundering Herd:
   Because the descriptor is never removed from epoll, epoll_pwait(5, ...) returns immediately on every single invocation without blocking. Multiple worker threads in the Go runtime get woken up concurrently:

   epoll_pwait(5, [{events=EPOLLIN|EPOLLOUT, data=...}], 128, 0, NULL, 0) = 1
   nanosleep({tv_sec=0, tv_nsec=3000}, NULL) = 0
   tgkill(141490, ..., SIGURG) = 0
   epoll_pwait(5, [{events=EPOLLIN|EPOLLOUT, data=...}], 128, 0, NULL, 0) = 1
   

   The 3-microsecond sleep and continuous preemption signals (SIGURG) keep 16 worker threads cycling endlessly, generating huge CPU overhead while performing 0 useful operations.

---

5. Diagnostic Evidence

A. Process CPU Usage (top -b -n 1 -p 141490)

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 141490 ai        20   0 2980388 474884 131648 S 150.0   1.5 137:58.00 agy

B. Leaked Descriptors (/proc/141490/fd)

lrwx------ 1 ai ai 64 Sep 8 21:22 66 -> /dev/ptmx
lr-x------ 1 ai ai 64 Sep 8 20:00 23 -> 'pipe:[1514615]'
l-wx------ 1 ai ai 64 Sep 8 20:00 25 -> 'pipe:[1514615]'
... [Dozens of uncollected pipes and ptmx descriptors]

C. Kernel Epoll State (/proc/141490/fdinfo/5)

pos:    0
flags:  02000002
tfd:       66 events: 8000201d data: 3fd360200400018c pos:0 ino:57 sdev:6

(Note: ino:57 sdev:6 corresponds directly to /dev/ptmx on devtmpfs).

D. System Call Trace (strace -p 141490 -e epoll_pwait)

[pid 141501] epoll_pwait(5, [{events=EPOLLIN|EPOLLOUT, data=...}], 128, 0, NULL, 0) = 1
[pid 141501] epoll_pwait(5, [{events=EPOLLIN|EPOLLOUT, data=...}], 128, 0, NULL, 0) = 1
[pid 144232] epoll_pwait(5, [{events=EPOLLIN|EPOLLOUT, data=...}], 128, 0, NULL, 0) = 1
[pid 141500] --- SIGURG {si_signo=SIGURG, ...} ---

---

6. Steps to Reproduce

1. Start an interactive session with agy --conversation <cid>.
2. Run a workflow that issues multiple tool executions (e.g., 20+ run_command invocations).
3. Complete the task and allow the agent to reach an idle prompt state (> ).
4. Monitor CPU usage with top or htop.
5. Observe that agy continues to consume 100%–160% CPU indefinitely despite zero user/agent activity.

---

7. Expected vs Actual Behavior

• Expected: When a child command terminates, its master PTY descriptor must be deregistered from epoll via EPOLL_CTL_DEL, all pipes and PTY descriptors closed, and agy CPU utilization during idle prompt must drop to 0.0%–0.2%.
• Actual: Master PTY descriptors leak inside epoll, triggering an unrecoverable busy-wait spinloop consuming 150%+ CPU.

---

8. Suggested Fix

In the subprocess / PTY management package:

1. Ensure that defer blocks in command execution handlers explicitly call epoll_ctl(epfd, unix.EPOLL_CTL_DEL, masterFd, nil) before closing masterFd.
2. Upon receiving EIO or EPOLLHUP from a PTY master in the reader loop, unconditionally remove the descriptor from the poller and close both ends.
3. Clean up associated stdout/stderr IPC pipes upon subprocess exit.