Skip to content

fix(tcp): prevent select error watchdog loop (EPROT-79) - #183

Open
zyk6271 wants to merge 2 commits into
espressif:mainfrom
zyk6271:agent/prevent-tcp-select-watchdog-loop
Open

fix(tcp): prevent select error watchdog loop (EPROT-79)#183
zyk6271 wants to merge 2 commits into
espressif:mainfrom
zyk6271:agent/prevent-tcp-select-watchdog-loop

Conversation

@zyk6271

@zyk6271 zyk6271 commented Aug 12, 2026

Copy link
Copy Markdown

Summary

  • validate descriptors before adding them to an fd_set
  • monitor exceptional conditions only for connected sockets and route them through the normal connection error path
  • add bounded exponential backoff and actionable errno logging when select() repeatedly fails
  • release every accepted socket and copied address string when the client table is full or node creation fails
  • recover stale TCP slave clients after fatal lwIP/network errors without deleting persistent TCP master node definitions
  • make close-all transaction cleanup and sparse-node timeout scanning deterministic

Root cause

Two failure modes combined in the reported long-running failure:

  1. The TCP slave accepted a socket and allocated its address string before checking the five-node limit. The rejection path closed the socket but did not release the copied address, so repeated connection attempts steadily exhausted internal heap. Under heap pressure, accept()/select() returned ENOMEM, Wi-Fi allocation failed, and connectivity was lost.
  2. The driver retried select() immediately on persistent errors. The high-priority mb_drv_tcp_task could therefore starve CPU0's IDLE task and trigger the task watchdog.

The previous code also copied the full read set into exceptfds, including non-socket descriptors, and did not consume exceptional client conditions.

Changes

  • check for a free virtual descriptor before allocating a node
  • centralize cleanup of rejected accepted sockets and address metadata
  • preserve accept() errno, avoid close(-1), and handle address/strdup() failures
  • close all stale slave clients on fatal network-stack errors, then purge their transactions
  • keep TCP master node descriptors intact so their reconnect configuration is not lost
  • use the virtual node index, never a negative socket descriptor, with open_set
  • scan all virtual descriptors during timeout processing so holes cannot skip active nodes
  • apply a 1-tick to 1-second bounded exponential backoff to repeated select() failures

Hardware validation

Target: ESP32-D0WD-V3, ESP-IDF v6.0.2, esp-modbus v2.1.3 component sources, Modbus TCP slave with five client slots.

A/B resource-pressure fault injection

The fault injector holds all five clients open, then repeatedly creates extra TCP connections.

Unpatched firmware:

overflow=100   free_internal=91836
overflow=800   free_internal=76084
overflow=2300  free_internal=4202
overflow=3600  free_internal=1264
held clients lost at approximately 3700 overflows
E (...) mb_driver: task select error, errno=12 (Not enough space)
W (...) wifi:mem fail
W (...) wifi:m f null
E (...) task_wdt: IDLE0 did not reset; CPU 0: mb_drv_tcp_task

Patched firmware, with the same temporary heap telemetry:

overflow=100   free_internal=94328
overflow=5000  free_internal=94348
PASS: attempts=6000 rejected=5999 admitted=1; service recovered

A fresh run after the final source-review fixes:

holding 5 active client(s)
progress=6000/6000 rejected=6000 admitted=0 held_alive=5
PASS: attempts=6000 rejected=6000 admitted=0; service recovered

Additional fault injection/regression

PASS: node-table holes/refill cycles=100
malformed MBAP/PID/UID/truncated frame cases: 4/4; service alive after each
Wi-Fi disabled for 15 seconds: held clients discarded; original IP and Modbus service recovered

During the patched test windows there was no errno=12, wifi:mem fail, wifi:m f null, select busy loop, or task WDT. A single errno=113 while Wi-Fi was physically unavailable was expected and recovery completed after reassociation.

Build validation

  • rebuilt TCP master, TCP slave, TCP driver, and TCP utility objects
  • linked and flashed the complete application successfully
  • firmware size: 0xfdb30; smallest app partition remained 32% free
  • git diff --check passed for all five changed library files

@zyk6271
zyk6271 marked this pull request as ready for review August 12, 2026 02:13
@github-actions github-actions Bot changed the title fix(tcp): prevent select error watchdog loop fix(tcp): prevent select error watchdog loop (EPROT-79) Aug 12, 2026
@alisitsyn

alisitsyn commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Hi @ zyk6271,

Thanks for the PR proposal. Unfortunately, I don't have time to review and test the related cases for now. I will take a look once have time for this. Please consider to extend your PR with the logs of stress testing with fault injection.

Release accepted sockets and address metadata on all failure paths, close
stale slave clients after fatal network-stack errors, and keep the master
node table intact for reconnects. Also fix sparse-node timeout scanning and
avoid using negative socket descriptors as fd_set indexes.
@zyk6271

zyk6271 commented Aug 17, 2026

Copy link
Copy Markdown
Author

Implemented the requested stress/fault-injection follow-up in commit 466cdb57f72f69a52707ba2b574a5939dd2dfc2a and added the complete A/B data to the PR description.

Key results on ESP32-D0WD-V3 / ESP-IDF v6.0.2:

unpatched:
  free_internal: 91836 @ 100 overflows
                 76084 @ 800
                  4202 @ 2300
                  1264 @ 3600
  held clients lost at ~3700, followed by errno=12 / Wi-Fi allocation failures / task WDT

patched:
  free_internal: 94328 @ 100 overflows
                 94348 @ 5000
  PASS: 6000/6000 extra connections rejected, all 5 held clients alive
  PASS: service recovered after held clients were released
  PASS: 100 sparse-node close/refill cycles
  PASS: 4 malformed-frame cases, service alive after each
  PASS: 15-second Wi-Fi outage and Modbus recovery

No errno=12, wifi:mem fail, wifi:m f null, select busy loop, or task WDT was observed in the patched windows.

The follow-up also fixes the accepted-socket/address leak itself, preserves TCP master reconnect definitions during fatal-error recovery, and removes negative-fd FD_ISSET calls in both master and slave close handlers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants