Server-ended room leaks every local track (publisher PC closed before the unpublish loop; AudioSourceCapture thread stranded per room)

[sgu-bithuman]

Summary

When a room ends server-side (room deleted / participant removed / Leave{Disconnect} from the signal), every local track the participant had published is leaked for the life of the process, together with its native RtpSender/MediaStreamTrack. For a local audio track this strands one AudioSourceCapture thread (~96 MB RSS on our workers) per server-ended room. A client-initiated Room::close() does not leak.

Observed on every python SDK version we tested (livekit 1.1.8 … 1.1.19, i.e. livekit-ffi up to 0.12.79 / livekit crate 0.9.1); the defect is in the Rust core, not the python wrapper. Line numbers below are at tag livekit-ffi/v0.12.79 (commit bae4df2); the same code is on main today.

Cause: the publisher PC is closed before the unpublish loop runs

Server-ended path:

1. livekit/src/rtc_engine/rtc_session.rs:1426 — on_signal_event(Leave) → on_session_disconnected → SessionEvent::Close (:2007)
2. livekit/src/rtc_engine/mod.rs:752 — EngineInner::close → session.close(reason) (:763)
3. livekit/src/rtc_engine/rtc_session.rs:2015 — SessionInner::close → self.publisher_pc.close() (:2023) with all senders still attached
4. livekit/src/rtc_engine/mod.rs:772 — EngineEvent::Disconnected → livekit/src/room/mod.rs:1807 handle_disconnected → spawns RoomSession::close (:1182)
5. livekit/src/room/mod.rs:1186-1188 — the unpublish loop, unpublish_track(sid) per publication
6. livekit/src/room/participant/local_participant.rs:670 — self.inner.rtc_engine.remove_track(sender)?;
7. livekit/src/rtc_engine/rtc_session.rs:1918 — publisher_pc.peer_connection().remove_track(sender)? → libwebrtc PeerConnection::RemoveTrackOrError refuses with INVALID_STATE "PeerConnection is closed." (pc/peer_connection.cc, IsClosed() check)
8. The ? at step 6 aborts unpublish_track, so track.set_transceiver(None), the local_track_unpublished callback and publication.set_track(None) (local_participant.rs:671-679) never run.

On the client-initiated path RoomSession::close runs the unpublish loop before rtc_engine.close, so remove_track succeeds and nothing leaks.

Why the closed PC keeps the track: RtpSenderBase::Stop() (pc/rtp_sender.cc) detaches but never releases track_, and both RemoveTrack and SetTrack(nullptr) are refused once the PC is closed — so after step 3 there is no API left that can drop the sender's reference to the MediaStreamTrack, and the AudioSourceCapture thread behind a local audio track runs until process exit.

Log line seen on the stock build during a server-ended disconnect (with the ? it is the error that aborts the unpublish):

livekit::room::participant::local_participant - failed to remove track TR_... from rtc engine: Rtc(RtcError { error_type: InvalidState, message: "PeerConnection is closed." })

Reproduction (python SDK, any 1.1.x)

# pip install livekit==1.1.19 livekit-api
import asyncio, os, gc
from livekit import rtc, api
from livekit.api import AccessToken, VideoGrants

def census():  # AudioSourceCapture threads in this process
    return sum(open(f"/proc/self/task/{t}/comm").read().startswith("AudioSourceCapt")
               for t in os.listdir("/proc/self/task"))

async def main():
    url, key, sec, name = os.environ["LIVEKIT_URL"], os.environ["LIVEKIT_API_KEY"], os.environ["LIVEKIT_API_SECRET"], "leak-repro"
    tok = AccessToken(key, sec).with_identity("p").with_grants(VideoGrants(room_join=True, room=name, can_publish=True)).to_jwt()
    room = rtc.Room(); ev = asyncio.Event(); room.on("disconnected", lambda *a: ev.set())
    await room.connect(url, tok)
    src = rtc.AudioSource(16000, 1); track = rtc.LocalAudioTrack.create_audio_track("a", src)
    await room.local_participant.publish_track(track, rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_MICROPHONE))
    await asyncio.sleep(1); print("after publish:", census())          # 1
    lk = api.LiveKitAPI(url, key, sec); await lk.room.delete_room(api.DeleteRoomRequest(room=name)); await lk.aclose()
    await asyncio.wait_for(ev.wait(), 15)                             # server-ended
    await room.disconnect()
    del room, src, track; gc.collect(); await asyncio.sleep(10); gc.collect()
    print("after server-ended + drop + 10 s:", census())              # stock: 1 (leaked)  patched: 0
asyncio.run(main())

Replace delete_room with await room.disconnect() while connected (client-ended) and the count goes to 0 on the stock build — that is the control.

Measurement

livekit 1.1.19 wheel, liblivekit_ffi.so 0.12.79, Linux x86_64, counting AudioSourceCapture threads after all python wrappers and FFI handles are dropped and a 10 s settle:

build	server-ended room	client-ended room (control)
stock 1.1.19	after publish 1 → final 1 (leaked; AudioSourceCapt, network_thread, signaling_threa, worker_thread 0, rtc-low-prio all still resident)	1 → 0
patched (PR below)	1 → 0	1 → 0

Dropping the FFI handles one at a time on the stock build (source, publication, track, room) never releases the thread on the server-ended path; on the client-ended path it is released when the track handle drops.

Fix

Two small changes in the Rust core plus one null guard in webrtc-sys (PR follows):

1. SessionInner::close — remove every sender from the publisher PC before publisher_pc.close(), so the tracks are released while libwebrtc still accepts RemoveTrack.
2. LocalParticipant::unpublish_track — do not abort the bookkeeping when rtc_engine.remove_track fails (log and continue), so the publication drops its track reference on the already-closed path.
3. webrtc-sys RtpSender::track() — a sender legitimately has no track after RemoveTrack; return null instead of passing a null scoped_refptr to get_or_create_media_stream_track (which dereferences it; without this guard SessionInner::remove_track's sender.track() crashes after change 1).