Crash in channelClosed

[fabianfett]

Describe the bug

When running the lambda tests we observed a crash in:

    private func channelClosed(_ channel: any Channel) {
        switch (self.connectionState, self.closingState) {
        case (_, .closed):
            fatalError("Invalid state: \(self.connectionState), \(self.closingState)") // <---- crash

fatalError log:

AWSLambdaRuntime/LambdaRuntimeClient.swift:270: Fatal error: Invalid state: connected(SocketChannel { BaseSocket { fd=214 }, active = false, localAddress = Optional([IPv4]127.0.0.1/127.0.0.1:36846), remoteAddress = Optional([IPv4]127.0.0.1/127.0.0.1:33103) }, AWSLambdaRuntime.LambdaChannelHandler<AWSLambdaRuntime.LambdaRuntimeClient>), closed

Crash - Logs:

Thread 34 "NIO-SGLTN-1-#12" crashed:

  0               0x00007f5df91f9718 _assertionFailure(_:_:file:line:flags:) + 264 in libswiftCore.so
  1 [ra]          0x0000560234e37935 LambdaRuntimeClient.channelClosed(_:) + 1636 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-aws-lambda-runtime/Sources/AWSLambdaRuntime/HTTPClient/LambdaRuntimeClient.swift:270:13
  2 [ra]          0x0000560234918a8d closure #1 in closure #3 in LambdaRuntimeClient.makeOrGetConnection() + 188 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-aws-lambda-runtime/Sources/AWSLambdaRuntime/HTTPClient/LambdaRuntimeClient.swift:371:35
  3 [ra] [thunk]  0x000056023491c234 partial apply for closure #1 in closure #3 in LambdaRuntimeClient.makeOrGetConnection() + 19 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
  4 [ra] [thunk]  0x000056023491bf8f thunk for @callee_guaranteed (@guaranteed isolated A) -> (@out A1, @error @owned Error) + 14 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
  5 [ra] [thunk]  0x000056023491c01b partial apply for thunk for @callee_guaranteed (@guaranteed isolated A) -> (@out A1, @error @owned Error) + 42 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
  6 [ra] [thunk]  0x000056023491c162 thunk for @escaping @callee_guaranteed (@guaranteed isolated A) -> (@out A1, @error @owned Error) + 17 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
  7 [ra] [thunk]  0x000056023491c20b partial apply for thunk for @escaping @callee_guaranteed (@guaranteed isolated A) -> (@out A1, @error @owned Error) + 42 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
  8 [ra] [system] 0x000056023491c0e6 closure #1 in Actor.assumeIsolated<A>(_:file:line:) + 181 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
  9 [ra] [system] 0x0000560234918cd6 Actor.assumeIsolated<A>(_:file:line:) + 469 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
 10 [ra]          0x00005602349189a1 closure #3 in LambdaRuntimeClient.makeOrGetConnection() + 208 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-aws-lambda-runtime/Sources/AWSLambdaRuntime/HTTPClient/LambdaRuntimeClient.swift:369:22
 11 [ra]          0x0000560234c48341 closure #1 in EventLoopFuture.whenComplete(_:) + 368 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOCore/EventLoopFuture.swift:900:13
 12 [ra]          0x0000560234f99b36 CallbackList._run() + 517 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOCore/EventLoopFuture.swift:106:40
 13 [ra]          0x0000560234f9a3f2 EventLoopPromise._resolve<A>(value:) + 273 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOCore/EventLoopFuture.swift:280:42
 14 [ra]          0x0000560234f9a2c1 EventLoopPromise.succeed(_:) + 176 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOCore/EventLoopFuture.swift:214:14
 15 [ra]          0x0000560234d09550 closure #3 in BaseSocketChannel.close0(error:mode:promise:) + 159 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/BaseSocketChannel.swift:922:31
 16 [ra]          0x0000560234d7ac80 closure #1 in SelectableEventLoop.run(_:) + 175 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:616:17
 17 [ra] [thunk]  0x0000560234d7f044 partial apply for closure #1 in SelectableEventLoop.run(_:) + 19 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
 18 [ra]          0x0000560235097eb2 withAutoReleasePool<A>(_:) + 33 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:47:16
 19 [ra]          0x000056023509be00 SelectableEventLoop.run(_:) + 95 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:613:9
 20 [ra]          0x000056023509d010 SelectableEventLoop.runOneLoopTick(selfIdentifier:) + 1311 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:807:22
 21 [ra]          0x000056023509d508 SelectableEventLoop.run() + 695 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:921:38
 22 [ra]          0x000056023507aa09 static MultiThreadedEventLoopGroup.runTheLoop(thread:uniqueID:parentGroup:canEventLoopBeShutdownIndividually:selectorFactory:initializer:metricsDelegate:_:) + 968 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:115:22
 23 [ra]          0x0000560234d401a0 closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:uniqueID:parentGroup:selectorFactory:initializer:metricsDelegate:) + 415 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:137:41
 24 [ra] [thunk]  0x0000560234d43808 partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:uniqueID:parentGroup:selectorFactory:initializer:metricsDelegate:) + 87 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
 25 [ra] [thunk]  0x0000560234d9816f thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () + 14 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>
 26 [ra]          0x0000560234d995e8 closure #1 in closure #1 in static ThreadOpsPosix.run(handle:args:) + 887 in swift-aws-lambda-runtimePackageTests.xctest at /workspace/session/repos/swift-nio/Sources/NIOPosix/ThreadPosix.swift:184:21
 27 [ra] [thunk]  0x0000560234d996a9 @objc closure #1 in closure #1 in static ThreadOpsPosix.run(handle:args:) + 8 in swift-aws-lambda-runtimePackageTests.xctest at //<compiler-generated>

Expected Behavior

Should not crash.

Current Behavior

Crashes.

Reproduction Steps

Run tests. Might crash eventually. Crashed on 6.0.

Possible Solution

No response

Additional Information/Context

No response

AWS SWIFT SDK version used

main

Compiler and Version used

Swift 6.0

Operating System and version

x86_64 Linux (Red Hat Enterprise Linux 9.7 (Plow))

[sebsto]

Analysis of Bug #624: Crash in channelClosed

Bug Report Summary

Issue: Fatal error crash in LambdaRuntimeClient.channelClosed() method

Error Message:

Fatal error: Invalid state: connected(SocketChannel { BaseSocket { fd=214 }, active = false, localAddress = Optional([IPv4]127.0.0.1/127.0.0.1:36846), remoteAddress = Optional([IPv4]127.0.0.1/127.0.0.1:33103) }, AWSLambdaRuntime.LambdaChannelHandler<AWSLambdaRuntime.LambdaRuntimeClient>), closed

Location: AWSLambdaRuntime/LambdaRuntimeClient.swift:270

Conditions: Intermittent crash during test execution, particularly under heavy load

---

Understanding the State Machines

1. LambdaRuntimeClient Actor State (Double State Management)

The LambdaRuntimeClient actor maintains three separate state machines:

• connectionState:

  • .disconnected
  • .connecting([continuations])
  • .connected(channel, handler)

• closingState:

  • .notClosing
  • .closing(continuation)
  • .closed

• lambdaState:

  • .idle(previousRequestID: String?)
  • .waitingForNextInvocation
  • .waitingForResponse(requestID: String)
  • .sendingResponse(requestID: String)
  • .sentResponse(requestID: String)

2. LambdaChannelHandler State (Inside the Channel Pipeline)

The LambdaChannelHandler maintains its own state machine:

• State:

  • .disconnected
  • .connected(context, lambdaState)
  • .closing

• Nested LambdaState:

  • .idle
  • .waitingForNextInvocation(continuation)
  • .waitingForResponse
  • .sendingResponse
  • .sentResponse(continuation)

---

The Race Condition

Crash Location

The crash occurs at line 270 in LambdaRuntimeClient.swift:

private func channelClosed(_ channel: any Channel) {
    switch (self.connectionState, self.closingState) {
    case (_, .closed):
        fatalError("Invalid state: \(self.connectionState), \(self.closingState)") // <---- CRASH

Crash State

The crash log shows:

Invalid state: connected(SocketChannel { ... active = false ... }), closed

This reveals the critical race condition: The connectionState is .connected while closingState is .closed, which should be impossible according to the state machine invariants.

---

Root Cause Hypothesis

The Race Occurs Between Two Asynchronous Callbacks

1. connectionWillClose(channel:) - Called from LambdaChannelHandler.channelRead() when the server sends "Connection: close" header
2. channel.closeFuture.whenComplete - Called when the channel actually closes

The Problematic Sequence

In makeOrGetConnection() at line 369:

channel.closeFuture.whenComplete { result in
    self.assumeIsolated { runtimeClient in
        runtimeClient.channelClosed(channel)
        runtimeClient.connectionState = .disconnected  // Line 373
    }
}

In connectionWillClose() at line 414:

nonisolated func connectionWillClose(channel: any Channel) {
    self.assumeIsolated { isolated in
        switch isolated.connectionState {
        case .connected(let stateChannel, _):
            guard channel === stateChannel else {
                isolated.closingConnections.append(channel)
                return
            }
            isolated.connectionState = .disconnected  // Line 434
        }
    }
}

---

The Race Condition Timeline

1. T1: Server sends response with "Connection: close" header
2. T2: LambdaChannelHandler.channelRead() processes response, calls delegate.connectionWillClose(channel)
3. T3: connectionWillClose() executes via assumeIsolated, sets connectionState = .disconnected
4. T4: Response continuation is resumed, closingState transitions through .closing(continuation) to .closed
5. T5: Channel actually closes, triggering closeFuture.whenComplete
6. T6: channelClosed() is called via assumeIsolated
7. T7: BUT - between T3 and T6, another request might have called makeOrGetConnection(), creating a NEW connection
8. T8: The new connection sets connectionState = .connected(newChannel, newHandler)
9. T9: Now channelClosed() executes with connectionState = .connected (the NEW channel) and closingState = .closed (from the OLD close operation)
10. CRASH: The guard case (_, .closed) triggers the fatalError

---

Key Issues

1. Temporal Coupling Between Two Callbacks

• connectionWillClose() is called BEFORE the channel actually closes
• closeFuture.whenComplete is called AFTER the channel closes
• Between these two events, the actor can process other requests and create new connections

2. State Confusion

• closingState = .closed refers to the OLD connection's close operation
• connectionState = .connected refers to a NEW connection
• channelClosed() receives the OLD channel but sees the NEW connection state

3. Missing Channel Identity Check

The channelClosed() method doesn't verify that the channel being closed matches the current connectionState.connected channel before asserting the state invariant.

---

Why It's Intermittent

The race only manifests when:

1. Server closes connection (keepAlive: false or "Connection: close")
2. The close operation completes (closingState reaches .closed)
3. A new request arrives and creates a new connection
4. The old channel's closeFuture fires AFTER the new connection is established
5. All of this happens in the narrow window between connectionWillClose() and closeFuture.whenComplete

This is why it's rare and only shows up under heavy test load on fast machines.

---

Proposed Fix Strategy

Option 1: Track Channel Identity in closingState

enum ClosingState {
    case notClosing
    case closing(CheckedContinuation<Void, Never>, channel: Channel)
    case closed(channel: Channel, until: ContinuousClock.Instant)
}

Option 2: Use Channel-Specific Tracking

Add a Set<ObjectIdentifier> to track channels that are in the process of closing, and check membership before asserting state invariants.

Option 3: Relax the Assertion

The case (_, .closed) might actually be valid if it refers to a different channel. The assertion should check channel identity:

case (_, .closed):
    // This is only invalid if we're trying to close the same channel twice
    // But if this is a different channel (old one), it's fine to ignore
    return

Option 4: Consolidate State Updates

Ensure connectionState and closingState are updated atomically, or use a single combined state enum.

---

Recommended Fix

I recommend Option 2 + Option 3 combined:

1. Track closing channels explicitly in a separate collection
2. In channelClosed(), check if the channel matches the current connection before asserting
3. If it's an old channel that's closing after we've moved on, simply remove it from tracking and return

This maintains the safety guarantees while handling the legitimate race condition where an old channel closes after a new connection has been established.

[sebsto]

I can't reproduce this. However, when running multiple swift test on a fast RHEL10 EC2 instance (c7a.12xlarge), one out of two or three run hangs. I have to kill the test suite.

I'm working on a fix based on the above proposal.