Time-skipping past activity’s startToCloseTimeout causes worker not to close

[ikonst]

Expected Behavior

worker.runUntil returns shortly after workflow ends

Actual Behavior

• worker.runUntil usually hangs, ostensibly because it fails to wait for the abandoned activity
• Logged SDK warning:

  WARN temporal_sdk_core::worker::activities: Activity not found on completion. This may happen if the activity has already been cancelled but completed anyway. task_token=TaskToken(<activity's token>) details=Status { code: NotFound, message: "invalid activityID or activity already timed out or invoking workflow is completed", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }
  

Steps to Reproduce the Problem

Pardon my TypeScript...

Assume activity foo:

export async function foo() {
}

and workflow workflow:

const { foo } = proxyActivities<typeof activities>({ startToCloseTimeout: "60 seconds" });

export async function workflow() {
  await foo();
}

Then let assume this test code:

test("foo", async () => {
  testEnv = await TestWorkflowEnvironment.createTimeSkipping();
  worker = Worker.create({
    connection: testEnv.nativeConnection,
    ...
  });

  let waited = false;
  const worker = await createWorker({
    activities: {
      foo: async function() { 
        if (!waited) {
          await testEnv.sleep("61 seconds");
          waited = true;
        }
      }
    },
  });
  
  await worker.runUntil(testEnv.client.workflow.execute(workflow, {}));
  // ^ gets stuck here:
}

Specifications

• Version: TypeScript SDK 1.11.1
• Platform: MacOS

Discussion

• Original Slack thread: https://temporalio.slack.com/archives/C01DKSMU94L/p1727789446627909