DELETE Request Stuck with @aws-sdk/client-s3 v3.717.0 When Using DigitalOcean Spaces

[mgsgde]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

The DELETE operation using @aws-sdk/client-s3 (version 3.717.0) gets stuck when interacting with DigitalOcean Spaces. The request is sent, but the process appears to hang indefinitely without receiving a response. This behavior is observed specifically with the DeleteObjectsCommand.

Environment:
• @aws-sdk/client-s3 version: 3.717.0
• Node.js version: v21.7.1
• DigitalOcean Spaces
• Operating System: macOS

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v21.7.1

Reproduction Steps

1.	Initialize an S3 client with DigitalOcean Spaces.
2.	Attempt to delete objects using the DeleteObjectsCommand.
3.	Observe that the operation hangs and does not complete.

Observed Behavior

The DELETE operation does not proceed past sending the HTTP request. The process does not throw any errors or complete the operation, making it impossible to debug effectively.

Expected Behavior

The DELETE operation should successfully remove the specified objects from the bucket and return a response without hanging.

Possible Solution

No response

Additional Information/Context

Other S3 commands (e.g., ListObjectsV2Command) work as expected. The DELETE issue seems isolated to the DeleteObjectsCommand.

[aBurmeseDev]

Hi @mgsgde - thanks for reaching out.

In order for us to further look into this, can you share the followings:

• A minimal SDK reproducible code
• Capture and share any logging requests by adding middleware to S3 client

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

• Try downgrading to an older version of SDK and share the result

Also are you able to test this same operation with AWS S3 bucket instead? We won't be able to provide support if the issue is specific to third-party service.

[n-lavrenko]

Hello guys!
and aBurmeseDev
Please, take a look, it's like a critical bug.

I have the same issue with the same environment.
Previously, 6 month ago everything worked, but now -- not.
I spent whole day to figure out what a hell is going on...

Upload is working. DeleteObjects - not.
After send a request DeleteObjects: in ~5 minutes I receive an error in try/catch:

catching callstack:

Error: char 's' is not expected.:1:1
  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.
    at XMLParser.parse (/Users/n-lavrenko/dev/joiny-api/node_modules/fast-xml-parser/src/xmlparser/XMLParser.js:30:21)
    at /Users/n-lavrenko/dev/joiny-api/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:177:26
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async parseXmlErrorBody (/Users/n-lavrenko/dev/joiny-api/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:198:17)
    at async de_CommandError (/Users/n-lavrenko/dev/joiny-api/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4874:11)
    at async /Users/n-lavrenko/dev/joiny-api/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/n-lavrenko/dev/joiny-api/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:485:18
    at async /Users/n-lavrenko/dev/joiny-api/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/n-lavrenko/dev/joiny-api/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:286:18
    at async /Users/n-lavrenko/dev/joiny-api/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22

and more data in error:

{
  message: "char 's' is not expected.:1:1
    Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.",
  $responseBodyText: "stream timeout",
  $response: below ⬇️:
}

error.$response:

{
  "statusCode": 408,
  "reason": "Request Timeout",
  "headers": {
    "strict-transport-security": "max-age=15552000; includeSubDomains; preload",
    "content-length": "14",
    "content-type": "text/plain",
    "x-envoy-upstream-healthchecked-cluster": "",
    "date": "Tue, 31 Dec 2024 13:10:23 GMT",
    "connection": "close"
  },
  "body": "stream timeout"
}

I've tried everything. It seems like a bug on maybe third-party lib. The request seems like not sending.
Here is the code to reproduce the issue:

import { DeleteObjectsCommand, S3Client } from '@aws-sdk/client-s3';

const client = new S3Client({
  endpoint: process.env.DIGITAL_OCEAN_SPACES_ENDPOINT,
  forcePathStyle: false,
  region: 'fra1',
  credentials: {
    accessKeyId: process.env.DIGITAL_OCEAN_SPACES_KEY_ID,
    secretAccessKey: process.env.DIGITAL_OCEAN_SPACES_KEY,
  },
});

(async () => {
  client.middlewareStack.add(
    (next, context) => async (args) => {
      console.log('AWS SDK context', context.clientName, context.commandName);
      console.log('AWS SDK request input', args.input);
      const result = await next(args);
      console.log('AWS SDK request output:', result.output);
      return result;
    },
    {
      name: 'MyMiddleware',
      step: 'build',
      override: true,
    },
  );

  try {
    console.log('Deleting object...');
    const res = await client.send(
      new DeleteObjectsCommand({
        Bucket: process.env.DIGITAL_OCEAN_SPACES_BUCKET,
        Delete: {
          Objects: [{ Key: 'order/6772f4f33c88f5ee2c9ceb6e.HEIC' }],
        },
      }),
    );
    console.log('Delete result:', res);
  } catch (err) {
    console.error(err);
  }
})();

[mgsgde]

Downgrading to version 3.645.0 resolved the issue. For now, I am reverting to the previous version as a workaround.

I have also contacted DigitalOcean support, as the issue could be on their side, even though backward compatibility in the SDK makes this less likely.

[n-lavrenko]

@mgsgde,
Thank you very much!
I'll try and return back with the result.

[n-lavrenko]

I've tested quickly - it doesn't work

[n-lavrenko]

FYI:
I've rewrote my code into single deletion files, it's still working using latest version of your library.

[aBurmeseDev]

According to the error response, the server seems to be responding with "stream timeout" instead of a valid response.

This may be because the connection to the DigitalOcean Spaces endpoint may have timed out, causing the server to return a "stream timeout" response.

I've rewrote my code into single deletion files, it's still working using latest version of your library.

@n-lavrenko - can you share the version that's last working for you? Also I noticed typo in your code for DeleteObjectCommand.

As next step of troubleshooting, I'd like to isolate the issue to SDK first, (third-party not involved). Can you try the SDK code below to deletes an object in S3 bucket?

import { S3Client, DeleteObjectCommand } from "@aws-sdk/client-s3"; 
const client = new S3Client();

  client.middlewareStack.add(
    (next, context) => async (args) => {
      console.log('AWS SDK context', context.clientName, context.commandName);
      console.log('AWS SDK request input', args.input);
      const result = await next(args);
      console.log('AWS SDK request output:', result.output);
      return result;
    },
    {
      name: 'MyMiddleware',
      step: 'build',
      override: true,
    },
  );

const input = { 
  Bucket: "STRING_VALUE", 
  Key: "STRING_VALUE", 
};
const command = new DeleteObjectCommand(input);
const response = await client.send(command);

Please share the output for our investigation, thank you for your corporation!

[n-lavrenko]

Hello @aBurmeseDev !

Also I noticed typo in your code for DeleteObjectCommand
What the typo?
FYI: I used DeleteObjectsCommand (not the DeleteObjectCommand).

As I said before, DeleteObjectsCommand previously worked, but now work only DeleteObjectCommand.
So, when I said above:

I've rewrote my code into single deletion files, it's still working using latest version of your library.

I meant that I rewrote DeleteObjectsCommand to DeleteObjectCommand which no have any problems.

So your code with single file deletion is not about the issue.
The issue is only with bulk deletion: DeleteObjectsCommand

[pdesmarais]

So, had the same issue as detailed in previous comments with DO Spaces timing out with the DeleteObjectsCommand (multiple) but not the DeleteObjectCommand (single).

Noticed in the error that it was smithy/node-http-handler that was complaining. Had updated packages recently.

First difference from the people above mentioning that downgrading to an older version of s3client fixed the issue for them is that it didn't fix it for us.

Then, I noticed that we are explicitly requiring @smithy/node-http-handler in our package.json. We were now using 3.3.1 of that package (before the latest package updates, we were using 3.3.0).

Downgraded smithy/node-http-handler back to 3.3.0 and everything was working normally.

Now, we're using latest s3client (3.723) and smithy/node-http-handler (3.3.0) and everything works.
What's weird is that the latest s3client uses smithy 4.0.0 but if I explicitly use that version in the package.json, it doesn't work.

Hope this helps anyone!