OOM Crashes when Resizing Multiple Image Sizes to AVIF and WebP Formats

[ainsleyclark]

Question about an existing feature

What are you trying to achieve?

I’m using sharp to resize images into multiple formats, specifically AVIF and WebP. I’m running the application on Digital Ocean App Platform with 1GB RAM. When uploading images (around 1.5MB in size), it takes about 20 seconds to resize them into six different image sizes, but this process causes the app to crash and restart due to high memory usage.

A Digital Ocean representative confirmed that even a small number of image requests leads to memory exhaustion and app restarts. I'm using Next.js alongside Payload CMS (v3.0.0-beta.76). Although the repo link I provided and tested with on App Platform just uses express alongside sharp.

I suspect the memory spike is related to processing AVIF and WebP formats, but I’m unsure how to mitigate the issue with my current memory constraints.

Any insights or suggestions for optimizing memory usage or handling large image workloads in Sharp would be highly appreciated.

When you searched for similar issues, what did you find that might be related?

I’ve seen similar issues regarding high memory usage when resizing to AVIF and WebP formats, but I haven’t found a specific solution or optimization recommendation for my case, where multiple sizes of these formats are being generated simultaneously.

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

• Repo Link with sample images and additional context.
• Discord Thread with the Payload Team

Please provide sample image(s) that help explain this question

Under /assets in repo link.

Image Sizes:

const imageSizes = [
  // Original Size (for WebP & Avif)
  {
    name: "original_webp",
    width: undefined,
    height: undefined,
    formatOptions: {
      format: "webp",
      options: {
        quality: 80,
      },
    },
  },
  {
    name: "original_avif",
    width: undefined,
    height: undefined,
    formatOptions: {
      format: "avif",
      options: {
        quality: 60,
        effort: 1,
        chromaSubsampling: "4:4:4",
        bitdepth: 8,
        lossless: false,
      },
    },
  },
  // Thumbnail Sizes
  {
    name: "thumbnail",
    width: 200,
    height: undefined,
    position: "centre",
  },
  {
    name: "thumbnail_webp",
    width: 200,
    height: undefined,
    position: "centre",
    formatOptions: {
      format: "webp",
      options: {
        quality: 80,
      },
    },
  },
  {
    name: "thumbnail_avif",
    width: 200,
    height: undefined,
    position: "centre",
    formatOptions: {
      format: "avif",
      options: {
        quality: 60,
        effort: 1,
        chromaSubsampling: "4:4:4",
        bitdepth: 8,
        lossless: false,
      },
    },
  },
  // Mobile Sizes
  {
    name: "mobile",
    width: 500,
    height: undefined,
  },
  {
    name: "mobile_webp",
    width: 500,
    height: undefined,
    formatOptions: {
      format: "webp",
      options: {
        quality: 80,
      },
    },
  },
  {
    name: "mobile_avif",
    width: 500,
    height: undefined,
    formatOptions: {
      format: "avif",
      options: {
        quality: 60,
        effort: 1,
        chromaSubsampling: "4:4:4",
        bitdepth: 8,
        lossless: false,
      },
    },
  },
  // Tablet Sizes
  {
    name: "tablet",
    width: 800,
    height: undefined,
  },
  {
    name: "tablet_webp",
    width: 800,
    height: undefined,
    formatOptions: {
      format: "webp",
      options: {
        quality: 80,
      },
    },
  },
  {
    name: "tablet_avif",
    width: 800,
    height: undefined,
    formatOptions: {
      format: "avif",
      options: {
        quality: 60,
        effort: 1,
        chromaSubsampling: "4:4:4",
        bitdepth: 8,
        lossless: false,
      },
    },
  },
  // Desktop Sizes
  {
    name: "desktop",
    width: 1200,
    height: undefined,
  },
  {
    name: "desktop_webp",
    width: 1200,
    height: undefined,
    formatOptions: {
      format: "webp",
      options: {
        quality: 80,
      },
    },
  },
  {
    name: "desktop_avif",
    width: 1200,
    height: undefined,
    formatOptions: {
      format: "avif",
      options: {
        quality: 60,
        effort: 1,
        chromaSubsampling: "4:4:4",
        bitdepth: 8,
        lossless: false,
      },
    },
  },
];

module.exports = imageSizes;

[lovell]

AVIF encoding is highly CPU and memory intensive, enough that 1GB should probably be considered a constrained environment for the task.

You can use .avif({ effort }) to reduce the CPU cost of AVIF encoding at the expense of larger file sizes - see https://sharp.pixelplumbing.com/api-output#avif

You might want to check that App Platform is reporting the relevant number of CPU cores and RAM available via cgroups, e.g. watch out for underlying physical cores vs available virtual cores.

You can use sharp.concurrency(1) to limit the number of threads that libvips will spawn - see https://sharp.pixelplumbing.com/api-utility#concurrency

The forthcoming libvips v8.16.0 contains logic to further limit the number of threads that libaom can spawn, which for many scenarios will reduce contention and speed things up.

[ainsleyclark]

Thanks for your reply @lovell, appreciate it.

• I did try to use avif:effort:0 as well as sharp.concurrency(1) but it didn't have much effect and the app still restarted.
• Would you mind elaborating on the cgroups part?

I think I may have to weigh up cost vs benefit. Do you know if the webp encoding has big affect on RAM/CPU resource in comparison to avif?

[Rushs321]

Thanks for your reply @lovell, appreciate it.

• I did try to use avif:effort:0 as well as sharp.concurrency(1) but it didn't have much effect and the app still restarted.
• Would you mind elaborating on the cgroups part?

I think I may have to weigh up cost vs benefit. Do you know if the webp encoding has big affect on RAM/CPU resource in comparison to avif?

webp is much better.