fix: disable libvips operation cache to stop image-conversion memory leak - #29
Merged
Conversation
…leak libvips keeps a process-global operation cache enabled by default. Every conversion caches its load operations, pinning image buffers across calls, so long-running services that convert many uploads leak memory. The cache never hits for one-shot conversions of unique buffers, so disable it via pyvips.cache_set_max(0), guarded by cache_get_max() so it runs once. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Long-running services that convert many uploads (e.g. an upload/S3 gateway) show steady process memory growth correlated with traffic that looks like a leak.
Root cause
FileValidator._convert_imageconverts every image with libvips viapyvips.Image.new_from_buffer(...).write_to_buffer(...). libvips keeps a process-global operation cache enabled by default (up to ~1000 operations, plus memory/file caps). Each cached operation pins references to its image buffers, so across conversions memory climbs until the caps are hit. Every upload is a unique buffer, so the cache never produces a hit: it is pure retained memory with no benefit for this workload.This is the standard libvips-in-a-server behavior; pyvips' own
examples/soak-test.pyprevents it withpyvips.cache_set_max(0).Measured (pyvips 3.1.1 / libvips 8.17.3)
Running the exact conversion in a loop and reading
pyvips.cache_get_size():new_from_bufferdecomposes into several cached ops, each pinning buffers)cache_set_max(0): stays 0Fix
Disable the libvips operation cache on the conversion path with
pyvips.cache_set_max(0), guarded by apyvips.cache_get_max()check so the setter effectively runs once. Theimport pyvipsstays local to_convert_image. No public API change; conversion output, formats, quality, and error handling are unchanged.Test
Added a TDD regression test asserting
pyvips.cache_get_size() == 0after a conversion. It only observes cache size (never mutates it), so it is order-independent: red onmain(nonzero), green with the fix. Full suite: 36 passing, 100% coverage.🤖 Generated with Claude Code