Skip to content

Conversation

@roberto-bayardo
Copy link
Collaborator

@roberto-bayardo roberto-bayardo commented Jan 3, 2026

The append write buffer is currently write-locked while we write data to underlying blob, which prevents any concurrent reads or buffered writes. This PR introduces a separate IO lock for the underlying blob so that buffer reads and appends can be performed concurrently with blob writing.

Impact on existing benchmarks are neutral, however we don't have any benchmarks that perform concurrent reads & writes, which is where this change would shine.

Resolves: #2673

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Jan 3, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
commonware-mcp 12d03ae Jan 05 2026, 07:16 PM

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves concurrency in the buffer pool by introducing a separate I/O lock (blob_io) for the underlying blob. The key change allows readers to access the buffer pool cache while writes to the blob are in progress, rather than blocking all reads during blob I/O operations.

Key changes:

  • Introduces AsyncMutex wrapper for async mutual exclusion primitives
  • Adds blob_io RwLock to coordinate reads/writes to the underlying blob separately from the buffer pool lock
  • Implements try_read_cached fast path to check cache without acquiring I/O locks

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
runtime/src/utils/mod.rs Adds AsyncMutex wrapper around async_lock::Mutex for consistency with existing synchronization primitives
runtime/src/utils/buffer/pool.rs Implements try_read_cached method to enable lock-free cache lookups
runtime/src/utils/buffer/append.rs Refactors blob I/O to use separate blob_io lock, allowing concurrent cache reads during blob writes

@roberto-bayardo roberto-bayardo force-pushed the avoid-write-lock-during-blob-io branch from 1d611a2 to a3e53db Compare January 3, 2026 05:16
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Jan 3, 2026

Deploying monorepo with  Cloudflare Pages  Cloudflare Pages

Latest commit: 12d03ae
Status: ✅  Deploy successful!
Preview URL: https://66afdd7c.monorepo-eu0.pages.dev
Branch Preview URL: https://avoid-write-lock-during-blob.monorepo-eu0.pages.dev

View logs

@commonwarexyz commonwarexyz deleted a comment from Copilot AI Jan 3, 2026
@commonwarexyz commonwarexyz deleted a comment from Copilot AI Jan 3, 2026
@roberto-bayardo roberto-bayardo force-pushed the avoid-write-lock-during-blob-io branch 2 times, most recently from 286f4ce to caffe14 Compare January 3, 2026 17:40
@roberto-bayardo roberto-bayardo force-pushed the avoid-write-lock-during-blob-io branch from caffe14 to b4fdfe9 Compare January 3, 2026 18:05
@roberto-bayardo roberto-bayardo marked this pull request as ready for review January 3, 2026 18:30
@roberto-bayardo roberto-bayardo changed the title [runtime/buffer] avoid write-locking the buffer pool during blob io [runtime/buffer] avoid write-locking the Append buffer during blob writes Jan 3, 2026
danlaine
danlaine previously approved these changes Jan 5, 2026
// Acquire buffer lock first.
// NOTE: We MUST acquire the buffer lock before the blob lock to avoid deadlocks with
// `append`, which acquires buffer then blob (via `flush_internal`).
let mut guard = self.buffer.write().await;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let mut guard = self.buffer.write().await;
let mut buf_guard = self.buffer.write().await;

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@roberto-bayardo roberto-bayardo added this pull request to the merge queue Jan 5, 2026
Merged via the queue into main with commit e78f1d4 Jan 5, 2026
127 checks passed
@roberto-bayardo roberto-bayardo deleted the avoid-write-lock-during-blob-io branch January 5, 2026 21:16
@codecov
Copy link

codecov bot commented Jan 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.83%. Comparing base (6deffb6) to head (12d03ae).
⚠️ Report is 10 commits behind head on main.

@@            Coverage Diff             @@
##             main    #2679      +/-   ##
==========================================
+ Coverage   92.63%   92.83%   +0.20%     
==========================================
  Files         357      361       +4     
  Lines      103102   106277    +3175     
==========================================
+ Hits        95505    98659    +3154     
- Misses       7597     7618      +21     
Files with missing lines Coverage Δ
runtime/src/utils/buffer/append.rs 99.63% <100.00%> (+0.05%) ⬆️
runtime/src/utils/buffer/pool.rs 96.05% <100.00%> (+0.31%) ⬆️
storage/src/journal/contiguous/fixed.rs 97.39% <100.00%> (-0.01%) ⬇️

... and 103 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d95a00...12d03ae. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[storage & runtime] Avoid holding write lock on append buffer during flush.

3 participants