Optimize allocator usage for import plugin: Use standard allocators for dependency files #15799
Closed as not planned
Description
Problem
When using the import plugin with external linters (JS plugins), all files currently use fixed-size allocators, including dependency files that are only parsed for module resolution and never linted.
Current behavior:
- Entry file to lint → Fixed-size allocator ✓ (needed for JS transfer)
- Dependency files → Fixed-size allocators ✗ (memory never used - never sent to JS)
Impact:
- Fixed-size allocators pre-allocate ~2 GiB address space each
- When using import plugin, we need many allocators at once to build the entire module graph
- This results in excessive memory allocation for dependency files that are never passed to external linters
Proposed Solution
Refactor AllocatorPool to contain both standard and fixed-size allocators, with runtime validation to ensure only fixed-size allocators are used when sending ASTs to JS.
Key Changes
1. Add type tracking:
pub enum AllocatorType { Standard, FixedSize }
pub struct AllocatorGuard<'alloc_pool> {
allocator: ManuallyDrop<Allocator>,
allocator_type: AllocatorType, // Track which type
pool: &'alloc_pool AllocatorPool,
}2. Single pool with both types:
pub struct AllocatorPool {
standard: StandardAllocatorPool,
fixed_size: Option<FixedSizeAllocatorPool>,
}
impl AllocatorPool {
pub fn new(thread_count: usize) -> Self // Standard only
pub fn new_dual(thread_count: usize) -> Self // Both types
pub fn get_standard(&self) -> AllocatorGuard<'_>
pub fn get_fixed_size(&self) -> AllocatorGuard<'_>
}3. Smart selection in Runtime:
// Create pool with both types when JS plugins present
let pool = if linter.has_external_linter() {
AllocatorPool::new_dual(thread_count)
} else {
AllocatorPool::new(thread_count)
};
// Choose allocator based on file purpose
let file_needs_linting = paths.contains(path);
let guard = if file_needs_linting && self.linter.has_external_linter() {
self.pool.get_fixed_size() // For linting + JS transfer
} else {
self.pool.get_standard() // For parsing only
};4. Runtime validation before JS transfer:
if linter.has_external_linter() {
assert_eq!(
allocator_guard.allocator_type(),
AllocatorType::FixedSize,
"External linters require fixed-size allocators"
);
}
linter.run(path, context, allocator_guard);