Skip to content

perf(download): use zero-copy NTFS hardlinks for cached payload extraction - #6766

Open
olahouze wants to merge 2 commits into
ScoopInstaller:masterfrom
olahouze:perf-zero-copy-hardlinks
Open

olahouze wants to merge 2 commits into
ScoopInstaller:masterfrom
olahouze:perf-zero-copy-hardlinks

Conversation

@olahouze

@olahouze olahouze commented Oct 5, 2026 •

Copy link
Copy Markdown

📋 Problème Traité

Dans lib/download.ps1, la fonction Invoke-CachedDownload effectue une duplication physique intégrale par Copy-Item de chaque archive téléchargée depuis le cache (~/.scoop/cache/) vers le répertoire d'installation temporaire de l'application :

if (!($null -eq $to)) {
    if ($use_cache) {
        Copy-Item $cached $to
    } else {
        Move-Item $cached $to -Force
    }
}

Conséquences :

  1. Consommation excessive d'espace disque : pour des applications volumineuses (ex. SDKs, navigateurs, Android Studio, rust, llvm pesant de 500 Mo à 2 Go), le fichier est stocké en double sur le même disque pendant toute la phase d'extraction.
  2. Latence I/O inutile : sur des disques HDD ou même des SSD NVMe saturés, copier 1 à 2 Go de données prend entre 2 et 10 secondes d'écritures superflues.
  3. Usure inutile du support SSD (Write Amplification).

💡 Solution Apportée

Remplacement de la copie physique aveugle par une stratégie zéro-copie via Hardlink NTFS (New-Item -ItemType HardLink) lorsque le cache et la cible se trouvent sur le même volume (cas par défaut de Scoop dans $env:USERPROFILE\scoop) :

  1. La création du HardLink est quasi-instantanée (0 ms), quel que soit le poids du fichier (10 Mo ou 5 Go).
  2. L'archive dans le cache n'est pas modifiée, et sa suppression ultérieure dans le répertoire d'installation par Invoke-Extraction -Removal ne fait que décrémenter le compteur de liens NTFS sans toucher au cache.
  3. En cas d'incompatibilité de volume (disques différents, FAT32/exFAT, restrictions de droits), un fallback automatique et transparent vers Copy-Item est assuré.
function Link-OrCopyFile($Source, $Destination) {
    if (Test-Path $Destination) {
        Remove-Item $Destination -Force
    }
    $sourceDrive = [System.IO.Path]::GetPathRoot([System.IO.Path]::GetFullPath($Source))
    $destDrive = [System.IO.Path]::GetPathRoot([System.IO.Path]::GetFullPath($Destination))

    if ($sourceDrive -and ($sourceDrive -eq $destDrive)) {
        try {
            $null = New-Item -ItemType HardLink -Path $Destination -Target $Source -Force -ErrorAction Stop
            return
        } catch {
            # Fallback to copy if hardlink is unsupported on volume
        }
    }
    Copy-Item $Source $Destination -Force
}

🔗 Issue Associée

Amélioration des performances I/O et réduction de l'empreinte disque lors de l'installation depuis le cache.


✅ Validation & Actions Réalisées

Note

  • Test de non-régression d'intégrité binaire par comparaison SHA-256 entre le fichier source et le fichier cible après lien.
  • Validation que la suppression de la cible après extraction laisse le fichier de cache intact et réutilisable.
  • Benchmark sur archive volumineuse (200 Mo) : temps passé de 133 ms à 14 ms (gain 9.5x et 0 octet d'I/O disque supplémentaire).
  • Add a comment starting with "/verify" after raising the PR - this will kick in the automatic manifest verifier.
  • Use conventional PR title: perf(download): use zero-copy NTFS hardlinks for cached payload extraction
  • I have read the Contributing Guide

🧪 Reproduction de l'Erreur

Script démontrant la lenteur et la duplication de Copy-Item sur un fichier de 200 Mo :

$testDir = "$env:TEMP\scoop_bench_copy"
New-Item -Path $testDir -ItemType Directory -Force | Out-Null
$src = "$testDir\cached_payload.bin"
$dst = "$testDir\install_payload.bin"

# Création d'un payload de 200 Mo
$bytes = New-Object byte[] (200 * 1024 * 1024)
[System.IO.File]::WriteAllBytes($src, $bytes)
$bytes = $null
[GC]::Collect()

$sw = [System.Diagnostics.Stopwatch]::StartNew()
Copy-Item $src $dst -Force
$sw.Stop()

Write-Host "Temps Copy-Item classique : $($sw.ElapsedMilliseconds) ms"
# Résultat mesuré : ~133 ms (sur SSD rapide) / plusieurs secondes sur HDD
Remove-Item $testDir -Recurse -Force

🔬 Test de la Correction

Script validant la correction, l'absence de duplication et la persistance du cache :

$testDir = "$env:TEMP\scoop_bench_link"
New-Item -Path $testDir -ItemType Directory -Force | Out-Null
$src = "$testDir\cached_payload.bin"
$dst = "$testDir\install_payload.bin"

$bytes = New-Object byte[] (200 * 1024 * 1024)
[System.IO.File]::WriteAllBytes($src, $bytes)
$bytes = $null
[GC]::Collect()

function Link-OrCopyFile($Source, $Destination) {
    if (Test-Path $Destination) { Remove-Item $Destination -Force }
    $sourceDrive = [System.IO.Path]::GetPathRoot([System.IO.Path]::GetFullPath($Source))
    $destDrive = [System.IO.Path]::GetPathRoot([System.IO.Path]::GetFullPath($Destination))
    if ($sourceDrive -and ($sourceDrive -eq $destDrive)) {
        try {
            $null = New-Item -ItemType HardLink -Path $Destination -Target $Source -Force -ErrorAction Stop
            return
        } catch {}
    }
    Copy-Item $Source $Destination -Force
}

$sw = [System.Diagnostics.Stopwatch]::StartNew()
Link-OrCopyFile $src $dst
$sw.Stop()

Write-Host "Temps HardLink zéro-copie : $($sw.ElapsedMilliseconds) ms"
# Résultat mesuré : 14 ms (gain x9.5)

# Validation d'intégrité
$hashSrc = (Get-FileHash $src -Algorithm SHA256).Hash
$hashDst = (Get-FileHash $dst -Algorithm SHA256).Hash
if ($hashSrc -ne $hashDst) { throw "Mismatch empreinte SHA256" }

# Validation persistance du cache lors de la suppression de la cible
Remove-Item $dst -Force
if (!(Test-Path $src)) { throw "Le fichier source du cache a été altéré !" }

Write-Host "Non-régression et intégrité validées avec succès !"
Remove-Item $testDir -Recurse -Force

RetriggerConfidence Score: 5/5

No new blocking issue was established, so the PR appears safe to merge on this review.

Summary

The PR stages cached archives with hardlinks where possible, retains copying for other payloads, and adds download tests. Since the previous review, it has changed the hardlink test to assert LinkType rather than Target.

Reviews (6) · Last reviewed commit: "perf(download): restrict zero-copy hardl..."

@olahouze
olahouze marked this pull request as ready for review October 5, 2026 13:04
Comment thread lib/download.ps1 Outdated
Comment thread lib/download.ps1 Outdated
@coderabbitai

coderabbitai Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Summary by CodeRabbit

  • Bug Fixes
    • Cached archive downloads now use a hard link when the source and destination are on the same drive. If hard linking is unavailable or fails, the archive is copied instead.
    • Other cached downloads are copied, keeping changes to the staged file separate from the cached file. Downloads with caching disabled continue to move the file.

Walkthrough

Cached downloads now use a helper that attempts a hard link for matching archive files when source and destination are on the same drive. If linking is not used or fails, the helper copies the file. Uncached downloads still move the file. Tests check archive hard links and copy behavior for scripts.

Changes

Cached download placement

Layer / File(s) Summary
Cached file placement
lib/download.ps1, test/Scoop-Download.Tests.ps1
Adds Link-OrCopyFile to remove an existing destination and attempt a hard link for matching archives on the same drive, with a copy fallback. Invoke-CachedDownload uses the helper for cached files; the uncached path remains a move. Tests check archive linking and script copy behavior.

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to e5642

The new hard-link test may fail on some PowerShell 7 versions even when the feature works, which could break CI. Asserting LinkType instead of Target fixes this before merge.

Architecture Summary

Architecture risk: 🔵 Low · up to a0025

The change affects 1 system.

Changed systems: lib

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — lib (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in lib/download.ps1: Added Link-OrCopyFile: it removes an existing destination, attempts a hard link when both full paths have the same nonempty drive root, and returns on success. If the roots differ or hard-link creation fails, it copies the source to the destination.
  • observed — Modified behavior in lib/download.ps1: Invoke-CachedDownload now calls Link-OrCopyFile instead of always copying the cached file when $use_cache is true. The $use_cache false path remains a move.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: using hard links for cached payloads to improve download extraction performance.
Description check ✅ Passed The description explains the cached-download hard-link approach, its copy fallback, and the intended performance and storage benefits. It is directly related to the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 99284feb-c958-4b8b-984e-db7f42cbb66e
📥 Commits

Reviewing files that changed from the base of the PR and between e6aa3b3 and a0025ea.

📒 Files selected for processing (1)
  • lib/download.ps1

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

Comment thread lib/download.ps1
@olahouze
olahouze force-pushed the perf-zero-copy-hardlinks branch 2 times, most recently from 8cc72c1 to b4ebd02 Compare October 6, 2026 07:11
@olahouze
olahouze force-pushed the perf-zero-copy-hardlinks branch from b4ebd02 to f0f9506 Compare October 6, 2026 07:15
Comment thread test/Scoop-Download.Tests.ps1 Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 20310013-2062-40c1-895a-10a333cec815
📥 Commits

Reviewing files that changed from the base of the PR and between f0f9506 and e5642de.

📒 Files selected for processing (2)
  • lib/download.ps1
  • test/Scoop-Download.Tests.ps1
🚧 Files skipped from review as they are similar to previous changes (1)
  • lib/download.ps1

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

Comment thread test/Scoop-Download.Tests.ps1 Outdated
$stagedZip = Join-Path $testDir 'staged.zip'
Link-OrCopyFile $cacheArchive $stagedZip
Test-Path $stagedZip | Should -BeTrue
(Get-Item $stagedZip).Target | Should -Not -BeNullOrEmpty

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
set -u
printf '%s\n' '--- diff ---'
git diff --no-ext-diff --unified=50 e6aa3b366bdee8ed138c1e0f7b85192ebdd35d0f e5642ded71e2d0035db6bf346bf2dac7d9db0cfc -- test/Scoop-Download.Tests.ps1
printf '%s\n' '--- target test and helper references ---'
rg -n -C 5 'Link-OrCopyFile|stagedZip|Target|LinkType' test/Scoop-Download.Tests.ps1
rg -n -F -- 'function Link-OrCopyFile' . --glob '!vendor/**' --glob '!node_modules/**' || test "$?" -eq 1
printf '%s\n' '--- PowerShell version configuration ---'
rg -n -i -C 3 'pwsh|powershell|PowerShellVersion|Pester|windows-latest|windows-' .github test appveyor.yml azure-pipelines.yml 2>/dev/null || test "$?" -eq 1
printf '%s\n' '--- repository files mentioning supported PowerShell ---'
rg -n -i -C 2 'PowerShell 5|PowerShell 7|PowerShell Core|PowerShell version|minimum.*PowerShell|requires.*PowerShell' README.md CONTRIBUTING.md .github docs 2>/dev/null || test "$?" -eq 1

Repository: ScoopInstaller/Scoop

Length of output: 17426


🤖 get_repo_knowledge executed:

get_repo_knowledge ScoopInstaller/Scoop /tmp/coderabbit-repo-knowledge/scoopinstaller-scoop-1ef5db97/learnings /tmp/coderabbit-repo-knowledge/scoopinstaller-scoop-1ef5db97/architecture

Length of output: 15047


🌐 Web query:

PowerShell FileSystemInfo Target property hard link empty PowerShell 7 LinkType hardlink behavior documentation issue 20938

💡 Result:

**Issue #20938 is closed as “Resolution-Won’t Fix.”** It reports that in PowerShell 7.4.0, `Get-Item` identifies a hard link as `LinkType = HardLink`, but `LinkTarget` and the legacy `Target` property are empty. The reporter contrasts that with Windows PowerShell 5, where `Target` was populated. (GitHub issue 20938 in PowerShell/PowerShell (link omitted to avoid creating a cross-reference))

The PowerShell 7.5 FileSystem provider docs describe `LinkType` and `LinkTarget`, and note that their values depend on the platform and filesystem—but they don’t promise a target path for hard links. ([learn.microsoft.com](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_filesystem_provider?view=powershell-7.5))

**Bottom line:** use `LinkType` to identify a hard link; don’t rely on `Target`/`LinkTarget` to enumerate its other names. The issue’s specific reproduction is for 7.4.0; the available issue and docs don’t establish behavior for every later PowerShell version.

Citations:

- 1: GitHub issue 20938 in PowerShell/PowerShell (link omitted to avoid creating a cross-reference)
- 2: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_filesystem_provider?view=powershell-7.5

🏁 Script executed:

#!/bin/bash
set -u
printf '%s\n' '--- CI workflow ---'
nl -ba .github/workflows/ci.yml | sed -n '1,36p'
printf '%s\n' '--- helper ---'
nl -ba lib/download.ps1 | sed -n '40,88p'
printf '%s\n' '--- test block ---'
nl -ba test/Scoop-Download.Tests.ps1 | sed -n '50,90p'
printf '%s\n' '--- test requirements ---'
nl -ba test/bin/test.ps1 | sed -n '1,26p'

Repository: ScoopInstaller/Scoop

Length of output: 6441


🌐 Web query:

PowerShell Windows PowerShell 5.1 Get-Item LinkType property unavailable LinkType PowerShell 6 file system provider

💡 Result:

`LinkType` is **not exclusive to PowerShell 6+**. Microsoft’s Windows PowerShell 5.1 documentation shows `LinkType` on a FileSystem item, and the 5.1 docs’ example output includes `LinkType : HardLink`. ([github.com](https://github.com/MicrosoftDocs/PowerShell-Docs/blob/main/reference/5.1/Microsoft.PowerShell.Core/About/about_Properties.md))

If `(Get-Item $path).LinkType` is blank or missing in 5.1, check that the item is actually a link and that the path uses the FileSystem provider. In PowerShell 7.5, Microsoft documents that `LinkType` describes linked items and that values depend on the platform and filesystem; Windows may also return no `LinkType` for certain `AppExecLink` items. ([learn.microsoft.com](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_filesystem_provider?view=powershell-7.5))

To inspect what your 5.1 session returns, try:

```powershell
$item = Get-Item -LiteralPath $path
$item | Get-Member
$item | Format-List LinkType, Target, Mode
```

The docs establish that 5.1 can expose `LinkType`; they don’t establish that every filesystem item or every link type will have a value.

Citations:

- 1: https://github.com/MicrosoftDocs/PowerShell-Docs/blob/main/reference/5.1/Microsoft.PowerShell.Core/About/about_Properties.md
- 2: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_filesystem_provider?view=powershell-7.5

Assert LinkType for the hard-link test.

When the pwsh leg uses an affected version (PowerShell 7.4 does), Get-Item can return an empty Target for a valid hard link, causing this assertion to fail. LinkType reports HardLink in both PowerShell 7.4 and Windows PowerShell 5.1.

🐛 Suggested fix
-        (Get-Item $stagedZip).Target | Should -Not -BeNullOrEmpty
+        (Get-Item $stagedZip).LinkType | Should -Be 'HardLink'
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
(Get-Item $stagedZip).Target | Should -Not -BeNullOrEmpty
(Get-Item $stagedZip).LinkType | Should -Be 'HardLink'

@olahouze
olahouze force-pushed the perf-zero-copy-hardlinks branch from e5642de to fa2284b Compare October 6, 2026 09:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant