Skip to content

Commit 5ddd846

Browse files
Update BENCHMARKS.md (#6258)
1 parent 2d41973 commit 5ddd846

File tree

2 files changed

+157
-123
lines changed

2 files changed

+157
-123
lines changed

BENCHMARKS.md

+21-26
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Benchmarks
22

3-
All benchmarks were computed on macOS using Python 3.12.0 (for non-uv tools), and come with a few
3+
All benchmarks were computed on macOS using Python 3.12.4 (for non-uv tools), and come with a few
44
important caveats:
55

66
- Benchmark performance may vary dramatically across different operating systems and filesystems. In
@@ -9,11 +9,6 @@ important caveats:
99
- Benchmark performance may vary dramatically depending on the set of packages being installed. For
1010
example, a resolution that requires building a single intensive source distribution may appear
1111
very similar across tools, since the bottleneck is tool-agnostic.
12-
- Unlike Poetry, both uv and pip-tools do _not_ generate platform-independent lockfiles. As such,
13-
Poetry is (by design) doing significantly more work than other tools in the resolution benchmarks.
14-
Poetry is included for completeness, as many projects may not _need_ a platform-independent
15-
lockfile. However, it's critical to understand that benchmarking uv's resolution time against
16-
Poetry is an unfair comparison. (Benchmarking installation, however, _is_ a fair comparison.)
1712

1813
This document benchmarks against Trio's `docs-requirements.in`, as a representative example of a
1914
real-world project.
@@ -22,35 +17,35 @@ In each case, a smaller bar (i.e., lower) is better.
2217

2318
## Warm Installation
2419

25-
Benchmarking package installation (e.g., `uv pip sync`) with a warm cache. This is equivalent to
20+
Benchmarking package installation (e.g., `uv sync`) with a warm cache. This is equivalent to
2621
removing and recreating a virtual environment, and then populating it with dependencies that you've
2722
installed previously on the same machine.
2823

29-
![install-warm](./assets/png/install-warm.png)
24+
![install-warm](https://github.com/user-attachments/assets/84118aaa-d030-4e29-8f1e-9483091ceca3)
3025

3126
## Cold Installation
3227

33-
Benchmarking package installation (e.g., `uv pip sync`) with a cold cache. This is equivalent to
34-
running `uv pip sync` on a new machine or in CI (assuming that the package manager cache is not
35-
shared across runs).
28+
Benchmarking package installation (e.g., `uv sync`) with a cold cache. This is equivalent to running
29+
`uv sync` on a new machine or in CI (assuming that the package manager cache is not shared across
30+
runs).
3631

37-
![install-cold](./assets/png/install-cold.png)
32+
![install-cold](https://github.com/user-attachments/assets/e7f5b203-7e84-452b-8c56-1ff6531c9898)
3833

3934
## Warm Resolution
4035

41-
Benchmarking dependency resolution (e.g., `uv pip compile`) with a warm cache, but no existing
42-
lockfile. This is equivalent to blowing away an existing `requirements.txt` file to regenerate it
43-
from a `requirements.in` file.
36+
Benchmarking dependency resolution (e.g., `uv lock`) with a warm cache, but no existing lockfile.
37+
This is equivalent to blowing away an existing `requirements.txt` file to regenerate it from a
38+
`requirements.in` file.
4439

45-
![resolve-warm](./assets/png/resolve-warm.png)
40+
![resolve-warm](https://github.com/user-attachments/assets/e1637a08-8b27-4077-8138-b3849e53eb04)
4641

4742
## Cold Resolution
4843

49-
Benchmarking dependency resolution (e.g., `uv pip compile`) with a cold cache. This is equivalent to
50-
running `uv pip compile` on a new machine or in CI (assuming that the package manager cache is not
51-
shared across runs).
44+
Benchmarking dependency resolution (e.g., `uv lock`) with a cold cache. This is equivalent to
45+
running `uv lock` on a new machine or in CI (assuming that the package manager cache is not shared
46+
across runs).
5247

53-
![resolve-cold](./assets/png/resolve-cold.png)
48+
![resolve-cold](https://github.com/user-attachments/assets/b578c264-c209-45ab-b4c3-54073d871e86)
5449

5550
## Reproduction
5651

@@ -69,7 +64,7 @@ To benchmark resolution against pip-compile, Poetry, and PDM:
6964

7065
```shell
7166
uv run resolver \
72-
--uv-pip \
67+
--uv-project \
7368
--poetry \
7469
--pdm \
7570
--pip-compile \
@@ -82,7 +77,7 @@ To benchmark installation against pip-sync, Poetry, and PDM:
8277

8378
```shell
8479
uv run resolver \
85-
--uv-pip \
80+
--uv-project \
8681
--poetry \
8782
--pdm \
8883
--pip-sync \
@@ -96,10 +91,10 @@ Both commands should be run from the `scripts/benchmark` directory.
9691
After running the benchmark script, you can generate the corresponding graph via:
9792

9893
```shell
99-
cargo run -p uv-dev render-benchmarks resolve-warm.json --title "Warm Resolution"
100-
cargo run -p uv-dev render-benchmarks resolve-cold.json --title "Cold Resolution"
101-
cargo run -p uv-dev render-benchmarks install-warm.json --title "Warm Installation"
102-
cargo run -p uv-dev render-benchmarks install-cold.json --title "Cold Installation"
94+
cargo run -p uv-dev --all-features render-benchmarks resolve-warm.json --title "Warm Resolution"
95+
cargo run -p uv-dev --all-features render-benchmarks resolve-cold.json --title "Cold Resolution"
96+
cargo run -p uv-dev --all-features render-benchmarks install-warm.json --title "Warm Installation"
97+
cargo run -p uv-dev --all-features render-benchmarks install-cold.json --title "Cold Installation"
10398
```
10499

105100
You need to install the [Roboto Font](https://fonts.google.com/specimen/Roboto) if the labels are

0 commit comments

Comments
 (0)