Skip to content
This repository was archived by the owner on Jun 16, 2026. It is now read-only.

Commit 6ef751e

Browse files
xenoscopicxenoscopic
authored
transport: add parallel transport and improve resumable transport (#130)
* transport: implement concurrent net/http.RoundTripper Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: extract common code used across transports Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: clean up semaphore implementation for unlimited case Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: remove unnecessary CloneHeader function Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: add better commenting to private field types Signed-off-by: Jacob Howard <jacob.howard@docker.com> * tools: add parallel download benchmark tool Signed-off-by: Jacob Howard <jacob.howard@docker.com> * tools: add progress output to parallelget benchmark Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: refactor parallel transport to use FIFO implementation Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: clean up FIFO tests and comments Signed-off-by: Jacob Howard <jacob.howard@docker.com> * tools: cleaned up parallelget implementation a bit Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: refactored tests to use shared code Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: improve code and comment styles Signed-off-by: Jacob Howard <jacob.howard@docker.com> * tools: clean up parallelget implementation and comments Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: clean up tests and some formatting Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: update code with GPT-5 recommendations Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: implement resumable improvements from GPT-5 Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: consolidate FIFO constructors Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: final fixups to AI-generated code Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: avoid using net/http/httptest package Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: clarify FIFO temporary file purpose Signed-off-by: Jacob Howard <jacob.howard@docker.com> * tools: don't show parallelget usage on error Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: fix test in bufferfile Signed-off-by: Jacob Howard <jacob.howard@docker.com> * transport: remove time.Sleep instances in concurrency test Signed-off-by: Jacob Howard <jacob.howard@docker.com> --------- Signed-off-by: Jacob Howard <jacob.howard@docker.com>
1 parent 7d9fc7b commit 6ef751e

19 files changed

Lines changed: 5662 additions & 1070 deletions

File tree

‎go.mod‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ require (
88
github.com/google/go-containerregistry v0.20.6
99
github.com/gpustack/gguf-parser-go v0.22.1
1010
github.com/sirupsen/logrus v1.9.3
11+
github.com/spf13/cobra v1.9.1
1112
)
1213

1314
require (
@@ -16,6 +17,7 @@ require (
1617
github.com/docker/distribution v2.8.3+incompatible // indirect
1718
github.com/docker/docker-credential-helpers v0.9.3 // indirect
1819
github.com/henvic/httpretty v0.1.4 // indirect
20+
github.com/inconshreveable/mousetrap v1.1.0 // indirect
1921
github.com/json-iterator/go v1.1.12 // indirect
2022
github.com/klauspost/compress v1.18.0 // indirect
2123
github.com/mitchellh/go-homedir v1.1.0 // indirect
@@ -25,6 +27,7 @@ require (
2527
github.com/opencontainers/image-spec v1.1.1 // indirect
2628
github.com/pkg/errors v0.9.1 // indirect
2729
github.com/smallnest/ringbuffer v0.0.0-20241116012123-461381446e3d // indirect
30+
github.com/spf13/pflag v1.0.6 // indirect
2831
github.com/stretchr/testify v1.10.0 // indirect
2932
github.com/vbatts/tar-split v0.12.1 // indirect
3033
golang.org/x/crypto v0.35.0 // indirect

‎go.sum‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
github.com/containerd/stargz-snapshotter/estargz v0.16.3 h1:7evrXtoh1mSbGj/pfRccTampEyKpjpOnS3CyiV1Ebr8=
22
github.com/containerd/stargz-snapshotter/estargz v0.16.3/go.mod h1:uyr4BfYfOj3G9WBVE8cOlQmXAbPN9VEQpBBeJIuOipU=
3+
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
34
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
45
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
56
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -18,6 +19,8 @@ github.com/gpustack/gguf-parser-go v0.22.1 h1:FRnEDWqT0Rcplr/R9ctCRSN2+3DhVsf6dn
1819
github.com/gpustack/gguf-parser-go v0.22.1/go.mod h1:y4TwTtDqFWTK+xvprOjRUh+dowgU2TKCX37vRKvGiZ0=
1920
github.com/henvic/httpretty v0.1.4 h1:Jo7uwIRWVFxkqOnErcoYfH90o3ddQyVrSANeS4cxYmU=
2021
github.com/henvic/httpretty v0.1.4/go.mod h1:Dn60sQTZfbt2dYsdUSNsCljyF4AfdqnuJFDLJA1I4AM=
22+
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
23+
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
2124
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
2225
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
2326
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
@@ -37,10 +40,15 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
3740
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
3841
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
3942
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
43+
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
4044
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
4145
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
4246
github.com/smallnest/ringbuffer v0.0.0-20241116012123-461381446e3d h1:3VwvTjiRPA7cqtgOWddEL+JrcijMlXUmj99c/6YyZoY=
4347
github.com/smallnest/ringbuffer v0.0.0-20241116012123-461381446e3d/go.mod h1:tAG61zBM1DYRaGIPloumExGvScf08oHuo0kFoOqdbT0=
48+
github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
49+
github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
50+
github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
51+
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
4452
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
4553
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
4654
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=

‎tools/benchmarks/parallelget/README.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# ParallelGet Benchmark Tool
2+
3+
A command-line benchmarking tool that compares the performance of standard HTTP GET requests against parallelized requests using the `transport/parallel` package.
4+
5+
## Features
6+
7+
- **Performance Comparison**: Downloads the same URL twice (standard vs parallel) and compares timing
8+
- **Response Validation**: Ensures both downloads produce identical results (byte-for-byte comparison)
9+
- **Configurable Parameters**: Adjustable chunk size and concurrency settings
10+
- **Detailed Metrics**: Reports download speeds, timing differences, and performance improvements
11+
- **Dynamic Progress Display**: Shows real-time progress bars during downloads with percentage and byte counts
12+
- **Clean Output**: User-friendly performance summary with emojis and clear formatting
13+
14+
## Usage
15+
16+
```bash
17+
go run ./tools/benchmarks/parallelget <url> [flags]
18+
```
19+
20+
or
21+
22+
```bash
23+
go build ./tools/benchmarks/parallelget
24+
./parallelget <url> [flags]
25+
```
26+
27+
### Arguments
28+
29+
- `<url>`: The HTTP URL to benchmark (required)
30+
31+
### Flags
32+
33+
- `--chunk-size int`: Minimum chunk size in bytes for parallelization (default: 1MB)
34+
- `--max-concurrent uint`: Maximum concurrent requests for parallel transport (default: 4)
35+
- `-h, --help`: Show help information
36+
37+
### Examples
38+
39+
```bash
40+
# Basic usage
41+
./parallelget https://example.com/large-file.zip
42+
43+
# Custom chunk size (512KB) and higher concurrency
44+
./parallelget https://example.com/large-file.zip --chunk-size 524288 --max-concurrent 8
45+
46+
# Small chunk size for testing with smaller files
47+
./parallelget https://httpbin.org/bytes/10485760 --chunk-size 262144 --max-concurrent 6
48+
```
49+
50+
## Output
51+
52+
The tool provides detailed output including:
53+
54+
1. **Configuration**: Shows the chunk size and concurrency settings
55+
2. **Progress**: Real-time updates for each benchmark phase
56+
3. **Individual Results**: Download speed and timing for each approach
57+
4. **Validation**: Confirms that both downloads produced identical content
58+
5. **Performance Summary**:
59+
- Speedup factor (e.g., "3.2x faster")
60+
- Time saved/penalty
61+
- Detailed timing breakdown
62+
63+
### Sample Output
64+
65+
```
66+
Benchmarking HTTP GET performance for: https://example.com/large-file.zip
67+
Configuration: chunk-size=1048576 bytes, max-concurrent=4
68+
69+
Running non-parallel benchmark...
70+
Progress: [██████████████████████████████] 100.0% (10485760/10485760 bytes)
71+
✓ Non-parallel: 10485760 bytes in 2.1s (4.76 MB/s)
72+
Running parallel benchmark...
73+
Progress: [██████████████████████████████] 100.0% (10485760/10485760 bytes)
74+
✓ Parallel: 10485760 bytes in 650ms (15.38 MB/s)
75+
Validating response consistency...
76+
✓ Responses match perfectly
77+
78+
============================================================
79+
PERFORMANCE COMPARISON
80+
============================================================
81+
🚀 Parallel was 3.23x faster than non-parallel
82+
⏱️ Time saved: 1.45s (69.0%)
83+
84+
Detailed timing:
85+
Non-parallel: 2.1s
86+
Parallel: 650ms
87+
Difference: -1.45s
88+
```
89+
90+
## How It Works
91+
92+
1. **Non-Parallel Benchmark**: Uses `net/http.DefaultClient` with `net/http.DefaultTransport`
93+
2. **Parallel Benchmark**: Uses `net/http.DefaultClient` with `transport/parallel.ParallelTransport` wrapping `net/http.DefaultTransport`
94+
3. **Response Storage**: Both responses are written to temporary files for validation
95+
4. **Validation**: Performs byte-by-byte comparison to ensure identical content
96+
5. **Cleanup**: Automatically removes temporary files after completion
97+
98+
## Notes
99+
100+
- The tool requires the server to support HTTP range requests (`Accept-Ranges: bytes`) for parallel downloads to work
101+
- If the server doesn't support range requests or the file is too small, the parallel transport will automatically fall back to a single request
102+
- Temporary files are automatically cleaned up, even if the tool exits unexpectedly
103+
- The tool validates that both downloads produce identical results before reporting performance metrics
104+
105+
## Use Cases
106+
107+
- **Performance Testing**: Evaluate the effectiveness of parallel downloads for different URLs
108+
- **Configuration Tuning**: Find optimal chunk size and concurrency settings for specific servers or file types
109+
- **Server Compatibility**: Test whether servers properly support range requests
110+
- **Network Optimization**: Understand the impact of parallel downloads on different network conditions

0 commit comments

Comments
 (0)