You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
***Danzo*** is a cross-platform and cross-architecture CLI downloader utility designed for fast parallel connections, progress tracking, and an easy to use binary. The tool aims to maximize download speeds by utilizing optimized buffer sizes and parallel processing.
10
+
***Danzo*** is a cross-platform and cross-architecture CLI downloader utility designed for multi-threaded downloads, progress tracking, and an intuitive command structure. Danzo maximizes download speeds by using a large number of goroutines.
11
11
12
-
Yes, the name is the same as a Naruto character who has a hobby of collecting many things, reprentative of parallel connections used in this tool.
12
+
*Side note - yes, the name is the same as a Naruto character with a hobby of collecting and using multiple "items", reprentative of parallel connections used in this tool.*
13
13
14
14
## Features
15
15
16
-
- Multi-connection downloads to improve speed
17
-
- Automatic chunk size optimization
18
-
- Real-time progress display with speed and ETA
19
-
- Batch downloading with YAML configuration
20
-
- Parallel downloading of multiple files
21
-
- Customizable user agent and timeout settings
16
+
- Multiple connection threads for high speed downloads and assembly
17
+
- Temporary directory for chunk downloads
18
+
- Automatic cleanup of temporary files
19
+
- Manual cleanup of temporary files in case of failures
20
+
- Automatic optimization of chunk size vs. threads
21
+
- Direct single-threaded download preference for small chunk sizes
22
+
- Fallback to single thread operation for lack of byte-range support
-l, --urllist string Path to YAML file containing URLs and output paths
77
-
-a, --user-agent string User agent (default "Danzo/1337")
78
-
-w, --workers int Number of links to download in parallel (default: 1) (default 1)
85
+
-a, --user-agent string User agent (default "danzo/1337")
86
+
-w, --workers int Number of links to download in parallel (default 1)
79
87
80
88
Use "danzo [command] --help" for more information about a command.
81
89
```
82
90
91
+
### Basic Usage
92
+
93
+
The simplest way to download a file is to provide a URL directly:
94
+
95
+
```bash
96
+
danzo https://example.com/largefile.zip
97
+
```
98
+
99
+
The output filename will be inferred from the URL and Danzo will use 4 connection threads by default. You can also specify an output filename manually with:
> The value for `-c` can go upto `64` for a single URL. Danzo creates chunks equal to number of connections requested. Once all chunks are downloaded, they are combined into a single file. If the decided number of chunks are smaller than 20 MB, Danzo falls back to a single threaded download for that file. This number was **arbitrarily** chosen based on heuristics.
107
+
108
+
You can customize the number of connections to use like so:
109
+
110
+
```bash
111
+
danzo "https://example.com/largefile.zip" -c 16
112
+
```
113
+
114
+
> [!WARNING]
115
+
> You should be careful of the disk IO as well. Multi-connection download takes disk IO, which can add to overall time before the file is ready.
116
+
>
117
+
> For example, a 1 GB file takes 54 seconds when using 50 connections vs. 62 seconds when using 64 connections. This is because combining 64 files takes longer than combining 50 files.
118
+
>
119
+
> Therefore, you need to find a balance where the number of connections maximize your network throughput without putting extra strain on disk IO. This effect is especially observable in HDDs.
120
+
121
+
Lastly, if a URL does not use byte-range requests (i.e., server doesn't support partial content downloads), Danzo automatically switches to a simple, single-threaded, direct download.
122
+
83
123
### Batch Download
84
124
85
-
For downloading multiple files, create a YAML file with the following format:
125
+
Danzo can be provided a YAML config to allow simultaneous downloads of several URLs. Each URL in turn will use multi-threaded connection mode by default to maximize throughput. The YAML file requires following format:
86
126
87
127
```yaml
88
128
- op: "./output1.zip"
89
129
link: "https://example.com/file1.zip"
90
130
- op: "./output2.zip"
91
131
link: "https://example.com/file2.zip"
132
+
# more entries with output path and urls...
92
133
```
93
134
94
-
Then run as:
135
+
Then run Danzo as:
95
136
96
137
```bash
97
-
./danzo --urllist "./downloads.yaml"
138
+
danzo -l config.yaml
98
139
```
99
140
100
-
Number of workers and connections per worker can be specified as follows:
141
+
The number of files being downloaded in parallel can be configured as workers (default: 1) and the number of connections would be applied per worker. Define these parameters as follows:
101
142
102
143
```bash
103
-
./danzo -l downloads.yaml -w 3 -c 16
144
+
danzo -l downloads.yaml -w 3 -c 16
104
145
```
105
146
106
147
> [!NOTE]
107
-
> Danzo caps the total number of parallel workers at 64. Specifically `#workers * #connections <= 64`. This is a sensible default to prevent overwhelming the system.
148
+
> Danzo caps the total number of parallel workers at 64. Specifically `#workers * #connections <= 64`. This is a generous default to prevent overwhelming the system.
108
149
109
150
### Cleaning Temporary Files
110
151
111
-
Danzo stores partial downloads on disk in the `.danzo-temp` directory (situated in the same path as the associated output path). If a download event is interrupted or failed, the temporary files can be cleared by specifying the output path like so:
152
+
Danzo stores partial downloads on disk in the `.danzo-temp` directory (situated in the same path as the associated output path). If a download event is interrupted or failed, the temporary files can be cleared by using the `clean` command:
112
153
113
154
```bash
114
-
./danzo clean --output"./downloaded-file.zip"
155
+
danzo clean -o"./path/for/download.zip"
115
156
```
116
157
158
+
For batch downloads, you may need to run the clean command for each output path individually if they don't share the same parent directory.
159
+
117
160
## Tips and Notes
118
161
119
-
- For optimal download speeds, the number of connections is automatically set to match your CPU cores, but you can adjust this with the -c flag
120
-
- Large files benefit the most from multiple connections
121
-
- If a download fails, Danzo will retry individual chunks up to 5 times
162
+
- Large files benefit the most from multiple connections, but also add to disk IO. Be mindful of the balance between network and disk IO.
163
+
- If a chunk download fails, Danzo will retry individual chunks up to 5 times.
122
164
- For downloading through a proxy, use the `--proxy` or `-p` flag with your proxy URL (you needn't provide the HTTP scheme, Danzo matches it to that of the URL)
123
-
-*Not all servers support multi-connection downloads (range requests)*
124
-
- For servers with rate limiting, reducing the number of connections might help
125
-
- Debug mode (`--debug`) provides detailed information about the download process
126
-
- Temporary files are stored in a .danzo-temp directory and automatically cleaned up after download
127
-
- Use `-a randomize` to randomize the user agent for every HTTP client
165
+
- Not all servers support multi-connection downloads (range requests), in which case, Danzo auto-switches to simple downloads.
166
+
- For servers with rate limiting, reducing the number of connections might help.
167
+
- Debug mode (`--debug`) provides detailed information about the download process.
168
+
- Temporary files are automatically cleaned up after successful downloads.
169
+
- Use `-a randomize` to randomly assign a user agent for every HTTP client. The full list of user agents considered are stored in the [helpers.go](https://github.com/Tanq16/danzo/blob/main/internal/helpers.go) file.
170
+
- The tool automatically activates "high-thread-mode" when using more than 6 connections, which optimizes socket buffer sizes for better performance.
0 commit comments