diff --git a/README.md b/README.md index d9adeddd2..a9005c3d6 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ | [explode](/src/cmd/explode.rs#L2)
🔣👆 | Explode rows into multiple ones by splitting a column value based on the given separator. | | [extdedup](/src/cmd/extdedup.rs#L2)
👆 | Remove duplicate rows from an arbitrarily large CSV/text file using a memory-mapped, [on-disk hash table](https://crates.io/crates/odht). Unlike the `dedup` command, this command does not load the entire file into memory nor does it sort the deduped file. | | [extsort](/src/cmd/extsort.rs#L2)
🚀📇👆 | Sort an arbitrarily large CSV/text file using a multithreaded [external merge sort](https://en.wikipedia.org/wiki/External_sorting) algorithm. | -| [fetch](/src/cmd/fetch.rs#L3)
✨🧠🌐 | Fetches data from web services for every row using **HTTP Get**. Comes with [HTTP/2](https://http2-explained.haxx.se/en/part1) [adaptive flow control](https://medium.com/coderscorner/http-2-flow-control-77e54f7fd518), [jaq](https://github.com/01mf02/jaq?tab=readme-ov-file#jaq) JSON query language support, dynamic throttling ([RateLimit](https://www.ietf.org/archive/id/draft-ietf-httpapi-ratelimit-headers-06.html)) & caching with available persistent caching using [Redis](https://redis.io/) or a disk-cache. | +| [fetch](/src/cmd/fetch.rs#L3)
✨🧠🌐 | Send/Fetch data to/from web services for every row using **HTTP Get**. Comes with [HTTP/2](https://http2-explained.haxx.se/en/part1) [adaptive flow control](https://medium.com/coderscorner/http-2-flow-control-77e54f7fd518), [jaq](https://github.com/01mf02/jaq?tab=readme-ov-file#jaq) JSON query language support, dynamic throttling ([RateLimit](https://www.ietf.org/archive/id/draft-ietf-httpapi-ratelimit-headers-06.html)) & caching with available persistent caching using [Redis](https://redis.io/) or a disk-cache. | | [fetchpost](/src/cmd/fetchpost.rs#L3)
✨🧠🌐 | Similar to `fetch`, but uses **HTTP Post** ([HTTP GET vs POST methods](https://www.geeksforgeeks.org/difference-between-http-get-and-post-methods/)). Supports both HTML form (application/x-www-form-urlencoded) and JSON (application/json) content types. | | [fill](/src/cmd/fill.rs#L2)
👆 | Fill empty values. | | [fixlengths](/src/cmd/fixlengths.rs#L2) | Force a CSV to have same-length records by either padding or truncating them. | diff --git a/src/cmd/fetch.rs b/src/cmd/fetch.rs index 42bab2859..3cae1e3bd 100644 --- a/src/cmd/fetch.rs +++ b/src/cmd/fetch.rs @@ -1,6 +1,6 @@ #![allow(unused_assignments)] static USAGE: &str = r#" -Fetches data from web services for every row using HTTP Get. +Send/Fetch data to/from web services for every row using HTTP Get. Fetch is integrated with `jaq` (a jq clone) to directly parse out values from an API JSON response. (See https://github.com/01mf02/jaq for more info on how to use the jaq JSON Query Language) @@ -43,8 +43,8 @@ If you don't want responses to be cached at all, use the --no-cache flag. NETWORK OPTIONS: Fetch recognizes RateLimit and Retry-After headers and dynamically throttles requests to be as fast as allowed. The --rate-limit option sets the maximum number of queries per second -(QPS) to be made. The default is 0, which means to go as fast as possible, -automatically throttling as required. +(QPS) to be made. The default is 0, which means to go as fast as possible, automatically +throttling as required, based on rate-limit and retry-after response headers. To use a proxy, set the environment variables HTTP_PROXY, HTTPS_PROXY or ALL_PROXY (e.g. export HTTPS_PROXY=socks5://127.0.0.1:1086). diff --git a/src/cmd/fetchpost.rs b/src/cmd/fetchpost.rs index 4ed984f97..ff362fe47 100644 --- a/src/cmd/fetchpost.rs +++ b/src/cmd/fetchpost.rs @@ -1,6 +1,6 @@ #![allow(unused_assignments)] static USAGE: &str = r#" -Fetchpost fetches data from web services for every row using HTTP Post. +Fetchpost sends/fetches data to/from web services for every row using HTTP Post. As opposed to fetch, which uses HTTP Get. CSV data is posted using two methods: @@ -58,8 +58,8 @@ If you don't want responses to be cached at all, use the --no-cache flag. NETWORK OPTIONS: Fetchpost recognizes RateLimit and Retry-After headers and dynamically throttles requests to be as fast as allowed. The --rate-limit option sets the maximum number of queries per second -(QPS) to be made. The default is 0, which means to go as fast as possible, -automatically throttling as required. +(QPS) to be made. The default is 0, which means to go as fast as possible, automatically +throttling as required, based on rate-limit and retry-after response headers. To use a proxy, please set env vars HTTP_PROXY, HTTPS_PROXY or ALL_PROXY (e.g. export HTTPS_PROXY=socks5://127.0.0.1:1086).