Skip to content

Commit 117e510

Browse files
robcaulkrobcaulk
committed
Merge branch 'develop' into improve-freqai-docs
2 parents 0e7ec18 + 8051235 commit 117e510

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

66 files changed

+3315
-578
lines changed

‎.github/workflows/ci.yml

+12-2
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,16 @@ jobs:
272272
pip install pyaml
273273
python build_helpers/pre_commit_update.py
274274
275+
pre-commit:
276+
runs-on: ubuntu-22.04
277+
steps:
278+
- uses: actions/checkout@v3
279+
280+
- uses: actions/setup-python@v4
281+
with:
282+
python-version: "3.10"
283+
- uses: pre-commit/[email protected]
284+
275285
docs_check:
276286
runs-on: ubuntu-20.04
277287
steps:
@@ -302,7 +312,7 @@ jobs:
302312

303313
# Notify only once - when CI completes (and after deploy) in case it's successfull
304314
notify-complete:
305-
needs: [ build_linux, build_macos, build_windows, docs_check, mypy_version_check ]
315+
needs: [ build_linux, build_macos, build_windows, docs_check, mypy_version_check, pre-commit ]
306316
runs-on: ubuntu-20.04
307317
# Discord notification can't handle schedule events
308318
if: (github.event_name != 'schedule')
@@ -327,7 +337,7 @@ jobs:
327337
webhookUrl: ${{ secrets.DISCORD_WEBHOOK }}
328338

329339
deploy:
330-
needs: [ build_linux, build_macos, build_windows, docs_check, mypy_version_check ]
340+
needs: [ build_linux, build_macos, build_windows, docs_check, mypy_version_check, pre-commit ]
331341
runs-on: ubuntu-20.04
332342

333343
if: (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'release') && github.repository == 'freqtrade/freqtrade'

‎config_examples/config_freqai.example.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -94,4 +94,4 @@
9494
"internals": {
9595
"process_throttle_secs": 5
9696
}
97-
}
97+
}

‎config_examples/config_full.example.json

+18-1
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,24 @@
172172
"jwt_secret_key": "somethingrandom",
173173
"CORS_origins": [],
174174
"username": "freqtrader",
175-
"password": "SuperSecurePassword"
175+
"password": "SuperSecurePassword",
176+
"ws_token": "secret_ws_t0ken."
177+
},
178+
"external_message_consumer": {
179+
"enabled": false,
180+
"producers": [
181+
{
182+
"name": "default",
183+
"host": "127.0.0.2",
184+
"port": 8080,
185+
"ws_token": "secret_ws_t0ken."
186+
}
187+
],
188+
"wait_timeout": 300,
189+
"ping_timeout": 10,
190+
"sleep_time": 10,
191+
"remove_entry_exit_signals": false,
192+
"message_size_limit": 8
176193
},
177194
"bot_name": "freqtrade",
178195
"db_url": "sqlite:///tradesv3.sqlite",

‎docker/Dockerfile.freqai

-1
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,3 @@ FROM ${sourceimage}:${sourcetag}
66
COPY requirements-freqai.txt /freqtrade/
77

88
RUN pip install -r requirements-freqai.txt --user --no-cache-dir
9-

‎docs/configuration.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -225,14 +225,16 @@ Mandatory parameters are marked as **Required**, which means that they are requi
225225
| `webhook.webhookexitcancel` | Payload to send on exit order cancel. Only required if `webhook.enabled` is `true`. See the [webhook documentation](webhook-config.md) for more details. <br> **Datatype:** String
226226
| `webhook.webhookexitfill` | Payload to send on exit order filled. Only required if `webhook.enabled` is `true`. See the [webhook documentation](webhook-config.md) for more details. <br> **Datatype:** String
227227
| `webhook.webhookstatus` | Payload to send on status calls. Only required if `webhook.enabled` is `true`. See the [webhook documentation](webhook-config.md) for more details. <br> **Datatype:** String
228-
| | **Rest API / FreqUI**
228+
| | **Rest API / FreqUI / Producer-Consumer**
229229
| `api_server.enabled` | Enable usage of API Server. See the [API Server documentation](rest-api.md) for more details. <br> **Datatype:** Boolean
230230
| `api_server.listen_ip_address` | Bind IP address. See the [API Server documentation](rest-api.md) for more details. <br> **Datatype:** IPv4
231231
| `api_server.listen_port` | Bind Port. See the [API Server documentation](rest-api.md) for more details. <br>**Datatype:** Integer between 1024 and 65535
232232
| `api_server.verbosity` | Logging verbosity. `info` will print all RPC Calls, while "error" will only display errors. <br>**Datatype:** Enum, either `info` or `error`. Defaults to `info`.
233233
| `api_server.username` | Username for API server. See the [API Server documentation](rest-api.md) for more details. <br>**Keep it in secret, do not disclose publicly.**<br> **Datatype:** String
234234
| `api_server.password` | Password for API server. See the [API Server documentation](rest-api.md) for more details. <br>**Keep it in secret, do not disclose publicly.**<br> **Datatype:** String
235+
| `api_server.ws_token` | API token for the Message WebSocket. See the [API Server documentation](rest-api.md) for more details. <br>**Keep it in secret, do not disclose publicly.** <br> **Datatype:** String
235236
| `bot_name` | Name of the bot. Passed via API to a client - can be shown to distinguish / name bots.<br> *Defaults to `freqtrade`*<br> **Datatype:** String
237+
| `external_message_consumer` | Enable [Producer/Consumer mode](producer-consumer.md) for more details. <br> **Datatype:** Dict
236238
| | **Other**
237239
| `initial_state` | Defines the initial application state. If set to stopped, then the bot has to be explicitly started via `/start` RPC command. <br>*Defaults to `stopped`.* <br> **Datatype:** Enum, either `stopped` or `running`
238240
| `force_entry_enable` | Enables the RPC Commands to force a Trade entry. More information below. <br> **Datatype:** Boolean

‎docs/data-download.md

+75-33
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ usage: freqtrade download-data [-h] [-v] [--logfile FILE] [-V] [-c PATH]
2626
[--timerange TIMERANGE] [--dl-trades]
2727
[--exchange EXCHANGE]
2828
[-t TIMEFRAMES [TIMEFRAMES ...]] [--erase]
29-
[--data-format-ohlcv {json,jsongz,hdf5}]
29+
[--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}]
3030
[--data-format-trades {json,jsongz,hdf5}]
3131
[--trading-mode {spot,margin,futures}]
3232
[--prepend]
@@ -55,7 +55,7 @@ optional arguments:
5555
list. Default: `1m 5m`.
5656
--erase Clean all existing data for the selected
5757
exchange/pairs/timeframes.
58-
--data-format-ohlcv {json,jsongz,hdf5}
58+
--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}
5959
Storage format for downloaded candle (OHLCV) data.
6060
(default: `json`).
6161
--data-format-trades {json,jsongz,hdf5}
@@ -76,7 +76,7 @@ Common arguments:
7676
`userdir/config.json` or `config.json` whichever
7777
exists). Multiple --config options may be used. Can be
7878
set to `-` to read config from stdin.
79-
-d PATH, --datadir PATH
79+
-d PATH, --datadir PATH, --data-dir PATH
8080
Path to directory with historical backtesting data.
8181
--userdir PATH, --user-data-dir PATH
8282
Path to userdata directory.
@@ -179,9 +179,11 @@ freqtrade download-data --exchange binance --pairs ETH/USDT XRP/USDT BTC/USDT --
179179

180180
Freqtrade currently supports 3 data-formats for both OHLCV and trades data:
181181

182-
* `json` (plain "text" json files)
183-
* `jsongz` (a gzip-zipped version of json files)
184-
* `hdf5` (a high performance datastore)
182+
* `json` - plain "text" json files
183+
* `jsongz` - a gzip-zipped version of json files
184+
* `hdf5` - a high performance datastore
185+
* `feather` - a dataformat based on Apache Arrow
186+
* `parquet` - columnar datastore
185187

186188
By default, OHLCV data is stored as `json` data, while trades data is stored as `jsongz` data.
187189

@@ -200,38 +202,74 @@ If the default data-format has been changed during download, then the keys `data
200202
!!! Note
201203
You can convert between data-formats using the [convert-data](#sub-command-convert-data) and [convert-trade-data](#sub-command-convert-trade-data) methods.
202204

205+
#### Dataformat comparison
206+
207+
The following comparisons have been made with the following data, and by using the linux `time` command.
208+
209+
```
210+
Found 6 pair / timeframe combinations.
211+
+----------+-------------+--------+---------------------+---------------------+
212+
| Pair | Timeframe | Type | From | To |
213+
|----------+-------------+--------+---------------------+---------------------|
214+
| BTC/USDT | 5m | spot | 2017-08-17 04:00:00 | 2022-09-13 19:25:00 |
215+
| ETH/USDT | 1m | spot | 2017-08-17 04:00:00 | 2022-09-13 19:26:00 |
216+
| BTC/USDT | 1m | spot | 2017-08-17 04:00:00 | 2022-09-13 19:30:00 |
217+
| XRP/USDT | 5m | spot | 2018-05-04 08:10:00 | 2022-09-13 19:15:00 |
218+
| XRP/USDT | 1m | spot | 2018-05-04 08:11:00 | 2022-09-13 19:22:00 |
219+
| ETH/USDT | 5m | spot | 2017-08-17 04:00:00 | 2022-09-13 19:20:00 |
220+
+----------+-------------+--------+---------------------+---------------------+
221+
```
222+
223+
Timings have been taken in a not very scientific way with the following command, which forces reading the data into memory.
224+
225+
``` bash
226+
time freqtrade list-data --show-timerange --data-format-ohlcv <dataformat>
227+
```
228+
229+
| Format | Size | timing |
230+
|------------|-------------|-------------|
231+
| `json` | 149Mb | 25.6s |
232+
| `jsongz` | 39Mb | 27s |
233+
| `hdf5` | 145Mb | 3.9s |
234+
| `feather` | 72Mb | 3.5s |
235+
| `parquet` | 83Mb | 3.8s |
236+
237+
Size has been taken from the BTC/USDT 1m spot combination for the timerange specified above.
238+
239+
To have a best performance/size mix, we recommend the use of either feather or parquet.
240+
203241
#### Sub-command convert data
204242

205243
```
206244
usage: freqtrade convert-data [-h] [-v] [--logfile FILE] [-V] [-c PATH]
207245
[-d PATH] [--userdir PATH]
208246
[-p PAIRS [PAIRS ...]] --format-from
209-
{json,jsongz,hdf5} --format-to
210-
{json,jsongz,hdf5} [--erase]
211-
[-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]]
247+
{json,jsongz,hdf5,feather,parquet} --format-to
248+
{json,jsongz,hdf5,feather,parquet} [--erase]
212249
[--exchange EXCHANGE]
250+
[-t TIMEFRAMES [TIMEFRAMES ...]]
213251
[--trading-mode {spot,margin,futures}]
214-
[--candle-types {spot,,futures,mark,index,premiumIndex,funding_rate} [{spot,,futures,mark,index,premiumIndex,funding_rate} ...]]
252+
[--candle-types {spot,futures,mark,index,premiumIndex,funding_rate} [{spot,futures,mark,index,premiumIndex,funding_rate} ...]]
215253
216254
optional arguments:
217255
-h, --help show this help message and exit
218256
-p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...]
219257
Limit command to these pairs. Pairs are space-
220258
separated.
221-
--format-from {json,jsongz,hdf5}
259+
--format-from {json,jsongz,hdf5,feather,parquet}
222260
Source format for data conversion.
223-
--format-to {json,jsongz,hdf5}
261+
--format-to {json,jsongz,hdf5,feather,parquet}
224262
Destination format for data conversion.
225263
--erase Clean all existing data for the selected
226264
exchange/pairs/timeframes.
227-
-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...], --timeframes {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]
228-
Specify which tickers to download. Space-separated
229-
list. Default: `1m 5m`.
230265
--exchange EXCHANGE Exchange name (default: `bittrex`). Only valid if no
231266
config is provided.
232-
--trading-mode {spot,margin,futures}
267+
-t TIMEFRAMES [TIMEFRAMES ...], --timeframes TIMEFRAMES [TIMEFRAMES ...]
268+
Specify which tickers to download. Space-separated
269+
list. Default: `1m 5m`.
270+
--trading-mode {spot,margin,futures}, --tradingmode {spot,margin,futures}
233271
Select Trading mode
234-
--candle-types {spot,,futures,mark,index,premiumIndex,funding_rate} [{spot,,futures,mark,index,premiumIndex,funding_rate} ...]
272+
--candle-types {spot,futures,mark,index,premiumIndex,funding_rate} [{spot,futures,mark,index,premiumIndex,funding_rate} ...]
235273
Select candle type to use
236274
237275
Common arguments:
@@ -245,7 +283,7 @@ Common arguments:
245283
`userdir/config.json` or `config.json` whichever
246284
exists). Multiple --config options may be used. Can be
247285
set to `-` to read config from stdin.
248-
-d PATH, --datadir PATH
286+
-d PATH, --datadir PATH, --data-dir PATH
249287
Path to directory with historical backtesting data.
250288
--userdir PATH, --user-data-dir PATH
251289
Path to userdata directory.
@@ -267,20 +305,24 @@ freqtrade convert-data --format-from json --format-to jsongz --datadir ~/.freqtr
267305
usage: freqtrade convert-trade-data [-h] [-v] [--logfile FILE] [-V] [-c PATH]
268306
[-d PATH] [--userdir PATH]
269307
[-p PAIRS [PAIRS ...]] --format-from
270-
{json,jsongz,hdf5} --format-to
271-
{json,jsongz,hdf5} [--erase]
308+
{json,jsongz,hdf5,feather,parquet}
309+
--format-to
310+
{json,jsongz,hdf5,feather,parquet}
311+
[--erase] [--exchange EXCHANGE]
272312
273313
optional arguments:
274314
-h, --help show this help message and exit
275315
-p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...]
276-
Show profits for only these pairs. Pairs are space-
316+
Limit command to these pairs. Pairs are space-
277317
separated.
278-
--format-from {json,jsongz,hdf5}
318+
--format-from {json,jsongz,hdf5,feather,parquet}
279319
Source format for data conversion.
280-
--format-to {json,jsongz,hdf5}
320+
--format-to {json,jsongz,hdf5,feather,parquet}
281321
Destination format for data conversion.
282322
--erase Clean all existing data for the selected
283323
exchange/pairs/timeframes.
324+
--exchange EXCHANGE Exchange name (default: `bittrex`). Only valid if no
325+
config is provided.
284326
285327
Common arguments:
286328
-v, --verbose Verbose mode (-vv for more, -vvv to get all messages).
@@ -293,7 +335,7 @@ Common arguments:
293335
`userdir/config.json` or `config.json` whichever
294336
exists). Multiple --config options may be used. Can be
295337
set to `-` to read config from stdin.
296-
-d PATH, --datadir PATH
338+
-d PATH, --datadir PATH, --data-dir PATH
297339
Path to directory with historical backtesting data.
298340
--userdir PATH, --user-data-dir PATH
299341
Path to userdata directory.
@@ -318,22 +360,22 @@ This command will allow you to repeat this last step for additional timeframes w
318360
usage: freqtrade trades-to-ohlcv [-h] [-v] [--logfile FILE] [-V] [-c PATH]
319361
[-d PATH] [--userdir PATH]
320362
[-p PAIRS [PAIRS ...]]
321-
[-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]]
363+
[-t TIMEFRAMES [TIMEFRAMES ...]]
322364
[--exchange EXCHANGE]
323-
[--data-format-ohlcv {json,jsongz,hdf5}]
365+
[--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}]
324366
[--data-format-trades {json,jsongz,hdf5}]
325367
326368
optional arguments:
327369
-h, --help show this help message and exit
328370
-p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...]
329371
Limit command to these pairs. Pairs are space-
330372
separated.
331-
-t {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...], --timeframes {1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} [{1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w,2w,1M,1y} ...]
373+
-t TIMEFRAMES [TIMEFRAMES ...], --timeframes TIMEFRAMES [TIMEFRAMES ...]
332374
Specify which tickers to download. Space-separated
333375
list. Default: `1m 5m`.
334376
--exchange EXCHANGE Exchange name (default: `bittrex`). Only valid if no
335377
config is provided.
336-
--data-format-ohlcv {json,jsongz,hdf5}
378+
--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}
337379
Storage format for downloaded candle (OHLCV) data.
338380
(default: `json`).
339381
--data-format-trades {json,jsongz,hdf5}
@@ -351,7 +393,7 @@ Common arguments:
351393
`userdir/config.json` or `config.json` whichever
352394
exists). Multiple --config options may be used. Can be
353395
set to `-` to read config from stdin.
354-
-d PATH, --datadir PATH
396+
-d PATH, --datadir PATH, --data-dir PATH
355397
Path to directory with historical backtesting data.
356398
--userdir PATH, --user-data-dir PATH
357399
Path to userdata directory.
@@ -371,7 +413,7 @@ You can get a list of downloaded data using the `list-data` sub-command.
371413
```
372414
usage: freqtrade list-data [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH]
373415
[--userdir PATH] [--exchange EXCHANGE]
374-
[--data-format-ohlcv {json,jsongz,hdf5}]
416+
[--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}]
375417
[-p PAIRS [PAIRS ...]]
376418
[--trading-mode {spot,margin,futures}]
377419
[--show-timerange]
@@ -380,13 +422,13 @@ optional arguments:
380422
-h, --help show this help message and exit
381423
--exchange EXCHANGE Exchange name (default: `bittrex`). Only valid if no
382424
config is provided.
383-
--data-format-ohlcv {json,jsongz,hdf5}
425+
--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}
384426
Storage format for downloaded candle (OHLCV) data.
385427
(default: `json`).
386428
-p PAIRS [PAIRS ...], --pairs PAIRS [PAIRS ...]
387429
Limit command to these pairs. Pairs are space-
388430
separated.
389-
--trading-mode {spot,margin,futures}
431+
--trading-mode {spot,margin,futures}, --tradingmode {spot,margin,futures}
390432
Select Trading mode
391433
--show-timerange Show timerange available for available data. (May take
392434
a while to calculate).
@@ -402,7 +444,7 @@ Common arguments:
402444
`userdir/config.json` or `config.json` whichever
403445
exists). Multiple --config options may be used. Can be
404446
set to `-` to read config from stdin.
405-
-d PATH, --datadir PATH
447+
-d PATH, --datadir PATH, --data-dir PATH
406448
Path to directory with historical backtesting data.
407449
--userdir PATH, --user-data-dir PATH
408450
Path to userdata directory.

0 commit comments

Comments
 (0)