-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Small fix for local storage #1556
base: main
Are you sure you want to change the base?
Conversation
@microsoft-github-policy-service agree |
@@ -89,13 +85,15 @@ def generate_target_weight_position(self, score, current, trade_start_time, trad | |||
max(1 / self.topk - final_stock_weight.get(stock_id, 0), 0.0), | |||
sold_stock_weight, | |||
) | |||
final_stock_weight[stock_id] = final_stock_weight.get(stock_id, 0.0) + add_weight | |||
final_stock_weight[stock_id] = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are parentheses added here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added by my lint tool with line width 80 and should be removed.
@@ -333,7 +333,7 @@ def generate_target_weight_position(self, score, current, trade_start_time, trad | |||
|
|||
Parameters | |||
----------- | |||
score : pd.Series |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The data structure of the score returned by get_signal is defined as Union[pd.Series, pd.DataFrame, None]:
qlib/qlib/contrib/strategy/signal_strategy.py
Line 353 in 05d67b3
pred_score = self.signal.get_signal(start_time=pred_start_time, end_time=pred_end_time) |
Is it appropriate to restrict the score to be only a DataFrame?
_old_df = pd.DataFrame( | ||
_old_data[1:], index=range(_old_index, _old_index + len(_old_data) - 1), columns=["old"] | ||
) | ||
fp.seek(0) | ||
_new_df = pd.DataFrame(data_array, index=range(index, index + len(data_array)), columns=["new"]) | ||
_df = pd.concat([_old_df, _new_df], sort=False, axis=1) | ||
_df = _df.reindex(range(_df.index.min(), _df.index.max() + 1)) | ||
_df["new"].fillna(_df["old"]).values.astype("<f").tofile(fp) | ||
np.hstack([_old_index, _df["new"].fillna(_df["old"]).values]).astype("<f").tofile(fp) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why fill the missing values in np.hstack([_old_index, _df["new"]) instead of _df["new"]?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The first item _old_index
is the start index not value. Data structure is [first_index, v0, v1, v2]
.
But I think current version still have bug when new index
is smaller than _old_index
@@ -234,7 +236,6 @@ def _write_instrument(self, data: Dict[InstKT, InstVT] = None) -> None: | |||
df.loc[:, [self.SYMBOL_FIELD_NAME, self.INSTRUMENT_START_FIELD, self.INSTRUMENT_END_FIELD]].to_csv( | |||
self.uri, header=False, sep=self.INSTRUMENT_SEP, index=False | |||
) | |||
df.to_csv(self.uri, sep="\t", encoding="utf-8", header=False, index=False) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why remove this to_csv method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previous line already have to_csv
at the end of line.
) | ||
final_stock_weight[stock_id] = final_stock_weight.get( | ||
stock_id, 0.0 | ||
) + sold_stock_weight / len(buy_signal_stocks) | ||
else: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes made here did not involve any modifications to the calculation logic, and the new format did not pass the CI testing. Can we revert back to the previous format?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but lint is another problem here and I am confused by conflict between ci and pre-commit, no flake8 in ci and it is not actually used in code. So I can not enable pre-commit in my develop env.
black . -l 120 --check --diff |
Lines 11 to 12 in 05d67b3
- id: flake8 | |
args: ["--ignore=E501,F541,E266,E402,W503,E731,E203"] |
pre-commit run --all-files
black....................................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1
examples/data_demo/data_mem_resuse_demo.py:10:1: F401 'pickle' imported but unused
examples/data_demo/data_mem_resuse_demo.py:12:1: F401 'subprocess' imported but unused
examples/online_srv/online_management_simulate.py:11:1: F401 'qlib.model.trainer.DelayTrainerR' imported but unused
examples/online_srv/online_management_simulate.py:11:1: F401 'qlib.model.trainer.DelayTrainerRM' imported but unused
examples/online_srv/online_management_simulate.py:103:9: F841 local variable 'CSI300_BENCH' is assigned to but never used
tests/rl/test_saoe_simple.py:20:1: F403 'from qlib.rl.order_execution import *' used; unable to detect undefined names
tests/rl/test_saoe_simple.py:51:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:85:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:108:21: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:111:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:121:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:140:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:148:19: F405 'FullHistoryStateInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:149:24: F405 'CurrentStepStateInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:150:26: F405 'CategoricalActionInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:151:31: F405 'TwapRelativeActionInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:221:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:227:19: F405 'FullHistoryStateInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:228:21: F405 'CategoricalActionInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:232:15: F405 'Recurrent' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:233:14: F405 'PPO' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:255:20: F405 'FullHistoryStateInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:256:21: F405 'TwapRelativeActionInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:257:14: F405 'AllOne' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:261:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:284:20: F405 'FullHistoryStateInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:285:21: F405 'CategoricalActionInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:286:15: F405 'Recurrent' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:287:14: F405 'PPO' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:292:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:315:20: F405 'FullHistoryStateInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:316:21: F405 'CategoricalActionInterpreter' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:317:15: F405 'Recurrent' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:318:14: F405 'PPO' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:321:17: F405 'SingleAssetOrderExecutionSimple' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/rl/test_saoe_simple.py:326:9: F405 'PAPenaltyReward' may be undefined, or defined from star imports: qlib.rl.order_execution
tests/test_contrib_model.py:14:17: F841 local variable 'model' is assigned to but never used
examples/benchmarks/TFT/tft.py:219:9: F841 local variable 'best_loss' is assigned to but never used
examples/benchmarks/TFT/tft.py:258:9: F841 local variable 'use_gpu' is assigned to but never used
examples/benchmarks/TFT/tft.py:273:13: F841 local variable 'targets' is assigned to but never used
tests/rolling_tests/test_update_pred.py:5:1: F401 'fire' imported but unused
tests/rolling_tests/test_update_pred.py:8:1: F401 'qlib' imported but unused
tests/rolling_tests/test_update_pred.py:50:9: F841 local variable 'pred' is assigned to but never used
tests/rolling_tests/test_update_pred.py:113:9: F841 local variable 'pred' is assigned to but never used
scripts/data_collector/cn_index/collector.py:7:1: F401 'datetime' imported but unused
tests/data_mid_layer_tests/test_handler_storage.py:2:1: F401 'time' imported but unused
examples/benchmarks/TFT/libs/utils.py:36:5: E741 ambiguous variable name 'l'
scripts/data_collector/yahoo/collector.py:5:1: F401 're.I' imported but unused
scripts/data_collector/yahoo/collector.py:114:9: F841 local variable 'e' is assigned to but never used
scripts/data_collector/yahoo/collector.py:144:9: F841 local variable 'e' is assigned to but never used
scripts/data_collector/yahoo/collector.py:173:13: F841 local variable 'e' is assigned to but never used
scripts/data_collector/yahoo/collector.py:183:17: F841 local variable 'e' is assigned to but never used
scripts/data_collector/fund/collector.py:92:9: F841 local variable 'e' is assigned to but never used
scripts/data_collector/us_index/collector.py:7:1: F401 'importlib' imported but unused
examples/benchmarks_dynamic/DDG-DA/vis_data.py:2:1: F401 'numpy as np' imported but unused
examples/benchmarks_dynamic/DDG-DA/vis_data.py:10:1: F401 'tqdm.auto.tqdm' imported but unused
examples/benchmarks/TRA/src/model.py:19:1: F401 'qlib.utils.get_or_create_path' imported but unused
examples/orderbook_data/create_dataset.py:9:1: F401 'datetime.date' imported but unused
examples/orderbook_data/create_dataset.py:9:1: F401 'datetime.datetime as dt' imported but unused
examples/orderbook_data/create_dataset.py:12:1: F401 'random' imported but unused
examples/orderbook_data/create_dataset.py:17:1: F401 'arctic.chunkstore' imported but unused
examples/orderbook_data/create_dataset.py:18:1: F401 'arctic' imported but unused
examples/orderbook_data/create_dataset.py:19:1: F811 redefinition of unused 'Arctic' from line 17
examples/orderbook_data/create_dataset.py:20:1: F401 'arctic.chunkstore.chunkstore.CHUNK_SIZE' imported but unused
examples/orderbook_data/create_dataset.py:22:1: F401 'joblib.parallel' imported but unused
examples/orderbook_data/create_dataset.py:23:1: F401 'numpy as np' imported but unused
examples/orderbook_data/create_dataset.py:25:1: F401 'pandas.DataFrame' imported but unused
examples/orderbook_data/create_dataset.py:26:1: F401 'pandas.core.indexes.datetimes.date_range' imported but unused
examples/orderbook_data/create_dataset.py:59:67: F811 redefinition of unused 'date' from line 9
examples/orderbook_data/create_dataset.py:94:45: E262 inline comment should start with '# '
examples/orderbook_data/create_dataset.py:102:64: E262 inline comment should start with '# '
examples/orderbook_data/create_dataset.py:127:21: E712 comparison to True should be 'if cond is True:' or 'if cond:'
examples/orderbook_data/create_dataset.py:136:77: F811 redefinition of unused 'date' from line 9
examples/orderbook_data/create_dataset.py:278:40: F811 redefinition of unused 'date' from line 9
examples/orderbook_data/create_dataset.py:310:72: F811 redefinition of unused 'date' from line 9
scripts/data_collector/br_index/collector.py:6:1: F401 'importlib' imported but unused
examples/highfreq/highfreq_ops.py:1:1: F401 'numpy as np' imported but unused
examples/highfreq/highfreq_ops.py:3:1: F401 'importlib' imported but unused
examples/highfreq/highfreq_ops.py:5:1: F401 'qlib.config.C' imported but unused
examples/highfreq/highfreq_ops.py:6:1: F401 'qlib.data.cache.H' imported but unused
examples/highfreq/highfreq_ops.py:7:1: F401 'qlib.data.data.Cal' imported but unused
examples/highfreq/highfreq_ops.py:149:33: E741 ambiguous variable name 'l'
examples/highfreq/highfreq_ops.py:150:14: E741 ambiguous variable name 'l'
examples/online_srv/rolling_online_management.py:16:1: F401 'qlib.model.trainer.DelayTrainerR' imported but unused
examples/online_srv/rolling_online_management.py:16:1: F401 'qlib.model.trainer.TrainerR' imported but unused
examples/online_srv/rolling_online_management.py:16:1: F401 'qlib.model.trainer.end_task_train' imported but unused
examples/online_srv/rolling_online_management.py:16:1: F401 'qlib.model.trainer.task_train' imported but unused
tests/data_mid_layer_tests/test_dataset.py:6:1: F401 'sys' imported but unused
tests/test_contrib_workflow.py:70:9: F841 local variable 'uri_path' is assigned to but never used
tests/test_contrib_workflow.py:74:9: F841 local variable 'uri_path' is assigned to but never used
tests/test_pit.py:11:1: F401 'baostock as bs' imported but unused
scripts/dump_pit.py:9:1: F401 'abc' imported but unused
scripts/dump_pit.py:12:1: F401 'traceback' imported but unused
scripts/dump_pit.py:14:1: F401 'typing.List' imported but unused
scripts/dump_pit.py:14:1: F401 'typing.Union' imported but unused
scripts/dump_pit.py:16:1: F401 'concurrent.futures.ThreadPoolExecutor' imported but unused
scripts/dump_pit.py:19:1: F401 'numpy as np' imported but unused
scripts/dump_pit.py:23:1: F401 'qlib.utils.code_to_fname' imported but unused
scripts/data_collector/utils.py:85:17: F811 redefinition of unused '_get_calendar' from line 70
scripts/data_collector/utils.py:206:9: F841 local variable '_retry' is assigned to but never used
tests/rl/test_data_queue.py:64:60: F841 local variable 'data_queue' is assigned to but never used
tests/rl/test_data_queue.py:78:66: F841 local variable 'data_queue' is assigned to but never used
tests/data_mid_layer_tests/test_handler.py:3:1: F401 'shutil' imported but unused
examples/workflow_by_code.py:4:39: W291 trailing whitespace
examples/highfreq/workflow.py:13:1: F401 'qlib.data.ops.Operators' imported but unused
examples/highfreq/workflow.py:122:9: E265 block comment should start with '# '
examples/highfreq/workflow.py:127:9: E265 block comment should start with '# '
examples/highfreq/workflow.py:135:9: E265 block comment should start with '# '
examples/highfreq/workflow.py:167:9: E265 block comment should start with '# '
scripts/data_collector/crypto/collector.py:8:1: F401 'requests' imported but unused
scripts/data_collector/crypto/collector.py:41:9: E722 do not use bare 'except'
scripts/data_collector/crypto/collector.py:124:9: F841 local variable 'e' is assigned to but never used
scripts/data_collector/base.py:11:1: F401 'concurrent.futures.ThreadPoolExecutor' imported but unused
scripts/data_collector/base.py:86:13: F841 local variable 'e' is assigned to but never used
tests/backtest/test_high_freq_trading.py:1:1: F401 'typing.List' imported but unused
tests/backtest/test_high_freq_trading.py:1:1: F401 'typing.Tuple' imported but unused
tests/backtest/test_high_freq_trading.py:1:1: F401 'typing.Union' imported but unused
tests/backtest/test_high_freq_trading.py:2:1: F401 'qlib.backtest.position.Position' imported but unused
tests/backtest/test_high_freq_trading.py:4:1: F401 'qlib.backtest.decision.BaseTradeDecision' imported but unused
tests/backtest/test_high_freq_trading.py:5:1: F401 'qlib' imported but unused
tests/backtest/test_high_freq_trading.py:124:9: F841 local variable 'report' is assigned to but never used
tests/backtest/test_high_freq_trading.py:127:9: F841 local variable 'f_dec' is assigned to but never used
tests/test_all_pipeline.py:4:1: F401 'sys' imported but unused
tests/test_all_pipeline.py:10:1: F401 'qlib' imported but unused
tests/test_all_pipeline.py:11:1: F401 'qlib.config.C' imported but unused
examples/benchmarks_dynamic/baseline/rolling_benchmark.py:11:1: F401 'tqdm.auto.tqdm' imported but unused
docs/conf.py:21:1: F401 'os' imported but unused
docs/conf.py:22:1: F401 'sys' imported but unused
examples/nested_decision_execution/workflow.py:105:1: F401 'qlib.data.D' imported but unused
examples/nested_decision_execution/workflow.py:106:1: F401 'qlib.utils.exists_qlib_data' imported but unused
examples/data_demo/data_cache_demo.py:10:1: F401 'pickle' imported but unused
examples/orderbook_data/example.py:4:1: F401 'arctic.arctic.Arctic' imported but unused
examples/rolling_process_data/rolling_handler.py:2:1: F401 'qlib.data.dataset.loader.DataLoaderDH' imported but unused
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I write a Makefile to support the pre-commit check.
Like run 'make precommit' before I commit the code.
Makefile:
.PHONY: precommit
precommit: ## Lint and static-check
black . -l 120
pylint --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,E0401,E1121,C0103,C0209,R0402,R1705,R1710,R1725,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0612,W0621,W0622,W0703,W1309,E1102,E1136 --const-rgx='[a-z_][a-z0-9_]{2,30}$' qlib --init-hook "import astroid; astroid.context.InferenceContext.max_inferred = 500; import sys; sys.setrecursionlimit(2000)"
flake8 --ignore=E501,F541,E266,E402,W503,E731,E203 --per-file-ignores="init.py:F401,F403" qlib
Yes, but lint is another problem here and I am confused by conflict between ci and pre-commit, no flake8 in ci and it is not actually used in code. So I can not enable pre-commit in my develop env.
* change_url * fix_CI * fix_CI_2 * fix_CI_3 * fix_CI_4 * fix_CI_5 * fix_CI_6 * fix_CI_7 * fix_CI_8 * fix_CI_9 * fix_CI_10 * fix_CI_11 * fix_CI_12 * fix_CI_13 * fix_CI_13 * fix_CI_14 * fix_CI_15 * fix_CI_16 * fix_CI_17 * fix_CI_18 * fix_CI_19 * fix_CI_20 * fix_CI_21 * fix_CI_22 * fix_CI_23 * fix_CI_24 * fix_CI_25 * fix_CI_26 * fix_CI_27 * fix_get_data_error * fix_get_data_error2 * modify_get_data * modify_get_data2 * modify_get_data3 * modify_get_data4 * fix_CI_28 * fix_CI_29 * fix_CI_30 --------- Co-authored-by: Linlang <[email protected]>
* Update release-drafter.yml * Update release-drafter.yml
* Update README.md * Update README.md
* fix_pip_ci * fix_ci_get_data_error --------- Co-authored-by: Linlang <[email protected]>
* Update qlibrl docs. * Update docs/component/rl/guidance.rst * Update docs/component/rl/guidance.rst * Update docs/component/rl/guidance.rst --------- Co-authored-by: Litzy <[email protected]> Co-authored-by: you-n-g <[email protected]>
* Intermediate version * Fix yaml template & Successfully run rolling * Be compatible with benchmark * Get same results with previous linear model * Black formatting * Update black * Update the placeholder mechanism * Update CI * Update CI * Upgrade Black * Fix CI and simplify code * Fix CI * Move the data processing caching mechanism into utils. * Adjusting DDG-DA * Organize import
* change_publish * Update .github/workflows/python-publish.yml --------- Co-authored-by: Linlang <[email protected]> Co-authored-by: you-n-g <[email protected]>
* Bump Version & Fix CI * Update test_qlib_from_pip.yml
Co-authored-by: Linlang <[email protected]>
Fixed a spelling mistake. I changed deicsions to decisions.
Co-authored-by: Cadenza-Li <[email protected]>
Co-authored-by: Linlang <[email protected]>
* Add multi pass port ana record * Add list function * Add documentation and support <MODEL> tag * Add drop in replacement example * reformat * Change according to comments * update format * Update record_temp.py Fix type hint * Update record_temp.py
* Update vessel.py Add exploration_noise=True to training collector * Update vessel.py Reformat
* CI failed to run on 23.1 and 23.1.1 * add pyproject.toml * upgrade pip in slow.yml * upgrade build-system requires * troubleshooting pytest problem * troubleshooting pytest problem * troubleshooting pytest problem * troubleshooting pytest problem * add qlib root path to python sys.path * add qlib root path to $PYTHONPATH * add qlib root path to $PYTHONPATH * add qlib root path to $PYTHONPATH * modify pytest root; * remove set env * change_pytest_command_CI * change_pytest_command_CI * fix_ci * fix_ci * fix_ci * fix_ci * fix_ci * fix_ci * fix_ci * remove_toml * recover_toml --------- Co-authored-by: lijinhui <[email protected]> Co-authored-by: linlang <[email protected]>
* df value is set as expected, suppress the warning; * depress warning with pandas option_context --------- Co-authored-by: Cadenza-Li <[email protected]>
Ensure localstorage folder is exist during init.
Description
I found although localstorage interface support write or update method, they are not actually used. They do not check folder exist during init and calling write method when only provider uri is exist will cause error.
Motivation and Context
no
How Has This Been Tested?
pytest qlib/tests/test_all_pipeline.py
under upper directory ofqlib
.Screenshots of Test Results (if appropriate):
(it is tested with temp fix from public access error when running get_data script #1547)
Types of changes