[perflint] fix invalid hoist in perf401
#14369
Conversation
|
| code | total | + violation | - violation | + fix | - fix |
|---|---|---|---|---|---|
| PERF401 | 43 | 14 | 29 | 0 | 0 |
| RUF100 | 3 | 3 | 0 | 0 | 0 |
Linter (preview)
ℹ️ ecosystem check detected linter changes. (+17 -29 violations, +0 -0 fixes in 5 projects; 49 projects unchanged)
apache/airflow (+6 -11 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
- dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3070:9: PERF401 Use `list.extend` to create a transformed list + dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3070:9: PERF401 Use a list comprehension to create a transformed list - dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use `list.extend` to create a transformed list + dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list comprehension to create a transformed list - dev/breeze/src/airflow_breeze/utils/packages.py:327:9: PERF401 Use `list.extend` to create a transformed list + dev/breeze/src/airflow_breeze/utils/packages.py:327:9: PERF401 Use a list comprehension to create a transformed list - docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use `list.extend` to create a transformed list + docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list comprehension to create a transformed list - docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use `list.extend` to create a transformed list - docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use `list.extend` to create a transformed list - providers/src/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:400:25: PERF401 Use a list comprehension to create a transformed list - providers/src/airflow/providers/microsoft/azure/hooks/wasb.py:721:13: PERF401 Use `list.extend` with an async comprehension to create a transformed list + providers/src/airflow/providers/microsoft/azure/hooks/wasb.py:721:13: PERF401 Use an async list comprehension to create a transformed list - scripts/in_container/run_provider_yaml_files_check.py:178:17: PERF401 Use `list.extend` to create a transformed list - scripts/in_container/update_quarantined_test_status.py:81:13: PERF401 Use `list.extend` to create a transformed list + scripts/in_container/update_quarantined_test_status.py:81:13: PERF401 Use a list comprehension to create a transformed list - tests/jobs/test_scheduler_job.py:759:13: PERF401 Use a list comprehension to create a transformed list
apache/superset (+4 -10 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
- scripts/benchmark_migration.py:128:21: PERF401 Use `list.extend` to create a transformed list - superset/db_engine_specs/base.py:107:9: PERF401 Use `list.extend` to create a transformed list + superset/db_engine_specs/base.py:107:9: PERF401 Use a list comprehension to create a transformed list + superset/db_engine_specs/lib.py:245:9: PERF401 Use `list.extend` to create a transformed list - superset/db_engine_specs/lib.py:245:9: PERF401 Use a list comprehension to create a transformed list + superset/db_engine_specs/lib.py:251:9: PERF401 Use `list.extend` to create a transformed list - superset/db_engine_specs/lib.py:251:9: PERF401 Use a list comprehension to create a transformed list - superset/db_engine_specs/lib.py:261:9: PERF401 Use a list comprehension to create a transformed list - superset/db_engine_specs/lib.py:281:9: PERF401 Use a list comprehension to create a transformed list - superset/db_engine_specs/lib.py:293:9: PERF401 Use a list comprehension to create a transformed list - superset/tasks/cache.py:208:13: PERF401 Use a list comprehension to create a transformed list - tests/integration_tests/annotation_layers/fixtures.py:85:9: PERF401 Use a list comprehension to create a transformed list + tests/integration_tests/security/migrate_roles_tests.py:50:13: PERF401 Use `list.extend` to create a transformed list - tests/integration_tests/security/migrate_roles_tests.py:50:13: PERF401 Use a list comprehension to create a transformed list
bokeh/bokeh (+1 -3 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
- src/bokeh/plotting/_figure.py:479:17: PERF401 Use `list.extend` to create a transformed list - src/bokeh/plotting/_figure.py:485:17: PERF401 Use `list.extend` to create a transformed list - src/bokeh/server/contexts.py:310:17: PERF401 Use `list.extend` to create a transformed list + src/bokeh/server/contexts.py:310:17: PERF401 Use a list comprehension to create a transformed list
latchbio/latch (+3 -5 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview
- src/latch/ldata/_transfer/upload.py:163:25: PERF401 Use `list.extend` to create a transformed list + src/latch/ldata/_transfer/upload.py:163:25: PERF401 Use a list comprehension to create a transformed list - src/latch/registry/utils.py:70:13: PERF401 Use `list.extend` to create a transformed list + src/latch/registry/utils.py:70:13: PERF401 Use a list comprehension to create a transformed list - src/latch_cli/services/cp/utils.py:54:9: PERF401 Use `list.extend` to create a transformed list + src/latch_cli/services/cp/utils.py:54:9: PERF401 Use a list comprehension to create a transformed list - src/latch_cli/snakemake/workflow.py:158:21: PERF401 Use `list.extend` to create a transformed list - src/latch_cli/snakemake/workflow.py:978:25: PERF401 Use a list comprehension to create a transformed list
indico/indico (+3 -0 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview
+ indico/modules/events/registration/lists.py:116:93: RUF100 [*] Unused `noqa` directive (unused: `PERF401`) + indico/modules/rb/models/rooms.py:513:91: RUF100 [*] Unused `noqa` directive (unused: `PERF401`) + indico/modules/rb/operations/rooms.py:60:87: RUF100 [*] Unused `noqa` directive (unused: `PERF401`)
Changes by rule (2 rules affected)
| code | total | + violation | - violation | + fix | - fix |
|---|---|---|---|---|---|
| PERF401 | 43 | 14 | 29 | 0 | 0 |
| RUF100 | 3 | 3 | 0 | 0 | 0 |
charliermarsh
added
bug
|
@w0nder1ng Another fun edge case that PR #14369 doesn't currently fix. Scoping rules are different between list comprehensions and for loops: another example from pytorch if kwargs is None:
kwargs = {}
- impl_args = []
- for a in args:
- impl_args.append(_helper(a, map_fn))
+ impl_args = [_helper(a, map_fn) for a in args]
impl_kwargs = {}
for k in kwargs.keys():
impl_kwargs[k] = _helper(a, map_fn)the @w0nder1ng If you want a good test bed, just run the autofix on PyTorch (or another large codebase) and see after applying the fixes if any other ruff rule violations are immediately detected. |
|
Also another really minor nit is that it can duplicate comments. - for dtype in ["f16", "bf16"]:
- kernels.append(
- cls(
- aligned=True,
- dtype=dtype,
- sm_range=(80, SM[SM.index(80) + 1]),
- apply_dropout=False,
- preload_mmas=True,
- block_i=128,
- block_j=64,
- max_k=96,
- # Sm80 has a faster kernel for this case
- dispatch_cond="cc == 86 || cc == 89",
- )
+ # Sm80 has a faster kernel for this case
+ kernels.extend(
+ cls(
+ aligned=True,
+ dtype=dtype,
+ sm_range=(80, SM[SM.index(80) + 1]),
+ apply_dropout=False,
+ preload_mmas=True,
+ block_i=128,
+ block_j=64,
+ max_k=96,
+ # Sm80 has a faster kernel for this case
+ dispatch_cond="cc == 86 || cc == 89",
)
+ for dtype in ["f16", "bf16"]
+ )See here. |
I think in this case, the lint shouldn't apply at all. If applying the fix breaks the code, then someone manually doing the same thing also wouldn't work. It should probably check that all references to the loop variable are inside the for loop before reporting the lint. I'll also try to fix the duplicate comments. I'm starting to see why this didn't have a fix before :) |
|
The comment duplication happened when the comment was inside the append body, it should be fixed now. |
| // ``` | ||
| let for_loop_target = checker | ||
| .semantic() | ||
| .lookup_symbol(id.as_str()) |
There was a problem hiding this comment.
For some reason, resolve_name returns None for the for-loop target. Also, references to it outside of the for-loop are not included in its list of references; e.g.
def f():
result = []
for val in range(5):
result.append(val * 2)
print(val) # this reference is not included in the list of references
There was a problem hiding this comment.
I think the problem here is that we build the semantic model lazily as we traverse the AST for checking and it hasn't reached the point after the for loop yet
There was a problem hiding this comment.
@charliermarsh any suggestions on how to best find all usages of target?
|
Could you take a look why https://github.com/apache/superset/blob/e528cb48c44543c14c1ac9a93528b147bcaecfde/scripts/benchmark_migration.py#L128 is no longer reported. I might have missed something obvious but it isn't clear to me why the diagnostic isn't reported anymore. |
|
I suspect the regressions are because of the |
Because the lambda arg here https://github.com/apache/superset/blob/e528cb48c44543c14c1ac9a93528b147bcaecfde/scripts/benchmark_migration.py#L131C28-L131C33 is shadowing the variable "model" in the for loop. In this specific case, it's probably okay since it's in a lambda arg, but in general it shouldn't apply the fix there. I suppose only references rvalues are problematic outside of the forloop, not lvalues (if all they do is immediately get overwritten after all). |
|
Okay hopefully last nit: reduction_axes: List[int] = []
- for i in range(input_rank):
- if i != axis:
- reduction_axes.append(i)
+ reduction_axes.extend(i for i in range(input_rank) if i != axis)Doesn't seem to like to hoist if there are any type_hints on the original |
|
I'll take a look |
|
Annotated assigns have a different statement type than normal assigns, and I wasn't handling it. The fix should work on type-annotated lists now. |
This one is still occurring sadly. Maybe because it references another loop lol? |
|
Despite this minor false positive, looks like all the other fixes worked well on the PyTorch codebase (for the torch/torchgen folders) pytorch/pytorch#140980. 👍 |
|
I'm still stuck on two things.
result = []; test = [] # deleting this whole line is wrong, but only deleting the binding statement also is wrong
# since it would leave the comment if there were no other statement
for ...
result = []
for i in range(10):
result.append(i+1)
print(i) # "fixing" this for loop is invalid because something relies on the loop variable, but the semantic model doesn't see this reference yet |
This should fix #14362. This new fix currently deletes lines like this:
Is there a convenient way to get every statement within a given
TextRangeto detect when this is happening?