Skip to content

[SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and lint-python #53412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gaogaotiantian wants to merge 3 commits into from
+6 −3
Closed

[SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and lint-python #53412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
094c097
Enable ruff on our CI and lint-python
gaogaotiantian Dec 10, 2025
b810e78
Pin ruff version
gaogaotiantian Dec 10, 2025
910d03c
Add E501 to ignore list
gaogaotiantian Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev/lint-python
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ if [[ -z "$COMPILE_TEST$BLACK_TEST$PYSPARK_CUSTOM_ERRORS_CHECK_TEST$FLAKE8_TEST$
BLACK_TEST=true
PYSPARK_CUSTOM_ERRORS_CHECK_TEST=true
FLAKE8_TEST=true
RUFF_TEST=true
Comment on lines 81 to +82
Copy link
Contributor

@Yicong-Huang Yicong-Huang Dec 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we enable both (flake8+black) and ruff in CI, will they have potential conflict on format? maybe better to have single one exist at a time?

Copy link
Contributor Author

@gaogaotiantian gaogaotiantian Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional - we want to make sure that ruff and flake8 are compatible at this point. We will eventually deprecate flake8.

MYPY_TEST=true
MYPY_EXAMPLES_TEST=true
MYPY_DATA_TEST=true
Expand Down
1 change: 1 addition & 0 deletions dev/spark-test-image/lint/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
RUN python3.11 -m pip install \
'black==23.12.1' \
'flake8==3.9.0' \
'ruff==0.14.8' \
'googleapis-common-protos-stubs==2.2.0' \
'grpc-stubs==1.24.11' \
'grpcio-status==1.76.0' \
Expand Down
2 changes: 1 addition & 1 deletion python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,7 +461,7 @@ def multiply_pandas(a: pd.Series, b: pd.Series) -> pd.Series:


if __name__ == "__main__":
from pyspark.sql.tests.arrow.test_arrow_udf_typehints import * # noqa: #F401
from pyspark.sql.tests.arrow.test_arrow_udf_typehints import * # noqa: F401

try:
import xmlrunner
Expand Down
2 changes: 1 addition & 1 deletion python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@ def multiply_arrow(a: pa.Array, b: pa.Array) -> pa.Array:


if __name__ == "__main__":
from pyspark.sql.tests.pandas.test_pandas_udf_typehints import * # noqa: #F401
from pyspark.sql.tests.pandas.test_pandas_udf_typehints import * # noqa: F401

try:
import xmlrunner
Expand Down
3 changes: 2 additions & 1 deletion python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,7 +367,8 @@ def func(col: "Union[pd.Series, pd.DataFrame]", *, col2: "pd.DataFrame") -> "pd.


if __name__ == "__main__":
from pyspark.sql.tests.pandas.test_pandas_udf_typehints_with_future_annotations import * # noqa: #F401
# E501: line too long
from pyspark.sql.tests.pandas.test_pandas_udf_typehints_with_future_annotations import * # noqa: F401, E501

try:
import xmlrunner
Expand Down