[SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and lint-python

### What changes were proposed in this pull request?

This add `ruff` to our docker image and enables ruff check in our CI. Also this adds ruff check in `dev/lint-python`.

We want to have both `ruff` and `flake8` run in CI for a while to confirm the compatibility then we will deprecate `flake8`.

~~It is intentional to leave ruff version blank - so it uses the latest version. I think the linter rule is pretty stable and the version upgrade should not affect our workflow too much. This is an experiment about using the latest version of devtool. Pinning the version has its advantage but upgrading the version is painful. If we can occasionally (per month?) fix a small amount of code for linter, I think that's acceptable.~~

After discussion with Yicong-Huang I think it's a good idea to pin the version - so we can do all the code fix with devtool change in a single PR. We should avoid having multiple people trying to fix the code when there is a new rule.

### Why are the changes needed?

`flake8` is too slow and our pinned version is just too old. We should replace it.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Local ruff test passes.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #53412 from gaogaotiantian/add-ruff-to-ci.

Authored-by: Tian Gao <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>

Author: gaogaotiantian

dev/lint-python

@@ -79,6 +79,7 @@ if [[ -z "$COMPILE_TEST$BLACK_TEST$PYSPARK_CUSTOM_ERRORS_CHECK_TEST$FLAKE8_TEST$
   BLACK_TEST=true
   PYSPARK_CUSTOM_ERRORS_CHECK_TEST=true
   FLAKE8_TEST=true
+  RUFF_TEST=true
   MYPY_TEST=true
   MYPY_EXAMPLES_TEST=true
   MYPY_DATA_TEST=true

dev/spark-test-image/lint/Dockerfile

@@ -80,6 +80,7 @@ RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
 RUN python3.11 -m pip install \
     'black==23.12.1' \
     'flake8==3.9.0' \
+    'ruff==0.14.8' \
     'googleapis-common-protos-stubs==2.2.0' \
     'grpc-stubs==1.24.11' \
     'grpcio-status==1.76.0' \

python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py

@@ -461,7 +461,7 @@ def multiply_pandas(a: pd.Series, b: pd.Series) -> pd.Series:
 
 
 if __name__ == "__main__":
-    from pyspark.sql.tests.arrow.test_arrow_udf_typehints import *  # noqa: #F401
+    from pyspark.sql.tests.arrow.test_arrow_udf_typehints import *  # noqa: F401
 
     try:
         import xmlrunner

python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py

@@ -444,7 +444,7 @@ def multiply_arrow(a: pa.Array, b: pa.Array) -> pa.Array:
 
 
 if __name__ == "__main__":
-    from pyspark.sql.tests.pandas.test_pandas_udf_typehints import *  # noqa: #F401
+    from pyspark.sql.tests.pandas.test_pandas_udf_typehints import *  # noqa: F401
 
     try:
         import xmlrunner

python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py

@@ -367,7 +367,8 @@ def func(col: "Union[pd.Series, pd.DataFrame]", *, col2: "pd.DataFrame") -> "pd.
 
 
 if __name__ == "__main__":
-    from pyspark.sql.tests.pandas.test_pandas_udf_typehints_with_future_annotations import *  # noqa: #F401
+    # E501: line too long
+    from pyspark.sql.tests.pandas.test_pandas_udf_typehints_with_future_annotations import *  # noqa: F401, E501
 
     try:
         import xmlrunner