Skip to content

Commit a82aee0

Browse files
committed
[SPARK-32435][PYTHON] Remove heapq3 port from Python 3
### What changes were proposed in this pull request? This PR removes the manual port of `heapq3.py` introduced from SPARK-3073. The main reason of this was to support Python 2.6 and 2.7 because Python 2's `heapq.merge()` doesn't not support `key` and `reverse`. See - https://docs.python.org/2/library/heapq.html#heapq.merge in Python 2 - https://docs.python.org/3.8/library/heapq.html#heapq.merge in Python 3 Since we dropped the Python 2 at SPARK-32138, we can remove this away. ### Why are the changes needed? To remove unnecessary codes. Also, we can leverage bug fixes made in Python 3.x at `heapq`. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Existing tests should cover. I locally ran and verified: ```bash ./python/run-tests --python-executable=python3 --testname="pyspark.tests.test_shuffle" ./python/run-tests --python-executable=python3 --testname="pyspark.shuffle ExternalSorter" ./python/run-tests --python-executable=python3 --testname="pyspark.tests.test_rdd RDDTests.test_external_group_by_key" ``` Closes apache#29229 from HyukjinKwon/SPARK-32435. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent 6ab29b3 commit a82aee0

File tree

9 files changed

+5
-1232
lines changed

9 files changed

+5
-1232
lines changed

LICENSE

-1
Original file line numberDiff line numberDiff line change
@@ -222,7 +222,6 @@ external/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaRe
222222
Python Software Foundation License
223223
----------------------------------
224224

225-
pyspark/heapq3.py
226225
python/docs/source/_static/copybutton.js
227226

228227
BSD 3-Clause

LICENSE-binary

-6
Original file line numberDiff line numberDiff line change
@@ -557,12 +557,6 @@ jakarta.ws.rs:jakarta.ws.rs-api https://github.com/eclipse-ee4j/jaxrs-api
557557
org.glassfish.hk2.external:jakarta.inject
558558

559559

560-
Python Software Foundation License
561-
----------------------------------
562-
563-
pyspark/heapq3.py
564-
565-
566560
Public Domain
567561
-------------
568562

dev/.rat-excludes

-1
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,6 @@ jsonFormatter.min.js
4949
.*log
5050
pyspark-coverage-site/*
5151
cloudpickle/*
52-
heapq3.py
5352
join.py
5453
SparkExprTyper.scala
5554
SparkILoop.scala

dev/tox.ini

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,4 @@
1616
[pycodestyle]
1717
ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504
1818
max-line-length=100
19-
exclude=python/pyspark/cloudpickle/*.py,heapq3.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
19+
exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*

licenses-binary/LICENSE-heapq.txt

-280
This file was deleted.

licenses/LICENSE-heapq.txt

-49
This file was deleted.

python/pylintrc

+1-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ profile=no
2929

3030
# Add files or directories to the ignoreList. They should be base names, not
3131
# paths.
32-
ignore=pyspark.heapq3
32+
#ignore=
3333

3434
# Pickle collected data for later comparisons.
3535
persistent=yes

0 commit comments

Comments
 (0)