Use PyPy

[VincenzoFerme]

Test the current code using PyPy: http://ianozsvald.com/2015/02/19/spark-1-2-pyspark-elasticsearch-pypy/

[Cerfoglg]

Note: pypy3 doesn't currently work with Spark, only pypy for python 2.7 does.

[Cerfoglg]

@VincenzoFerme I have tested pypy 4.0.1 for Python 2.7, and the code works. Pypy3 does not at the moment, so we will need to use Python 2.7 for now. In the future, if pypy3 becomes supported by Spark, we can consider using Python 3 with pypy3

[VincenzoFerme]

@Cerfoglg what did you actually do for testing this? I would like to try to replicate it.

[Cerfoglg]

@VincenzoFerme Download pypy 4.0.1, and set the env variable PYSPARK_PYTHON to the path of the pypy file in the bin folder, so that when launching the script on Spark it will use pypy as the interpreter

[VincenzoFerme]

@Cerfoglg is there already an issue opened on the Spark or Pypy repository or Issue tracker? If not, can you please open a new one and link it to this issue? Thank you.

[Cerfoglg]

@VincenzoFerme Doesn't look like you can report issues directly to Spark, and I'm not sure reporting issues to pypy makes sense, since it's not their problem Spark doesn't support pypy3. Something tells me that this wouldn't be high on their priority list though.

Let's keep this on the side in case a new version of Spark allows us to use pypy3, if not we stick with pypy2 for now.

[VincenzoFerme]

@Cerfoglg they use JIRA. Here you can find the instruction to post a new issue.