Communicate non-dataframe variables from livy to local

[sijunhe]

I am aware of the feature that communicate the contents of a dataframe/pandas from livy to local #333 . What about the communication of other basic data structures, like scalars, lists, etc?

[aggFTW]

That's not supported yet, and it's not in a roadmap. Can you please tell us about what you are trying to achieve?

[sijunhe]

The use case would be the same as #333, but just for normal variables on the driver. For example , I train a K means model on spark and store its cluster centers to a numpy array variable. If I need to get the variable to local, I'd need to do silly things of either converting it to a data frame and use -o or printing it out and defining the variable in local.

[aggFTW]

Got it. That is not in the works currently. What kind of user experience would you like to see? Would it just be a parameter sent to the %%spark api?

This would be a good community contribution.

[kuppu]

My vote for prioritizing this feature. I have had to use different workarounds to collect global variables.
This feature will also help in visualization, plots requires variables (data, legends, labels...) to be on single node.

%%collect - o <livy_variable>

[hanyucui]

Is this already taken care of by #432?