Collection of some interesting pieces from my projects.
Integration test, bash scripts
Python wrapper for JVM methods
Spark.ml, Python wrappers for JVM implementation of estimators and models (see spark.ml lib)
- ScoreEqualizeTransformer + ScoreEqualizeTransformerModel
- NEPriorClassProbaTransformer + NEPriorClassProbaTransformerModel
- ScoreQuantileThresholdTransformer + ScoreQuantileThresholdTransformerModel
ML model inference, migration first steps (to Scala-Apply)
- Convert npz file (from numpy.savez_compressed) to json file
- Simple spark job, apply ML model to test data
Scala-Apply, Python app and wrappers for JVM implementation
- Luigi 'Apply' spark-submit task and 'apply_scala_models' method
- Spark.ml transformer ApplyModelsTransformer
- Three ML models adopted for Scala-Apply
- Models json serialization code
- Tests for dynamic ML models serialization and Scala-Apply wrappers
Join-Features, Python app
Export (scored) records, app
Combine ML features, app
Experiments, test library
One-time scripts (adhoc)
TODO: all test scripts should run successfully (in proper docker container).