Skip to content
This repository has been archived by the owner on Oct 31, 2018. It is now read-only.

Support checkpoint RDDs #9

Open
kayousterhout opened this issue Jan 28, 2015 · 0 comments
Open

Support checkpoint RDDs #9

kayousterhout opened this issue Jan 28, 2015 · 0 comments
Labels

Comments

@kayousterhout
Copy link
Member

Support for checkpointing was removed with the monotasks change: if a RDD is checkpointed, the resulting job will fail with an exception that looks like: java.lang.Error: org.apache.spark.SparkException: Missing parent partition information for partition 0 of dependency org.apache.spark.OneToOneDependency@643812a8 (should have been set in DAGScheduler)

This can be fixed by fixing the way that the parent partitions get serialized in Macrotask, similar to what's done in other kinds of dependencies where the RDD can get checkpointed.

Once is done, the CheckpointSuite of tests should be re-enabled.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant