-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoiding recalculation. #20
Comments
Are you using scons with nestly? If so, then you certainly can (I do all the time). If not, I can't tell you, since I pretty much only use nestly in concert with scons. If you're not already, it's probably worth trying out scons given you have costly computations, since scons keeps track of dependencies/changes and only run things that actually need to be run. |
Ok thanks for the response. No I've not tried the scons integration. I'm unfamiliar with scons but I have used make and dependencies in C projects in the long ago past. Being a build system I expected that it would avoid repeated calculations between runs over the same nest, but can it also identify redundant calculations within the same run? |
If you've used make, scons should be a walk in the park :-) Still, I do wish we had a somewhat better tutorial/example for how to use these things together for someone not familiar with scons. Fortunately though, 98% of what you need to know to get started using scons for this kind of thing is the
Scons won't exactly find redundant calculations, but you get the desired behaviour based on where you add your targets. E.g.
Does that help? Note that there are all sorts of other fun things possible here:
|
Very cool and thanks for the great assistance. I'll dig into this tomorrow morning. :-) I used an ugly hack of calling qsub with "-W block=true" to imitate local processes. heh. |
It is likely I am approaching this in a dumb fashion, but is it possible to pin intermediate elements in a Nestly workflow higher in the tree and not calculate them at each tip?
I have intermediate results which are consumptive of both disk and cpu and at the tip there is duplication since these intermediates only depend on a subset of the parameter space. Do I simply partition this into separate nests?
The text was updated successfully, but these errors were encountered: