-
Notifications
You must be signed in to change notification settings - Fork 15
Improving Sparse Solving for Memory and Speed #69
Comments
@zasexton What is the motivation for this modification? Are you trying to use the solver and finding that it is still too slow? I'm curious about what the sizes of the matrices involved are. |
@ktbolt I won't really say this is a modification as much as fixing a bug because I don't believe that there should be much matrix reformatting during the time integration? Although I'm not a main contributor to this package so there could very well be something that I'm overlooking. The sparse solver seems to already be an option that is invoked for matrices larger than 800. Personally, my applications have vessel networks of at least 4000 vessels so sparse solving is really the only route for running on a personal machine. However, if I attempt this the solver will crash due to memory insufficiencies. |
I'd also like to add that I am not coding up anything for this issue right now. I was more interested in bouncing this to the team to sanity check what I am seeing on my end. I might not be the best person to fix this since I don't have the most knowledge about the zeroDsolver in general. |
If this is crashing for your application then it is a bug and it should be fixed. |
Yes, currently I've been moving larger vascular systems to sherlock where I can allocate more memory for the job...obviously not ideal. If no one has the bandwidth to take this on I can assign myself to it and I'll take a look later this week/weekend. |
@zasexton You might could try the C++ version implemented by @richterjakob located here https://github.com/richterjakob/svZeroDSolver/tree/cpp. It would be interesting to see how much memory it uses for your vessel networks. |
@zasexton In Python, I found that assembling into a dense matrix (and then converting to sparse for large systems) is much faster than assembling directly into a sparse matrix. This is counterintuitive but was also found by @richterjakob during his speedup experimentations. Of course, this ignores the memory demand which was never an issue for our models. However, I understand that assembling into a dense matrix is problematic for larger systems. Maybe it makes sense to include another switch and change the assembly strategy? But I don't know if we can do that in an easy, modular way. Let us know what happens in the C++ version! |
@mrp089 @ktbolt @richterjakob Yes, I will be running these tests with both versions (python & C++) on sherlock for my larger networks this weekend using some of my automatic scripting. I'll include what I notice in this thread. Thank you all so much for the advice so far! |
During initial construction a dense matrix M is used. It seems that csr_matrix has been imported from scipy but is never actually used to instantiate a sparse csr matrix. Also, it seems that for sparse solving we are supplying the vector b as an array type. If this is the case, I believe scipy.sparse.linalg.spsolve will convert between dense to sparse and then back to dense during each time step because spsolve cannot assume that the solution X will be sparse.
I think to get the full power of scipy's solvers we might need to add a few changes here and during inital matrix M construction.
https://github.com/SimVascular/svZeroDSolver/blob/817b95f6aedde24f158b1e7148920e2cf912030f/svzerodsolver/time_integration.py#L63-L68
https://github.com/SimVascular/svZeroDSolver/blob/817b95f6aedde24f158b1e7148920e2cf912030f/svzerodsolver/time_integration.py#L93-L104
Avoiding excessive matrix reformating should significantly speed up sparse applications.
Let me know what y'all think @ktbolt @mrp089 @JonathanPham @richterjakob
The text was updated successfully, but these errors were encountered: