Skip to content
This repository has been archived by the owner on Dec 16, 2023. It is now read-only.

Improving Sparse Solving for Memory and Speed #69

Open
zasexton opened this issue Jul 10, 2022 · 8 comments
Open

Improving Sparse Solving for Memory and Speed #69

zasexton opened this issue Jul 10, 2022 · 8 comments

Comments

@zasexton
Copy link

zasexton commented Jul 10, 2022

During initial construction a dense matrix M is used. It seems that csr_matrix has been imported from scipy but is never actually used to instantiate a sparse csr matrix. Also, it seems that for sparse solving we are supplying the vector b as an array type. If this is the case, I believe scipy.sparse.linalg.spsolve will convert between dense to sparse and then back to dense during each time step because spsolve cannot assume that the solution X will be sparse.

I think to get the full power of scipy's solvers we might need to add a few changes here and during inital matrix M construction.
https://github.com/SimVascular/svZeroDSolver/blob/817b95f6aedde24f158b1e7148920e2cf912030f/svzerodsolver/time_integration.py#L63-L68
https://github.com/SimVascular/svZeroDSolver/blob/817b95f6aedde24f158b1e7148920e2cf912030f/svzerodsolver/time_integration.py#L93-L104
Avoiding excessive matrix reformating should significantly speed up sparse applications.

Let me know what y'all think @ktbolt @mrp089 @JonathanPham @richterjakob

@ktbolt
Copy link
Contributor

ktbolt commented Jul 11, 2022

@zasexton What is the motivation for this modification? Are you trying to use the solver and finding that it is still too slow? I'm curious about what the sizes of the matrices involved are.

@zasexton
Copy link
Author

@ktbolt I won't really say this is a modification as much as fixing a bug because I don't believe that there should be much matrix reformatting during the time integration? Although I'm not a main contributor to this package so there could very well be something that I'm overlooking. The sparse solver seems to already be an option that is invoked for matrices larger than 800. Personally, my applications have vessel networks of at least 4000 vessels so sparse solving is really the only route for running on a personal machine. However, if I attempt this the solver will crash due to memory insufficiencies.

@zasexton
Copy link
Author

I'd also like to add that I am not coding up anything for this issue right now. I was more interested in bouncing this to the team to sanity check what I am seeing on my end. I might not be the best person to fix this since I don't have the most knowledge about the zeroDsolver in general.

@ktbolt
Copy link
Contributor

ktbolt commented Jul 11, 2022

If this is crashing for your application then it is a bug and it should be fixed.

@zasexton
Copy link
Author

Yes, currently I've been moving larger vascular systems to sherlock where I can allocate more memory for the job...obviously not ideal. If no one has the bandwidth to take this on I can assign myself to it and I'll take a look later this week/weekend.

@ktbolt
Copy link
Contributor

ktbolt commented Jul 12, 2022

@zasexton You might could try the C++ version implemented by @richterjakob located here https://github.com/richterjakob/svZeroDSolver/tree/cpp. It would be interesting to see how much memory it uses for your vessel networks.

@mrp089
Copy link
Member

mrp089 commented Jul 15, 2022

@zasexton In Python, I found that assembling into a dense matrix (and then converting to sparse for large systems) is much faster than assembling directly into a sparse matrix. This is counterintuitive but was also found by @richterjakob during his speedup experimentations.

Of course, this ignores the memory demand which was never an issue for our models. However, I understand that assembling into a dense matrix is problematic for larger systems. Maybe it makes sense to include another switch and change the assembly strategy? But I don't know if we can do that in an easy, modular way.

Let us know what happens in the C++ version!

@zasexton
Copy link
Author

@mrp089 @ktbolt @richterjakob Yes, I will be running these tests with both versions (python & C++) on sherlock for my larger networks this weekend using some of my automatic scripting. I'll include what I notice in this thread. Thank you all so much for the advice so far!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants