Simulating models in supercomputers

[optibikash]

I have been using PyBamm for a while and I really appreciate the efforts put forward and the wonderful results it gives. While working with the model, I experimented to simulate a code in the supercomputer, I observed that it was not utilizing multiple cores in multiple nodes to simulate the model. The program ran with only one node and multiple cores (the similar way it runs on a laptop or desktop) even when multiple nodes were allocated.
The implementation will be useful for the people who want to simulate aging effects by adding multiple sub-models as the computation time increases with the addition of sub-models.
Maybe someone can take up the task to run models using supercomputers whose jobs are managed by Slurm (opensource workload manager)

[valentinsulzer]

This is almost ready in #1261

[valentinsulzer]

The functionality for solving with multiple inputs in parallel is now merged into develop. However, we have not implemented anything to parallelize a single solve of the model. I don't think this is particularly high on any of the developers' priority list, but if someone wants to take it on that would be great.

[TomTranter]

Just thinking about this @optibikash lets say you want to run 1000 cycles. If you have a degradation model that modifies some transport property could you run these one cycle behind using the results of the previous solve and updating the relevant parameter for the next electrochemical cycle. I could see how this would parallelize well but I think introducing sub cycle feedback would not be worth the effort parallelizing unless you had a lot of concurrent calculations to make. I have made some code that runs multiple simulations in parallel and then does some thermal calculations on the group and updates temperature in this way. It works because different parts of the cell can be assumed to act independently when given the local voltage and current by an external solver. However parallelizing a single 1D model for different submodels is different. Would be interesting to know more about what you're modelling

[optibikash]

Thanks, @TomTranter , for the response. What you assumed is very different. I don't think at this point of time, we have any advanced model which can give simuation results for any future number of cycles. A rigorously tested prediction model will be required and the model will have to be tested on different conditions such as temperature, applied current density, etc.
One of my idea for parallelizing is w.r.t. to running multiple simulations for a varied range of parameters. Let us assume one wants to solve the model for different charging rates or discharging rates. At present, one has to wait for a particular simulation running at a particular C_rate to complete and then put another simulation only after completion. This is time-consuming. My idea is to simulate one model with different parameters at different cores concurrently. At one go, I get my model solved for a range of parameters.
Another idea is to check the possibility if I can run different submodels in different core. For example, I run a model with multiple submodels such as porosity, SEI layer, particle cracking, and thermal model at one go. It is obvious that the running of the model takes more time as compared to the one with fewer number of submodels. There are two possibilities here; first, to my understanding, most of the data computed in the submodel are not dependant on the data from another submodel, and second, in the model, a few parameters are passed from one submodel to another submodel. If we go with the first, the submodels can be assigned cores based on the system's availability, while for the second case, a mechanism needs to be developed by which we have to ensure that the parameters computed are exchanged between the cores to run the model smoothly.
In our lab, we have Ansys Electronics Desktop running in high-performance computing (HPC) servers, and in that case also, when we simulate any motor design with parameterization, we can see that the same model is being solved at multiple cores for different values of parameters. I replied late as I was in discussion with my colleagues regarding the mechanism.
I started to model in my institute HPC, but due to my PhD dissertation's urgent requirements, I got busy with some other works.

[valentinsulzer]

It is now possible to run simulations with different parameters in parallel, though we need to make an example to demonstrate this (also, it does not currently work with experiments).
Regarding, solving submodels independently, that is much more challenging, since all the submodels are coupled together. Perhaps we could evaluate the different parts of the state vector in parallel, need to think more about this.

[optibikash]

@tinosulzer Its good to hear Tino. As soon as the examples are ready, please share with us so that it can be tested in different workload managers. I have got Slurm workload manager installed in the HPC server. Maybe someone can try running it in PBS pro, XGrid, Spectrum or OpneHPC although many others are available.

[TomTranter]

Here's a little example of how to run an ensemble of input parameters. N.B this is running from our dev branch and is not finished yet and is still a work in progress.

if __name__ == "__main__":
    import pybamm
    import numpy as np
    
    model = pybamm.lithium_ion.SPMe()
    solver = pybamm.ScipySolver()
    param = model.default_parameter_values
    param.update({'Negative particle radius [m]': "[input]"})
    # 1 simulation - 1 input value
    sim = pybamm.Simulation(model=model, solver=solver, parameter_values=param)
    sim.solve(t_eval=np.linspace(0, 3600, 101),
              inputs={'Negative particle radius [m]': 10.0e-6})
    sim.plot()
    
    # list input value
    sim2 = pybamm.Simulation(model=model, solver=solver, parameter_values=param)
    sim2.solve(t_eval=np.linspace(0, 3600, 101),
              inputs=[{'Negative particle radius [m]': 5.0e-6},
                      {'Negative particle radius [m]': 10.0e-6},
                      {'Negative particle radius [m]': 15.0e-6}]
              )
    sim2.plot()

@tlestang will be able to explain more on our plans for this feature

[tlestang]

hi @optibikash . Apologies for the slow answer. I think Tino and Tom already summarised the situation pretty well: in the case where a model is solved for several input values, PyBaMM should be able to automatically disptach the computaion across available cores on the machine. I emphasize here because this feature is still experimental and hasn't really been "battle-tested" yet. Feedback is more than welcone.

As Tino pointed out, solving one model using multiple cores (or cpus for that matter) is something else entirely. Most of the computation actually happens outside of PyBaMM, in DAE or ODE solvers provided by CasADI or scikits.odes, and to my knowledge these are not meant to utilize several cores.

[optibikash]

Thanks, @tlestang for the response. For sure, I will get back with feedback.