CP2K jobs slower with higher number of cores per worker

[b-mazur]

I'm trying to reproduce mof_phase_transition.py example and I'm facing issue where with increasing number of cores per worker my calculations gets prohibitively slow. In all cases max_walltime: 20 results in AssertionError: atomic energy calculation of O failed because none of the CP2K tasks for oxygen are completed in 20 minutes.

I played a bit with different number of cores per worker and here are values of SCF steps reached in 20 minutes for oxygen task with multiplicity 5:

cores per worker	SCF steps
1	33
2	31
4	20
16	3

Finally I was able to finish this part by increasing max_walltime to 180 minutes and using only 1 core per worker but this will create another issue when ReferenceEvaluation is used for whole MOF in next steps.

I've never used CP2K but I feel that 180 minutes is far too long for single point of single atom. What else I observe is the surprisingly low CPU performance of slurm tasks, at levels of <10%. I checked timings in CP2K output but MPI timing doesn't seem to be such large (however, as I said, I have no experience so maybe I don't understand something). Here is an example:

 -------------------------------------------------------------------------------
 -                                                                             -
 -                                T I M I N G                                  -
 -                                                                             -
 -------------------------------------------------------------------------------
 SUBROUTINE                       CALLS  ASD         SELF TIME        TOTAL TIME
                                MAXIMUM       AVERAGE  MAXIMUM  AVERAGE  MAXIMUM
 CP2K                                 1  1.0    12.43    12.51 11438.24 11438.32
 qs_forces                            1  2.0     0.00     0.00 11307.17 11307.28
 qs_energies                          1  3.0     0.00     0.00 11272.36 11272.43
 scf_env_do_scf                       1  4.0     0.00     0.00 11209.10 11209.25
 scf_env_do_scf_inner_loop          125  5.0     0.00     0.01 10924.29 10924.59
 qs_scf_new_mos                     125  6.0     0.00     0.00  7315.99  7321.60
 qs_scf_loop_do_ot                  125  7.0     0.00     0.00  7315.99  7321.60
 ot_scf_mini                        125  8.0     0.00     0.00  6917.97  6923.00
 dbcsr_multiply_generic            3738 10.8     0.14     0.15  5700.36  5708.79
 ot_mini                            125  9.0     0.00     0.00  3021.17  3021.32
 mp_sum_l                         18209 11.7  2898.88  2915.89  2898.88  2915.89
 qs_ot_get_p                        256  9.0     0.00     0.00  2869.85  2871.50
 qs_ot_get_derivative               126 10.0     0.00     0.00  2737.75  2738.66
 rs_pw_transfer                    1905 10.0     0.02     0.02  2014.01  2029.50
 qs_ks_update_qs_env                128  6.0     0.00     0.00  1921.34  1922.16
 qs_ot_p2m_diag                     145 10.0     0.00     0.00  1907.22  1918.97
 rebuild_ks_matrix                  126  7.0     0.00     0.00  1867.10  1869.47
 qs_ks_build_kohn_sham_matrix       126  8.0     0.01     0.01  1867.10  1869.47
 qs_rho_update_rho_low              126  6.0     0.00     0.00  1717.53  1723.18
 calculate_rho_elec                 252  7.0     2.66    19.36  1717.53  1723.18
 density_rs2pw                      252  8.0     0.01     0.01  1667.59  1684.20
 cp_dbcsr_syevd                     145 11.0     0.01     0.01  1442.29  1459.78
 pw_transfer                       3737 10.9     0.13     0.17  1190.61  1208.75
 fft_wrap_pw1pw2                   3485 11.9     0.02     0.02  1190.36  1208.51
 fft3d_ps                          3485 13.9    69.25    75.81  1183.31  1202.91
 cp_fm_syevd                        145 12.0     0.00     0.00  1164.55  1177.19
 mp_alltoall_z22v                  3485 15.9  1095.45  1118.30  1095.45  1118.30
 qs_ot_get_derivative_diag           69 11.0     0.00     0.00  1104.83  1105.37
 mp_sum_b                          6644 12.1  1081.35  1095.09  1081.35  1095.09
 mp_waitall_1                    154259 15.1  1026.71  1085.45  1026.71  1085.45
 multiply_cannon                   3738 11.8     0.18     0.21  1069.89  1079.22
 fft_wrap_pw1pw2_500               1965 13.7     1.43     2.06  1032.01  1051.67
 qs_ot_get_derivative_taylor         57 11.0     0.00     0.00   978.10   978.50
 mp_waitany                        4620 12.0   869.01   955.24   869.01   955.24
 qs_vxc_create                      126  9.0     0.00     0.00   879.87   886.00
 qs_ot_get_orbitals                 250  9.0     0.00     0.00   799.42   800.84
 sum_up_and_integrate                64  9.0     0.08     0.08   790.31   799.75
 integrate_v_rspace                 128 10.0     0.00     0.00   790.23   799.67
 potential_pw2rs                    128 11.0     0.01     0.01   787.91   789.91
 make_m2s                          7476 11.8     0.07     0.07   704.01   711.12
 make_images                       7476 12.8     0.12     0.13   703.73   710.85
 make_images_sizes                 7476 13.8     0.01     0.01   703.28   710.42
 mp_alltoall_i44                   7476 14.8   703.27   710.41   703.27   710.41
 rs_pw_transfer_RS2PW_500           254 10.0     0.57     0.64   676.24   691.60
 xc_pw_derive                      1140 12.0     0.01     0.01   654.68   664.73
 mp_sendrecv_dv                    7056 11.0   659.52   660.69   659.52   660.69
 xc_rho_set_and_dset_create         126 11.0     2.11    10.54   491.93   604.77
 cp_fm_redistribute_start           145 13.0   443.54   480.41   587.12   600.87
 x_to_yz                           1712 15.9     1.63     1.75   590.30   599.08
 cp_fm_redistribute_end             145 13.0   417.14   571.08   430.35   590.60
 xc_vxc_pw_create                    64 10.0     1.70     8.42   581.92   584.35
 mp_sum_d                          3471 10.2   485.51   568.31   485.51   568.31
 multiply_cannon_loop              3738 12.8     0.07     0.08   543.02   555.38
 yz_to_x                           1773 14.1    16.93    19.20   523.71   539.79
 mp_allgather_i34                  3738 12.8   526.54   538.30   526.54   538.30
 multiply_cannon_metrocomm3       14952 13.8     0.03     0.04   487.40   508.68
 rs_pw_transfer_RS2PW_170           252 10.0     0.27     0.31   424.39   428.13
 calculate_dm_sparse                252  8.0     0.00     0.00   401.52   402.48
 rs_pw_transfer_PW2RS_500           131 12.9     0.27     0.29   336.32   337.31
 qs_ot_p2m_taylor                   111  9.9     0.00     0.00   320.88   328.70
 xc_pw_divergence                   128 11.0     0.00     0.00   315.44   324.80
 dbcsr_complete_redistribute        394 12.1     0.01     0.02   299.29   314.94
 copy_dbcsr_to_fm                   198 11.1     0.00     0.00   294.59   303.78
 xc_exc_calc                         62 10.0     0.26     0.77   297.95   301.64
 cp_fm_syevd_base                   145 13.0   147.08   300.70   147.08   300.70
 init_scf_loop                        3  5.0     0.00     0.00   281.84   282.02
 ot_new_cg_direction                 63 10.0     0.00     0.00   264.48   265.62
 mp_sum_dv                         2372 13.6   164.02   262.33   164.02   262.33
 arnoldi_normal_ev                  262 10.9     0.00     0.00   244.27   260.42
 arnoldi_extremal                   256 10.0     0.00     0.00   236.76   252.71
 -------------------------------------------------------------------------------

I'm using psiflow 3.0.4 and container oras://ghcr.io/molmod/psiflow:3.0.4_python3.10_cuda.

Any idea what I could check to find where the problem is? Also, wouldn't it be better to tabulate energy for all atoms in psiflow source files? Thanks in advance for any help!

[svandenhaute]

Hi Bartosz,

The energy of the isolated atoms depends on the specific pseudopotential used as well as the functional, so tabulating those would be quite an amount of work. In normal scenarios, these atomic energy calculations usually finish quite quickly (a few seconds -- a few minutes) so it's usually easier to do it on the fly.

This is a bug that I've encountered once on a very specific cluster here in Belgium, and I haven't quite figured out what causes it. Heuristically, I've found that by adding / removing a few MPI flags, CP2K performance will go back to normal again, but I don't quite understand why this is the case given that everything is executed within a container.

What is the host OS, and host container runtime (singularity/apptainer version)? Did you modify the default MPI command in the .yaml?