Enable GPU execution of mpas_reconstruct_2d via OpenACC

[gdicker1]

This PR slightly modifies and adds OpenACC directives to mpas_reconstruct_2d so it can execute on GPU(s).

Timing for the OpenACC data transfers in this routine is captured in the log file by a new timer: mpas_reconstruct_2d [ACC_data_xfer].

NOTE two things about this PR:

• The use of the sin and cos functions on the GPU causes answer differences from previous GPU results. You can merge the commit in gdicker1/MPAS-Model:framework/acc_mpas_reconstruct_2d-correctness-sincosloops to run these loops on the CPU and verify this.
• The time-invariant fields in this routine (coeffs_reconstruct, nEdgesOnCell, edgesOnCell, latCell, and lonCell) are handled in both mpas_reconstruct_2d and in mpas_atm_dynamics_{init,finalize}. Like Enable GPU execution of MPAS-Atmosphere initialization of coupled diagnostic fields via OpenACC #1216, this routine is called before mpas_atm_dynamics_init.

[gdicker1]

I used the compare_netcdf.py script and looked at the differences between log.atmosphere.0000.out files to characterize the differences. I used the 6 timestep regional testcase.

Running the loop near mpas_vector_reconstruct.F L273-283 in unmodified code on the CPU had no answer differences to a GPU run of the commit I started this PR-branch from.

When running this code on the GPU I observed differences in the u, w, and scalars [1-8] values reported in the log file. Unlike some other answer differences, the locations of mins and maxes did not change for u and w. Select output from compare_netcdf.py baseline_acc/restart.2019-09-01_00.06.00.nc port_att/restart.2019-09-01_00.06.00.nc ¹:

            Variable  Min       Max      
=========================================                                                
# ... omitting lines ...
                   u -0.041306  0.041719
                   w -0.040645  0.030554
              rho_zz -0.000086  0.000116                                                                                                                                          
             theta_m -0.054626  0.081970                                                 
          pressure_p -16.194092  11.606445
# ... omitting lines ...

Footnotes

1. Full output saved on Derecho within "/glade/work/gdicker/mpas-work/2025Feb06_PortMPASReconstruct/port_att1739582865/diff.baselineVtest.restart.txt" ↩

[gdicker1]

More on the answer differences, this does seem to be due to how the default CPU and GPU math implementations differ. I get no answer differences if I add the flags described in #1287 to my baseline_acc and PR branch builds (namely -gpu=math_uniform; -Mnofma is already in GPU builds).