Oscar/wet dep integration #303

odiazib · 2024-03-18T16:26:36Z

We are using this branch in PR # 2848

List of variables of the routine aero_model_wetdep

Inputs:

Variable	Description	Units	Comments
`cldt`	total cloud fraction	fraction
`cldn`	layer cloud fraction; it is used by modal_aero_wateruptake_dr cldn	fraction
`rprdsh`	rain production, shallow convection	kg/kg/s	`pbuf_get_index('RPRDSH')`
`rprddp`	rain production, deep convection	kg/kg/s
`evapcdp`	Evaporation rate of deep convective precipitation	kg/kg/s
`evapcsh`	Evaporation rate of shallow convective precipitation	kg/kg/s
`dp_frac`	Deep convective cloud fraction	fraction
`sh_frac`	Shallow convective cloud fraction	fraction
`icwmrdp`	in cloud water mixing ratio, deep convection	kg/kg
`icwmrsh`	in cloud water mixing ratio, shallow convection	kg/kg
`evapr`	evaporation from stratiform rain	kg/kg/s
`dlf`	Detrainment of convective condensate or shallow+deep convective detrainment	kg/kg/s
`prain`	stratiform rain production rate	kg/kg/s

In/Out:

Variable	Description	Units
`dgnum`	wet aerosol wet diameter	m
`dgncur_a`	aerosol particle diameter	m
`qaerwat`	aerosol water	kg/kg
`wetdens`	wet aerosol density	kg/m3
`ptend_lq`
`qqcw`

Output:

Variable	Description	Units	Comments
`aerdepwetis`	aerosol wet deposition (interstitial)	kg/m2/s
`aerdepwetcw`	aerosol wet deposition (cloud water)	kg/m2/s

@singhbalwinder,

In the tables above, could you please add comments indicating which process in EMAXX is responsible for computing the specific variables? This information can be added in the 'Comments' column.

Issue:

This code that produces this race condition is not needed by the eamxx-mam4xx interface. I commented out this code.

========= COMPUTE-SANITIZER
/home/odiazib/test_mam4/build/mam4xx/src/validation/aero_model/aero_model_driver: reading baseline_aero_model_wetdep_ts_379.yaml
aero_model_wetdep : 
========= Error: Race reported between Write access at 0x59a70 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>()
=========     and Read access at 0x58490 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [160 hazards]
=========     and Read access at 0x58500 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [336 hazards]
=========     and Read access at 0x58580 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [168 hazards]
=========     and Read access at 0x59330 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [152 hazards]
=========     and Read access at 0x593a0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [304 hazards]
=========     and Read access at 0x59420 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [152 hazards]
========= 
========= Error: Race reported between Write access at 0x30fc0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>()
=========     and Read access at 0x308d0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [8 hazards]
=========     and Read access at 0x30940 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [16 hazards]
=========     and Read access at 0x309c0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [8 hazards]
========= 
========= Error: Race reported between Write access at 0x31bd0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>()
=========     and Read access at 0x314e0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [8 hazards]
=========     and Read access at 0x31550 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [16 hazards]
=========     and Read access at 0x315d0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [8 hazards]
========= 
========= Error: Race reported between Write access at 0x303a0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>()
=========     and Read access at 0x2fd30 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [16 hazards]
=========     and Read access at 0x2fdb0 in void Kokkos::Impl::cuda_parallel_launch_constant_memory<Kokkos::Impl::ParallelFor<aero_model_wetdep(skywalker::Ensemble *)::[lambda(const skywalker::Input &, skywalker::Output &) (instance 1)]::operator ()(const skywalker::Input &, skywalker::Output &) const::[lambda(const Kokkos::Impl::CudaTeamMember &) (instance 1)], Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>>() [8 hazards]
========= 
/home/odiazib/test_mam4/build/mam4xx/src/validation/aero_model/aero_model_driver: writing mam4xx_baseline_aero_model_wetdep_ts_379.py
========= RACECHECK SUMMARY: 4 hazards displayed (4 errors, 0 warnings)

codecov · 2024-03-18T16:49:34Z

Codecov Report

Attention: Patch coverage is 99.15730% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 96.41%. Comparing base (7b0be56) to head (ce96755).

Files	Patch %	Lines
src/mam4xx/utils.hpp	93.54%	2 Missing ⚠️
src/mam4xx/wet_dep.hpp	99.68%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #303      +/-   ##
==========================================
+ Coverage   96.27%   96.41%   +0.14%     
==========================================
  Files          41       41              
  Lines        8365     8593     +228     
==========================================
+ Hits         8053     8285     +232     
+ Misses        312      308       -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ut calcsize.

… aero_model_wetdep.

…repancy in GPUs.

…nterface.

src/mam4xx/aero_model.hpp

overfelt · 2024-05-28T18:05:28Z

src/mam4xx/modal_aer_opt.hpp

@@ -26,7 +26,7 @@ constexpr int ntot_amode = mam4::AeroConfig::num_modes();
 // FIXME:  is top_lev equal to 1 in aerosol optics ?
 constexpr int top_lev = 0;
 //
-constexpr int pcnst = aero_model::pcnst;
+// constexpr int pcnst = mam4::pcnst;


Can probably delete this commented out line of code.

src/validation/wetdep/wetdepa_v2.cpp

singhbalwinder · 2024-05-28T18:58:05Z

src/validation/aero_model/aero_model_calcsize_water_uptake_dr.cpp

+                    acc_spec_in_ait,
+                    // outputs
+                    dgncur_i.data(), dgncur_c, ptend_q_k.data(),
+                    dqqcwdt_k.data());


Sometimes data layout is not very obvious when we use .data() in the output variables. We don't need to change if it is working but just a note about revisiting these in the source code when time allows.

@singhbalwinder , I agree we should re-visit the use of data() sometime. Using data() allows the function being called to use a simple Real* parameter. But it means that the Kokkos::view has to have a stride of 1. This is true for 1D views but 1D sub-views of 2D views might not have stride 1 which can cause hard to diagnose errors.

I agree with both of you. I have created issue #322.

…ce its passing errors.

odiazib force-pushed the oscar/wet_dep_integration branch 2 times, most recently from 0424c78 to d4a3009 Compare March 26, 2024 19:25

odiazib force-pushed the oscar/wet_dep_integration branch 2 times, most recently from 86c7a90 to 5d5119b Compare April 9, 2024 17:10

odiazib force-pushed the oscar/wet_dep_integration branch from b346ed2 to ef8870b Compare April 30, 2024 02:41

odiazib added 24 commits May 20, 2024 07:33

wet_dep - remove prog from compute_q_tendencies_phase_2

a4a5b3a

wet_dep - remove prog from inputs

40f5003

wet_dep - get state_q from prog.

176a9ac

wet_dep - using parallel_reduce

b35df65

wet_dep - replace KOKKOS_LAMBDA with [&]

25b5bc1

wet-dep - clang format

e7d9848

wet_dep - move pcnst to aero_config.hpp; interface for wet_scav witho…

bd90094

…ut calcsize.

wet_dep - interface for wet_scav

99bc70a

wet_dep - adding work len and check for work len.

da508d6

wet_dep - validation test for aero_model_wetdep.

4a51d29

wet_sav - a temporal test for aero_model_wetdep

1b264b8

wet_scav - validation tests runs.

e6baee6

wet_scav - clang format

b0a1caf

wet_dep - adding a few values to avoid FPE.

ca32603

wet_dep - clang format

a7ccf40

wet_dep - updating calcsize and modal_aero_water_uptake_dr calls.

d58ccd2

wet_dep - clang format

8d3d810

wet_dep - variables that part of pbuf need to be inputs.

ce72060

wet_dep - aerdepwetcw and aerdepwetis are 1d views and are outputs of…

66b38d7

… aero_model_wetdep.

wet_dep - modal_aero_water_uptake_dr relies on cldn_prev_step.

cb3e436

wet_dep - cldt is an input.

c3b5670

wet_dep - remove inputs.

e3d98d3

wet_dep - cldst is not an input.

1a3f3a4

wet_dep - delete old comments.

74dcf03

odiazib added 2 commits May 20, 2024 07:33

wet_scav - updating comments.

9080be4

wet_sav - update tol for aero_model_wetdep_ts_379 validation tests.

eff10eb

odiazib force-pushed the oscar/wet_dep_integration branch from 8a8ae57 to eff10eb Compare May 20, 2024 13:35

odiazib added 14 commits May 20, 2024 07:41

wet_scav - adding a baseline tests.

e18a32b

wet_sav - clang-format

41b3b74

wet_sav - reducing tol and updating mam_x_validation

ce5b245

wet_scav - Using View1D instead of Kokkos::View<Real *>.

b74ea18

wet_sav - remove qqcw_sav

287d8ee

wet_sav - remove std::cout.

febcc5c

wet_scav - using const View2D & instead of Kokkos::View<Real **>.

efa78f2

wet_sav - using view2d.

7f94325

wet_sav - clang format

b09c72b

wet_scav - upgrading mam_x_validation

064dc5b

wet_scav - Replacing 'for' with 'Kokkos::parallel_for' fixes the disc…

8e325b3

…repancy in GPUs.

wet_scav - Adding one parallel_for before for loop.

3bb37b0

wet_scav - clang-format

299c698

wet_scav - Commenting out code that is not used by the emaxx-mam4xx i…

666a291

…nterface.

odiazib requested review from overfelt and singhbalwinder May 28, 2024 15:32

overfelt reviewed May 28, 2024

View reviewed changes

src/mam4xx/aero_model.hpp Outdated Show resolved Hide resolved

overfelt reviewed May 28, 2024

View reviewed changes

overfelt approved these changes May 28, 2024

View reviewed changes

singhbalwinder approved these changes May 28, 2024

View reviewed changes

odiazib added 5 commits May 30, 2024 16:15

wet_scav - using static_cast.

22f1c81

wet_scav - Removing old comment.

8364d06

wet_scav - Removing commented code.

8f9e26a

wet_scav- Deleting old comment. This is a new test, so I did not redu…

25d6bbf

…ce its passing errors.

wet_scav - Updating mam_x_validation

ce96755

odiazib merged commit b1c7d98 into main Jun 3, 2024
8 checks passed

odiazib deleted the oscar/wet_dep_integration branch June 3, 2024 17:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Oscar/wet dep integration #303

Oscar/wet dep integration #303

odiazib commented Mar 18, 2024 •

edited

Loading

codecov bot commented Mar 18, 2024 •

edited

Loading

overfelt May 28, 2024

singhbalwinder May 28, 2024

overfelt May 28, 2024

odiazib May 30, 2024

Oscar/wet dep integration #303

Oscar/wet dep integration #303

Conversation

odiazib commented Mar 18, 2024 • edited Loading

Inputs:

In/Out:

Output:

codecov bot commented Mar 18, 2024 • edited Loading

Codecov Report

overfelt May 28, 2024

Choose a reason for hiding this comment

singhbalwinder May 28, 2024

Choose a reason for hiding this comment

overfelt May 28, 2024

Choose a reason for hiding this comment

odiazib May 30, 2024

Choose a reason for hiding this comment

odiazib commented Mar 18, 2024 •

edited

Loading

codecov bot commented Mar 18, 2024 •

edited

Loading