Multiple output points for ww3_outp when ITYPE==4

[dkokron]

I am testing ways to improve how the ww3_outp application interacts with the file system. One idea is to have each run of ww3_outp output more than one point. This could save a bunch of file system activity improving both runtime and runtime variability. I attempted to do this, but I was met with the following error message.
*** WAVEWATCH III ERROR IN W3OUTP : MULTIPLE OUTPUT POINTS DEFINED, ITYPE = 4 ONLY SINGLE POINT ALLOWED IN THIS VERSION

There is also this comment in the code.
! ... This version only allows single point output for bulletins

Is there a fundamental limitation that prevents multiple output points when ITYPE==4?
Is there a version of ww3_outp that supports multiple output points?

[JessicaMeixner-NOAA]

Pinging @AliS-Noaa

[JessicaMeixner-NOAA]

Pasting some comments from email:

• To help enable improving the point post processing, we converted the binary output to NetCDF. I completed this work with others in the Coupled Group and EIB here at EMC.
• The next step is to instead of parallelizing by scripting, we're looking at how to utilize actual MPI within the fortran code itself. One of the (many) reasons that this job is performing poorly is that we execute the point output for each hour instead of reading in all hours, and this is what Ali is currently working on right now. That alone should really help improve the file system impacts, but hopefully we can go a few steps further for additional improvements as well.

[dkokron]

@AliS-Noaa. What is the limitation that prevents serially looping over points (when itype==4) within a single run of ww3_outp?

[AliS-Noaa]

@dkokron So the issue comes form that fact W3IOPON and W3IOPO from w3iopomd.F90 in the 'READ' mode are not designed to read several inputs based on the TIMETAG (similar to 'WRITE' mode). I am working on the W3IOPON_READ (

WW3/model/src/w3iopomd.F90

Line 1151 in b4e119e

SUBROUTINE W3IOPON_READ(IOTST, IMOD_IN, filename, ncerr)

) to be able to read each netcdf files (for each timesteps). Similar to W3IOPON_WRITE (

WW3/model/src/w3iopomd.F90

Line 1426 in b4e119e

SUBROUTINE W3IOPON_WRITE(timestep_only,filename, ncerr)

) when OFILES(2)=1.

[JessicaMeixner-NOAA]

There might be a fundamental limitation in the sense that it then groups the output all together and not one output per file, but I'd have to check my memory on that.

[dkokron]

Having all outputs in a single file is fine for testing purposes. I'm just trying to test an idea for reducing file system activity.

[JessicaMeixner-NOAA]

Having all outputs in a single file is great for testing but since the product requires individual files for each point, we will have to come up with another solution for operations.

[dkokron]

I'm happy to test drive whatever solution you come up with.

[dkokron]

As a test of my idea, is it reasonable to simply comment these lines in WW3/model/ftn/ww3_outp.ftn

IF ( COUNT(FLREQ) .GT. 1 ) THEN WRITE (NDSE,1012) OTYPE CALL EXTCDE ( 45 ) ENDIF

[dkokron]

Commenting those lines in WW3/model/ftn/ww3_outp.ftn and putting all 3118 points in a single ww3_outp.inp file allows the ww3_outp executable to generate separate .bull and .cbull files for all 3118 points for a given time window in 10 seconds. The baseline run took 46 minutes to process (serially) five hourly time periods (5*3118 outputs).

I ran my executable on two of the five hourly periods and compared the generated .bull files against their corresponding file from the baseline run to find all are bit identical. Most (2043 of 3118 for each period) of the .cbull files are bit-identical. The ones that are different have what appears to be output from a csv file on the first line. The remaining contents of the cbull file are fine. Oddly, all but one of the .csv files generated by my executable are corrupt.

The total number of file system metadata operations (open, stat, close) to process all points within one instance of ww3_outp is 29,593. The total number of file system metadata operations to process one point using the one-point-at-a-time approach is 88. Thus the total number of file system metadata operations to process all points one at a time would be 88*3118=274,384.

There is clearly an issue causing the csv files and many of the cbull file to be corrupted. However, given the 9.2x reduction in file system load this approach has merit.

[dkokron]

Applying the attached patch onto revision fd9757a of ww3_outp.ftn will allow it to generate all three report types (bull, cbull and csv) for all points specified in ww3_outp.inp

ww3_outp.ftn.patch