Skip to content

Ensure vs30 values are written to disk and verified before exit #540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AndrewRidden-Harper wants to merge 1 commit into from
Closed

Ensure vs30 values are written to disk and verified before exit #540

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions workflow/calculation/hf_sim.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,9 @@ def run_hf(
out.seek(HEAD_STAT - 2 * FLOAT_SIZE, 1)
e_dist[i].tofile(out)
vs.tofile(out)
# Ensure vs30 values are fully written to disk before MPI terminates the process
Copy link
Member

@sungeunbae sungeunbae Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vs30 values?

Copy link
Author

@AndrewRidden-Harper AndrewRidden-Harper Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sungeunbae Sorry, I meant vs values

out.flush()
os.fsync(out.fileno())

# distribute work in a round-robin fashion across ranks for optimisation
# if size=4, rank 0 takes [0,4,8...], rank 1 takes [1,5,9...], rank 2 takes [2,6,10...],
Expand Down Expand Up @@ -553,5 +556,27 @@ def run_hf(
logger.error(msg)
comm.Abort(1)
else:
logger.debug("Simulation completed and size verified.")
print("✅ HF completed successfully", flush=True)
# open r+b and fsync to invalidate any stale NFS/Lustre client cache
with open(args.out_file, "r+b") as hff:
os.fsync(hff.fileno())
with open(args.out_file, "rb") as hff:
hff.seek(HEAD_SIZE)
vs_values = np.fromfile(
hff,
count=stations.size,
dtype={
"names": ["vs"],
"formats": ["f4"],
"offsets": [HEAD_STAT - FLOAT_SIZE],
"itemsize": HEAD_STAT,
},
)["vs"]
Comment on lines +560 to +573
Copy link

@gemini-code-assist gemini-code-assist bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For efficiency, you can combine these two with open(...) blocks. Opening the file in r+b mode allows for both calling os.fsync() and then reading from the file without needing to close and reopen it.

            with open(args.out_file, "r+b") as hff:
                os.fsync(hff.fileno())
                hff.seek(HEAD_SIZE)
                vs_values = np.fromfile(
                    hff,
                    count=stations.size,
                    dtype={
                        "names": ["vs"],
                        "formats": ["f4"],
                        "offsets": [HEAD_STAT - FLOAT_SIZE],
                        "itemsize": HEAD_STAT,
                    },
                )["vs"]

zero_vs = np.where(vs_values == 0)[0]
if zero_vs.size > 0:
msg = f"CRITICAL: {zero_vs.size} station(s) have vs=0 in header (indices: {zero_vs.tolist()})"
print(msg, flush=True)
logger.error(msg)
comm.Abort(1)
else:
logger.debug("Simulation completed, size and vs headers verified.")
print("✅ HF completed successfully", flush=True)