Skip to content

Conversation

@honzaflash
Copy link

@honzaflash honzaflash commented May 30, 2025

Changes

  • refactor to deduplicate code
    • the JSON file filtering gets its own function
  • simplify how dates are passed around for the kerchunk functions
  • and this way fix the date format mismatch when filtering the JSON files
    • start/end dates are kept as datetime and dates from the filenames get converted to it to compare

The issue before:

  • make_ciofs_kerchunk would receive dates as string formatted like YYYY_0DDD or YYYY-MM-DD
  • then parse the date from the filename like this: datetime.strptime(Path(j).stem, date_format).isoformat() getting YYYY-MM-DDT...
  • for CIOFS this would lead to comparison like this '2022_0213' <= '2022-09-01' which is false
    • even though '2022_0213' == datetime(2022,8,1).strftime('%Y_0%j') and 2022-08-01 is before 2022-09-01
    • because '_' <= '-' is false

The fixed issue after:

  • make_ciofs_kerchunk now receives start/end dates as datetime objects
  • filenames are parsed into datetime objects as well for the comparison
  • if future models require more flexibility in how the date time is represented in the file names the base_str could always be refactored to also accept a datetime object instead of the year 'YYYY' string

Testing

  • the project test suite is passing
  • seeding a CIOFS simulation now succeeds

@honzaflash honzaflash requested a review from kthyng May 30, 2025 01:55
- refactor to deduplicate code
- simplify how dates are passed around for the kerchunk functions
- and this way fix the date format mismatch when filtering the JSON files
@honzaflash
Copy link
Author

I hope I didn't miss some reason why the start/end dates were the way they were.

Also let me know if the changes in whats_new.md are what you were imagining.

@kthyng
Copy link
Member

kthyng commented May 30, 2025

Ok I see what you are doing here. I remember working on this but clearly didn't leave it working correctly!

Can you add a test to demonstrate this working correctly? Be sure to include the edge case of starting a test right at the end of the year that will need simulation output from both that year and the following year (ideally it would work if the simulation was running backward in time too, though I haven't been testing that behavior yet).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants