Implement dataset loading from Google Cloud Storage #191
Conversation
| """ | ||
| ) -> tuple[tf.data.Dataset, tf.data.Dataset, tf.data.Dataset]: | ||
|
|
||
| def pad_or_truncate_data(data, target_length, pad_value=0): |
There was a problem hiding this comment.
Why is this function nested? If it doesn't need to capture local variables then let's move it to module level.
| dataset = dataset.batch(batch_size) | ||
|
|
||
| # Use fixed sizes for splitting the dataset |
There was a problem hiding this comment.
Can we do this split outside of the function so the caller has complete control over how it's split?
|
|
||
|
|
||
| # Function to process inputs and labels for each day | ||
| # Modify process_day to handle no labels case |
There was a problem hiding this comment.
Can you remove these TODO notes before checking this in?
There was a problem hiding this comment.
Also add brief docstrings to these functions.
|
|
||
| # Create the dataset using the data generator | ||
| dataset = tf.data.Dataset.from_generator( | ||
| data_generator, |
There was a problem hiding this comment.
For my understanding, why do we specify an output signature in this dataset but not in the above datasets?
There was a problem hiding this comment.
I have considered that for create_atmo_input_output_sequencee, it generates batches from the data pipeline so it may not explicitly require an output signature. This is because TensorFlow can infer the shapes and types of the dataset elements during the execution pipeline if they follow a consistent structure. When the data is already pre-processed and consistently structured, the framework does not need the explicit signature to determine how to handle the elements. (This was similarly done for flood).
SallyElHajjar
changed the title
| ] | ||
|
|
||
| # Simulate spatial data blob | ||
| mock_spatial_blob = MagicMock() |
There was a problem hiding this comment.
Let's move this into a util function to avoid code duplication. Something like:
def mock_numpy_blob(array: np.ndarray) -> storage.Blob: ...
No description provided.