-
Notifications
You must be signed in to change notification settings - Fork 19.6k
Fix: added backend validation for dataset adapters across backends #21789
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
97285b4
8cd14f2
0d0a224
f842a5f
17fa695
69652de
70c8577
2a5f2d4
f374ea2
e19485d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,15 +17,28 @@ def __init__(self, dataset, class_weight=None, distribution=None): | |
| shard the input dataset into per worker/process dataset | ||
| instance. | ||
| """ | ||
| import keras | ||
| from keras.src.utils.module_utils import tensorflow as tf | ||
|
|
||
| # --- ✅ Backend compatibility check --- | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| backend = keras.backend.backend() | ||
| if backend not in ("tensorflow", "numpy", "torch", "jax"): | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a no-op, all the backends are in the list of supported backends. |
||
| raise ValueError( | ||
| f"Incompatible backend '{backend}' for TFDatasetAdapter. " | ||
| "This adapter only supports the TensorFlow , numpy , torch ," | ||
| " jax backend." | ||
| ) | ||
|
|
||
| # --- ✅ Dataset type validation --- | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| if not isinstance( | ||
| dataset, (tf.data.Dataset, tf.distribute.DistributedDataset) | ||
| ): | ||
| raise ValueError( | ||
| "Expected argument `dataset` to be a tf.data.Dataset. " | ||
| "Expected argument `dataset` to be a tf.data.Dataset or " | ||
| "tf.distribute.DistributedDataset. " | ||
| f"Received: {dataset}" | ||
| ) | ||
|
|
||
| if class_weight is not None: | ||
| dataset = dataset.map( | ||
| make_class_weight_map_fn(class_weight) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,8 +11,19 @@ class TorchDataLoaderAdapter(DataAdapter): | |
| """Adapter that handles `torch.utils.data.DataLoader`.""" | ||
|
|
||
| def __init__(self, dataloader): | ||
| # --- ✅ Backend compatibility check --- | ||
| import torch | ||
|
|
||
| import keras | ||
|
|
||
| backend = keras.backend.backend() | ||
| if backend not in ("torch", "tensorflow", "numpy", "jax"): | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a no-op, all the backends are in the list of supported backends. |
||
| raise ValueError( | ||
| f"Incompatible backend '{backend}' for TorchDataLoaderAdapter. " | ||
| "This adapter only supports the PyTorch, tensorflow, jax, numpy" | ||
| " backend. " | ||
| ) | ||
|
|
||
| if not isinstance(dataloader, torch.utils.data.DataLoader): | ||
| raise ValueError( | ||
| f"Expected argument `dataloader` to be an instance of" | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a no-op, all the backends are in the list of supported backends.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My code was raising an error if the dataset and the backend weren't from the same framework, ex: tensorflow dataset and tensorflow backend. But the code wasn't passing the tests, error basically said that tf dataset is ok to be used with any backend even though we get an error if the backend is not the same. If it's open I will would like to work on this issue.