Skip to content

Commit

Permalink
fix: _resolve_features check for future incompatabilities
Browse files Browse the repository at this point in the history
Signed-off-by: Mehant Kammakomati <[email protected]>
  • Loading branch information
kmehant committed Dec 5, 2024
1 parent 032d1a6 commit 0944056
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion tuning/utils/preprocessing_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,12 @@ def is_pretokenized_dataset(data: Union[str, Dataset, IterableDataset]):
raise DatasetGenerationError("failed to load the provided dataset") from e
if "column_names" not in data or data.column_names is None:
if isinstance(data, IterableDataset):
data = data._resolve_features()
if hasattr(data, "_resolve_features"):
data = data._resolve_features()
else:
raise ValueError(
"_resolve_features API is not available to fetch column names"
)
else:
raise ValueError(
f"not possible to fetch column names for the loaded dataset of type {type(data)}"
Expand Down

0 comments on commit 0944056

Please sign in to comment.