Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: how xorbits datastes export to json file #761

Open
simplew2011 opened this issue Dec 13, 2023 · 5 comments
Open

FEAT: how xorbits datastes export to json file #761

simplew2011 opened this issue Dec 13, 2023 · 5 comments
Labels
Milestone

Comments

@simplew2011
Copy link

Is your feature request related to a problem? Please describe

  • how xorbits datastes export to json file
  • in current, only support to arrow format file

Describe the solution you'd like

  • Add an interface for converting to the huggingface dataset
  • Or add an interface for export xorbits dataset to json format
@XprobeBot XprobeBot added this to the v0.7.2 milestone Dec 13, 2023
@simplew2011
Copy link
Author

需要实现如下接口
xorbits.datasets.to_huggingface
xorbits.datasets.Dataset.from_dataframe
xorbits.datasets.export_json

@simplew2011
Copy link
Author

需要向dataset.Dataset中新增一列用于记录中间值,如何处理,只看到__getitem__,没有实现__setitem__

@simplew2011
Copy link
Author

dataset.Dataset如何进行过滤,

类似于huggingface.dataset:https://github.com/huggingface/datasets/blob/ef0f986518bd252c5314a7e3a419dedcbb166630/src/datasets/arrow_dataset.py#L5061

@qinxuye
Copy link
Contributor

qinxuye commented Dec 15, 2023

@codingl2k1 看下这个问题。

@simplew2011 你有兴趣来贡献吗?

@codingl2k1
Copy link
Contributor

dataset.Dataset如何进行过滤,

类似于huggingface.dataset:https://github.com/huggingface/datasets/blob/ef0f986518bd252c5314a7e3a419dedcbb166630/src/datasets/arrow_dataset.py#L5061

Currently, xorbits dataframe can export the dataframe to csv, parquet, sql, and dataframe apply may be able to meet your needs. xorbits dataset can map data and convert the dataset to dataframe, but the filter is not implemented.

Could you provide some example code?

@XprobeBot XprobeBot modified the milestones: v0.7.2, v0.7.3 Jan 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants