Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Add documentation for data preprocessor release #423

Merged
merged 9 commits into from
Dec 23, 2024
Merged

docs: Add documentation for data preprocessor release #423

merged 9 commits into from
Dec 23, 2024

Conversation

dushyantbehl
Copy link
Contributor

Description of the change

Related issue number

How to verify the PR

Was the PR tested

  • I have added >=1 unit test(s) for every new method I have added.
  • I have ensured all unit tests pass

@github-actions GitHub Actions
Copy link

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

@willmj willmj changed the title [WIP] docs: Add documentation for data preprocessor release docs: [WIP] Add documentation for data preprocessor release Dec 19, 2024
@github-actions github-actions bot added the docs label Dec 19, 2024
Abhishek-TAMU
Abhishek-TAMU reviewed Dec 20, 2024
View reviewed changes
Copy link
Collaborator

@Abhishek-TAMU Abhishek-TAMU left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Dushyant for the PR. Just reviewed the file README.md for the first pass. The changes look good to me, with just small suggestions. These are optional inputs from my side. Feel free to cherry pick @dushyantbehl.

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
Abhishek-TAMU
Abhishek-TAMU reviewed Dec 20, 2024
View reviewed changes
Copy link
Collaborator

@Abhishek-TAMU Abhishek-TAMU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @willmj for adding the documentation content. Few optional suggestions in docs/advanced-data-preprocessing.md from my side. @dushyantbehl Feel free to modify my suggestions too.

docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
@dushyantbehl dushyantbehl changed the title docs: [WIP] Add documentation for data preprocessor release docs: Add documentation for data preprocessor release Dec 23, 2024
@dushyantbehl dushyantbehl marked this pull request as ready for review December 23, 2024 09:32
dushyantbehl
dushyantbehl commented Dec 23, 2024
View reviewed changes
docs/advanced-data-preprocessing.md Outdated

User can pass builder in `DataSetConfig` to mention the specific loader for the passed `file/folder/pattern`.
We support the following,
- Requested @Abhishek-TAMU to add the supported configuration here.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Abhishek-TAMU can you add the details here

ashokponkumar
ashokponkumar reviewed Dec 23, 2024
View reviewed changes
docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
@dushyantbehl
Update docs
d118ab6 
Signed-off-by: Dushyant Behl <[email protected]>
@dushyantbehl @ashokponkumar
Apply suggestions from code review
6a0c060 
Co-authored-by: Ashok Pon Kumar <[email protected]>
Signed-off-by: Dushyant Behl <[email protected]>
ashokponkumar
ashokponkumar reviewed Dec 23, 2024
View reviewed changes
Copy link
Collaborator

@ashokponkumar ashokponkumar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we support txt format and packing for pre-training?

docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
ashokponkumar
ashokponkumar reviewed Dec 23, 2024
View reviewed changes
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
docs/advanced-data-preprocessing.md Outdated Show resolved Hide resolved
@dushyantbehl
Copy link
Contributor Author

Do we support txt format and packing for pre-training?

txt format is supported but not tested so not called out

@Abhishek-TAMU Abhishek-TAMU mentioned this pull request Dec 23, 2024
Merged
2 tasks
ashokponkumar
ashokponkumar previously approved these changes Dec 23, 2024
View reviewed changes
@ashokponkumar ashokponkumar merged commit 594dd37 into foundation-model-stack:main Dec 23, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Abhishek-TAMU Abhishek-TAMU Abhishek-TAMU left review comments

@ashokponkumar ashokponkumar ashokponkumar approved these changes

@anhuong anhuong Awaiting requested review from anhuong anhuong is a code owner

@Ssukriti Ssukriti Awaiting requested review from Ssukriti Ssukriti is a code owner

@aluu317 aluu317 Awaiting requested review from aluu317 aluu317 is a code owner

@fabianlim fabianlim Awaiting requested review from fabianlim fabianlim is a code owner

@kmehant kmehant Awaiting requested review from kmehant kmehant is a code owner

Assignees
No one assigned
Labels
docs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dushyantbehl @ashokponkumar @Abhishek-TAMU @willmj