Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generating scripts/modules to scrape images/videos #6

Open
mchiquier opened this issue May 26, 2020 · 2 comments
Open

Generating scripts/modules to scrape images/videos #6

mchiquier opened this issue May 26, 2020 · 2 comments
Labels
enhancement New feature or request

Comments

@mchiquier
Copy link
Contributor

Ideally this is modular but it would be good to have a boilerplate for scraping data from:

-twitter
-youtube
-tiktok
-instagram

Or even just google drive & pipe-lining it into a PyTorch dataloader.
@mchiquier mchiquier changed the title scripts/modules to scrape images/videos Generating scripts/modules to scrape images/videos May 26, 2020
@goodmattg
Copy link
Collaborator

This is extremely difficult, and arguably it's own project. I consider this a "nice to have", but it should be the last thing we do.

@goodmattg goodmattg added the enhancement New feature or request label May 26, 2020
@ChickenTarm
Copy link
Collaborator

scraping data also leads to problematic territory like terms of service and them just straight up blocking your ass for taking their data, etc... And not all sites have nice apis for scraping, so some general scraper would be hard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@goodmattg @ChickenTarm @mchiquier