Create scraping.md #395
Open
Create scraping.md #395
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Scraping guidance
hancush
requested changes
Jan 9, 2025
etl/scraping.md
Outdated
| ## Libraries | ||
| DataMade prefers that web scrapers use the [`scrapy` framework](https://scrapy.org/). Here's what we appreciate about scrapy: | ||
|
|
||
| 1. Fast. `scrapy` wants to parallel, and so can pull a lot of data very quickly. |
| 1. Fast. `scrapy` wants to parallel, and so can pull a lot of data very quickly. | ||
| 2. Opinionated. `scrapy` scrapers expect files to be organized in particular ways. This is good for reviewing PRs. | ||
| 3. Popular. `scrapy` is the most popular scraping framework, so you can find lots of QAs and extensions on the internet. | ||
| 4. Extensible. If you need a scrapy that can run some javascript, you can stay within the `scrapy` framework and use middleware like [`scrapy-playwright`](https://github.com/scrapy-plugins/scrapy-playwright). If you need IP rotation or more advanced anti-bot circumventions, there is a good migration path from a normal scrapy script to [Zyte](https://scrapy-zyte-api.readthedocs.io/en/latest/). |
There was a problem hiding this comment.
Nice, thanks for including this resource!
| 2. Tolerant error handling. With most of the serial, `requests`-based scrapers DataMade has written, if there is an exception, everything comes to a grinding halt and the process exits with a non-zero exit code. This is sometimes annoying, but makes it quite clear that something has gone wrong and it is easy to incorporate into longer data pipelines that should continue or not based upon exit codes (Makefiles, github actions are two examples). `scrapy` has another different philosophy, and every request you make could fail and the process would still exit successfully. If you only want to scrape without errors, then you need to affirmatively change a setting to exit on the first error, and [you need to do something like this](https://github.com/scrapy/scrapy/issues/1231#issuecomment-102409470) to change the exit code if there is an error. | ||
| 3. Cache eviction. If you used `scrapy`'s caching mechanism, and you want to remove a small number of cached responses, it's pretty tricky to do the surgery to find the files and remove them. | ||
|
|
||
| We can put together a cookie cutter to help address some of those downsides. |
Co-authored-by: hannah cushman garland <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scraping guidance
This records the scraping guidance as discussed in #393