Intermediate Search: Implement a crawler that spans mastodon instances.

[berkes]

Expand the crawler that can crawl a list of mastodon instances and extracts public profiles with #for-hire tags.

Details

For the intermediate product "for hire search", we need to extend the ScraPy spider to crawl across multiple mastodon instances. Currently it only crawls one instance in a Proof of concept. TODO: release this scraper proof of concept in a flockingbird repo.

"Intermediate search" is explained in #37.

Deliverable

• Given a list (in JSON, text or STDIN), we crawl each instance on that list for public profiles
• As in the Proof of concept scraper, we only index public data.
• As in the Proof of concept scraper, we adhere to noindex, robots.txt, etc.
• It returns a JSON document either per instance, or of all instances, structural similar to the proof of concept scraper.