[RFC]webscraping shaders that aren't on API

[Vipitis]

This was considered in #25 however I am not sure we should include this.

It's really useful for debugging and testing, so I wanted to share this. Perhaps we can make this part of the optional dev dependencies? Basically looking for comments on this idea.

It works by spawning a browser, waiting for the shader to load and then calling a javascript function that produces almost exactly the same json as the api returns.

Pros:

• 50k additional shaders accessible (public and unlisted, instead of just public+api). API gives access to slightly more than 35k
• no API key required
• less hassle than copy & paste

Cons:

• maybe unethical, shader authors chose not to allow API access - and we just ignore that. (it's always possible to fork a shader and then download it, or make your fork available on api)
• additional dependencies: selenium, some webdriver
• additional setup required: webdriver available
• more maintaince: different browsers, website might change...

Perhaps there is also better implementation for this

[Korijn]

If you're able to get the info with just an HTTP request that could reduce the required dependencies.

You can also try to disable JS and see if the page is rendered differently.

I do think that this is entering a legal grey area, just because you can scrape it does not mean it's allowed.