[Feature] Trigger expansion from a 3rd party tool

[Rafiot]

I'm one of the core dev of lookyloo and I'd love not to reinvent the wheel when it gets to expanding URLs, and unfurl pretty much does exactly that. So I'd love to have a way to push URLs to a unfurl instance from lookyloo, get the expanded information (preferably in JSON, but I can work with CSV), displaying what makes sense to display on lookyloo (that part needs to be defined), and allowing the lookyloo users to click on a link to see the unfurl representation on an unfurl instance.

As a lookyloo capture often has a few hundred URLs, the submissions should be as automated as possible. For that, we need a way to trigger a submission to an existing unfurl instance, and as far as I can tell, it is not supported yet.

From the top of my mind, we need to following:

• REST API, I'd strongly recommend flask-restx for that (example for Lookyloo)
• Simple python client to communicate with the rest api, something like that

It's a bit of a long shot, but please let me know if that's something you'd be interested in the tool? If yes, I'll be happy to implement the REST API and the python lib/client that talks to it.

[obsidianforensics]

Hi there, this sounds great! Building in a proper API function to Unfurl has been on my list for a while, but there were a few issues (I haven't built one before and I wasn't sure what requirements a consumer of the API would have). If you wanted to build the API part, that'd be awesome and I'd be happy to help however I can. There's an existing /api/ route for the web app, but it's not a true API by any means. I'd welcome ideas/code from someone who actually knows what they're doing.

As for JSON output, getting JSON isn't hard (it's used internally and there's already a generate_json function that builds a JSON object of the nodes and edges - it's in the format need by vis.js Network, as that's what's used to display the Unfurl graph. I'm not sure what structure of JSON would be helpful for your use case, but I could tweak it or build another JSON output function if needed.

[Rafiot]

Great, I'll have a go at it in the coming days, probably next week.

[Rafiot]

I started to play with the app, but I have a quick question before I do so: the public instance expects links that look like that: https://dfir.blog/unfurl/?url=https://www.example.com/path/index.html?a=1&b=2 when the current HEAD on the master branch goes for http://localhost:5000/https://www.example.com/path/index.html?a=1&b=2

The URL to expand in the public instance is a parameter (url), which is not the case on the current code. Is it expected? Is there a missing commit for the code on github?

(sorry, editing this message as I look at the code) an URL like that:
http://localhost:5000/?url=magnet:?xt=urn:btih:c9e15763f722f23e98a29decdfae341b98d53056&dn=Cosmos+Laundromat&tr=udp%3A%2F%2Fexplodie.org%3A6969&tr=udp%3A%2F%2Ftracker.empire-js.us%3A1337&tr=wss%3A%2F%2Ftracker.btorrent.xyz&ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2F&xs=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fcosmos-laundromat.torrent
will not be processed properly because of the & that are interpreted as URL params by default:

ImmutableMultiDict([('url', 'magnet:?xt=urn:btih:c9e15763f722f23e98a29decdfae341b98d53056'), ('dn', 'Cosmos Laundromat'), ('tr', 'udp://explodie.org:6969'), ('tr', 'udp://tracker.empire-js.us:1337'), ('tr', 'wss://tracker.btorrent.xyz'), ('ws', 'https://webtorrent.io/torrents/'), ('xs', 'https://webtorrent.io/torrents/cosmos-laundromat.torrent')])

The reason it works right now is because you use the referrer to extract the URL to expand. I understand it makes things easier to copy/paste a URL, but would you consider requiring an url encoded parameter instead?

[Rafiot]

okay, so here is a small PR, I figured out a way to keep the current approach to submit a URL without requiring quoting it first. Please let me know what you think and if you're ok with that, I'll implement the non-visjs API part.

[obsidianforensics]

My initial idea for Unfurl was for the user to be able to be on a site, then insert unfurl.link/ in the address bar before the current page URL, then get the Unfurled result. This wasn't as easy as I had hoped and I got it (mostly) working on the public site via redirection shenanigans. It does work on the local app (substituting localhost:port for the domain of course).

For the public instance, it's powered by GCP cloud functions, so I actually base64 encode the url parameter before sending, then decode it in the cloud function. None of this is ideal, and I'll like redo a lot of it using the new API, as that seems a much more sane way.

I hope this at least explains why it is like it is. I'm happy to make changes to support the new API and make things more streamlined.

[obsidianforensics]

I think the PR looks good 👍

[Rafiot]

Great, thanks for the feedback, I'll keep working on it in this direction