Handling of URLs trigering a download

[Rafiot]

Right now, if you submit a URL that triggers a download on Lookyloo, you get an error.

The good thing is that playwright supports what we need to handle that (see Lookyloo/PlaywrightCapture#1).

When it is implement on PlaywrightCapture side, we need to handle that in Lookyloo.

• Build the tree as usual (we might have redirects, and all the usual stuff we have in a normal capture)
• Optional: mark in the tree the node that triggers a download (Can we have multiple?)
• In the hostnode popup, suggest to send the downloaded file to pandora
• Optional: if it is submitted to pandora, store the perma link to pandora somewhere in lookyloo so we can get back to it later
• On submit to pandora, open a new page to the analysis

[Rafiot]

It seems there are a few issues with the pdf handling in chromium headless: https://bugs.chromium.org/p/chromium/issues/detail?id=761295
But the problems are only relevant if we care about the bundled-in PDF viewer, and... we don't.

Now, let's get in the fun parts.

When we pass a URL that points to a file (exe, pdf, ...) to page.goto, the browser cannot render it, and it barfs something like that:

Traceback (most recent call last):
  File "/home/raphael/gits/lookyloo/testin/dl.py", line 39, in <module>
    run(playwright)
  File "/home/raphael/gits/lookyloo/testin/dl.py", line 14, in run
    page.goto("https://circl.lu/assets/files/circl-training-2016.pdf")
  File "/home/raphael/.local/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 7413, in goto
    self._sync(
  File "/home/raphael/.local/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 88, in _sync
    return task.result()
  File "/home/raphael/.local/lib/python3.10/site-packages/playwright/_impl/_page.py", line 493, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
  File "/home/raphael/.local/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 122, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/home/raphael/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 39, in send
    return await self.inner_send(method, params, False)
  File "/home/raphael/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 63, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.Error: net::ERR_ABORTED at https://circl.lu/assets/files/circl-training-2016.pdf
=========================== logs ===========================
navigating to "https://circl.lu/assets/files/circl-training-2016.pdf", waiting until "load"
============================================================

Which makes sense because we're not loading an HTML page or anything the browser can render. so we need to catch that and handle it appropriately.

The code below works as a standalone sample, and I'm not sure if it is intended by the playwright devs:

#!/usr/bin/env python3

import asyncio
from playwright.async_api import async_playwright
from playwright.async_api import Error, TimeoutError as PlaywrightTimeoutError


async def main() -> None:
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        # Open new page
        page = await context.new_page()

        # Go to the file to download
        try:
            # await page.goto("https://www.google.fr")
            await page.goto("https://the.earth.li/~sgtatham/putty/0.77/w64/putty.exe")
            # await page.goto("dfjdskfdsjgjhdsf")
        except Error as e:
            print(e)
            # page.goto failed, but it (might have) triggered a download event.
            # If it is the case, let's try to save it.
            try:
                async with page.expect_download(timeout=5) as download_info:
                    download = await download_info.value
                    await download.save_as(download.suggested_filename)
            except PlaywrightTimeoutError:
                print('nope, no download were triggered')


asyncio.run(main())