Chunked (pseudo-stream) encryption & upload

[drzraf]

piping-ui-web/src/components/DataUploader.vue

Lines 324 to 327 in e14a9f7

	// Get encrypted
	// NOTE: In the future, ReadableStream can be uploaded.
	// (see: https://github.com/whatwg/fetch/pull/425#issuecomment-518899855)
	const encrypted: Uint8Array = await utils.encrypt(plainBodyArray, password);

I think we are far from fetch's Request ReadableStream wide browser-support and the only suitable alternative for files > 200 MB is chunked upload.
Theoretically, openpgp.encrypt can accept a ReadableSteam and we could iteratively read() AEAD-encrypted chunks from it with no need to convey the full Blob in memory at once (neither plain nor encrypted).

The implementation challenge rather lies in the upload/chunking.
(A stream-friendly chunked-upload would also allow for live-video upload without the need for memory-hungry temporary storage) (flowjs/flow.js#295)

Have you considered this aspect and possible solutions for it?

BTW: Great work! This software uses state-of-art standards & components and further push an idea once illustrated by https://github.com/hellais/pgp-file-upload/ or encrypt.to. Streamlining these pipelines seems like the last missing piece.

[nwtgck]

@drzraf Thant's right! I also want the feature of stream encryption & upload.

Why?

Here is why Piping UI doesn't support chunked upload for now. The reason is a compatibility with curl or other related HTTP clients. Piping UI uses Piping Server for data transfer. So, if you send like the following on Piping UI, you can download and decrypt with curl https://ppng.io/myimage | gpg. That's is the compatibility with curl, which uses pure HTTP stream.

Let me know if a standard way to chunked-upload via HTTP. I'd like to use a standard way to implement chunked feature for Piping UI.

Pure HTTP stream on browser

I think we are far from fetch's Request ReadableStream wide browser-support

I think so too. I watching the issue about the issue you also commented on. According to the issue, Firefox, Safari and Chrome team seemed to be interested to upload any ReadableStream.
whatwg/fetch#425 (comment)

Alternative solution

Here is an alternative solution chunked stream encryption & upload & download via Piping Server.
Piping Chunk

Piping Chunk is a PoC application for chunked encryption stream upload via Piping Server. So, it's more free to implement an experimental no-standard chunked upload & download than this project. Because of PoC, if you try Piping Chunk, Google Chrome is recommended.

[drzraf]

We indeed lack a standard for chunked uploads so most software rolls their own (eg: owncloud).

There is no actual HTTP (1.1) client-side stream-upload implementation, nor resumable/chunked upload client/server standard (which would basically mandate server-side to implement a specific way to assemble segments packets, assuming some headers).

The nearest attempts are

1. https://tus.io/protocols/resumable-upload.html
2. https://tools.ietf.org/html/draft-wright-http-partial-upload-01

But aside from Tus protocol I'm personally interested in one specific (HTTP 1) server-side implementations : Large Object Storage, in S3 but mostly in OpenStack Swift object-storage

=> The server expects several HTTP requests representing segments but would transparently provide the full assembled file on GET requests (It sounds like flow.js possibly supports this out-of-the-box (I've yet to test this).

Independently to the way chunks are sent (incremented query-string parameter, headers, ...), each plaintext File/Blob (or camera) probably have to lead to a ReadableStream for which an openpgp.stream.getReader(openpgp.encrypt()) encryptor is created which provides a chunk of encrypted data every time it's .read() from.

[nwtgck]

@drzraf Thank you so much for the information. I heard tus before. Resumable is very attractive.

The purpose of Piping UI is to transfer data for every device in secure way over Piping Server, and I'd like to make Piping Server simple as possible. So, I'd like to use client-side data assembling instead of server-side.

Now, Piping UI has two E2E encryption methods - password-protection and passwordless protection. As I mentioned the previous comment, password-protection has a compatibility with curl or other pure HTTP client. On the other hand, passwordless-protection need a public key exchange which is used only Piping UI. After the key exchange, Piping UI uses OpenPGP to encrypt & decrypt, which is widely used.

So, I have a plan to implement chunked upload & download in passwordless protection. When key change process and some field like "chunked-transfer": true, PIping UI uses chunked upload & download. For password-protection, to make an option to enable "chunk-transfer" is an appropriate way for Piping UI for collabolating with other softwares.

[nwtgck]

Good news!

Google Chrome Canary has now an experimental FetchUploadStreaming flag to upload arbitrary ReadableStream.
chromium/chromium@4c75c0c#diff-0f684d35848d8674d6bd9c5673588856

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary  --enable-features=FetchUploadStreaming

Note that even if you run Chrome above, Piping UI does not support streaming encryption upload yet.

[nwtgck]

Good news again!

The latest stable Chrome (Version 90.0.4430.93 (Official Build) (x86_64)) now supports FetchUploadStreaming with enabling Experimental Web Platform features on chrome://flags/.

Demo video

Here is a full demo video transferring about 1GB file with E2E encryption. This is real stream not chunked (pseudo-stream).

piping-ui-e2ee-1g-trim1.mp4

Application: https://600584c9b454fcff8b585a46--piping-ui.netlify.app/

Branch

The feature is implemented in branch https://github.com/nwtgck/piping-ui-web/tree/feature/streaming-encrypting-upload.

[nwtgck]

Good news!

We will have the streaming feature in Chrome 105.

https://chromestatus.com/feature/5274139738767360

[nwtgck]

@drzraf
Finally, Piping UI 0.12.0 released!
That contains chunked encrypted upload called Piping UI Robust, which may be Piping Robust in the future.
This feature is for every browser which does and does not support fetch() upload streaming.

Use passwordless protection for this feature. Passwordless will be enabled by default in the future.

The current implementation of Piping UI Robust is here:
https://github.com/nwtgck/piping-ui-web/blob/eea7be163486b515dfdd8b6607899ae9b294ddc6/src/piping-ui-robust.ts