Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Common endpoint to upload videos #16

Open
ravi-sawlani-yral opened this issue Feb 6, 2025 · 23 comments · May be fixed by #17
Open

Common endpoint to upload videos #16

ravi-sawlani-yral opened this issue Feb 6, 2025 · 23 comments · May be fixed by #17
Assignees
@ravi-sawlani-yral

Comments

@ravi-sawlani-yral
Copy link

ravi-sawlani-yral commented Feb 6, 2025 •
edited
Loading

Motivation

We send multiple events during video upload like video_upload_unsuccessful and video_upload_successful and we do have multiple clients that uploads video Youtube video importer, IOS app, leptos-frontend. The idea was to have a common endpoint that could upload the video to canister and also emit these events correctly.

Suggested Solution

sequenceDiagram
   external-client ->> Cloudflare-worker: requests a signed url to upload user video.   
   Note over Cloudflare-worker: creates new entry in kv and stores video uid
   Cloudflare-worker -->> external-client: returns signed url to upload video.
   external-client ->> Cloudflare-streams: starts uloading video (This would happen even before the user clicks on the upload button)
    alt user uploads video
        external-client ->> Cloudflare-worker: sends metadata of video along with user delegated identity
        Note over Cloudflare-worker: stores the video metadata in kv for the video uid.
        Cloudflare-streams ->> Cloudflare-worker: sends request to mark video as ready
        Note over Cloudflare-worker: adds video to canisters using users delegated identity and emits respective events.
    else user cancels upload
        Note over Cloudflare-worker: deletes kv storage after one hour and deletes video.
    end
Loading
@ravi-sawlani-yral ravi-sawlani-yral self-assigned this Feb 6, 2025
@ravi-sawlani-yral
Copy link
Author

@komal-sai-yral I noticed that we are also emitting video_upload_video_selected in leptos-repo over here: VideoUploadVideoSelected
I don't think it's possible to extract out these into common endpoint as this should emitted from client side.

@komal-sai-yral
Copy link
Contributor

ya makes sense. Can be fired from client to offchain GRPC API. Although these are less priority events for the video upload pipleline. Just needed for analytics. The only important event that triggers the video processing pipeline is video_upload_successful

@ravi-sawlani-yral
Copy link
Author

@sarvesh-sharma-yral do we have setup to fire events from iOS?

@komal-sai-yral
Copy link
Contributor

komal-sai-yral commented Feb 6, 2025 •
edited
Loading

@sarvesh-sharma-yral also note that the event GRPC API is an authenticated call to offchain requiring an API key. Not sure its possible for sending the event directly from client device. Might have to plan accordingly.

Does it need to be authenticated is another debate

@sarvesh-sharma-yral
Copy link

We don't have a setup to fire events from client side yet

@ravi-sawlani-yral
Copy link
Author

@sarvesh-sharma-yral also note that the event GRPC API is an authenticated call to offchain requiring an API key. Not sure its possible for sending the event directly from client device. Might have to plan accordingly.

Does it need to be authenticated is another debate

if that's the case then we can skip the events that needs to be fired on client side for now. There is no option.

@ravi-sawlani-yral
Copy link
Author

ravi-sawlani-yral commented Feb 6, 2025 •
edited
Loading

@sarvesh-sharma-yral @komal-sai-yral is it possible to send events directly to google instead of routing it through offchain?

@sarvesh-sharma-yral
Copy link

do you mean google analytics?
if so, yes

@siyara-m-yral
Copy link

@ravi-sawlani-yral, we would need to keep description and hashtags as optional fields. User should be able to upload video without having to add these details

@ravi-sawlani-yral ravi-sawlani-yral transferred this issue from yral-dapp/off-chain-agent Feb 11, 2025
@harshita-srivastava-yral
Copy link

harshita-srivastava-yral commented Feb 12, 2025 •
edited
Loading

@sarvesh-sharma-yral Please have a look at the proposed architecture and ensure its compatibility with the proposed iOS app front.

@ravi-sawlani-yral
Copy link
Author

@sarvesh-sharma-yral upload design flow might change due to this architecture. We might need to add a toast that video would be uploaded soon.

@sarvesh-sharma-yral
Copy link

just wanted to confirm, clients are going to interact with streams directly?
I remember us discussing in the standup call using a backend endpoint and it handling the uploads.

Don't think we'd need a toast, basis the figma designs.
We can always come back to the video upload tab and look at the progress, a toast is shown if upload succeeds or fails.

@ravi-sawlani-yral
Copy link
Author

@sarvesh-sharma-yral I am not sure how would you show the progress bar as the client is never informed about the final upload. It would happen in the background.

@ravi-sawlani-yral
Copy link
Author

just wanted to confirm, clients are going to interact with streams directly? I remember us discussing in the standup call using a backend endpoint and it handling the uploads.

Don't think we'd need a toast, basis the figma designs. We can always come back to the video upload tab and look at the progress, a toast is shown if upload succeeds or fails.

Not directly as such. Backend would give you the url where you can upload the video which would start as soon as the client selects the video but that should not be communicated to client. When they press upload you send the rest of the metadata to endpoint and will rely on backend to make that video available eventually.

@ravi-sawlani-yral
Copy link
Author

@harshita-srivastava-yral as far as I can see the design needs to be modified.

@sarvesh-sharma-yral
Copy link

just wanted to confirm, clients are going to interact with streams directly? I remember us discussing in the standup call using a backend endpoint and it handling the uploads.

Don't think we'd need a toast, basis the figma designs. We can always come back to the video upload tab and look at the progress, a toast is shown if upload succeeds or fails.

Not directly as such. Backend would give you the url where you can upload the video which would start as soon as the client selects the video but that should not be communicated to client. When they press upload you send the rest of the metadata to endpoint and will rely on backend to make that video available eventually.

Is it going to be a cfstream url or a custom url?

@sarvesh-sharma-yral
Copy link

sarvesh-sharma-yral commented Feb 13, 2025 •
edited by ravi-sawlani-yral
Loading

@harshita-srivastava-yral as far as I can see the design needs to be modified.

Why?

@siyara-m-yral
Copy link

  • Working on event emitting and need clarification from Saikat for Corner cases
  • If user selects one video and then goes back and selects another one, client is responsible to handle this scenario

@siyara-m-yral
Copy link

  • Testing to be done today after which can be closed

@siyara-m-yral
Copy link

  • Testing cloudflare worker to upload video using common endpoint with api token.

@ravi-sawlani-yral ravi-sawlani-yral linked a pull request Feb 25, 2025 that will close this issue
Open
@harshita-srivastava-yral
Copy link

  • We need a secret for webhook then we can deploy

@harshita-srivastava-yral
Copy link

harshita-srivastava-yral commented Mar 3, 2025 •
edited
Loading

@siyara-m-yral
Copy link

  • This has been deployed, once we get go ahead from Sarvesh, will close the PR if no further dependencies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ravi-sawlani-yral ravi-sawlani-yral

Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@ravi-sawlani-yral @komal-sai-yral @harshita-srivastava-yral @sarvesh-sharma-yral @siyara-m-yral