added feed and live command with upload. (#162)

* added feed and live command with upload. Signed-off-by: Pradip Parmar <[email protected]> * changed readme file for feed and live commands Signed-off-by: Pradip Parmar <[email protected]> * read me updated for live and feed command Signed-off-by: Pradip Parmar <[email protected]> * Update README.md Co-authored-by: Lz <[email protected]> * Update README.md Co-authored-by: Lz <[email protected]> * Update README.md Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * readme updates from suggestion -- removed sync and live flag from uploda. from zbox code and readme both. Signed-off-by: Pradip Parmar <[email protected]> * image updated for sync Signed-off-by: Pradip Parmar <[email protected]> * updated flag description in code. Signed-off-by: Pradip Parmar <[email protected]> * image url update Signed-off-by: Pradip Parmar <[email protected]> * Update README.md * Update cmd/upload.go Co-authored-by: Vincenzo Palazzo <[email protected]> * Update cmd/upload.go Co-authored-by: Vincenzo Palazzo <[email protected]> * Update cmd/upload.go Co-authored-by: Vincenzo Palazzo <[email protected]> * build fix Signed-off-by: Pradip Parmar <[email protected]> * added feed and live command with upload. Signed-off-by: Pradip Parmar <[email protected]> * changed readme file for feed and live commands Signed-off-by: Pradip Parmar <[email protected]> * read me updated for live and feed command Signed-off-by: Pradip Parmar <[email protected]> * Update README.md Co-authored-by: Lz <[email protected]> * Update README.md Co-authored-by: Lz <[email protected]> * Update README.md Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * Update cmd/upload.go Co-authored-by: Lz <[email protected]> * readme updates from suggestion -- removed sync and live flag from uploda. from zbox code and readme both. Signed-off-by: Pradip Parmar <[email protected]> * image updated for sync Signed-off-by: Pradip Parmar <[email protected]> * updated flag description in code. Signed-off-by: Pradip Parmar <[email protected]> * image url update Signed-off-by: Pradip Parmar <[email protected]> * Update README.md * Update cmd/upload.go Co-authored-by: Vincenzo Palazzo <[email protected]> * Update cmd/upload.go Co-authored-by: Vincenzo Palazzo <[email protected]> * Update cmd/upload.go Co-authored-by: Vincenzo Palazzo <[email protected]> * build fix Signed-off-by: Pradip Parmar <[email protected]> Co-authored-by: Pradip Parmar <[email protected]> Co-authored-by: Lz <[email protected]> Co-authored-by: Vincenzo Palazzo <[email protected]>
0chain · Feb 15, 2022 · d172e70 · d172e70
1 parent eb3ff92
commit d172e70
Show file tree

Hide file tree

Showing 2 changed files with 247 additions and 39 deletions.
diff --git a/README.md b/README.md
@@ -30,6 +30,8 @@ zbox is a command line interface (CLI) tool to understand the capabilities of 0C
          - [Update blobber settings](#update-blobber-settings)
        - [Uploading and Managing Files](#uploading-and-managing-files)
          - [Upload](#upload)
+            - [Stream](#stream)
+            - [Feed](#feed)
          - [Download](#download)
          - [Update](#update)
          - [Delete](#delete)
@@ -673,12 +675,6 @@ The user must be the owner of the allocation.You can request the file be encrypt
 | remotepath              | yes      | remote path to upload file to, use to access file later            |         | string                 |
 | thumbnailpath           | no       | local path of thumbnaSil                                           |         | file path              |
 | chunksize               | no       | chunk size                                                         | 65536   | int                    |
-| delay                   | no       | set segment duration to seconds. only works with --live and --sync.| 5       | int                    |
-| sync                    | no       | enable SyncUpload from remote live feed. disabled it by default.   | false   | boolean                |
-| feed                    | no       | set remote live feed to url. only works with --sync.               | false   | url                    |
-| downloader-args         | no       | pass args to youtube-dl to download video. default is \"-q -f best\". only works with --sync.| -q -f best | [youtube-dl](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options)|
-| ffmpeg-args             | no       | pass args to ffmpeg to build segments. only works with --sync.     | -loglevel warning   | [ffmpeg](https://www.ffmpeg.org/ffmpeg.html)              |
-| live                    | no       | enable LiveUpload from local devices. disabled by default.         | false   | boolean                |
 
 
 <details>
@@ -720,9 +716,60 @@ Response:
 Status completed callback. Type = application/octet-stream. Name = sensitivedata.txt
 ```
 
-**Download segment files from remote live feed, re-encode and upload**
+## Stream
+
+Use `stream` to capture video and audio streaming form local devices, and upload
+
+The user must be the owner of the allocation.You can request the file be encrypted before upload, and can send thumbnails with the file. 
+
+| Parameter               | Required | Description                                          | Default | Valid values                            |
+|-------------------------|----------|------------------------------------------------------|---------|-----------------------------------------|
+| allocation              | yes      | allocation id, sender must be allocation owner                     |         | string                 |
+| commit                  | no       | save metadata to blockchain                                        | false   | boolean                |
+| encrypt                 | no       | encrypt file before upload                                         | false   | boolean                |
+| localpath               | yes      | local path of segment files to download, generate and upload                                   |         | file path              |
+| remotepath              | yes      | remote path to upload file to, use to access file later            |         | string                 |
+| thumbnailpath           | no       | local path of thumbnaSil                                           |         | file path              |
+| chunksize               | no       | chunk size                                                         | 65536   | int                    |
+| delay                   | no       | set segment duration to seconds.                                   | 5       | int                    |
+
+<details>
+  <summary>stream</summary>
+
+![image](https://github.com/0chain/blobber/wiki/uml/usecase/live_upload_live.png)
+
+</details>
+
+## Feed
+
+Use `feed` command to automatically download segment files from remote live feed with `--downloader-args "-q -f best"`
+ - encode them into new segment files with `--delay` and `--ffmpeg-args`, and upload. 
+ - please use `youtube-dl -F https://www.youtube.com/watch?v=pC5mGB5enkw` to list formats of video (see below).
+
+The user must be the owner of the allocation.You can request the file be encrypted before upload, and can send thumbnails with the file. 
+
+| Parameter               | Required | Description                                          | Default | Valid values                            |
+|-------------------------|----------|------------------------------------------------------|---------|-----------------------------------------|
+| allocation              | yes      | allocation id, sender must be allocation owner                     |         | string                 |
+| commit                  | no       | save metadata to blockchain                                        | false   | boolean                |
+| encrypt                 | no       | encrypt file before upload                                         | false   | boolean                |
+| localpath               | yes      | local path of segment files to download, generate and upload                                   |         | file path              |
+| remotepath              | yes      | remote path to upload file to, use to access file later            |         | string                 |
+| thumbnailpath           | no       | local path of thumbnaSil                                           |         | file path              |
+| chunksize               | no       | chunk size                                                         | 65536   | int                    |
+| delay                   | no       | set segment duration to seconds.                                   | 5       | int                    |
+| feed                    | no       | set remote live feed to url.                                       | false   | url                    |
+| downloader-args         | no       | pass args to youtube-dl to download video. default is \"-q -f best\". | -q -f best | [youtube-dl](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options)|
+| ffmpeg-args             | no       | pass args to ffmpeg to build segments.                             | -loglevel warning   | [ffmpeg](https://www.ffmpeg.org/ffmpeg.html)              |
+
+
+<details>
+  <summary>feed</summary>
+
+![image](https://github.com/0chain/blobber/wiki/uml/usecase/live_upload_sync.png)
+
+</details>
 
-Use `upload --sync` command to automatically download segment files from remove live feed with `--downloader-args "-f 22"`, encode them into new segment files with `--delay` and `--ffmpeg-args`, and upload. please use `youtube-dl -F https://www.youtube.com/watch?v=pC5mGB5enkw` to list formats of video (see below). 
 
 ```
 [youtube] pC5mGB5enkw: Downloading webpage
@@ -753,18 +800,9 @@ format code  extension  resolution note
 `--downloader-args "-f 22"` dowloads video with `22           mp4        1280x676   720p 2117k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)`
 
 ```
-./zbox upload --sync --localpath <absolute path to file>/tvshow.m3u8 --remotepath /videos/tvsho --allocation d0939e912851959637257573b08c748474f0dd0ebbc8e191e4f6ad69e4fdc7ac  --delay 10 --chunksize 655360 --downloader-args "-f 22" --feed https://www.youtube.com/watch?v=pC5mGB5enkw
-
-```
-
-**Capture streaming from local devices, encode with ffmpeg, and upload**
-
-Use `upload --live` to capture video and audio streaming form local devices, and upload
+./zbox feed --localpath <absolute path to file>/tvshow.m3u8 --remotepath /videos/tvsho --allocation d0939e912851959637257573b08c748474f0dd0ebbc8e191e4f6ad69e4fdc7ac  --delay 10 --chunksize 655360 --downloader-args "-f 22" --feed https://www.youtube.com/watch?v=pC5mGB5enkw
 
 ```
-./zbox upload --live --localpath <absolute path to file>/streaming.m3u8 --remotepath /videos/streaming --allocation d0939e912851959637257573b08c748474f0dd0ebbc8e191e4f6ad69e4fdc7ac  --delay 10 --chunksize 655360 
-```
-
 
 ## Download
 

diff --git a/cmd/upload.go b/cmd/upload.go
@@ -113,20 +113,160 @@ var uploadCmd = &cobra.Command{
 			attrs.WhoPaysForReads = wp // set given value
 		}
 
-		live, _ := cmd.Flags().GetBool("live")
-		sync, _ := cmd.Flags().GetBool("sync")
 		chunkSize, _ := cmd.Flags().GetInt("chunksize")
 
-		if live {
-			// capture video and audio from local default camera and micrlphone, and upload it to zcn
-			err = startLiveUpload(cmd, allocationObj, localpath, remotepath, encrypt, chunkSize, attrs)
-		} else if sync {
-			// download video from remote live feed(eg youtube), and sync it to zcn
-			err = startSyncUpload(cmd, allocationObj, localpath, remotepath, encrypt, chunkSize, attrs)
-		} else {
-			err = startChunkedUpload(cmd, allocationObj, localpath, thumbnailpath, remotepath, encrypt, chunkSize, attrs, statusBar, false)
+		if err := startChunkedUpload(cmd, allocationObj, localpath, thumbnailpath, remotepath, encrypt, chunkSize, attrs, statusBar, false); err != nil {
+			PrintError("Upload failed.", err)
+			os.Exit(1)
+		}
+		wg.Wait()
+		if !statusBar.success {
+			os.Exit(1)
+		}
+
+		if commit {
+			remotepath = zboxutil.GetFullRemotePath(localpath, remotepath)
+			statusBar.wg.Add(1)
+			commitMetaTxn(remotepath, "Upload", "", "", allocationObj, nil, statusBar)
+			statusBar.wg.Wait()
+		}
+	},
+}
+
+// feedCmd represents upload command with --sync flag
+var feedCmd = &cobra.Command{
+	Use:   "feed",
+	Short: "download segment files from remote live feed, and upload",
+	Long:  "download segment files from remote live feed, and upload",
+	Args:  cobra.MinimumNArgs(0),
+	Run: func(cmd *cobra.Command, args []string) {
+		fflags := cmd.Flags()              // fflags is a *flag.FlagSet
+		if !fflags.Changed("allocation") { // check if the flag "path" is set
+			PrintError("Error: allocation flag is missing") // If not, we'll let the user know
+			os.Exit(1)                                      // and return
+		}
+		if !fflags.Changed("remotepath") {
+			PrintError("Error: remotepath flag is missing")
+			os.Exit(1)
+		}
+
+		if !fflags.Changed("localpath") {
+			PrintError("Error: localpath flag is missing")
+			os.Exit(1)
+		}
+
+		allocationID := cmd.Flag("allocation").Value.String()
+		allocationObj, err := sdk.GetAllocation(allocationID)
+		if err != nil {
+			PrintError("Error fetching the allocation.", err)
+			os.Exit(1)
+		}
+		remotepath := cmd.Flag("remotepath").Value.String()
+		localpath := cmd.Flag("localpath").Value.String()
+		encrypt, _ := cmd.Flags().GetBool("encrypt")
+		commit, _ := cmd.Flags().GetBool("commit")
+
+		wg := &sync.WaitGroup{}
+		statusBar := &StatusBar{wg: wg}
+		wg.Add(1)
+		if strings.HasPrefix(remotepath, "/Encrypted") {
+			encrypt = true
+		}
+		var attrs fileref.Attributes
+		if fflags.Changed("attr-who-pays-for-reads") {
+			var (
+				wp  common.WhoPays
+				wps string
+			)
+			if wps, err = fflags.GetString("attr-who-pays-for-reads"); err != nil {
+				log.Fatalf("getting 'attr-who-pays-for-reads' flag: %v", err)
+			}
+			if err = wp.Parse(wps); err != nil {
+				log.Fatal(err)
+			}
+			attrs.WhoPaysForReads = wp // set given value
+		}
+
+		chunkSize, _ := cmd.Flags().GetInt("chunksize")
+
+		// download video from remote live feed(eg youtube), and sync it to zcn
+		err = startSyncUpload(cmd, allocationObj, localpath, remotepath, encrypt, chunkSize, attrs)
+
+		if err != nil {
+			PrintError("Upload failed.", err)
+			os.Exit(1)
+		}
+		wg.Wait()
+		if !statusBar.success {
+			os.Exit(1)
+		}
+
+		if commit {
+			remotepath = zboxutil.GetFullRemotePath(localpath, remotepath)
+			statusBar.wg.Add(1)
+			commitMetaTxn(remotepath, "Upload", "", "", allocationObj, nil, statusBar)
+			statusBar.wg.Wait()
+		}
+	},
+}
+
+// streamCmd represents upload command with --live flag
+var streamCmd = &cobra.Command{
+	Use:   "stream",
+	Short: "capture video and audio streaming form local devices, and upload",
+	Long:  "capture video and audio streaming form local devices, and upload",
+	Args:  cobra.MinimumNArgs(0),
+	Run: func(cmd *cobra.Command, args []string) {
+		fflags := cmd.Flags()              // fflags is a *flag.FlagSet
+		if !fflags.Changed("allocation") { // check if the flag "path" is set
+			PrintError("Error: allocation flag is missing") // If not, we'll let the user know
+			os.Exit(1)                                      // and return
+		}
+		if !fflags.Changed("remotepath") {
+			PrintError("Error: remotepath flag is missing")
+			os.Exit(1)
+		}
+
+		if !fflags.Changed("localpath") {
+			PrintError("Error: localpath flag is missing")
+			os.Exit(1)
 		}
 
+		allocationID := cmd.Flag("allocation").Value.String()
+		allocationObj, err := sdk.GetAllocation(allocationID)
+		if err != nil {
+			PrintError("Error fetching the allocation.", err)
+			os.Exit(1)
+		}
+		remotepath := cmd.Flag("remotepath").Value.String()
+		localpath := cmd.Flag("localpath").Value.String()
+		encrypt, _ := cmd.Flags().GetBool("encrypt")
+		commit, _ := cmd.Flags().GetBool("commit")
+
+		wg := &sync.WaitGroup{}
+		statusBar := &StatusBar{wg: wg}
+		wg.Add(1)
+		if strings.HasPrefix(remotepath, "/Encrypted") {
+			encrypt = true
+		}
+		var attrs fileref.Attributes
+		if fflags.Changed("attr-who-pays-for-reads") {
+			var wp common.WhoPays
+			var wps string
+			if wps, err = fflags.GetString("attr-who-pays-for-reads"); err != nil {
+				log.Fatalf("getting 'attr-who-pays-for-reads' flag: %v", err)
+			}
+			if err = wp.Parse(wps); err != nil {
+				log.Fatal(err)
+			}
+			attrs.WhoPaysForReads = wp // set given value
+		}
+
+		chunkSize, _ := cmd.Flags().GetInt("chunksize")
+
+		// capture video and audio from local default camera and micrlphone, and upload it to zcn
+		err = startLiveUpload(cmd, allocationObj, localpath, remotepath, encrypt, chunkSize, attrs)
+
 		if err != nil {
 			PrintError("Upload failed.", err)
 			os.Exit(1)
@@ -310,17 +450,6 @@ func init() {
 
 	uploadCmd.Flags().Int("chunksize", sdk.CHUNK_SIZE, "chunk size")
 
-	uploadCmd.Flags().Int("delay", 5, "set segment duration to seconds. only works with --live and --sync. default duration is 5s.")
-
-	// SyncUpload
-	uploadCmd.Flags().Bool("sync", false, "enable SyncUpload from remote live feed. disabled by default.")
-	uploadCmd.Flags().String("feed", "", "set remote live feed to url. only works with --sync.")
-	uploadCmd.Flags().String("downloader-args", "-q -f best", "pass args to youtube-dl to download video. default is \"-q\". only works with --sync.")
-	uploadCmd.Flags().String("ffmpeg-args", "-loglevel warning", "pass args to ffmpeg to build segments. default is \"-loglevel warning\". only works with --sync.")
-
-	// LiveUpload
-	uploadCmd.Flags().Bool("live", false, "enable LiveUpload from local devices. disabled by default.")
-
 	uploadCmd.MarkFlagRequired("allocation")
 	uploadCmd.MarkFlagRequired("remotepath")
 	uploadCmd.MarkFlagRequired("localpath")
@@ -330,4 +459,45 @@ func init() {
 	createDirCmd.MarkFlagRequired("allocation")
 	createDirCmd.MarkFlagRequired("dirname")
 
+	// feed command
+	rootCmd.AddCommand(feedCmd)
+	feedCmd.PersistentFlags().String("allocation", "", "Allocation ID")
+	feedCmd.PersistentFlags().String("remotepath", "", "Remote path to upload")
+	feedCmd.PersistentFlags().String("localpath", "", "Local path of file to upload")
+	feedCmd.PersistentFlags().String("thumbnailpath", "", "Local thumbnail path of file to upload")
+	feedCmd.PersistentFlags().String("attr-who-pays-for-reads", "owner", "Who pays for reads: owner or 3rd_party")
+	feedCmd.Flags().Bool("encrypt", false, "pass this option to encrypt and upload the file")
+	feedCmd.Flags().Bool("commit", false, "pass this option to commit the metadata transaction")
+
+	feedCmd.Flags().Int("chunksize", sdk.CHUNK_SIZE, "chunk size")
+
+	feedCmd.Flags().Int("delay", 5, "set segment duration to seconds.")
+
+	// SyncUpload
+	feedCmd.Flags().String("feed", "", "set remote live feed to url.")
+	feedCmd.Flags().String("downloader-args", "-q -f best", "pass args to youtube-dl to download video. default is \"-q\".")
+	feedCmd.Flags().String("ffmpeg-args", "-loglevel warning", "pass args to ffmpeg to build segments. default is \"-loglevel warning\".")
+
+	feedCmd.MarkFlagRequired("allocation")
+	feedCmd.MarkFlagRequired("remotepath")
+	feedCmd.MarkFlagRequired("localpath")
+
+	// stream Command
+	rootCmd.AddCommand(streamCmd)
+	streamCmd.PersistentFlags().String("allocation", "", "Allocation ID")
+	streamCmd.PersistentFlags().String("remotepath", "", "Remote path to upload")
+	streamCmd.PersistentFlags().String("localpath", "", "Local path of file to upload")
+	streamCmd.PersistentFlags().String("thumbnailpath", "", "Local thumbnail path of file to upload")
+	streamCmd.PersistentFlags().String("attr-who-pays-for-reads", "owner", "Who pays for reads: owner or 3rd_party")
+	streamCmd.Flags().Bool("encrypt", false, "pass this option to encrypt and upload the file")
+	streamCmd.Flags().Bool("commit", false, "pass this option to commit the metadata transaction")
+
+	streamCmd.Flags().Int("chunksize", sdk.CHUNK_SIZE, "chunk size")
+
+	streamCmd.Flags().Int("delay", 5, "set segment duration to seconds.")
+
+	streamCmd.MarkFlagRequired("allocation")
+	streamCmd.MarkFlagRequired("remotepath")
+	streamCmd.MarkFlagRequired("localpath")
+
 }