34 - adding operator (#39)

<!-- Replace {issue_number} with the issue that will be closed after
merging this PR -->
Fixes #34

<!-- Write an `x` in all the boxes that apply -->
- [x] New feature (non-breaking change which adds functionality)

data/models/README.md

@@ -4,13 +4,12 @@
 
 ### Trained models
 
-| Model (Size)  | Download | Notebook |
+| Model (Size) | Download | Notebook |
 | -- |-- | -- |
 | ColonSegNet-07112023-2359.onnx (20MB or 20,067,028) | [:link:](TOADD) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1s-eKu6QjaV54jva_ylG1VxwCj1B8nwJ3)
-| Geometric Tx | [:link:](https://drive.google.com/file/d/1vE023Thky0vKfoYD7wZecrglHEgp3z-W/view?usp=sharing) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1s-eKu6QjaV54jva_ylG1VxwCj1B8nwJ3) | 
-| Baseline model 4 epochs | [:link:](https://drive.google.com/file/d/1SWPzOeqKyKqBM9FVfj0qCPn8cxEhX9BK/view?usp=drive_link) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/TOADD) |
-Add Your model (copy and paste this line) | [:link:](TOADD) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/TOADD) |
-
+| Geometric Tx                                        | [:link:](https://drive.google.com/file/d/1vE023Thky0vKfoYD7wZecrglHEgp3z-W/view?usp=sharing) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1s-eKu6QjaV54jva_ylG1VxwCj1B8nwJ3) | 
+| Baseline model 4 epochs                             | [:link:](https://drive.google.com/file/d/1SWPzOeqKyKqBM9FVfj0qCPn8cxEhX9BK/view?usp=drive_link) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/TOADD) |
+| Add Your Model by creation a new row      | [:link:](COPY&PASTE_DRIVE_LINK) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/COPY&PASTE_DRIVE_LINK) |
 
 ### Examples to download models in your local machine
 * ColonSegNet-07112023-2359.onnx

data/polyp-dataset/README.md

@@ -1,41 +1,87 @@
 # Kvasir-SEG dataset
 > The Kvasir-SEG dataset (size 46.2 MB) contains 1000 polyp images and their corresponding ground truth from the Kvasir Dataset v2. The resolution of the images contained in Kvasir-SEG varies from 332x487 to 1920x1072 pixels. The images and its corresponding masks are stored in two separate folders with the same filename. The image files are encoded using JPEG compression, and online browsing is facilitated. The bounding box (coordinate points) for the corresponding images are stored in a JSON file. https://datasets.simula.no/downloads/kvasir-seg.zip
 
-
 ```
 wget https://datasets.simula.no/downloads/kvasir-seg.zip #(45M)
 unzip kvasir-seg.zip
 rm kvasir-seg.zip
 ```
 
 ## Converting images to video 
+* converting ck images to video
 ```
 cd data/polyp-dataset/Kvasir-SEG/images
-ffmpeg -framerate 1 -pattern_type glob -i 'Kvasir-SEG/images/ck*.jpg' -c:v libx264 out720x576.mp4
+ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/images/ck*.jpg' -c:v libx264 out720x576.mp4
+```
+* converting selected images to video
+```
+ #cp images/ck2bxknhjvs1x0794iogrq49k.jpg selected/
+ #cp images/cju16jgnyzp970878melv7r25.jpg selected/
+ #cp images/cju33w4sdcivk0855x879zht7.jpg selected/
+ #cp images/cju85omszllp30850b6rm9mi3.jpg selected/
+ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/selected-512x/*.jpg' -c:v libx264 out720x576-selection512x.mp4
+```
+
+* converting resized selected images to video
 ```
+ #convert cju16jgnyzp970878melv7r25.jpg -resize 512x cju16jgnyzp970878melv7r25.jpg
+ #convert cju33w4sdcivk0855x879zht7.jpg -resize 512x cju33w4sdcivk0855x879zht7.jpg
+ #convert cju85omszllp30850b6rm9mi3.jpg -resize 512x cju85omszllp30850b6rm9mi3.jpg
+ #convert ck2bxknhjvs1x0794iogrq49k.jpg -resize 512x ck2bxknhjvs1x0794iogrq49k.jpg
+ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/selected-512x/*.jpg' -c:v libx264 out720x576-selection512x.mp4
+```
+* converting one selected image to video
+```
+cd $HOME/repositories/scikit-surgery/cmicHACKS2/data/polyp-dataset/Kvasir-SEG/
+cp -r selected selected-ck2bxknhjvs1x0794iogrq49k-512x-01-frames #same for other frames
+cp -r selected selected-ck2bxknhjvs1x0794iogrq49k-512x-05-frames #and create copies of the same image
+convert *.jpg -resize 512x *.jpg #only height
+convert *.jpg -resize 512x512! *.jpg #hegith and width
+
+cd $HOME/repositories/scikit-surgery/cmicHACKS2/data/polyp-dataset
+
+#ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/selected-ck2bxknhjvs1x0794iogrq49k-512x-01-frames/*.jpg' -c:v libx264 out512x448-selected-ck2bxknhjvs1x0794iogrq49k-01-frames.mp4 #don't work 4 1frame
+ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/selected-ck2bxknhjvs1x0794iogrq49k-05-frames/*.jpg' -c:v libx264 -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4
 
+ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/selected-ck2bxknhjvs1x0794iogrq49k-512x-05-frames/*.jpg' -c:v libx264 -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" out512x448-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4
 
+ffmpeg -framerate 30 -pattern_type glob -i 'Kvasir-SEG/selected-ck2bxknhjvs1x0794iogrq49k-512x-512x-05-frames/*.jpg' -c:v libx264 -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" out512x512-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4
 
-## Using `convert_video_to_gxf_entities.py`  (Graph Execution Framework, GXF)
+
+## check video
+vlc out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4 
+
+```
+
+References: 
+* https://www.networkworld.com/article/3697731/resizing-images-on-the-linux-command-line.html
+* https://stackoverflow.com/questions/20847674/ffmpeg-libx264-height-not-divisible-by-2 
+
+## Using `ffmeg` and `convert_video_to_gxf_entities.py`  (Graph Execution Framework, GXF)
 ```
 mamba activate cagxVE
+#ffmpeg -i *.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 512 --height 484 --channels 3 --framerate 30 --basename *
 #ffmpeg -i out.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python convert_video_to_gxf_entities.py --width 1920 --height 1080 --channels 3 --framerate 1 --basename out
 #ffmpeg -i out.mp4 -pix_fmt yuvj422p -f rawvideo pipe:1 | python convert_video_to_gxf_entities.py --width 600 --height 500 --channels 3 --framerate 1 --basename out
-ffmpeg -i out720x576.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 720 --height 576 --channels 3 --framerate 1 --basename out720x576
+ffmpeg -i out720x576.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 720 --height 576 --channels 3 --framerate 30 --basename out720x576
+
+ffmpeg -i out720x576-selection512x.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 512 --height 484 --channels 3 --framerate 30 --basename out720x576-selection512x
+
+
+ffmpeg -i out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 602 --height 528 --channels 3 --framerate 30 --basename out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames
+
+
+ffmpeg -i out512x448-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 602 --height 528 --channels 3 --framerate 30 --basename out512x448-selected-ck2bxknhjvs1x0794iogrq49k-05-frames
+
+ffmpeg -i out512x512-selected-ck2bxknhjvs1x0794iogrq49k-05-frames.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python ../../nvidia-clara-agx/holoscan-sdk-scripts/convert_video_to_gxf_entities.py --width 602 --height 528 --channels 3 --framerate 30 --basename out512x512-selected-ck2bxknhjvs1x0794iogrq49k-05-frames
+
+
 ```
 https://github.com/nvidia-holoscan/holoscan-sdk/tree/main/scripts#convert_video_to_gxf_entitiespy
 https://docs.nvidia.com/holoscan/sdk-user-guide/gxf/gxf_by_example.html  
 https://docs.nvidia.com/clara-holoscan/archive/clara-holoscan-0.3.0/gxf/index.html
 
 
-## Coping files to clara-agx
-```
-cd /workspace/holohub/data/colonoscopy_segmentation
-cp ../../cmicHACKS2/data/polyp-dataset/out720x576.mp4 .
-cp ../../cmicHACKS2/data/polyp-dataset/out720x576.gxf_* .
-```
-
-
 
 
 ## Testing `raw.mp4` with `convert_video_to_gxf_entities.py`
@@ -52,25 +98,9 @@ ffmpeg -i colon_exam_720x576.mp4 -pix_fmt rgb24 -f rawvideo pipe:1 | python scri
 
 ```
 
-errors
-
-```
-
-av_interleaved_write_frame(): Broken pipe
-Error writing trailer of pipe:1: Broken pipeime=00:00:00.04 bitrate=248832.0kbits/s speed=1.83x    
-frame=    1 fps=0.0 q=-0.0 Lsize=    1215kB time=00:00:00.04 bitrate=248832.0kbits/s speed=1.22x    
-video:1215kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
-Conversion failed!
-
-```
-
 https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara-holoscan/resources/holoscan_endoscopy_sample_data/files?version=20221121
 
-
-
 ## References 
 https://stackoverflow.com/questions/22965569/convert-from-jpg-to-mp4-by-ffmpeg  
 https://docs.nvidia.com/holoscan/sdk-user-guide/gxf/gxf_by_example.html   
 
-
-

nvidia-clara-agx/colonoscopy_segmentation/README.md

@@ -16,6 +16,11 @@ cd /workspace/holohub/data/colonoscopy_segmentation
 cp ../../cmicHACKS2/data/models/ColonSegNet-07112023-2359.onnx .
 cp ../../cmicHACKS2/data/polyp-dataset/out720x576.mp4 .
 cp ../../cmicHACKS2/data/polyp-dataset/out720x576.gxf_* .
+
+
+cp ../../cmicHACKS2/data/polyp-dataset/*gxf_entities .
+cp ../../cmicHACKS2/data/polyp-dataset/*gxf_index .
+cp ../../cmicHACKS2/data/polyp-dataset/*mp4 .
 ```
 
 
@@ -44,9 +49,7 @@ python colonoscopy_segmentation.py --data /workspace/holohub/data/colonoscopy_se
 ```
 
 
-
-
-* log
+* Terminal log
 ```
 
 root@cagx-ucl:/workspace/holohub# ./run launch colonoscopy_segmentation       
@@ -185,9 +188,39 @@ NOTE: This could take a couple of minutes depending on your model size and GPU!
 [info] [gxf_executor.cpp:1787] Graph execution finished. Fragment: Colonoscopy App
 [info] [gxf_executor.cpp:229] Destroying context
 
+```
+## Screenshots of `colonoscopy_segmentation.py`
+* Using `colon.onnx` with replayer:baseline `out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames` 
+![fig](Screenshot%20from%202023-11-15%2008-31-11.png)
+inference_output_tensor for one frame
+```
+tensor.min()=-11.6~
+tensor.max()=10.7~
+tensor.mean()=-5.2~
+tensor.shape()=(1,1,512,512)
+```
+
+* Using `ColonSegNet-07112023-2359.onnx` with replayer:baseline `out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames`
+![fig](Screenshot%20from%202023-11-15%2008-23-30.png)
+inference_output_tensor for one frame
+```
+tensor.min()=-12.3~
+tensor.max()=2.2~
+tensor.mean()=-6.0~
+tensor.shape()=(1,1,512,512)
+```
+
+* Using `colon.onnx` with replayer:baseline `out512x448-selected-ck2bxknhjvs1x0794iogrq49k-05-frames`
+![fig](Screenshot%20from%202023-11-15%2008-35-29.png)
+inference_output_tensor for one frame
+```
+tensor.min()=-10.6~
+tensor.max()=2.2~
+tensor.mean()=-6.7~
+tensor.shape()=(1,1,512,512)
 ```
 
-## Optmise model
+## Optmisation of model
 
 ```
 [info] [core.cpp:46] TRT Inference: converting ONNX model at /workspace/holohub/data/colonoscopy_segmentation/colon.onnx

nvidia-clara-agx/colonoscopy_segmentation/colonoscopy_segmentation.py

@@ -16,7 +16,7 @@
 import os
 from argparse import ArgumentParser
 
-from holoscan.core import Application
+from holoscan.core import Application, Operator, OperatorSpec
 from holoscan.operators import (
     AJASourceOp,
     FormatConverterOp,
@@ -25,9 +25,39 @@
     SegmentationPostprocessorOp,
     VideoStreamReplayerOp,
 )
-from holoscan.resources import BlockMemoryPool, CudaStreamPool, MemoryStorageType
+from holoscan.gxf import Entity
+from holoscan.resources import BlockMemoryPool, CudaStreamPool, MemoryStorageType, UnboundedAllocator
+import cupy as cp
+import holoscan as hs
 
 
+class InfoOp(Operator):
+    """"""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+    def setup(self, spec: OperatorSpec):
+        spec.input("in")
+        spec.output("out")
+
+    def compute(self, op_input, op_output, context):
+        # Get input message
+        in_message = op_input.receive("in")
+
+        # Transpose
+        tensor = cp.asarray(in_message.get("inference_output_tensor"))
+        print(f"tensor.min()={cp.min(tensor)}")
+        print(f"tensor.max()={cp.max(tensor)}")
+        print(f"tensor.mean()={cp.mean(tensor)}")
+        print(f"tensor.shape={tensor.shape}")
+
+        # Create output message
+        out_message = Entity(context)
+        out_message.add(hs.as_tensor(tensor), "inference_output_tensor")
+        op_output.emit(out_message, "out")
+
+
 class ColonoscopyApp(Application):
     def __init__(self, data, source="replayer"):
         """Initialize the colonoscopy segmentation application
@@ -54,8 +84,9 @@ def __init__(self, data, source="replayer"):
         self.sample_data_path = data
 
         self.model_path_map = {
-            "ultrasound_seg": os.path.join(self.sample_data_path, "ColonSegNet-07112023-2359.onnx"),
             #"ultrasound_seg": os.path.join(self.sample_data_path, "colon.onnx"),
+            "ultrasound_seg": os.path.join(self.sample_data_path, "ColonSegNet-07112023-2359.onnx"),
+            #"ultrasound_seg": os.path.join(self.sample_data_path, "ColonSegNet_brightcontr.onnx"),
         }
 
     def compose(self):
@@ -143,6 +174,12 @@ def compose(self):
             transmit_on_cuda=True,
         )
 
+        info_op = InfoOp(
+            self,
+            name="info",
+            pool=UnboundedAllocator(self, name="pool"),
+        )
+
         postprocessor_block_size = width_inference * height_inference
         postprocessor_num_blocks = 2
         segmentation_postprocessor = SegmentationPostprocessorOp(
@@ -172,7 +209,9 @@ def compose(self):
             self.add_flow(source, segmentation_visualizer, {("", "receivers")})
             self.add_flow(source, segmentation_preprocessor)
         self.add_flow(segmentation_preprocessor, segmentation_inference, {("", "receivers")})
-        self.add_flow(segmentation_inference, segmentation_postprocessor, {("transmitter", "")})
+        self.add_flow(segmentation_inference, info_op, {("transmitter", "in")})
+        self.add_flow(info_op, segmentation_postprocessor, {("out", "")})
+        #self.add_flow(segmentation_inference, segmentation_postprocessor, {("transmitter", "")})
         self.add_flow(
             segmentation_postprocessor,
             segmentation_visualizer,

nvidia-clara-agx/colonoscopy_segmentation/colonoscopy_segmentation.yaml

@@ -16,7 +16,11 @@
 ---
 replayer:  # VideoStreamReplayer
   #basename: "colon_exam_720x576"
-  basename: "out720x576"
+  #basename: "out720x576"
+  #basename: "out720x576-selection512x"
+  #basename: "out512x448-selected-ck2bxknhjvs1x0794iogrq49k-05-frames"#not-rendered-properly
+  #basename: "out512x512-selected-ck2bxknhjvs1x0794iogrq49k-05-frames"#not-rendered-properly
+  basename: "out602x528-selected-ck2bxknhjvs1x0794iogrq49k-05-frames"
   frame_rate: 0 # as specified in timestamps
   repeat: true # default: false
   realtime: true # default: true
@@ -45,6 +49,8 @@ segmentation_postprocessor:  # Postprocessor
   data_format: nchw
 
 segmentation_visualizer:  # Holoviz
+  height: 512
+  width: 512
   color_lut: [
     [0.65, 0.81, 0.89, 0.1],
     [0.2, 0.63, 0.17, 0.7],