selfie_multiclass task with option delegate=BaseOptions.Delegate.GPU not working

[SoaringTiger]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Ubuntu 22.04.3 LTS (docker image: tensorflow:2.16.1-gpu-jupyter)

MediaPipe Tasks SDK version

0.10.14

Task name (e.g. Image classification, Gesture recognition etc.)

selfie_multiclass

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

delegate=BaseOptions.Delegate.GPU not work

Describe the expected behaviour

enable GPU support

Standalone code/steps you may have used to try to get what you need

import mediapipe as mp
import numpy as np
import cv2

model_path = '/models/mediapipe/selfie_multiclass_256x256.tflite'

IMAGE_FILE = 'input.jpg'

BaseOptions = mp.tasks.BaseOptions
ImageSegmenter = mp.tasks.vision.ImageSegmenter
ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions
VisionRunningMode = mp.tasks.vision.RunningMode


# ImageSegmenter 
options = ImageSegmenterOptions(
    #base_options=BaseOptions(model_asset_path=model_path),
    base_options = BaseOptions(model_asset_path=model_path, delegate=BaseOptions.Delegate.GPU),
    running_mode=VisionRunningMode.IMAGE,
    output_category_mask=True)

colors = [
    [255, 0, 0],  # background: blue
    [255, 255, 0],  # hair: skyblue
    [0, 255, 0],  # body-skin: green
    [255, 0, 128],  # face-skin: purple
    [255, 0, 255],  # clothes: magenta
    [0, 128, 255]  # others: orange
]

category = ["background", "hair", "body-skin", "face-skin", "clothes", "others"]


with ImageSegmenter.create_from_options(options) as segmenter:

    mp_image = mp.Image.create_from_file(IMAGE_FILE)

    original_image = mp_image.numpy_view()
    # original_image = original_image[:, :, :3]  # remove alpha channel

    # masks
    segmented_masks = segmenter.segment(mp_image)

    # category_mask
    category_mask = segmented_masks.category_mask
    category_mask_np = category_mask.numpy_view()

    h, w = category_mask_np.shape # numpy  width, height 
    color_image = np.zeros((h, w, 3), dtype=np.uint8) #generate same size empty image
    for i, color in enumerate(colors): # fill every class
        color_image[category_mask_np == i] = color

    alpha = 0.5
    blended_image = cv2.addWeighted(original_image, 1 - alpha, color_image, alpha, 0)

    status = cv2.imwrite('path_to_save_image.jpg', blended_image)
    print("Image written to file-system: ", status)

Other info / Complete Logs

When I set base_options=BaseOptions(model_asset_path=model_path), the code worked, when I set base_options = BaseOptions(model_asset_path=model_path, delegate=BaseOptions.Delegate.GPU), the code got error:
'''
2024-06-25 02:26:50.945231: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1719282412.338260     551 task_runner.cc:85] GPU suport is not available: INTERNAL: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
Traceback (most recent call last):
  File "/tools/mask_1_img.py", line 37, in <module>
    with ImageSegmenter.create_from_options(options) as segmenter:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/mediapipe/tasks/python/vision/image_segmenter.py", line 268, in create_from_options
    return cls(
           ^^^^
  File "/usr/local/lib/python3.11/dist-packages/mediapipe/tasks/python/vision/image_segmenter.py", line 145, in __init__
    super(ImageSegmenter, self).__init__(
  File "/usr/local/lib/python3.11/dist-packages/mediapipe/tasks/python/vision/core/base_vision_task_api.py", line 70, in __init__
    self._runner = _TaskRunner.create(graph_config, packet_callback)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Service "kGpuService", required by node mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator, was not provided and cannot be created: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
'''

run the script in tensorflow:2.16.1-gpu-jupyter container
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

'''
>>> import tensorflow as tf
2024-06-25 03:53:58.671721: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
2024-06-25 03:54:00.062634: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-25 03:54:00.067000: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-25 03:54:00.067152: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
Num GPUs Available:  1
'''

[kuaashish]

Hi @SoaringTiger,

Thank you for bringing this to our attention again. It appears that our multi-class selfie segmentation currently does not support GPU usage. A similar issue was reported previously at #5081. Please allow us some time to investigate further, and I will provide an update soon.

Thank you!!