How could you initialize the model to use it in code?

[andycga]

Before Asking

• I have read the README carefully. 我已经仔细阅读了README上的操作指引。

• I want to train my custom dataset, and I have read the tutorials for training your custom data carefully and organize my dataset correctly; (FYI: We recommand you to apply the config files of xx_finetune.py.) 我想训练自定义数据集，我已经仔细阅读了训练自定义数据的教程，以及按照正确的目录结构存放数据集。（FYI: 我们推荐使用xx_finetune.py等配置文件训练自定义数据集。）

• I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking

• I have searched the YOLOv6 issues and found no similar questions.

Question

I trained the yolov6s_finetune.py model with a custom dataset and obtained the .pt and .onnx files. Now I want convert it in .mlmodel for using in xcode (iOs app). but, I need inizalice the model in a python file and I don't know how do this.

currently, Coremltools support PyTorch model file to convertion with this code:

PyTorch Conversion Workflow from coreml documetation

import torch
import torchvision

# Load a pre-trained version of MobileNetV2
torch_model = torchvision.models.mobilenet_v2(pretrained=True) #This is the model inicialization
# Set the model in evaluation mode.
torch_model.eval()

# Trace the model with random data.
example_input = torch.rand(1, 3, 224, 224) 
traced_model = torch.jit.trace(torch_model, example_input)
out = traced_model(example_input)

import coremltools as ct

# Using image_input in the inputs parameter:
# Convert to Core ML program using the Unified Conversion API.
model = ct.convert(
    traced_model,
    convert_to="mlprogram",
    inputs=[ct.TensorType(shape=example_input.shape)]
 )
model.save("newmodel.mlpackage")

the main problen is the load of pytorch model. I tried to replicate the code but it doesn't work.

import torch
import torchvision.transforms as transforms
import cv2
import coremltools as ct

from yolov6.utils.checkpoint import load_state_dict, load_checkpoint
from yolov6.utils.events import LOGGER, load_yaml
from yolov6.layers.common import DetectBackend
from yolov6.utils.nms import non_max_suppression
from yolov6.core.inferer import Inferer

weights = 'Model1.pt'
yaml = 'data/dataset_Custom.yaml'

image = cv2.imread('image2.jpeg')
#image = cv2.resize(image, (640,640))
transform = transforms.Compose([ transforms.ToTensor()]) 
img_tensor = transform(image) #3D tensor
img_tensor = img_tensor.unsqueeze(0)# 4D tensor #torch.Size([1, 3, 640, 640])


#---Init model
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = DetectBackend(weights, device=device) #####<<---- attempt to initialize the model
class_labels = load_yaml(yaml)['names']

def model_switch(model):
    # Model switch to deploy status 
    from yolov6.layers.common import RepVGGBlock
    for layer in model.modules():
        if isinstance(layer, RepVGGBlock):
            layer.switch_to_deploy()
        elif isinstance(layer, torch.nn.Upsample) and not hasattr(layer, 'recompute_scale_factor'):
            layer.recompute_scale_factor = None  # torch 1.11.0 compatibility

    LOGGER.info("Switch model to deploy modality.")


model_switch(model.model)

model.model.float() #cpu

model.eval()

model(image) #### error in this line ###

traced_model = torch.jit.trace(model, img_tensor)

#---Convert to coreml
scale = 1/(0.226*255.0)
bias = [- 0.485/(0.229) , - 0.456/(0.224), - 0.406/(0.225)]

image_input = ct.ImageType(shape=img_tensor.shape,
                           scale=scale, bias=bias)

ml_model = ct.convert(traced_model,source="pytorch",inputs=[image_input],
                      classifier_config = ct.ClassifierConfig(class_labels),convert_to='neuralnetwork')

ml_model.save("alphabetModel_image.mlmodel")

The error shown is the following:

Loading checkpoint from Model1.pt

Fusing model...
Switch model to deploy modality.
Traceback (most recent call last):
  File "/Users/.../YOLOv6/convert_to_coreml.py", line 45, in <module>
    model(image)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/.../Desktop/YOLOv6/yolov6/layers/common.py", line 563, in forward
    y, _ = self.model(im)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/.../Desktop/YOLOv6/yolov6/models/yolo.py", line 35, in forward
    x = self.backbone(x)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/.../Desktop/YOLOv6/yolov6/models/efficientrep.py", line 107, in forward
    x = self.stem(x)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/.../Desktop/YOLOv6/yolov6/layers/common.py", line 248, in forward
    return self.nonlinearity(self.se(self.rbr_reparam(inputs)))
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
TypeError: conv2d() received an invalid combination of arguments - got (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (numpy.ndarray, Parameter, Parameter, tuple of (int, int), tuple of (int, int), tuple of (int, int), int)
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (numpy.ndarray, Parameter, Parameter, tuple of (int, int), tuple of (int, int), tuple of (int, int), int)

I don't know what I'm doing wrong... I tried to initialize the model as done in infer.py or eval.py but it seems I'm not doing it right.

Could someone help me figure out what I'm doing wrong or if there's an easier way to do it? Any help is really appreciated, thanks.

Additional

No response