Possible to get image embeddings out directly?

[nabsiddiqui]

Hello, would it be possible to get the image embeddings out directly? Say for each image in a folder?

[frankiethull]

As of now, this package will take images and encode them before returning an LLM's response. By embeddings, can you give an example of what the use case is or results you are after?

For the ollamar-backend, looks like there's a few ways to creating embeddings, we could possibly support.

For the ellmer-backend, I think there will be additional packages for embedding and RAG, so TBD on this side.

Let me know if you have an example and we can test support on the ollamar side, or if you are willing to put a PR, I can review it.

[nabsiddiqui]

I'm mainly interested in basic things, such as clustering of the images and cosine similarity between the vectors.

[frankiethull]

I guess embedding text is supported, but support for embedding images is TBD with ollama:
ollama/ollama#5304

will bump & track this issue to include image embedding!

[nabsiddiqui]

Thank you. No rush. I just noticed there was a need for image embeddings in R and this seemed like it was already doing something in that area.

[jrosell]

I used ollamar embeddings in the past but ragnar can be used too for openai embeddings.

[jrosell]

I checked this today. I think we should use python sentence_transformer as a backend. This way it would be possible to use modles such as this from hughingface https://huggingface.co/sentence-transformers/clip-ViT-B-32

[frankiethull]

would it be possible to use chat_huggingface for CLIP? I guess I haven't setup a HF account to hit an API or tried downloading the models. I have wondered about downloading HF's models via mlverse's hfhub repo as GGUF and setup locally through Ollama. I haven't tried that but if it worked we could have a tutorial on registering models for use. I have been thinking about this quite a bit as a helper library.

I'm not sure about adding Python dependency directly to kuzco but have been thinking about that too. Had me thinking about helper packages for ragnar and reticulate, helpers named as kronk, pacha, or yzma, respectively. A as groovyverse, ... I day dream a lot 😆

[jrosell]

As far as I know, there's still no support for multimodal embedding models in ollama ollama/ollama#5304

But we can use @hoeflechner implementation of oclip models using a similar ollama embedding interface https://github.com/hoeflechner/oclip

Here an example in R:

git clone https://github.com/hoeflechner/oclip
cd oclip
docker compose up -d --build

library(httr2)
library(jsonlite)

model <- "hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K"
image_file <- system.file("img/test_img.jpg", package = "kuzco")

oclip_image_embed <- \(model = "hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K", image, truncate = TRUE, normalize = TRUE, keep_alive = NULL, 
    endpoint = "/api/embed", host = NULL, timeout = 600, ...) 
{
  if(length(image) != 1 && !file.exists(image)) stop("One image file is required")
   data <- list(list(model = model))
  data_json <- jsonlite::toJSON(data, auto_unbox = TRUE)
  tmp_json_file <- tempfile(fileext = ".json")
  writeLines(data_json, tmp_json_file)
  on.exit(unlink(tmp_json_file), add = TRUE) 
  resp <- request("http://localhost:11435/api/embed") |>
    req_body_multipart(
      image = curl::form_file(image_file, type = "application/octet-stream"),
      data  = curl::form_file(tmp_json_file, type = "application/json")
    ) |>
    req_timeout(timeout) |>
    req_perform()

  content <- resp_body_json(resp)
  image_vec <- content$embeddings |> unlist()
  if (normalize) {
    image_vec <- image_vec / sqrt(sum(image_vec^2))
  }
  image_vec
}

oclip_image_embed(
  image = image_file,
  host = "http://localhost:11435"
)
#>   [1]  1.754675e-02  3.227077e-03  1.212710e-01 -7.285717e-03  2.596920e-02
#>   [6]  1.129095e-01 -5.069486e-03  4.461453e-02  8.353780e-03  3.188932e-02
#>  [11]  6.902740e-02 -1.165715e-02 -8.849667e-03 -2.561826e-02  4.302769e-02
#>  [16]  5.071775e-02 -1.209658e-01 -4.537743e-02  3.726015e-02  1.664653e-02
#>  [21]  3.701984e-03 -1.487660e-02 -3.292686e-02  2.342110e-02 -5.683622e-03
#>  [26] -3.722963e-02  6.514471e-05  3.655828e-02  4.565207e-02  1.290068e-02
#>  [31] -1.795872e-02  1.644817e-02 -3.062290e-02 -2.232681e-04  7.266645e-03
#>  [36] -3.182828e-02 -2.702200e-02  6.063548e-02 -2.957009e-02  4.894018e-03
#>  [41]  4.382874e-03  1.023052e-02  4.046434e-02 -3.351048e-03 -6.127631e-02
#>  [46] -6.099404e-03  5.111446e-02 -2.952432e-02 -4.428648e-03  4.571311e-02
#>  [51]  1.296934e-03  3.680241e-02 -3.176725e-02 -8.184416e-02  3.973195e-02
#>  [56]  5.950638e-02  1.240479e-02  6.782202e-03 -5.264026e-02  4.595724e-02
#>  [61] -1.644817e-02 -1.001691e-02 -7.934184e-03  1.940824e-02  3.152312e-02
#>  [66]  2.642694e-02  2.064414e-02 -4.672014e-02 -1.332027e-02  1.518176e-02
#>  [71] -2.712881e-02  3.457473e-02 -3.320151e-02  1.398400e-02  2.146807e-02
#>  [76]  7.146869e-02 -4.092208e-02 -4.110518e-02  2.128497e-03  3.016516e-02
#>  [81]  4.791027e-02 -5.694303e-02 -3.913689e-03 -2.026269e-02  2.772387e-02
#>  [86] -3.472731e-02  8.231001e-05  5.969711e-03 -5.743129e-02 -4.080002e-02
#>  [91] -4.432462e-03  7.579435e-03  1.356440e-02 -1.719582e-02 -4.635395e-02
#>  [96] -3.503247e-02 -4.531640e-02  2.200973e-03  5.001588e-02  2.651848e-02
#> [101]  5.514258e-02 -5.697355e-02  1.040599e-02  7.834053e-04 -1.270232e-02
#> [106] -1.068063e-03  6.170354e-02  1.353694e-01 -1.882843e-02 -2.970742e-02
#> [111] -4.495020e-02 -1.481556e-02 -2.326852e-02  3.039689e-04  1.898101e-02
#> [116]  4.229530e-02  2.061362e-02  1.846224e-03 -9.696489e-03  5.532568e-02
#> [121]  1.413658e-02 -1.094002e-02 -2.192581e-02  1.820285e-02  3.542918e-02
#> [126] -2.432133e-02  4.861214e-02 -8.285119e-03  2.491639e-02 -9.048022e-03
#> [131]  1.060434e-02  3.222499e-02 -9.963504e-03  2.590816e-02  4.054826e-03
#> [136]  1.238524e-04 -1.048533e-01 -2.410771e-02 -6.374049e-03  6.621992e-02
#> [141] -5.373693e-04  9.490505e-03  6.451102e-02  1.771459e-02  3.127900e-03
#> [146] -4.363801e-03 -6.957669e-03  1.086373e-01  2.987525e-02 -1.597517e-02
#> [151]  4.263098e-02  4.812388e-02  2.941751e-01  2.870420e-03  2.125446e-02
#> [156] -2.094167e-03  7.316233e-03  5.564419e-04 -2.966164e-02 -5.563084e-02
#> [161] -3.295738e-02  8.861873e-02 -3.750142e-04  3.310996e-02 -4.531640e-03
#> [166]  1.014965e-01  1.505206e-02  2.078146e-02  1.461721e-02 -1.146642e-02
#> [171]  4.238685e-02  1.951123e-03  4.296666e-02 -6.414483e-02  2.601497e-02
#> [176]  9.765150e-02 -3.201138e-02 -1.470876e-02  8.275964e-02 -3.460525e-02
#> [181] -2.023217e-02  6.921050e-02 -8.933586e-03 -1.923467e-03 -3.701602e-02
#> [186]  2.444339e-02 -7.921978e-02  4.845956e-02  5.212149e-02  5.212149e-02
#> [191] -1.267944e-02  3.570383e-02 -4.855110e-02  1.144353e-02  3.564661e-03
#> [196] -3.393390e-02  2.480958e-02  4.168498e-02 -4.534691e-02 -2.231489e-03
#> [201]  2.848677e-02  5.749232e-02 -6.255799e-02 -2.462649e-02 -6.339718e-03
#> [206] -9.742263e-03  6.124580e-02  5.651580e-02 -8.918328e-03  4.161632e-03
#> [211] -4.870368e-02  7.000392e-02 -2.865461e-02 -4.818491e-02 -1.433493e-02
#> [216]  2.650323e-02  6.240541e-02 -2.993629e-02 -1.733314e-02 -1.049754e-01
#> [221] -8.155426e-03  7.830429e-02  4.330234e-02 -6.823398e-02 -3.143158e-02
#> [226]  4.186808e-02  2.278026e-02  1.318867e-03  1.600569e-02  1.502918e-02
#> [231] -3.799254e-02 -4.672014e-02  4.348543e-02  3.071445e-02  2.615229e-02
#> [236] -1.858430e-02  3.487989e-02  2.122394e-02  1.609724e-02 -3.345327e-03
#> [241] -3.064960e-03 -2.636590e-02  7.860946e-02  1.243531e-02 -1.096291e-02
#> [246]  3.817563e-02 -1.514361e-03 -9.360811e-03 -1.010693e-01 -5.642426e-02
#> [251] -1.496814e-02  1.759253e-02 -5.221303e-02 -2.720510e-02  1.698221e-02
#> [256] -4.986330e-02  7.122456e-02 -6.701334e-02  8.648261e-02 -6.573166e-02
#> [261]  4.195963e-02  8.178313e-03  2.625910e-02  2.917338e-02  3.381183e-02
#> [266] -4.040331e-02 -1.968288e-02 -2.145281e-02  8.318687e-02 -4.939793e-03
#> [271] -3.509351e-02 -2.729665e-02 -3.099291e-03  6.015485e-03  3.187024e-03
#> [276]  1.960659e-02  3.106538e-02 -2.615229e-02  8.910699e-03  8.092868e-02
#> [281] -2.656426e-02 -3.140106e-02 -1.574630e-02  1.855378e-02  3.396823e-03
#> [286]  1.274810e-02  3.445267e-02 -5.889606e-03  8.758119e-03  9.722427e-02
#> [291]  6.597579e-02  9.009876e-03  3.027196e-02 -2.906658e-02  2.667107e-02
#> [296]  6.904266e-03  2.311594e-02  1.611250e-02 -1.303037e-02 -3.887750e-02
#> [301]  8.849667e-02  6.789831e-03 -3.356770e-02 -2.590816e-02  2.043358e-01
#> [306] -2.820355e-04  2.509949e-02 -2.929545e-01 -6.005567e-02  1.125281e-03
#> [311]  2.366523e-02 -2.641168e-02 -1.454855e-02  1.658550e-02  3.689396e-02
#> [316] -2.831893e-02 -1.218355e-02  1.933004e-03  1.904395e-03  7.940287e-02
#> [321] -1.258026e-02  1.576156e-02 -9.551537e-03 -8.385822e-02  6.057444e-02
#> [326] -3.729067e-02 -4.403472e-02  6.377387e-05  2.633539e-02 -1.208437e-02
#> [331] -2.378729e-02 -1.262603e-03  1.776037e-02  1.089425e-02 -7.098043e-02
#> [336]  4.177653e-02 -2.686942e-02 -1.089425e-02  2.621332e-02  5.645477e-02
#> [341] -1.081033e-02 -3.530712e-02 -2.466082e-03 -4.360750e-02  6.420586e-02
#> [346] -9.411163e-02 -6.667766e-03 -6.817295e-02  8.404132e-02 -6.359554e-02
#> [351] -2.834945e-02 -2.786119e-02  2.656807e-03 -9.009876e-03 -7.464236e-02
#> [356]  2.696097e-02  2.078909e-03  3.101961e-02  2.245984e-02  7.726675e-02
#> [361]  2.621332e-02  1.508258e-02  7.220870e-03  2.830368e-02 -1.250397e-02
#> [366]  1.417473e-02  3.549022e-02  1.846224e-02  4.290563e-02 -3.411699e-02
#> [371] -2.781542e-02 -3.155364e-02 -2.734242e-02 -4.263098e-02 -9.240654e-04
#> [376]  3.545970e-02 -1.612776e-02  2.381781e-02  2.325326e-02 -3.509351e-02
#> [381]  4.543846e-02 -2.026269e-02 -4.977175e-02 -1.946927e-02  2.314646e-02
#> [386]  6.188664e-02  1.988123e-02  6.385492e-03 -9.078538e-03  4.330234e-02
#> [391]  9.849069e-03  8.529248e-03 -2.409246e-02 -5.502052e-02 -5.425761e-02
#> [396] -2.795274e-02 -7.806017e-02 -5.267078e-02  4.974123e-02  6.385492e-03
#> [401]  1.467824e-02 -6.351162e-03 -1.711953e-02  2.285655e-02  2.200210e-02
#> [406] -4.479762e-02  2.661003e-02 -2.488587e-02 -3.335409e-02  1.423576e-02
#> [411]  1.386956e-02 -5.996412e-02  3.120270e-02 -3.150787e-03 -2.610652e-02
#> [416] -5.935380e-03 -2.649178e-03 -2.998206e-02  1.837641e-03  9.124312e-03
#> [421]  3.101961e-02  8.338522e-03  3.108064e-02  2.808434e-04  1.233613e-02
#> [426] -4.607930e-02  7.159075e-02 -4.092208e-02 -2.404668e-02 -1.399926e-02
#> [431] -5.267078e-02  1.285338e-01  2.006433e-02 -2.346688e-02 -1.843172e-02
#> [436] -1.067300e-02 -3.939628e-02 -4.340914e-03 -8.039083e-04 -1.782140e-02
#> [441] -1.308377e-02  4.962680e-03  7.766346e-03 -5.065671e-03  1.080270e-02
#> [446] -4.302769e-02  4.687272e-02  5.013794e-02  2.487062e-02 -4.202066e-02
#> [451]  1.965236e-02 -1.235139e-02  6.014722e-02 -1.815708e-02 -3.616157e-02
#> [456]  4.229530e-02  6.860018e-02 -2.519103e-02  3.111116e-02 -7.873152e-03
#> [461] -9.679323e-04 -4.147137e-02  8.611641e-02 -4.604878e-02  1.391534e-02
#> [466]  2.596920e-02 -4.778820e-02  7.079734e-02  6.168065e-03  1.320584e-02
#> [471] -7.022516e-03 -1.631085e-02 -1.479268e-02  5.996412e-02  3.240809e-02
#> [476]  3.649725e-02  1.956082e-02 -3.396441e-02  5.575290e-02  6.310728e-02
#> [481] -2.990577e-03 -1.708901e-02 -2.799852e-02  1.018475e-02  4.525537e-02
#> [486]  2.030846e-02  2.743397e-02  2.281078e-02  1.969814e-02 -3.811460e-02
#> [491] -4.449246e-02  2.410771e-02  7.677849e-02 -6.416009e-03 -3.298790e-02
#> [496]  3.578012e-03 -2.871564e-02  5.752284e-02  4.455350e-02 -2.624384e-02
#> [501]  9.178669e-04  3.973195e-02 -9.177715e-03 -2.541991e-02 -4.073898e-02
#> [506]  5.618013e-02 -3.221355e-03  1.287779e-01  4.598775e-02  6.438896e-03
#> [511]  1.995752e-02  7.061424e-02

^{Created on 2025-08-26 with reprex v2.1.1.9000}

I saw:

• openai clip https://cookbook.openai.com/examples/custom_image_embedding_search
• nomic https://docs.nomic.ai/reference/api/embed-image-v-1-embedding-image-post
• google gemini mediapipe https://ai.google.dev/edge/mediapipe/solutions/vision/image_embedder/python
• google vertex https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings
• sentence-transformers https://koaning.github.io/embetter/API/multimodal/
• hf transformers on amazon sagemaker https://huggingface.co/google/vit-base-patch16-224-in21k?sagemaker_deploy=true
• open clip on hf spaces https://huggingface.co/spaces/altera2015/open_clip_v2/blob/main/main.py

@nabsiddiqui @frankiethull what do you think? What API interfaces could be interesting?

[nabsiddiqui]

Thanks for following up on this. Any are really fine. I’m mainly thinking of embedding images so that I can run cosine similarity on them, umap, etc. As long as the embedding is sufficient for this purpose, anything is fine. The current solutions I’ve found all use really old embeddings (inception v2), so all of these are sufficient

[jrosell]

Can you try my example? Do you use docker? It would be great to see a small repeoducible example for your use case, so that @frankiethull can decide if it's a good addition to {kuzco}.

[nabsiddiqui]

Just tested your example. It looks good to me! I think it would suffice for what I’m trying to do.

[frankiethull]

@jrosell - did you want to add this to {kuzco}? oclip_image_embed seems like a fine name to me. I haven't used embeddings before so I don't have a preference at the moment but it works for @nabsiddiqui so I am happy.