text to image arraybuffer to imageData

[hpssjellis]

@coyotte508 @radames any suggestions for converting the text to image arrayBuffer to an imageData object. This does not seem to have the correct dimensions

      let UAC = new Uint8ClampedArray( objectDetectionRes, 768, 768);
      let imageData = new ImageData(UAC, 768, 768);

[coyotte508]

Hi @hpssjellis feel free to use discussions for usage questions. @TimMikeladze what do you think about returning a blob instead from @huggingface/inference, it will then contain the mime type of the image as well?

[coyotte508]

You need to use img and convert to a data url: https://stackoverflow.com/questions/8499633/how-to-display-base64-images-in-html

but for that you need to know the mime type (png or jpeg)

[radames]

right now @hpssjellis you need to manually set the mime type here is the full examples. But I guess it's a great idea to return the blob as well @coyotte508 !!

await hf
  .textToImage({
    inputs:
      "award winning high resolution photo of a giant tortoise/((ladybird)) hybrid, [trending on artstation]",
    negative_prompt: "blurry",
    model: "stabilityai/stable-diffusion-2"
  })
  .then((arrayBuffer) => {
    const image = new Image();
    const blob = new Blob([arrayBuffer], { type: "image/jpeg" });
    image.src = URL.createObjectURL(blob);
    return image;
  })

[coyotte508]

Alright @hpssjellis you can try the new version @huggingface/[email protected], a blob is directly returned!

You can also draw the img to a canvas if you want to get its ImageData;

[radames]

now you just need to remove the blob line

await hf
  .textToImage({
    inputs:
      "award winning high resolution photo of a giant tortoise/((ladybird)) hybrid, [trending on artstation]",
    negative_prompt: "blurry",
    model: "stabilityai/stable-diffusion-2"
  })
  .then((blob) => {
    const image = new Image();
    image.src = URL.createObjectURL(blob);
    return image;
  })

[hpssjellis]

Alright @hpssjellis you can try the new version @huggingface/[email protected], a blob is directly returned!

You can also draw the img to a canvas if you want to get its ImageData;

Working on this now. Thanks so much for making the new version. You are fast.

[hpssjellis]

Does anyone know if the generated arrayBuffer has an Alpha channel? I think it just has the RGB channels which is why imageData is unhappy. I could just inject an alpha of 255 for every pixel. Any opinions?

[coyotte508]

The arrayBuffer doesn't contain raw pixels, it contains a compressed jpg or png. You need to load it in an img or use a library like Jimp to decode it

[hpssjellis]

@coyotte508 @radames Amazing photo quality loads the turtle but strangely doesn't load anything else.

Demo here

[hpssjellis]

Actually I think it works fine. Just has a bit of a delay that I was too impatient to spot. Thanks for all the help. I only hve one more issue. I will pot it in the discussion.

[radames]

yes, it works for me. one reminder the API inference, a cache results based on the same input, and when it's new, it will take many seconds to load the model if it's not on memory yet, then run the inference