Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generate alt-text with ai #2875

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Shinigami92
Copy link
Member

@Shinigami92 Shinigami92 commented May 27, 2024

This feature adds a Copilot-AI to the image-alt-text editor so it is possible to click a button that just generates an example text for the image.

Aufzeichnung.2024-05-27.211052.mp4

Open TODOs:

  • Style the button so it is inside the textarea in the upper right corner and looks like in VSCode commit title field
  • Add a warning to inform the user that this feature will download ~250 MiB into the browsers cache
    The download will only be done when the user confirms
  • Display error messages if something went wrong
  • props.attachment.url might not be usable due to cors error?!

@Shinigami92 Shinigami92 added c: feature Request for new feature p: 1-normal Nothing urgent labels May 27, 2024
@Shinigami92 Shinigami92 self-assigned this May 27, 2024
Copy link

stackblitz bot commented May 27, 2024

Review PR in StackBlitz Codeflow Run & review this pull request in StackBlitz Codeflow.

Copy link

netlify bot commented May 27, 2024

Deploy Preview for elk-docs canceled.

Name Link
🔨 Latest commit 7c26cec
🔍 Latest deploy log https://app.netlify.com/sites/elk-docs/deploys/665618557bb4a400084cd59c

Copy link

netlify bot commented May 27, 2024

Deploy Preview for elk-zone ready!

Name Link
🔨 Latest commit 7c26cec
🔍 Latest deploy log https://app.netlify.com/sites/elk-zone/deploys/66561855b07a38000869e101
😎 Deploy Preview https://deploy-preview-2875--elk-zone.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

socket-security bot commented May 27, 2024

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher

View full report↗︎

@Shinigami92 Shinigami92 force-pushed the feat-generate-alt-text-with-ai branch from a25aec8 to 61196ad Compare May 27, 2024 23:32
Copy link

socket-security bot commented May 27, 2024

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report↗︎

@Shinigami92 Shinigami92 force-pushed the feat-generate-alt-text-with-ai branch 6 times, most recently from 41f00a7 to 61d4fd2 Compare May 28, 2024 12:03
@Shinigami92
Copy link
Member Author

Shinigami92 commented May 28, 2024

I run out of ideas how to get the image (either as dataUrl or blob) as every attempt results in cors issues 🙁

... trying out: https://www.hacksoft.io/blog/handle-images-cors-error-in-chrome

@Shinigami92 Shinigami92 force-pushed the feat-generate-alt-text-with-ai branch 4 times, most recently from 5f5c7a4 to b57be9f Compare May 28, 2024 13:12
@Shinigami92 Shinigami92 force-pushed the feat-generate-alt-text-with-ai branch from b57be9f to 83bcf82 Compare May 28, 2024 13:26
@Shinigami92
Copy link
Member Author

It looks like the cors error is only on https://media.universeodon.com, but when I use my own account from mas.to I dont get a cors error on https://media.mas.to and it works 👀

@userquin
Copy link
Member

userquin commented May 28, 2024

how about using fetch with mode: 'no-cors' + blob response + RawImage.fromBlob? (I guess you only need to provide the image to the transformer)

@Shinigami92
Copy link
Member Author

how about using fetch with mode: 'no-cors' + blob response + RawImage.fromBlob? (I guess you only need to provide the image to the transformer)

Will try later, need some sleep 😴
I have jetlag 😅

@userquin
Copy link
Member

userquin commented May 28, 2024

NOTE: maybe we can store the image temporarily in cache storage (similar to the model) instead of directly uploading the image to the server (we would also use the image from the cache in the preview and the dialog), this way, CORS problem will disappear (we'll need to change the publish logic)

m.webtoo.ls also failing:

imagen

@Shinigami92
Copy link
Member Author

Just as a note for FFR: I still have the idea in my head, however I observe the evolution of window.ai that is currently in experimental in chrome (e.g. https://syntackle.com/blog/window-ai-in-chrome/)
However, this is only for text and does not allow image as input... at least now...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: feature Request for new feature p: 1-normal Nothing urgent
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

2 participants