New Feature: Vision Support for Khoj #889
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MythicalCow
added
upgrade
sabaimran
reviewed
Aug 15, 2024
src/interface/web/app/components/chatInputArea/chatInputArea.tsx
Outdated
Show resolved
Hide resolved
sabaimran
reviewed
Aug 16, 2024
debanjum
reviewed
Aug 16, 2024
|
please upgrade and merge to master asap |
…hatml construction
…images for pending msgs
debanjum
reviewed
Sep 6, 2024
There was a problem hiding this comment.
Tested out the new vision support. Feels neat to be able to talk about visuals with Khoj!
Some general feedback on UX and response generation:
- When requesting a vision enabled response from web app's home screen, the response generation UX doesn't make it obvious that Khoj understood and is responding to that image
- The image attached to the chat message doesn't seem to be passed to Khoj's train of thought? This would reduce response quality and make the train of thought UX confusing (as Khoj doesn't acknowledge its using the image to generate it's response)
sabaimran
reviewed
Sep 7, 2024
debanjum
approved these changes
Sep 7, 2024
This should speed up image loading and reduce storage costs
The image itself isn't strictly required to infer output mode and data
sources to reference to generate chat response
So only share placeholder text for attached image rather than the
actual image itself with those chat actors. This should reduce
response latency and consume less tokens.
- Minor
- Reorder passing uploaded_image_url arg closer to query for better
code readability
The /api/chat API endpoint has been updated to a POST endpoint from a GET endpoint to support passing attached images from client
- Align rendering uploaded images to previous HTML DOM structure of chat messages - Support adding image attached to chat message by user or khoj to clipboard
sabaimran
approved these changes
Sep 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✨ Summary of Changes
vision_enabledoption in server admin panel while configuring models👁️ Demo Images
🛠️ Feedback