server/public_simplechat vision (basic ok), toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) #17142
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dont forget to map members of got entity from fetch to things from saved original promise, bcas remember what is got is a promise. also add some comments around certain decisions and needed exploration
A basic go at it
Declare the result of UrlReq as a DataClass, so that one doesnt goof up wrt updating and accessing members. Duplicate UrlRaw into UrlText, need to add Text extracting from html next for UrlText
As _UrlopenRet not exposed for use outside urllib, so decode and encode the data. Add skeleton to try get the html/xml tree top elements
As html can be malformed, xml ElementTree XMLParser cant handle the same properly, so switch to the HtmlParser helper class that is provided by python and try extend it. Currently a minimal skeleton to just start it out, which captures only the body contents.
First identify lines which have only whitespace and replace them with lines with only newline char in them. Next strip out adjacent lines, if they have only newlines
Ensures that if the url being requested as any query strings in them then things dont get messed up, when the url to get inc its query is extracted from the proxy request's query string
also now track header, footer and nav so that they arent captured
Add a new send headers common helper and use the same wrt the overridden send_error as well as do_OPTIONS This ensures that if there is any error during proxy opertions, the send_error propogates to the fetch from any browser properly without browser intercepting it with a CORS error
So that the same error path is used for logical error wrt http req also, without needing a different path for it. Dont forget to return the resp text/json/..., so that the contents are passed along the promise then chain
hanishkvc
changed the title
hanishkvc
changed the title
Had forgotten to move the one shot resp into try catch before. Fixed it. Ensure both oneshot multipart resp try catch Add some todos for later. Add a new check wrt response being normal or a error related one ie the content is actually a error message.
Trap the mouseenter and mouseleave events wrt the chatmessage block to show the chatmessage-popover ui inturn within the corresponding chatmessage. When ever user clicks the popover menu's delete button, the uniqId associated with each chat message, is now used to delete the user selected chat message. Initial skeleton wrt clipboard copy.
Use position fixed followed by using position-area so that one can use anchor passed through showPopover. Avoid the hidePopover from mouseLeave, because as the mouse moves over the poped up popover ui, the mouseLeave will be called on the chat message block/container. And that would have closed the popover and inturn mouseEnter would have got called on ChatMsg, given now with the popover hidden, the ChatMsg block is directly accessible. So this leads to twinkling / toggling ie continues show/hide of the popover. Inturn in mouseEnter itself decide when to close the previous popover (ie if the chatmessage id is different from what it was before).
As position-area is not yet officially supported in firefox (its only in nightly builds, as of now), so switching to the inset block/inline start/end css properties.
Currently only the text/content part is handled wrt copy. Image URL isnt handled, even if present.
Rename to get_chatmessage_index. Return the index of the chatmessage deleted. Add a scroll element into view helper. Now the UniqId wrt the ChatMessage is stored as a attribute wrt the corresponding chatmessage block. And same used wrt scrolling after deleting messages. When a chat message is deleted, previously the screen would have moved to last message in the chat session, now it tries to remain wrt adjacent messages on the screen.
This ensures that the tool call edit/trigger is reshown, when user submits a empty response and the logic stops from going ahead.
avoid refreshing the full chat session messages ui also Update readme
If deleting a non last (and non just before last) message, then just directly remove the corresponding chat message block from the ChatSession UI and be done with it. However if deleting the last (or just before last) message, then one needs to decide on whether tool call edit/trigger ui is shown or removed and so... and similarly wrt tool response edit/submit.
Make the toolresp non happy path use a direct insert chat msg wrt ui, rather than recreating the full chat session ui fresh.
Rename lastMsg to prevLastMsg to match semantic better Return seperate flags to indicate add and show success or not Update tool response happy path to use this optimised add show instead of chat_show (which recreates the full chat session ui).
Now even the user response handling flow (handle_user_submit) has been updated to use the optimised chat show logic which adjusts / updates the chat session ui wrt the last two messages only and not the full chat history in the chat session. Move the logic wrt last two messages and ui, into its own func and inturn use it everywhere as required.
Needed as some places we require to delete the chat message as well as update the corresponding ui block. And in other places we require to just remove the ui block from chat session ui and not the underlying chat message from chat history. Change ui updateprev appendlast to use the ui remove logic.
SC.add discards any temp role message and not just tool temp msg New SC.add_smart which either adds a new message to the chat session or replaces existing last message in the chat session, based on same role or different, given that the chat session cant have the same role type chat message occuring adjacent in it. Inturn rename MCUI chatmsg_add_uishow to chatmsg_addsmart_uishow and use add_smart * helps wrt trying to rerun a tool call with modified args or so. Rather prev discard temp role messages in SC.add not good enough as uniqId will change in its case. Helps avoid adding duplicate ToolTemp messages in chat session ui * can help if loading a prev chat session which ended in a user message without a ai response. User can type in a new message and continue that old chat session, with the new message, replacing the old user message as well as initiating handshake with ai server in a proper manner. Replace MCUI chatmsg_ui_updateprev_appendlast with chatmsg_ui_refresh which is a more generic flow, which takes care of updating the ui as needed irrespective of if the specified set of messages are already in the chat session ui or not. Also allows the caller to control how many messages at the end need to be refreshed wrt ui.
Rename chatmsg_ui_refresh to chat_uirefresh. Even in case of the delete path and inturn deleting one of the last two messages in a chat session, now use the generified chat uirefresh logic instead of the chat_show full session refresh / recreation of the session ui Inturn to make the uirefresh really generic and usable in all cases including above case, now take care of clearing the tool call edit / trigger at the beginning, so that the last 2 messages decide in turn whether to show a tool call edit/trigger ui or not, as well as the tool call response edit / submit ui.
also update some notes
Demote the Popover related mouse event trapping related logs to debug level, as they are no longer required. Avoid using brute force chat_show instead use only the optimal chat_uirefresh wrt all paths in handle_user_submit, rather had forgotten to change previously.
Avoid chat_show getting called twice during starting. Update description of sys_date_time to be more verbose Update the readme DONE section, where I had few days back forgotten and started making entries as parts of a item list or so.
ui_reset_userinput now resets role associated with user input to user always. chat_show now does a full on ui_reset_userinput, and thus inturn session switching now force cleans up User Input area, state and associated data (like dataURLs) with chat_show->ui_reset_userinput. UIRefresh helper does a partial ui_reset_userinput thus cleaning up the user input state. Inturn given that addsmart_uishow calls uirefresh, so avoid calling partial ui_reset_userinput after call to addsmart_uishow. show_message now sets user input area role to either ToolTemp or User (all cases other than ToolTemp).
Contributor
Author
|
Update chat session ui selectively, instead of recreating the full chat session ui elements always changes are made to chat session. Allow user to delete any chat message in the session, as well as copy the text content associated with the chat message. Other cleanups, finetuning and related refactoring... Some of these also allow for a minimal simple minded markdown to html conversion, if needed later and few other stuff... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the initial basic skeleton for supporting vision models going forward to the tools/server/public_simplechat alternate web client ui. Basic tested with Gemma3 and Qwen3VL
This builds on the previous PR in this series ie #17038 which completed the current initial go at tool calling which also includes a set of client side based builtin and bundled tool calls for code execution, data storage, web access, web search, pdf and xml use, ... as well as support for looking into reasoning/chainofthought of the models which share the same.