Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interactivity Overhaul (User Interface & Model Instrumentation & Network Comms) #1054

Open
wants to merge 162 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 152 commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
fd00ab7
Added prototype for state changes.
nopdive Sep 20, 2024
5d032e1
WIP state nodes now update.
nopdive Sep 26, 2024
b7c15ff
Added working role to trace tree.
nopdive Sep 27, 2024
e6c8420
Clean-up of data classes for trace.
nopdive Sep 27, 2024
6429d97
Various trace updates.
nopdive Sep 27, 2024
e95d4f7
Clean-up of trace module.
nopdive Sep 27, 2024
b69582e
Revert some documentation word swaps.
nopdive Sep 27, 2024
6652b90
Adding basic HTML renderer. Adjustments to trace.
nopdive Sep 28, 2024
43820d6
Working basic visualization.
nopdive Sep 28, 2024
0dc1014
Black format on visual and trace module.
nopdive Sep 29, 2024
7ac9f97
Added client skeletal for Jupyter rendering.
nopdive Sep 30, 2024
017988a
Adjustments to clientside. Fixed tooling issue.
nopdive Sep 30, 2024
af90c77
Connected widget system to renderer.
nopdive Sep 30, 2024
1a51086
Added stitch package.
nopdive Sep 30, 2024
b20780d
Fix copyright for stitch package.
nopdive Sep 30, 2024
a8a0f06
Serialization for trace. Minor fixes.
nopdive Sep 30, 2024
a65379b
WIP messaging in client.
nopdive Sep 30, 2024
9f9dcd9
Resolved race condition for Jupyter client.
nopdive Oct 2, 2024
2c64bbc
Callback for client messages.
nopdive Oct 2, 2024
37be3a3
Asyncio connected for renderers.
nopdive Oct 3, 2024
5f80680
WIP final message.
nopdive Oct 3, 2024
55b9487
HTML renderer now supports detecting end of cell execution.
nopdive Oct 3, 2024
0f6b59f
De(serialization) of guidance messages.
nopdive Oct 4, 2024
626880e
visualize 3 different colors for input/generated/force-forwarded texts
lochuynh1412 Oct 7, 2024
deeaddf
indev: add probs to UI
lochuynh1412 Oct 7, 2024
ee28e74
indev : add token-based metrics - latency
lochuynh1412 Oct 7, 2024
f28d8cc
Renderer update can be called via recv_msg.
nopdive Oct 7, 2024
09fdfa8
Merge branch 'state' into state_visualization
JC1DA Oct 7, 2024
b94db9c
indev - showing prob working
JC1DA Oct 7, 2024
ae9c347
Fixed queue without loop arguments.
nopdive Oct 7, 2024
3d14adf
Use BaseGenToken to return top_k
JC1DA Oct 7, 2024
c673754
add flag to use legacy renderer even if stitch is available
JC1DA Oct 7, 2024
3e69d37
Change bytes into text in TokenGen
JC1DA Oct 7, 2024
a71ef4a
Add associated gentokens into VisChunk
JC1DA Oct 7, 2024
30c66f2
use GenToken for JupyterCellExecutionCompletedOutputMessage
JC1DA Oct 8, 2024
a600f1e
refactor EngineCallResponse and VisBytesChunk
JC1DA Oct 8, 2024
33be67a
indev - fix input backtrack
JC1DA Oct 8, 2024
ac6e4fd
working but incorrect probs
lochuynh1412 Oct 8, 2024
8a69714
use state for final tokenization
lochuynh1412 Oct 8, 2024
3b75361
Mocks introduced for instrumentation integration.
nopdive Oct 8, 2024
bbf492d
fix incorrect input type for softmax arg
lochuynh1412 Oct 8, 2024
970bdf7
WIP: return token metrics as JupyterCellExecutionCompletedOutputMessage
lochuynh1412 Oct 8, 2024
0b5af66
Refactors in client.
nopdive Oct 8, 2024
156b283
Merge pull request #1 from JC1DA/state_loc
nopdive Oct 8, 2024
d54a3aa
Minor clean-ups.
nopdive Oct 8, 2024
cc47c13
Fix incorrect masked probs calculation
JC1DA Oct 8, 2024
9cd68fd
Fix incorrect masked probs calculation
JC1DA Oct 8, 2024
2348dc1
Merge pull request #2 from JC1DA/state_loc
nopdive Oct 8, 2024
72eb0ac
Open roles now working in client.
nopdive Oct 8, 2024
6617615
Fix more-than-two-tokens need to be merged to single token in state s…
JC1DA Oct 8, 2024
e756383
Merge pull request #3 from JC1DA/state_loc
nopdive Oct 8, 2024
4080cfd
Mouse hover now works correctly on text.
nopdive Oct 9, 2024
a80d95a
Minor style change on token grid items.
nopdive Oct 9, 2024
6dc22c3
Tooltip & highlight groundwork done.
nopdive Oct 9, 2024
22080a0
return JupyterCellExecutionCompletedOutputMessage only if we can matc…
lochuynh1412 Oct 9, 2024
38276db
Merge pull request #4 from JC1DA/state_loc
nopdive Oct 9, 2024
4af0a04
Minor adj to model.
nopdive Oct 9, 2024
7c2927f
Merge branch 'state' of https://github.com/nopdive/guidance into state
nopdive Oct 9, 2024
19f8943
Additional changes.
nopdive Oct 9, 2024
61b4786
Optimize latency per token metric
lochuynh1412 Oct 9, 2024
146cfb9
Update per token latency metric
lochuynh1412 Oct 10, 2024
6141e37
Basic tooltip exposed.
nopdive Oct 10, 2024
a666cc4
Updated html bundle.
nopdive Oct 10, 2024
e3f0477
Add metric worker
JC1DA Oct 10, 2024
9b36c7f
Add basic metric worker
lochuynh1412 Oct 10, 2024
b5ee7a0
Styling with big font.
nopdive Oct 10, 2024
5ddd90d
UI works with minimal real-estate.
nopdive Oct 10, 2024
8dbd57e
Minimum widths for metric cards.
nopdive Oct 10, 2024
e155f86
return if the token was filtered out by the parser (is_masked)
lochuynh1412 Oct 10, 2024
184e0eb
Merge pull request #5 from JC1DA/state_loc
nopdive Oct 11, 2024
1afa737
Working tooltip.
nopdive Oct 11, 2024
76d11e6
Merge branch 'state' of https://github.com/nopdive/guidance into state
nopdive Oct 11, 2024
ddd1ba4
Text strikethrough on tooltip.
nopdive Oct 11, 2024
fa51189
Add consumed tokens / token reduction / avg latency metrics
lochuynh1412 Oct 14, 2024
f0142b7
Update token reduction
lochuynh1412 Oct 14, 2024
b2af267
rename metrics generator
lochuynh1412 Oct 14, 2024
afa202e
Merged state branch
lochuynh1412 Oct 14, 2024
a3bd85a
Add get_token_probs into llama_cpp
JC1DA Oct 14, 2024
252b97c
add workaround for engines that don't support get_token_probs
JC1DA Oct 14, 2024
734df5a
Refactor get token stats out of _msg_recv func
lochuynh1412 Oct 15, 2024
6c8ec65
set new ui as default
JC1DA Oct 15, 2024
8fed48a
Merge pull request #6 from JC1DA/state_loc
nopdive Oct 15, 2024
c29accc
Client-side clean-ups.
nopdive Oct 15, 2024
7267dc7
Merge branch 'state' of https://github.com/nopdive/guidance into state
nopdive Oct 15, 2024
bd88ede
refactor get_per_token_stats function
JC1DA Oct 15, 2024
153846e
Fix for client hang.
nopdive Oct 15, 2024
a24c73e
Client should be able to take array metric messages.
nopdive Oct 15, 2024
2205eca
Removed mock inputs.
nopdive Oct 15, 2024
11f4262
Improve visualization for generated tokens
JC1DA Oct 15, 2024
5482a09
Merge branch 'loc_state_2' into state_loc
JC1DA Oct 15, 2024
f5f22a6
Add warning if gpustat is not installed
JC1DA Oct 15, 2024
d8f431b
Remove unused fields in VisBytesChunk
lochuynh1412 Oct 16, 2024
a71aa3e
1) rename get_top_k_probs into get_per_token_topk_probs 2) support no…
lochuynh1412 Oct 16, 2024
aa2d0e4
refactor get_next_token_with_top_k in Engine class to generate only l…
lochuynh1412 Oct 16, 2024
d63879f
Refactor token in BaseGenToken into token_id
lochuynh1412 Oct 16, 2024
2bf8a40
Fix token not in GenToken
lochuynh1412 Oct 16, 2024
9b2cda4
refactor get_per_token_stats func
lochuynh1412 Oct 16, 2024
cab4588
support getting per_token_stats for Phi3
lochuynh1412 Oct 17, 2024
9a6dd7f
WIP updated style.
nopdive Oct 17, 2024
c726a44
Mock for new design.
nopdive Oct 17, 2024
2d6750f
fallback to get_next_token if get_logits is not implemented in get_ne…
JC1DA Oct 18, 2024
aebfdef
Merge BaseGenToken into GenToken
JC1DA Oct 18, 2024
3c5f15a
Reuse softmax function in utils
lochuynh1412 Oct 18, 2024
2904040
Add comments for get_per_token_stats function
lochuynh1412 Oct 18, 2024
67631c8
Merge pull request #7 from JC1DA/state_loc
nopdive Oct 19, 2024
0a039ea
Working select and hide for client.
nopdive Oct 19, 2024
b33e61a
Colors updating for logscale on client.
nopdive Oct 19, 2024
61650bc
Merge branch 'state' of https://github.com/nopdive/guidance into state
nopdive Oct 19, 2024
2f50e11
Add docstring for get_next_token_with_top_k in Engine class
lochuynh1412 Oct 20, 2024
7eda82e
Add mock functions for get_next_token_with_top_k and get_per_token_to…
lochuynh1412 Oct 20, 2024
2ec8b86
Renderer looked at. Many changes.
nopdive Oct 20, 2024
af7ba94
Refactor get_per_token_topk_probs to return list[GenToken] instead of…
lochuynh1412 Oct 21, 2024
9f06270
Merge branch 'state' into state_loc
lochuynh1412 Oct 21, 2024
8d540db
Memory clean up and tests for trace.
nopdive Oct 21, 2024
9b657ef
Merge pull request #8 from JC1DA/state_loc
nopdive Oct 21, 2024
c10c749
Various tests and documentation added.
nopdive Oct 21, 2024
772a0f7
Tests for visual component.
nopdive Oct 21, 2024
b209c72
Util visual methods exposed.
nopdive Oct 21, 2024
7044f34
Merge branch 'state' of https://github.com/nopdive/guidance into state
nopdive Oct 21, 2024
8753156
Code fixes for merge.
nopdive Oct 21, 2024
0aa68ff
Merge branch 'guidance-ai:main' into state
nopdive Oct 21, 2024
9ed5d56
Merge branch 'state' of https://github.com/nopdive/guidance into state
nopdive Oct 21, 2024
84f5547
Basic docs for client layer.
nopdive Oct 21, 2024
18a2e4a
Typo fix for stitch.
nopdive Oct 21, 2024
f470fea
Roles aren't working well in client. Removing special background color.
nopdive Oct 21, 2024
9784c17
Add logging message to metrics gen.
nopdive Oct 21, 2024
8c1fc19
Fix incorrect log for non-nvidia gpu
lochuynh1412 Oct 21, 2024
eb26583
Prevent empty token to be sent to UI
lochuynh1412 Oct 21, 2024
f56bb20
Merge pull request #9 from JC1DA/overhaul_loc
nopdive Oct 21, 2024
8f72fa9
Merge branch 'overhaul' of https://github.com/nopdive/guidance into o…
nopdive Oct 21, 2024
e18a025
Patch for phi-3 on client.
nopdive Oct 21, 2024
2b93a5c
Color fix.
nopdive Oct 21, 2024
87202d0
Missing token fix on client.
nopdive Oct 21, 2024
e16586b
Adjustments to defaults on client. Fix for non underline.
nopdive Oct 21, 2024
c6880b6
Minor fix for client.
nopdive Oct 21, 2024
b5af07c
Blue steeeel.
nopdive Oct 22, 2024
0e632bf
Undo accidental change on test.
nopdive Oct 22, 2024
4f9d80d
fix None metrics for cpu/mem
lochuynh1412 Oct 22, 2024
7f58ddb
Merge pull request #10 from JC1DA/overhaul_loc
nopdive Oct 22, 2024
20082e5
Extra logging and client recompile.
nopdive Oct 22, 2024
7a90a2b
Merge branch 'overhaul' of https://github.com/nopdive/guidance into o…
nopdive Oct 22, 2024
b60ea87
Switch to CPU usage temp.
nopdive Oct 22, 2024
4f41cca
Fix for client completion.
nopdive Oct 22, 2024
a1105c5
fake an EngineOutput for ByteParser
hudson-ai Oct 22, 2024
44e9b87
Merge pull request #11 from hudson-ai/overhaul_fix_ByteParser
nopdive Oct 22, 2024
330a5e9
Fix for terminals.
nopdive Oct 22, 2024
a914537
Merge branch 'overhaul' of https://github.com/nopdive/guidance into o…
nopdive Oct 22, 2024
1deb47b
Add psutil as required libary for basic metrics measurement
JC1DA Oct 22, 2024
d9b7c91
Merge pull request #12 from JC1DA/overhaul_loc
nopdive Oct 22, 2024
4c11c5a
Temp disable of metric generators.
nopdive Oct 23, 2024
5743909
Merge branch 'overhaul' of https://github.com/nopdive/guidance into o…
nopdive Oct 23, 2024
ca88399
Pass selected renderer type for autorenderer.
nopdive Oct 23, 2024
0398b19
Added new cleanup method for engine and model.
nopdive Oct 24, 2024
13b501e
Engine and model now GC without metric gen running.
nopdive Oct 24, 2024
7ee8432
Further GC fix and fix for role errors.
nopdive Oct 24, 2024
5ef5387
Re-implementation of renderer.
nopdive Oct 28, 2024
d19f27b
Fix for divergence method.
nopdive Oct 28, 2024
3ccbd04
Remove verbose logging from messaging layer.
nopdive Oct 29, 2024
3c44577
Removal of most references required for GC on renderer.
nopdive Oct 29, 2024
fde6400
Make sure text starts with BOS token before calling get_per_token_stats
lochuynh1412 Nov 1, 2024
8626137
Make sure text starts with BOS token before calling get_per_token_stats
lochuynh1412 Nov 1, 2024
08ef9d0
Merge pull request #13 from JC1DA/overhaul_loc
nopdive Nov 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ notebooks/local_scratch
__pycache__/
.vscode
.vs
.idea/
/build
/dist
*.egg-info
*.diskcache
.ipynb_checkpoints
node_modules
/client
.eggs/
.env
.DS_Store
Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include resources/graphpaper-inline.html
3 changes: 3 additions & 0 deletions client/graphpaper-inline/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules/
build/
.DS_Store
3 changes: 3 additions & 0 deletions client/graphpaper-inline/TODO.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
- Remove CDN font links (googlefonts)
- Image integration
- Testing
5 changes: 5 additions & 0 deletions client/graphpaper-inline/build-to-guidance.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/bin/bash
set -x

npm run build
cp dist/index.html ../../guidance/resources/graphpaper-inline.html
2 changes: 2 additions & 0 deletions client/graphpaper-inline/dist/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*
!.gitignore
40 changes: 40 additions & 0 deletions client/graphpaper-inline/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"name": "graphpaper",
"version": "0.0.1",
"scripts": {
"build": "rollup -c",
"dev": "rollup -c -w",
"start": "sirv dist"
},
"devDependencies": {
"@rollup/plugin-commonjs": "^26.0.1",
"@rollup/plugin-node-resolve": "^15.2.3",
"@rollup/plugin-terser": "^0.4.4",
"@rollup/plugin-typescript": "^11.1.6",
"@types/d3-scale": "^4.0.8",
"@types/d3-scale-chromatic": "^3.0.3",
"@types/dompurify": "^3.0.5",
"autoprefixer": "^10.4.20",
"cssnano": "^7.0.5",
"postcss": "^8.4.41",
"rollup": "^4.21.0",
"rollup-plugin-copy": "^3.5.0",
"rollup-plugin-html-bundle": "^0.0.3",
"rollup-plugin-livereload": "^2.0.5",
"rollup-plugin-postcss": "^4.0.2",
"rollup-plugin-serve": "^1.1.1",
"rollup-plugin-svelte": "^7.2.2",
"sirv-cli": "^2.0.2",
"svelte": "^4.2.18",
"svelte-preprocess": "^6.0.2",
"tailwindcss": "^3.4.10",
"tslib": "^2.6.3",
"typescript": "^5.5.4"
},
"dependencies": {
"d3-interpolate": "^3.0.1",
"d3-scale": "^4.0.2",
"d3-scale-chromatic": "^3.1.0",
"dompurify": "^3.1.7"
}
}
Loading
Loading