optim on tinyblas #654

Djip007 · 2024-12-08T02:57:03Z

Djip007
Dec 8, 2024

ikawrakow/ik_llama.cpp#71 have a good idea.

I'll figure to add it in tinyblas and id work great. (and I add quant in FP16/BF16 in all case for B to reduce memory bandwidth. work nice for AVX512/AVX2 kernel)

https://github.com/Djip007/llama.cpp/blob/perfo/tinyblas/ggml/src/ggml-cpu/llamafile/sgemm.cpp#L297

Djip007 · 2024-12-08T03:12:28Z

Djip007
Dec 8, 2024
Author

[edit: 2024/12/09]: OK some correction and more tuning
some bench with Mistral-Nemo-Instruct-2407:

on ryzen 5950x (zen3) I get (from llama.cpp code):

with llamafile 0.8.17 vs this code

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp1	2.19	2.21
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp2	4.37	4.31
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp3	6.46	6.44
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp4	4.97	8.58
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp5	6.12	10.71
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp6	12.90	12.78
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp7	8.70	14.86
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp8	9.81	16.92
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp9	19.19	18.93
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp10	12.49	20.92
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp11	13.57	23.01
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp12	25.43	24.89
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp13	16.22	26.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp14	17.26	28.78
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp15	31.09	30.64
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp16	19.98	32.45
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp30	34.87	42.87
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp32	28.64	43.14
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp64	31.97	45.05
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp65	32.18	44.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp120	36.41	46.27
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp128	34.18	46.11
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp130	33.91	45.93
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp240	35.88	46.50
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp255	35.81	46.54
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp256	34.69	46.40
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp510	34.54	46.01
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp512	33.64	45.81
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp1024	33.36	45.05
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp1025	32.15	44.18
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	pp2048	32.76	43.68
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.BF16	tg128	2.19	2.21

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp1	2.19	2.19
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp2	4.37	4.28
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp3	6.44	6.41
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp4	4.90	8.53
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp5	6.19	10.64
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp6	12.83	12.71
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp7	8.54	14.81
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp8	9.85	16.88
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp9	19.10	18.90
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp10	12.24	20.88
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp11	13.53	22.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp12	25.38	24.89
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp13	16.00	26.87
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp14	17.27	28.74
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp15	31.60	30.66
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp16	19.67	32.55
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp30	39.28	54.23
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp32	30.30	55.37
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp64	34.77	58.46
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp65	35.08	57.47
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp120	40.33	58.01
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp128	37.34	57.76
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp130	37.36	58.28
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp240	40.43	58.66
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp255	40.45	58.62
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp256	38.88	58.57
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp510	39.67	57.94
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp512	38.94	57.51
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp1024	38.42	56.41
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp1025	37.90	53.96
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	pp2048	37.56	54.93
AMD R9 5950X 16-Core (znver3)	Mistral-Nemo-Instruct-2407.F16	tg128	2.19	2.19

Not the best that we can have with this CPU but we may need a true BLIS kernel for best (I think we can have ~80 t/s)
But that a "simple" change

on AMD Ryzen™ 9 7940HS (zen4)

with llamafile 0.8.17 vs this

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp1	2.50	2.51
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp2	4.98	4.74
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp3	4.02	7.23
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp4	9.91	9.91
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp5	12.33	12.37
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp6	7.91	14.78
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp7	9.26	17.09
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp8	7.30	19.41
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp9	11.78	21.63
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp10	24.02	24.02
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp11	14.41	26.30
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp12	15.67	28.43
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp13	11.70	29.41
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp14	18.10	31.56
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp15	34.31	33.51
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp16	20.57	35.87
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp30	50.37	51.09
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp32	36.26	51.04
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp64	50.78	57.23
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp65	54.97	57.33
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp120	61.28	65.51
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp128	47.93	65.81
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp130	61.80	66.31
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp240	64.80	70.40
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp255	64.09	70.32
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp256	60.79	70.34
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp510	59.59	68.26
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp512	56.96	67.97
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp1024	56.23	66.76
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp1025	55.21	64.84
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	pp2048	54.72	63.96
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.BF16	tg128	2.50	2.52

cpu_info	model_filename	test	0.8.17 t/s	new code
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp1	2.50	2.50
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp2	4.99	4.81
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp3	4.00	7.19
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp4	9.81	9.82
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp5	12.18	12.25
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp6	7.92	14.58
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp7	9.18	16.52
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp8	7.26	18.76
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp9	11.65	20.86
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp10	23.25	22.80
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp11	14.25	24.90
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp12	15.52	26.70
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp13	11.63	26.99
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp14	17.87	28.74
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp15	30.60	29.67
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp16	19.99	31.23
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp30	37.02	36.86
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp32	28.69	36.60
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp64	32.08	38.95
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp65	38.69	38.91
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp120	38.93	40.36
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp128	31.78	40.19
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp130	38.21	40.24
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp240	38.18	40.76
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp255	38.87	40.66
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp256	36.77	40.43
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp510	37.74	40.09
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp512	36.70	40.08
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp1024	36.44	39.03
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp1025	35.92	38.50
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	pp2048	35.78	37.95
AMD Ryzen 9 7940HS	Mistral-Nemo-Instruct-2407.F16	tg128	2.51	2.51

0 replies

Djip007 · 2024-12-08T03:13:08Z

Djip007
Dec 8, 2024
Author

@jart do you want I try it on llamafile ?

1 reply

Djip007 Dec 11, 2024
Author

#655 ...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

optim on tinyblas #654

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

optim on tinyblas #654

Djip007 Dec 8, 2024

Replies: 2 comments · 1 reply

Djip007 Dec 8, 2024 Author

Djip007 Dec 8, 2024 Author

Djip007 Dec 11, 2024 Author

Djip007
Dec 8, 2024

Replies: 2 comments 1 reply

Djip007
Dec 8, 2024
Author

Djip007
Dec 8, 2024
Author

Djip007 Dec 11, 2024
Author