Refactor/online repacking

[Djip007]

goal: consolidation of the cpu backend for reintegration of the AMX backend.

• remove Q4_0_N_M from ggml file tensor type, only the cpu backend have know of this type and do dynamic repacking for it on "ggml_backend_cpu_aarch64_buffer_type" only.
• "extract" extra_buffer_type part (aarch64/hbm) and move most in there .cpp/.h files (migrate aarch64 to c++)
• get more general structure for "extra_op"
• get GGML_OP_MUL_MAT_ID for Q4_0_N_M working with dynamic repacking (aarch64_buffer)

[Djip007]

Not sure it is good, can not test.
And may not work/build on master branch either.

[slaren]

It could probably be removed, the normal CPU buffer type calls ggml_aligned_malloc which already uses HBM. So at the moment this buffer type serves no purpose.

[Djip007]

Look for me it have not be write for dynamic repack but only with "native" Q4_0_N_M packing.
leave it commented it need some work to be usable on dynamic repacking.

[slaren]

We should try to fix to keep support for aarch64 types with models with experts.

[slaren]

Overall looks good. I am not sure about removing support for current Q4_0_x_x models, but I guess if we are going to do it, it is better to do it sooner than later.

[Djip007]

I am not sure about removing support for current Q4_0_x_x models, but I guess if we are going to do it, it is better to do it sooner than later.

yes it will be the main/difficult choice :

• Allow weight repacking only at load time, and reduce the interest of mmap...
• Allow to add new "bloc" type... and be prepare to have lot of new type (AVX512 will like bloc of 16xN, AVX512BF16 of 2x16xN, AVX2 of 8xN, RDNA3 of 16x16 ...)

[Djip007]

@slaren I still need your expertise so as not to make too many mistakes.

I was looking for where params->wdata was created.

llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c

Line 7497 in 9336db4

char * wdata = params->wdata;

for me look to be in this function:

llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c

Lines 13220 to 13223 in 9336db4

	struct ggml_cplan ggml_graph_plan(
	const struct ggml_cgraph * cgraph,
	int n_threads,
	struct ggml_threadpool * threadpool) {

Am I right?

If yes, look for me that the size is not calculated correctly for llamafile and Q4_0 repacking:

• llamafile: we may compute size for src[1] that may not be used.
• Q4_0_M_N: may be compute with wrong 'vec_dot_type'

llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c

Lines 13277 to 13284 in 9336db4

	case GGML_OP_MUL_MAT:
	{
	const enum ggml_type vec_dot_type = type_traits_cpu[node->src[0]->type].vec_dot_type;
	if (node->src[1]->type != vec_dot_type) {
	cur = ggml_row_size(vec_dot_type, ggml_nelements(node->src[1]));
	}
	} break;

Note: I'm trying to make it more generic to make it easier to reintegrate the AMX backend so maybe not useful to fix it for now.

[ggerganov]

llamafile: we may compute size for src[1] that may not be used.

It's OK if we over-allocate a bit of memory for wdata even if it ends up not being needed. It would be best to add asserts in the different branches that validate wdata is big enough.

Q4_0_M_N: may be compute with wrong 'vec_dot_type'

Isn't vec_dot_type always GGML_TYPE_Q8_0 for the Q4_0_M_N?

[Djip007]

Q4_0_M_N: may be compute with wrong 'vec_dot_type'

Isn't vec_dot_type always GGML_TYPE_Q8_0 for the Q4_0_M_N?

Yes it is the case for Q4_0_M_N for, so not critical for now. Even if internally it is more a Q8_0_N:

llama.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.c

Line 195 in cce5a90

block_q8_0x4 * restrict y = (block_q8_0x4 *) vy;

But may not work with other/future case.

[slaren]

If we remove the old API and make the CPU backend accessible only through ggml-backend, then there will be a context that can be used to store the work buffer. Then the work buffer could simply be a std::vector in the context, and each operation that uses it only needs to resize it to the amount of memory it needs. Then we can remove ggml_cplan and related functions. However at this point this would break a lot of code.

[Djip007]

If we remove the old API and make the CPU backend accessible only through ggml-backend, then there will be a context that can be used to store the work buffer. Then the work buffer could simply be a std::vector in the context, and each operation that uses it only needs to resize it to the amount of memory it needs. Then we can remove ggml_cplan and related functions. However at this point this would break a lot of code.

So you confirm that for now this is where the size is calculated.

[slaren]

Yes, the size is calculated in the function ggml_graph_plan.

[Djip007]

can find how to enable c++17 for macOS-latest-swift / xcode...

[Djip007]

OK now this is merged #10570 , and c++17 is the default I need some more work.

[Djip007]

@slaren @ggerganov what do you think with this refactor?

I tried to make adding a "cpu-extra-buffer" simpler and more general. 🤞

[slaren]

The design looks good, this is a good improvement. Just reformat the code according to the .clang-format file and remove outdated comments.

[slaren]

Nice job, this is a good improvement.

[Djip007]

I updated the size controls, it should be better like this. 🖕

[ggerganov]

We should add debug logs about what repacks are applied to what tensors, so that when running with --log-verbose 1 we would be able to understand the conversions. Currently, it is quite difficult to trace.

[Djip007]

We should add debug logs about what repacks are applied to what tensors, so that when running with --log-verbose 1 we would be able to understand the conversions. Currently, it is quite difficult to trace.

@ggerganov: I added 2 logs, is that what you were thinking?

[ggerganov]

Yes, perfect. Should we merge this or is there anything else you are planning to do?

[Djip007]

Yes, perfect. Should we merge this or is there anything else you are planning to do?

For me we can merge it. (if the CI success 🤞)

[Djip007]

@slaren @ggerganov thanks for all you reviews and time.

[bartowski1182]

Just saw that opened bug, is the implication of this change that I should no longer be making Q4_0_N_M quants then? They seem to be fully removed?

[slaren]

Yes, support for Q4_0_N_M model files has been removed and cannot be made anymore.