add ggml_backend_sched_dump_dot

[foldl]

This PR add a DOT dump function to sched. Comparing to the existing ggml_graph_dump_dot, this function:

• backend and buft are color coded into the background of nodes.
• Shows graph splits.

A demo of the usefulness: In this graph, we can find there is something abnormal at a glance. This is caused by ggml_rms_norm_inplace for input layer norm, which is probably an error in the scheduler.

Note:

1. ggml_graph_get_grad is also fixed for cgraph->grads is NULL.
2. define GGML_DOT_FULL_COLOR for full color, otherwise, a color scheme is used.

PS: Someone might say that the dumped graph is too large to be rendered. In my case (chatllm.cpp), I use --layer_spec to load only 2 or 3 layers.

[slaren]

The discrepancy between buffer type and backend may be caused by a few reasons. ggml_backend_sched ignores non-executable view ops, so they may end with random assignments that are not relevant while running the graph. Copies of tensors in the splits do not have assignments in the hash table at all, since they are implicitly allocated in the split backend.

Fundamentally, ggml_backend_sched does not work on the graph directly, it works on the list of nodes that are the topological representation of the graph. Thus, I do not think that trying to reason about it by representing what it does as a graph is going to be useful. The best way to understand what ggml_backend_sched is doing is by looking at the list of nodes, which can be obtained by setting the environment variable GGML_SCHED_DEBUG to 2. I am afraid that this is just going to result to misunderstandings about what is actually happening in ggml_backend_sched, and lead to false bug reports like the supposed problem that you found.

[foldl]

Update:

1. Color is deduced from index;
2. Hue is used for GGML_DOT_FULL_COLOR.

[foldl]

... lead to false bug reports like the supposed problem that you found.

The issue is:

• ggml API ggml_backend_sched_set_tensor_backend is not used correctly, or
• xxx_inplace, view, or reshape operators are not handled properly.

I used this to identify the issue, updated my code (chatllm.cpp), and things worked.

This proposed function is not for debugging the scheduler, but for visualization of graph splits and backends. It may also help for debugging.

[foldl]

Maintainers are not interested in this, and I am going to close this.

PS: This exists in https://github.com/foldl/chatllm.cpp of cause.