model test request

[dnhkng]

Now that the llama.cpp server is running correctly, would it be possible to have this model tested?

https://huggingface.co/Infinimol/miiqu-gguf
using ChatML format and context length >= 1024, please :)

It is a model I been working on it for some time, and I think it's interesting. It is not a fine-tune, but a merge, and I find it consistently scores higher than the base model (miqu), which I think is a first for a pure merge model. Eq-bench runs in about 15 mins on an A100.

The model is GGUF, but split to fit under the 50Gb limit on Huggingface, but the model card give the one-liner to reassemble the file.

[sam-paech]

Very cool! I'll try to put it through its paces today.

Can I ask what the merge params were?

[sam-paech]

Btw are you able to upload the full weights? Lm-eval (what I use for the MAGI metric) is super slow through llama.cpp

[dnhkng]

No full weights. I figured out how to model merge directly from EXL2 to avoid requantization; this can also be done dynamically (this is my reddit post on it), but it needs a modified version of ooba. I am also working together with a llama.cpp developer on a PR for the same for GGUF.

I also don't have the full weights, because I tried over 6000 permutation, and without dynamic merges it would be months of compute 😅

This particular merge gives me an average score of almost 84, using dynamic merging of EXL2. The static merges of EXL2 and GGUF give ~38.4. Not sure why it's slightly lower, I need to look into it. But when I started building the GGUF merging pipeline, I discovered the backend issue #16

If full weights are needed, I can try and make some, but I think it will take a few hours time to download, merge and upload to Huggingface. Before I start that, would using the EXL2 version be faster? https://huggingface.co/Infinimol/miiqu-exl2

[sam-paech]

Fair enough. Oh, that's cool that you can do dynamic merges!

I don't think eleuther eval harness supports EXL2. With gguf it said it was going to take 73 hours (on a A100) so that's not gonna work :(

If you are able to make the full weights, I'm sure others would make use of them as well if your model gets popular.

[dnhkng]

OK, if it takes 73 hours, then I'll make them for you :)

Did you get the eq-eval test done?

[sam-paech]

Yeah it scored 83.17 with chatml via ooba. Do you think it would score higher if I used EXL2?

[dnhkng]

I'm still trying to figure out why I get differences.

I built my own evaluation system, and maybe its something useful for eq-bench v3.

It works like this:
when you have a rating from 1-10, the scores are a) discrete and b) easy to mess up by the LLM.

so, I use a descriptive rating, i.e. for your system:
0: does not experience the emotion
1: subconsciously experiences the emotion, but does not recognise it
2: start to feel the emotion, but not enough to change behaviour
...

Then, I run the LLM and only select the logits for the symbols 0-9. These 10 logit values can then be converted to probabilities that sum to 1 for just these values. Lastly, you can use these probabilities as weights to get continuous score.
i.e., say the model gave all zeros to the scores except for 50% for 5, and 50% for 6, then we get a score of 5.5
This has the effect of reducing the standard deviation, as the model would pick 5 half the time or 6 half the time.

[dnhkng]

Yeah it scored 83.17 with chatml via ooba. Do you think it would score higher if I used EXL2?

It might do, but its still lower than my dynamic merges. I will put my fork of exllamaV2 up, and add in the integration for EQ-Bench. Then you can test on that; it's all done, but I figured its too custom a case to make a PR.

The huge benefit is that it reuses GPU weights! i.e. if you can run a 70B model, you can also run the 120B self-merge, as the weights are reused :)

[sam-paech]

Oh I like that approach. Is that something you added to the eq-bench questions, or another test?

I did try converting the test to work with eleuther harness using logprobs evaluation, which is more or less what you are describing except the targets were just the raw numbers like [emotion]: 0, etc. It didn't work very well. Maybe the text explanation plus your score aggregation method would produce better results.

The huge benefit is that it reuses GPU weights! i.e. if you can run a 70B model, you can also run the 120B self-merge, as the weights are reused :)

That's pretty amazing. How far do you think this could be pushed, in terms of scaling up the layer reuse? If it only increases computation time of inference I would be curious to just crank up the layer count and see what happens.

[dnhkng]

Oh I like that approach. Is that something you added to the eq-bench questions, or another test?

This was for my own test, but it would be straightforward to add to eq-bench on backends that support logits.

I did try converting the test to work with eleuther harness using logprobs evaluation, which is more or less what you are describing except the targets were just the raw numbers like [emotion]: 0, etc. It didn't work very well. Maybe the text explanation plus your score aggregation method would produce better results.

Yes, this only works if you add a description of the values. Otherwise, the model doesn't know the ranges, and the score ranges get compressed.

The huge benefit is that it reuses GPU weights! i.e. if you can run a 70B model, you can also run the 120B self-merge, as the weights are reused :)

That's pretty amazing. How far do you think this could be pushed, in terms of scaling up the layer reuse? If it only increases computation time of inference I would be curious to just crank up the layer count and see what happens.

I've been trying this a lot, you generally can change creativity, but only a few merge variants (<0.1%) increased eq-bench scores so far.

[dnhkng]

@sqrkl OK, f16 is up now, at: https://huggingface.co/Infinimol/miiqu-f16

Very curious to see how it does at MAGI... Can we really make models smarter without fine tuning?

[sam-paech]

|Tasks|Version|Filter|n-shot| Metric |Value |   |Stderr|
|-----|------:|------|-----:|--------|-----:|---|-----:|
|magi |      1|none  |     0|acc     |0.6328|±  |0.0085|
|     |       |none  |     0|acc_norm|0.6322|±  |0.0085|

It seems at the very least you didn't make it less smart. Which seems to be a difficult thing with these frankenmerges. It's scored pretty much exactly the same as Miqu.

I ran it through the creative writing benchmark as well, it scored 65.5 compared to 69.5 for mistral-medium. It was a bit prone to hallucinations with some of the prompts which is probably why the lower score there.

[dnhkng]

I noticed you need to lower the temperature a lot to chat with it, but then it is seems much better than miqu.

My 'hands on' testing is staring with a character card in ooba, and getting the model to write both sides of the conversation.

With the model merges, I found it's necessary to raise minP and lower the temp to about 0.3.

Most model merges degenerate into giggling fits. One character makes a weird mistake (repeats itself or leaves out spaces between words), and the other character finds that so hilarious, they both end up laughing hysterically.

In some other merges, the characters starts alliterating or using huge streams of adjectives in every sentence (e.g. the gorgeous, greenish, graceful goose leapt loosely, lively, lovingly like the splendid, soulful shining sun...)

But with lower temperature, this model is much more creative than base Miqu, while staying in character. I will give you the best settings later today.

[sam-paech]

That's hilarious. Sounds like merging makes for an intoxicated miqu.

People seem to like these big merges for writing more than anything else (after all they tend to do worse on benchmarks). I'm hoping the creative writing benchmark can capture this thing that other benchmarks seem to miss.

[dnhkng]

That's hilarious. Sounds like merging makes for an intoxicated miqu.

Yes, it has a very 'stoned' vibe sometimes! Where the repeats come from in the model changes how the model behaves. The model infinimol/miiqu model has the least 'personality' change but improved the EQ_bench score. Other merges increased writing creativity. I plan on mixing the merges once I get some more compute.

I ran it through the creative writing benchmark as well, it scored 65.5 compared to 69.5 for mistral-medium. It was a bit prone to hallucinations with some of the prompts which is probably why the lower score there.

Yep, it needs a lower temperature to stay on track.

Will you do a write-up for MAGI? I don't plan on fine-tuning this model but only work with self-merges for the time being. So I would like to evaluate using MAGI before I submit another model.