Qubes 4.1 Performance on Lenovo Ideapad 5 Unusable

[qubeslaptopissue]

How to file a helpful issue

Qubes OS release

4.1

Brief summary

I ran Qubes 4.0 on this laptop, which has 16GB of RAM and a Ryzen 7 4000 series for over a year. It not only ran good, it ran great. The performance was excellent with very little complaint. The other day, I installed 4.1 and have had countless issues with it, mostly performance related, sometimes it straight up doesn't work.

Steps to reproduce

Install Qubes 4.1 on laptop. restore from backup, Update qubes, install kernel-latest.

Expected behavior

I expect it to boot successfully, start the VMs that are supposed to start, take less than 5 minutes to boot. I expect it to work close to as good as 4.0 did. I expect that it is a usable operating system, not be extremely choppy.

Actual behavior

So first of all, half of the time it doesn't even boot or boots but none of the virtual machines start up and I cant open any applications. When it does boot, it takes at least 5 minutes after putting in the disk password to get to the login screen. I have timed it a couple of times. I am pretty sure that Qubes 4.1 does not need THAT much more computing power than 4.0. But sometimes, it boots and it actually works, maybe 60% of the time. Let's talk about what happens when it "works". It's extremely slow to the point where it's normally basically unusable. VMs will take minutes to start. The performance is extremely choppy. On 4.0, I can easily and smoothly drag a window across the screen. In Qubes 4.1, its like it chops across the screen, skipping places. I tried installing kernel-latest and updating the qubes. I tried booting from the other kernel option. I tried shutting down all of VMs and typing in dom0 and no improvement. It's still, text will take half a second or a few seconds to appear on screen after typing it, even in dom0. This exact same USB stick with the same iso was used to install 4.1 on my desktop and it works fine, so I know it cant be anything weird there. but the laptop is my main device, so I need it to actually work too. Another thing I tried was text mode during boot. It didn't seem like it gave much of anything useful and if it did it was gone quickly. For what it's worth, wifi seems to work fine despite needing to modify a small setting in 4.0 to make it work, it worked fine in 4.1.

I am at a loss for what to do. I have asked various people who may be knowledgeable. Someone suggested I post this here. For now, I have re-installed 4.0.4 on the laptop. If anyone can help me out or, rather, fix this bug whatever is causing it, I would greatly appreciate it, I need to get back to using my machine and need to upgrade before 4.0 is EOL. I'm already having to use Whonix anyways after its not supported for 4.0 anymore..

[DemiMarie]

What model of laptop is this?

[crat0z]

Most likely something like this on amazon?

For the first part: I have a very bad habit from the 3.2 days when Qubes would never properly shutdown and basically required holding down power button to turn off the PC. Recently, this has started causing problems with qubesd I think on startup, and the long boot time + no VMs started occurs. If you do something similar, I would suggest trying to break the habit :) Qubes can shutdown fairly quickly now.

Otherwise, try to update your UEFI. In #6055 there was a long discussion about Lenovo laptops and Renoir CPUs, which essentially boiled down to Lenovo shipping broken UEFIs. Otherwise, 4.1 should perform just as well as 4.0 did.

[DemiMarie]

For the first part: I have a very bad habit from the 3.2 days when Qubes would never properly shutdown and basically required holding down power button to turn off the PC. Recently, this has started causing problems with qubesd I think on startup, and the long boot time + no VMs started occurs. If you do something similar, I would suggest trying to break the habit :) Qubes can shutdown fairly quickly now.

Ugh, that might be my fault. Let me guess: qubesd is so slow to start that it eventually times out, causing VM start to fail, and repeated attempts to restart qubesd eventually succeed? If so, that’s the combination of QubesOS/qubes-core-admin#267 and QubesOS/qubes-core-admin#397, which could cause exactly the behavior you are seeing.

(Technical details: LVM ignores the devices/issue_discards option for thin, cache, mirror, and VDO devices. Because of this, QubesOS/qubes-core-admin#267 works around the problem by manually calling blkdiscard on the device before it is deallocated. It turns out that this is quite slow due to a kernel bug (#5426), which I suspect is why LVM does not do it automatically. This is merely annoying most of the time, but QubesOS/qubes-core-admin#397 fixed a nasty, long-standing bug in qubesd: it failed to reset storage in the event of an unclean shutdown. This is done by calling the same code used after a VM shutdown when the system is still running. If a single such call takes too long (more than 1 minute), qubesd startup could time out.

Otherwise, try to update your UEFI. In #6055 there was a long discussion about Lenovo laptops and Renoir CPUs, which essentially boiled down to Lenovo shipping broken UEFIs. Otherwise, 4.1 should perform just as well as 4.0 did.

Does the reported workaround (in the form of Xen command-line options) solve the problem? If so, @marmarek could the relevant changes in Xen 4.15 be backported?

[crat0z]

Yes, qubesd times out a bunch of times until it decides to work again. It could be related to those PRs, but I've only noticed the shutdown issue in last 2 months. Those PRs seem older.

IIRC, dom0_max_vcpus=1 dom0_vcpus_pin made Qubes usable as long as VMs didn't have more than 2-4 cores assigned, anything higher and performance would tank again. clocksource=tsc tsc=unstable hpetbroadcast=0 pretty much fixed the problem, but I don't know what that affects really otherwise... Either way, Lenovo frequently updates this lineup of laptops so I'm fairly confident a UEFI upgrade will fix this.

[qubeslaptopissue]

What model of laptop is this?

It's a Lenovo Ideapad 5. It looks like the one that person linked on Amazon but it was a bit more expensive (2020) and had more ssd space, so not sure if it's the same one.

I have typically not had issues shutting down Qubes in the past. I will look into the UFEI update but I'm not sure about that.

[DemiMarie]

Do the qubesd start problems go away if you use XFS or BTRFS (with the reflink driver) instead of LVM?

[qubeslaptopissue]

Should I maybe try to just reinstall? I'm doubtful it'll help but I'm not sure what to try here, if you're suggesting I do something with the above and not talking to the other user, then I'm not sure what.

[DemiMarie]

Should I maybe try to just reinstall? I'm doubtful it'll help but I'm not sure what to try here, if you're suggesting I do something with the above and not talking to the other user, then I'm not sure what.

Reinstalling with with XFS partitioning might help.

[qubeslaptopissue]

Should I maybe try to just reinstall? I'm doubtful it'll help but I'm not sure what to try here, if you're suggesting I do something with the above and not talking to the other user, then I'm not sure what.

Reinstalling with with XFS partitioning might help.

Not sure how to do that?

[DemiMarie]

Should I maybe try to just reinstall? I'm doubtful it'll help but I'm not sure what to try here, if you're suggesting I do something with the above and not talking to the other user, then I'm not sure what.

Reinstalling with with XFS partitioning might help.

Not sure how to do that?

I’m actually not sure either, at least not without looking at the installer.

[qubeslaptopissue]

Should I maybe try to just reinstall? I'm doubtful it'll help but I'm not sure what to try here, if you're suggesting I do something with the above and not talking to the other user, then I'm not sure what.

Reinstalling with with XFS partitioning might help.

Not sure how to do that?

I’m actually not sure either, at least not without looking at the installer.

I don't think I seen anything on the installer myself.

If anyone has anymore suggestions please tell.

[DemiMarie]

@fepitre can you help?

[ghost]

If anyone has anymore suggestions please tell.

take a look at this https://forum.qubes-os.org/t/qubes-os-installation-detached-encrypted-boot-and-header/6205 for picture, and this Steps to reproduce for xfs #7398

Not sure if using xfs could resolve the problem, I think if your bios can setting to use discrete gpu only, try choose that.

[qubeslaptopissue]

If anyone has anymore suggestions please tell.

take a look at this https://forum.qubes-os.org/t/qubes-os-installation-detached-encrypted-boot-and-header/6205 for picture, and this Steps to reproduce for xfs #7398

Not sure if using xfs could resolve the problem, I think if your bios can setting to use discrete gpu only, try choose that.

I will look into this ASAP first and then I will perhaps attempt with XFS. Hopefully something works because this is my full time device. Also does anyone have any tips for improving battery life? Probably not, but it feels kind of short on Qubes compared to how it should be for a fairly new laptop.

Can anyone even offer an explanation for why this is happening? Its such a drastic change in performance from 4.0 to 4.1 that it makes no sense to me.

Having a detached and encrypted boot partition is something that I wanted to do anyways, so this seems interesting.

[DemiMarie]

Also does anyone have any tips for improving battery life? Probably not, but it feels kind of short on Qubes compared to how it should be for a fairly new laptop.

That’s up to upstream Xen, sorry 😞.

[tzwcfq]

Also does anyone have any tips for improving battery life?

You can try different scaling governors. Maybe there will be difference for you:

xenpm get-cpufreq-para
xenpm set-scaling-governor ondemand
# or
xenpm set-scaling-governor powersave

Maybe disabling turbo mode:

xenpm disable-turbo-mode

UPD:
Also there seems to be no support for LFM (Low Frequency Mode). For my CPU Xen can only set cores to minimum frequency of 800 MHz and I couldn't find a way to enable LFM so I'll get lowest frequency of 400 MHz.
UPD2:
I've read some more about LFM and it seems that it's intended for thermal control to trottle CPU when it's overheating.
But in Windows LFM is used as default idle frequency. I've checked with bare-metal linux and it seems that linux don't support using LFM as minimum scaling frequency. So this is unrelated to Xen.

[qubeslaptopissue]

So after 4.1.1 released I decided to give it another try. It is now slightly more usable, but most users would still consider this to be unusable. I have pretty low standards. Lag and stuff has improved a little bit. But now half of the time the machine doesnt boot after GRUB, just nothing will happen, and about 25% of the time something goes wrong during the startup where my service qubes do not start up. This is very annoying and is making it very unreliable when most of the time my machine doesnt boot properly. The performance can still use some work, but I just wanted to update to say that the performance has improved but other issues have gotten worse even seemingly. The boot times are still occasionally about 5 minutes, but this has relaxed a bit too.

Please help anyone. Especially with this booting stuff.

Edited: Oh and I need to mention the shutdowns. They take way too long. What if I need to emergency shutdown for security? Or the shutdown time can really disrupt my workflow if I just need to restart. Sometimes they take 1-2 minutes, sometimes 5-10 minutes.

I bought this laptop SPECIFICALLY to use with Qubes. Is there any way we could just not EOL 4.0 until every device that 4.0 worked with works to the same standard as 4.1? I am wanting to downgrade back to 4.0 again because I'm not sure if I can use this. I can handle the poor performance on the desktop when using it, but I can't handle the 10 minute startups and shutdowns, especially when half the time the startups don't even do everything properly.

[andrewdavidwong]

Is there any way we could just not EOL 4.0 until every device that 4.0 worked with works to the same standard as 4.1?

I am truly sorry to hear about all the hardware troubles you're going through, but if we had done this for prior Qubes releases, we would still be on version 1.0. Our team is simply too small and our resources too limited to be able to support older releases indefinitely.

[DemiMarie]

Edited: Oh and I need to mention the shutdowns. They take way too long. What if I need to emergency shutdown for security? Or the shutdown time can really disrupt my workflow if I just need to restart. Sometimes they take 1-2 minutes, sometimes 5-10 minutes.

What happens if you do a hard reset?

[qubeslaptopissue]

Edited: Oh and I need to mention the shutdowns. They take way too long. What if I need to emergency shutdown for security? Or the shutdown time can really disrupt my workflow if I just need to restart. Sometimes they take 1-2 minutes, sometimes 5-10 minutes.

What happens if you do a hard reset?

Sorry, this is old kinda. But by a 'hard reset' do you mean just reinstalling the OS? I've done that. Doesn't make a difference.

[qubeslaptopissue]

Anyways, this issue is STILL happening but I have some updates
-performance has improved ever so slightly, but still practically at unusable levels
-all 3 options available in grub under advanced options have issues. the top one doesnt even boot at all, just a black screen, no passphrase prompt
-sometimes the system randomly starts shutting down
-sometimes it wont even shutdown. itll just randomly go black and then present me with my BIOS password prompt.
-there are occasional glitches
-boots are now usually more like 10-20 minutes instead of 5 minutes?!? wtf
-vms taking a few minutes to start sometimes

I have tried text mode, didn't see anything significant. I have kernel-latest installed, I have tried updating dom0 and all qubes. I have tried booting with no qubes that autostart and it still has these performance issues even in dom0 seemingly.
I have been trying to continue using it despite all of these issues but it's becoming pretty unbearable. I am at a loss for what to do. Why is 4.1 running so much worse than 4.0? I really just want to be able to reliably use my laptop again.

If anyone can suggest some troubleshooting steps? That would be nice. I will do anything at this point to just get a normally functioning computer!

oh and I wanted to include the detail that im pretty sure but not totally sure that the ufei version is the newest. .

[qubeslaptopissue]

At this point, I have been having the issue for 5 months. There has to be a solution at some point.

[github-actions]

This issue is being closed because:

• This issue is believed to affect only Qubes OS 4.1 (and possibly earlier).
• Qubes OS 4.1 has reached end-of-life (EOL).

If anyone believes that this issue should be reopened, please leave a comment saying so.
(For example, if a bug still affects Qubes OS 4.2, then the comment "Affects 4.2" will suffice.)