Hello everyone,
I have an issue with my desktop, which I really don’t understand.
It is a desktop PC, with a Nvidia GPU, which I use for development (of AI, but like the image analysis part, not the slop part). I usually let it always run, and my previous machine was working fine like this.
However, on this machine, I get crashes after either a few days or a few weeks.
These “crashes” are weird : the machine is still up (and I can ping it), however I can’t ssh into it, I can’t connect to cockpit either, and can’t login physically on it (the display is frozen).
The only way to get it back up is to forcefully shut it down, wait a little and then powr it back up; but this isn’t ideal, since it interrupts computations etc
The only crashes shown in journalctl are related to irrelevant (I believe) programs : KDEConnect, DrKronqi, etc; nothing right before the crash.
Do you have any idea what could cause this ?
I tried logging temperatures, but nothing abnormal showed up. I ran memtest86+, nothing showed up either. Smart tells me everything is alright for my drives as well.
Right now I’m thinking about changing the alimentation or the motherboard, but frankly I don’t know what’s going on and it’s really puzzling me.
Is it a Ryzen CPU? Do you have data fabric sync flood errors in the journal?
without knowing what you are running and your hardware, it’s hard to say. The symptoms sound similar to when I was running AI on a small server with 32gb ram only, and I hot oom, I’d be able to ping it but nothing else would work and would require a hard reset.
Well drkronqi is a crash handler, and KDE connect has its fingers in multiple pies? You say theyre not relevant but who do you know when the crash is, if you only notice it when you try to use the PC and its unresponsive? I wouldn’t dismiss the timing of these crashes if you can’t say exactly when it stopped functioning. They may even by symptoms if not the actual cause.
I’d start by increasing the logging levels to make them more detailed. You change the kernel and system logs to make them more detailed - how will differ slightly on your distro but essentially you want debug level system logs and lowest severity kernel logs. You can also setup some extra logging such as outputting system memory and vram usage to a dedicated file every second or so. This will.give you more information to hunt for the culprit.
As others have said, my first thought from your useage and description is a memory issue. That could be RAM or vram - especially with the AI useage. I’d start by logging both to their own files; out of memory can crash without much trace if the logs aren’t tracking the right things.
Also whatever process you are using - your AI tool for example - get that logging too.
Sounds like a long tedious elimination process is ahead.
But since the only errors you get regard KDE, I’d try installing another desktop environment. Afterwards you can generally choose which desktop to log into at the login screen. And when it’s resolved whether or not KDE is the problem, you can easily switch back if you want. So it’s very easy to try.
But each elimination will take 5 days!! If the computer behaves the same each time it crashes, it is definitely NOT random. If it was random it would be different things each time, and then it could be RAM or even CPU problems. But this sounds like a software problem.Thank you for your answer ! I’ll try it (nothing to lose), but more often than not I don’t login to the desktop; but maybe de login screen is enough ? I don’t know, I’ll try using gnome anyway.
I’ve had similar issues when I ran out of system memory. It also didn’t leave any traces in the journaling program, but I think it did say something about it in
dmesg. Not sure though. But since you’re probably using the VRAM and not the RAM, maybe this isn’t it at all? Just spitballing.I had a similar issue on bazzite and I think it had to do with the Out Of Memory service. This link might help if that’s it.
Disclaimer* if you look at my account I have mentioned that I only joined linux this year and have only distro hopped once so beyond this post I can’t really offer much assistance.
I did have an old laptop with bad SSD that would have similar problem - cannot ssh, but can ping, and desktop quickly gets unresponsive.
On root device issues kernel remounts root drive to read-only, and that breaks everything, but the machine still works - yet you get no logs because /var/log is also read-only.
I did some googling and it seems SSH requires writable /tmp, so you might setup some in-memory one, if you haven’t already - you could check if your machine uses tmpfs (in-memory tmp) to reject this hypothesis.
How much RAM/swap?
64G RAM, 8G Swap (i bought it about 2 years ago, and it’s worth a lot more now…)
i am going to follow this as i am having a similar issue (on multiple distros even and with nothing helpful in the journal either). i went as far as to replace the entire PC, suspecting a hardware issue (it was still under warranty) but the problem remained the same.
all i could find on the internet is it may be related to a combination of wayland and KDE plasma iirc. i deactivated stuff recommended by users and tried other suggestions but nothing ever resolved the issue. i have a feeling it has to do with certain updates after which the issue ebbs and flows but this might just be because the problem only occurs randomly in the first place, so this is an entirely “vibes based observation”.
my issue is definitely not an out of memory issue as i had btop open when it crashed and it showed memory usage of ~60%.Have you done a memtest? https://memtest.org/
