Recently I was investigating quite an interesting issue – there is Ubuntu based VM our testers run some tests on. It was reported they’re unable to log into the virtual machine. After a brief investigation it became clear the issue is not network or SSH key related.
These are records from auth log:
1 2 3 4 5 6
Quite an obscure error message, but smells like cgroup problem. Indeed:
1 2 3
Number of processes in the system was not this high, so, naturally the next guy to blame is the number of threads.
Pids of the most thread-heavy processes can be found using the following oneliner
1 2 3 4 5 6 7 8 9 10 11
First column here is a pid, second – number of threads.
Next it’s needed to find out what limit is being hit. Honestly it was quite a discovery for me that pids.max cgroup limit controls number of threads as well.
Limit set in the following file
Current usage can be found here:
user-1000 here is
ubuntu, confirmed by
As you can see the limit is almost exhausted. Once the limit was increased
it became possible to log in as user ‘ubuntu’. Tester folks were able to identify the reason of an excessive thread spawning and the issue should not reoccur.