Solved Reserve 2 cores for shell, sh from bootup

Hi,

I have 8 cores and I'm trying to reserve the first 2 for admin access and other important tasks. I found the cpuset, so I built 2 cpu sets.

I have no issues tying the sh and ssh to the first 2 cores however when I try to allocate the rest of the PIDs to the rest of the cores I get an error message :

cpuset: setaffinity: Resource deadlock avoided

The commands that I'm using :
cpuset -c -l 0-1 /bin/sh
cpuset -C -c -l 2-7 -p 0-99999

Any idea how I can do this ? Also, is it possible to implement the cpuset commands at boot time?

Thanks,
Ionut
 
cpuset():
Code:
Every process in the system belongs to a cpuset.  By default
     processes are started in set 1.
Code:
Modify the cpuset all threads are in by default to contain only the first
     4 CPUs, leaving the rest idle:
           cpuset -l 0-3 -s 1
So, for your use case: cpuset -l 2-7 -s 1 will assign all pids (0-99999) to set 1 using only CPUs 2-7. And your other command will assign /bin/sh to CPUs 0,1.

This is just a guess based off the text in the manpage and a few quick tests from the command line. I've never used cpuset before.

I don't see any mention of PID ranges in the manpage or examples, so it's not likely an available feature.
 
Thanks for your answer.

cpuset -l 2-7 -s 1 doesn't work for me : Resource deadlock avoided.

It work only if I include the first cores, I imagine it is because it can't transfer all the processes at the same time?
Sadly, I need to leave the first 2 cores open because of my business needs, otherwise I would be easily able to isolate the last 2 cores of my processor.
 
I imagine it is because it can't transfer all the processes at the same time?

I guess the "Resource deadlock avoided" could suggest that. What happens if you drop into single-user mode with very few processes running and try it again?

It might instead be that you can't de-link the first few cores from the default set.
 
You could try this:

# ps -A -o pid= | xargs -n 1 cpuset -l 2-7 -p

After this, all running PIDs should have a cpuset of 2-7.

You can then start a new shell on 0,1 with exec cpuset -l 0,1 sh, for example.

Be sure that cpuset -gr shows you 0-7 like you expect, too.
 
I guess the "Resource deadlock avoided" could suggest that. What happens if you drop into single-user mode with very few processes running and try it again?

It might instead be that you can't de-link the first few cores from the default set.

No difference for single user.

You could try this:

# ps -A -o pid= | xargs -n 1 cpuset -l 2-7 -p

After this, all running PIDs should have a cpuset of 2-7.

You can then start a new shell on 0,1 with exec cpuset -l 0,1 sh, for example.

Be sure that cpuset -gr shows you 0-7 like you expect, too.

I get the same resource deadlock error when I try ps -A -o pid= | xargs -n 1 cpuset -l 2-7 -p.
I also tried adding /usr/bin/cpuset -l 3-7 -s 1 at boot in crontab and in rc.conf, I get the same error in the logs.
 
I finally made it work, thanks to everyone for your suggestions, without them I wouldn't had advanced so fast.

For those interested, continue reading:
In crontab I added the following command @boot

ps -A -o pid=| xargs -n 1 echo cpuset -l 2-7 -p => all processes go to cores 2-7.
cpuset -l 2-7 -s 1 => /usr/bin/cpuset 1 is now 2-7, all new processes will go to these cores .
If I just cpuset without passing the processes to the cores 2-7 first, I get the the "resource deadlock error" in the var/log/cron.

Now I can create new cpusets in shell and allocate /bin/sh , sshd to them.

cpuset -C -l 0-1 /bin/sh etc.

Sorry for my English.
 
Looks like you already have it working but my suggestion would of been to try limiting the system to 0-5, then assign 6 & 7 to just your admin stuff. The first cores are likely to be the most active so restricting the entire system to 2-7 probably means it's having to try and move a huge number of threads around.
 
cpuset: setaffinity: Resource deadlock avoided

Yes, I got the same error when trying as suggested here.

cpuset -l 0-1 -s 1 should restrict the whole currently running system onto a limited set of cores - but that does not work. I tried to figure out why:
  • If you tell a cpuset to use only cpu 1 while it contains a thread that in already restricted to cpu 2, that must fail - and that's when this error appears.
  • When restricting all processes to a given number of cpus, this includes also pid 0, i.e. the kernel.
  • Within the kernel there are some threads which do already use cpusets and are restricted to a specific core. And so it fails.
On my system I found the if_io_tqg and softirq threads to be spread over individual cpus. The former seems to belong to iflib(), the latter to softirq tasklets for the linux emulation. (Check with procstat -a -S.)

ps -A -o pid=| xargs -n 1 echo cpuset -l 2-7 -p => all processes go to cores 2-7.
cpuset -l 2-7 -s 1 => /usr/bin/cpuset 1 is now 2-7, all new processes will go to these cores .
If I just cpuset without passing the processes to the cores 2-7 first, I get the the "resource deadlock error" in the var/log/cron.

Yes, this works, but it would be enough to do it only for pid 0 instead of all pids:
Code:
cpuset -l 2-7 -p 0
cpuset -l 2-7 -s 1
Caveat: hereby we destroy the specific function of beforementioned special kernel threads.

A more clean approach would be to move the kernel stuff out of the way unmodified:
Code:
cpuset -C -p 0
cpuset -l 2-7 -s 1
In this case the kernel threads will still be distributed over all cpus.
 
Back
Top