Linux-VServer - User contributions [en]

Memory Limits

2010-05-27T20:18:14Z

Petzsch: added note about cgroups

== Overview ==
A vserver kernel keeps track many resources used by each guest (context). Some of these relate to memory usage by the guest. You can place limits on these resources to prevent guests from using all the host memory and making the host unusable.

Two resources are particularly important in this regard:

* The '''Resident Set Size''' (<code>rss</code>) is the amount of pages currently present in RAM.
* The '''Address Space''' (<code>as</code>) is the total amount of memory (pages) mapped in all the processes in the context.

Both are measured in '''pages''', which are 4 kB each on Intel machines (i386). So a value of 200000 means a limit of 800,000 kB, a little less than 800 MB.

''To easily find out the page size on your host try this line and ignore the warnings''
<pre>echo 'int main () { printf ("%dKiB\n", getpagesize ()/1024); return 0; }' | gcc -xc - -o getpagesize && ./getpagesize</pre>
Each resource has a '''soft''' and a '''hard limit'''.

* If a guest exceeds the <code>rss</code> hard limit, the kernel will invoke the Out-of-Memory (OOM) killer to kill some process in the guest.
* The <code>rss</code> soft limit is shown inside the guest as the maximum available memory. If a guest exceeds the <code>rss</code> soft limit, it will get an extra "bonus" for the OOM killer (proportional to the oversize).
* If a guest exceeds the <code>as</code> hard limit, memory allocation attempts will return an error, but no process is killed.
* The <code>as</code> soft limit is not in utilized until now. In the future it may be used to penalizing guests over that limit or it could be used to force swapping on them and such ...

Bertl explained the difference between '''rss''' and '''as''' with the following example. If two processes share 100 MB of memory, then only 100 MB worth of virtual memory pages can be used at most, so the RSS use of the guest increases by 100 MB. However, two processes are using it, so the AS use increases by 200 MB.

This makes me think that limiting AS is less useful than limiting RSS, since it doesn't directly reflect real, limited resources (RAM and swap) on the host, that deprive other virtual machines of those resources. Bertl says that AS limits can be used to give guests a "gentle" warning that they are running out of memory, but I don't know how much more gentle it is, or how to set it accurately.

For example, 100 processes each mapping a 100 MB file would consume a total of 10 GB of address space (AS), but no more than 100 MB of resources on the host. But if you set the AS limit to 10 GB, then it will not stop one process from allocating 4 GB of RAM, which could kill the host or result in that process being killed by the OOM killer.

this is posted by siddharth

== Setting memory limits ==
'''Beginning with vs2.3.0.36.29 you should use cgroups to set memory limits. More information: [[util-vserver:Cgroups]]'''

You can set the hard limit on a particular context, effective immediately, with this command:

<pre>
/usr/sbin/vlimit -c <xid> --<resource> <value>
</pre>

<code><xid></code> is the context ID of the guest, which you can determine with the <code>/usr/sbin/vserver-stat</code> command.

For example, if you want to change the '''rss''' hard limit for the vserver with <code><xid></code> 49000, and limit it to 10,000 pages (40 MB), you could use this command:

<pre>
/usr/sbin/vlimit -c 49000 --rss 10000
</pre>

You can change the soft limit instead by adding the <code>-S</code> parameter.

Changes made with the vlimit command are effective only until the vserver is stopped. To make permanent changes, write the value to this file:

<pre>
/etc/vservers/<name>/rlimits/<resource>.hard
</pre>

To set a soft limit, use the same file name with the <code>.soft</code> extension. The <code>rlimits</code> directory is not created by default, so you may need to create it yourself.

If you omit the suffix after the <code>/etc/vservers/<name>/rlimits/rss</code> file, the value will be set for both, the hard and soft limit.

Changes to these files take effect only when the vserver is started. To make immediate and permanent changes to a running vserver, you need to run <code>vlimit</code> '''and''' update the rlimits file.

The safest setting, to prevent any guest from interfering with any other, is to set the total of all RSS hard limits (across all running guests) to be less than the total virtual memory (RAM and swap) on the host. It should be sufficiently less to leave room for processes running on the host, and some disk cache, perhaps 100 MB.

However, this is very conservative, since it assumes the worst case where all guests are using the maximum amount of memory at one time. In practice, you can usually get away with contended resources, i.e. allowing guests to use more than this value.

== Displaying current memory limits ==
To display the currently active RSS limits for a vserver execute the following command:
<pre>
vlimit -c <xid> -a -d | grep RSS
</pre>

The above command will display a similar result, whereas the third value (5000) is the soft limit and the last reflects the current hard limit (10000).
<pre>
RSS N/A 5000 10000
</pre>

== Display memory limits within a vserver ==
Normally the <code>top</code> and <code>free</code> command will display the RAM and Swap usage of the host while invoked within a vserver. To change this behavior, add the <code>VIRT_MEM</code> [[Capabilities_and_Flags#Context_flags_.28cflags.29|context flag]] to your vserver configuration:
<pre>
echo "VIRT_MEM" >> /etc/vservers/<name>/flags
</pre>

After a successful restart of the related vserver, the total available RAM will equal to the value of <code>rss.soft</code> while the difference of <code>rss.hard</code> - <code>rss.soft</code> will be displayed as swap space.

As an example, if you set the <code>rss.hard</code> limit to 10'000 pages and the <code>rss.soft</code> limit to 7'000 pages:
<pre>
vlimit -c <xid> --rss 10000
vlimit -c <xid> -S --rss 7000
vlimit -c <xid> -a -d | grep RSS
</pre>
<pre>
RSS N/A 7000 10000
</pre>

<code>free</code> will report 28'000 KB (7'000 pages) of total memory and 12'000 KB (10'000 - 7'000 pages) of Swap space (assuming that one page is 4 KB):
<pre>
free -k
</pre>
<pre>
total used free shared buffers cached
Mem: 28000 4396 23604 0 0 0
-/+ buffers/cache: 4396 23604
Swap: 12000 0 12000
</pre>

'''Note:''' According to Herbert, the kernel won't use any ''real'' swap as soon as the <code>rss.soft</code> limit has been reached. Swapping will be done on the host level, not per vserver (see [http://list.linux-vserver.org/archive?mss:653:200801:iiagcodpghkhehndkgjg "free" command inside vserver]).

util-vserver:Cgroups

2010-02-19T12:52:19Z

Petzsch: /* using cgroup to enforce memory limits */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you need to have <tt>/dev/cgroup</tt> mounted.

Recent versions of <tt>util-vserver</tt> sort this out for you by including the appropriate mount command in the <tt>util-vserver</tt> <tt>init</tt> (ie: runlevel) script included in the <tt>util-vserver</tt> distribution, however this apparently only works for the <tt>sysv</tt> <tt>init</tt> script, and not the Debian or Gentoo ones.

If you were to mount the <tt>cgroup</tt> Control Groups filesystem manually, you would use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o ''<subsystems>'' /dev/cgroup</tt>

Where <tt>''<subsystems>''</tt> is something like <tt>cpuset,memory</tt>.

To avoid the need for manual configuration after reboot, on Gentoo you may wish to add the cgroup mount to <tt>/etc/fstab</tt>. For Debian see the live examples section at the bottom of this page.
<pre>
none /dev/cgroup cgroup cpu,cpuset,memory 0 2
</pre>

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
When you wish the guests to see only their limited memory pool, be sure to include VIRT_MEM in your cflags config file.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-15T00:07:15Z

Petzsch:

Bears run away when you yell at them, even lynxes. ,

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure CONFIG_CGROUP_NS is unset so guests start properly for the time being.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

You can find documentation about the cfs hard limiting in Documentation/scheduler/sched-cfs-hard-limits.txt inside your kernel source dir.

This feature is currently available in patch-2.6.31.2-vs2.3.0.36.15.diff and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vserver start you need a recent utils package. It worked for me with: 0.30.216-pre2864.

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

Example for an upper bound of 2/5th (or 40%) of the all cpu power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the ressources used by the cgroup.
If you set both cpu share AND hard limit the system will do fine but hard limits takes priority over cpu share scheduling, so cpu share will do the job but each cgroup will have an upper bound that it cannot cross even if the cpu share you gived it is higher.

<pre>
Hard limit feature adds 3 cgroup files for CFS group scheduler:
cfs_runtime_us: Hard limit for the group in microseconds.
cfs_period_us: Time period in microseconds within which hard limits is enforced.
cfs_hard_limit: The control file to enable or disable hard limiting for the group.
</pre>

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

== real world exemples of scheduling ==

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

=== Ben's install on Debian Lenny ===

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. Stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Options used in the kernels from repo.psand.net were:

CONFIG_CGROUP_SCHED=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
# CONFIG_CGROUP_NS is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_MEM_RES_CTLR=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP is not set
CONFIG_NET_CLS_CGROUP=y

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line. I patched my with this patch found on the Linux-Vserver mailing list:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needs adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but worked for me) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. I fixed every guest to use only one same CPU (see below how I did this) and the percentages were pretty much exact! Easy process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processor to different guest servers ====

To limit each guest the cpuset of each cgroup needs to be changed. I found out the number of CPUs available by doing this:

$ cat /dev/cgroup/cpuset.cpus

Give me the result 0-1, meaning that the set consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset limited to CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

This meant that, on restarting, I could see with vtop that these guest were only using the CPU 0 (column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the percentages I had intended for my cpu shares were working.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs was I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".