Notes about IRQ Affinity.

1. irqaffinity

If irqaffinity is appended as described in Kernel Parameters, It’s used to set the default irq affinity mask.
The set value can be checked by:

1
$ cat /proc/irq/default_smp_affinity

2. IRQ Affinity

Binding IRQs to a group of CPUs is now a generic and independent kernel feature. Every IRQ source in Linux has an entry in /proc/irq directory. For example, the settings for IRQ 40 is stored in /proc/irq/40. IRQ affinity, or IRQ bindings, is configured though the smp_affinity setting in that directory. For example, the smp_affinity for IRQ 40 is in /proc/irq/40/smp_affinity. The value of the smp_affinity setting is a bitmask of all CPUs that are permitted as a resource for the given IRQ. The default value for smp_affinityis 0xffffffff. This means the processes for the IRQ are sent to all CPUs. You are not allowed to turn off all CPUs for an IRQ. If the IRQ controller does not support IRQ affinity, the value can not be changed from the default. If multiple CPUs are defined, then the IRQ source uses the least busy CPU. This is called lowest priority APIC routing. IRQ affinity is achieved by binding an IRQ to a specific CPU or group of CPUs by echoing a HEX value to smp_affinity for the IRQ.

3. Example

Here is an example of restricting IRQ44 (eth1) to CPU0-3 then restricting it to CPU4-7 (this is an 8-CPU SMP box):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@moon 44]# cd /proc/irq/44
[root@moon 44]# cat smp_affinity
ffffffff

[root@moon 44]# echo 0f > smp_affinity
[root@moon 44]# cat smp_affinity
0000000f
[root@moon 44]# ping -f h
PING hell (195.4.7.3): 56 data bytes
...
--- hell ping statistics ---
6029 packets transmitted, 6027 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.4 ms
[root@moon 44]# cat /proc/interrupts | grep 'CPU\|44:'
        CPU0       CPU1       CPU2       CPU3      CPU4       CPU5        CPU6       CPU7
44:       1068       1785       1785       1783         0          0           0         0    IO-APIC-level  eth1

As can be seen from the line above IRQ44 was delivered only to the first four processors (0-3). Now lets restrict that IRQ to CPU(4-7).

1
2
3
4
5
6
7
8
9
10
11
12
[root@moon 44]# echo f0 > smp_affinity
[root@moon 44]# cat smp_affinity
000000f0
[root@moon 44]# ping -f h
PING hell (195.4.7.3): 56 data bytes
..
--- hell ping statistics ---
2779 packets transmitted, 2777 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.5/585.4 ms
[root@moon 44]# cat /proc/interrupts |  'CPU\|44:'
        CPU0       CPU1       CPU2       CPU3      CPU4       CPU5        CPU6       CPU7
44:       1068       1785       1785       1783      1784       1069        1070       1069   IO-APIC-level  eth1

This time around IRQ44 was delivered only to the last four processors. i.e counters for the CPU0-3 did not change.

4. Irqbalance daemon

https://github.com/Irqbalance/irqbalance
开启irqbalance提升服务器性能

参考资料:

  1. SMP IRQ affinity
  2. linux irq/affinity理解
  3. kernel-parameters.txt