FreeBSD failed to report correct NUMA number on some AMD servers

J

JasonZ

FreeBSD failed to report correct NUMA number on some AMD servers.
Below is an example on a HP Server, DL385 Gen10 Plus V2, using "AMD EPYC 74F3 Processor".

When using OS Red Hat Enterprise Linux release 8.8, the NUMA number can be correctly reported as shown below.
Code: 
# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description:    Red Hat Enterprise Linux release 8.8 (Ootpa)
Release:        8.8
Codename:       Ootpa
#
# lscpu | grep Model
Model:               1
Model name:          AMD EPYC 74F3 24-Core Processor
BIOS Model name:     AMD EPYC 74F3 24-Core Processor
#
# numactl --hardware
available: 16 nodes (0-15)
node 0 cpus: 0 1 2 48 49 50
node 0 size: 7239 MB
node 0 free: 6780 MB
node 1 cpus: 3 4 5 51 52 53
node 1 size: 8062 MB
node 1 free: 7736 MB
node 2 cpus: 6 7 8 54 55 56
node 2 size: 8062 MB
node 2 free: 7928 MB
node 3 cpus: 9 10 11 57 58 59
node 3 size: 8062 MB
node 3 free: 7776 MB
node 4 cpus: 12 13 14 60 61 62
node 4 size: 8062 MB
node 4 free: 7504 MB
node 5 cpus: 15 16 17 63 64 65
node 5 size: 8062 MB
node 5 free: 7836 MB
node 6 cpus: 18 19 20 66 67 68
node 6 size: 8062 MB
node 6 free: 7910 MB
node 7 cpus: 21 22 23 69 70 71
node 7 size: 8045 MB
node 7 free: 7894 MB
node 8 cpus: 24 25 26 72 73 74
node 8 size: 8062 MB
node 8 free: 7892 MB
node 9 cpus: 27 28 29 75 76 77
node 9 size: 8062 MB
node 9 free: 7930 MB
node 10 cpus: 30 31 32 78 79 80
node 10 size: 8062 MB
node 10 free: 7919 MB
node 11 cpus: 33 34 35 81 82 83
node 11 size: 8062 MB
node 11 free: 7894 MB
node 12 cpus: 36 37 38 84 85 86
node 12 size: 8062 MB
node 12 free: 6777 MB
node 13 cpus: 39 40 41 87 88 89
node 13 size: 8018 MB
node 13 free: 7895 MB
node 14 cpus: 42 43 44 90 91 92
node 14 size: 8062 MB
node 14 free: 7947 MB
node 15 cpus: 45 46 47 93 94 95
node 15 size: 8057 MB
node 15 free: 7799 MB
node distances:
node   0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
  0:  10  11  11  11  11  11  11  11  32  32  32  32  32  32  32  32
  1:  11  10  11  11  11  11  11  11  32  32  32  32  32  32  32  32
  2:  11  11  10  11  11  11  11  11  32  32  32  32  32  32  32  32
  3:  11  11  11  10  11  11  11  11  32  32  32  32  32  32  32  32
  4:  11  11  11  11  10  11  11  11  32  32  32  32  32  32  32  32
  5:  11  11  11  11  11  10  11  11  32  32  32  32  32  32  32  32
  6:  11  11  11  11  11  11  10  11  32  32  32  32  32  32  32  32
  7:  11  11  11  11  11  11  11  10  32  32  32  32  32  32  32  32
  8:  32  32  32  32  32  32  32  32  10  11  11  11  11  11  11  11
  9:  32  32  32  32  32  32  32  32  11  10  11  11  11  11  11  11
 10:  32  32  32  32  32  32  32  32  11  11  10  11  11  11  11  11
 11:  32  32  32  32  32  32  32  32  11  11  11  10  11  11  11  11
 12:  32  32  32  32  32  32  32  32  11  11  11  11  10  11  11  11
 13:  32  32  32  32  32  32  32  32  11  11  11  11  11  10  11  11
 14:  32  32  32  32  32  32  32  32  11  11  11  11  11  11  10  11
 15:  32  32  32  32  32  32  32  32  11  11  11  11  11  11  11  10
#


However, when using OS FreeBSD 14.0-RELEASE, the NUMA number reported becomes 1, as shown below.
Code: 
# cat /etc/os-release
NAME=FreeBSD
VERSION="14.0-RELEASE"
VERSION_ID="14.0"
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 14.0-RELEASE"
CPE_NAME="cpe:/o:freebsd:freebsd:14.0"
HOME_URL="https://FreeBSD.org/"
BUG_REPORT_URL="https://bugs.FreeBSD.org/"
#
# sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: amd64
hw.model: AMD EPYC 74F3 24-Core Processor
hw.ncpu: 96
hw.machine_arch: amd64
#
# sysctl vm.ndomains
vm.ndomains: 1
#
# sysctl vm.phys_locality
vm.phys_locality:
0: -1
#
# cpuset -d 0 -g
domain 0 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95
#
# cpuset -d 1 -g
domain 1 mask:
#

Does it mean FreeBSD cannot support some AMD CPUs well?
Is there a workaround or config to solve this problem? Thanks in advance.
 
cracauer@ said:
Probably an ACPI problem that Linux has a hand-coded workaround for.
Click to expand...
Thanks cracauer for reply.
I tried changing the following definition from 8 to 16, then rebuild/reinstall the kernel.
Then the OS is able to show the correct NUMA number for the HP server (with 16 NUMA) using AMD CPU.
But I am not sure whether or not this is the only place needed to change.
Is there somewhere else also needed to change?
Thanks in advance.

1705291587572.png
 
You must log in or register to reply here.
Back
Top