winbind failure on ssh login, failed to mlock memory: Cannot allocate memory

DutchDaemon

Administrator
Staff member
Administrator
Moderator
Developer

Thanks: 2,716
Messages: 11,257

#2
The FreeBSD forum does not handle bugs. Contact the maintainer of the port: [cmd=]make -C /usr/ports/net/samba36 maintainer[/cmd] or wait for a fix.
 
OP
OP
M

mletzgus

Member


Messages: 23

#3
Thanks for the hint, but I'm not really sure it's a bug in winbind@samba.

If it's a problem with mlock() itself it's a problem with FreeBSD. I hope(!) it's only a config problem due to limited resources (ulimit).

But I don't know how to figure this out - windbind is running as root and there is no obvious limit:
Code:
ulimit -a
socket buffer size       (bytes, -b) unlimited
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 33554432
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 11095
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 524288
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5547
virtual memory          (kbytes, -v) unlimited
swap size               (kbytes, -w) unlimited
DutchDaemon said:
The FreeBSD forum does not handle bugs. Contact the maintainer of the port: [cmd=]make -C /usr/ports/net/samba36 maintainer[/cmd] or wait for a fix.
 

DutchDaemon

Administrator
Staff member
Administrator
Moderator
Developer

Thanks: 2,716
Messages: 11,257

#4
If it's not a simple configuration problem (which a fellow forum user might figure out from your report) it's still a better idea at a later stage to make a bug report upstream. Either the port maintainer will research it, or he will kick the bug to either FreeBSD developers or the original developers of the software. http://www.freebsd.org/send-pr.html
 
OP
OP
M

mletzgus

Member


Messages: 23

#5
Hmm, for the fellow forum user:

I think it might be a problem with memory limits.
  1. reboot (system is happy now)
  2. wbinfo -K username works fine
  3. start memtest 10G and wait a few seconds, keep it running
  4. wbinfo -K username now fails
  5. kill memtest
  6. restart Samba/winbind and wbinfo will succeed again
The reason why I'm thinking about some system memory limits is that memtest 10G says:
Code:
got  9976MB (10461159424 bytes), trying mlock ...too many pages, reducing...
and later on:
Code:
got  4717MB (4946239488 bytes), trying mlock ...over system/pre-process limit, reducing...
got  4717MB (4946231296 bytes), trying mlock ...locked.
Something is over some limit here, but what?

Code:
> getconf PAGE_SIZE
4096
Code:
>ipcs -M
shminfo:
        shmmax:    536870912    (max shared memory segment size)
        shmmin:            1    (min shared memory segment size)
        shmmni:          192    (max number of shared memory identifiers)
        shmseg:          128    (max shared memory segments per process)
        shmall:       131072    (max amount of shared memory in pages)
Code:
sysctl:
real memory  = 17179869184 (16384 MB)
avail memory = 16454688768 (15692 MB)
Virtual Memory:         (Total: 1075113720K Active: 1165400K)
Real Memory:            (Total: 302104K Active: 96528K)
Shared Virtual Memory:  (Total: 83104K Active: 17164K)
Shared Real Memory:     (Total: 22112K Active: 9192K)
Free Memory Pages:      15562976K
vm.vmtotal: 
vm.loadavg: { 0.03 0.08 0.04 }
vm.v_free_min: 25527
vm.v_free_target: 107422
vm.v_free_reserved: 5314
vm.v_inactive_target: 161133
vm.v_cache_min: 107422
vm.v_cache_max: 214844
vm.v_pageout_free_min: 34
vm.pageout_algorithm: 0
vm.swap_enabled: 1
vm.md_malloc_wait: 0
vm.kmem_map_free: 16479559680
vm.kmem_map_size: 79925248
vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 16559910912
vm.nswapdev: 2
vm.dmmax: 32
vm.swap_async_max: 4
vm.overcommit: 0
vm.swap_reserved: 538800128
vm.swap_total: 10511388672
vm.zone_count: 208
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.kstacks: 481
vm.kstack_cache_size: 128
vm.exec_map_entries: 16
vm.stats.misc.zero_page_count: 2327
vm.stats.misc.cnt_prezero: 0
vm.stats.vm.v_kthreadpages: 0
vm.stats.vm.v_rforkpages: 0
vm.stats.vm.v_vforkpages: 26244
vm.stats.vm.v_forkpages: 1115358
vm.stats.vm.v_kthreads: 21
vm.stats.vm.v_rforks: 0
vm.stats.vm.v_vforks: 50
vm.stats.vm.v_forks: 2212
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_cache_max: 214844
vm.stats.vm.v_cache_min: 107422
vm.stats.vm.v_cache_count: 173
vm.stats.vm.v_inactive_count: 17902
vm.stats.vm.v_inactive_target: 161133
vm.stats.vm.v_active_count: 8661
vm.stats.vm.v_wire_count: 125500
vm.stats.vm.v_free_count: 3890576
vm.stats.vm.v_free_min: 25527
vm.stats.vm.v_free_target: 107422
vm.stats.vm.v_free_reserved: 5314
vm.stats.vm.v_page_count: 4042947
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_tfree: 1872529
vm.stats.vm.v_pfree: 169189
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_tcached: 2677
vm.stats.vm.v_pdpages: 0
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_reactivated: 2349
vm.stats.vm.v_intrans: 112
vm.stats.vm.v_vnodepgsout: 7400
vm.stats.vm.v_vnodepgsin: 11730
vm.stats.vm.v_vnodeout: 1019
vm.stats.vm.v_vnodein: 1146
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_ozfod: 0
vm.stats.vm.v_zfod: 1668061
vm.stats.vm.v_cow_optim: 580
vm.stats.vm.v_cow_faults: 83746
vm.stats.vm.v_vm_faults: 1846961
vm.stats.sys.v_soft: 353664
vm.stats.sys.v_intr: 178630
vm.stats.sys.v_syscall: 13765943
vm.stats.sys.v_trap: 653376
vm.stats.sys.v_swtch: 2673027
vm.stats.object.bypasses: 698
vm.stats.object.collapses: 6703
vm.v_free_severe: 15420
vm.old_msync: 0
vm.tryrelock_restart: 0
vm.boot_pages: 64
vm.max_wired: 1333057
vm.pageout_lock_miss: 0
vm.disable_swapspace_pageouts: 0
vm.defer_swapspace_pageouts: 0
vm.swap_idle_enabled: 0
vm.pageout_stats_interval: 5
vm.pageout_full_stats_interval: 20
vm.pageout_stats_max: 107422
vm.max_launder: 32
vm.phys_segs: 
vm.phys_free: 
vm.reserv.reclaimed: 0
vm.reserv.partpopq: 
vm.reserv.freed: 8756
vm.reserv.broken: 0
vm.idlezero_enable: 0
vm.kvm_free: 530944356352
vm.kvm_size: 549755809792
vm.pmap.pmap_collect_active: 0
vm.pmap.pmap_collect_inactive: 0
vm.pmap.pv_entry_spare: 8221
vm.pmap.pv_entry_allocs: 4760314
vm.pmap.pv_entry_frees: 4691927
vm.pmap.pc_chunk_tryfail: 0
vm.pmap.pc_chunk_frees: 27823
vm.pmap.pc_chunk_allocs: 28279
vm.pmap.pc_chunk_count: 456
vm.pmap.pv_entry_count: 68387
vm.pmap.pdpe.demotions: 4
vm.pmap.pde.promotions: 3196
vm.pmap.pde.p_failures: 27420
vm.pmap.pde.mappings: 0
vm.pmap.pde.demotions: 3138
vm.pmap.shpgperproc: 200
vm.pmap.pv_entry_max: 5275747
vm.pmap.pg_ps_enabled: 1
vm.pmap.pat_works: 1
vfs.vmiodirenable: 1
compat.ia32.maxvmem: 0
dev.em.0.nvm: -1
And now we've reached the point where I need a hint. What's wrong here, what am I missing? Maybe it's not a problem with winbind itself - but with mlock() and/or my system settings.

Best regards,
Michael
 
OP
OP
M

mletzgus

Member


Messages: 23

#8
DutchDaemon said:
It's a network service, not a base system issue.
Ack, winbind is a network service - but not mlock(). Why do you think it's a network issue? My assumption was that mlock() (or a config problem) is the underlying problem. The winbind failure is only a symptom... maybe...

I'm still looking into the configuration of limits and the whole memory management stuff, the behavior of memtest might be a hint. :)

cu,
Michael
 

DutchDaemon

Administrator
Staff member
Administrator
Moderator
Developer

Thanks: 2,716
Messages: 11,257

#9
I'm going to advise you one last time to simply open a PR and let the porters/developers handle this. Most of them are not on here, as this is primarily a user forum. And so long as only your winbind has a problem. it's logical to assume it's that where the problem lies (a network service (i.e. Samba/winbind), not a "network issue"), and not the base system's mlock().
 

kpa

Beastie's Twin

Thanks: 1,791
Messages: 6,303

#12
The information about vm.kmem_size_max is totally wrong on the linked article, the value of the tunable is usually set to hundreds of gigabytes when automatically set by the kernel and that should be enough to hint that it's not an actual value for a maximum size for kmem but something else.

Are you using ZFS? If you are take note of the ARC usage stats, for example as reported by sysutils/zfs-stats. You may need to limit the maximum size of the ARC cache with vfs.zfs.arc_max so that enough wired memory is left for other uses.
 
OP
OP
M

mletzgus

Member


Messages: 23

#13
Hi kpa.

Yes, I'm using ZFS - and I'm currently lost in memory configration *G*

The unmodified default values on my system are:

Code:
real memory  = 17179869184 (16384 MB)
avail memory = 16454688768 (15692 MB)
real memory  = 17179869184 (16384 MB)
avail memory = 16454688768 (15692 MB)
Virtual Memory:         (Total: 1075068412K Active: 1134676K)
Real Memory:            (Total: 277512K Active: 84992K)
Shared Virtual Memory:  (Total: 74076K Active: 17596K)
Shared Real Memory:     (Total: 19496K Active: 9420K)
Free Memory Pages:      15600940K

vm.kmem_map_free: 16488685568
vm.kmem_map_size: 71143424
vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 16559910912
vm.max_wired: 1333057
vfs.zfs.arc_max: 15486169088
A little bit of math:

Code:
Total RAM: 16 G
kmem_size: 15.4 G [[color="Green"]OK[/color]]
vm.kmem_size_max: 307.2 G [[color="Green"]OK[/color]]
vm.max_wired: 5.1 G [[color="Red"]![/color]]
vfs.zfs.arc_max: 14.4 G [[color="Red"]![/color]]
My interpration: ZFS ARC can eat up wired pages so that calls to mlock() form winbindd, gpg and memtest fail.

You are totally right, the kmem-sizes are fine.

What about increasing wired pages to ~10 GB and decreasing arc_max to ~5 GB - or any other values?

Or are there other problems with increasing the wired pages limit?

By the way... does ZFSs AR cache uses wired pages via mlock() - and why can exceed the ARC size the wired_pages limit?

Or can ZFS do whatever it want and the wired_page limit is only applied to user processes?

Thanks in advance,
Michael
 

kpa

Beastie's Twin

Thanks: 1,791
Messages: 6,303

#14
Start by lowering the ARC maximum size to a low value, 3-4GBs for example. I'm not sure if vm.wired_max has an effect on how ZFS allocates the ARC cache, it looks like it doesn't looking at your numbers.
 
OP
OP
M

mletzgus

Member


Messages: 23

#15
I reduced ARC size to 4 GB and increased the wired page limit from 5 GB to 9 GB.

If the ARC is full there should be 5 GB left - exactly like without any ZFS/ARC memory usage.

What is the reason for the wired_pages limit? Why not allow anyone who needs it to lock much more pages?
 
OP
OP
M

mletzgus

Member


Messages: 23

#16
mletzgus said:
I reduced ARC size to 4 GB and increased the wired page limit from 5 GB to 9 GB.
Hm, somethings seems to wrong with my loader.conf. Setting wired_page doesn't work:

vboxdrv_load="YES"
coretemp_load="YES"
aio_load="YES"
linux_load="YES"
vfs.zfs.txg.timeout="5"
vfs.zfs.arc_max="4294967296"
vm.max_wired="2359296"

#EOF


Setting it manually works fine... strange.

[CMD=""]sysctl -a | grep -i wired
vm.max_wired: 1333034

sysctl vm.max_wired=2359296
vm.max_wired: 1333034 -> 2359296

sysctl -a | grep -i wired
vm.max_wired: 2359296[/CMD]
 

t1066

Active Member

Thanks: 84
Messages: 226

#17
Just a few remarks. You can use vfs.zfs.arc_max="4G" instead of numbers. And set vm.max_wired=2359296 in /etc/sysctl.conf instead.
 
OP
OP
M

mletzgus

Member


Messages: 23

#18
Setting it in sysctl.conf works - but there is an error message:

Code:
sysctl -a | grep -i wired
<118>/etc/rc: WARNING: unable to set vm.max_wired="2359296"
<118>/etc/rc.d/sysctl: WARNING: unable to set vm.max_wired="2359296"
vm.max_wired: 2359296
 

t1066

Active Member

Thanks: 84
Messages: 226

#19
At least for me, there is no need to put double quote around numeric values in /etc/sysctl.conf.
 
OP
OP
M

mletzgus

Member


Messages: 23

#20
t1066 said:
At least for me, there is no need to put double quote around numeric values in /etc/sysctl.conf.
Yes, loader.conf variables must be quoted - sysctl.conf variables must NOT be quoted.
When doing so - it works :)
 
Top