at-spi-bus-launcher eats all the memory

Emrion

Emrion

Hello,

When I launch vlc (90% of the time), I notice the process at-spi-bus-launcher eats all the memory and it is eventually terminated by the system. vlc finally run (after 20 - 30 seconds). But, meanwhile, it also kills a bhyve VM because this last uses a lot of memory (roughly 4 GiB, which is intended). I can get this behavior also by launching chrome (but, it's rare. Typically, it works without problem).

This is an example of what I get with top when the problem begins:
Code: 
PID        USERNAME    THR    PRI    NICE    SIZE    RES        STATE    C    TIME    WCPU    COMMAND
39619      Emrion        5     82    0       5174M  2397M        CPU0    0    0:04    99,78%  at-spi-bus-launcher
Then, the RES value increases until several GiB (the machine has 32 GiB total RAM, 2 GiB swap).

I read this in /var/log/messages:
Code: 
Jul  1 15:31:48 Central kernel: swap_pager: out of swap space
Jul  1 15:31:48 Central kernel: swp_pager_getswapspace(4): failed
Jul  1 15:31:51 Central kernel: pid 39619 (at-spi-bus-launcher), jid 0, uid 1001, was killed: failed to reclaim memory
Jul  1 15:31:55 Central kernel: pid 39260 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory

freebsd-version -uk
13.2-RELEASE-p1
13.2-RELEASE-p1
The software is up to date from the latest repository. It runs slim and lxde. Used graphic driver for Xorg is intel/i915kms.
This PC is based on an I3-7100T and Asus H170I-PRO motherboard. Root system lies in a zfs RAID1 (2 x Samsung SSD 860 EVO 250GB RVT02B6Q).

What makes me drive nut is that I can't reproduce this problem on other machines with a very close software configuration.

The first time it happened was at the end of march. I did pkg upgrade just before. It was on 13.1-RELEASE at this time. Here are the packages that were upgraded:
Mar 31 21:31:26 Central pkg[46810]: libxcb upgraded: 1.15 -> 1.15_1
Mar 31 21:31:26 Central pkg[46810]: pciids upgraded: 20230126 -> 20230223
Mar 31 21:31:26 Central pkg[46810]: libpciaccess upgraded: 0.16 -> 0.17
Mar 31 21:31:27 Central pkg[46810]: glib upgraded: 2.74.6,2 -> 2.76.1,2
Mar 31 21:31:27 Central pkg[46810]: zstd upgraded: 1.5.2_1 -> 1.5.4
Mar 31 21:31:27 Central pkg[46810]: ca_root_nss upgraded: 3.88.1 -> 3.89
Mar 31 21:31:27 Central pkg[46810]: libdatrie-0.2.13_1 installed
Mar 31 21:31:27 Central pkg[46810]: mesa-libs upgraded: 22.3.6 -> 22.3.7
Mar 31 21:31:27 Central pkg[46810]: mpg123 upgraded: 1.31.2 -> 1.31.3
Mar 31 21:31:27 Central pkg[46810]: vulkan-headers upgraded: 1.3.242 -> 1.3.245
Mar 31 21:31:27 Central pkg[46810]: libinput upgraded: 1.22.1 -> 1.23.0
Mar 31 21:31:27 Central pkg[46810]: harfbuzz reinstalled: 7.1.0 -> 7.1.0
Mar 31 21:31:28 Central pkg[46810]: openexr upgraded: 3.1.5_1 -> 3.1.6_1
Mar 31 21:31:28 Central pkg[46810]: libthai-0.1.29 installed
Mar 31 21:31:29 Central pkg[46810]: mesa-dri upgraded: 22.3.6 -> 22.3.7
Mar 31 21:31:29 Central pkg[46810]: highway upgraded: 1.0.3 -> 1.0.4_1
Mar 31 21:31:29 Central pkg[46810]: sqlite3 upgraded: 3.41.0,1 -> 3.41.0_1,1
Mar 31 21:31:29 Central pkg[46810]: libunibreak upgraded: 4.3,1 -> 5.1,1
Mar 31 21:31:29 Central pkg[46810]: pango upgraded: 1.50.9 -> 1.50.9_1
Mar 31 21:31:29 Central pkg[46810]: libva upgraded: 2.17.0 -> 2.18.0
Mar 31 21:31:29 Central pkg[46810]: libass upgraded: 0.17.1 -> 0.17.1_1
Mar 31 21:31:29 Central pkg[46810]: dav1d upgraded: 1.1.0 -> 1.1.0_1
Mar 31 21:31:30 Central pkg[46810]: librsvg2-rust upgraded: 2.54.5_5 -> 2.56.0
Mar 31 21:31:30 Central pkg[46810]: nss upgraded: 3.88.1 -> 3.89
Mar 31 21:31:30 Central pkg[46810]: libnghttp2 upgraded: 1.51.0_1 -> 1.52.0
Mar 31 21:31:31 Central pkg[46810]: ffmpeg upgraded: 4.4.3_5,1 -> 4.4.3_6,1
Mar 31 21:31:31 Central pkg[46810]: snappy upgraded: 1.1.9_1 -> 1.1.10
Mar 31 21:31:31 Central pkg[46810]: gobject-introspection upgraded: 1.74.0,1 -> 1.76.1,1
Mar 31 21:31:32 Central pkg[46810]: gdb upgraded: 12.1_3 -> 13.1
Mar 31 21:31:39 Central pkg[46810]: chromium upgraded: 110.0.5481.177_1 -> 111.0.5563.110
Mar 31 21:31:39 Central pkg[46810]: drm-510-kmod upgraded: 5.10.163_2 -> 5.10.163_3
Mar 31 21:31:40 Central pkg[46810]: binutils upgraded: 2.40,1 -> 2.40_2,1
Mar 31 21:31:42 Central pkg[46810]: vlc upgraded: 3.0.18_1,4 -> 3.0.18_2,4
Mar 31 21:31:42 Central pkg[46810]: sshpass upgraded: 1.09 -> 1.10
Mar 31 21:31:42 Central pkg[46810]: xorg-server upgraded: 21.1.4_1,1 -> 21.1.7,1
Mar 31 21:31:42 Central pkg[46810]: curl upgraded: 7.87.0_1 -> 7.88.1

I'm out of idea to diagnose the thing. Maybe someone here can point me in the right direction?
As this machine is the one that provides the internet to my home network, I can't crash or reboot it often. Not to mention that my wife works remotely from there...
 
I tried NO_AT_BRIDGE=1 on a similar machine ( lxde, slim with autologin). It seems that .profile isn't executed. So, I used .xinitrc but, this has no effect (i.e. at_spi_bus_launcher is launched anyway when I start chrome).

I don't need "assistive tech stuff". In fact, I don't need to run vlc or chrome on this machine, I don't even need Xorg but, I would like to mend the problem or, better, understand it.
 
Emrion said:
...I would like to mend the problem or, better, understand it.
Click to expand...
It seems that you have a pretty good handle on it. You know who the culprit is. Do you want to troubleshoot at_spi_bus_launcher itself? I've never done heap analysis on a C process, but there's gotta be tools for it. I guess you'd have to force it to core dump and then analyze the core file.
 
Jose said:
It seems that you have a pretty good handle on it. You know who the culprit is. Do you want to troubleshoot at_spi_bus_launcher itself? I've never done heap analysis on a C process, but there's gotta be tools for it. I guess you'd have to force it to core dump and then analyze the core file.
Click to expand...
Thank to take time on this problem. I know it's not easy at all. If only I could reproduce it on a separate machine, that would be a big step in its comprehension/resolution.

Yes, I would like to find some infos like a specific dbus or at-spi log file somewhere but, currently I failed on this. How can I force this process to core dump? I'm eager to any idea.
 
Emrion said:
Yes, I would like to find some infos like a specific dbus or at-spi log file somewhere but, currently I failed on this. How can I force this process to core dump? I'm eager to any idea.
Click to expand...
Can you run it in gdb and then ctrl-z? If you do this a number of times, based on likelihoods, it should help you to get a feel for the parts of code that it is spending the longest. A poor man's profiler basically.
 
Emrion said:
Yes, I would like to find some infos like a specific dbus or at-spi log file somewhere but, currently I failed on this. How can I force this process to core dump? I'm eager to any idea.
Click to expand...
The sigaction(2) man page suggests sending SIGQUIT to the process should cause it to create a core file and quit. I'm not sure how to look at the heap using lldb(1). Forum member Obsigna has a page on how to use LLDB in Freebsd:

Debugging a core dump with LLDB on FreeBSD

obsigna.com obsigna.com
 
Jose said:
The sigaction(2) man page suggests sending SIGQUIT to the process should cause it to create a core file and quit. I'm not sure how to look at the heap using lldb(1). Forum member Obsigna has a page on how to use LLDB in Freebsd:

Debugging a core dump with LLDB on FreeBSD

obsigna.com obsigna.com
Click to expand...
Ok, I'll try that during hollydays.

For the rest, I don't know how to run at_spi_bus_launcher inside gdb.

I have three machines with lxde (including this sick one). None of them do the same with at_spi_bus_launcher. One is running at_spi_bus_launcher at soon as you launch vlc or chrome (without problem) and the process stays in memory. The other machine doesn't run at_spi_bus_launcher at all (and at-spi2-core is well installed, it's a dependency of chromium). :-/
 
Emrion said:
For the rest, I don't know how to run at_spi_bus_launcher inside gdb.
Click to expand...
I think LLDB is the debugger on Freebsd nowadays. In any case, you don't have to run at_spi_bus_launcher inside it. You can attach to a running instance or force it to dump core and then run LLDB on that core file.
 
Some news about this topic.

It was and is yet difficult to sort out between abnormal and normal functioning of accessibility/at-spi2-core. So, conversely of what I was saying, multimedia/vlc doesn't launch at-spi-bus-launcher, but it's sensitive to its presence.

Many programs try to run at-spi-bus-launcher including chromium and firefox but not only. For example, in this list, there is geany and even acalc. No matter whether accessibility is disabled or not in the lxde / openbox settings. All those softwares are directly or indirectly dependent on accessibility/at-spi2-core.

It seems that a kill -QUIT doesn't generate a core file.

In the wait of understanding what happenned to my server (if one day I understand), I searched how to prevent at-spi-bus-launcher to run. I found several methods. The one with less side effect is:
In /usr/local/share/dbus-1/services/org.a11y.Bus.service, I commented out the "Exec" line.
Code: 
[D-BUS Service]
Name=org.a11y.Bus
#Exec=/usr/local/libexec/at-spi-bus-launcher
SystemdService=at-spi-dbus-bus.service
Then, I recomputed the checksum of the package to avoid complaint from periodic checking:
pkg check -r at-spi2-core

That seems efficient, but, obviously, this must be redone at each upgrade of accessibility/at-spi2-core.
 
Nice analysis! Yep, lots of things can pull that in:

forums.FreeBSD.org

Why do we need d-bus?

dbus is needed for some desktops, unfortunately, the more popular ones like KDE, GNOME. Quite a few applications (firefox) will actually call dbus-launch. A lot of times the applications use it to save configuration changes (preferences). A lot of non-Linux applications don't really need it...
forums.FreeBSD.org forums.FreeBSD.org
 
Emrion said:
It seems that a kill -QUIT doesn't generate a core file.
Click to expand...
The signal SIGQUIT can be intentionally caught by a program, and it can take it's own action. Commonly programs use a combination of INT and QUIT to signal that the program should end, more or less cleanly. Dumping core is not a common thing programs will do, as those that are well-developed enough to have signal handlers tend to also have their own debug facilities.

You could try sending one of the other signals that by default cause a core dump (ABRT, ILL, ...) instead; fewer programs catch those.
 
You must log in or register to reply here.
Back
Top