Xorg SEGFAULT on 14.0 in QEMU/KVM

anubsd · Nov 15, 2023

Hello, I did a fresh 14.0 installation on a QEMU/KVM virtual machine in Linux Ubuntu (vm configuration attached).
After the installation I installed Xorg, slim, xfce4 and xf86-video-qxl packages, then started xfce, but X crashed with a SEGFAULT (log attached).
I tried upgrading pkg from quarterly to latest, but nothing changed.

Any ideas?
Thanks!

anubsd · Nov 15, 2023

Output when starting X under gdb:

Code:

Starting program: /usr/local/libexec/Xorg
[New LWP 100308 of process 1302]

Thread 1 received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
0x00000008017da9ce in ?? () from /usr/local/lib/xorg/modules/drivers/spiceqxl_drv.so
#0  0x00000008017da9ce in  () at /usr/local/lib/xorg/modules/drivers/spiceqxl_drv.so
#1  0x00000008017ddb66 in  () at /usr/local/lib/xorg/modules/drivers/spiceqxl_drv.so
#2  0x00000000003cb154 in InitOutput ()
#3  0x00000000002cc94b in  ()
#4  0x00000008006eeafa in __libc_start1 () at /lib/libc.so.7
#5  0x000000000028e8e0 in _start ()

covacat · Nov 15, 2023

framebuffer is detected as having zero kb
this might be a problem

anubsd · Nov 15, 2023

Thanks for the answer covacat; I'm not an expert about Xorg and related technologies... any suggestion I could try?

covacat · Nov 15, 2023

look here

Xspice segfaults on launch (#13996) · Issues · alpine / aports · GitLab

~ # Xspice :1 X.Org X Server 1.21.1.3...

gitlab.alpinelinux.org

search for
Option "SurfaceBufferSize" "16"
Option "CommandBufferSize" "64"
Option "FrameBufferSize" "8"

dunno if the same root cause but thats what i quicked search

anubsd · Nov 15, 2023

Thanks for the link, but adding those options in the Device section of the xorg.conf doesn't seem to work in my case.

anubsd · Nov 18, 2023

I reinstalled the xf86-video-qxl after compiling from ports in debug mode and did some debugging:

Code:

Core was generated by `/usr/local/libexec/Xorg'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  0x00000000003cbb0f in ddxGiveUp ()
--Type <RET> for more, q to quit, c to continue without paging--bt
[Current thread is 1 (LWP 100245)]
(gdb) 
(gdb) bt
#0  0x00000000003cbb0f in ddxGiveUp ()
#1  0x00000000003b7882 in  ()
#2  0x00000000003b61aa in  ()
#3  0x00000000003ae395 in  ()
#4  0x000000082837c54d in  () at /lib/libthr.so.3
#5  0x000000082837bafb in  () at /lib/libthr.so.3
#6  0x00000008217472d3 in <signal handler called> ()
#7  0x0000000837c55fad in xspice_init_qxl_ram (qxl=0x1a1634985000) at spiceqxl_io_port.c:73
#8  0x0000000837c5aa12 in qxl_pre_init (pScrn=0x1a16348ecf00, flags=0) at qxl_driver.c:1120
#9  0x00000000003cb154 in InitOutput ()
#10 0x00000000002cc94b in  ()
#11 0x000000082a397afa in __libc_start1 () at /lib/libc.so.7
#12 0x000000000028e8e0 in _start ()

It seems that the get_ram_header() function returns a bad address... when debugging this code (spiceqxl_io_port.c)

Code:

void xspice_init_qxl_ram(qxl_screen_t *qxl)
{
    QXLRam *ram = get_ram_header(qxl);
    uint64_t *item;
    ram->magic       = QXL_RAM_MAGIC;

I get:

Code:

(gdb) step
544        return (struct QXLRam *)
(gdb) step
xspice_init_qxl_ram (qxl=0x801185000) at spiceqxl_io_port.c:73
73       ram->magic       = QXL_RAM_MAGIC;
(gdb) print ram
$2 = (QXLRam *) 0x9010067e0
(gdb) print *ram
Cannot access memory at address 0x9010067e0
(gdb) step

Thread 1 received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
0x00000008017d9fad in xspice_init_qxl_ram (qxl=0x801185000) at spiceqxl_io_port.c:73

So I suspect there's something wrong with the VM configuration or with the Xorg autoconfiguration.

_martin · Nov 21, 2023

I imported your profile, I was able to reproduce the crash. gdb shows memory range where sigsegv occurs is mapped. Without going deep into gdb I focus on config side first.

I never used spice before so I don't know what to expect exactly. I noticed X has special config for spice in /usr/local/etc/X11/xorg.conf.d/spiceqxl.xorg.conf. I noticed buffer settings there too. I uncommented few lines:

Code:

# grep -vE '^$|^[ ]*#' /usr/local/etc/X11/xorg.conf.d/spiceqxl.xorg.conf
Section "Device"
    Identifier "XSPICE"
    Driver "spiceqxl"
    Option "SpiceDisableTicketing" "False"
    Option "SpiceSasl" "False"
    Option "SpicePort" "5900"
    Option "SpiceIPV4Only" "True"
    Option "SurfaceBufferSize" "128"
    Option "CommandBufferSize" "128"
    Option "FrameBufferSize" "16"
EndSection
Section "InputDevice"
    Identifier "XSPICE POINTER"
    Driver     "xspice pointer"
EndSection
Section "InputDevice"
    Identifier "XSPICE KEYBOARD"
    Driver     "xspice keyboard"
EndSection
Section "Monitor"
    Identifier    "Configured Monitor"
EndSection
Section "Screen"
    Identifier     "XSPICE Screen"
    Monitor        "Configured Monitor"
    Device        "XSPICE"
    DefaultDepth    24
EndSection
Section "ServerLayout"
    Identifier "XSPICE Example"
    Screen "XSPICE Screen"
    InputDevice "XSPICE KEYBOARD"
    InputDevice "XSPICE POINTER"
EndSection
Section "ServerFlags"
    Option "AutoAddDevices" "False"
EndSection

And used startx to start the X. It went ok, I ended up with the interface_get_command: native on the screen. Using different virt-viewer I was able to connect and verify it works.

When I modified my ~/.xinitrc to start xfce it crashed after input devices. I'm debugging that but wanted to share this.

_martin · Nov 21, 2023

Ok, I got a bit more familiar with that. I was able to connect to slim but then Xorg catches the trap. That trap seems to be related to slim though. I'd need to recompile slim with debug symbols.
I did one test - I just used startx to start xfce. Xorg trapped again. So I installed openbox and tried again -- that works.

Picture is from a Windows machine (outer black background using remote viewer to connect to FreeBSD's spice Xorg session started from a user's openbox)

I guess that's something for start.

_martin · Nov 22, 2023

I was able to make this work as expected with xdm. gdm has its own problems I didn't dive into.

Slim starts, I'm able to connect. But after I send my password session crashes. State of the registers at the crash suggest some sort of bug (d'oh). Most likely buffer overflow, use after free or threads are stepping over each other.
Registers have bogus values full of strings. Main loop looks like this:

Code:

(gdb) bt
#0  0x000000000039ec5a in TimerCancel ()
#1  0x0000000801856f3a in  () at /usr/local/lib/libspice-server.so.1
#2  0x0000000801857377 in  () at /usr/local/lib/libspice-server.so.1
#3  0x000000080186a614 in  () at /usr/local/lib/libspice-server.so.1
#4  0x0000000801861925 in  () at /usr/local/lib/libspice-server.so.1
#5  0x000000080186c0d3 in  () at /usr/local/lib/libspice-server.so.1
#6  0x000000080186b5d5 in  () at /usr/local/lib/libspice-server.so.1
#7  0x00000000003a6394 in  ()
#8  0x000000000039e96b in WaitForSomething ()
#9  0x00000000002bfdc0 in  ()
#10 0x00000000002ca46c in  ()
#11 0x00000008006ddafa in __libc_start1 (argc=6, argv=0x7fffffffea40, env=0x7fffffffea78, cleanup=<optimized out>, mainX=0x41f630) at /usr/src/lib/libc/csu/libc_start1.c:157
#12 0x000000000028df30 in _start ()
(gdb) info threads
  Id   Target Id                                  Frame
* 1    LWP 100646 of process 48944 "MainThread"   0x000000000039ec5a in TimerCancel ()
  3    LWP 102697 of process 48944 "SPICE Worker" _poll () at _poll.S:4
  5    LWP 102701 of process 48944                _poll () at _poll.S:4
  6    LWP 102702 of process 48944 "InputThread"  _poll () at _poll.S:4
(gdb) i r
rax            0x78457475706e4958  8666461105690659160
rbx            0x82fc20d70         35160984944
rcx            0x3a6e6f69736e6574  4210425200336856436
rdx            0x10                16
rsi            0x8016054d0         34382828752
(gdb) x/2i $pc
=> 0x39ec5a <TimerCancel+26>:    mov    QWORD PTR [rax+0x8],rcx
   0x39ec5e <TimerCancel+30>:    mov    QWORD PTR [rcx],rax
(gdb)

Tracking what modifies that timer is hard as that event is being executed almost always (live timer).

I found some weird bugs about spice settings in spiceqxl.xorg.conf

Code:

    Option "SurfaceBufferSize" "256"
    Option "CommandBufferSize" "256"
    Option "FrameBufferSize" "64"

Where one has to pay attention on sizing of these.

anubsd · Nov 23, 2023

Hi _martin , thanks for taking the time to debug this. Unfortunately it seems I'm not able to replicate the behaviour on my VM.
I'm using the configs from spiceqxl.conf (adding, removing or modfying the SurfaceBufferSize, CommandBufferSize and FrameBufferSize options doesn't change anything).

Code:

Section "Device"
    Identifier "XSPICE"
    Driver "spiceqxl"
    Option "SpiceDisableTicketing" "False"
    Option "SpiceSasl" "False"
    Option "SpicePort" "5900"
    Option "SpiceIPV4Only" "True"
    Option "SurfaceBufferSize" "128"
    Option "CommandBufferSize" "128"
    Option "FrameBufferSize" "16"
EndSection
Section "InputDevice"
    Identifier "XSPICE POINTER"
    Driver     "xspice pointer"
EndSection
Section "InputDevice"
    Identifier "XSPICE KEYBOARD"
    Driver     "xspice keyboard"
EndSection
Section "Monitor"
    Identifier    "Configured Monitor"
EndSection
Section "Screen"
    Identifier     "XSPICE Screen"
    Monitor        "Configured Monitor"
    Device        "XSPICE"
    DefaultDepth    24
EndSection
Section "ServerLayout"
    Identifier "XSPICE Example"
    Screen "XSPICE Screen"
    InputDevice "XSPICE KEYBOARD"
    InputDevice "XSPICE POINTER"
EndSection
Section "ServerFlags"
    Option "AutoAddDevices" "False"
EndSection

Code:

[   239.743] 
X.Org X Server 1.21.1.9
X Protocol Version 11, Revision 0
[   239.743] Current Operating System: FreeBSD freebsd 14.0-RELEASE FreeBSD 14.0-RELEASE #0 releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
[   239.743]  
[   239.743] Current version of pixman: 0.42.2
[   239.743]     Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
[   239.743] Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[   239.743] (==) Log file: "/var/log/Xorg.0.log", Time: Thu Nov 23 15:00:55 2023
[   239.743] (==) Using config file: "/usr/local/etc/X11/xorg.conf"
[   239.743] (==) Using config directory: "/usr/local/etc/X11/xorg.conf.d"
[   239.743] (==) Using system config directory "/usr/local/share/X11/xorg.conf.d"
[   239.753] (==) ServerLayout "XSPICE Example"
[   239.753] (**) |-->Screen "XSPICE Screen" (0)
[   239.753] (**) |   |-->Monitor "Configured Monitor"
[   239.753] (**) |   |-->Device "XSPICE"
[   239.753] (**) |   |-->GPUDevice "XSPICE"
[   239.753] (**) |-->Input Device "XSPICE KEYBOARD"
[   239.753] (**) |-->Input Device "XSPICE POINTER"
[   239.753] (**) Option "AutoAddDevices" "False"
[   239.753] (**) Not automatically adding devices
[   239.753] (==) Automatically enabling devices
[   239.753] (==) Automatically adding GPU devices
[   239.753] (==) Automatically binding GPU devices
[   239.753] (==) Max clients allowed: 256, resource mask: 0x1fffff
[   239.753] (==) FontPath set to:
    /usr/local/share/fonts/misc/,
    /usr/local/share/fonts/TTF/,
    /usr/local/share/fonts/OTF/,
    /usr/local/share/fonts/Type1/,
    /usr/local/share/fonts/100dpi/,
    /usr/local/share/fonts/75dpi/,
    catalogue:/usr/local/etc/X11/fontpath.d
[   239.753] (==) ModulePath set to "/usr/local/lib/xorg/modules"
[   239.753] (==) |-->Input Device "<default pointer>"
[   239.753] (==) |-->Input Device "<default keyboard>"
[   239.753] (==) The core pointer device wasn't specified explicitly in the layout.
    Using the default mouse configuration.
[   239.753] (==) The core keyboard device wasn't specified explicitly in the layout.
    Using the default keyboard configuration.
[   239.753] (II) Module ABI versions:
[   239.753]     X.Org ANSI C Emulation: 0.4
[   239.753]     X.Org Video Driver: 25.2
[   239.753]     X.Org XInput driver : 24.4
[   239.753]     X.Org Server Extension : 10.0
[   239.753] (--) PCI:*(0@0:1:0) 1b36:0100:1af4:1100 rev 5, Mem @ 0xc4000000/67108864, 0xc0000000/67108864, 0xc8d80000/8192, I/O @ 0x0000c0c0/32, BIOS @ 0x????????/65536
[   239.753] (II) LoadModule: "glx"
[   239.753] (II) Loading /usr/local/lib/xorg/modules/extensions/libglx.so
[   239.753] (II) Module glx: vendor="X.Org Foundation"
[   239.753]     compiled for 1.21.1.9, module version = 1.0.0
[   239.753]     ABI class: X.Org Server Extension, version 10.0
[   239.753] (II) LoadModule: "spiceqxl"
[   239.753] (II) Loading /usr/local/lib/xorg/modules/drivers/spiceqxl_drv.so
[   239.753] (II) Module spiceqxl: vendor="X.Org Foundation"
[   239.753]     compiled for 1.21.1.9, module version = 0.1.6
[   239.753]     Module class: X.Org Video Driver
[   239.753]     ABI class: X.Org Video Driver, version 25.2
[   239.753] (II) LoadModule: "xspicekeyboard"
[   239.753] (WW) Warning, couldn't open module xspicekeyboard
[   239.753] (EE) Failed to load module "xspicekeyboard" (module does not exist, 0)
[   239.753] (II) LoadModule: "xspicepointer"
[   239.753] (WW) Warning, couldn't open module xspicepointer
[   239.753] (EE) Failed to load module "xspicepointer" (module does not exist, 0)
[   239.753] (II) LoadModule: "mouse"
[   239.753] (II) Loading /usr/local/lib/xorg/modules/input/mouse_drv.so
[   239.753] (II) Module mouse: vendor="X.Org Foundation"
[   239.753]     compiled for 1.21.1.9, module version = 1.9.3
[   239.753]     Module class: X.Org XInput Driver
[   239.753]     ABI class: X.Org XInput driver, version 24.4
[   239.753] (II) LoadModule: "kbd"
[   239.753] (II) Loading /usr/local/lib/xorg/modules/input/kbd_drv.so
[   239.753] (II) Module kbd: vendor="X.Org Foundation"
[   239.753]     compiled for 1.21.1.9, module version = 1.9.0
[   239.753]     Module class: X.Org XInput Driver
[   239.753]     ABI class: X.Org XInput driver, version 24.4
[   239.753] (WW) Falling back to old probe method for spiceqxl
[   239.753] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support
[   239.753] (II) spiceqxl(0): Creating default Display subsection in Screen section
    "XSPICE Screen" for depth/fbbpp 24/32
[   239.753] (**) spiceqxl(0): Depth 24, (--) framebuffer bpp 32
[   239.753] (==) spiceqxl(0): RGB weight 888
[   239.753] (==) spiceqxl(0): Default visual is TrueColor
[   239.753] (==) spiceqxl(0): Using gamma correction (1.0, 1.0, 1.0)
[   239.753] (DB) xf86MergeOutputClassOptions unsupported bus type 0
[   239.753] (II) spiceqxl(0): QXL_NUM_HEADS not configured, defaulting to 1
[   239.753] (II) spiceqxl(0): Deferred Frames: Disabled
[   239.753] (II) spiceqxl(0): Offscreen Surfaces: Disabled
[   239.753] (II) spiceqxl(0): Image Cache: Disabled
[   239.753] (II) spiceqxl(0): Fallback Cache: Disabled
[   239.753] (II) spiceqxl(0): framebuffer at 0xdcd4008830 (0 KB)
[   239.753] (II) spiceqxl(0): command ram at 0xdcd4008830 (4194296 KB)
[   239.753] (II) spiceqxl(0): vram at 0xdcd4008828 (0 KB)
[   239.753] (II) spiceqxl(0): rom at 0xdcd4665380
[   239.753] resizing surface0 to 0
[   239.753] memory space from 0xdcd4008830 to 0xdcd4005830
[   239.753] memory space from 0xdcd4008828 to 0xdcd4008828
[   239.753] (EE) Segmentation fault at address 0xddd4006830
[   239.753] (EE) 
Fatal server error:
[   239.753] (EE) Caught signal 11 (Segmentation fault). Server aborting
[   239.753] (EE) 
[   239.753] (EE) 
Please consult the The X.Org Foundation support 
     at http://wiki.x.org
 for help. 
[   239.753] (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
[   239.753] (EE)

I've also seen some weird behaviours in the qxl driver when reading the device options... maybe the white rabbit hole is deeper than I thought

I've removed all the .xinitrc, xfce, slim... just clean xorg installation from pkgs and xf86-video-qxl from ports.
By the way I'm on FreeBSD 14.0 targeting pkg and ports "latest" branches.. the host is a Ubuntu 22.04LTS running the 6.5.0-1008-oem Ubuntu kernel and QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.15)

_martin · Nov 23, 2023

Well you know you are falling into some deep sh^H^Hrabbit hole when you have to debug X

Very strange that changing those numbers don't have any effect. Can you make an obvious typo in the config file and try starting it that way? Or try maybe starting X with the -configure or specify custom path to file with -config. Check for possible local config overriding default configs (e.g. config in ~).

anubsd said:
By the way I'm on FreeBSD 14.0 targeting pkg and ports "latest" branches.

I left it on vanilla settinsgs, i.e. quarterly. Is there any difference now when 14 is still hot? But when I think about it most of the stuff (X) I had to recompile to have debugging symbols. Ports where sources by git, so I think I'm on somewhat latest too then.

anubsd · Nov 23, 2023

Yes I've already tried putting a typo in the xorg.conf and it failed with a parse error, so it was reading from the correct xorg.conf file (as stated also in the log file --> Using config file: "/usr/local/etc/X11/xorg.conf"). But the fun wasn't ending there, because debugging the spiceqxl driver I saw that the parsing of those pesky *size variables always failed, returning 0 (even if I was reading from the correct file and the option were there!).
Debug, debug, debug... and there it is: the qxl driver tries to read those variables from the file using the xf86GetOptValInteger() function and, if it fails, tries to read from env. So I defined this in the .shrc:

Code:

export QXL_FRAME_BUFFER_SIZE=16
export QXL_SURFACE_BUFFER_SIZE=128
export QXL_COMMAND_BUFFER_SIZE=128
export XSPICE_PORT=5900

relogged, and... whoaaaa, X started.

So, why reading from xorg.conf fails? Well I had both the files

Code:

/usr/local/etc/X11/xorg.conf.d/spiceqxl.xorg.conf

and the

Code:

/usr/local/etc/X11/xorg.conf

with the former having almost everything commented out and the second has the content I posted in the previous comment. The xorg.conf man page says that the definitons in the latter path have the highest priority, but it seems it's not true for spiceqxl custom options.

So, I removed the env variables, removed the xorg.conf and uncommented relevant lines in the xorg.conf.d/spiceqxl.xorg.conf file... and X started again without problems.

_martin · Nov 24, 2023

Sometimes it helps to remove /tmp/.X11-unix directory completely before restarting X. I noticed some weird behavior when I left it there.

Are you able to spawn X session from slim too ?

anubsd · Nov 27, 2023

SLIM starts but then crashes entering a session (root or normal user doesn't matter).
Xfce without SLIM crashes almost immediately with a "bus error" (log doesn't seem to have any meaningful info).
I think openbox without a login manager will do just fine for me

_martin · Nov 28, 2023

Yop, so the same as I'm experiencing. The "bus" error is because most likely address it crashed on is not a canonical address. ; in my example %rax had literal string in it (0x78457475706e4958). I attached debugger to a live session; the code that was called where it crashes is being called way too often (and data was ok everytime I checked). Then suddenly it crahed becase data was bogus. It had to be something during or after keyboard init finishes.
However really debugging it would require me to understand what X actually does. And that doesn't sound like fun to me. I was not able to tell if X is victim or a cause of the bug either.

I tried to uninstall fonts (string that it "crashed on" was a font name), but then it started to crash on 0, etc.
I had one unused 12.3 VM. I installed 12.4 ports on it, it was crashing the same way. So whatever there is it's there for a while.

anubsd said:
I think openbox without a login manager will do just fine for me

At least there's something.

Xorg SEGFAULT on 14.0 in QEMU/KVM

Attachments