How to add source file infomation to kgdb for my kernel module

snaily · Jan 20, 2010

Hi,

I have added a module to kernel,but it always crashed after running for some hous.I got the

problem function entry from the dump file vmcore.0 in kdbg.But when I list the source

code.It says "No source file for address XXXX".

Some suggestion from the maillist says that use asf command to add source file info to kgdb.
However, after I try
"asf -N /boot/kernel/kernel -M /var/crash/vmcore.0 -s /boot/kernel"
it prompts "asf: module mymodule.ko not found in search path"
I know little about this command and don't get enough message from man manual.

Any help or other suggestions will be appreciate!

PseudoCylon · Jan 21, 2010

snaily said:
It says "No source file for address XXXX".

I get the same message when the source code isn't accessible by kgdb. I don't know how you set up, but I save code on safe day-to-day-use box, and compile, run module, and debug on sacrificial box. When I forget to mount NFS (This is how make reads souce code.) before running kgdb, it says "no source".

snaily said:
However, after I try
"asf -N /boot/kernel/kernel -M /var/crash/vmcore.0 -s /boot/kernel"
it prompts "asf: module mymodule.ko not found in search path"

This sounds like mymodule.ko and mymodule.ko.symbols aren't in /boot/kernel. kgdb's default search directory is /boot/kernel/ If your symbol file is in different directory, you need to tell kgdb where it is.

If you add debug flag in Makefile, 'make' will run 'asf', and 'make install' will install .ko and .ko.symbols files (copy them into /boot/kernel/) Then, just running kgdb should do the trick.

--correction Jan 29,2010--
make doesn't run 'asf'. It run 'awk'. Those are totally different command. But, it looks the same isn't it? 3 letters begins with 'a'. Sorry for mistake.

snaily · Jan 25, 2010

PseudoCylon said:
If you add debug flag in Makefile, 'make' will run 'asf', and 'make install' will install .ko and .ko.symbols files (copy them into /boot/kernel/) Then, just running kgdb should do the trick.

3ku for answering.I find the following message in kmod.mk

Code:

# EXPORT_SYMS	A list of symbols that should be exported from the module,
#		or the name of a file containing a list of symbols, or YES
#		to export all symbols.  If not defined, no symbols are
#		exported.

I set EXPORT_SYMS=YES in Makefile,but got an empty export_syms file except the word "YES".What's wrong?

PseudoCylon · Jan 26, 2010

Well, you need

Code:

DEBUG_FLAGS = -g

(please read man gcc(1) for more options, i.e. you can set levels, like -g3.)
Then just simply run % make, there will be mymodule.ko, mymodule.ko.symbols, and mymodule.ko.debug in make dir. ("make" will do every thing for you.) .ko.symbols and .ko.debug files are much larger than .ko file since they contains debug symbols. # make install will copy .ko.symbols file into /boot/kernel/ When you run kgdb, it will automatically read .ko.symbols file, and you should see the source code.

snaily · Jan 27, 2010

PseudoCylon said:
Code:

DEBUG_FLAGS = -g

I added this in my makefile and also tried other options(e.g. -g3 -gstabs+ -ggdb) as man gcc list.But got another error "c1: error: unrecognised debug output level"

I tried to modify CFLAGS+=-g and got the same error.I test this option by directly compile one source file with gcc and it excuted normally.I don't know why this happens.

PseudoCylon · Jan 28, 2010

mmm... that's strange. The flag has worked countless time on me.

Delete -g flags from Makefile then try this in make dir
% make DEBUG_FLAGS =-g
If it works there is some thing wrong in Makefile. If you copy-and-pasted, maybe some extra invisible characters (i.e. CR) was copied too. The error message usually says,

Code:

c1: error: unrecognised debug output level "[I]oops[/I]"

what does "oops" part say? If not, there is something wrong with make. I cannot think of what goes wrong.

If it works,
# make install DEBUG_FLAGS =-g
installs .ko.symbols. You should see source code in kgdb.

snaily · Jan 31, 2010

Yeah,it works.The .ko and .ko.symbols file was copied to /boot/kernel.

Code:

#make install DEBUG_FLAGS=-g3
install -o root -g wheel -m 555   mymodule.ko /boot/kernel
install -o root -g wheel -m 555   mymodule.ko.symbols /boot/kernel
kldxref /boot/kernel

Unfortunatly,I got the same error while tried to list the source code.But I can list the system function source code successfully.

Code:

I can list calltrap() while fail to MyFunc().
#5  0xc0baf2b5 in trap (frame=0xc2bfa914) at /usr/src/sys/i386/i386/trap.c:528
#6  0xc0b91aab in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#7  0xc40cbfec in MyFunc() from /boot/kernel/./mymodule.ko

I just use the command "kldload" to load my module.And make install to add the .ko and .symbols files.

Is there any operation I missed to load a module,which can lead the kgdb to find my source code path like it does in"#6"?

I tried to recompile the whole kernel and install it after "make install" my module.But this doesn't work.

BTW,the path for my source code is "/usr/src/mycode/"

My makefile is simple as following:

Code:

KMOD=#Name of KLD to build
SRCS=#List of source files.
.include<bsd.kmod.mk>

Troubled by this problem for weeks,-_-!

PseudoCylon · Feb 1, 2010

Use the commands in following order
# make DEBUG_FLAGS=-g3
# make install DEBUG_FLAGS=-g3
# kldload mymodule
Then, let it panic.
Note no "./" before "mymodule" (.ko is optional.) Once installed the module, just giving kldload(8) a name of the module is sufficient because kldload(8) can find the modules in /boot/kernel/. You don't need to cd(1) to compile dir and prepend "./" to module name.

I'm assuming
1) your Makefile and your codes are in /usr/src/mycode/ and
2) your kgdb(1) version is 6.1.1. It used to be we need to walk though linker_file list. That will be totally different story.

Otherwise, ye shall see the code.

I'm still puzzling why DEBUG_FLAGS doesn't work. Makefile usually looks like

Code:

.PATH: ${.CURDIR}/../../../relative/path/to/src/dir/

KMOD= mymodule
SRCS= mymodule.c mymodule.h #and other source files.

DEBUG_FLAGS = -g3

CFLAGS += #as you need i.e. -DWITNESS -DINVARIANT_SUPPORT -DINVARIANTS

.include<bsd.kmod.mk>

${.CURDIR} is dir where you run make(1) or where Makefile is.

snaily · Feb 6, 2010

Thanks a lot!Your assumes are right.
Now I can list the source code and print parameters successfully
Perhaps the error is caused by that I don't use the command "make install" and always load module in current directory by "./"

There is still a little trouble.Sometimes the kgdb can't backtrace all the function calls and left the following result of bt.Have you ever met it,some reasons for this appearance?

Code:

#8  0xc1006174 in ?? ()
#9  0xc2bfab38 in ?? ()

I typed the makefile all by myself just without ".PATH" and "CFLAGS",but still got the same error.

"c1: error: unrecognised debug output level "3

It's always OK when I use "DEBUG_FLAGFS" as a command line parameter.

PseudoCylon · Feb 7, 2010

High five!

You can tell kgdb where to search .symbols files if you prefer not to install. (more info can be found in kgdb's help
[CMD="(kgdb)"]help[/CMD]
But, I've found installing module is the easiest.

snaily said:
Code:

#8 0xc1006174 in ?? () #9 0xc2bfab38 in ?? ()

I get those all the time. More often than getting complete back trace. Sometime I don't get any bt. Or sometime even dump fails. I guess it depends on what caused panic. Usually, right below calltrap() is the one causing trap. That's should point right at the problem.

I don't know why DEBUG_FLAGFS doesn't work. I've been using it since 7.2 (amd64). If you are using other arch, check out sys/conf/kmod.mk. That what you are including at the end of Makefile. You might find something.