ktrace/kdump inner workings

[ahev]

Could someone describe how ktrace/kdump work internally? For example, what is the communication between the kernel space and user space parts of it?

 

[covacat]

man 2 ktrace
/sys/kern/kern_ktrace.c

 

[ahev]

But kern_ktrace.c doesn't have a main function, does it?

 

[covacat]

no, its the actual kernel implementation if you care

 

[ahev]

Of course I care, I am trying to add some functionality to the ktrace/kdump. I want to make it so that I can see the arguments of `execve` in ktrace. For that, I need to modify the way kdump works so it dumps this new information, too.

 

[cy@]

DTrace can do that for you.

 

[ahev]

But is there a way to set breakpoints on kern_ktrace.c? I would like to debug and examine the stack trace.

 

[ahev]

DTrace can do that for you.

Could you please share a sample dtrace script that I can use?

dtrace -n 'fbt::ktrsyscall:entry { stack(); }'

I have found this one. Is it good enough or can be enhanced?

 

[cy@]

Could you please share a sample dtrace script that I can use?


dtrace -n 'fbt::ktrsyscall:entry { stack(); }'


I have found this one. Is it good enough or can be enhanced?

That's all you need. What else do you need to look at?

You can also add SDT probes in the code itself and print them like this:

dtrace -n 'sdt:::l4sums /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums1 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums2 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums3 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums4 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums5 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums6 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums7 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums8 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }' \
-n 'sdt:::l4sums9 /args[0] != args[1]/ { printf("%d %d\n",args[0],args[1]); }'

In the above I was looking at a layer 4 UDP checksum issue in ipfilter. Turned out that a change to FreeBSD's IP stack required changing the when and how (by calling the same function the IP stack does) to calculate UDP checksums.

Since it's a scripting language you can do things like this by making the script executable, like a shell script.

#!/usr/sbin/dtrace -s

dtrace:::BEGIN {
        in_checkin=0;
        mycount=0;
}

fbt::ipf_nat_checkin:entry /args[0]->fin_ifp->if_xname == "fxp0"/ {
        /* stack(); */
        printf("%s\n", args[0]->fin_ifp->if_xname); */
        print(args[0]->fin_fi); */
        printf("\n");
        /* print(args[0]->fin_dat); */
        /*printf("\n"); */
        in_checkin=1;
}

fbt::ipf_nat_checkin:return /in_checkin/ {
        print(arg1);
        in_checkin=0;
        mycount++;
}

fbt::ipf_nat_inlookup:entry /in_checkin/ {
}

fbt::ipf_nat_inlookup:return /in_checkin/ {
        print(arg1);
}

fbt::ipf_nat_in:entry /in_checkin/ {
}

fbt::ipf_nat_in:return /in_checkin/ {
        print(arg1);
}

fbt::ipf_nat_update:entry /in_checkin/ {
}

sdt:::frb_natv4in /in_checkin/ {
        print(arg1);
        /* print(*args[0]); */
}

fbt::ipf_p_ftp_port:entry /in_checkin/ {
}

fbt::ipf_p_ftp_addport:entry /in_checkin/ {
}

fbt::ipf_p_ftp_addport:return /in_checkin/ {
        print(arg1);
}

dtrace:::END {
        in_checkin=0;
}

proc:::exit /mycount > 1/ {
        exit(0);
}