How to debug a hang during booting.

How would I go about finding out why FreeBSD stops during booting?

Here's what happens:-
Code:
GoFlexHome> boot
starting USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
       scanning usb for storage devices...
Use USB retry period from the environment: 15 second(s)
1 Storage Device(s) found
reading ubldr
295973 bytes read in 90 ms (3.1 MiB/s)
CACHE: Misaligned operation at range [01000098, 01030c04]
CACHE: Misaligned operation at range [01030c04, 01031707]
CACHE: Misaligned operation at range [01031708, 010330f0]
CACHE: Misaligned operation at range [010330f0, 0103558b]
CACHE: Misaligned operation at range [01035590, 01036c98]
CACHE: Misaligned operation at range [01036c98, 01036cf4]
CACHE: Misaligned operation at range [01036cf4, 01036d5c]
CACHE: Misaligned operation at range [01036d5c, 01036e30]
CACHE: Misaligned operation at range [01036e30, 01036e3c]
## Starting application at 0x01000098 ...
Consoles: U-Boot console
Compatible U-Boot API signature found @0x7b12860

FreeBSD/arm U-Boot loader, Revision 1.2
(Thu Nov 30 14:06:37 GMT 2017 root@Test)

DRAM: 128MB
Number of U-Boot devices: 2
U-Boot env: loaderdev not set, will probe all devices.
Found U-Boot device: disk
  Probing all disk devices...
  Checking unit=0 slice=<auto> partition=<auto>... good.
Booting from disk0s2:
|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-Warning: unable to open file /boot/loader.conf.local
\|/-\|/-\|/-\|/-\|/-\|/-\|/-\/boot/kernel/kernel data=0x443064+0x30f9c |/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\syms=[0x4+0x932d0|/-\+0x4+0x667c5|/-\]
fuse...|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-can't find 'fuse'
failed!
umodem...\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-can't find 'umodem'
failed!
u3g...\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|can't find 'u3g'
failed!

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel] in 4 seconds... Booting [/boot/kernel/kernel] in 3 seconds... Booting [/boot/kernel/kernel]...              
Using DTB compiled into kernel.
/-\|/Kernel entry at 0x1200100...
Kernel args: (null)
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE #0: Thu Nov 30 14:22:01 GMT 2017
    root@Test:/usr/obj/arm.arm/usr/src/sys/DB-88F6XXX arm
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
module mvs already present!
CPU: Feroceon 88FR131 rev 1 (**unknown 4** core)
  Little-endian DC enabled IC disabled WA disabled DC streaming enabled
  BTB disabled L2 enabled L2 prefetch enabled
  WB enabled LABT branch prediction disabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 134213632 (127 MB)
avail memory = 123895808 (118 MB)
SOC: Marvell 88F6281 rev A1, TClock 200MHz
  Instruction cache prefetch disabled, data cache prefetch disabled
  256KB 4-way set-associative write-through unified L2 cache
random: entropy device external interface
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
localbus0: <Marvell device bus> on ofwbus0
nand0: <Marvell NAND controller> mem 0xf9300000-0xf93fffff on localbus0
nandbus0: <NAND bus> on nand0
onand0: <ONFI compliant NAND> on nandbus0
onand0: No BBT found. Prescan chip...
#####################
ic0: <Marvell Integrated Interrupt Controller> mem 0x20200-0x2023b on simplebus0
timer0: <Marvell CPU Timer> mem 0x20300-0x2032f irq 1 on simplebus0
Event timer "CPUTimer0" frequency 200000000 Hz quality 1000
Timecounter "CPUTimer1" frequency 200000000 Hz quality 1000
gpio0: <Marvell Integrated GPIO Controller> mem 0x10100-0x1011f irq 35,36,37,38,39,40,41 on simplebus0
rtc0: <Marvell Integrated RTC> mem 0x10300-0x10307 on simplebus0
twsi0: <Marvell Integrated I2C Bus Controller> mem 0x11000-0x1101f irq 43 on simplebus0
iicbus0: <Philips I2C bus> on twsi0
iic0: <I2C generic I/O> on iicbus0
mge0: <Marvell Gigabit Ethernet controller> mem 0x72000-0x73fff irq 12,13,14,11,46 on simplebus0
mge0: PHY8 attached, phy_sc points to mge0
mge0: Ethernet address: 52:3b:20:9c:11:51
mge0: MII failed to find PHY
device_attach: mge0 attach returned 6
uart0: <16550 or compatible> mem 0x12000-0x1201f irq 33 on simplebus0
uart0: console (1056,n,8,1)
uart1: <16550 or compatible> mem 0x12100-0x1211f irq 34 on simplebus0
cesa0: <Marvell Cryptographic Engine and Security Accelerator> mem 0x30000-0x30fff,0x3d000-0x3dfff irq 22 on simplebus0
ehci0: <Marvell Integrated USB 2.0 controller> mem 0x50000-0x50fff irq 48,19 on simplebus0
usbus0: EHCI version 1.0
usbus0 on ehci0
mvs0: <Marvell 88F6281 SATA controller> mem 0x80000-0x85fff irq 21 on simplebus0
mvs0: Gen-IIe, 2 3Gbps ports, Port Multiplier supported with FBS
mvsch0: <Marvell SATA channel> at channel 0 on mvs0
mvsch1: <Marvell SATA channel> at channel 1 on mvs0
pcib0: <Marvell Integrated PCI/PCI-E Controller> mem 0xf1040000-0xf1041fff irq 44 on ofwbus0
pci0: <PCI bus> on pcib0
cryptosoft0: <software crypto>
Timecounters tick every 10.000 msec
usbus0: 480Mbps High Speed USB v2.0
~
[EOT]

This is FreeBSD/ARM on a GoFlex Home.
 
Is there some device.hints setting I can use to trace what is going on?

There is no /var/run/dmesg.boot or /var/log/dmesg.today. When do these files start being written to?
 
When do these files start being written to?
After mountroot step.

So the last thing shown is USB right...

Glad to see you getting this far. So you compiled with FDT right?
If you noticed the second post in that crucial mailing list post he put the DTS changes needed in the post.(At very bottom)
This will allow for SATA.
You should be able to glean hints from these dmesg as well. For example what comes up after USB.
That could be the culprit.
From my looking this comes next:
ugen0.1: <Marvell EHCI root HUB> at usbus0
uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0


I would drop all the modules you are trying to load and just focus on getting the kernel to boot. fuse and u3g are not needed.
 
/usr/obj/arm.arm/usr/src/sys/DB-88F6XXX arm
This concerns me as I look closer at the mailing list post.
You might need a different kernel build.
They are using DOCKSTAR kernel and customizing it.
The DB-88F6XXX kernel is included with DOCKSTAR kernel with this kernconf line:
include "../mv/kirkwood/std.db88f6xxx"
This links the parent DB-88F6XXX and DOCKSTAR adds special settings for the device.
The mailing list post above renames DOCKSTAR to GOFLEX for his custom modifications.

Do notice the other kernconf modifications needed at the bottom of the post.
They are shown in a 'patch' format. The + adds a line and - minus removes a line.
You can hand patch these as needed.
 
After mountroot step.

So the last thing shown is USB right...

Glad to see you getting this far. So you compiled with FDT right?
make -j 8 buildworld TARGET_ARCH=arm -DWITH_FDT
make -j 8 buildkernel TARGET_ARCH=arm KERNCONF=DB-88F6XXX
If you noticed the second post in that crucial mailing list post he put the DTS changes needed in the post.(At very bottom)
This will allow for SATA.
You should be able to glean hints from these dmesg as well. For example what comes up after USB.
That could be the culprit.
From my looking this comes next:
ugen0.1: <Marvell EHCI root HUB> at usbus0
uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
It's very difficult to compare the logs, because of different versions of FreeBSD, also I'm booting from USB using bootelf to load ubldr @ 0x02000000, whilst he is booting from the network using go 0x1100000 and it's not altogether clear what he means by filename goflex.

I would drop all the modules you are trying to load and just focus on getting the kernel to boot. fuse and u3g are not needed.

That was fairly straightforward. I just commented out all the lines in /etc/boot/loader.conf
 
Everything looks good except this. You need to use DOCKSTAR kernconf.
make -j 8 buildkernel TARGET_ARCH=arm KERNCONF=DOCKSTAR


Here's what I got when building the kernel:-

Code:
--- buildkernel ---
--- buildkernel ---
--------------------------------------------------------------
>>> Kernel build for DOCKSTAR started on Fri Mar 22 03:10:29 UTC 2019
--------------------------------------------------------------
===> DOCKSTAR
mkdir -p /usr/obj/arm.arm/usr/src/sys
--------------------------------------------------------------
>>> stage 1: configuring the kernel
--------------------------------------------------------------
cd /usr/src/sys/arm/conf;  PATH=/usr/obj/arm.arm/usr/src/tmp/legacy/usr/sbin:/usr/obj/arm.arm/usr/src/tmp/legacy/usr/bin:/usr/obj/arm.arm/usr/src/tmp/legacy/bin:/usr/obj
/arm.arm/usr/src/tmp/usr/sbin:/usr/obj/arm.arm/usr/src/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin  config  -d /usr/obj/arm.arm/usr/src/sys/DOCKSTAR  -I '/usr/src/sys/arm/
conf' '/usr/src/sys/arm/conf/DOCKSTAR'
/usr/src/sys/arm/conf/DOCKSTAR: unknown option "IPSEC_NAT_T"
*** [buildkernel] Error code 1

make[1]: stopped in /usr/src
1 error

make[1]: stopped in /usr/src
*** [buildkernel] Error code 2

make: stopped in /usr/src
1 error

make: stopped in /usr/src

Command exit status: 2
Script done on Fri Mar 22 03:10:29 2019

I've seen references to IPSEC_NAT_T but can't recall what to do. Will check.
 
After commenting out some line referring to IPSEC_NAT_T in /usr/src/sys/arm/conf/DOCKSTAR

Code:
Script started on Fri Mar 22 11:25:42 2019
Command: cu -l /dev/cuaU0 -s 115200
Connected


U-Boot 2017.05-tld-2 (Jul 26 2017 - 02:37:42 -0700)
Seagate GoFlex Home

SoC:   Kirkwood 88F6281_A1
DRAM:  128 MiB
WARNING: Caches not enabled
NAND:  256 MiB
In:    serial
Out:   serial
Err:   serial
Net:   egiga0
88E1116 Initialized on egiga0
Hit any key to stop autoboot: 10  9  8  7  6  5  4  3  2  1  0
starting USB...
USB0:   USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
       scanning usb for storage devices...
Use USB retry period from the environment: 15 second(s)
1 Storage Device(s) found

Reset IDE: ide_preinit failed

Partition Map for USB device 0  --   Partition Type: DOS

Part    Start Sector    Num Sectors    UUID        Type
  1    63            65536         00000000-01    0c
  2    65599         3841984       00000000-02    a5

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0

## Unknown partition table type 0
loading envs from usb 0 ...
reading /boot/uEnv.txt
160 bytes read in 63 ms (2 KiB/s)

no IDE devices available
importing envs ...
running scan_disk ...
Scan device usb
device usb 0:1
reading /boot/uImage
** Unable to read file /boot/uImage **
device usb 1:1
** Bad device usb 1 **
device usb 2:1
** Bad device usb 2 **
device usb 3:1
** Bad device usb 3 **
Scan device ide

Reset IDE: ide_preinit failed
device ide 0:1
** Bad device ide 0 **
device ide 1:1
** Bad device ide 1 **
device ide 2:1
** Bad device ide 2 **
device ide 3:1
** Bad device ide 3 **
reading ubldr
295973 bytes read in 85 ms (3.3 MiB/s)
CACHE: Misaligned operation at range [01000098, 01030c04]
CACHE: Misaligned operation at range [01030c04, 01031707]
CACHE: Misaligned operation at range [01031708, 010330f0]
CACHE: Misaligned operation at range [010330f0, 0103558b]
CACHE: Misaligned operation at range [01035590, 01036c98]
CACHE: Misaligned operation at range [01036c98, 01036cf4]
CACHE: Misaligned operation at range [01036cf4, 01036d5c]
CACHE: Misaligned operation at range [01036d5c, 01036e30]
CACHE: Misaligned operation at range [01036e30, 01036e3c]
## Starting application at 0x01000098 ...
Consoles: U-Boot console 


Compatible U-Boot API signature found @0x7b12860





FreeBSD/arm U-Boot loader, Revision 1.2


(Thu Nov 30 14:06:37 GMT 2017 root@Test)





DRAM: 128MB


Number of U-Boot devices: 2


U-Boot env: loaderdev not set, will probe all devices.


Found U-Boot device: disk


  Probing all disk devices...


  Checking unit=0 slice=<auto> partition=<auto>... good.


Booting from disk0s2:


|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-/boot/kernel/kernel data=0x684724+0x378dc \|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|syms=[0x4+0xaa940/-\|/+0x4+0x88e8b-\|/]





Hit [Enter] to boot immediately, or any other key for command prompt.



Booting [/boot/kernel/kernel] in 1 second...
Booting [/boot/kernel/kernel]...               


Using DTB compiled into kernel.


-\|/Kernel entry at 0x1200100...


Kernel args: (null)


Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-RELEASE #0: Fri Mar 22 10:56:27 UTC 2019
    root@S07:/usr/obj/arm.arm/usr/src/sys/DOCKSTAR arm
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
CPU: Feroceon 88FR131 rev 1 (**unknown 4** core)
  Little-endian DC enabled IC disabled WA disabled DC streaming enabled
  BTB disabled L2 enabled L2 prefetch enabled
  WB enabled LABT branch prediction disabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 134213632 (127 MB)
avail memory = 121315328 (115 MB)
SOC: Marvell 88F6281 rev A1, TClock 200MHz
  Instruction cache prefetch disabled, data cache prefetch disabled
  256KB 4-way set-associative write-through unified L2 cache
random: entropy device external interface
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
localbus0: <Marvell device bus> on ofwbus0
ic0: <Marvell Integrated Interrupt Controller> mem 0x20200-0x2023b on simplebus0
timer0: <Marvell CPU Timer> mem 0x20300-0x2032f irq 1 on simplebus0
Event timer "CPUTimer0" frequency 200000000 Hz quality 1000
Timecounter "CPUTimer1" frequency 200000000 Hz quality 1000
gpio0: <Marvell Integrated GPIO Controller> mem 0x10100-0x1011f irq 35,36,37,38,39,40,41 on simplebus0
rtc0: <Marvell Integrated RTC> mem 0x10300-0x10307 on simplebus0
rtc0: registered as a time-of-day clock, resolution 1.000000s
twsi0: <Marvell Integrated I2C Bus Controller> mem 0x11000-0x1101f irq 43 on simplebus0
iicbus0: <Philips I2C bus> on twsi0
iic0: <I2C generic I/O> on iicbus0
mge0: <Marvell Gigabit Ethernet controller> mem 0x72000-0x73fff irq 12,13,14,11,46 on simplebus0
mge0: PHY0 attached, phy_sc points to mge0
mge0: Ethernet address: 52:3b:20:9c:11:51
miibus0: <MII bus> on mge0
e1000phy0: <Marvell 88E1116R Gigabit PHY> PHY 0 on miibus0
e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
uart0: <16550 or compatible> mem 0x12000-0x1201f irq 33 on simplebus0
uart0: console (1056,n,8,1)
uart1: <16550 or compatible> mem 0x12100-0x1211f irq 34 on simplebus0
cesa0: <Marvell Cryptographic Engine and Security Accelerator> mem 0x30000-0x30fff,0x3d000-0x3dfff irq 22 on simplebus0
ehci0: <Marvell Integrated USB 2.0 controller> mem 0x50000-0x50fff irq 48,19 on simplebus0
usbus0: EHCI version 1.0
usbus0 on ehci0
cryptosoft0: <software crypto>
Timecounters tick every 10.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0 with IPv6 initialized (100409)
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
usbus0: 480Mbps High Speed USB v2.0
Trying to mount root from ufs:/dev/da0s2 [rw,noclusterr,noclusterw]...
Root mount waiting for: usbus0
ugen0.1: <Marvell EHCI root HUB> at usbus0
uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <Generic USB2.0-CRW> at usbus0
umass0 on uhub0
umass0: <Bulk-In, Bulk-Out, Interface> on usbus0
mountroot: waiting for device /dev/da0s2...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Generic- SD/MMC/MS PRO 1.00> Removable Direct Access SPC-2 SCSI device
da0: Serial Number 20120926571200000
da0: 40.000MB/s transfers
da0: 15193MB (31116288 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
Mar 22 11:26:47 init: login_getclass: unknown class 'daemon'

sh: cannot open /etc/rc: No such file or directory
Enter full pathname of shell or RETURN for /bin/sh:
#
# ls
.snap    boot    etc    libexec    mnt    proc    root    sys    usr
bin    dev    lib    media    net    rescue    sbin    tmp    var
# pwd
/
# ls /etc
X11            gss            rc.conf.d
autofs            mail            rc.d
bluetooth        mtree            security
casper            newsyslog.conf.d    skel
cron.d            ntp            ssh
defaults        pam.d            ssl
devd            periodic        syslog.d
dma            pkg            zfs
fstab            ppp
# whoami
0
# reboot
~
[EOT]

Command exit status: 0
Script done on Fri Mar 22 11:27:41 2019

Good progress!

Not sure why you suggested using KERNCONF=DOCKSTAR, and to be honest I didn't think it would make any difference.

Now, I'm left wondering what the boot process is doing looking for /etc/rc and what tells it to look there.
 
Back
Top