Disk I/O seems to put processes into D state

eyousey · Mar 12, 2014

So I upgraded from 9.0 to 10.0 the other day. The upgrade via freebsd-update went just fine. I had applied the patches needed to do the update correctly. I'm currently running a 10.0 kernel and base system. While trying to update all of my ports ( portupgrade or portmaster) it will start the compile of the 1st program but eventually it will hang. When I do a ps, it shows the process ( rm, doxygen, tar, ruby, etc.) has changed state to D (waiting for disk input) and won't continue. It takes a reboot and a round of fsck to bring it back online. The computer runs fine in the background, only the processes needed for compiling/generating/untarring/cleaning/etc. hang. If i wait (as much as 6-8 hours) the process will still hang. If I restart the update or compile, it will stop at the same point as the previous with the same result.

It started to happen when I first ran portsnap. So, I thought something was up with it, and download the ports tarball. Tried to extract it, and it happened again. It will do it at seemingly random points. I finally got it to extract in single user mode, but it is not very practical to do a ports update while in single use mode.

I've tried turning off some of my start up services and some of my loader.conf tuning. I saw in the errata for 10.0 about the virtual machine mapping problems, and even tried that fix. It didn't work.

I'm at a loss. I haven't tested copying around large files.

The OS drive/partitions are formatted as UFS with SU+J. Smartctl doesn't show any concerns with the hardware.

I've thought about reinstalling from scratch, but would rather avoid that. What if I go through all that work, and it is still doing the same thing.

uname -a

Code:

FreeBSD foobar.org 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014     root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Let me know if you need any additional info. I'm not entirely sure what all would need to be posted up in a case like this.

Thanks for any help rendered.

Uniballer · Mar 12, 2014

Are you running on real hardware, or a virtual machine? Tell us the details.

What is in your loader.conf?

What is in your rc.conf?

eyousey · Mar 12, 2014

This is running on real hardware. I used the VM errata in the hopes it might work... it didn't.

The " #commented out while debugging this I/O thing" is to disable services until I can figure out what is going on.

loader.conf

Code:

snd_hda_load="YES"
##nvidia_load="YES"  #commented out while debugging this I/O thing
linux_load="YES"

# for Firefox 3.5 and HTML5
sem_load="YES"

# for cd burner
atapicam_load="YES"

# for system monitor
coretemp_load="YES"
smb_load="YES"

rc.conf

Code:

# Networking
gateway_enable="YES"
hostname="foobar.org"
ifconfig_em1="DHCP"
ifconfig_re0="inet 192.168.1.20 netmask 255.255.255.0"
ifconfig_re0_alias0="inet 192.168.1.100 netmask 255.255.255.0"
ifconfig_lo0="inet 127.0.0.1 netmask 255.0.0.0"

# Services
inetd_enable="YES"
inetd_flags="-wW -a 192.168.1.20"
##linux_enable="YES"  #commented out while debugging this I/O thing
sshd_enable="YES"
cupsd_enable="YES"
smbd_enable="YES"
nmbd_enable="YES"
winbindd_enable="YES"
apache22_enable="YES"
##webmin_enable="YES"  #commented out while debugging this I/O thing
squid_enable="YES"
sendmail_enable="NO"
syslogd_flags="-ss" #For no listening
syslogd_flags="-a 192.168.1.20"
opendd_enable="YES"
hald_enable="YES"  #for mouse & keyboard in X
dbus_enable="YES"  #for mouse & keyboard in X
ossechids_enable="YES"
##clamav_freshclam_enable="YES"  #commented out while debugging this I/O thing
##clamav_clamd_enable="YES"  #commented out while debugging this I/O thing
##kdm4_enable="YES"  #commented out while debugging this I/O thing
ntpdate_enable="YES"
##webcamd_enable="YES"    #commented out while debugging this I/O thing
samba_enable="YES"

### SNMP daemon ###
# Be sure to understand the security implications of running SNMP v1/v2
# in your network.
bsnmpd_enable="YES"              # Run the SNMP daemon (or NO).
bsnmpd_flags=""                 # Flags for bsnmpd.

# DHCP server stuff
dhcpd_enable="YES"                          # dhcpd enabled?
dhcpd_flags="-q"                            # command option(s)
dhcpd_conf="/usr/local/etc/dhcpd.conf"      # configuration file
dhcpd_ifaces="re0"                 # ethernet interface(s)
dhcpd_withumask="022"                       # file creation mask
dhcpd_chuser_enable="YES"                   # runs w/o privileges?
dhcpd_withuser="dhcpd"                      # user name to run as
dhcpd_withgroup="dhcpd"                     # group name to run as

# PF Stuff
pf_enable="YES"                 # Enable PF (load module if required
pf_rules="/etc/pf.conf"         # rules definition file for pf
pf_program="/sbin/pfctl"        # where the pfctl program lives
pf_flags=""                     # additional flags for pfctl startup
pflog_enable="YES"              # start pflogd(8)
pflog_logfile="/var/log/pflog"  # where pflogd should store the logfile
pflog_program="/sbin/pflogd"    # where the pflogd program lives
pflog_flags="not host 142.254.148.33"   # additional flags for pflogd startup

# Stuff for PF to work
ftpproxy_enable="YES"
ftpproxy_flags=""

# Jail Stuff - disabled for now
#jail_enable="YES"  # Set to NO to disable starting of any jails
jail_enable="NO"  # Set to NO to disable starting of any jails
jail_list="www"    # Space separated list of names of jails
jail_www_rootdir="/home/jail/www"     # jail's root directory
jail_www_hostname="jail.dually.dyndns.org"  # jail's hostname
jail_www_ip="192.168.1.100"           # jail's IP address
jail_www_devfs_enable="YES"          # mount devfs in the jail
jail_www_devfs_ruleset="devfsrules_jail" # devfs ruleset to apply to jail
jail_set_hostname_allow="NO" # Allow root user in a jail to change its hostname
jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail
jail_example_exec_stop="/bin/sh /etc/rc.shutdown" # command to execute in jail for stopping
jail_example_exec_afterstart0="/bin/sh /etc/rc.d/sendmail stop"
jail_www_flags="-l -U root -s 3"

# Misc stuff
saver="blank"
compat5x_enable="YES"

# Security stuff
clear_tmp_enable="YES"
icmp_drop_redirect="YES"
icmp_log_redirect="YES"
log_in_vain="YES"
kern_securelevel_enable="YES"
kern_securelevel="-1"

# NFS stuff
nfs_client_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
rpcbind_enable="YES"
mountd_enable="YES"
mountd_flags="-r"

# NFS v4 stuff
nfsv_server_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
nfscbd_enable="YES"

# power stuff
acpi_enable="YES"
powerd_enable="YES"
powerd_flags="-a adaptive -b adaptive"
est_enable="YES"
estctrl_enable="YES"
estctrl_speed_ac="adaptive"
#estctrl_speed_battery="adaptive"
apcupsd_enable="YES"

# -- sysinstall generated deltas -- # Mon Jan 12 19:55:52 2009
moused_port="/dev/psm0"
moused_type="auto"
moused_enable="YES"

# ZFS stuff
zfs_enable="YES"

# cd burner for k3b stuff
devfs_system_ruleset="system"

# device services
dbus_enable="YES"
devd_enable="YES"
devfs_system_ruleset="10"
devfs_enable="YES"
dumpdev="NO"
hald_enable="YES"
keyrate="FAST"
moused_enable="YES"

# VirtualBox
##vboxnet_enable="YES"  #commented out while debugging this I/O thing

trh411 · Mar 12, 2014

Are you running a GENERIC or CUSTOM kernel? How much memory and swap is on this system? Do tools like top(1) and gstat(8) show anything instructive when the processes are "waiting for disk"?

eyousey · Mar 12, 2014

I'm running the GENERIC kernel. The machine has 8GB of RAM and a 4GB swap. top and gstat don't show a lot. All you see is the process using less and less processing power, it just counts down. gstat doesn't really reveal a lot either.

wblock@ · Mar 12, 2014

The first thing I'd suggest is to back up, then boot into single user mode and turn off SUJ: tunefs -j disable /dev/ada0p2. Do that for all filesystems.

Uniballer · Mar 12, 2014

When I asked for hardware details I was hoping to hear what motherboard you have, what disk drive(s) are in use, how the disk drive is connected, etc. Is the power supply big enough for the load? Does it put out excessive ripple? Have you tested RAM with something like sysutils/memtest86+? Have you tested the disk subsystem with a standalone diagnostic program like MHDD32?

eyousey · Mar 12, 2014

Uniballer...

Sorry, didn't realize that is what you were asking for.

MB: GA-P55-USB3
PS: Don't remember wattage, but do remember specing dual 12v rails.

I don't think it is so much of a hardware problem; since it was never a problem in the past with 9.0. Also, it seems to behave correctly while in single user mode (untarring ports tarball). It seems to be more a software problem, like something that is being loaded during the transition to multi-user mode.

I can try a memory and maybe a hard drive test tonight. It will depend if I can get them installed or not.

dmesg

Code:

FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
    root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Core(TM) i3 CPU         540  @ 3.07GHz (3064.55-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x20655  Family = 0x6  Model = 0x25  Stepping = 5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ae3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8263327744 (7880 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  5
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
random: <Software, Yarrow> initialized
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, cf6d0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xef00-0xef7f mem 0xf9000000-0xf9ffffff,0xd0000000-0xdfffffff,0xee000000-0xefffffff irq 16 at device 0.0 on pci1
vgapci0: Boot video device
hdac0: <NVIDIA (0x0be2) HDA Controller> mem 0xfaffc000-0xfaffffff irq 16 at device 0.1 on pci1
uhci0: <UHCI (generic) USB controller> port 0xff00-0xff1f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <UHCI (generic) USB controller> port 0xfe00-0xfe1f irq 21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <UHCI (generic) USB controller> port 0xfd00-0xfd1f irq 18 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xfbfff000-0xfbfff3ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
hdac1: <Intel 5 Series/3400 Series HDA Controller> mem 0xfbff8000-0xfbffbfff irq 22 at device 27.0 on pci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci0: <JMicron JMB363 AHCI SATA controller> mem 0xfbefe000-0xfbefffff irq 16 at device 0.0 on pci2
ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
ahci0: quirks=0x1<NOFORCE>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
atapci0: <JMicron JMB363 UDMA133 controller> port 0xcf00-0xcf07,0xce00-0xce03,0xcd00-0xcd07,0xcc00-0xcc03,0xcb00-0xcb0f irq 17 at device 0.1 on pci2
ata2: <ATA channel> at channel 0 on atapci0
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xbe00-0xbeff mem 0xfbcff000-0xfbcfffff,0xfbcf8000-0xfbcfbfff irq 17 at device 0.0 on pci3
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00200000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 1c:6f:65:98:48:2c
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib4
xhci0: <NEC uPD720200 USB 3.0 controller> mem 0xfbbfe000-0xfbbfffff irq 18 at device 0.0 on pci4
xhci0: 32 byte context size.
usbus4 on xhci0
uhci3: <UHCI (generic) USB controller> port 0xfc00-0xfc1f irq 23 at device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus5 on uhci3
uhci4: <UHCI (generic) USB controller> port 0xfb00-0xfb1f irq 19 at device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus6 on uhci4
uhci5: <UHCI (generic) USB controller> port 0xfa00-0xfa1f irq 18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus7 on uhci5
uhci6: <UHCI (generic) USB controller> port 0xf900-0xf91f irq 16 at device 29.3 on pci0
uhci6: LegSup = 0x2f00
usbus8 on uhci6
ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xfbffe000-0xfbffe3ff irq 23 at device 29.7 on pci0
usbus9: EHCI version 1.0
usbus9 on ehci1
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0xdf00-0xdf3f mem 0xfbde0000-0xfbdfffff irq 18 at device 3.0 on pci5
em0: Ethernet address: 00:07:e9:0d:04:3c
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0xde00-0xde3f mem 0xfbdc0000-0xfbddffff irq 16 at device 3.1 on pci5
em1: Ethernet address: 00:07:e9:0d:04:3d
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel 5 Series/3400 Series AHCI SATA controller> port 0xf800-0xf807,0xf700-0xf703,0xf600-0xf607,0xf500-0xf503,0xf400-0xf41f mem 0xfbffd000-0xfbffd7ff irq 19 at device 31.2 on pci0
ahci1: AHCI v1.30 with 6 3Gbps ports, Port Multiplier supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich7: <AHCI channel> at channel 5 on ahci1
ahciem0: <AHCI enclosure management bridge> on ahci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77b irq 7 drq 3 flags 0x20 on acpi0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/16 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: IEEE1284 device found /NIBBLE
ppbus0: Probing for PnP devices:
ppbus0: <Hewlett-Packard HP LaserJet 6L/0101.01> PRINTER HP ENHANCED PCL5,PJL
lpt0: <Printer> on ppbus0
lpt0: Polled port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROM> at iomem 0xd2000-0xd4fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
hdacc0: <NVIDIA GT220 HDA CODEC> at cad 0 on hdac0
hdaa0: <NVIDIA GT220 Audio Function Group> at nid 1 on hdacc0
pcm0: <NVIDIA GT220 (HDMI/DP 8ch)> at nid 5 on hdaa0
hdacc1: <NVIDIA GT220 HDA CODEC> at cad 1 on hdac0
hdaa1: <NVIDIA GT220 Audio Function Group> at nid 1 on hdacc1
pcm1: <NVIDIA GT220 (HDMI/DP 8ch)> at nid 5 on hdaa1
hdacc2: <NVIDIA GT220 HDA CODEC> at cad 2 on hdac0
hdaa2: <NVIDIA GT220 Audio Function Group> at nid 1 on hdacc2
pcm2: <NVIDIA GT220 (HDMI/DP 8ch)> at nid 5 on hdaa2
hdacc3: <NVIDIA GT220 HDA CODEC> at cad 3 on hdac0
hdaa3: <NVIDIA GT220 Audio Function Group> at nid 1 on hdacc3
pcm3: <NVIDIA GT220 (HDMI/DP 8ch)> at nid 5 on hdaa3
hdacc4: <Realtek ALC892 HDA CODEC> at cad 2 on hdac1
hdaa4: <Realtek ALC892 Audio Function Group> at nid 1 on hdacc4
pcm4: <Realtek ALC892 (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and 24,26 on hdaa4
pcm5: <Realtek ALC892 (Front Analog)> at nid 27 and 25 on hdaa4
pcm6: <Realtek ALC892 (Rear Digital)> at nid 30 and 31 on hdaa4
pcm7: <Realtek ALC892 (Onboard Digital)> at nid 17 on hdaa4
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 5.0Gbps Super Speed USB v3.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 12Mbps Full Speed USB v1.0
usbus8: 12Mbps Full Speed USB v1.0
usbus9: 480Mbps High Speed USB v2.0
ugen1.1: <Intel> at usbus1
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen0.1: <Intel> at usbus0
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen4.1: <0x1033> at usbus4
uhub2: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus4
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <Intel> at usbus2
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen7.1: <Intel> at usbus7
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus7
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen5.1: <Intel> at usbus5
uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen9.1: <Intel> at usbus9
uhub8: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus9
ugen8.1: <Intel> at usbus8
uhub9: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus8
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD1600AAJS-22PSA0 05.06H05> ATA-7 SATA 2.x device
ada0: Serial Number WD-WMAP9E758462
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ata2 bus 0 scbus2 target 0 lun 0
ada1: <SAMSUNG SP2514N VF100-41> ATA-7 device
ada1: Serial Number S08BJ1JL416491
ada1: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
ada1: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad8
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <WDC WD10EADS-00L5B1 01.01A01> ATA-8 SATA 2.x device
ada2: Serial Number WD-WCAU46091201
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad10
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <WDC WD10EADS-00L5B1 01.01A01> ATA-8 SATA 2.x device
ada3: Serial Number WD-WCAU46286319
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad12
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <WDC WD10EADS-00L5B1 01.01A01> ATA-8 SATA 2.x device
ada4: Serial Number WD-WCAU46299551
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad14
ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
ada5: <WDC WD10EADS-00L5B1 01.01A01> ATA-8 SATA 2.x device
ada5: Serial Number WD-WCAU46271608
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada5: Previously was known as ad16
ses0 at ahciem0 bus 0 scbus9 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
cd0 at ata2 bus 0 scbus2 target 1 lun 0
cd0: <HL-DT-ST DVD-RAM GH20LP20 1.02> Removable CD-ROM SCSI-0 device
cd0: 66.700MB/s transfers (UDMA4, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
uhub2: 4 ports with 4 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
Netvsc initializing... SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1532276760 Hz quality 1000
uhub9: 2 ports with 2 removable, self powered
Root mount waiting for: usbus9 usbus3
uhub3: 6 ports with 6 removable, self powered
Root mount waiting for: usbus9 usbus3
uhub8: 8 ports with 8 removable, self powered
Trying to mount root from ufs:/dev/ada0s1a [rw]...
WARNING: / was not properly dismounted
ugen0.2: <Microsoft> at usbus0
ugen7.2: <American Power Conversion> at usbus7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ums0: <Microsoft Microsoft Wheel Mouse Optical, class 0/0, rev 1.10/1.21, addr 2> on usbus0
ums0: 3 buttons and [XYZ] coordinates ID=0

eyousey · Mar 12, 2014

wblock@ said:
The first thing I'd suggest is to back up, then boot into single user mode and turn off SUJ: tunefs -j disable /dev/ada0p2. Do that for all filesystems.

I did think about turning off journalling the other night. My understanding is that in, I think 9.0, it became stable to use both at the same time. I'll try it tonight and see what happens. This is probably one of the quicker tests.

eyousey · Mar 13, 2014

Well... it seems I may have found the problem. It seems to be something I did a *LONG* time ago to the system (FreeBSD 8ish timing). Back when the hardware was older and had less performance than it does now, I did some system tuning. It seems that FreeBSD 10.0 didn't like that tuning.

I first tried turning off journaling. I was being really hopeful that this might fix it. I fired up the compile that was killing it in the past and it died about midway again. Ok... I give up... going to fix another error I was seeing during start. In the process of Googling/remembering where that option was stored I remembered I did some tuning on the system years ago to improve performance. While it helped back then, it seems to have been killing me today. While digging around in my /etc/sysctl.conf I saw some hard drive tuning parameters I had put in there. I turned those off, changed a few other things around, and rebooted. Fired up the compile again... and... it finished. I fired up another batch of updates while I slept, and they completed with no problems.

It seems that the tuning created some sort of miscommunication between the process (rm, tar, doxygen, etc.) and the disk. While the disk was able to continue processing I/O for other processes, it would keep heavier I/O processes in a holding pattern after some amount I/O. Seems a buffer wasn't flushing or it kept waiting. Not too sure. I'm familiar with FreeBSD (14+ years of use), but I've never dug around too much under the hood.

I've still got lots of ports to update. Tonight I plan to do more testing. I'm going to turn most of the stuff I turned off, back on. Maybe even journalling, and see if truly was that tuning I did so long ago.

The below was giving me an error during boot, so I commented it out of my sysctl.conf

Code:

kern.ipc.nmbclusters=32768

Here are the hard drive specific ones that I also commented out of my sysctl.conf

Code:

vfs.read_max=128
vfs.hirunningspace=4194304

FreeBSD 10.0 default now.

Code:

vfs.read_max=64
vfs.hirunningspace=13565952