FreeBSD 9.2 does not boot after reboot

Hi

I have a server in production that makes two days was modified rules of PF and forget a "quick" I can not enter by ssh. Has spent two days online but today needed to access via ssh to correct this and make a few changes in websites and other software servers (have several jails with different services), thinking that it would be shortly fall, via the hosting panel is asked to do a soft reset (ctrl+alt+del) and take several minutes without seeing that services work, is remote hard reset (power failure).

Since the server does not start, cannot be accessed via ssh (was the reason for the restart, because pf.conf contained the version prior to the lockout), servers websites do not serve requests, or other service that you can check from the outside. I've tried to enter with rescue mode (mfsBSD) and I have been able to mount the partitions (I haven´t kvm and no option to do so). The server has soft raid based on GEOM with two 240GB hard disks.

I am very newbie and these are the steps that I've made and the system information; already the ideas have run me and I'm very desperate. They are already several hours of fall.

gpart
Code:
[root@rescue /mnt/log]# mount /dev/ada0p3 /mnt
[root@rescue /mnt/log]# cd /mnt/log
[root@rescue /mnt/log]# gpart show
=>  34  468862061  ada0  GPT  (224G)
  34  2014  - free -  (1.0M)
  2048  128  1  freebsd-boot  (64K)
  2176  62914560  2  freebsd-swap  (30G)
  62916736  4194304  3  freebsd-ufs  (2.0G)
  67111040  31457280  4  freebsd-ufs  (15G)
  98568320  10485760  5  freebsd-ufs  (5.0G)
  109054080  359808008  6  freebsd-ufs  (172G)
  468862088  7  - free -  (3.5K)

=>  34  468862061  ada1  GPT  (224G)
  34  2014  - free -  (1.0M)
  2048  128  1  freebsd-boot  (64K)
  2176  62914560  2  freebsd-swap  (30G)
  62916736  4194304  3  freebsd-ufs  (2.0G)
  67111040  31457280  4  freebsd-ufs  (15G)
  98568320  10485760  5  freebsd-ufs  (5.0G)
  109054080  359808008  6  freebsd-ufs  (172G)
  468862088  7  - free -  (3.5K)

=>  34  468862061  diskid/DISK-BTDA327407CT2403GN  GPT  (224G)
  34  2014  - free -  (1.0M)
  2048  128  1  freebsd-boot  (64K)
  2176  62914560  2  freebsd-swap  (30G)
  62916736  4194304  3  freebsd-ufs  (2.0G)
  67111040  31457280  4  freebsd-ufs  (15G)
  98568320  10485760  5  freebsd-ufs  (5.0G)
  109054080  359808008  6  freebsd-ufs  (172G)
  468862088  7  - free -  (3.5K)

fdisk output
Code:
[root@rescue /mnt/log]# fdisk /dev/ada0
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
  start 1, size 468862127 (228936 Meg), flag 0
  beg: cyl 0/ head 0/ sector 2;
  end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[root@rescue /mnt/log]#
[root@rescue /mnt/log]#
[root@rescue /mnt/log]# fdisk /dev/ada1
******* Working on device /dev/ada1 *******
parameters extracted from in-core disklabel are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
  start 1, size 468862127 (228936 Meg), flag 0
  beg: cyl 0/ head 0/ sector 2;
  end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[root@rescue /mnt/log]#

Load geom and gmirror status no output.
Code:
[root@rescue /mnt/log]# kldload geom_mirror
[root@rescue /mnt/log]# kldstat
Id Refs Address  Size  Name
1  5 0xffffffff80200000 1755658  kernel
2  1 0xffffffff86411000 a4c3  tmpfs.ko
3  1 0xffffffff8641c000 f9a0  geom_mirror.ko
[root@rescue /mnt/log]# gmirror status
[root@rescue /mnt/log]#

geom exist in loader.conf
Code:
[root@rescue ~]# more /mnt/boot/loader.conf
geom_mirror_load="YES"
vfs.zfs.zio.use_uma="0"
[root@rescue ~]#

File changed in /etc/ from hard disk
(
pf.conf to correct the blocking of sshd
rc.conf to add fsck in boot
fstab to change the size of /tmp from 2GB to 1GB (the last change, when I have proven that I am not able to solve the problem)
)
Code:
[root@rescue ~]# ls -ltr /mnt/etc/
-rw-r--r--  1 root  wheel  4596 Feb 15 03:22 pf.conf
-rw-r--r--  1 root  wheel  4948 Feb 15 03:56 rc.conf
-rw-r--r--  1 root  wheel  283 Feb 15 04:38 fstab

rc.conf
Code:
fsck_y_enable="YES" # added
background_fsck="NO" # added

sshd_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"

#openntpd_enable="YES"
ezjail_enable="YES"

sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

fstab
Code:
[root@rescue ~]# more /mnt/etc/fstab
/dev/mirror/gm0p2  none  swap  sw  0  0
/dev/mirror/gm0p3  /  ufs  rw,noatime  1  1
/dev/mirror/gm0p4  /usr  ufs  rw,noatime  2  2
/dev/mirror/gm0p5  /var  ufs  rw,noatime,noexec  2  2
/dev/mirror/gm0p6  /home  ufs  rw,noatime  2  2
md  /tmp  mfs  rw,noatime,noexec,-s1G  0  0

Output of dmesg related to hard disks (when loading geom not added log)
Code:
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
ada0: Serial Number BTDA3271039K2403GN
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
ada1: Serial Number BTDA327407CT2403GN
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
Contents of /var/log/messages mounted hard drive (RESUME)
Code:
Feb 15 00:59:56 srv0A ntpd[67247]: dispatch_imsg in main: pipe closed
Feb 15 00:59:56 srv0A kernel: Feb 15 00:59:56 srv0A ntpd[67247]: dispatch_imsg in main: pipe closed
Feb 15 00:59:56 srv0A syslogd: exiting on signal 15
Feb 15 00:49:13 srv0A syslogd: kernel boot file is /boot/kernel/kernel
Feb 15 00:49:13 srv0A kernel: Waiting (max 60 seconds) for system process `vnlru' to stop...done
Feb 15 00:49:13 srv0A kernel: Waiting (max 60 seconds) for system process `syncer' to stop...Syncing disks, vnodes remaining...3 3 2 2 1 1 0 0 done
Feb 15 00:49:13 srv0A kernel: Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Feb 15 00:58:04 srv0A syslogd: kernel boot file is /boot/kernel/kernel
Feb 15 00:58:04 srv0A kernel: Copyright (c) 1992-2013 The FreeBSD Project.
Feb 15 00:58:04 srv0A kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Feb 15 00:58:04 srv0A kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Feb 15 00:58:04 srv0A kernel: ada0: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
Feb 15 00:58:04 srv0A kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Feb 15 00:58:04 srv0A kernel: ada0: Command Queueing enabled
Feb 15 00:58:04 srv0A kernel: ada0: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
Feb 15 00:58:04 srv0A kernel: ada0: quirks=0x1<4K>
Feb 15 00:58:04 srv0A kernel: ada0: Previously was known as ad4
Feb 15 00:58:04 srv0A kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Feb 15 00:58:04 srv0A kernel: ada1: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
Feb 15 00:58:04 srv0A kernel: ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Feb 15 00:58:04 srv0A kernel: ada1: Command Queueing enabled
Feb 15 00:58:04 srv0A kernel: ada1: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
Feb 15 00:58:04 srv0A kernel: ada1: quirks=0x1<4K>
Feb 15 00:58:04 srv0A kernel: ada1: Previously was known as ad6
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #1 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #2 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #6 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #3 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #5 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #4 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #7 Launched!
Feb 15 00:58:04 srv0A kernel: Timecounter "TSC-low" frequency 1700035066 Hz quality 1000
Feb 15 00:58:04 srv0A kernel: GEOM_MIRROR: Device mirror/gm0 launched (2/2).
Feb 15 00:58:04 srv0A kernel: Root mount waiting for: usbus2 usbus1 usbus0
Feb 15 00:58:04 srv0A kernel: uhub1: 2 ports with 2 removable, self powered
Feb 15 00:58:04 srv0A kernel: uhub2: 2 ports with 2 removable, self powered
Feb 15 00:58:04 srv0A kernel: uhub0: 21 ports with 21 removable, self powered
Feb 15 00:58:04 srv0A kernel: ugen1.2: <vendor 0x8087> at usbus1
Feb 15 00:58:04 srv0A kernel: uhub3: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1
Feb 15 00:58:04 srv0A kernel: ugen2.2: <vendor 0x8087> at usbus2
Feb 15 00:58:04 srv0A kernel: uhub4: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2
Feb 15 00:58:04 srv0A kernel: Root mount waiting for: usbus2 usbus1
Feb 15 00:58:04 srv0A kernel: uhub3: 6 ports with 6 removable, self powered
Feb 15 00:58:04 srv0A kernel: uhub4: 8 ports with 8 removable, self powered
Feb 15 00:58:04 srv0A kernel: Trying to mount root from ufs:/dev/mirror/gm0p3 [rw,noatime]...
Feb 15 00:58:04 srv0A kernel: WARNING: / was not properly dismounted
Feb 15 00:58:04 srv0A kernel: WARNING: /usr was not properly dismounted
Feb 15 00:58:04 srv0A kernel: WARNING: /var was not properly dismounted
Feb 15 00:58:04 srv0A kernel: WARNING: /home was not properly dismounted
Feb 15 00:58:04 srv0A root: /etc/rc: WARNING: failed precmd routine for sshd
Feb 15 00:58:04 srv0A kernel: re0: link state changed to UP
Feb 15 01:01:51 srv0A fsck: /dev/mirror/gm0p4: 304128 files, 1065409 used, 2740926 free (8862 frags, 341508 blocks, 0.2% fragmentation)
Feb 15 01:01:57 srv0A fsck: /dev/mirror/gm0p5: 33612 files, 279563 used, 985844 free (23620 frags, 120278 blocks, 1.9% fragmentation)
Feb 15 01:03:08 srv0A ntpd[1329]: 1 out of 4 peers valid
Feb 15 01:03:08 srv0A ntpd[1329]: bad peer 1.europe.pool.ntp.org (193.225.190.4)
Feb 15 01:03:08 srv0A ntpd[1329]: bad peer 2.europe.pool.ntp.org (37.187.7.160)
Feb 15 01:03:08 srv0A ntpd[1329]: bad peer 3.europe.pool.ntp.org (178.63.135.195)
Feb 15 01:03:31 srv0A fsck: /dev/mirror/gm0p6: 1893255 files, 39568205 used, 3993778 free (272474 frags, 465163 blocks, 0.6% fragmentation)
Feb 15 01:06:29 srv0A kernel: .
Feb 15 01:06:29 srv0A ntpd[1330]: dispatch_imsg in main: pipe closed
Feb 15 01:06:29 srv0A syslogd: exiting on signal 15

I have executed fsck in all slices of both discs and has not found errors.

I restarted the server several times to see if it started, but I had to re-enter into rescue mode and "messages" file has not changed, I know if he starts or not, but posts it seems that server a time that if he did so.

I can only prove an occurrence, change fstab by removing the mirror and putting the slices of ada0 or ada1, but I have not tested if it could damage the contents of the discs.

You can see that I have tried almost everything, it is the proof that I have read and looked for solutions before asking, but I'm desperate because I cannot find the solution. Can someone guide me as start server? Thanks a lot.
 
FreeBSD 9.2 has reached its end-of-life and will no longer be supported by the FreeBSD Security Team. Users of FreeBSD 9.2 are strongly
encouraged to upgrade to a newer release ASAP.

Read for further details here.
 
Please, please stop using fdisk(8). You have GPT disks and it has no clue about their real layout.

You have a mirror of GPT partitions. That is a mistake. Don't do that. The Handbook shows the right way to use gmirror(8): https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom-mirror.html.

There are so many other micro-optimizations and customizations shown that it is really hard to tell what is happening.

If the system does not have a monitor so you can see what it does when booting, maybe connect a computer to the serial port to act as a terminal.
 
Many thanks to everyone for the support. After losing several hours without finding solution, choose to save data as I could, reinstall all the system and try to import data (some were lost). I have installed 10.2.

The problem see the error in the boot, is a dedicated server in a datacenter and did not have access or console, monitor, or neither option to hire a KVM or remote hands service.

Although I have already solved the problem (i.e. it was solution), I've been wanting to know the problem, so was know and learn more about this great operating system.
 
Back
Top