Hi
I have a server in production that makes two days was modified rules of PF and forget a "quick" I can not enter by ssh. Has spent two days online but today needed to access via ssh to correct this and make a few changes in websites and other software servers (have several jails with different services), thinking that it would be shortly fall, via the hosting panel is asked to do a soft reset (ctrl+alt+del) and take several minutes without seeing that services work, is remote hard reset (power failure).
Since the server does not start, cannot be accessed via ssh (was the reason for the restart, because pf.conf contained the version prior to the lockout), servers websites do not serve requests, or other service that you can check from the outside. I've tried to enter with rescue mode (mfsBSD) and I have been able to mount the partitions (I haven´t kvm and no option to do so). The server has soft raid based on GEOM with two 240GB hard disks.
I am very newbie and these are the steps that I've made and the system information; already the ideas have run me and I'm very desperate. They are already several hours of fall.
Load
geom exist in loader.conf
File changed in /etc/ from hard disk
(
pf.conf to correct the blocking of sshd
rc.conf to add fsck in boot
fstab to change the size of /tmp from 2GB to 1GB (the last change, when I have proven that I am not able to solve the problem)
)
rc.conf
fstab
Output of
Contents of /var/log/messages mounted hard drive (RESUME)
I have executed
I restarted the server several times to see if it started, but I had to re-enter into rescue mode and "messages" file has not changed, I know if he starts or not, but posts it seems that server a time that if he did so.
I can only prove an occurrence, change fstab by removing the mirror and putting the slices of ada0 or ada1, but I have not tested if it could damage the contents of the discs.
You can see that I have tried almost everything, it is the proof that I have read and looked for solutions before asking, but I'm desperate because I cannot find the solution. Can someone guide me as start server? Thanks a lot.
I have a server in production that makes two days was modified rules of PF and forget a "quick" I can not enter by ssh. Has spent two days online but today needed to access via ssh to correct this and make a few changes in websites and other software servers (have several jails with different services), thinking that it would be shortly fall, via the hosting panel is asked to do a soft reset (ctrl+alt+del) and take several minutes without seeing that services work, is remote hard reset (power failure).
Since the server does not start, cannot be accessed via ssh (was the reason for the restart, because pf.conf contained the version prior to the lockout), servers websites do not serve requests, or other service that you can check from the outside. I've tried to enter with rescue mode (mfsBSD) and I have been able to mount the partitions (I haven´t kvm and no option to do so). The server has soft raid based on GEOM with two 240GB hard disks.
I am very newbie and these are the steps that I've made and the system information; already the ideas have run me and I'm very desperate. They are already several hours of fall.
gpart
Code:
[root@rescue /mnt/log]# mount /dev/ada0p3 /mnt
[root@rescue /mnt/log]# cd /mnt/log
[root@rescue /mnt/log]# gpart show
=> 34 468862061 ada0 GPT (224G)
34 2014 - free - (1.0M)
2048 128 1 freebsd-boot (64K)
2176 62914560 2 freebsd-swap (30G)
62916736 4194304 3 freebsd-ufs (2.0G)
67111040 31457280 4 freebsd-ufs (15G)
98568320 10485760 5 freebsd-ufs (5.0G)
109054080 359808008 6 freebsd-ufs (172G)
468862088 7 - free - (3.5K)
=> 34 468862061 ada1 GPT (224G)
34 2014 - free - (1.0M)
2048 128 1 freebsd-boot (64K)
2176 62914560 2 freebsd-swap (30G)
62916736 4194304 3 freebsd-ufs (2.0G)
67111040 31457280 4 freebsd-ufs (15G)
98568320 10485760 5 freebsd-ufs (5.0G)
109054080 359808008 6 freebsd-ufs (172G)
468862088 7 - free - (3.5K)
=> 34 468862061 diskid/DISK-BTDA327407CT2403GN GPT (224G)
34 2014 - free - (1.0M)
2048 128 1 freebsd-boot (64K)
2176 62914560 2 freebsd-swap (30G)
62916736 4194304 3 freebsd-ufs (2.0G)
67111040 31457280 4 freebsd-ufs (15G)
98568320 10485760 5 freebsd-ufs (5.0G)
109054080 359808008 6 freebsd-ufs (172G)
468862088 7 - free - (3.5K) fdisk output
Code:
[root@rescue /mnt/log]# fdisk /dev/ada0
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
start 1, size 468862127 (228936 Meg), flag 0
beg: cyl 0/ head 0/ sector 2;
end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[root@rescue /mnt/log]#
[root@rescue /mnt/log]#
[root@rescue /mnt/log]# fdisk /dev/ada1
******* Working on device /dev/ada1 *******
parameters extracted from in-core disklabel are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=465141 heads=16 sectors/track=63 (1008 blks/cyl)
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
start 1, size 468862127 (228936 Meg), flag 0
beg: cyl 0/ head 0/ sector 2;
end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[root@rescue /mnt/log]#Load
geom and gmirror status no output.
Code:
[root@rescue /mnt/log]# kldload geom_mirror
[root@rescue /mnt/log]# kldstat
Id Refs Address Size Name
1 5 0xffffffff80200000 1755658 kernel
2 1 0xffffffff86411000 a4c3 tmpfs.ko
3 1 0xffffffff8641c000 f9a0 geom_mirror.ko
[root@rescue /mnt/log]# gmirror status
[root@rescue /mnt/log]#geom exist in loader.conf
Code:
[root@rescue ~]# more /mnt/boot/loader.conf
geom_mirror_load="YES"
vfs.zfs.zio.use_uma="0"
[root@rescue ~]#File changed in /etc/ from hard disk
(
pf.conf to correct the blocking of sshd
rc.conf to add fsck in boot
fstab to change the size of /tmp from 2GB to 1GB (the last change, when I have proven that I am not able to solve the problem)
)
Code:
[root@rescue ~]# ls -ltr /mnt/etc/
-rw-r--r-- 1 root wheel 4596 Feb 15 03:22 pf.conf
-rw-r--r-- 1 root wheel 4948 Feb 15 03:56 rc.conf
-rw-r--r-- 1 root wheel 283 Feb 15 04:38 fstabrc.conf
Code:
fsck_y_enable="YES" # added
background_fsck="NO" # added
sshd_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
#openntpd_enable="YES"
ezjail_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"fstab
Code:
[root@rescue ~]# more /mnt/etc/fstab
/dev/mirror/gm0p2 none swap sw 0 0
/dev/mirror/gm0p3 / ufs rw,noatime 1 1
/dev/mirror/gm0p4 /usr ufs rw,noatime 2 2
/dev/mirror/gm0p5 /var ufs rw,noatime,noexec 2 2
/dev/mirror/gm0p6 /home ufs rw,noatime 2 2
md /tmp mfs rw,noatime,noexec,-s1G 0 0Output of
dmesg related to hard disks (when loading geom not added log)
Code:
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
ada0: Serial Number BTDA3271039K2403GN
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
ada1: Serial Number BTDA327407CT2403GN
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
Code:
Feb 15 00:59:56 srv0A ntpd[67247]: dispatch_imsg in main: pipe closed
Feb 15 00:59:56 srv0A kernel: Feb 15 00:59:56 srv0A ntpd[67247]: dispatch_imsg in main: pipe closed
Feb 15 00:59:56 srv0A syslogd: exiting on signal 15
Feb 15 00:49:13 srv0A syslogd: kernel boot file is /boot/kernel/kernel
Feb 15 00:49:13 srv0A kernel: Waiting (max 60 seconds) for system process `vnlru' to stop...done
Feb 15 00:49:13 srv0A kernel: Waiting (max 60 seconds) for system process `syncer' to stop...Syncing disks, vnodes remaining...3 3 2 2 1 1 0 0 done
Feb 15 00:49:13 srv0A kernel: Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Feb 15 00:58:04 srv0A syslogd: kernel boot file is /boot/kernel/kernel
Feb 15 00:58:04 srv0A kernel: Copyright (c) 1992-2013 The FreeBSD Project.
Feb 15 00:58:04 srv0A kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Feb 15 00:58:04 srv0A kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Feb 15 00:58:04 srv0A kernel: ada0: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
Feb 15 00:58:04 srv0A kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Feb 15 00:58:04 srv0A kernel: ada0: Command Queueing enabled
Feb 15 00:58:04 srv0A kernel: ada0: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
Feb 15 00:58:04 srv0A kernel: ada0: quirks=0x1<4K>
Feb 15 00:58:04 srv0A kernel: ada0: Previously was known as ad4
Feb 15 00:58:04 srv0A kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Feb 15 00:58:04 srv0A kernel: ada1: <INTEL SSDSC2BW240A4 DC02> ATA-9 SATA 3.x device
Feb 15 00:58:04 srv0A kernel: ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Feb 15 00:58:04 srv0A kernel: ada1: Command Queueing enabled
Feb 15 00:58:04 srv0A kernel: ada1: 228936MB (468862128 512 byte sectors: 16H 63S/T 16383C)
Feb 15 00:58:04 srv0A kernel: ada1: quirks=0x1<4K>
Feb 15 00:58:04 srv0A kernel: ada1: Previously was known as ad6
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #1 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #2 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #6 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #3 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #5 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #4 Launched!
Feb 15 00:58:04 srv0A kernel: SMP: AP CPU #7 Launched!
Feb 15 00:58:04 srv0A kernel: Timecounter "TSC-low" frequency 1700035066 Hz quality 1000
Feb 15 00:58:04 srv0A kernel: GEOM_MIRROR: Device mirror/gm0 launched (2/2).
Feb 15 00:58:04 srv0A kernel: Root mount waiting for: usbus2 usbus1 usbus0
Feb 15 00:58:04 srv0A kernel: uhub1: 2 ports with 2 removable, self powered
Feb 15 00:58:04 srv0A kernel: uhub2: 2 ports with 2 removable, self powered
Feb 15 00:58:04 srv0A kernel: uhub0: 21 ports with 21 removable, self powered
Feb 15 00:58:04 srv0A kernel: ugen1.2: <vendor 0x8087> at usbus1
Feb 15 00:58:04 srv0A kernel: uhub3: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1
Feb 15 00:58:04 srv0A kernel: ugen2.2: <vendor 0x8087> at usbus2
Feb 15 00:58:04 srv0A kernel: uhub4: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2
Feb 15 00:58:04 srv0A kernel: Root mount waiting for: usbus2 usbus1
Feb 15 00:58:04 srv0A kernel: uhub3: 6 ports with 6 removable, self powered
Feb 15 00:58:04 srv0A kernel: uhub4: 8 ports with 8 removable, self powered
Feb 15 00:58:04 srv0A kernel: Trying to mount root from ufs:/dev/mirror/gm0p3 [rw,noatime]...
Feb 15 00:58:04 srv0A kernel: WARNING: / was not properly dismounted
Feb 15 00:58:04 srv0A kernel: WARNING: /usr was not properly dismounted
Feb 15 00:58:04 srv0A kernel: WARNING: /var was not properly dismounted
Feb 15 00:58:04 srv0A kernel: WARNING: /home was not properly dismounted
Feb 15 00:58:04 srv0A root: /etc/rc: WARNING: failed precmd routine for sshd
Feb 15 00:58:04 srv0A kernel: re0: link state changed to UP
Feb 15 01:01:51 srv0A fsck: /dev/mirror/gm0p4: 304128 files, 1065409 used, 2740926 free (8862 frags, 341508 blocks, 0.2% fragmentation)
Feb 15 01:01:57 srv0A fsck: /dev/mirror/gm0p5: 33612 files, 279563 used, 985844 free (23620 frags, 120278 blocks, 1.9% fragmentation)
Feb 15 01:03:08 srv0A ntpd[1329]: 1 out of 4 peers valid
Feb 15 01:03:08 srv0A ntpd[1329]: bad peer 1.europe.pool.ntp.org (193.225.190.4)
Feb 15 01:03:08 srv0A ntpd[1329]: bad peer 2.europe.pool.ntp.org (37.187.7.160)
Feb 15 01:03:08 srv0A ntpd[1329]: bad peer 3.europe.pool.ntp.org (178.63.135.195)
Feb 15 01:03:31 srv0A fsck: /dev/mirror/gm0p6: 1893255 files, 39568205 used, 3993778 free (272474 frags, 465163 blocks, 0.6% fragmentation)
Feb 15 01:06:29 srv0A kernel: .
Feb 15 01:06:29 srv0A ntpd[1330]: dispatch_imsg in main: pipe closed
Feb 15 01:06:29 srv0A syslogd: exiting on signal 15I have executed
fsck in all slices of both discs and has not found errors.I restarted the server several times to see if it started, but I had to re-enter into rescue mode and "messages" file has not changed, I know if he starts or not, but posts it seems that server a time that if he did so.
I can only prove an occurrence, change fstab by removing the mirror and putting the slices of ada0 or ada1, but I have not tested if it could damage the contents of the discs.
You can see that I have tried almost everything, it is the proof that I have read and looked for solutions before asking, but I'm desperate because I cannot find the solution. Can someone guide me as start server? Thanks a lot.
