Networking dead after unplanned reboot, ping sentTo: permission denied while pf off

J

JosefMeixner

I hope this is the right group for my problem.
Problem is, that the networking seems to have died after an unplanned restart. My server yesterday stopped responding, the console printed continuously messaged about something swap related (sorry, didn't take a screen shot, I regret it now). I couldn't log in, so forced a reset. System booted up, did some fscking (fixed some inodes, no prompt to repair anything) and booted up normally. I could log in as root and assumed it had returned to normal.

But the server was not reachable, I tried to ping google from the server, got "no such host", then I tried to contact bind, but got a permission denied from dig and telnet. Restarted bind, same result. Tried to ping local address, got
ping: sentTo: permission denied, tried to ping 127.0.0.1 and also got permission denied. Deactivated pf with service pf stop and ping still gave me a permission denied, so it is not the firewall.

So something is seriously wrong, so restarted the server, came back up without problems, networking still not working, effects as before. I tried ifconfig, but everything looks normal (I cannot attach, as I have no networking access, but as a USB-keyboard is working, I assume I could dump the output to a USB-stick if necessary). As I am a very long time user of the BSDs (started with 386bsd 0.1) I checked /dev for the device. As the networking devices were not there I fired up a FreeBSD-VM with 12.2 (I know, old, but I haven't needed it for quite some time) and checked and saw, it is normal, network devices are not appearing in /dev. Now I am running out of ideas, what could be wrong, where do I go to find information? I am a developer, so I can use debuggers if that is needed.

Kernel of the box is GENERIC 14.0-RELEASE-p5, ethernet is em0, em1, igb0 and igb1, all appearing in ifconfig output and devinfo.
freebsd-update IDS does not work, as it finds no network connection.
 
Update:
I started looking, where the ENOPERM might come from and found in the ping sources that it must be ping.c line 1072 which outputs the warning. So followed it through libc and as far as I see it must be coming from the kernel. As searching sources in cgit is a pain, I fetched the sources to my FreeBSD-VM so I can search and look into them. As I will probably have to add debugging statements to the kernel I have copied the sources to the box affected (USB falsh drive is indeed working) and built a GENERIC kernel. I just booted the system with it and no change, still networking is not working.

But I didn't expect that, as it would have surprised me, if an executable on /boot had gone bad, as nothing writes there so how should a simple fsck damage something there and a damaged binary would have probably led to a panic anyway.

So now I will try to add debugging output to the send-syscall. I will try to check the reference for working at the kernel sources, as I sure do not want to create a kernel which is unresponsive due to excessive output. I hope to locate the problem but will probably need help to understand what I am seeing.
 
Could you build another VM as close to the original box and checksum all the system binaries to see if something is different?

What is the VM software/hypervisor that is running on the host?

This seems like some sort of file system corruption after a restart? Or could just be that you had untested configuration changes and this restart has made them active?
 
I doubt, that this would help much. There were no recent configuration changes, the last were switching from ipfw to pf and after that I tried to enable sshguard which I had to undo (some weird perl error appeared when executing it, haven't looked into it). So no change I am aware of which could affect the kernel. Additionally the box is a server, so a dual CPU board, with server specific components and a hardware RAID controller. As I have no clue why only networking is affected I also have no idea, what could have happened and what would be needed to replicate it. And it has quite a history, it was set up with FreeBSD 9.1 and I made quite some upgrades..

I don't know if you saw my second post (new account, so it needs to be switched active by a mod), I compiled a new kernel and the problem still persists. So I assume something in some configuration is wrong, but I have not the slightest clue what it could be. But from the looks of it, only send is affected, it looks like the code to create a socket is working. I have forgotten to check, if listen works, but I think that daemons are listening for connections when I checked with netstat.
 
Next update:
I added printf-statements to uipc_syscall.c in /sys/kern and the ENOPERM is coming out of sousrsend in uipc_socket.c. As next step I have to find the function called in the statement
C: 
so->so_proto->pr_sosend(...)
for IPV4 localhost, so the loopback device for IPV4.
 
SirDice said:
Did you perhaps leave IPFW enabled? Causing it to get started when the system booted?
Click to expand...
I also had the idea and thought "that must be it" but no, it is off in rc.config. I plan to go over rc.config for the next reboot where I test the next printfs and switch off pf so it doesn't even start and take another look, if I did something stupid. I also checked sysctl.conf and found an old setting no longer supported, but I did that before the last reboot, it didn't fix the problem.

I am absolutely sure it is something really, really stupid I did. If only I knew what.
 
SirDice said:
Check /etc/defaults/rc.conf too, perhaps you edited that one?
Click to expand...
I am quite sure I didn't do that, but I will check. It is much easier than hunting through the kernel. And as I do not know what the fsck did, perhaps it impacted that file (though I doubt it, I use UFS on root and sure as hell didn't switch asynchronous metadata on so a file not being written should never be affected by a fsck as far as I understand).
 
You switched from IPFW to PF, may be you can try to switch back to IPFW again just to see how it goes and open it all with firewall_type="open".
I don't know if it can do anything but it's worth the try.
 
Box is back running, but it gets strange now. ipfw was indeed running, service ipfw status said so, but I could not switch it off with service ipfw stop, service told me, I had to set firewall_enable to "YES". So for some reason ipfw is started (with default settings, so even wrong internal IP-Address) at boot time. In /etc/rc.conf it was commented out and in /etc/default/rc.conf it is set to "NO". For the last boot I even set it explicitly to "NO" in /etc/rc.conf but it was still started.

I have no idea how that can happen, but I now at least know, that I have to switch it off after booting until I can find out what is starting it and why.
 
Maybe it got enabled in /etc/rc.local? Also check the /etc/rc.conf.d/ directory.
 
SirDice said:
Maybe it got enabled in /etc/rc.local? Also check the /etc/rc.conf.d/ directory.
Click to expand...
I don't have an /etc/rc.local at all and /etc/rc.conf.d/ is empty. dmesg shows this around the part where ipfw appears:
Code: 
ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe3fb000-0xfe3fbfff irq 18 at device 20.5 on pci0
usbus6 on ohci4
usbus6: 12Mbps Full Speed USB v1.0
acpi_button0: <Power Button> on acpi0
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xce7ff,0xce800-0xcefff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 15.
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounter "TSC" frequency 2000024221 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled
ugen5.1: <ATI EHCI root HUB> at usbus5
ugen0.1: <ATI OHCI root HUB> at usbus0
ugen4.1: <ATI OHCI root HUB> at usbus4
ugen6.1: <ATI OHCI root HUB> at usbus6
ugen3.1: <ATI OHCI root HUB> at usbus3
ugen1.1: <ATI OHCI root HUB> at usbus1
ugen2.1: <ATI EHCI root HUB> at usbus2
uhub0 on usbus5
uhub0: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub1 on usbus2
uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub2 on usbus4
uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub3 on usbus6
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub4 on usbus1
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub5 on usbus0
uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub6 on usbus3
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
And a recursive grep for ipfw in /etc results in this:
Code: 
./defaults/periodic.conf:# 500.ipfwdenied
./defaults/periodic.conf:security_status_ipfwdenied_enable="YES"
./defaults/periodic.conf:security_status_ipfwdenied_period="daily"
./defaults/periodic.conf:# 550.ipfwlimit
./defaults/periodic.conf:security_status_ipfwlimit_enable="YES"
./defaults/periodic.conf:security_status_ipfwlimit_period="daily"
./defaults/rc.conf:firewall_flags=""            # Flags passed to ipfw when type is a file
./defaults/rc.conf:ipfw_netflow_enable="NO"     # Enable netflow logging via ng_netflow
./mtree/BSD.usr.dist:            ipfw
./mtree/BSD.tests.dist:        ipfw
./mtree/BSD.tests.dist:            ipfw
./periodic/security/500.ipfwdenied:security_daily_compat_var security_status_ipfwdenied_enable
./periodic/security/500.ipfwdenied:if check_yesno_period security_status_ipfwdenied_enable
./periodic/security/500.ipfwdenied:     if ipfw -a list 2>/dev/null | egrep "deny|reset|unreach" > ${TMP}; then
./periodic/security/500.ipfwdenied:       check_diff new_only ipfw ${TMP} "${host} ipfw denied packets:"
./periodic/security/550.ipfwlimit:# Show ipfw rules which have reached the log limit
./periodic/security/550.ipfwlimit:security_daily_compat_var security_status_ipfwlimit_enable
./periodic/security/550.ipfwlimit:if check_yesno_period security_status_ipfwlimit_enable
./periodic/security/550.ipfwlimit:      ipfw -a list | grep " log " | \
./periodic/security/550.ipfwlimit:              echo 'ipfw log limit reached:'
./rc.d/NETWORKING:# REQUIRE: netif netwait netoptions routing ppp ipfw stf
./rc.d/ipfw:# PROVIDE: ipfw
./rc.d/ipfw:name="ipfw"
./rc.d/ipfw:start_cmd="ipfw_start"
./rc.d/ipfw:start_precmd="ipfw_prestart"
./rc.d/ipfw:start_postcmd="ipfw_poststart"
./rc.d/ipfw:stop_cmd="ipfw_stop"
./rc.d/ipfw:status_cmd="ipfw_status"
./rc.d/ipfw:required_modules="ipfw"
./rc.d/ipfw:ipfw_prestart()
./rc.d/ipfw:            required_modules="$required_modules ipfw_nat"
./rc.d/ipfw:            required_modules="$required_modules ipfw_nat64"
./rc.d/ipfw:            required_modules="$required_modules ipfw_nptv6"
./rc.d/ipfw:            required_modules="$required_modules ipfw_pmod"
./rc.d/ipfw:ipfw_start()
./rc.d/ipfw:    elif [ "`ipfw list 65535`" = "65535 deny ip from any to any" ]; then
./rc.d/ipfw:            if ! ifconfig ipfw0 >/dev/null 2>&1; then
./rc.d/ipfw:                    ifconfig ipfw0 create
./rc.d/ipfw:                    echo 'Firewall logging pseudo-interface (ipfw0)' \
./rc.d/ipfw:                    echo 'Firewall logging pseudo-interface (ipfw0)' \
./rc.d/ipfw:ipfw_poststart()
./rc.d/ipfw:ipfw_stop()
./rc.d/ipfw:ipfw_status()
./rc.d/ipfw:            echo "ipfw is not enabled"
./rc.d/ipfw:            echo "ipfw is enabled"
./rc.d/dnctl:# BEFORE: pf ipfw
./rc.d/ipfw_netflow:# PROVIDE: ipfw_netflow
./rc.d/ipfw_netflow:# REQUIRE: ipfw
./rc.d/ipfw_netflow:name="ipfw_netflow"
./rc.d/ipfw_netflow:desc="firewall, ipfw, netflow"
./rc.d/ipfw_netflow:required_modules="ipfw ng_netflow ng_ipfw"
./rc.d/ipfw_netflow:: ${ipfw_netflow_hook:=9995}
./rc.d/ipfw_netflow:: ${ipfw_netflow_rule:=01000}
./rc.d/ipfw_netflow:: ${ipfw_netflow_ip:=127.0.0.1}
./rc.d/ipfw_netflow:: ${ipfw_netflow_port:=9995}
./rc.d/ipfw_netflow:: ${ipfw_netflow_version:=}
./rc.d/ipfw_netflow:ipfw_netflow_test()
./rc.d/ipfw_netflow:    if [ "${ipfw_netflow_version}" != "" ] && [ "${ipfw_netflow_version}" != 9 ]; then
./rc.d/ipfw_netflow:    err 1 "Unknown netflow version \'${ipfw_netflow_version}\'"
./rc.d/ipfw_netflow:    case "${ipfw_netflow_hook}" in
./rc.d/ipfw_netflow:        err 1 "Bad value \"${ipfw_netflow_hook}\": Hook must be numerical"
./rc.d/ipfw_netflow:    case "${ipfw_netflow_rule}" in
./rc.d/ipfw_netflow:        err 1 "Bad value \"${ipfw_netflow_rule}\": Rule number must be numerical"
./rc.d/ipfw_netflow:ipfw_netflow_is_running()
./rc.d/ipfw_netflow:ipfw_netflow_status()
./rc.d/ipfw_netflow:    ipfw_netflow_is_running && echo "ipfw_netflow is active" || echo "ipfw_netflow is not active"
./rc.d/ipfw_netflow:ipfw_netflow_start()
./rc.d/ipfw_netflow:    ipfw_netflow_is_running && err 1 "ipfw_netflow is already active"
./rc.d/ipfw_netflow:    ipfw add ${ipfw_netflow_rule} ngtee ${ipfw_netflow_hook} ip from any to any ${ipfw_netflow_fib:+fib ${ipfw_netflow_fib}}
./rc.d/ipfw_netflow:    mkpeer ipfw: netflow ${ipfw_netflow_hook} iface0
./rc.d/ipfw_netflow:    name ipfw:${ipfw_netflow_hook} netflow
./rc.d/ipfw_netflow:    mkpeer netflow: ksocket export${ipfw_netflow_version} inet/dgram/udp
./rc.d/ipfw_netflow:    name netflow:export${ipfw_netflow_version} netflow_export
./rc.d/ipfw_netflow:    msg netflow:export${ipfw_netflow_version} connect inet/${ipfw_netflow_ip}:${ipfw_netflow_port}
./rc.d/ipfw_netflow:ipfw_netflow_stop()
./rc.d/ipfw_netflow:    ipfw_netflow_is_running || err 1 "ipfw_netflow is not active"
./rc.d/ipfw_netflow:    ipfw delete ${ipfw_netflow_rule}
./rc.d/netwait:# REQUIRE: devd ipfw pf routing
./rc.d/securelevel:# REQUIRE: adjkerntz ipfw pf sysctl_lastload
./network.subr: ipfw[0-9]*|\
./rc.firewall:# Setup system for ipfw(4) firewall service.
./rc.firewall:        load_rc_config ipfw
./rc.firewall:  fwcmd="/sbin/ipfw -q"
./rc.firewall:  fwcmd="/sbin/ipfw"
./rc.firewall:# If you just configured ipfw in the kernel as a tool to solve network
./netstart:/etc/rc.d/ipfw ${_start}
./rc.conf:# Firewall ipfw
./rc.conf:ipfw_enable="NO"
./firewall:fwcmd="/sbin/ipfw"
And the same looking for "firewall" ( dropped the rc.firewall:
Code: 
./defaults/rc.conf:### Basic network and firewall/security options: ###
./defaults/rc.conf:firewall_enable="NO"         # Set to YES to enable firewall functionality
./defaults/rc.conf:firewall_script="/etc/rc.firewall" # Which script to run to set up the firewall
./defaults/rc.conf:firewall_type="UNKNOWN"              # Firewall type (see /etc/rc.firewall)
./defaults/rc.conf:firewall_quiet="NO"          # Set to YES to suppress rule display
./defaults/rc.conf:firewall_logging="NO"                # Set to YES to enable events logging
./defaults/rc.conf:firewall_logif="NO"          # Set to YES to create logging-pseudo interface
./defaults/rc.conf:firewall_flags=""            # Flags passed to ipfw when type is a file
./defaults/rc.conf:firewall_coscripts=""                # List of executables/scripts to run after
./defaults/rc.conf:                             # firewall starts/stops
./defaults/rc.conf:firewall_client_net="192.0.2.0/24" # IPv4 Network address for "client"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:#firewall_client_net_ipv6="2001:db8:2:1::/64" # IPv6 network prefix for
./defaults/rc.conf:                             # "client" firewall.
./defaults/rc.conf:firewall_simple_iif="em1"    # Inside network interface for "simple"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:firewall_simple_inet="192.0.2.16/28" # Inside network address for "simple"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:firewall_simple_oif="em0"    # Outside network interface for "simple"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:firewall_simple_onet="192.0.2.0/28" # Outside network address for "simple"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:#firewall_simple_iif_ipv6="em1"      # Inside IPv6 network interface for "simple"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:#firewall_simple_inet_ipv6="2001:db8:2:800::/56" # Inside IPv6 network prefix
./defaults/rc.conf:                             # for "simple" firewall.
./defaults/rc.conf:#firewall_simple_oif_ipv6="em0"      # Outside IPv6 network interface for "simple"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:#firewall_simple_onet_ipv6="2001:db8:2:0::/56" # Outside IPv6 network prefix
./defaults/rc.conf:                             # for "simple" firewall.
./defaults/rc.conf:firewall_myservices=""               # List of ports/protocols on which this host
./defaults/rc.conf:                             # offers services for "workstation" firewall.
./defaults/rc.conf:firewall_allowservices=""    # List of IPs which have access to
./defaults/rc.conf:                             # $firewall_myservices for "workstation"
./defaults/rc.conf:                             # firewall.
./defaults/rc.conf:firewall_trusted=""          # List of IPs which have full access to this
./defaults/rc.conf:                             # host for "workstation" firewall.
./defaults/rc.conf:firewall_logdeny="NO"                # Set to YES to log default denied incoming
./defaults/rc.conf:                             # packets for "workstation" firewall.
./defaults/rc.conf:firewall_nologports="135-139,445 1026,1027 1433,1434" # List of TCP/UDP ports
./defaults/rc.conf:                             # logged for "workstation" firewall.
./defaults/rc.conf:firewall_nat_enable="NO"     # Enable kernel NAT (if firewall_enable == YES)
./defaults/rc.conf:firewall_nat_interface=""    # Public interface or IPaddress to use
./defaults/rc.conf:firewall_nat_flags=""                # Additional configuration parameters
./defaults/rc.conf:firewall_nat64_enable="NO"   # Enable kernel NAT64 module.
./defaults/rc.conf:firewall_nptv6_enable="NO"   # Enable kernel NPTv6 module.
./defaults/rc.conf:firewall_pmod_enable="NO"    # Enable kernel protocols modification module.
./defaults/rc.conf:natd_enable="NO"             # Enable natd (if firewall_enable == YES).
./rc.d/ipfilter:                echo "Saving firewall state tables"
./rc.d/ipfw:rcvar="firewall_enable"
./rc.d/ipfw:set_rcvar_obsolete ipv6_firewall_enable
./rc.d/ipfw:    if checkyesno firewall_nat_enable; then
./rc.d/ipfw:    if checkyesno firewall_nat64_enable; then
./rc.d/ipfw:    if checkyesno firewall_nptv6_enable; then
./rc.d/ipfw:    if checkyesno firewall_pmod_enable; then
./rc.d/ipfw:    local   _firewall_type _module _sysctl_reload
./rc.d/ipfw:            _firewall_type=$1
./rc.d/ipfw:            _firewall_type=${firewall_type}
./rc.d/ipfw:    # set the firewall rules script if none was specified
./rc.d/ipfw:    [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall
./rc.d/ipfw:    if [ -r "${firewall_script}" ]; then
./rc.d/ipfw:            /bin/sh "${firewall_script}" "${_firewall_type}"
./rc.d/ipfw:            echo 'Warning: kernel has firewall functionality, but' \
./rc.d/ipfw:                'firewall rules are not enabled.'
./rc.d/ipfw:    if checkyesno firewall_logging; then
./rc.d/ipfw:    if checkyesno firewall_logif; then
./rc.d/ipfw:    # Start firewall coscripts
./rc.d/ipfw:    for _coscript in ${firewall_coscripts} ; do
./rc.d/ipfw:    # Enable the firewall
./rc.d/ipfw:            warn "failed to enable IPv4 firewall"
./rc.d/ipfw:                    warn "failed to enable IPv6 firewall"
./rc.d/ipfw:    # Disable the firewall
./rc.d/ipfw:    # Stop firewall coscripts
./rc.d/ipfw:    for _coscript in `reverse_list ${firewall_coscripts}` ; do
./rc.d/ipfw:firewall_coscripts="/etc/rc.d/natd ${firewall_coscripts}"
./rc.d/ipfw_netflow:desc="firewall, ipfw, netflow"
./sysctl.conf:# Reinject packets passing through nat into firewall
./pf.os:# the case that X is a NAT firewall. While nmap is talking to the
./pf.os:# device itself, p0f is fingerprinting the guy behind the firewall
./pf.os:# caused by a commonly used software (personal firewalls, security
./pf.os:# KEEP IN MIND: Some packet firewalls configured to normalize outgoing
./pf.os:# system (and probably not quite to the firewall either).
./rc.conf:#firewall_enable="YES"
./rc.conf:firewall_enable="NO"
./rc.conf:#firewall_type="OPEN"
./rc.conf:#firewall_script="/etc/firewall"
./rc.conf:##firewall_logif="YES"
./firewall:# Define the firewall command (as in /etc/rc.firewall) for easy
As you can see, I left the configuration of ipfw in the rc.conf but all but firewall_enable="NO" has been commented out. I do not see an indication, that pf starts in dmesg so cannot see, if it is started at about the same time.
 
I still can see no reason, why ipfw is started. So instead I will try to get other things done to help me find it. For one I had been running an old version (14.0 p5, current would be 14.0 p8) and so I should update in any case. Perhaps that fixes the problem, though I think this is unlikely.

But to do that I first have to undo the damage I did by compiling a kernel. I forgot, that the 14.0 series and its patches are not in "stable14" but "releng14_0", so by accident built a 14.1 kernel. freebsd-update is not happy about this...

But at least ssh is working again, so I no longer have to work directly on the console.
 
You must log in or register to reply here.
Back
Top