Solved [Solved] tftp-hpa error: Open Timeout - help please!

T

trope

Hello everyone,

I'm currently running a PXE server to netboot various operating systems

Just a little background, I'm using isc-dhcp-server and tftp-hpa to serve gpxelinux.0 out to computers who boot via PXE.

I had no problems a few days ago and was able to boot into gpxelinux.0 fine, but yesterday in an attempt to install Windows through the same PXE server, I configured a Samba server to also run on this machine.

I can't recall I made any other changes to the box other than installing/configuring Samba, and yet today when I went to boot to my PXE menu I got the following error:
Code: 
PXE-E32: TFTP open timeout
I did some digging, and turned on verbose logging when tftp-hpa starts and found the following within /var/log/messages file:
Code: 
root@jumpstart:/var/log # less messages
Jun 12 20:00:00 jumpstart newsyslog[1173]: logfile turned over due to size>100K
Jun 12 20:00:22 jumpstart in.tftpd[1180]: RRQ from 10.0.10.2 filename gpxelinux.0
Jun 12 20:00:24 jumpstart in.tftpd[1181]: RRQ from 10.0.10.2 filename gpxelinux.0
Jun 12 20:00:25 jumpstart in.tftpd[1180]: tftpd: oack: Host is down
Jun 12 20:00:27 jumpstart in.tftpd[1181]: tftpd: oack: Host is down
Jun 12 20:00:28 jumpstart in.tftpd[1182]: RRQ from 10.0.10.2 filename gpxelinux.0
Jun 12 20:00:28 jumpstart in.tftpd[1182]: tftpd: oack: Host is down
Jun 12 20:00:34 jumpstart in.tftpd[1183]: RRQ from 10.0.10.2 filename gpxelinux.0
Jun 12 20:00:34 jumpstart in.tftpd[1183]: tftpd: oack: Host is down
Jun 12 20:00:42 jumpstart in.tftpd[1184]: RRQ from 10.0.10.2 filename gpxelinux.0
Jun 12 20:00:42 jumpstart in.tftpd[1184]: tftpd: oack: Host is down

It would seem that the PC attempting to PXE boot requests the file gpxelinux.0 but then thinks my TFTP server is down.. which it obviously is not.

FYI I am starting tftp-hpa from within inetd() and the line within inetd.conf looks like this:
Code: 
tftp    dgram   udp     wait    root    /usr/local/libexec/in.tftpd     in.tftpd -p -v -s /usr/tftpboot

Also if it's any help, I ran a quick netstat() :
Code: 
root@jumpstart:/var/log # netstat -a -p udp
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
udp4       0      0 *.tftp                 *.*                    
udp4       0      0 *.bootps               *.*                    
udp6       0      0 *.12358                *.*                    
udp4       0      0 *.61064                *.*                    
udp4       0      0 *.syslog               *.*                    
udp6       0      0 *.syslog               *.*

Any help would be great, this project needs to be completed soon and right now I am at a roadblock. Thank you in advance!
 
Re: tftp-hpa error: Open Timeout - help please!

Am I understanding it that after installing Samba so you can PXE boot Windows clients it broke PXE boot for everything? It's rather strange but the logs seem to say something. The TFTP service is obviously getting the read request for a file but can't return the request. The client appears to try multiple times indicated by the multiple request and new PIDs spawned by inetd for each with an immediate close afterwards. Is the client in the same directly connected subnet? Or is the client in a different subnet? If the client is in a different subnet, can you connect to something inside of that subnet? Example nc -nvvz 10.0.10.2. Maybe the work didn't change TFTP configuration but mucked with some routing that is causing the host is down error when sending the file.

Additionally, depending on the size of files you are sending around, particularly with getting Windows to install, I've seen it recommended to use a larger TFTP block size. The example at this site: http://www.wonkity.com/~wblock/docs/html/pxe.html shows using -B 1024 and that's what I based my config off us. Once you get this figured out keep that in mind.
 
Re: tftp-hpa error: Open Timeout - help please!

junovitch said:
Am I understanding it that after installing Samba so you can PXE boot Windows clients it broke PXE boot for everything?
Click to expand...
Yes that is why I installed Samba. I'm not sure if me installing Samba is what broke everything, but considering that's the last thing I did yesterday before it stopped working I assume it has something to do with it.

junovitch said:
Is the client in the same directly connected subnet? Or is the client in a different subnet? If the client is in a different subnet, can you connect to something inside of that subnet?
Click to expand...
My PXE server is on 10.0.0.250 and my DHCP server is assigning IP's in the range between 10.0.10.0 and 10.0.99.254 as seen in my dhcpd.conf below:
Code: 
authoritative;
option domain-name "webair.com";
option domain-name-servers 216.130.161.1, 216.130.161.6, 8.8.8.8;
ddns-update-style none;


subnet 10.0.0.0 netmask 255.0.0.0 {
  next-server 10.0.0.250;
  ddns-update-style none;
  filename "gpxelinux.0";

  pool {
    range 10.0.10.0 10.0.99.254;
    option routers 10.0.0.250;
    option subnet-mask 255.0.0.0;
    allow unknown-clients;
    max-lease-time 28800;
  }
}

group {
}

junovitch said:
Example nc -nvvz 10.0.10.2. Maybe the work didn't change TFTP configuration but mucked with some routing that is causing the host is down error when sending the file.
Click to expand...
That may be what happened. Sorry if this is a noobie question but can you tell me what nc -nvvz 10.0.10.2 does? I tried running the command and got a usage prompt.

I've still been investigating and unable to find a solution to this :/ any other ideas?
 
Re: tftp-hpa error: Open Timeout - help please!

trope said:
Yes that is why I installed Samba. I'm not sure if me installing Samba is what broke everything, but considering that's the last thing I did yesterday before it stopped working I assume it has something to do with it.
Click to expand...

Ok. I mainly just trying to put the chain of events into context. Samba shouldn't make a difference but maybe it was something done around the same time.

trope said:
That may be what happened. Sorry if this is a noobie question but can you tell me what nc -nvvz 10.0.10.2 does? I tried running the command and got a usage prompt.

I've still been investigating and unable to find a solution to this :/ any other ideas?
Click to expand...

Very sorry on that. It's nc -nvvz 10.0.10.2 22 assuming port 22 is running on that client. Mainly I was looking to see that you were getting a connecting. Netcat would have shown a TCP socket could connect. A ping would be alright as well. Since your dhcpd.conf shows that you are just using a big /8 10.0.0.0 network there shouldn't be any routing issues that come into play. If you can validate good connectivity that that should rule that out. I'm assuming you aren't using a firewall that could be blocking returning packets leaving from the server to the client, correct? How about running tcpdump -i em0 where em0 is your network interface on the TFTP server. Attempt to PXE boot a client and see if you see incoming traffic to port 69 on that interface followed by random high to random high outgoing traffic back to the client. If you only see incoming traffic that could indicate the problem is on the box itself. If you see the outgoing traffic but the client didn't seem to think it arrived maybe there is network issues elsewhere.
 
Re: tftp-hpa error: Open Timeout - help please!

Okay well I ran a tcpdump -i em1 -v and got the following output: (sorry, it's long!)

Code: 
root@jumpstart:~ # tcpdump -i -v em1 
tcpdump: -v: No such device exists
(BIOCSETIF failed: Device not configured)
root@jumpstart:~ # tcpdump -i em1 -v 
tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
19:50:15.927848 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:17.403519 IP (tos 0x0, ttl 20, id 2, offset 0, flags [none], proto UDP (17), length 576)
    0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 18:03:73:b9:f8:be (oui Unknown), length 548, xid 0x73b9f8be, secs 4, Flags [Broadcast]
          Client-Ethernet-Address 18:03:73:b9:f8:be (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Requested-IP Option 50, length 4: 10.0.10.2
            Parameter-Request Option 55, length 36: 
              Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
              IEN-Name-Server, Domain-Name-Server, RL, Hostname
              BS, Domain-Name, SS, RP
              EP, RSZ, TTL, BR
              YD, YS, NTP, Vendor-Option
              Requested-IP, Lease-Time, Server-ID, RN
              RB, Vendor-Class, TFTP, BF
              Option 128, Option 129, Option 130, Option 131
              Option 132, Option 133, Option 134, Option 135
            MSZ Option 57, length 2: 1260
            Server-ID Option 54, length 4: 10.0.0.250
            GUID Option 97, length 17: 0.68.69.76.76.50.0.16.84.128.50.198.192.79.49.82.49
            Client-ID Option 61, length 17: "DELL2^@^PTM-^@2M-FM-@O1R1"
            ARCH Option 93, length 2: 0
            NDI Option 94, length 3: 1.2.1
            Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"
19:50:17.405090 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.0.0.250.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 300, xid 0x73b9f8be, secs 4, Flags [Broadcast]
          Your-IP 10.0.10.2
          Server-IP 10.0.0.250
          Client-Ethernet-Address 18:03:73:b9:f8:be (oui Unknown)
          file "gpxelinux.0"
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.0.0.250
            Lease-Time Option 51, length 4: 28800
            Subnet-Mask Option 1, length 4: 255.0.0.0
            Default-Gateway Option 3, length 4: 10.0.0.250
            Domain-Name-Server Option 6, length 12: ns.webair.net,ns2.webair.net,google-public-dns-a.google.com
            Domain-Name Option 15, length 10: "webair.com"
19:50:17.405768 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.250 tell 10.0.10.2, length 46
19:50:17.405773 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.0.0.250 is-at 00:30:48:bd:45:fb (oui Unknown), length 28
19:50:17.405892 IP (tos 0x0, ttl 20, id 3, offset 0, flags [none], proto UDP (17), length 56)
    10.0.10.2.2070 > 10.0.0.250.tftp:  28 RRQ "gpxelinux.0" octet tsize 0
19:50:17.941050 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:19.435837 IP (tos 0x0, ttl 20, id 4, offset 0, flags [none], proto UDP (17), length 56)
    10.0.10.2.2071 > 10.0.0.250.tftp:  28 RRQ "gpxelinux.0" octet tsize 0 
19:50:19.954252 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:21.765834 
19:50:21.967580 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:23.445376 IP (tos 0x0, ttl 20, id 5, offset 0, flags [none], proto UDP (17), length 56)
    10.0.10.2.2072 > 10.0.0.250.tftp:  28 RRQ "gpxelinux.0" octet tsize 0 
19:50:23.980907 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:25.994233 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:28.007437 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:29.432268 IP (tos 0x0, ttl 20, id 6, offset 0, flags [none], proto UDP (17), length 56)
    10.0.10.2.2073 > 10.0.0.250.tftp:  28 RRQ "gpxelinux.0" octet tsize 0 
19:50:30.020763 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:31.773135 
19:50:32.033965 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:34.047667 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:36.060744 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:37.396634 IP (tos 0x0, ttl 20, id 7, offset 0, flags [none], proto UDP (17), length 56)
    10.0.10.2.2074 > 10.0.0.250.tftp:  28 RRQ "gpxelinux.0" octet tsize 0 
19:50:38.073823 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:40.087149 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:41.784432 
19:50:42.100352 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:44.113803 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:46.127132 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
19:50:47.585941 CDPv2, ttl: 180s, checksum: 692 (unverified), length 383
        Device-ID (0x01), length: 24 bytes: 'gsc120-l2.gsc.webair.net'
        Version String (0x05), length: 186 bytes: 
          Cisco IOS Software, C2970 Software (C2970-LANBASEK9-M), Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)
          Copyright (c) 1986-2009 by Cisco Systems, Inc.
          Compiled Mon 09-Mar-09 18:27 by gereddy
        Platform (0x06), length: 22 bytes: 'cisco WS-C2970G-24TS-E'
        Address (0x02), length: 13 bytes: IPv4 (1) gsc120-l2.gsc.webair.net
        Port-ID (0x03), length: 18 bytes: 'GigabitEthernet0/3'
        Capability (0x04), length: 4 bytes: (0x00000028): L2 Switch, IGMP snooping
        Protocol-Hello option (0x08), length: 32 bytes: 
        VTP Management Domain (0x09), length: 10 bytes: 'webair-gsc'
        Native VLAN ID (0x0a), length: 2 bytes: 3002
        Duplex (0x0b), length: 1 byte: full
        AVVID trust bitmap (0x12), length: 1 byte: 0x00
        AVVID untrusted ports CoS (0x13), length: 1 byte: 0x00
        Management Addresses (0x16), length: 13 bytes: IPv4 (1) gsc120-l2.gsc.webair.net
19:50:48.140209 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8bba.00:13:60:c0:04:00.8003, length 43
        message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
        root-id 5bba.00:08:a4:b0:65:00, root-pathcost 4, port-role Designated
^C
30 packets captured
30 packets received by filter
0 packets dropped by kernel

I don't really know what to make of this, perhaps you do?
 
Re: tftp-hpa error: Open Timeout - help please!

Well, most of it's just spanning tree noise. Didn't really need the -v option but I see what I was looking for. Each input packet for a file request is reflected in the packet capture by one of these entries.
Code: 
19:50:17.405892 IP (tos 0x0, ttl 20, id 3, offset 0, flags [none], proto UDP (17), length 56)
    10.0.10.2.2070 > 10.0.0.250.tftp:  28 RRQ "gpxelinux.0" octet tsize 0

In your TFTP log, each incoming RRQ is accompanied by an immediate failure. It's not the case that the TFTP service is sending a reply and an ICMP reachable is coming back to say it failed. According to the packet capture, the service is not attempting to send a single packet back to the client.
Code: 
Jun 12 20:00:28 jumpstart in.tftpd[1182]: RRQ from 10.0.10.2 filename gpxelinux.0
Jun 12 20:00:28 jumpstart in.tftpd[1182]: tftpd: oack: Host is down

My usual suspects would be to double check all my networking. Is there a firewall enabled? What is the output of ifconfig em1? What is the output of netstat -nr | grep "^10"?
 
Re: tftp-hpa error: Open Timeout - help please!

I was running pf but I explicitly disabled it a while ago because that was my first thought.. but still no luck.

If it's any help, here is the output of both ifconfig em1 and netstat -nr | grep "^10"
Code: 
root@jumpstart:~ # netstat -nr | grep "^10"
10.0.0.0/16        link#1             U           0       63   bge0 =>
10.0.0.0/8         link#3             U           0        0    em1
10.0.0.250         link#3             UHS         0        0    lo0
10.0.86.251        link#1             UHS         0        0    lo0
root@jumpstart:~ # ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:30:48:bd:45:fb
        inet 10.0.0.250 netmask 0xff000000 broadcast 10.255.255.255 
        inet6 fe80::230:48ff:febd:45fb%em1 prefixlen 64 scopeid 0x3 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
root@jumpstart:~ #
 
Re: tftp-hpa error: Open Timeout - help please!

What is the output of ifconfig bge0? Your routing table is showing you have 10.x.x.x subnets conflicting between both interfaces.
 
Re: tftp-hpa error: Open Timeout - help please!

bge0 is a third interface which is connected to a separate network (our backups network) and got its IP via DHCP.
Code: 
root@jumpstart:~ # ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:13:21:bd:3c:c1
        inet 10.0.86.251 netmask 0xffff0000 broadcast 10.0.255.255 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active

Can this still cause conflict even if they are on two separate physical networks?
 
Re: tftp-hpa error: Open Timeout - help please!

Without doing complicated stuff with multiple routing tables, the standard routing behavior would be to send traffic out the most specific route. A /16 route is more specific than a /8 and would take precedence. If you can, does doing ifconfig bge0 down and running PXE boot afterwards fix the issue? Make sure the /16 route entry is gone before running PXE boot. Once you've checked, you can do ifconfig bge0 up; dhclient bge0 to bring it back up to normal.
 
Re: tftp-hpa error: Open Timeout - help please!

Yep that was exactly the problem!
Disabling bge0 has fixed my issue entirely.

For now I suppose I will just leave that interface disabled until I have more time to investigate how routing tables in FreeBSD work :e

Thanks so much for the help, you truly saved me weeks of hair pulling frustration haha.

Have a good one man! Cheers.
 
Hey no problem. Glad we found it. FYI routing tables in general send packets using the most specific entry. It's not just FreeBSD. As far as multiple routing tables, it could solve the problem but the bit of experience I have with says it's a bit messy and hard to work with. And I'm not sure if dhclient plays nice when it comes up updating the default or another routing table. Having a network where subnets are scoped without any overlap is probably going to be easier solution to work with in the long run.
 
Sounds like something I'll have to bring up to the network admin lol, I'm not the most experienced when it comes to networking (obviously) ;)
 
You must log in or register to reply here.
Back
Top