Local traffic oddity with CARP

L

last1

I have 2 servers in HA mode using CARP, using virtual IP 192.168.1.95. Each server has a real IP of 192.168.2.75/76

When the haproxy that's running on the master server - 192.168.1.95 sends a reply back to a client, it first sends the reply back to the local network gateway ( 192.168.1.1 ) instead of directly to the client!

It's been driving me crazy because I can't figure out how this happens. This is on FreeBSD 12.1-p12

For example:
Code: 
arp -an
? (192.168.1.35) at 00:50:56:85:2c:f1 on em0 expires in 1199 seconds [ethernet]
? (192.168.2.1) at 00:50:56:bc:00:3c on em0 expires in 1123 seconds [ethernet]
? (192.168.1.1) at 00:50:56:bc:00:3c on em0 expires in 1190 seconds [ethernet]
tcpdump -en output:
Code: 
12:21:42.848617 00:0c:29:73:b4:bb > 00:50:56:bc:00:3c, ethertype IPv4 (0x0800), length 66: 192.168.1.95.3306 > 192.168.1.35.42691: Flags [.], ack 1713257743, win 513, options [nop,nop,TS val 1354880564 ecr 1570829278], length 0

12:21:42.848933 00:0c:29:73:b4:bb > 00:50:56:bc:00:3c, ethertype IPv4 (0x0800), length 77: 192.168.1.95.3306 > 192.168.1.35.42691: Flags [P.], seq 0:11, ack 1, win 513, options [nop,nop,TS val 1354880564 ecr 1570829278], length 11

The local firewall ( also FreeBSD ) receives the packets and redirects them. It looks like this:

Code: 
09:40:15.611039 00:0c:29:73:b4:bb > 00:50:56:bc:00:3c, ethertype IPv4 (0x0800), length 551: 192.168.1.95.3306 > 192.168.1.35.42691: Flags [P.], seq 5717:6202, ack 1460, win 513, options [nop,nop,TS val 1345193324 ecr 1561142039], length 485
09:40:15.611051 00:50:56:bc:00:3c > 00:50:56:85:2c:f1, ethertype IPv4 (0x0800), length 551: 192.168.1.95.3306 > 192.168.1.35.42691: Flags [P.], seq 5717:6202, ack 1460, win 513, options [nop,nop,TS val 1345193324 ecr 1561142039], length 485

For client->CARP server the communication happens directly. It's only the return packets that go through the local gateway.

Am I going crazy here ? How can packets destined for one IP/mac address go to my default gateway ??
 
If it were that easy...I do not. That's why I don't get why that traffic goes there.
Code: 
netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.2.1        UGS         em0
10.10.5.0/24       link#2             U           em1
10.10.5.10         link#2             UHS         lo0
127.0.0.1          link#4             UH          lo0
192.168.1.0/24     link#1             U           em0
192.168.1.95       link#1             UHS         lo0
192.168.2.0/24     link#1             U           em0
192.168.2.75       link#1             UHS         lo0


route get 192.168.1.35
   route to: 192.168.1.35
destination: 192.168.1.0
       mask: 255.255.255.0
        fib: 0
  interface: em0
      flags: <UP,DONE,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
 
I have no other fibs either ( just 0 )

route get -fib 1 192.168.1.35
route: invalid fib number: 1
 
weird. i assume you dont have any ipfw fwd or pf equiv tricks on the ha box
what if you add a /32 from 95 to 35 ?
 
192.168.1.35 is just an example. All ips in 192.168.1.0/24 get routed through 192.168.1.1 , it makes no sense.

I don't have ipfw installed, I do have pf installed but no rdr statements.

The majority of the packets appear to go through these rules:

scrub on em0 all fragment reassemble
[ Evaluations: 8315293736 Packets: 8283889573 Bytes: 1963306506718 States: 0 ]
[ Inserted: uid 0 pid 19260 State Creations: 0

pass out log all flags S/SA keep state allow-opts label "1232f88e5fac29a32501e3f051020cac"
[ Evaluations: 74445751 Packets: 1650057259 Bytes: 856410062079 States: 23525 ]
[ Inserted: uid 0 pid 19260 State Creations: 54832453]

pass in quick on em0 inet from (em0:network) to any flags S/SA keep state label "c9fb95f8275a8a73ce4a68190ed7bc51"
[ Evaluations: 19612546 Packets: 1666703150 Bytes: 821524496202 States: 11996 ]
[ Inserted: uid 0 pid 19260 State Creations: 19590990]

Where is the decision made/ who makes the decision to send to a particular mac address for a given IP in FreeBSD ? Can it be something else other than routing table, arp table, firewall ?
 
well i dont use pf so i dont know about
but you can override routing table with ipfw fwd
also ipsec policies override routing table
other then that routing table

is the problem present without the carp stuff too ?
 
I don't make use of NAT but here is my ruleset and currently at work With the rule I made when I caught Spectrum following me by the strange script IP# that NoScript let me catch in the act. Right before I called them, pinned them down on it and right before I stopped seeing it:

/etc/pf.conf:
Code: 
### Macro name for external interface
ext_if = "em0"
netbios_tcp = "{ 22, 23, 25, 80, 110, 111, 123, 512, 513, 514, 515, 6000, 6010 }"
netbios_udp = "{ 123, 512, 513, 514, 515, 5353, 6000, 6010 }"

### Reassemble fragmented packets
scrub in on $ext_if all fragment reassemble

### Default deny everything
block log all

### Pass loopback
set skip on lo0

### Block spooks
antispoof for lo0
antispoof for $ext_if inet
block in from no-route to any
block in from urpf-failed to any
block in quick on $ext_if from any to 255.255.255.255
block in quick log on $ext_if from { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 } to any

### Block all IPv6
block in quick inet6 all
block out quick inet6 all

### Block to and from port 0
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

### Block specific ports
block in quick log on $ext_if proto tcp from any to any port $netbios_tcp
block in quick log on $ext_if proto udp from any to any port $netbios_udp

###Block Charter Spy
#block in quick log on $ext_if from 24.217.1.158 to any
#block out quick log on $ext_if from any to 24.217.1.158

### Keep and modulate state of outbound tcp, udp and icmp traffic
pass out on $ext_if proto { tcp, udp, icmp } from any to any modulate state


Code: 
root@bakemono:/ # pfctl -s all
FILTER RULES:
scrub in on em0 all fragment reassemble
block drop log all
block drop in on ! lo0 inet from 127.0.0.0/8 to any
block drop in on ! em0 inet from 192.168.1.0/24 to any
block drop in inet from 192.168.1.74 to any
block drop in on ! lo0 inet6 from ::1 to any
block drop in from no-route to any
block drop in from urpf-failed to any
block drop in quick on em0 inet from any to 255.255.255.255
block drop in log quick on em0 inet from 10.0.0.0/8 to any
block drop in log quick on em0 inet from 172.16.0.0/12 to any
block drop in log quick on em0 inet from 192.168.0.0/16 to any
block drop in log quick on em0 inet from 255.255.255.255 to any
block drop in quick inet6 all
block drop out quick inet6 all
block drop quick proto tcp from any port = 0 to any
block drop quick proto tcp from any to any port = 0
block drop quick proto udp from any port = 0 to any
block drop quick proto udp from any to any port = 0
block drop in log quick on em0 proto tcp from any to any port = ssh
block drop in log quick on em0 proto tcp from any to any port = telnet
block drop in log quick on em0 proto tcp from any to any port = smtp
block drop in log quick on em0 proto tcp from any to any port = http
block drop in log quick on em0 proto tcp from any to any port = pop3
block drop in log quick on em0 proto tcp from any to any port = sunrpc
block drop in log quick on em0 proto tcp from any to any port = ntp
block drop in log quick on em0 proto tcp from any to any port = exec
block drop in log quick on em0 proto tcp from any to any port = login
block drop in log quick on em0 proto tcp from any to any port = shell
block drop in log quick on em0 proto tcp from any to any port = printer
block drop in log quick on em0 proto tcp from any to any port = x11
block drop in log quick on em0 proto tcp from any to any port = x11-ssh
block drop in log quick on em0 proto udp from any to any port = ntp
block drop in log quick on em0 proto udp from any to any port = biff
block drop in log quick on em0 proto udp from any to any port = who
block drop in log quick on em0 proto udp from any to any port = syslog
block drop in log quick on em0 proto udp from any to any port = printer
block drop in log quick on em0 proto udp from any to any port = mdns
block drop in log quick on em0 proto udp from any to any port = x11
block drop in log quick on em0 proto udp from any to any port = x11-ssh
pass out on em0 proto tcp all flags S/SA modulate state
pass out on em0 proto udp all keep state
pass out on em0 proto icmp all keep state

STATES:
all tcp 192.168.1.74:42959 -> 34.214.1.68:443       ESTABLISHED:ESTABLISHED
all tcp 192.168.1.74:52319 -> 204.109.59.195:443       TIME_WAIT:TIME_WAIT
all tcp 192.168.1.74:48612 -> 204.109.59.195:443       FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.1.74:30955 -> 204.109.59.195:443       TIME_WAIT:TIME_WAIT

INFO:
Status: Enabled for 49 days 03:59:43          Debug: Urgent

State Table                          Total             Rate
  current entries                        4               
  searches                        35744378            8.4/s
  inserts                           136549            0.0/s
  removals                          136545            0.0/s
Counters
  match                             306949            0.1/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start            60000 states
adaptive.end             120000 states
src.track                     0s

LIMITS:
states        hard limit   100000
src-nodes     hard limit    10000
frags         hard limit     5000
table-entries hard limit   200000

OS FINGERPRINTS:
762 fingerprints loaded
root@bakemono:/ #
 
I have solved the issue.

This was a production system so full restart of the OS was not possible.

At some point in the past, I did have a force gateway scenario but I had disabled it.

I also had sticky tables enabled on my haproxy which I always reloaded, but never restarted. Turns out sticky tables persist across reloads and they were the ones still forwarding to the gateway despite that config having changed.

I removed sticky tables for the front end in haproxy, restarted haproxy and the issue went away. Pretty crazy
 
You must log in or register to reply here.
Back
Top