Solved Reverse proxy ignored by FreeBSD box

[hakova]

Hi all,

I have a very weird situation. I have a reverse proxy (nginx) setup in my home lab that directs the https requests for media.example.org (not literally) to the jellyfin server in the LAN. Of note, IP number of the media.example.org resolves to my home IP; in other words, I am not using Cloudflare or such fancy reroutes at the moment. Please don't judge me. My media pc that has the FreeBSD 14.3 as its OS has the hostname htpc.example.org. In other words, it shares the same domain name. When I browse the https://media.example.org with this machine, the right IP address is resolved, but the browser lands on the http page of my router, although I specify it as https in the address. Of note, the ports 80 and 443 are forwarded to the reverse proxy host from my home router. With any other computer in my LAN, most of which are linux boxes, the same address is resolved to my home IP and the correct jellyfin login page is displayed on the browser. What am I missing?

 

[Jose]

...IP number of the media.example.org resolves to my home IP...

Do you mean the public IP your ISP gave you?

...When I browse the https://media.example.org with this machine, the right IP address is resolved, but the browser lands on the http page of my router, although I specify it as https in the address.

Same browser used for both Linux and Freebsd?

Of note, the ports 80 and 443 are forwarded to the reverse proxy host from my home router.

Plastic router running some firmware provided by its manufacturer?

 

[hakova]

Do you mean the public IP your ISP gave you?

Correct.

Same browser used for both Linux and Freebsd?

Correct. Both Firefox and Chromium work as expected in linux, both fail to work as expected and described above in FreeBSD.

Plastic router running some firmware provided by its manufacturer?

I am not sure what is implied here, but the router is not plastic, but is running firmware provided by its manufacturer and working well.

 

[Jose]

I am not sure what is implied here, but the router is not plastic, but is running firmware provided by its manufacturer and working well.

It's just a name I know for the average low-cost integrated router / access point you get from manufacturers like Asus. As opposed to a Unifi / Eero setup, or a custom router built from PC parts.

Anything in the Nginx logs? Compare what gets logged for Linux browsing with Freebsd browsing

 

[atax1a]

is the freebsd machine on the same subnet as the linux machines? this sounds like you have a "hairpin NAT" setup going wrong slightly.

 

[hakova]

is the freebsd machine on the same subnet as the linux machines? this sounds like you have a "hairpin NAT" setup going wrong slightly.

No they are not. FreeBSD in in 192.168.70.x whereas the linux machines and the router are on 192.168.1.x. The jellyfin host is on 192.168.20.x. I suspect you are on the right track but I am not sure how to troubleshoot this.

Anything in the Nginx logs? Compare what gets logged for Linux browsing with Freebsd browsing

Will check and get back...

 

[atax1a]

Ah, yeah, you need to properly hairpin-NAT the 192.168.70.0/24 subnet. as to how to accomplish that, it depends entirely on your router/firewall setup.

 

[ogogon]

Hi all,

I have a very weird situation. I have a reverse proxy (nginx) setup in my home lab that directs the https requests for media.example.org (not literally) to the jellyfin server in the LAN. Of note, IP number of the media.example.org resolves to my home IP; in other words, I am not using Cloudflare or such fancy reroutes at the moment. Please don't judge me. My media pc that has the FreeBSD 14.3 as its OS has the hostname htpc.example.org. In other words, it shares the same domain name. When I browse the https://media.example.org with this machine, the right IP address is resolved, but the browser lands on the http page of my router, although I specify it as https in the address. Of note, the ports 80 and 443 are forwarded to the reverse proxy host from my home router. With any other computer in my LAN, most of which are linux boxes, the same address is resolved to my home IP and the correct jellyfin login page is displayed on the browser. What am I missing?

I'm sorry, I'm not a native English speaker and unfortunately didn't get the full picture from your explanation.

If I understand you correctly, you have an internet connection and want to expose your web server to the outside world. I assume you have a static IP address.
You use a router connected to your ISP with this external address and redirect external ports 80 and 443 to the actual web server located on the internal network, in the intranet address space. In theory, this is a proven and reliable setup. (I have the same...)

Clarification: why do you need Nginx as a reverse proxy? Do you have huge traffic and a lot of static content? Do you experience DDOS attacks?
Also, how is your DNS architecture structured?

And most importantly, what's not working?

Ogogon.

 

[hakova]

Ah, yeah, you need to properly hairpin-NAT the 192.168.70.0/24 subnet. as to how to accomplish that, it depends entirely on your router/firewall setup.

Here is the port forwarding page of my router which is the only place I know hairpin NAT is mentioned. eth1 is the trunk for the whole LAN and includes the VLAN 70. Any pointers?

I can choose eth1.70 instead of eth1 from the dropdown list, but it appears that I cannot select both.

 

[hakova]

I'm sorry, I'm not a native English speaker and unfortunately didn't get the full picture from your explanation.

If I understand you correctly, you have an internet connection and want to expose your web server to the outside world. I assume you have a static IP address.
You use a router connected to your ISP with this external address and redirect external ports 80 and 443 to the actual web server located on the internal network, in the intranet address space. In theory, this is a proven and reliable setup. (I have the same...)

Clarification: why do you need Nginx as a reverse proxy? Do you have huge traffic and a lot of static content? Do you experience DDOS attacks?
Also, how is your DNS architecture structured?

And most importantly, what's not working?

Ogogon.

Thank you, ogogon, for your message. I have reverse proxy, because I have several services running in separate subdomains that need accessing from outside the LAN (e.g. mail.example.org, drive.example.org, etc.) . It manages the traffic. I am not aware of any DDOS attacks, although it doesn't mean that they are not happening. I use Adblock Home as a DNS server, content filter, etc.

I cannot get to one of my services from within LAN by typing its FQDN address in the browser. It looks like it is a hairpin NAT issue as atax1a has suggested. I just need to figure out how to overcome it.

 

[atax1a]

sorry, we only know how to remedy this issue with pf

 

[ogogon]

I cannot get to one of my services from within LAN by typing its FQDN address in the browser. It looks like it is a hairpin NAT issue as atax1a has suggested. I just need to figure out how to overcome it.

Let me clarify what you mean by "service." Is it a daemon with a separate address (in a jail, for example) or a separate computer (perhaps it would be more correct to call it a server rather than a service)? I'll assume the second.

This server, like the others, is located on the intranet, in the internal address space?
How will his domain address be resolved by your local DNS? What happens if you say dig my-problem-server.mydomain.com or nslookup my-problem-server.mydomain.com?

 

[hakova]

This server, like the others, is located on the intranet, in the internal address space?

Yes.

How will his domain address be resolved by your local DNS?

That is exactly what the reverse proxy does!

 

[ogogon]

That is exactly what the reverse proxy does!

I think you're a bit confused: the purpose of a reverse proxy is to cache static content elements requested by an external user, distribute external user requests across multiple web-servers to balance the load, and perform some security functions. Using it as a DNS server is somehow strange and unusual.
But what IP address of your problematic server does your DNS return on the internal network?

 

[atax1a]

the OP's problem has to do with the internal subnet reaching the public IP address from the internal side of the network, which requires additional translation rules on their firewall. there is no confusion on the OP's end about what their reverse proxy is doing.

 

[ogogon]

the OP's problem has to do with the internal subnet reaching the public IP address from the internal side of the network

I believe this could be solved quite elegantly by creating an internal DNS zone of the same name. This zone, by resolving this domain name, would direct the client to the desired intranet address.
And I don't really understand why a reverse proxy is needed at all in the architecture described. (It seems there is no particular need for it...)

 

[cy@]

is the freebsd machine on the same subnet as the linux machines? this sounds like you have a "hairpin NAT" setup going wrong slightly.

Many commodity routers do not support hairpin NAT. Some do. Others require firewall rules to make it work. Not all routers allow you to create firewall rules.

If your ISP's modem is running in routed mode (not running in bridge mode) it too will NAT. Double NAT complicates your connection. Most ISPs these days encourage routed mode. That replaces your commodity router.

When in routed mode your ISP is responsible for the local firewall.

 

[hakova]

I believe this could be solved quite elegantly by creating an internal DNS zone of the same name. This zone, by resolving this domain name, would direct the client to the desired intranet address.

Thank you, this sounds reasonable and it makes sense to avoid going out to internet to get back into my LAN again. Unfortunately, I am not sure how to do this on my local DNS server, which is Adguard Home.

Many commodity routers do not support hairpin NAT. Some do. Others require firewall rules to make it work. Not all routers allow you to create firewall rules.

If your ISP's modem is running in routed mode (not running in bridge mode) it too will NAT. Double NAT complicates your connection. Most ISPs these days encourage routed mode. That replaces your commodity router.

When in routed mode your ISP is responsible for the local firewall.

My ISP's modem is running in bridge mode, so there is no double NAT situation going on. I have firewall rules set on my router that is attached to ISP's modem, and they work as far as I can tell, at least the rules that define limitations and exceptions of communication between VLANs.

Thank you all for your suggestions. This looks like a very specific issue with local networking rather than FreeBSD. I don't want to take more time and attention. Looks like I have some learning to do.

 

[ogogon]

Unfortunately, I am not sure how to do this on my local DNS server, which is Adguard Home.

I apologize, but I think you've made an odd choice of DNS platform. This is a paid proprietary solution, originally written for the Windows platform by a certain group of programmers from Russia. At the same time, its main task is not resolving DNS requests, but filtering and selectively blocking them. (I wonder who creates ACL lists?)
I wasn't entirely clear from your explanation whether you have at least one decent Unix/Linux server on your internal network. If so, it's entirely possible to run an open-source DNS server on it, such as BIND. You could host a local zone on it and use it as a recurring server to forward requests to other zones on the internet. (Both into root servers and into your provider's servers.) You could configure your router to point to this server as the DNS server.
You'll gain valuable experience and solve your problem.

 

[hakova]

You got a good point there ogogon . I have set up two lxc containers to experiment with unbound server but didn't work on them ever since. I think I will use this opportunity to set them up as recursive DNS servers for my LAN, including creating the local zone you mentioned. Thank you for your great input!

 

[Abdelilah]

You got a good point there ogogon . I have set up two lxc containers to experiment with unbound server but didn't work on them ever since. I think I will use this opportunity to set them up as recursive DNS servers for my LAN, including creating the local zone you mentioned. Thank you for your great input!

The way I solve this issue in my network is that set up override entries in unbound DNS for the needed FQDN, for public access I configure the same FQDN in Cloudflare.
If my client is in the LAN or behind a my VPN he will resolve the FQDN through the local DNS server in this case unbound and will go to the local IP.
If the client is outside, the FQDN is resolved against the public DNS and will go to the public FQDN.
This is the first time I hear the term hairpin NAT and figured out I was using a medieval but simpler solution
I hope this helps.

 

[atax1a]

you'll occasionally hear people talk about "split-horizon DNS", this is what they're talking about

 

[cy@]

Thank you, this sounds reasonable and it makes sense to avoid going out to internet to get back into my LAN again. Unfortunately, I am not sure how to do this on my local DNS server, which is Adguard Home.


My ISP's modem is running in bridge mode, so there is no double NAT situation going on. I have firewall rules set on my router that is attached to ISP's modem, and they work as far as I can tell, at least the rules that define limitations and exceptions of communication between VLANs.

Not all routers have the ability to manage firweall rules. I have a many that don't.

Thank you all for your suggestions. This looks like a very specific issue with local networking rather than FreeBSD. I don't want to take more time and attention. Looks like I have some learning to do.

I have two Internet connections. One using a commodity router and the other through my FreeBSD ipfilter firewall.

Hairpin routing is not a FreeBSD problem. It's a problem that some but not all commodity routers have. Only one of my commodity routers does not have this problem.

 

[John24]

What you’re seeing is classic hairpin-NAT behaviour on a multi-VLAN setup.

Your Linux machines work because they’re on the router’s main LAN interface, where NAT loopback is applied.
Your FreeBSD host is in 192.168.70.0/24, and most consumer routers don’t apply hairpin NAT to additional VLAN interfaces.
So when it reaches your public IP, the router can’t loop it back to the reverse proxy and simply serves its own web UI.

Two usual fixes:

1. Split-horizon DNS
Internal clients resolve media.example.org to the internal reverse-proxy IP.
External clients resolve the public IP.
Unbound can do this easily.
https://en.wikipedia.org/wiki/Split-horizon_DNS

2. Check inter-VLAN filtering
When filtering is too strict, routed traffic can behave like a NAT failure.
Here are two clear references on how ACLs affect inter-subnet traffic:
https://pingmynetwork.com/network/ccna-200-301/how-acl-work
https://www.cisco.com/c/en/us/support/docs/ip/access-lists/26448-ACLsamples.html

 

[hakova]

Since the last post, I was able to install and set up unbound DNS server on a lxc container, defined local zone and A records of servers pointing to the internal LAN address of my reverse proxy host. Thank you all for your guidance, they are all very much appreciated! I did not define PTR records, since the A records were all the same, pointing to the reverse proxy's internal IP address. I am not sure if this is the underlying cause of my next problem. First, the good news:

I achieved my original goal. I can land on the correct web page of each server with https, lightning fast.

However, I noticed that I stopped getting emails from my FreeBSD VM, which contains the WebCal server. I am talking about the daily, weekly and monthly emails sent by cron. This server's A record is directed to the proxy server in unbound.conf. Similarly, I stopped getting backup reporting emails from my home lab server, which runs on Proxmox. These latter emails are daily emails reporting successful or failed backup of each container or VM on the server. This server's A record does NOT exist in unbound.conf. Curiously, I can still get emails from my NAS, which runs on FreeBSD14.3. This server's A record also does NOT exist in unbound.conf. The WebCal server is located on VLAN 1, whereas the other one with Proxmox is on VLAN 20. My NAS is also located on VLAN 1. I did some reading and based on that I suspect that enabling DNSSEC may have something to do with email transport from those hosts. However, I am having difficulty in finding more resources on how to solve this problem. Or is that happening because the PTR records not being listed on unbound.conf ?

I realize that the topic is now somehow outside the scope of FreeBSD, but I would very much trust and appreciate your expertise and suggestions.