Jails, inet aliases and hosts files

N

nanotek

Wasn't entirely sure where to post this, please relocate if it is inappropriate here.

Today, one of my alias local IPs for my Apache jail just disappeared and my web server went down. This has never happened before, is it "usual" or does it require further investigation? log/messages on the host and in the jail reveal nothing. I am not sure where else to look. If it is not usual, what should I look at to discover the cause? If it is, is a script or similar available that would be able to determine if such an event happens and recreate the required local IP alias and restart the jail? Thanks.

As an aside, here is my hosts /etc/hosts file:
Code: 
127.0.0.1       localhost localhost.bsdbox.co
127.0.0.1       nakatomi.bsdbox.co nakatomi localhost
10.0.0.50       nakatomi.bsdbox.co nakatomi
10.0.0.111      bsdbox.co
10.0.0.112      zero.bsdbox.co zero
10.0.0.113      mail.bsdbox.co mail

and the /etc/hosts file from the jail in question:
Code: 
127.0.0.1               localhost localhost.my.domain

Should I add these lines to my jail hosts file?
Code: 
127.0.0.1  bsdbox.co
10.0.0.111 bsdbox.co

Because I just noticed, this jail takes an inordinate amount of time to start.

And is the format of my host hosts file good?
 
Which or what alias did exactly disappear and how did you determine that it did? Are we talking about an IP alias which you can set using ifconfig or are we looking at an entry in the /etc/hosts file?

Because the latter would not make an Apache server suddenly stop working.

Keep in mind that the hosts file is only used by the system to do local lookups. By default, at its minimum, to be able and quickly lookup the localhost entry. Still; it's usage is also heavily influenced by the /etc/resolv.conf file.

So merely looking at the hosts file does not give you a good indication of what might have happened.

Edit:

As to the jail taking a long time to boot. That could be an indication of a DNS problem. But if localhost is properly defined this would usually mean that there are services which try to access other, perhaps foreign, locations. So is there anything else besides Apache working in this jail?
 
Yes; an alias set with ifconfig. I determined it disappeared because when I realized the web server was down, ifconfig from within the jail displayed no IP and ifconfig on the host showed only 3 IPs (2 jails and the host) instead of the usual 4; the missing IP was the Apache jail.

host /etc/rc.conf
Code: 
# bsdbox.co
ifconfig_rl0_alias0="inet 10.0.0.111 netmask 0xffffff00 broadcast 10.0.0.255"
# zeroAlpha
ifconfig_rl0_alias1="inet 10.0.0.112 netmask 0xffffff00 broadcast 10.0.0.255"
# mail.bsdbox.co
ifconfig_rl0_alias2="inet 10.0.0.113 netmask 0xffffff00 broadcast 10.0.0.255"

Normally, ifconfig on the host will display:
Code: 
ether xx:xx:xx:xx:xx:xx
        inet 10.0.0.113 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.0.0.112 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.0.0.50 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.0.0.111 netmask 0xffffff00 broadcast 10.0.0.255

When I realized the web server was down, the .111 IP was missing.

I just posted the /etc/hosts query because I wondered if an absence of anything more than a localhost entry within the jail hosts file is causing a protracted start. Not because I thought it brought down Apache. My /etc/resolv.conf file only has one entry; my router as nameserver. In this jail there is only Apache, MySQL and PHP running. When I say it takes a long time to start, I didn't time it, but it may be in the vicinity of 90-120 seconds.

The sudden disappearance of the ifconfig set alias brought the Apache server down. I have no idea how an IP can just disappear though!
 
Which FreeBSD version are you using? Are you by any chance using a customized kernel?

The only way I can see this happening is if there's something wrong on the host. Jails normally don't have the required permissions to change network settings.

How did you configure those jails and how do you manage them?
 
9.2-RELEASE, GENERIC kernel. Only changes made on the host were to my /etc/hosts file[1]. Jail management with ezjail:

Code: 
# ifconfig rl0 alias 10.0.0.22 netmask 0xffffff00 broadcast 10.0.0.255
# echo 'ifconfig_rl0_alias3="inet 10.0.0.22 netmask 0xffffff00 broadcast 10.0.0.255"' >> /etc/rc.conf
# ezjail-admin create leavenworth 10.0.0.22
# cp /etc/resolv.conf /usr/jails/leavenworth/etc/
# ezjail-admin start leavenworth

[1]
Code: 
#::1                    localhost localhost.nakatomi.bsdbox.co
#127.0.0.1              localhost localhost.nakatomi.bsdbox.co
127.0.0.1       localhost localhost.bsdbox.co
127.0.0.1       nakatomi.bsdbox.co localhost nakatomi
10.0.0.50      nakatomi.bsdbox.co nakatomi
10.0.0.111     bsdbox.co
10.0.0.112     zero.bsdbox.co zero
10.0.0.113     mail.bsdbox.co mail

Any ideas? I'm not even sure where to look to investigate.

ETA: Do you think the problem could be at the router? I recently made a hard reset on it when changing providers. In fact, that must be it. This only started occuring since I made the local IP additions to my hosts file and the hard reset at the router. Only the latter makes any sense.
 
You're not giving enough information for me to come to such conclusions. Personally I doubt that a router could be influencing all this, but it heavily depends on the environment. For example; if your server is using DHCP then I could see some outside interference causing issues on a machine itself. But if you're using static IP addresses (see /etc/rc.conf) then I think it's out of the question that an external router could cause for this to happen.

The only scenario where I could imagine something odd to happen is if you're creating a same IP address on the same network. Although I strongly doubt that FreeBSD would simply drop the IP address it's using I suppose it might be theoretically possible.

In my opinion the cause of the problem is on the server itself. But I have no idea what that might be, it most certainly isn't normal behaviour. I'd start by ruling out options like a security compromise. So other people trying weird things on your server which end up with the results you described.
 
ShelLuser said:
You're not giving enough information for me to come to such conclusions. Personally I doubt that a router could be influencing all this, but it heavily depends on the environment. For example; if your server is using DHCP then I could see some outside interference causing issues on a machine itself. But if you're using static IP addresses (see /etc/rc.conf) then I think it's out of the question that an external router could cause for this to happen.
Click to expand...

I failed to nominate a static local IP for the host (10.0.0.50) in my router administration. Therefore, on the router device list my host has been changing its IP to the various jail alias IPs. I think this may be the cause of the dropped aliases. For example, at this moment my router displays the host as possessing one of its jail IP addresses: 10.0.0.112. I am not sure of the best course of action to correct this suboptimal situation. Assigning IPs in /etc/rc.conf was one such attempt, but it seems to be ineffective. I think I need to effect a change in my router admininstration.

The only scenario where I could imagine something odd to happen is if you're creating a same IP address on the same network. Although I strongly doubt that FreeBSD would simply drop the IP address it's using I suppose it might be theoretically possible.
Click to expand...

I did not do this but as a result of neglecting to set the host IP as static on my router configuration it has had its IP changed to the different alias IPs I have created for the jails. Which, I guess, is essentially the same as creating an alias for a jail with an IP already in use by the host?

In my opinion the cause of the problem is on the server itself. But I have no idea what that might be, it most certainly isn't normal behaviour. I'd start by ruling out options like a security compromise. So other people trying weird things on your server which end up with the results you described.
Click to expand...

There were no security comprimises. Nevertheless, I've temporarily revoked all access to the host except through the terminal or SSH access to my IP and key. At the same time, I commented out the assigned local IPs in /etc/hosts. So far, no aliases have been dropped. However, I still need to somehow assign a new IP to the host at my router and make it static so it is not sharing the same local IP as one of its jails. I can't figure out how to do this without dropping connexions which I would rather not do. Would ifconfig interface inet desiredIP work? Example:

Code: 
# ifconfig rl0 inet 10.0.0.50
Would this change the host IP to 10.0.0.50 where I can then head to my router administration and set it to static?

If not, maybe I just have to wait for DHCP to assign a new IP to the host, hope its not another jail IP and then set it to static. I can't believe I forgot to set it to fixed after resetting and reconfiguring the router!
 
Use static addresses on the main host, DHCP interfaces may reset their aliases unexpectedly.


Code: 
ifconfig_rl0="inet 10.0.0.50/24"
defaultrouter="10.0.0.1"

I'm just guessing the default router is 10.0.0.1, change that with the real address is different. I'm also guessing that the netmask is a /24. You also have to edit /etc/resolv.conf to read at least:

Code: 
nameserver 10.0.0.1

Same deal with the address, change is necessary.
 
kpa said:
Use static addresses on the main host, DHCP interfaces may reset their aliases unexpectedly.

Code: 
ifconfig_rl0="inet 10.0.0.50/24"
defaultrouter="10.0.0.1"
Click to expand...

Thanks, @kpa! Out of interest, what's the purpose of specifying the class C netmask? Wouldn't this mean .50 is variable? Or is that the point: so that aliases on the host can be made for the jails?

I'm just guessing the default router is 10.0.0.1, change that with the real address is different. I'm also guessing that the netmask is a /24. You also have to edit /etc/resolv.conf to read at least:

Code: 
nameserver 10.0.0.1

Same deal with the address, change is necessary.
Click to expand...

I do have my router IP as the nameserver in /etc/resolv.conf on the hosts and in the jails. Thank you very much for chiming in. I really appreciate it.
 
Last edited by a moderator:
I believe that for historical reasons the 10.*.*.* addresses are still assumed to be /8 addresses if the netmask is not specified.
 
I was just reading through ifconfig(). This made me question if I've been creating my jail alias IP addresses incorrectly:

ifconfig manual page said:
alias Establish an additional network address for this interface. This
is sometimes useful when changing network numbers, and one wishes
to accept packets addressed to the old interface. If the address
is on the same subnet as the first network address for this
interface, a non-conflicting netmask must be given. Usually
0xffffffff is most appropriate.
Click to expand...

The italics: I create my jail aliases on the same subnet and use the same netmask. For example, to create a new alias IP for a new jail I issue this command: ifconfig rl0 alias 10.0.0.22 netmask 0xffffff00 broadcast 10.0.0.255 (only varying the local IP [10.0.0.22] for each jail). As a result, this is my current setup:

Code: 
# ifconfig
<snip>
ether xx:xx:xx:xx:xx:xx
        inet 10.0.0.50 netmask 0xffffff00 broadcast 10.0.0.255      <-- HOST IP
        inet 10.0.0.111 netmask 0xffffff00 broadcast 10.0.0.255     <-- Jail #1 (web server)
        inet 10.0.0.112 netmask 0xffffff00 broadcast 10.0.0.255     <-- Jail #2 (relay)
        inet 10.0.0.113 netmask 0xffffff00 broadcast 10.0.0.255     <-- Jail #3 (mail server)
        inet 10.0.0.114 netmask 0xffffff00 broadcast 10.0.0.255     <-- Jail #4 (research and technological development)
<snip>

Should I have assigned unique netmasks (0xffffffff) to each jail?
 
Yes since the aliases are in the same subnet as the main address you have to use a /32 (netmask 255.255.255.255 in other words) subnet mask.
 
And this is achievable with this command:

ifconfig rl0 alias 10.0.0.22 netmask [b]0xffffffff[/b] broadcast 10.0.0.255

The 0xffffffff designates a /32 CIDR?
 
nanotek said:
And this is achievable with this command:

ifconfig rl0 alias 10.0.0.22 netmask [b]0xffffffff[/b] broadcast 10.0.0.255

The 0xffffffff designates a /32 CIDR?
Click to expand...

Yes, same thing but different notation. The hex number 0xffffffff equals 255.255.255.255 when written as an IP address and in CIDR notation (/32) it reads "first 32 bits from the left make up the network part of the address".
 
Awesome. Thanks, @kpa.

What is involved to create the jails on a different subnet? Do I need to administer some configuration at my router? Or is it simply done by specifying a 10.0.1.xx (/16) alias IP? For example:

ifconfig rl0 alias [b]10.0.1.11[/b] netmask [b]0xfffff000[/b] broadcast [b]10.0.255.255[/b]
 
Last edited by a moderator:
The router would have to be made aware of the additional subnet. This is done by assigning its LAN interface an address from the additional subnet. Also the router would have to told to perform outbound NAT for the additional subnet. Be sure not to create overlapping subnets, your example 10.0.1.11 netmask 0xfffff000 (in CIDR 10.0.1.0/20) would overlap with the existing subnet 10.0.0.0/24 because it includes the smaller subnet's addresses.

If all this starts to sound too complicated you could use a cloned lo(4) interface or even more than one to bind the jails at. You would then use for example pf(4) to set up port forwarding from the main address of the host to the jail addresses and also outbound NAT for the jail addresses so they would not be visible outside the host.
 
kpa said:
The router would have to be made aware of the additional subnet. This is done by assigning its LAN interface an address from the additional subnet. Also the router would have to told to perform outbound NAT for the additional subnet.
Click to expand...

This is all beyond my current level of development. But I'm going to look into this. I guess some of this configuration would be done at my router administration and some on the host (rc.conf and hosts file)? When you say assign a new LAN IP to the router, I assume it's an additional IP; so it would possess 2 internal IPs (e.g., 10.0.0.138[existing] and 10.0.1.138[new LAN IP])?

Be sure not to create overlapping subnets, your example 10.0.1.11 netmask 0xfffff000 (in CIDR 10.0.1.0/20) would overlap with the existing subnet 10.0.0.0/24 because it includes the smaller subnet's addresses.
Click to expand...

This made my head hurt. I quickly tried to work out how to create a new small subnet (capable of servicing 16 local IPs) and came up short. My math is terrible. Fortunately, I start my math unit in February. However, I'd like to attempt this now. How does one formulate a feasible subnet that doesn't overlap with existing addresses? I was just reading that you need to subtract bits from the host portion of the address (viz. the last '0.xxx' portion) and add them to the new subnet while keeping the network portion of the address constant. So if the existing subnet is:

10.0.0.138/24

then 10.0 is the network address and 0.138 is the host address. And all IPs from 10.0.0.1 - 10.0.0.136 are within this existing subnet. Now, we want 16 potential new local addresses in the new subnet, so we need 8 bits from the host address:

10.0 remains the network and .18 (16 new IPs + 1 for network and 1 for subnet mask) becomes the new subnet and .120 remains the host address. That is, the new subnet would contain addresses such as.. I don't know???

This is really complicated! My current DHCP pool is only 10.0.0.1 - 10.0.0.136. So there's plenty of room to create subnets that don't overlap.

If all this starts to sound too complicated you could use a cloned lo(4) interface or even more than one to bind the jails at. You would then use for example pf(4) to set up port forwarding from the main address of the host to the jail addresses and also outbound NAT for the jail addresses so they would not be visible outside the host.
Click to expand...

It is complex for me but I need to learn this eventually. Further, subnetting appears to be the preferred practice so I should probably start with it before cloning.

There has to be a formula for working out how to create new subnets?

ETA: Thanks for educating me too by the way.
 
Same here. I got such a problem (12.0-RELEASE) at my jail hosting server.
Have no idea how to solve it. Network interface's alias randomly disappears, sometimes it can work for a week, sometimes not more than one day.
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
ether <skipped>
hwaddr <skipped>
inet 10.0.200.179 netmask 0xffffffff broadcast 10.0.200.179
inet 10.0.200.180 netmask 0xffffffff broadcast 10.0.200.180
inet 10.0.200.173 netmask 0xffffffff broadcast 10.0.200.173
inet 10.0.200.174 netmask 0xffffffff broadcast 10.0.200.174
inet 10.0.200.175 netmask 0xffffffff broadcast 10.0.200.175
inet 10.0.200.177 netmask 0xffffffff broadcast 10.0.200.177
inet 10.0.200.176 netmask 0xffffffff broadcast 10.0.200.176
inet 10.0.200.178 netmask 0xffffffff broadcast 10.0.200.178
inet 10.0.200.70 netmask 0xffffff00 broadcast 10.0.200.255
inet 10.0.200.172 netmask 0xffffffff broadcast 10.0.200.172
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 
You must log in or register to reply here.
Back
Top