Solved Starting settings IPFW for vnet&jail

jeltoesolnce

jeltoesolnce

Hello.
I'm not guru in IPFW, I use settings as
Code: 
firewall_enable="YES"
firewall_script="/etc/rc.firewall"
firewall_type="WORKSTATION"
firewall_myservices="ssh 667/udp 667/tcp 631/tcp ftp http"
firewall_allowservices="any"
and I created jail&vnet . Now, if host IPFW work, I can't run pkg -j jail install <some_package> - I get
Code: 
Updating FreeBSD repository catalogue...
pkg: http://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/meta.txz: Unknown resolver error
repository FreeBSD has no meta file, using default settings
pkg: http://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/packagesite.pkg: Unknown resolver error
pkg: http://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/packagesite.txz: Unknown resolver error
Unable to update repository FreeBSD
Error updating repositories!
What must I set in this notation?
(If IPFW stop, all works).
 
It looks like DNS problem.
Try to add the ipfw rule to enable DNS traffic
It may be something like:
Code: 
ipfw add 100 allow udp from me to any 53
ipfw add 105 allow udp from any 53 to me
 
For DNS you should also allow 53/TCP. DNS uses both UDP and TCP (lots of requests/responses don't fit in a UDP packet anymore).
 
Thanks, but does not work.
What I have: Jail & vnet, all works without firewalls. Host - FreeBSD 13.2, Jail - FreeBSD 13.2. If both IPFW stopped - all work fine. If host IPFW work - jail does not connect to Internet, host can't run pkg -j <jail> install some_package

Host Ifconfig
Code: 
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 04:d9:f5:32:65:64
        inet 192.168.1.51 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::6d9:f5ff:fe32:6564%re0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vboxnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0a:00:27:00:00:00
        media: Ethernet autoselect
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vboxnet1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0a:00:27:00:00:01
        media: Ethernet autoselect
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
...
<Vbox is here>
...
vboxnet9: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0a:00:27:00:00:09
        media: Ethernet autoselect
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
re0bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:47:7b
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_shumbely flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_shumbely: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:7e:e1:32:65:64
        hwaddr 02:16:aa:44:1e:0a
        inet6 fe80::16:aaff:fe44:1e0a%e0a_shumbely prefixlen 64 scopeid 0xe
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
[sailorsamoor@six ~]%
IPFW rules:
Code: 
sysctl: oid 'security.jail.jailed' is read only at line 72
Flushed all rules
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any icmp6types 1
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
00000 check-state :default
01200 allow tcp from me to any established
00000 allow tcp from me to any setup keep-state :default
00000 allow udp from me to any keep-state :default
00000 allow icmp from me to any keep-state :default
00000 allow ipv6-icmp from me to any keep-state :default
01700 allow udp from 0.0.0.0 68 to 255.255.255.255 67 out
01800 allow udp from any 67 to me 68 in
01900 allow udp from any 67 to 255.255.255.255 68 in
02000 allow udp from fe80::/10 to me 546 in
02100 allow icmp from any to any icmptypes 8
02200 allow ipv6-icmp from any to any icmp6types 128,129
02300 allow icmp from any to any icmptypes 3,4,11
02400 allow ipv6-icmp from any to any icmp6types 3
02500 allow udp from me to any 53
02600 allow udp from any to me 53
02700 allow tcp from me to any 53
02800 allow tcp from any to me 53
Consider using ssh/tcp in firewall_myservices.
02900 allow tcp from any to me 22
03000 allow udp from any to me 667
03100 allow tcp from any to me 667
03200 allow tcp from any to me 631
Consider using ftp/tcp in firewall_myservices.
03300 allow tcp from any to me 21
Consider using http/tcp in firewall_myservices.
03400 allow tcp from any to me 80
65000 count ip from any to any
65100 deny { tcp or udp } from any to any 135-139,445 in
65200 deny { tcp or udp } from any to any 1026,1027 in
65300 deny { tcp or udp } from any to any 1433,1434 in
65400 deny ip from any to 255.255.255.255
65500 deny ip from any to 224.0.0.0/24 in
65500 deny udp from any to any 520 in
65500 deny tcp from any 80,443 to any 1024-65535 in
65500 deny ip from any to any
Firewall rules loaded.
sysctl: oid 'security.jail.jailed' is read only at line 72 I get from Handbook.
 
Hi,

Not a pro, but last time I did use jail vnet a firewall configuration was required for the host and the jail/s.
In the rc.conf file jail side, just add this to test:
Code: 
firewall_enable="YES"
firewall_type="open"
 
gotnull said:
Hi,

Not a pro, but last time I did use jail vnet a firewall configuration was required for the host and the jail/s.
In the rc.conf file jail side, just add this to test:
Code: 
firewall_enable="YES"
firewall_type="open"
Click to expand...
Thank you, I did it, but the same situation... Network jail works without host-firewall only.. I suppose, I have to allow something on the host, but what?
 
May be try these settings (it's coming from iocage doc) :
Code: 
# Add these tunables to /etc/sysctl.conf:
net.inet.ip.forwarding=1             # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0       # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0      # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface

I've been on that situation too when I played with vnet jails, I took few notes of what I did but not enough sadly so I don't remember exactly all I did, but I remember it was a long road ...
 
Thanks, it works.
Itt turns out that as my virtual adapter not includes in "me" notation?
"allow udp from any to me 53": "me" in this case shows physical adapter only?

Ok, will be try further...
 
gotnull said:
May be try these settings (it's coming from iocage doc) :
Code: 
# Add these tunables to /etc/sysctl.conf:
net.inet.ip.forwarding=1             # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0       # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0      # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface

I've been on that situation too when I played with vnet jails, I took few notes of what I did but not enough sadly so I don't remember exactly all I did, but I remember it was a long road ...
Click to expand...
Thank you, I will allow all traffic into jail, and then in IPFW jail correct them.
 
jeltoesolnce said:
"me" in this case shows physical adapter only?
Click to expand...
That's funny because I checked it in the man page (man ipfw) 2 days ago because I was not sure about what was included in "me".
me Matches any IP address configured on an interface in the system.
Click to expand...
But honestly I can't tell if virtual adapter is taken into account or not, you'll need a response from a better experienced user.


So in your case it works without any jail's firewall config, without net.link.bridge.* bit , just only with the host's rc.conf firewall config ? that's interesting.
 
Good, well I learn something new today which appears to be confirmed by another discussion I came across, VNET is something!
I think that I will have to test this later, thank you.
 
jeltoesolnce said:
Hello.
I'm not guru in IPFW, I use settings as
Code: 
firewall_enable="YES"
firewall_script="/etc/rc.firewall"
firewall_type="WORKSTATION"
firewall_myservices="ssh 667/udp 667/tcp 631/tcp ftp http"
firewall_allowservices="any"
Click to expand...

'Workstation' is for a client enduser system; it's not appropriate for a gateway to other systems or networks, that is, it's not designed for a router.

jeltoesolnce said:
and I created jail&vnet . Now, if host IPFW work, I can't run
Click to expand...

(anything needing the host ipfw to forward packets to and from the vnet jail, firstly DNS ...)

Can you show configuration for your vnet jail, as opposed to an ordinary jail? I gather a vnet jail with its own network stack will require the host to act as a router, using

net.inet.ip.forwarding=1

in which case you would need a ruleset like rc.firewall 'simple' on the host, to both protect the host and forward (selected) traffic for the jail, on a distinct interface.

That said, I'm not familiar with vnet jails requirements or setup at all, but used 'simple' network ipfw for many years.
 
I read this tread about vnet VNET & Jail & FreeBSD on this forum. I don't understand the settings very well myself. I see that I need to setting two files, basically: host /etc/jail.conf, jail /etc/rc.conf.

That is HOST/etc/jail.conf
Code: 
exec.start = "/bin/sh /etc/rc";            # Start command
exec.stop = "/bin/sh /etc/rc.shutdown";    # Stop command

shumbely {
    path = "/usr/home/jails/shumbely";         # Path to the jail
    host.hostname = _shumbely_;                  # Hostname
    exec.consolelog = "/var/log/jail_shumbely_console.log";
    exec.system_user = "root";
    exec.jail_user = "root";
    exec.clean;

    vnet;
    vnet.interface = "e0b_shumbely";               # vnet interface(s) - e0b_ - it is important something to do with "bridge"
    exec.prestart += "/usr/local/sbin/jib addm shumbely re0"; #  "jib" - it is something about bridge (/usr/share/examples/jails/jib - needle to copy into /usr/local/sbin(?)
    exec.poststop += "/usr/local/sbin/jib destroy shumbely";
 
    mount.devfs;                               # Mount devfs inside the jail
    allow.raw_sockets;                            #allow ping-pong
    devfs_ruleset="5";                            #devfs ruleset for this jail
    allow.set_hostname = 1;
}
It is my first attempt jail settings, but it somehow works.
jib - sh script for manage bridges, as I understood.

That is JAIL/etc/rc.conf
Code: 
host_hostname="shumbely"
ifconfig_e0b_shumbely="inet 192.168.1.40/24" # - e0b - this is that I told upper.
defaultrouter="192.168.1.1"
firewall_enable="YES"
firewall_quiet="YES"
firewall_script="/etc/rc.firewall"
firewall_type="OPEN"

I thinking *e0a* and *e0b* are parts of bridge interface, but I dont understand how can I named them.
And firewall settings very simple: host IPFW see IP 192.168.1.40 as real IP, and notation "me" not includes this address. I'm thinking now how to add another one interface...

A part of HOST ifconfig output:
Code: 
re0bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:47:7b
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_shumbely flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_shumbely: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:7e:e1:32:65:64
        hwaddr 02:35:24:05:30:0a
        inet6 fe80::35:24ff:fe05:300a%e0a_shumbely prefixlen 64 scopeid 0xe
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

A part of JAIL ifconfig:
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_shumbely: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 0e:7e:e1:32:65:64
        hwaddr 02:35:24:05:30:0b
        inet 192.168.1.40 netmask 0xffffff00 broadcast 192.168.1.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Have a nice day, congratulations! (Today Programmer Day))
 
jeltoesolnce said:
I read this tread about vnet VNET & Jail & FreeBSD on this forum. I don't understand the settings very well myself. I see that I need to setting two files, basically: host /etc/jail.conf, jail /etc/rc.conf.
Click to expand...

Thanks for the reference, good article, and for the very detailed configurations. Please excuse my slowness getting back.

jeltoesolnce said:
I thinking *e0a* and *e0b* are parts of bridge interface, but I dont understand how can I named them.
Click to expand...

Usually epair0a and b, so I guess you must have renamed the epair interface somewhere?

jeltoesolnce said:
And firewall settings very simple: host IPFW see IP 192.168.1.40 as real IP, and notation "me" not includes this address.
Click to expand...

As that article very well explains; I had few clues before reading that. Did you need to enable
net.inet.ip.forwarding to pass the jail traffic from/to the net?

You've since marked this thread Solved ... so what did you finish up doing about host and jail ipfw rules to protect both?

cheers, Ian
 
smithi said:
You've since marked this thread Solved ...
Click to expand...
I understood how can I begin my work. I guess I'll set Workstation or Simple in my workstation ipfw and set some rules for vnet interfaces. Because I want to create small service (http, mysql etc.) I not to need in complicated rules for jail ipfw. But it is interesting.
 
smithi said:
Usually epair0a and b, so I guess you must have renamed the epair interface somewhere?
Click to expand...
I use sh script jid FreeBSD Git
Yes, it was in jid code:
Code: 
#     # NB: Below 2-lines required
#     # NB: The number of eNb_xxx interfaces should match the number of
#     #     arguments given to `jib addm xxx' in exec.prestart value.
#     #
#     vnet;
#     vnet.interface = e0b_xxx, e1b_xxx, ...;
 
You must log in or register to reply here.
Back
Top