Trying to run a traffic shaper in a VIMAGE jail. (It does not appear to work.)
The simple part:
For half a year I've now been running VIMAGE jails, and setting up ipfw within them, but didn't notice that they are all running with one_pass=1, in contrast to the base system. Only when confguring pipes/queues, this does obviousely not work, because the packets leave the ruleset and cannot be processed further after being transferred the queue.
The freaky part: pipes and queues exist only once.
After things were running as expected, I deleted the traffic shaper in the base system, as it should no longer be needed. And the one in the jail was also gone!
There is only one instance of pipes+queues. The same ones are visible in the base system and in every jail, and they can be created, modified and deleted from any jail! So, if you have a traffic shaper in the base system, and give a jail to some freaks, they can do with it whatever they want...
The known part: delays do not work.
This one is already on record: https://docs.freebsd.org/cgi/getmsg.cgi?fetch=40348+0+archive/2019/freebsd-net/20190728.freebsd-net
Sadly, there is no comment on the issue, whatsoever.
But i the light of the further findings, I might suppose these packets do not just "get lost", they rather reappear at some unexpected place...
The ugly part: data traverses between ipfw instances
Since I do not need delays on the pipes, I then started to run the whole thing. And I perceived very strange behaviour, e.g. I could access my own webserver with http, but not with https. I could read this forum more or less well (with occasionally missing CSS), but when I tried to compose a message, I got timeout and "secure connection failed".
Analyzing the whole packet's path I found that some packets, which should go through the pipe in the jail's ipfw, were rejected in the base system's ipfw, at a rule which they could impossibly have reached. Also, some of these packets were reported as incoming on a netif that did not belong to the base system, but to the jail!
Switching off the traffic shaper in the jail resolves all these issues.
The only explanation I have so far is: the pipes do not keep track on which ipfw-instance a packet belongs to, and re-insert it into some arbitrary ruleset, depending on packet-size, port-number or whatever.
The simple part:
sysctl net.inet.ip.fw.one_pass=0
must be set for every jail individually.For half a year I've now been running VIMAGE jails, and setting up ipfw within them, but didn't notice that they are all running with one_pass=1, in contrast to the base system. Only when confguring pipes/queues, this does obviousely not work, because the packets leave the ruleset and cannot be processed further after being transferred the queue.
The freaky part: pipes and queues exist only once.
After things were running as expected, I deleted the traffic shaper in the base system, as it should no longer be needed. And the one in the jail was also gone!
There is only one instance of pipes+queues. The same ones are visible in the base system and in every jail, and they can be created, modified and deleted from any jail! So, if you have a traffic shaper in the base system, and give a jail to some freaks, they can do with it whatever they want...
The known part: delays do not work.
This one is already on record: https://docs.freebsd.org/cgi/getmsg.cgi?fetch=40348+0+archive/2019/freebsd-net/20190728.freebsd-net
Sadly, there is no comment on the issue, whatsoever.
But i the light of the further findings, I might suppose these packets do not just "get lost", they rather reappear at some unexpected place...
The ugly part: data traverses between ipfw instances
Since I do not need delays on the pipes, I then started to run the whole thing. And I perceived very strange behaviour, e.g. I could access my own webserver with http, but not with https. I could read this forum more or less well (with occasionally missing CSS), but when I tried to compose a message, I got timeout and "secure connection failed".
Analyzing the whole packet's path I found that some packets, which should go through the pipe in the jail's ipfw, were rejected in the base system's ipfw, at a rule which they could impossibly have reached. Also, some of these packets were reported as incoming on a netif that did not belong to the base system, but to the jail!
Switching off the traffic shaper in the jail resolves all these issues.
The only explanation I have so far is: the pipes do not keep track on which ipfw-instance a packet belongs to, and re-insert it into some arbitrary ruleset, depending on packet-size, port-number or whatever.