Now I got this one here almost solved. It unravelled to a whole bunch of issues.
1. downloads hanging at the beginning of the last 128k block
This seems to be reproducible, coming from suricata when in IPS mode "running on one leg", i.e. only the incoming packets are fed to it, not the outgoing ones.
I think this is a malfunction and should not happen, otoh suricata is focused on analyzing flows, and so it needs to see both directions. And over all, suricata appears to be quite delicate about it's configuration being sensible.
2. various connection failures after fixing item 1.
That one was a bug created by me: I just lost these flows somewhere within the ruleset and they went onto a reject rule.
3. IPv4 header checksum errors
SURICATA IPv4 invalid checksum [Classification: Generic Protocol Command Decode]
This is normal and a design feature of the divert.
divert(4):
Packets are received and sent unchanged, except that packets read as outgoing have invalid IP header checksums
More precisely, when feeding an outgoing packets out of ipfw onto a divert socket, is has the hopcount already decremented, but the checksum not changed.
I would think, as suricata provides a special adapter to run inline in ipfw on a divert socket, it should also respect this feature and behave accordingly. But - it doesn't.
Using the option
-k none to switch off checksums does not really help - it does disable the messages for TCP traffic, but not for UDP, ICMP. (And assumingly this does also switch of the TCP checksum validataion, not only the IP header checksum.)
What does help is to disable the offending rule. Configure
suricata-update to use a
disable.conf file, and therein put a line with only
1:2200073.
4. suricata suddenly crashes with "permission denied"
[102574] <Warning> -- [ERRCODE: SC_WARN_IPFW_XMIT(84)] - Write to ipfw divert socket failed: Permission denied
[102843] <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - thread TX#00 failed
I have now seen this one more time, and I do not really understand what it means.
As mentioned, suricata is delicate about it's conditions. It appears to have orignally derived from some admin scripts, and it seems still to be a vast assortion of functionality packed together, lacking some means to ascertain proper over-all reliability.
The problem here is: when suricata runs inline on a divert socket, and it crashes, then those packets go nowhere and connectivity is lost. What can be done: putting this into
/etc/crontab to be run periodically:
service suricata status >/dev/null || service suricata start
)