You should be able to use max-src-conn, what rule did you come up with? Something like this might work...
pass in on $ext_if proto icmp all keep state (max-src-conn 100, max-src-conn-rate 5/5, overload <some_table> flush)
some_table being a table of hosts that you block/drop connections from...