Although we’ve tried to automate the process of testing and analyzing, there are cases in which you have to interpret the data yourself. One example happend when testing our firewall, running the ICMPv6 Filtering:
It showed that 4 ICMPv6 messages had been incorrectly dropped (Destination Unreachable, Packet Too Big, Time Exceeded and Parameter Problem) whilst all others had been handled correctly. Even when configuring policies to explicitly allow these packets, they still didn’t get through to the server. So we concluded that the firewall was not too stupid but in fact too smart: This becomes clear when thinkin about what these messages are saying: “There was a problem with the packet I just received”. Because the server never sent a packet that was too big, the client “responding” with Packet Too Big doesn’t make much sense. We guessed that the firewall was stateful enough to detect that.
Another issue occured when performing Tiny Fragments: The firewall appeared to be dropping all tiny fragments, even the ones that were valid. ft6 reported that the firewall was incorrectly dropping tiny fragments too early after 55 seconds and correctly not waiting too long, because after 65 seconds the packet didn’t arrive any more. Of course, when the firewall drops all tiny fragments, the statements about the timeout are not very meaningful any more.
After a lengthy discussion we still chose to keep it that way. There are just too many unforseen things that could happen that we feel we are never able to completely analyze the tests automatically. We think it’s best to keep the tool simple and trust in your ability to come up with smart interpretation.