VESARiA Network Security Specialists
About Vesaria Services Consulting Partners Research Customers Contact Us

Firewall Testing



What would "certification" mean in a firewall?

Certifying something means it passed some kind of test

Implicit in the issue of "certification" is the matter of testing, and that's a really tough nut to crack. Before you can certify a firewall, you need to be able to measure it against some kind of yardstick and determine if it is adequate. Even the concept of adequacy is slippery to come to grips with. A firewall may be adequate from a security perspective but unable to do the job because of some special requirement, cost, or whatever.

More importantly, a firewall needs to be correct for its proposed use and that needs to be taken into account when it is "certified." In the past I've given the example of a highly secure high assurance firewall for Email only, which can be easily implemented using a screening router and a UNIX machine. In some people's eyes that might not even be a "firewall" -- a rigid code for certification likely would not accept such a firewall as OK.

So how do you test a firewall?

I believe there are 2 approaches, which are not necessarily mutually exclusive or incompatible:

  • Programmed "checklist" testing or automated testing
  • Design-oriented testing

Checklist testing: Pro and Con

"Checklist" testing would amount to running SATAN++ against the firewall and failing it if SATAN++ found a hole. Do not pass go, do not collect $200. The problem with this approach is that it is very limited: a bug that we don't test for in SATAN++ could slice right through the firewall tomorrow and we'd have to invalidate the whole certification and recertify. The advantage of the "checklist" approach is that it's cheap, quick, easy, and it lets a vendor put a certification "seal of approval" on their product and everyone can get a quick set of warm fuzzies and tell their boss they have exercised due diligence.

Design-oriented testing: Pro and Con

Design-oriented testing is when you walk into the room where the engineers who wrote the firewall sit, and start with the question: "Why do you think this firewall protects networks and itself effectively?" and go from there. Depending on the answers they give you, you then formulate a set of tests which propose to verify the properties they claim the firewall has. So, if I tell you my firewall works by testing the psychicintent in each packet, a test would be derived whereby we would send malicious packets at the firewall and see if they were blocked. Then we'd send the same packets without thinking nasty thoughts while we did it, and see if they went through. In other words, the test is a custom-tailored approach that matches the design of the system as we understand it. The problem with design-oriented testing is that it's hard. It takes skills that are not presently common - I only know 5 people that I would believe could do a good job of it. It's expensive, slow, and it's hard to explain. To even explain or understand a serious red team design review requires a pretty high level of expertise.

Who is qualified to test a firewall?

I've heard scary stories of people doing "firewall testing" of UNIX-based firewalls who do not understand UNIX. So, for example, they will tell you the firewall is insecure if the sendmail executable has not been deleted. So their checklist is maybe a little bit off. :) I've heard of other scary stories about people getting an auditor for a firewall and having a CNE appear. It's a networking problem, so who is better qualified than a Certified Network Engineer, right? If someone hired me to do a design-oriented test of a VMS firewall, that'd be pretty ridiculous, too - I'm a UNIX guru, and am completely unqualified to find a hole in a VMS product.

Certificate or Rubber stamp?

The market is ripe right now for someone to come along and start certifying firewalls. NSA will probably do it for their customer base, which is government only. As such they will slant their "What is good" requirements to meet their political/technological agenda: NSA approved crypto only, and Fortezza. The question is: If someone starts certifying firewalls, will the certification have any intellectual integrity? Will the certificate just be a sticker that the vendor paid for, or will it mean something?

I recently rather derisively dismissed an RFI from a large consulting company that wants to hire "firewall test consultants" and which asked for a detailed writeup of the methodology used. (My response was a description of design-oriented testing) From the layout of the RFI it was pretty clear that they were building a laundry list and were canvassing other consultants to help fill out their own laundry list. Being certified as "secure" on those terms should not make anyone sleep better at night. Big laundry lists are better than small laundry lists but if you were to look at the set of facts that SATAN1.0 tested for, there are at least 4 new things since it's release that have been discovered. If SATAN1.0 were your sole firewall test "methodology" you could be wide open to attack, right now.

Who do you trust?

The testers and certification authorities should be the top experts in the field for the particular type of firewall you are talking about. That means that if it's a VMS based firewall, it'd better be a VMS guru, not smb, ches, or mjr. If it's a router, then it should be someone who really knows routers. The degree to which you can trust a certification depends a lot on the credentials of the certifiers, how much they have at stake in the process, and how well their test methodology has survived peer review.

What about liability?

Don't expect to see vendors or testers assuming liability for security problems with firewalls. Firewalls are, by their nature, easy to configure and reconfigure. They are also easy to bypass. For someone to assume liability for a firewall's installation, they'd have to have some kind of guarantee that the customer would not somehow alter, weaken, or bypass it. Another problem with assigning liability is that the provenance of the attack would have to be clearly identified. In many break-ins, it's almost impossible to tell how the attacker initially got into the network - no firewall vendor or testing authority is going to be very happy about having to defend their firewall against hordes of lawyers, when a customer's network gets broken into via a modem pool. But if there's no liability, what's the point of a certificate? The (intended to be humorous)"Marcus J. Ranum Certified firewall" logo at the top of this document is worth no more than the paper it's printed on; and it's not even printed on paper. Print it out and tape it to your firewall, then send me a check for $150: maybe you'll sleep better at night with it there. If you don't want to send $150, send $10, but it'll only be 1/15th as secure.

Vesaria, LLC

Ranum On Firewall Testing
Table of Contents

Previous Section: Should we have an authority that "certifies" firewalls?

Next Section: Let's imagine a firewall test programme

Find out more about VESARiA Firewall Testing.

© 2000 - 2017 Vesaria Network Security Specialists        
   About Vesaria   |   Legal   |   Privacy   |   Contact