VESARiA Network Security Specialists
About Vesaria Services Consulting Partners Research Customers Contact Us

Firewall Testing



Let's imagine a firewall test programme

Attacking the firewall host

First, let's pretend we're going to attack the firewall host. So we grab our copy of SATAN++, give it the IP address of the firewall, and tell it to attack. In some cases, the results we get back will indicate something valuable. But, let's pretend that the firewall we're testing is a firewall such as a SunScreen or a bridging firewall, and it has no IP address! Suddenly, one of our most important testing tools and methods has become meaningless. Does that mean the firewall is completely secure against attack? In some cases, it may, but in other cases it may be that the firewall is accessible via some other control means that our copy of SATAN++ does not know about.

Another scary scenario in the SATAN++ testing approach is a firewall that is actually not a firewall at all, but which simply passes SATAN++ tests by luck or design. For example, one thing SATAN checks for is the version of Sendmail a system is running, if it is running Sendmail. Let's suppose that our fictional firewall is running Sendmail V3.0 (an ancient version) with no bug fixes applied. However, when the firewall's mailer was configured, it was configured to not display the version prompt in the SMTP welcome banner. So SATAN++ decides it's a new version of the mailer and does not complain. I can imagine developing a "firewall" that is a complete leaking sieve, but which passes SATAN with no warning. Presently, however, one of the questions most often asked of firewall vendors is: "does it pass SATAN?" There's a lot of ignorance out there for us to overcome.

Attacking hosts behind the firewall

Let's pretend that the next phase of our checklist testing procedure is that we run SATAN++ against all the systems behind the firewall. The rationale for this is that if the firewall is a screening router-type firewall, we'll get an idea about what it screens or permits, and if the firewall is a dual-homed-block-everything-type firewall, we'll verify that it does, indeed, block everything. The problems with this thinking are serious. If the firewall is a screening router-type firewall, we may simply wind up measuring what version of Sendmail is running on the test-bed network, and not the customer network. Also, we may not probe a system that happens to be down right now, which is otherwise wide open to attack.

With a screening router-type firewall, the deployed configuration is critical to its configuration. Suppose I submit a firewall for testing that consists of a screening router with a default set of rules that block all traffic except outgoing WWW? It's going to register on the tests as impregnable, but just about any customer who installs it is going to have to weaken its default rules considerably in order to get any use out of it. Is the test I performed still valid? What about a firewall that ships with several "standard operating policies" of which the default (how the firewall is tested) is very restrictive and conservative, and the remainder of which are weak?

In the firewall product summaries effort, I have tried to encourage the community to recognize that firewalls embody two protective relationships:

  • How the firewall protects itself from attack
  • How the firewall protects attached networks from attack

Testing a firewall needs to take this into consideration, as well! To meaningfully test the firewall, we need to consider the strengths and weaknesses of the systems behind it. Doing this in a lab is impossible, since very few real world sites will be exactly like the test lab.

Attacking firewall protocols

Last but not least, many firewalls embody their own protocols, which the firewall may or may not rely on for its security. An example would be the authentication server (authsrv) from the firewall toolkit. Suppose there is a hole in the protocol or its implementation? (There are none that I know of; this is just an example) An automated testing suite will not uncover that fact. One firewall vendor that shall remain nameless used to sell a firewall that they remotely managed for customers that needed direct support. When remote management was necessary, the vendor telnetted into the firewall, over the Internet, using a clear-text root password. What is the point of testing a firewall for elaborate holes, when there is a gaping weakness in its managment procedures? From a testing standpoint, such weaknesses will not be visible. They only become visible if a detailed and very intensive penetration test is attempted.

A picture of disaster

The preceeding examples paint a deliberately bleak picture. But it's important to look at the ways in which a test may accidentally become meaningless, in order to decide if it's worth doing, and how much faith to place in it. Automated testing in a lab is not going to provide adequate coverage, and firewalls that are highly adaptable and user-customizable will be practically impossible to test, except in a very general way.

Vesaria, LLC

Ranum On Firewall Testing
Table of Contents

Previous Section: What would "certification" mean in a firewall?

Next Section: How I test a firewall

Find out more about VESARiA Firewall Testing.

© 2000 - 2017 Vesaria Network Security Specialists        
   About Vesaria   |   Legal   |   Privacy   |   Contact