Network World Clear Choice Test: IPS Performance

Dear PR Person:

(if you're not a PR person, please forward this to your marketing department)

We are pleased to invite your company or client to participate in an evaluation of intrusion prevention system performance, with results scheduled for publication in Network World. Results are tentatively planned for publication in late spring.

Given the large number of IPS devices claiming multi-gigabit performance, we expect the results of this test to be of great interest to Network World’s readers.

As product reviews editor, I've commissioned David Newman of Network Test (dnewman@networktest.com) and Joel Snyder of Opus One (jms@opus1.com) to conduct this comparative review.

Please read on for information about deadlines and timeframes. I know that this

message is long, but please read at least up to the FAQs and I trust that most of your

questions will be answered.

EXECUTIVE SUMMARY

What: IPS Clear Choice comparative test

Who: Network World

DEADLINES

Expression of Interest: by 20 February 2006

Product to lab: by 27 February 2006

Contacts: David Newman dnewman@networktest.com, Joel Snyder (jms@opus1.com)

Shipping Address: Network Test, 31324 Via Colinas, Suite 113, Westlake Village, CA, 91362, +1-818-889-0011

OVERVIEW

A primary goal of this test is to identify a field of intrusion prevention systems suitable for deployment in high-performance enterprise networking environments. In addition to assessing device performance – both with and without attack traffic present-- we also will evaluate devices in terms of how completely and correctly they act in response to various forms of attack.

As in all Network World tests, we will be combining objective results, such as performance testing results, with subjective evaluation of the features, manageability, and usability of the products.

Our primary goal is to offer our readers an intelligent discussion of the features that differentiate products, and to point out critical issues in evaluation of this product category. We also want to share the results of our testing with readers, so that they can shorten their own buying cycle by not having to start from scratch when evaluating products. And, because our readers demand it, we will have, in addition to the 5,000 or so words of discussion about products and results, a scorecard.

METHODOLOGY

Our evaluation criteria are available in a separate document at the following URL:

http://networktest.com/ips06/ips06meth.html

Products submitted for this test will be evaluated using criteria detailed in that document.

This invitation is available at the following URL:

http://networktest.com/ips06/ips06invite.html

Because we expect to be testing performance, products in the test should be matched for size and expected performance. Vendors should submit devices that have at least three gigabit Ethernet interfaces (two for monitoring traffic and one for management). Performance numbers will be reported with pricing, so there is an advantage to submitting a product that is suited to this configuration.

We’re happy to test products with more interfaces as well. Vendors who wish to submit devices with more interfaces should contact the authors to discuss this.

ACTION ITEMS FOR YOU

If you would like to participate in this test, we will need the following from you:

0) An expression of interest. Please let me (Christine Burns cburns@nww.com, the Network World editor working on this project) know if you are interested in participating at the earliest opportunity before Feb. 20. This is not a commitment on your part, but if we have more submissions than available testing slots, early responses will be given priority.

1) Product. You should have a copy of your product delivered to our labs on or before 27 February 2006. (If you already have a copy in our lab, please check to make sure that we have the latest version of your software).

2) Name, version, and pricing. Vendors often forget to tell us what the product is “officially” called, what version we are testing and how much it costs. Please provide prices as tested. If you have additional hardware or software options, please provide pricing and nomenclature for these options as well. Vendors must supply this information before testing begins.

3) Contact information for you. We’ll need both engineering and marketing points of contact. Please supply both voice numbers and email addresses (Note: This information will not be published.)

4) Screen shots/product shots. Don't send those unless we specifically ask for them. Network World does not run these very often, so it's better not to waste time and effort mailing them around.

DATES

Our target date to being testing is approximately 27 February (or sooner). Your product should be in our hands by then. If this closely coincides with a software release and you need a week of leeway, please contact the authors. Because we can’t test everything at the same instant, it is generally possible to accommodate you. If you want any hardware back, send along appropriate shipping instructions with airbill numbers. Anything without this information will be treated as a loan (not a gift) to our labs. Standard operating practice in Network World tests is that participating vendors are responsible for all shipping costs.

The publication date is not yet set, but we are aiming for late spring 2006.

INSTALLATION

No, we do not want you to send someone out to install the software/hardware, unless you always do that for all your customers. However, we would like to be able to call technical support for questions so that we appear as an ordinary customer. We would prefer to not call your PR or Marketing people to get technical support.

CONTACTS

David and Joel will be doing all the testing and writing at their labs in Westlake Village, CA and Tucson, AZ, respectively, under the close supervision of Christine Burns, Network World’s executive reviews editor. You can contact Christine at cburns@nww.com. David’s email is dnewman@networktest.com and Joel’s email is jms@opus1.com If you actually want to talk to any of us, please send email.

SHIPPING ADDRESS

The shipping address for everything is:

David Newman

Network Test Inc.

31324 Via Colinas, Suite 113

Westlake Village, CA 91362

+1 818-889-0011

FAQs

After writing a lot of testing based articles on this and other topics, we thought we'd save some time and share some common questions and answers with you.

Q1: Our software will require you to call some support line that is only open from 9AM to noon, Eastern Standard Time, to get a serial number to unlock it to make it work. Is this OK?

A1: No. Because of the nature of testing, we do a lot of it at odd hours and on weekends. If we get your product out of the box and suddenly discover that it won't work without some magic key that takes 12 to 48 hours to get, this can throw things off. Please make sure that anything needed to make your product work is in the box you send us.

Q2: We think that you're incompetent and want to send one of our engineers to configure the software so that you can understand its cosmic wonderfulness and harmonic goodness. Is this OK?

A2: Only if you absolutely insist, or if you always do that for all of your customers.Have your engineer send me email and we'll work out a mutually acceptable day. Our take is that if your software does require an engineer to install it, or takes more than a few hours to install, that you've done a poor job at figuring out how to productize and document your product.

Q3: Understood, but we do send engineers to installation sites and we definitely want to do so here. Can we also send a couple of marketing managers to show you PowerPoint slides on our philosophy of networking, how we fit in magic quadrant boxes, etc.?

Q3: No, we’ll be too busy setting up and testing for that.

Q4: I have this marketing manager who wants to chat you up for a couple of hours on his car phone while driving home. This will make you like us better and be less likely to tell people our product sucks. Plus, it’s a good way to pass the time while stuck in traffic. Can I have him call you?

A4: Please don't. The deadlines on this test are very tight and such conversations, while often interesting, generally are a waste of time.

Q5: We have a new version of the product coming out and want to send you beta stuff. Can you ignore any crashing-and-burning behavior?

A5: No. We will test what you send us as if it is a released and supported product. We will be happy to test late betas if the product will be released by the time the test is printed.

Q6: Our documentation is online! All you have to do is click on ...

A6: No. If you can't be bothered to print it, we can't be bothered to read it. Also, be aware that anything that isn’t in the documentation will be assumed not to exist. Thus, if you have a way to handle Russian characters that requires we do something obscure, that’s OK---as long as you document it. If you don’t put it in the printed documentation, we will not go looking for it. In other words, we expect you to document your product in the documentation, not in a search engine.

Q7: I don't want to tell you how much it costs. Is this OK?

A7: Sure, so long as you don't mind us telling people it costs $999,999 per port.

Q8: We only want to be tested if we get top billing. Can we call you and have 9 people crowded around a speakerphone quiz you for a couple of hours about your methodology so we can predict whether or not we're going to win before we send the stuff?

A8: Yes and no. Your feedback on proper testing methodology is always welcome and any ideas you have on how to fairly compare products with disparate design goals is very useful. Please send us these ideas via email.

Also, the subjective test methodology may change as we look at products. In most cases, products are tested 3 times: once with a first pass, once again after we learn what the bugs are in our testing, and a third time just to make sure that we have all the facts right. We don't believe we can design a perfect test before ever looking at the products, and we don't believe that it's in your best interest for us to lock down the test before seeing the nifty new features which you've added since we last looked at them. We’ll keep you abreast of any changes as we proceed.

Q9: Should I send you <x>? (<x> is usually a white paper, competitive review, or

explanation of why everyone else's product loses big)

A9: Yes.

Q10: Will you tell us who else is in the test?

A10: The usual suspects, including all your major competitors.

Q11: I cannot possibly make your deadlines. Can you completely rearrange your print schedule because some dweeb in our company lost the invitation and didn't get it to the right person (or: we didn't think it was important so I didn't read this until too late)?

A11: No. Someone decided months ago to do this test now and not six months from now. We’re not allowed to revisit that decision. We understand that this can mean that significant lacunae will be present in the product lineup, but that's the way the presses run. If you want to avoid this kind of problem in the future, you can always call your top-10 trade magazines each October and ask them to send you their editorial calendars for next year. They will be happy to do this and you'll be prepared for what's coming up. We hear you can also find this on the web; if you don't know what the web is, call 1-900-GET-AOL and the nice man will explain it to you.

Q12: Can I call you every few days to see how things are going?

A12: Yes. In fact, this is not a supremely bad idea. However, you should not be insulted if no one takes your calls or calls you back. Generally, this means that things are going fine. Your success rate for status reports will increase approximately 1000% if you use email instead of the phone.

Q13: Can you be bribed?

A13: Unfortunately, no. You are welcome, however, to put us on your Christmas gift list for chocolate chip cookies.

Q14: If I leave my <hardware>/<software> in your lab after the test, will that positively affect the outcome? (alternatively: if I insist that you return the <hardware>/<software>, will that negatively affect the outcome?)

A14: No to both.

Q15: Our product doesn't fit in the test, but we'd love to have you write a sidebar just about us. Should we call you a lot to discuss this idea?

A15: No. If you have ideas on other interesting sidebars, call the editor and talk her into it. Hint: she probably won't call you back.

Q16: We think that we’re not going to participate because of <X>. Do you have a problem with that?

A16: No, of course not. However, this is Network World, the closest thing we have to a “journal of record” for product testing in network security, and it is important to us that our tests be seen as credible and complete. This may mean that we will test your product anyway, if its absence would be a significant gap.

Q17: I realize that your email address and shipping address are in this invitation, and they’re at the bottom of your email messages, and they’re on your web page, and they’ve been the same since 1999, but I still want to verify that I have the address right. Whom should I contact?

A17: Let me think about that and get back to you.

Q18: Can we see our results prior to publication?

A18: We will send you your performance results prior to publication and give you a chance to comment on them. We cannot share subjective comments about your product prior to publication, so asking isn’t worth the trouble. See next question.

Q19: Will you share other vendors’ results with us or tell us “how we did” prior to publication?

A19: No. Just as we don’t tell other vendors anything about your testing, we don’t tell you anything about them.

Q20: Can we see a draft of the article before publication?

A20: Are you kidding?