Current Status on testing efforts
|Routing Protocol||Compliance||Performance/Scale/additional Feature tests||Resilience|
|BGPv4||tested||test plans written (comments welcome)||tested|
|BGPv6||tested||test plans written (comments welcome)||tested|
|ISIS||tested||test plans written (comments welcome)||tested|
|OSPFv2||tested||test plans written (comments welcome)||tested|
|OSPFv3||tested||test plans written (comments welcome)||tested|
|RIP||tested||test plans in progress||tested|
|RIPnG||tested||test plans in progress||tested|
See individual sections compliance, performance/scale/additional feature tests, and resilience for details.
Not all the tests have been analyzed but results are being put into the appropriate sections regularly.
Testing Efforts Overview
OSR will test Quagga across various areas (listed below). OSR will do its best to publish plans and results for each of these areas. This will enable the community to review, contribute, and use the information as needed.
The general testing support OSR provides to quagga.net is described below.
As code arrives (bug fixes, new features, patches, etc) arrives into the quagga.net community, OSR will pick up these components and
a) review the code (holding an open code review) based on GNU standards. This will include review of not only style and standard implementation methodologies, but also review for comments, and for applicability of the code. Applicability of the code refers to - does it make sense from a perspective of compliance or useability.
b) As with BIND, OSR will ask that most code follow TDD (Test driven development) methodology. It helps OSR unit test specific patches, bug fixes etc. Otherwise OSR will try and integrate into the appropriate process and unit test the feature according to the fixes. Bugs will regularly placed into Quagga.net bugzilla
c) Once unit testing is accomplished then OSR will run the following list of testing areas:
- Protocol conformance - OSR uses a commercial tool for protocol compliance testing. In particular is runs against OSPF, BGP, ISIS and RIP. These tests are run whenever major features, patches or a merge occurs.
- Interoperability - OSR will ensure that quagga will work with existing vendor routers. OSR has Cisco and Juniper routers to help verify interoperability. Most interoperability is covered via conformance, but there are specific differences in implementations with Cisco and Juniper where they do not follow compliance. Hence OSR will run interop only where exceptions are reported.
- Resiliency testing - OSR uses a commercial tool for resiliency. As with conformance it runs against all the protocols performing negative testing. "Fuzz" testing.
- Vulnerability testing - OSR will develop a set of tests checking the vulnerability of Quagga. These will be published only for members and with CERT-IF on an as needed basis.
- Scale and performance - Using a strong set of traffic sources, from hardware testers, to multiple ISPs BGP feeds, to software generation, to replay and manipulation tools. We are running through these tests for all protocols. Performance metrics will be varied and continuously modified based on community interest
- Functional tests - there are specific tests that are not covered in conformance, interoperability or scale tests. These relate to potential "real world" scenarios and usage of the routing protocols.
- Field trials - OSR plans to gather a number of groups in diverse situations who are willing to introduce new turns of quagga into places in their network. In anything other than a lab, this will need to be part of a release testing process, as there needs to be reasonable assurance that the software is well tested and expected to work.
d) Release to the community - code review and testing is complete, we will release out results to the community. We may have even merged the code streams together if needed. It will be up to the community to integrate this into the mainline.
As of now, all existing code bugs found or new features tested will be added into quagga.net’s bugzilla. If it is decided that some other process would be better, we will work with the community to develop what makes the most sense for the community
One of our focuses is to make the testing understood and available to the community. This will include test plans, test cases, guidelines, support tools and anything else we can share. Community feedback on this makes the overall testing stronger. This is and will always be a work in progress, new features and new bugs create new tests and regression cases. We hope this will make it easy for others to use our tests and increase the overall testing of quagga.