Article:SOA Testing using Black, White and Gray Box Techniques


Category: Web Services based SOA testing

Introduction:

Web services are the foundations of modern Service Oriented Architecture (SOA). Typical Web Services include message exchange between a consumer and a producer using SOAP request and responses over the ubiquitous HTTP protocol. A Web service producer advertises its services to potential consumers through Web Services Description Language (WSDL) – an XML file that contains details of available operations, execution endpoints and expected SOAP request-response structures.

Many testing techniques and methodologies developed over the years apply to Web services-based SOA systems as well. Through functional, regression, unit, integration, system and process level testing, the primary objective of testing methodologies is to increase confidence that the target system will deliver functionality in a robust, scalable, interoperable and secure manner.

Techniques such as Black, White and Gray Box testing applied to traditional systems map well into Web Services deployments. However, such deployments introduce unique testing challenges since Web Services:

 Are intrinsically distributed and are platform and language agnostic.

 Can be chained with dependencies on other 3rd party Web services that can change without notice.

 Share Ownership across various stakeholders.

 Client developers typically only have access to interfaces (WSDLs) and lack access to code.

In this paper, we investigate testing techniques and their application to Web Services. We will explore of the relative strengths and weaknesses of such techniques. Finally, a novel approach that extends Gray Box’s reach into realm of White Box testing by leveraging the rich information provided in the WSDL file is described.

Black Box Testing

Definition: Black Box testing refers to the technique of testing a system with no knowledge of the internals of the system. Black Box testers do not have access to the source code and are oblivious of the system architecture. A Black Box tester typically interacts with a system through a user interface by providing inputs and examining outputs without knowing where and how the inputs were operated upon. In Black Box testing, target software is exercised over a range of inputs and the outputs are observed for correctness.

Advantages

 Efficient Testing – Well suited and efficient for large code segments or units.

 Unbiased Testing – clearly separates user’s perspective from developer’s perspective through separation of QA and Development responsibilities.

 Non intrusive – code access not required.

 Easy to execute – can be scaled to large number of moderately skilled testers with no knowledge of implementation, programming language, operating systems or networks.

Disadvantages

 Localized Testing – Limited code path coverage since only a limited number of test inputs are actually tested.

 Inefficient Test Authoring – without implementation information, exhaustive input coverage would take forever and would require tremendous resources.

 Blind Coverage – cannot control targeting code segments or paths which may be more error prone than others.


Black Box testing is best suited for rapid test scenario testing and quick Web Service prototyping. This testing technique for Web Services provides quick feedback on the functional readiness of operations through quick spot checking.

White Box Testing

Definition: White Box testing refers to the technique of testing a system with knowledge of the internals of the system. White Box testers have access to the source code and are aware of the system architecture. A White Box tester typically analyzes source code, derives test cases from knowledge about the source code, and finally targets specific code paths to achieve a certain level of code coverage. A White Box tester with access to details about both operations can readily craft efficient test cases that exercise boundary conditions.

Advantages

 Increased Effectiveness – Crosschecking design decisions and assumptions against source code may outline a robust design, but the implementation may not align with the design intent.

 Full Code Pathway Capable – all the possible code pathways can be tested including error handling, resource dependencies, and additional internal code logic/flow.

 Early Defect Identification – Analyzing source code and developing tests based on the implementation details enables testers to find programming errors quickly.

 Reveal Hidden Code Flaws – access to source code improves understanding and uncovering unintended hidden behavior of program modules.

Disadvantages

 Difficult To Scale – requires intimate knowledge of target system, testing tools and coding languages, and modeling. It suffers for scalability of skilled and expert testers.

 Difficult to Maintain – requires specialized tools such as source code analyzers, debuggers, and fault injectors.

 Cultural Stress – the demarcation between developer and testers starts to blur which may become a cultural stress.

 Highly intrusive – requires code modification has been done using interactive debuggers, or by actually changing the source code. This may be adequate for small programs; however, it does not scale well to larger applications. Not useful for networked or distributed systems.

White Box testing is most suited for Web Services early in the development cycle where the developer and the tester may collaborate to identify defects. White Box testing is problematic for large SOA deployments where the distributed nature of services makes it easy for 3rd party web services to be invoked from within other web services. This results in the lack of knowledge of programming language, operating systems and hardware platforms.

Gray Box Testing

Definition: Gray Box testing refers to the technique of testing a system with limited knowledge of the internals of the system. Gray Box testers have access to detailed design documents with information beyond requirement documents. Gray Box tests are generated based on information such as state-based models or architecture diagrams of the target system.

Advantages

 Offers Combined Benefits – Leverage strengths of both Black Box and White Box testing wherever possible.

 Non Intrusive – Gray Box does not rely on access to source code or binaries. Instead, based on interface definition, functional specifications, and application architecture.

 Intelligent Test Authoring – Based on the limited information available, a Gray Box tester can author intelligent test scenarios, especially around data type handling, communication protocols and exception handling.

 Unbiased Testing – The demarcation between testers and developer is still maintained. The handoff is only around interface definitions and documentation without access to source code or binaries.

Disadvantages

 Partial Code Coverage – Since the source code or binaries are not available, the ability to traverse code paths is still limited by the tests deduced through available information. The coverage depends on the tester authoring skills.

 Defect Identification – Inherent to distributed application is the difficulty associated in defect identification. Gray Box testing is still at the mercy of how well systems throw exceptions and how well are these exceptions propagated with a distributed Web Services environment.

The inherent distributed nature of web services and lack of source code or program binaries access makes White Box testing impossible within a SOA. With WSDLs as the de facto contract between consumers and producers in a web services-based SOA, significant information is available to construct intelligent and efficient gray Box tests. WSDLs provide rich information to construct and automate such tests to improve web services deployments.

Summary & Recommendations

Web Services-based SOA plays an important role in facilitating the integration of disparate applications from various departments or trading partners and thus increasing business productivity. The distributed nature of web services makes Gray Box testing ideal for detecting defects within a SOA. Black Box testing provides rapid functional testing that can be used across distributed services; however, owing to the “blind” nature of Black Box testing, test coverage is limited, inefficient and redundant. White Box testing is not practical for web services since access to source code or binaries in a web services deployment is usually impossible. By leveraging the rich information presented in WSDL files, intelligent and efficient Gray Box test can be generated. Further state-of-the-art techniques such as message mutation can be used to auto-generate a large array of test that can extract program internals – exception handling, states, flows – without having access to source or binaries. Such techniques push the Gray Box testing closer to the results of White Box testing without dealing with its expense or intrusive characteristics.

About the Authors:

Rizwan Mallal is the Managing Director at Crosscheck Networks. Also, as the founding member and Chief Security Architect of Forum Systems, Rizwan is responsible for all security related aspects of Forum’s technology. Previously, Rizwan was the Director of Engineering at Sonicwall (SNWL). He joined Sonicwall through the Phobos acquisition, where he was the Chief Architect of the SSL product line. Before joining Phobos, he was the technical architect at Raptor Systems where he was one of the pioneers of VPN/Firewall space in the mid 1990s. Raptor after its successful IPO in 1996 was later acquired by Axent/Symantec (SYMC).

Mamoon Yunus is an industry-honored CTO and visionary in Web Services-based SOA technologies. As the founder of Forum Systems, Mamoon pioneered Web Services Security Gateways & Firewalls. He has spearheaded Forum's direction and strategy for six generations of award-winning Web Services Security products. Prior to Forum Systems, Mr. Yunus was a Global Systems Engineer for webMethods (NASD: WEBM) where he developed XML-based business integration and architecture plans for Global 2000 companies. He has held various high-level executive positions at Informix (acquired by IBM) and Cambridge Technology Group. InfoWorld recognized Mamoon as one of 4 "Up and coming CTOs to watch in 2004." He is a sought after speaker at industry conferences such as RSA, Gartner, Web Services Edge, CSI, Network Interop, and Microsoft TechEd. Mamoon has the distinction of showcasing Forum Systems' entrepreneurial leadership as a case study at the MIT Sloan School of Management. He has also been featured on CNBC as Terry Bradshaw's "Pick of the Week.” Mamoon holds two Graduate Degrees in Engineering from MIT.

Most Recent

Most Popular

Most Active Categories


Back To Top Add New Article Printable Page
MediaWiki

This page has been accessed 3,250 times.

This page was last modified 20:11, 30 August 2006.