CS302: Software Engineering

Developing a Test Strategy 

There are no rules for developing a test strategy. Rather, loose heuristics are provided. Testing, like everything else in software engineering, is a skill that comes with practice. Good testers are among the most highly skilled workers on a development team. A career can revolve around testing because skilled testers are in short supply.

As with all other testing projects, the strategy should be designed to prove the application works and that it is stable in its operational environment. While scheduling and time allotted are not most important, when the strategy is devised, one subgoal is to minimize the amount of time and resources (both human and computer) that are devoted to testing.

The first decision is whether and what to test top-down and bottom-up. There are no rules, or even heuristics, for this decision. Commitment to top-down testing is as much cultural and philosophical as it is technical. To provide some heuristics, in general, the more critical, the larger, and the more complex an application, the more top-down benefits outweigh bottom-up benefits.

TABLE 17-2 Test Level and Test Strategy

The heuristics of testing are dependent on the language, timing, and operational environment of the application. Significantly different testing strategies are needed for third (e.g., COBOL, Pl/1), fourth (e.g., Focus, SQL), and semantic (e.g., Lisp, PROLOG) languages or current languages like Python, Java, or C++. Application timing is either batch, online, or real-time. The operational environment includes hardware, software, and other co-resident applications. Heuristics for each of these are summarized in Table 17-3.

Package testing differs significantly from self-developed code. More often, when you purchase package software, you are not given the source code or the specifications. You are given user documentation and an executable code. By definition, you have to treat the software as a black box. Further, top-down testing does not make sense because you are presented with a complete, supposedly working application. Testing should be at the system level only, including functional, volume, intermodular communications, and data-related black-box tests. Next, we consider the ABC test strategy.

ABC Video Test Strategy 

The ABC application will be developed using some SQL-based language. SQL is a fourth-generation language which simplifies the testing process and suggests certain testing strategies. The design from Chapter 10, Data-Oriented Design, is used as the basis for testing, although the arguments are the same for the other methodologies. 

First, we need to decide the major questions: Who? What? Where? When? How?

TABLE 17-3 Test Strategy Design Heuristics

Who? The test coordinator should be a member of the team. Assume it is yourself. Put yourself into this role and think about the remaining questions and how you would answer them if you were testing this application. 

What? All application functions, constraints, user acceptance criteria, human interface, peak performance, recoverability, and other possible tests must be performed to exercise the system and prove its functioning. 

Where? The ABC application should be tested in its operational environment to also test the environment. This means that all hardware and software of the operational environment should be installed and tested. If Vic, or the responsible project team member, has not yet installed and tested the equipment, they are now delaying the conduct of application testing. 

When? Since a 4GL is being used, we can begin testing as soon as code is ready. An iterative, top-down approach will be used. This approach allows Vic and his staff early access to familiarize themselves with the application. Testing at the system level needs to include the scaffold code to support top-down testing. The schedule for module coding should identify and schedule all critical modules for early coding. The tasks identified so far are:

1. Build scaffold code and test it. 

2. Identify critical modules. 

3. Schedule coding of critical modules first. 

4. Test and validate modules as developed using the strategy developed.

How? Since a top-down strategy is being used, we should identify critical modules first. Since the application is completely on-line, the screen controls and navigation modules must be developed before anything else can be tested. Also, since the application is being developed specifically to perform rental/return processing, rental/return processing should be the second priority. Rental/return cannot be performed without a customer file and a video file, both of which try to access the respective create modules. Therefore, the creation modules for the two files have a high priority.

The priority definition of create and rental/return modules provides a prioritized list for development. The scaffolding should include the test screens, navigation, and stubs for all other processing. The last item, backup and recovery testing, can be parallel to the others. 

Next, we want to separate the activities into parallel equivalent chunks for testing. By having parallel testing streams, we can work through the system tests for each parallel stream simultaneously, speeding the testing process. For ABC, Customer Maintenance, Video Maintenance, Rental/Return, and Periodic processing can all be treated as stand-alone processes. Notice that in Information Engineering (IE), this independence of processes is at the activity level. If we were testing object-oriented design, we would look at processes from the Rooch diagram as the independent and parallel test units. If we were testing process design, we would use the structure charts to decide parallel sets of processes. 

Of the ABC processes, Rental/ Return is the most complex and is discussed in detail. Rental/Return assumes that all files are present, so the DBA must have files defined and populated with data before Rental/Return can be tested. Note that even though files must be present, it is neither important nor required that the file maintenance processes be present. For the two create processes that are called, program stubs that return only a new Customer ID, or Video ID IC opy ID, are sufficient for testing.

In addition to parallel streams of testing, we might also want to further divide Rental/Return into several streams of testing by level of complexity, by transaction type, or by equivalent processes to further subdivide the code generation and testing processes. We choose such a division so that the same person can write all of the code but testing can proceed without all variations completed. For example, we will divide Rental/Return by transaction type as we did in IE. The four transaction types are rentals with and without returns, and returns with and without rentals. This particular work breakdown allows us to test all major variations of all inputs and outputs, and allows us to proceed from simple to complex as well. In the next sections, we will discuss from bottom-up how testing at each level is designed and conducted using Rental/Return as the ABC example.

Next, we define the integration test strategy. The IE design resulted in small modules that are called for execution, some of which are used in more than one process. At the integration level, we define inputs and predict outputs of each module, using a black-box approach. Because SQL calls do not pass data, predicting SQL set output is more important than creating input. An important consideration with the number of modules is that intermodular errors that are created in one module but not evidenced until they are used in another module. The top-down approach should help focus attention on critical modules for this problem.

Because SQL is a declarative language, blackbox testing at the unit level is also appropriate. The SQL code that provides the control structure is logic and becomes an important test item. White-box tests are most appropriate to testing the control logic. Therefore, a mix of black- and white-box testing will be done at the unit level. 

To summarize, the top-down strategy for testing the application includes:

1. Test screen design and navigation, including validation of security and access controls. 

2. Test the call structure for all modules. 

3. Test rental/return processing. 

4. Test create processing for customers and videos. 

5. Test remaining individual processes and file contents as parallel streams. 

6. Test multiple processes and file manipulations together, including validation of response time and peak system performance. The test will use many users doing the same and different processes, simultaneously. 

7. Test backup and recovery strategies.

Now, we develop and try a unit test to test the strategy. If a small test of the strategy works, we implement the strategy.

Source: Adapted from Sue Conger, https://resources.saylor.org/CS/CS302/OER/The_New_Software_Engineering.pdf
This work is licensed under a Creative Commons Attribution 3.0 License.