Test Plan for ABC Video Order Processing

Test Strategy

Developing a Test Strategy 

There are no rules for developing a test strategy. Rather, loose heuristics are provided. Testing, like everything else in software engineering, is a skill that comes with practice. Good testers are among the most highly skilled workers on a development team. A career can revolve around testing because skilled testers are in short supply.

As with all other testing projects, the strategy should be designed to prove the application works and that it is stable in its operational environment. While scheduling and time allotted are not most important, when the strategy is devised, one subgoal is to minimize the amount of time and resources (both human and computer) that are devoted to testing.

The first decision is whether and what to test top-down and bottom-up. There are no rules, or even heuristics, for this decision. Commitment to top-down testing is as much cultural and philosophical as it is technical. To provide some heuristics, in general, the more critical, the larger, and the more complex an application, the more top-down benefits outweigh bottom-up benefits.

TABLE 17-2 Test Level and Test Strategy

The heuristics of testing are dependent on the language, timing, and operational environment of the application. Significantly different testing strategies are needed for third (e.g., COBOL, Pl/1), fourth (e.g., Focus, SQL), and semantic (e.g., Lisp, PROLOG) languages or current languages like Python, Java, or C++. Application timing is either batch, online, or real-time. The operational environment includes hardware, software, and other co-resident applications. Heuristics for each of these are summarized in Table 17-3.

Package testing differs significantly from self-developed code. More often, when you purchase package software, you are not given the source code or the specifications. You are given user documentation and an executable code. By definition, you have to treat the software as a black box. Further, top-down testing does not make sense because you are presented with a complete, supposedly working application. Testing should be at the system level only, including functional, volume, intermodular communications, and data-related black-box tests. Next, we consider the ABC test strategy.

ABC Video Test Strategy 

The ABC application will be developed using some SQL-based language. SQL is a fourth-generation language which simplifies the testing process and suggests certain testing strategies. The design from Chapter 10, Data-Oriented Design, is used as the basis for testing, although the arguments are the same for the other methodologies. 

First, we need to decide the major questions: Who? What? Where? When? How?

TABLE 17-3 Test Strategy Design Heuristics

Who? The test coordinator should be a member of the team. Assume it is yourself. Put yourself into this role and think about the remaining questions and how you would answer them if you were testing this application. 

What? All application functions, constraints, user acceptance criteria, human interface, peak performance, recoverability, and other possible tests must be performed to exercise the system and prove its functioning. 

Where? The ABC application should be tested in its operational environment to also test the environment. This means that all hardware and software of the operational environment should be installed and tested. If Vic, or the responsible project team member, has not yet installed and tested the equipment, they are now delaying the conduct of application testing. 

When? Since a 4GL is being used, we can begin testing as soon as code is ready. An iterative, top-down approach will be used. This approach allows Vic and his staff early access to familiarize themselves with the application. Testing at the system level needs to include the scaffold code to support top-down testing. The schedule for module coding should identify and schedule all critical modules for early coding. The tasks identified so far are:

1. Build scaffold code and test it. 

2. Identify critical modules. 

3. Schedule coding of critical modules first. 

4. Test and validate modules as developed using the strategy developed.

How? Since a top-down strategy is being used, we should identify critical modules first. Since the application is completely on-line, the screen controls and navigation modules must be developed before anything else can be tested. Also, since the application is being developed specifically to perform rental/return processing, rental/return processing should be the second priority. Rental/return cannot be performed without a customer file and a video file, both of which try to access the respective create modules. Therefore, the creation modules for the two files have a high priority.

The priority definition of create and rental/return modules provides a prioritized list for development. The scaffolding should include the test screens, navigation, and stubs for all other processing. The last item, backup and recovery testing, can be parallel to the others. 

Next, we want to separate the activities into parallel equivalent chunks for testing. By having parallel testing streams, we can work through the system tests for each parallel stream simultaneously, speeding the testing process. For ABC, Customer Maintenance, Video Maintenance, Rental/Return, and Periodic processing can all be treated as stand-alone processes. Notice that in Information Engineering (IE), this independence of processes is at the activity level. If we were testing object-oriented design, we would look at processes from the Rooch diagram as the independent and parallel test units. If we were testing process design, we would use the structure charts to decide parallel sets of processes. 

Of the ABC processes, Rental/ Return is the most complex and is discussed in detail. Rental/Return assumes that all files are present, so the DBA must have files defined and populated with data before Rental/Return can be tested. Note that even though files must be present, it is neither important nor required that the file maintenance processes be present. For the two create processes that are called, program stubs that return only a new Customer ID, or Video ID IC opy ID, are sufficient for testing.

In addition to parallel streams of testing, we might also want to further divide Rental/Return into several streams of testing by level of complexity, by transaction type, or by equivalent processes to further subdivide the code generation and testing processes. We choose such a division so that the same person can write all of the code but testing can proceed without all variations completed. For example, we will divide Rental/Return by transaction type as we did in IE. The four transaction types are rentals with and without returns, and returns with and without rentals. This particular work breakdown allows us to test all major variations of all inputs and outputs, and allows us to proceed from simple to complex as well. In the next sections, we will discuss from bottom-up how testing at each level is designed and conducted using Rental/Return as the ABC example.

Next, we define the integration test strategy. The IE design resulted in small modules that are called for execution, some of which are used in more than one process. At the integration level, we define inputs and predict outputs of each module, using a black-box approach. Because SQL calls do not pass data, predicting SQL set output is more important than creating input. An important consideration with the number of modules is that intermodular errors that are created in one module but not evidenced until they are used in another module. The top-down approach should help focus attention on critical modules for this problem.

Because SQL is a declarative language, blackbox testing at the unit level is also appropriate. The SQL code that provides the control structure is logic and becomes an important test item. White-box tests are most appropriate to testing the control logic. Therefore, a mix of black- and white-box testing will be done at the unit level. 

To summarize, the top-down strategy for testing the application includes:

1. Test screen design and navigation, including validation of security and access controls. 

2. Test the call structure for all modules. 

3. Test rental/return processing. 

4. Test create processing for customers and videos. 

5. Test remaining individual processes and file contents as parallel streams. 

6. Test multiple processes and file manipulations together, including validation of response time and peak system performance. The test will use many users doing the same and different processes, simultaneously. 

7. Test backup and recovery strategies.

Now, we develop and try a unit test to test the strategy. If a small test of the strategy works, we implement the strategy.

Source: Adapted from Sue Conger, https://resources.saylor.org/CS/CS302/OER/The_New_Software_Engineering.pdf
This work is licensed under a Creative Commons Attribution 3.0 License.

Unit Testing

Guidelines for Developing a Unit Test 

Unit tests verify that a specific program, module, or routine (all referred to as 'module' in the remaining discussion) fulfills its requirements as stated in related program and design specifications. The two primary goals of unit testing are conformance to specifications and processing accuracy. 

For conformance, unit tests determine the extent to which processing logic satisfies the functions assigned to the module. The logical and operational requirements of each module are taken from the program specifications. Test cases are designed to verify that the module meets the requirements. The test is designed from the specification, not the code. 

Processing accuracy has three components: input, process, and output. First, each module must process all allowable types of input data in a stable, predictable, and accurate manner. Second, all possible errors should be found and treated according to the specifications. Third, all output should be consistent with results predicted from the specification. Outputs might include hard copy, terminal displays, electronic transmissions, or file contents; all are tested. 

There is no one strategy for unit testing. For input/output bound applications, black-box strategies are normally used. For process logic, either or both strategies can be used. In general, the more critical to the organization or the more damaging the possible errors, the more detailed and extensively white-box testing is used. For example, organizationally critical processes might be defined as any process that affects the financial books of the organization', meets legal requirements, or deals with client relationships. Examples of application damage might include life-threatening situations such as in nuclear power plant support systems, life support systems in hospitals, or test systems for car or plane parts.

Since most business applications combine approaches, an example combining black- and white-box strategies is described here. Using a white-box approach, each program specification is analyzed to identify the distinct logic paths which serve as the basis for unit test design. This analysis is simplified by the use of tables, lists, matrices, diagrams, or decision tables to document the logic paths of the program. Then, the logic paths most critical in performing the functions are selected for white-box testing. Next, to verify that all logic paths not white-box tested are functioning at an acceptable level of accuracy, black-box testing of input and output is designed. This is a common approach that we will apply to ABC Video. 

When top-down unit testing is used, control structure logic paths are tested first. When each path is successfully tested, combinations of paths may be tested in increasingly complex relationships until all possible processing combinations are satisfactorily tested. This process of simple-to-complex testing ensures that all logic paths in a module are performing both individually and collectively as intended.

Similarly, unit testing of multiuser applications also uses the simple-to-complex approach. Each program is tested first for single users. Then multiuser tests of the single functions follow. Finally, multiuser tests of multiple functions are performed.

Unit tests of relatively large, complex programs may be facilitated by reducing them to smaller, more manageable equivalent components such as 

• transaction type: e.g., Debit/Credit, Edit/Update/Report/Error 
• functional component activity e.g., Preparing, Sending, Receiving, Processing 
• decision option e.g., If true ... If false ...

When the general process of reduction is accomplished, both black-box and white-box approaches are applied to the process of actually defining test cases and their corresponding output. The black-box approach should provide both good and bad data inputs and examine the outputs for correctness of processing. In addition, at least one white-box strategy should be used to test specific critical logic of the tested item. 

Test cases should be both exhaustive and minimal. This means that test cases should test every condition or data domain possible but that no extra tests are necessary. For example, the most common errors in data inputs are for edit/validate criteria. Boundary conditions of fields should be tested. Using equivalence partitioning of the sets of allowable values for each field we develop the test for a date formatted YYYYMMDD (that is, 4-digit year, 2-digit month, and 2-digit day). A good year test will test last year, this year, next year, change of century, all zeros, and all nines. A good month test will test zeros, 1, 2, 4 (representative of months with 30 days), 12, and 13. Only 1 and 12 are required for the boundary month test, but the other months are required to test day boundaries. A good day test will test zeros, 1,28,29,30,31, and 32, depending on the final day of each month. Only one test for zero and one are required, based on the assumption that if one month processes correctly, all months will. Leap years and non-leap years should also be tested. An example of test cases for these date criteria is presented. Figure 17 -14 shows the equivalent sets of data for each domain. Table 17-4 lists exhaustive test cases for each set in the figure. Table 17-5 lists the reduced set after extra tests are removed.

FIGURE 17-14 Unit Test Equivalent Sets for a Date

TABLE 17-4 Exhaustive Set of Unit Test Cases for a Date

TABLE 17-5 Minimal Set of Unit Test Cases for a Date

Other frequently executed tests are for character, field, batch, and control field checks. Table 17-6 lists a sampling of errors found during unit tests. Character checks include tests for blanks, signs, length, and data types (e.g., numeric, alpha, or other). Field checks include sequence, reasonableness, consistency, range of values, or specific contents. Control fields are most common in batch applications and are used to verify that the file being used is the correct one and that all records have been processed. Usually the control field includes the last execution date and file name which are both checked for accuracy. Record counts are only necessary when not using a declarative language. 

Once all test cases are defined, tests are run and results are compared to the predictions. Any result that does not exactly match the prediction must be reconciled. The only possible choices are that the tested item is in error or the prediction is in error. If the tested item is in error, it is fixed and retested. Retests should follow the approach used in the first tests. If the prediction is in error, the prediction is researched and corrected so that specifications are accurate and documentation shows the correct predictions. 

Unit tests are conducted and reviewed by the author of the code item being tested, with final test results approved by the project test coordinator. 

How do you know when to stop unit testing? While there is no simple answer to this question, there are practical guidelines. When testing, each tester should keep track of the number of errors found (and resolved) in each test. The errors should be plotted by test shot to show the pattern. A typical module test curve is skewed left with a decreasing number of errors found in each test (see Figure 17-15). When the number of errors found approaches zero, or when the slope is negative and approaching zero, the module can be moved forward to the next level of testing. If the number of errors found stays constant or increases, you should seek help either in interpreting the specifications or in testing the program.

ABC Video Unit Test 

Above, we said we would use a combination of black- and white-box testing for ABC unit tests. The application is being implemented using a SQL software package. Therefore, all code is assumed to be in SQL. The control logic and nonSELECT code is subject to white-box tests, while the SELECT modules will be subject to black-box tests.

TABLE 17-6 Sample Unit Test Errors

In Chapter 10, we defined Rent/Return processing as an execute unit with many independent code units. Figure 17-16 shows partial SQL code from two Rent/Return modules. Notice that most of the code is defining data and establishing screen addressability. As soon as two or three modules that have such strikingly similar characteristics are built, the need to further consolidate the design to accommodate the implementation language should be obvious. With the current design, more code is spent in overhead tasks than in application tasks. Overhead code means that users will have long wait times while the system changes modules. The current design also means that debugging the individual modules would require considerable work to verify that the modules perform collectively as expected. Memory locations would need to be printed many times in such testing.

FIGURE 17-15 Unit Test Errors Found Over Test Shots

To restructure the code, we examine what all of the Rent/Return modules have in common-Open Rentals data. We can redefine the data in terms of Open Rentals with a single-user view used for all Rent/Return processing. This simplifies the data part of the processing but increases the vulnerability of the data to integrity problems. Problems might increase because the global view of data violates the principle of information hiding. The risk must be taken, however, to accommodate reasonable user response time. 

The common format of the restructured SQL code is shown in Figure 17-17. In the restructured version, data is defined once at the beginning of Rent/Return processing. The cursor name is declared once and the data is retrieved into memory based on the data entered through the Get Request module. The remaining Rent/Return modules are called in sequence. The modules have a similar structure for handling memory addressing. The problems with many prints of memory are reduced because once the data is brought into memory, no more retrievals are necessary until updates take place at the end of the transaction. Processing is simplified by unifying the application's view of the data.

          ADD RETURN DATE (Boldface code Is redundant)
                    DCL   INPUT_VIDEOID	     CHAR(8);
                    DCL	  INPUT COPY_ID	     CHAR(2);
                    DCL	  INPUT CUST_ID      CHAR (9);
                    DCL	  AMT                PAID	DECIMAL (4,2);
                    DCL	  CUST_ID	           CHAR(9);
                    ...
                       CONTINUE UNTIL ALL HELDS USED ON THE SCREEN OR USED TO
                    CONTROL SCREEN PROCESSING ARE DECLARED
                    DCL	  TOTAL_AMT_DUE	     D€CIMAL(5.2);
                    DCL	  CHANGE	           DECIMAL(4,2);
                    DCL	  MORE OPEN RENTALS	 BIT(1);
                    DCL	  MORE NEW_RENTALS	 BIT(1);
                    EXEC  SQL INCLUDE SQLCA: /"COMMUNICATION AREA"/".
                    EXEC  SQL DECLARE CUSTOMER TABLE
                          (FIELD DEFINITIONS FOR CUSTOMER RELATION);
                    EXEC SOL DECLARE VIDEO TABLE
                          (FIELD DERNmONS FOR VIDEO RELATION);
                    EXEC SQL DECLARE COPY TABLE
                          (FIELD DEFINITIONS FOR COPY RELATION);
                    EXEC SQL DECLARE OPENRENTAL TABLE
                          (FIELD DEFINITIONS FOR OPENRENTAL RELATION);
                    EXEC SQL DECLARE SCREEN_CURSOR CURSOR FOR
                          SELECT ’ FROM OPEN_RENTAL
                                   WHERE VIDEOID - ORVlDEOID
                                   AND COPYID = ORCOPYID;
                    EXEC SQL OPEN SCREEN_CURSOR
                    GOTOLABEL
                    EXEC SQL FETCH SCREEN CURSOR INTO TARGET
                          :CUSTID
                          :VIDEOID
                          :COPYID
                          :RENTALDATE
                    IF SQLCOOE - 100 GOTO GOTOEXIT;
                    EXEC SQL SET :RETURNOATE - TODAYS.DATE
                    WHERE CURRENT OF SCREEN_CURSOR;
                    EXEC SQL UPDATE OPEN RENTAL
                          SET ORRETURNDATE - TODAYS,DATE
                          WHERE CURRENT OF SCREEN_CURSOR;
                    GOTO GOTOLABEL.
                    GOTOEXIT;
                    EXEC SQL CLOSE SCREEN CURSOR;


      

FIGURE 17-16 Two Modules Sample Code

The restructuring now requires a change to the testing strategy for Rent/Return. A strictly top-down approach cannot work because the Rent/Return modules are no longer independent. Rather, a combined top-down and bottom-up approach is warranted. A sequential bottom-up approach is more effective for the functional Rent/Return processing. Top-down, black-box tests of the SELECT code are done before being embedded in the execute unit. Black-box testing for the SELECT is used because SQL controls all data input and output. Complete SELECT statements are the test unit.

        DCL  	INPUT_VIDEO_ID	    CHAR(8);
        DCL	  INPUT_COPY_ID	      CHAR(2);
        DCL	  INPUT_CUST_ID	      CHAR(9);
        DCL	  AMT_PAID	          DECIMAL (4.2);
        DCL	  CUST_ID	            CHAR(9);
        ...
        continue until all fields used on the screen or used to control screen processing are 
        declared...
        DCL	  TOTAL_AMT_DUE	      DECIMALS(5,2);
        DCL	  CHANGE	            DECIMAL(4,2);
        DCL	  MORE_OPEN RENTALS	  BIT(1);
        DCL	  MORE_NEW_RENTALS	  BIT(1);
        EXEC SQL INCLUDE SQLCA: /*COMMUNICATION AREA*/
        EXEC SQL DECLARE RENTRETURN TABLE
        (field definitions for user view including all fields from customer, video, copy,
        open rental, and customer history relations);
        EXEC SQL DECLARE SCREEN_CURSOR CURSOR FOR
                  SELECT * from rentretum
                  where (:videoid = orvideojd and xopyid = orcopyid) 
                  or xustid = orcustid)
        EXEC SQL OPEN SCREEN_CURSOR
        EXEC SQL FETCH SCREEN_CURSOR INTO TARGET
                 :Request
        If :request eq "C?" set custid = :request
        else      set :videoid = :request 1
                  set icopyid = :request;
                      
                  (At this point the memory contains the related relation data
                  and the remaining rent/return processing can be done.)
                      
        All the other modules are called and contain the following common format:
        GOTOLABEL
        EXEC SQL FETCH SCREEN.CURSOR INTO TARGET
        :screen fields
            
        IF SQLCODE = 0 next step; (return code of zero means no errors)
        IF SQLCODE - 100 (not found condition) CREATE DATA or CALL END PROCESS;
        IF SQLCODE < 0 CALL ERROR PROCESS, ERROR-TYPE;
        Set screen variables (which displays new data)
        Prompt next action
            
        GOTO GOTOLABEL;
        GOTOEXIT;
        EXEC SQL CLOSE SCREEN_CURSOR;
      

FIGURE 17-17 Restructured SQL Code-Common Format

FIGURE 17-18 Unit Test Strategy

The screen interaction and module logic can be tested as either white box or black box. At the unit level, white-box testing will be used to test intermodule control logic. A combination of white-box and black-box testing should be used to test intramodule control and process logic. 

The strategy for unit testing, then, is to test data retrievals first, to verify screen processing, including SQL cursor and data addressability second, and to sequentially test all remaining code last (see Figure 17-18). 

Because all processing in the ABC application is on-line, an interactive dialogue test script is developed. All file interactions predict data retrieved and written, as appropriate. The individual unit test scripts begin processing at the execute unit boundary. This means that menus are not necessarily tested. A test script has three columns of information developed. The first column shows the computer messages or prompts displayed on the screen. The second column shows data entered by the user. The third column shows comments or explanations of the interactions taking place.

A partial test script for Rent/Return processing is shown in Figure 17-19. The example shows the script for a return with a rental transaction. Notice that the test begins at the Rent/Return screen and that both error and correct data are entered for each field. After all errors are detected and dispatched properly, only correct data is required. This script shows one of the four types of transactions. It shows only one return and one rental, however, and should be expanded in another transaction to do several rentals and several returns; returns should include on-time and late videos and should not include all tapes checked out. This type of transaction represents the requisite variety to test returns with rentals. Of course, other test scripts for the other three types of transactions should also be developed. This is left as an extra-credit activity.

Subsystem or Integration Testing

Guidelines for Integration Testing 

The purpose of integration testing is to verify that groups of interacting modules that comprise an execute unit perform in a stable, predictable, and accurate manner that is consistent with all related program and systems design specifications. 

Integration tests are considered distinct from unit tests. That is, as unit tests are successful, integration testing for the tested units can begin. The two primary goals of integration testing are compatibility and intermodule processing accuracy.

FIGURE 17-19 ABC Video Unit Test Example-Rent/Return

Integration tests are considered distinct from unit tests. That is, as unit tests are successful, integration testing for the tested units can begin. The two primary goals of integration testing are compatibility and intermodule processing accuracy. 

Compatibility relates to calling modules in an operational environment. The test verifies first that all modules are called correctly, and, even with errors, do not cause abends. Intermodule tests check that data transfers between modules operate as intended within constraints of CPU time, memory, and response time. Data transfers tested include sorted and extracted data provided by utility programs, as well as data provided by other application modules.

Test cases developed for integration testing should be sufficiently exhaustive to test all possible interactions and may include a subset of unit test cases as well as special test cases used only in this test. The integration test does not test logic paths within the modules as the unit test does. Instead, it tests interactions between modules only. Thus, a black-box strategy works well in integration testing. 

If modules are called in a sequence, checking of inputs and outputs to each module simplifies the identification of computational and data transfer errors. Special care must be taken to identify the source of errors, not just the location of bad data. Frequently, in complex applications, errors may not be apparent until several modules have touched the data and the true source of problems can be difficult to locate. Representative integration test errors are listed in Table 17-7.

Integration testing can begin as soon as two or more modules are successfully unit tested. When to end integration tests is more subjective. When exceptions are detected, the results of all other test processing become suspect. Depending on the severity and criticality of the errors to overall process integrity, all previous levels of testing might be re-executed to re-verify processing. Changes in one module may cause tests of other modules to become invalid. Therefore, integration tests should be considered successful only when the entire group of modules in an execute unit are run individually and collectively without error. Integration test curves usually start low, increase and peak, then decrease (see Figure 17-20). If there is pressure to terminate integration testing before all errors are found, the rule of thumb is to continue testing until fewer errors are found on several successive test runs.

TABLE 1 7-7 Sample Integration Test Errors

ABC Video Integration Test 

Because of the redesign of execution units for more efficient SQL processing, integration testing can be concurrent with unit code and test work, and should integrate and test the unit functions as they are complete. The application control structure for screen processing and for calling modules is the focus of the test.

FIGURE 17-20 Integration Test Errors Found . Over Test Shots

Black-box, top-down testing is used for the integration test. Because SQL does not pass data as input, we predict the sets that SQL will generate during SELECT processing. The output sets are then passed to the control code and used for screen processing, both of which have been unit tested and should work. To verify the unit tests at the integration level, we should: 

1. Ensure that the screen control structure works and that execute units are invoked as intended. 

2. Ensure that screens contain expected data from SELECT processing. 

3. Ensure that files contain all updates and created records as expected. 

4. Ensure that printed output contains expected information in the correct format.

First, we want to define equivalent sets of processes and the sets' equivalent sets of data inputs. For instance, the high level processes from IE analysis constitute approximately equivalent sets. These were translated into modules during design and, with the exception of integrating data access and use across modules, have not changed. These processes include Rent/Return, Customer Maintenance, Video Maintenance, and Other processing. If the personnel are available, four people could be assigned to develop one script each for these equivalent sets of processing. Since we named Rent/Return as the highest priority for development, its test should be developed first. The others can follow in any order, although the start-up and shutdown scripts should be developed soon after Rent/Return to allow many tests of the entire interface. 

First, we test screen process control, then individual screens. Since security and access control are embedded in the screen access structure, this test should be white box and test every possible access path, including invalid ones. Each type of access rights and screen processing should be tested . For the individual screens, spelling, positioning, color, highlighting, message placement, consistency of design, and accuracy of information are all validated (see Figure 17-21).

The integration test example in Figure 17-22 is the script for testing the start-up procedure and security access control for the application. This script would be repeated for each valid and invalid user including the other clerks and accountant. The startup should only work for Vic, the temporary test account, and the chief clerk. The account numbers that work should not be documented in the test script. Rather, a note should refer the reader to the person responsible for maintaining passwords.

1. Define equivalent sets of processes and data inputs. 

2. Define the priorities of equivalent sets for testing. 

3. Develop test scrips for Rent/Return, Other processing, Customer Maintenance, Video Maintenance. 

4. For each of the above scripts, the testing will proceed as follows: 

FIGURE 17-21 ABC Integration Test Plan

FIGURE 17-22 ABC Video Integration Test Script

In the integration portion of the test, multiuser processing might take place, but it is not necessarily fully tested at this point. File contents are verified after each transaction is entered to ensure that file updates and additions are correct. If the integration test is approached as iteratively adding modules for testing, the final run-through of the test script should include all functions of the application, including start-up, shutdown, generation and printing of all reports, queries on all files, all file maintenance, and all transaction types. At least several days and one monthly cycle of processing should be simulated for ABC's test to ensure that end-oi-day and end-of-month processing work. 

Next, we discuss system testing and continue the example from ABC with a functional test that is equally appropriate at the integration, system, or QA levels.

System and Quality Assurance Testing

Guidelines for Developing System and Quality Assurance Tests 

The system test is used to demonstrate an application's ability to operate satisfactorily in a simulated production environment using its intended hardware and software configuration. The quality assurance test (QA) is both a system test and a documentation test. Both tests also verify that all of the system's interacting modules do the following:

1. Fulfill the user's functional requirements as contained in the business system design specifications and as translated into design requirements in the design spec and any documents controlling interfaces to other systems. 

2. The human interface works as intended. Screen design, navigation, and work interruptabilityare the test objects for human interface testing. All words on screens should be spelled properly. All screens should share a common format that is presented consistently throughout the application. This format includes the assignment of program function keys as well as the physical screen format. Navigation is the movement between screens. All menu selections should bring up the correct next screen. All screens should return to a location designated somewhere on the screen. If direct navigation from one screen to any other is provided, the syntax for that movement should be consistent and correct. If transactions are to be interruptible, the manner of saving partial transactions and calling them back should be the same for all screens. System level testing should test all of these capabilities. 

3. All processing is within constraints. General constraints can relate to prerequisites, post-requisites, time, structure, control, and inferences (see Chapter 1). Constraints can be internally controlled by the application or can be externally determined with the application simply meeting the constraint. Internally controlled constraints are tested through test cases specifically designed for that purpose. For instance, if response time limits have been stated, the longest possible transaction with the most possible errors or other delays should be designed to test response. If response time for a certain number of users is limited, then the test must have all users doing the most complex of actions to prove the response time constraint is met. Externally controlled constraints are those that the application either meets or does not. If the constraints are not met, then some redesign is probably required. 

4. All modules are compatible and, in event of failures, degrade gracefully. System tests of compatibility prove that all system components are capable of operating together as designed. System components include programs, modules, utilities, hardware, database, network, and other specialized software. 

5. Has sufficient procedures and code to provide disaster, restart, and application error recovery in both the designed and host software (e.g., DB2) 

6. All operations procedures for the system are useful and complete. Operations procedures include start-up, shutdown, normal processing, exception processing, special operator interventions, periodic processing, system specific errors, and the three types of recovery.

In addition, the QA test evaluates the accuracy, consistency, format, and content of application documentation, including technical, user, on-line, and operations documentation. Ideally, the individual performing the QA test does not work on the project team but can deal with them effectively in the adversarial role of QA. Quality assurance in some companies is called the acceptance test and is performed by the user. In other companies, QA is performed within the IS department and precedes the user acceptance test. 

The system test is the final developmental test under the control of the project team and is considered distinct from integration tests. That is, the successful completion of integration testing of successively larger groups of programs eventually leads to a test of the entire system. The system test is conducted by the project team and is analogous to the quality assurance acceptance test which is conducted by the user (or an agent of the user). Sample system test errors are shown in Table 17-8.

Test cases used in both QA and system testing should include as many normal operating conditions as possible. System test cases may include subsets of all previous test cases created for unit and integration tests as well as global test cases for system level requirements. The combined effect of test data used should be to verify all major logic paths (for both normal and exception processing), protection mechanisms, and audit trails. 

QA tests are developed completely from analysis and design documentation. The goal of the test is to verify that the system does what the documentation describes and that all documents, screens, and processing are consistent. Therefore, QA tests go beyond system testing by specifically evaluating application information consistency across environments in addition to testing functional software accuracy. QA tests find a broader range of errors than system tests; a sampling of QA errors is in Table 17-9.

System testing affords the first opportunity to observe the system's hardware components operating as they would in a production mode. This enables the project's test coordinator to verify that response time and performance requirements are satisfied.

Since system testing is used to check the entire system, any errors detected and corrected may require retesting of previously tested items. The system test, therefore, is considered successful only when the entire system runs without error for all test types.

TABLE 17-8 Sample System Test Errors

TABLE 17-9 Sample ON Acceptance Test Errors

The test design should include all possible legal and illegal transactions, good and bad data in transactions, and enough volume to measure response time and peak transaction processing performance. As the test proceeds, each person notes on the test script whether an item worked or not. If a tested interaction had unexpected results, the result obtained is marked in the margin and noted for review.

The first step is to list all actions, functions, and transactions to be tested. The information for this list is developed from the analysis document for all required functions in the application and from the design document for security, audit, backup, and interface designs.

The second step is to design transactions to test all actions, functions and transactions. Third, the transactions are developed into a test script for a single user as a general test of system functioning. This test proves that the system works for one user and all transactions. Fourth, the transactions are interleaved across the participating number of users for multiuser testing. In general, the required transactions are only a subset of the total transactions included in the multiuser test. Required transactions test the variations of processing and should be specifically designed to provide for exhaustive transaction coverage. The other transactions can be a mix of simple and complex transactions at the designer's discretion. If wanted, the same transaction with variations to allow multiple use can be used. Fifth, test scripts for each user are then developed. Last, the test is conducted. These steps in developing system/QA tests are summarized as follows:

QA testing is considered complete when the errors do not interfere with application functioning. A complete list of errors to be fixed is developed and given to the project manager and his or her manager to track. In addition, a QA test report is developed to summarize the severity and types of errors found over the testing cycle. Errors that are corrected before the QA test completes are noted as such in the report.

FIGURE 17-23 System Test Errors Found Over Test Shots

ABC Video System Test 

Because ABC's application is completely on-line, the system test is essentially a repeat of the integration test for much of the functional testing. The system test, in addition, evaluates response time, audit, recovery, security, and multiuser processing. The functional tests do not duplicate the integration test exactly, however. The first user might use the integration test scripts. Other user(s) dialogues are designed to try to corrupt processing of the first user data and processes and to do other independent processing. If the total number of independent processing. If the total number of expected system users is six people simultaneously, then the system test should be designed for six simultaneous users.

FIGURE 17-24 ABC Video System Test Overview-Rent/Return Transactions

The first step is to list all actions, functions, and transactions to be tested. For example, Figure 17-24 lists required transactions to test multiple days and all transaction types for each major file and processing activity for Rent/Return. These transactions would be developed into a test script for a single user test of the application. 

FIGURE 17-25 ABC Video System Test Overview-Test Schedule

Then, the transactions are interleaved with other erroneous and legal transactions for the other ABC processes as planned in Figure 17-25. Notice that the required transactions are only a subset of the total transactions included in the test. The required transactions provide for exhaustive transaction coverage. The other transactions in Figure 17-25 are a mix of simple and complex transactions. Test scripts to follow the plan for each user are then developed; this is left as a student exercise.

left as a student exercise. Last, the test is conducted. During each shutdown procedure, the end-of-day reports are generated and reset. The data mayor may not be checked after the first day to verify that they are correct. If errors are suspected, the files and report should be checked to verify accuracy. When one whole day is run through without errors, the entire set of test scripts can be executed. After an entire execution of each test script completes, the test team convenes and reviews all test scripts together to discuss unexpected results. All data from the files are verified for their predicted final contents. That is, unless a problem is suspected, intermediate intraday results are not verified during system testing. Errors that are found are reconciled and fixed as required. The test scripts are run through repeatedly until no errors are generated. Then, the test team should take real transactions for several days of activity and do the same type of test all over again. These transactions should also have file and report contents predicted. This 'live-data' test should be successful if system testing has been successful. If it is not, the errors found should be corrected and transactions to cause the same errors should be added to the system test. After the test is complete, the regression test package is developed for use during application maintenance.

Automated Support Tools for Testing

Many CASE tools now support the automatic generation of test data for the specifications in their design products. There are also hundreds of different types of automated testing support tools that are not related to CASE. Some of the functions of these tools include

• static code analyzers 
• dynamic code analyzers 
• assertion generators and processors 
• test data generators 
• test driver 
• output comparators

In Table 17-10, several examples of CASE testing tools are presented. Many other types of testing support tools are available for use outside of a CASE environment. The most common test support tools are summarized below and sample products are listed in Table 17-11.

A code analyzer can range from simple to complex. In general, static code analyzers evaluate the syntax and executability of code without ever executing the code. They cross-reference all references to a line of code. Analyzers can determine code that is never executed, infinite loops, files that are only read once, data type errors, global, common, or parameter errors, and other common problems. Another output of some static analyzers is a cross-reference of all variables and the lines of code on which they are referenced. They are a useful tool, but they cannot determine the worth or reliability of the code which are desired functions. 

A special type of code analyzer audits code for compliance to standards and structured programming (or other) guidelines. Auditors can be customized by each using company to check their conventions for code structure.

A more complex type of code analyzer is a dynamic tool. Dynamic code analyzers run while the program is executing, hence the term dynamic. They can determine one or more of: coverage, tracing, tuning, timing, resource use, symbolic execution, and assertion checking. Coverage analysis of test data determines how much of the program is exercised by the set of test data. Tracing shows the execution path by statement of code. Some tools list values of key variables identified by the programmer. Languages on PCs usually have dynamic tracers as an execute option. Tuning analyzers identify the parts of the program executed most frequently, thus identifying code for tuning should a timing problem occur. Timing analysis reports CPU time used by a module or program. Resource usage software reports physicall/Os, CPU time, number of database transactions, and other hardware and software utilization. Symbolic executors run with symbolic, rather than real data, to identify the logic paths and computations for programmer-specified levels of coverage.

TABLE 17-10 CASE Test Tools

TABLE 17-11 Other Testing Support Tools

On-line test drivers are of several types. Batch simulators generate transactions in batch-mode processing to simulate multi-user, on-line processing. Transaction simulators copy a test script as entered in single-user mode for later re-execution with other copied test scripts to simulate multi-user interactions.

Output comparators compare two files and identify differences. This makes checking of databases and large files less time-consuming than it would otherwise be.

Summary

Testing is the process of finding errors in an application's code and documentation. Testing is a difficult activity because it is a high-cost, time-consuming activity for which the returns diminish upon success. As such, it is frequently difficult for managers to understand the importance of testing in application development.

The levels of developmental testing include unit, integration, and system. In addition, an agent, who is not a project team member, performs quality assurance testing to validate the documentation and processing for the user. Code tests are on subroutines, modules, and programs to verify that individual code units work as expected. Integration tests verify the logic and processing for suites of modules, verifying intermodular communications. Systems tests verify that the application operates in its intended environment" and meets requirements for constraints, response time, peak processing, backup and recovery, and security, access, and audit controls.

Strategies of testing are either white-box, black-box, top-down, or bottom-up. White-box tests verify that specific logic of the application works as intended. White-box strategies include logic tests, mathematical proof tests, and cleanroom tests. Black-box strategies include equivalence partitioning, boundary value analysis, and error guessing. Heuristics for matching the test level to the strategy were provided.