CS302: Software Engineering (2021.B.01)

Guidelines for Developing System and Quality Assurance Tests 

The system test is used to demonstrate an application's ability to operate satisfactorily in a simulated production environment using its intended hardware and software configuration. The quality assurance test (QA) is both a system test and a documentation test. Both tests also verify that all of the system's interacting modules do the following:

1. Fulfill the user's functional requirements as contained in the business system design specifications and as translated into design requirements in the design spec and any documents controlling interfaces to other systems. 

2. The human interface works as intended. Screen design, navigation, and work interruptabilityare the test objects for human interface testing. All words on screens should be spelled properly. All screens should share a common format that is presented consistently throughout the application. This format includes the assignment of program function keys as well as the physical screen format. Navigation is the movement between screens. All menu selections should bring up the correct next screen. All screens should return to a location designated somewhere on the screen. If direct navigation from one screen to any other is provided, the syntax for that movement should be consistent and correct. If transactions are to be interruptible, the manner of saving partial transactions and calling them back should be the same for all screens. System level testing should test all of these capabilities. 

3. All processing is within constraints. General constraints can relate to prerequisites, post-requisites, time, structure, control, and inferences (see Chapter 1). Constraints can be internally controlled by the application or can be externally determined with the application simply meeting the constraint. Internally controlled constraints are tested through test cases specifically designed for that purpose. For instance, if response time limits have been stated, the longest possible transaction with the most possible errors or other delays should be designed to test response. If response time for a certain number of users is limited, then the test must have all users doing the most complex of actions to prove the response time constraint is met. Externally controlled constraints are those that the application either meets or does not. If the constraints are not met, then some redesign is probably required. 

4. All modules are compatible and, in event of failures, degrade gracefully. System tests of compatibility prove that all system components are capable of operating together as designed. System components include programs, modules, utilities, hardware, database, network, and other specialized software. 

5. Has sufficient procedures and code to provide disaster, restart, and application error recovery in both the designed and host software (e.g., DB2) 

6. All operations procedures for the system are useful and complete. Operations procedures include start-up, shutdown, normal processing, exception processing, special operator interventions, periodic processing, system specific errors, and the three types of recovery.

In addition, the QA test evaluates the accuracy, consistency, format, and content of application documentation, including technical, user, on-line, and operations documentation. Ideally, the individual performing the QA test does not work on the project team but can deal with them effectively in the adversarial role of QA. Quality assurance in some companies is called the acceptance test and is performed by the user. In other companies, QA is performed within the IS department and precedes the user acceptance test. 

The system test is the final developmental test under the control of the project team and is considered distinct from integration tests. That is, the successful completion of integration testing of successively larger groups of programs eventually leads to a test of the entire system. The system test is conducted by the project team and is analogous to the quality assurance acceptance test which is conducted by the user (or an agent of the user). Sample system test errors are shown in Table 17-8.

Test cases used in both QA and system testing should include as many normal operating conditions as possible. System test cases may include subsets of all previous test cases created for unit and integration tests as well as global test cases for system level requirements. The combined effect of test data used should be to verify all major logic paths (for both normal and exception processing), protection mechanisms, and audit trails. 

QA tests are developed completely from analysis and design documentation. The goal of the test is to verify that the system does what the documentation describes and that all documents, screens, and processing are consistent. Therefore, QA tests go beyond system testing by specifically evaluating application information consistency across environments in addition to testing functional software accuracy. QA tests find a broader range of errors than system tests; a sampling of QA errors is in Table 17-9.

System testing affords the first opportunity to observe the system's hardware components operating as they would in a production mode. This enables the project's test coordinator to verify that response time and performance requirements are satisfied.

Since system testing is used to check the entire system, any errors detected and corrected may require retesting of previously tested items. The system test, therefore, is considered successful only when the entire system runs without error for all test types.

TABLE 17-8 Sample System Test Errors

Functional Application does not perform a function in the functional specification Application does not meet all functional acceptance criteria Human Interface Screen format, spelling, content errors Navigation does not meet user requirements Interruption of transaction processing does not meet user requirements Constraints Prerequisites treated as sequential and should be parallel ... must all be checked by (x) module Prerequisite not checked Response Time/Peak Performance Response time not within requirements for file updates, start-up, shutdown, query, etc. Volume of transactions expected cannot be processed within the specified run-time intervals Batch processing cannot be completed in the time allotted Expected number of peak users cannot be accommodated Restart/Recovery Program-Interrupted printout fails to restart at the point of failure (necessary for check processing and some confidential/financial reporting) Software-Checkpoint/restart routine is not called properly Hardware-Printer cannot be accessed from main terminal Switches incorrectly set System re-IPL called for in procedures cannot be done without impacting other users not of this application Expected hardware configuration has incompatible components

TABLE 17-9 Sample ON Acceptance Test Errors

Documentation Two or more documents inconsistent Document does not accurately reflect system feature Edit/Validate Invalid transaction accepted Valid transaction rejected Screen Navigation, format, content, processing inconsistent with functional specification Data Integrity Multifile, multitransaction, multimatches are incorrect File File create, update, delete, query not present or not working Sequence, data, or other criteria for processing not checked Report specification Navigation, format, content, processing inconsistent with functional Recovery Printer, storage, memory, software, or application recovery not correct Performance Process, response, user, peak, or other performance criteria not met User ProceduresDo not match processing Incomplete, inconsistent, incomprehensible On-line help differs from paper documents Operations Procedures Do not match processing Incomplete, inconsistent, incomprehensible

The test design should include all possible legal and illegal transactions, good and bad data in transactions, and enough volume to measure response time and peak transaction processing performance. As the test proceeds, each person notes on the test script whether an item worked or not. If a tested interaction had unexpected results, the result obtained is marked in the margin and noted for review.

The first step is to list all actions, functions, and transactions to be tested. The information for this list is developed from the analysis document for all required functions in the application and from the design document for security, audit, backup, and interface designs.

The second step is to design transactions to test all actions, functions and transactions. Third, the transactions are developed into a test script for a single user as a general test of system functioning. This test proves that the system works for one user and all transactions. Fourth, the transactions are interleaved across the participating number of users for multiuser testing. In general, the required transactions are only a subset of the total transactions included in the multiuser test. Required transactions test the variations of processing and should be specifically designed to provide for exhaustive transaction coverage. The other transactions can be a mix of simple and complex transactions at the designer's discretion. If wanted, the same transaction with variations to allow multiple use can be used. Fifth, test scripts for each user are then developed. Last, the test is conducted. These steps in developing system/QA tests are summarized as follows:

List all actions, functions, and transactions to be tested. Design transactions to test all actions, functions, and transactions. Develop a single-user test script for above. Interleave the tests across the users participating in the test to fully test multiuser functioning of the application. Develop test scripts for each user. Conduct the test. Review test results and reconcile anomalous findings.

Designing multiuser test scripts is a tedious and lengthy process. Doing multiuser tests is equally time-consuming. Batch test simulator (BTS) software is an on-line test aid available in mainframe environments. BTSs generate data transactions based on designer-specified attribute domain characteristics. Some BTSs can read data dictionaries and can directly generate transactions. The simulation portion of the software executes the interactive programs using the automatically generated transactions and can, in seconds, perform a test that might take people several hours. BTSs are not generally available on PCs or LANs yet, but they should be in the future.

Finally, after the system and QA tests are successful, the minimal set of transactions to test the application are compiled into test scripts for a regression test package. A regression test package is a set of tests that is executed every time a change is made to the application. The purpose of the regression test is to ensure that the changes do not cause the application to regress to a nonfunctional state, that is, that the changes do not introduce errors into the processing.

Deciding when to stop system testing is as subjective as the same decision for other tests. Unlike module and integration tests, system tests might have several peaks in the number of errors found over time (see Figure 17-23). Each peak might represent new modules or subsystems introduced for testing or might demonstrate application regression due to fixes of old errors that cause new errors. Because of this multipeak phenomenon, system testing is the most difficult to decide to end. If a decreasing number of errors have not begun to be found, that is, the curve is still rising, do not stop testing. If all modules have been through the system test at least once, and the curve is moving toward zero, then testing can be stopped if the absolute number of errors is acceptable. Testing should continue with a high number of errors regardless of the slope of the line. What constitutes an acceptable number of errors, however, is decided by the project manager, user, and IS managers; there is no right number.

QA testing is considered complete when the errors do not interfere with application functioning. A complete list of errors to be fixed is developed and given to the project manager and his or her manager to track. In addition, a QA test report is developed to summarize the severity and types of errors found over the testing cycle. Errors that are corrected before the QA test completes are noted as such in the report.

The QA report is useful for several purposes. The report gives feedback to the project manager ~bout the efficacy of the team-testing effort and can Identify weaknesses that need correcting. The reports are useful for management to gain confidence (or lose it) in project managers and testing groups. Projects that reach the QA stage and are then stalled for several months because of errors identify training needs that might not otherwise surface.

FIGURE 17-23 System Test Errors Found Over Test Shots

ABC Video System Test 

Because ABC's application is completely on-line, the system test is essentially a repeat of the integration test for much of the functional testing. The system test, in addition, evaluates response time, audit, recovery, security, and multiuser processing. The functional tests do not duplicate the integration test exactly, however. The first user might use the integration test scripts. Other user(s) dialogues are designed to try to corrupt processing of the first user data and processes and to do other independent processing. If the total number of independent processing. If the total number of expected system users is six people simultaneously, then the system test should be designed for six simultaneous users.

Trans #	Rents	Returns	Late Fees	Payment	Receipt
T111	2	0	-	Exact	Automatic
T112	1	0	-	Over	Automatic
T113	1	1 (Total)	No	Over	Automatic
T121	10	0	-	Over	Automatic
T122	0	2 (From 121)	No	-	No
T141	0	2 (From 121)	2, 4 days	Over	Automatic
T151	4	2 (From 121)	2, 5 days	Over	Automatic
T211	1	1 (Total)	1 day	Exact	Automatic
T212	0	1 (Total)	No	-	No
T213	0	1 (Total)	No	-	Requested
T214	0	1 (Total)	2 days	Under, then exact	Automatic
T221	2	0	-	Under-abort	No
T222-Wait required	0	2 (From T121)	No	-	Requested
T311	0	1 (Total)	10 days	Over	Automatic
T312	1 (with other open rentals)	0	-	Over	Automatic
T313	6 (with other open rentals), error then rent 5	1	0	Exact	Automatic
T411=T311 Err	0	1 (Total)	10 days	Over	Automatic
T412=T312 Err	1 (with other open rentals)	0	-	Over	Automatic
T413=T313 Err	6 (with other open rentals), error then rent 5	1	0	Exact	Automatic
T331	0	2 (From 121)	2.2 days	Exact	Automatic
T322	2	0	-	Under-abort	No
T511	5 (with other open rentals)	2	1 tape, 3 days	Over	Automatic
NOTE: Txyz TransactionID: x = User, x = Day, z = Transaction number

FIGURE 17-24 ABC Video System Test Overview-Rent/Return Transactions

The first step is to list all actions, functions, and transactions to be tested. For example, Figure 17-24 lists required transactions to test multiple days and all transaction types for each major file and processing activity for Rent/Return. These transactions would be developed into a test script for a single user test of the application. 

User 1	User 2	User 3	User 4	User 5	User 6
Start-up–success	Start-up–Err	Start-up–Err	Password–Err	Logon–Err
Logon	Logon	Logon	Logon	Logon	Logon
Rent–T111 Errs + Good data	Rent–T211 Errs + Good data	Cust Add Errs + Good data	Cust Change–Err, Abort	Video add, Errs + Good data	Shutdown–Err
Rent–T112	Rent–T111	Rent–T311	Cust–Change	Copy Change– Errs + Good data	Try to crash system with bad trans
Rent–T113	Rent–T112–Err	Rent–T312	Rent–T411	Rent–T511	Delete Cust–Errs + Good data
Rent–T114	Rent–T213	Rent–T313	Rent–T412	Rent–any trans	Delete Video Errs
Rent–any trans	Rent–any trans	Rent–any trans	Rent–any trans	Rent–any trans	Delete Copy–Errs + Good data
END OF DAY, SHUT-DOWN, and STARTUP
Rent–T121	Rent–T221	Rent–any trans	Rent–any trans	Rent–any trans	Rent–any trans
Rent–T122	Rent–T111	Rent–any trans	Rent–any trans	Rent–any trans	Rent–any trans
END OF DAY, SHUT-DOWN, and STARTUP
Cust Add Errs + Good data	Cust Change–Err, Abort	Rent–T331	Copy Change–Errors + Good data	Try to crash system 2ith bad trans	Rent–any trans
Delete Cust–Errs + Good data	Delete Video Errs	Rent–T332	Cust–Change	Video Add	Rent–any trans
END OF DAY, SHUT-DOWN, and STARTUP END OF MONTH NOTE: Txyz TransactionID: x = User, x = Day, z = Transaction number

FIGURE 17-25 ABC Video System Test Overview-Test Schedule

Then, the transactions are interleaved with other erroneous and legal transactions for the other ABC processes as planned in Figure 17-25. Notice that the required transactions are only a subset of the total transactions included in the test. The required transactions provide for exhaustive transaction coverage. The other transactions in Figure 17-25 are a mix of simple and complex transactions. Test scripts to follow the plan for each user are then developed; this is left as a student exercise.

left as a student exercise. Last, the test is conducted. During each shutdown procedure, the end-of-day reports are generated and reset. The data mayor may not be checked after the first day to verify that they are correct. If errors are suspected, the files and report should be checked to verify accuracy. When one whole day is run through without errors, the entire set of test scripts can be executed. After an entire execution of each test script completes, the test team convenes and reviews all test scripts together to discuss unexpected results. All data from the files are verified for their predicted final contents. That is, unless a problem is suspected, intermediate intraday results are not verified during system testing. Errors that are found are reconciled and fixed as required. The test scripts are run through repeatedly until no errors are generated. Then, the test team should take real transactions for several days of activity and do the same type of test all over again. These transactions should also have file and report contents predicted. This 'live-data' test should be successful if system testing has been successful. If it is not, the errors found should be corrected and transactions to cause the same errors should be added to the system test. After the test is complete, the regression test package is developed for use during application maintenance.