CS302: Software Engineering

8a. Describe basic software testing terms, such as test case, traceability matrix, and acceptance criteria

• What is software testing?
• When is software testing performed?
• Where is software testing performed?
• Why is software testing performed?
• How is software testing performed?
• Who performs software testing?
• How would you define the test case, traceability matrix, and acceptance criteria?

What: Software testing is a process of executing software to find errors. It is a type of verification. Other types of verification include reviews and analysis. If the software satisfies its specification, it is said to be verified. If it satisfies the user requirements, it is validated. Errors are either errors of omission or errors of commission. Projects also usually categorize errors by severity and by-product, such as design, documentation, and levels 1 and 2. You will find errors in a trouble reporting or ticket system linked to a version control or configuration management system.

When: Software testing is performed primarily in the testing phases of development, after the implementation phase. However, we perform testing with scaffolding software in other phases, such as the design phase. Each test level is verified against the specification for that level (and by implication of all prior levels). Scaffolding software is test software (not part of the application) that provides an environment enabling partial code execution. Testing stops when all test cases and procedures are complete. 

Where: Software testing can be performed in a development, test, or operational environment. 

Why: Software testing is performed to find errors and verify and validate the software against the functional and non-functional requirements. We acquire error data to measure quality factors, predict future errors, and make management and technical/software decisions. 

How: Software testing is performed using test cases, test procedures, test data, operational data, test software, and test tools. Furthermore, we conduct testing management according to a test plan, process, and procedures using test management software. 

Who: Testing is performed by the developers, by project testers, by internal company testers, by external testers, by customer testers, and by software tools. 

A test case is a data record or transaction to test whether a component or the system satisfies its software requirements. A traceability matrix is an actual table or a graph that links artifacts or elements of the artifacts of each SDLC phase to related ones in the other phases. A test traceability matrix does the same for the elements of the Test Phase; for example, it links a test case to software code and to errors it discovers. Test acceptance criteria are the exit conditions for stopping testing. 

To review, see Test Terminology.

8b. Compare the use of various testing strategies, including black-box, white-box, top-down, and bottom-up

• What are the major software testing strategies?

Four major testing strategies are black-box, white-box, top-down, and bottom-up. A module has three parts: input, process, and output. A strategy for testing software includes a way of selecting the test data. 

If the basis of the specified test data is on the design specifications rather than the logic of the software, we refer to the test strategy as black-box testing. In black-box testing, we execute a module with the test or input data and compare the output to the required or expected output. If they match, the test is considered successful; otherwise, it fails. We call it "black-box testing" because we do not utilize the processing part of the module in selecting the test data; we treat it as hidden.

We call the test strategy white-box testing if we use information on the module's processing to choose the input data. For example, suppose a module has two execution paths controlled by a binary condition. We would select test data to exercise each path in white-box testing. In general, a module could have many execution paths, and analysis would determine the test data to target which pathways to execute.

The software design consists of a collection of modules organized as two structures, static and dynamic, in the form of a module hierarchy. For the static structure, the relationships reflect the parts of a module. For the dynamic structure, the relationships reflect the execution paths of the collection. In bottom-up testing, testing begins when the coding of a module at the bottom of the hierarchy executes to completion. We test the parent module when all parts of a module have been coded and tested. Thus, the order of testing proceeds from the bottom of the hierarchy to the top.

In top-down testing, the testing order proceeds from the top of the hierarchy along execution paths. We use test code, known as "scaffolding", to execute a module when only a part of that module is in development. This test code is in addition to the module code and will be replaced by the module code when it becomes available. 

To review, read Test Strategies.

8c. Identify different types of tests, including functional,non-functional, performance, and user tests

• What are the types of tests?
• What is the purpose of testing?

Each of these is either a functional or non-functional type of test. The type of test depends on the following:

1. the requirements and purpose of the test
2. the phase and phase artifacts tested
3. the customer/client/user
4. the type of requirement

A test may fall into several types; for example, if the purpose of a test is verification of a requirement on performance, then it is also a performance or stress test.

There are several reasons for testing: to verify a phase deliverable, to validate the software system, to improve quality, to determine an attribute (such as complexity), to re-test (called regression testing), to gather data (such as to determine the complexity, to support a decision, or to make a prediction)

Each deliverable phase artifact has a corresponding test: requirements test (validation), design test (verification), code test (verification), integration (verification and validation)

When we perform a test by an external group, it is called an acceptance test. The external group could be the customer, client, user, or independent quality/assessment team.

Requirements include product and process requirements, functional, non-functional, and quality.

To review, see Test Terminology and Software Test.

8d. Design a test plan that includes unit, integration, and system levels of test coverage

• What should we include in a software test plan?
• What are the activities of a software test phase?
• In addition to verification and validation, are there other benefits from the test activities?
• What are several types of tools available to support software testing?
• How are tests verified and validated?
• Do data-oriented, process-oriented, and object-oriented methodologies differ, or are they the same concerning testing?

Testing is a critical, costly, and challenging activity that verifies and validates the code, software design, and software requirements. The requirements, design, and implementation phases have corresponding levels of test activities: system, integration, and unit, respectively. We perform each group of tests according to a test plan. A test plan addresses why testing is done (test goals); who does the testing, what we test and coverage; when testing starts, stops, and is redone (regression); where testing occurs; and how it is done. System and quality assurance testing are similar and share test strategies, cases, and scripts. However, quality assurance testing is a pure validation test. Various software tools are available to support the test activities, and those we select are in the test plan. 

Test activities include:

1. test planning
2. test case development
3. test environment development
4. test execution
5. test logging
6. test result evaluation
7. defect recording and tracking

Test activities are described in the test process (what) and test procedures (how to perform processes). 

The test work products are:

1. test plan
2. test cases
3. test environment
4. test results
5. test log
6. test results
7. test defect resolutions

We can see that testing produces benefits in addition to improving the quality of the software. Testing has information and measures about the application system to help make management and technical decisions during further development and maintenance. For example, traceability information identifies the impact of errors on other parts of the system. It helps determine the scope of related changes, the effort needed for regression testing, and the system's complexity. Moreover, we can use defect measures from tests and technical reviews to determine the probability of remaining errors and to target test resources to the parts of the software with the most significant risk. 

Test tools include test management systems, test scaffolding, test case tracing, and defect management systems. As with all technology, test tools change and evolve due to business, economic, or new technological reasons; old tools are replaced or evolve into new ones. However, types and functions remain the same (or extended). 

Logically, data-oriented, process-oriented, and object-oriented methodologies activities are the same. The artifacts and diagrams used by the respective activities differ, particularly in the emphasis on data, process, or objects. Testing is a recursive activity in that tests need to be verified, validated, and themselves tested. The test code used for the first time should be tested, particularly for critical software, in the same way as the system code. The test software's historical performance, test case coverage reviews, and analysis of test results or errors, either found or not found, are also used to verify and validate the test software.

To review, see Test Plan for ABC Video Order Processing and Software Test.

Unit 8 Vocabulary

This vocabulary list includes terms you must know to complete the final exam successfully.

• acceptance test
• black-box testing
• bottom-up testing
• regression testing
• software testing
• stress test
• test acceptance criteria
• test case
• top-down testing
• traceability matrix
• white-box testing