Lecture 7B - Program based Testing

Question 1

What is Mutation testing?

Answer 1

Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.
- The quality of your tests can be gauged from the percentage of mutations killed.

Question 2

Why use mutation testing?

Answer 2

We are essentially asking the question: What’s wrong with line coverage?

- Traditional test coverage (i.e. line, statement, branch, etc.) measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code. It is therefore only able to identify code that is definitely not tested.

Question 3

In general what are some useful techniques for working out if there are gaps in our tests are?

Answer 3

THESE HAVE BEEN STUDIED
- graph coverage techniques -|- may show us that there are portions of code that have never been tested
- logic coverage techniques -|- may show us we haven’t tested different sub-parts of conditions
- program-based mutation testing can be used to “test the tests” -|- if we mutate a program, and it still passes all our tests, something is wrong

Question 4

What is Program-Based Mutation Testing?

Answer 4

Program-Based Mutation testing (also called “mutation analysis”) is a technique for evaluating the quality of a suite of software tests

Question 5

What is an example of Program-Based Mutation Testing?

Answer 5

• Suppose we have some program under a test, and a suite of tests designed to identify defects in it.
  
  • Mutation testing works by modifying the program under test in small ways (e.g. flipping a less-than sign to a greater-than; changing a hard-coded number from 0 to 1).
  • Mutations are usually designed to mimic typically programming errors, such as typographical errors, wrong choice of operator, or off-by-one errors
  • If one of our tests behaves differently for the mutant (vs on the original program), we say that test kills the mutant
    If the test suite doesn’t detect and reject the mutated code, we consider the test suite is defective, and we need to consider more/different test cases.

REFER TO SLIDES FOR EXAMPLE

Question 6

What is the Theory of mutation test?

Answer 6

Mutation testing is a form of syntax-based testing
- Mutations (changes) to the program under test can be defined by a grammar

Question 7

What is a ground string?

Answer 7

A string in the grammar
- (The term “ground” basically means “not having any variables” – in this context, not having any non-terminals)

Question 8

What is a mutation operator?

Answer 8

A rule that specifies syntactic variations of strings generated from a grammar

Question 9

What is a mutant?

Answer 9

The result of one application of a mutation operator on the ground string
- A mutant is a string

Question 10

What do we mean by killing mutants?

Answer 10

If we have some mutant generated from the original ground string, and we look at one or several of our tests, we can ask: do they “kill” the mutant?
- i.e. Does the test(s) give a different result for the mutant, compared to the original?
- If it does, it’s said to “kill” the mutant.

Question 11

Syntax-based coverage criteria – mutant coverage

Answer 11

When creating mutants (i.e. invalid strings), two simple criteria:
- use every operator once or
- use every production once

Question 12

What are test requirements?

Answer 12

TR is test requirements. TR are descriptions of test cases that will be refined into executable tests

Question 13

What is Mutation Production Coverage (MPC)?

Answer 13

For each mutation operator, TR contains several requirements, to create one mutated string m that includes every production that can be mutated by that operator

Question 14

What is Mutation Operator Coverage (MOC)?

Answer 14

For each mutation operator, TR contains exactly one requirement, to create a mutated string m that is derived using the mutation operator

Question 15

What are some practical concerns with mutation - Coverage criteria – practical concerns?

Answer 15

The number of test requirements for mutation is somewhat difficult to quantify because it depends on the syntax of the artifact as well as the mutation operators
- In most situations, mutation yields more test requirements than any other coverage criterion

Question 16

What are some frameworks we can use for mutation testing?

Answer 16

Mutation testing is difficult to apply by hand and automation is more complicated than for most other criteria.
- As a result, mutation is widely considered a high-end coverage criterion, more effective than most but also more expensive.
- One common use of mutation is as a sort of gold standard in experimental studies for comparative evaluation of other test criteria.

Question 17

What are the advantages and disadvantages of using the frameworks?

Answer 17

Advantages:
- Identifies weak/ineffective tests I Very effective at finding problems
- Helps quantify how useful your tests are

Disadvantages:
- Can be time-consuming (large number of mutants to generate, whole test suite needs to be run many times)
- Results require some familiarity with mutation testing to be properly understood

Question 18

Mutation Testing Example

Answer 18

REFER TO SLIDES