9. Statistical Debugging

Question 1

Problem with random statistical testing

Answer 1

too slow

Question 2

every kth site issue

Answer 2

violates independence

might miss predicates “out of place”

Question 3

periodic hardware time or interrupt issue

Answer 3

might miss rare event

not portable across systems

Question 4

amortized coin tossing

Answer 4

implement as countdown values selected from geometric distribution

Question 5

F(P) is

S(P) is

Answer 5

# of failing runs where P is observed to be true
# of successful runs where P is observed to be true

Question 6

Failure(P) =

Answer 6

F(P)+S(P)

Question 7

Context Statistic

Answer 7

background chance of failure regardless of P’s value

Question 8

F(P observed) =

S(P observed) =

Answer 8

# of failing runs observing P
# of successful runs observing P

Question 9

Context(P) =

Answer 9

F(P observed) + S(P observed)

Question 10

Increase()

Answer 10

Does P being true increase chance of failure over the background rate?

Question 11

Increase(P) =

Answer 11

Failure(P) - Context(P)

Question 12

Finding Bugs First Algorithm

Answer 12

1. Discard predicates having Increase(P)<=0

2. Sort remaining predicates by Increase(P)

Question 13

Revised Algorithm

Answer 13

1. Compute Increase() for all predicates
2. Rank predicates
3. Add top ranked predicate P to the results list
4. Remove P and discard all runs where P is true

Question 14

sub-bug predictor

Answer 14

covers special cases of a more general bug

Question 15

Ranking by Increase(P) problem

Answer 15

high increase() scores but few failing runs

Question 16

Ranking by F(P) problem

Answer 16

many failing runs but low increase() scores

Question 17

Precision =

Answer 17

fraction of retrieved instances that are relevant

Question 18

recall =

Answer 18

fraction of relevant instances that are retrieved

Question 19

retrieved instances

Answer 19

predicates reported as bug predictors

Question 20

relevant instances

Answer 20

predicates that are actually bugs

Question 21

It (is)/(is not) trivial to achieve only high precision or only high recall

Answer 21

is

Question 22

Increase(P) has ___ precision, ___ recall

Answer 22

high precision

low recall

Question 23

F(P) has ___ precision, ___ recall

Answer 23

high recall

low precision

Question 24

Combining Precision and Recall: Harmonic Mean

Answer 24

1/Increase(P) + 1/F(P)

Rewards high scores in both dimensions

Question 25

sorting by the harmonic mean

Answer 25

does good