Reviews and Evaluating Testing Methods

Question 1

What constitutes a review?

Answer 1

Qualitative evaluation of correctness based on informal techniques.
ex. Checklist

Question 2

Define Analysis

Answer 2

provides repeatable and detailed evidence of correctness.

Question 3

Do Reviews or Analysis involve running code?

Answer 3

No

Question 4

Peer Review

Answer 4

Evaluation conducted by one or more people with similar competencies and expertise to the author.

Question 5

Why is Peer review useful?

Answer 5

• People most fit to assess expert’s work is other experts.
• Large and diverse group finds more weaknesses.
• Provides external validation (stamp of approval)

Question 6

Limitations of peer review?

Answer 6

• Slower than checking own work.
• Asking people to point out errors in your work is uncomfortable.
• Encourages Opportunism (let reviewers find errors and how to fix) and Perfectionism.

Question 7

Software Review

Answer 7

Evaluation of software elements or project status to identify discrepancies from planned results and to recommend improvement.

Question 8

Name types of Review

Answer 8

-inspection
-team review
-walkthrough
-pair programming
-peer check (desk check)
-ad hoc review
(from formal to least formal)

Question 9

Ad hoc review

Answer 9

Focus is on problem solving (“help me figure out this bug for 5 minutes”)

Question 10

Peer desk check/passaround

Answer 10

desk check is one person, passaround many. Author only sees output.

(“check out this pull request please?”)

Question 11

Pair programming

Answer 11

Observer reviews each line of code as it is written.

improves quality, more a development strategy.

Question 12

Walkthrough

Answer 12

Formal meeting where developers will attend and a demo of the code is shown.

Goal is to evaluate software or educate an audience about a piece of software. Author guides participants through code and explains how it works.

Question 13

Inspection

Answer 13

Systematically identify defects with program. Reader presents software to team and outcome is (accept, accept with re-work, re-inspect).

• formal, systematic and rigorous review.
• Carried out by inspectors trained to find common defects using checklists and analysis.
• May be used for conformance to regulations or certifications.
• Can be used to collect metrics on defect rates.

Question 14

Checklist for code

Answer 14

Reliability: fault tolerant? effective exception handling?

Efficiency: how much memory consumed? Optimized algorithms?

Reusability: Can components be used in other places?

Scalability: Can system grow?

Question 15

Why independence in review?

Answer 15

Political view? commercial incentives
Psychological view? reviewing own work is difficult.
Technical view? independence builds redundancy
Systems view? review creates feedback loop.

Question 16

Types of independence

Answer 16

Financial, organizational, task independence, knowledge independence (a, b make different assumptions.)

Question 17

Trade off of independence

Answer 17

• Financial (expensive),
• system knowledge (you’re an outsider.)
• Relationship (seen as an outsider.)

Question 18

Review in support of design?

Answer 18

not usually good.

big loss developer is more suited to understand failures.

design and verification out of sync.

verfication doesn’t influence design.

Question 19

When to use walkthrough, review and inspection

Answer 19

W: for training.
R: for consensus.
I: improve quality of document.

Question 20

Error seeding

Answer 20

Early example of fault injection.

plant bugs and see how many are caught.

Question 21

Mutation testing.

Answer 21

Assesses ability of your test set to distinguish between original program and minor small changes (mutants). More caught mutants the better.

Question 22

Common mutations

Answer 22

Operand replacement

Statement modification operators (while-> repeat until)

Expression modification operators.

Question 23

Types of mutants

Answer 23

Do Fewer: Mutant sampling, constrained mutation, mutant clustering.

Do Faster: Schema based mutation, separate compilation approach. Runtime optimization

Do Smarter: Human error vs completeness.

Question 24

Limitations of mutation testing

Answer 24

too expensive. still lack of safety critical systems support.

Question 25

Benefits of mutation Testing

Answer 25

Increasing developments towards automation.