Final Review Flashcards
(168 cards)
1
Q
Entry Criteria For Peformance Testing
A
- Quantitative and measurable performance requirements.
- Reasonably stable system.
- Test environment representative of customer site.
- Tools:
a. Load Generator
b. Resource Monitor
2
Q
Load May Reflect
A
- Different Volumes of Activity
- Different Mixes of Activity
3
Q
Why should load be varied for relevant use cases and response time tracked?
A
It assists in verifying performance requirements.
4
Q
What is identified when resource usage is tracked?
A
The ability to identify potential bottlenecks or sources of performance problems..
5
Q
(True or False): Performance testing involves varying the load on the system and comparing the results against the performance requirements.
A
True
6
Q
Question 3
True or False? Analysis of resource usage helps identify potential sources of performance issues.
A
True
7
Q
What type of testing? Verify the behavior of the system meets its requirements when its resources are saturated and pushed beyond their limits.
A
Stress Testing
8
Q
What does Stress Testing attempt to find?
A
It attempts to find the stress points and ensure the system perfoms as specified.
9
Q
What are the 4 Stress Testing Steps?
A
- Identify Stress Points.
- Develop a strategy to stress the system at points identified in step 1.
- Verify that intended stress is actually generated.
- Observe Behavior.
10
Q
True or False? Stress testing should be scheduled during the last couple of weeks of the project.
A
False. Stress Testing should begin earlier.
11
Q
What does Volume Testing? Achieve
A
Verification that the behavior of the system meets its requirements when the system is subjected to a large volume of acitvity over an extended period of time.
12
Q
What Types of Testing would lead to these types of Errors;
a. Memory Leaks
b. Counter Overflow
c. Resource Depletion.
A
Volume Testing will lead to these types of errors.
13
Q
True or False? Volume testing looks to verify that a system meets its requirements when it is subjected to a large volume of activity over a short amount of time.
A
False. Volume Testing is done over an extended period of time.
14
Q
True or False? Configuration Testing should be done for every potential configuration of a system.
A
False.
15
Q
What are the steps needed for configuration Testing?
A
- Identify the parameters that define a configuration. (These parameters are ones that have an impact on the system’s ability to meet its functional and performance requirements)
- Partition. (Grouping of similar parameters, this helps in reducing number of configurations)
- Identify configuration combinations to test.
16
Q
What are some of the criterias needed when creating configuration combinations?
A
a. Using boundaries (Maximum and Minimum)
b. Risk Based.
c. Design of Experiments Pairwise combinations.
17
Q
True or False? Configuration testing looks to verify that the functional and performance requirements of a system are met for different configurations of the system.
A
True.
18
Q
Define Configuration Testing?
A
Configuration testing looks to verify that the functional and performance requirements of a system are met for different configurations of the system.
19
Q
True or False? Modifying existing software is a lowrisk-activity.
A
False. The modification of software is a high risk-activity.
20
Q
Why are modifications needed?
A
- Error Fixes.
- The addition of new functions.
21
Q
What are some of the reasons Modifications introduce errors?
A
a. Code Ripple Effects
b. Unintended feature interactions.
c. Changes in performance synchronization, resource sharing, etc.
22
Q
What is the purpose of Regression testing?
A
Regression testing is used to ensure that previously developed and tested functions continue to work as specified after software modifications have been made.
23
Q
When should regression testing be done?
A
At a multi-level:
a. Unit Level.
b. Integration Level.
c. System Level
24
Q
What is the overall process for incremental development and testing with regression Testing?
A
a. Implement first set of functions and release a build.
b. Test the build, create regression tests and report problems.
c. Incorporate the fixes into a new build. Implement next set of functions into a new build.
d. Test Fixes, Run regression tests for old functions, test new functions, and finally update regression test set.
25
What is a Full Regression Testing?
Rerun all existing tests in response to a code modification.
Note: This is normally impractical.
26
What is Selective Regression Testing?
Rerun a selected subset of tests based on the modifications made.
Finally, execute a standard confidence test irregardless of the modification.
27
Information of Code Deltas
Requires coverage tool for mapping test cases to code at desired granularity level.
Requires configuration management tool to identify code change deltas.
Strategy suggest rerunning test that traversed changed or deleted code.
28
What is Ripple Effect Analysis?
This requires developers to identify the impact of changes on other requirements or features.
29
Selective Regression Testing Using a Confidence Test Suite. How is this accomplished?
- Selects a subset of tests to execute to verify previous functionality.
- Include Tests Addressing:
a. High Frequency use cases.
b. Critical Functionality.
c. Functional breadth.
30
What is the 'Revalidation Issue'?
Regression tests must be revalidated to ensure they are consistent with the software modification.
This is accomplished by ensuring test inputs and expected outputs are re-examined for correctness.
31
What is a system's verall reliability and availability is
| dependent upon?
It is dependent on the ability of the system to Detect and Recover from a variety of failures.
Examples:
- User
- Hardware,
- Software
- Other Systems
32
(True or False) It is essential to have a list of the errors to recover from specified
in the requirements
True.
33
What is the usual approach for Error and Recovery Testing?
The approach will be error injection.
34
Why is Serviceability Testing Important?
It is important for systems availability.
35
What is the purpose of Serviceabiltiy testing?
Objective is to verify serviceability requirements are
| being met.
36
What does Serviceability Include?
- Problem reporting.
- Isolation
- Correction
- Verification
- Fix Release
37
What does Usability Testing provide?
Verify the behavior of the system meets its requirements when its resources are saturated and pushed beyond their limits.
38
What test will find stress points in a system and ensures the system performs as specified?
Usability Testing
39
The usability requirements of a system is stated in what terms?
- Learnability
- Memorability
- Errors
- Efficiency
- Subjective Satisfaction
40
The type and amount of training required to bring users to a desired level of performance
Learnability
41
The addressing of the ability to retain skills in using a product once it is learned
Memorability
42
The measure of the number of incorrect actions a user makes in trying to accomplish a task
Errors
43
The measure of the speed with which tasks can be performed
Efficiency
44
The user's overall feeling about the product
Subjective Satisfaction
45
Explain the concern with Reliability where it comes to usability testing?
Are we able to get the same results if the test is repeated.
46
Explain the concern with Validity where it comes to usability testing?
Is the test measuring something of relevance.
47
Usability Testing: Formative Evaluation
- Learning which aspects of interface are good and bad
| Answers the question of what can be improved.
48
Usablity Testing: Summative Evaluation
- The assessing of the overall quality of the interface.
| These are like measurement test.
49
What are the concerns of a Test Plan
- Who are the Users?
- What task will they perform?
- What user aids will be available?
- What data is to be collected?
- What criteria will be used to determine success?
50
When should test procedures be tried out? Explain what is accomplished.
Test procedures must be tried out in a pilot study.
The pilot test will evaluate instructions, success criteria, the time to perform tasks, and evaluation criteria.
51
What kind of test users should be used?
Users must be representative. Evaluation should be done of both novice and expert users.
Note: We should be prepared to train users to achieve expert level.
52
What type of Usability Comparison is prefered?
"Within Subject Testing" is preferable.
53
What are the stages of a Test
- Preparation: Ensure Environment is set-up
- Introduction: Welcome, Purpose and Overview.
- Running the Test.
- Debriefing
54
What percentage of an application is composed of the user interface?
Around 50%
55
What does Usability is the degree in which users are able to:
The ability to perform tasks the product is intended to support in intended environment.
The satisfaction by the procedures the must follow and the resultant output.
The protection from consequences of their actions.
56
Define Reliability:
The probability that a system or a capability functions without failure for a specified time or number of natural units in a specified environment
57
Define Availability:
The Probability at any given time that a system or capability of a system functions satisfactorily in a specified environment.
58
What is the Formula for Availability?
(MTTF / (MTTF + MTTR) ) x 100%
MTTF - Mean Time To Failure
MTTR - Mean Time To Repair
59
What does High Reliability and Availability depend on?
Fault Prevention and Fault Tolerance
60
What is an Operational Profile?
An Operational Profile describes how users utilize a product.
61
What does an operational profile consists?
A set of major functions performed by the system and their occurrence probabilities.
62
What is needed for Reliability Prediction?
An Operational Profile is needed for Reliability Prediction.
63
Three Steps needed for Operational Profile Creation:
1. Identify the major functions performed by system.
2. Identify the occurrence rates.
3. Calculate the occurrence probability.
64
True or False: Tests are developed based on operational profiles.
True.
65
What is the Goal of Development Testing?
The goal is to remove faults that have caused failures.
66
What is the Goal of Certification Testing?
The goal is to determine whether a software component or system should be accepted or rejected.
67
True or False. Operational Profiles assist in the development of priorities and assists in performance analysis
True.
68
Which method is not needed for obtaining operational profile data?
a. Marketing
b. Competing Systems
c. Existing Systems
d. Tester Opinions
Tester Opinions.
Operational Profiles are created based upon user data and a tester opinions would not always be representative of users of the system.
69
What does Reliability Growth Model gives insight into?
It will give users to see how reliability changes over time.
70
What question does Software Modeling help to answer?
When should we stop testing?
71
Describe and explain Failure Intensity?
The number of failures per natural or time unit.
72
What is needed for an effective model prediction?
To ensure effective reliability model is chosen it should be used in conjunction with a operational profile.
73
What is statistical Testing?
It is the testing software for reliability rather than fault detection.
This reliability should be specified and the software tested and amended until that specific level is reached.
74
What are some of the Problems that arise with Reliability Testing?
Operational Profile Uncertainty: It is very difficult to test at times if the operational profile being used is truly representative of the real use of the system.
High Costs of Test Data Generation.
Statistical Uncertainty: It may be impossible to generate enough failures to draw statistically valid conclusions
75
Selecting Growth Model Selection:
- Many Different reliability growth models have been proposed.
- No universally applicable growth model
- Reliability should be measured and observed data should be fitted to several models.
- Best-fit model should be used for reliability prediction.
76
True or False: Reliability Growth models utilize historic data to predict system reliability.
False.
Data for system reliability doesn't come from historic data, but from system testing.
77
True or False: Software Correctness and Security are the same.
False.
78
What is the ultimate goal of security testing?
The goal in security testing is to ensure private data is protected from unauthorized users.
79
Security Fundamentals and Testing Areas.
Confidentiality:
- Application
- Data
Integrity
- Data Modification.
- Functions Performed
Availability
- Denial of Service
80
Why is it important to do security testing?
Software may have unintended or unknown functionality that may produce side-effects contributing to security problems.
81
GUI Security Risks
Verify Access Control
- Entry to system
- Access to functions and data
Look for all possible access methods to data
- Cut and Paste
- Screen capture
Evaluate malicious input
- Denial of Service
82
File System Security Risks
Evaluate how data is stored and retrieved
Focus on encryption and data protection.
83
OS Security Risks
Evaluate decrypted data storage in memory
Stress test with low memory
- System under memory stress may leave data unprotected.
84
Security Testing Strategies
- Deny Application access to libraries it needs.
- Try to overflow input buffers by inputting long strings.
- Try special characters as inputs
- Try default or common user names and passwords
- Attempt to fake the source of data.
- Force system to use default values
- Test all routes to perform a task
- Produce each error message and ensure that it does not compromise security
85
True or False? System availability is affected by the ability of the system to detect and recover from failures.
True
86
True or False? Use cases can help with developing quantitative and measurable usability tests
True
87
True or False? Reliability models require system testing to be performed with an operational profile.
True
88
True or False? The goal of certification testing is to remove faults that have caused failures.
False
89
True or False? Test plans should be written for all testing levels.
True.
Unit, Integration, System, Beta, and Acceptance.
90
When should System Test Planning Begin?
System test planning must begin early and used during all phases of software development..
91
What should a test plan reflect?
It should reflect an in-depth understanding of the objective of the system test as well as project constraints.
92
What does the System Test Plan Address:
- System test objectives
- Dependencies and assumptions
- Adopted test strategy
- Specification of the test environment
- Specification of system test entry and exit criteria
- Schedule
- Risk Management
93
True or False. The system plan should give a general understanding of system test activity.
False.
The system plan must Clearly Define the objective.
94
How should dependencies be accounted for?
When creating a system test plan all dependencies and assumptions need to be identified.
95
What is Testing Strategy
Testing strategy defines how testing objectives will be met within project constraints.
96
What is a major criteria that testing strategy is based on?
Risk.
97
What does Testing Strategy determine?
- Techniques to be used for test data generation.
- Test Environment
- Entry and Exit Criteria
- Schedule
98
What does Test Environment Include?
- Platforms to test on. (Windows, Linux, Mac, etc.)
- Simulators
- Testing tools
99
How is an environment selected?
It is selected based on the objective of testing and the testing strategy.
Example:
- Performance testing objective may require load generation tools.
- Configuration testing objective may require additional resources and simulation tools
100
System Test Entry Criteria
It is established based on test strategy to maximize test effectiveness.
Possible Entry Criteria:
- Code under configuration management
- Completion of integration test
- No outstanding high priority problems
- Successful completion of system test readiness assessment
101
What issues may arise if system testing starts to early?
- Inability to run all tests.
- Excessive communication with developers on problem fixes.
- High degree of retest
102
System Test Readiness Assessment. What does it provide.
It identifies functions and code stability needed to effectively begin system test.
It Provides a concrete entry criteria for system test.
Provides a way for development to prioritize their activies as athe start of system test grows near.
Note: This is developed early in the project in conjunction with development.
103
What activities are needed for the creation of System Test Schedule?
- Identify all of the testing tasks to be performed.
- Identify dependencies among the testing tasks.
- Estimate the effort and resources needed to perform each task
- Assign tasks to individuals or groups.
- Map testing tasks to a time line.
104
Why are Test Plan Risk Management important?
It helps to identify the risk that correspond to scenarios that could impact testing schedule and effectiveness.
105
True or False? Testing risks can be identified from previous projects
True.
Testing risks can be identified via checklists or previous project "lessons learned"
106
True or False? Testing Risks do not have to prioritized.
False.
Testing risks must be prioritized and mitigated.
Prioritization is based on likelihood of risk occurring and consequences.
107
What does Risk Mitigation accomplish?
Risk mitigation involves reducing the likelihood of the risk occurring and/or developing contingency plans to minimize impact of the risk should it occur.
108
True or False? Test Planning is only needed for system testing.
False.
109
True or False? System test planning involves both an understanding of the test objectives and the constraints
True.
110
What kind of Chart documents dependencies in a system?
PERT Chart documents dependencies.
PERT - Program Evaluation and Review Technique
111
What is the Critical Path in a PERT Chart?
It is the path with no slack time.
112
What are some of the consequences of overestimating system testing time?
Inefficient Testing and delayed product release.
113
What are some of the consequences of underestimating system testing time?
Lots of overtime, high stress, and probable ineffective testing.
Explanation: Because of underestimating, much more work needs to be accomplished therefore stress increases and the needed work to complete the test in short period of time leads to overtime. Some times testing will be pushed to the backburner leading to inefficient testing.
114
What must be identified in order to create an effective timeline?
- Constraints
- Task Dependencies
- Availability of personnel
- Risks
115
What kind of chart can be used to document a schedule?
Gantt Chart can be used to document a schedule.
Gantt charts identifies duration of tasks along with their starting and ending dates.
Gantt Charts identify parallel tasks.
116
What must be used in order to provide the proper scheduling buffer?
Risk management must be used to guide the team in the amount of contingency time which must be allocated to the schedule.
117
True or False? A PERT chart helps document dependencies in a test schedule.
True
118
What is needed in the development of an estimate?
The variables need to be identified. Then an estimate based upon them can be produced.
119
Approaches for developing estimates based on variables include:
- Using historical data.
- Calculation via a cost estimation model
- Generating a test estimate based on a percentage of development estimate
120
Major Causes of Inaccurate Estimates
- Misunderstanding of requirements
- Overlooked tasks
- Insufficient analysiis when developing estimates due to time pressure.
- Lack of guidelines for estimating
- Lack of historical data.
- Pressure to reduce estimates.
121
Steps for Generic Estimation Process
1. Determine Estimation Responsibilities
2. Review and clarify testing objectives, deliverables, milestones, and constraints.
3. Identify testing tasks.
4. Select appropriate size measure for testing work.
5. Select size estimation method.
6. Estimate and document size.
7. Estimate and document effort.
122
What is Top-Down Estimation?
It is the development of an estimate based on past similar projects.
123
True or False? Top-Down Estimation works well with new types of projects
False.
124
What is Bottom-Up Estimation?
It is the breaking of testing effort into parts.
Each part is estimated separately and the parts are summed to create the estimate.
125
What is the 80/20 Rule?
80% of the effects come from 20% of the causes.
126
What is an example of the size unit for effort?
- Staff hours
| - Staff months
127
True or False? Effort estimation can take part before, concurrently, or after size estimation.
False.
Effort is based upon the size of the project, and can only be determined after the size of the project is estimated.
128
True or False? A work breakdown structure (WBS) helps in bottom up estimation.
True.
129
True or False? It is important to test high risk areas early.
True.
130
True or False? Testing high risk areas are need to be tested partially
False.
High risk areas need to be tested thoroughly.
131
True or False? It is good to test high risk areas early, but they do not need to be tested thoroughly.
False.
Testing Areas that are high risk should be tested early and thoroughly.
132
True or False? Testing should be prioritized based on the risk exposure.
True.
133
What is the best criteria needed for stopping the testing?
The best criteria to conclude testing is when the test objective have been met.
134
Define Defect Density
The number of defects per thousand lines of code.
135
How is defect density used as a Test Exit Criteria
Historical defect density for similar projects can be used to determine if a project is ready for release.
136
What is Defect Pooling?
It is the reporting of defects from 2 groups which can be tracked separately.
This is normally used with operational profiles.
137
What is the Formula to calculate Unique Defects?
(Defects(A) + Defects(B)) - Defects(A+B)
Defects Found in Group A + Defects Found in Group B Subtracted from the common Defects found in Both Groups.
138
What is the Formula to calculate Estimated Total Defects?
(Defects(A) x Defects(B)) / (Defects(A+B)
Defects found in Group A multiplied by Defects Found in Group B, Divided by the Common Defects found in both groups.
139
What is the Formula for Estimated Defects Remaining?
Estimated Total Defects - Unique Defects
140
What is Defect Seeding?
It is the approach of "seeding" defects into the system. It uses the Seeded Defects Planted Divided by Seeded Defects Found then multiplied by normal defects found.
141
What is the formula for Estimated Total Defects?
(Seeded Defects Planted / Seeded Defects Found) x (Normal Defects Found)
142
What are some of the metrics used for Trend Analysis?
- Time to Failure
- Cumulative number of failures.
- Number of failures per unit of time (Failure Intensity)
143
What are the Three Types of Trends seen in Trend Analysis?
- Increasing Reliability
- Decreasing Reliability
- Stable Reliability
144
True or False? Increasing Reliability is normally good news
True.
145
Give an example what may cause a sudden increase in reliability.
- Changing Test Effort.
- Test Burnout.
- Unrecorded Failures
146
True or False? Using historical data to predict defect density may not always lead to accurate predictions.
True.
147
True or False? Defect seeding is typically used with operational profiles.
False
Defect Pooling is typically used with Defect Pooling
148
True or False? Good test cases ideally map back to requirements and have their own identifiers.
True
149
True or False? Severity reflects the customer impact of the error.
True
150
Who does severity impact?
Severity reflects the customer impact of the error
151
Define Priority
Priority reflects project considerations.
152
What is assessed during a system test?
- Product Quality
| - Testing Progress
153
Metrics used to assessed test progress
- Percentage of tests developed
- Percentage of tests executed
- Percentage of requirements tested
154
What does BCWS stand for?
Budgeted Cost of Work Scheduled
155
What does BCWP stand for?
Budgeted Cost of Work Performed
156
What does ACWP stand for?
Actual Cost of Work Performed.
157
Describe the state of this project:
BCWS = 210
BCWP = 220
This project is ahead of schedule.
158
Describe the state of this project:
BCWS = 220
BCWP = 220
This project is on schedule.
159
Describe the state of this project:
BCWS = 190
BCWP = 180
This project is behind schedule.
160
Describe the state of this project:
BCWS = 180
BCWP = 180
ACWP = 190
This project is on schedule and above budget.
161
Describe the state of this project:
BCWS = 170
BCWP = 180
ACWP = 150
This project is ahead of schedule and under budget.
162
True or False? Earned values are used primarily to identify schedule variance.
False.
Earned values are used not only for scheduling but also for budgeting.
163
Define Perceived Process.
What you think you do.
164
Define Official Process
What you are supposed to do.
165
Define Actual Process
What you do.
166
What is the GQM Paradigm
GQM stands for Goal-Question-Metric
- Define the goals of the measurement process.
- Derive the questions that must be answered to meet the goals
- Develop metrics to answer the questions.
167
What are the Process Improvement phases
- Characterize the current process.
- Analyze current Process
- Characterize target process
- Process redesign
- Implement
168
True or False? When doing process improvement in an organization, the focus should be on examining the “official process”.
False.
The official process is what you are supposed to do.
Process begins with examining "the actual process."
