4452 - Testing Final Flashcards
1
Q
What does statistical testing reflect and what does it measure?
A
reflects: frequency of user inputs
measures: system reliability
2
Q
What does defect testing discover?
A
system defects
3
Q
What are the 4 testing stages?
A
- unit testing
- integration testing
- system testing
- user/acceptance testing
4
Q
What is unit testing?
A
the testing of individual components
5
Q
What is integration testing?
A
the testing of interactions between components
6
Q
What is system testing?
A
testing the complete system prior to delivery
7
Q
What is acceptance testing?
A
testing done by end users to ensure that the system delivers on business and user needs
8
Q
What is black box testing?
A
Its where something is inputted to the system and then the output is compared to the expected output
9
Q
What is white box testing?
A
Its where the input is selected with explicit knowledge of the internal workings of the system
10
Q
What types of testing are included in the white box test suite?
A
- edge (decision) testing
- path testing
- statement testing
- condition testing
11
Q
What is regression testing? What does it ensure?
A
- Regression testing is used to make sure that a defect fix, added functionality, really any changes to the code does not change functionalities or behaviors that should not be affected
- ensures that modified code has not broken the code it modified
12
Q
What are the two broad techniques used during unit testing?
A
functional testing and structural testing
13
Q
What are the two types of structural testing methods? What is the difference between them?
A
Control flow testing, data flow testing
- one is based on control flow graphs the other is based on data flow graphs
14
Q
What does integration testing attempt to construct?
A
the system architecture
15
Q
Integration testing wants to uncover defects associated with what?
A
interfaces
16
Q
When is integration testing complete?
A
- when all modules are fully integrated together
- when all the test cases have been executed
- when all the severe and moderate defects have been found and fixed
17
Q
What does system testing check the system against?
A
the quality requirements
18
Q
A system must be constructed in a ________ for system testing to be performed
A
an emulated environment that is as close to the real production environment as possible
19
Q
Where is acceptance testing performed and who performs it?
A
in the production environment by the end users
20
Q
what is the key difference between testing and debugging?
A
testing confirms the presence of defects, debugging actually locates and fixes them
21
Q
What are the activities in testing?
A
IDBEC:
- identify
- design
- build
- execute
- compare
22
Q
What is “built” during the build activity in testing?
A
test cases
23
Q
What are the activities in debugging?
A
LDRRe
- locate
- design
- repair
- re-test
24
Q
What is a test condition?
A
it is the goal of a test case, it is something you want to verify
25
What does a test case consist of?
test data to be inputted and an expected output
26
What is a test set?
a collection of test cases
27
What do test scripts contain?
data and instructions for testing
28
What is the system that performs the checking of the observed behavior referred to as?
oracle
29
program behavior can be specified in which three ways?
- plain natural language
- state diagrams
- formal mathematical specifications
30
What are automated oracles?
a pre determined input output relationship that is checked for all outputs
31
What is considered completeness for functionality testing and structural testing?
functional; the behavior of all functional partitions are tested
structural; all statement, all branches, all loops
32
What are the three reasons for test stopping?
1. budget exceeded or deadline needs to be met
2. desired coverage reached
3. desired level of failure intensity reached
33
What does boundary value testing do?
targets failures in application at the boundary of the equivalence classes
34
what does equivalence class partitioning do?
selects partitions from WITHIN equivalence classes
35
boundary value testing and equivalence partitioning tests may overlap (T/F)
T
36
What are nodes in a CFG?
statements or groups of statements
37
what are edges in CFG?
the flow of control; there is an edge between i and j if j can be executed immediately after i completes
38
what is statement coverage?
run tests until all statements have been executed
39
what is edge coverage?
run tests until all edges in control flow graph have been executed at least once
40
what is condition coverage?
like edge coverage but with additional attention being paid to conditionals - ensuring that all combinations have been tested
41
what is path coverage?
ensuring that all paths (start node to end node) have been executed multiple times
42
for loop based paths what test cases should you write?
three cases:
1. skip the loop
2. execute it once
3. execute it multiple times
43
what does the driver component do in unit tests?
it takes test case data and passes it into the component being tested and returns the results
44
what do stubs do? what MUST they use?
replace the modules that are subordinate to the subsystem being tested. They MUST use the modules exact interface
45
why is test selection difficult?
because of two aspects of the input domain for the partition; its large size and complexity
46
how does equivalence portioning solve the input domain size issue?
by creating a relatively small number of disjoint sub-domains that allow the system to be tested with N test cases for N sub domains or partitions
47
what are the two objectives for test generation?
completeness and efficiency
48
what is the difference between weak and strong equivalence class testing?
```
weak; uses one variable from each equivalence class in a test case
strong; uses the Cartesian product of the partition subset
```
49
what is the minimum number of test cases that can be in weak equivalence class testing?
the number of classes in the partition with the largest number of subsets
50
in strong equivalence classes the interactions between the represented values and any ______ is tested?
subsets
51
What is the foundation for partitioning equivalence classes?
expected behavior
52
what is the difference between normal and robust testing strategies
normal: equivalence classes of valid values of inputs
robust: equivalence classes of valid and invalid values of inputs
53
what is efficiency for equivalence classes?
identify test cases using only one element from each equivalence classes
54
what ratio is effectiveness evaluated on for equivalence class testing
failures exposed by equivalence class testing vs total failures in the system
55
what is the typically effectiveness of equivalence partitioning?
less than 1
56
what are ways to improve the effectiveness of equivalence class testing?
1. unambiguous and complete specifications of the requirements
2. carefully selected equivalence partitioning
57
what is uni dimensional partitioning for compound data types?
when you consider only 1 variable at a time
58
what is multi-dimensional partitioning for compound data types?
the input domain is the set product of the the variable inputs
59
what are the 4 steps of equivalence classing?
1. identify the input domain
2. equivalence classing: partition the set of values of each variable into disjoint subsets
3. combine the equivalence classing
4. identify the infeasible equivalence classes
60
what is the main source for identifying inputs and outputs for equivalence testing?
the software requirements
61
input variables don't include environmental variables (T/F)
F; they do
62
how can equivalence classes be defined?
using multidimensional partitioning
63
what inputs are used in infeasible equivalence classes?
inputs that cannot be generated during the test
64
what are the three steps to boundary value testing?
1. partition the input domain
2. identify the boundary
3. select test data
65
when does BVA work well?
when program to be tested is a function of several independent variables that are physically bounded quantities and when the selected test data is chosen with no consideration for the function of the program or the semantic meaning of the variable
66
what do decision variables make easier?
to make sure all possible conditions have been accounted for
67
what is logic based modeling?
associating conditions with actions to be performed
68
what are default rules in limited entry decision tables?
indicate actions to be taken when none of the rules apply
69
what must you ensure before using a decision table?
that the rules are complete and consistent
70
how do you create a decision table? (what are the steps)
1. determine the actions, conditions and values, and how many rules you want to specify
2. encode all the possible rules and the actions for each rule
3. reduce where you can
71
what is TAP?
test anything protocol :)
72
what does it mean that drivers and stubs are considered "overhead"?
they must be written but are not considered part of the testing software product
73
what is the mocha testing tool based off of?
the javasript test framework
74
what runs mocha on server and client side
node-js and browser
75
mocha tests run asynchronously (T/F)
True
76
mocha test run in parallel (T/F)
False; they run serially
77
what is a test runner?
the driver; takes test cases and returns results
78
what is a test fixture?
a test environment used by multiple test cases
79
what does TAP allows in terms of communications?
it allows for communication between unit tests and the test harness
80
what is the test harness?
the automated test framework
81
what was tap initially designed for and when?
for the perl interpreter in 1987
82
what are the two inputs of the describe function in mocha?
the name of the test group and the all back function
83
what are the two inputs of the it function
the name of the test case and the call back function that contains the actual test
84
what is the difference between the it and describe function
describe --> test group (higher level)
| it --> test case
85
what are the 4 hooks in mocha?
- beforeEach
- afterEach
- after
- before
86
what runs first beforeEach() or before()?
before
87
what does SDK testing use for white box testing?
user code test coverage reports
88
what kind of testing does api testing use?
black box testing
89
what does api testing use of the server side?
REST
90
what function are stubs written in for EQT?
beforeEach and afterEach
91
what does the spy function solve?
it prevents the program from not calling subsequent arguments after an invalid return
92
what are the three types of functional testing?
- equivalence class
- boundary value
- decision table
93
what three basic metrics can be used to evaluate functional testing methods?
- effort
- efficiency
- effectiveness
94
what functional testing method requires the highest identification effort?
decision tables
95
what functional testing method requires the highest execution effort?
boundary value testing
96
what is the most efficient functional testing technique?
decision tables
97
How do we measure efficiency in functional testing?
you can't really do it
98
what is single fault assumption?
that faults rarely occur as the result of two or more simultaneous faults
99
if single fault assumption is assumed what functional testing method should be considered?
equivalence class testing with robustness
100
if variables are independent and refer to physical domains what functional testing method should be considered?
equivalence class testing
101
if variables are logical quantities what functional testing methods can be considered?
equivalence class testing or decision table
102
if variables are dependent what functional testing methods can be considered?
decision table
103
structural testing provides test definitions relating to what?
- control flow
- data flow
- program language semantics
- coverage criteria
104
what is used to select test data in structural testing?
the internal workings of the system
105
data flow testing uses a test strategy based on what?
selecting paths through the programs control flow graph
106
if a program has compound conditionals what must you ensure?
that all the combinations have been tested
107
what is the fault assumption in path testing?q
that the program may make take a different path then the one expected
108
what are the characteristics of a dd path graph
1. that all nodes are DD paths from the CFG
| 2. edges represent control flow between DD paths
109
what graph enables the precise description of test coverage?
DDpath graphs
110
DD path testing provides metrics and coverage criteria for what?
effectiveness
111
what does predicate coverage aim to reduce all program predicates to?
a true or false (single predicate)
112
for n conditions how many test cases are there typically for MC/DV coverage testing? For regular multiple decision coverage how many should there be?
MC/DC: n + 1
| MDC: 2^n
113
what is the issue with C1 (simple coverage criteria) for DD path testing?
it only covers all edges of the DD path but does not take into consideration dependencies between paths; for example if a variable is declared in one path and then referenced in another, they will be tested independently from each other which isn't what happens in practice
114
what is Cd and what does it solve?
data flow dependency modeling coverage criteria for DD path testing; ensures that all edges as well as all dependent DD path pairs are tested
115
what is C2 and what are the two basic outcomes?
loop testing
1. traverse loop
2. exit loop
116
how can C2 be extended to consider the loop index?
by doing a modified boundary value analysis on the index
117
statement coverage ensures that all loops reach their termination condition (T/F_)
F; only ensures that all statement within the loop were executed
118
it is not acceptable to release code based exclusively on statement coverage (T/F)
T
119
what is the flaw with decision coverage?
it can produce gaps in programming languages that support short circuit logic operators (will skip the evaluation of logic if some other condition is accepted)
120
what is function coverage?
it ensures that all functions in the program have been called
121
what is call coverage?
ensures that all function calls in the program have been executed
122
what is mutliple decision coverage?
ensures that all possible combinations of Boolean sub expressions occurs
123
for languages without short circuit logic operators what is multiple decision coverage equal to ?
path coverage
124
what is MC/DC?
multiple decision coverage but only combinations whose result can effect the decision are evaluated
125
what are the problems with path testing?
1. the number of paths has an exponential relationship to the number of branches
2. some paths cannot be exercised due to the relationship of the data
126
path coverage includes decision coverage (T/F)
T
127
statement coverage includes decision coverage (T/F)
F; the other way around
128
what is a baseline path?
corresponds to a normal execution of the program, from start node to end node, that encompasses as many decisions as possible
129
what is the difference between topological feasible paths and logically feasible paths?
topological is strictly based on the graph, logical incorporates program logic and identifies constraints that eliminate certain paths that are present in the topological model
130
what selection criteria exercises every statement or instruction?
C0; statement coverage
131
what selection criteria exercises each branch and case statement at least once in each direction?
C1; branch coverage
132
what selection criteria exercises each condition in a decision with all possible outcomes
C1p; predicate to outcome
133
what selection criteria exercises every compound predicate outcome?
C2; multiple condition coverage (MCC)
134
what selection criteria exercises every path from entry to exit?
C(infinite), all possible execution paths
135
what is the core difference between functional and structural (path) testing?
functional testing is directly derived from the software quality requirements and specifications and is not based on the source code, whereas structural testing is directly based on the the source code and the resulting graph
136
path testing is based on the logic of the code (T/F)
F; it is based on the graph
137
data flow diagrams explore the events related to what?
the status of variables
138
what are the 3 types of data flow anomalies?
1. variable is defined then defined again
2. variable is referenced but no defined
3. variable is defined but not referenced
139
what is the difference between a defining node and a usage node in CFG?
a defining node is DEFINED at the statement
| a usage node is USED at the statement
140
what is the triple that defines def-use?
x - variable
d - node containing definition of x
u - statement or predicate node that uses x
141
what is a du-path
for variable x, its a path whose first node is a definition for x and last node is a usage node for x
142
what is a dc-path
a du-path where there are no other defining nodes for x
143
what is AduP
a testing coverage where every du-path from every definition of every variable to every use of that definition is exercised under some test
144
what does integration testing attempt to construct?
the system architecture
145
integration testing attempts to uncover errors associated with what?
interfaces and interactions among units
146
what are the three testing areas of integration testing
internal, external, interactions
147
what are the three approaches to base integration testing off of and what is the associated dependency?
- function decomposition (functional dependency)
- call graph (interaction dependency)
- paths (flow dependency)
148
what are the three types of incremental integration functional decomposition?
```
top down (module --> stub)
bottom up (module --> driver)
sandwich testing (mix of above two)
```
149
what are the pros and cons of top down integration testing?
pros
- test cases can be created based on the functionality of the system
- makes sure that the majority of control points are tested early on
cons
- stubs are a waste
- stubs are tricky
- stubs make it so no significant data flow can occur until later level testing
150
what is a test driver?
a fake routine that requires a valid subsystem and passes a test case to it
151
what are the pros and cons of bottom up integration testing?
pros
- low level data processes are tested early on
- no stubs
cons
- you need to build test drivers which constitutes overhead
- testing may be incomplete
152
how does sandwich testing work?
functional groups are tested alternatively between high level and low level modules, once complete its on to the next functional group
153
what is the advantage to sandwich testing?
it has all the positives of top down and bottom down while minimizing the need for stubs or drivers
154
what are the two methods of call based integration testing?
pair wise and neighbourhood
155
what is pair wise integration testing?
uses test sessions that are restricted to a particular pair of units, corresponding to one edge in the call graph equally to one integration test session
156
what is neighborhood integration testing?
where a test session makes up all nodes that are one edge away from the current node, meaning that all predecessor and successor nodes are in the session
157
neighborhood testing has more test sessions than pair wise testing (T/F)
F; pair wise has way more
158
interface based testing is behavioral (T/F)
F; its structural
159
interaction based testing is behavioral (T/F)
T
160
what is a source node?
a node where the program execution begins or resumes (in the case of control transfer from one unit to another)
161
what is a sink node?
a node where the program terminates can be both the end of all execution or when control is transfered to another unit
162
what is a module execution path (MEP)?
a sequence of statements that begins with a source node and ends in a sink node with no intervening sink nodes
163
what is the program mechanism that transfer flow of control from one unit to another?
a message
164
what is an MM-path?
a chain of MEP & messages
165
what are the nodes and edges in an MM-path graph
nodes - MEP
| edges - messages
166
what three things does continuous integration try and automate?
the build, tests, and deployment
167
what are the three key steps to the CI build cycle?
1. developer commits code to the source code repository
2. build server executes the master build script which does the following
- checks source code
- builds an executable version
- runs other jobs like testing
3. team is notified of build results
168
what is the criteria for adding new models to CI build?
- must be unit tested
- must be inspected/reviewed
- must have some release note
169
what are the 4 types of devops processes?
CI - uses tools to orchestrate software changes and tests
CT - uses tools to orchestrate test environment and management
CD - uses tools to manage deliverable
CM - uses tools to run analytics
170
when does regression testing occur?
many times throughout the life of a product
171
what does regression testing ensure?
that changes do not affect functionalities that should not be affected
172
what tests should not be included in regression tests?
any tests that have become obsolete as a result to the changes
173
what is test re validation
identifying obsolete tests in regression testing
174
what are the 3 types of regression testing?
1. test all
2. random
3. selected modification traversing tests
175
what tests are included in the test all approach?
all valid tests from the previous versions and any new tests added to test new functionality
176
what is the problem with the test all approach?
it may not be feasible depending on the size of the program
177
what tests are included in the random approach for regression testing?
random tests from valid previous and new
178
what is selected modified traversing testing for regression testing?
selecting a subset of the original test set such that the execution of the modified code against this subset implies that all functionality was carried over from the previous version, and then also testing any new test cases developed to test new functionality
179
what is a "safe" regression technique?
where no test cases that will traverse modified code are disgarded
180
the sequence of node that a test traverses in a CFG is called what?
its execution trace
181
what are some examples of testable entities?
statements, decisions, branches, def-us path
182
what does the naive algorithm do?
computes all subsets of the test set starting by the smallest subset and stops when one that covers all the entities covered by the test set is found
183
what does the greedy algorithm do?
finds the local optimum choice at each stage and hopes to find the global optimum
184
when do you use test prioritization instead of test minimization?
when you can't discard any test cases due to quality requirements
185
what is the goal of test prioritization
to identify faults as early as possible in the test process
186
which tests get maximum priority?
tests with the most coverage
187
what type of environment is system testing typically performed on?
a constructed by reasonably stable emulated production environment
188
what is perceived performance
user perspective of system performance
189
what are the two approaches to performance testing?
bench marking and profiling
190
comparing performance against some pre determine criteria is called what?
benchmarking
191
comparing performance against measures relative to the system statistics is called what?
profiling
192
what type of performance testing measures the absolute performance of the application?
benchmarking
193
what type of performance testing measures the run-time performance of the application?
profiling
194
what are the typical criteria used in benchmarking?
execution time (cpu usage), response time and memory usage
195
what are some ways to optimize to improve performance?
reduce I/O routines, lazy evaluation saves (computing/loading), hashing/cashing save (reloading resource)
196
what is the difference between contiguous and non contiguous memory testing
contiguous; assigns the consecutive blocks of memory to a process requesting for memory
non contiguous; assigns the separate memory blocks at different locations in memory space in a non consecutive manner for a process requesting memory
197
how do you optimize memory access?
changing non contiguous memory access to contiguous
198
how do you improve data structures?
by using hash based functions and dealing with their limitations in an intelligent way
199
how do you optimize evaluation saves?
- don't do computation when you already know the answer
- hoist computation heavy code outside of a loop
- use dynamic programming
200
what is the classic question that define profiling?
where is the most time being spent
201
what are the two types of profiling (with definitions)
point; events are simple and independent
| context; event consist of simple ordered events
202
what is the difference between insertion and sampling profiling?
- insertion means writing profiling code
| - sampling means monitoring cpu at regular intervals
203
what is hprof?
a tool built into JDK for profiling
204
why should you design your profiler code to be short?
because profiling slows down code a ton
205
where should security mechanisms be positioned?
in subsystems and in the communication links between sub systems
206
what are the 6 steps to the treat modeling?
1. identify assets with security quality requirements
2. create an architecture overview
3. decompose the application in terms of processes
4. identify threats
5. document threat
6. rank threat severity
207
what are security testing mechanisms (VESPP)
vulnerability scanning, ethical hacking, security scanning, penetration testing, password crackingq
208
malware exploits arise from what coding problems?
- buffer overflow
- format string vulnerabilities
- integer vulnerabilities
209
how is a buffer overflow attack done?
injecting attack code into the buffer and overflow the buffer so that it jumps to the attack code
210
how do you defend against an overflow buffer attack
- non overflow buffers testing
- array bound checking
- code pointer integrity
211
what does non overflow buffer testing do?
makes attack code injected into input buffer non-executable
212
what does array bound checking do?
ensure that all read and writes to arrays are within array bounds
213
what does code pointer integrity testing detect?
if a code pointer has been corrupted before it was de-referenced
214
what is XSS?
cross site scripting which allows you to inject code directly into contents of a website
215
what is SQL injection?
when you fuck with queries
216
what is important do not do when building SQL queries?
string concatenation
217
what is XSSI?
an attack that forces the server to perform actions that the client never requested
218
what is path traversal?
a vulnerability that allows attackers to read from folder that they don't have access to
219
what is DDOS attack?
an attack where a server is unable to fulfill ordinary requests
220
what are configuration vulnerabilities?
vulnerabilities that can allow an attacker to take control of a system given that they have knowledge of the default system configuration
221
what is malware?
any software program that works against the interest of the user
222
how to antiviruses identify malware?
by looking for specific signatures in the code
223
what are viruses?
self replicating programs
224
what are worms?
malware that uses the internet to spread
225
what is the difference between a worm and a virus (not internet propogation)
a virus requires user intervention to initiate replication, worms do not
226
what is a trojan horse?
a seemingly innocent program that contains malware
227
what is a backdoor?
malware that creates a covert access channel
228
what is sticky software?
malware that prevents the user from uninstalling it manually
229
what does load testing try and expose?
faults that would not come up under normal conditions (only occur after a long time under full load)
230
what type of testing is used to evaulate software under full load?
stress testing
231
what do availability tests verify?
the systems ability to recover from hardware and software failures without having adverse impacts on other functionality
232
what do degreaded node test verify?
the ability of the system to remain operational after a portion of the system become non-operational
233
what is a p-use?
predicate use, statement is a predicate statement
234
what is a c-use?
computational use, statement is a computation
