Security Testing

Question 1

What is security testing?

Answer 1

A systematic method of testing security flaws in a system.

Question 2

What is “a method of testing running applications manually” in security testing?

Answer 2

Penetration Testing or Pentesting

Question 3

What is “a method of testing static applications manually” in security testing?

Answer 3

Manual Code Review

Question 4

What is “an automated system that tests running applications” in security testing?

Answer 4

DAST, or Dynamic Application Security Testing

Question 5

What is “a system used to secure software by reviewing the source code” in security testing?

Answer 5

SAST, or Static Application Security Testing

Question 6

What is the difference between dynamic and static security testing?

Answer 6

Dynamic security testing involves starting the application and feeding it strange input to observe how it responds.

Static security testing, on the other hand, involves parsing the source code of a program, analysing the parsed files and reporting any problems found.

Question 7

What are two forms of static security testing?

Answer 7

SAST / Static Application Security Testing
Manual Code Review

Question 8

What are two forms of dynamic security testing?

Answer 8

DAST / Dynamic Application Security Testing
Penetration Testing

Question 9

What is a compromise we may have to make when building a fully automated security testing tool?

Answer 9

We may have to consider whether to alert the user of every vulnerability we find, even if we’re still in doubt.

Two options are to either always report it, at risk of annoying the user, or to stay silent, and risk missing severe issues.

Question 10

What is the difference between a false negative and a false positive?

Answer 10

A false negative is where the tool reports no error when there is actually one.

A true positive is where the tool reports an error, but there isn’t one.

Question 11

Why may we find false negatives in our tool?

Answer 11

We may lack knowledge of known insecure frameworks, of certain security threats, or simply have an under-approximation of the tool, missing language features or support for complete syntax.

Question 12

Why may we find false positives in our tool?

Answer 12

We may lack knowledge of the security framework, or have an over-approximation of the tool (the tool got too crazy).

There may also be situations where the false positive is in fact a true positive, but the severity of the weakness is mitigated by the attack surface. For example, if the admin is able to breach the central database, it doesn’t matter if they already have access.

Question 13

What do we want to look out for when doing static security analysis?

Answer 13

Local issues, which have no dependency on the dataflow - for example, known insecure functions like Math.random() in Javascript, or secrets stored in the source code.

Dataflow-related issues, such as a cross-site scripting attack, or secrets being initialised earlier in the code.

Question 14

How would you analyse a control flow?

Answer 14

One way to analyse a control flow is to evaluate all different cases a function may have. In a function with four cases, we can read all the different outputs of that method and where it is all going.

Question 15

Why may you choose a dynamic analysis tool over a static analysis tool?

Answer 15

Static analysis tools are very good at detecting generic defects, and sometimes at finding context-specific defects, but they aren’t suitable for projects stored only in the design, as issues like this aren’t visible at source code level.

Question 16

What are some examples of pragmatic static analysis tools?

Answer 16

Type checkers, style checkers or bug finders are good examples of pragmatic static analysis tools that don’t look for exploits per se, but instead look for petty improvements in the code to make it nicer overall.

Question 17

What is Fuzzing?

Answer 17

Fuzzing is a method of testing software by feeding a method invalid or malicious inputs and seeing how they handle it.

TRIVIA: The term comes from a 1988 student project who developed the “Fuzz” algorithm to test Unix commands.

Question 18

What is Cluster Fuzzing?

Answer 18

Cluster fuzzing is a method of fuzzing where we allocate different VMs to different fuzzing instances.

TRIVIA: Google reported using 700 VMs, with 500 allocated to ASan, 100 allocated to LSan and another 100 to UBSan.

Question 19

What are some of the 3 challenges we may face when creating dynamic analysis tools?

HINT: Consider how you would run it

Answer 19

We may struggle to generate the fuzzing inputs, as modern scenarios often require valid input files, and so cannot be easily randomized.

Furthermore, we need to understand and make sure we know what a “valid response” actually looks like, so we can automatically test for it.

Finally, we may struggle to know when we have tested enough, and may stop.

Question 20

What is random fuzzing?

Answer 20

Random fuzzing is a simple but inefficient method of generating fuzzing inputs for a fuzzing algorithm.

It will often get rejected, as a lot of outputs may just be straight up invalid syntax. For example, how can we randomise a HTML file?

Question 21

What is mutation-based fuzzing?

Answer 21

Mutation-based fuzzing is a method used to generate fuzzing inputs for a fuzzing algorithm by mutating existing valid inputs to try to induce a fuzzing error.

We use this where we have little to no knowledge of the input structure (pentesting).

Question 22

What is generation-based fuzzing?

Answer 22

Generation-based fuzzing is a technique that defines new tests based on known models of the input format.

We generate possible inputs with the input specification in mind, and add all possible anomalies to fish for an error. Inputs can be specified by a grammar - called grammar-based fuzzing.

Question 23

“Requires little to no knowledge of the input format/protocol” is an advantage of what fuzzing technique?

Answer 23

Mutation-Based Fuzzing

BECAUSE: We mutate from known valid inputs instead of knowing the specification outright.

Question 24

“We need to have the specification on hand to create the generator” is a disadvantage of what fuzzing technique?

Answer 24

Generation-Based Fuzzing

Question 25

“We may run into problems that require valid checksums” is a disadvantage of what fuzzing technique?

Answer 25

Mutation-Based Fuzzing

BECAUSE: If we take valid inputs that already contain a checksum and mutate them, they may be completely valid inputs, but the checksum failure makes it invalid by default, stunting mutation.

Question 26

Is static analysis better than dynamic analysis for security testing?

Answer 26

Neither is better than the other - in fact, they compliment eachother.

Dynamic tools are able to cover all languages and layers, but have limited application coverage.

On the other hand, static tools have full coverage of the code, but don’t support all programming languages.

Question 27

Why are dynamic tools costly to run?

Answer 27

Generating good test cases is hard, and in many cases becomes undecidable, and deciding if we have reached a vulnerability is also undecidable.

They may also require dedicated testing systems, and needs to parse the output of the system under testing conditions.

Question 28

Why should you never run dynamic security tools on a production system?

Answer 28

Dynamic security tools mutate and generate into thousands of invalid inputs, and apply them straight to the running product. If we’re not careful, we could end up polluting our main production databases with junk data and destroying the general IT landscape!