CIA.Models

Question 1

Define ‘model’

Answer 1

A simplification of reality using financial, economical, mathematical and statistical concepts
or
A practical representation of relationships among entities using FEMS concepts.

[Financial, Economical, Mathematical, Statistical]

Question 2

What are the 3 elements of a ‘model’ (Hint = SIR)

Answer 2

1. Specification
2. Implementation
3. Run

Question 3

Define ‘model specification’ a model elements

Answer 3

A description of the parts of a model and their interactions
(data, assumptions, methods, entities, events)

Question 4

Define ‘model implementation’ a model elements

Answer 4

The systems that perform the calculations
(computer programs, spreadsheets,…)

Question 5

Define ‘model run’ a model elements

Answer 5

The inputs/outputs of the implementation

Question 6

Define ‘model risk’

Answer 6

The risk that the user will draw inappropriate conclusions due to shortcomings of the model or its use

Question 7

Why is there always risk in using a model?

Answer 7

Because a model is a simplification of reality

Question 8

How can model risk be measured? (2)

Answer 8

1. Severity of model failure
2. Likelihood of model failure

Question 9

Identify 3 considerations in assessing the severity of model failure

Answer 9

1. Financial significance
   (Ex: severity is higher if estimating a major balance sheet item)
2. Importance of model
   (Ex: severity is lower if multiple models are being used)
3. Frequency of use of model
   (Ex: severity is higher if model is used frequently)

Question 10

Identify 4 considerations in assessing the likelihood of model failure

Answer 10

1. Complexity
   (Ex: higher complexity means higher likelihood of misuse of model)
2. Expertise
   (Ex: non-expert users may not understand model limitations)
3. Docs
   (Ex: bad docs means high likelihood of model failure!!!)
4. Testing
   (Ex: inadequate testing means high likelihood of model failure)

Question 11

Does the actuary have more control over the SEVERITY or LIKELIHOOD of model failure? (justify)

Answer 11

More control over likelihood
- CHOOSE a more reliable model (within the actuary’s control)
- TEST the model more thoroughly (within the actuary’s control)

Question 12

Identify the 4 steps an actuary should take before using a new model

Answer 12

1. Review specification
2. Validate implementation
3. Deal with limitations
4. Keep documentation

Question 13

Describe 3 things an actuary does when reviewing a model’s SPECIFICATIONS

Answer 13

Verify DAMs:
- your DATA fits the models requirements
- METHODS are sound
- ASSUMPTIONS are appropriate

Question 14

Describe 5 things an actuary does when validating a model’s IMPLEMENTATION

Answer 14

1. Compare with other tested models
2. Maintain a set of test cases
3. Backtesting (testing with historical data where you already know the answer)
4. Run an entire live file through successive versions of the model (for models with a higher risk-rating)
5. Peer review of testing procedure

Question 15

Describe what an actuary does when dealing with model’s LIMITATIONS

Answer 15

Understand the range of uses for which the model was designed and tested

Question 16

Identify 3 things an actuary should include when DOCUMENTING a model

Answer 16

1. How the model was CHOSEN
2. How it was TESTED
3. What are its limitations

Question 17

Identify an important tool for validating models

Answer 17

A model’s risk rating (riskier models need more thorough validation)

Question 18

Identify 2 things an actuary should evaluate when looking at an existing model that’s being used in a NEW WAY

Answer 18

1. Check that the initial model was properly validated
2. Review limitations in the new application that may not have been relevant in the initial application

Question 19

How should an actuary evaluate a model approved for use BY OTHERS?

Answer 19

Should review & approve the initial validation report

Question 20

How should an actuary evaluate a model OUTSIDE ACTUARY’S EXPERTISE

Answer 20

Make a reasonable attempt at understanding the model’s…
- specifications
- validation (extent to which experts were involved)
- risk-rating
- complexity
- control framework

Question 21

Give an example of a model outside actuary’s expertise

Answer 21

A credit-scoring model

Question 22

What is the purpose (3) of sensitivity testing regarding models

Answer 22

1. Validate model
2. Understand the relationship between inputs/outputs
3. Develop a sense of comfort with the model

Question 23

How can model assumptions be tested in the context of sensitivity-testing? (3 ways)

Answer 23

1. Test assumptions OUTSIDE the expected range
2. Test assumptions singly and then COMBINED
3. Test assumptions with a NONLINEAR relationship between inputs/outputs

Question 24

Identify 3 types of validation that should be done when USING a model

(Hint: DAR)

Answer 24

DAR: validation of Data, Assumptions, Results

• data should be Reliable & Sufficient
• validate non-global assumptions that vary by model run
• results should be “reasonable” relative to input

Question 25

What does it mean for data to be RELIABLE

Answer 25

• Data RECONCILES to audited sources (Ex: balance sheet)
• Data is REASONABLE with respect to prior period data (Ex: data from prior quarter shouldn’t be hugely different)

Question 26

What does it mean for data to be SUFFICIENT

Answer 26

• Data FITS model specification
• Data is available in a CONSISTENT format

Question 27

Identify 2 checks thant can be done to validate the results of a model

Answer 27

1. Inputs/outputs should be CONSISTENT
   (input data should match similar fields in output file)
2. Results should be REASONABLE in both magnitude & direction
   (small change in inputs causes a small change in outputs)

Question 28

Compare uni-dimensional model risk-rating with two-dimensional rating

Answer 28

Uni-dimensional approach:
- rating from 1-20 (20 is high)
- based on financial significance, complexity, expertise of users, docs
Two-dimensional approach:
- assessed separately for severity & likelihood of failure
- final rating is a balance of these