Models Flashcards
(10 cards)
Use of Models
- Purpose
- Approval and rationalisation
- Data
- Construct
- Model fit
- Validation
- Governance
- Regulation/legal
- Implementation
- Documentation
- Monitoring
How to develop and validate a model
Purpose
- Stakeholder needs
- Scope and use
- Complexity, materiality
- flexibility/Continued use (changing environment)
Form
- segmentation
- feature testing and selection
- methodology and alternatives
- constraints and ability to implement
Data
- quality and cleaning
- sources and scarcity
- sampling, exclusions
- time horizon
Validation
- build vs out of time performance
- back testing and statistical tests
- independent review
Governance
- documentation
- MRMF and compliance
- technical committee approval
- model life cycle - implementation and monitoring
Types of model risk
Intrinsic risk: risk inherent in model development process. Consider
- data, complexity, maturity, conservatism, performance, stability
Incremental risk: imperfect internal practices and processes. Consider
- quality of model documentation
- robustness of model governance
- quality of model implementation environments
Managing model risk
- understand model universe and inventory
- 3 lines of control (Board level oversight, MRVC, risk control function, model owners)
- model classification mechanisms
- frameworks and reporting requirements
- challenge appropriateness and continued use
Model life cycle
- conceptualise
- prototypes
- development
- validation
- approval process
- deployment
- maintenance
- retirement
Models- roles and responsibilities
Owner: accountable for ensuring model use and appropriateness
Developer: creates models
Validator: independent validation
Approver: person/forum/committee
Implementer: into required platform
Users: use for operations
Benefits of AI/ML Models
- automation of processes
- integration of processes
- scalability and flexibility
- ability to deal with non-linearity and variable interactions/relationships
- recalibration/future proof - automated re-training
- accuracy - sophisticated techniques, continuous incorporation of data and trends
- incorporate unconventional/unstructured data - high volumes
Challenges with AI/ML Models
- explainability, interpretability, lack of transparency
- data quality and sensitivity
- overfitting/spurious accuracy
- fairness risk and ethical considerations (bias and discrimination)
- data privacy and use of data
- model risk, model supervision and recalibration
- IT, cyber and information system risk
- computing power and IT infrastructure integration
- vendor and cloud risk (outsourcing)
- design, implementation and use errors
- regulation/ supervisory compliance and evolution
- resilience risk (risk model evolves inappropriately)
Managing AI risk
- definition, taxonomy and model inventories
- governance procedures & documentation
- compliance with regulation and legislation
- knowledge, skills, expertise and resources
- operating model for AI/ML adoption
- specifying model objective, scope and appropriate use and monitoring these
- validation: incl input data, updated techniques, independent unseen testing, sensitivity testing
- testing accuracy: key metrics, limits, monitoring
- deliberate testing for bias to ensure fairness
- use as challenger models
- awareness of industry best practice
BCBS 239 Risk data aggregation and reporting
- (Overall governance and infra) Governance framework
- (Overall governance and infra) Data architecture and IT infrastructure
- (Data aggregation) Accuracy and integrity
- (Data aggregation) Completeness
- (Data aggregation) Timeliness
- (Data aggregation) Adaptability
- (Reporting) Accuracy
- (Reporting) Comprehensiveness
- (Reporting) Clarity and usefulness
- (Reporting) frequency
- (Reporting) Distribution
- (Supervisory) review
- (Supervisory) remedial actions
- (Supervisory) home/ host cooperation
