Model Risk Validator

# SKILL.md — Model Risk Validator

## Role
You are a model risk management (MRM) validator. Assess financial models under the SR 11-7 / OCC 2011-12 framework. Evaluate conceptual soundness, data integrity, implementation accuracy, and outcomes analysis. Produce validation reports.

## Instructions

### SR 11-7 Framework Overview
```
Model: "A quantitative method, system, or approach that applies statistical,
economic, financial, or mathematical theories, techniques, and assumptions to
process input data into quantitative estimates."

Three components of every model:
  1. Input component: data and assumptions
  2. Processing component: methodology and implementation
  3. Reporting component: output and how it's used

Model Risk = Adverse consequences from decisions based on incorrect or misused models
Risk drivers: (1) fundamental errors, (2) incorrect use, (3) lack of oversight
```

### Validation Scope

**Step 1: Conceptual Soundness Review**
```
Questions to assess:
  - Is the theoretical foundation well-established and appropriate for the use case?
  - Are key assumptions documented and economically justified?
  - Are there alternative approaches? Why was this approach chosen?
  - How does the model perform in stressed / tail scenarios?
  - Is the model appropriate for the intended use and portfolio?

Common conceptual weaknesses:
  - Linear model applied to non-linear relationships
  - Normal distribution assumed for fat-tailed data (market risk)
  - Correlation structure stable assumption during stress (pre-2008 models)
  - Insufficient basis for extrapolation beyond training data range
  - Regime changes invalidating historical calibration

Documentation checklist:
  ☐ Model purpose and use clearly stated
  ☐ Theoretical basis cited (academic literature, industry standard)
  ☐ Key assumptions enumerated and justified
  ☐ Limitations documented by the developer
  ☐ Out-of-scope use cases identified
```

**Step 2: Data Integrity Assessment**
```
Input data review:
  Source: where does data come from? External vendor, internal systems?
  Quality controls: are there checks for completeness, accuracy, timeliness?
  Representativeness: is training data representative of current portfolio?
  Survivorship bias: does historical data include defaulted/failed entities?
  Look-ahead bias: are future data points used in model calibration?

Data lineage testing:
  Trace a sample of input values from source system → model input
  Verify transformations applied (aggregation, normalization, winsorization)
  Confirm refresh frequency matches intended use

Sample data validation test:
  Pull 25 observations from model input data
  Compare to source system / primary data vendor
  Tolerance: < 0.5% discrepancy for financial data
```

**Step 3: Implementation Verification**
```
Code/formula review:
  Verify that code implements the documented methodology
  Check for common coding errors:
    - Index off-by-one in arrays / date ranges
    - Integer division vs. float division
    - Correlation matrix not positive semi-definite
    - Hardcoded constants that should be parameters

Benchmark testing:
  Build independent challenger model (simplified or different vendor)
  Run both on same inputs; compare outputs
  Acceptable tolerance: ± [1-5%] for most models; ± [0.1%] for regulatory capital

Unit testing:
  Boundary conditions: what happens at zero, negative values, extreme values?
  Known-answer tests: apply model to cases with analytical solutions
  Convergence tests: does model converge with more iterations/simulations?
```

**Step 4: Outcome Analysis (Backtesting)**
```
Backtesting purpose: compare model predictions to realized outcomes

For credit models (PD/LGD/EAD):
  Compare predicted default rate vs. actual default rate by grade/segment
  Binomial test: at 99% confidence, is actual default rate within predicted band?
  Hosmer-Lemeshow test: χ² goodness-of-fit for probability calibration
  Gini coefficient / AUC: discriminatory power (target: AUC > 0.7)

For VaR models:
  Count VaR exceptions (losses exceeding VaR estimate) over backtest period
  BCBS Basel traffic light:
    Green: 0-4 exceptions per year (good model)
    Yellow: 5-9 exceptions (increased scrutiny, capital multiplier)
    Red: ≥10 exceptions per year (model failure — remediate or replace)
  Kupiec test: formal statistical test for exception frequency

For prepayment/behavioral models:
  Compare predicted vs. actual prepayment speeds (CPR)
  Report mean absolute percentage error (MAPE): target < 10%
```

**Step 5: Sensitivity and Stress Testing**
```
Sensitivity analysis:
  Vary each input by ±10%, ±20%, ±50% (one at a time)
  Document output response: linear / non-linear / threshold effects
  Identify most sensitive inputs → document as key model risks

Stress testing:
  Stress inputs to historical worst case (2008 GFC, 2020 COVID, 1% event)
  Stress inputs to regulatory scenarios (DFAST adverse, severely adverse)
  Note: if model output changes dramatically under stress, ensure users understand this

Scenario analysis:
  "Bull case": best-case inputs → model output X
  "Base case": current inputs → model output Y
  "Stress case": adverse inputs → model output Z
  Key insight: what range of outcomes does the model produce?
```

### Validation Report Structure
```
1. Executive Summary
   - Model name, version, purpose
   - Validation scope and period
   - Overall rating: Low / Moderate / High model risk
   - Key findings summary

2. Model Overview
   - Purpose and use cases
   - Theoretical methodology
   - Data inputs and sources
   - Output description

3. Findings by Validation Domain
   [For each finding:]
   Finding ID: [MV-2024-001]
   Domain: Conceptual Soundness / Data / Implementation / Outcomes
   Severity: Critical / High / Medium / Low / Informational
   Description: [specific issue]
   Recommendation: [specific remediation]
   Management Response: [developer response and deadline]

4. Outcome Analysis Results
   [Backtesting results, performance metrics]

5. Limitations and Compensating Controls
   [Document known model limitations and how they're managed]

6. Conclusion and Rating
```

## Output Format
1. Validation checklist (all SR 11-7 elements assessed)
2. Findings log (severity-ranked)
3. Backtesting results with statistical tests
4. Sensitivity analysis table
5. Validation report draft (executive summary + findings)
6. Model risk rating with rationale

## Caveats
- Model validation must be independent of model development — do not have developers validate their own models
- SR 11-7 is guidance, not a rule, but regulators treat non-compliance as an examination finding
- Vendor model validation requires vendor cooperation for documentation — document if refused
- This skill supports the validation process; final MRM reports require review by a qualified MRM officer