Validation
The Two Questions We Ask
We have written and reviewed policies and procedures many times to fully know their necessity and value; which, gives us the understanding to explain and describe validation with two short questions:
Is the math right?
While clearly important, it has, unfortunately, been overemphasized because it’s the easiest to audit, and that’s been very costly.
This first question is the easier one to answer, and it involves checking the calculations, code, algorithms, and other mechanical aspects of a model’s construction, implementation, and processing.
Obviously, ensuring that the math is correct is necessary to have a valid model, but it’s not sufficient, and it leads to the misallocation of resources.
At Spero Risk Associates we have deep business experience and expert quantitative skills. We can fully validate, where other firms can only partially “vet” or “verify”.
Consider what overemphasis on the first question looks like in practice. A validation team checks every formula, reruns every calculation, and confirms that the code produces the output it was designed to produce — and concludes the model is valid. But if the underlying methodology was never the right choice for the portfolio, a perfectly calculated wrong answer is still wrong. The model passes review. The institution relies on it. The examiner accepts it. And the risk is never actually measured or controlled.
This is not a hypothetical. We have seen widely-used methodologies — including transition matrix approaches for calculating probabilities of default — applied to portfolios where the fundamental assumptions of the method cannot be satisfied. The math was right. The method was not. That distinction is the difference between validation and verification, and it is why we ask both questions.
Is it the right math?
The old regulatory definition of a “model” was a process with inputs and outputs, and that view is still extremely useful in many situations, but this isn’t one of them.
This is the more important question, which involves conceptual soundness. It cannot be easily answered by the inexperienced in this topic.
BySR 26-2 uses the ancient definition of a model as a “simplified representation of reality,” and that is the best way of understanding this question. To determine if something (a mathematical construct) is a faithful, reliable representation of reality it helps a whole lot to know something about reality
Fortunately, for our clients, we understand the business, not just the math.
This question is where ALM and IRR validation becomes particularly demanding. Validating, say, a QRM model — confirming that NII, EVE, and rate sensitivity estimates are correct — requires understanding not just the calculations but the assumptions embedded in the balance sheet representation. We have validated these models at institutions with over $100B in assets and in IRRBB frameworks following significant model updates.
Asking whether it is the right math is what it means to elevate the industry standard — most validators confirm the calculation; through our experience and nature, we challenge the assumption.
Vendor Model Challenge
Banks and credit unions frequently use models that were built and are maintained by vendors. SR 26-2 explicitly mentions vendor models inside the model risk framework — they require validation just as internally-built models do. The challenge is that vendors frequently cannot or will not explain what their platforms are computing at a level of detail that permits genuine validation.
Our responses to this is practical: extensive testing with varying inputs or even parallel frameworks. These are the best ways to provide genuine independent challenge of a black-box tool.
For CECL platforms, for example, we can build benchmark models and forecasting platforms in separate environments and compare outputs. Significant divergences indicate either a problem with the vendor model, a problem with our model, or — most commonly — a data preparation issue that neither the institution nor the vendor had identified. We have applied this approach at multiple institutions while building and implementing CECL models and implementations.
We can and do validate models built with AI-assisted methods — including those where the developer used machine learning for feature selection or generative AI to accelerate code production. We know what to look for because we use these tools ourselves. The same discipline we apply to vendor model challenge — build an independent framework, compare outputs, document divergences — applies to AI-generated model components.
For banks under $30B in assets, are you getting what you were promised and what you pay for? Our experience says that it’s likely worthwhile to investigate and find out for yourself.
Common Validation Mistakes
Through this simple framework, we can also explain typical validation mistakes and how we avoid them. Frequently, we see others make mistakes on two extremes:
- False Positives: validators, who lack industry experience, confuse being difficult (about stupid stuff) with effective challenge. Seemingly important findings that have little or no potential for adverse consequences—for anyone—are over-emphasized. It’s a lack of proportionality that’s usually oriented to easily-verified, non-conceptual issues.
- False Negatives: for short-term benefit, expedient validators provide a “light touch” to appease noisy developers or managers and possibly to hide their lack of experience and understanding. This does nothing to benefit the firm.
Both extremes waste a lot of time and a lot of resources, especially in the latter case, when a poorly-implemented or conceptually-unsound model could lead to failure costs associated with use or regulatory findings.
Our Approach to the Three Components of Validation
While the guidance lists three components of validation, executives need to understand that these components are not to be taken as independently important on their own or even mutually exclusive, e.g., initial outcomes analysis provides developmental evidence and, therefore, helps prove conceptual soundness.
We like to think of the three main components of validation like Cerebus – validation encompasses the past, present, and future.
Evaluation of conceptual soundness
Including developmental evidence
Ongoing monitoring
Including process verification and benchmarking
Outcome analysis
Including back-testing
Validations and validation reports provide crucial and, frequently, the only feedback to developers. Proper validations provide constructive criticism leading to more relevant and reliable models that improve decision-making and, therefore, profits.
The principals at Spero Risk Associates have been applying SR 11-7 guidance since its publication in 2011 — including building MRM programs that passed Federal Reserve horizontal review. We integrate all three validation components in our loss-forecasting and benchmark frameworks, and we have applied the same approach to validating models at institutions from $20 billion to $75 billion across credit risk, CECL, ALM/IRR, and vendor platforms.
Sometimes you may have some of the pieces completed and just be looking for an ongoing monitoring package which we are happy to custom tailor to your needs (and possible black box platform model outputs).