The Scientific Method Applied to Credit Risk

Credit model development is as much research as development. Don’t let anyone tell you otherwise.

We don’t design a model until we understand the problem(s). Our starting point is always the institution: your portfolio, your concentrations, your data. That allows us to form the right answer before we calculate anything. If the output conflicts with the view, we investigate why — because the conflict is usually where the insights live.

This is the scientific method applied to credit risk: understand the environment, form a hypothesis, then let rigorous analysis confirm or challenge it.

When “Industry Standard” Is Not a Justification

In model development, “industry standard” is what a developer says when someone asks, “Why did you choose this method?” and they cannot justify their methodology. We have seen it since 2011 — methods like the transition matrix approach for calculating probabilities of default across economic conditions, whose underlying assumptions are never met, adopted not because they are optimal but because they are familiar. No one asked whether the method was right for the portfolio. (The devil truly is in the details.) No one asked what problem it was actually solving or how it worked.

That is not a standard worth defending. Our approach starts with a different question: not “What does everyone else do?” but “What does this portfolio actually require?” That is what Thought Before Calculation means in practice and why we trademarked, “Elevating the Industry Standard.” It’s what we do.

We Talk to Both

Before we build a credit model, we meet with the line-of-business portfolio managers and executives who manage the loans. We have been told by a senior executive that this was a unique request. No other outside firm had ever asked for the same meeting. In fact, many internal developers won’t or don’t do it.

LOB managers hold knowledge that never appears in the data: the “Why?” the portfolio is what it is. We want to know about the judgment calls, credit culture shifts, the mergers and their effects. A model built without these conversations is built on an incomplete picture; so, it can’t possibly represent what it should.

Reasoning, Not Just Conclusions

Every model we build, validate, or review is fully explainable—to you, to your board, your auditor, or to your examiner.

Our documentation captures not just what we found, but how we found it, what alternatives we considered, and what constraints shaped our approach of transparent justification and defendability for you. In a regulatory examination, the question is never only whether the model is correct; it is whether the institution understands it. Documented weaknesses are manageable. The unexplained black box is not.

Not the Other Way Around

Roughly two-thirds of any development engagement is data preparation: cleaning, reconciling, back-filling, understanding what the historical record can and cannot support. Once the data is understood, the viable model specifications are sharply reduced and the best design from the constrained optimization problem is nearly determined.

We follow the signal, and that’s how we choose a method, a specification, qualitative adjustment framework.

The alternative is using a predetermined methodology and forcing your data into it. You’ll get a model that runs, but it won’t represent the portfolio, especially when you need it to the most. This is the square-peg-round-hole failure mode of most vendor and large-firm approaches.

No Overhead, No Entertainment, No Layers

We are less expensive than the large firms, and it’s not because we provide a discounted version of the same service or inferior service. Large firms recover their giant overhead somewhere—associate layers, partner-level reviews from a distance, client entertainment charged back (to you). We keep overhead low internally and with our clients. No expensive dinners charged back. No junior staff, doing it for the first time, billed at senior rates.

You pay for is direct senior expertise applied to your problem, and that’s what you get. The fee reflects the actual cost of doing the work well and nothing else.

Poor Data Governance Is the Rule, Not the Exception

Spending two-thirds of development time on data prep is not the natural cost of modeling. It is the cost of data that was never properly governed or maintained. We once completed a model development engagement for a client who was later acquired. When the acquirer requested the institution’s data, the client sent our developmental data set and not their own official data because ours was cleaner and more reliable. That is a remarkable statement and condition.

We have built second-generation CECL models for clients where data prep took as long as the first build because the institution never updated its production systems with the corrected data from the original engagement. The institutional lesson was never operationalized.

This byproduct of development is more valuable than the development! Don’t throw it away!

The answer to “How should we do data governance?” shouldn’t be “Snowflake.” It should be a classics major from Yale who knows how to write a dictionary, even if it’s on paper. That’s your first step to eliminate silos and the “towers of Babel” that they create.

Math Isn’t Enough

Many believe that quantitative skills are sufficient to build strong models. Unfortunately, that includes a lot of developers and their bosses, but it’s not true. In fact, that a very dangerous mentality. Quant skills are indeed necessary for development, but they are not sufficient. “Model development” may remind you of software development, but there are important differences. Model development, especially for loss forecasting, is as much research as development, and the research involves understanding borrowers and their environments and how those environments influence and interact with creditworthiness, i.e., your portfolio.

Datamining doesn’t compensate for talking to LOB and portfolio managers. Developers who only focus on test statistics and have no understanding or appreciation for past events and qualitative information will likely deliver an over-fitted model that’s fine in good times, probably fails in bad times, and likely can’t be explained. That’s silly and expensive and drive model risk

Get Your Money’s Worth from Developers and Vendors

If a validation surfaces problems with a model, that is not an inconvenience. It is information about your internal developers, your vendors, and whether the institution is getting its money’s worth.

There is no equivalent to USDA inspections for models. Validation, done well, allows boards and executives to understand whether the forecasts and estimates they are consuming by assessing represent what they purport to represent.

It’s how an institution holds (or should hold) its model developers and platform vendors to the standard they were hired to meet. That’s what we do when we validate. We don’t check boxes. We tell you the truth about your models.

BB&T (now Truist) · Regions Bank · CCAR Regulatory Environment Andy Spero held senior roles at BB&T (now Truist) and Regions Bank during the period of most intense regulatory scrutiny in the industry. At Regions, as Head of Model Risk Management, the department received the highest rating in the Federal Reserve’s 2014 horizontal review of model risk management across major U.S. financial institutions. The reasons were not complicated: we found every model deficiency ourselves before the examiners did, and we documented everything clearly, thoroughly, and honestly. That is still how we work.

Mellon Bank · Carnegie Mellon · Washington University · University of Minnesota Our approach is grounded in 40 years of quantitative and analytical work — beginning at Mellon Bank building credit risk models, deepened through 15 years of rigorous mathematical research as a doctoral student at Carnegie Mellon and as a professor at Washington University and the University of Minnesota. That academic work was theorem-and-proof mathematical economics: formal models of how incentives, information, and behavior interact under uncertainty. It is the foundation that explains why our solutions are grounded in theory rather than statistical regularities (that may not persist when you need them to).

150+ Years of Combined Experience

Spero Risk is not a one-person shop.

Our team brings more than 150 years of combined quantitative experience across credit risk, model development, validation, ALM/IRR and market risk, model governance, valuation, and data warehouses and governance. Phil brings 40+ years across major financial institutions. Ramesh brings nearly 30 years in quantitative modeling, valuation, and risk at bulge-bracket banks. Haoyu brings 20+ years, including years at the Federal Reserve. Ben, Chrissy, and Dexter each bring eight years of hands-on development, analysis, and project management across all portfolios. When we staff an engagement, you get this team and not a junior proxy for it.